diff --git a/.agents/skills/rustfs-release-publish/SKILL.md b/.agents/skills/rustfs-release-publish/SKILL.md index f15c72471..063b77a9a 100644 --- a/.agents/skills/rustfs-release-publish/SKILL.md +++ b/.agents/skills/rustfs-release-publish/SKILL.md @@ -25,7 +25,7 @@ Read only the current phase's procedure; retain completed evidence across turns. | 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. | | Confirmation | Complete the gate below and end the turn. | | 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. | -| 7 | [Maintain milestones, installation references, and the website](references/post-release-updates.md) only after complete publication. | +| 7 | [Reconcile Security Advisories, milestones, installation references, and the website](references/post-release-updates.md) only after complete publication. | ## Human Confirmation @@ -39,4 +39,4 @@ Only a new affirmative user reply to that exact report authorizes Phase 6. The o - Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases. - If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting candidate branches or preview tags requires an explicit decision. -Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization. +Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization. diff --git a/.agents/skills/rustfs-release-publish/references/post-release-updates.md b/.agents/skills/rustfs-release-publish/references/post-release-updates.md index 9506f393f..885275af4 100644 --- a/.agents/skills/rustfs-release-publish/references/post-release-updates.md +++ b/.agents/skills/rustfs-release-publish/references/post-release-updates.md @@ -1,7 +1,17 @@ -# Post-release Milestone, Installation, and Website Updates +# Post-release Security Advisory, Milestone, Installation, and Website Updates Read only after the final non-preview tag has passed Phase 6. Tag creation or a green binary build alone is insufficient: the release assets, source archive, container images, Helm package, and applicable latest channels must be available. Preview tags never enter this phase. +## Security Advisories + +Reconcile advisories after every completed final release, including Console-only releases. Read-only preparation may start earlier; do not claim an unpublished artifact is an available fix. + +1. Fetch every page of the repository's live Security Advisories, including triage, draft, published, and closed states. A permission error or partial response blocks reconciliation; it does not mean there are no advisories. Keep unpublished advisory details in private task artifacts, never in a public PR, release note, or issue. +2. For each potentially relevant advisory, map its fix to the exact published tag and verify the regression evidence. Main-branch reachability, a merged private fix, or a green build alone does not prove the shipped version is fixed. Check cherry-picked equivalents and reverted fixes explicitly. For embedded Console or dependency fixes, verify the shipped component version. +3. Determine the first available fixed release and affected version ranges from evidence. Preserve an earlier correct patched version; do not relabel all fixes as the current release. An unchanged server baseline such as a Console-only release requires checking the earlier server release as well. Leave unresolved reports and fixes absent from the tag pending; do not close them merely because a release shipped. +4. Prepare any necessary corrections to package identity, affected ranges, patched versions, fix/release links, and advisory text. Preserve unrelated fields, credit, CVE identifiers, and disclosure coordination. Apply updates only within existing user authorization; read back the result to verify it. Publishing a draft, requesting a CVE, or closing a report is a separate action: require explicit authorization for the concrete advisories and respect any embargo. Release-tag approval alone does not authorize those actions. Prepare the exact proposed changes before asking for missing authorization. +5. Record each relevant advisory's URL, current state, fix evidence, first fixed version, action taken or proposed, and remaining blocker in private task artifacts. Report advisory follow-up separately from artifact publication. A successful release with unresolved advisory work is not complete follow-up; retain that work for the release monitor or a concrete user decision. + ## Version milestones 1. Query all pages of both open and closed milestones in the RustFS release repository, matching titles exactly to version strings without a `v` prefix. A failed or incomplete query is not evidence that a milestone is absent; resolve the lookup before making changes. diff --git a/.agents/skills/security-advisory-lessons/references/advisory-patterns.md b/.agents/skills/security-advisory-lessons/references/advisory-patterns.md index fd8521534..98310295e 100644 --- a/.agents/skills/security-advisory-lessons/references/advisory-patterns.md +++ b/.agents/skills/security-advisory-lessons/references/advisory-patterns.md @@ -45,11 +45,13 @@ Update this file only when an advisory adds or changes a reusable lesson, affect - `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate. - `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets. - `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide. +- `GHSA-9wgh-5r68-49p7`: substituted policy-variable values could recreate `${...}` and loop inside a single resolution pass, never reaching the outer iteration limit. Lesson: bound substitutions within each pass and test cycles formed across otherwise literal resolver values; confirm whether attacker-controlled claims can reach those values before assigning remote impact. ### S3 object actions, copy, multipart, and upload policy validation - `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage. - `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial. +- `GHSA-cg33-pxfp-qcj3` and `GHSA-65wq-gjw8-xr9w`: `HeadObject` and object-tagging paths authorized current-object actions while operating on an explicit historical `versionId`. Lesson: select the corresponding version action for metadata and tag reads and writes too; audit every operation that accepts a version ID. - `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history. - `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies. - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`. @@ -110,6 +112,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect - `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction. - `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers. +- `GHSA-q23w-fp7p-472v`: `PutObjectRetention` policy conditions read `x-amz-object-lock-*` headers while the handler applied mode and date from the XML body. Lesson: authorize the parsed retention that will be stored, or reject mismatched header/body values; compare retention dates as instants so offsets cannot change the applied bound. ### Serde deserialization and input validation @@ -143,13 +146,13 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy. - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases. - Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work. -- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases. +- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, multipart-copy source, `HeadObject`, and tag read/write cases. - Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately. -- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode. +- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, cyclic variable substitutions, plugin mode, and built-in policy mode. - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets. - Logging fixes: assert redacted output for structs and response bodies that may contain credentials. - IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them. - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls. - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content. - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects. -- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. +- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. For `PutObjectRetention`, test conflicting header/body values and offset dates against policy bounds and read-back retention. diff --git a/.config/e2e-full-selection.txt b/.config/e2e-full-selection.txt index 8669a2126..4e5beaea0 100644 --- a/.config/e2e-full-selection.txt +++ b/.config/e2e-full-selection.txt @@ -1,2 +1,2 @@ -sha256-darwin=98dff0961c6eb21d6128bfd2d4a6100a624ddf1fb7db05dba360a5d3d0a310c2 -sha256-linux=890f5925b87e1bf3710c356253f3b2276d901e156df9ccc04ec9315374de6d22 +sha256-darwin=11f953852ca8305e132cb25bfa33a494fe0e4b34847f9e96ce7097d79acccf3d +sha256-linux=d5f1f598f10680a790fb854f5eec884e4c1dd95eb94ab4b33c791a8fab998e9c diff --git a/.config/e2e-nightly-selection.txt b/.config/e2e-nightly-selection.txt index 07dfe2811..6dcd1745d 100644 --- a/.config/e2e-nightly-selection.txt +++ b/.config/e2e-nightly-selection.txt @@ -1,2 +1,2 @@ -sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db -sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1 +sha256-darwin=347a0d4e5e547d5388703062ce66a66167a845b6e57ec591ccb4c732a567c68d +sha256-linux=4f9efa47194279185bb68249e82c788fa147abe944b6c764050e3d47f31e7926 diff --git a/.config/e2e-smoke-selection.txt b/.config/e2e-smoke-selection.txt index eb68a79f1..9430a4d40 100644 --- a/.config/e2e-smoke-selection.txt +++ b/.config/e2e-smoke-selection.txt @@ -1 +1 @@ -sha256=ba8f392b617cf9f2338142233a417e43927b28096635e628233ce8a60cf49d10 +sha256=ce16185f87c6cd30aa0891ebd22be3352aedc10040d48cc5b00652d72f7be3b5 diff --git a/.config/make/tests.mak b/.config/make/tests.mak index 72f39e663..00bb7641f 100644 --- a/.config/make/tests.mak +++ b/.config/make/tests.mak @@ -52,6 +52,9 @@ script-tests: ## Run shell script tests ./scripts/check_embedded_secrets.sh --self-test $(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test $(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py + $(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py + $(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py + $(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py $(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py $(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test $(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test diff --git a/.config/nextest.toml b/.config/nextest.toml index 3881b9036..92674305d 100644 --- a/.config/nextest.toml +++ b/.config/nextest.toml @@ -76,6 +76,13 @@ threads-required = "num-test-threads" filter = 'package(rustfs-ecstore) & (test(/^set_disk::tests::streaming_get_blocks_concurrent_/) | test(=store::object::tests::retired_marker_converges_after_bucket_recreation_on_sixteen_disks))' threads-required = "num-test-threads" +# These real-disk lock probes must finish durable IO within production's 5s +# acquisition budget. Reserve capacity from sibling test processes while +# preserving the probes' internal concurrency, deadlines, and assertions. +[[profile.default.overrides]] +filter = '(package(rustfs-ecstore) & test(=core::pools_test::decommission_lock_order_tests::active_decommission_migration_does_not_invert_capacity_and_delete_object_tag_locks)) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))' +threads-required = "num-test-threads" + [[profile.default.overrides]] filter = 'package(rustfs-heal) & (binary(mrf_partial_write_test) | (binary(mrf_pipeline_test) & test(/^journal_replay_survives_/)) | test(=admin_selector_survives_sigkill_with_original_scope_and_receipts))' threads-required = "num-test-threads" @@ -88,6 +95,14 @@ threads-required = "num-test-threads" filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)' setup = 'ecstore-large-stack' +# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests. +# The admin target-repair fixtures compose the full bucket delete and recreate +# path on a real store; measured with RUST_MIN_STACK on 2026-10-15 they need +# 2016-2040 KiB of the 2 MiB libtest default and overflowed on Linux CI. +[[profile.default.scripts]] +filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))' +setup = 'ecstore-base-stack' + [[profile.default.scripts]] filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))' setup = 'ecstore-base-stack' @@ -327,6 +342,13 @@ threads-required = "num-test-threads" filter = 'package(rustfs-ecstore) & (test(/^set_disk::tests::streaming_get_blocks_concurrent_/) | test(=store::object::tests::retired_marker_converges_after_bucket_recreation_on_sixteen_disks))' threads-required = "num-test-threads" +# These real-disk lock probes must finish durable IO within production's 5s +# acquisition budget. Reserve capacity from sibling test processes while +# preserving the probes' internal concurrency, deadlines, and assertions. +[[profile.ci.overrides]] +filter = '(package(rustfs-ecstore) & test(=core::pools_test::decommission_lock_order_tests::active_decommission_migration_does_not_invert_capacity_and_delete_object_tag_locks)) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))' +threads-required = "num-test-threads" + [[profile.ci.overrides]] filter = 'package(rustfs-heal) & (binary(mrf_partial_write_test) | (binary(mrf_pipeline_test) & test(/^journal_replay_survives_/)) | test(=admin_selector_survives_sigkill_with_original_scope_and_receipts))' threads-required = "num-test-threads" @@ -339,6 +361,11 @@ threads-required = "num-test-threads" filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)' setup = 'ecstore-large-stack' +# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests. +[[profile.ci.scripts]] +filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))' +setup = 'ecstore-base-stack' + [[profile.ci.scripts]] filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))' setup = 'ecstore-base-stack' @@ -595,6 +622,7 @@ default-filter = """ test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat|data_usage)_test::|^fake_s3_target::/) | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/) | test(/^reliant::lifecycle::/) + | test(/^conditional_delete_objects_test::/) | test(/^reliant::tiering::/) | test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/) | test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/) diff --git a/.docker/observability/grafana/dashboards/rustfs.json b/.docker/observability/grafana/dashboards/rustfs.json index 15fbdc34b..1ca9a90a7 100644 --- a/.docker/observability/grafana/dashboards/rustfs.json +++ b/.docker/observability/grafana/dashboards/rustfs.json @@ -1748,7 +1748,7 @@ "uid": "${datasource}" }, "editorMode": "code", - "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count{job=~\"$job\", bucket=~\"$bucket\"})", + "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})", "legendFormat": "{{bucket}}", "range": true, "refId": "A" @@ -1848,7 +1848,7 @@ "uid": "${datasource}" }, "editorMode": "code", - "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes{job=~\"$job\", bucket=~\"$bucket\"})", + "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})", "legendFormat": "{{bucket}}", "range": true, "refId": "A" diff --git a/.docker/observability/prometheus.yml b/.docker/observability/prometheus.yml index 1039e2972..4ac483e6e 100644 --- a/.docker/observability/prometheus.yml +++ b/.docker/observability/prometheus.yml @@ -61,6 +61,7 @@ scrape_configs: otlp: promote_resource_attributes: - service.instance.id + - process.pid - service.name - service.namespace - cloud.availability_zone diff --git a/.docker/observability/tests/bucket-usage.test.yml b/.docker/observability/tests/bucket-usage.test.yml new file mode 100644 index 000000000..d886a7dc9 --- /dev/null +++ b/.docker/observability/tests/bucket-usage.test.yml @@ -0,0 +1,48 @@ +evaluation_interval: 1m +tests: +- name: bucket usage keeps clusters separate and deduplicates reporting nodes + interval: 1m + input_series: + - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"} + values: '4096 1024' + - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"} + values: '4096 1024' + - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"} + values: '65536 65536' + - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"} + values: '0 0' + - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"} + values: '3 1' + - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"} + values: '3 1' + - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"} + values: '70 70' + - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"} + values: '0 0' + promql_expr_test: + - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"}) + eval_time: 1m + exp_samples: + - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}' + value: 1024 + - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}' + value: 0 + - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"}) + eval_time: 1m + exp_samples: + - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}' + value: 1 + - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}' + value: 0 + - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes) + eval_time: 1m + exp_samples: + - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}' + value: 1024 + - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}' + value: 0 + - labels: '{rustfs_cluster_id="b",job="rustfs",bucket="shared"}' + value: 65536 + - expr: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",bucket="unknown"} + eval_time: 1m + exp_samples: [] diff --git a/.github/actions/setup-docker-compose/action.yml b/.github/actions/setup-docker-compose/action.yml new file mode 100644 index 000000000..17fd3f7ab --- /dev/null +++ b/.github/actions/setup-docker-compose/action.yml @@ -0,0 +1,39 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Set up Docker Compose +description: Install the verified Compose CLI used by native Linux x86_64 Connect acceptance + +runs: + using: composite + steps: + - name: Install verified Docker Compose CLI plugin + shell: bash + run: | + set -euo pipefail + [[ $(uname -s) == Linux && $(uname -m) == x86_64 ]] + plugin_dir="${DOCKER_CONFIG:-$HOME/.docker}/cli-plugins" + mkdir -p "$plugin_dir" + compose_download=$(mktemp "${RUNNER_TEMP:?}/connect-compose.XXXXXX") + compose_staged=$(mktemp "$plugin_dir/.docker-compose.XXXXXX") + trap 'rm -f "$compose_download" "$compose_staged"' EXIT + curl --fail --location --retry 3 \ + https://github.com/docker/compose/releases/download/v2.40.3/docker-compose-linux-x86_64 \ + --output "$compose_download" + printf '%s %s\n' \ + dba9d98e1ba5bfe11d88c99b9bd32fc4a0624a30fafe68eea34d61a3e42fd372 \ + "$compose_download" | sha256sum --check --strict + install -m 0755 "$compose_download" "$compose_staged" + mv -f "$compose_staged" "$plugin_dir/docker-compose" + [[ $(docker compose version --short) == 2.40.3 ]] diff --git a/.github/actions/setup/action.yml b/.github/actions/setup/action.yml index eb8f8b377..f9e178569 100644 --- a/.github/actions/setup/action.yml +++ b/.github/actions/setup/action.yml @@ -36,6 +36,16 @@ inputs: description: "Install cross-compilation tools" required: false default: "false" + zig-version: + description: >- + Zig version for cargo-zigbuild builds. Pinned so the `.` + baseline semantics cannot drift with an untested zig release. + required: false + default: "0.17.0" + cargo-zigbuild-version: + description: "cargo-zigbuild version installed for cross builds." + required: false + default: "0.23.4" target: description: "Target architecture to add" required: false @@ -99,10 +109,14 @@ runs: - name: Install Zig if: inputs.install-cross-tools == 'true' uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2 + with: + version: ${{ inputs.zig-version }} - name: Install cargo-zigbuild if: inputs.install-cross-tools == 'true' uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild + with: + tool: cargo-zigbuild@${{ inputs.cargo-zigbuild-version }} - name: Install cargo-nextest if: inputs.install-test-tools == 'true' @@ -126,5 +140,5 @@ runs: # normalises, so cargo fingerprints stay valid. cache-all-crates: false cache-on-failure: true - shared-key: ${{ inputs.cache-shared-key }} + shared-key: ${{ inputs.cache-shared-key }}${{ inputs.install-cross-tools == 'true' && format('-zig-{0}-zigbuild-{1}', inputs.zig-version, inputs.cargo-zigbuild-version) || '' }} save-if: ${{ inputs.cache-save-if }} diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index a647019c5..48d17292f 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -213,12 +213,12 @@ jobs: fi all='{"include":[ - {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""}, - {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""}, - {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"}, - {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"}, - {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"}, - {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""} + {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""}, + {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""}, + {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"}, + {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"}, + {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""}, + {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""} ]}' if [[ "${selected}" == "all" ]]; then @@ -246,7 +246,8 @@ jobs: echo "Selected platforms: ${selected}" build-rustfs: - name: Build RustFS + # Acceptance workflows select artifacts by this stable platform name. + name: Build RustFS (${{ matrix.target_id }}) needs: [ build-check, prepare-platform-matrix ] if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success' runs-on: ${{ matrix.os }} @@ -273,7 +274,7 @@ jobs: with: rust-version: stable target: ${{ matrix.target }} - cache-shared-key: build-${{ matrix.target }} + cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }} # main only. A cache saved on refs/tags/X is scoped to that tag: no # other tag, no main run and no PR can restore it, so every release # cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag, @@ -410,35 +411,53 @@ jobs: shell: bash env: BUILD_TARGET: ${{ matrix.target }} + BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }} run: | # Force rebuild by touching build.rs touch rustfs/build.rs - BINARY_NAMES=(rustfs) - if [[ "${{ matrix.platform }}" == "linux" ]]; then - BINARY_NAMES+=(rustfs-cli) - fi FEATURE_ARGS=() if [[ -n "${{ matrix.features }}" ]]; then FEATURE_ARGS=(--features "${{ matrix.features }}") fi - # Build one binary at a time so release LTO links cannot exhaust a - # self-hosted runner by running concurrently. - for binary in "${BINARY_NAMES[@]}"; do - PROFILE_ARGS=() - if [[ "$binary" == "rustfs" && "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then - # The in-process profiler resolves names from the final executable. - # Retain function symbols without enabling DWARF source information. - PROFILE_ARGS=(--config 'profile.release.strip="none"') - fi - if [[ "${{ matrix.cross }}" == "true" ]]; then - # All cross targets in the matrix are Linux; zigbuild handles them. - cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary" - else - cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary" - fi - done + # gnu targets pin an explicit glibc baseline (`.` for + # cargo-zigbuild) so release binaries link no symbols newer than it + # and run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062. + TARGET_ARG="$BUILD_TARGET" + if [[ -n "${{ matrix.glibc }}" ]]; then + TARGET_ARG="${TARGET_ARG}.${{ matrix.glibc }}" + fi + + PROFILE_ARGS=() + if [[ "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then + # The in-process profiler resolves names from the final executable. + # Retain function symbols without enabling DWARF source information. + PROFILE_ARGS=(--config 'profile.release.strip="none"') + fi + if [[ "${{ matrix.cross }}" == "true" ]]; then + cargo zigbuild --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs + else + cargo build --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs + fi + + # All release targets use same-architecture runners, including GNU + # builds that use Zig only to select the glibc ABI floor. + bin="target/${{ matrix.target }}/release/rustfs" + if [[ -f "${bin}.exe" ]]; then + bin="${bin}.exe" + fi + version_output=$("$bin" --version) + if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \ + ! grep -Fxq "build os : ${BUILD_OS}" <<< "$version_output" || \ + ! grep -Fxq "rust channel : stable-${BUILD_TARGET}" <<< "$version_output"; then + echo "release binary reports an unexpected compile target" >&2 + exit 1 + fi + if [[ -n "${{ matrix.glibc }}" ]]; then + python3 scripts/release/check_glibc.py "${{ matrix.glibc }}" \ + "target/${{ matrix.target }}/release/rustfs" + fi - name: Verify macOS binary has no dynamic liblzma dependency if: matrix.platform == 'macos' @@ -554,9 +573,6 @@ jobs: fi BINARY_FILES=("$BINARY_NAME") - if [[ "${{ matrix.platform }}" == "linux" ]]; then - BINARY_FILES+=("rustfs-cli") - fi if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256) @@ -671,7 +687,7 @@ jobs: echo "🔧 Build type: ${BUILD_TYPE}" echo "📊 Version: ${VERSION}" - - name: Verify packaged Linux binaries + - name: Verify packaged Linux binary if: matrix.platform == 'linux' shell: bash run: | @@ -679,7 +695,7 @@ jobs: package_file="${{ steps.package.outputs.package_file }}" archive_members="$(unzip -Z1 "$package_file" | sort)" - expected_members=$'rustfs\nrustfs-cli' + expected_members='rustfs' target="${{ matrix.target }}" features="${{ matrix.features }}" if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then @@ -691,13 +707,14 @@ jobs: exit 1 fi - # Every Linux target builds natively, so executing the binaries here + # musl targets build natively; the gnu legs are same-arch + # cargo-zigbuild builds pinned to the release glibc baseline. Either + # way the binaries run on their runner, so executing them here # proves each artifact matches its runner architecture. - cli_dir="$(mktemp -d)" - trap 'rm -rf "$cli_dir"' EXIT - unzip -q "$package_file" rustfs rustfs-cli -d "$cli_dir" - "$cli_dir/rustfs" --version - "$cli_dir/rustfs-cli" --help >/dev/null + binary_dir="$(mktemp -d)" + trap 'rm -rf "$binary_dir"' EXIT + unzip -q "$package_file" rustfs -d "$binary_dir" + "$binary_dir/rustfs" --version - name: Verify packaged console if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu' diff --git a/.github/workflows/cache-warm.yml b/.github/workflows/cache-warm.yml index 699ba2990..c01dee315 100644 --- a/.github/workflows/cache-warm.yml +++ b/.github/workflows/cache-warm.yml @@ -99,7 +99,7 @@ env: RUST_BACKTRACE: 1 jobs: - # Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary, + # Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary, # e2e-tests, e2e-full. warm-ci-dev: name: Warm ci-dev diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fe1907458..2e92b226e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -168,13 +168,10 @@ jobs: echo "started_at=$(date --utc --iso-8601=seconds)" } > artifacts/test-and-lint/run-metadata.txt - # Clippy runs before the test pass: lint failures are the most common - # CI-only breakage and should surface in minutes, not after 20+ minutes - # of tests. - # Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS - # experiment, since --all-targets is check-only for workspace members and - # never links the ~100 test binaries the limit exists to throttle. + # Keep full Clippy validation on main, merge groups, and scheduled/manual + # runs. PRs retain the complete default test suite and compile-free checks. - name: Run clippy lints + if: github.event_name != 'pull_request' run: | ./scripts/ci/resource_sampler.sh start clippy trap './scripts/ci/resource_sampler.sh stop' EXIT @@ -283,6 +280,49 @@ jobs: - name: Verify rebalance/decommission migration proofs run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml + # The workspace already compiled these ignored tests. Keep the same Cargo + # package selection to reuse their binaries, then run them serially in + # separate processes because they share fixed ports and global state. + # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303: + # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded + # disk_index 0), not an EC metadata-distribution issue. + # restore_object_usecase_reports_ongoing_conflict_and_completion was + # re-enabled by backlog#1304 (restore accepts serialize on a short CAS + # guard; the copy-back no longer holds the #4877 whole-copy-back lock, + # so the mid-restore ongoing read and fast 409 rejection it asserts are + # the implemented contract). + - name: Run ignored ILM integration tests serially + if: needs.classify-changes.outputs.mode == 'full' + env: + # Match the preceding workspace build and reuse its feature graph. + CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }} + run: | + mkdir -p artifacts/ilm-integration + set +e + NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \ + cargo nextest run --profile default -j1 --run-ignored ignored-only \ + --all --exclude e2e_test \ + -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \ + --status-level all --final-status-level all \ + 2>&1 | tee artifacts/ilm-integration/nextest.log + status=${PIPESTATUS[0]} + { + echo "exit_status=${status}" + echo "finished_at=$(date --utc --iso-8601=seconds)" + echo + echo "Remaining test-related processes:" + pgrep -af 'cargo|nextest|target/.*/deps/' || true + } > artifacts/ilm-integration/diagnostics.txt + exit "${status}" + + - name: Upload ILM test diagnostics + if: always() && needs.classify-changes.outputs.mode == 'full' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }} + path: artifacts/ilm-integration + retention-days: 3 + # This gate builds into a fresh target directory to isolate the E2E root. # Give its cold build a separate budget from workspace tests and migration proofs. offline-enrollment-e2e: @@ -314,83 +354,11 @@ jobs: - name: Check offline enrollment E2E root boundary run: ./scripts/check_offline_enrollment_e2e.sh - # Dedicated serial lane for the ILM / lifecycle integration tests. These tests - # drive the object layer through process-global singletons (the GLOBAL_ENV - # ECStore, the global tier-config manager, background-expiry workers) and bind - # fixed ports (9002 for the scanner suite, 9003 for the app suite), so they are - # marked #[ignore] to keep them out of the default parallel `cargo nextest run - # --all` pass. Run them here explicitly with `--run-ignored ignored-only` and - # `-j1` so no two of them share global state or race for a port at once. - # serial_test's #[serial] does NOT serialize across nextest's process-per-test - # boundary (see .config/nextest.toml); `-j1` is what actually serializes them. - # See rustfs/backlog#1148 (ilm-1) and #1155. - test-ilm-integration-serial: - name: ILM Integration (serial) - if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed') - needs: [ quick-checks, classify-changes ] - runs-on: sm-standard-4 - timeout-minutes: 90 - env: - FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" - steps: - - name: Checkout repository - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7 - with: - persist-credentials: false - - - name: Setup Rust environment - uses: ./.github/actions/setup - with: - rust-version: stable - cache-shared-key: ci-dev - cache-save-if: 'false' - install-build-packaging-tools: 'false' - - # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303: - # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded - # disk_index 0), not an EC metadata-distribution issue. - # restore_object_usecase_reports_ongoing_conflict_and_completion was - # re-enabled by backlog#1304 (restore accepts serialize on a short CAS - # guard; the copy-back no longer holds the #4877 whole-copy-back lock, - # so the mid-restore ongoing read and fast 409 rejection it asserts are - # the implemented contract). - - name: Run ignored ILM integration tests serially - env: - # Match the measured Test and Lint link budget. The default exposed - # all 14 pod CPUs and a cold cache spent the full 80m compiling - # without starting one ILM test (main run 32982910990). - CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }} - run: | - mkdir -p artifacts/ilm-integration - set +e - NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \ - cargo nextest run -j1 --run-ignored ignored-only \ - -p rustfs-scanner -p rustfs \ - -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \ - --status-level all --final-status-level all \ - 2>&1 | tee artifacts/ilm-integration/nextest.log - status=${PIPESTATUS[0]} - { - echo "exit_status=${status}" - echo "finished_at=$(date --utc --iso-8601=seconds)" - echo - echo "Remaining test-related processes:" - pgrep -af 'cargo|nextest|target/.*/deps/' || true - } > artifacts/ilm-integration/diagnostics.txt - exit "${status}" - - - name: Upload ILM test diagnostics - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 - with: - name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }} - path: | - artifacts/ilm-integration - target/nextest/ci/junit.xml - test-and-lint-rio-v2: name: Test and Lint (rio-v2) - if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed') + # Alternate feature configurations run on main, merge groups, and scheduled + # or manual validation; the default workspace remains the PR gate. + if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request' needs: [ quick-checks, classify-changes ] runs-on: sm-standard-4 timeout-minutes: 90 @@ -506,7 +474,7 @@ jobs: test-and-lint-protocols: name: "Test and Lint (${{ matrix.features.name }})" - if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed') + if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request' needs: [ quick-checks, classify-changes ] runs-on: sm-standard-4 timeout-minutes: 90 @@ -1264,7 +1232,6 @@ jobs: - quick-checks - test-and-lint - offline-enrollment-e2e - - test-ilm-integration-serial - test-and-lint-rio-v2 - connect-short-credential-boundary - test-and-lint-protocols @@ -1298,7 +1265,6 @@ jobs: - quick-checks - test-and-lint - offline-enrollment-e2e - - test-ilm-integration-serial - test-and-lint-rio-v2 - test-and-lint-protocols - build-rustfs-debug-binary diff --git a/.github/workflows/connect-diagnostic-schedule-acceptance.yml b/.github/workflows/connect-diagnostic-schedule-acceptance.yml index 4b12335a3..0a69329df 100644 --- a/.github/workflows/connect-diagnostic-schedule-acceptance.yml +++ b/.github/workflows/connect-diagnostic-schedule-acceptance.yml @@ -57,6 +57,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -94,7 +102,7 @@ jobs: ' <<<"$run" >/dev/null jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e ' - [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches + [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches | ($matches | length) == 1 and $matches[0].conclusion == "success" ' <<<"$jobs" >/dev/null artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}") @@ -137,7 +145,7 @@ jobs: assert outer.testzip() is None with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner: entries = inner.infolist() - assert {item.filename for item in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2 + assert {item.filename for item in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3 assert all(regular(item) and 0 < item.file_size <= 1073741824 for item in entries) assert inner.testzip() is None pathlib.Path('binary').mkdir() @@ -162,7 +170,7 @@ jobs: connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check (cd connect-harness && ./tests/connectivity/diagnostic-schedule-evidence.test.sh) - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_DIAGNOSTIC_SCHEDULE_EVIDENCE="$GITHUB_WORKSPACE/diagnostic-schedule-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=diagnostic-schedule diff --git a/.github/workflows/connect-health-service-acceptance.yml b/.github/workflows/connect-health-service-acceptance.yml index b0de48ea9..0423f10cc 100644 --- a/.github/workflows/connect-health-service-acceptance.yml +++ b/.github/workflows/connect-health-service-acceptance.yml @@ -57,6 +57,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -127,7 +135,7 @@ jobs: ' <<<"$run" >/dev/null jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e ' - [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches + [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches | ($matches | length) == 1 and $matches[0].conclusion == "success" ' <<<"$jobs" >/dev/null artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}") @@ -172,7 +180,7 @@ jobs: assert outer.testzip() is None with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner: entries = inner.infolist() - assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4 + assert {entry.filename for entry in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3 assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries) assert inner.testzip() is None pathlib.Path('binary').mkdir() @@ -199,7 +207,7 @@ jobs: connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check (cd connect-harness && ./tests/connectivity/health-service-evidence.test.sh) - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_HEALTH_SERVICE_EVIDENCE="$GITHUB_WORKSPACE/health-service-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=health-service diff --git a/.github/workflows/connect-performance-drive-acceptance.yml b/.github/workflows/connect-performance-drive-acceptance.yml index 1513ddfb3..b1624dc3f 100644 --- a/.github/workflows/connect-performance-drive-acceptance.yml +++ b/.github/workflows/connect-performance-drive-acceptance.yml @@ -73,6 +73,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -107,7 +115,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.[].jobs[] | select(.name == $name)] as $matches @@ -154,7 +162,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_PERFORMANCE_DRIVE_EVIDENCE="$GITHUB_WORKSPACE/performance-drive-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=performance-drive diff --git a/.github/workflows/connect-performance-network-acceptance.yml b/.github/workflows/connect-performance-network-acceptance.yml index cd4b727af..f686f3288 100644 --- a/.github/workflows/connect-performance-network-acceptance.yml +++ b/.github/workflows/connect-performance-network-acceptance.yml @@ -73,6 +73,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -107,7 +115,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.[].jobs[] | select(.name == $name)] as $matches @@ -154,7 +162,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_PERFORMANCE_NETWORK_EVIDENCE="$GITHUB_WORKSPACE/performance-network-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=performance-network diff --git a/.github/workflows/connect-profile-cpu-acceptance.yml b/.github/workflows/connect-profile-cpu-acceptance.yml index c91bd257b..496769dac 100644 --- a/.github/workflows/connect-profile-cpu-acceptance.yml +++ b/.github/workflows/connect-profile-cpu-acceptance.yml @@ -77,6 +77,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -118,7 +126,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.jobs[] | select(.name == $name)] as $matches @@ -173,7 +181,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_CPU_CATALOG="$GITHUB_WORKSPACE/binary/rustfs.cpu-symbol-catalog.json" \ CONNECT_E2E_CPU_CATALOG_SHA256="$CATALOG_SHA256" \ diff --git a/.github/workflows/connect-profile-memory-acceptance.yml b/.github/workflows/connect-profile-memory-acceptance.yml index 6474f0896..77d226e04 100644 --- a/.github/workflows/connect-profile-memory-acceptance.yml +++ b/.github/workflows/connect-profile-memory-acceptance.yml @@ -57,6 +57,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -127,7 +135,7 @@ jobs: ' <<<"$run" >/dev/null jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e ' - [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches + [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches | ($matches | length) == 1 and $matches[0].conclusion == "success" ' <<<"$jobs" >/dev/null artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}") @@ -172,7 +180,7 @@ jobs: assert outer.testzip() is None with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner: entries = inner.infolist() - expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} + expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} assert {entry.filename for entry in entries} == expected and len(entries) == len(expected) assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries) assert inner.testzip() is None @@ -198,7 +206,7 @@ jobs: connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check (cd connect-harness && ./tests/connectivity/profile-memory-evidence.test.sh) - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_PROFILE_MEMORY_EVIDENCE="$GITHUB_WORKSPACE/profile-memory-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=profile-memory diff --git a/.github/workflows/connect-profile-threads-acceptance.yml b/.github/workflows/connect-profile-threads-acceptance.yml index 25479d60d..5cc0c67f1 100644 --- a/.github/workflows/connect-profile-threads-acceptance.yml +++ b/.github/workflows/connect-profile-threads-acceptance.yml @@ -81,6 +81,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -148,7 +156,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.[].jobs[] | select(.name == $name)] as $matches @@ -201,7 +209,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_PROFILE_THREADS_EVIDENCE="$GITHUB_WORKSPACE/profile-threads-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=profile-threads diff --git a/.github/workflows/connect-top-api-acceptance.yml b/.github/workflows/connect-top-api-acceptance.yml index 2c112dffd..c76184149 100644 --- a/.github/workflows/connect-top-api-acceptance.yml +++ b/.github/workflows/connect-top-api-acceptance.yml @@ -73,6 +73,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -107,7 +115,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.[].jobs[] | select(.name == $name)] as $matches @@ -154,7 +162,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_TOP_API_EVIDENCE="$GITHUB_WORKSPACE/top-api-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=top-api diff --git a/.github/workflows/connect-top-disk-acceptance.yml b/.github/workflows/connect-top-disk-acceptance.yml index e6b32ce43..58f5ccada 100644 --- a/.github/workflows/connect-top-disk-acceptance.yml +++ b/.github/workflows/connect-top-disk-acceptance.yml @@ -68,6 +68,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -138,7 +146,7 @@ jobs: ' <<<"$run" >/dev/null jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e ' - [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches + [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches | ($matches | length) == 1 and $matches[0].conclusion == "success" ' <<<"$jobs" >/dev/null artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}") @@ -183,7 +191,7 @@ jobs: assert outer.testzip() is None with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner: entries = inner.infolist() - expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} + expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} assert {entry.filename for entry in entries} == expected and len(entries) == len(expected) assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries) assert inner.testzip() is None @@ -213,7 +221,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_TOP_DISK_EVIDENCE="$GITHUB_WORKSPACE/top-disk-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=top-disk diff --git a/.github/workflows/connect-top-locks-acceptance.yml b/.github/workflows/connect-top-locks-acceptance.yml index edd8e93b0..6d0051a5a 100644 --- a/.github/workflows/connect-top-locks-acceptance.yml +++ b/.github/workflows/connect-top-locks-acceptance.yml @@ -73,6 +73,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -107,7 +115,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.[].jobs[] | select(.name == $name)] as $matches @@ -154,7 +162,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_TOP_LOCKS_EVIDENCE="$GITHUB_WORKSPACE/top-locks-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=top-locks diff --git a/.github/workflows/connect-top-rpc-acceptance.yml b/.github/workflows/connect-top-rpc-acceptance.yml index 5a8cf4da9..7128bc885 100644 --- a/.github/workflows/connect-top-rpc-acceptance.yml +++ b/.github/workflows/connect-top-rpc-acceptance.yml @@ -73,6 +73,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -107,7 +115,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.[].jobs[] | select(.name == $name)] as $matches @@ -154,7 +162,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_TOP_RPC_EVIDENCE="$GITHUB_WORKSPACE/top-rpc-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=top-rpc diff --git a/.github/workflows/e2e-upgrade.yml b/.github/workflows/e2e-upgrade.yml index 21125fe13..4e36da370 100644 --- a/.github/workflows/e2e-upgrade.yml +++ b/.github/workflows/e2e-upgrade.yml @@ -46,9 +46,6 @@ concurrency: env: CARGO_TERM_COLOR: always RUST_BACKTRACE: 1 - UPGRADE_SOURCE_VERSION: 1.0.0-rc.5 - UPGRADE_SOURCE_ASSET: rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip - UPGRADE_SOURCE_SHA256: 3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d jobs: build: @@ -71,14 +68,6 @@ jobs: install-build-packaging-tools: "false" install-test-tools: "false" - - name: Download pinned previous release - run: | - mkdir -p target/debug - archive="target/debug/$UPGRADE_SOURCE_ASSET" - curl --fail --location --retry 3 --output "$archive" \ - "https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}" - echo "$UPGRADE_SOURCE_SHA256 $archive" | sha256sum --check --strict - - name: Build current RustFS binary run: python3 scripts/e2e_binary.py build @@ -89,49 +78,31 @@ jobs: path: | target/debug/rustfs target/debug/rustfs.e2e.json - target/debug/${{ env.UPGRADE_SOURCE_ASSET }} if-no-files-found: error retention-days: 3 compression-level: 0 overwrite: true upgrade: - name: ${{ matrix.name }} + name: Upgrade compatibility (${{ matrix.source.version }}) needs: build + runs-on: sm-standard-2 + timeout-minutes: 60 strategy: fail-fast: false matrix: - include: - # The two `_from_rc2_` tests keep their names: they assert - # release-independent object contracts and pass unchanged against the - # newer pinned source, so renaming them would only churn history and - # the CI required-check names. UPGRADE_SOURCE_VERSION above is the - # single source of truth for which release they actually run against. - - name: Direct upgrade from the previous release - test: direct_upgrade_from_rc2_preserves_object_contracts - artifact: direct-upgrade - - name: Mixed-version rolling upgrade from the previous release - test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts - artifact: mixed-version-upgrade - - name: Bucket configuration survives the upgrade - test: direct_upgrade_from_previous_release_preserves_bucket_configuration - artifact: bucket-config-upgrade - - name: Rollback reads current bucket metadata - test: rollback_to_previous_release_reads_current_bucket_metadata - artifact: bucket-config-rollback - - name: ODM configuration recovery after rc.5 rollback - test: rc5_rollback_requires_restoring_odm_configuration - artifact: odm-config-rollback - - name: Multipart layouts survive the rc.5 upgrade - test: direct_upgrade_from_rc5_preserves_multipart_layouts - artifact: multipart-layout-upgrade - - name: rc.5 multipart replication baseline - test: rc5_baseline_replicates_multipart_layouts - artifact: multipart-layout-baseline - runs-on: sm-standard-2 - timeout-minutes: 60 + source: + - version: 1.0.0-rc.5 + asset: rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip + sha256: 3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d + - version: 1.0.1 + asset: rustfs-linux-x86_64-gnu-v1.0.1.zip + sha256: 9742e183cb0dfb0d48fa909797af572fe47b7f827569036baa4156e0d3958ee5 env: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" + UPGRADE_SOURCE_VERSION: ${{ matrix.source.version }} + UPGRADE_SOURCE_ASSET: ${{ matrix.source.asset }} + UPGRADE_SOURCE_SHA256: ${{ matrix.source.sha256 }} steps: - name: Checkout repository uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7 @@ -142,8 +113,7 @@ jobs: uses: ./.github/actions/setup with: cache-shared-key: e2e-upgrade-tests - # One main-branch consumer warms the shared test-harness dependencies. - cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }} + cache-save-if: ${{ github.ref == 'refs/heads/main' }} install-build-packaging-tools: "false" install-test-tools: "false" @@ -163,6 +133,8 @@ jobs: set -euo pipefail mkdir -p "$SOURCE_DIR" archive="target/debug/$UPGRADE_SOURCE_ASSET" + curl --fail --location --retry 3 --output "$archive" \ + "https://github.com/rustfs/rustfs/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}" echo "$UPGRADE_SOURCE_SHA256 $archive" | sha256sum --check --strict unzip -q "$archive" -d "$SOURCE_DIR" chmod +x "$SOURCE_DIR/rustfs" @@ -170,22 +142,53 @@ jobs: echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV" echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV" - - name: Run upgrade compatibility test + # Reuse one compiled harness, while keeping each case in its own process. + # Continue after a failure to retain the former matrix's complete verdict. + - name: Run upgrade compatibility tests + shell: bash env: - RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }} NO_PROXY: 127.0.0.1,localhost,::1 run: | export no_proxy="$NO_PROXY" - python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \ - "upgrade_compatibility_test::${{ matrix.test }}" \ - -- --ignored --exact --nocapture + cases=( + direct-upgrade:direct_upgrade_from_previous_release_preserves_object_contracts + mixed-version-upgrade:rolling_upgrade_from_previous_release_preserves_mixed_version_contracts + bucket-config-upgrade:direct_upgrade_from_previous_release_preserves_bucket_configuration + bucket-config-rollback:rollback_to_previous_release_reads_current_bucket_metadata + multipart-layout-upgrade:direct_upgrade_from_previous_release_preserves_multipart_layouts + multipart-layout-baseline:previous_release_baseline_replicates_multipart_layouts + ) + if [[ "$UPGRADE_SOURCE_VERSION" == "1.0.0-rc.5" ]]; then + cases+=(odm-config-rollback:rc5_rollback_requires_restoring_odm_configuration) + else + cases+=(legacy-delete-reservation:stable_upgrade_deletes_foreign_reservations_without_losing_live_objects) + fi + status=0 + for entry in "${cases[@]}"; do + artifact="${entry%%:*}" + test="${entry#*:}" + export RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR="$RUNNER_TEMP/rustfs-upgrade-g09-evidence/$artifact" + export RUSTFS_E2E_LOG_DIR="$RUNNER_TEMP/rustfs-upgrade-logs/$artifact" + mkdir -p "$RUSTFS_E2E_LOG_DIR" + echo "::group::$test" + if python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \ + "upgrade_compatibility_test::$test" -- --ignored --exact --nocapture \ + 2>&1 | tee "$RUSTFS_E2E_LOG_DIR/test.log"; then + echo "PASS: $test" + else + echo "::error::Upgrade compatibility failed: $test" + status=1 + fi + echo "::endgroup::" + done + exit "$status" - name: Upload scanner/heal G09 evidence if: always() uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: - name: ${{ matrix.artifact }}-g09-evidence-${{ github.run_number }} - path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }} + name: upgrade-g09-evidence-${{ matrix.source.version }}-${{ github.run_number }} + path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence if-no-files-found: ignore retention-days: 14 @@ -193,7 +196,7 @@ jobs: if: always() uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: - name: ${{ matrix.artifact }}-server-logs-${{ github.run_number }} + name: upgrade-server-logs-${{ matrix.source.version }}-${{ github.run_number }} path: ${{ runner.temp }}/rustfs-upgrade-logs if-no-files-found: warn retention-days: 14 diff --git a/.github/workflows/gateway-integration-sync.yml b/.github/workflows/gateway-integration-sync.yml index f80fbfeb2..2209099e7 100644 --- a/.github/workflows/gateway-integration-sync.yml +++ b/.github/workflows/gateway-integration-sync.yml @@ -6,29 +6,20 @@ name: Gateway Integration Sync # Merges `main` into the long-lived `gateway/integration` branch every weekday # so the s3s -> gateway migration (rustfs/backlog#2734, task T0.1 in #2736) # never accumulates merge debt. A clean merge lands as a `--no-ff` merge commit -# pushed straight to the branch. A conflicting merge is aborted and surfaced as -# a pull request from `sync/-`, whose head is the un-merged -# source commit, so the conflict is resolved the same day, in review, with no -# other change riding along. The run itself stays green in that case: the pull -# request is the signal. Any other failure (push rejected, pull request not -# created) fails the run and, on the schedule, files the shared failure issue. +# pushed straight to the branch. A conflicting merge is aborted and fails the +# run after its summary records the source and target SHAs, run URL, and exact +# conflicting paths. It creates no sync branch or pull request. Resolve the +# merge in an isolated worktree based on the latest target, preserve the +# two-parent merge commit, and push only after the required checks pass and the +# target still matches the merge's first parent. If the target moves, restart +# from its new head. # -# Resolving a conflict pull request: merge `origin/gateway/integration` into -# the sync branch, resolve, push, get the review. Land it with a fast-forward -# push of the resolved merge commit to `gateway/integration`, not with the -# squash button: a squash drops the merge ancestry and the same hunks conflict -# again on the next run. -# -# Write scope: only `gateway/integration` and `sync/*` are ever pushed, and -# both push targets below are literal refs. The pushes use a one-hour -# installation token of the organisation's own GitHub App (contents: write) -# because that app is the only non-admin bypass actor of the +# Write scope: only `gateway/integration` is ever pushed. The push uses a +# one-hour installation token of the organisation's GitHub App (contents: +# write) because that app is the only non-admin bypass actor of the # `gateway/integration` ruleset; GITHUB_TOKEN cannot be a ruleset bypass actor, -# so it cannot push through the required reviews and checks. The conflict pull -# request is opened with FLAKE_UPDATE_TOKEN, the PAT that already opens the -# weekly flake.lock pull requests, because this repository keeps "Allow GitHub -# Actions to create and approve pull requests" switched off. GITHUB_TOKEN is -# therefore read-only here. +# so it cannot push through the required reviews and checks. GITHUB_TOKEN is +# read-only here. # # `schedule` fires only from the default branch, so the cron is live once this # file exists on `main`. `workflow_dispatch` runs from any ref that carries the @@ -94,6 +85,7 @@ jobs: id: merge env: SOURCE_REF: ${{ inputs.source_ref || 'main' }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} run: | set -euo pipefail if ! [[ "$SOURCE_REF" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] \ @@ -119,13 +111,25 @@ jobs: fi exit 0 fi - git diff --name-only --diff-filter=U > "$RUNNER_TEMP/conflicts.txt" - git merge --abort - if [[ ! -s "$RUNNER_TEMP/conflicts.txt" ]]; then + conflicts=$(git diff --name-only --diff-filter=U) + if [[ -z "$conflicts" ]]; then + git merge --abort || true echo "::error::git merge failed without reporting conflicting paths; this is not a conflict" exit 1 fi - echo "result=conflict" >> "$GITHUB_OUTPUT" + { + echo "## Gateway integration sync stopped on conflicts" + echo + echo "- Run: ${RUN_URL}" + echo "- Source: \`${SOURCE_REF}\` at ${source_sha}" + echo "- Target: \`${TARGET_BRANCH}\` at ${target_sha}" + echo + echo "Conflicting files:" + printf '%s\n' "$conflicts" | sed 's/^/- /' + } >> "$GITHUB_STEP_SUMMARY" + git merge --abort + echo "::error::${SOURCE_REF} cannot be merged cleanly into ${TARGET_BRANCH}; no branch or pull request was created" + exit 1 - name: Push the merge commit if: steps.merge.outputs.result == 'merged' @@ -137,50 +141,12 @@ jobs: git push origin HEAD:refs/heads/gateway/integration echo "Merged \`${SOURCE_REF}\` (${SOURCE_SHA}) into \`${TARGET_BRANCH}\` as $(git rev-parse HEAD)." >> "$GITHUB_STEP_SUMMARY" - - name: Open the conflict pull request - if: steps.merge.outputs.result == 'conflict' - env: - GH_TOKEN: ${{ secrets.FLAKE_UPDATE_TOKEN }} - SOURCE_REF: ${{ steps.merge.outputs.source_ref }} - SOURCE_SHA: ${{ steps.merge.outputs.source_sha }} - TARGET_SHA: ${{ steps.merge.outputs.target_sha }} - RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - run: | - set -euo pipefail - date_utc=$(date -u +%Y-%m-%d) - sync_branch="sync/${SOURCE_REF//\//-}-${date_utc}" - git push origin "refs/remotes/origin/${SOURCE_REF}:refs/heads/${sync_branch}" - { - echo "The scheduled merge of \`${SOURCE_REF}\` into \`${TARGET_BRANCH}\` stopped on conflicts and was aborted; \`${TARGET_BRANCH}\` is unchanged." - echo - echo "This branch is the un-merged \`${SOURCE_REF}\` head. Merge \`origin/${TARGET_BRANCH}\` into it, resolve the files below, push, and get the review. Land the resolved merge commit with a fast-forward push to \`${TARGET_BRANCH}\`; a squash merge drops the merge ancestry and the same hunks conflict again on the next run. Do not include any change that is not conflict resolution." - echo - echo "- Run: ${RUN_URL}" - echo "- Source: \`${SOURCE_REF}\` at ${SOURCE_SHA}" - echo "- Target: \`${TARGET_BRANCH}\` at ${TARGET_SHA}" - echo - echo "## Conflicting files" - echo - echo "Output of \`git diff --name-only --diff-filter=U\` captured before \`git merge --abort\`:" - echo - sed 's/^/- /' "$RUNNER_TEMP/conflicts.txt" - } > "$RUNNER_TEMP/pr-body.md" - title="sync: merge ${SOURCE_REF} ${date_utc} (conflicts)" - existing=$(gh pr list --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --state open --json number --jq '.[0].number // empty') - if [[ -n "$existing" ]]; then - gh pr edit "$existing" --repo "$GITHUB_REPOSITORY" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md" - pr_url=$(gh pr view "$existing" --repo "$GITHUB_REPOSITORY" --json url --jq .url) - else - pr_url=$(gh pr create --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md") - fi - echo "Conflicts in \`${SOURCE_REF}\` -> \`${TARGET_BRANCH}\`; pull request: ${pr_url}" >> "$GITHUB_STEP_SUMMARY" - alert-on-failure: name: Alert on scheduled failure needs: [sync] # Same ci-8 mechanism audit.yml and coverage.yml use: a scheduled run that - # could not push or could not open the conflict pull request must not stay - # visible only in the Actions tab. + # could not push or encountered merge conflicts must not stay visible only + # in the Actions tab. if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')) runs-on: ubuntu-latest timeout-minutes: 10 diff --git a/.github/workflows/nightly-gnu.yml b/.github/workflows/nightly-gnu.yml index afbbb210c..3f72a6311 100644 --- a/.github/workflows/nightly-gnu.yml +++ b/.github/workflows/nightly-gnu.yml @@ -80,13 +80,24 @@ jobs: - name: Setup Rust environment uses: ./.github/actions/setup with: - cache-shared-key: build-x86_64-unknown-linux-gnu + cache-shared-key: build-x86_64-unknown-linux-gnu-glibc-2.31 cache-save-if: 'false' + install-cross-tools: 'true' + target: x86_64-unknown-linux-gnu install-build-packaging-tools: 'false' install-test-tools: 'false' - name: Build RustFS - run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins + # Pin the glibc baseline (`.` for cargo-zigbuild) so + # nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062. + run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins + + - name: Verify glibc symbol baseline + run: | + python3 scripts/release/check_glibc.py 2.31 \ + target/x86_64-unknown-linux-gnu/release/rustfs \ + target/x86_64-unknown-linux-gnu/release/rustfs-cli + target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu' - name: Build DEB package id: deb diff --git a/.github/workflows/oidc-keycloak.yml b/.github/workflows/oidc-keycloak.yml index 16b736521..f40f77859 100644 --- a/.github/workflows/oidc-keycloak.yml +++ b/.github/workflows/oidc-keycloak.yml @@ -20,8 +20,9 @@ on: - ".github/workflows/oidc-keycloak.yml" - "crates/config/src/constants/oidc.rs" - "crates/iam/src/federation/**" - - "crates/iam/src/oidc.rs" + - "crates/iam/src/oidc/**" - "rustfs/src/admin/handlers/oidc.rs" + - "rustfs/src/admin/service/oidc_config.rs" - "rustfs/src/admin/handlers/sts.rs" - "scripts/test/oidc_keycloak_live.sh" - "scripts/test/fixtures/keycloak-rustfs-ci-realm.json" @@ -31,8 +32,9 @@ on: - ".github/workflows/oidc-keycloak.yml" - "crates/config/src/constants/oidc.rs" - "crates/iam/src/federation/**" - - "crates/iam/src/oidc.rs" + - "crates/iam/src/oidc/**" - "rustfs/src/admin/handlers/oidc.rs" + - "rustfs/src/admin/service/oidc_config.rs" - "rustfs/src/admin/handlers/sts.rs" - "scripts/test/oidc_keycloak_live.sh" - "scripts/test/fixtures/keycloak-rustfs-ci-realm.json" diff --git a/.github/workflows/windows-filesystem.yml b/.github/workflows/windows-filesystem.yml index 46a395456..fe6ca7000 100644 --- a/.github/workflows/windows-filesystem.yml +++ b/.github/workflows/windows-filesystem.yml @@ -19,8 +19,21 @@ on: branches: [ main ] paths: - "crates/ecstore/src/disk/**" + - "crates/ecstore/src/layout/**" - "crates/ecstore/src/store/init_format.rs" + - "crates/ecstore/src/set_disk/ops/locking.rs" + - "crates/ecstore/src/object_api/write_commit_context.rs" + - "crates/ecstore/src/bucket/utils.rs" - "crates/ecstore/Cargo.toml" + - "crates/utils/src/dunce.rs" + - "crates/utils/src/os/**" + - "crates/utils/Cargo.toml" + - "crates/utils/src/path.rs" + - "rustfs/build.rs" + - "rustfs/Cargo.toml" + - "rustfs/src/main.rs" + - "rustfs/src/server/runtime.rs" + - "rustfs/src/startup_*.rs" - "Cargo.toml" - "Cargo.lock" - ".github/actions/setup/**" @@ -29,8 +42,21 @@ on: branches: [ main ] paths: - "crates/ecstore/src/disk/**" + - "crates/ecstore/src/layout/**" - "crates/ecstore/src/store/init_format.rs" + - "crates/ecstore/src/set_disk/ops/locking.rs" + - "crates/ecstore/src/object_api/write_commit_context.rs" + - "crates/ecstore/src/bucket/utils.rs" - "crates/ecstore/Cargo.toml" + - "crates/utils/src/dunce.rs" + - "crates/utils/src/os/**" + - "crates/utils/Cargo.toml" + - "crates/utils/src/path.rs" + - "rustfs/build.rs" + - "rustfs/Cargo.toml" + - "rustfs/src/main.rs" + - "rustfs/src/server/runtime.rs" + - "rustfs/src/startup_*.rs" - "Cargo.toml" - "Cargo.lock" - ".github/actions/setup/**" @@ -77,14 +103,118 @@ jobs: shell: pwsh run: cargo test -p rustfs-ecstore --lib rename_all_ -- --nocapture + - name: Test Windows path conversion + shell: pwsh + run: "cargo test -p rustfs-utils --no-default-features --features path --lib dunce:: -- --nocapture" + + - name: Test Windows volume and disk-space probes + shell: pwsh + run: | + # Map a drive letter to the temp drive's administrative share so the + # remote-volume probe also covers mapped network drives. + $drive = (Split-Path -Qualifier ([System.IO.Path]::GetTempPath())).TrimEnd(':') + net use R: "\\localhost\$drive`$" /persistent:no + if ($LASTEXITCODE -eq 0) { + $env:RUSTFS_TEST_WINDOWS_MAPPED_DRIVE = 'R:' + } else { + Write-Output "::warning::could not map R: to \\localhost\$drive`$; mapped-drive test will skip" + } + cargo test -p rustfs-utils --features os --lib os:: -- --nocapture + $status = $LASTEXITCODE + if ($env:RUSTFS_TEST_WINDOWS_MAPPED_DRIVE) { net use R: /delete /y | Out-Null } + exit $status + - name: Test Windows handle guards shell: pwsh run: cargo test -p rustfs-ecstore --lib windows_ -- --nocapture + - name: Test endpoint path parsing + shell: pwsh + run: "cargo test -p rustfs-ecstore --lib layout::endpoint:: -- --nocapture" + - name: Test startup temporary-directory cleanup shell: pwsh run: cargo test -p rustfs-ecstore --lib cleanup_tmp_on_startup_ -- --nocapture + - name: Test object key Windows admission + shell: pwsh + run: "cargo test -p rustfs-ecstore --lib bucket::utils:: -- --nocapture" + + - name: Test Windows file-name rules + shell: pwsh + run: "cargo test -p rustfs-utils --no-default-features --features path --lib dunce:: -- --nocapture" + - name: Test fresh format publication shell: pwsh run: cargo test -p rustfs-ecstore --lib fresh_format_load_initializes_all_disks -- --nocapture + + # The native Windows main thread only gets the 1 MiB PE stack reserve, while + # Linux and macOS hand out 8 MiB. Startup runs the whole root future on the + # thread that calls `block_on`, so a debug server once overflowed before it + # listened on Windows only. Start a real server to keep that covered. + startup-smoke: + name: Startup Smoke + runs-on: windows-latest + timeout-minutes: 90 + steps: + - name: Checkout repository + uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7 + with: + persist-credentials: false + + - name: Setup Rust environment + uses: ./.github/actions/setup + with: + rust-version: stable + cache-shared-key: build-x86_64-pc-windows-msvc + cache-save-if: 'false' + install-build-packaging-tools: 'false' + install-test-tools: 'false' + + - name: Build debug server binary + shell: pwsh + run: cargo build --locked -p rustfs --bin rustfs + + - name: Print version + shell: pwsh + run: | + & .\target\debug\rustfs.exe --version + if ($LASTEXITCODE -ne 0) { throw "rustfs --version exited with $LASTEXITCODE" } + + - name: Start server and wait for readiness + shell: pwsh + env: + RUSTFS_ACCESS_KEY: rustfsadmin + RUSTFS_SECRET_KEY: rustfsadmin + run: | + $volume = Join-Path $env:RUNNER_TEMP 'rustfs-startup-smoke' + $stdout = Join-Path $env:RUNNER_TEMP 'rustfs-startup-smoke.out.log' + $stderr = Join-Path $env:RUNNER_TEMP 'rustfs-startup-smoke.err.log' + New-Item -ItemType Directory -Force -Path $volume | Out-Null + $server = Start-Process -FilePath .\target\debug\rustfs.exe ` + -ArgumentList @('--address', '127.0.0.1:9000', "`"$volume`"") ` + -RedirectStandardOutput $stdout -RedirectStandardError $stderr ` + -NoNewWindow -PassThru + # Cache the process handle now; otherwise ExitCode reads back empty. + $null = $server.Handle + $ready = $false + try { + for ($i = 0; $i -lt 180; $i++) { + if ($server.HasExited) { break } + try { + $response = Invoke-WebRequest -Uri 'http://127.0.0.1:9000/health/ready' -UseBasicParsing -NoProxy -TimeoutSec 2 + if ($response.StatusCode -eq 200) { $ready = $true; break } + } catch {} + Start-Sleep -Seconds 1 + } + } finally { + if (-not $server.HasExited) { Stop-Process -Id $server.Id -Force } + $server.WaitForExit() + Write-Host '--- server stdout (tail) ---' + Get-Content $stdout -Tail 100 -ErrorAction SilentlyContinue + Write-Host '--- server stderr (tail) ---' + Get-Content $stderr -Tail 100 -ErrorAction SilentlyContinue + } + if (-not $ready) { + throw ("rustfs did not report ready on /health/ready (exit code: {0})" -f $server.ExitCode) + } diff --git a/.gitignore b/.gitignore index 8a26b9e94..1dad3b899 100644 --- a/.gitignore +++ b/.gitignore @@ -90,3 +90,4 @@ worktrees/* # insta scratch files; the accepted .snap files ARE the assertions and are committed *.snap.new *.pending-snap +.mimosa \ No newline at end of file diff --git a/.vscode/launch.json b/.vscode/launch.json index 394c1e979..d842e5118 100644 --- a/.vscode/launch.json +++ b/.vscode/launch.json @@ -7,7 +7,9 @@ "name": "Debug(only) executable 'rustfs'", "env": { "RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info", - "RUSTFS_SKIP_BACKGROUND_TASK": "on" + "RUSTFS_SKIP_BACKGROUND_TASK": "on", + // Local test volumes share one physical disk. + "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true" //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs", // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow", // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token" @@ -44,6 +46,7 @@ "env": { "RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug", "RUSTFS_SKIP_BACKGROUND_TASK": "on", + "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true", //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs", // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow", // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token" @@ -117,7 +120,8 @@ "env": { "RUSTFS_ACCESS_KEY": "rustfsadmin", "RUSTFS_SECRET_KEY": "rustfsadmin", - //"RUSTFS_VOLUMES": "./target/volume/test{1...4}", + "RUSTFS_VOLUMES": "./target/volume/test{1...4}", + "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true", "RUSTFS_ADDRESS": ":9000", "RUSTFS_CONSOLE_ENABLE": "true", // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint @@ -135,41 +139,30 @@ { "type": "lldb", "request": "launch", - "name": "Debug test_lifecycle_transition_basic", + "name": "Debug test_transition_and_restore_flows", "cargo": { "args": [ "test", + "--no-run", "-p", "rustfs-scanner", "--test", "lifecycle_integration_test", - "serial_tests::test_lifecycle_transition_basic", "-j", "1" - ] + ], + "filter": { + "name": "lifecycle_integration_test", + "kind": "test" + } }, - "args": [], - "cwd": "${workspaceFolder}" - }, - { - "name": "Debug executable target/debug/test", - "type": "lldb", - "request": "launch", - "program": "${workspaceFolder}/target/debug/deps/lifecycle_integration_test-5915cbfcab491b3b", "args": [ - "--skip", - "test_lifecycle_expiry_basic", - "--skip", - "test_lifecycle_expiry_deletemarker", - //"--skip", - //"test_lifecycle_transition_basic", - ], - "cwd": "${workspaceFolder}", - //"stopAtEntry": false, - //"preLaunchTask": "cargo build", - "sourceLanguages": [ - "rust" + "serial_tests::test_transition_and_restore_flows", + "--exact", + "--ignored", + "--test-threads=1" ], + "cwd": "${workspaceFolder}" }, { "name": "Debug executable target/debug/rustfs with sse kms", @@ -285,7 +278,7 @@ "RUSTFS_COMPRESSION_ENABLED": "true", "RUSTFS_KMS_ENABLE": "true", "RUSTFS_KMS_BACKEND": "local", - "RUSTFS_KMS_KEY_DIR": "./target/rio-v2/kms-key-dir", + "RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/rio-v2/kms-key-dir", "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key", }, @@ -307,6 +300,7 @@ "RUSTFS_ACCESS_KEY": "rustfsadmin", "RUSTFS_SECRET_KEY": "rustfsadmin", "RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}", + "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true", "RUSTFS_REGION": "us-east-1", "RUSTFS_ADDRESS": ":9000", "RUSTFS_CONSOLE_ENABLE": "true", @@ -315,7 +309,7 @@ "RUSTFS_KMS_ENABLE": "true", "RUSTFS_KMS_BACKEND": "local", - "RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir", + "RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/e2e-key-dir", "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key", }, diff --git a/CHANGELOG.md b/CHANGELOG.md index 00c2b33b8..02eaf4e62 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Presigned URLs honour only signed headers** (GHSA-g8w9-qw9q-fghr): a SigV4 presigned request that carries an `x-amz-*` request header not listed in `X-Amz-SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3. Previously the holder of a presigned `PutObject` URL could add unsigned `x-amz-tagging`, `x-amz-storage-class`, `x-amz-website-redirect-location`, ACL, metadata, Object Lock or SSE headers and have them applied. Presigners that intend a property must set it before signing so the SDK lists the header in `SignedHeaders`; `x-amz-cf-id` (CloudFront) remains tolerated unsigned. Header-signed SigV4 and SigV2 requests are unchanged. ### Fixed +- **Replication status writebacks no longer invalidate scanner cycles**: every replication status writeback (`persist_replication_state_if_current` in `Update` mode, including the heals the scanner queues during its own walk) marked the bucket dirty on the namespace path, advanced the dirty-usage generation and woke the scanner, so a site-replication resync or any steadily replicated bucket ended every cycle `superseded`, the authoritative usage baseline was never certified, and the console reported usage freshness as stale for as long as replication ran (#8375). Those writebacks are now status class: they do not touch the dirty-usage generation, the activity snapshot, the dirty-usage wake or the durable dirty journal, and only the process-local producer coverage is recorded. They need no rescan queue, because a deployment with an active replication rule already walks every bucket on every cycle and that walk refreshes `replication_stats`. Tier-expiration cleanup, which removes data, stays on the namespace path. No persisted format, protocol, admin API or configuration changes. Design: `docs/architecture/scanner-per-bucket-convergence.md`. +- **A bucket written during a scanner walk no longer supersedes the whole cycle**: the post-walk cluster activity proof required the entire activity snapshot and the dirty-usage generation to be unchanged, so a single-node deployment with one bucket written every few seconds never reached a `Complete` cycle, `usage_snapshot_converged` never became true and the authoritative usage baseline was never certified (#8494). Dirty usage is now certified per bucket: a structurally complete walk that the guardrail did not veto is `Complete`, buckets whose dirty generation still matches the start snapshot are cleared, and buckets written during the walk stay dirty and are published as the pending set, so `usage_snapshot_converged` means "no bucket pending" and the next cycle is scoped to the pending buckets. Such a cycle writes the authoritative usage object; the observed object is now selected by an explicit observational mark that vetoed cycles carry. The guardrail itself vetoes only on the fields that change which namespace the walk described (peer set, process instance, protocol version, topology, maintenance generation, data movement, movement generation, publication blocked) and reports the field and host in the scanner log and in the new `usage_freshness` fields `guardrail_veto_total`, `last_guardrail_veto_unix_secs` and `last_guardrail_veto_reason`. In a distributed deployment a remote peer whose dirty usage moved during the walk still supersedes the cycle until scoped acknowledgements can name its buckets. `.usage.v2.json` gains the additive, default-when-absent fields `usage_snapshot_pending_buckets`, `usage_snapshot_pending_bucket_count` and `usage_snapshot_observational`; older readers ignore them and treat the snapshot as an authoritative but unconverged publication. Quota is unchanged: an uncertified bucket keeps its last usage value as the floor and live writes do not advance it. No protocol or configuration changes. Design: `docs/architecture/scanner-per-bucket-convergence.md`. - **SSE-C TLS requirement now covers copy-source keys**: with `RUSTFS_SSE_C_REQUIRE_TLS=true`, a `CopyObject` or `UploadPartCopy` that carried only the `x-amz-copy-source-server-side-encryption-customer-*` headers (an SSE-C source copied to a destination that is not SSE-C) was accepted on a plaintext transport and was not counted in `rustfs_ssec_plaintext_requests_total`, although those headers carry the source object's customer key. They now count as SSE-C headers: such requests are counted, and refused with `400 InvalidRequest` when the switch is on. Requests over TLS are unaffected. - **SSE-KMS encryption context could replace the location entry**: data keys are wrapped under an encryption-context entry, keyed by the bucket name, that ties them to the object's location. A client `x-amz-server-side-encryption-context` entry under the same key replaced it, and because the client context is stored with the object, the replacement also applied on every read. SSE-KMS writes now refuse such a context with `400 InvalidArgument`; other keys, including the names of other buckets, are unaffected. Objects stored with such an entry by an earlier release remain readable. See `docs/operations/kms-backend-security.md`. - **Two admin handlers reported a missing policy or user as `500 InternalError`**: `GET /rustfs/admin/v3/info-canned-policy` and `DELETE /rustfs/admin/v3/remove-user` wrapped every IAM error as an internal fault instead of classifying it, so a lookup for something that does not exist answered `500 InternalError` where the rest of the admin API answers `404 NoSuchResource` — `GET /rustfs/admin/v3/user-info` already did, and `iam_error_to_s3_error` already mapped `NoSuchPolicy`, `NoSuchUser`, `NoSuchServiceAccount` and `NoSuchTempAccount` to `NoSuchResource`. A client could not tell "does not exist" from a real server failure, and a controller that checks existence before creating (a Kubernetes operator reconciling policies or users) could never create the object: the missing-policy lookup failed with a `500` and it retried forever. Both handlers now map their IAM errors through the shared `iam_error_to_s3_error` helper, so a missing policy or user is `404 NoSuchResource` and the `error` is preserved as the S3 error's source; the three IAM calls in the removal path (temporary-user lookup, service-account lookup, delete) are all reclassified, and every other IAM error stays `500 InternalError`. Fixes #8117. @@ -85,6 +87,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Four-layer regression-prevention tests guard against silent feature deletion: compile-time module assertion, module-presence unit test, cross-module `Protocol` enum assertion, end-to-end SSH banner test against the running binary ### Changed +- **Scanner bucket walkers are no longer tied to the disk count**: a new scanner key `walkers_per_disk` (`RUSTFS_SCANNER_WALKERS_PER_DISK`, `1` through `16`, default `1`) spawns that many bucket walkers per local disk of an erasure set, so a single-disk deployment can walk several buckets at once instead of one bucket, one object at a time. The walkers of one disk share its scanner admission, so a peer still sees one scan session per physical disk, and remote disks keep one walker. The walker count is what `max_concurrent_disk_scans` caps and the disk count stays the floor; the `disk_concurrency_budget` log line now reports `walkers_per_disk`, `walkers`, and the `reason` for the effective limit. The default keeps today's I/O profile because a local disk cannot yet report whether it is rotational; the design intends a higher default on flash once it can (`docs/architecture/scanner-per-bucket-convergence.md`). As groundwork for pacing, the raw-enumeration page writer inserts into its building page in place and commits pages without copying them, and a regression test pins the cost of recording 20,000 entries in one prefix so the quadratic bookkeeping removed by [#8114](https://github.com/rustfs/rustfs/pull/8114) (rustfs#8113) cannot return. No persisted format, protocol or admin API changes beyond the additional config key, which older releases preserve and ignore. - **Encryption and KMS work merged since `1.0.0-rc.5`** (entries were missing from this section): - **Persisted KMS configuration secrets** are sealed field-by-field with `RUSTFS_KMS_CONFIG_SECRET`. **When the variable is unset the secrets are persisted in cleartext and the server only warns** (`persisted KMS configuration carries cleartext secrets`); it never refuses the write. Set it, identically, on every node, and re-save the configuration to seal an existing one. - **New v2 ciphertext frame format** with per-frame index binding and final-frame authentication. Its write switch `RUSTFS_ENCRYPTION_FRAME_V2` is **off by default**: v2 frames are unreadable by nodes without v2 read support, and encrypted ciphertext travels verbatim through transition, decommission and SSE-C replication passthrough, so turn it on only after every node — and every RustFS warm/replication target that receives raw ciphertext — runs a release with v2 read support. Reading v2 objects needs no switch. diff --git a/Cargo.lock b/Cargo.lock index 932d315dc..19a417978 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -30,7 +30,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1973cfbc1a2daf9cf550e74e1f088c28e7f7d8c1e1418fb6c9dc5184b7e84c99" dependencies = [ "crypto-common 0.2.2", - "inout 0.2.2", + "inout 0.2.4", ] [[package]] @@ -328,9 +328,9 @@ dependencies = [ [[package]] name = "archive-trait" -version = "0.0.14" +version = "0.0.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c6080ea14ccf9019d7ce572c319e581c20a805c9bdc6dbfb9b988da003cbd1a3" +checksum = "999bf88cc483664cab77700ca2fdfb0415a73412abe284ef5bc2319d00938e00" dependencies = [ "cap-std", "thiserror 2.0.21", @@ -371,9 +371,9 @@ checksum = "d3fb67a6e08acf24fdeccbac2cb6ac4305825bd1f117462e0e6f2f193345ad56" [[package]] name = "arrow" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c14b3d39f306bc28fd639d59f06e17a0f377d0021e1b7e9054e4d6fedc98774" +checksum = "4c25317ebddb8f2e6f006759336c95e5ded847a446f3524c27853bfe7d77a69b" dependencies = [ "arrow-arith", "arrow-array", @@ -392,9 +392,9 @@ dependencies = [ [[package]] name = "arrow-arith" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce2961626677665b2195eb59242af4c7befe7b8737ca2050295389362380104e" +checksum = "2ab555c0ffe9cb817041c0cee80baa9663a22225d0a035693579771f2693cadc" dependencies = [ "arrow-array", "arrow-buffer", @@ -406,9 +406,9 @@ dependencies = [ [[package]] name = "arrow-array" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e5f6adeffdf587d7a31db5d2266189624b526730cd3627f9ff9fedae97ad584" +checksum = "2dae1d91737421e92c6dd7c7f9d0f2d25198fd6fc539738b628f4dad10adf533" dependencies = [ "ahash", "arrow-buffer", @@ -425,9 +425,9 @@ dependencies = [ [[package]] name = "arrow-buffer" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "097d193003ce7995d5d087089069ec2a6e0187faf5a6f8c9f38af2645d987182" +checksum = "1fe9cd0064f2d7cf8ecfaf77cd70e67a980bc71aea92b63102bd37a7d3379a09" dependencies = [ "bytes", "half", @@ -437,9 +437,9 @@ dependencies = [ [[package]] name = "arrow-cast" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "635c9c635668ad26adf76cce8fb276c4be7cf06e63bd516de7da514f9680ee53" +checksum = "ee510bc252781d700400979a96bd1eadebeae23901132bbabd92118da1b376c5" dependencies = [ "arrow-array", "arrow-buffer", @@ -459,9 +459,9 @@ dependencies = [ [[package]] name = "arrow-csv" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c2ebf8d631e79b02c16cf5ae860561272c26024ec88fce389a56aaddd558e86" +checksum = "ace83d5935421b950fea3a0a3f9972c0837c297ef98ade57127a5661b681112c" dependencies = [ "arrow-array", "arrow-cast", @@ -474,9 +474,9 @@ dependencies = [ [[package]] name = "arrow-data" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ba2f832eaeca24b8f26143dba750e42ee4ab51cf7d65e701ca9607cfda9f358" +checksum = "54f128c73bb7fe9f7137218f910edd5b436f390e0f040b1956f5028390787897" dependencies = [ "arrow-buffer", "arrow-schema", @@ -487,9 +487,9 @@ dependencies = [ [[package]] name = "arrow-ipc" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dcc41681ea80f521df14c36725b74d4c60702c47f0793af2be469c04527e2599" +checksum = "8e244f39b82fa7a823e8b5984a0bd988234f9c92c1df79378fb85090e6e4c821" dependencies = [ "arrow-array", "arrow-buffer", @@ -503,9 +503,9 @@ dependencies = [ [[package]] name = "arrow-json" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a2f57d7a81969f24ccf80809587b76c09897e6f829d2d65a5976bfb3218851f1" +checksum = "727b21ebf8894dcd9504d6ab66bffdac338391e1f85cde594526a0210299ea40" dependencies = [ "arrow-array", "arrow-buffer", @@ -528,9 +528,9 @@ dependencies = [ [[package]] name = "arrow-ord" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2c900759f3bd8354fd4196bc4403eee846894dc2adf66b4225472006a0bf18c5" +checksum = "70d376b9b883239ae0f9963cdeeb59126ccaac08d60adf208a6ad3e2dd4048c3" dependencies = [ "arrow-array", "arrow-buffer", @@ -541,9 +541,9 @@ dependencies = [ [[package]] name = "arrow-row" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f4c6425032e28266e3fc4ff680805e57e670d6ea92473043f3e65b7ed6ac79f2" +checksum = "48096811f92b5c3a0c59d33cfecd7d7d2ed037d81e54a27a42d232e7700e6de2" dependencies = [ "arrow-array", "arrow-buffer", @@ -554,9 +554,9 @@ dependencies = [ [[package]] name = "arrow-schema" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10fab8d4563491417ba801fab29d205104d20d4bdf37bda6cd1cf425cff598cd" +checksum = "37b6c8a6b12cf3869d299ce36ed8e1cd6fe9836264b68fb6c6538473a36124a1" dependencies = [ "serde_core", "serde_json", @@ -564,9 +564,9 @@ dependencies = [ [[package]] name = "arrow-select" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc58569193c2525915f3cc6310edba3792f1200f65d6e9ed330aa33e691493b8" +checksum = "e523e85fa9fb3181a391bf0edfa6a1ae0dc46df8cbf7f6b96034dcb1e2a7a04a" dependencies = [ "ahash", "arrow-array", @@ -578,9 +578,9 @@ dependencies = [ [[package]] name = "arrow-string" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e0813f3c35c1cfea65e14c20a953440f7783c088b7ad2d0db162ccdeefcec14" +checksum = "b931a905a0e4b8832d9f54fb369cf6e545f9dc03cb3d9611216a7bd2653df60a" dependencies = [ "arrow-array", "arrow-buffer", @@ -791,7 +791,7 @@ checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -916,9 +916,9 @@ dependencies = [ [[package]] name = "aws-lc-rs" -version = "1.18.1" +version = "1.18.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b281d307588d634de920874890732659e2e7672f72b5e10e81badc1a8a83621e" +checksum = "faac5829c2b74c28f830747e7818ccfb684261b5f48a1118b1e2a13d36dfab13" dependencies = [ "aws-lc-sys", "untrusted 0.7.1", @@ -927,9 +927,9 @@ dependencies = [ [[package]] name = "aws-lc-sys" -version = "0.45.0" +version = "0.46.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9bff6c3b54fad79a2e60b8102caf565819711497c1f5f092f49508e2f5c31b27" +checksum = "f1622d8446a2d4b2ce0c7eefc73dd43a99779028d5ee5c2dd8073a658ba8a2bc" dependencies = [ "cc", "cmake", @@ -1020,7 +1020,7 @@ dependencies = [ "lru 0.18.5", "percent-encoding", "regex-lite", - "sha2 0.11.0", + "sha2 0.11.1", "tracing", "url", ] @@ -1120,7 +1120,7 @@ dependencies = [ "http 1.5.0", "p256 0.13.2", "percent-encoding", - "sha2 0.11.0", + "sha2 0.11.1", "subtle", "time", "tracing", @@ -1155,7 +1155,7 @@ dependencies = [ "md-5 0.11.0", "pin-project-lite", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "tracing", ] @@ -1518,7 +1518,7 @@ checksum = "144e573728da132683b9488acd528274c790e07fc06ff81ee29f9d8f8b1041e0" dependencies = [ "blowfish", "pbkdf2 0.13.0", - "sha2 0.11.0", + "sha2 0.11.1", ] [[package]] @@ -1561,19 +1561,13 @@ dependencies = [ [[package]] name = "bit-set" -version = "0.8.0" +version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3" +checksum = "56d87354e4229f54a44f7bf2435906a4656dba36026ab6eaca629a2c436a691c" dependencies = [ - "bit-vec 0.8.0", + "bit-vec 0.10.1", ] -[[package]] -name = "bit-vec" -version = "0.8.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7" - [[package]] name = "bit-vec" version = "0.9.1" @@ -1583,6 +1577,16 @@ dependencies = [ "serde", ] +[[package]] +name = "bit-vec" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5727b15fa97d4f4fee0a3b7c3d550ed0269f54329207b86388de918604e31269" +dependencies = [ + "borsh", + "serde", +] + [[package]] name = "bitflags" version = "1.3.2" @@ -1703,7 +1707,31 @@ dependencies = [ "prettyplease 0.3.0", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", +] + +[[package]] +name = "borsh" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "553c5d846a6ba5150c65e3b1b8ec073bcf1abc20f9b7220de384a4443ea4e20a" +dependencies = [ + "borsh-derive", + "bytes", + "cfg_aliases", +] + +[[package]] +name = "borsh-derive" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12cdfe656708a01f89b451a7d36466e6fe6c414de0aa18fc54f864f6f9ca9f56" +dependencies = [ + "once_cell", + "proc-macro-crate", + "proc-macro2", + "quote", + "syn 3.0.9", ] [[package]] @@ -1950,9 +1978,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.6.0" +version = "1.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f74872d07caf508b30a21f6836e7d7016a2eaf7d9ff4f48deaa58cd8a0407630" +checksum = "6651c9ed80effdc7db0ff72512157f901af5e3549e341e24b1dd4887d836d838" dependencies = [ "find-msvc-tools", "jobserver", @@ -2072,7 +2100,7 @@ checksum = "e8cf2a2c93cd704877c0858356ed03480ff301ee950b43f1cbe4573b088bfa6c" dependencies = [ "block-buffer 0.12.1", "crypto-common 0.2.2", - "inout 0.2.2", + "inout 0.2.4", "zeroize", ] @@ -2107,7 +2135,7 @@ dependencies = [ "heck 0.5.0", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -2151,9 +2179,9 @@ checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" [[package]] name = "combine" -version = "4.6.8" +version = "4.6.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfc320937d09e6de266b31b9afb480f197d7a861be86be7cb2ea7e5d1bfffc5e" +checksum = "4356eba6976435498d05a311ee9fc486da3bc81c200d3f0d7dd9703b41913f92" dependencies = [ "bytes", "futures-core", @@ -2199,8 +2227,8 @@ dependencies = [ "liblzma", "lz4", "memchr", - "zstd 0.14.0", - "zstd-safe 8.0.0", + "zstd 0.14.2", + "zstd-safe 8.1.0", ] [[package]] @@ -2304,9 +2332,12 @@ dependencies = [ [[package]] name = "constant_time_eq" -version = "0.4.2" +version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" +checksum = "9eb6feb175df65d0a3ccd3e73484e940d0ac664e8034f0519fd49a93db3071d9" +dependencies = [ + "rustversion", +] [[package]] name = "convert_case" @@ -2764,7 +2795,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -2808,7 +2839,7 @@ checksum = "2ac7135c3ef02b2f7833bbeb1be5ba7f966dcde8a87c6b87f65a778d71a02785" dependencies = [ "darling_core 0.24.1", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -2839,9 +2870,9 @@ checksum = "4583a4551df46e2792f82ceeac45e850d2e2d5debba0b91f102385cda5b11f06" [[package]] name = "datafusion" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "14313430439ef858ed5f52ca9c840e9eea9844da34d3716d05a476449634da5d" +checksum = "4cd5317db77ba6bca25c761a8091b9d294a9f4188e4f9dcd0243e6a57de862ac" dependencies = [ "arrow", "arrow-schema", @@ -2888,9 +2919,9 @@ dependencies = [ [[package]] name = "datafusion-catalog" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2c0756ebc770bcebae366dfd273dad50935f6103b4d212253cb17e2e53c5427f" +checksum = "f35d36794509b7bfcbf90e11a1b5e4d047123c34610105e2e80a2048351cdb2f" dependencies = [ "arrow", "async-trait", @@ -2913,9 +2944,9 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bbde2961698705d8402f0f382f08876fc6f1126ffde790ef17833cf0ab87cd59" +checksum = "a6454bfb7f43b83942e2ca2576fd09342eedbd0d323e1eef72c3c8e4d57d7203" dependencies = [ "arrow", "async-trait", @@ -2937,9 +2968,9 @@ dependencies = [ [[package]] name = "datafusion-common" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6e06fe711ff815c8cf3143158547aea599702fd43f76f5866c70d5a5b3693a0" +checksum = "d26626e12b224b83351239465dcd00f72d47b02fb5e5b65a643e2dc8ed37f687" dependencies = [ "arrow", "arrow-ipc", @@ -2964,9 +2995,9 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f7a5107acff4e3ed50a7732acec06bba40e74ab49df021397ed250d60f04e42a" +checksum = "a106420cbfa980f242c22243ae9a6cce661d4d86ce96cff87dacabf7754a4470" dependencies = [ "futures", "log", @@ -2975,9 +3006,9 @@ dependencies = [ [[package]] name = "datafusion-datasource" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f69f45d2d700785be8df9d4b44410e1c8e4a856f10cc728b418a1066c0b594fb" +checksum = "210aa1e9525e9454f2399dbb8b0b95510145bd8c1fdc35de74e9ccbf81bcb699" dependencies = [ "arrow", "async-trait", @@ -3005,9 +3036,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-arrow" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "81650af64894ef34b46d2ef818ef244b40a81d90818f3bc642bf8fbf6758f8e0" +checksum = "65b1b041688cad707fa103c3e7f4c6d89b3134ebf67da0129070dad3381851eb" dependencies = [ "arrow", "arrow-ipc", @@ -3029,9 +3060,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-csv" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5e2519bed59a6907c79298b9c21fe807bbc34e4ff8414577d699b0741083593" +checksum = "2a7b34e253c4fa5251f174977af910a2f2ec831ec39dbbb56b29ce505cff278b" dependencies = [ "arrow", "async-trait", @@ -3052,9 +3083,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "635e61290e171597f6e50427e0cb486eda468ab47e880fa24d15bbb0fa24ac7e" +checksum = "1663338f43be6a5746473008e2d0fbde282577ddf0bae90d6c7386e988d71b29" dependencies = [ "arrow", "async-trait", @@ -3075,9 +3106,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-parquet" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07cd518f85b3a027d913047b4ab0811111a2b734fdf84a51b78234455f10d8fa" +checksum = "7c521ae91b6d899d7190812881f45fe9463c222c11d90149f5332a2e56555904" dependencies = [ "arrow", "arrow-schema", @@ -3107,15 +3138,15 @@ dependencies = [ [[package]] name = "datafusion-doc" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d44c5919b560862dd1cb20a5c75b3e48c3d9e49719c91d78da5c38eaad5495c4" +checksum = "0c2569c2e74a34fcae3a369ab3912d61e41a062592510f11e31b1dae2725ecdc" [[package]] name = "datafusion-execution" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7620ad535af8730186bdec05d720e40d9a4584b40c7eb75f72c43aa7978e5484" +checksum = "39cfdd5ac0769d1e74a885450e49fde8f63ca7ab41eb479a068564aaea80b837" dependencies = [ "arrow", "arrow-buffer", @@ -3139,9 +3170,9 @@ dependencies = [ [[package]] name = "datafusion-expr" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea229af8b73e566fc6ebb1c8dd42ab250f8855c5304bd173f4b50ead78b64304" +checksum = "86560fd7df4dda924f5474b588c56e7ae8e96dcf158c7327c9cfe9daff448ec5" dependencies = [ "arrow", "arrow-schema", @@ -3156,15 +3187,16 @@ dependencies = [ "indexmap 2.14.2", "itertools 0.15.0", "recursive", + "serde", "serde_json", "sqlparser", ] [[package]] name = "datafusion-expr-common" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a18baa840f77760dcc9eaad281329bc57536c5c39e65e8649596f2a51216be7" +checksum = "e82e313cf655cf6bc2dce6cbd23bcb0d20236a54a1b09c436615e4bac4d8c49f" dependencies = [ "arrow", "datafusion-common", @@ -3174,9 +3206,9 @@ dependencies = [ [[package]] name = "datafusion-functions" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "763bbfeec40b55a8a961d7349c0fcfcec4f3f66cc234d6295250b95371b85e34" +checksum = "939ae2771da8a3fdc23ab4de610ae8fef9515476626b49d96d0d5c782b105771" dependencies = [ "arrow", "arrow-buffer", @@ -3202,9 +3234,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "46c0c2b2e856883ecf6a4fa8b23e7c6041183af8a68d0583dc95d1d21e4eaee0" +checksum = "fd6c3aee9f2bc65716787d651e023743a90b4efbc57a2f679f036632e16fb110" dependencies = [ "arrow", "datafusion-common", @@ -3223,9 +3255,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e35e42c87dfbff06934ea0e99cfed3440ac614c3f3bfb8c33be35ca0587d4140" +checksum = "988b5d3dfb4939d81c2f9fab267063610fe52a843620f2fa6cb2a1598e43369c" dependencies = [ "arrow", "datafusion-common", @@ -3235,9 +3267,9 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08079b0604cdcc43c7d6cf006cfd5d3ed9c235ade0c1ebe91085a62f875603df" +checksum = "08149d433a97c9ea3cdf80de06480ab5c7f5453485f1e452d5b3583c81d5c0dd" dependencies = [ "arrow", "arrow-ord", @@ -3260,9 +3292,9 @@ dependencies = [ [[package]] name = "datafusion-functions-table" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fe2070ecf5638a93c81c8babc7858339ef52165a9f54da1411afd69adf7d85ce" +checksum = "2c767567cce2782d4d0bdfa55fab010aaba0bdbe533509e9e0b9a6a3ee29ed59" dependencies = [ "arrow", "async-trait", @@ -3276,9 +3308,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0e9a164ebe1998c5c3f9e15c9a0c4e6b599a0c8739b5471bfb089752441963f1" +checksum = "0a311c1f558d1f792f1f631c3670da12ee72a20fbf5d6ca151325d66191c03c1" dependencies = [ "arrow", "datafusion-common", @@ -3293,9 +3325,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window-common" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "98965b183f92201c16798994820ae032649c4821e089c4c80e7d0f4485d41df5" +checksum = "fffd54b8acf6ecef284cc0677e01e1b8c71be64fec50612ba00a0ff0d9b41a52" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -3303,20 +3335,20 @@ dependencies = [ [[package]] name = "datafusion-macros" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a711bd5a06ec85d9683e142d28a98ad0679a7220669f5c1b8589e57e9b29e46" +checksum = "14d6df9763c8fb968b2417d797aef4c55ac2e4a895aaef38c72c16a287aa4a40" dependencies = [ "datafusion-doc", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] name = "datafusion-optimizer" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff3256b62ff44616ec83c7f04741d21124970833bb377ed56d9abc5ec6f707ec" +checksum = "acfb8cea6e05e2e75bae183340cf7d53cccb3db0449e2f1fbc07654d980fd19f" dependencies = [ "arrow", "chrono", @@ -3334,9 +3366,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1e37a25b5aa1f3a1db114f0ee3605bf01639b83a831dcde35119ee61f0b11d6" +checksum = "9f002cef849e10c0597a217ec7891c2796c0f8b42d739f5d24abc24496373406" dependencies = [ "arrow", "arrow-schema", @@ -3357,9 +3389,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-adapter" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a4d12787d42e88b26b0d3ec006192fa23ceb832ef2a9e87439ce820a8e038cc" +checksum = "82761522c4536e7059d3b185cd99f10525c2bf48f1bd26a0a57e73e19eadc0fb" dependencies = [ "arrow", "datafusion-common", @@ -3372,9 +3404,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-common" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d13bf7f50fd902f5afc46d50759f5185b5620c40b24054f80f7c648897d3e5a6" +checksum = "f8c5c67aa252886f4d6b331c8114797f5b140464e774c4da36889269a1bf1bd8" dependencies = [ "arrow", "chrono", @@ -3389,9 +3421,9 @@ dependencies = [ [[package]] name = "datafusion-physical-optimizer" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "11fbe7e6665072957d44ed9587676e17a1f9ab3acfadfc287dac23750585f18a" +checksum = "f1a49dd3a76e4879012682c59dc3735282e60640c97483546a4113564a60bf8f" dependencies = [ "arrow", "datafusion-common", @@ -3409,9 +3441,9 @@ dependencies = [ [[package]] name = "datafusion-physical-plan" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1265d58e5bce07d154e642a51ff43033b576a6ae40d50a29ac2b9f311004eb52" +checksum = "b1451d6817e5ac0dd21819b19949d1c420ab51d748f55dd9a96ecb5418a24c97" dependencies = [ "arrow", "arrow-data", @@ -3438,15 +3470,16 @@ dependencies = [ "num-traits", "parking_lot", "pin-project-lite", + "serde", "serde_json", "tokio", ] [[package]] name = "datafusion-pruning" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "74b06b333405c05015836ed36cea02cc29d1e2ee8fb414a77532d8346fb56492" +checksum = "f11f0dbe2b5a642f6106e44de8bbd6654fa7220ca14f32313d4efd5514a86968" dependencies = [ "arrow", "datafusion-common", @@ -3460,9 +3493,9 @@ dependencies = [ [[package]] name = "datafusion-session" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7616b17b1d2fee3e9b13c651eac91b768d153265e6ba88e06e0b4af7883f69c" +checksum = "62dc5ce37a5ce1ff5362949776829aded2529a11dbaa13d24b0ffa6a059e106a" dependencies = [ "arrow-schema", "async-trait", @@ -3475,9 +3508,9 @@ dependencies = [ [[package]] name = "datafusion-sql" -version = "55.1.0" +version = "55.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b4848be66dd008d55adbe1dbc8cbd6a2502588ad0c0b8483bab0f43fd4495d0" +checksum = "816c224a0ef5469a32a562b5d4830544f714f97c5f50142109286d40a288c7db" dependencies = [ "arrow", "bigdecimal", @@ -3575,18 +3608,18 @@ checksum = "ac6b926516df9c60bfa16e107b21086399f8285a44ca9711344b9e553c5146e2" [[package]] name = "defmt" -version = "0.3.100" +version = "0.3.101" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0963443817029b2024136fc4dd07a5107eb8f977eaf18fcd1fdeb11306b64ad" +checksum = "4deef977660e373c458df605a913d4ab121db0ccd18d1ad6976dff7d37cddb53" dependencies = [ - "defmt 1.1.1", + "defmt 1.2.0", ] [[package]] name = "defmt" -version = "1.1.1" +version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" +checksum = "e4a0795ac740cb54d7b3775c9c17345f6aaecf884c01733b39d6017738e0e3e1" dependencies = [ "bitflags 1.3.2", "defmt-macros", @@ -3594,21 +3627,21 @@ dependencies = [ [[package]] name = "defmt-macros" -version = "1.1.1" +version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" +checksum = "840bb056bd99c495c9188198a258ab787c79f306d864db3e6e8451442d6d8366" dependencies = [ "defmt-parser", "proc-macro2", "quote", - "syn 2.0.119", + "syn 3.0.9", ] [[package]] name = "defmt-parser" -version = "1.0.0" +version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" +checksum = "71958f69fb2de856c5dd51ac9377e986ca1aa5919f931bf5395a807277f889a0" dependencies = [ "thiserror 2.0.21", ] @@ -3675,9 +3708,9 @@ dependencies = [ [[package]] name = "deranged" -version = "0.5.8" +version = "0.5.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" +checksum = "9e9de72ce2ad1f90dc62fa25f0f430ef85eb4b0d8fa0be4f30373bc40a21d28e" dependencies = [ "serde_core", ] @@ -3690,7 +3723,7 @@ checksum = "2e2b94854e8576378ccda7c8de8a66ed8b4e8acbd2c50ec3418ea6c8aaf4b567" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -3767,9 +3800,9 @@ dependencies = [ [[package]] name = "dial9" -version = "0.5.3" +version = "0.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b35f95885a2167602b457c77caf898670c02eb0aaba72bef76ac5b0faef86e14" +checksum = "734474ca4e814f893385c8fdea63688a9e6034ea19a5ec9846f8f515a69c9bed" dependencies = [ "bon", "dial9-core", @@ -3784,9 +3817,9 @@ dependencies = [ [[package]] name = "dial9-core" -version = "0.5.3" +version = "0.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "82f826078c782919cf9a4fb0b2b6755dab24d6c908b5181469b23cf6eb8f6fff" +checksum = "4ff5074fab1f810a73d58336275eb163e795ab0ae923fcc8437ba58bf2994e33" dependencies = [ "ahash", "arc-swap", @@ -3818,9 +3851,9 @@ dependencies = [ [[package]] name = "dial9-metrique" -version = "0.5.3" +version = "0.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c59e35e7b38ce255d3dc6310b48c3ea54307c42a0589eda44ea91ba24eadf67" +checksum = "3dd61f8f3d44db27e7a5e970c216bf4c19db66ebb68b7f58db2549f010c3a02e" dependencies = [ "dial9-core", "dial9-trace-format", @@ -3832,9 +3865,9 @@ dependencies = [ [[package]] name = "dial9-perf-self-profile" -version = "0.5.3" +version = "0.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c429b209bc07612b2e60267e8bca80cd2b249f498e9a0b83f7caf514d5f5388" +checksum = "d320bef9be4ef9c20ca9800cbfc369c319304b5e11d0a7aa931f5482aa5c1bd8" dependencies = [ "blazesym", "bon", @@ -3851,9 +3884,9 @@ dependencies = [ [[package]] name = "dial9-tokio-telemetry" -version = "0.5.3" +version = "0.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aee46c0a97dd88ba151f9688c79de60c41cd7efc6855e6eb369af06326cba8f3" +checksum = "1a98e1e673f2dd9e223811832be7a4344ddfe48282a2e7efbf1289e47e4322ce" dependencies = [ "bon", "bytes", @@ -3968,7 +4001,7 @@ checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -3997,7 +4030,7 @@ checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" [[package]] name = "e2e_test" -version = "1.0.1" +version = "1.0.2" dependencies = [ "anyhow", "async-compression", @@ -4051,7 +4084,7 @@ dependencies = [ "serde_json", "serde_urlencoded", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "suppaftp", "tempfile", "time", @@ -4064,7 +4097,7 @@ dependencies = [ "uuid", "walkdir", "zip", - "zstd 0.14.0", + "zstd 0.14.2", ] [[package]] @@ -4141,7 +4174,7 @@ dependencies = [ "ed25519 3.0.0", "rand_core 0.10.1", "serde", - "sha2 0.11.0", + "sha2 0.11.1", "signature 3.0.0", "subtle", "zeroize", @@ -4277,9 +4310,9 @@ dependencies = [ [[package]] name = "equivalent" -version = "1.0.2" +version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" +checksum = "00d174d5400e5e8fd687ad1049e2f578285fa914201b1af7e8b112a4546bd826" [[package]] name = "erased-serde" @@ -4338,7 +4371,7 @@ version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6d8d873f78e1d3e1998ed2084da9147e3cd2bb34b67814cd49a3902b970eb124" dependencies = [ - "defmt 0.3.100", + "defmt 0.3.101", "serde", ] @@ -4579,7 +4612,7 @@ checksum = "9fb9654ba8355388abeb8dcb4fc62f511300867002afc858860463bdd9fe0c44" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -4762,7 +4795,7 @@ dependencies = [ "rustls-pki-types", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "thiserror 2.0.21", "time", "tokio", @@ -4924,7 +4957,7 @@ dependencies = [ "serde", "serde_json", "serde_with", - "sha2 0.11.0", + "sha2 0.11.1", "thiserror 2.0.21", "tokio", "tokio-stream", @@ -5297,9 +5330,9 @@ dependencies = [ [[package]] name = "hotpath" -version = "0.28.5" +version = "0.28.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "34aabdeba3c81cef2075a2961a370ebfe1d9cc3f8d2242fd400ce6ef6f3bbfcb" +checksum = "e2ee4f8d17ebe2400fa845f079acf9ede235be6472512876779af45f7890f934" dependencies = [ "arc-swap", "async-channel", @@ -5332,15 +5365,15 @@ dependencies = [ [[package]] name = "hotpath-drain" -version = "0.28.5" +version = "0.28.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b4b61a4a77e6563bdc37f831cd6aa5e2089429b62ea377dd759bbabafc207d7a" +checksum = "fd9e65572a429f44a48c360559a3f59028579b53e1df07c5f31dc147d2b94a50" [[package]] name = "hotpath-macros" -version = "0.28.5" +version = "0.28.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c6275c8d1931d280bd505debb7ef91dbeae2c23477aabbfe10487b7430bdeb3" +checksum = "2b90aafbaf5f6af01ee9db2888644b15995f5306e5c280aec8261feb3e5e214a" dependencies = [ "proc-macro2", "quote", @@ -5674,9 +5707,9 @@ dependencies = [ [[package]] name = "inout" -version = "0.2.2" +version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4250ce6452e92010fdf7268ccc5d14faa80bb12fc741938534c58f16804e03c7" +checksum = "3ce983bb6fb399067d101fbfc9e74174e07e9f29a2bc679efefeb8c645819c3f" dependencies = [ "block-padding 0.4.2", "hybrid-array", @@ -5819,7 +5852,7 @@ version = "0.2.38" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b2b005715dcbeb0089a3c0dab99f2ff1cc3b2525323552703d648585d342a383" dependencies = [ - "defmt 1.1.1", + "defmt 1.2.0", "jiff-core", "jiff-static", "jiff-tzdb-platform", @@ -5836,7 +5869,7 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5e52fe76043ccecc9005d2305ebaadf7d7fc0cc89ca6baa10a94d6bc68c7128c" dependencies = [ - "defmt 1.1.1", + "defmt 1.2.0", "log", ] @@ -6019,9 +6052,9 @@ checksum = "a4933f3f57a8e9d9da04db23fb153356ecaf00cbd14aee46279c33dc80925c37" [[package]] name = "lapin" -version = "4.12.0" +version = "4.12.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b86b2a46f42a1d565c7ba327e03725708c1ff212baa4c271cec291e265ae542" +checksum = "3a585eb3060f00f09a1d231e3030afe59f2c2d63f794787ab1dd87d2f72df6a3" dependencies = [ "amq-protocol", "async-rs", @@ -6055,7 +6088,7 @@ dependencies = [ "proc-macro2", "quote", "regex", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -6347,11 +6380,11 @@ dependencies = [ [[package]] name = "lzma-rust2" -version = "0.16.5" +version = "0.21.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ca93e534d1142d1d0dcca6d25fe302508a5dfb40b302802904577725ea0b695b" +checksum = "fde178a3caf126c440fa15628147772d8ee590a39477d711353fbe2e58a73a5b" dependencies = [ - "sha2 0.11.0", + "sha2 0.11.1", ] [[package]] @@ -6534,7 +6567,7 @@ dependencies = [ "metrique-core", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -6597,7 +6630,7 @@ dependencies = [ "metrique-core", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", "synstructure 0.14.0", ] @@ -7224,9 +7257,7 @@ version = "0.39.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2e5a6c098c7a3b6547378093f5cc30bc54fd361ce711e05293a5cc589562739b" dependencies = [ - "flate2", "memchr", - "ruzstd", ] [[package]] @@ -7236,9 +7267,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dd229a0361b9d0d4396176e02d65897f487eebeab7caa6d443855ee152ca0b9c" dependencies = [ "crc32fast", + "flate2", "hashbrown 0.17.1", "indexmap 2.14.2", "memchr", + "ruzstd", ] [[package]] @@ -7337,9 +7370,9 @@ checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" [[package]] name = "opentelemetry" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6cdb0b1b267eb9db3331b434ed9ddab10d50e280a9adf9d13e5233e2002b61b5" +checksum = "8c1f1a6ac9c875a4941e4c7c085b61e8f4daa326de03b4701768c3491a6ad9ee" dependencies = [ "futures-core", "futures-sink", @@ -7351,9 +7384,9 @@ dependencies = [ [[package]] name = "opentelemetry-appender-tracing" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2b304e8d8f11326ba366a99ca5c7aaf7cd397b67268a332437175aeb42a40db" +checksum = "1f01f1c53c46ce80e5fd8cb7a646f13766fd8f315a8a5b75b9bad59a08d2c5c7" dependencies = [ "opentelemetry", "tracing", @@ -7364,9 +7397,9 @@ dependencies = [ [[package]] name = "opentelemetry-http" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee2c3625b8aa04209f7e01e513bc8044da9687fa38a9eacf59628ca5f3b87300" +checksum = "302249bd8ee804789faaa7b5b2cbf79bc61cb864714079c644e3828af5b7e811" dependencies = [ "async-trait", "bytes", @@ -7377,9 +7410,9 @@ dependencies = [ [[package]] name = "opentelemetry-otlp" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "699a67345e21962a231955b9059a157e428b219fa5df8efe11f08346fb23a344" +checksum = "0756af24841e367128ef31c831999937b8b5c5197d47914590d8f4e2e88c2e1a" dependencies = [ "flate2", "http 1.5.0", @@ -7395,9 +7428,9 @@ dependencies = [ [[package]] name = "opentelemetry-proto" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "25da1ac11a0aeccf38d7f77ee0348715adaf8340f65ad46c94a02c6b20e2f65d" +checksum = "d576f5dddc3299341bcd85a04be17b2564844c0c9563214f02d85f2524d1afae" dependencies = [ "opentelemetry", "opentelemetry_sdk", @@ -7406,15 +7439,15 @@ dependencies = [ [[package]] name = "opentelemetry-semantic-conventions" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d288b1e1bc3590052fbb8f0f635ef248dd9022cf1fbaeb6582b7ad6081a17b87" +checksum = "7a3cf639fefd3e852869114f9925b578da6dd8e99281a2089bb0e773ead46962" [[package]] name = "opentelemetry-stdout" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f71803f481d3f5ccbe477edb42a78f473613712b2327835911a9ea0ab069824" +checksum = "2afbc6661934787ce79bef1e35d0ffe9d62e945f07f664dc1283ce83292b908a" dependencies = [ "chrono", "opentelemetry", @@ -7423,9 +7456,9 @@ dependencies = [ [[package]] name = "opentelemetry_sdk" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cb39533d9d1c912123efd7d41d7e0c29d16917b60ce15b4c8d87cb1af7f67520" +checksum = "372a912db2974d1990207006fd0e3aa0ef3fd4ad3be43c65cfaadf2206866061" dependencies = [ "futures-channel", "futures-executor", @@ -7544,7 +7577,7 @@ dependencies = [ "elliptic-curve 0.14.1", "primefield", "primeorder 0.14.0", - "sha2 0.11.0", + "sha2 0.11.1", ] [[package]] @@ -7570,7 +7603,7 @@ dependencies = [ "fiat-crypto 0.3.0", "primefield", "primeorder 0.14.0", - "sha2 0.11.0", + "sha2 0.11.1", ] [[package]] @@ -7584,7 +7617,7 @@ dependencies = [ "elliptic-curve 0.14.1", "primefield", "primeorder 0.14.0", - "sha2 0.11.0", + "sha2 0.11.1", ] [[package]] @@ -7610,7 +7643,7 @@ dependencies = [ "futures", "log", "rand 0.10.3", - "sha2 0.11.0", + "sha2 0.11.1", "thiserror 2.0.21", "tokio", "windows", @@ -7648,9 +7681,9 @@ dependencies = [ [[package]] name = "parquet" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff322f54b1a0f9288e614ed1f2d329b380af5476420db19f46ffb865e1163d73" +checksum = "8b3d2ef0fb136e9341570f219ecb168902e76079c6d81ad555f0a4840b1e08a3" dependencies = [ "ahash", "arrow-array", @@ -7999,7 +8032,7 @@ dependencies = [ "pbkdf2 0.13.0", "rand_core 0.10.1", "scrypt 0.12.0", - "sha2 0.11.0", + "sha2 0.11.1", "spki 0.8.1", ] @@ -8117,7 +8150,7 @@ dependencies = [ "md-5 0.11.0", "memchr", "rand 0.10.3", - "sha2 0.11.0", + "sha2 0.11.1", "stringprep", ] @@ -8202,7 +8235,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2bfe0f4c752e450fc2faf62654f1c134747922825d5b04ca717b8874f41a40c0" dependencies = [ "proc-macro2", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -8299,16 +8332,17 @@ dependencies = [ [[package]] name = "proptest" -version = "1.11.0" +version = "1.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4b45fcc2344c680f5025fe57779faef368840d0bd1f42f216291f0dc4ace4744" +checksum = "b8530004ccb15eae51c7e40009fbe317f341f804db54dc033eec1c50be28cfa0" dependencies = [ "bit-set", - "bit-vec 0.8.0", + "bit-vec 0.10.1", "bitflags 2.13.2", + "chacha20", + "core_detect", "num-traits", - "rand 0.9.5", - "rand_chacha 0.9.0", + "rand 0.10.3", "rand_xorshift", "regex-syntax", "rusty-fork", @@ -8495,9 +8529,9 @@ dependencies = [ [[package]] name = "pyroscope" -version = "2.1.1" +version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59b1989a61e856eba4ef903eadad8313f8376e88e7c9cc5d5db8217d832a64d9" +checksum = "357782a4af9b6b43bd3a9caa64dd802967498b1fce1d4e0bda552267d62de7cf" dependencies = [ "aligned-vec", "backtrace", @@ -8508,10 +8542,11 @@ dependencies = [ "log", "memmap2", "nix 0.31.3", - "object 0.39.1", + "object 0.40.0", "once_cell", "prost 0.14.4", "reqwest", + "ruzstd", "serde_json", "smallvec", "spin 0.12.3", @@ -8846,11 +8881,11 @@ dependencies = [ [[package]] name = "rand_xorshift" -version = "0.4.0" +version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "513962919efc330f829edb2535844d1b912b0fbe2ca165d613e4e8788bb05a5a" +checksum = "60aa6af80be32871323012e02e6e65f8a7cc7890931ae421d217ad8fe0df2ccf" dependencies = [ - "rand_core 0.9.5", + "rand_core 0.10.1", ] [[package]] @@ -9037,7 +9072,7 @@ checksum = "92ecd8964f8453721699a1ed72037b0db49ce2f5a5138486ee89bed6f67cdf3a" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -9255,7 +9290,7 @@ dependencies = [ "pkcs1 0.8.0-rc.4", "pkcs8 0.11.0", "rand_core 0.10.1", - "sha2 0.11.0", + "sha2 0.11.1", "signature 3.0.0", "spki 0.8.1", "zeroize", @@ -9345,7 +9380,7 @@ dependencies = [ "ghash", "hex-literal", "hmac 0.13.0", - "inout 0.2.2", + "inout 0.2.4", "keccak", "log", "md5", @@ -9370,7 +9405,7 @@ dependencies = [ "scrypt 0.12.0", "sec1 0.8.1", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "sha3 0.12.0", "signature 3.0.0", "spki 0.8.1", @@ -9430,9 +9465,9 @@ dependencies = [ [[package]] name = "rust-embed" -version = "8.12.0" +version = "8.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e9e7760e252aaba7b09f4be00e36476cf585bdb68a53552ac954cdf504ab4bc9" +checksum = "19afa5b4b6a611de00bd1bdae6ae6f39084c9399f0679c3f52d8469cf335cc23" dependencies = [ "rust-embed-impl", "rust-embed-utils", @@ -9441,9 +9476,9 @@ dependencies = [ [[package]] name = "rust-embed-impl" -version = "8.12.0" +version = "8.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3bcfc4d6f53af43755f7a723e4b6b8794fcce052a178dd8c6c1dadc5f5343097" +checksum = "e0d8afda6374eac59e066abee06d265247ebbaf3006cf878e2879e8356e34053" dependencies = [ "mime_guess", "proc-macro2", @@ -9456,11 +9491,11 @@ dependencies = [ [[package]] name = "rust-embed-utils" -version = "8.12.0" +version = "8.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42ffa149f6aa81b58a5b3011d01a857c4ed12c7a732d2c51947a4c7c692185f0" +checksum = "0d84e8ba78bd384263e5922f084cbe1b081c3b7e69add59c8fb097b879ba968a" dependencies = [ - "sha2 0.11.0", + "sha2 0.11.1", "walkdir", ] @@ -9487,7 +9522,7 @@ dependencies = [ [[package]] name = "rustfs" -version = "1.0.1" +version = "1.0.2" dependencies = [ "aes-gcm", "anyhow", @@ -9613,7 +9648,7 @@ dependencies = [ "serde_json", "serde_urlencoded", "serial_test", - "sha2 0.11.0", + "sha2 0.11.1", "shadow-rs", "snap", "socket2", @@ -9641,12 +9676,12 @@ dependencies = [ "xxhash-rust", "zeroize", "zip", - "zstd 0.14.0", + "zstd 0.14.2", ] [[package]] name = "rustfs-audit" -version = "1.0.1" +version = "1.0.2" dependencies = [ "const-str", "futures", @@ -9668,7 +9703,7 @@ dependencies = [ [[package]] name = "rustfs-checksums" -version = "1.0.1" +version = "1.0.2" dependencies = [ "base64-simd", "bytes", @@ -9678,14 +9713,15 @@ dependencies = [ "md-5 0.11.0", "pretty_assertions", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "xxhash-rust", ] [[package]] name = "rustfs-common" -version = "1.0.1" +version = "1.0.2" dependencies = [ + "arc-swap", "hotpath", "metrics", "smallvec", @@ -9698,7 +9734,7 @@ dependencies = [ [[package]] name = "rustfs-concurrency" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "insta", @@ -9711,7 +9747,7 @@ dependencies = [ [[package]] name = "rustfs-config" -version = "1.0.1" +version = "1.0.2" dependencies = [ "const-str", "hotpath", @@ -9721,19 +9757,19 @@ dependencies = [ [[package]] name = "rustfs-connect-symbol-catalog" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hex-simd", "object 0.40.0", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "symbolic-demangle", ] [[package]] name = "rustfs-credentials" -version = "1.0.1" +version = "1.0.2" dependencies = [ "base64-simd", "hmac 0.13.0", @@ -9741,13 +9777,13 @@ dependencies = [ "rand 0.10.3", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "time", ] [[package]] name = "rustfs-crypto" -version = "1.0.1" +version = "1.0.2" dependencies = [ "aes-gcm", "argon2", @@ -9760,7 +9796,7 @@ dependencies = [ "rsa 0.10.0-rc.18", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "test-case", "thiserror 2.0.21", "time", @@ -9768,7 +9804,7 @@ dependencies = [ [[package]] name = "rustfs-data-usage" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "rmp-serde", @@ -9778,7 +9814,7 @@ dependencies = [ [[package]] name = "rustfs-ecstore" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "async-channel", @@ -9801,6 +9837,7 @@ dependencies = [ "futures-util", "google-cloud-auth", "google-cloud-storage", + "h2", "hex-simd", "hmac 0.13.0", "hostname", @@ -9856,6 +9893,7 @@ dependencies = [ "rustfs-rio", "rustfs-rio-v2", "rustfs-s3-client", + "rustfs-s3-ops", "rustfs-s3-types", "rustfs-scanner-metrics", "rustfs-storage-api", @@ -9868,7 +9906,7 @@ dependencies = [ "serde", "serde_json", "serial_test", - "sha2 0.11.0", + "sha2 0.11.1", "shadow-rs", "smallvec", "temp-env", @@ -9910,7 +9948,7 @@ dependencies = [ [[package]] name = "rustfs-extension-schema" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "serde", @@ -9920,7 +9958,7 @@ dependencies = [ [[package]] name = "rustfs-filemeta" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "base64-simd", @@ -9937,7 +9975,7 @@ dependencies = [ "rustfs-utils", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "tempfile", "thiserror 2.0.21", "time", @@ -9949,7 +9987,7 @@ dependencies = [ [[package]] name = "rustfs-heal" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "base64-simd", @@ -9977,7 +10015,7 @@ dependencies = [ "serde", "serde_json", "serial_test", - "sha2 0.11.0", + "sha2 0.11.1", "temp-env", "tempfile", "thiserror 2.0.21", @@ -9991,7 +10029,7 @@ dependencies = [ [[package]] name = "rustfs-heal-contracts" -version = "1.0.1" +version = "1.0.2" dependencies = [ "serde", "serde_json", @@ -10001,7 +10039,7 @@ dependencies = [ [[package]] name = "rustfs-iam" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "async-trait", @@ -10036,7 +10074,7 @@ dependencies = [ "serde_with", "serial_test", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "subtle", "temp-env", "tempfile", @@ -10045,12 +10083,13 @@ dependencies = [ "tokio", "tokio-util", "tracing", + "tracing-subscriber", "url", ] [[package]] name = "rustfs-io-core" -version = "1.0.1" +version = "1.0.2" dependencies = [ "bytes", "hotpath", @@ -10062,7 +10101,7 @@ dependencies = [ [[package]] name = "rustfs-io-metrics" -version = "1.0.1" +version = "1.0.2" dependencies = [ "criterion", "hotpath", @@ -10092,7 +10131,7 @@ dependencies = [ "pbkdf2 0.13.0", "rand 0.10.3", "rustls", - "sha2 0.11.0", + "sha2 0.11.1", "socket2", "thiserror 2.0.21", "tracing", @@ -10116,7 +10155,7 @@ dependencies = [ "rand 0.10.3", "rustfs-kafka", "rustls", - "sha2 0.11.0", + "sha2 0.11.1", "tokio", "tokio-rustls", "tracing", @@ -10126,7 +10165,7 @@ dependencies = [ [[package]] name = "rustfs-keystone" -version = "1.0.1" +version = "1.0.2" dependencies = [ "bytes", "futures", @@ -10153,7 +10192,7 @@ dependencies = [ [[package]] name = "rustfs-kms" -version = "1.0.1" +version = "1.0.2" dependencies = [ "aes-gcm", "anyhow", @@ -10186,7 +10225,7 @@ dependencies = [ "serde", "serde_ignored", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "subtle", "temp-env", "tempfile", @@ -10203,14 +10242,14 @@ dependencies = [ [[package]] name = "rustfs-license" -version = "1.0.1" +version = "1.0.2" dependencies = [ "thiserror 2.0.21", ] [[package]] name = "rustfs-lifecycle" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "hotpath", @@ -10232,7 +10271,7 @@ dependencies = [ [[package]] name = "rustfs-lock" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "compact_str", @@ -10255,7 +10294,7 @@ dependencies = [ [[package]] name = "rustfs-log-analyzer" -version = "1.0.1" +version = "1.0.2" dependencies = [ "chrono", "flate2", @@ -10263,18 +10302,18 @@ dependencies = [ "regex", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "tar", "tempfile", "thiserror 2.0.21", "walkdir", "zip", - "zstd 0.14.0", + "zstd 0.14.2", ] [[package]] name = "rustfs-madmin" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "http 1.5.0", @@ -10312,7 +10351,7 @@ dependencies = [ [[package]] name = "rustfs-notify" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "async-trait", @@ -10347,7 +10386,7 @@ dependencies = [ [[package]] name = "rustfs-object-capacity" -version = "1.0.1" +version = "1.0.2" dependencies = [ "criterion", "futures", @@ -10366,7 +10405,7 @@ dependencies = [ [[package]] name = "rustfs-object-data-cache" -version = "1.0.1" +version = "1.0.2" dependencies = [ "bytes", "criterion", @@ -10383,7 +10422,7 @@ dependencies = [ [[package]] name = "rustfs-obs" -version = "1.0.1" +version = "1.0.2" dependencies = [ "chrono", "crossbeam-channel", @@ -10441,12 +10480,12 @@ dependencies = [ "tracing-opentelemetry", "tracing-subscriber", "url", - "zstd 0.14.0", + "zstd 0.14.2", ] [[package]] name = "rustfs-policy" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "base64-simd", @@ -10478,7 +10517,7 @@ dependencies = [ [[package]] name = "rustfs-protocols" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-compression", "async-trait", @@ -10521,7 +10560,7 @@ dependencies = [ "serde", "serde_json", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "socket2", "subtle", "tempfile", @@ -10540,7 +10579,7 @@ dependencies = [ [[package]] name = "rustfs-protos" -version = "1.0.1" +version = "1.0.2" dependencies = [ "flatbuffers", "hotpath", @@ -10565,7 +10604,7 @@ dependencies = [ [[package]] name = "rustfs-replication" -version = "1.0.1" +version = "1.0.2" dependencies = [ "byteorder", "bytes", @@ -10583,7 +10622,7 @@ dependencies = [ [[package]] name = "rustfs-rio" -version = "1.0.1" +version = "1.0.2" dependencies = [ "aes-gcm", "arc-swap", @@ -10615,7 +10654,7 @@ dependencies = [ "serde", "serde_json", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "temp-env", "thiserror 2.0.21", "tokio", @@ -10627,7 +10666,7 @@ dependencies = [ [[package]] name = "rustfs-rio-v2" -version = "1.0.1" +version = "1.0.2" dependencies = [ "aes-gcm", "bytes", @@ -10643,14 +10682,14 @@ dependencies = [ "rustfs-utils", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "tokio", "walkdir", ] [[package]] name = "rustfs-s3-client" -version = "1.0.1" +version = "1.0.2" dependencies = [ "base64-simd", "bytes", @@ -10680,7 +10719,7 @@ dependencies = [ "s3s", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "thiserror 2.0.21", "time", "tokio", @@ -10694,7 +10733,7 @@ dependencies = [ [[package]] name = "rustfs-s3-ops" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "rustfs-s3-types", @@ -10702,7 +10741,7 @@ dependencies = [ [[package]] name = "rustfs-s3-types" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "serde", @@ -10711,7 +10750,7 @@ dependencies = [ [[package]] name = "rustfs-s3select-api" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "async-compression", @@ -10747,7 +10786,7 @@ dependencies = [ [[package]] name = "rustfs-s3select-query" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "bytes", @@ -10767,7 +10806,7 @@ dependencies = [ [[package]] name = "rustfs-scanner" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "bytes", @@ -10800,7 +10839,7 @@ dependencies = [ "serde", "serde_json", "serial_test", - "sha2 0.11.0", + "sha2 0.11.1", "temp-env", "tempfile", "thiserror 2.0.21", @@ -10814,7 +10853,7 @@ dependencies = [ [[package]] name = "rustfs-scanner-metrics" -version = "1.0.1" +version = "1.0.2" dependencies = [ "chrono", "jiff", @@ -10829,7 +10868,7 @@ dependencies = [ [[package]] name = "rustfs-security-governance" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "thiserror 2.0.21", @@ -10837,7 +10876,7 @@ dependencies = [ [[package]] name = "rustfs-signer" -version = "1.0.1" +version = "1.0.2" dependencies = [ "base64-simd", "bytes", @@ -10855,7 +10894,7 @@ dependencies = [ [[package]] name = "rustfs-storage-api" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "hotpath", @@ -10870,7 +10909,7 @@ dependencies = [ [[package]] name = "rustfs-targets" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "async-nats", @@ -10906,7 +10945,7 @@ dependencies = [ "s3s", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "snap", "sysinfo", "tempfile", @@ -10924,7 +10963,7 @@ dependencies = [ [[package]] name = "rustfs-test-utils" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "rustfs-data-usage", @@ -10940,7 +10979,7 @@ dependencies = [ [[package]] name = "rustfs-tls-runtime" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "hotpath", @@ -10952,11 +10991,12 @@ dependencies = [ "rustls-pki-types", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "tempfile", "thiserror 2.0.21", "tokio", "tracing", + "x509-parser", ] [[package]] @@ -10978,7 +11018,7 @@ dependencies = [ [[package]] name = "rustfs-trusted-proxies" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "axum", @@ -11015,7 +11055,7 @@ dependencies = [ [[package]] name = "rustfs-utils" -version = "1.0.1" +version = "1.0.2" dependencies = [ "base64-simd", "blake2", @@ -11042,7 +11082,7 @@ dependencies = [ "rustix", "serde", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "siphasher", "snap", "temp-env", @@ -11052,12 +11092,12 @@ dependencies = [ "transform-stream", "url", "windows", - "zstd 0.14.0", + "zstd 0.14.2", ] [[package]] name = "rustfs-zip" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-compression", "futures", @@ -11067,7 +11107,7 @@ dependencies = [ "rustfs-tokio-tar", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "tar-codec", "tar-framing", "thiserror 2.0.21", @@ -11142,9 +11182,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.45" +version = "0.23.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634" +checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf" dependencies = [ "aws-lc-rs", "log", @@ -11172,9 +11212,9 @@ dependencies = [ [[package]] name = "rustls-native-certs" -version = "0.8.4" +version = "0.8.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d" +checksum = "4a073f5dc7469f984c52ad2752b63b0807745133b6de880b7b64c1ac4c48aec4" dependencies = [ "openssl-probe", "rustls-pki-types", @@ -11251,9 +11291,9 @@ dependencies = [ [[package]] name = "ruzstd" -version = "0.8.3" +version = "0.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7c1c839d570d835527c9a5e4db7cb2198683a988cb9d7293fc8674e6bd58fc8" +checksum = "69e4ce4a039f42bf25920b95667e4d789e98749ab11bbb8d2c2936f788421bb5" dependencies = [ "twox-hash", ] @@ -11304,7 +11344,7 @@ dependencies = [ "serde_json", "serde_urlencoded", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "smallvec", "std-next", "subtle", @@ -11368,7 +11408,7 @@ dependencies = [ "jiff", "nom 8.0.0", "serde", - "sha2 0.11.0", + "sha2 0.11.1", "smallvec", "std-next", "subtle", @@ -11433,9 +11473,9 @@ dependencies = [ [[package]] name = "schemars" -version = "1.2.2" +version = "1.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "687274d293b6cdc6e73e0fee520bf2049650090d7164f87672d212a3c530cf4a" +checksum = "ab508826f74a77ca9d5aba6ff19b522583ee3eaf28a19384ff3d0e5835fadf6e" dependencies = [ "dyn-clone", "ref-cast", @@ -11469,7 +11509,7 @@ dependencies = [ "cfg-if", "pbkdf2 0.13.0", "salsa20 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", ] [[package]] @@ -11592,7 +11632,7 @@ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -11607,10 +11647,11 @@ dependencies = [ [[package]] name = "serde_json" -version = "1.0.151" +version = "1.0.154" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6" dependencies = [ + "foldhash 0.2.0", "indexmap 2.14.2", "itoa", "memchr", @@ -11656,7 +11697,7 @@ checksum = "8d3b1629de253c70a0508c3899572da79ca359fdab27c7920ff00406df418906" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -11685,7 +11726,7 @@ dependencies = [ "indexmap 2.14.2", "jiff", "schemars 0.9.0", - "schemars 1.2.2", + "schemars 1.2.3", "serde_core", "serde_json", "serde_with_macros", @@ -11701,7 +11742,7 @@ dependencies = [ "darling 0.24.1", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -11736,7 +11777,7 @@ checksum = "a22144e767da4ddd8416dbf383700542ffd8a5dc493dfecedfe1fe3ad03c98ae" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -11780,9 +11821,9 @@ dependencies = [ [[package]] name = "sha2" -version = "0.11.0" +version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" +checksum = "47d7069beb7d6ac7b9acd1039986e73443f24234f41074da099d6f994ac9ad19" dependencies = [ "cfg-if", "cpufeatures 0.3.1", @@ -11983,9 +12024,9 @@ dependencies = [ [[package]] name = "smallvec" -version = "1.16.2" +version = "1.16.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9395f0f0eee849a9b707b2f06bb92a6a422090e2123bb2ef8e87a0e61892a8e" +checksum = "5b3dc8af474f516a851ff4bd12db780f948b9250ad37211e4eec0bccea54e01b" dependencies = [ "serde", ] @@ -12152,7 +12193,7 @@ dependencies = [ "sec1 0.8.1", "serde", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "signature 3.0.0", "ssh-cipher", "ssh-encoding", @@ -12255,9 +12296,9 @@ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" [[package]] name = "suppaftp" -version = "12.1.1" +version = "12.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b161d71217da0cc91bd09c8f98ac5ad2a5e5b2735c6dfe1c6cb08b4471612f35" +checksum = "2b58fcf921745cca6f0127c9f822069947da8c6dbeff52e9004750bcc448171f" dependencies = [ "async-trait", "chrono", @@ -12323,9 +12364,9 @@ dependencies = [ [[package]] name = "syn" -version = "3.0.6" +version = "3.0.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee" +checksum = "d78c8dee4c7bf0e14673097256fed6142ce9d3b85a408189d07482442145823b" dependencies = [ "proc-macro2", "quote", @@ -12372,7 +12413,7 @@ checksum = "901704edd0dfe137f1987838ee4f259e4e063c31371bdb423f7ae38ec6f77f02" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -12431,9 +12472,9 @@ dependencies = [ [[package]] name = "tar-codec" -version = "0.0.14" +version = "0.0.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ea42eb144d30fcbf32c26dfea8959175bb8335bfb7b18d20c6ed32edecf0551" +checksum = "b9dad615e50d181afea4bafea23dacb8e33ed52dde06503813570634b932d467" dependencies = [ "archive-trait", "tar-framing", @@ -12443,9 +12484,9 @@ dependencies = [ [[package]] name = "tar-framing" -version = "0.0.14" +version = "0.0.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "783223a7a6590be4227cb821e7ce80372575511f67c824921aba0752d8ad5573" +checksum = "563e8181b47c9cdcce569eae3ad431eec71543d6269dd60c7bd2350f70bcd233" dependencies = [ "thiserror 2.0.21", "tokio", @@ -12514,7 +12555,7 @@ dependencies = [ "cfg-if", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -12525,7 +12566,7 @@ checksum = "f4ce35ece947bccac166e1ded639133827a953f69a48ba134d17f91108333082" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", "test-case-core", ] @@ -12566,7 +12607,7 @@ checksum = "fe5197923287db20a58125f0bc85c062f7f2c892de97b18c356f9efb14b28524" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -12681,9 +12722,9 @@ dependencies = [ [[package]] name = "tokio" -version = "1.53.2" +version = "1.53.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e95f91fcc7a621e8b030f6aa23c71fe9838ae2fb4d8118b75602a328f5144044" +checksum = "ce3335fa71841cda333a58d7615b03901380ecf09d59b3296d21f8bbac0dde4e" dependencies = [ "bytes", "libc", @@ -12703,7 +12744,7 @@ checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -12739,7 +12780,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4c2ad44aa0ae96db89c4742212ed41645b2f597311ff6e1945542a4d9fadc2fb" dependencies = [ "rustls", - "sha2 0.11.0", + "sha2 0.11.1", "tokio", "tokio-postgres", "tokio-rustls", @@ -12781,9 +12822,9 @@ dependencies = [ [[package]] name = "tokio-util" -version = "0.7.19" +version = "0.7.20" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "494815d09bf52b5548659851081238f0ca39ff638363907596da739561c62c52" +checksum = "e464cf451ba96ebfc6f9b6542f17ee8b8956e33f1e40d9690624e59d7a7f8a4b" dependencies = [ "bytes", "futures-core", @@ -12819,18 +12860,18 @@ dependencies = [ [[package]] name = "toml_datetime" -version = "1.1.1+spec-1.1.0" +version = "1.1.2+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3165f65f62e28e0115a00b2ebdd37eb6f3b641855f9d636d3cd4103767159ad7" +checksum = "2b86d767906c6c42421dcba507eb9d203e779497710a47782a224bb871653053" dependencies = [ "serde_core", ] [[package]] name = "toml_edit" -version = "0.25.15+spec-1.1.0" +version = "0.25.17+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1340ea94a5856333492c9064b02c778b191dd2c853778d9609debdcdfea3a614" +checksum = "e3641d5bbb5349a79e1020a242d251efbc546ad8048d133958323ce9c40a9c9c" dependencies = [ "indexmap 2.14.2", "toml_datetime", @@ -12840,9 +12881,9 @@ dependencies = [ [[package]] name = "toml_parser" -version = "1.1.3+spec-1.1.0" +version = "1.1.5+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d38ac1cf9b95face32296c0a3ede1fdc270627c9d9c02a7274dd6d960dc4d56" +checksum = "baa693a8032d7e1cada7d0041e96126df243179ff061456783ac7f12bda4744c" dependencies = [ "winnow", ] @@ -13219,9 +13260,9 @@ checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5" [[package]] name = "unicode-ident" -version = "1.0.26" +version = "1.0.27" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954" +checksum = "a2c754d6c33795a1c324727428e5a7dedb5b06195f9890bdbcba760d3e246563" [[package]] name = "unicode-normalization" @@ -13311,9 +13352,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.27.0" +version = "1.28.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97277d36b9c3ace13e58fa6e753f8b0bbbf302a18dd193240d70f9e29681059a" +checksum = "7cc1186384beb7dd8eedea376413fd654937285ea6c9cfbb928dc3043ea4b606" dependencies = [ "getrandom 0.4.3", "js-sys", @@ -13464,7 +13505,7 @@ dependencies = [ "bumpalo", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", "wasm-bindgen-shared", ] @@ -14039,24 +14080,24 @@ checksum = "ec8ebde2db3681e8c9980cc27822030e68752690ddfa9473e739aeb4dbde6d71" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", "synstructure 0.14.0", ] [[package]] name = "zerocopy" -version = "0.8.60" +version = "0.8.63" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6400fc4bb7426f6faac3fb07566122bb18622e3022bd2c36426a92ed489b1dbe" +checksum = "e5fe1f8f1b06191a00962174c61aa5005e0bb391a6d80d07e24d115c01a92ed8" dependencies = [ "zerocopy-derive", ] [[package]] name = "zerocopy-derive" -version = "0.8.60" +version = "0.8.63" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "99c27cc7525bad4df59a45e073c45b2c2aa820efb588234cc23a547ec69a34db" +checksum = "863ad3ac83293fb4d740aedbfdc9240dd8d1a50c1099acd76ce80ce7c7230c7f" dependencies = [ "proc-macro2", "quote", @@ -14080,7 +14121,7 @@ checksum = "f75b4683f6c7f45248d4d64056a24298c6281e0993356d7d1b4a1a962ef10d4a" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", "synstructure 0.14.0", ] @@ -14134,14 +14175,14 @@ checksum = "34df6fc39dbd26ddc9c10e6a2984476e13acce22e64e4487636ef494369225da" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] name = "zip" -version = "8.6.0" +version = "9.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2d04a6b5381502aa6087c94c669499eb1602eb9c5e8198e534de571f7154809b" +checksum = "c2d98561b612a84ae525331a7a094429ef8a5772514ed507ff6db5f54badc971" dependencies = [ "aes 0.9.3", "bzip2", @@ -14154,6 +14195,7 @@ dependencies = [ "indexmap 2.14.2", "lzma-rust2", "memchr", + "nix 0.31.3", "pbkdf2 0.13.0", "ppmd-rust", "sha1 0.11.0", @@ -14161,7 +14203,7 @@ dependencies = [ "typed-path", "zeroize", "zopfli", - "zstd 0.13.3", + "zstd 0.14.2", ] [[package]] @@ -14178,9 +14220,9 @@ checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" [[package]] name = "zopfli" -version = "0.8.3" +version = "0.8.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f05cd8797d63865425ff89b5c4a48804f35ba0ce8d125800027ad6017d2b5249" +checksum = "aaf7fc5d30c28483d93805c4a5e12b05bbb52407fa67c5f8bd552374cd01fb11" dependencies = [ "bumpalo", "crc32fast", @@ -14199,11 +14241,11 @@ dependencies = [ [[package]] name = "zstd" -version = "0.14.0" +version = "0.14.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf06bd8162af0734b344780deb55b42a2429ae430870d13fcc12f238e880fe6e" +checksum = "057cfd910cfac363a0ada849592624b4c9ff2e10bef504c3433810d78ed96f93" dependencies = [ - "zstd-safe 8.0.0", + "zstd-safe 8.1.0", ] [[package]] @@ -14217,18 +14259,18 @@ dependencies = [ [[package]] name = "zstd-safe" -version = "8.0.0" +version = "8.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae42c0555055784c70058d19ba8e275528e8a99a706684868ace5da4e716a4ab" +checksum = "bdd44c6a7284e91f3717755b24315a302edd9153a01f753c3cba3d765e8eafac" dependencies = [ "zstd-sys", ] [[package]] name = "zstd-sys" -version = "2.1.0+zstd.1.5.7" +version = "2.1.1+zstd.1.5.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ef0a8027ec3ee71300ab3bcbcd0393f434aa72b91ca6d635a39941deae8eea0" +checksum = "aeec9eaf2dffbbd09201e23bd0ffcbaa33bb8e9266a10734fd7ed90a85eca078" dependencies = [ "cc", "pkg-config", diff --git a/Cargo.toml b/Cargo.toml index 6a5791efb..a87735faa 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -74,7 +74,7 @@ edition = "2024" license = "Apache-2.0" repository = "https://github.com/rustfs/rustfs" rust-version = "1.99.0" -version = "1.0.1" +version = "1.0.2" homepage = "https://rustfs.com" description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. " keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"] @@ -91,56 +91,56 @@ redundant_clone = "warn" [workspace.dependencies] # RustFS Internal Crates -rustfs = { path = "./rustfs", version = "1.0.1" } -rustfs-heal = { path = "crates/heal", version = "1.0.1" } -rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.1" } -rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.1" } -rustfs-audit = { path = "crates/audit", version = "1.0.1" } -rustfs-checksums = { path = "crates/checksums", version = "1.0.1" } -rustfs-common = { path = "crates/common", version = "1.0.1" } -rustfs-data-usage = { path = "crates/data-usage", version = "1.0.1" } -rustfs-config = { path = "./crates/config", version = "1.0.1" } -rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.1" } -rustfs-credentials = { path = "crates/credentials", version = "1.0.1" } -rustfs-crypto = { path = "crates/crypto", version = "1.0.1" } -rustfs-ecstore = { path = "crates/ecstore", version = "1.0.1" } -rustfs-filemeta = { path = "crates/filemeta", version = "1.0.1" } -rustfs-iam = { path = "crates/iam", version = "1.0.1" } -rustfs-keystone = { path = "crates/keystone", version = "1.0.1" } -rustfs-license = { path = "crates/license", version = "1.0.1" } -rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.1" } -rustfs-kms = { path = "crates/kms", version = "1.0.1" } -rustfs-lock = { path = "crates/lock", version = "1.0.1" } -rustfs-madmin = { path = "crates/madmin", version = "1.0.1" } -rustfs-notify = { path = "crates/notify", version = "1.0.1" } -rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.1" } -rustfs-io-core = { path = "crates/io-core", version = "1.0.1" } -rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.1" } -rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.1", default-features = false } -rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.1" } -rustfs-obs = { path = "crates/obs", version = "1.0.1" } -rustfs-policy = { path = "crates/policy", version = "1.0.1" } -rustfs-protos = { path = "crates/protos", version = "1.0.1" } -rustfs-protocols = { path = "crates/protocols", version = "1.0.1" } -rustfs-replication = { path = "crates/replication", version = "1.0.1" } -rustfs-rio = { path = "crates/rio", version = "1.0.1" } -rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.1" } -rustfs-s3-client = { path = "crates/s3-client", version = "1.0.1" } -rustfs-s3-types = { path = "crates/s3-types", version = "1.0.1" } -rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.1" } -rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.1" } -rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.1" } -rustfs-scanner = { path = "crates/scanner", version = "1.0.1" } -rustfs-security-governance = { path = "crates/security-governance", version = "1.0.1" } -rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.1" } -rustfs-signer = { path = "crates/signer", version = "1.0.1" } -rustfs-storage-api = { path = "crates/storage-api", version = "1.0.1" } -rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.1" } -rustfs-targets = { path = "crates/targets", version = "1.0.1" } -rustfs-test-utils = { path = "crates/test-utils", version = "1.0.1" } -rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.1" } -rustfs-utils = { path = "crates/utils", version = "1.0.1" } -rustfs-zip = { path = "./crates/zip", version = "1.0.1" } +rustfs = { path = "./rustfs", version = "1.0.2" } +rustfs-heal = { path = "crates/heal", version = "1.0.2" } +rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.2" } +rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.2" } +rustfs-audit = { path = "crates/audit", version = "1.0.2" } +rustfs-checksums = { path = "crates/checksums", version = "1.0.2" } +rustfs-common = { path = "crates/common", version = "1.0.2" } +rustfs-data-usage = { path = "crates/data-usage", version = "1.0.2" } +rustfs-config = { path = "./crates/config", version = "1.0.2" } +rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.2" } +rustfs-credentials = { path = "crates/credentials", version = "1.0.2" } +rustfs-crypto = { path = "crates/crypto", version = "1.0.2" } +rustfs-ecstore = { path = "crates/ecstore", version = "1.0.2" } +rustfs-filemeta = { path = "crates/filemeta", version = "1.0.2" } +rustfs-iam = { path = "crates/iam", version = "1.0.2" } +rustfs-keystone = { path = "crates/keystone", version = "1.0.2" } +rustfs-license = { path = "crates/license", version = "1.0.2" } +rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.2" } +rustfs-kms = { path = "crates/kms", version = "1.0.2" } +rustfs-lock = { path = "crates/lock", version = "1.0.2" } +rustfs-madmin = { path = "crates/madmin", version = "1.0.2" } +rustfs-notify = { path = "crates/notify", version = "1.0.2" } +rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.2" } +rustfs-io-core = { path = "crates/io-core", version = "1.0.2" } +rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.2" } +rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.2", default-features = false } +rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.2" } +rustfs-obs = { path = "crates/obs", version = "1.0.2" } +rustfs-policy = { path = "crates/policy", version = "1.0.2" } +rustfs-protos = { path = "crates/protos", version = "1.0.2" } +rustfs-protocols = { path = "crates/protocols", version = "1.0.2" } +rustfs-replication = { path = "crates/replication", version = "1.0.2" } +rustfs-rio = { path = "crates/rio", version = "1.0.2" } +rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.2" } +rustfs-s3-client = { path = "crates/s3-client", version = "1.0.2" } +rustfs-s3-types = { path = "crates/s3-types", version = "1.0.2" } +rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.2" } +rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.2" } +rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.2" } +rustfs-scanner = { path = "crates/scanner", version = "1.0.2" } +rustfs-security-governance = { path = "crates/security-governance", version = "1.0.2" } +rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.2" } +rustfs-signer = { path = "crates/signer", version = "1.0.2" } +rustfs-storage-api = { path = "crates/storage-api", version = "1.0.2" } +rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.2" } +rustfs-targets = { path = "crates/targets", version = "1.0.2" } +rustfs-test-utils = { path = "crates/test-utils", version = "1.0.2" } +rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" } +rustfs-utils = { path = "crates/utils", version = "1.0.2" } +rustfs-zip = { path = "./crates/zip", version = "1.0.2" } # Async Runtime and Networking async-channel = "2.5.0" @@ -155,9 +155,10 @@ futures-core = "0.3.34" futures-lite = "2.6.1" futures-util = "0.3.34" pollster = "1.0.1" -pulsar = { default-features = false, version = "6.9.0" } -lapin = { default-features = false, version = "4.12.0" } +pulsar = { default-features = false, version = "=6.9.0" } +lapin = { default-features = false, version = "4.12.2" } hyper = { version = "1.12.0" } +h2 = { version = "0.4.20" } hyper-rustls = { default-features = false, version = "0.27.10" } hyper-util = { version = "0.1.21" } http = "1.5.0" @@ -168,11 +169,11 @@ minlz = "1.2.3" reqwest = "0.13.5" rustfs-kafka-async = { version = "1.4.0" } socket2 = { version = "0.6.5" } -tokio = { version = "1.53.2" } +tokio = { version = "1.53.3" } tokio-rustls = { default-features = false, version = "0.26.6" } tokio-stream = { version = "0.1.19" } tokio-test = "0.4.6" -tokio-util = { version = "0.7.19" } +tokio-util = { version = "0.7.20" } tonic = { version = "0.14.6" } tonic-prost = { version = "0.14.6" } tonic-prost-build = { version = "0.14.6" } @@ -180,7 +181,7 @@ tower = { version = "0.5.3" } tower-http = { version = "0.7.1" } h3 = "0.0.8" h3-quinn = "0.0.10" -quinn = { version = "0.11.11", default-features = false, features = [ +quinn = { version = "0.11.12", default-features = false, features = [ "futures-io", "runtime-tokio", "rustls-aws-lc-rs", @@ -200,7 +201,7 @@ rmp-serde = { version = "1.3.1" } serde = { version = "1.0.229" } serde_ignored = { version = "0.1" } serde_with = { version = "3", default-features = false, features = ["macros", "std"] } -serde_json = { version = "1.0.151" } +serde_json = { version = "1.0.154" } serde_urlencoded = "0.7.1" # Cryptography and Security @@ -219,12 +220,12 @@ openidconnect = { default-features = false, version = "4.0" } pbkdf2 = "0.13.0" p256 = { version = "0.14.0", features = ["ecdsa", "pkcs8"] } rsa = { version = "=0.10.0-rc.18" } -rustls = { default-features = false, version = "0.23.45" } +rustls = { default-features = false, version = "0.23.46" } rustls-native-certs = "0.8" rustls-pki-types = "1.15.1" x509-parser = "0.18.1" sha1 = "0.11.0" -sha2 = "0.11.0" +sha2 = "0.11.1" subtle = "2.6" zeroize = { version = "1.9.1" } proptest = "1" @@ -245,8 +246,8 @@ anyhow = "1.0.104" arc-swap = "1.9.2" rustfs-tokio-tar = { version = "0.7.1" } # Candidate Snowball parser versions exercised by rustfs-zip compatibility fixtures. -tar-codec = "0.0.14" -tar-framing = "0.0.14" +tar-codec = "0.0.15" +tar-framing = "0.0.15" atoi = "3.1.0" atomic_enum = "0.3.0" aws-config = { version = "1.12.0" } @@ -268,7 +269,7 @@ crossbeam-queue = "0.3.14" crossbeam-channel = "0.5.17" crossbeam-deque = "0.8.8" crossbeam-utils = "0.8.23" -datafusion = { default-features = false, version = "55.1.0" } +datafusion = { default-features = false, version = "55.2.0" } derive_builder = "0.20.2" ed25519-dalek = "3.0.0" enumset = "1.1.14" @@ -319,14 +320,14 @@ rumqttc = { package = "rumqttc-next", version = "0.34.0" } redis = { version = "1.7.1" } rustify = { version = "0.7", default-features = false } rustix = { version = "1.1.5" } -rust-embed = { version = "8.12.0" } +rust-embed = { version = "8.13.0" } rustc-hash = { version = "2.1.3" } s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] } s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" } serial_test = "4.0.1" shadow-rs = { default-features = false, version = "2.0.0" } siphasher = "1.0.4" -smallvec = { version = "1.16.2" } +smallvec = { version = "1.16.3" } compact_str = "0.10.0" snap = "1.1.2" starshard = { version = "2.4.0" } @@ -340,12 +341,12 @@ tracing = { version = "0.1.44" } tracing-appender = "0.2.5" tracing-core = "0.1.36" tracing-error = "0.2.1" -tracing-opentelemetry = { version = "0.34" } +tracing-opentelemetry = { version = "0.34.0" } tracing-subscriber = { version = "0.3.23" } transform-stream = "0.3.1" url = "2.5.8" urlencoding = "2.1.3" -uuid = { version = "1.27.0" } +uuid = { version = "1.28.0" } vaultrs = { version = "0.8.0" } tar = "0.4.46" walkdir = "2.5.0" @@ -353,26 +354,26 @@ winapi-util = "0.1.11" windows = { version = "0.62.2" } windows-sys = "0.61.2" xxhash-rust = { version = "0.8.19" } -zip = "8.6.0" -zstd = "0.14.0" +zip = "9.0.3" +zstd = "0.14.2" # Observability and Metrics metrics = "0.24.6" metrics-util = "0.20" -dial9 = { version = "0.5.3", default-features = false } -opentelemetry = { version = "0.33.0" } -opentelemetry-appender-tracing = { version = "0.33.0" } -opentelemetry-otlp = { version = "0.33.0" } -opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] } -opentelemetry_sdk = { version = "0.33.0" } -opentelemetry-semantic-conventions = { version = "0.33.0" } -opentelemetry-stdout = { version = "0.33.0" } -pyroscope = { version = "2.1.1" } +dial9 = { version = "0.5.4", default-features = false } +opentelemetry = { version = "0.33.1" } +opentelemetry-appender-tracing = { version = "0.33.1" } +opentelemetry-otlp = { version = "0.33.1" } +opentelemetry-proto = { version = "0.33.1", default-features = false, features = ["metrics", "gen-tonic-messages"] } +opentelemetry_sdk = { version = "0.33.1" } +opentelemetry-semantic-conventions = { version = "0.33.1" } +opentelemetry-stdout = { version = "0.33.1" } +pyroscope = { version = "2.1.2" } # FTP and SFTP libunftp = { version = "0.23.1" } unftp-core = "0.1.0" -suppaftp = { version = "12.1.1" } +suppaftp = { version = "12.2.1" } rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] } russh = { version = "0.64.1" } russh-sftp = "3.0.1" @@ -383,7 +384,7 @@ dav-server = "0.11.0" # Performance Analysis and Memory Profiling rustfs-mimalloc = { version = "0.5.6" } # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved. -hotpath = { version = "0.28.5", default-features = false } +hotpath = { version = "0.28.6", default-features = false } # Snapshot testing for output format regression detection insta = { version = "1.49" } diff --git a/README.md b/README.md index bfbeac208..74093efc2 100644 --- a/README.md +++ b/README.md @@ -141,7 +141,7 @@ chown -R 10001:10001 data logs docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest # Using specific version -docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1 +docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.3 ``` If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command @@ -220,7 +220,7 @@ For developers who want to build RustFS Docker images from source with multi-arc ./docker-buildx.sh --push # Build specific version -./docker-buildx.sh --release 1.0.1 --push +./docker-buildx.sh --release 1.0.3 --push # Build for custom registry ./docker-buildx.sh --registry your-registry.com --namespace yourname --push @@ -238,7 +238,7 @@ You can also use Make targets for convenience: ```bash make docker-buildx # Build locally make docker-buildx-push # Build and push -make docker-buildx-version VERSION=1.0.1 # Build specific version +make docker-buildx-version VERSION=1.0.3 # Build specific version make help-docker # Show all Docker-related commands ``` diff --git a/README_ZH.md b/README_ZH.md index 0d2fcee43..9ca9cedec 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -138,7 +138,7 @@ chown -R 10001:10001 data logs docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest # 使用指定版本运行 -docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1 +docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.3 ``` 如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录: @@ -176,7 +176,7 @@ docker compose -f docker-compose-simple.yml up -d ./docker-buildx.sh --push # 构建指定版本 -./docker-buildx.sh --release 1.0.1 --push +./docker-buildx.sh --release 1.0.3 --push # 构建并推送到自定义仓库 ./docker-buildx.sh --registry your-registry.com --namespace yourname --push @@ -193,7 +193,7 @@ docker compose -f docker-compose-simple.yml up -d ```bash make docker-buildx # 本地构建 make docker-buildx-push # 构建并推送 -make docker-buildx-version VERSION=1.0.1 # 构建指定版本 +make docker-buildx-version VERSION=1.0.3 # 构建指定版本 make help-docker # 显示所有 Docker 相关命令 ``` diff --git a/_typos.toml b/_typos.toml index 7ff843137..e11c3c036 100644 --- a/_typos.toml +++ b/_typos.toml @@ -51,6 +51,8 @@ unparseable = "unparseable" # Disaster-recovery objectives: recovery time and recovery point. RTO = "RTO" rto = "rto" +# rustix OFlags variant (POSIX O_WRONLY equivalent). +WRONLY = "WRONLY" [files] extend-exclude = [] diff --git a/crates/common/Cargo.toml b/crates/common/Cargo.toml index 3749c929d..b772c459d 100644 --- a/crates/common/Cargo.toml +++ b/crates/common/Cargo.toml @@ -34,6 +34,7 @@ hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"] hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"] [dependencies] +arc-swap.workspace = true hotpath.workspace = true tokio = { workspace = true, features = ["fs", "rt-multi-thread"] } tonic = { workspace = true, features = ["gzip", "deflate"] } diff --git a/crates/common/src/globals.rs b/crates/common/src/globals.rs index e4cd89c88..03814d454 100644 --- a/crates/common/src/globals.rs +++ b/crates/common/src/globals.rs @@ -12,9 +12,10 @@ // See the License for the specific language governing permissions and // limitations under the License. +use arc_swap::ArcSwap; use std::collections::HashMap; -use std::sync::LazyLock; use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, LazyLock}; use tokio::sync::RwLock; use tonic::transport::Channel; @@ -27,6 +28,42 @@ pub static GLOBAL_ROOT_CERT: LazyLock>>> = LazyLock::new(| pub static GLOBAL_MTLS_IDENTITY: LazyLock>> = LazyLock::new(|| RwLock::new(None)); pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock = LazyLock::new(|| AtomicU64::new(0)); +#[derive(Clone, Default)] +pub struct OutboundTlsSnapshot { + pub generation: u64, + pub root_ca_pem: Option>, + pub mtls_identity: Option, +} + +static GLOBAL_OUTBOUND_TLS_STATE: LazyLock> = + LazyLock::new(|| ArcSwap::from_pointee(OutboundTlsSnapshot::default())); + +pub fn get_global_outbound_tls_snapshot() -> Arc { + GLOBAL_OUTBOUND_TLS_STATE.load_full() +} + +/// Publish identity, trust and generation together; the individual globals are legacy views. +pub async fn publish_global_outbound_tls_snapshot(snapshot: OutboundTlsSnapshot) -> u64 { + GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| { + if current.generation != 0 + && current.root_ca_pem == snapshot.root_ca_pem + && current.mtls_identity == snapshot.mtls_identity + { + return Arc::clone(current); + } + Arc::new(OutboundTlsSnapshot { + generation: snapshot.generation.max(current.generation.saturating_add(1)), + root_ca_pem: snapshot.root_ca_pem.clone(), + mtls_identity: snapshot.mtls_identity.clone(), + }) + }); + let accepted = get_global_outbound_tls_snapshot(); + *GLOBAL_ROOT_CERT.write().await = accepted.root_ca_pem.clone(); + *GLOBAL_MTLS_IDENTITY.write().await = accepted.mtls_identity.clone(); + GLOBAL_OUTBOUND_TLS_GENERATION.store(accepted.generation, Ordering::Relaxed); + accepted.generation +} + /// Log level to use when reporting cached gRPC connection eviction. #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub enum ConnectionEvictionLogLevel { @@ -85,17 +122,27 @@ pub async fn get_global_addr() -> String { /// # Arguments /// * `cert` - A vector of bytes representing the PEM-encoded root CA certificate. pub async fn set_global_root_cert(cert: Vec) { + GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| { + let mut next = (**current).clone(); + next.root_ca_pem = Some(cert.clone()); + Arc::new(next) + }); *GLOBAL_ROOT_CERT.write().await = Some(cert); } /// Clear the global root CA certificate for outbound gRPC clients. pub async fn clear_global_root_cert() { + GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| { + let mut next = (**current).clone(); + next.root_ca_pem = None; + Arc::new(next) + }); *GLOBAL_ROOT_CERT.write().await = None; } /// Get the global root CA certificate for outbound gRPC clients. pub async fn get_global_root_cert() -> Option> { - GLOBAL_ROOT_CERT.read().await.clone() + get_global_outbound_tls_snapshot().root_ca_pem.clone() } /// Set the global mTLS identity (cert+key PEM) for outbound gRPC clients. @@ -105,22 +152,32 @@ pub async fn get_global_root_cert() -> Option> { /// # Arguments /// * `identity` - An optional MtlsIdentityPem struct containing the cert and key PEM. pub async fn set_global_mtls_identity(identity: Option) { + GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| { + let mut next = (**current).clone(); + next.mtls_identity = identity.clone(); + Arc::new(next) + }); *GLOBAL_MTLS_IDENTITY.write().await = identity; } /// Get the global mTLS identity for outbound gRPC clients. pub async fn get_global_mtls_identity() -> Option { - GLOBAL_MTLS_IDENTITY.read().await.clone() + get_global_outbound_tls_snapshot().mtls_identity.clone() } /// Set the global outbound TLS generation. pub fn set_global_outbound_tls_generation(generation: u64) { + GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| { + let mut next = (**current).clone(); + next.generation = generation; + Arc::new(next) + }); GLOBAL_OUTBOUND_TLS_GENERATION.store(generation, Ordering::Relaxed); } /// Get the global outbound TLS generation. pub fn get_global_outbound_tls_generation() -> u64 { - GLOBAL_OUTBOUND_TLS_GENERATION.load(Ordering::Relaxed) + get_global_outbound_tls_snapshot().generation } /// Evict a stale/dead connection from the global connection cache. @@ -194,8 +251,77 @@ pub async fn clear_all_connections() { /// /// When present, gRPC clients will present this identity to servers requesting/requiring mTLS. /// When absent, clients proceed with standard server-authenticated TLS. -#[derive(Clone, Debug)] +#[derive(Clone, PartialEq, Eq)] pub struct MtlsIdentityPem { pub cert_pem: Vec, pub key_pem: Vec, } + +impl std::fmt::Debug for MtlsIdentityPem { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("MtlsIdentityPem") + .field("cert_bytes", &self.cert_pem.len()) + .field("key_pem", &"[REDACTED]") + .finish() + } +} + +#[cfg(test)] +mod tls_snapshot_tests { + use super::*; + + #[tokio::test] + async fn outbound_snapshot_publication_is_coherent_and_skips_unchanged_material() { + let start = get_global_outbound_tls_generation().saturating_add(1); + let snapshot = OutboundTlsSnapshot { + generation: start, + root_ca_pem: Some(vec![1]), + mtls_identity: Some(MtlsIdentityPem { + cert_pem: vec![1], + key_pem: vec![1], + }), + }; + let first = publish_global_outbound_tls_snapshot(snapshot.clone()).await; + assert_eq!( + publish_global_outbound_tls_snapshot(OutboundTlsSnapshot { + generation: first.saturating_add(1), + ..snapshot + }) + .await, + first + ); + let writer = tokio::spawn(async move { + for tag in 2_u8..=255 { + publish_global_outbound_tls_snapshot(OutboundTlsSnapshot { + generation: start.saturating_add(u64::from(tag)), + root_ca_pem: Some(vec![tag]), + mtls_identity: Some(MtlsIdentityPem { + cert_pem: vec![tag], + key_pem: vec![tag], + }), + }) + .await; + tokio::task::yield_now().await; + } + }); + while !writer.is_finished() { + let state = get_global_outbound_tls_snapshot(); + let identity = state.mtls_identity.as_ref().expect("identity in complete snapshot"); + assert_eq!(state.root_ca_pem.as_ref().expect("roots in complete snapshot"), &identity.cert_pem); + assert_eq!(identity.cert_pem, identity.key_pem); + tokio::task::yield_now().await; + } + writer.await.expect("writer completes"); + } + + #[test] + fn mtls_identity_debug_redacts_private_key() { + let identity = MtlsIdentityPem { + cert_pem: b"public certificate".to_vec(), + key_pem: b"PRIVATE-KEY-MARKER".to_vec(), + }; + let debug = format!("{identity:?}"); + assert!(!debug.contains("PRIVATE-KEY-MARKER")); + assert!(debug.contains("[REDACTED]")); + } +} diff --git a/crates/config/README.md b/crates/config/README.md index 81600cff9..fadcbac0d 100644 --- a/crates/config/README.md +++ b/crates/config/README.md @@ -119,11 +119,19 @@ when restoring `/rustfs/console` for a custom OEM build. - `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` (canonical) - `RUSTFS_SCANNER_CYCLE_MAX_OBJECTS` (canonical) - `RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES` (canonical) +- `RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS` (canonical) +- `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` (canonical) +- `RUSTFS_SCANNER_WALKERS_PER_DISK` (canonical) Scanner cycle budget controls: - When `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` is unset, the finite default is 1800 seconds (30 minutes), matching the scanner benchmark guidance. - An explicit `0` preserves the compatibility behavior of an unbounded runtime budget. Object and directory budgets likewise remain unbounded when explicitly set to `0`. +Scanner walker controls: + +- `RUSTFS_SCANNER_WALKERS_PER_DISK` sets how many bucket walkers each local disk of an erasure set runs; the default `1` walks one bucket per disk at a time, which is the right value for rotational media. On flash, `2` to `4` lets one disk walk several buckets concurrently, which matters most for single-disk deployments with many buckets, where the walk is otherwise fully serial. Values outside `1` through `16` are rejected by `admin config set`; an out-of-range or non-numeric environment value is logged as a scanner runtime config error (`state = "startup_apply_failed"`) and the scanner keeps the default. The walkers of one disk share that disk's scanner admission, so peers still see one scan session per physical disk, and `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` still caps the total. +- The same value is available as the persistent `scanner.walkers_per_disk` key and is reported under `runtime_config.walkers_per_disk` in the admin scanner status; see [Scanner Runtime Controls](../../docs/operations/scanner-runtime-controls.md). + - A timed-out cycle cancels cooperative scanner work, then fences its leader epoch before releasing the lease. An uncooperative I/O operation is dropped after the bounded shutdown window; its cursor is not claimed to be durable and the scanner reports `recovery-required` when the worker cannot stop cooperatively, the cycle state was not confirmed durable, or epoch fencing cannot be persisted. ## Mmap read environment aliases @@ -266,7 +274,7 @@ Drive timeout profile preset: - `warn` (default): log warning and continue startup. - `fail`: abort startup with an error. -RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example `nfs`, `cifs`, `smb2`, and `fuse.*`) are treated as unsupported by this startup guard. +RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example `nfs`, `cifs`, `smb2`, and `fuse.*`) are treated as unsupported by this startup guard. On Windows, endpoints on network drives (UNC shares such as `\\server\share` and mapped drive letters) are treated the same way, even though Windows reports the server's filesystem (usually `NTFS`) for them. ## 📄 License diff --git a/crates/config/src/constants/console.rs b/crates/config/src/constants/console.rs index 617aa020d..a8dd5f764 100644 --- a/crates/config/src/constants/console.rs +++ b/crates/config/src/constants/console.rs @@ -69,15 +69,17 @@ pub const ENV_CONSOLE_RATE_LIMIT_RPM: &str = "RUSTFS_CONSOLE_RATE_LIMIT_RPM"; pub const DEFAULT_CONSOLE_RATE_LIMIT_RPM: u32 = 100; /// Set the console authentication timeout in seconds -/// Specifies how long a console authentication session remains valid +/// Bounds how long the console waits for a single authenticated HTTP request before it is +/// considered timed out. This is a request timeout, not the session lifetime: the Console +/// credential lifetime issued by the OIDC callback is `RUSTFS_CONSOLE_OIDC_SESSION_DURATION`. /// Default: 3600 seconds (1 hour) /// Minimum: 300 seconds (5 minutes) /// Maximum: 86400 seconds (24 hours) pub const ENV_CONSOLE_AUTH_TIMEOUT: &str = "RUSTFS_CONSOLE_AUTH_TIMEOUT"; /// Default console authentication timeout in seconds -/// This is the default timeout for console authentication sessions. -/// After this timeout, users need to re-authenticate to access the console. +/// This is the default timeout for console authentication requests. +/// After this timeout, a pending console request is abandoned. /// Default value: 3600 seconds (1 hour) /// Environment variable: RUSTFS_CONSOLE_AUTH_TIMEOUT /// Command line argument: --console-auth-timeout @@ -85,6 +87,26 @@ pub const ENV_CONSOLE_AUTH_TIMEOUT: &str = "RUSTFS_CONSOLE_AUTH_TIMEOUT"; /// Example: --console-auth-timeout 3600 pub const DEFAULT_CONSOLE_AUTH_TIMEOUT: u64 = 3600; +/// Console credential lifetime, in seconds, for Console credentials issued by the OIDC callback. +/// +/// Process-level, Console-scoped setting: it applies to every OIDC provider and deliberately +/// lives outside the per-provider `RUSTFS_IDENTITY_OPENID_*` namespace, so it is never +/// suffix-scanned into a provider id and has no `identity_openid` KVS equivalent. The same value +/// also sets how long the Console logout token stays valid. +pub const ENV_CONSOLE_OIDC_SESSION_DURATION: &str = "RUSTFS_CONSOLE_OIDC_SESSION_DURATION"; + +/// Default Console credential lifetime in seconds, used when the environment variable is unset, +/// not a positive integer, or zero. Matches the value this path hard-coded before it was +/// parameterised. +pub const DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS: usize = 3600; + +/// Upper bound on the Console credential lifetime in seconds (7 days). +/// +/// Values above this are clamped with a warning. The bound keeps the moka logout cache's +/// `time_to_live` well under its 1000-year assertion, and stops an operator from minting an +/// effectively permanent Console STS credential on every node. +pub const CONSOLE_OIDC_SESSION_DURATION_MAX_SECS: usize = 604_800; + /// Toggle update check /// It controls whether to check for newer versions of rustfs /// Default value: true diff --git a/crates/config/src/constants/scanner.rs b/crates/config/src/constants/scanner.rs index 8c5b9231b..1465ad29c 100644 --- a/crates/config/src/constants/scanner.rs +++ b/crates/config/src/constants/scanner.rs @@ -59,6 +59,9 @@ pub const SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "max_concurrent_set_scans"; /// Scanner config key capping concurrent scanner disk bucket walks per set. pub const SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "max_concurrent_disk_scans"; +/// Scanner config key setting how many bucket walkers each local disk runs. +pub const SCANNER_WALKERS_PER_DISK: &str = "walkers_per_disk"; + /// Scanner config key controlling how often object loops yield. pub const SCANNER_YIELD_EVERY_N_OBJECTS: &str = "yield_every_n_objects"; @@ -86,6 +89,7 @@ pub const SCANNER_KEYS: &[&str] = &[ SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_CONCURRENT_DISK_SCANS, + SCANNER_WALKERS_PER_DISK, SCANNER_YIELD_EVERY_N_OBJECTS, SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_ALERT_EXCESS_VERSION_SIZE, @@ -214,6 +218,14 @@ pub const ENV_SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCU /// - Example: `export RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS=1` pub const ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS"; +/// Environment variable that sets how many bucket walkers each local disk runs. +/// The walkers of one disk share its scanner admission, so a peer still sees one +/// scan session per physical disk. Remote disks keep one walker. The product of +/// local disks and this factor is what `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` +/// caps; the disk count stays the floor. Valid range is `1` through `16`. +/// - Example: `export RUSTFS_SCANNER_WALKERS_PER_DISK=4` +pub const ENV_SCANNER_WALKERS_PER_DISK: &str = "RUSTFS_SCANNER_WALKERS_PER_DISK"; + /// Environment variable that controls how often scanner object loops yield to the async runtime. /// A value of `0` disables this extra object-count yield. /// - Example: `export RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS=32` @@ -228,6 +240,18 @@ pub const DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS: usize = 4; /// Default disk scan concurrency budget. pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 4; +/// Default bucket walkers per local disk. +/// +/// One walker per disk keeps today's I/O profile, which is what rotational +/// media needs. Flash is meant to default higher once a local disk can report +/// its media type; `rustfs_utils::os::Info::rotational` is never populated +/// today, so every deployment starts from one walker. +pub const DEFAULT_SCANNER_WALKERS_PER_DISK: usize = 1; + +/// Upper bound for `walkers_per_disk`; a larger factor only multiplies +/// contention on one spindle or NVMe queue without more disks to spread it. +pub const MAX_SCANNER_WALKERS_PER_DISK: usize = 16; + /// Default object interval for cooperative scanner yields. pub const DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS: u64 = 128; diff --git a/crates/config/src/observability/mod.rs b/crates/config/src/observability/mod.rs index 9a1acc6c4..ca001fc8b 100644 --- a/crates/config/src/observability/mod.rs +++ b/crates/config/src/observability/mod.rs @@ -51,6 +51,11 @@ pub const ENV_OBS_PROFILING_EXPORT_ENABLED: &str = "RUSTFS_OBS_PROFILING_EXPORT_ /// PUT records multiple timers and histograms when attribution is active. pub const ENV_OBS_PUT_STAGE_METRICS_ENABLED: &str = "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED"; +/// Enables sampled in-process GetObject stage summaries without OTLP histograms. +pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED"; +pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE"; +pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS"; + pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL"; pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED"; pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY"; @@ -159,6 +164,15 @@ mod tests { assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED"); assert_eq!(ENV_OBS_PROFILING_EXPORT_ENABLED, "RUSTFS_OBS_PROFILING_EXPORT_ENABLED"); assert_eq!(ENV_OBS_PUT_STAGE_METRICS_ENABLED, "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED"); + assert_eq!(ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED, "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED"); + assert_eq!( + ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE, + "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE" + ); + assert_eq!( + ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS, + "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS" + ); // Test log cleanup related env keys assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES"); assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES"); diff --git a/crates/data-usage/src/data_usage.rs b/crates/data-usage/src/data_usage.rs index 3b0e81699..fd9681064 100644 --- a/crates/data-usage/src/data_usage.rs +++ b/crates/data-usage/src/data_usage.rs @@ -569,6 +569,25 @@ pub struct DataUsageInfo { /// which last-known-good generation contributed to the view. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub usage_snapshot_set_states: Vec, + /// Buckets whose dirty-usage generation moved while this complete + /// snapshot was being walked. Their `buckets_usage` values are the walk + /// result, which the cycle could not certify as current; the next cycle + /// rescans them. Sorted. Empty on a converged snapshot and on an + /// observational one. A bucket created during the walk may be listed + /// here without a `buckets_usage` entry. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub usage_snapshot_pending_buckets: Vec, + /// Length of `usage_snapshot_pending_buckets`, carried for readers that + /// only need the count. The scanner writes both from one set. + #[serde(default, skip_serializing_if = "u64_is_zero")] + pub usage_snapshot_pending_bucket_count: u64, + /// Set by the scanner on a snapshot whose authoritative publication was + /// vetoed or could not be verified. It is routed to the observed object + /// and never replaces the authoritative snapshot. Absent on snapshots + /// written before per-bucket certification, which keep + /// `usage_snapshot_converged == Some(false)` as their observed marker. + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub usage_snapshot_observational: bool, /// An observational view may contain only the sets that completed this /// cycle (or retained a compatible last-known-good cache). #[serde(default)] @@ -584,6 +603,10 @@ pub struct DataUsageInfo { pub disk_usage_status: Vec, } +fn u64_is_zero(value: &u64) -> bool { + *value == 0 +} + /// Stable identity fields changed by both coordinated scanner publication and /// backward-compatible bucket namespace cleanup. #[derive(Debug, Default, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] @@ -3011,6 +3034,58 @@ mod tests { assert_eq!(current.usage_snapshot_converged, Some(false)); } + #[test] + fn pending_bucket_certification_fields_round_trip_and_stay_additive() { + let certified = DataUsageInfo { + last_update: Some(SystemTime::UNIX_EPOCH), + usage_snapshot_complete: true, + usage_snapshot_converged: Some(false), + usage_snapshot_pending_buckets: vec!["hot-a".to_string(), "hot-b".to_string()], + usage_snapshot_pending_bucket_count: 2, + ..Default::default() + }; + let encoded = rmp_serde::to_vec_named(&certified).expect("encode certified snapshot"); + let decoded: DataUsageInfo = rmp_serde::from_slice(&encoded).expect("decode certified snapshot"); + assert_eq!(decoded, certified); + assert!(!decoded.usage_snapshot_observational); + let legacy: LegacyUsageReader = rmp_serde::from_slice(&encoded).expect("legacy reader should ignore pending fields"); + assert_eq!(legacy.buckets_count, 0); + + let observed = DataUsageInfo { + usage_snapshot_observational: true, + ..certified + }; + let encoded = rmp_serde::to_vec_named(&observed).expect("encode observed snapshot"); + let decoded: DataUsageInfo = rmp_serde::from_slice(&encoded).expect("decode observed snapshot"); + assert!(decoded.usage_snapshot_observational); + + // Older snapshots carry none of the fields and decode to the defaults; + // a converged snapshot writes none of them (named encoding carries + // field names, so a skipped field leaves no trace in the bytes). + let converged = DataUsageInfo { + last_update: Some(SystemTime::UNIX_EPOCH), + usage_snapshot_complete: true, + usage_snapshot_converged: Some(true), + ..Default::default() + }; + let encoded = rmp_serde::to_vec_named(&converged).expect("encode converged snapshot"); + for key in [ + "usage_snapshot_pending_buckets", + "usage_snapshot_pending_bucket_count", + "usage_snapshot_observational", + ] { + assert!( + !encoded.windows(key.len()).any(|window| window == key.as_bytes()), + "{key} must be skipped when empty" + ); + } + let decoded: DataUsageInfo = rmp_serde::from_slice(&encoded).expect("decode converged snapshot"); + assert!(decoded.usage_snapshot_pending_buckets.is_empty()); + assert_eq!(decoded.usage_snapshot_pending_bucket_count, 0); + assert!(!decoded.usage_snapshot_observational); + assert!(decoded.is_complete_bucket_usage_snapshot()); + } + #[test] fn convergence_marker_defaults_to_unknown_for_older_snapshots() { let encoded = rmp_serde::to_vec_named(&DataUsageInfo { diff --git a/crates/e2e_test/Cargo.toml b/crates/e2e_test/Cargo.toml index 6b1dd5ccf..63c3d75de 100644 --- a/crates/e2e_test/Cargo.toml +++ b/crates/e2e_test/Cargo.toml @@ -146,4 +146,6 @@ zip.workspace = true clap = { workspace = true, features = ["derive", "env"] } [dev-dependencies] +aws-smithy-http-client = { workspace = true, default-features = false, features = ["test-util"] } tempfile.workspace = true +tokio = { workspace = true, features = ["test-util"] } diff --git a/crates/e2e_test/README.md b/crates/e2e_test/README.md index ea2ce2b85..60f781898 100644 --- a/crates/e2e_test/README.md +++ b/crates/e2e_test/README.md @@ -22,7 +22,7 @@ Registered in [`src/lib.rs`](src/lib.rs). Grouped by concern: | **functional** | top-level `*_test.rs` | S3 data plane: `list_objects_*`, `copy_object_*`, `delete_objects_versioning`, `head_object_*`, `checksum_upload`, `compression`, `content_encoding`, `special_chars`, `leading_slash_key`, `create_bucket_region`, `quota`, `data_usage`, `snowball_auto_extract`, `mc_mirror_small_bucket`, `archive_download_integrity`, `version_id_regression`, `delete_marker_migration_semantics` | | **object_lock** | [`src/object_lock/`](src/object_lock) | Retention / legal-hold / WORM semantics | | **kms** | [`src/kms/`](src/kms) | SSE-S3 / SSE-KMS / SSE-C, local + Vault backends, multipart encryption. Own guide: [`src/kms/README.md`](src/kms/README.md) | -| **policy** | [`src/policy/`](src/policy), `existing_object_tag_policy_test`, `bucket_policy_check_test`, `anonymous_access_test`, `security_boundary_test`, `multipart_auth_test` | IAM / bucket-policy / STS session policy, policy variables, anonymous access, DoS/SSRF boundaries. Own guide: [`src/policy/README.md`](src/policy/README.md) | +| **policy** | [`src/policy/`](src/policy), `existing_object_tag_policy_test`, `conditional_write_policy_test`, `bucket_policy_check_test`, `anonymous_access_test`, `security_boundary_test`, `multipart_auth_test` | IAM / bucket-policy / STS session policy, policy variables, anonymous access, DoS/SSRF boundaries. Own guide: [`src/policy/README.md`](src/policy/README.md) | | **protocols** | [`src/protocols/`](src/protocols) | FTPS, WebDAV, SFTP compliance. Fixed ports, own guide: [`src/protocols/README.md`](src/protocols/README.md) | | **reliant** | [`src/reliant/`](src/reliant) | Tests that reuse an **externally started** server (SQL/select, conditional writes, lifecycle, deleted-object reads, node-interact). Run via [`scripts/run_e2e_tests.sh`](../../scripts/run_e2e_tests.sh); see [`src/reliant/README.md`](src/reliant/README.md) | | **cluster** | `cluster_concurrency_test`, `stale_multipart_cleanup_cluster_test`, `namespace_lock_quorum_test`, `admin_timeout_regression_test`, `object_lambda_test`, `replication_extension_test`, `tier_stats_cluster_test` | Multi-node scenarios via `RustFSTestClusterEnvironment` | diff --git a/crates/e2e_test/src/common.rs b/crates/e2e_test/src/common.rs index 622fe9015..74d7f8729 100644 --- a/crates/e2e_test/src/common.rs +++ b/crates/e2e_test/src/common.rs @@ -1037,6 +1037,22 @@ pub async fn awscurl_post_sts_form_urlencoded( Ok(response) } +fn extract_xml_tag(xml: &str, tag: &str) -> Option { + let open = format!("<{tag}>"); + let close = format!(""); + let start = xml.find(&open)? + open.len(); + let end = xml[start..].find(&close)? + start; + Some(xml[start..end].to_string()) +} + +/// Returns `(access_key, secret_key, session_token)` from an STS `AssumeRole` XML response. +pub fn parse_assume_role_credentials(xml: &str) -> Result<(String, String, String), Box> { + let access_key = extract_xml_tag(xml, "AccessKeyId").ok_or("missing AccessKeyId in AssumeRole response")?; + let secret_key = extract_xml_tag(xml, "SecretAccessKey").ok_or("missing SecretAccessKey in AssumeRole response")?; + let session_token = extract_xml_tag(xml, "SessionToken").ok_or("missing SessionToken in AssumeRole response")?; + Ok((access_key, secret_key, session_token)) +} + /// Helper function for POST requests pub async fn awscurl_post( url: &str, @@ -1542,6 +1558,15 @@ impl RustFSTestClusterEnvironment { self.start_node_from_binary(node_idx, &binary_path).await } + /// Start a process and wait only for TCP, allowing tests to bring up a + /// quorum one node at a time before the S3 service can become ready. + pub async fn start_node_process(&mut self, node_idx: usize) -> Result<(), Box> { + let binary_path = rustfs_binary_path(); + let volumes_arg = self.build_volumes_arg(); + self.spawn_node(node_idx, &binary_path, &volumes_arg)?; + self.wait_for_node_ready(&self.nodes[node_idx].address, node_idx).await + } + /// Start one stopped cluster node with a specific RustFS binary while /// preserving the cluster's volume layout and that node's data directory. pub async fn start_node_from_binary( diff --git a/crates/e2e_test/src/conditional_delete_objects_test.rs b/crates/e2e_test/src/conditional_delete_objects_test.rs new file mode 100644 index 000000000..753ed42a7 --- /dev/null +++ b/crates/e2e_test/src/conditional_delete_objects_test.rs @@ -0,0 +1,381 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! S3 wire semantics for per-object conditional bulk deletion. + +use crate::common::RustFSTestEnvironment; +use aws_sdk_s3::Client; +use aws_sdk_s3::error::ProvideErrorMetadata; +use aws_sdk_s3::operation::delete_objects::DeleteObjectsOutput; +use aws_sdk_s3::operation::put_object::PutObjectOutput; +use aws_sdk_s3::primitives::ByteStream; +use aws_sdk_s3::types::{BucketVersioningStatus, Delete, ObjectIdentifier, VersioningConfiguration}; +use std::collections::BTreeSet; +use std::error::Error; + +type TestResult = Result>; +type VersionSnapshot = BTreeSet<(String, bool)>; + +async fn put(client: &Client, bucket: &str, key: &str, body: &'static [u8]) -> TestResult { + let result = client + .put_object() + .bucket(bucket) + .key(key) + .body(ByteStream::from_static(body)) + .send() + .await?; + assert!(result.e_tag().is_some(), "fixture PUT must provide an ETag for the following conditions"); + Ok(result) +} + +fn identifier(key: &str, etag: Option<&str>, version: Option<&str>) -> TestResult { + Ok(ObjectIdentifier::builder() + .key(key) + .set_e_tag(etag.map(str::to_owned)) + .set_version_id(version.map(str::to_owned)) + .build()?) +} + +async fn delete_batch( + client: &Client, + bucket: &str, + objects: Vec, + quiet: bool, +) -> TestResult { + Ok(client + .delete_objects() + .bucket(bucket) + .delete(Delete::builder().set_objects(Some(objects)).quiet(quiet).build()?) + .send() + .await?) +} + +async fn assert_body(client: &Client, bucket: &str, key: &str, expected: &[u8]) -> TestResult { + let result = client.get_object().bucket(bucket).key(key).send().await?; + assert_eq!(result.body.collect().await?.into_bytes().as_ref(), expected); + Ok(()) +} + +async fn assert_absent(client: &Client, bucket: &str, key: &str) { + let error = client + .head_object() + .bucket(bucket) + .key(key) + .send() + .await + .expect_err("deleted key must be absent"); + assert_eq!(error.as_service_error().and_then(|error| error.code()), Some("NotFound")); +} + +async fn versions(client: &Client, bucket: &str, key: &str) -> TestResult { + let result = client.list_object_versions().bucket(bucket).prefix(key).send().await?; + assert!(!result.is_truncated().unwrap_or_default(), "focused version fixture fits a single page"); + let mut snapshot = BTreeSet::new(); + for version in result.versions().iter().filter(|version| version.key() == Some(key)) { + snapshot.insert((version.version_id().expect("version has an identity").to_owned(), false)); + } + for marker in result.delete_markers().iter().filter(|marker| marker.key() == Some(key)) { + snapshot.insert((marker.version_id().expect("delete marker has an identity").to_owned(), true)); + } + Ok(snapshot) +} + +#[tokio::test] +async fn sdk_conditional_delete_objects_mixed_results_overwrites_duplicates_and_quiet() -> TestResult { + let mut env = RustFSTestEnvironment::new().await?; + env.start_rustfs_server(vec![]).await?; + let client = env.create_s3_client(); + let bucket = "conditional-bulk-mixed"; + client.create_bucket().bucket(bucket).send().await?; + + let old = put(&client, bucket, "overwritten", b"generation A").await?; + let current = put(&client, bucket, "overwritten", b"generation B").await?; + let matched = put(&client, bucket, "matched", b"matching object").await?; + put(&client, bucket, "exists", b"existence object").await?; + put(&client, bucket, "unconditional", b"unconditional object").await?; + put(&client, bucket, "empty", b"empty condition must preserve this").await?; + put(&client, bucket, "empty-quoted", b"quoted empty condition must preserve this").await?; + let weak = put(&client, bucket, "weak", b"weak condition must preserve this").await?; + let weak_etag = format!("W/\"{}\"", weak.e_tag().expect("PUT supplies an ETag").trim_matches('"')); + assert_ne!(old.e_tag(), current.e_tag(), "overwrite fixture must have distinct ETags"); + let matched_etag = matched.e_tag().expect("PUT supplies an ETag"); + let result = delete_batch( + &client, + bucket, + vec![ + identifier("matched", Some(matched_etag), None)?, + identifier("overwritten", old.e_tag(), None)?, + identifier("matched", Some("stale-etag"), None)?, + identifier("matched", Some(matched_etag), None)?, + identifier("unconditional", None, None)?, + identifier("exists", Some("*"), None)?, + identifier("missing", Some("*"), None)?, + identifier("empty", Some(""), None)?, + identifier("empty-quoted", Some("\"\""), None)?, + identifier("weak", Some(&weak_etag), None)?, + identifier("unconditional-missing", None, None)?, + identifier("missing-specific", Some("opaque-etag"), None)?, + ], + false, + ) + .await?; + assert_eq!( + result.deleted().iter().map(|object| object.key()).collect::>(), + [ + Some("matched"), + Some("matched"), + Some("unconditional"), + Some("exists"), + Some("unconditional-missing") + ], + "duplicate successful inputs retain their separate response entries" + ); + assert_eq!( + result + .errors() + .iter() + .map(|error| (error.key(), error.code())) + .collect::>(), + [ + (Some("overwritten"), Some("PreconditionFailed")), + (Some("matched"), Some("PreconditionFailed")), + (Some("missing"), Some("NoSuchKey")), + (Some("empty"), Some("InvalidArgument")), + (Some("empty-quoted"), Some("InvalidArgument")), + (Some("weak"), Some("InvalidArgument")), + (Some("missing-specific"), Some("NoSuchKey")), + ] + ); + assert_body(&client, bucket, "overwritten", b"generation B").await?; + assert_body(&client, bucket, "empty", b"empty condition must preserve this").await?; + assert_body(&client, bucket, "empty-quoted", b"quoted empty condition must preserve this").await?; + assert_body(&client, bucket, "weak", b"weak condition must preserve this").await?; + for key in ["matched", "unconditional", "exists"] { + assert_absent(&client, bucket, key).await; + } + + let quiet = put(&client, bucket, "quiet-match", b"quiet matching object").await?; + put(&client, bucket, "quiet-rejected", b"quiet rejected object").await?; + let result = delete_batch( + &client, + bucket, + vec![ + identifier("quiet-match", quiet.e_tag(), None)?, + identifier("quiet-match", quiet.e_tag(), None)?, + identifier("quiet-rejected", Some("stale-etag"), None)?, + identifier("missing", Some("*"), None)?, + ], + true, + ) + .await?; + assert!(result.deleted().is_empty(), "Quiet suppresses successful entries"); + assert_eq!( + result + .errors() + .iter() + .map(|error| (error.key(), error.code())) + .collect::>(), + [ + (Some("quiet-rejected"), Some("PreconditionFailed")), + (Some("missing"), Some("NoSuchKey")) + ] + ); + assert_absent(&client, bucket, "quiet-match").await; + assert_body(&client, bucket, "quiet-rejected", b"quiet rejected object").await?; + Ok(()) +} + +#[tokio::test] +async fn sdk_conditional_delete_objects_versioning_current_markers_and_explicit_versions() -> TestResult { + let mut env = RustFSTestEnvironment::new().await?; + env.start_rustfs_server(vec![]).await?; + let client = env.create_s3_client(); + for suspended in [false, true] { + let bucket = if suspended { + "conditional-bulk-suspended" + } else { + "conditional-bulk-enabled" + }; + client.create_bucket().bucket(bucket).send().await?; + client + .put_bucket_versioning() + .bucket(bucket) + .versioning_configuration( + VersioningConfiguration::builder() + .status(BucketVersioningStatus::Enabled) + .build(), + ) + .send() + .await?; + let historical = put(&client, bucket, "history", b"historical payload").await?; + let invalid_historical = put(&client, bucket, "invalid", b"historical rejected payload").await?; + if suspended { + client + .put_bucket_versioning() + .bucket(bucket) + .versioning_configuration( + VersioningConfiguration::builder() + .status(BucketVersioningStatus::Suspended) + .build(), + ) + .send() + .await?; + } + let current = put(&client, bucket, "history", b"current payload").await?; + let invalid = put(&client, bucket, "invalid", b"current rejected payload").await?; + let invalid_before = versions(&client, bucket, "invalid").await?; + let historical_id = historical.version_id().expect("enabled fixture has a UUID version"); + let invalid_historical_id = invalid_historical + .version_id() + .expect("enabled rejected fixture has a UUID version"); + let result = delete_batch( + &client, + bucket, + vec![ + identifier("history", current.e_tag(), None)?, + identifier("history", None, Some(historical_id))?, + identifier("history", current.e_tag(), None)?, + identifier("invalid", invalid.e_tag(), Some(invalid_historical_id))?, + identifier("invalid", Some("*"), Some("null"))?, + identifier("invalid", Some("stale-etag"), None)?, + ], + false, + ) + .await?; + assert_eq!(result.deleted().len(), 3, "both current input entries and the historical delete succeed"); + assert_eq!( + result + .errors() + .iter() + .map(|error| (error.key(), error.code())) + .collect::>(), + [ + (Some("invalid"), Some("InvalidRequest")), + (Some("invalid"), Some("InvalidRequest")), + (Some("invalid"), Some("PreconditionFailed")), + ] + ); + assert_eq!(result.errors()[0].version_id(), Some(invalid_historical_id)); + assert_eq!(result.errors()[1].version_id(), Some("null")); + assert_eq!(versions(&client, bucket, "invalid").await?, invalid_before); + assert_body(&client, bucket, "invalid", b"current rejected payload").await?; + + let marker = result.deleted()[0] + .delete_marker_version_id() + .expect("current conditional delete returns a marker"); + assert_eq!(result.deleted()[0].delete_marker(), Some(true)); + assert_eq!(result.deleted()[2].delete_marker_version_id(), Some(marker)); + assert_eq!(result.deleted()[1].version_id(), Some(historical_id)); + if suspended { + assert_eq!(marker, "null", "a suspended current delete returns the created null marker identity"); + } + let history_after = versions(&client, bucket, "history").await?; + assert_eq!( + history_after.iter().filter(|(_, marker)| *marker).count(), + 1, + "duplicate actions create one marker" + ); + assert!( + !history_after.contains(&(historical_id.to_string(), false)), + "distinct historical target is removed" + ); + if suspended { + assert_eq!(history_after, BTreeSet::from([("null".to_string(), true)])); + } else { + assert_eq!( + history_after, + BTreeSet::from([ + (current.version_id().expect("current enabled object has a UUID").to_string(), false), + (marker.to_string(), true), + ]), + "enabled conditional current deletion retains the current data version behind the marker" + ); + } + + let result = delete_batch( + &client, + bucket, + vec![ + identifier("history", Some("*"), None)?, + identifier("history", current.e_tag(), None)?, + identifier("missing", Some("*"), None)?, + ], + false, + ) + .await?; + assert!( + result.deleted().is_empty(), + "current markers and absent objects cannot satisfy a live-object condition" + ); + assert_eq!(result.errors().len(), 3); + assert!(result.errors().iter().all(|error| error.code() == Some("NoSuchKey"))); + assert_eq!( + versions(&client, bucket, "history").await?, + history_after, + "failed conditions create no marker" + ); + assert!(versions(&client, bucket, "missing").await?.is_empty()); + + put(&client, bucket, "exists", b"versioned existence payload").await?; + let result = delete_batch(&client, bucket, vec![identifier("exists", Some("*"), None)?], false).await?; + assert!(result.errors().is_empty()); + assert_eq!(result.deleted().len(), 1); + assert_eq!(result.deleted()[0].delete_marker(), Some(true)); + assert_eq!( + versions(&client, bucket, "exists") + .await? + .iter() + .filter(|(_, marker)| *marker) + .count(), + 1 + ); + + if !suspended { + let directory = put(&client, bucket, "dir/", b"directory null-version payload").await?; + assert_eq!( + versions(&client, bucket, "dir/").await?, + BTreeSet::from([("null".to_string(), false)]), + "the directory PUT fixture must identify the synthetic null data version" + ); + let result = delete_batch( + &client, + bucket, + vec![identifier("dir/", directory.e_tag(), None)?, identifier("dir/", None, None)?], + false, + ) + .await?; + assert!(result.errors().is_empty(), "both distinct directory actions must succeed"); + assert_eq!(result.deleted().len(), 2); + assert_eq!(result.deleted()[0].key(), Some("dir/")); + assert_eq!(result.deleted()[1].key(), Some("dir/")); + assert_eq!(result.deleted()[0].delete_marker(), Some(true)); + let marker = result.deleted()[0] + .delete_marker_version_id() + .expect("conditional current-directory deletion creates a versioned marker"); + assert!( + !uuid::Uuid::parse_str(marker)?.is_nil(), + "enabled current-directory deletion creates a UUID marker" + ); + assert_eq!(result.deleted()[1].delete_marker(), None); + assert_eq!(result.deleted()[1].delete_marker_version_id(), None); + assert_eq!(result.deleted()[1].version_id(), None); + assert_eq!( + versions(&client, bucket, "dir/").await?, + BTreeSet::from([(marker.to_string(), true)]), + "current-marker creation and synthetic null-version removal are distinct actions: one marker remains and the null data version is gone" + ); + } + } + Ok(()) +} diff --git a/crates/e2e_test/src/conditional_write_policy_test.rs b/crates/e2e_test/src/conditional_write_policy_test.rs new file mode 100644 index 000000000..5b610b692 --- /dev/null +++ b/crates/e2e_test/src/conditional_write_policy_test.rs @@ -0,0 +1,377 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! E2E: conditional write enforcement with the `s3:if-none-match` and `s3:if-match` +//! condition keys in a **bucket policy**, an **IAM identity policy**, and an **STS +//! AssumeRole session policy**. Covers PutObject and CompleteMultipartUpload, the two +//! upload operations that accept the conditional headers. The plain precondition +//! semantics without a policy live in `reliant/conditional_writes.rs`. + +use crate::common::{ + AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user, + awscurl_post_sts_form_urlencoded, build_test_s3_config, init_logging, parse_assume_role_credentials, +}; +use aws_sdk_s3::Client; +use aws_sdk_s3::config::http::HttpResponse; +use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError}; +use aws_sdk_s3::primitives::ByteStream; +use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart}; +use tracing::info; +use uuid::Uuid; + +type TestResult = Result<(), Box>; + +fn assert_error( + err: &SdkError, + code: &str, + status: u16, + context: &str, +) { + assert_eq!( + err.as_service_error().and_then(ProvideErrorMetadata::code), + Some(code), + "{context}: {err:?}" + ); + assert_eq!( + err.raw_response().map(|response| response.status().as_u16()), + Some(status), + "{context}: {err:?}" + ); +} + +async fn admin_add_and_attach_policy( + env: &RustFSTestEnvironment, + policy_name: &str, + policy_json: &str, + username: &str, +) -> TestResult { + admin_add_canned_policy_via( + AdminTransport::Signed, + &env.url, + &env.access_key, + &env.secret_key, + policy_name, + policy_json, + ) + .await?; + admin_attach_user_policy_via(AdminTransport::Signed, &env.url, &env.access_key, &env.secret_key, policy_name, username).await +} + +/// Starts a one-part multipart upload as the admin so that only the complete step +/// runs under the policy being tested. CreateMultipartUpload and UploadPart are both +/// authorized as `s3:PutObject` and cannot carry a conditional header, so a policy +/// that requires `s3:if-none-match` denies them. AWS exempts them with +/// `s3:ObjectCreationOperation`, which RustFS does not support yet. +async fn admin_start_upload( + admin: &Client, + bucket: &str, + key: &str, +) -> Result<(String, CompletedMultipartUpload), Box> { + let upload = admin.create_multipart_upload().bucket(bucket).key(key).send().await?; + let upload_id = upload.upload_id().ok_or("missing upload id")?.to_string(); + let part = admin + .upload_part() + .bucket(bucket) + .key(key) + .upload_id(&upload_id) + .part_number(1) + .body(ByteStream::from_static(b"multipart-conditional-write")) + .send() + .await?; + let completed = CompletedMultipartUpload::builder() + .parts( + CompletedPart::builder() + .part_number(1) + .set_e_tag(part.e_tag().map(str::to_string)) + .build(), + ) + .build(); + Ok((upload_id, completed)) +} + +/// One server for the three policy kinds; each scenario uses its own principal and bucket. +#[tokio::test] +async fn test_e2e_policies_enforce_conditional_write_keys() -> TestResult { + init_logging(); + let mut env = RustFSTestEnvironment::new().await?; + env.start_rustfs_server(vec![]).await?; + let admin = env.create_s3_client(); + let suffix = Uuid::new_v4(); + + bucket_policy_requires_if_none_match(&env, &admin, suffix).await?; + iam_policy_requires_if_match(&env, &admin, suffix).await?; + sts_session_policy_requires_if_none_match(&env, &admin, suffix).await?; + + info!("test_e2e_policies_enforce_conditional_write_keys passed"); + Ok(()) +} + +/// Bucket policy in the shape of the AWS enforcement example: PutObject is allowed only +/// when the request carries `If-None-Match`, which makes the bucket create-only. +async fn bucket_policy_requires_if_none_match(env: &RustFSTestEnvironment, admin: &Client, suffix: Uuid) -> TestResult { + let user = format!("e2econdwrite-{suffix}"); + let user_secret = "longSecretKeyForCondWrite1!"; + let bucket = format!("e2e-cond-write-bkt-{suffix}"); + + admin_create_user(env, &user, user_secret).await?; + admin.create_bucket().bucket(&bucket).send().await?; + let bucket_policy = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": { "AWS": [user.clone()] }, + "Action": ["s3:PutObject"], + "Resource": [format!("arn:aws:s3:::{bucket}/*")], + "Condition": { "Null": { "s3:if-none-match": "false" } } + }] + }) + .to_string(); + admin + .put_bucket_policy() + .bucket(&bucket) + .policy(&bucket_policy) + .send() + .await?; + + let client = env.create_s3_client_with_credentials(&user, user_secret); + + let denied = client + .put_object() + .bucket(&bucket) + .key("plain.txt") + .body(ByteStream::from_static(b"plain")) + .send() + .await + .expect_err("PutObject without If-None-Match must be denied"); + assert_error(&denied, "AccessDenied", 403, "unconditional PutObject"); + + let created = client + .put_object() + .bucket(&bucket) + .key("created.txt") + .if_none_match("*") + .body(ByteStream::from_static(b"created")) + .send() + .await?; + let etag = created.e_tag().ok_or("missing ETag on conditional PutObject")?.to_string(); + + let overwrite = client + .put_object() + .bucket(&bucket) + .key("created.txt") + .if_none_match("*") + .body(ByteStream::from_static(b"overwrite")) + .send() + .await + .expect_err("If-None-Match: * must not overwrite an existing key"); + assert_error( + &overwrite, + "PreconditionFailed", + 412, + "allowed PutObject still evaluates the precondition", + ); + + // A non-`*` value satisfies `Null: false` but must not act as an ETag comparison + // that lets the write overwrite the key. + let overwrite = client + .put_object() + .bucket(&bucket) + .key("created.txt") + .if_none_match("\"0123456789abcdef0123456789abcdef\"") + .body(ByteStream::from_static(b"overwrite")) + .send() + .await + .expect_err("If-None-Match with an ETag value must be rejected"); + assert_error(&overwrite, "NotImplemented", 501, "PutObject with an ETag If-None-Match"); + let current = admin.head_object().bucket(&bucket).key("created.txt").send().await?; + assert_eq!(current.e_tag(), Some(etag.as_str()), "created.txt must not be overwritten"); + + let (upload_id, completed) = admin_start_upload(admin, &bucket, "multipart.bin").await?; + let denied = client + .complete_multipart_upload() + .bucket(&bucket) + .key("multipart.bin") + .upload_id(&upload_id) + .multipart_upload(completed.clone()) + .send() + .await + .expect_err("CompleteMultipartUpload without If-None-Match must be denied"); + assert_error(&denied, "AccessDenied", 403, "unconditional CompleteMultipartUpload"); + + client + .complete_multipart_upload() + .bucket(&bucket) + .key("multipart.bin") + .upload_id(&upload_id) + .if_none_match("*") + .multipart_upload(completed) + .send() + .await?; + + let (upload_id, completed) = admin_start_upload(admin, &bucket, "multipart.bin").await?; + let overwrite = client + .complete_multipart_upload() + .bucket(&bucket) + .key("multipart.bin") + .upload_id(&upload_id) + .if_none_match("*") + .multipart_upload(completed) + .send() + .await + .expect_err("CompleteMultipartUpload with If-None-Match: * must not overwrite an existing key"); + assert_error( + &overwrite, + "PreconditionFailed", + 412, + "allowed CompleteMultipartUpload still evaluates the precondition", + ); + + let _ = admin + .abort_multipart_upload() + .bucket(&bucket) + .key("multipart.bin") + .upload_id(&upload_id) + .send() + .await; + for key in ["created.txt", "multipart.bin"] { + let _ = admin.delete_object().bucket(&bucket).key(key).send().await; + } + let _ = admin.delete_bucket().bucket(&bucket).send().await; + Ok(()) +} + +/// IAM identity policy: PutObject is allowed only when the request carries `If-Match`. +async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Client, suffix: Uuid) -> TestResult { + let user = format!("e2econdmatch-{suffix}"); + let user_secret = "longSecretKeyForCondMatch2!"; + let policy_name = format!("e2e-cond-match-pol-{suffix}"); + let bucket = format!("e2e-cond-match-bkt-{suffix}"); + let key = "guarded.txt"; + + admin_create_user(env, &user, user_secret).await?; + let policy = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Action": ["s3:PutObject"], + "Resource": [format!("arn:aws:s3:::{bucket}/*")], + "Condition": { "Null": { "s3:if-match": "false" } } + }] + }) + .to_string(); + admin_add_and_attach_policy(env, &policy_name, &policy, &user).await?; + + admin.create_bucket().bucket(&bucket).send().await?; + let original = admin + .put_object() + .bucket(&bucket) + .key(key) + .body(ByteStream::from_static(b"v1")) + .send() + .await?; + let etag = original.e_tag().ok_or("missing ETag on initial PutObject")?.to_string(); + + let client = env.create_s3_client_with_credentials(&user, user_secret); + let denied = client + .put_object() + .bucket(&bucket) + .key(key) + .body(ByteStream::from_static(b"blind-overwrite")) + .send() + .await + .expect_err("PutObject without If-Match must be denied"); + assert_error(&denied, "AccessDenied", 403, "unconditional PutObject"); + + client + .put_object() + .bucket(&bucket) + .key(key) + .if_match(&etag) + .body(ByteStream::from_static(b"v2")) + .send() + .await?; + + let _ = admin.delete_object().bucket(&bucket).key(key).send().await; + let _ = admin.delete_bucket().bucket(&bucket).send().await; + Ok(()) +} + +/// STS session policy: AssumeRole must accept `s3:if-none-match`, and the session may +/// only create objects with `If-None-Match: *`. +async fn sts_session_policy_requires_if_none_match(env: &RustFSTestEnvironment, admin: &Client, suffix: Uuid) -> TestResult { + let parent = format!("e2e-sts-cond-par-{suffix}"); + let parent_secret = "longSecretKeyForCondParent3!"; + let policy_name = format!("e2e-sts-cond-rw-{suffix}"); + let bucket = format!("e2e-sts-cond-bkt-{suffix}"); + + admin_create_user(env, &parent, parent_secret).await?; + let parent_policy = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [ + { "Effect": "Allow", "Action": ["s3:*"], "Resource": ["arn:aws:s3:::*"] }, + { "Effect": "Allow", "Action": ["sts:AssumeRole"], "Resource": ["arn:aws:s3:::*"] } + ] + }) + .to_string(); + admin_add_and_attach_policy(env, &policy_name, &parent_policy, &parent).await?; + admin.create_bucket().bucket(&bucket).send().await?; + + let session_policy = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Action": ["s3:PutObject"], + "Resource": [format!("arn:aws:s3:::{bucket}/*")], + "Condition": { "StringEquals": { "s3:if-none-match": "*" } } + }] + }) + .to_string(); + let body = format!( + "Action=AssumeRole&Version=2011-06-15&DurationSeconds=3600&Policy={}", + urlencoding::encode(&session_policy) + ); + let url = format!("{}/", env.url.trim_end_matches('/')); + let xml = awscurl_post_sts_form_urlencoded(&url, &body, &parent, parent_secret).await?; + let (access_key, secret_key, session_token) = parse_assume_role_credentials(&xml)?; + let session = Client::from_conf(build_test_s3_config( + &env.url, + &access_key, + &secret_key, + Some(&session_token), + "e2e-sts-session", + )); + + let denied = session + .put_object() + .bucket(&bucket) + .key("plain.txt") + .body(ByteStream::from_static(b"plain")) + .send() + .await + .expect_err("session PutObject without If-None-Match must be denied"); + assert_error(&denied, "AccessDenied", 403, "unconditional session PutObject"); + + session + .put_object() + .bucket(&bucket) + .key("created.txt") + .if_none_match("*") + .body(ByteStream::from_static(b"created")) + .send() + .await?; + + let _ = admin.delete_object().bucket(&bucket).key("created.txt").send().await; + let _ = admin.delete_bucket().bucket(&bucket).send().await; + Ok(()) +} diff --git a/crates/e2e_test/src/distributed/harness.rs b/crates/e2e_test/src/distributed/harness.rs index 364034477..3ce00fa01 100644 --- a/crates/e2e_test/src/distributed/harness.rs +++ b/crates/e2e_test/src/distributed/harness.rs @@ -860,10 +860,12 @@ pub(crate) async fn start_rebalance(cluster: &RustFSTestClusterEnvironment) -> T if status.is_success() { break response; } - if status == StatusCode::INTERNAL_SERVER_ERROR - && response.contains("requires a live fleet capability proof") - && Instant::now() < deadline - { + // The init stage answers 503 + Retry-After while the cluster capability + // proof is still missing; older binaries surfaced the same condition as + // a 500 carrying the proof marker. + let waiting_for_readiness = status == StatusCode::SERVICE_UNAVAILABLE + || (status == StatusCode::INTERNAL_SERVER_ERROR && response.contains("requires a live fleet capability proof")); + if waiting_for_readiness && Instant::now() < deadline { sleep(Duration::from_secs(1)).await; continue; } diff --git a/crates/e2e_test/src/existing_object_tag_policy_test.rs b/crates/e2e_test/src/existing_object_tag_policy_test.rs index 57a21f48e..ff32787d9 100644 --- a/crates/e2e_test/src/existing_object_tag_policy_test.rs +++ b/crates/e2e_test/src/existing_object_tag_policy_test.rs @@ -18,7 +18,7 @@ use crate::common::{ AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, - awscurl_delete, awscurl_post_sts_form_urlencoded, build_test_s3_config, init_logging, + awscurl_delete, awscurl_post_sts_form_urlencoded, build_test_s3_config, init_logging, parse_assume_role_credentials, }; use aws_sdk_s3::Client; use aws_sdk_s3::error::ProvideErrorMetadata; @@ -41,21 +41,6 @@ fn sts_session_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: )) } -fn extract_xml_tag(xml: &str, tag: &str) -> Option { - let open = format!("<{tag}>"); - let close = format!(""); - let start = xml.find(&open)? + open.len(); - let end = xml[start..].find(&close)? + start; - Some(xml[start..end].to_string()) -} - -fn parse_assume_role_credentials(xml: &str) -> Result<(String, String, String), Box> { - let ak = extract_xml_tag(xml, "AccessKeyId").ok_or("missing AccessKeyId in AssumeRole response")?; - let sk = extract_xml_tag(xml, "SecretAccessKey").ok_or("missing SecretAccessKey in AssumeRole response")?; - let token = extract_xml_tag(xml, "SessionToken").ok_or("missing SessionToken in AssumeRole response")?; - Ok((ak, sk, token)) -} - async fn assume_role_with_session_policy( env: &RustFSTestEnvironment, parent_ak: &str, diff --git a/crates/e2e_test/src/fake_s3_target/README.md b/crates/e2e_test/src/fake_s3_target/README.md index bcd2e8ca6..dda6ee019 100644 --- a/crates/e2e_test/src/fake_s3_target/README.md +++ b/crates/e2e_test/src/fake_s3_target/README.md @@ -6,7 +6,7 @@ This module is the shared failure-injection boundary for replication end-to-end Supported data operations are HeadBucket, GetBucketVersioning, ListObjectsV2, PUT/GET/HEAD/DELETE Object, Get/Put/Delete ObjectTagging (tags live per version; Put replaces the whole set, Delete clears it), and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets created with `create_bucket` are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB. -`create_bucket_with_object_lock(name)` creates a versioned bucket whose GetObjectLockConfiguration reports `Enabled`; every other bucket answers `ObjectLockConfigurationNotFoundError`, the code RustFS's replication-check classifies as "not enabled". Three switches model remote-target behaviors the fleet has shown, so the outbound target matrix (`crates/e2e_test/src/replication_target_matrix_test.rs`) can replicate every object shape against each: `assign_own_version_ids(true)` ignores the source version id and mints its own (AWS S3 / Wasabi); `reject_aws_chunked_uploads(true)` refuses any PutObject or UploadPart announcing `aws-chunked` framing (`Content-Encoding: aws-chunked`, an `x-amz-trailer`, or a `STREAMING-*` payload hash) with `InvalidRequest` before the body is read (SeaweedFS 3.97, rustfs#6853); `require_checksum_for_object_lock(true)` rejects a PutObject carrying any `x-amz-object-lock-*` header unless it also carries `Content-MD5`, an `x-amz-checksum-*` header, or `x-amz-sdk-checksum-algorithm` (AWS S3 / MinIO, rustfs#7082). Independently of that switch, a `Content-MD5` header is always verified against the body and a mismatch answers `BadDigest`. +`create_bucket_with_object_lock(name)` creates a versioned bucket whose GetObjectLockConfiguration reports `Enabled`; every other bucket answers `ObjectLockConfigurationNotFoundError`, the code RustFS's replication-check classifies as "not enabled". Four switches model remote-target behaviors the fleet has shown, so the outbound target matrix (`crates/e2e_test/src/replication_target_matrix_test.rs`) can replicate every object shape against each: `assign_own_version_ids(true)` ignores the source version id and mints its own (AWS S3 / Wasabi); `reject_aws_chunked_uploads(true)` refuses any PutObject or UploadPart announcing `aws-chunked` framing (`Content-Encoding: aws-chunked`, an `x-amz-trailer`, or a `STREAMING-*` payload hash) with `InvalidRequest` before the body is read (SeaweedFS 3.97, rustfs#6853); `require_checksum_for_object_lock(true)` rejects a PutObject carrying any `x-amz-object-lock-*` header unless it also carries `Content-MD5`, an `x-amz-checksum-*` header, or `x-amz-sdk-checksum-algorithm` (AWS S3 / MinIO, rustfs#7082). `ignore_source_etags(true)` ignores the `x-rustfs-source-etag` / `x-minio-source-etag` replication headers and reports the ETag of the received bytes (AWS S3 and other generic S3 services; the default adopts the source ETag like RustFS/MinIO). Independently of that switch, a `Content-MD5` header is always verified against the body and a mismatch answers `BadDigest`. `create_bucket_with_mode(name, BucketMode::Unversioned)` models a plain migration source: PUT overwrites in place, DELETE removes the key without a delete marker, GetBucketVersioning reports no status, and no `x-amz-version-id` is returned by PUT, GET, HEAD, tagging, or multipart completion. The only `versionId` such a bucket accepts is `null`; any other value is rejected with `InvalidArgument`. The mode is fixed at creation. diff --git a/crates/e2e_test/src/fake_s3_target/mod.rs b/crates/e2e_test/src/fake_s3_target/mod.rs index 3f8bf88d8..411c4133d 100644 --- a/crates/e2e_test/src/fake_s3_target/mod.rs +++ b/crates/e2e_test/src/fake_s3_target/mod.rs @@ -537,6 +537,10 @@ struct StoreState { /// PutObject carrying any `x-amz-object-lock-*` header must also carry /// `Content-MD5` or an `x-amz-checksum-*` header. require_checksum_for_object_lock: bool, + /// Models AWS S3 and other generic S3 services: the `x-rustfs-source-etag` + /// / `x-minio-source-etag` replication headers are ignored and every + /// stored object reports the ETag computed from the bytes it received. + ignore_source_etags: bool, /// Models Wasabi (rustfs/backlog#2340): a version-addressed DELETE of a /// version id the target never had answers 404 `NoSuchVersion` instead of /// the idempotent 204 RustFS/MinIO give. @@ -1052,6 +1056,13 @@ impl FakeS3Target { lock(&self.backend.store).require_checksum_for_object_lock = enabled; } + /// AWS-like mode: ignore the replication source-ETag headers and report + /// the ETag of the received bytes, as AWS S3 does for SSE-S3 and + /// unencrypted objects. The default adopts the source ETag (RustFS/MinIO). + pub fn ignore_source_etags(&self, enabled: bool) { + lock(&self.backend.store).ignore_source_etags = enabled; + } + pub fn assign_own_multipart_version_ids(&self, enabled: bool) { lock(&self.backend.store).assign_own_multipart_version_ids = enabled; } @@ -2444,12 +2455,13 @@ impl S3 for FakeBackend { .map_err(|_| s3s::s3_error!(ServiceUnavailable, "fake target body limiter closed"))?; let headers = req.headers; let input = req.input; - let (assign_own, drop_unlisted, require_checksum, limits, versioned) = { + let (assign_own, drop_unlisted, require_checksum, ignore_source_etags, limits, versioned) = { let state = lock(&self.store); ( state.assign_own_version_ids, state.drop_unlisted_replication_headers, state.require_checksum_for_object_lock, + state.ignore_source_etags, state.limits, bucket_versioned(&state, &input.bucket), ) @@ -2489,7 +2501,7 @@ impl S3 for FakeBackend { } else { NULL_VERSION_ID.to_string() }; - let e_tag = match source_etag(&headers)? { + let e_tag = match source_etag(&headers)?.filter(|_| !ignore_source_etags) { Some(value) => value, None => { let (digest, permit) = md5_digest(body.clone(), body_permit).await?; @@ -3112,7 +3124,10 @@ impl S3 for FakeBackend { .collect(); let (body, digests, _body_permits) = assemble_multipart(assembly_parts, total_len, _body_permits).await?; let part_count = requested.len(); - let e_tag = source_etag(&headers)?.unwrap_or_else(|| format!("{}-{part_count}", md5_hex(digests))); + let ignore_source_etags = lock(&self.store).ignore_source_etags; + let e_tag = source_etag(&headers)? + .filter(|_| !ignore_source_etags) + .unwrap_or_else(|| format!("{}-{part_count}", md5_hex(digests))); let version = ObjectVersion { version_id: upload.version_id.clone(), body, diff --git a/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs b/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs index aaf45f5de..f5187aa77 100644 --- a/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs +++ b/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs @@ -1032,7 +1032,7 @@ mod tests { let heal_body = r#"{"recursive":true,"dryRun":false,"remove":false,"recreate":true,"scanMode":2,"updateParity":false,"nolock":false}"#; let heal_url = format!("{}/rustfs/admin/v3/heal/{}?forceStart=true", env.url, bucket); - signed_admin_post(&heal_url, Some(heal_body), &env.access_key, &env.secret_key) + start_root_heal_when_control_ready(&heal_url, heal_body, &env.access_key, &env.secret_key) .await .expect("admin deep heal should be accepted"); @@ -1191,11 +1191,59 @@ mod tests { .into()) } + fn scanner_cycle_completion_budget( + cycle_end: u64, + interval: Option, + now: u64, + active: bool, + ) -> Result { + if cycle_end > now { + return Err("scanner cycle timestamp is in the future"); + } + if active { + return Ok(Duration::from_secs(60)); + } + let interval = interval + .filter(|interval| *interval != 0) + .ok_or("scanner schedule has no positive interval")?; + let next_cycle = cycle_end + .checked_add(interval) + .ok_or("scanner schedule timestamp overflows")?; + let budget = next_cycle + .saturating_sub(now) + .checked_add(60) + .ok_or("scanner completion budget overflows")?; + Ok(Duration::from_secs(budget)) + } + + #[test] + fn scanner_cycle_completion_budget_accounts_for_captured_retry_schedule() { + assert_eq!( + scanner_cycle_completion_budget(1_791_815_390, Some(256), 1_791_815_447, false), + Ok(Duration::from_secs(259)), + ); + assert_eq!(scanner_cycle_completion_budget(100, Some(1), 110, false), Ok(Duration::from_secs(60))); + } + + #[test] + fn scanner_cycle_completion_budget_does_not_grant_planned_sleep_to_active_cycle() { + assert_eq!(scanner_cycle_completion_budget(100, Some(256), 110, true), Ok(Duration::from_secs(60))); + assert_eq!(scanner_cycle_completion_budget(0, None, 110, true), Ok(Duration::from_secs(60))); + } + + #[test] + fn scanner_cycle_completion_budget_rejects_invalid_schedule() { + for (end, interval, now) in [(101, 1, 100), (100, 0, 100), (u64::MAX, 1, u64::MAX), (1, u64::MAX - 1, 1)] { + assert!(scanner_cycle_completion_budget(end, Some(interval), now, false).is_err()); + } + } + async fn wait_for_scanner_cycle_after( cluster: &RustFSTestClusterEnvironment, previous_cycle_end: u64, ) -> Result> { - let deadline = Instant::now() + Duration::from_secs(60); + let mut deadline = Instant::now() + Duration::from_secs(60); + let mut schedule_accounted = false; loop { let mut latest_cycle_end = 0; let mut versions_observed = false; @@ -1237,14 +1285,36 @@ mod tests { let versions_scanned = metrics["versions_scanned"] .as_u64() .ok_or("scanner status is missing its version-coverage counter")?; - latest_cycle_end = latest_cycle_end.max(cycle_end); + let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs(); + if cycle_end > now { + return Err("scanner cycle timestamp is in the future".into()); + } + let schedule = &status["cycle_schedule"]; + if !schedule_accounted && schedule["execution_role"].as_str() == Some("leader") { + let active = metrics["current_cycle_active"] + .as_bool() + .ok_or("scanner status is missing its active-cycle flag")?; + if active || (cycle_end != 0 && schedule["effective_interval_available"].as_bool() == Some(true)) { + let interval = schedule["effective_interval_seconds"].as_u64(); + let budget = scanner_cycle_completion_budget(cycle_end, interval, now, active)?; + deadline = Instant::now().checked_add(budget).ok_or("scanner deadline overflows")?; + schedule_accounted = true; + } + } + // Superseded walks complete all sets while withholding quota authority. + // Deferred and partial attempts do not certify completed scanning. + if matches!(metrics["last_cycle_result"].as_str(), Some("success" | "superseded")) { + latest_cycle_end = latest_cycle_end.max(cycle_end); + } versions_observed |= versions_scanned > 0; observations.push(format!( - "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}", + "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}, partial_reason={}, schedule={}", metrics["current_cycle"], metrics["current_cycle_active"], metrics["leader_lock_state"], metrics["last_cycle_result"], + metrics["last_cycle_partial_reason"], + schedule, )); } // The coordinator records cycle completion, but remote workers @@ -1555,7 +1625,8 @@ mod tests { } if background_enabled { - wait_for_scanner_cycle_after(&cluster, 0).await?; + let baseline_cycle_floor = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs(); + wait_for_scanner_cycle_after(&cluster, baseline_cycle_floor).await?; } cluster.stop_node(1)?; @@ -1898,11 +1969,16 @@ mod tests { let status_body = signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key).await?; let status: serde_json::Value = serde_json::from_str(&status_body) .map_err(|err| format!("pre-interrupt background heal status is not JSON ({err}): {status_body}"))?; - if background_enabled || only_admin_heal_is_active(&status) { + let admin_active = background_rejoin_heal_evidence + || (status["state"].as_str() == Some("active") + && status["healOperations"]["activeBySource"]["admin"].as_u64() == Some(1)); + if admin_active && (background_enabled || only_admin_heal_is_active(&status)) { break status; } if Instant::now() >= partial_deadline { - return Err(format!("non-admin heal did not drain before the interruption deadline: {status}").into()); + return Err( + format!("heal tasks did not reach the required interruption state before the deadline: {status}").into(), + ); } sleep(Duration::from_millis(50)).await; }; @@ -2562,4 +2638,157 @@ mod tests { }; assert!(!outage_candidate_targets_replacement(&insufficient_peer_shards, 12, replacement_set_slot)); } + async fn two_node_heal_status( + cluster: &RustFSTestClusterEnvironment, + bucket: Option<&str>, + recreate: bool, + scan_mode: u8, + ) -> Result> { + let url = format!("{}/rustfs/admin/v3/heal/{}", cluster.nodes[0].url, bucket.unwrap_or_default()); + let options = serde_json::json!({ + "recursive": true, "dryRun": false, "remove": false, + "recreate": recreate, "scanMode": scan_mode, "updateParity": false, "nolock": false, + }); + let started = + start_root_heal_when_control_ready(&url, &options.to_string(), &cluster.access_key, &cluster.secret_key).await?; + let started: serde_json::Value = serde_json::from_str(&started)?; + let token = started["clientToken"] + .as_str() + .ok_or_else(|| format!("missing heal token: {started}"))?; + let status_url = format!("{url}?clientToken={token}"); + let deadline = Instant::now() + Duration::from_secs(240); + loop { + let body = timeout( + Duration::from_secs(20), + signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key), + ) + .await??; + let status: serde_json::Value = serde_json::from_str(&body)?; + match status["summary"].as_str() { + Some("finished" | "stopped") => return Ok(status), + Some("running") if Instant::now() < deadline => sleep(Duration::from_millis(250)).await, + _ => return Err(format!("heal did not reach a terminal state: {status}").into()), + } + } + } + + async fn verify_two_node_directory_recovery( + remove_bucket: bool, + scan_mode: u8, + bucket_scoped: bool, + recreate: bool, + ) -> Result<(), Box> { + init_logging(); + let mut cluster = RustFSTestClusterEnvironment::new(2).await?; + // Isolate explicit administrator recovery from scanner, MRF, and disk healing. + for key in [ + "RUSTFS_HEAL_AUTO_HEAL_ENABLE", + "RUSTFS_HEAL_MRF_ENABLE", + "RUSTFS_SCANNER_ENABLED", + "RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", + ] { + cluster.set_env(key, "false"); + } + cluster.start().await?; + let source = cluster.create_s3_client(0)?; + let target = cluster.create_s3_client(1)?; + let bucket = "two-node-directory-recovery"; + source.create_bucket().bucket(bucket).send().await?; + let objects = [ + ("empty", Vec::new()), + ("nested/inline", vec![0x3c; 1024]), + ("nested/external", (0_u8..=255).cycle().take(1024 * 1024).collect::>()), + ]; + for (key, bytes) in &objects { + source + .put_object() + .bucket(bucket) + .key(*key) + .body(ByteStream::from(bytes.clone())) + .send() + .await?; + } + let target_bucket = Path::new(&cluster.nodes[1].data_dir).join(bucket); + let original = objects + .iter() + .map(|(key, _)| census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None)) + .collect::, _>>()?; + for census in &original { + assert!(census.has_xl_meta, "fixture must have a target metadata copy: {census:?}"); + } + assert!( + !original[2].present_part_fingerprints.is_empty(), + "large fixture must exercise external parts" + ); + let removed = Path::new(&cluster.temp_dir).join("removed"); + tokio::fs::create_dir(&removed).await?; + if remove_bucket { + tokio::fs::rename(&target_bucket, removed.join(bucket)).await?; + } else { + for (index, (key, _)) in objects.iter().enumerate() { + tokio::fs::rename(target_bucket.join(key), removed.join(index.to_string())).await?; + } + } + let status = two_node_heal_status(&cluster, bucket_scoped.then_some(bucket), recreate, scan_mode).await?; + assert_eq!(status["summary"], "finished", "administrator recovery should complete: {status}"); + assert_eq!(status["outcome"]["execution"]["state"], "completed"); + assert_eq!(status["outcome"]["coverage"], "complete"); + for ((key, _), before) in objects.iter().zip(&original) { + let after = census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None)?; + assert!(after.has_xl_meta, "metadata must be restored before GET: {key}"); + assert_eq!( + after.present_part_fingerprints, before.present_part_fingerprints, + "external shard bytes must match: {key}" + ); + assert_eq!( + after.inline_data_fingerprint, before.inline_data_fingerprint, + "inline shard bytes must match: {key}" + ); + } + cluster.stop_node(0)?; + for (key, expected) in &objects { + let actual = timeout(Duration::from_secs(30), async { + target + .get_object() + .bucket(bucket) + .key(*key) + .send() + .await? + .body + .collect() + .await + .map(|body| body.into_bytes()) + .map_err(Box::::from) + }) + .await??; + assert_eq!(actual.len(), expected.len(), "target-only read length: {key}"); + assert_eq!(sha256_hex(&actual), sha256_hex(expected), "target-only content: {key}"); + } + Ok(()) + } + + #[tokio::test] + async fn test_two_node_bucket_discovery_and_explicit_recreate() -> Result<(), Box> { + verify_two_node_directory_recovery(true, 2, true, true).await + } + + #[tokio::test] + async fn test_two_node_missing_bucket_default_cluster_recovery() -> Result<(), Box> { + verify_two_node_directory_recovery(true, 2, false, false).await + } + + #[tokio::test] + async fn test_two_node_missing_bucket_default_scoped_recovery() -> Result<(), Box> { + verify_two_node_directory_recovery(true, 2, true, false).await + } + + #[tokio::test] + async fn test_two_node_object_directory_recovery_normal() -> Result<(), Box> { + verify_two_node_directory_recovery(false, 1, false, false).await + } + + #[tokio::test] + async fn test_two_node_object_directory_recovery_deep() -> Result<(), Box> { + verify_two_node_directory_recovery(false, 2, false, false).await + } } diff --git a/crates/e2e_test/src/internode_rpc_signature_e2e_test.rs b/crates/e2e_test/src/internode_rpc_signature_e2e_test.rs index a311aa76e..dcc911e77 100644 --- a/crates/e2e_test/src/internode_rpc_signature_e2e_test.rs +++ b/crates/e2e_test/src/internode_rpc_signature_e2e_test.rs @@ -264,7 +264,7 @@ async fn call_make_volume_response( request: MakeVolumeRequest, headers: HeaderMap, ) -> Result, Status> { - let mut client = node_service_time_out_client_no_auth(&url.to_string()) + let mut client = node_service_time_out_client_no_auth(url) .await .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?; let mut rpc_request = Request::new(request); @@ -273,7 +273,7 @@ async fn call_make_volume_response( } async fn call_ping_response(url: &str, headers: HeaderMap) -> Result, Status> { - let mut client = node_service_time_out_client_no_auth(&url.to_string()) + let mut client = node_service_time_out_client_no_auth(url) .await .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?; let mut rpc_request = Request::new(PingRequest { diff --git a/crates/e2e_test/src/lib.rs b/crates/e2e_test/src/lib.rs index b4e0f4ce9..939544ed7 100644 --- a/crates/e2e_test/src/lib.rs +++ b/crates/e2e_test/src/lib.rs @@ -138,6 +138,10 @@ mod admin_auth_test; #[cfg(test)] mod existing_object_tag_policy_test; +/// IAM / bucket / STS session policy with `s3:if-none-match` / `s3:if-match` conditions (E2E). +#[cfg(test)] +mod conditional_write_policy_test; + #[cfg(test)] mod sts_query_compat_test; @@ -179,6 +183,9 @@ mod compression_test; #[cfg(test)] mod delete_objects_versioning_test; +#[cfg(test)] +mod conditional_delete_objects_test; + #[cfg(test)] mod delete_authorization_test; diff --git a/crates/e2e_test/src/multipart_auth_test.rs b/crates/e2e_test/src/multipart_auth_test.rs index 613bf0839..4ce076808 100644 --- a/crates/e2e_test/src/multipart_auth_test.rs +++ b/crates/e2e_test/src/multipart_auth_test.rs @@ -1001,6 +1001,382 @@ async fn test_anonymous_post_object_rejects_sse_kms_params_outside_policy_condit Ok(()) } +#[tokio::test] +async fn test_object_creation_operation_separates_staging_from_publication() -> Result<(), Box> { + use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart}; + init_logging(); + let mut env = RustFSTestEnvironment::new().await?; + env.start_rustfs_server(vec![]).await?; + let client = env.create_s3_client(); + let bucket = "object-creation-operation"; + client.create_bucket().bucket(bucket).send().await?; + client + .put_object() + .bucket(bucket) + .key("source") + .body(ByteStream::from_static(b"copied-part")) + .send() + .await?; + let policy = |deny_creation: bool| { + serde_json::json!({ + "Version": "2012-10-17", + "Statement": [ + { "Effect": "Allow", "Principal": "*", "Action": "s3:PutObject", "Resource": format!("arn:aws:s3:::{bucket}/*") }, + { "Effect": "Deny", "Principal": "*", "Action": "s3:PutObject", "Resource": format!("arn:aws:s3:::{bucket}/target*"), + "Condition": { "Bool": { "s3:ObjectCreationOperation": deny_creation.to_string() } } }, + { "Effect": "Deny", "Principal": "*", "Action": "s3:GetObject", "Resource": format!("arn:aws:s3:::{bucket}/*"), + "Condition": { "Null": { "s3:ObjectCreationOperation": "false" } } } + ] + }).to_string() + }; + client.put_bucket_policy().bucket(bucket).policy(policy(true)).send().await?; + let upload = client + .create_multipart_upload() + .bucket(bucket) + .key("target-multipart") + .send() + .await?; + let upload_id = upload.upload_id().expect("multipart initiation should return upload ID"); + let first_body = vec![b'a'; 5 * 1024 * 1024]; + let part = client + .upload_part() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .part_number(1) + .body(ByteStream::from(first_body.clone())) + .send() + .await?; + let copy = client + .upload_part_copy() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .part_number(2) + .copy_source(format!("{bucket}/source")) + .send() + .await?; + let completed = CompletedMultipartUpload::builder() + .parts( + CompletedPart::builder() + .part_number(1) + .e_tag(part.e_tag().expect("part ETag")) + .build(), + ) + .parts( + CompletedPart::builder() + .part_number(2) + .e_tag(copy.copy_part_result().and_then(|part| part.e_tag()).expect("copy part ETag")) + .build(), + ) + .build(); + let error = client + .complete_multipart_upload() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .multipart_upload(completed.clone()) + .send() + .await + .expect_err("publication must be denied"); + assert_eq!(error.as_service_error().and_then(|error| error.code()), Some("AccessDenied")); + let error = client + .put_object() + .bucket(bucket) + .key("target-put") + .body(ByteStream::from_static(b"blocked")) + .customize() + .mutate_request(|req| { + req.headers_mut().insert("objectcreationoperation", "false"); + }) + .send() + .await + .expect_err("a spoofed header must not exempt PutObject"); + assert_eq!(error.as_service_error().and_then(|error| error.code()), Some("AccessDenied")); + let error = client + .copy_object() + .bucket(bucket) + .key("target-copy") + .copy_source(format!("{bucket}/source")) + .send() + .await + .expect_err("CopyObject publishes an object"); + assert_eq!(error.as_service_error().and_then(|error| error.code()), Some("AccessDenied")); + let archive = make_tar(&[("target-member", b"blocked member")], &[]).await; + let error = client + .put_object() + .bucket(bucket) + .key("archive.tar") + .body(ByteStream::from(archive)) + .customize() + .mutate_request(|req| { + req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true"); + }) + .send() + .await + .expect_err("each extracted member must authorize object creation"); + assert_eq!(error.as_service_error().and_then(|error| error.code()), Some("AccessDenied")); + let missing = client + .head_object() + .bucket(bucket) + .key("target-member") + .send() + .await + .expect_err("denied archive member must not exist"); + assert!(missing.as_service_error().is_some_and(|error| error.is_not_found())); + for deny_creation in [true, false] { + client + .put_bucket_policy() + .bucket(bucket) + .policy(policy(deny_creation)) + .send() + .await?; + let form = reqwest::multipart::Form::new() + .text("key", "target-post") + .text("success_action_status", "201") + .part( + "file", + reqwest::multipart::Part::bytes(b"post body".to_vec()) + .file_name("upload.txt") + .mime_str("text/plain")?, + ); + let response = local_http_client() + .post(format!("{}/{bucket}", env.url)) + .multipart(form) + .send() + .await?; + assert_eq!( + response.status(), + if deny_creation { + reqwest::StatusCode::FORBIDDEN + } else { + reqwest::StatusCode::CREATED + }, + "POST object creation={deny_creation}" + ); + } + let error = client + .create_multipart_upload() + .bucket(bucket) + .key("target-denied") + .send() + .await + .expect_err("Bool=false must deny multipart staging under the inverted policy"); + assert_eq!(error.as_service_error().and_then(|error| error.code()), Some("AccessDenied")); + client + .complete_multipart_upload() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .multipart_upload(completed) + .send() + .await?; + let object = client + .get_object() + .bucket(bucket) + .key("target-multipart") + .send() + .await? + .body + .collect() + .await? + .into_bytes(); + let mut expected = first_body; + expected.extend_from_slice(b"copied-part"); + assert_eq!(object.as_ref(), expected, "completion must publish both regular and copied parts"); + Ok(()) +} + +/// A vended STS credential whose inline session policy follows AWS's conditional-write +/// example 1: object publication needs a request header, multipart staging is exempt +/// through `s3:ObjectCreationOperation`. `s3:x-amz-storage-class` stands in for +/// `s3:if-none-match` because only the former is a header SDK requests can carry here. +#[tokio::test] +async fn test_object_creation_operation_in_sts_session_policy() -> Result<(), Box> { + use crate::common::{ + AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, + awscurl_post_sts_form_urlencoded, build_test_s3_config, + }; + use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart, StorageClass}; + init_logging(); + let mut env = RustFSTestEnvironment::new().await?; + env.start_rustfs_server(vec![]).await?; + let admin = env.create_s3_client(); + let bucket = "object-creation-sts"; + admin.create_bucket().bucket(bucket).send().await?; + admin + .put_object() + .bucket(bucket) + .key("source") + .body(ByteStream::from_static(b"copied-part")) + .send() + .await?; + + let (parent, parent_secret, parent_policy) = ("objcreate-sts-parent", "objcreate-sts-parent-secret", "objcreate-sts-rw"); + let (url, access_key, secret_key) = (env.url.as_str(), env.access_key.as_str(), env.secret_key.as_str()); + admin_create_user_via(AdminTransport::Signed, url, access_key, secret_key, parent, parent_secret).await?; + let read_write = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [ + { "Effect": "Allow", "Action": ["s3:*"], "Resource": ["arn:aws:s3:::*"] }, + { "Effect": "Allow", "Action": ["sts:AssumeRole"], "Resource": ["arn:aws:s3:::*"] } + ] + }) + .to_string(); + admin_add_canned_policy_via(AdminTransport::Signed, url, access_key, secret_key, parent_policy, &read_write).await?; + admin_attach_user_policy_via(AdminTransport::Signed, url, access_key, secret_key, parent_policy, parent).await?; + + let objects = format!("arn:aws:s3:::{bucket}/*"); + let session_policy = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [ + { "Effect": "Allow", "Action": "s3:PutObject", "Resource": objects, + "Condition": { "Null": { "s3:x-amz-storage-class": "false" } } }, + { "Effect": "Allow", "Action": "s3:PutObject", "Resource": objects, + "Condition": { "Bool": { "s3:ObjectCreationOperation": "false" } } }, + { "Effect": "Allow", "Action": "s3:GetObject", "Resource": objects } + ] + }) + .to_string(); + let body = format!( + "Action=AssumeRole&Version=2011-06-15&DurationSeconds=3600&Policy={}", + urlencoding::encode(&session_policy) + ); + let xml = + awscurl_post_sts_form_urlencoded(&format!("{}/", env.url.trim_end_matches('/')), &body, parent, parent_secret).await?; + let field = |tag: &str| -> Result> { + let open = format!("<{tag}>"); + let start = xml.find(&open).ok_or(format!("missing {tag} in AssumeRole response"))? + open.len(); + let end = xml[start..].find(&format!("")).ok_or(format!("unterminated {tag}"))? + start; + Ok(xml[start..end].to_string()) + }; + let session = aws_sdk_s3::Client::from_conf(build_test_s3_config( + &env.url, + &field("AccessKeyId")?, + &field("SecretAccessKey")?, + Some(&field("SessionToken")?), + "object-creation-sts-session", + )); + let denied = |error: Option<&str>, what: &str| assert_eq!(error, Some("AccessDenied"), "{what}"); + + let error = session + .put_object() + .bucket(bucket) + .key("target-put") + .body(ByteStream::from_static(b"blocked")) + .send() + .await + .expect_err("PutObject without the header must be denied"); + denied(error.as_service_error().and_then(|error| error.code()), "PutObject without the header"); + session + .put_object() + .bucket(bucket) + .key("target-put") + .storage_class(StorageClass::Standard) + .body(ByteStream::from_static(b"allowed")) + .send() + .await?; + let error = session + .copy_object() + .bucket(bucket) + .key("target-copy") + .copy_source(format!("{bucket}/source")) + .send() + .await + .expect_err("CopyObject without the header must be denied"); + denied(error.as_service_error().and_then(|error| error.code()), "CopyObject without the header"); + session + .copy_object() + .bucket(bucket) + .key("target-copy") + .copy_source(format!("{bucket}/source")) + .storage_class(StorageClass::Standard) + .send() + .await?; + + let upload = session + .create_multipart_upload() + .bucket(bucket) + .key("target-multipart") + .send() + .await?; + let upload_id = upload.upload_id().expect("multipart initiation should return upload ID"); + let first_body = vec![b'a'; 5 * 1024 * 1024]; + let part = session + .upload_part() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .part_number(1) + .body(ByteStream::from(first_body.clone())) + .send() + .await?; + let copy = session + .upload_part_copy() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .part_number(2) + .copy_source(format!("{bucket}/source")) + .send() + .await?; + let completed = CompletedMultipartUpload::builder() + .parts( + CompletedPart::builder() + .part_number(1) + .e_tag(part.e_tag().expect("part ETag")) + .build(), + ) + .parts( + CompletedPart::builder() + .part_number(2) + .e_tag(copy.copy_part_result().and_then(|part| part.e_tag()).expect("copy part ETag")) + .build(), + ) + .build(); + let error = session + .complete_multipart_upload() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .multipart_upload(completed.clone()) + .send() + .await + .expect_err("CompleteMultipartUpload without the header must be denied"); + denied( + error.as_service_error().and_then(|error| error.code()), + "CompleteMultipartUpload without the header", + ); + session + .complete_multipart_upload() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .multipart_upload(completed) + .customize() + .mutate_request(|req| { + req.headers_mut() + .insert("x-amz-storage-class", HeaderValue::from_static("STANDARD")); + }) + .send() + .await?; + + let object = session + .get_object() + .bucket(bucket) + .key("target-multipart") + .send() + .await? + .body + .collect() + .await? + .into_bytes(); + let mut expected = first_body; + expected.extend_from_slice(b"copied-part"); + assert_eq!(object.as_ref(), expected, "completion must publish both regular and copied parts"); + Ok(()) +} + #[tokio::test] async fn test_anonymous_multipart_control_apis_require_auth() -> Result<(), Box> { init_logging(); diff --git a/crates/e2e_test/src/namespace_lock_quorum_test.rs b/crates/e2e_test/src/namespace_lock_quorum_test.rs index 46a59a943..7f9e8c760 100644 --- a/crates/e2e_test/src/namespace_lock_quorum_test.rs +++ b/crates/e2e_test/src/namespace_lock_quorum_test.rs @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::common::RustFSTestClusterEnvironment; +use crate::common::{ClusterTopology, RustFSTestClusterEnvironment}; use aws_sdk_s3::Client; use aws_sdk_s3::error::SdkError; use bytes::Bytes; @@ -185,21 +185,113 @@ async fn wait_for_quorum_read_admission(clients: &[Client], bucket: &str) -> Tes return Ok(()); } if tokio::time::Instant::now() >= deadline { - return Err(format!("read quorum did not become available after lease convergence for {bucket}").into()); + let diagnostics = clients[0].get_object().bucket(bucket).key("warm-small").send().await; + return Err(format!( + "read quorum did not become available after lease convergence for {bucket}: GET={diagnostics:?}" + ) + .into()); } tokio::time::sleep(std::time::Duration::from_millis(100)).await; } } +#[tokio::test] +async fn test_cold_degraded_object_mutations_require_only_storage_quorum() -> TestResult { + crate::common::init_logging(); + let mut cluster = RustFSTestClusterEnvironment::new(4).await?; + cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2"); + cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false"); + cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false"); + cluster.set_env("RUSTFS_GET_OBJECT_METADATA_CACHE_DISTRIBUTED_ENABLE", "false"); + cluster.start().await?; + let clients = cluster + .create_all_clients()? + .into_iter() + .map(|client| { + Client::from_conf( + client + .config() + .to_builder() + .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1)) + .build(), + ) + }) + .collect::>(); + let bucket = "cold-degraded-mutations"; + clients[0].create_bucket().bucket(bucket).send().await?; + // No object mutation may warm a process-wide configuration probe before + // the missing peer is introduced. Three shards still satisfy EC:2 writes. + cluster.stop_node(3)?; + let admission_deadline = Instant::now() + Duration::from_secs(30); + for client in &clients[..3] { + loop { + match client + .get_object() + .bucket(bucket) + .key("never-created-readiness-probe") + .send() + .await + { + Err(error) if error.as_service_error().and_then(|error| error.meta().code()) == Some("NoSuchKey") => break, + Err(error) + if error.raw_response().is_some_and(|response| response.status().as_u16() == 503) + && Instant::now() < admission_deadline => + { + tokio::time::sleep(Duration::from_millis(100)).await; + } + Err(error) => return Err(error.into()), + Ok(_) => panic!("a cold readiness probe must not create an object"), + } + } + } + let body = b"first write while one peer is offline"; + clients[0] + .put_object() + .bucket(bucket) + .key("object") + .body(Bytes::from_static(body).into()) + .send() + .await?; + for client in &clients[..3] { + assert_quorum_object_body(client, bucket, "object", body).await?; + } + // This coordinator has not previously committed an object mutation either. + clients[1].delete_object().bucket(bucket).key("object").send().await?; + let error = clients[2] + .get_object() + .bucket(bucket) + .key("object") + .send() + .await + .expect_err("acknowledged deletion must be visible at another entrypoint"); + assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("NoSuchKey")); + + cluster.stop_node(2)?; + let error = clients[0] + .put_object() + .bucket(bucket) + .key("below-quorum") + .body(Bytes::from_static(body).into()) + .send() + .await + .expect_err("removing the cache gate must not weaken storage write quorum"); + assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("ServiceUnavailable")); + cluster.stop(); + Ok(()) +} + #[tokio::test] async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResult { crate::common::init_logging(); - for (node_count, parity) in [(4, 2), (6, 3), (6, 2)] { + for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] { let read_quorum = node_count - parity; let write_quorum = read_quorum + usize::from(read_quorum == parity); let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?; cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}")); + if parity == 0 { + cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0"); + } // Wait for every seed fanout before removing any physical shard. cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false"); cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false"); @@ -338,6 +430,277 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul Ok(()) } +#[tokio::test] +async fn test_cold_start_read_quorum_follows_erasure_layout() -> TestResult { + crate::common::init_logging(); + for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] { + let read_quorum = node_count - parity; + let write_quorum = read_quorum + usize::from(read_quorum == parity); + let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?; + cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}")); + if parity == 0 { + cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0"); + } + cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false"); + cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false"); + cluster.set_env("RUST_LOG", "warn"); + cluster.start().await?; + let clients = cluster + .create_all_clients()? + .into_iter() + .map(|client| { + Client::from_conf( + client + .config() + .to_builder() + .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1)) + .build(), + ) + }) + .collect::>(); + let bucket = format!("cold-read-quorum-{node_count}-{parity}"); + clients[0].create_bucket().bucket(&bucket).send().await?; + let small = b"cold startup follows the stored EC layout".to_vec(); + let large = vec![73; 256 * 1024]; + for (key, body) in [ + ("warm-small", &small), + ("warm-large", &large), + ("cold-large", &large), + ("below-quorum", &large), + ] { + clients[0] + .put_object() + .bucket(&bucket) + .key(key) + .body(Bytes::copy_from_slice(body).into()) + .send() + .await?; + for node in &cluster.nodes { + let census = + crate::chaos::census_object_version_on_disk(std::path::Path::new(&node.data_dir), &bucket, key, None)?; + assert!(census.is_complete(), "seed shard must exist before cold startup: {census:?}"); + assert_eq!((census.data_blocks, census.parity_blocks), (Some(read_quorum), Some(parity))); + } + } + for node in 0..node_count { + cluster.stop_node(node)?; + } + // Rebuild every process's in-memory state below read quorum, then + // cross the EC boundary. A format bootstrap must not admit S3 reads + // merely because half the peers are online (e.g. six disks, EC:2). + for node in 0..read_quorum - 1 { + cluster.start_node_process(node).await?; + } + let error = clients[0] + .get_object() + .bucket(&bucket) + .key("below-quorum") + .send() + .await + .expect_err("cold startup below the configured EC read quorum must stay unavailable"); + assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503)); + cluster.start_node_process(read_quorum - 1).await?; + wait_for_quorum_read_admission(&clients[..read_quorum], &bucket).await?; + for client in clients.iter().take(read_quorum) { + assert_quorum_object_body(client, &bucket, "warm-small", &small).await?; + assert_quorum_object_body(client, &bucket, "cold-large", &large).await?; + let write = client + .put_object() + .bucket(&bucket) + .key("cold-quorum-write") + .body(Bytes::copy_from_slice(&small).into()) + .send() + .await; + if read_quorum >= write_quorum { + write?; + assert_quorum_object_body(client, &bucket, "cold-quorum-write", &small).await?; + } else { + let error = write.expect_err("cold startup at read quorum must preserve the EC write quorum"); + assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503)); + } + } + for node in read_quorum..node_count { + cluster.start_node(node).await?; + } + for client in &clients { + assert_quorum_object_body(client, &bucket, "cold-large", &large).await?; + } + } + Ok(()) +} + +#[tokio::test] +async fn test_four_by_six_cold_start_at_read_quorum() -> TestResult { + crate::common::init_logging(); + let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 6)).await?; + // Explicit URLs are consumed in order. Place one drive from each node in + // every set, matching a four-host, six-drive-per-host deployment. + let volumes = (0..6) + .flat_map(|drive| { + cluster + .nodes + .iter() + .map(move |node| format!("http://{}{}", node.address, node.data_dirs[drive])) + }) + .collect::>() + .join(" "); + cluster.set_env("RUSTFS_VOLUMES", volumes); + cluster.set_env("RUSTFS_ERASURE_SET_DRIVE_COUNT", "4"); + cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2"); + cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false"); + cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false"); + cluster.set_env("RUST_LOG", "error"); + cluster.start().await?; + let clients = cluster + .create_all_clients()? + .into_iter() + .map(|client| { + Client::from_conf( + client + .config() + .to_builder() + .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1)) + .build(), + ) + }) + .collect::>(); + let bucket = "cold-start-read-quorum"; + clients[0].create_bucket().bucket(bucket).send().await?; + let mut fixtures = Vec::new(); + let mut covered = [[false; 2]; 6]; + for index in 0..128 { + let key = match index { + 0 => "warm-small".to_string(), + 1 => "warm-large".to_string(), + _ => format!("cold-{index}"), + }; + let size_class = index % 2; + let size = if size_class == 0 { 71 } else { 256 * 1024 }; + let body = (0..size) + .map(|offset| u8::try_from(offset % 251).expect("bounded fixture byte")) + .collect::>(); + clients[0] + .put_object() + .bucket(bucket) + .key(&key) + .body(Bytes::copy_from_slice(&body).into()) + .send() + .await?; + for node in &cluster.nodes { + let mut complete_shards = 0; + for (drive, path) in node.data_dirs.iter().enumerate() { + let census = crate::chaos::census_object_version_on_disk(std::path::Path::new(path), bucket, &key, None)?; + if census.has_xl_meta { + assert!(census.is_complete(), "seed shard must be complete before shutdown: {census:?}"); + assert_eq!((census.data_blocks, census.parity_blocks), (Some(2), Some(2))); + covered[drive][size_class] = true; + complete_shards += 1; + } + } + assert_eq!(complete_shards, 1, "each node must own one shard of every seed object"); + } + fixtures.push((key, body)); + if covered.iter().flatten().all(|covered| *covered) { + break; + } + } + assert!( + covered.iter().flatten().all(|covered| *covered), + "fixtures must cover all six sets with inline and large objects" + ); + + for first in 0..4 { + for second in first + 1..4 { + for node in 0..4 { + cluster.stop_node(node)?; + } + cluster.start_node_process(first).await?; + let error = clients[first] + .get_object() + .bucket(bucket) + .key("warm-large") + .send() + .await + .expect_err("one node cannot satisfy read quorum after a cold start"); + assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503)); + cluster.start_node_process(second).await?; + wait_for_quorum_read_admission(&[clients[first].clone(), clients[second].clone()], bucket).await?; + for node in [first, second] { + for (key, body) in &fixtures { + assert_quorum_object_body(&clients[node], bucket, key, body).await?; + } + let range = clients[node] + .get_object() + .bucket(bucket) + .key("warm-large") + .range("bytes=17-1023") + .send() + .await?; + assert_eq!(range.body.collect().await?.into_bytes().as_ref(), &fixtures[1].1[17..1024]); + let listing = clients[node].list_objects_v2().bucket(bucket).send().await?; + for (key, _) in &fixtures { + assert!( + listing.contents().iter().any(|object| object.key() == Some(key.as_str())), + "listing omitted {key}" + ); + } + let error = clients[node] + .put_object() + .bucket(bucket) + .key("recovery-write") + .body(Bytes::from_static(b"after recovery").into()) + .send() + .await + .expect_err("two nodes must not authorize a write"); + assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503)); + } + let error = clients[first] + .delete_object() + .bucket(bucket) + .key("warm-large") + .send() + .await + .expect_err("two nodes must not authorize a delete"); + assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503)); + assert_quorum_object_body(&clients[first], bucket, "warm-large", &fixtures[1].1).await?; + + let remaining = (0..4).filter(|node| *node != first && *node != second).collect::>(); + cluster.start_node(remaining[0]).await?; + let deadline = tokio::time::Instant::now() + Duration::from_secs(90); + loop { + match clients[first] + .put_object() + .bucket(bucket) + .key("recovery-write") + .body(Bytes::from_static(b"after recovery").into()) + .send() + .await + { + Ok(_) => break, + Err(error) + if error.raw_response().is_some_and(|response| response.status().as_u16() == 503) + && tokio::time::Instant::now() < deadline => + { + tokio::time::sleep(Duration::from_millis(200)).await + } + Err(error) => return Err(error.into()), + } + } + for node in [first, second, remaining[0]] { + assert_quorum_object_body(&clients[node], bucket, "recovery-write", b"after recovery").await?; + } + cluster.start_node(remaining[1]).await?; + for client in &clients { + for (key, body) in &fixtures { + assert_quorum_object_body(client, bucket, key, body).await?; + } + assert_quorum_object_body(client, bucket, "recovery-write", b"after recovery").await?; + } + } + } + Ok(()) +} + async fn put_object(client: Client, payload: Vec, writer_id: usize) -> Result<(), String> { client .put_object() diff --git a/crates/e2e_test/src/quota_test.rs b/crates/e2e_test/src/quota_test.rs index 7acb51e02..eb5593607 100644 --- a/crates/e2e_test/src/quota_test.rs +++ b/crates/e2e_test/src/quota_test.rs @@ -851,13 +851,20 @@ mod integration_tests { .delete( aws_sdk_s3::types::Delete::builder() .set_objects(Some(objects)) - .quiet(true) + // Quiet mode hides successful entries, so the batch must be + // verbose for the assertion on the deleted list below. + .quiet(false) .build() .unwrap(), ) .send() .await?; + assert!( + delete_result.errors().is_empty(), + "batch delete must not report errors: {:?}", + delete_result.errors() + ); assert_eq!(delete_result.deleted().len(), 2); // Now should be able to upload again (quota freed up) diff --git a/crates/e2e_test/src/reliant/conditional_writes.rs b/crates/e2e_test/src/reliant/conditional_writes.rs index 6bc37980c..60505eb35 100644 --- a/crates/e2e_test/src/reliant/conditional_writes.rs +++ b/crates/e2e_test/src/reliant/conditional_writes.rs @@ -70,7 +70,7 @@ async fn test_conditional_put_okay() -> TestResult { let test_key = "conditional-put-ok"; let initial_data = generate_test_data(1024); // 1KB test data let matching_data = generate_test_data(2048); // 2KB updated data - let non_matching_data = generate_test_data(3072); // 3KB updated data + let created_data = generate_test_data(3072); // 3KB new object // Upload initial object and get its ETag let initial_etag = upload_object_with_metadata(&client, TEST_BUCKET, test_key, &initial_data).await?; @@ -86,17 +86,17 @@ async fn test_conditional_put_okay() -> TestResult { .await?; assert_eq!(object_body(&client, test_key).await?.as_ref(), matching_data); - // Test 2: PUT with non-matching If-None-Match condition (should succeed) - let fake_etag = "\"fake-etag-12345\""; + // Test 2: PUT with If-None-Match: * on a missing key (should succeed) + let created_key = "conditional-put-ok-created"; client .put_object() .bucket(TEST_BUCKET) - .key(test_key) - .body(Bytes::from(non_matching_data.clone()).into()) - .if_none_match(fake_etag) + .key(created_key) + .body(Bytes::from(created_data.clone()).into()) + .if_none_match("*") .send() .await?; - assert_eq!(object_body(&client, test_key).await?.as_ref(), non_matching_data); + assert_eq!(object_body(&client, created_key).await?.as_ref(), created_data); Ok(()) } @@ -130,17 +130,30 @@ async fn test_conditional_put_failed() -> TestResult { assert_s3_error_code(response1, "PreconditionFailed"); assert_eq!(object_body(&client, test_key).await?.as_ref(), initial_data); - // Test 2: PUT with matching If-None-Match condition (should fail with 412) - let response2 = client - .put_object() + // Test 2: writes accept If-None-Match only as `*`; RustFS rejects an ETag + // value with 501 NotImplemented whether or not it matches. + for if_none_match in [initial_etag.as_str(), "\"fake-etag-12345\""] { + let response = client + .put_object() + .bucket(TEST_BUCKET) + .key(test_key) + .body(Bytes::from(updated_data.clone()).into()) + .if_none_match(if_none_match) + .send() + .await; + assert_s3_error_code(response, "NotImplemented"); + } + + let response = client + .copy_object() .bucket(TEST_BUCKET) .key(test_key) - .body(Bytes::from(updated_data.clone()).into()) - .if_none_match(&initial_etag) + .copy_source(format!("{TEST_BUCKET}/{test_key}")) + .metadata_directive(aws_sdk_s3::types::MetadataDirective::Replace) + .if_none_match("\"fake-etag-12345\"") .send() .await; - - assert_s3_error_code(response2, "PreconditionFailed"); + assert_s3_error_code(response, "NotImplemented"); assert_eq!(object_body(&client, test_key).await?.as_ref(), initial_data); Ok(()) @@ -252,7 +265,7 @@ async fn test_conditional_multi_part_upload() -> TestResult { assert_s3_error_code(complete_response, "PreconditionFailed"); - // Test 2: Multipart upload with matching If-None-Match, should fail + // Test 2: Multipart upload with an unsupported ETag If-None-Match let complete_response = client .complete_multipart_upload() .bucket(TEST_BUCKET) @@ -263,7 +276,7 @@ async fn test_conditional_multi_part_upload() -> TestResult { .send() .await; - assert_s3_error_code(complete_response, "PreconditionFailed"); + assert_s3_error_code(complete_response, "NotImplemented"); // Test 3: Multipart upload with unmatching If-Match, should fail let complete_response = client diff --git a/crates/e2e_test/src/reliant/grpc_lock_server.rs b/crates/e2e_test/src/reliant/grpc_lock_server.rs index 1bffebdb4..386d9b155 100644 --- a/crates/e2e_test/src/reliant/grpc_lock_server.rs +++ b/crates/e2e_test/src/reliant/grpc_lock_server.rs @@ -113,6 +113,13 @@ impl NodeService for MinimalLockNodeService { Err(Status::unimplemented("MinimalLockNodeService only supports lock RPCs")) } + async fn acquire_quota_mutation_fence( + &self, + _request: Request, + ) -> Result, Status> { + Err(Status::unimplemented("MinimalLockNodeService only supports lock RPCs")) + } + async fn renew_snapshot_lease( &self, _request: Request, @@ -127,6 +134,13 @@ impl NodeService for MinimalLockNodeService { Err(Status::unimplemented("MinimalLockNodeService only supports lock RPCs")) } + async fn release_quota_mutation_fence( + &self, + _request: Request, + ) -> Result, Status> { + Err(Status::unimplemented("MinimalLockNodeService only supports lock RPCs")) + } + async fn acquire_scanner_publication_lease( &self, _request: Request, @@ -465,6 +479,13 @@ impl NodeService for MinimalLockNodeService { Err(Status::unimplemented("lock-only test server")) } + async fn recover_delete_transaction( + &self, + _request: Request, + ) -> Result, Status> { + Err(Status::unimplemented("delete recovery is not supported by the lock fixture")) + } + async fn settle_part_transaction( &self, _request: Request, @@ -569,6 +590,13 @@ impl NodeService for MinimalLockNodeService { Err(Status::unimplemented("lock-only test server")) } + async fn rename_data_with_quota_fence( + &self, + _request: Request, + ) -> Result, Status> { + Err(Status::unimplemented("lock-only test server")) + } + async fn make_volumes( &self, _request: Request, @@ -667,6 +695,13 @@ impl NodeService for MinimalLockNodeService { Err(Status::unimplemented("lock-only test server")) } + async fn guarded_delete_versions( + &self, + _request: Request, + ) -> Result, Status> { + Err(Status::unimplemented("lock-only test server")) + } + async fn delete_versions( &self, _request: Request, diff --git a/crates/e2e_test/src/replication_extension_test.rs b/crates/e2e_test/src/replication_extension_test.rs index ab98d0cb0..e4a4c72a1 100644 --- a/crates/e2e_test/src/replication_extension_test.rs +++ b/crates/e2e_test/src/replication_extension_test.rs @@ -14,8 +14,8 @@ use crate::common::{ AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user, - awscurl_post_sts_form_urlencoded, init_logging, local_http_client, replication_fast_env, rustfs_binary_path, signed_request, - signed_request_with_client, signed_request_with_session_token, + awscurl_post_sts_form_urlencoded, init_logging, local_http_client, parse_assume_role_credentials, replication_fast_env, + rustfs_binary_path, signed_request, signed_request_with_client, signed_request_with_session_token, }; use crate::fake_s3_target::{ FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction as FakeTargetFault, Operation as FakeTargetOperation, @@ -387,21 +387,6 @@ pub(crate) struct ReplicationResetStatusTarget { pub(crate) object: String, } -fn extract_xml_tag(xml: &str, tag: &str) -> Option { - let open = format!("<{tag}>"); - let close = format!(""); - let start = xml.find(&open)? + open.len(); - let end = xml[start..].find(&close)? + start; - Some(xml[start..end].to_string()) -} - -fn parse_assume_role_credentials(xml: &str) -> Result<(String, String, String), Box> { - let access_key = extract_xml_tag(xml, "AccessKeyId").ok_or("missing AccessKeyId in AssumeRole response")?; - let secret_key = extract_xml_tag(xml, "SecretAccessKey").ok_or("missing SecretAccessKey in AssumeRole response")?; - let session_token = extract_xml_tag(xml, "SessionToken").ok_or("missing SessionToken in AssumeRole response")?; - Ok((access_key, secret_key, session_token)) -} - pub(crate) struct ReplicationTargetOptions<'a> { pub(crate) endpoint: &'a str, pub(crate) access_key: &'a str, @@ -3945,7 +3930,15 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou (sse_key, sse_put.version_id().ok_or("SSE PUT omitted version ID")?.to_string()), (large_key, large_put.version_id().ok_or("large PUT omitted version ID")?.to_string()), ]; - assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?; + // Managed-SSE listings may omit ETags. Check the replicated version's body + // before deleting it rather than comparing keyless listing fingerprints. + for (key, version_id) in &purged { + let source_body = get_version_body(&source_client, &source_bucket, key, version_id).await?; + let expected_sha256: [u8; 32] = Sha256::digest(&source_body).into(); + wait_for_replicated_sha256(&target_client, &target_bucket, key, expected_sha256).await?; + let target_body = get_version_body(&target_client, &target_bucket, key, version_id).await?; + assert_eq!(target_body, source_body, "replicated version body differs for {key}/{version_id}"); + } for (key, version_id) in &purged { source_client @@ -3956,10 +3949,6 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou .send() .await?; } - assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?; - let target_state = list_replication_state(&target_client, &target_bucket).await?; - assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}"); - // The purge state is applied on the source asynchronously after the target // acknowledges the delete; only then does the retained version go away and // the bucket become deletable. A listing that is empty while DeleteBucket @@ -3984,6 +3973,9 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou } } + let target_state = list_replication_state(&target_client, &target_bucket).await?; + assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}"); + Ok(()) } diff --git a/crates/e2e_test/src/replication_target_matrix_test.rs b/crates/e2e_test/src/replication_target_matrix_test.rs index 2a3cb973e..743650e72 100644 --- a/crates/e2e_test/src/replication_target_matrix_test.rs +++ b/crates/e2e_test/src/replication_target_matrix_test.rs @@ -45,9 +45,10 @@ use aws_sdk_s3::error::ProvideErrorMetadata; use aws_sdk_s3::primitives::{ByteStream, DateTime}; use aws_sdk_s3::types::{ Checksum, ChecksumAlgorithm, CompletedMultipartUpload, CompletedPart, ObjectAttributes, ObjectLockLegalHold, - ObjectLockLegalHoldStatus, ObjectLockMode, ObjectLockRetention, ObjectLockRetentionMode, Tag, Tagging, + ObjectLockLegalHoldStatus, ObjectLockMode, ObjectLockRetention, ObjectLockRetentionMode, ServerSideEncryption, Tag, Tagging, }; use bytes::Bytes; +use md5::{Digest as _, Md5}; use std::error::Error; use std::time::{SystemTime, UNIX_EPOCH}; use tokio::time::{Duration, sleep, timeout}; @@ -73,14 +74,20 @@ enum TargetMode { /// land, and every version-addressed mutation must resolve the replica /// through the target-version ledger. MintOwnVersionIds, + /// AWS S3 and other generic S3 services: ignores the replication + /// source-ETag headers and reports the MD5 of the plaintext it received. + /// Every single-part replica ETag is therefore checked against the source + /// ETag for real (the other modes echo the source ETag back). + ComputesOwnEtags, } impl TargetMode { - const ALL: [TargetMode; 4] = [ + const ALL: [TargetMode; 5] = [ TargetMode::Baseline, TargetMode::RejectAwsChunked, TargetMode::RequireChecksumWithObjectLock, TargetMode::MintOwnVersionIds, + TargetMode::ComputesOwnEtags, ]; fn apply(self, target: &FakeS3Target) { @@ -92,6 +99,7 @@ impl TargetMode { target.assign_own_version_ids(true); target.reject_unknown_version_deletes(true); } + TargetMode::ComputesOwnEtags => target.ignore_source_etags(true), } } @@ -101,6 +109,7 @@ impl TargetMode { TargetMode::RejectAwsChunked => "reject-aws-chunked", TargetMode::RequireChecksumWithObjectLock => "require-checksum-object-lock", TargetMode::MintOwnVersionIds => "mint-own-version-ids", + TargetMode::ComputesOwnEtags => "computes-own-etags", } } } @@ -126,10 +135,14 @@ enum ObjectShape { /// Single-part object uploaded with `x-amz-checksum-sha256`; the replica /// must carry the same header (rustfs/backlog#2340). Checksummed, + /// Single-part SSE-S3 object. Its ETag is a keyed fingerprint, not the + /// plaintext MD5 (docs/operations/encrypted-etags.md), while the replica + /// travels as plaintext under `x-amz-server-side-encryption: AES256`. + SseS3, } impl ObjectShape { - const ALL: [ObjectShape; 8] = [ + const ALL: [ObjectShape; 9] = [ ObjectShape::Empty, ObjectShape::Plain, ObjectShape::Retention, @@ -138,6 +151,7 @@ impl ObjectShape { ObjectShape::LockedMultipart, ObjectShape::OdmPreservedMd5Multipart, ObjectShape::Checksummed, + ObjectShape::SseS3, ]; fn key(self) -> &'static str { @@ -150,6 +164,7 @@ impl ObjectShape { ObjectShape::LockedMultipart => "matrix/locked-multipart.bin", ObjectShape::OdmPreservedMd5Multipart => "matrix/odm-preserved-md5.bin", ObjectShape::Checksummed => "matrix/checksummed.bin", + ObjectShape::SseS3 => "matrix/sse-s3.bin", } } @@ -233,6 +248,24 @@ impl ObjectShape { .await?; Ok(body) } + ObjectShape::SseS3 => { + let body = payload(56 * 1024, 0x77); + let put = client + .put_object() + .bucket(bucket) + .key(key) + .body(ByteStream::from(body.clone())) + .server_side_encryption(ServerSideEncryption::Aes256) + .send() + .await?; + // The cell only proves something while the source ETag is + // not the content MD5 a generic target reports back. + let plaintext_md5 = hex_simd::encode_to_string(Md5::digest(&body), hex_simd::AsciiCase::Lower); + if put.e_tag().map(|etag| etag.trim_matches('"')) == Some(plaintext_md5.as_str()) { + return Err("SSE-S3 source ETag is the plaintext MD5; the cell no longer exercises encrypted ETags".into()); + } + Ok(body) + } } } } @@ -270,14 +303,40 @@ fn expectation(mode: TargetMode, shape: ObjectShape) -> Expectation { /// - the first PUT lands but its response is lost, so the object is FAILED /// and the scanner heal pass re-drives it; /// - an existing-object resync re-drives a COMPLETED object unconditionally. +/// +/// A permanent delete of the healed version then has to remove exactly the +/// replica the heal pass located. #[tokio::test] async fn matrix_mint_own_version_ids_redrive_does_not_duplicate() -> TestResult { + run_redrive_does_not_duplicate(RedriveObjects::Plaintext).await +} + +/// The same re-drives for SSE-S3 objects against a target that also computes +/// its own ETags (AWS S3). The source ETag is a keyed fingerprint +/// (docs/operations/encrypted-etags.md) while the replica reports the MD5 of +/// the plaintext it received, so no exact-ETag lookup can ever find the +/// replica: the lost-response heal, the resync and the purge must locate it +/// without comparing ETags. +#[tokio::test] +async fn matrix_mint_own_version_ids_encrypted_redrive_does_not_duplicate() -> TestResult { + run_redrive_does_not_duplicate(RedriveObjects::SseS3OnEtagComputingTarget).await +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum RedriveObjects { + Plaintext, + SseS3OnEtagComputingTarget, +} + +async fn run_redrive_does_not_duplicate(objects: RedriveObjects) -> TestResult { init_logging(); + let encrypted = objects == RedriveObjects::SseS3OnEtagComputingTarget; let target = FakeS3Target::start().await?; let target_bucket = "matrix-mint-own-redrive-dst".to_string(); target.create_bucket_with_object_lock(target_bucket.clone()); target.assign_own_version_ids(true); + target.ignore_source_etags(encrypted); let mut env_vars = replication_fast_env(); env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV); @@ -291,6 +350,7 @@ async fn matrix_mint_own_version_ids_redrive_does_not_duplicate() -> TestResult ]); let env = OdmTestEnv::start_with(OdmEnvOptions { env: env_vars, + local_kms: encrypted, ..OdmEnvOptions::default() }) .await?; @@ -319,45 +379,63 @@ async fn matrix_mint_own_version_ids_redrive_does_not_duplicate() -> TestResult }, ) .await?; - put_bucket_replication(source_env, source_bucket, &target_arn).await?; + put_bucket_replication_with_delete_statuses(source_env, source_bucket, &target_arn, "Enabled", Some("Enabled")).await?; + + let put = |key: &'static str, fill: u8, len: usize| { + let request = source_client + .put_object() + .bucket(source_bucket) + .key(key) + .body(ByteStream::from(payload(len, fill))); + let request = if encrypted { + request.server_side_encryption(ServerSideEncryption::Aes256) + } else { + request + }; + async move { request.send().await } + }; // Teach the worker the target's identity contract with one ordinary // write, exactly as production learns it (the PUT response carries the // minted id). let probe_key = "redrive/identity-probe.bin"; - source_client - .put_object() - .bucket(source_bucket) - .key(probe_key) - .body(ByteStream::from(payload(4 * 1024, 0x01))) - .send() - .await?; + let probe = put(probe_key, 0x01, 4 * 1024).await?; assert_eq!( wait_for_terminal_replication_status(&source_client, source_bucket, probe_key).await?, "COMPLETED" ); + if encrypted { + // The variant only proves something while the replica's ETag differs + // from the source's encrypted-ETag fingerprint. + let replica = single_target_version(&target, &target_bucket, probe_key)?; + let replica_etag = fake_source_client(&target) + .head_object() + .bucket(&target_bucket) + .key(probe_key) + .version_id(&replica) + .send() + .await? + .e_tag() + .map(str::to_string); + if replica_etag.is_none() || replica_etag.as_deref() == probe.e_tag() { + return Err(format!( + "the replica ETag {replica_etag:?} must differ from the source fingerprint {:?}", + probe.e_tag() + ) + .into()); + } + } // Shape 1: the PUT is stored, its response never arrives, heal re-drives. let heal_key = "redrive/heal.bin"; target.inject_for_key(FakeTargetOperation::PutObject, heal_key, FakeTargetFault::DisconnectAfterResponse, 1); - source_client - .put_object() - .bucket(source_bucket) - .key(heal_key) - .body(ByteStream::from(payload(8 * 1024, 0x02))) - .send() - .await?; + let healed = put(heal_key, 0x02, 8 * 1024).await?; + let healed_version = healed.version_id().ok_or("source PUT returned no version id")?.to_string(); wait_for_replication_status_and_single_version(&source_client, source_bucket, &target, &target_bucket, heal_key).await?; // Shape 2: an existing-object resync re-drives a COMPLETED object. let resync_key = "redrive/resync.bin"; - source_client - .put_object() - .bucket(source_bucket) - .key(resync_key) - .body(ByteStream::from(payload(8 * 1024, 0x03))) - .send() - .await?; + put(resync_key, 0x03, 8 * 1024).await?; assert_eq!( wait_for_terminal_replication_status(&source_client, source_bucket, resync_key).await?, "COMPLETED" @@ -388,6 +466,20 @@ async fn matrix_mint_own_version_ids_redrive_does_not_duplicate() -> TestResult ); } + // The heal pass recorded the replica it located; a permanent delete of + // that version must remove exactly it. + source_client + .delete_object() + .bucket(source_bucket) + .key(heal_key) + .version_id(&healed_version) + .send() + .await?; + wait_until("permanent delete of the healed replica", || async { + Ok(target.stored_versions(&target_bucket, heal_key).is_empty()) + }) + .await?; + target.shutdown().await; Ok(()) } @@ -871,6 +963,11 @@ async fn matrix_mint_own_version_ids_target() -> TestResult { run_row(TargetMode::MintOwnVersionIds).await } +#[tokio::test] +async fn matrix_computes_own_etags_target() -> TestResult { + run_row(TargetMode::ComputesOwnEtags).await +} + /// Every known-red entry must name a real cell and an issue, and the lookup /// must round-trip, so a stale or mistyped entry cannot silently pin nothing. #[test] @@ -907,6 +1004,8 @@ async fn run_row(mode: TargetMode) -> TestResult { env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]); let env = OdmTestEnv::start_with(OdmEnvOptions { env: env_vars, + // The SseS3 shape needs a KMS for SSE-S3. + local_kms: true, ..OdmEnvOptions::default() }) .await?; diff --git a/crates/e2e_test/src/sts_query_compat_test.rs b/crates/e2e_test/src/sts_query_compat_test.rs index 134b3bfc6..3a703be6d 100644 --- a/crates/e2e_test/src/sts_query_compat_test.rs +++ b/crates/e2e_test/src/sts_query_compat_test.rs @@ -740,7 +740,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult { .version_id(version) .send() .await?; - opa.expect_bucket_tags("s3:GetObject", source, Some("finance")).await?; + opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?; client.list_objects_v2().bucket(source).send().await?; opa.expect_bucket_tags("s3:ListBucket", source, Some("finance")).await?; client.list_object_versions().bucket(source).send().await?; diff --git a/crates/e2e_test/src/upgrade_compatibility_test.rs b/crates/e2e_test/src/upgrade_compatibility_test.rs index 51ff93d6f..01db59002 100644 --- a/crates/e2e_test/src/upgrade_compatibility_test.rs +++ b/crates/e2e_test/src/upgrade_compatibility_test.rs @@ -25,11 +25,11 @@ use aws_sdk_s3::Client; use aws_sdk_s3::error::ProvideErrorMetadata; use aws_sdk_s3::primitives::ByteStream; use aws_sdk_s3::types::{ - BucketLifecycleConfiguration, BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, DefaultRetention, - ExpirationStatus, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter, ObjectAttributes, ObjectLockConfiguration, - ObjectLockEnabled, ObjectLockRetentionMode, ObjectLockRule, PublicAccessBlockConfiguration, ServerSideEncryption, - ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule, Tag, Tagging, - VersioningConfiguration, + BucketLifecycleConfiguration, BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, DefaultRetention, Delete, + ExpirationStatus, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter, ObjectAttributes, ObjectIdentifier, + ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRetentionMode, ObjectLockRule, PublicAccessBlockConfiguration, + ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule, + StorageClass, Tag, Tagging, VersioningConfiguration, }; use http::{Method, StatusCode}; use serde_json::Value; @@ -46,6 +46,7 @@ type BoxError = Box; const SOURCE_BINARY_ENV: &str = "RUSTFS_UPGRADE_SOURCE_BINARY"; const G09_EVIDENCE_DIR_ENV: &str = "RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR"; const RC5_COMMIT: &str = "40a2470feb567201165a5b809b7598bb4b1f68f5"; +const STABLE_1_0_1_COMMIT: &str = "6de965ae3c965a78ff819fbcd7acd4aa44177d92"; const SSE_MASTER_KEY_ENV: &str = "RUSTFS_SSE_S3_MASTER_KEY"; const SSE_MASTER_KEY: &str = "QkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkI="; const PLAIN_BUCKET: &str = "upgrade-plain-data"; @@ -66,6 +67,16 @@ const MIXED_MULTIPART_UPLOAD_TIMEOUT: Duration = Duration::from_secs(30); // probe_interval (2s) x success_threshold (3) after it comes back; 30s // comfortably covers that window plus CI scheduling jitter. const LISTING_CONVERGENCE_TIMEOUT: Duration = Duration::from_secs(30); +// Parity 0 for the reduced-redundancy class only. A parity-0 write on the +// four-drive set needs all four drives (`write_quorum = data_drives`), so a +// PUT with that class succeeds only once the serving node writes through every +// one of its own drive handles. Standard-class objects keep the default layout. +const ALL_DRIVES_STORAGE_CLASS_ENV: (&str, &str) = ("RUSTFS_STORAGE_CLASS_RRS", "EC:0"); +// A peer drops a restarted node's drive to Offline after two failed +// operations and only re-admits it through probe_interval (2s) x +// success_threshold (3) once the node answers again. The budget covers that +// for every node after the slowest restart in the sequence, plus CI jitter. +const DRIVE_READMISSION_TIMEOUT: Duration = Duration::from_secs(60); // Bucket-configuration upgrade/rollback scenarios (rustfs#7172, #7183, #7089). const CONFIG_PLAIN_BUCKET: &str = "upgrade-config-plain"; @@ -509,6 +520,9 @@ async fn observe_mixed_multipart_failure( let network_deadline = started + Duration::from_secs(4); let path = format!("/{MIXED_BUCKET}/{}", failure.key); let _ = tokio::time::timeout_at(deadline, async { + // Every node keeps its own lock table, and the writer that stalls is + // usually a previous-release node; the current node's snapshot alone + // cannot show which holder blocks the upload. let (heads, locks) = tokio::join!( futures::future::join_all((0..cluster.nodes.len()).map(|node| probe_mixed_multipart_failure( cluster, @@ -517,7 +531,13 @@ async fn observe_mixed_multipart_failure( true, network_deadline ))), - probe_mixed_multipart_failure(cluster, 0, "/rustfs/admin/v3/top/locks?count=1000", false, network_deadline), + futures::future::join_all((0..cluster.nodes.len()).map(|node| probe_mixed_multipart_failure( + cluster, + node, + "/rustfs/admin/v3/top/locks?count=1000", + false, + network_deadline + ))), ); let snapshot = serde_json::json!({ "phase": phase, "bucket": MIXED_BUCKET, "key": failure.key, @@ -579,6 +599,7 @@ async fn exercise_mixed_cluster( ) -> TestResult { let clients = cluster.create_all_clients()?; wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, phase, LISTING_CONVERGENCE_TIMEOUT).await?; + wait_for_drive_readmission(&clients, MIXED_BUCKET, phase, DRIVE_READMISSION_TIMEOUT).await?; let current_client = &clients[current_node]; let previous_client = &clients[previous_node]; @@ -633,6 +654,26 @@ async fn exercise_mixed_cluster( vec![u8::try_from(MULTIPART_WORKERS - 1)?; 64 * 1024] ); + // Every online participant must understand the delete decision protocol. + // An upgrade phase can serve reads/writes while intentionally refusing a + // modern DELETE; rejecting it must leave the old writer's seed intact. + let error = upgrade_probe_client(current_client) + .delete_object() + .bucket(MIXED_BUCKET) + .key(PREVIOUS_RELEASE_SEED_KEY) + .send() + .await + .expect_err("a modern DELETE must fail closed while an old participant is online"); + assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503), "{error:?}"); + assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("ServiceUnavailable")); + for (node, client) in clients.iter().enumerate() { + assert_eq!( + read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1, + UPGRADE_READINESS_BODY, + "{phase}: rejected DELETE must preserve the seed through node {node}" + ); + } + Ok(()) } @@ -656,16 +697,51 @@ pub(crate) async fn wait_for_upgrade_write_readiness( // health also accepts Returning disks whose write health is still FAULTY. // Probe every writer outside the asserted phase prefix; compatibility // writes still execute once and retain their original assertions. + wait_for_upgrade_writes(clients, bucket, phase, "write readiness", None, budget).await +} + +/// Block until every node commits a parity-0 write, which needs all of the +/// node's own drive handles. +/// +/// Write readiness proves that a node accepts a standard write at quorum; it +/// does not prove that the node has re-admitted the drives it marked Offline +/// while a peer restarted. A writer that still excludes one drive commits at +/// exactly three of four drives and cannot ride out any other slow drive. The +/// cluster must run with [`ALL_DRIVES_STORAGE_CLASS_ENV`]; a write with the +/// reduced-redundancy class then fails with 503 `ServiceUnavailable` until the +/// serving node writes through every drive, on the previous release and on +/// the current build alike. +async fn wait_for_drive_readmission(clients: &[Client], bucket: &str, context: &str, budget: Duration) -> TestResult { + wait_for_upgrade_writes( + clients, + bucket, + context, + "drive re-admission", + Some(StorageClass::ReducedRedundancy), + budget, + ) + .await +} + +async fn wait_for_upgrade_writes( + clients: &[Client], + bucket: &str, + phase: &str, + probe: &str, + storage_class: Option, + budget: Duration, +) -> TestResult { let deadline = Instant::now() + budget; for (node, client) in clients.iter().enumerate() { let client = upgrade_probe_client(client); - let key = format!(".upgrade-readiness/{phase}/node-{node}"); + let key = match storage_class { + Some(_) => format!(".upgrade-readiness/{phase}/all-drives/node-{node}"), + None => format!(".upgrade-readiness/{phase}/node-{node}"), + }; let mut last_response = "no response".to_string(); loop { if Instant::now() >= deadline { - return Err( - format!("{phase}: node {node} write readiness deadline exceeded; last response: {last_response}").into(), - ); + return Err(format!("{phase}: node {node} {probe} deadline exceeded; last response: {last_response}").into()); } let response = tokio::time::timeout_at( deadline, @@ -673,6 +749,7 @@ pub(crate) async fn wait_for_upgrade_write_readiness( .put_object() .bucket(bucket) .key(&key) + .set_storage_class(storage_class.clone()) .body(ByteStream::from_static(UPGRADE_READINESS_BODY)) .send(), ) @@ -683,13 +760,13 @@ pub(crate) async fn wait_for_upgrade_write_readiness( if error.raw_response().map(|response| response.status().as_u16()) != Some(503) || error.as_service_error().and_then(ProvideErrorMetadata::code) != Some("ServiceUnavailable") { - return Err(format!("{phase}: node {node} write readiness failed: {error:?}").into()); + return Err(format!("{phase}: node {node} {probe} failed: {error:?}").into()); } last_response = format!("{error:?}"); } Err(_) => { return Err(format!( - "{phase}: node {node} write readiness deadline exceeded during PutObject; last response: {last_response}" + "{phase}: node {node} {probe} deadline exceeded during PutObject; last response: {last_response}" ) .into()); } @@ -700,6 +777,18 @@ pub(crate) async fn wait_for_upgrade_write_readiness( Ok(()) } +fn previous_baseline_writers(clients: &[Client], restarted_node: usize) -> &[Client] { + // Node 0 has persisted the seed. With restart order [1, 2, 3, 0], only + // this prefix has also cleared rc.5's startup write fence. Every RRS + // probe still needs all drives, including those of not-yet-restarted peers. + let ready_end = if restarted_node == 0 { + clients.len() + } else { + restarted_node + 1 + }; + &clients[..ready_end] +} + async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnvironment, previous_binary: &Path) -> TestResult { let clients: Vec<_> = cluster.create_all_clients()?.iter().map(upgrade_probe_client).collect(); // rc.5 can latch a non-elected node's write fence while its first startup @@ -707,16 +796,30 @@ async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnviro // persist data, then restart each old process once with three peers still // readable. This preparation ends before any current binary is started. wait_for_upgrade_write_readiness(&clients[..1], MIXED_BUCKET, "previous-seed", LISTING_CONVERGENCE_TIMEOUT).await?; + // A node that just came back is still replaying its startup traffic against + // every peer. Killing the next node during that burst gives the fresh node + // two failures in a row and it drops that peer's drive to Offline, where + // only the slow recovery probe can bring it back. Let every writable node + // re-admit every drive before the next kill. Nodes still awaiting their + // restart may retain the startup write fence and cannot coordinate a PUT. for node in [1, 2, 3, 0] { cluster.stop_node(node)?; cluster.start_node_from_binary(node, previous_binary).await?; + let context = format!("previous-restart-{node}"); wait_for_upgrade_write_readiness( std::slice::from_ref(&clients[node]), MIXED_BUCKET, - &format!("previous-restart-{node}"), + &context, LISTING_CONVERGENCE_TIMEOUT, ) .await?; + wait_for_drive_readmission( + previous_baseline_writers(&clients, node), + MIXED_BUCKET, + &context, + DRIVE_READMISSION_TIMEOUT, + ) + .await?; } wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?; @@ -738,61 +841,243 @@ async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnviro Ok(()) } +/// The drive re-admission gate assumes that a parity-0 write needs every +/// drive. Prove it on the workspace build: with one node down a standard +/// write still commits, the reduced-redundancy write fails with 503, and once +/// the node is back the gate converges within its budget. +#[tokio::test] +async fn parity_zero_write_needs_every_drive() -> TestResult { + init_logging(); + let mut cluster = RustFSTestClusterEnvironment::new(MIXED_NODE_COUNT).await?; + cluster.set_env("RUST_LOG", "rustfs=warn,rustfs_notify=warn"); + cluster.set_env(ALL_DRIVES_STORAGE_CLASS_ENV.0, ALL_DRIVES_STORAGE_CLASS_ENV.1); + cluster.start().await?; + cluster.create_test_bucket(MIXED_BUCKET).await?; + let clients = cluster.create_all_clients()?; + wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, "parity-zero-seed", LISTING_CONVERGENCE_TIMEOUT).await?; + wait_for_drive_readmission(&clients, MIXED_BUCKET, "parity-zero-healthy", DRIVE_READMISSION_TIMEOUT).await?; + + cluster.stop_node(3)?; + let writer = upgrade_probe_client(&clients[0]); + writer + .put_object() + .bucket(MIXED_BUCKET) + .key("parity-zero/standard-with-one-node-down") + .body(ByteStream::from_static(UPGRADE_READINESS_BODY)) + .send() + .await?; + let error = writer + .put_object() + .bucket(MIXED_BUCKET) + .key("parity-zero/all-drives-with-one-node-down") + .storage_class(StorageClass::ReducedRedundancy) + .body(ByteStream::from_static(UPGRADE_READINESS_BODY)) + .send() + .await + .expect_err("a parity-0 write must not commit at three of four drives"); + assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503), "{error:?}"); + assert_eq!( + error.as_service_error().and_then(ProvideErrorMetadata::code), + Some("ServiceUnavailable"), + "{error:?}" + ); + + cluster.start_node(3).await?; + wait_for_drive_readmission(&clients, MIXED_BUCKET, "parity-zero-returned", DRIVE_READMISSION_TIMEOUT).await?; + Ok(()) +} + #[cfg(test)] mod upgrade_write_readiness_tests { use super::*; use crate::fake_s3_target::FaultAction; + use aws_sdk_s3::config::interceptors::BeforeTransmitInterceptorContextRef; + use aws_sdk_s3::config::{ConfigBag, Intercept, RuntimeComponents}; + + #[derive(Debug)] + struct RequireAllDrivesProbe; + + impl Intercept for RequireAllDrivesProbe { + fn name(&self) -> &'static str { + "require-all-drives-probe" + } + + fn read_before_transmit( + &self, + context: &BeforeTransmitInterceptorContextRef<'_>, + _runtime_components: &RuntimeComponents, + _cfg: &mut ConfigBag, + ) -> Result<(), BoxError> { + assert_eq!(context.request().method(), "PUT"); + assert_eq!( + context.request().headers().get("x-amz-storage-class"), + Some("REDUCED_REDUNDANCY"), + "every admitted coordinator must use the all-drive storage class" + ); + Ok(()) + } + } #[tokio::test] - async fn waits_for_each_writer_after_metadata_is_ready() -> TestResult { - let target = FakeS3Target::start().await?; - let bucket = "distributed-upgrade-history"; - target.create_bucket(bucket); - let client = fake_source_client(&target); - client.head_bucket().bucket(bucket).send().await?; + async fn previous_baseline_probes_only_writable_coordinators_until_all_restart() -> TestResult { + let mut targets = Vec::new(); + let mut clients = Vec::new(); + for _ in 0..MIXED_NODE_COUNT { + let target = FakeS3Target::start().await?; + target.create_bucket(MIXED_BUCKET); + clients.push(Client::from_conf( + fake_source_client(&target) + .config() + .to_builder() + .interceptor(RequireAllDrivesProbe) + .build(), + )); + targets.push(target); + } + for target in &targets[2..] { + target.inject(FakeTargetOperation::PutObject, FaultAction::Status(StatusCode::INTERNAL_SERVER_ERROR), 10); + } + // Reproduce the original all-coordinator gate: node 0 and restarted + // node 1 can write, but unrestarted node 2 still returns a latched 500. + let error = wait_for_drive_readmission(&clients, MIXED_BUCKET, "premature", Duration::from_secs(5)) + .await + .expect_err("an unrestarted coordinator must retain its startup write failure"); + assert!(error.to_string().contains("node 2 drive re-admission failed"), "{error}"); + let failed = targets[2].requests(); + assert_eq!(failed.len(), 1, "the startup 500 must fail without retrying"); + assert_eq!(failed[0].fault, Some(FaultAction::Status(StatusCode::INTERNAL_SERVER_ERROR))); + for target in &targets { + target.take_requests(); + } + + for (restarted_node, expected_writers) in [(1, 2), (2, 3), (3, 4), (0, 4)] { + // A restart clears only this process's startup write fence. + targets[restarted_node].clear_faults(); + let phase = format!("previous-restart-{restarted_node}"); + wait_for_drive_readmission( + previous_baseline_writers(&clients, restarted_node), + MIXED_BUCKET, + &phase, + Duration::from_secs(5), + ) + .await?; + for (node, target) in targets.iter().enumerate() { + let key = format!(".upgrade-readiness/{phase}/all-drives/node-{node}"); + let admitted = node < expected_writers; + assert_eq!( + target.count_requests(FakeTargetOperation::PutObject, &key), + usize::from(admitted), + "{phase}: only writable node {node} may coordinate an all-drive probe" + ); + assert_eq!(target.has_object(MIXED_BUCKET, &key), admitted); + assert_eq!(target.take_requests().len(), usize::from(admitted)); + } + } + + // Once every node is admitted, its permanent errors must still fail + // the full gate immediately rather than being skipped or retried. + let phase = "previous-ready-error"; + let key = format!(".upgrade-readiness/{phase}/all-drives/node-3"); + targets[3].inject_for_key( + FakeTargetOperation::PutObject, + &key, + FaultAction::Status(StatusCode::INTERNAL_SERVER_ERROR), + 1, + ); + let error = + wait_for_drive_readmission(previous_baseline_writers(&clients, 0), MIXED_BUCKET, phase, Duration::from_secs(5)) + .await + .expect_err("a recovered coordinator's 500 must still fail the final gate"); + assert!(error.to_string().contains("node 3 drive re-admission failed"), "{error}"); + assert_eq!(targets[3].count_requests(FakeTargetOperation::PutObject, &key), 1); + assert!(!targets[3].has_object(MIXED_BUCKET, &key)); + for target in targets { + target.shutdown().await; + } + Ok(()) + } + + #[tokio::test(start_paused = true)] + async fn waits_for_each_writer_after_metadata_is_ready() -> TestResult { + use aws_sdk_s3::config::{Credentials, Region, RequestChecksumCalculation}; + use aws_smithy_http_client::test_util::{ReplayEvent, StaticReplayClient}; + use aws_smithy_types::body::SdkBody; + + let bucket = "distributed-upgrade-history"; let phase = "one-previous-node"; let first_key = format!(".upgrade-readiness/{phase}/node-0"); let second_key = format!(".upgrade-readiness/{phase}/node-1"); - target.inject_for_key( - FakeTargetOperation::PutObject, - &first_key, - FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE), - 2, - ); - target.inject_for_key( - FakeTargetOperation::PutObject, - &second_key, - FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE), - 1, + let endpoint = "http://upgrade-readiness.example.invalid"; + let event = |method: Method, key: &str, status: StatusCode| { + let (body, query) = if method == Method::PUT { + (SdkBody::from(UPGRADE_READINESS_BODY), "?x-id=PutObject") + } else { + (SdkBody::empty(), "") + }; + ReplayEvent::new( + http::Request::builder() + .method(method) + .uri(format!("{endpoint}/{bucket}/{key}{query}")) + .body(body) + .unwrap(), + http::Response::builder() + .status(status) + .header("content-type", "application/xml") + .body(if status == StatusCode::SERVICE_UNAVAILABLE { + SdkBody::from("ServiceUnavailable") + } else { + SdkBody::empty() + }) + .unwrap(), + ) + }; + // Keep transport scheduling outside the probe's time budget. The + // replay checks every writer's key and body without real network I/O. + let transport = StaticReplayClient::new(vec![ + event(Method::HEAD, "", StatusCode::OK), + event(Method::PUT, &first_key, StatusCode::SERVICE_UNAVAILABLE), + event(Method::PUT, &first_key, StatusCode::SERVICE_UNAVAILABLE), + event(Method::PUT, &first_key, StatusCode::OK), + event(Method::PUT, &second_key, StatusCode::SERVICE_UNAVAILABLE), + event(Method::PUT, &second_key, StatusCode::OK), + ]); + let client = Client::from_conf( + aws_sdk_s3::Config::builder() + .behavior_version_latest() + .credentials_provider(Credentials::new(FAKE_ACCESS_KEY, FAKE_SECRET_KEY, None, None, "readiness-test")) + .region(Region::new("us-east-1")) + .endpoint_url(endpoint) + .force_path_style(true) + .retry_config(aws_sdk_s3::config::retry::RetryConfig::disabled()) + .request_checksum_calculation(RequestChecksumCalculation::WhenRequired) + .http_client(transport.clone()) + .build(), ); + client.head_bucket().bucket(bucket).send().await?; + // Metadata readiness does not prove that a data write can succeed. let premature = client .put_object() .bucket(bucket) .key(&first_key) - .body(ByteStream::from_static(b"upgrade write readiness")) + .body(ByteStream::from_static(UPGRADE_READINESS_BODY)) .send() .await .expect_err("metadata readiness does not prove write readiness"); assert_eq!(premature.raw_response().map(|response| response.status().as_u16()), Some(503)); - wait_for_upgrade_write_readiness(&[client.clone(), client.clone()], bucket, phase, Duration::from_secs(5)).await?; - assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &first_key), 3); - assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &second_key), 2); - assert!(target.has_object(bucket, &first_key)); - assert!(target.has_object(bucket, &second_key)); - assert!( - client - .list_objects_v2() - .bucket(bucket) - .prefix(format!("{phase}/")) - .send() - .await? - .contents() - .is_empty() + let started = Instant::now(); + wait_for_upgrade_write_readiness(&[client.clone(), client], bucket, phase, Duration::from_secs(5)).await?; + assert_eq!(started.elapsed(), Duration::from_secs(1), "each unavailable writer must be polled again"); + transport.assert_requests_match(&[]); + assert_eq!( + transport + .actual_requests() + .map(|request| request.method().to_string()) + .collect::>(), + ["HEAD", "PUT", "PUT", "PUT", "PUT", "PUT"], ); - target.shutdown().await; Ok(()) } @@ -963,7 +1248,14 @@ mod upgrade_write_readiness_tests { ); assert_eq!(target.count_requests(FakeTargetOperation::HeadObject, stalled_key), MIXED_NODE_COUNT); assert_eq!(target.count_requests(FakeTargetOperation::CompleteMultipartUpload, stalled_key), 1); - assert_eq!(snapshot["top_locks"]["observation"], "unknown", "the fake peer has no admin lock API"); + let locks = snapshot["top_locks"] + .as_array() + .expect("failure artifact must retain every node's lock probe"); + assert_eq!(locks.len(), MIXED_NODE_COUNT); + for (node, probe) in locks.iter().enumerate() { + assert_eq!(probe["node"], node); + assert_eq!(probe["observation"], "unknown", "the fake peer has no admin lock API"); + } let locks: MixedLockSnapshot = serde_json::from_value(serde_json::json!({ "total": 1, "truncated": true, "capability_note": "partial snapshot", @@ -1082,7 +1374,7 @@ async fn rc5_rollback_requires_restoring_odm_configuration() -> TestResult { #[tokio::test] #[ignore = "requires a pinned previous RustFS release binary"] -async fn direct_upgrade_from_rc2_preserves_object_contracts() -> TestResult { +async fn direct_upgrade_from_previous_release_preserves_object_contracts() -> TestResult { init_logging(); let previous_binary = source_binary()?; let mut env = RustFSTestEnvironment::new().await?; @@ -1232,13 +1524,14 @@ async fn direct_upgrade_from_rc2_preserves_object_contracts() -> TestResult { #[tokio::test] #[ignore = "requires a pinned previous RustFS release binary"] -async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestResult { +async fn rolling_upgrade_from_previous_release_preserves_mixed_version_contracts() -> TestResult { init_logging(); let previous_binary = source_binary()?; let evidence_context = g09_evidence_context(&previous_binary).await?; let current_binary = rustfs_binary_path(); let mut cluster = RustFSTestClusterEnvironment::new(MIXED_NODE_COUNT).await?; cluster.set_env("RUST_LOG", "rustfs=warn,rustfs_notify=warn"); + cluster.set_env(ALL_DRIVES_STORAGE_CLASS_ENV.0, ALL_DRIVES_STORAGE_CLASS_ENV.1); configure_cluster_logs(&mut cluster)?; cluster.start_with_binary(&previous_binary).await?; cluster.create_test_bucket(MIXED_BUCKET).await?; @@ -1248,22 +1541,24 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes cluster.start_node_from_binary(0, ¤t_binary).await?; exercise_mixed_cluster(&cluster, "one-current-node", 0, 1).await?; - for node_idx in [1, 2] { + for (node_idx, phase) in [(1, "two-current-nodes"), (2, "one-previous-node")] { cluster.stop_node(node_idx)?; cluster.start_node_from_binary(node_idx, ¤t_binary).await?; + exercise_mixed_cluster(&cluster, phase, 0, 3).await?; } - exercise_mixed_cluster(&cluster, "one-previous-node", 0, 3).await?; cluster.stop_node(3)?; cluster.start_node_from_binary(3, ¤t_binary).await?; + let clients = cluster.create_all_clients()?; + wait_for_drive_readmission(&clients, MIXED_BUCKET, "homogeneous-current", DRIVE_READMISSION_TIMEOUT).await?; - for (node_idx, client) in cluster.create_all_clients()?.iter().enumerate() { + for (node_idx, client) in clients.iter().enumerate() { assert_eq!( read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1, UPGRADE_READINESS_BODY, "current node {node_idx} must retain the previous-release seed" ); - for phase in ["one-current-node", "one-previous-node"] { + for phase in ["one-current-node", "two-current-nodes", "one-previous-node"] { wait_for_phase_listing( client, phase, @@ -1275,14 +1570,14 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes } if let Some(context) = evidence_context.as_ref() { - let phases = ["one-current-node", "one-previous-node"]; + let phases = ["one-current-node", "two-current-nodes", "one-previous-node"]; let objects_per_phase = MULTIPART_WORKERS * MULTIPART_UPLOADS_PER_WORKER + 2; write_g09_evidence( context, "mixed_version_reader_evidence", "mixed-version-reader", &["old-writer-new-reader", "new-writer-old-reader"], - "upgrade_compatibility_test::rolling_upgrade_from_rc2_preserves_mixed_version_contracts", + "upgrade_compatibility_test::rolling_upgrade_from_previous_release_preserves_mixed_version_contracts", serde_json::json!({ "bucket": MIXED_BUCKET, "phases": phases, @@ -1300,7 +1595,7 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes "mixed_version_writer_evidence", "mixed-version-writer", &["old-reader-new-writer", "new-reader-old-writer"], - "upgrade_compatibility_test::rolling_upgrade_from_rc2_preserves_mixed_version_contracts", + "upgrade_compatibility_test::rolling_upgrade_from_previous_release_preserves_mixed_version_contracts", serde_json::json!({ "bucket": MIXED_BUCKET, "phases": phases, @@ -1308,12 +1603,337 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes "assertions": [ "current writer publishes objects readable by previous-release node", "previous-release writer publishes objects readable by current node", - "multipart writers continue under one-current-node and one-previous-node layouts" + "multipart writers continue at every individual node upgrade", + "modern DELETE fails closed and preserves every byte while old peers remain" ], }), )?; } + let clients = cluster.create_all_clients()?; + wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, "all-current", LISTING_CONVERGENCE_TIMEOUT).await?; + clients[0] + .delete_object() + .bucket(MIXED_BUCKET) + .key(PREVIOUS_RELEASE_SEED_KEY) + .send() + .await?; + for (node, client) in clients.iter().enumerate() { + let error = client + .get_object() + .bucket(MIXED_BUCKET) + .key(PREVIOUS_RELEASE_SEED_KEY) + .send() + .await + .expect_err("the homogeneous cluster must apply the previously blocked DELETE"); + assert_eq!( + error.as_service_error().and_then(ProvideErrorMetadata::code), + Some("NoSuchKey"), + "node {node}" + ); + } + + Ok(()) +} + +struct LegacyDeleteObject { + bucket: String, + key: String, + version: Option, + etag: String, + body: Vec, + data_dirs: Vec, +} + +fn legacy_payload_directories(object_dir: &Path) -> Result, BoxError> { + let entries = match std::fs::read_dir(object_dir) { + Ok(entries) => entries, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()), + Err(error) => return Err(error.into()), + }; + let mut directories = Vec::new(); + for entry in entries { + let entry = entry?; + if entry.file_type()?.is_dir() && uuid::Uuid::parse_str(&entry.file_name().to_string_lossy()).is_ok() { + for child in std::fs::read_dir(entry.path())? { + let child = child?; + if child.file_type()?.is_file() && child.file_name().to_string_lossy().starts_with("part.") { + directories.push(entry.path()); + break; + } + } + } + } + Ok(directories) +} + +async fn seed_legacy_delete_object( + env: &RustFSTestEnvironment, + client: &Client, + bucket: &str, + key: &str, + seed: u64, +) -> Result { + let object_dir = Path::new(&env.temp_dir).join(bucket).join(key); + let before = legacy_payload_directories(&object_dir)?; + // Incompressible external shards ensure the regression exercises payload + // reclamation, rather than only the inline xl.meta path. + let body = layout_noise(1024 * 1024 + 17, seed); + let response = client + .put_object() + .bucket(bucket) + .key(key) + .body(ByteStream::from(body.clone())) + .send() + .await?; + let data_dirs: Vec<_> = legacy_payload_directories(&object_dir)? + .into_iter() + .filter(|path| !before.contains(path)) + .collect(); + assert_eq!(data_dirs.len(), 1, "the old binary must create one external payload for {bucket}/{key}"); + // A suspended PUT can omit the version header while reads identify the + // stored generation as "null". Preserve the old reader's exact identity. + let head = client + .head_object() + .bucket(bucket) + .key(key) + .set_version_id(response.version_id().map(str::to_owned)) + .send() + .await?; + assert_eq!(head.e_tag(), response.e_tag(), "old writer and reader agree on the stored bytes"); + if let Some(version) = response.version_id() { + assert_eq!(head.version_id(), Some(version), "old reader resolves the written version"); + } + Ok(LegacyDeleteObject { + bucket: bucket.to_owned(), + key: key.to_owned(), + version: head.version_id().map(str::to_owned), + etag: response.e_tag().ok_or("the old writer omitted its ETag")?.to_owned(), + body, + data_dirs, + }) +} + +async fn assert_legacy_delete_object(client: &Client, object: &LegacyDeleteObject) -> TestResult { + let response = client + .get_object() + .bucket(&object.bucket) + .key(&object.key) + .set_version_id(object.version.clone()) + .send() + .await + .map_err(|error| format!("GET {}/{} version {:?}: {error:?}", object.bucket, object.key, object.version))?; + assert_eq!( + response.e_tag(), + Some(object.etag.as_str()), + "{}/{}: original ETag", + object.bucket, + object.key + ); + assert_eq!( + response.version_id(), + object.version.as_deref(), + "{}/{}: original version", + object.bucket, + object.key + ); + assert_eq!(response.body.collect().await?.into_bytes().as_ref(), object.body); + Ok(()) +} + +fn preserve_legacy_upgrade_volume(env: &RustFSTestEnvironment, label: &str) -> TestResult { + let Some(log_dir) = std::env::var_os("RUSTFS_E2E_LOG_DIR") else { return Ok(()) }; + let root = Path::new(&env.temp_dir); + let destination = Path::new(&log_dir).join(label); + for entry in walkdir::WalkDir::new(root).follow_links(false) { + let entry = entry?; + let target = destination.join(entry.path().strip_prefix(root)?); + if entry.file_type().is_dir() { + std::fs::create_dir_all(target)?; + } else if entry.file_type().is_file() { + std::fs::copy(entry.path(), target)?; + } else { + return Err("the stopped upgrade fixture contains a non-regular entry".into()); + } + } + Ok(()) +} + +#[tokio::test] +#[ignore = "requires the pinned stable 1.0.1 release binary"] +async fn stable_upgrade_deletes_foreign_reservations_without_losing_live_objects() -> TestResult { + use sha2::Digest; + + init_logging(); + let previous_binary = source_binary()?; + assert_eq!(binary_source_revision(&previous_binary).await?, STABLE_1_0_1_COMMIT); + let mut env = RustFSTestEnvironment::new().await?; + env.start_rustfs_server_from_binary(&previous_binary, vec![], &[]).await?; + let old_client = env.create_s3_client(); + let plain_bucket = "legacy-delete-plain"; + let versioned_bucket = "legacy-delete-versioned"; + let suspended_bucket = "legacy-delete-suspended"; + for bucket in [plain_bucket, versioned_bucket, suspended_bucket] { + env.create_test_bucket(bucket).await?; + } + enable_versioning(&old_client, versioned_bucket).await?; + enable_versioning(&old_client, suspended_bucket).await?; + old_client + .put_bucket_versioning() + .bucket(suspended_bucket) + .versioning_configuration( + VersioningConfiguration::builder() + .status(BucketVersioningStatus::Suspended) + .build(), + ) + .send() + .await?; + + let mut deleted = Vec::new(); + for (index, (bucket, key)) in [ + (plain_bucket, "single"), + (plain_bucket, "batch-one"), + (plain_bucket, "batch-two"), + (versioned_bucket, "versioned"), + (suspended_bucket, "suspended"), + ] + .into_iter() + .enumerate() + { + deleted.push(seed_legacy_delete_object(&env, &old_client, bucket, key, index as u64 + 1).await?); + } + let successor = seed_legacy_delete_object(&env, &old_client, versioned_bucket, "versioned", 20).await?; + let child_key = format!("single/{}/child", uuid::Uuid::new_v4()); + let child = seed_legacy_delete_object(&env, &old_client, plain_bucket, &child_key, 21).await?; + for object in deleted.iter().chain([&successor, &child]) { + assert_legacy_delete_object(&old_client, object).await?; + } + env.stop_server(); + + // The stable release produced the complete volume and every xl.meta/part + // byte above. Only the empty legacy marker is fault-injected after stop; + // this deterministically models an interrupted reservation, not a claimed + // reproduction of the old binary's SIGKILL timing window. + let mut evidence = Vec::new(); + for object in &deleted { + let metadata_path = Path::new(&env.temp_dir) + .join(&object.bucket) + .join(&object.key) + .join("xl.meta"); + let metadata = std::fs::read(&metadata_path)?; + for directory in &object.data_dirs { + let marker = directory.join(format!("reserve-delete-data.{}", uuid::Uuid::new_v4())); + std::fs::File::create(marker)?.sync_all()?; + } + assert_eq!( + std::fs::read(&metadata_path)?, + metadata, + "fault injection must preserve the old xl.meta bytes" + ); + evidence.push(serde_json::json!({ + "bucket": object.bucket, + "key": object.key, + "version": object.version, + "etag": object.etag, + "xl_meta_sha256": hex_simd::encode_to_string(sha2::Sha256::digest(&metadata), hex_simd::AsciiCase::Lower), + })); + } + if let Some(directory) = std::env::var_os("RUSTFS_E2E_LOG_DIR") { + std::fs::create_dir_all(&directory)?; + std::fs::write( + Path::new(&directory).join("stable-delete-source.json"), + serde_json::to_vec_pretty(&serde_json::json!({ + "writer_revision": STABLE_1_0_1_COMMIT, + "candidate_revision": current_source_revision()?, + "fault_injection": "empty foreign reservation marker after stopping the genuine old writer", + "objects": evidence, + }))?, + )?; + } + + preserve_legacy_upgrade_volume(&env, "stable-source-volume")?; + env.restart_server_preserving_data(vec![], &[]).await?; + let client = env.create_s3_client(); + for object in deleted.iter().chain([&successor, &child]) { + if let Err(error) = assert_legacy_delete_object(&client, object).await { + env.stop_server(); + preserve_legacy_upgrade_volume(&env, "failed-upgrade-volume")?; + return Err(error); + } + } + client.delete_object().bucket(plain_bucket).key("single").send().await?; + let response = client + .delete_objects() + .bucket(plain_bucket) + .delete( + Delete::builder() + .objects(ObjectIdentifier::builder().key("batch-one").build()?) + .objects(ObjectIdentifier::builder().key("batch-two").build()?) + .build()?, + ) + .send() + .await?; + assert!(response.errors().is_empty(), "batch deletion must converge: {:?}", response.errors()); + assert_eq!(response.deleted().len(), 2); + for object in &deleted[3..] { + client + .delete_object() + .bucket(&object.bucket) + .key(&object.key) + .set_version_id(object.version.clone()) + .send() + .await?; + } + + // Repeat recovery across process restarts, then retry the same requests. + // The old committed payload must disappear while the successor and the + // UUID-shaped prefix child remain intact through every pass. + for pass in 0..2 { + env.restart_server_preserving_data(vec![], &[]).await?; + let client = env.create_s3_client(); + for object in &deleted { + let error = client + .get_object() + .bucket(&object.bucket) + .key(&object.key) + .set_version_id(object.version.clone()) + .send() + .await + .expect_err("an acknowledged deletion must not resurrect after restart"); + assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(404)); + for directory in &object.data_dirs { + assert!( + !directory.exists(), + "pass {pass}: deleted payload and foreign reservation must be physically reclaimed" + ); + } + client + .delete_object() + .bucket(&object.bucket) + .key(&object.key) + .set_version_id(object.version.clone()) + .send() + .await?; + } + for object in [&successor, &child] { + assert_legacy_delete_object(&client, object).await?; + } + assert_eq!( + read_object(&client, versioned_bucket, "versioned", None).await?.1, + successor.body, + "deleting the old version must preserve the current successor" + ); + let versions = client.list_object_versions().bucket(versioned_bucket).send().await?; + assert_eq!(versions.versions().len(), 1); + assert_eq!(versions.versions()[0].version_id(), successor.version.as_deref()); + assert!(versions.delete_markers().is_empty()); + let listed = client.list_objects_v2().bucket(plain_bucket).send().await?; + assert_eq!(listed.contents().len(), 1); + assert_eq!(listed.contents()[0].key(), Some(child_key.as_str())); + } + + client.delete_object().bucket(plain_bucket).key(&child_key).send().await?; + client.delete_bucket().bucket(plain_bucket).send().await?; Ok(()) } @@ -2028,14 +2648,14 @@ async fn rollback_to_previous_release_reads_current_bucket_metadata() -> TestRes } // --------------------------------------------------------------------------- -// rc.5 multipart layouts under the current build (backlog#2147 follow-up to +// Previous-release multipart layouts under the current build (backlog#2147 follow-up to // rustfs#7305) // --------------------------------------------------------------------------- // // rustfs#7305 changed `ObjectInfo::is_multipart` to consult the stored part // list before the ETag shape. Every earlier check of that change used // synthetic metadata; this scenario writes the layouts with the published -// rc.5 binary and then reads, describes, and replicates them with the +// previous-release binary and then reads, describes, and replicates them with the // current build on the same data directory. const LAYOUT_PLAIN_BUCKET: &str = "upgrade-layout-plain"; @@ -2056,10 +2676,10 @@ struct LayoutCase { /// `false` for layouts whose replication is a known pre-existing failure; /// their outcome is logged, not asserted. assert_replication: bool, - /// Recorded from the rc.5 writer. - rc5_etag: String, - /// Whether rc.5 reported `ObjectParts` for the object. - rc5_reported_parts: Option, + /// Recorded from the previous-release writer. + source_etag: String, + /// Whether the previous release reported `ObjectParts` for the object. + source_reported_parts: Option, } impl LayoutCase { @@ -2164,7 +2784,7 @@ async fn layout_attributes( Ok(request.send().await?) } -/// Write `case` with the rc.5 client; single PUT when `part_sizes` is empty. +/// Write `case` with the source client; single PUT when `part_sizes` is empty. async fn layout_write(client: &Client, case: &LayoutCase) -> Result<(), BoxError> { let content_type = "text/plain"; if case.part_sizes.is_empty() { @@ -2251,8 +2871,8 @@ fn layout_cases() -> Vec { body, ssec, assert_replication: true, - rc5_etag: String::new(), - rc5_reported_parts: None, + source_etag: String::new(), + source_reported_parts: None, }; vec![ case(LAYOUT_PLAIN_BUCKET, "plain/single.bin", vec![], layout_noise(1024 * 1024 + 17, 1), false), @@ -2295,7 +2915,7 @@ fn layout_cases() -> Vec { // compressible first part is stored well below 5 MiB, so the sender // declares each part's plaintext length and the target validates the // 5 MiB minimum against it (rustfs/backlog#2363). rc.5 as the sender - // still fails this layout (see `rc5_baseline_replicates_multipart_layouts`). + // still fails this layout (see `previous_release_baseline_replicates_multipart_layouts`). case( LAYOUT_PLAIN_BUCKET, "plain/ssec-compressed-multipart-2.txt", @@ -2345,13 +2965,13 @@ fn layout_reported_parts(attributes: &aws_sdk_s3::operation::get_object_attribut attributes.object_parts().map(|parts| parts.parts().len()) } -async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult { +async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &str) -> TestResult { let label = case.label(); let head = layout_head(client, case).await?; assert_eq!( head.e_tag().map(|etag| etag.trim_matches('"')), - Some(case.rc5_etag.as_str()), - "{context}: {label}: the ETag written by rc.5 must be reported unchanged" + Some(case.source_etag.as_str()), + "{context}: {label}: the ETag written by the previous release must be reported unchanged" ); assert_eq!( head.content_length(), @@ -2361,8 +2981,16 @@ async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &st let full = layout_get(client, case, None, None).await?.body.collect().await?.into_bytes(); assert_eq!(full.len(), case.body.len(), "{context}: {label}: full GET length"); - assert!(full == case.body, "{context}: {label}: full GET body must equal the rc.5 upload"); + assert!( + full == case.body, + "{context}: {label}: full GET body must equal the previous-release upload" + ); + Ok(()) +} +async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult { + assert_layout_full_body(client, case, context).await?; + let label = case.label(); if case.is_multipart_layout() { let first = case.part_sizes[0]; let range = format!("bytes={}-{}", first - 32, first + 31); @@ -2398,7 +3026,7 @@ async fn assert_layout_attributes(client: &Client, case: &LayoutCase, context: & let attributes = layout_attributes(client, case).await?; assert_eq!( attributes.e_tag().map(|etag| etag.trim_matches('"')), - Some(case.rc5_etag.as_str()), + Some(case.source_etag.as_str()), "{context}: {label}: attributes ETag" ); assert_eq!( @@ -2597,13 +3225,13 @@ async fn replicate_layouts( Ok((target, transports)) } -/// rc.5 writes single-PUT, multipart, compressed, SSE-C and SSE-S3 layouts; +/// The previous release writes single-PUT, multipart, compressed, SSE-C and SSE-S3 layouts; /// the current build must read every byte, expose the stored part layout /// through GetObjectAttributes and partNumber reads, and replicate the objects /// with the transport that matches their stored parts. #[tokio::test] -#[ignore = "requires the pinned 1.0.0-rc.5 release binary"] -async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult { +#[ignore = "requires a pinned previous RustFS release binary"] +async fn direct_upgrade_from_previous_release_preserves_multipart_layouts() -> TestResult { init_logging(); let previous_binary = source_binary()?; let server_env = layout_server_env(); @@ -2617,18 +3245,18 @@ async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult { enable_versioning(&old_client, LAYOUT_PLAIN_BUCKET).await?; enable_versioning(&old_client, LAYOUT_ENCRYPTED_BUCKET).await?; put_default_sse_s3_encryption(&old_client, LAYOUT_ENCRYPTED_BUCKET).await?; - assert_default_sse_s3_encryption(&old_client, LAYOUT_ENCRYPTED_BUCKET, "rc.5").await?; + assert_default_sse_s3_encryption(&old_client, LAYOUT_ENCRYPTED_BUCKET, "previous release").await?; let mut cases = layout_cases(); for case in cases.iter_mut() { layout_write(&old_client, case).await?; let head = layout_head(&old_client, case).await?; - case.rc5_etag = head + case.source_etag = head .e_tag() - .ok_or_else(|| format!("{}: rc.5 HEAD omitted the ETag", case.label()))? + .ok_or_else(|| format!("{}: previous-release HEAD omitted the ETag", case.label()))? .trim_matches('"') .to_string(); - case.rc5_reported_parts = layout_attributes(&old_client, case) + case.source_reported_parts = layout_attributes(&old_client, case) .await .ok() .and_then(|a| layout_reported_parts(&a)); @@ -2636,15 +3264,16 @@ async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult { target: "e2e_test::upgrade_compatibility_test", object = %case.label(), parts = case.part_sizes.len(), - etag = %case.rc5_etag, - rc5_reported_parts = ?case.rc5_reported_parts, - "rc.5 wrote a legacy layout" + etag = %case.source_etag, + source_reported_parts = ?case.source_reported_parts, + "previous release wrote a layout" ); } - // The rc.5 writer must itself still read what it wrote, so a later - // failure is attributable to the upgrade rather than to the fixture. + // Prove the previous-release writer persisted every original byte before upgrading. + // Its compressed cross-part range reader has a legacy limitation; the + // upgraded reader must still satisfy every range and partNumber assertion. for case in &cases { - assert_layout_readable(&old_client, case, "rc.5").await?; + assert_layout_full_body(&old_client, case, "previous release").await?; } // Upgrade in place. @@ -2699,19 +3328,19 @@ async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult { .await? .into_bytes(); assert_eq!(replica.len(), case.body.len(), "{label}: replica length"); - assert!(replica == case.body, "{label}: replica body must equal the rc.5 upload"); + assert!(replica == case.body, "{label}: replica body must equal the previous-release upload"); } } Ok(()) } -/// The same layouts replicated by rc.5 itself, without an upgrade. This is the +/// The same layouts replicated by the previous release itself, without an upgrade. This is the /// baseline that tells a pre-existing transport failure apart from one the /// current build introduced; it records the outcome per layout and only fails /// when the fixture cannot run. #[tokio::test] -#[ignore = "requires the pinned 1.0.0-rc.5 release binary"] -async fn rc5_baseline_replicates_multipart_layouts() -> TestResult { +#[ignore = "requires a pinned previous RustFS release binary"] +async fn previous_release_baseline_replicates_multipart_layouts() -> TestResult { init_logging(); let previous_binary = source_binary()?; let server_env = layout_server_env(); @@ -2730,9 +3359,9 @@ async fn rc5_baseline_replicates_multipart_layouts() -> TestResult { for case in cases.iter_mut() { layout_write(&client, case).await?; let head = layout_head(&client, case).await?; - case.rc5_etag = head + case.source_etag = head .e_tag() - .ok_or_else(|| format!("{}: rc.5 HEAD omitted the ETag", case.label()))? + .ok_or_else(|| format!("{}: previous-release HEAD omitted the ETag", case.label()))? .trim_matches('"') .to_string(); } @@ -2750,7 +3379,7 @@ async fn rc5_baseline_replicates_multipart_layouts() -> TestResult { ) }) .collect(); - tracing::info!(target: "e2e_test::upgrade_compatibility_test", ?summary, "rc.5 baseline replication outcomes"); + tracing::info!(target: "e2e_test::upgrade_compatibility_test", ?summary, "previous-release baseline replication outcomes"); Ok(()) } @@ -2777,7 +3406,7 @@ async fn existing_object_replication_drives_each_layout_once() -> TestResult { for case in cases.iter_mut() { layout_write(&writer, case).await?; let head = layout_head(&writer, case).await?; - case.rc5_etag = head + case.source_etag = head .e_tag() .ok_or_else(|| format!("{}: HEAD omitted the ETag", case.label()))? .trim_matches('"') diff --git a/crates/ecstore/Cargo.toml b/crates/ecstore/Cargo.toml index 5a41e01f4..d659ee917 100644 --- a/crates/ecstore/Cargo.toml +++ b/crates/ecstore/Cargo.toml @@ -31,6 +31,7 @@ workspace = true [features] default = [] +diagnostic-read-version-caller = ["rustfs-io-metrics/diagnostic-read-version-caller"] gcs = ["dep:google-cloud-storage", "dep:google-cloud-auth"] # Compiles the controlled list-objects namespace-journal chaos injector into a # production binary (it is always available to tests). Off by default so the @@ -183,6 +184,7 @@ sha2 = { workspace = true } hex-simd = { workspace = true } tempfile.workspace = true hyper = { workspace = true, features = ["http2", "http1", "server"] } +h2 = { workspace = true } hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] } hyper-rustls = { workspace = true, default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs"] } hostname.workspace = true @@ -200,8 +202,8 @@ md-5.workspace = true memmap2 = { workspace = true } libc.workspace = true # "process" adds getrlimit for the io_uring fd-cache RLIMIT_NOFILE gate -# (backlog#1178); "fs" comes from the workspace default. -rustix = { workspace = true, features = ["process", "fs"] } +# (backlog#1178); "thread" provides cleanup affinity; "fs" is a workspace default. +rustix = { workspace = true, features = ["process", "fs", "thread"] } rustfs-madmin.workspace = true reqwest = { workspace = true } aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] } @@ -248,6 +250,7 @@ windows-sys = { workspace = true, features = ["Win32_System_Ioctl"] } [dev-dependencies] aws-smithy-async.workspace = true +rustfs-s3-ops.workspace = true tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] } criterion = { workspace = true, features = ["html_reports"] } temp-env = { workspace = true, features = ["async_closure"] } diff --git a/crates/ecstore/src/api/mod.rs b/crates/ecstore/src/api/mod.rs index d8d3a85b7..6c07fae84 100644 --- a/crates/ecstore/src/api/mod.rs +++ b/crates/ecstore/src/api/mod.rs @@ -14,6 +14,15 @@ //! Explicit ECStore public facades for outer crate compatibility boundaries. +#[cfg(feature = "diagnostic-read-version-caller")] +pub mod diagnostic { + pub use crate::cluster::rpc::runtime_sources::{ + current_read_version_entry_source, observe_read_version_context_boundary, with_read_version_entry, + with_read_version_spawn_context, + }; + pub use rustfs_io_metrics::internode_metrics::{ReadVersionContextBoundary, ReadVersionEntrySource}; +} + pub mod admin { pub use crate::diagnostics::admin_server_info::{get_local_server_property, get_server_info}; } @@ -199,10 +208,10 @@ pub mod bucket { get_object_lock_config, get_object_lock_config_state, get_on_demand_migration_config, get_on_demand_migration_config_in, get_public_access_block_config, get_quota_config, get_replication_config, get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config, - init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, set_bucket_metadata, - update, update_bucket_targets_under_transaction_lock, update_config_with, update_if_incarnation, - update_if_incarnation_at, update_quota_if_incarnation, update_quota_if_incarnation_at, update_under_transaction_lock, - update_under_transaction_lock_at, + init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, + remove_bucket_metadata_if_absent, set_bucket_metadata, update, update_bucket_targets_under_transaction_lock, + update_config_with, update_if_incarnation, update_if_incarnation_at, update_quota_if_incarnation, + update_quota_if_incarnation_at, update_under_transaction_lock, update_under_transaction_lock_at, }; #[cfg(feature = "test-util")] pub use crate::bucket::metadata_sys::{ConfigWriteLockProbe, test_support}; @@ -210,7 +219,8 @@ pub mod bucket { pub mod migration { pub use crate::bucket::migration::{ - LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_iam_config, + LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_bucket_metadata_for_buckets, + try_migrate_iam_config, }; } @@ -459,12 +469,16 @@ pub mod disk { pub use crate::disk::os::{LocalPublicationPause, LocalPublicationStage}; pub use crate::disk::{ BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp, - CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, - DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, LegacyFileStream, LegacyFileWrite, + CheckPartsResp, ConditionalFileUpdate, CurrentDeleteGeneration, CurrentDeleteTransition, + DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, + DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DeleteRecoveryState, + Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore, FileInfoVersions, FileReader, FileWriter, + GuardedDeleteVersionsPayload, GuardedFileInfoVersions, HEALING_MARKER_PATH, LegacyFileStream, LegacyFileWrite, LegacyMetadataWrite, LegacyPartMutation, LegacyPartOperation, LegacyPathDelete, LegacyPathDeleteOperation, - LegacyRenameData, LegacyRenameFile, NsScannerOpenRequest, OldCurrentSize, PartTransactionAction, RUSTFS_META_BUCKET, - ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE, SnapshotLeaseToken, - UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk, validate_batch_read_version_item_count, + LegacyRenameData, LegacyRenameFile, NsScannerOpenRequest, OldCurrentSize, PartTransactionAction, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, + STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, is_quota_mutation_fence_path, + new_disk, validate_batch_read_version_item_count, }; pub use bytes::Bytes; pub use endpoint::Endpoint; @@ -533,17 +547,19 @@ pub mod metrics { } pub mod notification { - #[cfg(any(test, feature = "test-util"))] - pub use crate::services::notification_sys::rotate_cross_pool_fence_fleet_proof_for_test; pub use crate::services::notification_sys::{ ClusterTierDailyStats, CrossPoolFenceFleetProofToken, IlmRecoveryExportFleetProofToken, LegacyTransitionStateReconcileFleetProofToken, NotificationPeerErr, NotificationSys, ScannerPublicationLeaseGrant, acquire_cross_pool_fence_fleet_proof, acquire_ilm_recovery_export_fleet_proof, - acquire_legacy_transition_state_reconcile_fleet_proof, cross_pool_fence_fleet_proof_matches, get_global_notification_sys, - ilm_recovery_export_fleet_proof_matches, ilm_recovery_export_local_process_epoch, - ilm_recovery_export_member_epochs_sha256, ilm_recovery_export_topology_generation, - legacy_transition_state_reconcile_fleet_proof_matches, new_global_notification_sys, - scanner_peer_transport_error_message_is_retryable, start_remote_version_state_fleet_probe, + acquire_legacy_transition_state_reconcile_fleet_proof, configured_fleet_topology_fingerprint, + cross_pool_fence_fleet_proof_matches, get_global_notification_sys, ilm_recovery_export_fleet_proof_matches, + ilm_recovery_export_local_process_epoch, ilm_recovery_export_member_epochs_sha256, + ilm_recovery_export_topology_generation, legacy_transition_state_reconcile_fleet_proof_matches, + new_global_notification_sys, scanner_peer_transport_error_message_is_retryable, start_remote_version_state_fleet_probe, + }; + #[cfg(any(test, feature = "test-util"))] + pub use crate::services::notification_sys::{ + rotate_cross_pool_fence_fleet_proof_for_test, without_cross_pool_fence_fleet_proof_for_test, }; } @@ -557,13 +573,14 @@ pub mod integrity { pub mod object { pub use crate::object_api::{ BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook, - GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver, - ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission, - RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest, + GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReadCondition, GetObjectReadMetadata, GetObjectReader, + NamespaceLockFence, ObjectEncryptionResolver, ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, + PutObjReader, QuotaAdmission, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest, SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, ScannerPublicationCommitScope, ScannerPublicationCommitStartError, ScannerPublicationCommitState, ShardIntegrityWriteMode, StreamConsumer, WriteCommitGuard, WriteCompletion, get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook, register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook, + with_get_object_read_condition, }; pub use crate::store::{ PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError, @@ -628,6 +645,7 @@ pub mod set_disk { #[cfg(feature = "test-util")] pub mod test_util { pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test; + pub use crate::set_disk::ConditionalReadBenchmarkMetadataGuard; pub use crate::set_disk::{MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause}; /// Keep a namespace commit pending until the returned owner is dropped. diff --git a/crates/ecstore/src/bucket/bucket_target_sys.rs b/crates/ecstore/src/bucket/bucket_target_sys.rs index a8edc849d..a5561717e 100644 --- a/crates/ecstore/src/bucket/bucket_target_sys.rs +++ b/crates/ecstore/src/bucket/bucket_target_sys.rs @@ -142,11 +142,11 @@ fn same_replication_service(edited: &BucketTarget, previous: &BucketTarget) -> b && access_key(edited) == access_key(previous) } -/// Page size and page budget for [`TargetClient::locate_replica_by_etag`]. +/// Page size and page budget for [`TargetClient::replica_candidates`]. const FIND_VERSION_BY_ETAG_PAGE_SIZE: i32 = 1000; const FIND_VERSION_BY_ETAG_MAX_PAGES: usize = 8; -/// Candidate cap for [`TargetClient::replica_candidates_by_etag`]: more than -/// this many same-content versions of one key is ambiguity by any measure. +/// Candidate cap for [`TargetClient::replica_candidates`]: more than this +/// many matching versions of one key is ambiguity by any measure. const FIND_VERSION_BY_ETAG_MAX_MATCHES: usize = 16; pub type GetObjectSdkError = Box>; pub type GetObjectTaggingSdkError = Box>; @@ -2030,27 +2030,28 @@ impl TargetClient { /// Candidate replicas by content identity on a target that mints its own /// version ids: page `ListObjectVersions` under the exact key and report - /// the live versions whose ETag matches `source_etag`, newest first. - /// Delete markers and prefix siblings never match. Bounded to + /// the live versions that match `identity`, newest first. Delete markers + /// and prefix siblings never match. Bounded to /// [`FIND_VERSION_BY_ETAG_MAX_PAGES`] pages and - /// [`FIND_VERSION_BY_ETAG_MAX_MATCHES`] candidates so a key with a very - /// deep history cannot turn one convergence check into an unbounded scan; - /// a replica beyond that window reads as missing, which only costs a - /// re-PUT (today's behaviour), never a lost object. + /// [`FIND_VERSION_BY_ETAG_MAX_MATCHES`] candidates per kind so a key with + /// a very deep history cannot turn one convergence check into an + /// unbounded scan; a replica beyond that window reads as missing, which + /// only costs a re-PUT (today's behaviour), never a lost object. /// /// Content identity is not version identity: two source generations with - /// the same bytes have the same ETag. Callers drop the candidates other - /// source versions already claim through their ledgers and refuse an - /// [`ReplicaLocation::Ambiguous`] remainder before mutating or deleting. - pub async fn replica_candidates_by_etag( + /// the same bytes have the same ETag, and two with the same size have the + /// same shape. Callers drop the candidates other source versions already + /// claim through their ledgers and decide through + /// [`ReplicaCandidates::locate`]. + pub async fn replica_candidates( &self, bucket: &str, object: &str, - source_etag: &str, - ) -> Result, Box>> { + identity: &ReplicaIdentity<'_>, + ) -> Result>> { let mut key_marker: Option = None; let mut version_id_marker: Option = None; - let mut matches: Vec = Vec::new(); + let mut candidates = ReplicaCandidates::default(); for _ in 0..FIND_VERSION_BY_ETAG_MAX_PAGES { let page = self .client @@ -2063,19 +2064,27 @@ impl TargetClient { .send() .await .map_err(Box::new)?; - matches.extend( - page.versions() - .iter() - .filter(|version| { - version.key() == Some(object) - && version.version_id().is_some_and(|id| !id.is_empty()) - && replication_etags_match(Some(source_etag), version.e_tag()) - }) - .filter_map(|version| version.version_id().map(str::to_string)), - ); + for version in page.versions() { + let Some(version_id) = version.version_id().filter(|id| !id.is_empty()) else { + continue; + }; + if version.key() != Some(object) { + continue; + } + match identity.evidence(version.e_tag(), version.size(), version.last_modified()) { + Some(ReplicaEvidence::Etag) => candidates.exact.push(version_id.to_string()), + // An unversioned target overwrites the key in place: its + // `null` version holds whichever generation was written + // last, so a shape match there identifies nothing. + Some(ReplicaEvidence::Shape) if version_id != "null" => candidates.shaped.push(version_id.to_string()), + Some(ReplicaEvidence::Shape) => {} + None => {} + } + } // A listing that moved past the exact key (every listed key is >= - // the prefix), ended, or already filled the candidate cap decides. - if matches.len() >= FIND_VERSION_BY_ETAG_MAX_MATCHES + // the prefix), ended, or already filled a candidate cap decides. + if candidates.exact.len() >= FIND_VERSION_BY_ETAG_MAX_MATCHES + || candidates.shaped.len() >= FIND_VERSION_BY_ETAG_MAX_MATCHES || page .versions() .iter() @@ -2090,8 +2099,9 @@ impl TargetClient { break; } } - matches.truncate(FIND_VERSION_BY_ETAG_MAX_MATCHES); - Ok(matches) + candidates.exact.truncate(FIND_VERSION_BY_ETAG_MAX_MATCHES); + candidates.shaped.truncate(FIND_VERSION_BY_ETAG_MAX_MATCHES); + Ok(candidates) } /// PutObjectRetention against a replica version on a target that does not @@ -2614,19 +2624,18 @@ impl TargetClient { } /// Where a replica stands on a target that mints its own version ids, by -/// content identity (exact key + ETag) after the candidates other source -/// versions claim were removed. See -/// [`TargetClient::replica_candidates_by_etag`]. +/// content identity after the candidates other source versions claim were +/// removed. See [`TargetClient::replica_candidates`]. #[derive(Debug, Clone, PartialEq, Eq)] pub enum ReplicaLocation { - /// No live version under the key carries the source ETag. + /// No live version under the key matches the source. Missing, - /// Exactly one live version carries it: safe to address. + /// Exactly one live version matches: safe to address. Unique(String), - /// More than one live version carries it (same bytes replicated for - /// several source generations). `newest` is the most recently listed - /// one — good enough to prove the replica exists, never good enough to - /// pick which one to mutate or delete. + /// More than one live version carries the source ETag (same bytes + /// replicated for several source generations). `newest` is the most + /// recently listed one — good enough to prove the replica exists, never + /// good enough to pick which one to mutate or delete. Ambiguous { newest: String }, } @@ -2652,6 +2661,107 @@ impl ReplicaLocation { } } +/// What a replica of an encrypted source version still has in common with +/// it on a target that computes its own ETags. The source ETag is a keyed +/// fingerprint (docs/operations/encrypted-etags.md) or a legacy plaintext +/// MD5 the target's re-encrypted replica never reports, so the replica is +/// recognised by the byte count the replication PUT sent and by a write time +/// no earlier than the source version's own. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct EncryptedReplicaShape { + /// Bytes on the replication wire: the plaintext size for managed SSE, + /// the stored ciphertext size for an SSE-C passthrough. + pub size: i64, + /// The source version's mtime in whole seconds; listings and HEAD report + /// second-precision times. + pub not_before_secs: i64, +} + +impl EncryptedReplicaShape { + pub fn matches(&self, size: Option, last_modified: Option<&aws_sdk_s3::primitives::DateTime>) -> bool { + size == Some(self.size) && last_modified.is_some_and(|written| written.secs() >= self.not_before_secs) + } +} + +/// How a source version recognises its replica among a key's target +/// versions. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ReplicaIdentity<'a> { + /// Unencrypted source: its ETag is the MD5 of the replicated bytes, which + /// every target reports back — exact identity. + Etag(&'a str), + /// Encrypted source: a target that adopts the source ETag (RustFS, + /// MinIO) still reports `etag`; any other target only keeps the shape. + Encrypted { + etag: Option<&'a str>, + shape: EncryptedReplicaShape, + }, +} + +/// Which kind of evidence tied a target version to the source. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ReplicaEvidence { + /// The target reports the source ETag. + Etag, + /// Encrypted source on a target that computes its own ETag: same size, + /// written no earlier than the source version. + Shape, +} + +impl ReplicaIdentity<'_> { + pub fn evidence( + &self, + etag: Option<&str>, + size: Option, + last_modified: Option<&aws_sdk_s3::primitives::DateTime>, + ) -> Option { + match *self { + Self::Etag(source_etag) => replication_etags_match(Some(source_etag), etag).then_some(ReplicaEvidence::Etag), + Self::Encrypted { + etag: source_etag, + shape, + } => { + if replication_etags_match(source_etag, etag) { + Some(ReplicaEvidence::Etag) + } else { + shape.matches(size, last_modified).then_some(ReplicaEvidence::Shape) + } + } + } + } +} + +/// Content-identity candidates of one key, newest first, by evidence kind. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct ReplicaCandidates { + /// Versions that report the source ETag. + pub exact: Vec, + /// Encrypted sources only: versions without the source ETag whose shape + /// matches. + pub shaped: Vec, +} + +impl ReplicaCandidates { + pub fn is_empty(&self) -> bool { + self.exact.is_empty() && self.shaped.is_empty() + } + + /// Drop the candidates other source versions claim and decide. An ETag + /// match outranks any shape match. A shape is weaker evidence than an + /// ETag — two generations of one size share it — so several shape + /// matches are never resolved by picking one: they read as + /// [`ReplicaLocation::Missing`] for convergence (the caller re-PUTs, a + /// duplicate at worst) and as ambiguous for a purge (it refuses). + pub fn locate(mut self, claimed: &HashSet) -> (ReplicaLocation, ReplicaEvidence) { + self.exact.retain(|candidate| !claimed.contains(candidate)); + if !self.exact.is_empty() { + return (ReplicaLocation::from_matches(self.exact), ReplicaEvidence::Etag); + } + self.shaped.retain(|candidate| !claimed.contains(candidate)); + (ReplicaLocation::from_matches(self.shaped), ReplicaEvidence::Shape) + } +} + #[derive(Debug)] pub enum BucketTargetError { BucketRemoteTargetNotFound { diff --git a/crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs b/crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs index 345dc5601..c7f8c5751 100644 --- a/crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs +++ b/crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs @@ -2418,6 +2418,16 @@ fn is_slow_down(err: &Error) -> bool { matches!(err, Error::SlowDown) } +/// Lifecycle expiry is idempotent when the exact object or version has already +/// disappeared between the scanner listing and the delete admission. +/// +/// Keep this predicate narrower than bucket/volume "not found" helpers: an +/// unavailable bucket or disk is an operational failure and must not be +/// reported as a completed expiry. +fn is_lifecycle_expiry_already_gone(err: &Error) -> bool { + is_err_object_not_found(err) || is_err_version_not_found(err) +} + /// Resolves the expiry worker count from the single documented knob, /// `RUSTFS_MAX_EXPIRY_WORKERS`: a set, parsable, non-zero value wins; /// anything else falls back to `min(cpus, 16)`. The historical @@ -2452,7 +2462,13 @@ fn spawn_recovery_disposition_maintenance_once(api: Arc) -> Option) { return; } tokio::spawn(async move { - run_tier_delete_journal_recovery_loop(api, cancel_token).await; + let recovery = run_tier_delete_journal_recovery_loop(api, cancel_token); + #[cfg(feature = "diagnostic-read-version-caller")] + let recovery = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::LifecycleRecovery, + recovery, + ); + recovery.await; }); } @@ -3354,7 +3376,13 @@ fn spawn_transition_transaction_recovery_once(api: Arc) { return; } tokio::spawn(async move { - run_transition_transaction_recovery_loop(api, cancel_token).await; + let recovery = run_transition_transaction_recovery_loop(api, cancel_token); + #[cfg(feature = "diagnostic-read-version-caller")] + let recovery = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::LifecycleRecovery, + recovery, + ); + recovery.await; }); } @@ -3939,44 +3967,28 @@ pub(crate) async fn enqueue_immediate_expiry(api: Arc, oi: &ObjectInfo, return; }; - let mut marker = None; - let mut version_marker = None; - let mut object_infos = Vec::new(); - - loop { - let page = match api - .clone() - .list_object_versions_for_lifecycle(&oi.bucket, &oi.name, marker.clone(), version_marker.clone(), None, 1000) - .await - { - Ok(page) => page, - Err(err) => { - warn!( - event = EVENT_LIFECYCLE_EVALUATION_FAILED, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_LIFECYCLE, - bucket = %oi.bucket, - object = %oi.name, - error = %err, - reason = "list_versions_failed", - "Failed to load lifecycle version group" - ); - return; - } - }; - - object_infos.extend(page.objects.into_iter().filter(|object| object.name == oi.name)); - - if !page.is_truncated { - break; + let object_infos = match api + .get_object_versions_for_lifecycle(&oi.bucket, &oi.name, opts, configs.bucket_incarnation_id) + .await + { + Ok(objects) => objects, + Err(err) => { + warn!( + event = EVENT_LIFECYCLE_EVALUATION_FAILED, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_LIFECYCLE, + result = "failed", + bucket = %oi.bucket, + object = %oi.name, + error = %err, + reason = "read_versions_failed", + "Failed to load lifecycle version group" + ); + return; } - - marker = page.next_marker; - version_marker = page.next_version_idmarker; - } - + }; if object_infos.is_empty() { - object_infos.push(oi.clone()); + return; } let object_opts = object_infos @@ -4901,6 +4913,14 @@ async fn expire_transitioned_object_with_lock_lost_signal( let (versioned, version_suspended) = snapshot.versioning_config().delete_state(&oi.name); let mut opts = transitioned_object_delete_opts(oi, lc_event.action, versioned, version_suspended, bucket_incarnation_id) .map_err(std::io::Error::other)?; + if lc_event.requires_expiry_policy() { + opts.lifecycle_expiry_policy = Some( + lc_event + .expiry_policy + .clone() + .ok_or_else(|| std::io::Error::other("lifecycle expiration is missing its policy snapshot"))?, + ); + } opts.add_namespace_lock_guard(&publication_guard); if let Some(signal) = lock_lost_signal { opts.add_namespace_lock_lost_signal(signal); @@ -4921,6 +4941,20 @@ async fn expire_transitioned_object_with_lock_lost_signal( //audit_log_lifecycle(*oi, ILMExpiry, tags, traceFn); Ok(dobj) } + Err(err) if is_lifecycle_expiry_already_gone(&err) => { + debug!( + event = EVENT_LIFECYCLE_DELETE_COMPLETED, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_LIFECYCLE, + bucket = %oi.bucket, + object = %oi.name, + operation = "delete_restored_object", + reason = "already_gone", + "Lifecycle delete target was already absent" + ); + crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await; + Ok(oi.clone()) + } Err(err) => Err(std::io::Error::other(err)), }; } @@ -4931,6 +4965,8 @@ async fn expire_transitioned_object_with_lock_lost_signal( // pointing to a remote tier version that has already been removed. If this // only creates a delete marker, remote cleanup must be driven by persisted // free-version recovery rather than the visible delete result. + // Keep remote transitioned not-found errors visible: local metadata + // absence alone does not prove that the tier object was cleaned up. let dobj = match api.delete_object(&oi.bucket, &oi.name, opts).await { Ok(obj) => obj, Err(e) => { @@ -5735,6 +5771,9 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal( bucket_incarnation_id: Uuid, lock_lost_signal: Option>, ) -> bool { + let Some(expiry_policy) = lc_event.expiry_policy.clone() else { + return false; + }; let Some(publication_guard) = lifecycle_expiry_publication_guard(&api, &oi.bucket, &oi.name, bucket_incarnation_id).await else { return false; @@ -5765,6 +5804,7 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal( versioned, version_suspended, expiration: ExpirationOptions { expire: true }, + lifecycle_expiry_policy: Some(expiry_policy), delete_replication_config_snapshot: Some(Arc::new(snapshot)), expected_bucket_incarnation_id: Some(bucket_incarnation_id), ..Default::default() @@ -5803,6 +5843,20 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal( .await { Ok(dobj) => dobj, + Err(err) if is_lifecycle_expiry_already_gone(&err) => { + debug!( + event = EVENT_LIFECYCLE_DELETE_COMPLETED, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_LIFECYCLE, + bucket = %oi.bucket, + object = %oi.name, + operation = "delete_object", + reason = "already_gone", + "Lifecycle delete target was already absent" + ); + crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await; + return true; + } Err(e) => { error!( event = EVENT_LIFECYCLE_DELETE_FAILED, @@ -6032,6 +6086,7 @@ pub async fn apply_lifecycle_action(event: &lifecycle::Event, src: &LcEventSrc, mod tests { use super::expiry_worker_count; use super::lifecycle_transition_publication_guard; + use super::metadata_boundary::get_expiry_configs; use super::{ DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS, DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX, DEFAULT_TRANSITION_WORKERS_CAP, EVENT_LIFECYCLE_EVALUATION_FAILED, EVENT_LIFECYCLE_EXPIRED_DETECTED, @@ -6042,18 +6097,19 @@ mod tests { cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at, enqueue_recovered_free_version_with_state, enqueue_transition_for_existing_objects_scoped, enqueue_transition_with_lifecycle, enqueue_transition_with_lifecycle_report, eval_action_from_lifecycle, - get_lock_acquire_timeout, jitter_tier_free_version_recovery_delay, lifecycle_action_blocked_by_replication, - lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object, lifecycle_replication_blocks_action, - lifecycle_rule_has_date_expiration, lifecycle_version_delete_target, manual_transition_duration_elapsed, - manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink, manual_transition_version_marker, - manual_transition_worker_failure_reason, mark_delete_opts_skip_decommissioned_on_remote_success, - merge_stale_multipart_candidate, persist_manual_transition_job_progress_if_owned, - persist_manual_transition_page_checkpoint, recover_manual_transition_job, recover_manual_transition_jobs, - resolve_tier_free_version_recovery_enabled, resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, - resolve_transition_worker_count, resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator, - run_tier_free_version_recovery_loop, select_restore_s3_location, set_lifecycle_observability_observer, - set_recovered_free_version_enqueue_observer, should_defer_date_expiry_for_recent_config_update, - transitioned_cleanup_tuple, transitioned_object_delete_opts, wait_for_tier_free_version_recovery, + get_lock_acquire_timeout, is_lifecycle_expiry_already_gone, jitter_tier_free_version_recovery_delay, + lifecycle_action_blocked_by_replication, lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object, + lifecycle_replication_blocks_action, lifecycle_rule_has_date_expiration, lifecycle_version_delete_target, + manual_transition_duration_elapsed, manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink, + manual_transition_version_marker, manual_transition_worker_failure_reason, + mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate, + persist_manual_transition_job_progress_if_owned, persist_manual_transition_page_checkpoint, + recover_manual_transition_job, recover_manual_transition_jobs, resolve_tier_free_version_recovery_enabled, + resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count, + resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator, run_tier_free_version_recovery_loop, + select_restore_s3_location, set_lifecycle_observability_observer, set_recovered_free_version_enqueue_observer, + should_defer_date_expiry_for_recent_config_update, transitioned_cleanup_tuple, transitioned_object_delete_opts, + wait_for_tier_free_version_recovery, }; #[cfg(feature = "test-util")] use super::{delete_free_version_remote_object_then, encode_dir_object, get_transitioned_object_reader_with_tier_manager}; @@ -6079,7 +6135,7 @@ mod tests { save_manual_transition_scope_admission_if_current, save_manual_transition_task_if_absent, save_manual_transition_worker_result_if_absent, }; - use crate::bucket::lifecycle::replication_sink::{ReplicationStatusType, VersionPurgeStatusType}; + use crate::bucket::lifecycle::replication_sink::{self, ReplicationStatusType, VersionPurgeStatusType}; use crate::bucket::lifecycle::runtime_boundary as runtime_sources; use crate::bucket::lifecycle::tier_free_version_recovery::{ FreeVersionRecoveryStats, RecoveryWalkTestAction, RecoveryWorkBudget, list_tier_free_versions, @@ -10610,6 +10666,26 @@ mod tests { ); } + #[test] + fn lifecycle_expiry_already_gone_only_accepts_object_or_version_absence() { + assert!(is_lifecycle_expiry_already_gone(&Error::FileNotFound)); + assert!(is_lifecycle_expiry_already_gone(&Error::ObjectNotFound( + "bucket".to_string(), + "object".to_string(), + ))); + assert!(is_lifecycle_expiry_already_gone(&Error::FileVersionNotFound)); + assert!(is_lifecycle_expiry_already_gone(&Error::VersionNotFound( + "bucket".to_string(), + "object".to_string(), + "version".to_string(), + ))); + + assert!(!is_lifecycle_expiry_already_gone(&Error::VolumeNotFound)); + assert!(!is_lifecycle_expiry_already_gone(&Error::DiskNotFound)); + assert!(!is_lifecycle_expiry_already_gone(&Error::ErasureReadQuorum)); + assert!(!is_lifecycle_expiry_already_gone(&Error::MethodNotAllowed)); + } + #[tokio::test] async fn manual_transition_counts_already_transitioned_object() { let lc = latest_transition_lifecycle(); @@ -12931,6 +13007,7 @@ mod tests { .await .expect("bucket incarnation should be available"); + let expiry_policy = install_test_expiry_policy(&ecstore, &bucket).await; test_recorder::install(); // One version that exists and one that never did: `delete_objects` @@ -12949,15 +13026,24 @@ mod tests { ..Default::default() }, ]; + let _ = replication_sink::take_scheduled_deletes_for_test(); let failed = crate::bucket::lifecycle::object_handlers_common::delete_object_versions( &ecstore, &bucket, &targets, - lifecycle::Event::default(), + lifecycle::Event { + action: IlmAction::DeleteVersionAction, + expiry_policy: Some(expiry_policy.into()), + ..Default::default() + }, incarnation, ) .await; assert_eq!(failed, 0, "a missing version is not a batch failure"); + assert!( + replication_sink::take_scheduled_deletes_for_test().is_empty(), + "a bucket without replication must not schedule deletes for removed or missing versions" + ); let events = test_recorder::recorded_for_bucket(&bucket); let announced = events.iter().map(|event| event.version_id).collect::>(); @@ -12984,6 +13070,97 @@ mod tests { assert_eq!(remaining[0].version_id, Some(current_version_id)); } + #[tokio::test] + #[serial] + async fn lifecycle_noncurrent_batch_expiry_schedules_only_matching_delete_rules() { + use crate::bucket::metadata::BUCKET_REPLICATION_CONFIG; + + let (_disk_paths, ecstore) = setup_test_env().await; + for (prefix, delete_status, should_replicate) in [ + ("other/", "Enabled", false), + ("logs/", "Disabled", false), + ("logs/", "Enabled", true), + ] { + let bucket = format!("lifecycle-repl-admission-{}", Uuid::new_v4().simple()); + let object = "logs/object"; + create_test_bucket(&ecstore, &bucket).await; + metadata_sys::update_in( + &ecstore.ctx, + &bucket, + BUCKET_VERSIONING_CONFIG, + b"Enabled".to_vec(), + ) + .await + .expect("versioning should be enabled"); + + let mut versions = Vec::new(); + for body in [b"noncurrent".as_slice(), b"current".as_slice()] { + let mut reader = PutObjReader::from_vec(body.to_vec()); + let version = ecstore + .put_object( + &bucket, + object, + &mut reader, + &ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect("source version should be created"); + versions.push(version.version_id.expect("versioned PUT must have an identity")); + } + let config = format!( + "delete-ruleEnabled\ + {prefix}arn:test\ + {delete_status}\ + " + ); + metadata_sys::update_in(&ecstore.ctx, &bucket, BUCKET_REPLICATION_CONFIG, config.into_bytes()) + .await + .expect("replication configuration should be persisted"); + let expiry_policy = install_test_expiry_policy(&ecstore, &bucket).await; + let incarnation = ecstore + .bucket_incarnation_id_from_disk(&bucket) + .await + .expect("bucket incarnation should be available"); + let _ = replication_sink::take_scheduled_deletes_for_test(); + let failed = crate::bucket::lifecycle::object_handlers_common::delete_object_versions( + &ecstore, + &bucket, + &[ObjectToDelete { + object_name: object.to_string(), + version_id: Some(versions[0]), + ..Default::default() + }], + lifecycle::Event { + action: IlmAction::DeleteVersionAction, + expiry_policy: Some(expiry_policy.into()), + ..Default::default() + }, + incarnation, + ) + .await; + assert_eq!(failed, 0, "batch expiry should commit"); + let scheduled = replication_sink::take_scheduled_deletes_for_test(); + assert_eq!(scheduled.len(), usize::from(should_replicate), "prefix={prefix}, delete={delete_status}"); + if should_replicate { + assert_eq!(scheduled[0].version_id, Some(versions[0])); + let state = scheduled[0] + .replication_state + .as_ref() + .expect("admitted purge state must survive"); + assert_eq!(state.version_purge_status_internal.as_deref(), Some("arn:test=PENDING;")); + assert!(state.purge_targets.contains_key("arn:test")); + } + let current = ecstore + .get_object_info(&bucket, object, &ObjectOptions::default()) + .await + .expect("batch expiry must retain the current version"); + assert_eq!(current.version_id, Some(versions[1])); + } + } + #[tokio::test] #[serial] async fn lifecycle_deletes_only_the_historical_null_version_after_versioning_is_reenabled() { @@ -13071,6 +13248,14 @@ mod tests { assert!(!object_infos[null_index].is_latest); let lifecycle = current_and_noncurrent_expiration_lifecycle(); + metadata_sys::update_in( + &ecstore.ctx, + &bucket, + BUCKET_LIFECYCLE_CONFIG, + crate::bucket::utils::serialize(&lifecycle).unwrap(), + ) + .await + .unwrap(); let object_opts = object_infos .iter() .map(lifecycle::object_opts_from_object_info) @@ -13241,6 +13426,81 @@ mod tests { assert_eq!(event.action, IlmAction::NoneAction); } + #[tokio::test] + #[serial] + async fn lifecycle_exact_immediate_expiry_preserves_guards_and_skips_stale_writes() { + let (paths, ecstore) = setup_test_env().await; + let bucket = format!("lifecycle-exact-expiry-{}", Uuid::new_v4().simple()); + let object = "object"; + create_test_bucket(&ecstore, &bucket).await; + let oi = ecstore + .put_object( + &bucket, + object, + &mut PutObjReader::from_vec(b"payload".to_vec()), + &ObjectOptions::default(), + ) + .await + .expect("create object before configuring expiry"); + + // Exercise persisted legacy immediate rules without relaxing the S3 + // validator, which correctly rejects newly submitted Expiration.Days=0. + let sys = metadata_sys::bucket_metadata_sys_of(&ecstore.ctx).expect("metadata system"); + let sys = sys.read().await.clone(); + let mut metadata = (*sys.get(&bucket).await.expect("bucket metadata")).clone(); + let mut lifecycle = latest_expiration_lifecycle(); + lifecycle.rules[0].expiration.as_mut().expect("expiration rule").days = Some(0); + metadata.lifecycle_config_xml = crate::bucket::utils::serialize(&lifecycle).expect("encode legacy policy"); + metadata.lifecycle_config = Some(lifecycle); + sys.persist_and_set(metadata).await.expect("persist legacy policy"); + + let state = install_unconsumed_runtime_expiry_worker(&ecstore, 4).await; + super::enqueue_immediate_expiry(ecstore.clone(), &oi, LcEventSrc::S3PutObject, &ObjectOptions::default()).await; + assert_eq!(state.read().await.stats.pending_tasks(), 1, "exact reads must preserve immediate expiry"); + + let state = install_unconsumed_runtime_expiry_worker(&ecstore, 4).await; + for path in &paths { + let metadata_path = path.join(&bucket).join(object).join(crate::disk::STORAGE_FORMAT_FILE); + let mut meta = rustfs_filemeta::FileMeta::load(&fs::read(&metadata_path).await.expect("read object metadata")) + .expect("decode object metadata"); + let mut version = meta + .get_all_file_info_versions(&bucket, object, true) + .expect("version group") + .versions + .remove(0); + version.metadata.insert("x-amz-object-lock-legal-hold".into(), "ON".into()); + meta.add_version(version).expect("persist legal hold"); + fs::write(metadata_path, meta.marshal_msg().expect("encode held metadata")) + .await + .expect("write legal hold"); + } + super::enqueue_immediate_expiry(ecstore.clone(), &oi, LcEventSrc::S3PutObject, &ObjectOptions::default()).await; + assert_eq!( + state.read().await.stats.pending_tasks(), + 0, + "use current lock metadata, not the stale write result" + ); + + for path in &paths { + fs::write(path.join(&bucket).join(object).join(crate::disk::STORAGE_FORMAT_FILE), b"corrupt") + .await + .expect("inject corrupt metadata"); + } + super::enqueue_immediate_expiry(ecstore.clone(), &oi, LcEventSrc::S3PutObject, &ObjectOptions::default()).await; + assert_eq!(state.read().await.stats.pending_tasks(), 0, "read failure must never enqueue expiry"); + for path in &paths { + fs::remove_dir_all(path.join(&bucket).join(object)) + .await + .expect("simulate concurrent object removal"); + } + super::enqueue_immediate_expiry(ecstore.clone(), &oi, LcEventSrc::S3PutObject, &ObjectOptions::default()).await; + assert_eq!( + state.read().await.stats.pending_tasks(), + 0, + "confirmed absence must not fall back to the old ObjectInfo" + ); + } + #[tokio::test] #[serial] async fn lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata() { @@ -13350,6 +13610,7 @@ mod tests { .await .expect("table data object should be created"); + let expiry_policy = install_test_expiry_policy(&ecstore, &bucket).await; let publication_lock = ecstore .new_ns_lock(&bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH) .await @@ -13364,6 +13625,7 @@ mod tests { let mut expiry = tokio::spawn(async move { let event = crate::bucket::lifecycle::lifecycle::Event { action: IlmAction::DeleteAction, + expiry_policy: Some(expiry_policy.into()), ..Default::default() }; let bucket_incarnation_id = expiry_store @@ -13478,13 +13740,16 @@ mod tests { queued.size = 1; let event = lifecycle::Event { action: IlmAction::DeleteAction, + expiry_policy: Some(install_test_expiry_policy(&ecstore, &bucket).await.into()), ..Default::default() }; let incarnation = ecstore .bucket_incarnation_id_from_disk(&bucket) .await .expect("bucket incarnation"); - for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, false)] { + // Replaying the same scanner snapshot after the delete is an + // already-gone success and must not be reported as a failure. + for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, true)] { assert_eq!( super::apply_expiry_on_non_transitioned_objects( ecstore.clone(), @@ -13587,6 +13852,7 @@ mod tests { .bucket_incarnation_id_from_disk(&bucket) .await .expect("bucket incarnation should be available"); + let expiry_policy = install_test_expiry_policy(&ecstore, &bucket).await; let publication_lock = ecstore .new_ns_lock(&bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH) .await @@ -13609,7 +13875,11 @@ mod tests { &delete_store, &delete_bucket, std::slice::from_ref(&target), - lifecycle::Event::default(), + lifecycle::Event { + action: IlmAction::DeleteVersionAction, + expiry_policy: Some(expiry_policy.into()), + ..Default::default() + }, incarnation, ) .await @@ -13765,6 +14035,7 @@ mod tests { let queued_event = crate::bucket::lifecycle::lifecycle::Event { action: IlmAction::DelMarkerDeleteAllVersionsAction, rule_id: "delete-marker-history".to_string(), + expiry_policy: get_expiry_configs(&ecstore, &bucket).await.unwrap().lifecycle.map(Into::into), ..Default::default() }; metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, lifecycle_xml(30).into_bytes()) @@ -13882,6 +14153,7 @@ mod tests { &crate::bucket::lifecycle::lifecycle::Event { action: IlmAction::DeleteAllVersionsAction, rule_id: "delete-all-versions".to_string(), + expiry_policy: get_expiry_configs(&ecstore, &bucket).await.unwrap().lifecycle.map(Into::into), ..Default::default() }, &LcEventSrc::Scanner, @@ -14068,6 +14340,221 @@ mod tests { (disk_paths, ecstore) } + #[tokio::test] + #[serial] + async fn queued_expiry_rejects_withdrawn_or_changed_policy() { + let (_disk_paths, ecstore) = setup_test_env().await; + for case in [ + "removed", + "disabled", + "longer-days", + "different-prefix", + "unreadable", + "missing-snapshot", + "unchanged", + ] { + let bucket = format!("expiry-policy-{case}"); + create_test_bucket(&ecstore, &bucket).await; + metadata_sys::update_in( + &ecstore.ctx, + &bucket, + BUCKET_VERSIONING_CONFIG, + b"Enabled".to_vec(), + ) + .await + .unwrap(); + for object in ["single", "batch"] { + for age in [40, 2] { + let mut reader = PutObjReader::from_vec(vec![1; 8]); + ecstore + .put_object( + &bucket, + object, + &mut reader, + &ObjectOptions { + versioned: true, + mod_time: Some(OffsetDateTime::now_utc() - time::Duration::days(age)), + ..Default::default() + }, + ) + .await + .unwrap(); + } + } + let policy = current_and_noncurrent_expiration_lifecycle(); + metadata_sys::update_in( + &ecstore.ctx, + &bucket, + BUCKET_LIFECYCLE_CONFIG, + crate::bucket::utils::serialize(&policy).unwrap(), + ) + .await + .unwrap(); + let incarnation = ecstore.bucket_incarnation_id_from_disk(&bucket).await.unwrap(); + let state = install_unconsumed_runtime_expiry_worker(&ecstore, 3).await; + for object in ["single", "batch"] { + let versions = ecstore + .clone() + .list_object_versions(&bucket, object, None, None, None, 10) + .await + .unwrap() + .objects; + let opts = versions + .iter() + .map(lifecycle::object_opts_from_object_info) + .collect::>(); + let events = Evaluator::new(Arc::new(policy.clone())).eval(&opts).await.unwrap(); + let index = versions.iter().position(|version| !version.is_latest).unwrap(); + let mut event = events[index].clone(); + assert_eq!(event.action, IlmAction::DeleteVersionAction); + if case == "missing-snapshot" { + event.expiry_policy = None; + } + let queued = if object == "single" { + state + .write() + .await + .enqueue_by_days(&versions[index], &event, &LcEventSrc::Scanner, incarnation) + } else { + state.write().await.enqueue_by_newer_noncurrent( + &bucket, + vec![lifecycle_version_delete_target(&versions[index]).unwrap()], + event, + &LcEventSrc::Scanner, + incarnation, + ) + }; + assert!(queued); + } + let mut changed = policy.clone(); + match case { + "removed" => { + metadata_sys::delete(&bucket, BUCKET_LIFECYCLE_CONFIG).await.unwrap(); + } + "disabled" => changed.rules[0].status = ExpirationStatus::from_static(ExpirationStatus::DISABLED), + "longer-days" => { + changed.rules[0] + .noncurrent_version_expiration + .as_mut() + .unwrap() + .noncurrent_days = Some(60) + } + "different-prefix" => changed.rules[0].prefix = Some("keep/".to_string()), + "unreadable" => { + let sys = metadata_sys::bucket_metadata_sys_of(&ecstore.ctx) + .unwrap() + .read() + .await + .clone(); + let mut metadata = (*sys.get(&bucket).await.unwrap()).clone(); + metadata.lifecycle_config_xml = b"".to_vec(); + metadata.lifecycle_config = None; + sys.persist_and_set(metadata).await.unwrap(); + } + _ => {} + } + if !matches!(case, "removed" | "unchanged" | "missing-snapshot" | "unreadable") { + metadata_sys::update_in( + &ecstore.ctx, + &bucket, + BUCKET_LIFECYCLE_CONFIG, + crate::bucket::utils::serialize(&changed).unwrap(), + ) + .await + .unwrap(); + } + let (rx, stats, recovery_notify) = { + let state = state.read().await; + state.tasks_tx[0].try_send(None).unwrap(); + (state.tasks_rx[0].clone(), state.stats.clone(), state.recovery_notify.clone()) + }; + tokio::time::timeout(StdDuration::from_secs(30), async { + let mut receiver = rx.lock().await; + ExpiryState::worker(&mut receiver, Arc::downgrade(&ecstore), stats, recovery_notify).await; + }) + .await + .expect("queued expiry worker must drain"); + let versions = ecstore + .clone() + .list_object_versions(&bucket, "", None, None, None, 10) + .await + .unwrap() + .objects; + assert_eq!(versions.len(), if case == "unchanged" { 2 } else { 4 }, "case: {case}"); + assert_eq!(versions.iter().filter(|version| version.is_latest).count(), 2); + } + } + + #[tokio::test] + #[serial] + async fn lifecycle_policy_update_waits_for_admitted_delete_commit() { + let (_disk_paths, ecstore) = setup_test_env().await; + let bucket = "expiry-policy-commit-fence"; + create_test_bucket(&ecstore, bucket).await; + let mut reader = PutObjReader::from_vec(b"expired".to_vec()); + let object = ecstore + .put_object( + bucket, + "object", + &mut reader, + &ObjectOptions { + mod_time: Some(OffsetDateTime::now_utc() - time::Duration::days(40)), + ..Default::default() + }, + ) + .await + .unwrap(); + let policy = install_test_expiry_policy(&ecstore, bucket).await; + let event = eval_action_from_lifecycle(&policy, None, &object).await; + assert_eq!(event.action, IlmAction::DeleteAction); + let incarnation = ecstore.bucket_incarnation_id_from_disk(bucket).await.unwrap(); + let barrier = crate::store::DeleteAfterObjectLockSnapshotBarrier::install(bucket); + let deleting_store = ecstore.clone(); + let deleting = tokio::spawn(async move { + super::apply_expiry_on_non_transitioned_objects(deleting_store, &object, &event, &LcEventSrc::Scanner, incarnation) + .await + }); + tokio::time::timeout(StdDuration::from_secs(30), barrier.wait_until_paused()) + .await + .unwrap(); + let probe = metadata_sys::ConfigWriteLockProbe::install(bucket); + let mut removing = tokio::spawn(async move { metadata_sys::delete(bucket, BUCKET_LIFECYCLE_CONFIG).await }); + probe.wait_until_attempted().await; + assert!( + tokio::time::timeout(StdDuration::from_millis(100), &mut removing) + .await + .is_err(), + "configuration removal must wait for the admitted delete" + ); + barrier.release(); + assert!( + tokio::time::timeout(StdDuration::from_secs(30), deleting) + .await + .unwrap() + .unwrap() + ); + tokio::time::timeout(StdDuration::from_secs(30), removing) + .await + .unwrap() + .unwrap() + .unwrap(); + assert!(get_expiry_configs(&ecstore, bucket).await.unwrap().lifecycle.is_none()); + assert!( + ecstore + .get_object_info(bucket, "object", &ObjectOptions::default()) + .await + .is_err() + ); + } + + async fn install_test_expiry_policy(ecstore: &Arc, bucket: &str) -> Arc { + let xml = br#"expireEnabled00"#; + metadata_sys::update_in(&ecstore.ctx, bucket, BUCKET_LIFECYCLE_CONFIG, xml.to_vec()) + .await + .expect("install expiration policy"); + get_expiry_configs(ecstore, bucket).await.unwrap().lifecycle.unwrap() + } + async fn create_test_bucket(ecstore: &Arc, bucket: &str) { ecstore .make_bucket(bucket, &MakeBucketOptions::default()) diff --git a/crates/ecstore/src/bucket/lifecycle/object_handlers_common.rs b/crates/ecstore/src/bucket/lifecycle/object_handlers_common.rs index 5df0d848e..a28d518e2 100644 --- a/crates/ecstore/src/bucket/lifecycle/object_handlers_common.rs +++ b/crates/ecstore/src/bucket/lifecycle/object_handlers_common.rs @@ -33,13 +33,17 @@ pub async fn delete_object_versions( api: &Arc, bucket: &str, to_del: &[ObjectToDelete], - _lc_event: lifecycle::Event, + lc_event: lifecycle::Event, bucket_incarnation_id: Uuid, ) -> usize { if to_del.is_empty() { return 0; } + let Some(expiry_policy) = lc_event.expiry_policy else { + return to_del.len(); + }; + if to_del.iter().any(|target| { target.version_id.is_none() || (target.version_id.is_some_and(|version_id| version_id.is_nil()) && target.expected_identity.is_none()) @@ -104,6 +108,7 @@ pub async fn delete_object_versions( let mut delete_options = ObjectOptions { delete_replication_config_snapshot: Some(Arc::clone(&delete_config_snapshot)), expected_bucket_incarnation_id: Some(bucket_incarnation_id), + lifecycle_expiry_policy: Some(expiry_policy.clone()), ..Default::default() }; delete_options.add_namespace_lock_guard(&publication_guard); @@ -125,7 +130,11 @@ pub async fn delete_object_versions( // already happened (backlog#2202). emit_noncurrent_expiration_event(bucket, target, deleted_obj, false); } - if deleted_obj.replication_state.is_none() { + if !deleted_obj + .replication_state + .as_ref() + .is_some_and(|state| !state.targets.is_empty() || !state.purge_targets.is_empty()) + { continue; } replication_sink::schedule_delete(bucket.to_string(), deleted_obj.clone()).await; diff --git a/crates/ecstore/src/bucket/lifecycle/replication_sink.rs b/crates/ecstore/src/bucket/lifecycle/replication_sink.rs index 042759176..2334d57b3 100644 --- a/crates/ecstore/src/bucket/lifecycle/replication_sink.rs +++ b/crates/ecstore/src/bucket/lifecycle/replication_sink.rs @@ -70,6 +70,11 @@ pub(crate) async fn schedule_delete(bucket: String, delete_object: DeletedObject ReplicationLifecycleBridge::schedule_delete(bucket, delete_object).await; } +#[cfg(test)] +pub(crate) fn take_scheduled_deletes_for_test() -> Vec { + ReplicationLifecycleBridge::take_scheduled_deletes_for_test() +} + #[cfg(test)] mod tests { use std::collections::HashMap; diff --git a/crates/ecstore/src/bucket/metadata_sys.rs b/crates/ecstore/src/bucket/metadata_sys.rs index 3306d37b2..78a4b30c9 100644 --- a/crates/ecstore/src/bucket/metadata_sys.rs +++ b/crates/ecstore/src/bucket/metadata_sys.rs @@ -178,6 +178,13 @@ pub(crate) fn object_lock_config_state_from_authoritative_metadata(bm: &BucketMe Ok(ObjectLockConfigState::ConfirmedAbsent) } +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct CachedQuotaMetadataSnapshot { + pub(crate) quota: Option, + pub(crate) bucket_incarnation: Uuid, + pub(crate) quota_revision: OffsetDateTime, +} + /// Convert the persisted serving-layer configuration into the storage-level /// [`DefaultRetention`](crate::bucket::object_lock::types::DefaultRetention) /// the WORM evaluation code consumes (rustfs/backlog#1842). A rule without a @@ -316,6 +323,33 @@ pub async fn remove_bucket_metadata(bucket: &str) -> Result { Ok(lock.remove(bucket).await) } +/// Remove a peer's cached metadata only when the bucket is absent under the +/// namespace read fence. A delayed delete notification must not withdraw the +/// metadata of a same-name bucket that has already been recreated. +pub async fn remove_bucket_metadata_if_absent(api: Arc, bucket: &str) -> Result { + if is_meta_bucketname(bucket) { + return Ok(false); + } + // Lock order: bucket namespace read -> metadata system read (released) + // -> metadata publish, matching peer reloads. + let namespace_lock = api.new_ns_lock(bucket, bucket).await?; + let namespace_guard = namespace_lock + .get_read_lock(crate::set_disk::get_lock_acquire_timeout()) + .await?; + let sys = bucket_metadata_sys_of(&api.ctx)?; + let sys = sys.read().await.clone(); + if sys + .bucket_exists(bucket, &namespace_guard, "peer bucket metadata removal existence check") + .await? + { + return Ok(false); + } + await_bucket_namespace_operation(Some(&namespace_guard), bucket, "peer bucket metadata removal", async { + Ok(sys.remove(bucket).await) + }) + .await +} + fn start_refresh_buckets_metadata_loop(sys: Arc>) { let Some(cancel_token) = runtime_sources::background_services_cancel_token() else { warn!("bucket metadata refresh loop skipped because background cancellation token is not initialized"); @@ -493,6 +527,36 @@ pub(crate) async fn get_bucket_incarnation_id_in(ctx: &crate::runtime::instance: sys.get_bucket_incarnation_id_from_disk(bucket).await } +/// Directory loss is recoverable only for a committed, non-retired generation. +/// The caller retains the bucket lifecycle fence through the ensuing reconstruction. +pub(crate) async fn validate_bucket_reconstruction_in( + ctx: &crate::runtime::instance::InstanceContext, + bucket: &str, + expected: Uuid, +) -> Result<()> { + let api = bucket_metadata_sys_of(ctx)?.read().await.object_store(); + let lock = api + .new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket)) + .await?; + let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?; + let (metadata, persisted) = load_bucket_metadata_parse_with_presence(api.clone(), bucket, true).await?; + if !persisted + || !metadata.bucket_incarnation_sidecar + || expected.is_nil() + || metadata.bucket_incarnation_id != expected + || !metadata.bucket_creation_committed + { + return Err(Error::PreconditionFailed); + } + if crate::bucket::retirement::is_retired(api, bucket, expected).await? { + return Err(Error::BucketNotFound(bucket.to_owned())); + } + if guard.is_lock_lost() { + return Err(Error::PreconditionFailed); + } + Ok(()) +} + /// Validate against disk while retaining an already-held Object Lock fence. pub(crate) async fn get_bucket_incarnation_id_for_options_in( ctx: &crate::runtime::instance::InstanceContext, @@ -1314,6 +1378,18 @@ pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_with_guard_ ctx: &crate::runtime::instance::InstanceContext, bucket: &str, guard: Option<&rustfs_lock::NamespaceLockGuard>, +) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> { + get_delete_config_from_disk_in(ctx, bucket, guard, None).await +} + +/// Validate queued expiration against the same fresh metadata used for Object +/// Lock admission. The caller holds the transaction read lock through commit, +/// so a successful lifecycle update cannot be followed by an old-policy delete. +pub(crate) async fn get_delete_config_from_disk_in( + ctx: &crate::runtime::instance::InstanceContext, + bucket: &str, + guard: Option<&rustfs_lock::NamespaceLockGuard>, + expiry_policy: Option<&rustfs_lifecycle::ExpiryPolicySnapshot>, ) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> { let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?; let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone(); @@ -1334,6 +1410,13 @@ pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_with_guard_ BucketMetadataAuthority::Authoritative(metadata) if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() => { + if let Some(expected) = expiry_policy { + let current = ConfigState::of(&metadata.lifecycle_config_xml, &metadata.lifecycle_config) + .require(bucket, BUCKET_LIFECYCLE_CONFIG)?; + if !expected.matches_configuration(current) { + return Err(Error::PreconditionFailed); + } + } let object_lock_config_updated_at = metadata.object_lock_config_updated_at; Ok(( object_lock_config_state_from_authoritative_metadata(&metadata)?, @@ -1434,6 +1517,9 @@ fn quota_config_and_incarnation_from_authority( BucketMetadataAuthority::Authoritative(metadata) if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() => { + if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() { + return Err(Error::other("persisted bucket quota configuration is invalid")); + } Ok(( metadata.quota_config.clone(), metadata.bucket_incarnation_id, @@ -1448,6 +1534,15 @@ fn quota_config_and_incarnation_from_authority( } } +pub(crate) async fn get_cached_quota_config_and_incarnation_in( + ctx: &crate::runtime::instance::InstanceContext, + bucket: &str, +) -> Result> { + let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?; + let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone(); + bucket_meta_sys.cached_quota_config_and_incarnation(bucket).await +} + pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> { let bucket_meta_sys_lock = get_bucket_metadata_sys()?; let bucket_meta_sys = bucket_meta_sys_lock.read().await; @@ -2982,6 +3077,27 @@ impl BucketMetadataSys { } } + async fn cached_quota_config_and_incarnation(&self, bucket: &str) -> Result> { + let metadata = { + let metadata_map = self.metadata_map.read().await; + metadata_map.get(bucket).cloned() + }; + let Some(metadata) = metadata else { + return Ok(None); + }; + if !metadata.bucket_incarnation_sidecar || metadata.bucket_incarnation_id.is_nil() { + return Ok(None); + } + if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() { + return Err(Error::other("persisted bucket quota configuration is invalid")); + } + Ok(Some(CachedQuotaMetadataSnapshot { + quota: metadata.quota_config.clone(), + bucket_incarnation: metadata.bucket_incarnation_id, + quota_revision: metadata.quota_config_updated_at, + })) + } + pub async fn get_replication_config(&self, bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> { let (bm, _) = self.get_config(bucket).await?; @@ -4886,6 +5002,154 @@ mod tests { ); } + #[tokio::test] + #[serial] + async fn quota_admission_rejects_stale_cached_no_quota() { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let sys = bucket_metadata_sys_of(&store.ctx).expect("scoped metadata system"); + let bucket = "stale-no-quota-admission"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let (stale, _) = sys.read().await.get_config(bucket).await.expect("cache initial metadata"); + store + .update_bucket_metadata_config(bucket, "quota.json", br#"{"quota":1}"#.to_vec()) + .await + .expect("publish quota to authoritative storage"); + // Model a peer that has not received the newer configuration snapshot. + sys.read().await.set(bucket.to_owned(), stale).await; + assert!( + get_cached_quota_config_and_incarnation_in(&store.ctx, bucket) + .await + .expect("read stale cache") + .expect("cached bucket identity") + .quota + .is_none() + ); + + temp_env::async_with_vars([("RUSTFS_QUOTA_BEGIN_SAFE_NO_QUOTA_CACHE_FAST_PATH", Some("true"))], async { + let error = crate::bucket::quota::reservation::begin(&store.ctx, bucket, "object", &ObjectOptions::default(), 0, 0) + .await + .err() + .expect("persisted quota must reject admission without its snapshot"); + assert!(matches!(error, Error::PartMissingOrCorrupt), "{error:?}"); + }) + .await; + } + + #[tokio::test] + #[serial] + async fn peer_reload_clears_negative_cache_after_remote_bucket_creation() { + let (dirs, node_a, node_b) = two_nodes_over_one_store().await; + let bucket = "cross-node-negative-cache-refresh"; + let node_b_sys = node_b.read().await.clone(); + + assert_eq!( + node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap_err(), + Error::BucketNotFound(bucket.to_string()), + "the probe must populate node B's missing-bucket cache" + ); + + for dir in &dirs { + std::fs::create_dir_all(dir.path().join(bucket)).expect("physical bucket should exist before creation metadata"); + } + let created = BucketMetadata::new(bucket); + let created_incarnation = created.bucket_incarnation_id; + node_a + .read() + .await + .persist_new_and_set(created) + .await + .expect("node A should publish the created bucket metadata"); + + node_b_sys + .reload_from_store(bucket) + .await + .expect("peer reload should clear the stale negative cache"); + assert_eq!( + node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap(), + created_incarnation, + "node B should resolve the newly created bucket immediately" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + #[serial] + async fn peer_delete_clears_metadata_for_absent_bucket() { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner"); + let bucket = "peer-delete-absent-bucket"; + + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("initial bucket creation"); + let cached = sys.read().await.get(bucket).await.expect("initial cached metadata"); + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("bucket deletion"); + + // Model a peer that still caches the metadata after the physical delete. + sys.read().await.set(bucket.to_string(), cached).await; + assert!( + remove_bucket_metadata_if_absent(store.clone(), bucket) + .await + .expect("conditional peer delete"), + "the peer must evict the deleted bucket's cached metadata" + ); + let sys = sys.read().await; + assert!(matches!(sys.get(bucket).await, Err(Error::ConfigNotFound))); + assert_eq!( + sys.get_bucket_incarnation_id(bucket) + .await + .expect_err("deleted bucket lookup"), + Error::BucketNotFound(bucket.to_string()), + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + #[serial] + async fn delayed_peer_delete_preserves_recreated_bucket_metadata() { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner"); + let bucket = "delayed-peer-delete-recreated-bucket"; + + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("initial bucket creation"); + let old_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("old incarnation"); + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("bucket deletion"); + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("bucket recreation"); + let new_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("new incarnation"); + assert_ne!(old_incarnation, new_incarnation); + + let removed = remove_bucket_metadata_if_absent(store.clone(), bucket) + .await + .expect("conditional peer delete"); + assert!(!removed, "a delayed delete must not remove metadata for a live recreated bucket"); + assert_eq!( + sys.read() + .await + .get(bucket) + .await + .expect("recreated metadata should remain cached") + .bucket_incarnation_id, + new_incarnation, + ); + } + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] #[serial] async fn stale_config_request_cannot_mutate_a_recreated_bucket() { diff --git a/crates/ecstore/src/bucket/migration.rs b/crates/ecstore/src/bucket/migration.rs index 723b75c69..6785795c2 100644 --- a/crates/ecstore/src/bucket/migration.rs +++ b/crates/ecstore/src/bucket/migration.rs @@ -273,6 +273,30 @@ where let buckets: Vec = buckets_list.into_iter().map(|b| b.name).collect(); + try_migrate_bucket_metadata_for_buckets(store, &buckets).await +} + +/// Migrate the already discovered startup candidates without repeating a +/// majority-only namespace listing. Every migration write retains its quorum. +pub async fn try_migrate_bucket_metadata_for_buckets(store: Arc, buckets: &[String]) -> Result<()> +where + S: ObjectIO< + Error = Error, + RangeSpec = HTTPRangeSpec, + HeaderMap = HeaderMap, + ObjectOptions = ObjectOptions, + ObjectInfo = ObjectInfo, + GetObjectReader = GetObjectReader, + PutObjectReader = PutObjReader, + > + ObjectOperations< + Error = Error, + ObjectInfo = ObjectInfo, + ObjectOptions = ObjectOptions, + FileInfo = FileInfo, + ObjectToDelete = ObjectToDelete, + DeletedObject = DeletedObject, + >, +{ if buckets.is_empty() { debug!("No migrating bucket metadata found"); return Ok(()); diff --git a/crates/ecstore/src/bucket/quota/reservation.rs b/crates/ecstore/src/bucket/quota/reservation.rs index 59c51c301..f28918085 100644 --- a/crates/ecstore/src/bucket/quota/reservation.rs +++ b/crates/ecstore/src/bucket/quota/reservation.rs @@ -419,7 +419,6 @@ pub(crate) struct QuotaContext { quota_revision: Option, quota_limit: Option, reservation_protocol: Option, - capability_proof: Option, snapshot_admission: Option, legacy_data_movement: bool, metadata_guard: Option>, @@ -479,7 +478,6 @@ impl QuotaContext { reservation: reservation.clone(), }; let metadata_guard = self.metadata_guard; - let capability_proof = self.capability_proof; tokio::spawn(async move { reap_stale_reservations(Arc::clone(&store), &ledger_data.bucket, &ledger_data.ledger_object).await?; @@ -526,7 +524,6 @@ impl QuotaContext { ledger: Some(ReservationLedgerData::Single(ledger_data)), operation_guard: Some(operation_guard), metadata_guard, - capability_proof, state: ReservationState::Pending, }) }) @@ -555,7 +552,6 @@ async fn reserve_sharded(context: QuotaContext, old_size: u64, new_size: u64) -> commit_started: false, }; let metadata_guard = context.metadata_guard; - let capability_proof = context.capability_proof; let data = ShardedReservationData { store: Arc::clone(&store), bucket: context.bucket, @@ -640,7 +636,6 @@ async fn reserve_sharded(context: QuotaContext, old_size: u64, new_size: u64) -> ledger: Some(ReservationLedgerData::Sharded(data)), operation_guard: Some(operation_guard), metadata_guard, - capability_proof, state: ReservationState::Pending, }); } @@ -725,20 +720,12 @@ impl ReservationLedgerData { Self::Sharded(data) => &data.bucket, } } - - fn error_object(&self) -> &str { - match self { - Self::Single(data) => &data.ledger_object, - Self::Sharded(data) => &data.shard_object, - } - } } pub(crate) struct QuotaReservation { ledger: Option, operation_guard: Option, metadata_guard: Option>, - capability_proof: Option, state: ReservationState, } @@ -756,7 +743,6 @@ impl QuotaReservation { ledger: None, operation_guard: None, metadata_guard, - capability_proof: None, state: ReservationState::Pending, } } @@ -766,17 +752,7 @@ impl QuotaReservation { || self.metadata_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) } - pub(crate) fn capability_proof_matches(&self) -> bool { - self.capability_proof - .as_ref() - .is_none_or(crate::services::notification_sys::cross_pool_fence_fleet_proof_matches) - } - pub(crate) async fn mark_commit_started(&mut self) -> Result<()> { - if !self.capability_proof_matches() { - let ledger = self.ledger.as_ref().ok_or(StorageError::PartMissingOrCorrupt)?; - return Err(quota_capability_error(ledger.bucket(), ledger.error_object())); - } if let Some(ledger) = self.ledger.as_ref() { mark_commit_started(ledger).await?; } @@ -850,6 +826,11 @@ pub(crate) async fn begin( let snapshot_admission = opts.quota_admission; let data_movement = opts.data_movement; if crate::bucket::utils::is_meta_bucketname(bucket) { + let meta_bucket_fast_path_started = rustfs_io_metrics::put_stage_timer(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_META_BUCKET_FAST_PATH, + meta_bucket_fast_path_started, + ); return Ok(QuotaContext { store: None, bucket: bucket.to_string(), @@ -859,7 +840,6 @@ pub(crate) async fn begin( quota_revision: None, quota_limit: None, reservation_protocol: None, - capability_proof: None, snapshot_admission: None, legacy_data_movement: false, metadata_guard: None, @@ -878,7 +858,6 @@ pub(crate) async fn begin( quota_revision: None, quota_limit: Some(snapshot_admission.quota_limit()), reservation_protocol: None, - capability_proof: None, snapshot_admission: Some(snapshot_admission), legacy_data_movement: false, metadata_guard: None, @@ -897,7 +876,6 @@ pub(crate) async fn begin( quota_revision: None, quota_limit: None, reservation_protocol: None, - capability_proof: None, snapshot_admission: None, legacy_data_movement: false, metadata_guard: None, @@ -906,9 +884,63 @@ pub(crate) async fn begin( }); } + let metadata_lock_started = rustfs_io_metrics::put_stage_timer(); let metadata_guard = metadata_sys::acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts).await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_METADATA_LOCK, + metadata_lock_started, + ); + let cache_lookup_started = rustfs_io_metrics::put_stage_timer(); + let cached_quota_snapshot = if rustfs_io_metrics::put_stage_metrics_enabled() { + match metadata_sys::get_cached_quota_config_and_incarnation_in(ctx, bucket).await { + Ok(Some(snapshot)) => { + rustfs_io_metrics::record_put_object_quota_cache_decision( + "candidate", + quota_cache_reason(snapshot.quota.as_ref()), + ); + Some(snapshot) + } + Ok(None) => { + rustfs_io_metrics::record_put_object_quota_cache_decision("miss", "no_authoritative_cache"); + None + } + Err(_) => { + rustfs_io_metrics::record_put_object_quota_cache_decision("miss", "invalid_cached_quota"); + None + } + } + } else { + None + }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CACHE_LOOKUP, + cache_lookup_started, + ); + let config_read_started = rustfs_io_metrics::put_stage_timer(); let (quota, bucket_incarnation, quota_revision) = metadata_sys::get_quota_config_and_incarnation_from_disk_for_options_in(ctx, bucket, opts, &metadata_guard).await?; + if let Some(cached) = cached_quota_snapshot.as_ref() { + let reason = quota_cache_reason(quota.as_ref()); + if cached.quota == quota && cached.bucket_incarnation == bucket_incarnation && cached.quota_revision == quota_revision { + rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_match", reason); + } else { + rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch", reason); + if cached.quota != quota { + rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch_field", "quota"); + } + if cached.bucket_incarnation != bucket_incarnation { + rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch_field", "bucket_incarnation"); + } + if cached.quota_revision != quota_revision { + rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch_field", "quota_revision"); + } + } + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONFIG_READ, + config_read_started, + ); + let lock_lost_check_started = rustfs_io_metrics::put_stage_timer(); if metadata_guard.is_lock_lost() { return Err(StorageError::NamespaceLockQuorumUnavailable { mode: "quota_config", @@ -918,6 +950,11 @@ pub(crate) async fn begin( achieved: 0, }); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_LOCK_LOST_CHECK, + lock_lost_check_started, + ); + let policy_evaluate_started = rustfs_io_metrics::put_stage_timer(); if quota .as_ref() .is_some_and(|quota| quota.has_unsupported_reservation_protocol()) @@ -926,15 +963,15 @@ pub(crate) async fn begin( return Err(StorageError::PartMissingOrCorrupt); } let durable_quota = quota.as_ref().filter(|quota| quota.uses_durable_reservations()); - let capability_proof = if durable_quota.is_some() { - Some( - crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof() - .ok_or_else(|| quota_capability_error(bucket, &ledger_object(bucket)))?, - ) - } else { - None - }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_POLICY_EVALUATE, + policy_evaluate_started, + ); + // Durable writes are fenced by a write quorum of participating disks that + // enforce the configured topology. Fleet-wide proof remains a configuration gate; + // losing an unrelated peer must not revoke an admitted object write. let durable_quota_limit = durable_quota.and_then(|quota| quota.quota); + let snapshot_admission_started = rustfs_io_metrics::put_stage_timer(); let reservation_protocol = durable_quota.and_then(|quota| quota.reservation_protocol); let snapshot_admission = match quota.as_ref().filter(|quota| !quota.uses_durable_reservations()) { Some(quota) => match (quota.quota, snapshot_admission) { @@ -954,6 +991,10 @@ pub(crate) async fn begin( }, None => None, }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_SNAPSHOT_ADMISSION, + snapshot_admission_started, + ); let legacy_data_movement = durable_quota_limit.is_none() && quota.as_ref().and_then(|quota| quota.quota).is_some() && snapshot_admission.is_none() @@ -965,11 +1006,21 @@ pub(crate) async fn begin( .then(|| quota.as_ref().and_then(|quota| quota.quota)) .flatten() }); + let object_store_lookup_started = rustfs_io_metrics::put_stage_timer(); let store = if durable_quota_limit.is_some() { Some(metadata_sys::object_store_in(ctx).await?) } else { None }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_OBJECT_STORE_LOOKUP, + object_store_lookup_started, + ); + let context_build_started = rustfs_io_metrics::put_stage_timer(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONTEXT_BUILD, + context_build_started, + ); Ok(QuotaContext { store, bucket: bucket.to_string(), @@ -979,7 +1030,6 @@ pub(crate) async fn begin( quota_revision: Some(quota_revision), quota_limit, reservation_protocol, - capability_proof, snapshot_admission, legacy_data_movement, metadata_guard: Some(metadata_guard), @@ -988,13 +1038,12 @@ pub(crate) async fn begin( }) } -fn quota_capability_error(bucket: &str, object: &str) -> StorageError { - StorageError::NamespaceLockQuorumUnavailable { - mode: "quota_capability", - bucket: bucket.to_string(), - object: object.to_string(), - required: 1, - achieved: 0, +fn quota_cache_reason(quota: Option<&crate::bucket::quota::BucketQuota>) -> &'static str { + match quota { + None => "no_quota", + Some(quota) if quota.uses_durable_reservations() => "durable_quota", + Some(quota) if quota.quota.is_some() => "snapshot_quota", + Some(_) => "quota_without_limit", } } diff --git a/crates/ecstore/src/bucket/replication/replication_pool.rs b/crates/ecstore/src/bucket/replication/replication_pool.rs index f0eb97e2d..1c0d8cd52 100644 --- a/crates/ecstore/src/bucket/replication/replication_pool.rs +++ b/crates/ecstore/src/bucket/replication/replication_pool.rs @@ -3383,7 +3383,7 @@ async fn queue_replicate_deletes(batch: ReplicationHealResyncDeletes) -> Replica #[cfg(test)] mod tests { - use super::super::replication_filemeta_boundary::ReplicateTargetDecision; + use super::super::replication_filemeta_boundary::{ReplicateTargetDecision, ReplicationState, VersionPurgeStatusType}; use super::super::replication_resync_boundary::{decode_mrf_file, encode_mrf_file, encode_resync_file}; use super::super::replication_storage_boundary::{ DeletedObject, FileInfo, GetObjectReader, HTTPRangeSpec, ListOperations, ObjectIO, ObjectOperations, PutObjReader, @@ -3420,6 +3420,7 @@ mod tests { hold_first_read: AtomicBool, allow_first_read: Notify, read_count: AtomicUsize, + delete_storage_calls: AtomicUsize, reported_size: StdMutex>, stream_read_bytes: Arc, write_count: AtomicUsize, @@ -3628,6 +3629,7 @@ mod tests { _object: &str, _opts: &Self::ObjectOptions, ) -> Result { + self.shared.delete_storage_calls.fetch_add(1, Ordering::SeqCst); Err(EcstoreError::NotImplemented) } @@ -3669,6 +3671,7 @@ mod tests { _object: &str, _opts: Self::ObjectOptions, ) -> Result { + self.shared.delete_storage_calls.fetch_add(1, Ordering::SeqCst); Err(EcstoreError::NotImplemented) } @@ -3790,6 +3793,7 @@ mod tests { type NamespaceLock = rustfs_lock::NamespaceLockWrapper; async fn new_ns_lock(&self, bucket: &str, object: &str) -> Result { + self.shared.delete_storage_calls.fetch_add(1, Ordering::SeqCst); let lock = rustfs_lock::NamespaceLock::with_local_manager("load-resync-test".to_string(), self.shared.lock_manager.clone()); Ok(rustfs_lock::NamespaceLockWrapper::new( @@ -4166,6 +4170,7 @@ mod tests { hold_first_read: AtomicBool::new(false), allow_first_read: Notify::new(), read_count: AtomicUsize::new(0), + delete_storage_calls: AtomicUsize::new(0), reported_size: StdMutex::new(None), stream_read_bytes: Arc::new(AtomicUsize::new(0)), write_count: AtomicUsize::new(0), @@ -4177,6 +4182,106 @@ mod tests { }) } + #[tokio::test] + async fn delete_worker_empty_admission_skips_storage() { + let mut declined = ReplicateDecision::default(); + declined.set(ReplicateTargetDecision::new("arn:test".to_string(), false, false)); + for state in [ + None, + Some(ReplicationState::default()), + Some(ReplicationState { + replicate_decision_str: declined.to_string(), + ..Default::default() + }), + ] { + for delete_marker in [false, true] { + let shared = empty_resync_shared_state(); + let storage = Arc::new(LoadResyncNodeStore::new("empty-delete", shared.clone())); + let version_id = Uuid::new_v4(); + let delete = DeletedObjectReplicationInfo { + bucket: "no-replication".to_string(), + delete_object: ReplicationDeletedObject { + object_name: "expired-version".to_string(), + version_id: (!delete_marker).then_some(version_id), + delete_marker_version_id: delete_marker.then_some(version_id), + delete_marker, + replication_state: state.clone(), + ..Default::default() + }, + ..Default::default() + }; + let completed = replicate_delete_with_outcome(delete, storage).await; + assert_eq!( + shared.delete_storage_calls.load(Ordering::SeqCst), + 0, + "empty admission must not lock, read, or write storage (marker={delete_marker}, state={state:?})" + ); + assert!(completed, "an empty delete is a completed no-op"); + } + } + } + + #[tokio::test] + async fn delete_worker_preserves_persisted_admission_without_live_config() { + let arn = "arn:test"; + for state in [ + ReplicationState { + replicate_decision_str: test_replicate_decision(&[arn]).to_string(), + ..Default::default() + }, + ReplicationState { + targets: HashMap::from([(arn.to_string(), ReplicationStatusType::Pending)]), + ..Default::default() + }, + ReplicationState { + purge_targets: HashMap::from([(arn.to_string(), VersionPurgeStatusType::Pending)]), + ..Default::default() + }, + ] { + let shared = empty_resync_shared_state(); + let storage = Arc::new(LoadResyncNodeStore::new("pending-delete", shared.clone())); + let delete = DeletedObjectReplicationInfo { + bucket: "no-live-config".to_string(), + delete_object: ReplicationDeletedObject { + object_name: "pending-version".to_string(), + version_id: Some(Uuid::new_v4()), + replication_state: Some(state), + ..Default::default() + }, + ..Default::default() + }; + assert!( + !replicate_delete_with_outcome(delete, storage).await, + "the injected storage failure must retain admitted work for retry" + ); + assert!(shared.delete_storage_calls.load(Ordering::SeqCst) > 0); + } + } + + #[tokio::test] + async fn delete_worker_rejects_malformed_empty_admission() { + let shared = empty_resync_shared_state(); + let storage = Arc::new(LoadResyncNodeStore::new("invalid-delete", shared.clone())); + let delete = DeletedObjectReplicationInfo { + bucket: "no-replication".to_string(), + delete_object: ReplicationDeletedObject { + object_name: "invalid-decision".to_string(), + version_id: Some(Uuid::new_v4()), + replication_state: Some(ReplicationState { + replicate_decision_str: "invalid".to_string(), + ..Default::default() + }), + ..Default::default() + }, + ..Default::default() + }; + assert!( + !replicate_delete_with_outcome(delete, storage).await, + "malformed decisions must not be acknowledged as empty work" + ); + assert_eq!(shared.delete_storage_calls.load(Ordering::SeqCst), 0); + } + #[tokio::test] async fn durable_resync_read_observes_cancellation_after_cached_pending_intent() { let bucket = "startup-cancel-race"; @@ -5198,6 +5303,7 @@ mod tests { hold_first_read: AtomicBool::new(false), allow_first_read: Notify::new(), read_count: AtomicUsize::new(0), + delete_storage_calls: AtomicUsize::new(0), reported_size: StdMutex::new(None), stream_read_bytes: Arc::new(AtomicUsize::new(0)), write_count: AtomicUsize::new(0), diff --git a/crates/ecstore/src/bucket/replication/replication_resyncer.rs b/crates/ecstore/src/bucket/replication/replication_resyncer.rs index f69e662b4..e2b642681 100644 --- a/crates/ecstore/src/bucket/replication/replication_resyncer.rs +++ b/crates/ecstore/src/bucket/replication/replication_resyncer.rs @@ -36,9 +36,9 @@ use super::replication_object_decision_boundary::{ MustReplicateOptions, ReplicationMultipartPartInput, delete_marker_purge_mrf_entry, delete_marker_purge_version_id, delete_replication_creates_marker, delete_replication_target_version_id, heal_uses_delete_replication_path, is_object_lock_denied_delete, is_retryable_delete_replication_head_error, is_version_delete_replication, - replicate_delete_outcome, replication_etags_match, replication_multipart_complete_actual_size, - replication_multipart_part_plan, replication_single_put_size_error, resync_existing_delete_replication_info, - should_retry_delete_marker_purge, single_part_replica_etag_mismatch, + replicate_delete_outcome, replication_multipart_complete_actual_size, replication_multipart_part_plan, + replication_single_put_size_error, resync_existing_delete_replication_info, should_retry_delete_marker_purge, + single_part_replica_etag_mismatch, }; use super::replication_queue_boundary::{DeletedObjectReplicationInfo, ReplicationQueueAdmission}; use super::replication_resync_boundary::ResyncStatusType; @@ -60,12 +60,13 @@ use super::replication_storage_boundary::{ use super::replication_storage_boundary::{NamespaceLockFence, NamespaceLockSignalTestFence, ReplicationDeletedObject}; use super::replication_tagging_boundary::ReplicationTagFilter; use super::replication_target_boundary::{ - ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED, HeadObjectSdkError, PutObjectOptions, PutObjectPartOptions, - RemotePutObjectResponse, ReplicaLocation, ReplicationTargetStore, S3ClientError, SsecPassthroughCapability, - SsecPassthroughGate, TargetClient, VersionIdentityCapability, is_replication_target_offline_error, - replication_action_for_target_head, replication_complete_multipart_options, replication_delete_marker_purge_remove_options, - replication_delete_remove_options, replication_force_delete_remove_options, replication_object_is_ssec_encrypted, - replication_put_object_header_size, replication_put_object_options, replication_target_head_is_newer_null_version, + ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED, EncryptedReplicaShape, HeadObjectSdkError, PutObjectOptions, + PutObjectPartOptions, RemotePutObjectResponse, ReplicaCandidates, ReplicaEvidence, ReplicaIdentity, ReplicaLocation, + ReplicationTargetStore, S3ClientError, SsecPassthroughCapability, SsecPassthroughGate, TargetClient, + VersionIdentityCapability, is_replication_target_offline_error, replication_action_for_target_head, + replication_complete_multipart_options, replication_delete_marker_purge_remove_options, replication_delete_remove_options, + replication_force_delete_remove_options, replication_object_is_ssec_encrypted, replication_put_object_header_size, + replication_put_object_options, replication_source_etag_is_comparable, replication_target_head_is_newer_null_version, resolve_read_api_version_id, ssec_passthrough_evidence_present, ssec_passthrough_gate, version_identity_capability_from_put, version_identity_drifted, }; @@ -349,15 +350,23 @@ fn replica_etag_verification_enabled() -> bool { /// target that stores a transformed payload (e.g. undecoded `aws-chunked` /// frames, #6853) returns the ETag of what it actually wrote. Reporting /// COMPLETED over such a replica is silent corruption, so a decidable -/// mismatch fails the replication instead. An SSE-C ciphertext passthrough -/// transfer is exempt: the wire bytes are ciphertext while the source ETag is -/// the plaintext MD5, and that path has its own HEAD-back audit. +/// mismatch fails the replication instead. Encrypted sources are exempt: an +/// SSE-C ciphertext passthrough sends ciphertext while the source ETag is the +/// plaintext MD5 (that path has its own HEAD-back audit), and a managed-SSE +/// (SSE-S3 / SSE-KMS) source stores a keyed fingerprint, or a legacy plaintext +/// MD5, that the ETag of the target's re-encrypted replica can neither confirm +/// nor refute. A managed-SSE source forwards no checksum either +/// (`replication_put_object_options`), so its replica has no post-PUT +/// content audit. fn verify_single_part_replica( object_info: &ObjectInfo, response: &RemotePutObjectResponse, ciphertext_passthrough: bool, ) -> std::result::Result<(), std::io::Error> { - if ciphertext_passthrough || !replica_etag_verification_enabled() { + if ciphertext_passthrough + || !replication_source_etag_is_comparable(&object_info.user_defined) + || !replica_etag_verification_enabled() + { return Ok(()); } if single_part_replica_etag_mismatch(object_info.etag.as_deref(), response.etag.as_deref()) { @@ -448,6 +457,26 @@ fn is_version_id_format_mismatch(err: &SdkError) -> bool { is_version_id_mismatch(code, raw_status) } +/// A version-addressed HEAD rejected because the target cannot address the +/// source version id (AWS S3: bodiless 400). Narrower than +/// [`is_version_id_format_mismatch`]: a bodiless 403 may be a genuine +/// `s3:GetObjectVersion` denial, so it is not accepted where a mutation or a +/// completion verdict follows instead of another non-destructive HEAD. +fn rejects_source_version_id(err: &SdkError) -> bool { + is_version_id_format_mismatch(err) && !has_raw_status(err, 403) +} + +/// Whether the target's latest version of `object` is a delete marker. For +/// targets that mint their own version ids, which cannot be asked about the +/// source marker id: S3 answers a versionless HEAD on such a key with 404 +/// (405 on some targets) plus `x-amz-delete-marker: true`. +async fn latest_target_version_is_delete_marker(tgt_client: &TargetClient, object: &str) -> bool { + match head_object_for_worker(tgt_client, &tgt_client.bucket, object, None).await { + Ok(head) => head.delete_marker == Some(true), + Err(e) => raw_header_value(e.as_ref(), "x-amz-delete-marker").as_deref() == Some("true"), + } +} + async fn mark_replication_target_offline_if_needed(target_client: &Arc, err: &(impl Display + ?Sized)) { if is_replication_target_offline_error(err) { ReplicationTargetStore::mark_target_offline(target_client).await; @@ -462,6 +491,7 @@ async fn head_object_fallback( Ok(head) => Ok(Some(LocatedReplica { version_id: head.version_id.clone(), head, + by: LocatedBy::CurrentVersion, })), Err(e) if head_object_not_found(&e) => Ok(None), Err(e) => Err(e), @@ -475,6 +505,63 @@ async fn head_object_fallback( struct LocatedReplica { head: HeadObjectOutput, version_id: Option, + by: LocatedBy, +} + +/// How [`replica_head_fallback`] reached a replica, which decides what may +/// confirm it (see [`located_replica_matches`]). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum LocatedBy { + /// The target version the source ledger recorded for this version. + Ledger, + /// The one candidate content identity left (`ListObjectVersions`). + ContentIdentity, + /// Whatever version is current under the key (AWS-style fallback): + /// nothing ties it to this source version but an ETag match. + CurrentVersion, +} + +/// How this source version recognises its replica on a target. `None` when +/// the source carries nothing to compare (no ETag on a plaintext source). +fn replica_identity(object_info: &ObjectInfo) -> Option> { + let etag = object_info.etag.as_deref().filter(|etag| !etag.trim().is_empty()); + if replication_source_etag_is_comparable(&object_info.user_defined) { + return etag.map(ReplicaIdentity::Etag); + } + // Same byte count the replication PUT sends: SSE-C passes the stored + // ciphertext through, managed SSE sends the decrypted plaintext. + let size = if replication_object_is_ssec_encrypted(&object_info.user_defined) { + object_info.size + } else { + object_info.get_actual_size().ok()? + }; + match object_info.mod_time { + Some(mod_time) if size >= 0 => Some(ReplicaIdentity::Encrypted { + etag, + shape: EncryptedReplicaShape { + size, + not_before_secs: mod_time.unix_timestamp(), + }, + }), + _ => etag.map(ReplicaIdentity::Etag), + } +} + +/// Whether a replica reached through [`replica_head_fallback`] holds this +/// source version. An ETag match always does. For an encrypted source on a +/// target that computes its own ETag, the shape does too — but only for a +/// replica the ledger or content identity singled out, never for whichever +/// version happens to be current. A plaintext source still needs its ETag. +fn located_replica_matches(identity: Option<&ReplicaIdentity<'_>>, located: &LocatedReplica) -> bool { + let Some(identity) = identity else { + return false; + }; + let head = &located.head; + match identity.evidence(head.e_tag(), head.content_length(), head.last_modified()) { + Some(ReplicaEvidence::Etag) => true, + Some(ReplicaEvidence::Shape) => located.by != LocatedBy::CurrentVersion, + None => false, + } } fn head_object_not_found(err: &HeadObjectSdkError) -> bool { @@ -556,15 +643,20 @@ fn ledger_target_version_id(user_defined: &HashMap, arn: &str) - /// Second look at a replica whose version-addressed HEAD failed, for the two /// target shapes where that failure is not a verdict on the replica: /// -/// - AWS-style 400/403 (the RustFS uuid is rejected as malformed): HEAD the -/// current version without a version id; callers compare ETags. +/// - AWS-style 400/403 (the RustFS uuid is rejected as malformed) and /// - 404 on a target that mints its own version ids (the Wasabi shape, -/// rustfs/backlog#2340): the source id never existed there. HEAD the id -/// the source ledger recorded for this target when it has one; otherwise, -/// once the target is known to mint ids, locate the replica by exact key -/// and ETag through ListObjectVersions and HEAD the id the target assigned. -/// Without this, every heal, MRF retry and existing-object resync re-drive -/// PUTs the object again and mints one more target version. +/// rustfs/backlog#2340): the source id never existed there. +/// +/// Both HEAD the id the source ledger recorded for this target first. Without +/// one, the AWS shape HEADs the current version without a version id +/// (callers compare ETags); the Wasabi shape, once the target is known to +/// mint ids, locates the replica by exact key and content identity through +/// ListObjectVersions and HEADs the id the target assigned. An encrypted +/// source takes that listing on the AWS shape too: its ETag never matches a +/// target-computed one (docs/operations/encrypted-etags.md), so only a unique +/// shape match can stand in for it. Without this, every heal, MRF retry and +/// existing-object resync re-drive PUTs the object again and mints one more +/// target version. /// /// `content_identity` is the sibling-ledger view used to disown candidates /// other source versions already claim; `None` disables content identity @@ -581,7 +673,7 @@ struct ReplicaSource<'a> { bucket: &'a str, object: &'a str, version_id: Option, - etag: Option<&'a str>, + identity: Option>, ledger_version_id: Option<&'a str>, } @@ -595,13 +687,11 @@ async fn replica_head_fallback( bucket: source_bucket, object, version_id: source_version_id, - etag: source_etag, + identity, ledger_version_id, } = source; - if is_version_id_format_mismatch(err) { - return Some(head_object_fallback(tgt_client, object).await); - } - if !head_object_not_found(err) { + let format_mismatch = is_version_id_format_mismatch(err); + if !format_mismatch && !head_object_not_found(err) { return None; } if let Some(ledger_version_id) = ledger_version_id { @@ -613,24 +703,45 @@ async fn replica_head_fallback( return Some(Ok(Some(LocatedReplica { head, version_id: Some(ledger_version_id.to_string()), + by: LocatedBy::Ledger, }))); } // The recorded version is gone: fall through to content identity // before concluding that the replica is missing. - Err(e) if head_object_not_found(&e) => {} + Err(e) if head_object_not_found(&e) || is_version_id_format_mismatch(&e) => {} Err(e) => return Some(Err(e)), } } - if !ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable() { + let encrypted = matches!(identity, Some(ReplicaIdentity::Encrypted { .. })); + if format_mismatch { + if !(encrypted && content_identity.is_some()) { + return Some(head_object_fallback(tgt_client, object).await); + } + } else if !ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable() { return None; } let siblings = content_identity?; - let etag = source_etag.filter(|etag| !etag.trim().is_empty())?; - Some(match tgt_client.replica_candidates_by_etag(&tgt_client.bucket, object, etag).await { + let identity = identity?; + let candidates = match tgt_client.replica_candidates(&tgt_client.bucket, object, &identity).await { + Ok(candidates) => Ok(candidates), + // A target that refused the version-addressed HEAD (a bodiless 403 + // can be a permission denial) may refuse the listing too: keep the + // current-version fallback, which re-PUTs, instead of stalling. + Err(_) if format_mismatch => return Some(head_object_fallback(tgt_client, object).await), + Err(list_err) => Err(list_err), + }; + Some(match candidates { Ok(candidates) => { - let location = + let (location, evidence) = disowned_replica_location(siblings, &tgt_client.arn, source_bucket, object, source_version_id, candidates).await; - match location.any_version_id() { + // Several same-ETag versions all hold the source bytes, so any of + // them proves the replica exists. Several same-shape versions do + // not: re-PUT rather than adopt the wrong generation. + let assigned_version_id = match (&location, evidence) { + (ReplicaLocation::Ambiguous { .. }, ReplicaEvidence::Shape) => None, + _ => location.any_version_id(), + }; + match assigned_version_id { Some(assigned_version_id) => { debug!( event = EVENT_REPLICATION_DRIFTED_REPLICA_LOCATED, @@ -641,6 +752,7 @@ async fn replica_head_fallback( arn = %tgt_client.arn, assigned_version_id = %assigned_version_id, ambiguous = matches!(location, ReplicaLocation::Ambiguous { .. }), + evidence = ?evidence, "Located replica by content identity on a target that mints its own version ids" ); match head_object_for_worker(tgt_client, &tgt_client.bucket, object, Some(assigned_version_id.to_string())) @@ -649,6 +761,7 @@ async fn replica_head_fallback( Ok(head) => Ok(Some(LocatedReplica { head, version_id: Some(assigned_version_id.to_string()), + by: LocatedBy::ContentIdentity, })), // The located version disappeared between LIST and HEAD. Err(e) if head_object_not_found(&e) => Ok(None), @@ -671,16 +784,15 @@ async fn disowned_replica_location( source_bucket: &str, object: &str, source_version_id: Option, - mut candidates: Vec, -) -> ReplicaLocation { + candidates: ReplicaCandidates, +) -> (ReplicaLocation, ReplicaEvidence) { if candidates.is_empty() { - return ReplicaLocation::Missing; + return (ReplicaLocation::Missing, ReplicaEvidence::Etag); } let claimed = siblings .claimed_target_versions(source_bucket, object, source_version_id, arn) .await; - candidates.retain(|candidate| !claimed.contains(candidate)); - ReplicaLocation::from_matches(candidates) + candidates.locate(&claimed) } /// The id the target assigned to a located replica when it is not the source @@ -1827,10 +1939,14 @@ fn spawn_resync_walk_task( /// Classify the target HEAD verification result for one resynced object, /// updating the per-object status counters and returning the accounted size -/// together with any verification error. +/// together with any verification error. `source` is the source version as +/// stored after the resync replicated it (its ledger and encryption decide +/// how the replica is recognised); without it the queued ETag is all there +/// is to go on. async fn verify_resync_head_result( head_result: std::result::Result, roi: &ReplicateObjectInfo, + source: Option<&ObjectInfo>, st: &mut TargetReplicationResyncStatus, target_client: &Arc, ) -> (i64, Option) { @@ -1846,7 +1962,17 @@ async fn verify_resync_head_result( // only absence proves completion. All other outcomes remain // failed (backlog#862 / #799 B13). let retryable = if roi.version_purge_status.is_empty() { - matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Ambiguous) + match classify_delete_marker_head_error(&err) { + // A target that mints its own version ids never holds the + // source marker id; verify the marker on the latest version. + DeleteMarkerHeadOutcome::Ambiguous => { + let mints_own = rejects_source_version_id(&err) + || ReplicationTargetStore::version_identity_capability(&target_client.arn) + .version_addressing_unreliable(); + !(mints_own && latest_target_version_is_delete_marker(target_client, &roi.name).await) + } + DeleteMarkerHeadOutcome::Exists | DeleteMarkerHeadOutcome::Missing => false, + } } else { !matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Missing) }; @@ -1862,14 +1988,23 @@ async fn verify_resync_head_result( // A version-addressed HEAD is not the last word on every target: // re-verify through the fallback before counting a well-replicated // object as failed (see `replica_head_fallback`). + let ledger_version_id = source.and_then(|info| ledger_target_version_id(&info.user_defined, &target_client.arn)); + let identity = match source { + Some(info) => replica_identity(info), + None => roi + .etag + .as_deref() + .filter(|etag| !etag.trim().is_empty()) + .map(ReplicaIdentity::Etag), + }; match replica_head_fallback( target_client.as_ref(), ReplicaSource { bucket: &roi.bucket, object: &roi.name, version_id: roi.version_id, - etag: roi.etag.as_deref(), - ledger_version_id: None, + identity, + ledger_version_id: ledger_version_id.as_deref(), }, Some(&NoSiblingLedger), &err, @@ -1894,6 +2029,24 @@ async fn verify_resync_head_result( } } +/// The resynced source version as stored now, ledger included, for the +/// verification fallback. `None` when it cannot be read. +async fn resync_source_object_info(storage: &Arc, roi: &ReplicateObjectInfo) -> Option { + storage + .get_object_info( + &roi.bucket, + &roi.name, + &ObjectOptions { + version_id: roi.version_id.map(|version_id| version_id.to_string()), + versioned: ReplicationVersioningStore::prefix_enabled(&roi.bucket, &roi.name).await, + version_suspended: ReplicationVersioningStore::prefix_suspended(&roi.bucket, &roi.name).await, + ..Default::default() + }, + ) + .await + .ok() +} + /// Replicate one existing object (or delete marker / version purge) to the /// resync target, verify the outcome via a target HEAD, and produce the /// per-object resync status update. @@ -1928,7 +2081,12 @@ async fn resync_worker_process_object( roi.version_id.map(|v| v.to_string()), ) .await; - let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, target_client).await; + let source = if head_result.is_err() && !roi.delete_marker { + resync_source_object_info(storage, &roi).await + } else { + None + }; + let (size, err) = verify_resync_head_result(head_result, &roi, source.as_ref(), &mut st, target_client).await; if err.is_some() { debug!( @@ -2269,6 +2427,51 @@ pub(crate) async fn replicate_delete_with_outcome( dobj.delete_object.version_id }; + let dsc = match parse_replicate_decision( + &bucket, + &dobj + .delete_object + .replication_state + .as_ref() + .map(|v| v.replicate_decision_str.clone()) + .unwrap_or_default(), + ) { + Ok(dsc) => dsc, + Err(err) => { + debug!( + event = EVENT_REPLICATION_DELETE_SKIPPED, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC, + bucket = %bucket, + arn = %dobj.target_arn, + error = %err, + reason = "replicate_decision_parse_failed", + "Failed to parse replicate decision" + ); + send_local_event(EventArgs { + event_name: EventName::ObjectReplicationNotTracked.to_string(), + bucket_name: bucket.clone(), + object: ObjectInfo { + bucket: bucket.clone(), + name: dobj.delete_object.object_name.clone(), + version_id, + delete_marker: dobj.delete_object.delete_marker, + ..Default::default() + }, + user_agent: "Internal: [Replication]".to_string(), + ..Default::default() + }); + return false; + } + }; + let target_arns = dobj.admitted_target_arns(); + if !dsc.replicate_any() && target_arns.is_empty() { + // Batch expiry can return an empty state after removing the source. + // It owes no remote work or local writeback. Persisted target admission + // must still be honored even when the live configuration is gone. + return true; + } + if dobj.delete_object.delete_marker && let Some(delete_marker_version_id) = dobj.delete_object.delete_marker_version_id { @@ -2355,43 +2558,6 @@ pub(crate) async fn replicate_delete_with_outcome( } } - let dsc = match parse_replicate_decision( - &bucket, - &dobj - .delete_object - .replication_state - .as_ref() - .map(|v| v.replicate_decision_str.clone()) - .unwrap_or_default(), - ) { - Ok(dsc) => dsc, - Err(err) => { - debug!( - event = EVENT_REPLICATION_DELETE_SKIPPED, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC, - bucket = %bucket, - arn = %dobj.target_arn, - error = %err, - reason = "replicate_decision_parse_failed", - "Failed to parse replicate decision" - ); - send_local_event(EventArgs { - event_name: EventName::ObjectReplicationNotTracked.to_string(), - bucket_name: bucket.clone(), - object: ObjectInfo { - bucket: bucket.clone(), - name: dobj.delete_object.object_name.clone(), - version_id, - delete_marker: dobj.delete_object.delete_marker, - ..Default::default() - }, - user_agent: "Internal: [Replication]".to_string(), - ..Default::default() - }); - return false; - } - }; let ns_lock = match storage .new_ns_lock(&bucket, format!("/[replicate]/{}", dobj.delete_object.object_name).as_str()) .await @@ -2471,7 +2637,6 @@ pub(crate) async fn replicate_delete_with_outcome( let mut join_set = JoinSet::new(); // Process each target - let target_arns = dobj.admitted_target_arns(); let expected_targets = dsc .targets_map .values() @@ -3325,6 +3490,9 @@ fn abandoned_purge_target_info(bucket: &str, dobj: &DeletedObjectReplicationInfo /// to resolve the replica on a target that mints its own version ids. struct VersionPurgeSource { etag: Option, + /// Encrypted sources: what identifies the replica when the target + /// computes its own ETag (see [`replica_identity`]). + encrypted_shape: Option, ledger: HashMap, ledger_corrupt: bool, /// Target versions other generations of the key claim, per target ARN: @@ -3370,8 +3538,13 @@ async fn version_purge_source( claimed_by_siblings.insert(target.arn.clone(), claimed); } } + let encrypted_shape = match replica_identity(&info) { + Some(ReplicaIdentity::Encrypted { shape, .. }) => Some(shape), + _ => None, + }; Some(VersionPurgeSource { etag: info.etag, + encrypted_shape, ledger, ledger_corrupt, claimed_by_siblings, @@ -3389,7 +3562,8 @@ enum PurgeReplicaResolution { /// goal is already met there. Absent, /// Refuse: no ledger entry and more than one target version carries the - /// source ETag, so any pick could destroy a live generation. + /// source ETag (or, for an encrypted source on a target that computes its + /// own ETag, its shape), so any pick could destroy a live generation. Ambiguous, /// Refuse: the persisted ledger is inconsistent. LedgerCorrupt, @@ -3417,15 +3591,19 @@ async fn resolve_purge_replica( if !force_lookup && !ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable() { return PurgeReplicaResolution::SourceVersion; } - let Some(etag) = source.etag.as_deref().filter(|etag| !etag.trim().is_empty()) else { - return PurgeReplicaResolution::SourceVersion; + let etag = source.etag.as_deref().filter(|etag| !etag.trim().is_empty()); + let identity = match (source.encrypted_shape, etag) { + (Some(shape), etag) => ReplicaIdentity::Encrypted { etag, shape }, + (None, Some(etag)) => ReplicaIdentity::Etag(etag), + (None, None) => return PurgeReplicaResolution::SourceVersion, }; - match tgt_client.replica_candidates_by_etag(&tgt_client.bucket, object, etag).await { - Ok(mut candidates) => { - if let Some(claimed) = source.claimed_by_siblings.get(&tgt_client.arn) { - candidates.retain(|candidate| !claimed.contains(candidate)); - } - match ReplicaLocation::from_matches(candidates) { + match tgt_client.replica_candidates(&tgt_client.bucket, object, &identity).await { + Ok(candidates) => { + let no_claims = HashSet::new(); + let claimed = source.claimed_by_siblings.get(&tgt_client.arn).unwrap_or(&no_claims); + // Several shape matches stay ambiguous here: a purge never picks + // one to delete. + match candidates.locate(claimed).0 { ReplicaLocation::Unique(assigned) => PurgeReplicaResolution::Resolved(assigned), ReplicaLocation::Missing => PurgeReplicaResolution::Absent, ReplicaLocation::Ambiguous { .. } => PurgeReplicaResolution::Ambiguous, @@ -3567,7 +3745,16 @@ async fn replicate_delete_to_target( } } - if !is_version_purge && dobj.delete_object.delete_marker && dobj.delete_object.delete_marker_version_id.is_some() { + // A target that mints its own version ids never holds the source marker + // id, so a version-addressed pre-check cannot succeed there; the marker + // is created by the versionless DELETE below. + let version_addressable = + !ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable(); + if version_addressable + && !is_version_purge + && dobj.delete_object.delete_marker + && dobj.delete_object.delete_marker_version_id.is_some() + { match head_object_for_worker( tgt_client.as_ref(), &tgt_client.bucket, @@ -3576,6 +3763,11 @@ async fn replicate_delete_to_target( ) .await { + // The verdict is not known yet (e.g. no PUT since restart) and the + // target rejected the source id: learn it as the PUT path does. + Err(e) if rejects_source_version_id(e.as_ref()) => { + ReplicationTargetStore::record_version_identity_capability(&tgt_client.arn, VersionIdentityCapability::MintsOwn); + } Ok(head) if head.delete_marker == Some(true) => { // Some targets expose an existing marker as a successful HEAD // with the marker bit instead of the usual 405. Treat that as @@ -4465,13 +4657,14 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo { // generation with the same bytes would be mistaken for one. // Only the ledger (a lost response of this very version) and // the AWS-style format fallback apply here. + let identity = replica_identity(&object_info); if let Some(fallback) = replica_head_fallback( &tgt_client, ReplicaSource { bucket: &bucket, object: &object, version_id: self.version_id, - etag: object_info.etag.as_deref(), + identity, ledger_version_id: ledger_version_id.as_deref(), }, None, @@ -4480,9 +4673,7 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo { .await { match fallback { - Ok(Some(located)) - if replication_etags_match(object_info.etag.as_deref(), located.head.e_tag.as_deref()) => - { + Ok(Some(located)) if located_replica_matches(identity.as_ref(), &located) => { if ssec_audit_required && !settle_ssec_passthrough_evidence(&located.head, &tgt_client, &bucket, &object, &mut rinfo) .await @@ -5130,13 +5321,14 @@ async fn resolve_replicate_all_action( } Err(e) => { let ledger_version_id = ledger_target_version_id(&object_info.user_defined, &tgt_client.arn); + let identity = replica_identity(&object_info); if let Some(fallback) = replica_head_fallback( tgt_client, ReplicaSource { bucket, object, version_id: roi.version_id, - etag: object_info.etag.as_deref(), + identity, ledger_version_id: ledger_version_id.as_deref(), }, Some(siblings), @@ -5147,20 +5339,20 @@ async fn resolve_replicate_all_action( match fallback { Ok(Some(located)) => { let oi = &located.head; - let etags_match = replication_etags_match(object_info.etag.as_deref(), oi.e_tag.as_deref()); - if require_existing_target && !etags_match { + let replica_matches = located_replica_matches(identity.as_ref(), &located); + if require_existing_target && !replica_matches { rinfo.error = Some("replica metadata target does not contain matching object data".to_string()); rinfo.duration = (OffsetDateTime::now_utc() - start_time).unsigned_abs(); return None; } - if etags_match + if replica_matches && ssec_audit_required && !settle_ssec_passthrough_evidence(oi, tgt_client, bucket, object, rinfo).await { rinfo.duration = (OffsetDateTime::now_utc() - start_time).unsigned_abs(); return None; } - if etags_match && let Some(assigned) = drifted_replica_version_id(&located, roi.version_id) { + if replica_matches && let Some(assigned) = drifted_replica_version_id(&located, roi.version_id) { // The replica lives under an id the target minted: // record it (the ledger every later mutation // resolves through) and apply metadata in place — @@ -5193,7 +5385,7 @@ async fn resolve_replicate_all_action( // records the sync without running the PUT. replication_action = ReplicationAction::None; } else { - replication_action = if etags_match { + replication_action = if replica_matches { ReplicationAction::None } else { ReplicationAction::All @@ -5826,6 +6018,57 @@ mod tests { use super::super::replication_filemeta_boundary::ReplicateTargetDecision; use super::super::replication_object_decision_boundary::ReplicationMultipartPlanError; + #[test] + fn verify_single_part_replica_exempts_managed_sse_sources() { + // A new encrypted write stores a keyed fingerprint; a target that + // computes its own ETag answers with the MD5 of the plaintext it + // received. Both are 32 hex characters, so without the exemption the + // audit would fail every SSE-S3 / SSE-KMS replica (rustfs/backlog). + let fingerprint = "00112233445566778899aabbccddeeff"; + let replica_md5 = "ffeeddccbbaa99887766554433221100"; + let response = RemotePutObjectResponse { + version_id: None, + etag: Some(replica_md5.to_string()), + }; + for sse in ["AES256", "aws:kms"] { + let object_info = ObjectInfo { + bucket: "source".to_string(), + name: "encrypted".to_string(), + etag: Some(fingerprint.to_string()), + user_defined: HashMap::from([("x-amz-server-side-encryption".to_string(), sse.to_string())]).into(), + ..Default::default() + }; + verify_single_part_replica(&object_info, &response, false) + .unwrap_or_else(|err| panic!("{sse}: a managed-SSE source must not be audited against the replica MD5: {err}")); + } + } + + #[test] + fn verify_single_part_replica_still_fails_plaintext_mismatch() { + if !replica_etag_verification_enabled() { + return; + } + let object_info = ObjectInfo { + bucket: "source".to_string(), + name: "plain".to_string(), + etag: Some("00112233445566778899aabbccddeeff".to_string()), + ..Default::default() + }; + let mismatch = RemotePutObjectResponse { + version_id: None, + etag: Some("ffeeddccbbaa99887766554433221100".to_string()), + }; + let err = verify_single_part_replica(&object_info, &mismatch, false) + .expect_err("a plaintext source whose replica MD5 differs must fail the replication"); + assert_eq!(err.to_string(), REPLICA_ETAG_MISMATCH_ERROR); + + let matching = RemotePutObjectResponse { + version_id: None, + etag: object_info.etag.clone(), + }; + verify_single_part_replica(&object_info, &matching, false).expect("matching plaintext MD5s pass the audit"); + } + #[test] fn multipart_read_plan_preserves_legacy_plain_part_ranges() { const MIB: usize = 1024 * 1024; @@ -6552,13 +6795,361 @@ mod tests { let head_result = head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; - let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; assert!(err.is_none(), "{err:?}"); assert_eq!((size, st.replicated_count, st.failed_count), (4, 1, 0)); server.join().expect("test HTTP server should finish"); } + /// What a target that computes its own ETag (AWS S3) reports for the + /// replica of an encrypted source: the MD5 of the plaintext it received, + /// never the source's encrypted-ETag fingerprint ([`DRIFTED_ETAG`] here). + const COMPUTED_REPLICA_ETAG: &str = "0f0e0d0c0b0a09080706050403020100"; + /// The replica's write time on the scripted targets below. + const REPLICA_WRITTEN: &str = "2026-09-06T10:00:00.000Z"; + const REPLICA_WRITTEN_HTTP: &str = "Sun, 06 Sep 2026 10:00:00 GMT"; + + /// An SSE-S3 source version: 4 plaintext bytes, stored as ciphertext, its + /// ETag a keyed fingerprint, written at `written` (unix seconds). + fn encrypted_roi_and_object(written: i64) -> (ReplicateObjectInfo, ObjectInfo) { + let (roi, mut object_info) = drifted_roi_and_object(); + object_info.size = 36; + object_info.actual_size = 4; + object_info.mod_time = Some(OffsetDateTime::from_unix_timestamp(written).expect("valid timestamp")); + object_info.user_defined = HashMap::from([("x-amz-server-side-encryption".to_string(), "AES256".to_string())]).into(); + (roi, object_info) + } + + fn replica_written_secs() -> i64 { + OffsetDateTime::parse(REPLICA_WRITTEN, &Rfc3339) + .expect("valid replica time") + .unix_timestamp() + } + + fn computed_etag_versions_response(versions: &[(&str, i64)]) -> String { + let versions = versions + .iter() + .enumerate() + .map(|(index, (version_id, size))| { + format!( + "object{version_id}{}{REPLICA_WRITTEN}"{COMPUTED_REPLICA_ETAG}"{size}STANDARD", + index == 0 + ) + }) + .collect::(); + xml_response( + "200 OK", + format!( + "target-bucketobject1000false{versions}" + ), + ) + } + + fn computed_etag_head_response(version_id: &str, size: i64) -> String { + format!( + "HTTP/1.1 200 OK\r\nETag: \"{COMPUTED_REPLICA_ETAG}\"\r\nContent-Length: {size}\r\nx-amz-version-id: {version_id}\r\nLast-Modified: {REPLICA_WRITTEN_HTTP}\r\nConnection: close\r\n\r\n" + ) + } + + /// A Wasabi/AWS-shaped target that computes its own ETags: 404 on the + /// source uuid, the listed versions, and a HEAD for any listed id. + fn spawn_computed_etag_target( + requests: usize, + versions: &'static [(&'static str, i64)], + ) -> (String, std::thread::JoinHandle>) { + spawn_scripted_target_server(requests, move |line| { + if line.starts_with("GET ") && line.contains("versions") { + return computed_etag_versions_response(versions); + } + if line.starts_with("HEAD ") + && let Some((version_id, size)) = versions + .iter() + .find(|(version_id, _)| line.contains(&format!("versionId={version_id}"))) + { + return computed_etag_head_response(version_id, *size); + } + if line.starts_with("HEAD ") { + return empty_response("404 Not Found"); + } + empty_response("500 Unexpected") + }) + } + + async fn resolve_heal_action( + target: &Arc, + roi: &ReplicateObjectInfo, + object_info: ObjectInfo, + rinfo: &mut ReplicatedTargetInfo, + ) -> Option<(ReplicationAction, ObjectInfo)> { + resolve_replicate_all_action( + ReplicateAllActionContext { + siblings: &NoSiblingLedger, + roi, + tgt_client: target, + bucket: &roi.bucket, + object: &roi.name, + start_time: OffsetDateTime::now_utc(), + ssec_audit_required: false, + }, + object_info, + rinfo, + ) + .await + } + + #[tokio::test] + async fn heal_redrive_locates_encrypted_replica_by_shape_on_target_that_computes_own_etags() { + static VERSIONS: &[(&str, i64)] = &[(DRIFTED_ASSIGNED_VERSION_ID, 4)]; + let (endpoint, server) = spawn_computed_etag_target(3, VERSIONS); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut rinfo = replicate_all_target_info(&roi, &target); + + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + + assert!( + matches!(action, Some((ReplicationAction::None, _))), + "an encrypted replica located by its shape must not be re-driven: {action:?}" + ); + assert!(rinfo.error.is_none(), "{:?}", rinfo.error); + assert_eq!(rinfo.target_version_id.as_deref(), Some(DRIFTED_ASSIGNED_VERSION_ID)); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 3, "HEAD by source id, ListObjectVersions, HEAD by assigned id: {seen:?}"); + } + + #[tokio::test] + async fn heal_redrive_does_not_adopt_an_encrypted_replica_of_another_size_or_an_older_write() { + // Size differs from the 4 plaintext bytes. + static OTHER_SIZE: &[(&str, i64)] = &[(DRIFTED_ASSIGNED_VERSION_ID, 36)]; + let (endpoint, server) = spawn_computed_etag_target(2, OTHER_SIZE); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut rinfo = replicate_all_target_info(&roi, &target); + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + assert!(matches!(action, Some((ReplicationAction::All, _))), "{action:?}"); + server.join().expect("test HTTP server should finish"); + + // Same size, but written before the source version existed. + static SAME_SIZE: &[(&str, i64)] = &[(DRIFTED_ASSIGNED_VERSION_ID, 4)]; + let (endpoint, server) = spawn_computed_etag_target(2, SAME_SIZE); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() + 1); + let mut rinfo = replicate_all_target_info(&roi, &target); + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + assert!(matches!(action, Some((ReplicationAction::All, _))), "{action:?}"); + server.join().expect("test HTTP server should finish"); + } + + #[tokio::test] + async fn heal_redrive_does_not_pick_among_several_encrypted_shape_matches() { + static VERSIONS: &[(&str, i64)] = &[(DRIFTED_ASSIGNED_VERSION_ID, 4), ("another-minted-id", 4)]; + let (endpoint, server) = spawn_computed_etag_target(2, VERSIONS); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut rinfo = replicate_all_target_info(&roi, &target); + + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + + assert!( + matches!(action, Some((ReplicationAction::All, _))), + "two same-shape versions are not evidence of either: {action:?}" + ); + assert!(rinfo.target_version_id.is_none(), "{:?}", rinfo.target_version_id); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 2, "HEAD by source id, ListObjectVersions, no HEAD of a guess: {seen:?}"); + } + + #[tokio::test] + async fn heal_redrive_keeps_exact_etag_identity_for_plaintext_sources() { + // A same-size replica whose ETag differs is not this plaintext + // version's replica: the shape criterion is for encrypted sources only. + static VERSIONS: &[(&str, i64)] = &[(DRIFTED_ASSIGNED_VERSION_ID, 4)]; + let (endpoint, server) = spawn_computed_etag_target(2, VERSIONS); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, mut object_info) = drifted_roi_and_object(); + object_info.mod_time = Some(OffsetDateTime::from_unix_timestamp(replica_written_secs() - 1).expect("valid timestamp")); + let mut rinfo = replicate_all_target_info(&roi, &target); + + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + + assert!(matches!(action, Some((ReplicationAction::All, _))), "{action:?}"); + server.join().expect("test HTTP server should finish"); + } + + #[tokio::test] + async fn heal_redrive_never_adopts_an_unversioned_null_version_by_shape() { + // An unversioned target's `null` version holds whatever generation + // was written last; a same-size newer generation must not be claimed. + static VERSIONS: &[(&str, i64)] = &[("null", 4)]; + let (endpoint, server) = spawn_computed_etag_target(2, VERSIONS); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut rinfo = replicate_all_target_info(&roi, &target); + + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + + assert!(matches!(action, Some((ReplicationAction::All, _))), "{action:?}"); + assert!(rinfo.target_version_id.is_none(), "{:?}", rinfo.target_version_id); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 2, "HEAD by source id, ListObjectVersions, nothing adopted: {seen:?}"); + } + + #[tokio::test] + async fn encrypted_lookup_keeps_the_current_version_fallback_when_listing_is_denied() { + // A bodiless 403 on the version-addressed HEAD may be a permission + // denial; the same credentials then refuse ListObjectVersions. The + // object must still be re-driven, as before content identity existed. + let (endpoint, server) = spawn_scripted_target_server(3, |line| { + let listing = line.starts_with("GET ") && line.contains("versions"); + let version_addressed_head = line.starts_with("HEAD ") && line.contains("versionId="); + if listing || version_addressed_head { + empty_response("403 Forbidden") + } else if line.starts_with("HEAD ") { + computed_etag_head_response(DRIFTED_ASSIGNED_VERSION_ID, 4) + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut rinfo = replicate_all_target_info(&roi, &target); + + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + + assert!( + matches!(action, Some((ReplicationAction::All, _))), + "a denied listing must fall back to the re-PUT, not stall: {action:?}" + ); + assert!(rinfo.error.is_none(), "{:?}", rinfo.error); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!( + seen.len(), + 3, + "HEAD by source id, ListObjectVersions, HEAD of the current version: {seen:?}" + ); + assert!(seen[2].starts_with("HEAD ") && !seen[2].contains("versionId="), "{}", seen[2]); + } + + #[tokio::test] + async fn encrypted_ledger_replica_converges_on_an_aws_style_target() { + // AWS rejects the RustFS uuid as malformed (400). The ledger names the + // replica; its computed ETag cannot confirm it, its shape does. + let (endpoint, server) = spawn_scripted_target_server(2, |line| { + if line.starts_with("HEAD ") && line.contains(&format!("versionId={DRIFTED_ASSIGNED_VERSION_ID}")) { + computed_etag_head_response(DRIFTED_ASSIGNED_VERSION_ID, 4) + } else if line.starts_with("HEAD ") { + empty_response("400 Bad Request") + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + let (roi, mut object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut user_defined = (*object_info.user_defined).clone(); + insert_str( + &mut user_defined, + &format!("{SUFFIX_REPLICATION_TARGET_VERSION_ARN_PREFIX}{}", target.arn), + DRIFTED_ASSIGNED_VERSION_ID.to_string(), + ); + object_info.user_defined = Arc::new(user_defined); + let mut rinfo = replicate_all_target_info(&roi, &target); + + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + + assert!(matches!(action, Some((ReplicationAction::None, _))), "{action:?}"); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 2, "HEAD by source id, HEAD by the ledger id: {seen:?}"); + assert!(seen[1].contains(DRIFTED_ASSIGNED_VERSION_ID), "{}", seen[1]); + } + + #[tokio::test] + async fn resync_verification_counts_encrypted_replica_by_shape() { + static VERSIONS: &[(&str, i64)] = &[(DRIFTED_ASSIGNED_VERSION_ID, 4)]; + let (endpoint, server) = spawn_computed_etag_target(3, VERSIONS); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut st = TargetReplicationResyncStatus::default(); + + let head_result = + head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; + let (_, err) = verify_resync_head_result(head_result, &roi, Some(&object_info), &mut st, &target).await; + + assert!(err.is_none(), "{err:?}"); + assert_eq!((st.replicated_count, st.failed_count), (1, 0)); + server.join().expect("test HTTP server should finish"); + } + + fn encrypted_purge_source() -> Arc { + let (_, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let Some(ReplicaIdentity::Encrypted { shape, .. }) = replica_identity(&object_info) else { + panic!("an SSE-S3 source must be identified by its shape"); + }; + Arc::new(VersionPurgeSource { + etag: object_info.etag, + encrypted_shape: Some(shape), + ledger: HashMap::new(), + ledger_corrupt: false, + claimed_by_siblings: HashMap::new(), + }) + } + + #[tokio::test] + async fn encrypted_version_purge_without_ledger_deletes_the_unique_shape_match() { + let (endpoint, server) = spawn_scripted_target_server(2, |line| { + if line.starts_with("GET ") && line.contains("versions") { + computed_etag_versions_response(&[(DRIFTED_ASSIGNED_VERSION_ID, 4)]) + } else if line.starts_with("DELETE ") && line.contains(&format!("versionId={DRIFTED_ASSIGNED_VERSION_ID}")) { + empty_response("204 No Content") + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + register_test_target(&target).await; + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let dobj = version_purge_dobj(&target.arn); + + let rinfo = replicate_delete_to_target(&dobj, target.clone(), Some(encrypted_purge_source())).await; + + assert_eq!(rinfo.version_purge_status, VersionPurgeStatusType::Complete, "{:?}", rinfo.error); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 2, "ListObjectVersions, then DELETE by the located id: {seen:?}"); + assert!( + seen[1].starts_with("DELETE ") && seen[1].contains(DRIFTED_ASSIGNED_VERSION_ID), + "{}", + seen[1] + ); + } + + #[tokio::test] + async fn encrypted_version_purge_refuses_several_shape_matches() { + let (endpoint, server) = spawn_scripted_target_server(1, |line| { + if line.starts_with("GET ") && line.contains("versions") { + computed_etag_versions_response(&[(DRIFTED_ASSIGNED_VERSION_ID, 4), ("another-minted-id", 4)]) + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + register_test_target(&target).await; + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let dobj = version_purge_dobj(&target.arn); + + let rinfo = replicate_delete_to_target(&dobj, target.clone(), Some(encrypted_purge_source())).await; + + assert_eq!(rinfo.version_purge_status, VersionPurgeStatusType::Failed); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 1, "no DELETE of a guessed replica: {seen:?}"); + } + fn delete_marker_roi() -> ReplicateObjectInfo { ReplicateObjectInfo { bucket: "source".to_string(), @@ -6675,6 +7266,93 @@ mod tests { assert!(seen[0].starts_with("HEAD "), "{seen:?}"); } + /// AWS S3 answers a HEAD for a version id it did not mint with a bodiless + /// 400. On a target with no verdict yet, that teaches `MintsOwn` and the + /// marker is still created with one versionless DELETE. + #[tokio::test] + async fn live_delete_marker_worker_learns_mints_own_from_bodiless_400() { + let (endpoint, server) = spawn_scripted_target_server(2, |line| { + if line.starts_with("HEAD ") { + empty_response("400 Bad Request") + } else if line.starts_with("DELETE ") { + empty_response("204 No Content") + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + register_test_target(&target).await; + let dobj = marker_creation_dobj(&target.arn); + + let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await; + + assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}"); + assert_eq!( + ReplicationTargetStore::version_identity_capability(&target.arn), + VersionIdentityCapability::MintsOwn + ); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 2, "{seen:?}"); + assert!(seen[0].starts_with("HEAD "), "{seen:?}"); + assert!(seen[1].starts_with("DELETE "), "{seen:?}"); + assert!(!seen[1].contains("versionId="), "marker creation must remain versionless: {seen:?}"); + } + + /// A bodiless 403 on the version-addressed HEAD may be a genuine + /// GetObjectVersion denial on an id-adopting target. It must fail closed: + /// no DELETE, and no `MintsOwn` verdict cached for the ARN. + #[tokio::test] + async fn live_delete_marker_worker_keeps_bodiless_403_failure_closed() { + let (endpoint, server) = spawn_scripted_target_server(1, |line| { + if line.starts_with("HEAD ") { + empty_response("403 Forbidden") + } else if line.starts_with("DELETE ") { + empty_response("204 No Content") + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + register_test_target(&target).await; + let dobj = marker_creation_dobj(&target.arn); + + let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await; + + assert_eq!(rinfo.replication_status, ReplicationStatusType::Failed, "{rinfo:?}"); + assert_ne!( + ReplicationTargetStore::version_identity_capability(&target.arn), + VersionIdentityCapability::MintsOwn + ); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 1, "a denied HEAD must not trigger DELETE: {seen:?}"); + assert!(seen[0].starts_with("HEAD "), "{seen:?}"); + } + + /// A target already known to mint its own version ids is never asked for + /// the source marker id: the marker is created by the versionless DELETE. + #[tokio::test] + async fn live_delete_marker_worker_skips_version_head_on_mints_own_target() { + let (endpoint, server) = spawn_scripted_target_server(1, |line| { + if line.starts_with("DELETE ") { + empty_response("204 No Content") + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + register_test_target(&target).await; + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let dobj = marker_creation_dobj(&target.arn); + + let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await; + + assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}"); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 1, "no version-addressed HEAD on a MintsOwn target: {seen:?}"); + assert!(seen[0].starts_with("DELETE "), "{seen:?}"); + assert!(!seen[0].contains("versionId="), "marker creation must remain versionless: {seen:?}"); + } + /// A target answers `HEAD ?versionId=` with a bodiless /// 405, so the SDK reports no error code. That is the marker having /// propagated, not a failure (rustfs/backlog#2479). @@ -6696,7 +7374,7 @@ mod tests { None, "the fixture must reproduce the codeless 405 the SDK yields for a bodiless HEAD" ); - let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; assert!(err.is_none(), "{err:?}"); assert_eq!((size, st.replicated_count, st.failed_count), (0, 1, 0)); @@ -6723,7 +7401,7 @@ mod tests { let head_result = head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; - let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; assert!(err.is_some(), "a 405 must not count as a completed purge ({status:?})"); assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1), "{status:?}"); @@ -6741,7 +7419,7 @@ mod tests { let head_result = head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; - let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; assert!(err.is_none(), "{err:?}"); assert_eq!((size, st.replicated_count, st.failed_count), (0, 1, 0)); @@ -6758,13 +7436,83 @@ mod tests { let head_result = head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; - let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; assert!(err.is_some(), "a 503 must not count as a propagated delete marker"); assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1)); server.join().expect("test HTTP server should finish"); } + /// AWS answers `HEAD ?versionId=` with a bodiless 400. + /// The marker is verified through the latest version instead: a + /// versionless 404 with `x-amz-delete-marker: true` proves it propagated. + #[tokio::test] + async fn resync_verification_accepts_latest_delete_marker_when_target_rejects_source_id() { + let (endpoint, server) = spawn_scripted_target_server(2, |line| { + if line.contains("versionId=") { + empty_response("400 Bad Request") + } else { + "HTTP/1.1 404 Not Found\r\nX-Amz-Delete-Marker: true\r\nX-Amz-Version-Id: target-marker\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" + .to_string() + } + }); + let target = test_target_client(endpoint); + let roi = delete_marker_roi(); + let mut st = TargetReplicationResyncStatus::default(); + + let head_result = + head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; + + assert!(err.is_none(), "{err:?}"); + assert_eq!((size, st.replicated_count, st.failed_count), (0, 1, 0)); + let seen = server.join().expect("test HTTP server should finish"); + assert!(!seen[1].contains("versionId="), "the re-check must address the latest version: {seen:?}"); + } + + /// A live latest version on a target that rejects the source id means the + /// marker has not propagated: the verification stays failed. + #[tokio::test] + async fn resync_verification_fails_delete_marker_when_latest_target_version_is_live() { + let (endpoint, server) = spawn_scripted_target_server(2, |line| { + if line.contains("versionId=") { + empty_response("400 Bad Request") + } else { + empty_response("200 OK") + } + }); + let target = test_target_client(endpoint); + let roi = delete_marker_roi(); + let mut st = TargetReplicationResyncStatus::default(); + + let head_result = + head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; + + assert!(err.is_some(), "a live latest version must not verify the marker"); + assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1)); + server.join().expect("test HTTP server should finish"); + } + + /// A bodiless 403 may be a genuine permission denial: it is not evidence + /// that the target mints its own ids, so no latest-version re-check runs. + #[tokio::test] + async fn resync_verification_keeps_delete_marker_failed_on_bodiless_403() { + let (endpoint, server) = spawn_scripted_target_server(1, |_| empty_response("403 Forbidden")); + let target = test_target_client(endpoint); + let roi = delete_marker_roi(); + let mut st = TargetReplicationResyncStatus::default(); + + let head_result = + head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; + + assert!(err.is_some(), "a 403 must not count as a propagated delete marker"); + assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1)); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 1, "{seen:?}"); + } + /// Serves exactly `requests` connections, answering each from /// `respond(request_line)`, and returns the request lines it saw. Reads /// the whole request (headers plus `Content-Length` body, honoring @@ -6874,6 +7622,7 @@ mod tests { fn purge_source(ledger: &[(&str, &str)]) -> Arc { Arc::new(VersionPurgeSource { etag: Some(DRIFTED_ETAG.to_string()), + encrypted_shape: None, ledger: ledger .iter() .map(|(arn, version_id)| (arn.to_string(), version_id.to_string())) @@ -7078,6 +7827,7 @@ mod tests { let dobj = version_purge_dobj(&target.arn); let source = Arc::new(VersionPurgeSource { etag: Some(DRIFTED_ETAG.to_string()), + encrypted_shape: None, ledger: HashMap::new(), ledger_corrupt: false, claimed_by_siblings: HashMap::from([(target.arn.clone(), HashSet::from([DRIFTED_ASSIGNED_VERSION_ID.to_string()]))]), @@ -7165,6 +7915,7 @@ mod tests { let dobj = version_purge_dobj(&target.arn); let source = Arc::new(VersionPurgeSource { etag: Some(DRIFTED_ETAG.to_string()), + encrypted_shape: None, ledger: HashMap::new(), ledger_corrupt: true, claimed_by_siblings: HashMap::new(), diff --git a/crates/ecstore/src/bucket/replication/replication_target_boundary.rs b/crates/ecstore/src/bucket/replication/replication_target_boundary.rs index e71ae9f19..2d7fba2dd 100644 --- a/crates/ecstore/src/bucket/replication/replication_target_boundary.rs +++ b/crates/ecstore/src/bucket/replication/replication_target_boundary.rs @@ -37,8 +37,9 @@ use time::OffsetDateTime; use time::format_description::well_known::Rfc3339; pub(crate) use crate::bucket::bucket_target_sys::{ - AdvancedPutOptions, HeadObjectSdkError, PutObjectOptions, PutObjectPartOptions, RemotePutObjectResponse, RemoveObjectOptions, - ReplicaLocation, S3ClientError, TargetClient, resolve_read_api_version_id, + AdvancedPutOptions, EncryptedReplicaShape, HeadObjectSdkError, PutObjectOptions, PutObjectPartOptions, + RemotePutObjectResponse, RemoveObjectOptions, ReplicaCandidates, ReplicaEvidence, ReplicaIdentity, ReplicaLocation, + S3ClientError, TargetClient, resolve_read_api_version_id, }; #[cfg(test)] pub(crate) use crate::bucket::target::BucketTarget; @@ -157,6 +158,19 @@ pub(crate) fn replication_object_is_ssec_encrypted(user_defined: &HashMap) -> bool { + matches!( + classify_replication_source_encryption(user_defined), + ReplicationSourceEncryption::Plaintext + ) +} + /// HeadObjectOutput adapter over the pure SSE-C passthrough evidence /// judgment owned by `rustfs-replication`: extract the echoed /// customer-algorithm header and let the crate-owned policy decide. @@ -578,6 +592,22 @@ mod tests { use time::Duration; use uuid::Uuid; + #[test] + fn replication_source_etag_is_comparable_only_for_plaintext_sources() { + assert!(replication_source_etag_is_comparable(&HashMap::new())); + for (key, value) in [ + (metadata_keys::SERVER_SIDE_ENCRYPTION, "AES256"), + (metadata_keys::SERVER_SIDE_ENCRYPTION, "aws:kms"), + (SSEC_ALGORITHM_HEADER, "AES256"), + ] { + let metadata = HashMap::from([(key.to_string(), value.to_string())]); + assert!( + !replication_source_etag_is_comparable(&metadata), + "{key}={value} must not expose its stored ETag to the replica audit" + ); + } + } + /// backlog#1735 A3b SSE key-case finding. Every RustFS writer persists the /// SSE intent as lowercase `x-amz-server-side-encryption` /// (`encryption_material_to_metadata`; header-derived keys come from a diff --git a/crates/ecstore/src/bucket/utils.rs b/crates/ecstore/src/bucket/utils.rs index f659468bb..850e60840 100644 --- a/crates/ecstore/src/bucket/utils.rs +++ b/crates/ecstore/src/bucket/utils.rs @@ -15,6 +15,7 @@ use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET}; use crate::error::{Error, Result, StorageError}; use regex::Regex; +use rustfs_utils::WindowsFileNameError; use rustfs_utils::path::SLASH_SEPARATOR; use s3s::xml; use tracing::instrument; @@ -178,8 +179,7 @@ pub fn is_valid_object_name(object: &str) -> bool { dead_code, reason = "live on Windows: callers sit inside the #[cfg(target_os = \"windows\")] block in check_object_name_for_length_and_slash (backlog#1823)" )] -pub const WINDOWS_RESERVED_CHARACTERS_REASON: &str = - "object key contains characters unsupported on Windows hosts (one of ':', '*', '?', '\"', '|', '<', '>')"; +pub const WINDOWS_RESERVED_CHARACTERS_REASON: &str = "object key contains characters unsupported on Windows hosts (one of ':', '*', '?', '\"', '|', '<', '>', or a control character)"; /// Client-facing reason for path segments Windows can store but not address /// afterwards (issue #3449): trailing dot/space or reserved DOS device names. @@ -187,37 +187,46 @@ pub const WINDOWS_RESERVED_CHARACTERS_REASON: &str = dead_code, reason = "live on Windows: callers sit inside the #[cfg(target_os = \"windows\")] block in check_object_name_for_length_and_slash (backlog#1823)" )] -pub const WINDOWS_RESERVED_SEGMENT_REASON: &str = "object key contains a path segment unsupported on Windows hosts (trailing dot or space, or a reserved device name such as NUL/CON/COM1)"; +pub const WINDOWS_RESERVED_SEGMENT_REASON: &str = "object key contains a path segment unsupported on Windows hosts (trailing dot or space, or a reserved device name such as NUL/CON/COM1/CONIN$)"; -/// Reserved DOS device names that shadow regular files on Windows, even when -/// an extension is appended (e.g. `NUL.txt` resolves to the `NUL` device). +/// Client-facing reason for keys containing `\`: Windows hosts treat it as a +/// path separator, so `a\b` would land on the same file as `a/b`. +pub const WINDOWS_BACKSLASH_REASON: &str = + "object key contains '\\', which Windows hosts treat as a path separator (it would alias the same key spelled with '/')"; + +/// Returns the client-facing reason when `object` cannot round-trip through +/// the Win32 path layer, or `None` when it can. +/// +/// Each segment is checked with [`rustfs_utils::check_windows_file_name`]. +/// `\` splits segments here, like on a Windows disk, so this check never +/// rejects it: an existing `a\b` key must stay readable and deletable. New +/// writes reject `\` separately in [`check_new_object_args`]. #[allow( dead_code, reason = "live on Windows: callers sit inside the #[cfg(target_os = \"windows\")] block in check_object_name_for_length_and_slash (backlog#1823)" )] -const WINDOWS_RESERVED_NAMES: &[&str] = &[ - "CON", "PRN", "AUX", "NUL", "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9", "LPT1", "LPT2", "LPT3", - "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9", -]; +pub fn windows_object_key_violation(object: &str) -> Option<&'static str> { + // A NUL byte is rejected on every platform as ObjectNameInvalid by + // is_valid_object_prefix; keep that answer on Windows too. + if object.contains('\0') { + return None; + } -/// Returns true when `object` contains a path segment that NTFS can store but -/// the Win32 API cannot address afterwards (issue #3449): segments ending in a -/// dot or a space, and reserved DOS device names — bare or with an extension -/// (`NUL.txt`), matching classic Win32 path resolution semantics. -#[allow( - dead_code, - reason = "live on Windows: callers sit inside the #[cfg(target_os = \"windows\")] block in check_object_name_for_length_and_slash (backlog#1823)" -)] -pub fn object_name_has_windows_incompatible_segment(object: &str) -> bool { - object.split(['/', '\\']).any(|segment| { - if segment.ends_with('.') || segment.ends_with(' ') { - return true; + let mut segment_violation = false; + for segment in object.split(['/', '\\']).filter(|segment| !segment.is_empty()) { + match rustfs_utils::check_windows_file_name(segment) { + Err(WindowsFileNameError::ReservedCharacter | WindowsFileNameError::ControlCharacter) => { + return Some(WINDOWS_RESERVED_CHARACTERS_REASON); + } + Err(WindowsFileNameError::TrailingDotOrSpace | WindowsFileNameError::ReservedDeviceName) => { + segment_violation = true; + } + // Segment length is enforced in bytes by object_key_segments_fit_on_disk, + // which is stricter than the UTF-16 limit and answers ObjectNameInvalid. + Ok(()) | Err(WindowsFileNameError::TooLong | WindowsFileNameError::Empty) => {} } - // Device names are matched on the part before the first dot, with - // trailing spaces ignored (so `NUL .txt` is also reserved). - let base = segment.split('.').next().unwrap_or(segment).trim_end_matches(' '); - WINDOWS_RESERVED_NAMES.iter().any(|name| base.eq_ignore_ascii_case(name)) - }) + } + segment_violation.then_some(WINDOWS_RESERVED_SEGMENT_REASON) } pub fn check_object_name_for_length_and_slash(bucket: &str, object: &str) -> Result<()> { @@ -229,33 +238,33 @@ pub fn check_object_name_for_length_and_slash(bucket: &str, object: &str) -> Res return Err(StorageError::ObjectNamePrefixAsSlash(bucket.to_owned(), object.to_owned())); } + // Reject names that NTFS would refuse, or create but the Win32 path layer + // cannot read back (os error 3), e.g. `a:b`, `baddir.` or `NUL.txt`. #[cfg(target_os = "windows")] - { - if object.contains(':') - || object.contains('*') - || object.contains('?') - || object.contains('"') - || object.contains('|') - || object.contains('<') - || object.contains('>') - // || object.contains('\\') - { - return Err(StorageError::InvalidArgument( - bucket.to_owned(), - object.to_owned(), - WINDOWS_RESERVED_CHARACTERS_REASON.to_owned(), - )); - } + if let Some(reason) = windows_object_key_violation(object) { + return Err(StorageError::InvalidArgument(bucket.to_owned(), object.to_owned(), reason.to_owned())); + } - // Reject names that NTFS would happily create but the Win32 path - // layer cannot read back (os error 3), e.g. `baddir.` or `NUL.txt`. - if object_name_has_windows_incompatible_segment(object) { - return Err(StorageError::InvalidArgument( - bucket.to_owned(), - object.to_owned(), - WINDOWS_RESERVED_SEGMENT_REASON.to_owned(), - )); - } + Ok(()) +} + +/// Admission check for every request that introduces `object` into the +/// namespace: PutObject, a CopyObject destination, and CreateMultipartUpload. +/// Sharing one entry keeps their rejections (and S3 error codes) identical. +/// +/// `data_movement` writes (decommission, rebalance) relocate keys that already +/// exist, so they skip the new-key-only rules and stay able to move old data. +/// Internal meta buckets are exempt too: their keys are produced by RustFS +/// itself and resolve to the same path either way. +pub fn check_new_object_args(bucket: &str, object: &str, data_movement: bool) -> Result<()> { + check_put_object_args(bucket, object)?; + + if cfg!(target_os = "windows") && !data_movement && !is_meta_bucketname(bucket) && object.contains('\\') { + return Err(StorageError::InvalidArgument( + bucket.to_owned(), + object.to_owned(), + WINDOWS_BACKSLASH_REASON.to_owned(), + )); } Ok(()) @@ -316,10 +325,6 @@ pub fn check_bucket_and_object_names(bucket: &str, object: &str) -> Result<()> { return Err(StorageError::ObjectNameInvalid(bucket.to_string(), object.to_string())); } - // if cfg!(target_os = "windows") && object.contains('\\') { - // return Err(StorageError::ObjectNameInvalid(bucket.to_string(), object.to_string())); - // } - Ok(()) } @@ -586,103 +591,193 @@ mod tests { } #[test] - fn test_object_name_has_windows_incompatible_segment() { + fn test_windows_object_key_violation_segments() { + let segment = Some(WINDOWS_RESERVED_SEGMENT_REASON); + // Segments ending in a dot are writable on NTFS but unreadable via Win32 paths. - assert!(object_name_has_windows_incompatible_segment("baddir.")); - assert!(object_name_has_windows_incompatible_segment("dir1/dir2/baddir.")); - assert!(object_name_has_windows_incompatible_segment("dir1/baddir./file.txt")); - assert!(object_name_has_windows_incompatible_segment("file.txt.")); + assert_eq!(windows_object_key_violation("baddir."), segment); + assert_eq!(windows_object_key_violation("dir1/dir2/baddir."), segment); + assert_eq!(windows_object_key_violation("dir1/baddir./file.txt"), segment); + assert_eq!(windows_object_key_violation("file.txt."), segment); // Segments ending in a space. - assert!(object_name_has_windows_incompatible_segment("file.txt ")); - assert!(object_name_has_windows_incompatible_segment("dir /file.txt")); + assert_eq!(windows_object_key_violation("file.txt "), segment); + assert_eq!(windows_object_key_violation("dir /file.txt"), segment); // Reserved DOS device names, bare and case-insensitive. - assert!(object_name_has_windows_incompatible_segment("NUL")); - assert!(object_name_has_windows_incompatible_segment("nul")); - assert!(object_name_has_windows_incompatible_segment("CON")); - assert!(object_name_has_windows_incompatible_segment("prn")); - assert!(object_name_has_windows_incompatible_segment("Aux")); - assert!(object_name_has_windows_incompatible_segment("COM1")); - assert!(object_name_has_windows_incompatible_segment("com9")); - assert!(object_name_has_windows_incompatible_segment("LPT1")); - assert!(object_name_has_windows_incompatible_segment("lpt9")); - assert!(object_name_has_windows_incompatible_segment("dir/NUL/file.txt")); + for object in [ + "NUL", + "nul", + "CON", + "prn", + "Aux", + "COM1", + "com9", + "LPT1", + "lpt9", + "dir/NUL/file.txt", + // Superscript port numbers and console aliases are devices too. + "COM\u{b9}", + "dir/lpt\u{b3}", + "CONIN$", + "conout$", + ] { + assert_eq!(windows_object_key_violation(object), segment, "{object:?}"); + } // Reserved device names with an extension still resolve to the device. - assert!(object_name_has_windows_incompatible_segment("NUL.txt")); - assert!(object_name_has_windows_incompatible_segment("dir/aux.log")); - assert!(object_name_has_windows_incompatible_segment("con.tar.gz")); - assert!(object_name_has_windows_incompatible_segment("com1.dat")); - assert!(object_name_has_windows_incompatible_segment("NUL .txt")); + for object in [ + "NUL.txt", + "dir/aux.log", + "con.tar.gz", + "com1.dat", + "NUL .txt", + "COM\u{b2}.log", + ] { + assert_eq!(windows_object_key_violation(object), segment, "{object:?}"); + } // Backslash-separated segments are checked as well. - assert!(object_name_has_windows_incompatible_segment("dir\\baddir.\\file.txt")); - assert!(object_name_has_windows_incompatible_segment("dir\\nul")); + assert_eq!(windows_object_key_violation("dir\\baddir.\\file.txt"), segment); + assert_eq!(windows_object_key_violation("dir\\nul"), segment); // Valid names must not be flagged. - assert!(!object_name_has_windows_incompatible_segment("file.txt")); - assert!(!object_name_has_windows_incompatible_segment("dir.name/file")); - assert!(!object_name_has_windows_incompatible_segment("path/to/file.tar.gz")); - assert!(!object_name_has_windows_incompatible_segment("nullable")); - assert!(!object_name_has_windows_incompatible_segment("CONSOLE")); - assert!(!object_name_has_windows_incompatible_segment("com10")); - assert!(!object_name_has_windows_incompatible_segment("com0")); - assert!(!object_name_has_windows_incompatible_segment("lpt")); - assert!(!object_name_has_windows_incompatible_segment("aux-data/file")); - assert!(!object_name_has_windows_incompatible_segment("object with spaces inside")); - assert!(!object_name_has_windows_incompatible_segment("")); + for object in [ + "file.txt", + "dir.name/file", + "path/to/file.tar.gz", + "nullable", + "CONSOLE", + "com10", + "com0", + "lpt", + "CONIN", + "aux-data/file", + "object with spaces inside", + "dir/", + "", + ] { + assert_eq!(windows_object_key_violation(object), None, "{object:?}"); + } } #[test] - fn test_check_object_name_windows_incompatible_segments() { - // Rejected on Windows (would be written but unreadable, issue #3449); - // valid on non-Windows platforms. - for object in [ - "baddir.", - "dir1/dir2/baddir.", - "dir1/baddir./file.txt", - "file.txt ", - "NUL", - "nul.txt", - "COM1", - "dir/LPT9.log", - ] { - let result = check_object_name_for_length_and_slash("test-bucket", object); - if cfg!(target_os = "windows") { - assert!( - matches!(&result, Err(StorageError::InvalidArgument(_, _, reason)) if reason == WINDOWS_RESERVED_SEGMENT_REASON), - "object name must be rejected on Windows with a descriptive reason: {object:?}, got {result:?}" - ); - } else { - assert!(result.is_ok(), "object name must remain valid on non-Windows: {object:?}"); - } + fn test_windows_object_key_violation_characters() { + let characters = Some(WINDOWS_RESERVED_CHARACTERS_REASON); + for object in ["path/*sUt*mykey", "a:b", "what?", "pipe|name", "quote\"d", "lt"] { + assert_eq!(windows_object_key_violation(object), characters, "{object:?}"); } - // Valid on every platform. - for object in ["file.txt", "dir.name/file", "nullable", "CONSOLE", "com10"] { - assert!( - check_object_name_for_length_and_slash("test-bucket", object).is_ok(), - "object name must be valid on all platforms: {object:?}" + // Control characters 0x01-0x1F cannot appear in a Win32 file name. + for object in ["a\u{1}b", "tab\tname", "dir/line\nbreak", "esc\u{1b}", "unit\u{1f}"] { + assert_eq!(windows_object_key_violation(object), characters, "{object:?}"); + } + + // NUL keeps its platform-independent ObjectNameInvalid answer. + assert_eq!(windows_object_key_violation("a\0b"), None); + + // A character violation outranks a segment violation anywhere in the key. + assert_eq!(windows_object_key_violation("baddir./a:b"), characters); + } + + /// Existing `a\b` keys must stay readable, deletable and abortable, so the + /// shared name check never rejects `\` (only new writes do, on Windows). + #[test] + fn test_backslash_passes_the_shared_name_check() { + assert_eq!(windows_object_key_violation("a\\b"), None); + assert_eq!(windows_object_key_violation("dir\\sub\\file.txt"), None); + assert!(check_object_name_for_length_and_slash("test-bucket", "a\\b").is_ok()); + assert!(check_object_args("test-bucket", "a\\b").is_ok()); + assert!(check_bucket_and_object_names("test-bucket", "a\\b").is_ok()); + // Data movement relocates keys that already exist. + assert!(check_new_object_args("test-bucket", "a\\b", true).is_ok()); + assert!(check_new_object_args(RUSTFS_META_BUCKET, "config\\a", false).is_ok()); + } + + #[test] + fn test_check_new_object_args_matches_put_object_args() { + for object in [ + "file.txt", + "", + "/leading", + "a//b", + "x/../y", + &"k".repeat(1025), + &"s".repeat(256), + "baddir.", + "a:b", + "NUL.txt", + ] { + assert_eq!( + format!("{:?}", check_new_object_args("test-bucket", object, false)), + format!("{:?}", check_put_object_args("test-bucket", object)), + "{object:?}" ); } + assert!(matches!( + check_new_object_args("Bad_Bucket", "file.txt", false), + Err(StorageError::BucketNameInvalid(_)) + )); } #[test] - fn test_check_object_name_windows_reserved_characters() { - // Keys containing Win32-reserved characters are rejected on Windows - // with a descriptive reason (issue #3299); valid elsewhere. - for object in ["path/*sUt*mykey", "a:b", "what?", "pipe|name", "quote\"d", "lt"] { - let result = check_object_name_for_length_and_slash("test-bucket", object); - if cfg!(target_os = "windows") { + #[cfg(not(target_os = "windows"))] + fn test_check_new_object_args_keeps_full_key_set_off_windows() { + for object in ["a\\b", "a:b", "baddir.", "NUL", "CONIN$", "tab\tname"] { + assert!(check_new_object_args("test-bucket", object, false).is_ok(), "{object:?}"); + } + } + + /// `a\b` and `a/b` resolve to the same file on a Windows disk, so a new + /// `a\b` would overwrite (or be overwritten by) `a/b`. + #[test] + #[cfg(target_os = "windows")] + fn windows_new_object_key_rejects_backslash_alias() { + assert!(check_new_object_args("test-bucket", "a/b", false).is_ok()); + for object in ["a\\b", "dir\\", "dir/sub\\file"] { + assert!( + matches!( + &check_new_object_args("test-bucket", object, false), + Err(StorageError::InvalidArgument(_, _, reason)) if reason == WINDOWS_BACKSLASH_REASON + ), + "{object:?}" + ); + } + // Reads, deletes and in-flight multipart uploads of existing keys keep working. + assert!(check_object_args("test-bucket", "a\\b").is_ok()); + assert!(check_del_obj_args("test-bucket", "a\\b").is_ok()); + assert!(check_new_object_args("test-bucket", "a\\b", true).is_ok()); + } + + #[test] + #[cfg(target_os = "windows")] + fn windows_new_object_key_rejects_unaddressable_names() { + for (object, expected) in [ + ("baddir.", WINDOWS_RESERVED_SEGMENT_REASON), + ("dir/LPT9.log", WINDOWS_RESERVED_SEGMENT_REASON), + ("COM\u{b9}", WINDOWS_RESERVED_SEGMENT_REASON), + ("CONOUT$", WINDOWS_RESERVED_SEGMENT_REASON), + ("a:b", WINDOWS_RESERVED_CHARACTERS_REASON), + ("tab\tname", WINDOWS_RESERVED_CHARACTERS_REASON), + ("esc\u{1b}", WINDOWS_RESERVED_CHARACTERS_REASON), + ] { + for result in [ + check_object_name_for_length_and_slash("test-bucket", object), + check_put_object_args("test-bucket", object), + check_new_object_args("test-bucket", object, false), + check_object_args("test-bucket", object), + ] { assert!( - matches!(&result, Err(StorageError::InvalidArgument(_, _, reason)) if reason == WINDOWS_RESERVED_CHARACTERS_REASON), - "object name must be rejected on Windows with a descriptive reason: {object:?}, got {result:?}" + matches!(&result, Err(StorageError::InvalidArgument(_, _, reason)) if reason == expected), + "{object:?} must be rejected with {expected:?}, got {result:?}" ); - } else { - assert!(result.is_ok(), "object name must remain valid on non-Windows: {object:?}"); } } + // NUL keeps the platform-independent answer. + assert!(matches!( + check_put_object_args("test-bucket", "a\0b"), + Err(StorageError::ObjectNameInvalid(_, _)) + )); } #[test] diff --git a/crates/ecstore/src/cluster/rpc/client.rs b/crates/ecstore/src/cluster/rpc/client.rs index 76ae3bb36..307b6ded0 100644 --- a/crates/ecstore/src/cluster/rpc/client.rs +++ b/crates/ecstore/src/cluster/rpc/client.rs @@ -24,10 +24,11 @@ use crate::cluster::rpc::{gen_tonic_replay_scope_headers, gen_tonic_signature_he #[cfg(test)] use crate::cluster::rpc::{tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers}; use crate::disk::error::{DiskError, Error as DiskErrorType, RpcStatusError}; +#[cfg(test)] use crate::runtime::sources as runtime_sources; use http::{Request as HttpRequest, Response as HttpResponse, Uri}; use rustfs_protos::{ - ChannelClass, create_new_channel, get_channel_for_class, + ChannelClass, get_channel_for_class, proto_gen::node_service::{ heal_control_service_client::HealControlServiceClient, node_service_client::NodeServiceClient, scanner_control_service_client::ScannerControlServiceClient, @@ -53,7 +54,7 @@ use super::context_propagation::{inject_request_id_into_metadata, inject_trace_c /// 3. Subsequent calls will attempt fresh connections /// 4. If node is still down, connection will fail fast (3s timeout) pub async fn node_service_time_out_client( - addr: &String, + addr: &str, interceptor: TonicInterceptor, ) -> Result>, Box> { // Default to the latency-sensitive control channel; bulk `bytes` RPCs opt in via the @@ -66,12 +67,9 @@ pub(crate) async fn scanner_control_time_out_client( interceptor: TonicInterceptor, ) -> crate::error::Result>> { let interceptor = interceptor.with_rpc_audience(addr)?; - let channel = match runtime_sources::cached_node_channel(addr).await { - Some(channel) => channel, - None => create_new_channel(addr) - .await - .map_err(|err| crate::error::Error::other(err.to_string()))?, - }; + let channel = get_channel_for_class(addr, ChannelClass::Control) + .await + .map_err(|err| crate::error::Error::other(err.to_string()))?; let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience()); let limit = rustfs_protos::scoped_dirty_usage::SCOPED_DIRTY_USAGE_MAX_REQUEST_BYTES as usize; Ok(ScannerControlServiceClient::with_interceptor(channel, interceptor) @@ -84,10 +82,7 @@ pub async fn heal_control_time_out_client( interceptor: TonicInterceptor, ) -> Result>, Box> { let interceptor = interceptor.with_rpc_audience(addr)?; - let channel = match runtime_sources::cached_node_channel(addr).await { - Some(channel) => channel, - None => create_new_channel(addr).await?, - }; + let channel = get_channel_for_class(addr, ChannelClass::Control).await?; let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE; let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience()); Ok(HealControlServiceClient::with_interceptor(channel, interceptor) @@ -100,10 +95,7 @@ pub async fn tier_mutation_control_time_out_client( interceptor: TonicInterceptor, ) -> Result>, Box> { let interceptor = interceptor.with_rpc_audience(addr)?; - let channel = match runtime_sources::cached_node_channel(addr).await { - Some(channel) => channel, - None => create_new_channel(addr).await?, - }; + let channel = get_channel_for_class(addr, ChannelClass::Control).await?; let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE; let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience()); Ok(TierMutationControlServiceClient::with_interceptor(channel, interceptor) @@ -118,22 +110,12 @@ pub async fn tier_mutation_control_time_out_client( /// from lock/health RPCs; everything else uses [`ChannelClass::Control`]. When isolation is /// disabled the two classes resolve to the same cached channel, i.e. legacy behavior. pub async fn node_service_time_out_client_for_class( - addr: &String, + addr: &str, interceptor: TonicInterceptor, class: ChannelClass, ) -> Result>, Box> { let interceptor = interceptor.with_rpc_audience(addr)?; - let channel = match class { - ChannelClass::Control => match runtime_sources::cached_node_channel(addr).await { - Some(channel) => { - debug!("Using cached gRPC channel for: {}", addr); - channel - } - // No cached connection, create new one. - None => create_new_channel(addr).await?, - }, - ChannelClass::Bulk => get_channel_for_class(addr, ChannelClass::Bulk).await?, - }; + let channel = get_channel_for_class(addr, class).await?; let max_message_size = rustfs_protos::internode_rpc_max_message_size(); let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience()); @@ -142,8 +124,24 @@ pub async fn node_service_time_out_client_for_class( .max_encoding_message_size(max_message_size)) } +pub(crate) async fn node_service_delete_recovery_client( + addr: &str, + interceptor: TonicInterceptor, + timeout: std::time::Duration, +) -> Result>, DiskError> { + let interceptor = interceptor.with_rpc_audience(addr).map_err(DiskError::from)?; + let channel = rustfs_protos::create_channel_with_timeout(addr, timeout) + .await + .map_err(|error| DiskError::RemoteClientUnavailable(error.to_string()))?; + let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience()); + let max_message_size = rustfs_protos::internode_rpc_max_message_size(); + Ok(NodeServiceClient::with_interceptor(channel, interceptor) + .max_decoding_message_size(max_message_size) + .max_encoding_message_size(max_message_size)) +} + pub async fn node_service_time_out_client_no_auth( - addr: &String, + addr: &str, ) -> Result>, Box> { node_service_time_out_client(addr, TonicInterceptor::NoOp(NoOpInterceptor)).await } @@ -355,6 +353,15 @@ fn apply_peer_replay_response( } } +fn rename_data_grpc_stage(path: &str) -> bool { + matches!( + path, + "/node_service.NodeService/RenameData" + | "/node_service.NodeService/RenameDataAtIncarnation" + | "/node_service.NodeService/RenameDataWithQuotaFence" + ) +} + impl Service> for ReplayScopeChannel where S: Service, Response = HttpResponse>, @@ -372,6 +379,12 @@ where } fn call(&mut self, mut request: HttpRequest) -> Self::Future { + let observe_rename_data = rustfs_io_metrics::put_stage_metrics_enabled() && rename_data_grpc_stage(request.uri().path()); + let request_scope_started = if observe_rename_data { + rustfs_io_metrics::put_stage_timer() + } else { + None + }; let authenticated = self.audience.as_ref().is_some_and(|_| { request .headers() @@ -406,11 +419,30 @@ where } } } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REQUEST_SCOPE, + request_scope_started, + ); let audience = self.audience.clone(); let future = self.inner.call(request); Box::pin(async move { - let response = future.await?; + let response = if observe_rename_data { + rustfs_io_metrics::observe_put_stage_future( + future, + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL, + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_POLL_PENDING_COUNT, + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_FIRST_PENDING_TO_READY, + ) + .await? + } else { + future.await? + }; + let replay_response_started = if observe_rename_data { + rustfs_io_metrics::put_stage_timer() + } else { + None + }; if let (Some(audience), Some(challenge)) = (audience, challenge) { let response_state = verify_tonic_peer_replay_capabilities_response(&audience, challenge, response.headers()); if let Err(error) = &response_state @@ -428,6 +460,10 @@ where } apply_peer_replay_response(audience, sent_state, response_state); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REPLAY_RESPONSE, + replay_response_started, + ); Ok(response) }) } @@ -546,6 +582,126 @@ mod tests { use tracing_opentelemetry::OpenTelemetrySpanExt; use tracing_subscriber::{Registry, layer::SubscriberExt}; + #[tokio::test] + async fn delete_recovery_channel_uses_its_scoped_deadline() { + use bytes::Bytes; + use http_body_util::StreamBody; + use hyper::{body::Frame, service::service_fn}; + use hyper_util::rt::{TokioExecutor, TokioIo}; + use rustfs_protos::proto_gen::node_service::{RecoverDeleteTransactionRequest, RecoverDeleteTransactionResponse}; + use tonic_prost::prost::Message; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind recovery peer"); + let addr = format!("http://{}", listener.local_addr().expect("peer address")); + let server = tokio::spawn(async move { + loop { + let (stream, _) = listener.accept().await.expect("accept recovery connection"); + tokio::spawn(async move { + let service = service_fn(|_request: HttpRequest| async { + tokio::time::sleep(std::time::Duration::from_millis(500)).await; + let payload = RecoverDeleteTransactionResponse { + success: true, + protocol_version: 1, + result: Bytes::new(), + error: None, + } + .encode_to_vec(); + let mut data = vec![0]; + data.extend_from_slice(&u32::try_from(payload.len()).expect("small response").to_be_bytes()); + data.extend_from_slice(&payload); + let mut trailers = http::HeaderMap::new(); + trailers.insert("grpc-status", "0".parse().expect("status")); + let frames: Vec, std::convert::Infallible>> = + vec![Ok(Frame::data(data.into())), Ok(Frame::trailers(trailers))]; + let mut response = HttpResponse::new(StreamBody::new(futures::stream::iter(frames))); + response + .headers_mut() + .insert("content-type", "application/grpc".parse().expect("content type")); + Ok::<_, std::convert::Infallible>(response) + }); + let _ = hyper::server::conn::http2::Builder::new(TokioExecutor::new()) + .serve_connection(TokioIo::new(stream), service) + .await; + }); + } + }); + let request = || RecoverDeleteTransactionRequest { + disk: "disk".into(), + volume: "bucket".into(), + protocol_version: 1, + operation: Bytes::new(), + }; + // Seed an existing short-lived control channel to reproduce the inherited deadline. + let control = tonic::transport::Endpoint::from_shared(addr.clone()) + .expect("test endpoint") + .timeout(std::time::Duration::from_millis(50)) + .connect() + .await + .expect("control connect"); + rustfs_common::cache_connection(addr.clone(), control).await; + let mut foreground = node_service_time_out_client(&addr, TonicInterceptor::NoOp(NoOpInterceptor)) + .await + .expect("control client"); + let error = foreground + .recover_delete_transaction(request()) + .await + .expect_err("short control deadline"); + assert!(matches!(error.code(), tonic::Code::Cancelled | tonic::Code::DeadlineExceeded)); + // The 2-second test override bypasses the production range parser, which rejects 2. + let mut recovery = node_service_delete_recovery_client( + &addr, + TonicInterceptor::NoOp(NoOpInterceptor), + std::time::Duration::from_secs(2), + ) + .await + .expect("scoped recovery client"); + assert!( + recovery + .recover_delete_transaction(request()) + .await + .expect("scoped deadline must survive control deadline") + .into_inner() + .success + ); + server.abort(); + rustfs_protos::evict_failed_connection(&addr).await; + } + + #[tokio::test] + async fn rpc_client_factories_do_not_reuse_https_channels_without_tls_generation() { + let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("reserve local address"); + let addr = format!("https://{}", listener.local_addr().expect("local address")); + drop(listener); + for factory in 0..4 { + let placeholder = tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy(); + runtime_sources::cache_test_node_channel(addr.clone(), placeholder).await; + let interceptor = TonicInterceptor::NoOp(NoOpInterceptor); + let _result = match factory { + 0 => scanner_control_time_out_client(&addr, interceptor) + .await + .map(|_| ()) + .map_err(|e| e.to_string()), + 1 => heal_control_time_out_client(&addr, interceptor) + .await + .map(|_| ()) + .map_err(|e| e.to_string()), + 2 => tier_mutation_control_time_out_client(&addr, interceptor) + .await + .map(|_| ()) + .map_err(|e| e.to_string()), + _ => node_service_time_out_client(&addr, interceptor) + .await + .map(|_| ()) + .map_err(|e| e.to_string()), + }; + assert!( + !runtime_sources::test_node_channel_is_cached(&addr).await, + "factory {factory} must evict an unbound HTTPS cache entry before attempting a fresh dial" + ); + } + } + #[derive(Clone)] struct EpochProofService { audience: String, diff --git a/crates/ecstore/src/cluster/rpc/network_probe.rs b/crates/ecstore/src/cluster/rpc/network_probe.rs index 75d9049ea..db2221cfb 100644 --- a/crates/ecstore/src/cluster/rpc/network_probe.rs +++ b/crates/ecstore/src/cluster/rpc/network_probe.rs @@ -289,13 +289,9 @@ async fn client( >, NetworkPeerProbeError, > { - node_service_time_out_client_for_class( - &address.to_owned(), - TonicInterceptor::Signature(gen_tonic_signature_interceptor()), - class, - ) - .await - .map_err(|_| NetworkPeerProbeError::Unreachable) + node_service_time_out_client_for_class(address, TonicInterceptor::Signature(gen_tonic_signature_interceptor()), class) + .await + .map_err(|_| NetworkPeerProbeError::Unreachable) } fn ping_request(payload: &[u8]) -> PingRequest { @@ -329,6 +325,24 @@ fn map_status(status: tonic::Status) -> NetworkPeerProbeError { match status.code() { Code::Unavailable | Code::Unknown => NetworkPeerProbeError::Unreachable, Code::DeadlineExceeded => NetworkPeerProbeError::TimedOut, + Code::Cancelled | Code::Internal => { + let mut source = std::error::Error::source(&status); + while let Some(error) = source { + if error + .downcast_ref::() + .is_some_and(|error| matches!(error.reason(), Some(h2::Reason::CANCEL | h2::Reason::NO_ERROR))) + { + return NetworkPeerProbeError::Unreachable; + } + source = if let Some(error) = error.downcast_ref::() { + // io::Error::source skips its immediate custom error. + error.get_ref().map(|inner| inner as &(dyn std::error::Error + 'static)) + } else { + error.source() + }; + } + NetworkPeerProbeError::ProtocolFailure + } _ => NetworkPeerProbeError::ProtocolFailure, } } @@ -339,6 +353,35 @@ mod tests { use crate::layout::endpoint::Endpoint; use crate::layout::endpoints::{Endpoints, PoolEndpoints}; + #[test] + fn peer_transport_resets_are_unreachable_without_reclassifying_application_errors() { + for reason in [h2::Reason::CANCEL, h2::Reason::NO_ERROR] { + let status = tonic::Status::from_error(Box::new(h2::Error::from(reason))); + assert!(matches!(status.code(), Code::Cancelled | Code::Internal)); + let code = status.code(); + assert_eq!(map_status(status), NetworkPeerProbeError::Unreachable); + let mut wrapped = tonic::Status::new(code, "wrapped transport reset"); + wrapped.set_source(Arc::new(std::io::Error::other(h2::Error::from(reason)))); + assert_eq!(map_status(wrapped), NetworkPeerProbeError::Unreachable); + } + for reason in [h2::Reason::PROTOCOL_ERROR, h2::Reason::COMPRESSION_ERROR] { + let status = tonic::Status::from_error(Box::new(h2::Error::from(reason))); + assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure); + } + for status in [ + tonic::Status::cancelled("peer cancelled the application request"), + tonic::Status::internal("peer rejected the probe"), + tonic::Status::unauthenticated("authentication rejected"), + tonic::Status::permission_denied("authorization rejected"), + ] { + assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure); + } + assert_eq!( + map_status(tonic::Status::deadline_exceeded("probe deadline elapsed")), + NetworkPeerProbeError::TimedOut + ); + } + fn pacing(duration: Duration) -> Arc { NetworkPeerProbeClient::from_endpoint_pools(&topology()) .with_diagnostic_pacing(Instant::now(), duration) diff --git a/crates/ecstore/src/cluster/rpc/remote_disk.rs b/crates/ecstore/src/cluster/rpc/remote_disk.rs index 551011d62..071f858e5 100644 --- a/crates/ecstore/src/cluster/rpc/remote_disk.rs +++ b/crates/ecstore/src/cluster/rpc/remote_disk.rs @@ -14,7 +14,8 @@ use crate::cluster::rpc::client::{ AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error, - node_service_time_out_client, node_service_time_out_client_for_class, node_service_time_out_client_no_auth, + node_service_delete_recovery_client, node_service_time_out_client, node_service_time_out_client_for_class, + node_service_time_out_client_no_auth, }; use crate::cluster::rpc::internode_data_transport::{ InternodeDataTransport, NsScannerCapabilityRequest, NsScannerStreamRequest, ReadStreamRequest, WalkDirStreamRequest, @@ -22,10 +23,12 @@ use crate::cluster::rpc::internode_data_transport::{ }; use crate::disk::error::{Error, Result}; use crate::disk::{ - BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DeleteOptions, DiskAPI, DiskInfo, - DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq, - ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, - batch_read_version_one_by_one, + BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION, + DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection, + DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, + DiskOption, FileInfoVersions, FileReader, FileWriter, GuardedDeleteVersionsPayload, GuardedFileInfoVersions, + PartTransactionAction, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, + UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one, disk_store::{ DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE, get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout, @@ -57,12 +60,12 @@ use rustfs_protos::proto_gen::node_service::RenamePartRequest; use rustfs_protos::proto_gen::node_service::{ BatchReadVersionRequest, BatchReadVersionResponse, CheckPartsRequest, CompareAndUpdateFileOutcome, CompareAndUpdateFileRequest, DeletePathsRequest, DeleteRequest, DeleteVersionRequest, DeleteVersionsRequest, - DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, ListVolumesRequest, MakeVolumeRequest, - MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, ReadMetadataRequest, ReadMultipleRequest, - ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest, RenameFileRequest, - SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest, - SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest, - node_service_client::NodeServiceClient, + DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, GuardedDeleteVersionsRequest, ListDirRequest, + ListVolumesRequest, MakeVolumeRequest, MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, + ReadMetadataRequest, ReadMultipleRequest, ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, + RecoverDeleteTransactionRequest, RenameDataRequest, RenameFileRequest, SettlePartTransactionRequest, + SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest, SnapshotLeaseResponse, StatVolumeRequest, + UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest, node_service_client::NodeServiceClient, }; use serde::{Serialize, de::DeserializeOwned}; use std::{ @@ -142,6 +145,45 @@ fn decode_delete_versions_errors(response: DeleteVersionsResponse, expected_len: .collect() } +fn decode_delete_recovery_batch_errors(codes: Vec>, expected_len: usize) -> Result>> { + if codes.len() != expected_len { + return Err(Error::other("malformed recoverable batch delete response length")); + } + codes + .into_iter() + .map(|code| match code { + None => Ok(None), + Some(code) => DiskError::from_u32(code) + .map(Some) + .ok_or_else(|| Error::other("invalid recoverable batch delete error code")), + }) + .collect() +} + +fn validate_delete_recovery_inspection( + volume: &str, + object: &str, + transaction: Uuid, + inspection: &DeleteRecoveryInspection, +) -> Result<()> { + if inspection.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION + || inspection.id.object != object + || inspection.id.transaction != transaction + || transaction.is_nil() + { + return Err(Error::other("delete recovery inspection does not match request")); + } + if let Some(journal) = &inspection.journal + && (journal.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION + || journal.volume != volume + || journal.object != object + || journal.transaction != transaction) + { + return Err(Error::other("delete recovery journal does not match request")); + } + Ok(()) +} + fn snapshot_lease_token_from_response(response: SnapshotLeaseResponse) -> Result { if !response.success { return Err(response.error.unwrap_or_default().into()); @@ -176,6 +218,24 @@ fn attach_mutation_body_digest( crate::cluster::rpc::set_tonic_rolling_canonical_body_digest(request, &canonical_body).map_err(Error::other) } +fn attach_quota_mutation_body_digest( + request: &mut Request, + canonical_body: std::result::Result, std::num::TryFromIntError>, +) -> Result<()> { + let topology = crate::services::notification_sys::configured_fleet_topology_fingerprint() + .ok_or_else(|| Error::other("quota mutation topology is not initialized or conflicts with startup"))?; + let body = canonical_body.map_err(|_| Error::other("quota request length cannot be represented"))?; + let body = rustfs_protos::canonical_quota_mutation_request_body(topology, &body) + .map_err(|_| Error::other("quota topology envelope length cannot be represented"))?; + request.metadata_mut().insert( + rustfs_protos::QUOTA_MUTATION_TOPOLOGY_HEADER, + topology + .parse() + .map_err(|_| Error::other("quota topology cannot be encoded"))?, + ); + crate::cluster::rpc::set_tonic_canonical_body_digest(request, &body).map_err(Error::other) +} + fn decode_volume_infos(volume_infos: Vec) -> Result> { volume_infos .into_iter() @@ -858,6 +918,132 @@ fn spawn_control_channel_prewarm(addr: String) { } impl RemoteDisk { + async fn guarded_delete_versions_rpc( + &self, + volume: &str, + payload: Vec, + probe_only: bool, + ) -> Result { + if payload.len() > rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES { + return Err(DiskError::FileCorrupt); + } + self.execute_with_timeout( + || async { + let mut client = self.get_client().await?; + let mut request = Request::new(GuardedDeleteVersionsRequest { + disk: self.endpoint.to_string(), + volume: volume.to_owned(), + payload: payload.into(), + probe_only, + }); + let body = rustfs_protos::canonical_guarded_delete_versions_request_body(request.get_ref()) + .map_err(|_| DiskError::FileCorrupt)?; + crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &body).map_err(Error::other)?; + // The additive method is mandatory: an old peer cannot safely delete + // a guarded request through any legacy batch method. + let response = client + .guarded_delete_versions(request) + .await + .map_err(|status| { + if status.code() == Code::Unimplemented { + DiskError::GuardedDeleteUnsupported + } else { + status.into() + } + })? + .into_inner(); + if !response.success { + return Err(response.error.ok_or(DiskError::FileCorrupt)?.into()); + } + if response.error.is_some() { + return Err(DiskError::FileCorrupt); + } + Ok(response) + }, + get_max_timeout_duration(), + ) + .await + } + + pub(crate) async fn is_plain_orphan_data_dir(&self, volume: &str, path: &str) -> Result { + match self + .recover_delete_transaction(volume, DeleteRecoveryRpcOperation::InspectOrphanDataDir { path: path.to_owned() }) + .await? + { + DeleteRecoveryRpcReply::OrphanDataDir { plain } => Ok(plain), + _ => Err(Error::other("unexpected orphan data directory inspection reply")), + } + } + + async fn recover_delete_transaction( + &self, + volume: &str, + operation: DeleteRecoveryRpcOperation, + ) -> Result { + let payload = rmp_serde::to_vec_named(&operation)?; + if payload.len() > operation.max_wire_payload_bytes() { + return Err(Error::other("delete recovery operation exceeds payload limit")); + } + let settle_timeout = if matches!(&operation, DeleteRecoveryRpcOperation::Settle { .. }) { + Some(crate::disk::get_delete_recovery_timeout()?) + } else { + None + }; + self.execute_with_timeout( + || async { + let mut client = if let Some(timeout) = settle_timeout { + if let Some(error) = self.offline_bypass_error() { + return Err(error); + } + node_service_delete_recovery_client( + &self.addr, + TonicInterceptor::Signature(gen_tonic_signature_interceptor()), + timeout, + ) + .await? + } else { + self.get_client().await? + }; + let mut request = Request::new(RecoverDeleteTransactionRequest { + disk: self.endpoint.to_string(), + volume: volume.to_owned(), + protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION, + operation: payload.into(), + }); + let canonical_body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()) + .map_err(|_| Error::other("recover_delete_transaction request length cannot be represented"))?; + crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical_body).map_err(Error::other)?; + // UNIMPLEMENTED is deliberately propagated: older peers cannot settle transactions. + let response = client + .recover_delete_transaction(request) + .await + .map_err(|status| { + if status.code() == Code::Unimplemented { + DiskError::UnsupportedDisk + } else { + status.into() + } + })? + .into_inner(); + if response.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION { + return Err(DiskError::UnsupportedDisk); + } + if !response.success { + return Err(response + .error + .ok_or_else(|| Error::other("missing delete recovery error"))? + .into()); + } + if response.error.is_some() { + return Err(Error::other("malformed delete recovery success")); + } + rustfs_protos::decode_delete_recovery_payload(&response.result).map_err(Into::into) + }, + settle_timeout.unwrap_or_else(get_max_timeout_duration), + ) + .await + } + async fn rename_file_with_durability( &self, src_volume: &str, @@ -1920,6 +2106,12 @@ fn encode_msgpack(value: &T) -> Result> { encode_msgpack_with_capacity(value, MSGPACK_ENCODE_CAPACITY_HINT) } +// Optional fields are omitted by serde; named maps keep rollback flags and +// transaction UUIDs independent of field order across peer versions. +fn encode_delete_options_msgpack(value: &DeleteOptions) -> Result> { + Ok(rmp_serde::to_vec_named(value)?) +} + fn encode_file_info_msgpack(value: &FileInfo) -> Result> { encode_msgpack_with_capacity(value, FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT) } @@ -2376,6 +2568,7 @@ impl RemoteDisk { self.execute_with_timeout_for_op( "rename_data", || async { + let prepare_started = rustfs_io_metrics::put_stage_timer(); let file_info = compat_json(fi)?; let file_info_bin = encode_file_info_msgpack(fi)?; let mut client = self.get_client().await?; @@ -2403,33 +2596,85 @@ impl RemoteDisk { rustfs_protos::canonical_rename_data_request_body(request.get_ref()) }; let incarnation_bound = !request.get_ref().bucket_incarnation_id.is_empty(); - if scanner_publication_lease_token.is_some() || incarnation_bound { + let quota_fenced = rustfs_utils::http::metadata_compat::contains_key_str( + &fi.metadata, + crate::disk::QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + ); + if quota_fenced { + attach_quota_mutation_body_digest(&mut request, canonical_body)?; + } else if scanner_publication_lease_token.is_some() || incarnation_bound { let canonical_body = canonical_body.map_err(|_| Error::other("rename_data request length cannot be represented"))?; crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical_body).map_err(Error::other)?; } else { attach_mutation_body_digest(&mut request, canonical_body, "rename_data")?; } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE, + prepare_started, + ); - let response = if artifact_incarnation.is_some() { - client.artifact_rename_data(request).await? - } else if incarnation_bound { - // Older peers return Unimplemented before mutation; never downgrade. - client.rename_data_at_incarnation(request).await? - } else { - client.rename_data(request).await? - } - .into_inner(); + let rpc_started = rustfs_io_metrics::put_stage_timer(); + let response = rustfs_io_metrics::observe_put_stage_future( + async { + if artifact_incarnation.is_some() { + client.artifact_rename_data(request).await + } else if quota_fenced { + // The method itself proves quota-token enforcement; an old peer must fail closed. + client.rename_data_with_quota_fence(request).await + } else if incarnation_bound { + // Older peers return Unimplemented before mutation; never downgrade. + client.rename_data_at_incarnation(request).await + } else { + client.rename_data(request).await + } + }, + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT, + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT, + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY, + ) + .await?; + let response_process_started = rustfs_io_metrics::put_stage_timer(); + let response_headers_started = rustfs_io_metrics::put_stage_timer(); + let _ = response.metadata().len(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_HEADERS, + response_headers_started, + ); + let into_inner_started = rustfs_io_metrics::put_stage_timer(); + let response = response.into_inner(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_INTO_INNER, + into_inner_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC, + rpc_started, + ); + let success_check_started = rustfs_io_metrics::put_stage_timer(); if !response.success { return Err(response.error.unwrap_or_default().into()); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_SUCCESS_CHECK, + success_check_started, + ); + let decode_started = rustfs_io_metrics::put_stage_timer(); let rename_data_resp = decode_msgpack_or_json::( &response.rename_data_resp_bin, &response.rename_data_resp, "RenameDataResp", )?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_DECODE, + decode_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_PROCESS, + response_process_started, + ); Ok(rename_data_resp) }, @@ -2510,6 +2755,39 @@ impl RemoteDisk { #[async_trait::async_trait] impl DiskAPI for RemoteDisk { + async fn supports_guarded_delete_versions(&self) -> Result<()> { + let response = self.guarded_delete_versions_rpc("", Vec::new(), true).await?; + if !response.errors.is_empty() || !response.item_errors.is_empty() { + return Err(DiskError::FileCorrupt); + } + Ok(()) + } + async fn delete_versions_guarded_recoverable( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + ) -> Vec> { + let count = versions.len(); + let payload = GuardedDeleteVersionsPayload { versions, opts }; + if let Err(error) = payload.validate() { + return vec![Some(error); count]; + } + let encoded = match rmp_serde::to_vec_named(&payload) { + Ok(encoded) => encoded, + Err(error) => return vec![Some(error.into()); count], + }; + match self.guarded_delete_versions_rpc(volume, encoded, false).await { + Ok(response) => { + // New guarded peers must return typed errors, never lossy strings. + if response.item_errors.len() != count { + return vec![Some(DiskError::FileCorrupt); count]; + } + decode_delete_versions_errors(response, count) + } + Err(error) => vec![Some(error); count], + } + } #[tracing::instrument(level = "trace", skip_all)] fn to_string(&self) -> String { self.endpoint.to_string() @@ -2788,7 +3066,7 @@ impl DiskAPI for RemoteDisk { // `_bin` support for DeleteVersion is new (grpc-optimization P2); always dual-write // JSON + msgpack until its fallback counter has read zero across a release window. let file_info_bin = encode_file_info_msgpack(&fi)?; - let opts_bin = encode_msgpack(&opts)?; + let opts_bin = encode_delete_options_msgpack(&opts)?; let conditional_marker = opts.expected_delete_marker.is_some(); let file_info = serde_json::to_string(&fi)?; let opts = serde_json::to_string(&opts)?; @@ -2860,7 +3138,7 @@ impl DiskAPI for RemoteDisk { // `_bin` support for DeleteVersions is new (grpc-optimization P2); always dual-write JSON + // msgpack until its fallback counter has read zero across a release window. - let opts_bin = match encode_msgpack(&opts) { + let opts_bin = match encode_delete_options_msgpack(&opts) { Ok(opts_bin) => opts_bin, Err(err) => { let mut errors = Vec::with_capacity(versions.len()); @@ -3017,8 +3295,15 @@ impl DiskAPI for RemoteDisk { .map_err(|_| Error::other("snapshot lease TTL cannot be represented"))?, }); let canonical_body = rustfs_protos::canonical_snapshot_lease_request_body(request.get_ref()); - attach_mutation_body_digest(&mut request, canonical_body, "acquire_snapshot_lease")?; - let response = client.acquire_snapshot_lease(request).await?.into_inner(); + let quota_fence = volume == crate::disk::RUSTFS_META_BUCKET && crate::disk::is_quota_mutation_fence_path(path); + let response = if quota_fence { + attach_quota_mutation_body_digest(&mut request, canonical_body)?; + client.acquire_quota_mutation_fence(request).await? + } else { + attach_mutation_body_digest(&mut request, canonical_body, "acquire_snapshot_lease")?; + client.acquire_snapshot_lease(request).await? + } + .into_inner(); snapshot_lease_token_from_response(response) }, get_max_timeout_duration(), @@ -3059,8 +3344,15 @@ impl DiskAPI for RemoteDisk { token: token.as_bytes().to_vec().into(), }); let canonical_body = rustfs_protos::canonical_snapshot_lease_release_request_body(request.get_ref()); - attach_mutation_body_digest(&mut request, canonical_body, "release_snapshot_lease")?; - let response = client.release_snapshot_lease(request).await?.into_inner(); + let quota_fence = volume == crate::disk::RUSTFS_META_BUCKET && crate::disk::is_quota_mutation_fence_path(path); + let response = if quota_fence { + attach_quota_mutation_body_digest(&mut request, canonical_body)?; + client.release_quota_mutation_fence(request).await? + } else { + attach_mutation_body_digest(&mut request, canonical_body, "release_snapshot_lease")?; + client.release_snapshot_lease(request).await? + } + .into_inner(); if !response.success { return Err(response.error.unwrap_or_default().into()); } @@ -3270,7 +3562,7 @@ impl DiskAPI for RemoteDisk { opts_bin: opts_bin.into(), }); - crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request(); + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request(opts); if let Some(request_payload_bytes) = request_payload_bytes { crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_sent_bytes(request_payload_bytes); } @@ -3348,6 +3640,7 @@ impl DiskAPI for RemoteDisk { record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_REQUEST_ENCODE, encode_started); let request_payload_bytes = batch_read_version_attribution_enabled .then(|| batch_read_version_request_payload_len(&req, &batch_read_version_req, &batch_read_version_req_bin)); + let request_opts = req.opts; let batch_result = self .execute_with_timeout_for_op( "batch_read_version", @@ -3361,7 +3654,7 @@ impl DiskAPI for RemoteDisk { batch_read_version_req_bin: batch_read_version_req_bin.into(), }); - crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_request(); + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_request(&request_opts); if let Some(request_payload_bytes) = request_payload_bytes { crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_sent_bytes( request_payload_bytes.saturating_add(disk_len), @@ -3854,6 +4147,153 @@ impl DiskAPI for RemoteDisk { } #[tracing::instrument(level = "trace", skip_all)] + async fn delete_version_recoverable( + &self, + volume: &str, + object: &str, + file_info: FileInfo, + force_del_marker: bool, + opts: DeleteOptions, + ) -> Result<()> { + match self + .recover_delete_transaction( + volume, + DeleteRecoveryRpcOperation::ApplyVersion { + object: object.to_owned(), + file_info: Box::new(file_info), + force_del_marker, + opts, + }, + ) + .await? + { + DeleteRecoveryRpcReply::AppliedVersion => Ok(()), + _ => Err(Error::other("unexpected recoverable delete reply")), + } + } + + async fn delete_versions_recoverable( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + ) -> Vec> { + let expected_len = versions.len(); + let result = match self + .recover_delete_transaction(volume, DeleteRecoveryRpcOperation::ApplyVersions { versions, opts }) + .await + { + Ok(DeleteRecoveryRpcReply::AppliedVersions(codes)) => decode_delete_recovery_batch_errors(codes, expected_len), + Ok(_) => Err(Error::other("unexpected recoverable batch delete reply")), + Err(error) => Err(error), + }; + match result { + Ok(errors) => errors, + Err(error) => vec![Some(error); expected_len], + } + } + + async fn has_delete_transaction_evidence(&self, volume: &str, object: &str) -> Result { + match self + .recover_delete_transaction( + volume, + DeleteRecoveryRpcOperation::Exists { + object: object.to_owned(), + }, + ) + .await? + { + DeleteRecoveryRpcReply::Evidence { present } => Ok(present), + _ => Err(Error::other("unexpected delete recovery existence reply")), + } + } + + async fn discover_delete_transactions( + &self, + volume: &str, + object: Option<&str>, + cursor: Option, + limit: u32, + ) -> Result { + match self + .recover_delete_transaction( + volume, + DeleteRecoveryRpcOperation::Discover { + object: object.map(str::to_owned), + cursor, + limit, + }, + ) + .await? + { + DeleteRecoveryRpcReply::Discovered(page) => { + if object.is_some_and(|object| page.transactions.iter().any(|id| id.object != object)) { + return Err(Error::other("delete recovery discovery escaped requested object")); + } + if page.transactions.len() > usize::try_from(limit).map_err(|_| Error::other("invalid discovery limit"))? + || page + .transactions + .iter() + .any(|id| id.transaction.is_nil() || id.object.is_empty()) + { + return Err(Error::other("malformed delete discovery page")); + } + Ok(page) + } + _ => Err(Error::other("unexpected delete discovery reply")), + } + } + async fn inspect_delete_transaction( + &self, + volume: &str, + object: &str, + transaction: Uuid, + ) -> Result { + match self + .recover_delete_transaction( + volume, + DeleteRecoveryRpcOperation::Inspect { + object: object.to_owned(), + transaction, + }, + ) + .await? + { + DeleteRecoveryRpcReply::Inspected(inspection) => { + validate_delete_recovery_inspection(volume, object, transaction, &inspection)?; + Ok(inspection) + } + _ => Err(Error::other("unexpected delete inspection reply")), + } + } + async fn settle_delete_transaction( + &self, + volume: &str, + object: &str, + transaction: Uuid, + expected_fingerprint: DeleteRecoveryDigest, + action: DeleteRecoveryAction, + ) -> Result { + match self + .recover_delete_transaction( + volume, + DeleteRecoveryRpcOperation::Settle { + object: object.to_owned(), + transaction, + expected_fingerprint, + action, + }, + ) + .await? + { + DeleteRecoveryRpcReply::Settled(inspection) => { + validate_delete_recovery_inspection(volume, object, transaction, &inspection)?; + Ok(inspection) + } + _ => Err(Error::other("unexpected delete settlement reply")), + } + } + async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> { self.execute_with_timeout( || async { @@ -4344,6 +4784,101 @@ mod tests { peer.stop().await; } + #[tokio::test] + async fn quota_fence_old_peer_rejects_without_legacy_fallback() { + runtime_sources::ensure_test_rpc_secret(); + crate::services::notification_sys::install_cross_pool_fence_fleet_proof_for_test(); + let _without_proof = crate::services::notification_sys::without_live_cross_pool_fence_fleet_proof_for_test(); + assert!(crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_none()); + assert!(crate::services::notification_sys::configured_fleet_topology_fingerprint().is_some()); + let peer = TestGrpcPeer::spawn(Bytes::new(), Bytes::new()) + .await + .expect("old peer must bind"); + let remote = remote_disk_for_addr(&peer.addr).await; + let fence_path = crate::disk::quota_mutation_fence_path("bucket", "object"); + assert!( + remote + .acquire_snapshot_lease(crate::disk::RUSTFS_META_BUCKET, &fence_path) + .await + .is_err() + ); + assert!( + remote + .release_snapshot_lease(crate::disk::RUSTFS_META_BUCKET, &fence_path, SnapshotLeaseToken::revoke_all()) + .await + .is_err() + ); + let mut file_info = sample_rename_data_file_info(); + rustfs_utils::http::metadata_compat::insert_str( + &mut file_info.metadata, + crate::disk::QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + Uuid::new_v4().to_string(), + ); + assert!( + remote + .rename_data("tmp", "source", file_info, "bucket", "object") + .await + .is_err() + ); + let methods = peer + .peer + .unsupported_methods + .lock() + .expect("unsupported method list lock") + .clone(); + assert_eq!( + methods, + vec![ + "/node_service.NodeService/AcquireQuotaMutationFence", + "/node_service.NodeService/ReleaseQuotaMutationFence", + "/node_service.NodeService/RenameDataWithQuotaFence", + ], + "quota operations must never fall back to snapshot or unfenced rename methods" + ); + peer.stop().await; + } + + #[tokio::test] + async fn guarded_delete_old_peer_rejects_without_legacy_fallback() { + runtime_sources::ensure_test_rpc_secret(); + let peer = TestGrpcPeer::spawn(Bytes::new(), Bytes::new()) + .await + .expect("old peer must bind"); + let remote = remote_disk_for_addr(&peer.addr).await; + assert_eq!(remote.supports_guarded_delete_versions().await, Err(DiskError::GuardedDeleteUnsupported)); + let group = GuardedFileInfoVersions { + group: FileInfoVersions { + name: "object".into(), + versions: vec![FileInfo::default()], + ..Default::default() + }, + expected_current: crate::disk::CurrentDeleteGeneration::Absent, + }; + let errors = remote + .delete_versions_guarded_recoverable( + "bucket", + vec![group], + DeleteOptions { + old_data_dir: Some(Uuid::new_v4()), + ..Default::default() + }, + ) + .await; + assert_eq!(errors, vec![Some(DiskError::GuardedDeleteUnsupported)]); + let methods = peer + .peer + .unsupported_methods + .lock() + .expect("unsupported method list lock") + .clone(); + assert_eq!( + methods, + vec!["/node_service.NodeService/GuardedDeleteVersions"; 2], + "neither capability failure nor guarded application may call a legacy deletion method" + ); + peer.stop().await; + } + #[test] fn artifact_repair_seal_response_requires_proof_full_request_and_physical_identity() { use rustfs_protos::proto_gen::node_service::{ @@ -4431,6 +4966,45 @@ mod tests { ); } + #[test] + fn recoverable_batch_delete_rejects_invalid_error_codes_and_length() { + let errors = decode_delete_recovery_batch_errors(vec![None, Some(DiskError::FileNotFound.to_u32())], 2) + .expect("valid per-item errors"); + assert!(errors[0].is_none()); + assert_eq!(errors[1], Some(DiskError::FileNotFound)); + assert!(decode_delete_recovery_batch_errors(vec![None], 2).is_err()); + assert!(decode_delete_recovery_batch_errors(vec![Some(0)], 1).is_err()); + assert!(decode_delete_recovery_batch_errors(vec![Some(u32::MAX)], 1).is_err()); + } + + #[test] + fn delete_recovery_inspection_rejects_wrong_transaction_object_and_protocol() { + let transaction = Uuid::new_v4(); + let inspection = DeleteRecoveryInspection { + protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION, + id: crate::disk::DeleteTransactionId { + object: "object".into(), + transaction, + }, + state: crate::disk::DeleteRecoveryState::Absent, + journal: None, + current_digest: None, + current_versions: Vec::new(), + backup_digest: None, + rollback_data_complete: false, + commit_data_safe: false, + explicit_delete_safe: false, + fingerprint: [0; 32], + }; + validate_delete_recovery_inspection("bucket", "object", transaction, &inspection).expect("matching inspection"); + assert!(validate_delete_recovery_inspection("bucket", "other", transaction, &inspection).is_err()); + assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::new_v4(), &inspection).is_err()); + assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::nil(), &inspection).is_err()); + let mut incompatible = inspection; + incompatible.protocol_version += 1; + assert!(validate_delete_recovery_inspection("bucket", "object", transaction, &incompatible).is_err()); + } + #[test] fn compare_and_update_wire_outcome_fails_closed() { assert_eq!( @@ -4464,32 +5038,19 @@ mod tests { } #[test] - fn retired_marker_options_preserve_legacy_positional_wire_shape() { + fn delete_options_named_wire_preserves_rollback_and_legacy_marker_fields() { #[derive(Debug, serde::Serialize, serde::Deserialize, PartialEq)] struct LegacyDeleteOptions { recursive: bool, immediate: bool, undo_write: bool, + #[serde(default)] undo_delete: bool, old_data_dir: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + expected_delete_marker: Option, } - let legacy = LegacyDeleteOptions { - recursive: false, - immediate: false, - undo_write: false, - undo_delete: false, - old_data_dir: None, - }; - let original = encode_msgpack(&legacy).unwrap(); - let current = encode_msgpack(&DeleteOptions::default()).unwrap(); - assert_eq!(current, original, "ordinary deletes must retain the older peer's positional payload"); - assert_eq!(rmp_serde::from_slice::(¤t).unwrap(), legacy); - assert!( - rmp_serde::from_slice::(&original) - .unwrap() - .expected_delete_marker - .is_none() - ); + let transaction = Uuid::new_v4(); let mut marker = FileInfo { deleted: true, version_id: Some(Uuid::new_v4()), @@ -4498,12 +5059,69 @@ mod tests { }; marker.set_delete_marker_incarnation(Uuid::new_v4()); let marker = rustfs_filemeta::MetaDeleteMarker::from(marker); + for expected_delete_marker in [None, Some(marker)] { + for (undo_write, undo_delete) in [(true, false), (false, true)] { + let legacy = LegacyDeleteOptions { + recursive: false, + immediate: false, + undo_write, + undo_delete, + old_data_dir: Some(transaction), + expected_delete_marker: expected_delete_marker.clone(), + }; + // Older senders use positional arrays. Their unchanged field prefix and + // missing optional fields must also decode on the named-map sender's peer. + for (encoding, payload) in [ + ("positional", rmp_serde::to_vec(&legacy).expect("encode legacy positional options")), + ("named", rmp_serde::to_vec_named(&legacy).expect("encode legacy named options")), + ] { + let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode legacy options on current peer"); + assert_eq!(decoded.undo_write, undo_write, "{encoding} undo-write flag"); + assert_eq!(decoded.undo_delete, undo_delete, "{encoding} undo-delete flag"); + assert_eq!(decoded.old_data_dir, Some(transaction), "{encoding} transaction UUID"); + assert_eq!(decoded.expected_delete_marker, expected_delete_marker, "{encoding} marker precondition"); + assert!(decoded.delete_recovery_bucket_incarnation.is_none(), "{encoding} omitted incarnation"); + assert!(!decoded.recoverable_delete, "{encoding} omitted recovery flag"); + } + for incarnation in [None, Some(Uuid::new_v4())] { + let current = DeleteOptions { + undo_write, + undo_delete, + old_data_dir: Some(transaction), + expected_delete_marker: expected_delete_marker.clone(), + delete_recovery_bucket_incarnation: incarnation, + recoverable_delete: true, + ..Default::default() + }; + let payload = encode_delete_options_msgpack(¤t).expect("encode current named options"); + let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode current options on current peer"); + assert_eq!(decoded.undo_write, undo_write); + assert_eq!(decoded.undo_delete, undo_delete); + assert_eq!(decoded.old_data_dir, Some(transaction)); + assert_eq!(decoded.expected_delete_marker, expected_delete_marker); + assert_eq!(decoded.delete_recovery_bucket_incarnation, incarnation); + assert!(decoded.recoverable_delete); + let decoded: LegacyDeleteOptions = + rmp_serde::from_slice(&payload).expect("decode current options on legacy peer"); + assert_eq!(decoded, legacy, "unknown recovery fields cannot shift rollback or marker values"); + } + } + } + } + + #[test] + fn malformed_nonempty_delete_options_do_not_fall_back_to_json() { let options = DeleteOptions { - expected_delete_marker: Some(marker.clone()), + undo_write: true, + old_data_dir: Some(Uuid::new_v4()), ..Default::default() }; - let decoded: DeleteOptions = rmp_serde::from_slice(&encode_msgpack(&options).unwrap()).unwrap(); - assert_eq!(decoded.expected_delete_marker, Some(marker)); + let json = serde_json::to_string(&options).expect("valid compatibility JSON"); + assert!(decode_msgpack_or_json::(&[0xc1], &json, "DeleteOptions").is_err()); + let decoded: DeleteOptions = + decode_msgpack_or_json(&[], &json, "DeleteOptions").expect("absent MessagePack permits JSON compatibility"); + assert!(decoded.undo_write); + assert_eq!(decoded.old_data_dir, options.old_data_dir); } #[test] @@ -4737,6 +5355,7 @@ mod tests { read_all_data: Bytes, next_format_read_gate: Arc>>>, rpc_paths: Arc>>, + unsupported_methods: Arc>>, } #[derive(Debug, Default)] @@ -4756,6 +5375,7 @@ mod tests { read_all_data, next_format_read_gate: Arc::default(), rpc_paths: Arc::default(), + unsupported_methods: Arc::default(), } } @@ -4907,13 +5527,20 @@ mod tests { Ok(grpc.unary(method, request).await) }) } - _ => Box::pin(async move { - let mut response = http::Response::new(tonic::body::Body::default()); - let headers = response.headers_mut(); - headers.insert(tonic::Status::GRPC_STATUS, (tonic::Code::Unimplemented as i32).into()); - headers.insert(http::header::CONTENT_TYPE, tonic::metadata::GRPC_CONTENT_TYPE); - Ok(response) - }), + _ => { + self.peer + .unsupported_methods + .lock() + .expect("unsupported method list lock") + .push(request.uri().path().to_owned()); + Box::pin(async move { + let mut response = http::Response::new(tonic::body::Body::default()); + let headers = response.headers_mut(); + headers.insert(tonic::Status::GRPC_STATUS, (tonic::Code::Unimplemented as i32).into()); + headers.insert(http::header::CONTENT_TYPE, tonic::metadata::GRPC_CONTENT_TYPE); + Ok(response) + }) + } } } } @@ -5193,6 +5820,8 @@ mod tests { cleanup_data_dir: Some(Uuid::new_v4()), sign: Some(vec![0x14, 0x35]), old_current_size: Some(crate::disk::OldCurrentSize::Present(64 * 1024)), + old_current_source_checked: false, + old_current_source: None, }; let json = serde_json::to_string(&response).expect("legacy rename_data JSON response should encode"); let decode_before = rustfs_io_metrics::internode_metrics::global_internode_metrics().msgpack_json_decode_total_for_test(); @@ -5595,6 +6224,8 @@ mod tests { cleanup_data_dir: Some(Uuid::new_v4()), sign: Some(vec![1_u8; 32]), old_current_size: Some(crate::disk::OldCurrentSize::Present(4096)), + old_current_source_checked: false, + old_current_source: None, }; let json = serde_json::to_vec(&response).expect("rename data response json should encode"); let named_msgpack = rmp_serde::encode::to_vec_named(&response).expect("rename data response named msgpack should encode"); diff --git a/crates/ecstore/src/cluster/rpc/runtime_sources.rs b/crates/ecstore/src/cluster/rpc/runtime_sources.rs index 4ba4ef06d..ca0968b9d 100644 --- a/crates/ecstore/src/cluster/rpc/runtime_sources.rs +++ b/crates/ecstore/src/cluster/rpc/runtime_sources.rs @@ -12,6 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +use crate::disk::ReadOptions; use rustfs_io_metrics::internode_metrics::{ INTERNODE_MSGPACK_CODEC_JSON, INTERNODE_MSGPACK_CODEC_MSGPACK, INTERNODE_MSGPACK_DIRECTION_RESPONSE, INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, INTERNODE_OPERATION_GRPC_COMPARE_AND_UPDATE_FILE, @@ -19,8 +20,409 @@ use rustfs_io_metrics::internode_metrics::{ INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_OPERATION_PUT_FILE_STREAM, INTERNODE_OPERATION_READ_FILE_STREAM, INTERNODE_TRANSPORT_BACKEND_GRPC, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, global_internode_metrics, }; +#[cfg(feature = "diagnostic-read-version-caller")] +use rustfs_io_metrics::internode_metrics::{ + ReadVersionRequestCaller, ReadVersionRequestMode, record_read_version_request_by_caller, +}; use std::time::Duration; +#[cfg(feature = "diagnostic-read-version-caller")] +fn is_path_separator(byte: u8) -> bool { + matches!(byte, b'/' | b'\\') +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn path_byte_matches(path_byte: u8, pattern_byte: u8) -> bool { + path_byte == pattern_byte || (pattern_byte == b'/' && is_path_separator(path_byte)) +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn path_ends_with(path: &str, suffix: &str) -> bool { + let path = path.as_bytes(); + let suffix = suffix.as_bytes(); + let Some(start) = path.len().checked_sub(suffix.len()) else { + return false; + }; + path[start..] + .iter() + .zip(suffix) + .all(|(&path_byte, &pattern_byte)| path_byte_matches(path_byte, pattern_byte)) +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn path_contains(path: &str, pattern: &str) -> bool { + let path = path.as_bytes(); + let pattern = pattern.as_bytes(); + !pattern.is_empty() + && path.len() >= pattern.len() + && path.windows(pattern.len()).any(|window| { + window + .iter() + .zip(pattern) + .all(|(&path_byte, &pattern_byte)| path_byte_matches(path_byte, pattern_byte)) + }) +} + +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) fn read_version_request_caller_from_location(file: &str) -> ReadVersionRequestCaller { + if path_ends_with(file, "/store/peer.rs") { + ReadVersionRequestCaller::Peer + } else if path_ends_with(file, "/store/init.rs") { + ReadVersionRequestCaller::Startup + } else if path_ends_with(file, "/store/heal.rs") + || path_ends_with(file, "/set_disk/ops/heal.rs") + || path_ends_with(file, "/set_disk/ops/heal_walk.rs") + { + ReadVersionRequestCaller::Heal + } else if path_ends_with(file, "/store/list_objects.rs") { + ReadVersionRequestCaller::ListObjects + } else if path_ends_with(file, "/store/object.rs") || path_ends_with(file, "/set_disk/ops/object.rs") { + ReadVersionRequestCaller::Object + } else if path_ends_with(file, "/set_disk/core/metadata_read.rs") { + ReadVersionRequestCaller::MetadataRead + } else if path_ends_with(file, "/set_disk/ops/multipart.rs") { + ReadVersionRequestCaller::Multipart + } else if path_ends_with(file, "/disk/disk_store.rs") { + ReadVersionRequestCaller::DiskStore + } else if path_ends_with(file, "/disk/mod.rs") { + ReadVersionRequestCaller::DiskDispatch + } else if path_ends_with(file, "/set_disk/read.rs") { + ReadVersionRequestCaller::SetDiskRead + } else if path_ends_with(file, "/set_disk/core/io_primitives.rs") { + ReadVersionRequestCaller::SetDiskIo + } else if path_ends_with(file, "/services/rebalance/worker.rs") { + ReadVersionRequestCaller::Rebalance + } else if path_ends_with(file, "/core/pools.rs") { + ReadVersionRequestCaller::Decommission + } else if path_ends_with(file, "/tier_free_version_recovery.rs") { + ReadVersionRequestCaller::TierRecovery + } else if path_contains(file, "/bucket/lifecycle/") { + ReadVersionRequestCaller::Lifecycle + } else if path_contains(file, "/cache_value/") { + ReadVersionRequestCaller::CacheValue + } else if path_contains(file, "/ecstore/src/") { + ReadVersionRequestCaller::Other + } else { + ReadVersionRequestCaller::Unattributed + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) fn current_read_version_request_caller() -> Option { + current_read_version_task_context().and_then(|context| context.caller) +} + +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) async fn with_read_version_request_caller(caller: ReadVersionRequestCaller, future: F) -> F::Output +where + F: std::future::Future, +{ + let mut context = current_read_version_task_context().unwrap_or_default(); + context.caller = Some(caller); + with_read_version_task_context(context, future).await +} + +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) async fn with_read_version_request_caller_if_absent(caller: ReadVersionRequestCaller, future: F) -> F::Output +where + F: std::future::Future, +{ + if current_read_version_request_caller().is_some() { + future.await + } else { + with_read_version_request_caller(caller, future).await + } +} + +#[cfg(all(test, feature = "diagnostic-read-version-caller"))] +pub(crate) async fn with_optional_read_version_request_caller(caller: Option, future: F) -> F::Output +where + F: std::future::Future, +{ + match caller { + Some(caller) => with_read_version_request_caller(caller, future).await, + None => future.await, + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +use rustfs_io_metrics::internode_metrics::{ReadVersionAttribution, ReadVersionContextBoundary, ReadVersionEntrySource}; +#[cfg(feature = "diagnostic-read-version-caller")] +use rustfs_io_metrics::s3_http_metrics::{S3HttpOperationContext, current_s3_http_operation_context}; + +#[cfg(feature = "diagnostic-read-version-caller")] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +struct ReadVersionEntryContext { + source: ReadVersionEntrySource, + api_operation: Option, + caller: Option, + // Entry provenance stays fixed; the parent snapshot follows each handoff. + parent_api_operation: Option, + parent_caller: Option, +} + +#[cfg(feature = "diagnostic-read-version-caller")] +impl ReadVersionEntryContext { + fn new( + source: ReadVersionEntrySource, + api_operation: Option, + caller: Option, + ) -> Self { + Self { + source, + api_operation, + caller, + parent_api_operation: api_operation, + parent_caller: caller, + } + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] +struct ReadVersionTaskContext { + caller: Option, + entry: Option, +} + +/// Capture values before spawning or enqueueing; absence is part of the value. +#[cfg(feature = "diagnostic-read-version-caller")] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) struct ReadVersionDiagnosticContext { + pub(crate) api_operation: Option, + pub(crate) caller: Option, + entry: Option, +} + +#[cfg(feature = "diagnostic-read-version-caller")] +tokio::task_local! { + static CURRENT_READ_VERSION_CONTEXT: std::cell::Cell; + static CURRENT_READ_VERSION_BATCH_CONTEXTS: Vec; +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn current_read_version_task_context() -> Option { + CURRENT_READ_VERSION_CONTEXT.try_with(std::cell::Cell::get).ok() +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn with_read_version_task_context(context: ReadVersionTaskContext, future: F) -> impl std::future::Future +where + F: std::future::Future, +{ + CURRENT_READ_VERSION_CONTEXT.scope(std::cell::Cell::new(context), future) +} + +#[cfg(feature = "diagnostic-read-version-caller")] +impl ReadVersionDiagnosticContext { + pub(crate) fn capture() -> Self { + let task_context = current_read_version_task_context(); + Self { + api_operation: current_s3_http_operation_context(), + caller: task_context.and_then(|context| context.caller), + entry: task_context.and_then(|context| context.entry), + } + } + + pub(crate) fn capture_entry(source: ReadVersionEntrySource) -> Self { + let mut context = Self::capture(); + if context.entry.is_none() { + context.entry = Some(ReadVersionEntryContext::new(source, context.api_operation, context.caller)); + context.observe(ReadVersionContextBoundary::Entry); + } + context + } + + pub(crate) fn observe(self, boundary: ReadVersionContextBoundary) { + self.observe_with_mode(boundary, None); + } + + fn observe_with_mode(self, boundary: ReadVersionContextBoundary, mode: Option) { + rustfs_io_metrics::internode_metrics::record_read_version_context_transition( + self.entry.map_or(ReadVersionEntrySource::Unclassified, |entry| entry.source), + boundary, + self.api_operation, + current_s3_http_operation_context(), + self.caller, + current_read_version_request_caller(), + mode, + ); + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +pub fn with_read_version_entry(source: ReadVersionEntrySource, future: F) -> impl std::future::Future +where + F: std::future::Future, +{ + // Box before constructing the scope future so large storage futures do not + // inflate every caller frame. Capture task-local values only when polled. + let future = Box::pin(future); + async move { + let context = ReadVersionDiagnosticContext::capture_entry(source); + with_read_version_task_context( + ReadVersionTaskContext { + caller: context.caller, + entry: context.entry, + }, + future, + ) + .await + } +} + +/// Set a missing caller and entry source in one task-local scope for the object-read boundary. +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) fn with_read_version_caller_and_entry_if_absent( + caller: ReadVersionRequestCaller, + source: ReadVersionEntrySource, + future: F, +) -> impl std::future::Future +where + F: std::future::Future, +{ + let future = Box::pin(future); + async move { + let api_operation = current_s3_http_operation_context(); + let mut task_context = current_read_version_task_context().unwrap_or_default(); + if task_context.caller.is_none() { + task_context.caller = Some(caller); + } + let new_entry = task_context.entry.is_none(); + if new_entry { + task_context.entry = Some(ReadVersionEntryContext::new(source, api_operation, task_context.caller)); + } + let diagnostic = ReadVersionDiagnosticContext { + api_operation, + caller: task_context.caller, + entry: task_context.entry, + }; + let observed = async move { + if new_entry { + diagnostic.observe(ReadVersionContextBoundary::Entry); + } + future.await + }; + with_read_version_task_context(task_context, observed).await + } +} + +/// Capture on the caller before spawning, then restore when the child is polled. +#[cfg(feature = "diagnostic-read-version-caller")] +pub fn with_read_version_spawn_context( + source: ReadVersionEntrySource, + boundary: ReadVersionContextBoundary, + future: F, +) -> impl std::future::Future +where + F: std::future::Future, +{ + let context = ReadVersionDiagnosticContext::capture_entry(source); + let future = Box::pin(future); + with_read_version_diagnostic_context(context, boundary, future) +} + +/// Observe a named entry even when its scope retains an earlier source. +#[cfg(feature = "diagnostic-read-version-caller")] +pub fn observe_read_version_context_boundary(boundary: ReadVersionContextBoundary) { + ReadVersionDiagnosticContext::capture().observe(boundary); +} + +/// The earliest observed storage entry, for diagnostic consumers only. +#[cfg(feature = "diagnostic-read-version-caller")] +pub fn current_read_version_entry_source() -> Option { + current_read_version_task_context().and_then(|context| context.entry.map(|entry| entry.source)) +} + +/// Restore only this item's context and observe the child after restoration. +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) async fn with_read_version_diagnostic_context( + context: ReadVersionDiagnosticContext, + boundary: ReadVersionContextBoundary, + future: F, +) -> F::Output +where + F: std::future::Future, +{ + let observed = async { + context.observe(boundary); + future.await + }; + let mut forwarded_entry = context.entry; + if let Some(entry) = &mut forwarded_entry { + entry.parent_api_operation = context.api_operation; + entry.parent_caller = context.caller; + } + let task_context = ReadVersionTaskContext { + caller: context.caller, + entry: forwarded_entry, + }; + let scoped = with_read_version_task_context(task_context, observed); + rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(context.api_operation, scoped).await +} + +/// A batch has item contexts, not one owner's S3 task-local operation. +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) async fn with_read_version_batch_contexts(contexts: Vec, future: F) -> F::Output +where + F: std::future::Future, +{ + let batch = CURRENT_READ_VERSION_BATCH_CONTEXTS.scope(contexts, future); + let scoped = with_read_version_task_context(ReadVersionTaskContext::default(), batch); + rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(None, scoped).await +} + +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) async fn with_batch_read_version_item_context(index: usize, item_count: usize, future: F) -> F::Output +where + F: std::future::Future, +{ + let context = CURRENT_READ_VERSION_BATCH_CONTEXTS + .try_with(|contexts| (contexts.len() == item_count).then(|| contexts.get(index).copied()).flatten()) + .ok() + .flatten(); + match context { + Some(context) => with_read_version_diagnostic_context(context, ReadVersionContextBoundary::UnaryFallback, future).await, + None => future.await, + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn common_read_version_attribution(values: impl Iterator>) -> ReadVersionAttribution { + values + .map(|value| value.map_or(ReadVersionAttribution::Absent, ReadVersionAttribution::Common)) + .reduce(|common, value| { + if common == value { + common + } else { + ReadVersionAttribution::Mixed + } + }) + .unwrap_or(ReadVersionAttribution::Absent) +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn record_read_version_entry_to_rpc_context(opts: &ReadOptions) { + let context = ReadVersionDiagnosticContext::capture(); + let mode = Some(read_version_request_mode(opts)); + if let Some(entry) = context.entry { + ReadVersionDiagnosticContext { + api_operation: entry.api_operation, + caller: entry.caller, + ..context + } + .observe_with_mode(ReadVersionContextBoundary::EntryToRpc, mode); + ReadVersionDiagnosticContext { + api_operation: entry.parent_api_operation, + caller: entry.parent_caller, + ..context + } + .observe_with_mode(ReadVersionContextBoundary::RemoteRpc, mode); + } else { + context.observe_with_mode(ReadVersionContextBoundary::RemoteRpc, mode); + } +} + #[cfg(test)] use rustfs_io_metrics::internode_metrics::InternodeMetricsSnapshot; @@ -106,7 +508,18 @@ pub(crate) fn record_remote_disk_grpc_read_all_request() { .record_outgoing_request_for_operation_and_backend(INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC); } -pub(crate) fn record_remote_disk_grpc_read_version_request() { +#[cfg(feature = "diagnostic-read-version-caller")] +fn read_version_request_mode(opts: &ReadOptions) -> ReadVersionRequestMode { + match (opts.read_data, opts.healing, opts.incl_free_versions) { + (false, false, false) => ReadVersionRequestMode::MetadataOnly, + (true, false, false) => ReadVersionRequestMode::Data, + (false, true, false) => ReadVersionRequestMode::Healing, + (false, false, true) => ReadVersionRequestMode::IncludeFreeVersions, + _ => ReadVersionRequestMode::Combined, + } +} + +pub(crate) fn record_remote_disk_grpc_read_version_request(_opts: &ReadOptions) { if !rustfs_io_metrics::get_stage_metrics_enabled() { return; } @@ -114,9 +527,19 @@ pub(crate) fn record_remote_disk_grpc_read_version_request() { INTERNODE_OPERATION_GRPC_READ_VERSION, INTERNODE_TRANSPORT_BACKEND_GRPC, ); + #[cfg(feature = "diagnostic-read-version-caller")] + let api_operation = rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_context(); + #[cfg(feature = "diagnostic-read-version-caller")] + record_read_version_request_by_caller( + api_operation, + current_read_version_request_caller().unwrap_or_default(), + read_version_request_mode(_opts), + ); + #[cfg(feature = "diagnostic-read-version-caller")] + record_read_version_entry_to_rpc_context(_opts); } -pub(crate) fn record_remote_disk_grpc_batch_read_version_request() { +pub(crate) fn record_remote_disk_grpc_batch_read_version_request(_opts: &ReadOptions) { if !rustfs_io_metrics::get_stage_metrics_enabled() { return; } @@ -124,6 +547,30 @@ pub(crate) fn record_remote_disk_grpc_batch_read_version_request() { INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, INTERNODE_TRANSPORT_BACKEND_GRPC, ); + #[cfg(feature = "diagnostic-read-version-caller")] + { + let attribution = CURRENT_READ_VERSION_BATCH_CONTEXTS.try_with(|contexts| { + ( + common_read_version_attribution(contexts.iter().map(|context| context.api_operation)), + common_read_version_attribution(contexts.iter().map(|context| context.caller)), + common_read_version_attribution(contexts.iter().map(|context| context.entry.map(|entry| entry.source))), + ) + }); + let (api, caller, source) = attribution.unwrap_or_else(|_| { + let context = ReadVersionDiagnosticContext::capture(); + ( + common_read_version_attribution(std::iter::once(context.api_operation)), + common_read_version_attribution(std::iter::once(context.caller)), + common_read_version_attribution(std::iter::once(context.entry.map(|entry| entry.source))), + ) + }); + rustfs_io_metrics::internode_metrics::record_batch_read_version_request_by_caller( + api, + caller, + source, + read_version_request_mode(_opts), + ); + } } pub(crate) fn record_remote_disk_grpc_batch_read_version_stage(stage: &'static str, duration: Duration) { @@ -303,3 +750,591 @@ pub(crate) fn internode_metrics_snapshot_for_test() -> InternodeMetricsSnapshot pub(crate) fn internode_msgpack_json_decode_error_total_for_test() -> u64 { global_internode_metrics().msgpack_json_decode_error_total_for_test() } + +#[cfg(test)] +mod tests { + #[cfg(feature = "diagnostic-read-version-caller")] + use super::*; + + #[cfg(feature = "diagnostic-read-version-caller")] + fn diagnostic_operation(operation: rustfs_s3_ops::S3Operation) -> S3HttpOperationContext { + let mut request = rustfs_io_metrics::s3_http_metrics::S3HttpRequestGuard::new("GET"); + request.in_scope(|| { + rustfs_io_metrics::record_s3_op(operation); + current_s3_http_operation_context().expect("operation should be captured") + }) + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn combined_caller_and_entry_scope_captures_once_and_restores() { + let operation = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + rustfs_io_metrics::s3_http_metrics::with_s3_http_operation_context(Some(operation), async { + let before = ReadVersionDiagnosticContext::capture(); + let captured = with_read_version_caller_and_entry_if_absent( + ReadVersionRequestCaller::Object, + ReadVersionEntrySource::ObjectRead, + async { ReadVersionDiagnosticContext::capture() }, + ) + .await; + assert_eq!(captured.api_operation, Some(operation)); + assert_eq!(captured.caller, Some(ReadVersionRequestCaller::Object)); + assert_eq!( + captured.entry.map(|entry| (entry.source, entry.caller)), + Some((ReadVersionEntrySource::ObjectRead, Some(ReadVersionRequestCaller::Object))) + ); + assert_eq!(ReadVersionDiagnosticContext::capture(), before); + + let existing_entry = ReadVersionEntryContext::new( + ReadVersionEntrySource::ConfigRead, + Some(operation), + Some(ReadVersionRequestCaller::Peer), + ); + let existing = ReadVersionTaskContext { + caller: Some(ReadVersionRequestCaller::Peer), + entry: Some(existing_entry), + }; + let preserved = with_read_version_task_context(existing, async { + with_read_version_caller_and_entry_if_absent( + ReadVersionRequestCaller::Object, + ReadVersionEntrySource::ObjectRead, + async { ReadVersionDiagnosticContext::capture() }, + ) + .await + }) + .await; + assert_eq!(preserved.caller, Some(ReadVersionRequestCaller::Peer)); + assert_eq!( + preserved.entry.map(|entry| (entry.source, entry.caller)), + Some((ReadVersionEntrySource::ConfigRead, Some(ReadVersionRequestCaller::Peer))) + ); + }) + .await; + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_list_retains_background_entry_through_shared_workers() { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + runtime.block_on(async { + tokio::spawn(with_read_version_entry(ReadVersionEntrySource::RecoveryDispositionMaintenance, async { + let entry = ReadVersionDiagnosticContext::capture(); + assert_eq!(entry.api_operation, None); + assert_eq!(entry.caller, None); + let expected_entry = entry.entry.expect("background entry is observed"); + assert_eq!(expected_entry.source.as_str(), "recovery_disposition_maintenance"); + with_read_version_entry(ReadVersionEntrySource::ListObjects, async { + with_read_version_request_caller(ReadVersionRequestCaller::ListObjects, async { + let parent = ReadVersionDiagnosticContext::capture_entry(ReadVersionEntrySource::ListObjects); + assert_eq!(parent.entry, Some(expected_entry)); + for boundary in [ + ReadVersionContextBoundary::ListProducer, + ReadVersionContextBoundary::ListGather, + ] { + tokio::spawn(with_read_version_diagnostic_context(parent, boundary, async move { + let child = ReadVersionDiagnosticContext::capture(); + assert_eq!(child.api_operation, None); + assert_eq!(child.caller, Some(ReadVersionRequestCaller::ListObjects)); + let child_entry = child.entry.expect("worker retains background provenance"); + assert_eq!(child_entry.source, expected_entry.source); + assert_eq!(child_entry.api_operation, None); + assert_eq!(child_entry.caller, None); + assert_eq!(child_entry.parent_api_operation, None); + assert_eq!(child_entry.parent_caller, Some(ReadVersionRequestCaller::ListObjects)); + })) + .await + .expect("list worker completes"); + } + }) + .await; + }) + .await; + assert_eq!(ReadVersionDiagnosticContext::capture(), entry); + })) + .await + .expect("background task completes"); + assert_eq!(ReadVersionDiagnosticContext::capture().entry, None); + }); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_spawn_scope_captures_parent_and_isolates_absence() { + struct LargeFuture([u8; 64 * 1024]); + impl std::future::Future for LargeFuture { + type Output = ReadVersionDiagnosticContext; + fn poll(self: std::pin::Pin<&mut Self>, _cx: &mut std::task::Context<'_>) -> std::task::Poll { + assert_eq!(self.0[0], 7); + std::task::Poll::Ready(ReadVersionDiagnosticContext::capture()) + } + } + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let put = diagnostic_operation(rustfs_s3_ops::S3Operation::PutObject); + let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + runtime.block_on(async { + let child = rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context( + Some(put), + with_read_version_request_caller( + ReadVersionRequestCaller::Object, + with_read_version_entry(ReadVersionEntrySource::ConfigWrite, async { + let child = with_read_version_spawn_context( + ReadVersionEntrySource::AppPutOwner, + ReadVersionContextBoundary::AppPutOwner, + LargeFuture([7; 64 * 1024]), + ); + assert!(std::mem::size_of_val(&child) < 1024, "spawn scope must not embed the storage future"); + tokio::spawn(child).await.expect("child task") + }), + ), + ) + .await; + assert_eq!(child.api_operation, Some(put)); + assert_eq!(child.caller, Some(ReadVersionRequestCaller::Object)); + assert_eq!(child.entry.expect("earlier source retained").source, ReadVersionEntrySource::ConfigWrite); + let internal = with_read_version_spawn_context( + ReadVersionEntrySource::AppPutOwner, + ReadVersionContextBoundary::AppPutOwner, + LargeFuture([7; 64 * 1024]), + ); + let observed = rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context( + Some(get), + with_read_version_request_caller( + ReadVersionRequestCaller::Peer, + with_read_version_entry(ReadVersionEntrySource::ConfigRead, async { + let observed = internal.await; + assert_eq!(current_s3_http_operation_context(), Some(get)); + assert_eq!(current_read_version_entry_source(), Some(ReadVersionEntrySource::ConfigRead)); + observed + }), + ), + ) + .await; + assert_eq!(observed.api_operation, None); + assert_eq!(observed.caller, None); + assert_eq!( + observed.entry.expect("internal source retained").source, + ReadVersionEntrySource::AppPutOwner + ); + assert_eq!(current_s3_http_operation_context(), None); + assert_eq!(current_read_version_entry_source(), None); + }); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_entry_keeps_large_future_off_the_scope_frame_and_captures_when_polled() { + struct LargeFuture([u8; 64 * 1024]); + impl std::future::Future for LargeFuture { + type Output = ReadVersionDiagnosticContext; + fn poll(self: std::pin::Pin<&mut Self>, _cx: &mut std::task::Context<'_>) -> std::task::Poll { + assert_eq!(self.0[0], 7); + std::task::Poll::Ready(ReadVersionDiagnosticContext::capture()) + } + } + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + let read = with_read_version_entry(ReadVersionEntrySource::ObjectInfoRead, LargeFuture([7; 64 * 1024])); + assert!(std::mem::size_of_val(&read) < 1024, "scope frame must not embed the storage future"); + assert_eq!(current_read_version_entry_source(), None); + let observed = runtime.block_on(rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context( + Some(get), + with_read_version_entry(ReadVersionEntrySource::ConfigRead, read), + )); + assert_eq!(observed.entry.expect("entry captured at poll").source, ReadVersionEntrySource::ConfigRead); + assert_eq!(observed.api_operation, Some(get)); + assert_eq!(current_read_version_entry_source(), None); + assert_eq!(current_s3_http_operation_context(), None); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_named_entry_is_observed_when_earlier_source_is_preserved() { + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("runtime"); + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + for earlier in [None, Some(ReadVersionEntrySource::ConfigRead)] { + let api = earlier.map(|_| get); + let entry = with_read_version_entry(ReadVersionEntrySource::DataUsageCacheRead, async { + observe_read_version_context_boundary(ReadVersionContextBoundary::ScannerCacheLoad); + assert_eq!(current_s3_http_operation_context(), api); + }); + rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(api, async { + match earlier { + Some(source) => with_read_version_entry(source, entry).await, + None => entry.await, + } + }) + .await; + assert_eq!(current_read_version_entry_source(), None); + assert_eq!(current_s3_http_operation_context(), None); + } + }); + }); + let rows = snapshots.snapshot().into_vec(); + for (source, api, state) in [ + ("data_usage_cache_read", "unattributed", "absent"), + ("config_read", "s3:GetObject", "preserved"), + ] { + assert!( + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("boundary") == Some(&"scanner_cache_load") + && labels.get("entry_source") == Some(&source) + && labels.get("parent_api_operation") == Some(&api) + && labels.get("child_api_operation") == Some(&api) + && labels.get("api_context") == Some(&state) + && matches!(value, DebugValue::Counter(1)) + }), + "a named cache entry remains observable with source {source}" + ); + } + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_context_records_absent_internal_entry_and_real_child_loss() { + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + use rustfs_io_metrics::s3_http_metrics::{with_isolated_s3_http_operation_context, with_s3_http_operation_context}; + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + with_read_version_entry(ReadVersionEntrySource::ConfigRead, async { + let context = ReadVersionDiagnosticContext::capture(); + tokio::spawn(with_read_version_diagnostic_context( + context, + ReadVersionContextBoundary::FullWait, + async { + assert_eq!(current_s3_http_operation_context(), None); + record_read_version_entry_to_rpc_context(&ReadOptions::default()); + }, + )) + .await + .expect("internal child should complete"); + }) + .await; + with_s3_http_operation_context( + Some(get), + with_read_version_entry( + ReadVersionEntrySource::ObjectRead, + with_read_version_request_caller(ReadVersionRequestCaller::Object, async { + let context = ReadVersionDiagnosticContext::capture(); + tokio::spawn(with_read_version_diagnostic_context( + context, + ReadVersionContextBoundary::EarlyStop, + async move { + assert_eq!(current_s3_http_operation_context(), Some(get)); + record_read_version_entry_to_rpc_context(&ReadOptions::default()); + with_isolated_s3_http_operation_context(None, async { + record_read_version_entry_to_rpc_context(&ReadOptions::default()); + }) + .await; + let mut missing_caller = + current_read_version_task_context().expect("diagnostic scope should be active"); + missing_caller.caller = None; + with_read_version_task_context(missing_caller, async { + record_read_version_entry_to_rpc_context(&ReadOptions::default()); + }) + .await; + }, + )) + .await + .expect("external child should complete"); + }), + ), + ) + .await; + assert_eq!(ReadVersionDiagnosticContext::capture().entry, None); + record_read_version_entry_to_rpc_context(&ReadOptions::default()); + }) + }); + let rows = snapshots.snapshot().into_vec(); + let observed = |source: &str, state: &str, parent: &str, child: &str| { + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("entry_source") == Some(&source) + && labels.get("boundary") == Some(&"remote_rpc") + && labels.get("api_context") == Some(&state) + && labels.get("parent_api_operation") == Some(&parent) + && labels.get("child_api_operation") == Some(&child) + && labels.get("mode") == Some(&"metadata_only") + && matches!(value, DebugValue::Counter(1)) + }) + }; + assert!(observed("config_read", "absent", "unattributed", "unattributed")); + assert!(observed("object_read", "preserved", "s3:GetObject", "s3:GetObject")); + assert!(observed("object_read", "lost", "s3:GetObject", "unattributed")); + assert!(observed("unclassified", "unobserved", "unattributed", "unattributed")); + assert!( + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("boundary") == Some(&"remote_rpc") + && labels.get("parent_caller") == Some(&"object") + && labels.get("child_caller") == Some(&"unattributed") + && labels.get("caller_context") == Some(&"lost") + && matches!(value, DebugValue::Counter(1)) + }), + "RPC caller must be compared with its direct parent's seed, not an earlier unseeded entry" + ); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn diagnostic_context_restores_ambient_scope_after_pending_item_is_cancelled() { + use std::future::Future; + let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + let parent = ReadVersionDiagnosticContext { + api_operation: Some(get), + caller: Some(ReadVersionRequestCaller::Object), + entry: Some(ReadVersionEntryContext::new( + ReadVersionEntrySource::ObjectRead, + Some(get), + Some(ReadVersionRequestCaller::Object), + )), + }; + let internal = ReadVersionDiagnosticContext { + api_operation: None, + caller: None, + entry: Some(ReadVersionEntryContext::new(ReadVersionEntrySource::ConfigRead, None, None)), + }; + with_read_version_diagnostic_context(parent, ReadVersionContextBoundary::Entry, async { + let mut pending = Box::pin(with_read_version_diagnostic_context( + internal, + ReadVersionContextBoundary::UnaryFallback, + async { + assert_eq!(ReadVersionDiagnosticContext::capture(), internal); + std::future::pending::<()>().await; + }, + )); + std::future::poll_fn(|cx| { + assert!(pending.as_mut().poll(cx).is_pending()); + std::task::Poll::Ready(()) + }) + .await; + assert_eq!( + ReadVersionDiagnosticContext::capture(), + parent, + "a pending item must not leak into its poller's scope" + ); + drop(pending); + assert_eq!( + ReadVersionDiagnosticContext::capture(), + parent, + "cancellation must retain the ambient request" + ); + }) + .await; + assert_eq!(ReadVersionDiagnosticContext::capture().entry, None); + assert_eq!(current_s3_http_operation_context(), None); + assert_eq!(current_read_version_request_caller(), None); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_batch_attributes_each_dimension_and_counts_one_rpc() { + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + let list = diagnostic_operation(rustfs_s3_ops::S3Operation::ListObjectsV2); + let context = |api, caller, source| ReadVersionDiagnosticContext { + api_operation: api, + caller, + entry: Some(ReadVersionEntryContext::new(source, api, caller)), + }; + let external = context(Some(get), Some(ReadVersionRequestCaller::Object), ReadVersionEntrySource::ObjectRead); + let internal = context(None, None, ReadVersionEntrySource::ConfigRead); + let internal_object = context(None, Some(ReadVersionRequestCaller::Object), ReadVersionEntrySource::ConfigRead); + let heal = context(Some(get), Some(ReadVersionRequestCaller::Heal), ReadVersionEntrySource::ObjectRead); + let other_api = context(Some(list), Some(ReadVersionRequestCaller::Object), ReadVersionEntrySource::ObjectRead); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + for contexts in [ + vec![external, external], + vec![external, internal], + vec![external, internal_object], + vec![external, heal], + vec![external, other_api], + vec![internal, internal], + ] { + with_read_version_batch_contexts(contexts, async { + assert_eq!(current_s3_http_operation_context(), None); + assert_eq!(current_read_version_request_caller(), None); + record_remote_disk_grpc_batch_read_version_request(&ReadOptions { + read_data: true, + ..Default::default() + }); + }) + .await; + } + }) + }); + let labels = snapshots + .snapshot() + .into_vec() + .into_iter() + .filter(|(key, _, _, _)| { + key.key().name() == "rustfs_system_network_internode_batch_read_version_requests_by_caller_total" + }) + .map(|(key, _, _, value)| { + assert!(matches!(value, DebugValue::Counter(1)), "batch must count one RPC, not its items"); + let labels = key + .key() + .labels() + .map(|label| (label.key().to_owned(), label.value().to_owned())) + .collect::>(); + assert_eq!(labels["mode"], "data"); + (labels["api_operation"].clone(), labels["caller"].clone(), labels["entry_source"].clone()) + }) + .collect::>(); + assert_eq!(labels.len(), 6); + for (api, caller, source) in [ + ("s3:GetObject", "object", "object_read"), + ("mixed", "mixed", "mixed"), + ("mixed", "object", "mixed"), + ("s3:GetObject", "mixed", "object_read"), + ("mixed", "object", "object_read"), + ("unattributed", "unattributed", "config_read"), + ] { + assert!(labels.contains(&(api.to_owned(), caller.to_owned(), source.to_owned()))); + } + } + + #[test] + #[cfg(feature = "diagnostic-read-version-caller")] + fn read_version_request_mode_uses_only_bounded_read_flags() { + assert_eq!(read_version_request_mode(&ReadOptions::default()), ReadVersionRequestMode::MetadataOnly); + assert_eq!( + read_version_request_mode(&ReadOptions { + read_data: true, + ..Default::default() + }), + ReadVersionRequestMode::Data + ); + assert_eq!( + read_version_request_mode(&ReadOptions { + healing: true, + ..Default::default() + }), + ReadVersionRequestMode::Healing + ); + assert_eq!( + read_version_request_mode(&ReadOptions { + incl_free_versions: true, + ..Default::default() + }), + ReadVersionRequestMode::IncludeFreeVersions + ); + assert_eq!( + read_version_request_mode(&ReadOptions { + read_data: true, + healing: true, + ..Default::default() + }), + ReadVersionRequestMode::Combined + ); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn read_version_request_caller_classification_is_separator_agnostic() { + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/store/peer.rs"), + ReadVersionRequestCaller::Peer + ); + assert_eq!( + read_version_request_caller_from_location(r"C:\repo\crates\ecstore\src\store\peer.rs"), + ReadVersionRequestCaller::Peer + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/notstore/peer.rs"), + ReadVersionRequestCaller::Other + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/store/object.rs"), + ReadVersionRequestCaller::Object + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/set_disk/core/metadata_read.rs"), + ReadVersionRequestCaller::MetadataRead + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/set_disk/ops/heal.rs"), + ReadVersionRequestCaller::Heal + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/set_disk/ops/multipart.rs"), + ReadVersionRequestCaller::Multipart + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/disk/disk_store.rs"), + ReadVersionRequestCaller::DiskStore + ); + assert_eq!( + read_version_request_caller_from_location(r"C:\repo\crates\ecstore\src\bucket\lifecycle\worker.rs"), + ReadVersionRequestCaller::Lifecycle + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/other/module.rs"), + ReadVersionRequestCaller::Other + ); + assert_eq!( + read_version_request_caller_from_location("/repo/external/module.rs"), + ReadVersionRequestCaller::Unattributed + ); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn read_version_request_caller_context_is_scoped_to_the_read_future() { + assert_eq!(current_read_version_request_caller(), None); + let caller = + with_read_version_request_caller(ReadVersionRequestCaller::Peer, async { current_read_version_request_caller() }) + .await; + assert_eq!(caller, Some(ReadVersionRequestCaller::Peer)); + assert_eq!(current_read_version_request_caller(), None); + } +} diff --git a/crates/ecstore/src/config/com.rs b/crates/ecstore/src/config/com.rs index fa0a9eeb4..ab51073c8 100644 --- a/crates/ecstore/src/config/com.rs +++ b/crates/ecstore/src/config/com.rs @@ -642,17 +642,20 @@ where >, { let h = HeaderMap::new(); - let mut rd = api - .get_object_reader(RUSTFS_META_BUCKET, file, None, h, opts) - .await - .map_err(|err| { - if err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _)) { - Error::ConfigNotFound - } else { - warn!("read_config_with_metadata: err: {:?}, file: {}", err, file); - err - } - })?; + let read = api.get_object_reader(RUSTFS_META_BUCKET, file, None, h, opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ConfigRead, + read, + ); + let mut rd = read.await.map_err(|err| { + if err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _)) { + Error::ConfigNotFound + } else { + warn!("read_config_with_metadata: err: {:?}, file: {}", err, file); + err + } + })?; let data = if let Some(max_bytes) = max_bytes { let object_size = usize::try_from(rd.object_info.size).map_err(|_| Error::CorruptedFormat)?; @@ -875,7 +878,18 @@ where >, { let mut put_data = PutObjReader::from_vec(data); - match api.put_object(RUSTFS_META_BUCKET, file, &mut put_data, opts).await { + let write = api.put_object(RUSTFS_META_BUCKET, file, &mut put_data, opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let write = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ConfigWrite, + async { + crate::cluster::rpc::runtime_sources::observe_read_version_context_boundary( + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::ConfigWrite, + ); + write.await + }, + ); + match write.await { Ok(object_info) => Ok(object_info), Err(err) => { if log_error { @@ -2908,7 +2922,7 @@ mod tests { use rustfs_config::{ DEFAULT_DELIMITER, ENABLE_KEY, EnableState, HEAL_BITROT_CYCLE, HEAL_SUB_SYS, MYSQL_DSN_STRING, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_TABLE, SCANNER_CYCLE, SCANNER_IDLE_MODE, SCANNER_KEYS, SCANNER_SPEED, - SCANNER_SUB_SYS, + SCANNER_SUB_SYS, SCANNER_WALKERS_PER_DISK, }; #[tokio::test] @@ -2938,6 +2952,11 @@ mod tests { #[derive(Debug, Default)] struct MetadataWriteStore { error: Option, + #[cfg(feature = "diagnostic-read-version-caller")] + observations: + std::sync::Mutex, Option<&'static str>)>>, + #[cfg(feature = "diagnostic-read-version-caller")] + pending: bool, } #[async_trait::async_trait] @@ -2968,7 +2987,20 @@ mod tests { _data: &mut Self::PutObjectReader, _opts: &Self::ObjectOptions, ) -> core::result::Result { - Err(self.error.clone().expect("test store error should be configured")) + #[cfg(feature = "diagnostic-read-version-caller")] + { + self.observations.lock().expect("observations lock").push(( + crate::cluster::rpc::runtime_sources::current_read_version_entry_source(), + rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(), + )); + if self.pending { + std::future::pending::<()>().await; + } + } + match self.error.clone() { + Some(error) => Err(error), + None => Ok(ObjectInfo::default()), + } } } @@ -2976,6 +3008,10 @@ mod tests { async fn save_config_preserves_retryable_system_volume_errors() { let store = Arc::new(MetadataWriteStore { error: Some(Error::BucketNotFound(RUSTFS_META_BUCKET.to_string())), + #[cfg(feature = "diagnostic-read-version-caller")] + observations: Default::default(), + #[cfg(feature = "diagnostic-read-version-caller")] + pending: false, }); let error = save_config_with_opts_inner(store, "buckets/example/.metadata.bin", Vec::new(), &ObjectOptions::default(), false) @@ -2987,6 +3023,59 @@ mod tests { Error::InsufficientWriteQuorum(RUSTFS_META_BUCKET.to_string(), "buckets/example/.metadata.bin".to_string()) ); } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation() { + use crate::cluster::rpc::runtime_sources::{current_read_version_entry_source, with_read_version_entry}; + use rustfs_io_metrics::internode_metrics::ReadVersionEntrySource; + use rustfs_io_metrics::s3_http_metrics::{S3HttpRequestGuard, current_s3_http_operation_context}; + + let mut guard = S3HttpRequestGuard::new("PUT"); + let put = guard.in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::PutObject); + current_s3_http_operation_context().expect("captured PutObject") + }); + for earlier in [None, Some(ReadVersionEntrySource::TierConfigReload)] { + for api in [None, Some(put)] { + for case in ["success", "error", "cancelled"] { + let store = Arc::new(MetadataWriteStore { + error: (case == "error").then_some(Error::VolumeNotFound), + pending: case == "cancelled", + ..Default::default() + }); + let opts = ObjectOptions::default(); + let write = async { + let result = tokio::time::timeout( + std::time::Duration::from_secs(1), + save_config_with_opts_inner(store.clone(), "buckets/diagnostic/state", Vec::new(), &opts, false), + ) + .await; + match case { + "success" => assert!(result.expect("write completes").is_ok()), + "error" => { + assert!(matches!(result.expect("error completes"), Err(Error::InsufficientWriteQuorum(_, _)))) + } + _ => assert!(result.is_err(), "pending write is cancelled"), + } + }; + rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(api, async { + match earlier { + Some(source) => with_read_version_entry(source, write).await, + None => write.await, + } + }) + .await; + assert_eq!( + *store.observations.lock().expect("observations lock"), + vec![(Some(earlier.unwrap_or(ReadVersionEntrySource::ConfigWrite)), api.map(|_| "s3:PutObject"))] + ); + assert_eq!(current_read_version_entry_source(), None, "write scope restores: {case}"); + assert_eq!(current_s3_http_operation_context(), None, "API scope restores: {case}"); + } + } + } + } use rustfs_lock::client::LockClient; use rustfs_lock::client::local::LocalClient; use rustfs_lock::{LockError, LockInfo, LockResponse, LockStats}; @@ -4181,6 +4270,8 @@ mod tests { let value = match *key { SCANNER_SPEED => "fast", SCANNER_IDLE_MODE => "false", + // "1" is this key's default, which the diff encoding omits. + SCANNER_WALKERS_PER_DISK => "2", _ => "1", }; scanner_kvs.insert((*key).to_string(), value.to_string()); @@ -4202,6 +4293,8 @@ mod tests { let expected = match *key { SCANNER_SPEED => "fast", SCANNER_IDLE_MODE => "false", + // "1" is this key's default, which the diff encoding omits. + SCANNER_WALKERS_PER_DISK => "2", _ => "1", }; assert_eq!(decoded_scanner.get(key), expected, "scanner key {key} should survive roundtrip"); diff --git a/crates/ecstore/src/config/scanner.rs b/crates/ecstore/src/config/scanner.rs index 140f7c186..7d3f6e038 100644 --- a/crates/ecstore/src/config/scanner.rs +++ b/crates/ecstore/src/config/scanner.rs @@ -17,11 +17,11 @@ use rustfs_config::{ DEFAULT_SCANNER_BITROT_CYCLE_SECS, DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS, DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES, DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS, DEFAULT_SCANNER_CYCLE_MAX_OBJECTS, DEFAULT_SCANNER_IDLE_MODE, DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS, DEFAULT_SCANNER_SPEED, - DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, SCANNER_ALERT_EXCESS_FOLDERS, SCANNER_ALERT_EXCESS_VERSION_SIZE, - SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, - SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE, - SCANNER_MAX_CONCURRENT_DISK_SCANS, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_SPEED, SCANNER_START_DELAY, - SCANNER_YIELD_EVERY_N_OBJECTS, + DEFAULT_SCANNER_WALKERS_PER_DISK, DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, SCANNER_ALERT_EXCESS_FOLDERS, + SCANNER_ALERT_EXCESS_VERSION_SIZE, SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, + SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, + SCANNER_IDLE_MODE, SCANNER_MAX_CONCURRENT_DISK_SCANS, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_SPEED, + SCANNER_START_DELAY, SCANNER_WALKERS_PER_DISK, SCANNER_YIELD_EVERY_N_OBJECTS, }; use std::sync::LazyLock; @@ -92,6 +92,11 @@ pub static DEFAULT_KVS: LazyLock = LazyLock::new(|| { value: DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS.to_string(), hidden_if_empty: false, }, + KV { + key: SCANNER_WALKERS_PER_DISK.to_owned(), + value: DEFAULT_SCANNER_WALKERS_PER_DISK.to_string(), + hidden_if_empty: false, + }, KV { key: SCANNER_YIELD_EVERY_N_OBJECTS.to_owned(), value: DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS.to_string(), diff --git a/crates/ecstore/src/core/pools.rs b/crates/ecstore/src/core/pools.rs index f807eb603..a79f08956 100644 --- a/crates/ecstore/src/core/pools.rs +++ b/crates/ecstore/src/core/pools.rs @@ -3691,6 +3691,55 @@ fn activation_pool_meta_lock_error(err: rustfs_lock::LockError) -> Error { } } +fn rebalance_worker_read_lock_error(err: rustfs_lock::LockError, object: &str) -> Error { + match err { + rustfs_lock::LockError::QuorumNotReached { required, achieved } => Error::NamespaceLockQuorumUnavailable { + mode: "read", + bucket: RUSTFS_META_BUCKET.to_string(), + object: object.to_string(), + required, + achieved, + }, + other => Error::Lock(other), + } +} + +/// Read-only admission to an already durable rebalance run. Keeping this +/// separate from the activation writer prevents admission from authorizing a +/// pool or rebalance metadata write under shared namespace locks. +pub(crate) struct PoolRebalanceWorkerFence { + pool_meta_guard: rustfs_lock::NamespaceLockGuard, + rebalance_meta_guard: rustfs_lock::NamespaceLockGuard, + fleet_proof: Option, +} + +impl PoolRebalanceWorkerFence { + pub(crate) fn set_fleet_proof( + &mut self, + fleet_proof: Option, + ) { + self.fleet_proof = fleet_proof; + } + + pub(crate) fn ensure_held(&self) -> Result<()> { + if self.pool_meta_guard.is_lock_lost() + || self.pool_meta_guard.is_released() + || self.rebalance_meta_guard.is_lock_lost() + || self.rebalance_meta_guard.is_released() + { + return Err(Error::other("worker admission lock lost before rebalance activation")); + } + if self + .fleet_proof + .as_ref() + .is_some_and(|proof| !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof)) + { + return Err(Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)); + } + Ok(()) + } +} + pub(crate) struct PoolRebalanceActivationFence { pool_meta_guard: Arc, rebalance_meta_guard: Arc, @@ -3762,6 +3811,35 @@ tokio::task_local! { pub(crate) static REBALANCE_ACTIVATION_LOCK_ATTEMPT: Arc; } +pub(crate) async fn acquire_pool_rebalance_worker_locks(pool: Arc) -> Result +where + S: crate::storage_api_contracts::namespace::NamespaceLocking< + Error = Error, + NamespaceLock = rustfs_lock::NamespaceLockWrapper, + >, +{ + // Match activation and entry admission: rebalance.bin -> pool.bin. Joining + // the same durable run must coexist with entries retaining their run fence. + let rebalance_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, REBAL_META_NAME).await?; + #[cfg(test)] + let _ = REBALANCE_ACTIVATION_LOCK_ATTEMPT.try_with(|attempted| attempted.notify_one()); + let rebalance_meta_guard = rebalance_meta_lock + .get_read_lock(get_lock_acquire_timeout()) + .await + .map_err(|err| rebalance_worker_read_lock_error(err, REBAL_META_NAME))?; + let pool_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, POOL_META_NAME).await?; + let pool_meta_guard = pool_meta_lock + .get_read_lock(get_lock_acquire_timeout()) + .await + .map_err(|err| rebalance_worker_read_lock_error(err, POOL_META_NAME))?; + + Ok(PoolRebalanceWorkerFence { + pool_meta_guard, + rebalance_meta_guard, + fleet_proof: None, + }) +} + pub(crate) async fn acquire_pool_rebalance_activation_locks( pool: Arc, fleet_proof: Option, @@ -6685,12 +6763,12 @@ fn pool_meta_cas_options(object: &str, token: &PoolMetaCasToken, fence: &PoolMet fence.ensure_held()?; let mut opts = ObjectOptions { max_parity: true, - write_completion: crate::object_api::WriteCompletion::TailDrained, no_lock: true, http_preconditions: Some(pool_meta_cas_preconditions(token, object)?), ..Default::default() }; fence.add_to_options(&mut opts, object); + opts.write_completion = crate::object_api::WriteCompletion::TailDrainedWithOwnerRepair; Ok(opts) } @@ -6718,6 +6796,72 @@ async fn execute_pool_meta_object_cas( fence: &PoolMetaPersistenceFence<'_>, phase: &'static str, ) -> PoolMetaCasWriteOutcome +where + S: EcstoreObjectIO, +{ + // Bound the CAS state retained by callers, including ordinary metadata + // transactions that never enter identity repair. Large inline retry states + // otherwise propagate through startup and cross-crate storage futures. + Box::pin(async move { + if object != POOL_META_IDENTITY_NAME { + return execute_pool_meta_object_cas_once(pool, object, data, opts, fence, phase).await; + } + // Keep the exact identity owned by this transaction. A quorum read alone + // cannot discharge repair responsibility for a missing disk replica. + let identity = data; + let mut outcome = execute_pool_meta_object_cas_once(pool.clone(), object, identity.clone(), opts, fence, phase).await; + for delay_secs in [1, 2, 4] { + if !outcome.source.as_deref().is_some_and(|err| { + matches!(err, Error::Io(inner) if inner.get_ref().is_some_and(|source| source.is::())) + }) { + break; + } + // The caller's transaction arm remains live across reconnect waits, + // verification and repair. Cancellation must retain its sticky gate. + tokio::time::sleep(std::time::Duration::from_secs(delay_secs)).await; + let repair_options = async { + fence.ensure_held()?; + let (current, info) = read_config_no_lock_preserve_empty_with_metadata(pool.clone(), object).await?; + if current != identity { + return Err(Error::PreconditionFailed); + } + let token = info + .etag + .filter(|etag| !etag.trim().is_empty()) + .map(PoolMetaCasToken::Existing) + .unwrap_or(PoolMetaCasToken::Unsafe); + pool_meta_cas_options(object, &token, fence) + } + .await; + outcome = match repair_options { + Ok(opts) => execute_pool_meta_object_cas_once(pool.clone(), object, identity.clone(), opts, fence, phase).await, + Err(err) => PoolMetaCasWriteOutcome::not_dispatched(err), + }; + // A rejected repair cannot turn the preceding committed write into a + // harmless CAS conflict that the outer loop could accept on readback. + outcome.may_have_mutated = true; + outcome.result = outcome.result.map_err(|err| { + let original = outcome.source.get_or_insert_with(|| Arc::new(err)); + Error::other(pool_metadata_error( + crate::error::PoolMetadataFailure::TransactionUnknown, + phase, + Some(Arc::clone(original)), + )) + }); + } + outcome + }) + .await +} + +async fn execute_pool_meta_object_cas_once( + pool: Arc, + object: &str, + data: Vec, + opts: ObjectOptions, + fence: &PoolMetaPersistenceFence<'_>, + phase: &'static str, +) -> PoolMetaCasWriteOutcome where S: EcstoreObjectIO, { @@ -6735,7 +6879,11 @@ where "payload_sha256": rustfs_utils::crypto::hex(Sha256::digest(&data)), "if_match": opts.http_preconditions.as_ref().and_then(|p| p.if_match.as_deref()), "if_none_match": opts.http_preconditions.as_ref().and_then(|p| p.if_none_match.as_deref()), - "tail_drained": opts.write_completion == crate::object_api::WriteCompletion::TailDrained, + "tail_drained": matches!( + opts.write_completion, + crate::object_api::WriteCompletion::TailDrained + | crate::object_api::WriteCompletion::TailDrainedWithOwnerRepair + ), "no_lock": opts.no_lock, }) }); @@ -12430,6 +12578,18 @@ impl ECStore { Ok(pool_meta) } + pub(crate) async fn load_runtime_pool_meta_under_worker_fence( + &self, + write_state: &mut PoolMetaWriteState, + worker_fence: &PoolRebalanceWorkerFence, + operation: &str, + ) -> Result { + worker_fence.ensure_held()?; + let pool_meta = self.load_runtime_pool_meta_observing(write_state, operation).await?; + worker_fence.ensure_held()?; + Ok(pool_meta) + } + pub(crate) async fn load_runtime_pool_meta_under_activation_fence( &self, write_state: &mut PoolMetaWriteState, @@ -24216,6 +24376,335 @@ mod pools_tests { identity: StdMutex, String)>>, } + #[test] + fn pool_meta_cas_future_remains_stack_bounded() { + let pool = Arc::new(PartialPoolMetaWriteStorage::default()); + let fence = PoolMetaPersistenceFence::OfflineRecovery; + let future = super::execute_pool_meta_object_cas( + pool.clone(), + POOL_META_IDENTITY_NAME, + b"owned-identity".to_vec(), + ObjectOptions::default(), + &fence, + "identity_cas", + ); + let original = super::execute_pool_meta_object_cas_once( + pool, + POOL_META_IDENTITY_NAME, + b"owned-identity".to_vec(), + ObjectOptions::default(), + &fence, + "identity_cas", + ); + let future_size = std::mem::size_of_val(&future); + let original_size = std::mem::size_of_val(&original); + assert!( + future_size <= 4 * 1024, + "pool metadata CAS future must remain stack-bounded; measured {future_size} bytes versus {original_size} without repair" + ); + } + + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + enum IdentityRepairFault { + Healthy, + Once, + Always, + Changed, + Unreadable, + TypedReadFailure, + MissingEtag, + EmptyEtag, + Reject, + Unknown, + BlockBackoff, + BlockRead, + BlockWrite, + } + + #[derive(Debug)] + struct IdentityRepairStorage { + inner: PartialPoolMetaWriteStorage, + fault: IdentityRepairFault, + writes: AtomicUsize, + reads: AtomicUsize, + blocked: tokio::sync::Notify, + } + + impl IdentityRepairStorage { + fn new(fault: IdentityRepairFault) -> Arc { + Arc::new(Self { + inner: PartialPoolMetaWriteStorage::default(), + fault, + writes: AtomicUsize::new(0), + reads: AtomicUsize::new(0), + blocked: tokio::sync::Notify::new(), + }) + } + } + + #[async_trait::async_trait] + impl ObjectIO for IdentityRepairStorage { + type Error = Error; + type RangeSpec = HTTPRangeSpec; + type HeaderMap = http::HeaderMap; + type ObjectOptions = ObjectOptions; + type ObjectInfo = ObjectInfo; + type GetObjectReader = crate::object_api::GetObjectReader; + type PutObjectReader = crate::object_api::PutObjReader; + + async fn get_object_reader( + &self, + bucket: &str, + object: &str, + range: Option, + headers: http::HeaderMap, + opts: &ObjectOptions, + ) -> std::result::Result { + self.reads.fetch_add(1, Ordering::SeqCst); + if self.fault == IdentityRepairFault::Unreadable { + return Err(Error::ErasureReadQuorum); + } + if self.fault == IdentityRepairFault::TypedReadFailure { + return Err(Error::other(super::pool_metadata_error( + crate::error::PoolMetadataFailure::ReadUnavailable, + "identity_repair_read", + Some(Arc::new(Error::ErasureReadQuorum)), + ))); + } + if self.fault == IdentityRepairFault::BlockRead { + self.blocked.notify_one(); + std::future::pending::<()>().await; + } + let mut reader = self.inner.get_object_reader(bucket, object, range, headers, opts).await?; + match self.fault { + IdentityRepairFault::Changed => { + reader.stream = Box::new(Cursor::new(b"other-identity".to_vec())); + reader.object_info.size = 14; + } + IdentityRepairFault::MissingEtag => reader.object_info.etag = None, + IdentityRepairFault::EmptyEtag => reader.object_info.etag = Some(" ".to_owned()), + _ => {} + } + Ok(reader) + } + + async fn put_object( + &self, + bucket: &str, + object: &str, + data: &mut Self::PutObjectReader, + opts: &ObjectOptions, + ) -> std::result::Result { + let attempt = self.writes.fetch_add(1, Ordering::SeqCst); + assert_eq!(opts.write_completion, crate::object_api::WriteCompletion::TailDrainedWithOwnerRepair); + assert!(opts.max_parity && opts.no_lock); + if attempt > 0 { + assert_eq!( + opts.http_preconditions + .as_ref() + .and_then(HTTPPreconditions::if_none_match_value), + None + ); + assert_eq!( + opts.http_preconditions.as_ref().and_then(HTTPPreconditions::if_match_value), + self.inner + .identity + .lock() + .expect("identity lock") + .as_ref() + .map(|(_, etag)| etag.as_str()), + "repair must use the observed post-commit CAS token", + ); + if self.fault == IdentityRepairFault::Reject { + return Err(Error::PreconditionFailed); + } + if self.fault == IdentityRepairFault::BlockWrite { + self.blocked.notify_one(); + std::future::pending::<()>().await; + } + } + let info = self.inner.put_object(bucket, object, data, opts).await?; + if self.fault == IdentityRepairFault::Unknown { + return Err(Error::Timeout); + } + if self.fault != IdentityRepairFault::Healthy && (attempt == 0 || self.fault == IdentityRepairFault::Always) { + if self.fault == IdentityRepairFault::BlockBackoff { + self.blocked.notify_one(); + } + return Err(Error::other(crate::set_disk::SystemMetadataRepairRequired)); + } + Ok(info) + } + } + + #[tokio::test(start_paused = true)] + async fn pool_identity_owner_repair_preserves_cas_and_fail_closed_boundaries() { + for (fault, object, expected_writes, expected_reads, succeeds) in [ + (IdentityRepairFault::Healthy, POOL_META_IDENTITY_NAME, 1, 0, true), + (IdentityRepairFault::Once, POOL_META_IDENTITY_NAME, 2, 1, true), + (IdentityRepairFault::Always, POOL_META_IDENTITY_NAME, 4, 3, false), + (IdentityRepairFault::Changed, POOL_META_IDENTITY_NAME, 1, 1, false), + (IdentityRepairFault::Unreadable, POOL_META_IDENTITY_NAME, 1, 1, false), + (IdentityRepairFault::TypedReadFailure, POOL_META_IDENTITY_NAME, 1, 1, false), + (IdentityRepairFault::MissingEtag, POOL_META_IDENTITY_NAME, 1, 1, false), + (IdentityRepairFault::EmptyEtag, POOL_META_IDENTITY_NAME, 1, 1, false), + (IdentityRepairFault::Reject, POOL_META_IDENTITY_NAME, 2, 1, false), + (IdentityRepairFault::Unknown, POOL_META_IDENTITY_NAME, 1, 0, false), + (IdentityRepairFault::Once, POOL_META_NAME, 1, 0, false), + ] { + let pool = IdentityRepairStorage::new(fault); + let state = super::PoolMetaWriteState::default(); + let mut transaction = state.arm_transaction(); + let started = tokio::time::Instant::now(); + let result = save_pool_meta_object_cas( + pool.clone(), + object, + b"owned-identity".to_vec(), + &PoolMetaCasToken::Missing, + &PoolMetaPersistenceFence::OfflineRecovery, + "identity_cas", + &mut transaction, + ) + .await; + assert_eq!(result.is_ok(), succeeds, "{fault:?}: {result:?}"); + let original_source = result + .as_ref() + .err() + .and_then(|err| err.pool_metadata_failure()) + .and_then(|failure| failure.source.clone()); + if fault == IdentityRepairFault::TypedReadFailure { + assert_eq!( + original_source + .as_ref() + .expect("read failure source") + .pool_metadata_failure() + .expect("retain the original read context") + .kind, + crate::error::PoolMetadataFailure::ReadUnavailable + ); + } + if succeeds { + transaction.disarm(); + } else { + assert_eq!( + result + .expect_err("failed repair") + .pool_metadata_failure() + .expect("typed uncertainty") + .kind, + crate::error::PoolMetadataFailure::TransactionUnknown, + "{fault:?}" + ); + } + drop(transaction); + let status = state.ensure_write_safe("after repair"); + assert_eq!(status.is_ok(), succeeds, "{fault:?}"); + if let Err(blocked) = status { + let source = blocked + .pool_metadata_failure() + .expect("blocked transaction context") + .source + .as_ref() + .expect("blocked transaction source"); + assert!( + Arc::ptr_eq(source, original_source.as_ref().expect("caller failure source")), + "caller and latch must retain the same original cause: {fault:?}" + ); + } + assert_eq!(pool.writes.load(Ordering::SeqCst), expected_writes, "{fault:?}"); + assert_eq!(pool.reads.load(Ordering::SeqCst), expected_reads, "{fault:?}"); + if fault == IdentityRepairFault::Always { + assert_eq!(started.elapsed(), StdDuration::from_secs(7), "repair backoff must be bounded"); + } + } + } + + #[tokio::test(start_paused = true)] + async fn pool_identity_owner_repair_cancellation_keeps_the_transaction_armed() { + for fault in [ + IdentityRepairFault::BlockBackoff, + IdentityRepairFault::BlockRead, + IdentityRepairFault::BlockWrite, + ] { + let pool = IdentityRepairStorage::new(fault); + let state = Arc::new(tokio::sync::Mutex::new(super::PoolMetaWriteState::default())); + let task = tokio::spawn({ + let pool = pool.clone(); + let state = state.clone(); + async move { + let state = state.lock().await; + let mut transaction = state.arm_transaction(); + save_pool_meta_object_cas( + pool, + POOL_META_IDENTITY_NAME, + b"owned-identity".to_vec(), + &PoolMetaCasToken::Missing, + &PoolMetaPersistenceFence::OfflineRecovery, + "identity_cas", + &mut transaction, + ) + .await + } + }); + pool.blocked.notified().await; + task.abort(); + assert!(task.await.expect_err("cancelled repair").is_cancelled()); + let state = state.lock().await; + let error = state + .ensure_write_safe("cancelled repair") + .expect_err("partial identity remains blocked"); + assert_eq!(error.pool_metadata_failure().expect("typed uncertainty").phase, "identity_cas"); + assert_eq!(state.active_transactions.load(Ordering::SeqCst), 0); + assert!(pool.inner.identity.lock().expect("identity lock").is_some()); + } + } + + #[tokio::test] + async fn pool_identity_owner_repair_stops_after_fence_loss() { + let client = Arc::new(LocalClient::with_manager(Arc::new(GlobalLockManager::new()))); + let lock = NamespaceLock::with_clients_and_quorum("identity-repair-fence".to_string(), vec![client], 1); + let request = LockRequest::new( + ObjectKey::new(super::RUSTFS_META_BUCKET, POOL_META_NAME), + LockType::Exclusive, + "identity-repair-owner", + ) + .with_acquire_timeout(StdDuration::from_secs(1)) + .with_ttl(StdDuration::from_millis(50)) + .with_refresh_interval(StdDuration::from_millis(50)); + let owner = Arc::new( + lock.acquire_guard(&request) + .await + .expect("acquire fence") + .expect("available fence"), + ); + let fence = PoolMetaPersistenceFence::Owned(owner.clone()); + let pool = IdentityRepairStorage::new(IdentityRepairFault::Once); + let state = super::PoolMetaWriteState::default(); + let mut transaction = state.arm_transaction(); + let error = save_pool_meta_object_cas( + pool.clone(), + POOL_META_IDENTITY_NAME, + b"owned-identity".to_vec(), + &PoolMetaCasToken::Missing, + &fence, + "identity_cas", + &mut transaction, + ) + .await + .expect_err("expired owner must not repair"); + assert!(owner.is_lock_lost()); + assert_eq!( + error.pool_metadata_failure().expect("typed uncertainty").kind, + crate::error::PoolMetadataFailure::TransactionUnknown + ); + assert_eq!(pool.writes.load(Ordering::SeqCst), 1); + assert_eq!(pool.reads.load(Ordering::SeqCst), 0); + drop(transaction); + state + .ensure_write_safe("lost repair fence") + .expect_err("lost fence retains the write gate"); + } + #[test] fn decommission_target_gate_retry_is_bounded() { assert!(!decommission_capacity_target_gate_retry_exhausted(0)); @@ -24692,7 +25181,7 @@ mod pools_tests { _ => panic!("phase fixture must receive valid pool metadata"), }; assert!(opts.max_parity && opts.no_lock); - assert_eq!(opts.write_completion, crate::object_api::WriteCompletion::TailDrained); + assert_eq!(opts.write_completion, crate::object_api::WriteCompletion::TailDrainedWithOwnerRepair); self.trace.calls.lock().expect("trace lock").push(WriteCall { pool: self.pool, phase, diff --git a/crates/ecstore/src/data_movement/mod.rs b/crates/ecstore/src/data_movement/mod.rs index 6370da0eb..c2f888937 100644 --- a/crates/ecstore/src/data_movement/mod.rs +++ b/crates/ecstore/src/data_movement/mod.rs @@ -893,6 +893,8 @@ fn is_data_movement_rewritten_metadata(object_info: &ObjectInfo, key: &str, norm SUFFIX_ACTUAL_SIZE, SUFFIX_CRC, SUFFIX_PART_CHECKSUMS, + // Inline placement is local to each physical copy, like its data directory. + rustfs_utils::http::SUFFIX_INLINE_DATA, crate::object_api::ENCRYPTED_PART_LAYOUT_QUORUM_SUFFIX, ] .iter() @@ -3532,6 +3534,23 @@ mod tests { assert!(overwrite_resume_for_target(&source, source.clone())); } + #[test] + fn test_data_movement_identity_accepts_inline_layout_change_only() { + let source = overwrite_equivalence_source(); + let mut target = source.clone(); + rustfs_utils::http::insert_str( + Arc::make_mut(&mut target.user_defined), + rustfs_utils::http::SUFFIX_INLINE_DATA, + "true".to_string(), + ); + assert!(is_equivalent_data_movement_object(&source, &target)); + assert!(is_equivalent_data_movement_object(&target, &source)); + + Arc::make_mut(&mut target.user_defined).insert("x-amz-meta-inline-data".to_string(), "user value".to_string()); + assert!(!is_equivalent_data_movement_object(&source, &target)); + assert!(!is_equivalent_data_movement_object(&target, &source)); + } + #[test] fn test_data_movement_overwrite_resume_accepts_destination_marker() { let source = overwrite_equivalence_source(); diff --git a/crates/ecstore/src/data_movement/scanner_backlog.rs b/crates/ecstore/src/data_movement/scanner_backlog.rs index 22c4a925c..6f7a1d058 100644 --- a/crates/ecstore/src/data_movement/scanner_backlog.rs +++ b/crates/ecstore/src/data_movement/scanner_backlog.rs @@ -110,19 +110,17 @@ async fn read_replica(set: Arc) -> Result reader, Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => { return Ok(ScannerPauseBacklogRetirementRead { replica, etag: None }); diff --git a/crates/ecstore/src/data_usage/mod.rs b/crates/ecstore/src/data_usage/mod.rs index af803a3a2..75aaaa601 100644 --- a/crates/ecstore/src/data_usage/mod.rs +++ b/crates/ecstore/src/data_usage/mod.rs @@ -1325,6 +1325,18 @@ fn select_admin_data_usage_snapshot( && authoritative.is_complete_bucket_usage_snapshot() && observed_data_usage_is_newer(&observed, &authoritative) => { + // A bucket can span several sets. Without a current or retained + // generation for every live set, its observed sum is only a subset, + // not a replacement for the last complete bucket total. Keep the + // baseline's timestamp and provenance until coverage is available. + if authoritative_format + && observed + .usage_snapshot_set_states + .iter() + .any(|state| !state.tombstone && state.scanner_cycle.is_none()) + { + return (authoritative, authoritative_format); + } (merge_partial_observation_for_admin(authoritative, observed), true) } Some(observed) if observed_data_usage_is_newer(&observed, &authoritative) => (observed, true), @@ -2466,7 +2478,13 @@ async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mu } continue; } - if !cached.stale_snapshot_pending && persisted_update.is_some_and(|persisted| cached.usage_updated_at <= persisted) { + // An observed scan can finish after mutations it never enumerated. Its + // completion time cannot supersede unreconciled single-process counts. + let observed_dirty_usage = data_usage_info.usage_snapshot_converged == Some(false) && cached.dirty; + if !cached.stale_snapshot_pending + && !observed_dirty_usage + && persisted_update.is_some_and(|persisted| cached.usage_updated_at <= persisted) + { continue; } @@ -2529,19 +2547,17 @@ where use crate::object_api::ObjectOptions; use http::HeaderMap; - match store - .get_object_reader( - RUSTFS_META_BUCKET, - key, - None, - HeaderMap::new(), - &ObjectOptions { - no_lock: true, - ..Default::default() - }, - ) - .await - { + let opts = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let read = store.get_object_reader(RUSTFS_META_BUCKET, key, None, HeaderMap::new(), &opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::DataUsageCacheRead, + read, + ); + match read.await { // A cache object that fails to decode is treated as absent rather than // as an error: a corrupt cache should not stall the caller, and the // next scanner pass rewrites it. @@ -3064,6 +3080,8 @@ mod tests { transient_failures: Mutex, reads: Mutex>, terminal_error: Mutex>, + #[cfg(feature = "diagnostic-read-version-caller")] + record_context: bool, } impl UsageCacheReadStore { @@ -3072,6 +3090,8 @@ mod tests { transient_failures: Mutex::new(n), reads: Mutex::new(Vec::new()), terminal_error: Mutex::new(None), + #[cfg(feature = "diagnostic-read-version-caller")] + record_context: false, } } @@ -3080,6 +3100,8 @@ mod tests { transient_failures: Mutex::new(0), reads: Mutex::new(Vec::new()), terminal_error: Mutex::new(Some(error)), + #[cfg(feature = "diagnostic-read-version-caller")] + record_context: false, } } @@ -3106,6 +3128,12 @@ mod tests { _h: Self::HeaderMap, _opts: &Self::ObjectOptions, ) -> Result { + #[cfg(feature = "diagnostic-read-version-caller")] + if self.record_context { + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request( + &crate::disk::ReadOptions::default(), + ); + } self.reads.lock().await.push(object.to_string()); if let Some(error) = self.terminal_error.lock().await.clone() { return Err(error); @@ -3205,6 +3233,89 @@ mod tests { ); } + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_usage_cache_read_preserves_api_and_earlier_entry() { + use crate::cluster::rpc::runtime_sources::with_read_version_entry; + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + use rustfs_io_metrics::internode_metrics::ReadVersionEntrySource; + use rustfs_io_metrics::s3_http_metrics::{ + S3HttpRequestGuard, current_s3_http_operation_context, with_s3_http_operation_context, + }; + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("diagnostic usage cache runtime"); + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let mut request = S3HttpRequestGuard::new("GET"); + let get = request.in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::GetObject); + current_s3_http_operation_context().expect("dispatched GET operation") + }); + request.response(200); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + let ambient = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture(); + for (operation, source) in [ + (None, None), + (Some(get), None), + (None, Some(ReadVersionEntrySource::ServerInfoPools)), + (Some(get), Some(ReadVersionEntrySource::StoreObjectReader)), + ] { + let store = UsageCacheReadStore { + record_context: true, + ..Default::default() + }; + with_s3_http_operation_context(operation, async { + let load = load_data_usage_cache(&store, "usage-cache"); + let cache = match source { + Some(source) => with_read_version_entry(source, load).await, + None => load.await, + } + .expect("missing diagnostic cache is harmless"); + assert!(cache.cache.is_empty()); + assert_eq!(store.read_keys().await.len(), 2, "both prefixed and legacy cache reads are observed"); + }) + .await; + assert_eq!( + crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture(), + ambient, + "cache reads must restore the caller's complete diagnostic scope" + ); + } + assert_eq!(current_s3_http_operation_context(), None); + }); + }); + let rows = snapshots.snapshot().into_vec(); + for (source, operation, state) in [ + ("data_usage_cache_read", "unattributed", "absent"), + ("data_usage_cache_read", "s3:GetObject", "preserved"), + ("server_info_pools", "unattributed", "absent"), + ("store_object_reader", "s3:GetObject", "preserved"), + ] { + assert!( + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("entry_source") == Some(&source) + && labels.get("boundary") == Some(&"remote_rpc") + && labels.get("parent_api_operation") == Some(&operation) + && labels.get("child_api_operation") == Some(&operation) + && labels.get("api_context") == Some(&state) + && matches!(value, DebugValue::Counter(2)) + }), + "cache reads must retain API and the earliest observed entry: {source}/{operation}" + ); + } + } + #[tokio::test] async fn data_usage_removal_surfaces_missing_system_volume_as_read_quorum() { for cause in [Error::BucketNotFound(RUSTFS_META_BUCKET.to_string()), Error::VolumeNotFound] { @@ -3718,6 +3829,169 @@ mod tests { assert!(selected.unknown_tier_stats.is_none()); } + fn usage_snapshots_with_missing_observed_set() -> (DataUsageInfo, DataUsageInfo) { + let baseline_time = SystemTime::UNIX_EPOCH + Duration::from_secs(10); + let mut authoritative = data_usage_info_for_test("large-bucket", 510_000, 580_000_000_000, baseline_time); + authoritative.scanner_epoch = Some(4); + authoritative.scanner_cycle = Some(10); + authoritative.usage_snapshot_converged = Some(true); + + let mut observed = data_usage_info_for_test("large-bucket", 3_000, 3_000_000_000, baseline_time + Duration::from_secs(1)); + observed.scanner_epoch = Some(4); + observed.scanner_cycle = Some(11); + observed.usage_snapshot_complete = false; + observed.usage_snapshot_partial = true; + observed.usage_snapshot_converged = Some(false); + observed.usage_snapshot_authoritative_baseline = Some(authoritative.snapshot_identity()); + observed.usage_snapshot_set_states = vec![ + rustfs_data_usage::DataUsageSnapshotSetState { + pool_index: 0, + set_index: 0, + scanner_epoch: Some(4), + scanner_cycle: Some(11), + scan_plan_digest: Some([1; 32]), + complete: true, + ..Default::default() + }, + rustfs_data_usage::DataUsageSnapshotSetState { + pool_index: 1, + set_index: 0, + scan_plan_digest: Some([1; 32]), + ..Default::default() + }, + ]; + assert!( + observed.is_valid_partial_snapshot(), + "a missing set is valid diagnostic progress, not a bucket total" + ); + (authoritative, observed) + } + + #[test] + fn partial_admin_observation_does_not_replace_totals_with_a_missing_set() { + let (authoritative, observed) = usage_snapshots_with_missing_observed_set(); + let identity = authoritative.snapshot_identity(); + + let (selected, _) = select_admin_data_usage_snapshot(authoritative, true, Some(observed)); + + assert_eq!(selected.buckets_usage["large-bucket"].objects_count, 510_000); + assert_eq!(selected.buckets_usage["large-bucket"].size, 580_000_000_000); + assert_eq!(selected.objects_total_count, 510_000); + assert_eq!(selected.objects_total_size, 580_000_000_000); + assert_eq!( + selected.snapshot_identity(), + identity, + "retained counts must not gain a newer timestamp or generation" + ); + assert!(selected.is_complete_bucket_usage_snapshot()); + } + + #[tokio::test] + async fn partial_admin_observation_does_not_promote_a_legacy_baseline() { + let (legacy, observed) = usage_snapshots_with_missing_observed_set(); + let (selected, current_format) = select_admin_data_usage_snapshot(legacy, false, Some(observed)); + let (normalized, _) = normalize_loaded_data_usage(selected, current_format).await; + + assert!(normalized.is_valid_partial_snapshot()); + assert!(!normalized.is_complete_bucket_usage_snapshot()); + assert_eq!(normalized.usage_snapshot_converged, Some(false)); + assert_eq!(normalized.objects_total_count, 3_000); + assert_eq!(normalized.objects_total_size, 3_000_000_000); + } + + #[test] + fn partial_admin_observation_accepts_decreases_when_every_set_is_covered() { + let (authoritative, mut observed) = usage_snapshots_with_missing_observed_set(); + // A compatible retained generation is useful even when it is not a + // completed scan of the current cycle. A tombstone explicitly removes + // a set; neither case is the same as an unavailable live set. + for tombstone in [false, true] { + observed.usage_snapshot_set_states[1].tombstone = tombstone; + observed.usage_snapshot_set_states[1].scanner_epoch = (!tombstone).then_some(4); + observed.usage_snapshot_set_states[1].scanner_cycle = (!tombstone).then_some(10); + let (selected, _) = select_admin_data_usage_snapshot(authoritative.clone(), true, Some(observed.clone())); + + assert_eq!(selected.objects_total_count, 3_000); + assert_eq!(selected.objects_total_size, 3_000_000_000); + assert!(selected.is_valid_partial_snapshot()); + assert!(!selected.is_complete_bucket_usage_snapshot()); + assert_eq!(selected.usage_snapshot_converged, Some(false)); + } + } + + #[tokio::test] + #[serial] + async fn partial_admin_observation_preserves_persisted_totals_across_cache_reload() { + use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks; + use crate::config::com::save_config; + + let (_dirs, store) = isolated_store_over_temp_disks().await; + let (authoritative, observed) = usage_snapshots_with_missing_observed_set(); + let primary_bytes = serde_json::to_vec(&authoritative).expect("encode complete usage"); + let observed_bytes = serde_json::to_vec(&observed).expect("encode partial observation"); + save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone()) + .await + .expect("persist complete usage"); + save_config(store.clone(), DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(), observed_bytes.clone()) + .await + .expect("persist observation with a missing set"); + clear_usage_memory_cache_for_test().await; + invalidate_data_usage_snapshot_cache().await; + + for reload in [false, false, true] { + if reload { + // Recreate process-local usage state, not an OS process restart. + clear_usage_memory_cache_for_test().await; + invalidate_data_usage_snapshot_cache().await; + } + let mut displayed = load_admin_data_usage_from_backend_cached(store.clone()) + .await + .expect("load admin usage through the real snapshot cache"); + apply_bucket_usage_memory_overlay(&mut displayed).await; + assert_eq!(displayed.objects_total_count, 510_000); + assert_eq!(displayed.objects_total_size, 580_000_000_000); + assert_eq!(displayed.bucket_sizes["large-bucket"], 580_000_000_000); + assert_eq!(displayed.snapshot_identity(), authoritative.snapshot_identity()); + } + let quota = load_data_usage_from_backend_cached(store.clone()) + .await + .expect("load quota authority"); + assert_eq!(quota.objects_total_size, 580_000_000_000); + assert_eq!( + read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("read primary"), + primary_bytes + ); + assert_eq!( + read_config(store.clone(), DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str()) + .await + .expect("read diagnostic observation"), + observed_bytes, + "admin reads must not rewrite either persisted snapshot" + ); + + let mut reconciled = data_usage_info_for_test("large-bucket", 2, 42, SystemTime::UNIX_EPOCH + Duration::from_secs(12)); + reconciled.scanner_epoch = Some(4); + reconciled.scanner_cycle = Some(12); + reconciled.usage_snapshot_converged = Some(true); + save_config( + store.clone(), + DATA_USAGE_OBJ_NAME_PATH.as_str(), + serde_json::to_vec(&reconciled).expect("encode confirmed deletion"), + ) + .await + .expect("persist a later complete scan after deletion"); + invalidate_data_usage_snapshot_cache().await; + let displayed = load_admin_data_usage_from_backend_cached(store) + .await + .expect("load confirmed decrease"); + assert_eq!((displayed.objects_total_count, displayed.objects_total_size), (2, 42)); + assert_eq!(displayed.snapshot_identity(), reconciled.snapshot_identity()); + clear_usage_memory_cache_for_test().await; + invalidate_data_usage_snapshot_cache().await; + } + #[tokio::test] async fn authoritative_save_cleanup_removes_observed_snapshot_best_effort() { let store = UsageCasStore::default(); @@ -6522,6 +6796,117 @@ mod tests { ); } + #[tokio::test] + #[serial] + async fn observed_admin_snapshot_preserves_unreconciled_mutations() { + let bucket = "observed-mutations"; + for partial in [false, true] { + for delete in [false, true] { + for timestamp_offset in [0, 10] { + clear_usage_memory_cache_for_test().await; + let mut baseline = data_usage_info_for_test(bucket, 2, 20, SystemTime::now() - Duration::from_secs(10)); + baseline.scanner_epoch = Some(4); + baseline.scanner_cycle = Some(10); + baseline.usage_snapshot_converged = Some(true); + replace_bucket_usage_memory_from_info(&baseline).await; + let (expected_count, expected_size) = if delete { + record_bucket_object_delete_memory(bucket, 10, true).await; + (1, 10) + } else { + record_bucket_object_write_memory(bucket, None, 10).await; + (3, 30) + }; + let mutation_time = memory_cache().read().await[bucket].usage_updated_at; + let mut observed = baseline.clone(); + observed.last_update = Some(mutation_time + Duration::from_secs(timestamp_offset)); + observed.scanner_cycle = Some(11); + observed.usage_snapshot_complete = !partial; + observed.usage_snapshot_partial = partial; + observed.usage_snapshot_converged = Some(false); + observed.usage_snapshot_authoritative_baseline = Some(baseline.snapshot_identity()); + if partial { + observed.usage_snapshot_set_states = vec![rustfs_data_usage::DataUsageSnapshotSetState { + pool_index: 0, + set_index: 0, + scanner_cycle: Some(11), + scanner_epoch: Some(4), + scan_plan_digest: Some([1; 32]), + complete: false, + tombstone: false, + segment_invalidation_proof: None, + }]; + assert!(observed.is_valid_partial_snapshot()); + } + + // Observational saves invalidate the admin cache without reconciling + // memory. Selecting that newer view must not erase local mutations. + let (selected, current_format) = select_admin_data_usage_snapshot(baseline, true, Some(observed)); + assert_eq!(selected.scanner_cycle, Some(11)); + let (selected, _) = normalize_loaded_data_usage(selected, current_format).await; + let mut distributed = selected.clone(); + apply_bucket_usage_memory_overlay_if_authoritative(&mut distributed, false).await; + assert_eq!((distributed.objects_total_count, distributed.objects_total_size), (2, 20)); + + let mut response = selected; + apply_bucket_usage_memory_overlay_if_authoritative(&mut response, true).await; + assert_eq!( + (response.objects_total_count, response.objects_total_size), + (expected_count, expected_size), + "partial={partial}, delete={delete}, timestamp_offset={timestamp_offset}" + ); + assert_eq!(response.versions_total_count, expected_count); + assert_eq!(response.bucket_sizes[bucket], expected_size); + assert_eq!(response.buckets_usage[bucket].objects_count, expected_count); + assert_eq!(response.usage_snapshot_converged, Some(false)); + assert_eq!(response.usage_snapshot_partial, partial); + let cache = memory_cache().read().await; + assert!(cache[bucket].dirty, "an admin read cannot acknowledge mutations"); + assert!(!cache[bucket].stale_snapshot_pending); + assert_eq!(cache[bucket].pending_scanner_position, None); + drop(cache); + + // A later converged scan can legitimately correct the count downward; + // the display fix must not retain dirty counts or a historical maximum. + let mut converged = data_usage_info_for_test(bucket, 0, 0, mutation_time + Duration::from_secs(20)); + converged.scanner_epoch = Some(4); + converged.scanner_cycle = Some(12); + converged.usage_snapshot_converged = Some(true); + replace_bucket_usage_memory_from_info(&converged).await; + apply_bucket_usage_memory_overlay_if_authoritative(&mut converged, true).await; + assert_eq!((converged.objects_total_count, converged.objects_total_size), (0, 0)); + assert!(!memory_cache().read().await[bucket].dirty); + } + } + } + } + + #[tokio::test] + #[serial] + async fn observed_admin_snapshot_does_not_overlay_clean_or_unknown_usage() { + let bucket = "observed-baseline"; + for unknown in [false, true] { + clear_usage_memory_cache_for_test().await; + let baseline = data_usage_info_for_test(bucket, 2, 20, SystemTime::now() - Duration::from_secs(10)); + replace_bucket_usage_memory_from_info(&baseline).await; + if unknown { + memory_cache().write().await.insert( + bucket.to_string(), + cached_bucket_usage_from_backend(BucketUsageInfo::default(), SystemTime::UNIX_EPOCH, false), + ); + record_bucket_object_write_memory(bucket, None, 10).await; + } + let mutation_time = memory_cache().read().await[bucket].usage_updated_at; + let mut observed = data_usage_info_for_test(bucket, 100, 1_000, mutation_time + Duration::from_secs(10)); + observed.usage_snapshot_converged = Some(false); + observed.usage_snapshot_authoritative_baseline = Some(baseline.snapshot_identity()); + let (mut response, _) = select_admin_data_usage_snapshot(baseline, true, Some(observed)); + apply_bucket_usage_memory_overlay_if_authoritative(&mut response, true).await; + assert_eq!((response.objects_total_count, response.objects_total_size), (100, 1_000)); + assert_eq!(response.usage_snapshot_complete, !unknown); + assert_eq!(response.usage_snapshot_converged, Some(false)); + } + } + #[tokio::test] #[serial] async fn scanner_sync_preserves_dirty_memory_update_with_later_partial_snapshot() { diff --git a/crates/ecstore/src/diagnostics/admin_server_info.rs b/crates/ecstore/src/diagnostics/admin_server_info.rs index 3bbf7cf05..41b6e8647 100644 --- a/crates/ecstore/src/diagnostics/admin_server_info.rs +++ b/crates/ecstore/src/diagnostics/admin_server_info.rs @@ -15,7 +15,9 @@ use crate::cluster::rpc::{ ScannerBucketListing, TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client, }; -use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend_cached}; +use crate::data_usage::{ + DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_admin_data_usage_from_backend_cached as load_server_info_usage_snapshot, +}; use crate::error::{Error, Result}; use crate::{ disk::endpoint::{Endpoint, EndpointType}, @@ -37,12 +39,16 @@ use rustfs_protos::{ }; use std::{ collections::{HashMap, HashSet}, - time::Duration, + time::{Duration, Instant}, }; -use time::OffsetDateTime; use tokio::time::timeout; use tonic::Request; -use tracing::warn; +use tracing::{error, trace, warn}; + +const EVENT_SERVER_PING: &str = "server_ping"; +const EVENT_SERVER_INFO: &str = "server_info"; +const LOG_COMPONENT_ECSTORE: &str = "ecstore"; +const LOG_SUBSYSTEM_DIAGNOSTICS: &str = "diagnostics"; use shadow_rs::shadow; @@ -157,11 +163,22 @@ async fn is_server_resolvable(endpoint: &Endpoint) -> Result<()> { let response: PingResponse = client.ping(request).await?.into_inner(); - let ping_response_body = flatbuffers::root::(&response.body); - if let Err(e) = ping_response_body { - eprintln!("{e}"); + if flatbuffers::root::(&response.body).is_err() { + error!( + event = EVENT_SERVER_PING, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DIAGNOSTICS, + result = "invalid_response", + "Server ping response decoding failed" + ); } else { - println!("ping_resp:body(flatbuffer): {ping_response_body:?}"); + trace!( + event = EVENT_SERVER_PING, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DIAGNOSTICS, + result = "success", + "Server ping completed" + ); } Ok(()) @@ -259,16 +276,10 @@ fn collect_runtime_server_stats() -> (MemStats, u64, u64) { } pub async fn get_server_info(get_pools: bool) -> InfoMessage { - let nowt: OffsetDateTime = OffsetDateTime::now_utc(); - - warn!("get_server_info start {:?}", nowt); + let started = tracing::enabled!(tracing::Level::TRACE).then(Instant::now); let local = get_local_server_property().await; - let after1 = OffsetDateTime::now_utc(); - - warn!("get_local_server_property end {:?}", after1 - nowt); - let mut servers = { if let Some(sys) = runtime_sources::notification_sys() { sys.server_info().await @@ -277,9 +288,6 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage { } }; - let after2 = OffsetDateTime::now_utc(); - - warn!("server_info end {:?}", after2 - after1); servers.push(local); let mut buckets = rustfs_madmin::Buckets::default(); @@ -294,7 +302,7 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage { if let Some(store) = runtime_sources::object_store_handle() { mode = ITEM_ONLINE; apply_data_usage_result( - load_data_usage_from_backend_cached(store.clone()).await, + load_server_info_usage_snapshot(store.clone()).await, &mut buckets, &mut objects, &mut versions, @@ -314,15 +322,8 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage { ); } - let after3 = OffsetDateTime::now_utc(); - - warn!("load_data_usage_from_backend end {:?}", after3 - after2); - let backend_info = StorageAdminApi::backend_info(store.as_ref()).await; - let after4 = OffsetDateTime::now_utc(); - - warn!("backend_info end {:?}", after4 - after3); if let Some(endpoints) = runtime_sources::endpoint_pools() { let (added, report) = reconcile_servers_with_endpoint_topology(&mut servers, &endpoints); if added > 0 || !report.is_complete() { @@ -344,9 +345,6 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage { } let (online_disks, offline_disks, unknown_disks) = get_online_offline_disks_stats(&all_disks); - let after5 = OffsetDateTime::now_utc(); - - warn!("get_online_offline_disks_stats end {:?}", after5 - after4); backend = rustfs_madmin::ErasureBackend { backend_type: rustfs_madmin::BackendType::ErasureType, online_disks: online_disks.sum(), @@ -359,14 +357,24 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage { }; if get_pools { pools = get_pools_info(&all_disks).await.unwrap_or_default(); - let after6 = OffsetDateTime::now_utc(); - - warn!("get_pools_info end {:?}", after6 - after5); } } let services = rustfs_madmin::Services::default(); + if let Some(started) = started { + trace!( + event = EVENT_SERVER_INFO, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DIAGNOSTICS, + state = "completed", + mode, + include_pools = get_pools, + duration_ms = started.elapsed().as_secs_f64() * 1000.0, + "Server info collected" + ); + } + InfoMessage { mode: Some(mode.to_string()), domain: None, @@ -678,12 +686,16 @@ async fn get_pools_info(all_disks: &[Disk]) -> Result apply_erasure_set_usage(&cache, erasure_set), Err(_) => erasure_set.usage_error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string()), } @@ -967,6 +979,93 @@ mod tests { assert_eq!(usage.size, 6); } + #[tokio::test] + #[serial] + async fn server_info_reads_complete_observation_without_promoting_authority() { + use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks; + use crate::config::com::{read_config, save_config}; + use crate::data_usage::{ + DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache, + load_data_usage_from_backend_cached, + }; + use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo}; + use std::time::{Duration, SystemTime}; + + let (_dirs, store) = isolated_store_over_temp_disks().await; + let primary = DataUsageInfo { + last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)), + scanner_epoch: Some(1), + usage_snapshot_bootstrap_pending: true, + usage_snapshot_converged: Some(false), + ..Default::default() + }; + let observed = DataUsageInfo { + last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(2)), + scanner_epoch: Some(1), + scanner_cycle: Some(1), + buckets_count: 1, + objects_total_count: 3, + versions_total_count: 4, + delete_markers_total_count: 1, + objects_total_size: 6, + buckets_usage: [( + "bucket".to_string(), + BucketUsageInfo { + objects_count: 3, + versions_count: 4, + delete_markers_count: 1, + size: 6, + ..Default::default() + }, + )] + .into_iter() + .collect(), + usage_snapshot_complete: true, + usage_snapshot_converged: Some(false), + usage_snapshot_authoritative_baseline: Some(primary.snapshot_identity()), + ..Default::default() + }; + let primary_bytes = serde_json::to_vec(&primary).expect("bootstrap should encode"); + save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone()) + .await + .expect("bootstrap should persist"); + save_config( + store.clone(), + DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(), + serde_json::to_vec(&observed).expect("observation should encode"), + ) + .await + .expect("observation should persist"); + invalidate_data_usage_snapshot_cache().await; + + let displayed = super::load_server_info_usage_snapshot(store.clone()) + .await + .expect("server info should load the complete observation"); + assert_eq!(displayed.usage_snapshot_converged, Some(false)); + let mut buckets = rustfs_madmin::Buckets::default(); + let mut objects = rustfs_madmin::Objects::default(); + let mut versions = rustfs_madmin::Versions::default(); + let mut delete_markers = rustfs_madmin::DeleteMarkers::default(); + let mut usage = rustfs_madmin::Usage::default(); + apply_data_usage_result(Ok(displayed), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage); + assert!(objects.error.is_none()); + assert_eq!( + (buckets.count, objects.count, versions.count, delete_markers.count, usage.size), + (1, 3, 4, 1, 6) + ); + let quota = load_data_usage_from_backend_cached(store.clone()) + .await + .expect("quota should retain its bootstrap state"); + assert!(!quota.is_complete_bucket_usage_snapshot()); + assert_eq!( + read_config(store, DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("primary should remain readable"), + primary_bytes + ); + invalidate_data_usage_snapshot_cache().await; + } + #[test] fn incomplete_data_usage_is_unavailable_in_server_info() { let mut buckets = rustfs_madmin::Buckets::default(); diff --git a/crates/ecstore/src/diagnostics/get.rs b/crates/ecstore/src/diagnostics/get.rs index fecb3c701..b0f85383c 100644 --- a/crates/ecstore/src/diagnostics/get.rs +++ b/crates/ecstore/src/diagnostics/get.rs @@ -182,6 +182,7 @@ pub(crate) const GET_METADATA_CACHE_REASON_DELETE_MARKER: &str = "delete_marker" pub(crate) const GET_METADATA_CACHE_REASON_DIST_ERASURE: &str = "dist_erasure"; pub(crate) const GET_METADATA_CACHE_REASON_INCL_FREE_VERSIONS: &str = "incl_free_versions"; pub(crate) const GET_METADATA_CACHE_REASON_INSUFFICIENT_CACHED_QUORUM: &str = "insufficient_cached_quorum"; +pub(crate) const GET_METADATA_CACHE_REASON_BODY_LAYOUT_REQUIRED: &str = "body_layout_required"; pub(crate) const GET_METADATA_CACHE_REASON_META_BUCKET: &str = "meta_bucket"; pub(crate) const GET_METADATA_CACHE_REASON_NO_LOCK: &str = "no_lock"; pub(crate) const GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED: &str = "not_found_or_expired"; diff --git a/crates/ecstore/src/disk/cleanup_runtime.rs b/crates/ecstore/src/disk/cleanup_runtime.rs new file mode 100644 index 000000000..8293e577a --- /dev/null +++ b/crates/ecstore/src/disk/cleanup_runtime.rs @@ -0,0 +1,1292 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Admission and execution isolation for PUT, local/RPC disk cleanup and GC. +//! PUT admission precedes staging; receiver and GC disk work share a separate +//! budget so coordinators waiting for peers cannot exhaust receiver capacity. + +use crate::disk::{LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_DISK}; +use std::collections::BTreeSet; +use std::future::Future; +use std::io; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, OnceLock}; +use std::time::Instant; +use tokio::runtime::{Handle, Runtime}; +use tokio::sync::{OwnedSemaphorePermit, Semaphore}; + +const ENABLE: &str = "RUSTFS_CLEANUP_ISOLATE_ENABLE"; +const BUDGET: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_MAX_PENDING"; +const BLOCKING_THREADS: &str = "RUSTFS_CLEANUP_BLOCKING_THREADS"; +const ASYNC_THREADS: &str = "RUSTFS_CLEANUP_ASYNC_THREADS"; +const CPUS: &str = "RUSTFS_CLEANUP_CPUS"; +const WORKERS: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKERS"; +const RENAME_TAIL_CLEANUP_LANE_ENABLE: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_ENABLE"; +const DISK_PENDING: &str = "RUSTFS_CLEANUP_DISK_MAX_PENDING"; +const DISK_WORKERS: &str = "RUSTFS_CLEANUP_DISK_WORKERS"; +const GC_WORKERS: &str = "RUSTFS_CLEANUP_GC_WORKERS"; +const EVENT_CLEANUP_RUNTIME: &str = "disk_cleanup_runtime"; +pub(crate) const OLD_DATA_CLEANUP_RECEIPT_FILE: &str = ".rustfs-old-data-cleanup-receipt.json"; + +#[derive(Clone, Copy)] +enum DiskAdmission { + Request, + Guarded, +} + +#[derive(Clone)] +pub(crate) struct Reservation { + _admission: Arc, + execution: Arc, + handle: Handle, +} + +struct Admission { + _permit: OwnedSemaphorePermit, + metrics_enabled: bool, + metric: &'static str, +} + +impl Admission { + fn new(permit: OwnedSemaphorePermit, metric: &'static str) -> Self { + let metrics_enabled = rustfs_io_metrics::put_stage_metrics_enabled(); + if metrics_enabled { + metrics::gauge!(metric).increment(1.0); + } + Self { + _permit: permit, + metrics_enabled, + metric, + } + } +} + +impl Drop for Admission { + fn drop(&mut self) { + if self.metrics_enabled { + metrics::gauge!(self.metric).decrement(1.0); + } + } +} + +/// A lease release cannot queue behind work that might need the releasing +/// reader's locks. It relinquishes the lease first, then tries this disk budget. +pub(crate) struct DiskExecution { + _active: Admission, + _pending: Admission, +} + +/// Created only inside an owned scan. Steps can borrow the scan's directory +/// iterator and paths: cancelling its waiter cannot cancel a step's syscalls. +#[cfg_attr(test, derive(Default))] +pub(crate) struct GcBudget { + gates: Option<(Arc, Arc)>, +} + +impl GcBudget { + pub(crate) async fn step(&self, future: F) -> super::error::Result + where + F: Future>, + { + let Some((pending, execution)) = &self.gates else { + return future.await; + }; + // Scan producers are bounded and hold no object locks here. Unlike + // requests, they can wait for admission without an unbounded backlog. + let started = rustfs_io_metrics::put_stage_metrics_enabled().then(Instant::now); + let pending = pending.clone().acquire_owned().await.map_err(io::Error::other)?; + execute_disk( + Admission::new(pending, "rustfs_cleanup_disk_pending"), + None, + execution.clone(), + started, + future, + ) + .await + } +} + +fn try_disk_permit(semaphore: &Arc) -> io::Result { + semaphore.clone().try_acquire_owned().map_err(|err| { + if rustfs_io_metrics::put_stage_metrics_enabled() { + metrics::counter!("rustfs_cleanup_disk_admission_rejected_total").increment(1); + } + io::Error::new(io::ErrorKind::WouldBlock, err) + }) +} + +async fn execute_disk( + pending: Admission, + active: Option, + execution: Arc, + started: Option, + future: F, +) -> super::error::Result +where + F: Future>, +{ + let _pending = pending; + let active = match active { + Some(active) => active, + None => execution.acquire_owned().await.map_err(super::error::DiskError::other)?, + }; + let _active = Admission::new(active, "rustfs_cleanup_disk_active"); + if let Some(started) = started { + metrics::histogram!("rustfs_cleanup_disk_execution_wait_seconds").record(started.elapsed().as_secs_f64()); + } + future.await +} + +struct Config { + pending: usize, + workers: usize, + async_threads: usize, + blocking_threads: usize, + cpus: Vec, + disk_pending: usize, + disk_workers: usize, + gc_workers: usize, +} + +fn positive_env(name: &str, default: usize) -> io::Result { + match std::env::var(name) { + Ok(value) => value + .parse::() + .ok() + .filter(|v| *v > 0 && *v <= Semaphore::MAX_PERMITS) + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, format!("{name} must be a positive supported count"))), + Err(std::env::VarError::NotPresent) => Ok(default), + Err(err) => Err(io::Error::new(io::ErrorKind::InvalidInput, err)), + } +} + +// Keep the guard's accepted values aligned with the lane's existing parser. +fn legacy_rename_tail_cleanup_lane_enabled(value: Option<&str>) -> bool { + matches!(value, Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON")) +} + +fn parse_cpus(value: &str) -> io::Result> { + let invalid = || { + io::Error::new( + io::ErrorKind::InvalidInput, + "cleanup CPUs must be comma-separated CPU IDs or ascending ranges", + ) + }; + if value.trim().is_empty() { + return Err(invalid()); + } + let mut cpus = BTreeSet::new(); + for part in value.split(',') { + let mut ends = part.trim().split('-'); + let first = ends.next().and_then(|s| s.parse::().ok()).ok_or_else(invalid)?; + let last = match ends.next() { + Some(s) => s.parse::().map_err(|_| invalid())?, + None => first, + }; + if ends.next().is_some() || first > last || last >= 1024 { + return Err(invalid()); + } + cpus.extend(first..=last); + } + Ok(cpus.into_iter().collect()) +} + +impl Config { + fn from_env() -> io::Result { + for name in [ + "RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP", + "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER", + "RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP", + ] { + if rustfs_utils::get_env_bool(name, false) { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + format!("{name} cannot be combined with cleanup isolation"), + )); + } + } + if legacy_rename_tail_cleanup_lane_enabled(std::env::var(RENAME_TAIL_CLEANUP_LANE_ENABLE).ok().as_deref()) { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + format!("{RENAME_TAIL_CLEANUP_LANE_ENABLE} cannot be combined with cleanup isolation"), + )); + } + let workers = positive_env(WORKERS, 2)?; + let pending = positive_env(BUDGET, 1024)?; + let blocking_threads = positive_env(BLOCKING_THREADS, 4)?; + let cpus = match std::env::var(CPUS) { + Ok(value) if value.trim().eq_ignore_ascii_case("none") => Vec::new(), + Ok(value) => parse_cpus(&value)?, + Err(std::env::VarError::NotPresent) if cfg!(target_os = "linux") => vec![1, 2], + Err(std::env::VarError::NotPresent) => Vec::new(), + Err(err) => return Err(io::Error::new(io::ErrorKind::InvalidInput, err)), + }; + let available = std::thread::available_parallelism().map_or(1, usize::from); + let async_threads = positive_env(ASYNC_THREADS, available.min(if cpus.is_empty() { 2 } else { cpus.len().min(2) }))?; + let disk_workers = positive_env(DISK_WORKERS, blocking_threads)?; + Ok(Self { + pending, + workers, + async_threads, + blocking_threads, + cpus, + // Bound queueing to sixteen service waves by default. Increasing + // async coordinator concurrency does not create more OS threads. + disk_pending: positive_env(DISK_PENDING, disk_workers.saturating_mul(16).min(Semaphore::MAX_PERMITS))?, + disk_workers, + gc_workers: positive_env(GC_WORKERS, 1)?, + }) + } + + fn validate(&self) -> io::Result<()> { + if self.pending == 0 + || self.pending > Semaphore::MAX_PERMITS + || self.workers == 0 + || self.workers > self.pending + || self.async_threads == 0 + || self.async_threads > 1024 + || self.blocking_threads == 0 + || self.blocking_threads > 1024 + || self.disk_pending == 0 + || self.disk_pending > Semaphore::MAX_PERMITS + || self.disk_workers == 0 + || self.disk_workers > self.disk_pending + || self.gc_workers == 0 + || self.gc_workers > self.disk_workers + { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "cleanup counts must be positive: PUT workers <= PUT pending, GC workers <= disk workers <= disk pending, async/blocking threads <= 1024", + )); + } + validate_cpus(&self.cpus) + } +} + +#[cfg(target_os = "linux")] +fn validate_cpus(cpus: &[usize]) -> io::Result<()> { + if cpus.is_empty() { + return Ok(()); + } + let allowed = rustix::thread::sched_getaffinity(None)?; + if cpus + .iter() + .any(|cpu| *cpu >= rustix::thread::CpuSet::MAX_CPU || !allowed.is_set(*cpu)) + { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "cleanup CPUs must be a subset of the caller's allowed CPU set", + )); + } + Ok(()) +} + +#[cfg(not(target_os = "linux"))] +fn validate_cpus(cpus: &[usize]) -> io::Result<()> { + if cpus.is_empty() { + Ok(()) + } else { + Err(io::Error::new(io::ErrorKind::Unsupported, "cleanup CPU affinity requires Linux")) + } +} + +fn bind_thread(cpus: &[usize]) -> io::Result<()> { + if cpus.is_empty() { + return Ok(()); + } + #[cfg(target_os = "linux")] + { + let mut set = rustix::thread::CpuSet::new(); + for cpu in cpus { + set.set(*cpu); + } + rustix::thread::sched_setaffinity(None, &set)?; + if rustix::thread::sched_getaffinity(None)? != set { + return Err(io::Error::other("cleanup CPU affinity changed during thread startup")); + } + Ok(()) + } + #[cfg(not(target_os = "linux"))] + { + validate_cpus(cpus) + } +} + +struct CleanupRuntime { + runtime: Option, + admission: Arc, + execution: Arc, + affinity_failed: Arc, + disk_admission: Arc, + disk_execution: Arc, + gc_execution: Arc, +} + +impl CleanupRuntime { + fn new(config: Config) -> io::Result { + config.validate()?; + let affinity_failed = Arc::new(AtomicBool::new(false)); + let failed = affinity_failed.clone(); + let cpus = config.cpus; + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(config.async_threads) + .max_blocking_threads(config.blocking_threads) + .thread_name("rustfs-cleanup") + .enable_all() + .on_thread_start(move || { + // Explicit fsync dispatch must not escape into the shared + // fsync runtime. This callback also runs on blocking threads. + crate::disk::os::use_current_runtime_for_fsync(); + if let Err(err) = bind_thread(&cpus) + && !failed.swap(true, Ordering::AcqRel) + { + tracing::error!(event = EVENT_CLEANUP_RUNTIME, component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DISK, state = "affinity_failed", error = %err, + "Cleanup thread affinity failed"); + } + }) + .build()?; + Ok(Self { + runtime: Some(runtime), + admission: Arc::new(Semaphore::new(config.pending)), + execution: Arc::new(Semaphore::new(config.workers)), + affinity_failed, + disk_admission: Arc::new(Semaphore::new(config.disk_pending)), + disk_execution: Arc::new(Semaphore::new(config.disk_workers)), + gc_execution: Arc::new(Semaphore::new(config.gc_workers)), + }) + } + + fn reserve(&self) -> io::Result { + if self.affinity_failed.load(Ordering::Acquire) { + return Err(io::Error::other("cleanup thread affinity failed; refusing new cleanup admission")); + } + let metrics_enabled = rustfs_io_metrics::put_stage_metrics_enabled(); + // Callers may already own source-reader or data-movement locks. Do + // not wait here behind a PUT whose completion requires those locks. + let permit = self.admission.clone().try_acquire_owned().map_err(|err| { + if metrics_enabled { + metrics::counter!("rustfs_cleanup_admission_rejected_total").increment(1); + } + io::Error::new(io::ErrorKind::WouldBlock, err) + })?; + let runtime = self + .runtime + .as_ref() + .ok_or_else(|| io::Error::other("cleanup runtime stopped"))?; + Ok(Reservation { + _admission: Arc::new(Admission::new(permit, "rustfs_cleanup_admitted_puts")), + execution: self.execution.clone(), + handle: runtime.handle().clone(), + }) + } + + async fn disk_operation(&self, future: F, mode: DiskAdmission) -> super::error::Result + where + F: Future> + Send + 'static, + T: Send + 'static, + { + if self.affinity_failed.load(Ordering::Acquire) && !super::os::is_isolated_io_thread() { + return Err(io::Error::other("cleanup thread affinity failed; refusing new disk cleanup admission").into()); + } + let started = rustfs_io_metrics::put_stage_metrics_enabled().then(Instant::now); + let pending = try_disk_permit(&self.disk_admission)?; + let pending = Admission::new(pending, "rustfs_cleanup_disk_pending"); + let runtime = self + .runtime + .as_ref() + .ok_or_else(|| io::Error::other("cleanup runtime stopped"))?; + let execution = self.disk_execution.clone(); + // An externally guarded mutation may already hold a namespace lock + // needed by an active disk job. Never queue it behind that job. + let active = if matches!(mode, DiskAdmission::Guarded) { + try_disk_permit(&execution)? + } else { + // Keep queued work in the caller's existing future. A waiter + // timeout must discard an unstarted delete, not leave a task that + // can later delete a reused path after its caller released locks. + execution.clone().acquire_owned().await.map_err(io::Error::other)? + }; + // A permit does not mean the worker has polled this operation. Use + // the result receiver's lifetime to discard work cancelled during + // runtime scheduling, before its first poll can start filesystem I/O. + let (result_tx, result_rx) = tokio::sync::oneshot::channel(); + let task = runtime.spawn(async move { + if result_tx.is_closed() { + return; + } + // After the first poll, the owned job must retain both permits + // through I/O even if the waiter cancels and closes result_rx. + let result = execute_disk(pending, Some(active), execution, started, future).await; + let _ = result_tx.send(result); + }); + match result_rx.await { + Ok(result) => result, + Err(err) => { + // Preserve a panic's JoinError rather than erasing its cause + // into a closed-channel error. + task.await.map_err(super::error::DiskError::other)?; + Err(super::error::DiskError::other(err)) + } + } + } + + fn try_disk_execution(&self) -> io::Result { + let pending = try_disk_permit(&self.disk_admission)?; + let active = try_disk_permit(&self.disk_execution)?; + Ok(DiskExecution { + _pending: Admission::new(pending, "rustfs_cleanup_disk_pending"), + _active: Admission::new(active, "rustfs_cleanup_disk_active"), + }) + } + + async fn gc_operation(&self, scan: F) -> super::error::Result<()> + where + F: FnOnce(GcBudget) -> Fut + Send + 'static, + Fut: Future> + Send + 'static, + { + if self.affinity_failed.load(Ordering::Acquire) { + return Err(io::Error::other("cleanup thread affinity failed; refusing new GC scan").into()); + } + let handle = self + .runtime + .as_ref() + .ok_or_else(|| io::Error::other("cleanup runtime stopped"))? + .handle(); + let execution = self.gc_execution.clone(); + let budget = GcBudget { + gates: Some((self.disk_admission.clone(), self.disk_execution.clone())), + }; + handle + .spawn(async move { + let active = execution.acquire_owned().await.map_err(super::error::DiskError::other)?; + let _active = Admission::new(active, "rustfs_cleanup_gc_active"); + scan(budget).await + }) + .await + .map_err(super::error::DiskError::other)? + } +} + +impl Drop for CleanupRuntime { + fn drop(&mut self) { + if let Some(runtime) = self.runtime.take() { + runtime.shutdown_background(); + } + } +} + +fn instance() -> io::Result> { + static INSTANCE: OnceLock, String>> = OnceLock::new(); + match INSTANCE.get_or_init(|| { + if !rustfs_utils::get_env_bool(ENABLE, false) { + return Ok(None); + } + Config::from_env() + .and_then(CleanupRuntime::new) + .map(Some) + .map_err(|err| err.to_string()) + }) { + Ok(runtime) => Ok(runtime.as_ref()), + Err(err) => Err(io::Error::other(err.clone())), + } +} + +pub(crate) fn reserve() -> io::Result> { + match instance()? { + Some(runtime) => runtime.reserve().map(Some), + None => Ok(None), + } +} + +pub(crate) fn enabled() -> io::Result { + Ok(instance()?.is_some()) +} + +/// Local cleanup entry point, also used by remote RPC receivers. This budget +/// is distinct from coordinator execution: coordinators can wait for peers +/// without exhausting the peers' capacity to service disk cleanup. +pub(crate) async fn disk_operation(future: F) -> super::error::Result +where + F: Future> + Send + 'static, + T: Send + 'static, +{ + match instance()? { + Some(runtime) => runtime.disk_operation(future, DiskAdmission::Request).await, + None => future.await, + } +} + +pub(crate) async fn guarded_disk_operation(future: F) -> super::error::Result +where + F: Future> + Send + 'static, + T: Send + 'static, +{ + match instance()? { + Some(runtime) => runtime.disk_operation(future, DiskAdmission::Guarded).await, + None => future.await, + } +} + +/// Lease release must run even under cleanup saturation, or reader tokens +/// would leak. Only its optional deferred deletion consumes disk permits. +pub(crate) async fn release_lease(future: F) -> super::error::Result<()> +where + F: Future> + Send + 'static, +{ + match instance()? { + Some(runtime) => runtime + .runtime + .as_ref() + .ok_or_else(|| io::Error::other("cleanup runtime stopped"))? + .spawn(future) + .await + .map_err(super::error::DiskError::other)?, + None => future.await, + } +} + +pub(crate) fn try_disk_execution() -> io::Result> { + instance()?.map(CleanupRuntime::try_disk_execution).transpose() +} + +/// At most gc_workers scans can produce work. Each scan yields the shared +/// disk execution budget between entries without spawning a task per entry. +pub(crate) async fn run_gc(scan: F) -> super::error::Result<()> +where + F: FnOnce(GcBudget) -> Fut + Send + 'static, + Fut: Future> + Send + 'static, +{ + match instance()? { + Some(runtime) => runtime.gc_operation(scan).await, + None => scan(GcBudget { gates: None }).await, + } +} + +/// Dispatch only already-admitted work. Keeping the runtime handle in the +/// reservation prevents configuration errors from dropping committed cleanup. +pub(crate) fn spawn(reservation: Option, future: F) -> tokio::task::JoinHandle +where + F: Future + Send + 'static, + F::Output: Send + 'static, +{ + let started = rustfs_io_metrics::put_stage_metrics_enabled().then(Instant::now); + match reservation { + Some(reservation) => reservation.handle.clone().spawn(async move { + // This semaphore is never closed. Both queued work and overflow + // use this gate, and cancellation of the waiter cannot release it. + let _permit = reservation.execution.clone().acquire_owned().await; + if let Some(started) = started { + metrics::histogram!("rustfs_cleanup_execution_wait_seconds").record(started.elapsed().as_secs_f64()); + } + let result = future.await; + drop(reservation); + result + }), + None => tokio::spawn(future), + } +} + +/// Queue workers, queue overflow and synchronous failure cleanup share a gate. +/// With isolation disabled, retain the original inline polling behavior. +pub(crate) async fn run(reservation: Option, future: F) +where + F: Future + Send + 'static, +{ + if reservation.is_none() { + future.await; + } else if let Err(err) = spawn(reservation, future).await { + tracing::error!(event = EVENT_CLEANUP_RUNTIME, component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DISK, state = "task_failed", error = %err, + "Cleanup task failed"); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::atomic::AtomicUsize; + use std::time::Duration; + use tokio::sync::oneshot; + + fn config(pending: usize, workers: usize, blocking_threads: usize) -> Config { + Config { + pending, + workers, + async_threads: workers.min(2), + blocking_threads, + cpus: Vec::new(), + disk_pending: 4, + disk_workers: 2, + gc_workers: 1, + } + } + + async fn deadline(future: F) -> F::Output { + tokio::time::timeout(Duration::from_secs(20), future) + .await + .expect("cleanup test timed out") + } + + #[test] + fn cpu_list_rejects_invalid_or_unbounded_ranges() { + assert_eq!(parse_cpus("3, 1-3,6").unwrap(), vec![1, 2, 3, 6]); + for value in ["", " ", "1,", "-1", "2-1", "1-2-3", "no", "1024", "0-18446744073709551615"] { + assert_eq!(parse_cpus(value).unwrap_err().kind(), io::ErrorKind::InvalidInput, "{value}"); + } + } + + #[test] + fn cleanup_lane_guard_matches_legacy_boolean_values() { + for (value, expected) in [ + (Some("1"), true), + (Some("true"), true), + (Some("TRUE"), true), + (Some("yes"), true), + (Some("YES"), true), + (Some("on"), true), + (Some("ON"), true), + (Some("True"), false), + (Some("enabled"), false), + (None, false), + ] { + assert_eq!(legacy_rename_tail_cleanup_lane_enabled(value), expected, "{value:?}"); + } + } + + #[test] + fn invalid_environment_is_rejected_before_admission() { + const CHILD: &str = "RUSTFS_CLEANUP_CONFIG_TEST_CHILD"; + if std::env::var_os(CHILD).is_some() { + assert!(reserve().is_err()); + return; + } + // Each configuration needs a fresh process: production intentionally + // caches configuration, and mutating process env races parallel tests. + for (name, value) in [ + (WORKERS, "0"), + (WORKERS, "5"), + (BUDGET, "invalid"), + (BLOCKING_THREADS, "0"), + (BLOCKING_THREADS, "1025"), + (ASYNC_THREADS, "0"), + (ASYNC_THREADS, "1025"), + (DISK_PENDING, "0"), + (DISK_WORKERS, "3"), + (GC_WORKERS, "3"), + (CPUS, "1-0"), + ("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP", "true"), + ("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER", "true"), + ("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP", "true"), + (RENAME_TAIL_CLEANUP_LANE_ENABLE, "true"), + ] { + let result = std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "disk::cleanup_runtime::tests::invalid_environment_is_rejected_before_admission", + "--nocapture", + ]) + .env(CHILD, "1") + .env(ENABLE, "true") + .env(WORKERS, "1") + .env(BUDGET, "4") + .env(BLOCKING_THREADS, "1") + .env(ASYNC_THREADS, "1") + .env(DISK_PENDING, "2") + .env(DISK_WORKERS, "2") + .env(GC_WORKERS, "1") + .env_remove(CPUS) + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP") + .env_remove(RENAME_TAIL_CLEANUP_LANE_ENABLE) + .env(name, value) + .output() + .unwrap(); + assert!(result.status.success(), "{name}={value}: {}", String::from_utf8_lossy(&result.stderr)); + assert!( + String::from_utf8_lossy(&result.stdout).contains("1 passed"), + "child must execute the admission assertion" + ); + } + } + + #[test] + fn invalid_budgets_fail_before_building_threads() { + for config in [config(0, 1, 1), config(1, 0, 1), config(1, 2, 1), config(1, 1, 0)] { + assert!(matches!(CleanupRuntime::new(config), Err(err) if err.kind() == io::ErrorKind::InvalidInput)); + } + } + + #[tokio::test] + async fn disk_work_keeps_budget_after_rpc_waiter_cancellation() { + let mut settings = config(1, 1, 1); + settings.disk_pending = 1; + settings.disk_workers = 1; + let runtime = Arc::new(CleanupRuntime::new(settings).unwrap()); + let receiver = runtime.clone(); + let (entered, entry) = oneshot::channel(); + let (release, released) = std::sync::mpsc::channel(); + let rpc = tokio::spawn(async move { + receiver + .disk_operation( + async move { + tokio::task::spawn_blocking(move || { + assert_eq!(std::thread::current().name(), Some("rustfs-cleanup")); + entered.send(()).unwrap(); + released.recv_timeout(Duration::from_secs(20)).unwrap(); + }) + .await + .unwrap(); + Ok(()) + }, + DiskAdmission::Request, + ) + .await + }); + deadline(entry).await.unwrap(); + rpc.abort(); + assert!(rpc.await.unwrap_err().is_cancelled()); + assert_eq!(runtime.disk_admission.available_permits(), 0); + assert_eq!(runtime.disk_execution.available_permits(), 0); + let ran = Arc::new(AtomicBool::new(false)); + let invoked = ran.clone(); + let rejected = runtime + .disk_operation( + async move { + invoked.store(true, Ordering::SeqCst); + Ok(()) + }, + DiskAdmission::Request, + ) + .await; + assert!(matches!(rejected, Err(super::super::error::DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + assert!(!ran.load(Ordering::SeqCst), "rejected work must not execute"); + release.send(()).unwrap(); + deadline(async { + while runtime.disk_admission.available_permits() == 0 { + tokio::task::yield_now().await; + } + }) + .await; + assert_eq!(runtime.disk_execution.available_permits(), 1); + } + + #[tokio::test] + async fn cancelling_queued_disk_work_discards_it_before_execution() { + let runtime = Arc::new(CleanupRuntime::new(config(1, 1, 1)).expect("cleanup runtime")); + let occupied = runtime + .disk_execution + .clone() + .acquire_many_owned(2) + .await + .expect("occupy all execution slots"); + let receiver = runtime.clone(); + let ran = Arc::new(AtomicBool::new(false)); + let invoked = ran.clone(); + let waiter = tokio::spawn(async move { + receiver + .disk_operation( + async move { + invoked.store(true, Ordering::SeqCst); + Ok(()) + }, + DiskAdmission::Request, + ) + .await + }); + deadline(async { + while runtime.disk_admission.available_permits() == 4 { + tokio::task::yield_now().await; + } + }) + .await; + waiter.abort(); + assert!(waiter.await.expect_err("cancel queued waiter").is_cancelled()); + assert_eq!( + runtime.disk_admission.available_permits(), + 4, + "unstarted work must release pending capacity immediately" + ); + drop(occupied); + deadline(runtime.disk_operation(async { Ok(()) }, DiskAdmission::Request)) + .await + .expect("later work must progress"); + assert!(!ran.load(Ordering::SeqCst), "cancelled queued deletion must never execute later"); + } + + #[tokio::test] + async fn cancelling_disk_work_before_worker_poll_preserves_reused_path() { + let runtime = Arc::new(CleanupRuntime::new(config(1, 1, 1)).expect("cleanup runtime")); + let (entered, entry) = oneshot::channel(); + let (release, released) = std::sync::mpsc::channel(); + let stalled = runtime.runtime.as_ref().expect("live runtime").spawn(async move { + entered.send(()).expect("worker entered"); + released + .recv_timeout(Duration::from_secs(30)) + .expect("release stalled worker"); + }); + deadline(entry).await.expect("worker must stall before dispatch"); + let dir = tempfile::tempdir().expect("temporary disk"); + let backup = dir.path().join("xl.meta.bkp"); + std::fs::write(&backup, b"previous transaction").expect("old backup"); + { + let path = backup.clone(); + let deletion = runtime.disk_operation( + async move { + tokio::fs::remove_file(path).await?; + Ok(()) + }, + DiskAdmission::Request, + ); + tokio::pin!(deletion); + assert!(futures::poll!(&mut deletion).is_pending()); + assert_eq!(runtime.disk_execution.available_permits(), 1, "execution was admitted but not polled"); + } + std::fs::write(&backup, b"next transaction").expect("reuse path after cancelled waiter"); + release.send(()).expect("resume worker"); + deadline(stalled).await.expect("worker resumed"); + deadline(async { + while runtime.disk_admission.available_permits() != 4 { + tokio::task::yield_now().await; + } + }) + .await; + assert_eq!( + std::fs::read(backup).expect("unstarted deletion must not execute after cancellation"), + b"next transaction" + ); + assert_eq!(runtime.disk_execution.available_permits(), 2); + } + + #[tokio::test] + async fn disk_task_panic_preserves_cause_and_releases_budget() { + async fn panicking_job() -> super::super::error::Result<()> { + panic!("injected cleanup panic"); + } + let runtime = CleanupRuntime::new(config(1, 1, 1)).expect("cleanup runtime"); + let error = deadline(runtime.disk_operation(panicking_job(), DiskAdmission::Request)) + .await + .expect_err("worker panic must reach its caller"); + let super::super::error::DiskError::Io(error) = error else { + panic!("worker panic must retain its I/O error source"); + }; + let cause = error + .get_ref() + .expect("panic source") + .downcast_ref::() + .expect("preserved JoinError"); + assert!(cause.is_panic()); + assert_eq!(runtime.disk_admission.available_permits(), 4); + assert_eq!(runtime.disk_execution.available_permits(), 2); + } + + #[tokio::test] + async fn coordinator_waiting_for_disk_does_not_consume_receiver_capacity() { + let mut settings = config(1, 1, 1); + settings.disk_pending = 1; + settings.disk_workers = 1; + let runtime = Arc::new(CleanupRuntime::new(settings).unwrap()); + let reservation = runtime.reserve().unwrap(); + let receiver = runtime.clone(); + let result = deadline(spawn(Some(reservation), async move { + receiver + .disk_operation(async { Ok(42) }, DiskAdmission::Request) + .await + .unwrap() + })) + .await + .unwrap(); + assert_eq!(result, 42); + } + + #[tokio::test] + async fn gc_shares_disk_budget_and_guarded_requests_do_not_wait_under_locks() { + let mut settings = config(1, 1, 1); + settings.disk_pending = 1; + settings.disk_workers = 1; + let runtime = Arc::new(CleanupRuntime::new(settings).unwrap()); + let receiver = runtime.clone(); + let (entered, entry) = oneshot::channel(); + let (release, released) = oneshot::channel(); + let rpc = tokio::spawn(async move { + receiver + .disk_operation( + async move { + entered.send(()).unwrap(); + released.await.unwrap(); + Ok(()) + }, + DiskAdmission::Request, + ) + .await + }); + deadline(entry).await.unwrap(); + let guarded = runtime.disk_operation(async { Ok(()) }, DiskAdmission::Guarded).await; + assert!(matches!(guarded, Err(super::super::error::DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + let budget = GcBudget { + gates: Some((runtime.disk_admission.clone(), runtime.disk_execution.clone())), + }; + let gc = budget.step(async { Ok(7) }); + tokio::pin!(gc); + assert!( + futures::poll!(&mut gc).is_pending(), + "GC must join the shared budget rather than bypass it" + ); + release.send(()).unwrap(); + deadline(rpc).await.unwrap().unwrap(); + assert_eq!(deadline(gc).await.unwrap(), 7); + + // Also reject a guarded mutation when pending capacity is free but + // execution is occupied. This prevents namespace -> budget inversion. + let execution = runtime.disk_execution.clone().acquire_owned().await.unwrap(); + let guarded = runtime.disk_operation(async { Ok(()) }, DiskAdmission::Guarded).await; + assert!(matches!(guarded, Err(super::super::error::DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + assert_eq!(runtime.disk_admission.available_permits(), 1); + assert!(matches!(runtime.try_disk_execution(), Err(err) if err.kind() == io::ErrorKind::WouldBlock)); + assert_eq!(runtime.disk_admission.available_permits(), 1); + drop(execution); + let lease_cleanup = runtime.try_disk_execution().unwrap(); + assert_eq!(runtime.disk_admission.available_permits(), 0); + assert_eq!(runtime.disk_execution.available_permits(), 0); + drop(lease_cleanup); + assert_eq!(runtime.disk_execution.available_permits(), 1); + } + + #[tokio::test] + async fn gc_scan_limit_leaves_disk_slots_available_between_entries() { + let runtime = Arc::new(CleanupRuntime::new(config(1, 1, 1)).unwrap()); + let first_runtime = runtime.clone(); + let (entered, entry) = oneshot::channel(); + let (release, released) = oneshot::channel(); + let first = tokio::spawn(async move { + first_runtime + .gc_operation(move |_| async move { + entered.send(()).unwrap(); + released.await.unwrap(); + Ok(()) + }) + .await + }); + deadline(entry).await.unwrap(); + assert_eq!(runtime.gc_execution.available_permits(), 0); + assert_eq!( + deadline(runtime.disk_operation(async { Ok(42) }, DiskAdmission::Request)) + .await + .unwrap(), + 42 + ); + let second = runtime.gc_operation(|_| async { Ok(()) }); + tokio::pin!(second); + assert!(futures::poll!(&mut second).is_pending()); + release.send(()).unwrap(); + deadline(first).await.unwrap().unwrap(); + deadline(second).await.unwrap(); + } + + #[tokio::test] + async fn gc_steps_borrow_scan_state_and_keep_budget_after_waiter_cancellation() { + let runtime = Arc::new(CleanupRuntime::new(config(1, 1, 1)).expect("cleanup runtime")); + let receiver = runtime.clone(); + let (entered, entry) = oneshot::channel(); + let (release, released) = std::sync::mpsc::channel(); + let waiter = tokio::spawn(async move { + receiver + .gc_operation(move |budget| async move { + let scan_task = tokio::task::id(); + let mut count = 0; + for _ in 0..32 { + budget + .step(async { + assert_eq!(tokio::task::id(), scan_task, "each entry must stay on the owned scan task"); + count += 1; + Ok(()) + }) + .await?; + } + assert_eq!(count, 32); + budget + .step(async move { + tokio::task::spawn_blocking(move || { + entered.send(()).expect("notify started syscall"); + released.recv_timeout(Duration::from_secs(20)).expect("release syscall"); + }) + .await + .expect("blocking cleanup"); + Ok(()) + }) + .await + }) + .await + }); + deadline(entry).await.expect("GC must start"); + waiter.abort(); + assert!(waiter.await.expect_err("cancelled waiter").is_cancelled()); + assert_eq!(runtime.gc_execution.available_permits(), 0); + assert_eq!(runtime.disk_admission.available_permits(), 3); + assert_eq!(runtime.disk_execution.available_permits(), 1); + release.send(()).expect("release blocking cleanup"); + deadline(async { + while runtime.gc_execution.available_permits() == 0 { + tokio::task::yield_now().await; + } + }) + .await; + assert_eq!(runtime.disk_admission.available_permits(), 4); + assert_eq!(runtime.disk_execution.available_permits(), 2); + } + + #[test] + fn defaults_bound_queue_waves_and_separate_threads_from_jobs() { + const CHILD: &str = "RUSTFS_CLEANUP_DEFAULTS_TEST_CHILD"; + if std::env::var_os(CHILD).is_some() { + let mut config = Config::from_env().expect("default configuration"); + let expected_cpus = match std::env::var(CPUS).as_deref() { + Ok("none") => Vec::new(), + Ok("3-4") => vec![3, 4], + _ if cfg!(target_os = "linux") => vec![1, 2], + _ => Vec::new(), + }; + assert_eq!(config.cpus, expected_cpus); + assert_eq!(config.workers, 64); + assert!(config.async_threads <= 2); + assert_eq!(config.blocking_threads, 4); + assert_eq!(config.disk_workers, 4); + assert_eq!(config.disk_pending, 64); + // Queue/thread defaults are independent of the host's CPU mask. + // The integration test exercises actual default/custom affinity. + config.cpus.clear(); + config.validate().expect("defaults must be internally consistent"); + let threads = config.async_threads; + let runtime = CleanupRuntime::new(config).expect("runtime with many logical jobs"); + assert_eq!(runtime.runtime.as_ref().expect("live runtime").metrics().num_workers(), threads); + return; + } + for cpus in [None, Some("3-4"), Some("none")] { + let mut command = std::process::Command::new(std::env::current_exe().expect("test executable")); + command + .args([ + "--exact", + "disk::cleanup_runtime::tests::defaults_bound_queue_waves_and_separate_threads_from_jobs", + ]) + .env(CHILD, "1") + .env(WORKERS, "64") + .env(BUDGET, "1024") + .env(GC_WORKERS, "1") + .env_remove(ASYNC_THREADS) + .env_remove(BLOCKING_THREADS) + .env_remove(DISK_WORKERS) + .env_remove(DISK_PENDING) + .env_remove(CPUS) + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP"); + if let Some(cpus) = cpus { + command.env(CPUS, cpus); + } + let output = command.output().expect("run defaults test in a fresh process"); + assert!(output.status.success(), "{}", String::from_utf8_lossy(&output.stderr)); + assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed")); + } + } + + #[tokio::test] + async fn affinity_failure_rejects_new_admission_without_dropping_admitted_work() { + let runtime = CleanupRuntime::new(config(2, 1, 1)).unwrap(); + let reservation = runtime.reserve().unwrap(); + runtime.affinity_failed.store(true, Ordering::Release); + assert!(runtime.reserve().is_err()); + assert_eq!(deadline(spawn(Some(reservation), async { 42 })).await.unwrap(), 42); + assert_eq!(runtime.admission.available_permits(), 2); + } + + #[tokio::test] + async fn admission_is_retained_by_every_commit_and_cleanup_owner() { + let runtime = CleanupRuntime::new(config(1, 1, 1)).unwrap(); + let request = runtime.reserve().unwrap(); + let commit = request.clone(); + let (release, released) = oneshot::channel(); + let cleanup = spawn(Some(commit.clone()), async move { + released.await.unwrap(); + }); + drop(request); + drop(commit); + assert!( + matches!(runtime.reserve(), Err(err) if err.kind() == io::ErrorKind::WouldBlock), + "queued and running cleanup must retain admission without waiting on external locks" + ); + release.send(()).unwrap(); + deadline(cleanup).await.unwrap(); + let next = runtime.reserve().unwrap(); + assert_eq!(runtime.admission.available_permits(), 0); + drop(next); + assert_eq!(runtime.admission.available_permits(), 1); + } + + #[tokio::test] + async fn cancelling_cleanup_waiter_does_not_cancel_work_or_release_budget() { + let runtime = CleanupRuntime::new(config(1, 1, 1)).unwrap(); + let reservation = runtime.reserve().unwrap(); + let (entered, entry) = oneshot::channel(); + let (release, released) = oneshot::channel(); + let waiter = tokio::spawn(run(Some(reservation), async move { + entered.send(()).unwrap(); + released.await.unwrap(); + })); + deadline(entry).await.unwrap(); + waiter.abort(); + assert!(waiter.await.unwrap_err().is_cancelled()); + assert_eq!(runtime.admission.available_permits(), 0); + assert_eq!(runtime.execution.available_permits(), 0); + release.send(()).unwrap(); + deadline(async { + loop { + if let Ok(reservation) = runtime.reserve() { + drop(reservation); + break; + } + tokio::task::yield_now().await; + } + }) + .await; + assert_eq!(runtime.admission.available_permits(), 1); + } + + #[tokio::test] + async fn worker_and_overflow_dispatch_share_one_execution_budget() { + let runtime = CleanupRuntime::new(config(8, 2, 1)).unwrap(); + let active = Arc::new(AtomicUsize::new(0)); + let peak = Arc::new(AtomicUsize::new(0)); + let gate = Arc::new(Semaphore::new(0)); + let (entered, mut entries) = tokio::sync::mpsc::unbounded_channel(); + let mut jobs = Vec::new(); + for index in 0..8 { + let reservation = runtime.reserve().unwrap(); + let active = active.clone(); + let peak = peak.clone(); + let gate = gate.clone(); + let entered = entered.clone(); + let work = async move { + let running = active.fetch_add(1, Ordering::SeqCst) + 1; + peak.fetch_max(running, Ordering::SeqCst); + entered.send(()).unwrap(); + gate.acquire().await.unwrap().forget(); + active.fetch_sub(1, Ordering::SeqCst); + }; + // Both a dequeued job and a queue-full continuation enter `run`; + // no-tail temporary cleanup uses direct owned dispatch. + jobs.push(if index % 2 == 0 { + tokio::spawn(run(Some(reservation), work)) + } else { + spawn(Some(reservation), work) + }); + } + deadline(entries.recv()).await.unwrap(); + deadline(entries.recv()).await.unwrap(); + assert_eq!(runtime.execution.available_permits(), 0); + assert!(entries.try_recv().is_err()); + assert_eq!(runtime.admission.available_permits(), 0); + gate.add_permits(8); + for job in jobs { + deadline(job).await.unwrap(); + } + assert_eq!(peak.load(Ordering::SeqCst), 2); + assert_eq!(runtime.admission.available_permits(), 8); + } + + #[tokio::test] + async fn cleanup_children_and_blocking_work_use_dedicated_threads() { + let runtime = CleanupRuntime::new(config(1, 1, 1)).unwrap(); + let task = spawn(Some(runtime.reserve().unwrap()), async move { + assert_eq!(std::thread::current().name(), Some("rustfs-cleanup")); + tokio::spawn(async { + assert_eq!(std::thread::current().name(), Some("rustfs-cleanup")); + tokio::task::spawn_blocking(|| { + assert_eq!(std::thread::current().name(), Some("rustfs-cleanup")); + }) + .await + .unwrap(); + }) + .await + .unwrap(); + }); + deadline(task).await.unwrap(); + assert_ne!(std::thread::current().name(), Some("rustfs-cleanup")); + } + + #[tokio::test] + async fn blocking_pool_saturation_does_not_block_foreground_pool() { + let runtime = CleanupRuntime::new(config(2, 2, 1)).unwrap(); + let (entered, entry) = oneshot::channel(); + let (release, released) = std::sync::mpsc::channel(); + let first = spawn(Some(runtime.reserve().unwrap()), async move { + tokio::task::spawn_blocking(move || { + entered.send(()).unwrap(); + released.recv_timeout(Duration::from_secs(20)).unwrap(); + }) + .await + .unwrap(); + }); + deadline(entry).await.unwrap(); + let (submitted, submission) = oneshot::channel(); + let (second_entered, mut second_entry) = oneshot::channel(); + let second = spawn(Some(runtime.reserve().unwrap()), async move { + let task = tokio::task::spawn_blocking(move || { + second_entered.send(()).unwrap(); + }); + submitted.send(()).unwrap(); + task.await.unwrap(); + }); + deadline(submission).await.unwrap(); + assert!(matches!(second_entry.try_recv(), Err(oneshot::error::TryRecvError::Empty))); + assert_eq!(deadline(tokio::task::spawn_blocking(|| 42)).await.unwrap(), 42); + release.send(()).unwrap(); + deadline(first).await.unwrap(); + deadline(second).await.unwrap(); + deadline(second_entry).await.unwrap(); + } + + #[cfg(target_os = "linux")] + #[tokio::test] + async fn affinity_covers_async_and_blocking_threads_without_changing_caller() { + let caller = rustix::thread::sched_getaffinity(None).unwrap(); + let cpu = (0..rustix::thread::CpuSet::MAX_CPU).find(|cpu| caller.is_set(*cpu)).unwrap(); + let mut config = config(1, 1, 1); + config.cpus = vec![cpu]; + let runtime = CleanupRuntime::new(config).unwrap(); + deadline(spawn(Some(runtime.reserve().unwrap()), async move { + let expected = { + let mut set = rustix::thread::CpuSet::new(); + set.set(cpu); + set + }; + assert_eq!(rustix::thread::sched_getaffinity(None).unwrap(), expected); + let actual = tokio::task::spawn_blocking(|| rustix::thread::sched_getaffinity(None).unwrap()) + .await + .unwrap(); + assert_eq!(actual, expected); + })) + .await + .unwrap(); + assert_eq!(rustix::thread::sched_getaffinity(None).unwrap(), caller); + assert!(validate_cpus(&[rustix::thread::CpuSet::MAX_CPU]).is_err()); + } + + #[cfg(not(target_os = "linux"))] + #[test] + fn explicit_cpu_affinity_is_rejected_on_unsupported_platforms() { + assert_eq!(validate_cpus(&[0]).unwrap_err().kind(), io::ErrorKind::Unsupported); + validate_cpus(&[]).unwrap(); + } +} diff --git a/crates/ecstore/src/disk/disk_store.rs b/crates/ecstore/src/disk/disk_store.rs index 95c9fd941..0871aa124 100644 --- a/crates/ecstore/src/disk/disk_store.rs +++ b/crates/ecstore/src/disk/disk_store.rs @@ -12,14 +12,15 @@ // See the License for the specific language governing permissions and // limitations under the License. +use crate::disk::cleanup_runtime; use crate::disk::io_admission::{ AdmissionReader, BoxedChunkReader, DiskIoAdmission, DiskIoClass, DiskIoContext, current_disk_io_context, with_disk_io_context, with_disk_io_permit, }; use crate::disk::{ CheckPartsResp, DataDirDeleteStatus, DeleteOptions, DiskAPI, DiskError, DiskInfo, DiskInfoOptions, DiskLocation, Endpoint, - Error, FileInfoVersions, MmapCopyStageMetrics, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, Result, - SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, + Error, FileInfoVersions, GuardedFileInfoVersions, MmapCopyStageMetrics, ReadMultipleReq, ReadMultipleResp, ReadOptions, + RenameDataResp, Result, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, health_state::{ RuntimeDriveHealthState, classify_drive_recovery, get_drive_returning_probe_interval, get_drive_returning_success_threshold, get_drive_suspect_failure_threshold, record_drive_offline_duration, @@ -466,12 +467,11 @@ impl LocalDiskWrapper { self.track_disk_health_mutation( "delete_version", DiskMetricMutation::Delete, - || async { + || { Box::pin( self.disk .delete_version_with_namespace_owner(volume, path, fi, force_del_marker, opts, namespace_owner), ) - .await }, get_max_timeout_duration(), ) @@ -485,10 +485,10 @@ impl LocalDiskWrapper { opts: DeleteOptions, namespace_owner: Option>, ) -> Result<()> { - self.track_disk_health_mutation( + self.track_cleanup_mutation( "delete", DiskMetricMutation::Delete, - || async { Box::pin(self.disk.delete_with_namespace_owner(volume, path, opts, namespace_owner)).await }, + || async { Box::pin(self.cleanup_delete(volume, path, opts, namespace_owner)).await }, get_max_timeout_duration(), ) .await @@ -1539,6 +1539,25 @@ impl LegacyFileWrite { .await } + pub async fn delete_versions_guarded( + self, + versions: Vec, + opts: DeleteOptions, + ) -> Vec> { + let payload = super::GuardedDeleteVersionsPayload { versions, opts }; + if let Err(error) = payload.validate() { + return vec![Some(error); payload.versions.len()]; + } + let super::GuardedDeleteVersionsPayload { versions, mut opts } = payload; + opts.recoverable_delete = true; + with_disk_io_permit( + Some(self.permit.clone()), + self.disk + .execute_guarded_delete_versions(&self.volume, versions, opts, None, Some(self.owner)), + ) + .await + } + pub async fn write_all(&self, path: &str, data: Bytes) -> Result<()> { crate::disk::io_admission::with_disk_io_permit( Some(self.permit.clone()), @@ -1759,7 +1778,28 @@ enum PathDelete<'a> { Many(&'a [String]), } +enum OwnedPathDelete { + One(String, DeleteOptions), + Many(Vec), +} + +impl OwnedPathDelete { + async fn apply(self, disk: Arc, volume: String, owner: Arc) -> Result<()> { + match self { + Self::One(path, options) => PathDelete::One(&path, options).apply(&disk, &volume, owner).await, + Self::Many(paths) => PathDelete::Many(&paths).apply(&disk, &volume, owner).await, + } + } +} + impl PathDelete<'_> { + fn into_owned(self) -> OwnedPathDelete { + match self { + Self::One(path, options) => OwnedPathDelete::One(path.to_owned(), options), + Self::Many(paths) => OwnedPathDelete::Many(paths.to_vec()), + } + } + fn operation(&self) -> LegacyPathDeleteOperation { match self { Self::One(..) => LegacyPathDeleteOperation::Delete, @@ -1895,6 +1935,36 @@ impl LegacyMetadataWrite { .await } + pub async fn settle_delete_transaction( + self, + transaction: Uuid, + expected_fingerprint: super::DeleteRecoveryDigest, + action: super::DeleteRecoveryAction, + ) -> Result { + with_disk_io_permit( + Some(self.permit.clone()), + self.disk.track_disk_health_with_deadline( + "settle_delete_transaction", + |_permit| async { + Box::pin(self.disk.disk.delete_recovery_settle_prepared( + &self.volume, + &self.path, + transaction, + expected_fingerprint, + action, + self.publication.clone(), + )) + .await + }, + super::get_delete_recovery_timeout()?, + TimeoutHealthAction::MarkFailure, + DiskMetricMutation::Delete, + self.deadline, + ), + ) + .await + } + pub async fn update_metadata(self, fi: FileInfo, opts: &UpdateMetadataOpts) -> Result<()> { if opts.transition_reconcile.is_some() { return Err(DiskError::MethodNotAllowed); @@ -2196,21 +2266,43 @@ impl LocalDiskWrapper { action: PathDelete<'_>, external_guard: Option>, ) -> Result<()> { - let duration = get_max_timeout_duration(); - let deadline = tokio::time::Instant::now() + duration; + let duration = if external_guard.is_some() { + Duration::ZERO + } else { + get_max_timeout_duration() + }; + let deadline = external_guard.is_none().then(|| tokio::time::Instant::now() + duration); let permit = self.io_admission.foreground(); + let cleanup_enabled = cleanup_runtime::enabled()?; + let guarded = external_guard.is_some(); with_disk_io_permit( Some(permit.clone()), self.track_disk_health_with_deadline( action.operation().name(), |_permit| async { let owner = self.disk.begin_legacy_file_write(volume, permit).await?; - action.apply(&self.disk, volume, Arc::new((owner, external_guard))).await + let physical_owner: Arc = Arc::new((owner, external_guard)); + if cleanup_enabled { + let mutation = action + .into_owned() + .apply(self.disk.clone(), volume.to_owned(), physical_owner); + if guarded { + cleanup_runtime::guarded_disk_operation(mutation).await + } else { + cleanup_runtime::disk_operation(mutation).await + } + } else { + action.apply(&self.disk, volume, physical_owner).await + } }, duration, - TimeoutHealthAction::MarkFailure, + if cleanup_enabled { + TimeoutHealthAction::IgnoreFailure + } else { + TimeoutHealthAction::MarkFailure + }, DiskMetricMutation::Delete, - Some(deadline), + deadline, ), ) .await @@ -2449,6 +2541,28 @@ impl LocalDiskWrapper { ) } + async fn cleanup_delete( + &self, + volume: &str, + path: &str, + options: DeleteOptions, + owner: Option>, + ) -> Result<()> { + if !cleanup_runtime::enabled()? { + return self.disk.delete_with_namespace_owner(volume, path, options, owner).await; + } + let disk = self.disk.clone(); + let volume = volume.to_owned(); + let path = path.to_owned(); + let guarded = owner.is_some(); + let operation = async move { disk.delete_with_namespace_owner(&volume, &path, options, owner).await }; + if guarded { + cleanup_runtime::guarded_disk_operation(operation).await + } else { + cleanup_runtime::disk_operation(operation).await + } + } + /// Run a delete under an owned coordinator task when a publication guard /// is present. This keeps the guard alive if the RPC waiter is cancelled /// while the local namespace mutation is still in progress. @@ -2770,19 +2884,7 @@ impl LocalDiskWrapper { } // Clean up - disk.delete( - test_bucket, - test_filename, - DeleteOptions { - recursive: false, - immediate: false, - undo_write: false, - undo_delete: false, - old_data_dir: None, - expected_delete_marker: None, - }, - ) - .await?; + disk.delete(test_bucket, test_filename, DeleteOptions::default()).await?; Ok(()) }) @@ -3031,10 +3133,155 @@ impl LocalDiskWrapper { } /// Get the current disk ID + pub(crate) async fn delete_versions_with_namespace_owner( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + namespace_owner: Option>, + ) -> Vec> { + let version_count = versions.len(); + self.metrics.record_operation_call("delete_versions"); + // Check if disk is faulty before proceeding + if self.health.is_faulty() { + self.metrics.record_availability_error(); + return vec![Some(DiskError::FaultyDisk); versions.len()]; + } + + // Check if disk is stale + if let Err(e) = self.check_disk_stale().await { + self.metrics.record_availability_error(); + return vec![Some(e); versions.len()]; + } + + let _io_permit = match self.acquire_disk_io_permit().await { + Ok(permit) => permit, + Err(error) => return vec![Some(error); version_count], + }; + + // Record operation start + self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed); + let waiting_guard = self.health.waiting_guard(); + let metric_waiting_guard = self.metrics.waiting_guard(); + let started = Instant::now(); + + // Execute the operation + let result = self + .disk + .delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner) + .await; + self.metrics.record_operation_latency("delete_versions", started.elapsed()); + self.record_batch_delete_error_metrics(&result); + + drop(waiting_guard); + drop(metric_waiting_guard); + let has_err = result.iter().any(|e| e.is_some()); + if !has_err { + // Log success and decrement waiting counter + self.health.record_operation_success(&self.endpoint(), "operation_success"); + self.metrics.record_mutation_success(DiskMetricMutation::Delete); + } + + result + } + + pub(crate) async fn delete_versions_guarded_with_namespace_owner( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + namespace_owner: Option>, + ) -> Vec> { + self.execute_guarded_delete_versions(volume, versions, opts, namespace_owner, None) + .await + } + + async fn execute_guarded_delete_versions( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + namespace_owner: Option>, + admitted_volume_owner: Option>, + ) -> Vec> { + let version_count = versions.len(); + self.metrics.record_operation_call("delete_versions"); + // Check if disk is faulty before proceeding + if self.health.is_faulty() { + self.metrics.record_availability_error(); + return vec![Some(DiskError::FaultyDisk); versions.len()]; + } + + // Check if disk is stale + if let Err(e) = self.check_disk_stale().await { + self.metrics.record_availability_error(); + return vec![Some(e); versions.len()]; + } + + let _io_permit = match self.acquire_disk_io_permit().await { + Ok(permit) => permit, + Err(error) => return vec![Some(error); version_count], + }; + + // Record operation start + self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed); + let waiting_guard = self.health.waiting_guard(); + let metric_waiting_guard = self.metrics.waiting_guard(); + let started = Instant::now(); + + // Execute the operation + let result = if let Some(owner) = admitted_volume_owner { + self.disk + .delete_versions_guarded_with_owner(volume, versions, opts, owner) + .await + } else { + self.disk + .delete_versions_guarded_with_namespace_owner(volume, versions, opts, namespace_owner) + .await + }; + self.metrics.record_operation_latency("delete_versions", started.elapsed()); + self.record_batch_delete_error_metrics(&result); + + drop(waiting_guard); + drop(metric_waiting_guard); + let has_err = result.iter().any(|e| e.is_some()); + if !has_err { + // Log success and decrement waiting counter + self.health.record_operation_success(&self.endpoint(), "operation_success"); + self.metrics.record_mutation_success(DiskMetricMutation::Delete); + } + + result + } + pub async fn get_current_disk_id(&self) -> Option { *self.disk_id.read().await } + pub(crate) async fn settle_delete_transaction_with_namespace_owner( + &self, + volume: &str, + object: &str, + transaction: uuid::Uuid, + expected_fingerprint: crate::disk::DeleteRecoveryDigest, + action: crate::disk::DeleteRecoveryAction, + namespace_owner: Option>, + ) -> Result { + self.track_disk_health_mutation( + "settle_delete_transaction", + DiskMetricMutation::Delete, + || { + Box::pin(async { + self.disk + .delete_recovery_settle(volume, object, transaction, expected_fingerprint, action, namespace_owner) + .await + }) + }, + crate::disk::get_delete_recovery_timeout()?, + ) + .await + } + /// Track disk health for an operation. /// This method should wrap disk operations to ensure health checking. pub async fn track_disk_health(&self, operation: F, timeout_duration: Duration) -> Result @@ -3045,6 +3292,31 @@ impl LocalDiskWrapper { self.track_disk_health_with_op("unknown", operation, timeout_duration).await } + /// Cleanup latency includes waiting behind a node-wide execution budget, + /// so a waiter deadline alone is not evidence that this disk is faulty. + /// The active disk health probe retains its original timeout policy. + async fn track_cleanup_mutation( + &self, + op: &'static str, + mutation: DiskMetricMutation, + operation: F, + timeout_duration: Duration, + ) -> Result + where + F: FnOnce() -> Fut, + Fut: std::future::Future>, + { + // Runtime initialization must not prevent control-plane release of + // reader tokens. Disk I/O entries still propagate configuration errors. + let action = if cleanup_runtime::enabled().unwrap_or(false) { + TimeoutHealthAction::IgnoreFailure + } else { + TimeoutHealthAction::MarkFailure + }; + self.track_disk_health_with_op_timeout_action_and_mutation(op, operation, timeout_duration, action, mutation) + .await + } + async fn track_disk_health_mutation( &self, op: &'static str, @@ -3270,6 +3542,110 @@ impl LocalDiskWrapper { #[async_trait::async_trait] impl DiskAPI for LocalDiskWrapper { + async fn supports_guarded_delete_versions(&self) -> Result<()> { + // The local implementation shares this binary's capability. Availability + // and disk identity remain mutation checks and write-quorum inputs. + self.disk.supports_guarded_delete_versions().await + } + async fn delete_versions_guarded_recoverable( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + ) -> Vec> { + self.delete_versions_guarded_with_namespace_owner(volume, versions, opts, None) + .await + } + + async fn delete_version_recoverable( + &self, + volume: &str, + object: &str, + file_info: FileInfo, + force_del_marker: bool, + mut opts: DeleteOptions, + ) -> Result<()> { + crate::disk::validate_delete_recovery_options(&opts)?; + if object.starts_with('/') { + return Err(DiskError::InvalidPath); + } + opts.recoverable_delete = true; + self.delete_version(volume, object, file_info, force_del_marker, opts).await + } + async fn delete_versions_recoverable( + &self, + volume: &str, + versions: Vec, + mut opts: DeleteOptions, + ) -> Vec> { + if let Err(err) = crate::disk::validate_delete_recovery_options(&opts) { + return vec![Some(err); versions.len()]; + } + if versions.iter().any(|version| version.name.starts_with('/')) { + return vec![Some(DiskError::InvalidPath); versions.len()]; + } + opts.recoverable_delete = true; + self.delete_versions(volume, versions, opts).await + } + async fn has_delete_transaction_evidence(&self, volume: &str, object: &str) -> Result { + self.track_disk_health_with_op( + "has_delete_transaction_evidence", + || async { self.disk.has_delete_transaction_evidence(volume, object).await }, + get_max_timeout_duration(), + ) + .await + } + async fn discover_delete_transactions( + &self, + volume: &str, + object: Option<&str>, + cursor: Option, + limit: u32, + ) -> Result { + self.track_disk_health_with_op( + "discover_delete_transactions", + || async { + self.disk + .discover_delete_transactions(volume, object, cursor.clone(), limit) + .await + }, + get_max_timeout_duration(), + ) + .await + } + async fn inspect_delete_transaction( + &self, + volume: &str, + object: &str, + transaction: uuid::Uuid, + ) -> Result { + self.track_disk_health_with_op( + "inspect_delete_transaction", + || async { self.disk.inspect_delete_transaction(volume, object, transaction).await }, + get_max_timeout_duration(), + ) + .await + } + async fn settle_delete_transaction( + &self, + volume: &str, + object: &str, + transaction: uuid::Uuid, + expected_fingerprint: crate::disk::DeleteRecoveryDigest, + action: crate::disk::DeleteRecoveryAction, + ) -> Result { + self.track_disk_health_mutation( + "settle_delete_transaction", + DiskMetricMutation::Delete, + || async { + self.disk + .settle_delete_transaction(volume, object, transaction, expected_fingerprint, action) + .await + }, + crate::disk::get_delete_recovery_timeout()?, + ) + .await + } fn has_replacement_mount_lease(&self) -> bool { self.disk.has_replacement_mount_lease() } @@ -3500,9 +3876,37 @@ impl DiskAPI for LocalDiskWrapper { } async fn release_snapshot_lease(&self, volume: &str, path: &str, token: SnapshotLeaseToken) -> Result<()> { - self.track_disk_health_with_op( + // Quota fence release is control-plane work and may wait for in-flight + // mutations. Keep its original waiter lifecycle instead of creating + // detached cleanup tasks for repeated fence-release RPCs. + if volume == super::RUSTFS_META_BUCKET && super::is_quota_mutation_fence_path(path) { + return self + .track_disk_health_with_op( + "release_snapshot_lease", + || async { self.disk.release_snapshot_lease(volume, path, token).await }, + get_max_timeout_duration(), + ) + .await; + } + self.track_cleanup_mutation( "release_snapshot_lease", - || async { self.disk.release_snapshot_lease(volume, path, token).await }, + DiskMetricMutation::None, + || async { + if self.disk.release_snapshot_lease_without_cleanup(volume, path, token).await { + return Ok(()); + } + if !cleanup_runtime::enabled().unwrap_or(false) { + // The raw release removes the token before trying the + // reclaim budget. Initialization failure therefore keeps + // the cleanup intent and data, without leaking a reader + // or falling back to unbudgeted filesystem deletion. + return self.disk.release_snapshot_lease(volume, path, token).await; + } + let disk = self.disk.clone(); + let volume = volume.to_owned(); + let path = path.to_owned(); + cleanup_runtime::release_lease(async move { disk.release_snapshot_lease(&volume, &path, token).await }).await + }, get_max_timeout_duration(), ) .await @@ -3527,10 +3931,22 @@ impl DiskAPI for LocalDiskWrapper { } else { get_max_timeout_duration() }; - self.track_disk_health_mutation( + self.track_cleanup_mutation( "delete_data_dir", DiskMetricMutation::Delete, - || async { self.disk.delete_data_dir(volume, path, opts).await }, + || async { + if !cleanup_runtime::enabled()? { + return self.disk.delete_data_dir(volume, path, opts).await; + } + let disk = self.disk.clone(); + let volume = volume.to_owned(); + let path = path.to_owned(); + cleanup_runtime::disk_operation(async move { + let _scope = scope; + disk.delete_data_dir(&volume, &path, opts).await + }) + .await + }, timeout, ) .await @@ -3565,6 +3981,7 @@ impl DiskAPI for LocalDiskWrapper { .await } + #[track_caller] async fn read_version( &self, org_volume: &str, @@ -3573,7 +3990,7 @@ impl DiskAPI for LocalDiskWrapper { version_id: &str, opts: &ReadOptions, ) -> Result { - self.track_disk_health_with_op_timeout_action_and_mutation_with_io_permit( + let operation = self.track_disk_health_with_op_timeout_action_and_mutation_with_io_permit( "read_version", |io_permit| async move { with_disk_io_permit(io_permit, self.disk.read_version(org_volume, volume, path, version_id, opts)).await @@ -3581,8 +3998,20 @@ impl DiskAPI for LocalDiskWrapper { get_max_timeout_duration(), TimeoutHealthAction::MarkFailure, DiskMetricMutation::None, - ) - .await + ); + #[cfg(feature = "diagnostic-read-version-caller")] + { + if crate::cluster::rpc::runtime_sources::current_read_version_request_caller().is_some() { + operation.await + } else { + let caller = crate::cluster::rpc::runtime_sources::read_version_request_caller_from_location( + std::panic::Location::caller().file(), + ); + crate::cluster::rpc::runtime_sources::with_read_version_request_caller(caller, operation).await + } + } + #[cfg(not(feature = "diagnostic-read-version-caller"))] + operation.await } async fn read_xl(&self, volume: &str, path: &str, read_data: bool) -> Result { @@ -3788,11 +4217,25 @@ impl DiskAPI for LocalDiskWrapper { } async fn write_all(&self, volume: &str, path: &str, data: Bytes) -> Result<()> { - self.track_disk_health_mutation( + let cleanup = path.rsplit('/').next() == Some(cleanup_runtime::OLD_DATA_CLEANUP_RECEIPT_FILE); + self.track_disk_health_with_op_timeout_action_and_mutation( "write_all", - DiskMetricMutation::Write, - || async { self.disk.write_all(volume, path, data).await }, + || async { + if !cleanup || !cleanup_runtime::enabled()? { + return self.disk.write_all(volume, path, data).await; + } + let disk = self.disk.clone(); + let volume = volume.to_owned(); + let path = path.to_owned(); + cleanup_runtime::disk_operation(async move { disk.write_all(&volume, &path, data).await }).await + }, get_max_timeout_duration(), + if cleanup && cleanup_runtime::enabled()? { + TimeoutHealthAction::IgnoreFailure + } else { + TimeoutHealthAction::MarkFailure + }, + DiskMetricMutation::Write, ) .await } @@ -4312,6 +4755,256 @@ mod tests { let _ = walk.await; } + #[tokio::test] + async fn invalid_cleanup_config_does_not_leak_snapshot_readers() { + const CHILD: &str = "RUSTFS_CLEANUP_INVALID_LEASE_TEST_CHILD"; + if std::env::var_os(CHILD).is_none() { + let output = tokio::task::spawn_blocking(|| { + std::process::Command::new(std::env::current_exe().expect("test executable")) + .args([ + "--exact", + "disk::disk_store::tests::invalid_cleanup_config_does_not_leak_snapshot_readers", + "--nocapture", + ]) + .env(CHILD, "1") + .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true") + .env("RUSTFS_CLEANUP_ASYNC_THREADS", "0") + .output() + .expect("run with invalid cleanup configuration") + }) + .await + .expect("child process task"); + assert!( + output.status.success(), + "{}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed")); + return; + } + let dir = tempfile::tempdir().expect("temporary disk"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk")); + let wrapper = LocalDiskWrapper::new(disk.clone(), false); + let volume = "invalid-cleanup-config"; + let path = "object/data"; + wrapper.make_volume(volume).await.expect("bucket"); + wrapper + .write_all(volume, "object/data/part.1", Bytes::from_static(b"live shard")) + .await + .expect("write shard"); + let token = wrapper.acquire_snapshot_lease(volume, path).await.expect("GET lease"); + wrapper + .release_snapshot_lease(volume, path, token) + .await + .expect("control-only release must not depend on cleanup initialization"); + assert!(matches!( + wrapper.renew_snapshot_lease(volume, path, token).await, + Err(DiskError::FileNotFound) + )); + + let token = wrapper + .acquire_snapshot_lease(volume, path) + .await + .expect("reader of old data"); + // Register reclaim intent directly; this only updates the lease + // registry while a reader is present and performs no filesystem I/O. + assert_eq!( + disk.delete_data_dir( + volume, + path, + DeleteOptions { + recursive: true, + ..Default::default() + } + ) + .await + .expect("defer reclaim"), + DataDirDeleteStatus::Deferred + ); + assert!( + wrapper.release_snapshot_lease(volume, path, token).await.is_err(), + "invalid configuration must still reject reclamation" + ); + assert!( + matches!(wrapper.renew_snapshot_lease(volume, path, token).await, Err(DiskError::FileNotFound)), + "failed reclamation must not strand a reader token" + ); + assert_eq!( + wrapper.read_all(volume, "object/data/part.1").await.expect("retained data"), + Bytes::from_static(b"live shard"), + "configuration failure must not fall back to unbudgeted deletion" + ); + } + + #[tokio::test] + async fn rpc_cleanup_entries_share_disk_budget() { + const CHILD: &str = "RUSTFS_CLEANUP_RPC_TEST_CHILD"; + if std::env::var_os(CHILD).is_none() { + let output = tokio::task::spawn_blocking(|| { + std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "disk::disk_store::tests::rpc_cleanup_entries_share_disk_budget", + "--nocapture", + ]) + .env(CHILD, "1") + .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true") + .env("RUSTFS_CLEANUP_ASYNC_THREADS", "1") + .env("RUSTFS_CLEANUP_DISK_MAX_PENDING", "1") + .env("RUSTFS_CLEANUP_DISK_WORKERS", "1") + .env("RUSTFS_CLEANUP_GC_WORKERS", "1") + .env("RUSTFS_CLEANUP_CPUS", "none") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP") + .output() + .unwrap() + }) + .await + .unwrap(); + assert!( + output.status.success(), + "{}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed")); + return; + } + let dir = tempfile::tempdir().unwrap(); + let endpoint = Endpoint::try_from(dir.path().to_str().unwrap()).unwrap(); + let disk = Arc::new(LocalDisk::new(&endpoint, false).await.unwrap()); + let wrapper = LocalDiskWrapper::new(disk, false); + let volume = "cleanup-receiver"; + let path = "object/old-data"; + let body_path = format!("{path}/part.1"); + let receipt_path = format!("{path}/{}", cleanup_runtime::OLD_DATA_CLEANUP_RECEIPT_FILE); + wrapper.make_volume(volume).await.unwrap(); + wrapper + .write_all(volume, &body_path, Bytes::from_static(b"old body")) + .await + .unwrap(); + let read_lease = wrapper + .acquire_snapshot_lease(volume, path) + .await + .expect("ordinary GET lease"); + let (entered, entry) = tokio::sync::oneshot::channel(); + let (release, released) = std::sync::mpsc::channel(); + let occupied = tokio::spawn(cleanup_runtime::disk_operation(async move { + entered.send(()).unwrap(); + // Deliberately stall the only cleanup scheduler to prove that + // ordinary GET completion does not dispatch to it. + released + .recv_timeout(Duration::from_secs(30)) + .expect("release stalled scheduler"); + Ok(()) + })); + tokio::time::timeout(Duration::from_secs(20), entry).await.unwrap().unwrap(); + tokio::time::timeout(Duration::from_secs(5), wrapper.release_snapshot_lease(volume, path, read_lease)) + .await + .expect("ordinary GET release must progress while the cleanup scheduler is stalled") + .expect("release without reclamation"); + let fence_path = super::super::quota_mutation_fence_path(volume, "object"); + let fence = wrapper + .acquire_snapshot_lease(super::super::RUSTFS_META_BUCKET, &fence_path) + .await + .unwrap(); + wrapper + .release_snapshot_lease(super::super::RUSTFS_META_BUCKET, &fence_path, fence) + .await + .unwrap(); + let timeout = wrapper + .track_cleanup_mutation( + "delete", + DiskMetricMutation::Delete, + std::future::pending::>, + Duration::from_millis(10), + ) + .await; + assert!(matches!(timeout, Err(DiskError::Timeout))); + assert!(!wrapper.health.is_faulty(), "cleanup queue delay must not mark a healthy disk faulty"); + let options = DeleteOptions { + recursive: true, + ..Default::default() + }; + let drops = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + // This is the same entry point used after RPC authentication and + // scanner publication-lease validation by handle_delete. + let result = wrapper + .delete_with_publication_guard(volume, path, options.clone(), Some(Arc::new(DropProbe(drops.clone())))) + .await; + assert!(matches!(result, Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + assert_eq!(drops.load(Ordering::SeqCst), 1); + assert!(matches!(wrapper.delete_data_dir(volume, path, options.clone()).await, + Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + assert!(matches!(wrapper.write_all(volume, &receipt_path, Bytes::from_static(b"receipt")).await, + Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + assert_eq!(wrapper.read_all(volume, &body_path).await.unwrap(), Bytes::from_static(b"old body")); + release.send(()).unwrap(); + occupied.await.unwrap().unwrap(); + wrapper + .write_all(volume, &receipt_path, Bytes::from_static(b"receipt")) + .await + .unwrap(); + wrapper + .delete_with_publication_guard(volume, path, options, None) + .await + .unwrap(); + assert!(matches!(wrapper.read_all(volume, &body_path).await, Err(DiskError::FileNotFound))); + // Snapshot readers defer old-data cleanup. Saturation must release + // their token while retaining reclaim intent, rather than leaking a + // live token or performing an unbudgeted delete on the RPC runtime. + wrapper + .write_all(volume, &body_path, Bytes::from_static(b"leased body")) + .await + .unwrap(); + let lease = wrapper.acquire_snapshot_lease(volume, path).await.unwrap(); + assert_eq!( + wrapper + .delete_data_dir( + volume, + path, + DeleteOptions { + recursive: true, + ..Default::default() + } + ) + .await + .unwrap(), + DataDirDeleteStatus::Deferred + ); + let (entered, entry) = tokio::sync::oneshot::channel(); + let (release, released) = tokio::sync::oneshot::channel(); + let occupied = tokio::spawn(cleanup_runtime::disk_operation(async move { + entered.send(()).unwrap(); + released.await.unwrap(); + Ok(()) + })); + tokio::time::timeout(Duration::from_secs(20), entry).await.unwrap().unwrap(); + assert!(matches!(wrapper.release_snapshot_lease(volume, path, lease).await, + Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + assert_eq!(wrapper.read_all(volume, &body_path).await.unwrap(), Bytes::from_static(b"leased body")); + release.send(()).unwrap(); + occupied.await.unwrap().unwrap(); + assert_eq!( + wrapper + .delete_data_dir( + volume, + path, + DeleteOptions { + recursive: true, + ..Default::default() + } + ) + .await + .unwrap(), + DataDirDeleteStatus::Deleted + ); + assert!(matches!(wrapper.read_all(volume, &body_path).await, Err(DiskError::FileNotFound))); + } + #[tokio::test] async fn owned_mutation_keeps_publication_guard_after_waiter_cancellation() { let drops = Arc::new(std::sync::atomic::AtomicUsize::new(0)); @@ -4581,6 +5274,97 @@ mod tests { assert_eq!(snapshot.total_errors_availability, 1); } + #[tokio::test] + async fn local_guarded_delete_capability_is_independent_of_faulty_or_stale_mutation_rejection() { + for stale in [false, true] { + let dir = tempfile::tempdir().expect("create local disk directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse local endpoint"); + let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("create local disk")); + let volume = "guarded-capability"; + let object = "object"; + disk.make_volume(volume).await.expect("create object volume"); + let mut info = FileInfo::new(object, 1, 0); + info.name = object.to_owned(); + info.version_id = Some(Uuid::nil()); + info.data_dir = Some(Uuid::new_v4()); + info.mod_time = Some(::time::OffsetDateTime::now_utc()); + info.erasure.index = 1; + info.size = 8; + info.parts = vec![ObjectPartInfo { + number: 1, + size: 8, + actual_size: 8, + ..Default::default() + }]; + info.data = Some(Bytes::from_static(b"original")); + info.metadata.insert("etag".into(), "etag-a".into()); + info.set_inline_data(); + disk.write_metadata(volume, volume, object, info.clone()) + .await + .expect("publish guarded fixture metadata"); + let generation = crate::disk::CurrentDeleteGeneration::from_file_info(&info).expect("capture object generation"); + let wrapper = LocalDiskWrapper::new(Arc::clone(&disk), false); + let expected_error = if stale { + let mut format_info = disk.format_info.write().await; + format_info.id = Some(Uuid::new_v4()); + format_info.file_info = Some(tokio::fs::metadata(dir.path()).await.expect("local disk directory metadata")); + format_info.last_check = Some(::time::OffsetDateTime::now_utc()); + drop(format_info); + wrapper.set_disk_id_state(Some(Uuid::new_v4())).await; + DiskError::DiskNotFound + } else { + wrapper.force_offline_for_test(); + DiskError::FaultyDisk + }; + let object_dir = dir.path().join(volume).join(object); + let metadata_path = object_dir.join(crate::disk::STORAGE_FORMAT_FILE); + let before = tokio::fs::read(&metadata_path).await.expect("read original metadata"); + wrapper + .supports_guarded_delete_versions() + .await + .expect("local binary capability is known even when the drive is unavailable"); + assert_eq!( + wrapper.metrics_snapshot().total_errors_availability, + 0, + "capability does not probe drive availability" + ); + let transaction = Uuid::new_v4(); + let result = wrapper + .delete_versions_guarded_recoverable( + volume, + vec![GuardedFileInfoVersions { + group: FileInfoVersions { + name: object.to_owned(), + versions: vec![FileInfo { + name: object.to_owned(), + version_id: info.version_id, + ..Default::default() + }], + ..Default::default() + }, + expected_current: generation, + }], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await; + assert_eq!(result, vec![Some(expected_error)], "mutation availability checks remain enforced"); + assert_eq!( + tokio::fs::read(&metadata_path) + .await + .expect("metadata survives rejected mutation"), + before + ); + assert!( + !object_dir.join(transaction.to_string()).exists(), + "rejected mutation creates no recovery record" + ); + assert_eq!(wrapper.metrics_snapshot().total_errors_availability, 1); + } + } + #[tokio::test] async fn local_disk_health_wrapper_counts_faulty_precheck_rejections() { let dir = tempfile::tempdir().expect("temp dir should be created"); diff --git a/crates/ecstore/src/disk/error.rs b/crates/ecstore/src/disk/error.rs index 9c31377c0..ade7c9dbd 100644 --- a/crates/ecstore/src/disk/error.rs +++ b/crates/ecstore/src/disk/error.rs @@ -167,6 +167,12 @@ pub enum DiskError { #[error("method not allowed")] MethodNotAllowed, + #[error("conditional delete object generation changed")] + ConditionalDeleteConflict, + + #[error("guarded batch deletion is not supported by this disk")] + GuardedDeleteUnsupported, + #[error("erasure write quorum")] ErasureWriteQuorum, @@ -762,6 +768,8 @@ impl Clone for DiskError { DiskError::PartMissingOrCorrupt => DiskError::PartMissingOrCorrupt, DiskError::NoHealRequired => DiskError::NoHealRequired, DiskError::MethodNotAllowed => DiskError::MethodNotAllowed, + DiskError::ConditionalDeleteConflict => DiskError::ConditionalDeleteConflict, + DiskError::GuardedDeleteUnsupported => DiskError::GuardedDeleteUnsupported, DiskError::ErasureWriteQuorum => DiskError::ErasureWriteQuorum, DiskError::ErasureReadQuorum => DiskError::ErasureReadQuorum, DiskError::ShortWrite => DiskError::ShortWrite, @@ -819,6 +827,8 @@ impl DiskError { DiskError::Timeout => 0x29, DiskError::InvalidPath => 0x2A, DiskError::RemoteClientUnavailable(_) => 0x2B, + DiskError::ConditionalDeleteConflict => 0x2C, + DiskError::GuardedDeleteUnsupported => 0x2D, } } @@ -867,6 +877,8 @@ impl DiskError { 0x29 => Some(DiskError::Timeout), 0x2A => Some(DiskError::InvalidPath), 0x2B => Some(DiskError::RemoteClientUnavailable(String::new())), + 0x2C => Some(DiskError::ConditionalDeleteConflict), + 0x2D => Some(DiskError::GuardedDeleteUnsupported), _ => None, } } @@ -1645,6 +1657,21 @@ mod tests { assert!(rendered.contains("transport error: connection refused")); } + #[test] + fn guarded_delete_errors_preserve_wire_and_io_types() { + for (error, code) in [ + (DiskError::ConditionalDeleteConflict, 0x2C), + (DiskError::GuardedDeleteUnsupported, 0x2D), + ] { + assert_eq!(error.to_u32(), code); + assert_eq!(DiskError::from_u32(code), Some(error.clone())); + let wire: rustfs_protos::proto_gen::node_service::Error = error.clone().into(); + assert_eq!(DiskError::from(wire), error); + let io_error: std::io::Error = error.clone().into(); + assert_eq!(DiskError::from(io_error), error); + } + } + #[test] fn remote_client_unavailable_wire_roundtrip_keeps_variant_and_detail() { let original = DiskError::RemoteClientUnavailable("dial tcp: connection refused".to_string()); diff --git a/crates/ecstore/src/disk/error_reduce.rs b/crates/ecstore/src/disk/error_reduce.rs index 298905459..34d13fade 100644 --- a/crates/ecstore/src/disk/error_reduce.rs +++ b/crates/ecstore/src/disk/error_reduce.rs @@ -131,6 +131,15 @@ pub fn build_write_quorum_failure_summary( fn dominant_error_label(errors: &[Option], ignored_errs: &[Error], dominant_error: Option<&Error>) -> &'static str { let Some(dominant_error) = dominant_error else { + // The reducer also returns None when every failure is ignored for + // quorum voting. That case contains no successful nil entry. + if !errors.is_empty() && errors.iter().all(Option::is_some) { + return if errors.iter().all(|error| error.as_ref() == Some(&Error::DiskNotFound)) { + "disk_not_found" + } else { + "all_ignored" + }; + } return "nil_dominated"; }; @@ -387,9 +396,9 @@ mod tests { } #[test] - fn test_build_write_quorum_failure_summary_all_ignored_is_nil_dominated() { - // All disks offline (all DiskNotFound). The dominant error must be None and - // the label must be "nil_dominated" rather than falling back to "other_error". + fn test_build_write_quorum_failure_summary_all_missing_is_disk_not_found() { + // Ignoring unavailable disks for quorum reduction must not imply a + // successful nil entry in the diagnostic label. let errors = vec![ Some(Error::DiskNotFound), Some(Error::DiskNotFound), @@ -397,7 +406,21 @@ mod tests { ]; let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2); assert_eq!(summary.dominant_error, None); - assert_eq!(summary.dominant_error_label, "nil_dominated"); + assert_eq!(summary.achieved, 0); + assert_eq!(summary.dominant_error_label, "disk_not_found"); + } + + #[test] + fn test_build_write_quorum_failure_summary_all_ignored_has_no_success() { + let errors = [Some(Error::FaultyDisk), Some(Error::DiskNotFound)]; + let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2); + assert_eq!(summary.dominant_error, None); + assert_eq!(summary.achieved, 0); + assert_eq!(summary.dominant_error_label, "all_ignored"); + assert_eq!( + reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 2), + Some(Error::ErasureWriteQuorum) + ); } #[test] diff --git a/crates/ecstore/src/disk/fs.rs b/crates/ecstore/src/disk/fs.rs index 63a37c70f..d7c542bd2 100644 --- a/crates/ecstore/src/disk/fs.rs +++ b/crates/ecstore/src/disk/fs.rs @@ -215,12 +215,6 @@ pub fn rename_std(from: impl AsRef, to: impl AsRef) -> io::Result<() std::fs::rename(from, to) } -#[cfg(any(not(windows), test))] -#[tracing::instrument(level = "debug", skip_all)] -pub async fn read_file(path: impl AsRef) -> io::Result> { - fs::read(path.as_ref()).await -} - // Bucket existence cache - reduces statx syscalls for repeated bucket checks /// Cache for bucket directory existence checks. @@ -584,28 +578,6 @@ mod tests { assert_eq!(content, "test content"); } - #[tokio::test] - async fn test_read_file() { - let temp_dir = TempDir::new().unwrap(); - let file_path = temp_dir.path().join("test_read.txt"); - - let test_content = b"This is test content for reading"; - tokio::fs::write(&file_path, test_content).await.unwrap(); - - // Read file - let read_content = read_file(&file_path).await.unwrap(); - assert_eq!(read_content, test_content); - } - - #[tokio::test] - async fn test_read_file_nonexistent() { - let temp_dir = TempDir::new().unwrap(); - let file_path = temp_dir.path().join("nonexistent.txt"); - - // Should fail for non-existent file - assert!(read_file(&file_path).await.is_err()); - } - #[tokio::test] async fn test_same_file() { let temp_dir = TempDir::new().unwrap(); diff --git a/crates/ecstore/src/disk/lifecycle_test_io.rs b/crates/ecstore/src/disk/lifecycle_test_io.rs new file mode 100644 index 000000000..e331b8869 --- /dev/null +++ b/crates/ecstore/src/disk/lifecycle_test_io.rs @@ -0,0 +1,72 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Bucket-scoped observations distinguish exact metadata reads from walks. +use std::collections::HashMap; +use std::sync::{Arc, Mutex, OnceLock}; + +#[derive(Default, Clone)] +pub(crate) struct Counts { + pub walks: usize, + pub reads: Vec, +} + +fn observers() -> &'static Mutex>>> { + static OBSERVERS: OnceLock>>>> = OnceLock::new(); + OBSERVERS.get_or_init(Mutex::default) +} + +pub(crate) struct Observer { + bucket: String, + counts: Arc>, +} + +impl Observer { + pub fn new(bucket: &str) -> Self { + let counts = Arc::new(Mutex::new(Counts::default())); + assert!( + observers() + .lock() + .expect("I/O observers lock") + .insert(bucket.to_owned(), counts.clone()) + .is_none(), + "each test must use its own bucket" + ); + Self { + bucket: bucket.to_owned(), + counts, + } + } + + pub fn snapshot(&self) -> Counts { + self.counts.lock().expect("I/O counts lock").clone() + } +} + +impl Drop for Observer { + fn drop(&mut self) { + observers().lock().expect("I/O observers lock").remove(&self.bucket); + } +} + +pub(crate) fn observe(bucket: &str, object: Option<&str>) { + let observer = observers().lock().expect("I/O observers lock").get(bucket).cloned(); + if let Some(observer) = observer { + let mut counts = observer.lock().expect("I/O counts lock"); + match object { + Some(object) => counts.reads.push(object.to_owned()), + None => counts.walks += 1, + } + } +} diff --git a/crates/ecstore/src/disk/local.rs b/crates/ecstore/src/disk/local.rs index 4c40eda3e..8dbbc3c8d 100644 --- a/crates/ecstore/src/disk/local.rs +++ b/crates/ecstore/src/disk/local.rs @@ -18,6 +18,7 @@ pub(in crate::disk) use self::commit::{LocalRenamePreflightRejection, PreparedRe mod artifact_lease; mod commit; +pub(super) mod delete_recovery; #[cfg(all(unix, test))] #[path = "fd_cache_tests.rs"] mod fd_cache_tests; @@ -40,6 +41,7 @@ mod uring_result_budget; pub(crate) use artifact_lease::Domain as ArtifactLeaseDomain; pub use artifact_lease::{ArtifactPutLease, ArtifactRepairLease}; pub use replacement_lease::ReplacementExecutionLease; +pub(crate) use replacement_lease::is_blank_target as is_blank_replacement_target; use crate::crash_inject::{self, CrashPoint}; use crate::data_usage::local_snapshot::ensure_data_usage_layout; @@ -54,12 +56,12 @@ use crate::disk::disk_store::{get_drive_walkdir_stall_timeout, get_object_disk_r use crate::disk::io_admission::{DiskIoPermit, spawn_blocking_with_disk_io_permit}; use crate::disk::{ BUCKET_META_PREFIX, CHECK_PART_FILE_CORRUPT, CHECK_PART_FILE_NOT_FOUND, CHECK_PART_SUCCESS, CHECK_PART_UNKNOWN, - CHECK_PART_VOLUME_NOT_FOUND, CheckPartsResp, ConditionalFileUpdate, DataDirDeleteStatus, DeleteOptions, DiskAPI, DiskInfo, - DiskInfoOptions, DiskLocation, DiskMetrics, FileInfoVersions, FileReader, FileWriter, MmapCopyStageMetrics, OldCurrentSize, - PART_TRANSACTION_NEW_META, PART_TRANSACTION_OLD_META, PART_TRANSACTION_ROLLBACK, PartTransactionAction, RUSTFS_META_BUCKET, - RUSTFS_META_TMP_BUCKET, RUSTFS_META_TMP_DELETED_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, - STORAGE_FORMAT_FILE, STORAGE_FORMAT_FILE_BACKUP, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, - acquire_context_disk_io_permit, conv_part_err_to_int, + CHECK_PART_VOLUME_NOT_FOUND, CheckPartsResp, ConditionalFileUpdate, CurrentDeleteGeneration, DataDirDeleteStatus, + DeleteOptions, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskMetrics, FileInfoVersions, FileReader, FileWriter, + GuardedFileInfoVersions, MmapCopyStageMetrics, OldCurrentSize, PART_TRANSACTION_NEW_META, PART_TRANSACTION_OLD_META, + PART_TRANSACTION_ROLLBACK, PartTransactionAction, RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET, RUSTFS_META_TMP_DELETED_BUCKET, + ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE, STORAGE_FORMAT_FILE_BACKUP, + SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, acquire_context_disk_io_permit, conv_part_err_to_int, endpoint::Endpoint, error::{DiskError, Error, FileAccessDeniedWithContext, Result}, error_conv::{to_access_error, to_file_error, to_unformatted_disk_error, to_volume_error}, @@ -133,8 +135,10 @@ tokio::task_local! { const RUSTFS_META_TMP_OLD_BUCKET: &str = ".rustfs.sys/tmp-old"; const INLINE_METADATA_ROLLBACK_DIR_XOR: u128 = 0x7275737466735f696e6c696e655f7262; const DELETE_MARKER_ROLLBACK_FILE: &str = "xl.meta.delete-marker.rollback"; +pub(crate) const DELETE_QUORUM_COMMITTED_FILE: &str = "delete-quorum-committed"; pub(crate) const DELETE_DATA_DIR_MARKER_PREFIX: &str = "delete-data."; pub(crate) const RESERVED_DELETE_DATA_DIR_MARKER_PREFIX: &str = "reserve-delete-data."; +pub(crate) const RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX: &str = "recovery-delete-data."; /// Largest directory read the delete-residue probe issues before it must /// fall back to a complete read. Residue holds one or two data dirs, so an /// under-filled batch settles the common case without materializing large @@ -164,6 +168,34 @@ pub(crate) fn metadata_less_part_file(entry: &str) -> bool { .is_some_and(|part_number| part_number.parse::().is_ok_and(|part_number| part_number > 0)) } +/// Native multipart sidecars travel with their part during delete recovery. +/// Callers must additionally require a regular file inside a bound data dir. +fn delete_recovery_part_file(entry: &str) -> bool { + metadata_less_part_file(entry) + || entry.strip_suffix(".meta").is_some_and(metadata_less_part_file) + || entry.split_once(".integrity.").is_some_and(|(part, generation)| { + metadata_less_part_file(part) && Uuid::parse_str(generation).is_ok_and(|id| !id.is_nil()) + }) +} + +/// Classify the entire leaf before unlinking anything. UUID-shaped S3 prefixes +/// and transaction witnesses share this namespace with erasure data dirs. +fn orphan_data_dir_parts(path: &Path) -> std::io::Result> { + let protected = || std::io::Error::new(ErrorKind::InvalidData, "orphan data directory contains protected entries"); + if !std::fs::symlink_metadata(path)?.file_type().is_dir() { + return Err(protected()); + } + let mut parts = Vec::new(); + for entry in std::fs::read_dir(path)? { + let entry = entry?; + if !entry.file_type()?.is_file() || !entry.file_name().to_str().is_some_and(delete_recovery_part_file) { + return Err(protected()); + } + parts.push(entry.path()); + } + Ok(parts) +} + fn delete_transaction_marker_id(entry: &str, prefix: &str) -> Option { entry .strip_prefix(prefix) @@ -180,10 +212,15 @@ fn is_delete_transaction_marker(entry: &str, prefix: &str) -> bool { /// an `xl.meta`, an unknown file) means the directory is not plain delete residue. fn is_metadata_less_data_dir_entry(entry: &str) -> bool { !entry.ends_with(SLASH_SEPARATOR) - && (metadata_less_part_file(entry) + && (delete_recovery_part_file(entry) || entry == STORAGE_FORMAT_FILE_BACKUP + || entry == DELETE_QUORUM_COMMITTED_FILE + || entry == delete_recovery::PREPARED + || entry == delete_recovery::TARGET + || entry == delete_recovery::FINISHED || is_delete_transaction_marker(entry, DELETE_DATA_DIR_MARKER_PREFIX) - || is_delete_transaction_marker(entry, RESERVED_DELETE_DATA_DIR_MARKER_PREFIX)) + || is_delete_transaction_marker(entry, RESERVED_DELETE_DATA_DIR_MARKER_PREFIX) + || is_delete_transaction_marker(entry, RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX)) } const STARTUP_CLEANUP_WAIT_TIMEOUT: Duration = Duration::from_secs(2); const ENV_BITROT_SIZE_MISMATCH_RETRY_COUNT: &str = "RUSTFS_BITROT_SIZE_MISMATCH_RETRY_COUNT"; @@ -206,8 +243,10 @@ struct ListingMetadataRead { has_namespace_child_candidate: bool, } -fn read_all_data_std(path: &Path) -> core::result::Result<(Vec, Option), ReadAllError> { - let mut file = std::fs::File::open(path).map_err(ReadAllError::Open)?; +fn read_all_data_std( + opened: std::io::Result, +) -> core::result::Result<(Vec, Option), ReadAllError> { + let mut file = opened.map_err(ReadAllError::Open)?; let metadata = file.metadata().map_err(|err| ReadAllError::Disk(to_file_error(err).into()))?; if metadata.is_dir() { @@ -227,6 +266,10 @@ fn read_all_data_std(path: &Path) -> core::result::Result<(Vec, Option Uuid { let used_data_dirs: HashSet = meta.get_data_dirs().unwrap_or_default().into_iter().flatten().collect(); + inline_metadata_rollback_dir_with_used_dirs(version_id, &used_data_dirs) +} + +fn inline_metadata_rollback_dir_with_used_dirs(version_id: Uuid, used_data_dirs: &HashSet) -> Uuid { let base = version_id.as_u128() ^ INLINE_METADATA_ROLLBACK_DIR_XOR; let mut salt = 0u128; @@ -324,21 +367,44 @@ fn restore_part_transaction_file(current: &Path, backup: &Path, absent: &Path, r #[cfg(test)] async fn write_metadata_rollback_backup(object_dir: &Path, rollback_dir: Uuid, data: &[u8]) -> Result<()> { - write_delete_rollback_file(object_dir, rollback_dir, STORAGE_FORMAT_FILE_BACKUP, data, None).await + write_delete_rollback_file( + object_dir.parent().unwrap_or(object_dir), + "test-backup", + object_dir, + rollback_dir, + STORAGE_FORMAT_FILE_BACKUP, + data, + None, + ) + .await } #[cfg(test)] async fn write_delete_rollback_file( + io_root: &Path, + volume: &str, object_dir: &Path, rollback_dir: Uuid, - name: &str, + name: &'static str, data: &[u8], namespace_owner: Option>, ) -> Result<()> { - write_delete_rollback_file_with_prepared_publication(object_dir, rollback_dir, name, data, namespace_owner, None).await + write_delete_rollback_file_with_prepared_publication( + io_root, + volume, + object_dir, + rollback_dir, + name, + data, + namespace_owner, + None, + ) + .await } async fn write_delete_rollback_file_with_prepared_publication( + io_root: &Path, + volume: &str, object_dir: &Path, rollback_dir: Uuid, name: &str, @@ -354,16 +420,50 @@ async fn write_delete_rollback_file_with_prepared_publication( os::acquire_namespace_mutation_lease_with_owner(&path, namespace_owner).await }; let data = data.to_vec(); - let object_dir = object_dir.to_path_buf(); + let sync = effective_durability(volume).syncs_commit_metadata(); + let staging_root = io_root.join(RUSTFS_META_TMP_BUCKET); + let io_root = io_root.to_path_buf(); + let name = name.to_owned(); os::run_blocking_namespace_operation(lease, move || { + reject_local_disk_symlink_components(&io_root, &path).map_err(std::io::Error::from)?; + reject_local_disk_symlink_components(&io_root, &staging_root).map_err(std::io::Error::from)?; std::fs::create_dir_all(&backup_dir)?; + std::fs::create_dir_all(&staging_root)?; #[cfg(test)] run_owned_file_write_before_open(&path); - // Rollback must never consume a partially written backup. Stage outside - // the rollback directory so interrupted writes cannot become staged data. - let mut staged = tempfile::Builder::new().prefix(".delete-rollback-").tempfile_in(object_dir)?; + // tempfile's Windows `persist` hands both the staged and the final path + // straight to `MoveFileExW`, bypassing std's long-path handling, so a + // deep object key failed every delete at MAX_PATH (rustfs#8545). + let staging_root = rustfs_utils::extended_length_path(&staging_root)?; + let destination = rustfs_utils::extended_length_path(&path)?; + // Private directory staging is reclaimed by existing tmp cleanup after + // a crash and cannot leave a regular artifact in the user's bucket. + let staging_dir = tempfile::Builder::new() + .prefix(".delete-rollback-") + .tempdir_in(&staging_root)?; + let staged_path = staging_dir.path().join(name); + let mut staged_options = std::fs::OpenOptions::new(); + staged_options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + staged_options.mode(0o600); + } + let mut staged = staged_options.open(&staged_path)?; std::io::Write::write_all(&mut staged, &data)?; - staged.persist(path).map_err(|err| err.error)?; + if sync { + staged.sync_all()?; + } + // Close the writer before publishing via std::fs::rename, which applies + // verbatim long-path normalization and POSIX-semantics replacement on Windows. + drop(staged); + std::fs::rename(&staged_path, &destination)?; + if sync { + os::fsync_dir_std(&backup_dir)?; + if let Some(parent) = backup_dir.parent() { + os::fsync_dir_std(parent)?; + } + } Ok(()) }) .await @@ -425,14 +525,23 @@ async fn restore_metadata_backup_with_prepared_publication( } async fn restore_delete_rollback_with_namespace_owner( + volume: &str, object_dir: &Path, xl_path: &Path, rollback_dir: Uuid, publication_root: &os::PublicationRoot, namespace_owner: Option>, ) -> Result<()> { - restore_delete_rollback_with_prepared_publication(object_dir, xl_path, rollback_dir, publication_root, namespace_owner, None) - .await + restore_delete_rollback_with_prepared_publication( + volume, + object_dir, + xl_path, + rollback_dir, + publication_root, + namespace_owner, + None, + ) + .await } // The frozen paths are shared by forward staging and its error recovery. The @@ -443,14 +552,36 @@ struct DeleteRollbackPlan { staged_paths: Vec<(PathBuf, PathBuf)>, rollback_exists: bool, remove_new_metadata: bool, + recovery_journal: Option, publication: Option>, rollback_dir_owned: bool, rollback_marker_owned: bool, } impl DeleteRollbackPlan { - async fn read(object_dir: &Path, rollback_dir: Uuid, marker_paths: Vec) -> Result { + async fn read(volume: &str, object_dir: &Path, rollback_dir: Uuid, marker_paths: Vec) -> Result { let rollback_path = object_dir.join(rollback_dir.to_string()); + match fs::symlink_metadata(rollback_path.join(DELETE_QUORUM_COMMITTED_FILE)).await { + Ok(_) => return Err(DiskError::FileCorrupt), + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + let recovery_journal = match fs::symlink_metadata(rollback_path.join(delete_recovery::PREPARED)).await { + Ok(metadata) if metadata.is_file() => { + let journal: super::DeleteRecoveryJournal = serde_json::from_slice( + &fs::read(rollback_path.join(delete_recovery::PREPARED)) + .await + .map_err(to_file_error)?, + )?; + if journal.volume != volume || journal.transaction != rollback_dir { + return Err(DiskError::FileCorrupt); + } + Some(journal) + } + Ok(_) => return Err(DiskError::InvalidPath), + Err(err) if err.kind() == ErrorKind::NotFound => None, + Err(err) => return Err(to_file_error(err).into()), + }; let mut staged_paths = Vec::new(); let mut remove_new_metadata = false; let rollback_exists = match fs::read_dir(&rollback_path).await { @@ -459,7 +590,21 @@ impl DeleteRollbackPlan { let name = entry.file_name(); if name == DELETE_MARKER_ROLLBACK_FILE { remove_new_metadata = true; - } else if name != STORAGE_FORMAT_FILE_BACKUP { + } else if name != STORAGE_FORMAT_FILE_BACKUP + && name != DELETE_QUORUM_COMMITTED_FILE + && name != delete_recovery::PREPARED + && name != delete_recovery::TARGET + && name != delete_recovery::FINISHED + { + if let Some(journal) = &recovery_journal { + let id = name + .to_str() + .and_then(|name| Uuid::parse_str(name).ok()) + .ok_or(DiskError::FileCorrupt)?; + if !journal.data_dirs.contains(&id) || !entry.file_type().await.map_err(to_file_error)?.is_dir() { + return Err(DiskError::FileCorrupt); + } + } staged_paths.push((entry.path(), object_dir.join(name))); } } @@ -473,6 +618,7 @@ impl DeleteRollbackPlan { staged_paths, rollback_exists, remove_new_metadata, + recovery_journal, publication: None, rollback_dir_owned: false, rollback_marker_owned: false, @@ -488,6 +634,11 @@ impl DeleteRollbackPlan { ) -> Result<()> { let rollback_path = object_dir.join(rollback_dir.to_string()); let mut paths = self.marker_paths.clone(); + if self.rollback_exists || self.rollback_dir_owned { + for name in [delete_recovery::TARGET, delete_recovery::PREPARED, delete_recovery::FINISHED] { + paths.push(rollback_path.join(name)); + } + } if self.rollback_exists || self.rollback_dir_owned { paths.push(rollback_path.join(STORAGE_FORMAT_FILE_BACKUP)); for (src, dst) in &self.staged_paths { @@ -520,6 +671,7 @@ impl DeleteRollbackPlan { } async fn prepare_forward( + volume: &str, object_dir: &Path, xl_path: &Path, rollback_dir: Uuid, @@ -528,7 +680,7 @@ impl DeleteRollbackPlan { publication: Arc, ) -> Result { let marker_paths = version_delete_marker_paths(object_dir, rollback_dir).await?; - let mut plan = Self::read(object_dir, rollback_dir, marker_paths).await?; + let mut plan = Self::read(volume, object_dir, rollback_dir, marker_paths).await?; let rollback_path = object_dir.join(rollback_dir.to_string()); plan.rollback_dir_owned = true; plan.rollback_marker_owned = new_marker || plan.remove_new_metadata; @@ -539,6 +691,8 @@ impl DeleteRollbackPlan { plan.marker_paths.push(reserved); plan.marker_paths .push(data_path.join(format!("{DELETE_DATA_DIR_MARKER_PREFIX}{rollback_dir}"))); + plan.marker_paths + .push(data_path.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{rollback_dir}"))); } } for data_dir in data_dirs { @@ -552,7 +706,10 @@ impl DeleteRollbackPlan { Ok(plan) } - async fn refresh_forward(&mut self, object_dir: &Path, rollback_dir: Uuid) -> Result<()> { + async fn refresh_forward(&mut self, volume: &str, object_dir: &Path, rollback_dir: Uuid) -> Result<()> { + self.recovery_journal = Self::read(volume, object_dir, rollback_dir, self.marker_paths.clone()) + .await? + .recovery_journal; let rollback_path = object_dir.join(rollback_dir.to_string()); self.rollback_exists = match fs::read_dir(&rollback_path).await { Ok(_) => true, @@ -582,6 +739,7 @@ impl DeleteRollbackPlan { } async fn restore_delete_rollback_with_prepared_publication( + volume: &str, object_dir: &Path, xl_path: &Path, rollback_dir: Uuid, @@ -590,17 +748,15 @@ async fn restore_delete_rollback_with_prepared_publication( prepared: Option>, ) -> Result<()> { let marker_paths = version_delete_marker_paths(object_dir, rollback_dir).await?; - if prepared.is_none() { - remove_version_delete_marker_paths(&marker_paths, namespace_owner.clone(), None).await?; - } - let mut plan = DeleteRollbackPlan::read(object_dir, rollback_dir, marker_paths).await?; + let mut plan = DeleteRollbackPlan::read(volume, object_dir, rollback_dir, marker_paths).await?; if let Some(publication) = prepared { plan.acquire(object_dir, xl_path, rollback_dir, publication)?; } - restore_delete_rollback_plan(object_dir, xl_path, rollback_dir, publication_root, namespace_owner, plan).await + restore_delete_rollback_plan(volume, object_dir, xl_path, rollback_dir, publication_root, namespace_owner, plan).await } async fn restore_delete_rollback_plan( + volume: &str, object_dir: &Path, xl_path: &Path, rollback_dir: Uuid, @@ -613,48 +769,137 @@ async fn restore_delete_rollback_plan( staged_paths, rollback_exists, remove_new_metadata, + recovery_journal, publication: prepared, rollback_dir_owned, rollback_marker_owned, } = plan; let rollback_path = object_dir.join(rollback_dir.to_string()); - let backup_path = rollback_path.join(STORAGE_FORMAT_FILE_BACKUP); - if prepared.is_some() { - remove_version_delete_marker_paths(&marker_paths, namespace_owner.clone(), prepared.clone()).await?; - } if !rollback_exists { - return Ok(()); + return remove_version_delete_marker_paths(volume, &marker_paths, namespace_owner, prepared).await; } let had_staged_paths = !staged_paths.is_empty(); for (src, dst) in staged_paths { - if let Some(publication) = &prepared { - os::rename_all_with_lease(&src, &dst, object_dir, publication_root, publication.clone()).await?; + if recovery_journal.is_some() { + let mut entries = fs::read_dir(&src).await.map_err(to_file_error)?; + let mut files = Vec::new(); + while let Some(entry) = entries.next_entry().await.map_err(to_file_error)? { + let name = entry.file_name(); + let name = name.to_str().ok_or(DiskError::FileCorrupt)?; + if !entry.file_type().await.map_err(to_file_error)?.is_file() + || (!delete_recovery_part_file(name) + && ![ + DELETE_DATA_DIR_MARKER_PREFIX, + RESERVED_DELETE_DATA_DIR_MARKER_PREFIX, + RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX, + ] + .iter() + .any(|prefix| name == format!("{prefix}{rollback_dir}"))) + { + return Err(DiskError::FileCorrupt); + } + files.push((entry.path(), dst.join(name))); + } + if let Some(publication) = &prepared { + os::create_dir_all_with_lease(dst.clone(), publication.clone()) + .await + .map_err(to_file_error)?; + } else { + os::create_dir_all_with_namespace_owner(&dst, namespace_owner.clone()) + .await + .map_err(to_file_error)?; + } + for (source, destination) in files { + match fs::symlink_metadata(&destination).await { + Ok(metadata) if metadata.is_file() => { + let lease = match &prepared { + Some(publication) => { + os::try_acquire_namespace_mutation_lease_with_owner(&destination, publication.clone())? + } + None => os::acquire_namespace_mutation_lease_with_owner(&destination, namespace_owner.clone()).await, + }; + let identical = os::run_blocking_namespace_operation(lease, move || { + let source_meta = std::fs::symlink_metadata(&source)?; + let destination_meta = std::fs::symlink_metadata(&destination)?; + if !source_meta.is_file() + || !destination_meta.is_file() + || source_meta.len() != destination_meta.len() + { + return Ok(false); + } + let mut source_file = std::fs::File::open(&source)?; + let mut destination_file = std::fs::File::open(&destination)?; + let mut left = [0u8; 65536]; + let mut right = [0u8; 65536]; + let mut remaining = source_meta.len(); + while remaining > 0 { + let length = usize::try_from(remaining.min(left.len() as u64)).map_err(std::io::Error::other)?; + std::io::Read::read_exact(&mut source_file, &mut left[..length])?; + std::io::Read::read_exact(&mut destination_file, &mut right[..length])?; + if left[..length] != right[..length] { + return Ok(false); + } + remaining -= length as u64; + } + std::fs::remove_file(source)?; + Ok(true) + }) + .await + .map_err(to_file_error)?; + if !identical { + return Err(DiskError::FileCorrupt); + } + } + Ok(_) => return Err(DiskError::InvalidPath), + Err(err) if err.kind() == ErrorKind::NotFound => { + if let Some(publication) = &prepared { + os::rename_all_with_lease(&source, &destination, object_dir, publication_root, publication.clone()) + .await?; + } else { + os::rename_all_with_owner( + &source, + &destination, + object_dir, + publication_root, + namespace_owner.clone(), + ) + .await?; + } + } + Err(err) => return Err(to_file_error(err).into()), + } + } + if effective_durability(volume).syncs_commit_metadata() { + os::fsync_dir_with_owner(&dst, namespace_owner.clone()) + .await + .map_err(to_file_error)?; + os::fsync_dir_with_owner(&src, namespace_owner.clone()) + .await + .map_err(to_file_error)?; + } + if let Some(publication) = &prepared { + let _ = os::remove_dir_with_lease(&src, publication.clone()).await; + } else { + let _ = os::remove_dir_with_owner(&src, namespace_owner.clone()).await; + } } else { - os::rename_all_with_owner(&src, &dst, object_dir, publication_root, namespace_owner.clone()).await?; + if let Some(publication) = &prepared { + os::rename_all_with_lease(&src, &dst, object_dir, publication_root, publication.clone()).await?; + } else { + os::rename_all_with_owner(&src, &dst, object_dir, publication_root, namespace_owner.clone()).await?; + } } } + let backup_path = rollback_path.join(STORAGE_FORMAT_FILE_BACKUP); let restore = if let Some(publication) = &prepared { os::rename_all_with_lease(&backup_path, xl_path, object_dir, publication_root, publication.clone()).await } else { os::rename_all_with_owner(&backup_path, xl_path, object_dir, publication_root, namespace_owner.clone()).await }; match restore { - Ok(()) => { - cleanup_delete_rollback( - &rollback_path, - false, - namespace_owner, - prepared, - rollback_dir_owned, - rollback_marker_owned, - ) - .await; - Ok(()) - } - // A missing backup only means "remove the newly-created delete marker" - // when the marker proves there was no old metadata to restore. + Ok(()) => {} Err(DiskError::FileNotFound) if remove_new_metadata => { let remove = if let Some(publication) = &prepared { os::remove_file_with_lease(xl_path.to_path_buf(), publication.clone()).await @@ -666,29 +911,48 @@ async fn restore_delete_rollback_plan( Err(err) if err.kind() == ErrorKind::NotFound => {} Err(err) => return Err(to_file_error(err).into()), } - cleanup_delete_rollback(&rollback_path, true, namespace_owner, prepared, rollback_dir_owned, rollback_marker_owned) - .await; - Ok(()) } - Err(DiskError::FileNotFound) if had_staged_paths => Err(DiskError::FileNotFound), + Err(DiskError::FileNotFound) if had_staged_paths => return Err(DiskError::FileNotFound), Err(DiskError::FileNotFound) => match fs::metadata(xl_path).await { - Ok(_) => { - cleanup_delete_rollback( - &rollback_path, - false, - namespace_owner, - prepared, - rollback_dir_owned, - rollback_marker_owned, - ) - .await; - Ok(()) - } - Err(err) if err.kind() == ErrorKind::NotFound => Err(DiskError::FileNotFound), - Err(err) => Err(to_file_error(err).into()), + Ok(_) => {} + Err(err) if err.kind() == ErrorKind::NotFound => return Err(DiskError::FileNotFound), + Err(err) => return Err(to_file_error(err).into()), }, - Err(err) => Err(err), + Err(err) => return Err(err), } + delete_recovery::restore_barrier(volume, object_dir, rollback_dir).await?; + // Persist both sides of the restore rename before discarding its evidence. + if effective_durability(volume).syncs_commit_metadata() { + os::fsync_dir_with_owner(object_dir, namespace_owner.clone()) + .await + .map_err(to_file_error)?; + os::fsync_dir_with_owner(&rollback_path, namespace_owner.clone()) + .await + .map_err(to_file_error)?; + } + remove_version_delete_marker_paths(volume, &marker_paths, namespace_owner.clone(), prepared.clone()).await?; + for name in [delete_recovery::TARGET, delete_recovery::PREPARED, delete_recovery::FINISHED] { + let remove = if let Some(publication) = &prepared { + os::remove_file_with_lease(rollback_path.join(name), publication.clone()).await + } else { + os::remove_file_with_owner(rollback_path.join(name), namespace_owner.clone()).await + }; + match remove { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + } + cleanup_delete_rollback( + &rollback_path, + remove_new_metadata, + namespace_owner, + prepared, + rollback_dir_owned, + rollback_marker_owned, + ) + .await; + Ok(()) } async fn cleanup_delete_rollback( @@ -733,6 +997,7 @@ async fn cleanup_delete_rollback( async fn version_delete_marker_paths(object_dir: &Path, rollback_dir: Uuid) -> Result> { let reserved_name = format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{rollback_dir}"); let committed_name = format!("{DELETE_DATA_DIR_MARKER_PREFIX}{rollback_dir}"); + let recovery_name = format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{rollback_dir}"); let mut entries = match fs::read_dir(object_dir).await { Ok(entries) => entries, Err(err) if err.kind() == ErrorKind::NotFound => return Ok(Vec::new()), @@ -745,7 +1010,7 @@ async fn version_delete_marker_paths(object_dir: &Path, rollback_dir: Uuid) -> R { continue; } - for marker_name in [&reserved_name, &committed_name] { + for marker_name in [&reserved_name, &committed_name, &recovery_name] { paths.push(entry.path().join(marker_name)); } } @@ -753,6 +1018,7 @@ async fn version_delete_marker_paths(object_dir: &Path, rollback_dir: Uuid) -> R } async fn remove_version_delete_marker_paths( + volume: &str, paths: &[PathBuf], namespace_owner: Option>, prepared: Option>, @@ -769,6 +1035,18 @@ async fn remove_version_delete_marker_paths( Err(err) => return Err(to_file_error(err).into()), } } + if effective_durability(volume).syncs_commit_metadata() { + // Finish every unlink in a directory before its durability barrier. + // Retry still syncs directories whose markers are already absent. + let parents: BTreeSet<_> = paths.iter().filter_map(|path| path.parent()).collect(); + for parent in parents { + match os::fsync_dir_with_owner(parent, namespace_owner.clone()).await { + Ok(()) => {} + Err(error) if error.kind() == ErrorKind::NotFound => {} + Err(error) => return Err(to_file_error(error).into()), + } + } + } Ok(()) } @@ -806,14 +1084,16 @@ async fn restore_delete_rollback_after_error( }; let restore = if let Some(mut plan) = prepared { - match plan.refresh_forward(object_dir, rollback_dir).await { + match plan.refresh_forward(volume, object_dir, rollback_dir).await { Ok(()) => { - restore_delete_rollback_plan(object_dir, xl_path, rollback_dir, publication_root, namespace_owner, plan).await + restore_delete_rollback_plan(volume, object_dir, xl_path, rollback_dir, publication_root, namespace_owner, plan) + .await } Err(error) => Err(error), } } else { - restore_delete_rollback_with_namespace_owner(object_dir, xl_path, rollback_dir, publication_root, namespace_owner).await + restore_delete_rollback_with_namespace_owner(volume, object_dir, xl_path, rollback_dir, publication_root, namespace_owner) + .await }; if let Err(restore_err) = restore { warn!( @@ -6223,24 +6503,6 @@ fn mount_id_from_mountinfo_contents(mountinfo: &str, path: &Path) -> Option } impl LocalDisk { - #[cfg(target_os = "linux")] - fn open_mount_lease(root: &Path) -> Result<(std::fs::File, PathBuf, Option)> { - use rustix::fs::{Mode, OFlags, open}; - use std::os::fd::AsRawFd as _; - - let fd = open( - root, - OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, - Mode::empty(), - ) - .map_err(std::io::Error::from) - .map_err(DiskError::from)?; - let lease = std::fs::File::from(fd); - let io_root = PathBuf::from(format!("/proc/self/fd/{}/.", lease.as_raw_fd())); - let mount_id = mount_id_for_fd(&lease); - Ok((lease, io_root, mount_id)) - } - #[cfg(not(target_os = "linux"))] fn open_mount_lease(root: &Path) -> Result { Ok(root.to_path_buf()) @@ -6433,6 +6695,7 @@ impl LocalDisk { source.map_or(destination.0, |s| s.0), destination.0, &dst, + None, Some(owner), ) .await; @@ -6995,7 +7258,8 @@ impl LocalDisk { // Destination namespace -> sorted, deduplicated READs -> try source. // The source lease owns the first lease, with no reverse edge. let lease = - os::acquire_rename_data_mutation_lease_for_volumes(&self.root, source.0, destination.0, &dst, Some(owner)).await; + os::acquire_rename_data_mutation_lease_for_volumes(&self.root, source.0, destination.0, &dst, None, Some(owner)) + .await; let lease = if src == dst { lease } else { @@ -7310,7 +7574,7 @@ impl LocalDisk { // reuses these exact locks, including when a DeleteVolume writer queues. let metadata = os::acquire_metadata_mutation_lease(&object_path, Some(owner)).await; let publication = - os::acquire_rename_data_mutation_lease_with_owner(&self.root, volume, &metadata_path, Some(metadata)).await; + os::acquire_rename_data_mutation_lease_with_owner(&self.root, volume, &metadata_path, None, Some(metadata)).await; if !skip_access_checks(volume) { let metadata = fs::symlink_metadata(volume_path).await.map_err(to_volume_error)?; if !metadata.is_dir() { @@ -7500,7 +7764,11 @@ impl LocalDisk { let root = publication_root.path().to_path_buf(); #[cfg(target_os = "linux")] - let (mount_lease, io_root, mount_lease_mount_id) = Self::open_mount_lease(&root)?; + let (mount_lease, io_root, mount_lease_mount_id) = ( + publication_root.directory().clone(), + publication_root.io_path().to_path_buf(), + mount_id_for_fd(publication_root.directory()), + ); #[cfg(not(target_os = "linux"))] let io_root = Self::open_mount_lease(&root)?; @@ -7551,9 +7819,11 @@ impl LocalDisk { state = "format_path_resolved", "Local disk format path resolved" ); - let (format_data, format_meta) = read_file_exists(&io_format_path).await.inspect_err(|err| { - log_startup_disk_error("read_format_json", &io_format_path, err); - })?; + let (format_data, format_meta) = read_file_exists(&io_format_path, &publication_root) + .await + .inspect_err(|err| { + log_startup_disk_error("read_format_json", &io_format_path, err); + })?; let mut id = None; // let mut format_legacy = false; @@ -7586,8 +7856,9 @@ impl LocalDisk { Box::pin(async move { match get_disk_info(root.clone()).await { Ok((info, is_root_disk)) => { - let physical_device_ids = match rustfs_utils::os::get_physical_device_ids(root.to_string_lossy().as_ref()) - { + let physical_device_ids = match win32_volume_path(&root).and_then(|volume_path| { + rustfs_utils::os::get_physical_device_ids(volume_path.to_string_lossy().as_ref()) + }) { Ok(ids) => ids, Err(err) => { warn!( @@ -7701,7 +7972,13 @@ impl LocalDisk { let io_root = disk.io_root.clone(); let publication_root = disk.publication_root.clone(); - tokio::spawn(Self::cleanup_deleted_objects_loop(io_root, publication_root, exit_rx)); + tokio::spawn(Self::cleanup_deleted_objects_loop( + io_root, + publication_root, + #[cfg(target_os = "linux")] + disk.mount_lease.clone(), + exit_rx, + )); debug!( event = EVENT_DISK_LOCAL_STARTUP_CLEANUP, component = LOG_COMPONENT_ECSTORE, @@ -7717,30 +7994,53 @@ impl LocalDisk { async fn cleanup_deleted_objects_loop( root: PathBuf, publication_root: os::PublicationRoot, + #[cfg(target_os = "linux")] mount_lease: Arc, mut exit_rx: tokio::sync::broadcast::Receiver<()>, ) { let start_at = Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL; let mut interval = interval_at(start_at, DELETED_OBJECTS_CLEANUP_INTERVAL); + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); loop { tokio::select! { _ = interval.tick() => { - if let Err(err) = Self::cleanup_deleted_objects(root.clone()).await { + let root = root.clone(); + let publication_root = publication_root.clone(); + #[cfg(target_os = "linux")] + let mount_lease = mount_lease.clone(); + if let Err(err) = super::cleanup_runtime::run_gc(move |budget| async move { + // Queued/detached scans must keep /proc/self/fd paths + // attached to the original mount after LocalDisk drops. + #[cfg(target_os = "linux")] + let _mount_lease = mount_lease; + if let Err(err) = Self::cleanup_deleted_objects(root.clone(), &budget).await { + error!( + event = EVENT_DISK_LOCAL_BACKGROUND_CLEANUP, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DISK_LOCAL, + task = "deleted_objects", + state = "failed", + error = ?err, + "Disk local background cleanup failed" + ); + } + if let Err(err) = Self::cleanup_stale_tmp_objects(root.clone(), &publication_root, &budget).await { + error!( + event = EVENT_DISK_LOCAL_BACKGROUND_CLEANUP, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DISK_LOCAL, + task = "stale_tmp_objects", + state = "failed", + error = ?err, + "Disk local background cleanup failed" + ); + } + Ok(()) + }).await { error!( event = EVENT_DISK_LOCAL_BACKGROUND_CLEANUP, component = LOG_COMPONENT_ECSTORE, subsystem = LOG_SUBSYSTEM_DISK_LOCAL, - task = "deleted_objects", - state = "failed", - error = ?err, - "Disk local background cleanup failed" - ); - } - if let Err(err) = Self::cleanup_stale_tmp_objects(root.clone(), &publication_root).await { - error!( - event = EVENT_DISK_LOCAL_BACKGROUND_CLEANUP, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_DISK_LOCAL, - task = "stale_tmp_objects", + task = "cleanup_dispatch", state = "failed", error = ?err, "Disk local background cleanup failed" @@ -7831,95 +8131,148 @@ impl LocalDisk { } } - async fn cleanup_stale_tmp_objects(root: PathBuf, publication_root: &os::PublicationRoot) -> Result<()> { - Self::cleanup_stale_tmp_objects_with_expiry(root, publication_root, STALE_TMP_OBJECT_EXPIRY).await + async fn cleanup_stale_tmp_objects( + root: PathBuf, + publication_root: &os::PublicationRoot, + budget: &super::cleanup_runtime::GcBudget, + ) -> Result<()> { + Self::cleanup_stale_tmp_objects_with_expiry(root, publication_root, STALE_TMP_OBJECT_EXPIRY, budget).await } async fn cleanup_stale_tmp_objects_with_expiry( root: PathBuf, publication_root: &os::PublicationRoot, expiry: Duration, + budget: &super::cleanup_runtime::GcBudget, ) -> Result<()> { let tmp_path = Self::meta_path(&root, RUSTFS_META_TMP_BUCKET); - let mut entries = match fs::read_dir(&tmp_path).await { + let mut entries = match budget.step(async { Ok(fs::read_dir(&tmp_path).await?) }).await { Ok(entries) => entries, Err(e) => { - if e.kind() == ErrorKind::NotFound { + if matches!(&e, DiskError::Io(err) if err.kind() == ErrorKind::NotFound) { return Ok(()); } - return Err(e.into()); + return Err(e); } }; - while let Some(entry) = entries.next_entry().await? { - let name = entry.file_name().to_string_lossy().to_string(); - if name.is_empty() || name == "." || name == ".." || name == ".trash" { - continue; + loop { + let more = budget + .step(async { + let Some(entry) = entries.next_entry().await? else { + return Ok(false); + }; + Self::cleanup_stale_tmp_entry(entry, &root, publication_root, expiry).await?; + Ok(true) + }) + .await?; + if !more { + break; } - - let file_type = entry.file_type().await?; - if !file_type.is_dir() { - continue; - } - - let Some(age) = entry - .metadata() - .await? - .modified() - .ok() - .and_then(|modified| modified.elapsed().ok()) - else { - continue; - }; - if age <= expiry { - continue; - } - - let target_path = Self::meta_path(&root, RUSTFS_META_TMP_DELETED_BUCKET).join(Uuid::new_v4().to_string()); - rename_all(entry.path(), target_path, Self::meta_path(&root, RUSTFS_META_BUCKET), publication_root).await?; } Ok(()) } - async fn cleanup_deleted_objects(root: PathBuf) -> Result<()> { + async fn cleanup_stale_tmp_entry( + entry: fs::DirEntry, + root: &Path, + publication_root: &os::PublicationRoot, + expiry: Duration, + ) -> Result<()> { + let name = entry.file_name().to_string_lossy().to_string(); + if name.is_empty() || name == "." || name == ".." || name == ".trash" || !entry.file_type().await?.is_dir() { + return Ok(()); + } + let Some(age) = entry + .metadata() + .await? + .modified() + .ok() + .and_then(|modified| modified.elapsed().ok()) + else { + return Ok(()); + }; + if age <= expiry { + return Ok(()); + } + let staging = Self::meta_path(publication_root.path(), RUSTFS_META_TMP_BUCKET).join(entry.file_name()); + let owner = os::acquire_namespace_mutation_lease_with_owner(&staging, None).await; + // An active publisher may have consumed or refreshed this entry while + // GC waited. Recheck the actual path before moving the whole prefix. + let metadata = match fs::symlink_metadata(entry.path()).await { + Ok(metadata) => metadata, + Err(error) if error.kind() == ErrorKind::NotFound => return Ok(()), + Err(error) => return Err(error.into()), + }; + if !metadata.is_dir() + || metadata + .modified() + .ok() + .and_then(|modified| modified.elapsed().ok()) + .is_none_or(|age| age <= expiry) + { + return Ok(()); + } + let target_path = Self::meta_path(root, RUSTFS_META_TMP_DELETED_BUCKET).join(Uuid::new_v4().to_string()); + os::rename_all_with_owner( + entry.path(), + target_path, + Self::meta_path(root, RUSTFS_META_BUCKET), + publication_root, + Some(owner), + ) + .await?; + Ok(()) + } + + async fn cleanup_deleted_objects(root: PathBuf, budget: &super::cleanup_runtime::GcBudget) -> Result<()> { let trash = Self::meta_path(&root, RUSTFS_META_TMP_DELETED_BUCKET); - let mut entries = match fs::read_dir(&trash).await { + let mut entries = match budget.step(async { Ok(fs::read_dir(&trash).await?) }).await { Ok(entries) => entries, Err(e) => { - if e.kind() == ErrorKind::NotFound { + if matches!(&e, DiskError::Io(err) if err.kind() == ErrorKind::NotFound) { return Ok(()); } - return Err(e.into()); + return Err(e); } }; - while let Some(entry) = entries.next_entry().await? { - let name = entry.file_name().to_string_lossy().to_string(); - if name.is_empty() || name == "." || name == ".." { - continue; - } - - let file_type = entry.file_type().await?; - - let path = trash.join(name); - - if file_type.is_dir() { - if let Err(e) = tokio::fs::remove_dir_all(path).await - && e.kind() != ErrorKind::NotFound - { - return Err(e.into()); - } - } else if let Err(e) = tokio::fs::remove_file(path).await - && e.kind() != ErrorKind::NotFound - { - return Err(e.into()); + loop { + let more = budget + .step(async { + let Some(entry) = entries.next_entry().await? else { + return Ok(false); + }; + Self::cleanup_trash_entry(entry, &trash).await?; + Ok(true) + }) + .await?; + if !more { + break; } } Ok(()) } + async fn cleanup_trash_entry(entry: fs::DirEntry, trash: &Path) -> Result<()> { + let name = entry.file_name().to_string_lossy().to_string(); + if name.is_empty() || name == "." || name == ".." { + return Ok(()); + } + let path = trash.join(name); + let result = if entry.file_type().await?.is_dir() { + fs::remove_dir_all(path).await + } else { + fs::remove_file(path).await + }; + match result { + Err(err) if err.kind() != ErrorKind::NotFound => Err(err.into()), + _ => Ok(()), + } + } + fn is_valid_volname(volname: &str) -> bool { if volname.len() < 3 { return false; @@ -8046,6 +8399,19 @@ impl LocalDisk { self.local_disk_object_path(self.io_root(), bucket, key) } + /// Only for operations that open through PublicationRoot or a commit guard: + /// containment and symlink checks happen with the actual open, off the runtime. + fn io_get_object_open_path(&self, bucket: &str, key: &str) -> Result { + #[cfg(target_os = "linux")] + { + let (bucket_path, path) = build_local_disk_object_path(self.io_root(), bucket, key); + check_local_disk_object_path_components(self.io_root(), &bucket_path, &path)?; + Ok(path) + } + #[cfg(not(target_os = "linux"))] + self.io_get_object_path(bucket, key) + } + fn io_get_bucket_path(&self, bucket: &str) -> Result { self.local_disk_bucket_path(self.io_root(), bucket) } @@ -8256,7 +8622,7 @@ impl LocalDisk { let mut meta = FileMeta::new(); if !fi.fresh { - let (buf, _) = read_file_exists(&p).await?; + let (buf, _) = read_file_exists(&p, &self.publication_root).await?; if !buf.is_empty() { let _ = meta.unmarshal_msg(&buf).map_err(|_| { meta = FileMeta::new(); @@ -8283,6 +8649,36 @@ impl LocalDisk { Ok(()) } + /// Keep the common GET completion on its caller. Only a last-reader + /// release with pending reclamation needs an owned cleanup task. Leave + /// that token untouched so the slow path owns the complete transition. + pub(in crate::disk) async fn release_snapshot_lease_without_cleanup( + &self, + volume: &str, + path: &str, + token: SnapshotLeaseToken, + ) -> bool { + let key = SnapshotLeaseKey { + volume: volume.to_owned(), + path: path.to_owned(), + }; + let mut registry = self.snapshot_leases.lock(); + let Some(entry) = registry.entries.get_mut(&key) else { + return true; + }; + if !entry.deleting + && entry.pending_delete.is_some() + && (entry.tokens.is_empty() || (entry.tokens.len() == 1 && entry.tokens.contains(&token))) + { + return false; + } + entry.tokens.remove(&token); + if entry.tokens.is_empty() && !entry.deleting { + registry.entries.remove(&key); + } + true + } + async fn delete_data_dir_with_namespace_owner( &self, volume: &str, @@ -8301,6 +8697,19 @@ impl LocalDisk { opts: DeleteOptions, namespace_owner: Option>, prepared: Option, + ) -> Result { + self.delete_data_dir_with_prepared_metadata(volume, path, opts, namespace_owner, prepared, None) + .await + } + + async fn delete_data_dir_with_prepared_metadata( + &self, + volume: &str, + path: &str, + opts: DeleteOptions, + namespace_owner: Option>, + prepared: Option, + metadata_owner: Option>, ) -> Result { let key = SnapshotLeaseKey { volume: volume.to_string(), @@ -8336,13 +8745,91 @@ impl LocalDisk { let result = if let Some(prepared) = prepared { self.delete_unleased_prepared(volume, path, &opts, prepared).await } else { - self.delete_unleased_with_namespace_owner(volume, path, &opts, namespace_owner) + self.delete_unleased_with_prepared_metadata(volume, path, &opts, namespace_owner, None, metadata_owner, None) .await }; claim.finish(result.is_ok()); result.map(|()| DataDirDeleteStatus::Deleted) } + pub(crate) async fn is_plain_orphan_data_dir(&self, volume: &str, path: &str) -> Result { + let path = self.io_get_object_path(volume, path)?; + let permit = acquire_context_disk_io_permit().await.map_err(DiskError::Io)?; + spawn_blocking_with_disk_io_permit(permit, move || match orphan_data_dir_parts(&path) { + Ok(_) => Ok(true), + Err(err) if matches!(err.kind(), ErrorKind::NotFound | ErrorKind::InvalidData) => Ok(false), + Err(err) => Err(to_file_error(err).into()), + }) + .await + .map_err(DiskError::from)? + } + + pub(crate) async fn delete_versions_with_namespace_owner( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + namespace_owner: Option>, + ) -> Vec> { + let mut errs = Vec::with_capacity(versions.len()); + for _ in 0..versions.len() { + errs.push(None); + } + + for (i, ver) in versions.iter().enumerate() { + if let Err(e) = self + .delete_versions_guarded_internal( + volume, + ver.name.as_str(), + &ver.versions, + &opts, + namespace_owner.clone(), + None, + None, + ) + .await + { + errs[i] = Some(e); + } else { + errs[i] = None; + } + } + + errs + } + + pub(crate) async fn delete_versions_guarded_with_namespace_owner( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + namespace_owner: Option>, + ) -> Vec> { + let payload = super::GuardedDeleteVersionsPayload { versions, opts }; + if let Err(error) = payload.validate() { + return vec![Some(error); payload.versions.len()]; + } + let super::GuardedDeleteVersionsPayload { versions, mut opts } = payload; + opts.recoverable_delete = true; + let mut errors = Vec::with_capacity(versions.len()); + for version in &versions { + errors.push( + self.delete_versions_guarded_internal( + volume, + &version.group.name, + &version.group.versions, + &opts, + namespace_owner.clone(), + Some(&version.expected_current), + None, + ) + .await + .err(), + ); + } + errors + } + async fn delete_version_inner(&self, volume: &str, path: &str, fi: FileInfo, mutation: DeleteVersionMutation) -> Result<()> { let DeleteVersionMutation { force_del_marker, @@ -8413,11 +8900,15 @@ impl LocalDisk { check_path_length(file_path.to_string_lossy().as_ref())?; let xl_path = path_join(&[file_path.as_path(), Path::new(STORAGE_FORMAT_FILE)]); - let namespace_owner: Option> = if let Some(publication) = &prepared_publication { + let mut namespace_owner: Option> = if let Some(publication) = &prepared_publication { Some(publication.clone()) } else { Some(os::acquire_metadata_mutation_lease(&self.get_object_path(volume, path)?, namespace_owner).await) }; + if prepared_publication.is_none() && (opts.recoverable_delete || (opts.undo_write && opts.undo_delete)) { + let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await; + namespace_owner = Some(Arc::new((namespace_owner, volume_guard))); + } // Reserve fresh trash destinations before changing metadata or data. A // held volume READ must never queue for another namespace behind delete. let prepared_trash = if let Some(publication) = &prepared_publication @@ -8430,8 +8921,17 @@ impl LocalDisk { None }; if opts.old_data_dir.is_some() && opts.undo_write { + if opts.undo_delete + && let Some(transaction) = opts.old_data_dir + && self + .rollback_prepared_delete(volume, path, transaction, namespace_owner.clone(), prepared_publication.clone()) + .await? + { + return Ok(()); + } return self .undo_write( + volume, file_path.as_path(), &fi, &opts, @@ -8454,20 +8954,13 @@ impl LocalDisk { // Leave residue for the existing unprepared cleanup/retry path // and preserve the original missing-file/version failure. if prepared_publication.is_none() { - self.finish_committed_delete_residue(volume, path, namespace_owner.clone()) + self.finish_internal_delete_residue(volume, path, namespace_owner.clone()) .await; } if fi.deleted && force_del_marker { return self - .write_missing_delete_marker( - volume, - path, - fi, - rollback_dir, - namespace_owner.clone(), - prepared_publication, - ) + .write_missing_delete_marker(volume, path, fi, &opts, namespace_owner.clone(), prepared_publication) .await; } @@ -8501,6 +8994,7 @@ impl LocalDisk { let prepared_rollback = if let (Some(rollback), Some(publication)) = (rollback_dir, &prepared_publication) { Some( DeleteRollbackPlan::prepare_forward( + volume, &file_path, &xl_path, rollback, @@ -8536,16 +9030,37 @@ impl LocalDisk { prepared_trash }; let mut reserved_version_delete = false; + let mut recovery_target = None; if let Some(rollback_dir) = rollback_dir { - write_delete_rollback_file_with_prepared_publication( - file_path.as_path(), - rollback_dir, - STORAGE_FORMAT_FILE_BACKUP, - &buf, - namespace_owner.clone(), - prepared_publication.clone(), - ) - .await?; + if opts.recoverable_delete { + recovery_target = self + .prepare_delete_recovery( + volume, + path, + Some(&buf), + std::slice::from_ref(&fi), + None, + delete_recovery::DeletePreparationContext { + transaction: rollback_dir, + bucket_incarnation: opts.delete_recovery_bucket_incarnation, + namespace_owner: namespace_owner.clone(), + prepared_publication: prepared_publication.clone(), + }, + ) + .await?; + } else { + write_delete_rollback_file_with_prepared_publication( + self.io_root(), + volume, + file_path.as_path(), + rollback_dir, + STORAGE_FORMAT_FILE_BACKUP, + &buf, + namespace_owner.clone(), + prepared_publication.clone(), + ) + .await?; + } } if let Some(uuid) = old_dir { @@ -8739,7 +9254,7 @@ impl LocalDisk { } let commit_result = if !meta.versions.is_empty() { - let buf = match meta.marshal_msg() { + let buf = match recovery_target.map(Ok).unwrap_or_else(|| meta.marshal_msg()) { Ok(buf) => buf, Err(err) => { let err: DiskError = err.into(); @@ -8832,6 +9347,10 @@ impl LocalDisk { .await); } + if let Some(transaction) = rollback_dir { + self.delete_recovery_barrier(volume, path, transaction, "APPLIED").await?; + } + if reserved_version_delete && let Some(rollback_dir) = rollback_dir && let Err(err) = self @@ -8919,6 +9438,7 @@ impl LocalDisk { async fn undo_write( &self, + volume: &str, file_path: &Path, fi: &FileInfo, opts: &DeleteOptions, @@ -8930,6 +9450,7 @@ impl LocalDisk { let xl_path = path_join(&[file_path, Path::new(STORAGE_FORMAT_FILE)]); if opts.undo_delete { restore_delete_rollback_with_prepared_publication( + volume, file_path, &xl_path, old_data_dir, @@ -9235,6 +9756,19 @@ impl LocalDisk { opts: &DeleteOptions, plan: PreparedPathDelete, ) -> Result<()> { + if opts.recursive && opts.old_data_dir.is_some() { + return self + .delete_unleased_with_prepared_metadata( + volume, + path, + opts, + Some(plan.lease.clone()), + None, + None, + Some(plan.lease), + ) + .await; + } let volume_dir = self.io_get_bucket_path(volume)?; let file_path = self.io_get_object_path(volume, path)?; let lease = plan.lease.clone(); @@ -9247,6 +9781,7 @@ impl LocalDisk { Some(lease), None, Some(plan), + None, ) .await?; self.io_backend.invalidate_cached_fds_under(volume, path); @@ -9264,6 +9799,37 @@ impl LocalDisk { .then(|| path.rsplit_once('/')) .flatten() .and_then(|(object, id)| Uuid::parse_str(id).ok().map(|id| (object, id))); + // Recovery receipts and orphan validation use main's exact cleanup + // executor. Admission has already captured the real volume owner, but + // no per-object namespace has been taken yet on this branch. + if opts.orphan_data_dir || transaction.is_some_and(|(_, id)| opts.old_data_dir == Some(id)) { + let object = if opts.orphan_data_dir { + path.rsplit_once('/') + .map(|(object, _)| object) + .ok_or(DiskError::InvalidPath)? + } else { + transaction.map(|(object, _)| object).ok_or(DiskError::InvalidPath)? + }; + let metadata = + os::try_acquire_metadata_mutation_lease_with_owner(&self.get_object_path(volume, object)?, owner.clone())?; + let publication = if opts.orphan_data_dir { + None + } else { + let transaction_path = self.io_get_object_path(volume, path)?; + let mut publication = os::try_acquire_namespace_mutation_lease_with_owner(&transaction_path, metadata.clone())?; + for name in [ + STORAGE_FORMAT_FILE_BACKUP, + DELETE_MARKER_ROLLBACK_FILE, + DELETE_QUORUM_COMMITTED_FILE, + ] { + publication = os::try_acquire_namespace_mutation_lease_with_owner(&transaction_path.join(name), publication)?; + } + Some(publication) + }; + return self + .delete_with_prepared_metadata(volume, path, opts, Some(owner), Some(metadata), publication) + .await; + } let object = transaction.map_or(path, |(object, _)| object); let publication = self.prepare_path_delete_object(volume, object, owner)?; let (data_paths, scan_error) = if let Some((object, transaction)) = transaction { @@ -9371,20 +9937,86 @@ impl LocalDisk { path: &str, opt: DeleteOptions, namespace_owner: Option>, + ) -> Result<()> { + self.delete_with_prepared_metadata(volume, path, opt, namespace_owner, None, None) + .await + } + + async fn delete_with_prepared_metadata( + &self, + volume: &str, + path: &str, + opt: DeleteOptions, + namespace_owner: Option>, + metadata_owner: Option>, + publication: Option>, ) -> Result<()> { crate::hp_guard!("LocalDisk::delete"); + if opt.orphan_data_dir { + return self + .delete_data_dir_with_prepared_metadata(volume, path, opt, namespace_owner, None, metadata_owner) + .await + .and_then(|status| match status { + DataDirDeleteStatus::Deleted => Ok(()), + DataDirDeleteStatus::Deferred => Err(DiskError::DiskOngoingReq), + }); + } + let mut opt = opt; + let mut namespace_owner = namespace_owner; let handled_version_delete = if opt.recursive && opt.immediate && let Some((object, transaction_id)) = path.rsplit_once('/') && let Ok(transaction_id) = Uuid::parse_str(transaction_id) + && !transaction_id.is_nil() { - self.finish_version_delete(volume, object, transaction_id, namespace_owner.clone()) - .await? + if opt.old_data_dir == Some(transaction_id) { + let object_path = self.get_object_path(volume, object)?; + namespace_owner = Some(match metadata_owner.as_ref() { + Some(metadata) => metadata.clone(), + None => os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await, + }); + let handled = self + .finish_delete_transaction_with_prepared_publication( + volume, + object, + transaction_id, + namespace_owner.clone(), + publication.clone(), + ) + .await?; + // A key below the UUID prefix may be created concurrently with + // cleanup. Never recursively remove newly published descendants. + opt.recursive = false; + handled + } else { + self.finish_version_delete(volume, object, transaction_id, false, namespace_owner.clone()) + .await? + .0 + } } else { false }; + let staging_cleanup = if volume == RUSTFS_META_TMP_BUCKET && opt.recursive { + let staging = tmp_staging_mutation_path(&self.root, path)?; + let lease = os::acquire_namespace_mutation_lease_with_owner(&staging, namespace_owner.take()).await; + namespace_owner = Some(lease.clone()); + Some(StagingCleanupLease { + io_prefix: tmp_staging_mutation_path(self.io_root(), path)?, + lease, + }) + } else { + None + }; match self - .delete_unleased_with_namespace_owner(volume, path, &opt, namespace_owner) + .delete_unleased_with_prepared_metadata( + volume, + path, + &opt, + namespace_owner, + staging_cleanup.as_ref(), + metadata_owner, + publication, + ) .await { Err(DiskError::FileNotFound) if handled_version_delete => Ok(()), @@ -9392,12 +10024,155 @@ impl LocalDisk { } } + // The caller owns the object metadata lease; retry recovery must not + // reacquire it while holding the same mutation domain. + async fn finish_delete_transaction_with_namespace_owner( + &self, + volume: &str, + object: &str, + transaction_id: Uuid, + namespace_owner: Option>, + ) -> Result { + self.finish_delete_transaction_with_prepared_publication(volume, object, transaction_id, namespace_owner, None) + .await + } + + async fn finish_delete_transaction_with_prepared_publication( + &self, + volume: &str, + object: &str, + transaction_id: Uuid, + namespace_owner: Option>, + publication: Option>, + ) -> Result { + let path = format!("{object}/{transaction_id}"); + let transaction_path = self.io_get_object_path(volume, &path)?; + let mut transaction_cleanup = false; + match fs::read_dir(&transaction_path).await { + Ok(mut entries) => { + while let Some(entry) = entries.next_entry().await.map_err(to_file_error)? { + let name = entry.file_name(); + if !entry.file_type().await.map_err(to_file_error)?.is_file() + || (name != STORAGE_FORMAT_FILE_BACKUP + && name != DELETE_MARKER_ROLLBACK_FILE + && name != DELETE_QUORUM_COMMITTED_FILE) + { + return Err(DiskError::FileCorrupt); + } + transaction_cleanup = true; + } + } + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + if transaction_cleanup { + // The set invokes transaction cleanup only after observing write + // quorum. Unlike data-dir markers, this receipt proves that + // decision and also covers inline objects with no data directory. + let receipt_path = transaction_path.join(DELETE_QUORUM_COMMITTED_FILE); + let lease = match &publication { + Some(publication) => publication.clone(), + None => os::acquire_namespace_mutation_lease_with_owner(&receipt_path, namespace_owner.clone()).await, + }; + let sync = effective_durability(volume).syncs_commit_metadata(); + os::run_blocking_namespace_operation(lease, move || { + let receipt = match std::fs::OpenOptions::new().write(true).create_new(true).open(&receipt_path) { + Ok(receipt) => receipt, + Err(err) if err.kind() == ErrorKind::AlreadyExists => { + if !std::fs::symlink_metadata(&receipt_path)?.is_file() { + return Err(std::io::Error::new(ErrorKind::InvalidData, "delete receipt is not a regular file")); + } + std::fs::File::open(receipt_path)? + } + Err(err) => return Err(err), + }; + if sync { + receipt.sync_all()?; + os::fsync_dir_std(&transaction_path)?; + if let Some(parent) = transaction_path.parent() { + os::fsync_dir_std(parent)?; + } + } + Ok(()) + }) + .await + .map_err(to_file_error)?; + // Retain the receipt until both the backup and marked data are + // gone. Every interrupted cleanup remains recognizable. + for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] { + let file = self.io_get_object_path(volume, &path)?.join(name); + let remove = match &publication { + Some(publication) => os::remove_file_with_lease(file, publication.clone()).await, + None => os::remove_file_with_owner(file, namespace_owner.clone()).await, + }; + match remove { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + } + if effective_durability(volume).syncs_commit_metadata() { + os::fsync_dir_with_owner(&self.io_get_object_path(volume, &path)?, namespace_owner.clone()) + .await + .map_err(to_file_error)?; + } + } + let (found, physically_deleted) = self + .finish_version_delete_with_prepared_publication( + volume, + object, + transaction_id, + true, + namespace_owner.clone(), + publication.clone(), + ) + .await?; + if !physically_deleted { + // Snapshot readers still own these parts. Keep the quorum receipt + // until their release finishes the physical data removal. + return Err(DiskError::DiskOngoingReq); + } + if transaction_cleanup { + let file = self.io_get_object_path(volume, &path)?.join(DELETE_QUORUM_COMMITTED_FILE); + let remove = match publication { + Some(publication) => os::remove_file_with_lease(file, publication).await, + None => os::remove_file_with_owner(file, namespace_owner).await, + }; + match remove { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + } + Ok(found || transaction_cleanup) + } + + #[cfg(test)] + async fn delete_unleased(&self, volume: &str, path: &str, opt: &DeleteOptions) -> Result<()> { + self.delete_unleased_with_namespace_owner(volume, path, opt, None, None).await + } + async fn delete_unleased_with_namespace_owner( &self, volume: &str, path: &str, opt: &DeleteOptions, namespace_owner: Option>, + staging_cleanup: Option<&StagingCleanupLease>, + ) -> Result<()> { + self.delete_unleased_with_prepared_metadata(volume, path, opt, namespace_owner, staging_cleanup, None, None) + .await + } + + async fn delete_unleased_with_prepared_metadata( + &self, + volume: &str, + path: &str, + opt: &DeleteOptions, + namespace_owner: Option>, + staging_cleanup: Option<&StagingCleanupLease>, + metadata_owner: Option>, + publication: Option>, ) -> Result<()> { let volume_dir = self.io_get_bucket_path(volume)?; if !skip_access_checks(volume) @@ -9408,8 +10183,187 @@ impl LocalDisk { let file_path = self.io_get_object_path(volume, path)?; check_path_length(file_path.to_string_lossy().as_ref())?; - self.delete_file_with_namespace_owner(&volume_dir, &file_path, opt.recursive, opt.immediate, namespace_owner) - .await?; + if opt.orphan_data_dir { + if opt.recursive || opt.old_data_dir.is_some() || opt.recoverable_delete { + return Err(DiskError::FileCorrupt); + } + let (object, data_dir) = path.rsplit_once('/').ok_or(DiskError::InvalidPath)?; + let data_dir = Uuid::parse_str(data_dir).map_err(|_| DiskError::InvalidPath)?; + if data_dir.is_nil() { + return Err(DiskError::InvalidPath); + } + // A snapshot may postpone this mutation until after the caller's + // object lock is released. Recheck metadata and recovery obligations + // under the same metadata -> namespace order as normal publication. + let owner = match metadata_owner.as_ref() { + Some(metadata) => metadata.clone(), + None => os::acquire_metadata_mutation_lease(&self.get_object_path(volume, object)?, namespace_owner).await, + }; + if self.has_delete_transaction_evidence(volume, object).await? { + return Err(DiskError::DiskOngoingReq); + } + let metadata_path = self.io_get_object_path(volume, &format!("{object}/{STORAGE_FORMAT_FILE}"))?; + let lease = match metadata_owner { + Some(_) => os::try_acquire_namespace_mutation_lease_with_owner(&file_path, owner)?, + None => os::acquire_namespace_mutation_lease_with_owner(&file_path, Some(owner)).await, + }; + let sync = effective_durability(volume).syncs_commit_metadata(); + let result = os::run_blocking_namespace_operation(lease, move || { + let bytes = std::fs::read(metadata_path)?; + let metadata = FileMeta::load(&bytes).map_err(|_| std::io::Error::from(ErrorKind::InvalidData))?; + if metadata + .get_data_dirs() + .map_err(|_| std::io::Error::from(ErrorKind::InvalidData))? + .into_iter() + .flatten() + .any(|referenced| referenced == data_dir) + { + return Err(std::io::Error::from(ErrorKind::InvalidData)); + } + let parts = orphan_data_dir_parts(&file_path)?; + for part in parts { + #[cfg(all(any(test, feature = "test-util"), not(windows)))] + os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &part); + // A descendant may be published after preflight. Unlinking + // only the classified files and using rmdir never removes it. + match std::fs::remove_file(part) { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(err), + } + } + if sync { + os::fsync_dir_std(&file_path)?; + } + std::fs::remove_dir(&file_path)?; + if sync && let Some(parent) = file_path.parent() { + os::fsync_dir_std(parent)?; + } + Ok(()) + }) + .await + .map_err(|err| { + if err.kind() == ErrorKind::InvalidData { + DiskError::FileCorrupt + } else { + to_file_error(err).into() + } + }); + self.io_backend.invalidate_cached_fds_under(volume, path); + return result; + } + let mut recursive = opt.recursive; + if recursive + && let Some(transaction) = opt.old_data_dir + && path + .rsplit_once('/') + .is_some_and(|(_, name)| Uuid::parse_str(name).is_ok_and(|id| id != transaction || opt.recoverable_delete)) + { + // Snapshot release replays these same options. Remove only this + // transaction's regular files; UUID-prefix child keys can appear + // after classification and must never be recursively removed. + let lease = match &publication { + Some(publication) => publication.clone(), + None => os::acquire_namespace_mutation_lease_with_owner(&file_path, namespace_owner.clone()).await, + }; + let data_path = file_path.clone(); + let sync = effective_durability(volume).syncs_commit_metadata(); + let keep_rollback_witnesses = opt.recoverable_delete; + os::run_blocking_namespace_operation(lease, move || { + let entries = match std::fs::read_dir(&data_path) { + Ok(entries) => entries, + Err(err) if err.kind() == ErrorKind::NotFound => return Ok(()), + Err(err) => return Err(err), + }; + let committed = format!("{DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"); + let recovery_committed = format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"); + let reserved = format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"); + let corrupt = + || std::io::Error::new(ErrorKind::InvalidData, "delete data directory contains unrecognized entries"); + let mut parts = Vec::new(); + let mut markers = Vec::new(); + let mut has_committed_marker = false; + for entry in entries { + let entry = entry?; + let name = entry.file_name(); + let name = name.to_str().ok_or_else(corrupt)?; + if !entry.file_type()?.is_file() { + return Err(corrupt()); + } + if delete_recovery_part_file(name) + || (name != reserved && is_delete_transaction_marker(name, RESERVED_DELETE_DATA_DIR_MARKER_PREFIX)) + { + // An interrupted legacy delete can reserve a still-live + // data dir. Only this transaction's commit marker below + // authorizes reclaiming it; remove the old reservation + // with the payload before dropping that proof. + parts.push(entry.path()); + } else if name == committed || name == recovery_committed || name == reserved { + has_committed_marker |= name == committed || name == recovery_committed; + markers.push(entry.path()); + } else if keep_rollback_witnesses + && [ + STORAGE_FORMAT_FILE_BACKUP, + DELETE_MARKER_ROLLBACK_FILE, + DELETE_QUORUM_COMMITTED_FILE, + ] + .contains(&name) + { + // Explicit legacy deletion keeps rollback witnesses until + // every snapshot-protected payload has disappeared. + } else { + return Err(corrupt()); + } + } + if !parts.is_empty() && !has_committed_marker { + return Err(corrupt()); + } + for files in [parts, markers] { + for entry in files { + #[cfg(all(any(test, feature = "test-util"), not(windows)))] + os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &entry); + match std::fs::remove_file(entry) { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(err), + } + } + if sync { + os::fsync_dir_std(&data_path)?; + } + } + match std::fs::remove_dir(data_path) { + Ok(()) => Ok(()), + Err(err) if is_benign_object_rmdir_error(&err) => Ok(()), + Err(err) => Err(err), + } + }) + .await + .map_err(|err| { + if err.kind() == ErrorKind::InvalidData { + DiskError::FileCorrupt + } else { + to_file_error(err).into() + } + })?; + recursive = false; + } + self.delete_file_with_prepared_entry( + &volume_dir, + &file_path, + recursive, + opt.immediate, + namespace_owner, + publication.clone(), + None, + publication.map(|lease| PreparedPathDelete { + lease, + trash: None, + encoded_trash: None, + }), + staging_cleanup, + ) + .await?; // A deleted shard must not remain readable through the io_uring fd cache. self.io_backend.invalidate_cached_fds_under(volume, path); Ok(()) @@ -9422,7 +10376,7 @@ impl LocalDisk { recursive: bool, immediate_purge: bool, ) -> Result<()> { - self.delete_file_with_namespace_owner(base_path, delete_path, recursive, immediate_purge, None) + self.delete_file_with_namespace_owner(base_path, delete_path, recursive, immediate_purge, None, None) .await } @@ -9434,8 +10388,9 @@ impl LocalDisk { recursive: bool, immediate_purge: bool, namespace_owner: Option>, + staging_cleanup: Option<&'a StagingCleanupLease>, ) -> BoxFuture<'a, Result<()>> { - self.delete_file_with_prepared_publication( + self.delete_file_with_prepared_entry( base_path, delete_path, recursive, @@ -9443,6 +10398,8 @@ impl LocalDisk { namespace_owner, None, None, + None, + staging_cleanup, ) } @@ -9465,6 +10422,7 @@ impl LocalDisk { prepared_publication, prepared_trash, None, + None, ) } @@ -9478,6 +10436,7 @@ impl LocalDisk { prepared_publication: Option>, prepared_trash: Option<(PathBuf, Arc)>, prepared_entry: Option, + staging_cleanup: Option<&'a StagingCleanupLease>, ) -> BoxFuture<'a, Result<()>> { Box::pin(async move { // debug!("delete_file {:?}\n base_path:{:?}", &delete_path, &base_path); @@ -9516,9 +10475,11 @@ impl LocalDisk { ) .await?; } - } else if delete_path.is_dir() { + } else if staging_cleanup.is_some_and(|staging| staging.io_prefix == *delete_path) || delete_path.is_dir() { // debug!("delete_file remove_dir {:?}", &delete_path); - let result = if let Some(publication) = &prepared_publication { + let result = if let Some(staging) = staging_cleanup.filter(|staging| staging.io_prefix == *delete_path) { + os::remove_dir_with_lease(delete_path, staging.lease.clone()).await + } else if let Some(publication) = &prepared_publication { let lease = if let Some(entry) = &prepared_entry { entry.lease.clone() } else { @@ -9602,7 +10563,7 @@ impl LocalDisk { } if let Some(dir_path) = delete_path.parent() { let result = self - .delete_file_with_prepared_publication( + .delete_file_with_prepared_entry( base_path, &PathBuf::from(dir_path), false, @@ -9610,6 +10571,8 @@ impl LocalDisk { namespace_owner.clone(), prepared_publication.clone(), None, + None, + staging_cleanup, ) .await; match result { @@ -9688,10 +10651,11 @@ impl LocalDisk { // - metadata failure -> to_file_error // - parse failure -> propagated verbatim from read_xl_meta_no_data_sync (`?`) let path = file_path.as_ref().to_path_buf(); + let root = self.publication_root.clone(); let (data, modtime) = spawn_blocking_with_disk_io_permit(io_permit, move || -> Result<(Vec, Option)> { // Read-only open, equivalent to O_RDONLY (get_readonly_options only sets read(true)). - let mut f = std::fs::File::open(&path).map_err(to_file_error)?; + let mut f = root.open_readonly(&path).map_err(to_file_error)?; let meta = f.metadata().map_err(to_file_error)?; @@ -9745,6 +10709,7 @@ impl LocalDisk { let metadata_path = self.io_get_object_path(bucket, path.to_string_lossy().as_ref()); // A part's existence check, metadata read and decode share one dispatch. // Keep open errors unmapped for the existing missing-volume fallback. + let root = self.publication_root.clone(); let result = spawn_blocking_with_disk_io_permit(io_permit, move || -> Result<_> { let part_error = |error: String| ObjectPartInfo { number: num, @@ -9757,7 +10722,7 @@ impl LocalDisk { // Invalid metadata paths remain request errors, but missing data wins // first, as it does in the serial reader. let metadata_path = metadata_path?; - Ok(read_all_data_std(&metadata_path) + Ok(read_all_data_std(root.open_readonly(&metadata_path)) .map(|(data, _)| ObjectPartInfo::unmarshal(&data).unwrap_or_else(|err| part_error(err.to_string())))) }) .await; @@ -9777,8 +10742,9 @@ impl LocalDisk { let object_dir = self.io_get_object_path(volume, object_name)?; let metadata_path = object_dir.join(STORAGE_FORMAT_FILE); let volume_dir = self.io_get_bucket_path(volume)?; + let root = self.publication_root.clone(); let result = spawn_blocking_with_disk_io_permit(io_permit, move || { - let (bytes, _) = read_all_data_std(&metadata_path)?; + let (bytes, _) = read_all_data_std(root.open_readonly(&metadata_path))?; let file_meta = FileMeta::load(&bytes).ok(); let data_dirs: HashSet = file_meta .as_ref() @@ -9876,7 +10842,8 @@ impl LocalDisk { // gating the volume fallback on the open error alone is equivalent to // the original code, where the fallback lived solely in the open match arm. let path = file_path.as_ref().to_path_buf(); - let res = spawn_blocking_with_disk_io_permit(io_permit, move || read_all_data_std(&path)) + let root = self.publication_root.clone(); + let res = spawn_blocking_with_disk_io_permit(io_permit, move || read_all_data_std(root.open_readonly(&path))) .await .map_err(DiskError::from)?; @@ -9890,14 +10857,18 @@ impl LocalDisk { volume: &str, path: &str, fi: FileInfo, - rollback_dir: Option, + opts: &DeleteOptions, namespace_owner: Option>, prepared_publication: Option>, ) -> Result<()> { + let rollback_dir = opts.old_data_dir; let object_dir = self.io_get_object_path(volume, path)?; let xl_path = object_dir.join(STORAGE_FORMAT_FILE); let prepared_rollback = if let (Some(rollback), Some(publication)) = (rollback_dir, &prepared_publication) { - Some(DeleteRollbackPlan::prepare_forward(&object_dir, &xl_path, rollback, &[], true, publication.clone()).await?) + Some( + DeleteRollbackPlan::prepare_forward(volume, &object_dir, &xl_path, rollback, &[], true, publication.clone()) + .await?, + ) } else { None }; @@ -9909,26 +10880,63 @@ impl LocalDisk { .as_ref() .map(|owner| owner.clone() as Arc) .or(namespace_owner); + let mut recovery_target = None; if let Some(rollback_dir) = rollback_dir { - write_delete_rollback_file_with_prepared_publication( - &object_dir, - rollback_dir, - DELETE_MARKER_ROLLBACK_FILE, - &[], - namespace_owner.clone(), - prepared_publication.clone(), - ) - .await?; + if opts.recoverable_delete { + recovery_target = self + .prepare_delete_recovery( + volume, + path, + None, + std::slice::from_ref(&fi), + None, + delete_recovery::DeletePreparationContext { + transaction: rollback_dir, + bucket_incarnation: opts.delete_recovery_bucket_incarnation, + namespace_owner: namespace_owner.clone(), + prepared_publication: prepared_publication.clone(), + }, + ) + .await?; + } else { + write_delete_rollback_file_with_prepared_publication( + self.io_root(), + volume, + &object_dir, + rollback_dir, + DELETE_MARKER_ROLLBACK_FILE, + &[], + namespace_owner.clone(), + prepared_publication.clone(), + ) + .await?; + } } - if let Err(err) = self - .write_metadata_with_namespace_owner(volume, path, fi, namespace_owner.clone(), prepared_publication) + let committed = if let Some(target) = recovery_target { + async { + fi.validate_for_metadata_read()?; + self.write_all_meta_with_namespace_owner( + volume, + &format!("{path}/{STORAGE_FORMAT_FILE}"), + &target, + true, + namespace_owner.clone(), + prepared_publication, + ) + .await + } .await - { + } else { + self.write_metadata_with_namespace_owner(volume, path, fi, namespace_owner.clone(), prepared_publication) + .await + }; + if let Err(err) = committed { let restore = if let Some(rollback_dir) = rollback_dir { if let Some(mut plan) = prepared_rollback { - match plan.refresh_forward(&object_dir, rollback_dir).await { + match plan.refresh_forward(volume, &object_dir, rollback_dir).await { Ok(()) => { restore_delete_rollback_plan( + volume, &object_dir, &xl_path, rollback_dir, @@ -9942,6 +10950,7 @@ impl LocalDisk { } } else { restore_delete_rollback_with_namespace_owner( + volume, &object_dir, &xl_path, rollback_dir, @@ -9970,6 +10979,9 @@ impl LocalDisk { } return Err(err); } + if let Some(transaction) = rollback_dir { + self.delete_recovery_barrier(volume, path, transaction, "APPLIED").await?; + } Ok(()) } @@ -10029,9 +11041,45 @@ impl LocalDisk { errors } + pub(in crate::disk) async fn delete_versions_guarded_with_owner( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + owner: Arc, + ) -> Vec> { + let mut errors = Vec::with_capacity(versions.len()); + for version in versions { + let result = async { + let publication = self.prepare_bulk_delete_group(volume, &version.group.name, owner.clone())?; + self.delete_versions_guarded_internal( + volume, + &version.group.name, + &version.group.versions, + &opts, + None, + Some(&version.expected_current), + Some(publication), + ) + .await + } + .await; + errors.push(result.err()); + } + errors + } + #[cfg(test)] - async fn delete_versions_internal(&self, volume: &str, path: &str, fis: &[FileInfo], opts: &DeleteOptions) -> Result<()> { - self.delete_versions_internal_prepared(volume, path, fis, opts, None).await + async fn delete_versions_internal( + &self, + volume: &str, + path: &str, + fis: &[FileInfo], + opts: &DeleteOptions, + namespace_owner: Option>, + ) -> Result<()> { + self.delete_versions_guarded_internal(volume, path, fis, opts, namespace_owner, None, None) + .await } async fn delete_versions_internal_prepared( @@ -10041,14 +11089,32 @@ impl LocalDisk { fis: &[FileInfo], opts: &DeleteOptions, prepared_publication: Option>, + ) -> Result<()> { + self.delete_versions_guarded_internal(volume, path, fis, opts, None, None, prepared_publication) + .await + } + + async fn delete_versions_guarded_internal( + &self, + volume: &str, + path: &str, + fis: &[FileInfo], + opts: &DeleteOptions, + namespace_owner: Option>, + expected_current: Option<&CurrentDeleteGeneration>, + prepared_publication: Option>, ) -> Result<()> { let volume_dir = self.io_get_bucket_path(volume)?; let object_path = self.get_object_path(volume, path)?; - let namespace_owner: Option> = if let Some(publication) = &prepared_publication { + let mut namespace_owner: Option> = if let Some(publication) = &prepared_publication { Some(publication.clone()) } else { - Some(os::acquire_metadata_mutation_lease(&object_path, None).await) + Some(os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await) }; + if prepared_publication.is_none() && (opts.recoverable_delete || (opts.undo_write && opts.undo_delete)) { + let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await; + namespace_owner = Some(Arc::new((namespace_owner, volume_guard))); + } let xlpath = self.io_get_object_path(volume, format!("{path}/{STORAGE_FORMAT_FILE}").as_str())?; let object_dir = xlpath .parent() @@ -10058,7 +11124,14 @@ impl LocalDisk { && opts.undo_write { if opts.undo_delete { + if self + .rollback_prepared_delete(volume, path, rollback_dir, namespace_owner.clone(), prepared_publication.clone()) + .await? + { + return Ok(()); + } return restore_delete_rollback_with_prepared_publication( + volume, object_dir, &xlpath, rollback_dir, @@ -10086,8 +11159,11 @@ impl LocalDisk { { Ok(data) => data, Err(DiskError::FileNotFound) => { + if expected_current.is_some_and(|expected| *expected != CurrentDeleteGeneration::Absent) { + return Err(DiskError::ConditionalDeleteConflict); + } if prepared_publication.is_none() { - self.finish_committed_delete_residue(volume, path, namespace_owner.clone()) + self.finish_internal_delete_residue(volume, path, namespace_owner.clone()) .await; } // `deleted` alone can be an explicit marker purge; only @@ -10096,14 +11172,7 @@ impl LocalDisk { return Err(DiskError::FileNotFound); }; return self - .write_missing_delete_marker( - volume, - path, - delete_marker, - opts.old_data_dir, - namespace_owner.clone(), - prepared_publication, - ) + .write_missing_delete_marker(volume, path, delete_marker, opts, namespace_owner.clone(), prepared_publication) .await; } Err(err) => return Err(err), @@ -10116,45 +11185,39 @@ impl LocalDisk { let mut fm = FileMeta::default(); fm.unmarshal_msg(&data)?; - let rollback_dir = opts.old_data_dir; - // Apply the original metadata operations once, in request order. Keep - // each outcome so filesystem errors retain precedence over later - // metadata errors while all required paths are known before writing. - let mut planned = if prepared_publication.is_some() { - let mut outcomes = Vec::with_capacity(fis.len()); - for fi in fis { - let outcome = fm.delete_version(fi).map_err(DiskError::from); - if let Ok(Some(dir)) = &outcome { - let _ = fm.data.remove(vec![fi.version_id.unwrap_or_default(), *dir]); - } - let fatal = outcome.as_ref().is_err_and(|error| { - fi.deleted || (*error != DiskError::FileNotFound && *error != DiskError::FileVersionNotFound) - }); - outcomes.push(outcome); - if fatal { - break; - } + if let Some(expected) = expected_current { + let actual = match fm.into_fileinfo(volume, path, "", false, false, false) { + Ok(fi) => CurrentDeleteGeneration::from_file_info(&fi)?, + Err(rustfs_filemeta::Error::FileNotFound) => CurrentDeleteGeneration::Absent, + Err(error) => return Err(error.into()), + }; + // Metadata mutation ownership is held from this comparison through + // recovery preparation and publication, so a stale request cannot + // stage a replacement generation for deletion. + if &actual != expected { + return Err(DiskError::ConditionalDeleteConflict); } - Some(outcomes.into_iter()) - } else { - None - }; - let mut data_dirs: Vec = planned + } + // Compact inline data once, but preserve backup syscall error priority. + // The metadata result is propagated only after rollback evidence exists. + let metadata_result = fm.delete_versions(fis); + let data_dirs: Vec = metadata_result .as_ref() - .map(|outcomes| { - outcomes - .as_slice() - .iter() - .filter_map(|result| result.as_ref().ok().copied().flatten()) - .collect() - }) + .map(|dirs| dirs.iter().copied().flatten().collect()) .unwrap_or_default(); - data_dirs.sort(); - data_dirs.dedup(); + let rollback_dir = opts.old_data_dir; let prepared_rollback = if let (Some(rollback), Some(publication)) = (rollback_dir, &prepared_publication) { Some( - DeleteRollbackPlan::prepare_forward(object_dir, &xlpath, rollback, &data_dirs, false, publication.clone()) - .await?, + DeleteRollbackPlan::prepare_forward( + volume, + object_dir, + &xlpath, + rollback, + &data_dirs, + false, + publication.clone(), + ) + .await?, ) } else { None @@ -10179,30 +11242,61 @@ impl LocalDisk { None }; let mut reserved_version_delete = false; + let mut recovery_target = None; if let Some(rollback_dir) = rollback_dir { - write_delete_rollback_file_with_prepared_publication( - object_dir, - rollback_dir, - STORAGE_FORMAT_FILE_BACKUP, - &data, - namespace_owner.clone(), - prepared_publication.clone(), - ) - .await?; + if opts.recoverable_delete { + recovery_target = self + .prepare_delete_recovery( + volume, + path, + Some(&data), + fis, + None, + delete_recovery::DeletePreparationContext { + transaction: rollback_dir, + bucket_incarnation: opts.delete_recovery_bucket_incarnation, + namespace_owner: namespace_owner.clone(), + prepared_publication: prepared_publication.clone(), + }, + ) + .await?; + } else { + write_delete_rollback_file_with_prepared_publication( + self.io_root(), + volume, + object_dir, + rollback_dir, + STORAGE_FORMAT_FILE_BACKUP, + &data, + namespace_owner.clone(), + prepared_publication.clone(), + ) + .await?; + } } - for fi in fis.iter() { - let outcome = match &mut planned { - Some(outcomes) => outcomes.next().ok_or(DiskError::Unexpected)?, - None => fm.delete_version(fi).map_err(DiskError::from), - }; - let data_dir = match outcome { - Ok(res) => res, - Err(err) => { - if !fi.deleted && (err == DiskError::FileNotFound || err == DiskError::FileVersionNotFound) { - continue; - } + if let Err(error) = metadata_result { + return Err(restore_delete_rollback_after_error( + object_dir, + &xlpath, + rollback_dir, + volume, + path, + DeleteRollbackFailure { + stage: "delete_versions_metadata_plan", + error: error.into(), + namespace_owner: namespace_owner.clone(), + prepared: prepared_rollback.clone(), + }, + &self.publication_root, + ) + .await); + } + for dir in data_dirs { + let dir_path = match self.io_get_object_path(volume, format!("{path}/{dir}").as_str()) { + Ok(dir_path) => dir_path, + Err(err) => { if reserved_version_delete && let Some(rollback_dir) = rollback_dir { return Err(self .abort_reserved_version_delete_with_failure( @@ -10211,7 +11305,7 @@ impl LocalDisk { volume, path, DeleteRollbackFailure { - stage: "delete_versions_metadata_update", + stage: "delete_versions_data_path", error: err, namespace_owner: namespace_owner.clone(), prepared: prepared_rollback.clone(), @@ -10226,7 +11320,7 @@ impl LocalDisk { volume, path, DeleteRollbackFailure { - stage: "delete_versions_metadata_update", + stage: "delete_versions_data_path", error: err, namespace_owner: namespace_owner.clone(), prepared: prepared_rollback.clone(), @@ -10236,205 +11330,158 @@ impl LocalDisk { .await); } }; - - if let Some(dir) = data_dir { - let vid = fi.version_id.unwrap_or_default(); - if prepared_publication.is_none() { - let _ = fm.data.remove(vec![vid, dir]); + if let Some(rollback_dir) = rollback_dir { + let rollback_path = object_dir.join(rollback_dir.to_string()); + let create = if let Some(publication) = &prepared_publication { + os::create_dir_all_with_lease(rollback_path.clone(), publication.clone()).await + } else { + fs::create_dir_all(&rollback_path).await + }; + if let Err(err) = create { + let err: DiskError = to_file_error(err).into(); + if reserved_version_delete { + return Err(self + .abort_reserved_version_delete_with_failure( + object_dir, + rollback_dir, + volume, + path, + DeleteRollbackFailure { + stage: "delete_versions_rollback_dir", + error: err, + namespace_owner: namespace_owner.clone(), + prepared: prepared_rollback.clone(), + }, + ) + .await); + } + return Err(restore_delete_rollback_after_error( + object_dir, + &xlpath, + Some(rollback_dir), + volume, + path, + DeleteRollbackFailure { + stage: "delete_versions_rollback_dir", + error: err, + namespace_owner: namespace_owner.clone(), + prepared: prepared_rollback.clone(), + }, + &self.publication_root, + ) + .await); } - - let dir_path = match self.io_get_object_path(volume, format!("{path}/{dir}").as_str()) { - Ok(dir_path) => dir_path, - Err(err) => { - if reserved_version_delete && let Some(rollback_dir) = rollback_dir { - return Err(self - .abort_reserved_version_delete_with_failure( - object_dir, - rollback_dir, - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_data_path", - error: err, - namespace_owner: namespace_owner.clone(), - prepared: prepared_rollback.clone(), - }, - ) - .await); - } - return Err(restore_delete_rollback_after_error( - object_dir, - &xlpath, - rollback_dir, - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_data_path", - error: err, - namespace_owner: namespace_owner.clone(), - prepared: prepared_rollback.clone(), - }, - &self.publication_root, - ) - .await); - } - }; - if let Some(rollback_dir) = rollback_dir { - let rollback_path = object_dir.join(rollback_dir.to_string()); - let create = if let Some(publication) = &prepared_publication { - os::create_dir_all_with_lease(rollback_path.clone(), publication.clone()).await - } else { - fs::create_dir_all(&rollback_path).await - }; - if let Err(err) = create { - let err: DiskError = to_file_error(err).into(); - if reserved_version_delete { - return Err(self - .abort_reserved_version_delete_with_failure( - object_dir, - rollback_dir, - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_rollback_dir", - error: err, - namespace_owner: namespace_owner.clone(), - prepared: prepared_rollback.clone(), - }, - ) - .await); - } - return Err(restore_delete_rollback_after_error( - object_dir, - &xlpath, - Some(rollback_dir), - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_rollback_dir", - error: err, - namespace_owner: namespace_owner.clone(), - prepared: prepared_rollback.clone(), - }, - &self.publication_root, - ) - .await); - } - let reserved = match self - .reserve_version_delete_with_prepared_publication( - volume, - path, - dir, - rollback_dir, - namespace_owner.clone(), - prepared_publication.clone(), - ) - .await - { - Ok(reserved) => reserved, - Err(err) => { - return Err(self - .abort_reserved_version_delete_with_failure( - object_dir, - rollback_dir, - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_reserve_data", - error: err, - namespace_owner: namespace_owner.clone(), - prepared: prepared_rollback.clone(), - }, - ) - .await); - } - }; - reserved_version_delete |= reserved; - let rollback_data_path = rollback_path.join(dir.to_string()); - let stage = if reserved { - Ok(()) - } else if let Some(publication) = &prepared_publication { - os::rename_all_ignore_missing_source_with_lease( - &dir_path, - &rollback_data_path, - &rollback_path, - &self.publication_root, - publication.clone(), - ) - .await - } else { - os::rename_all_ignore_missing_source_with_owner( - &dir_path, - &rollback_data_path, - &rollback_path, - &self.publication_root, - namespace_owner.clone(), - ) - .await - }; - if let Err(err) = stage { - return Err(restore_delete_rollback_after_error( - object_dir, - &xlpath, - Some(rollback_dir), - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_stage_data", - error: err, - namespace_owner: namespace_owner.clone(), - prepared: prepared_rollback.clone(), - }, - &self.publication_root, - ) - .await); - } - if should_fail_after_delete_data_staged(path) { - if reserved_version_delete { - return Err(self - .abort_reserved_version_delete_with_failure( - object_dir, - rollback_dir, - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_test_after_stage", - error: DiskError::Unexpected, - namespace_owner: namespace_owner.clone(), - prepared: prepared_rollback.clone(), - }, - ) - .await); - } - return Err(restore_delete_rollback_after_error( - object_dir, - &xlpath, - Some(rollback_dir), - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_test_after_stage", - error: DiskError::Unexpected, - namespace_owner: namespace_owner.clone(), - prepared: prepared_rollback.clone(), - }, - &self.publication_root, - ) - .await); - } - } else if let Err(err) = self - .move_to_trash_with_prepared_target(&dir_path, true, false, namespace_owner.clone(), data_trash.remove(&dir)) + let reserved = match self + .reserve_version_delete_with_prepared_publication( + volume, + path, + dir, + rollback_dir, + namespace_owner.clone(), + prepared_publication.clone(), + ) .await - && !(err == DiskError::FileNotFound || err == DiskError::VolumeNotFound) { - return Err(err); + Ok(reserved) => reserved, + Err(err) => { + return Err(self + .abort_reserved_version_delete_with_failure( + object_dir, + rollback_dir, + volume, + path, + DeleteRollbackFailure { + stage: "delete_versions_reserve_data", + error: err, + namespace_owner: namespace_owner.clone(), + prepared: prepared_rollback.clone(), + }, + ) + .await); + } }; - - // The version's data dir was staged or trashed; drop any cached - // io_uring descriptors under it so a deleted `part.N` cannot keep - // answering reads (rustfs/backlog#1175). - self.io_backend.invalidate_cached_fds_under(volume, &format!("{path}/{dir}")); + reserved_version_delete |= reserved; + let rollback_data_path = rollback_path.join(dir.to_string()); + let stage = if reserved { + Ok(()) + } else if let Some(publication) = &prepared_publication { + os::rename_all_ignore_missing_source_with_lease( + &dir_path, + &rollback_data_path, + &rollback_path, + &self.publication_root, + publication.clone(), + ) + .await + } else { + os::rename_all_ignore_missing_source_with_owner( + &dir_path, + &rollback_data_path, + &rollback_path, + &self.publication_root, + namespace_owner.clone(), + ) + .await + }; + if let Err(err) = stage { + return Err(restore_delete_rollback_after_error( + object_dir, + &xlpath, + Some(rollback_dir), + volume, + path, + DeleteRollbackFailure { + stage: "delete_versions_stage_data", + error: err, + namespace_owner: namespace_owner.clone(), + prepared: prepared_rollback.clone(), + }, + &self.publication_root, + ) + .await); + } + if should_fail_after_delete_data_staged(path) { + if reserved_version_delete { + return Err(self + .abort_reserved_version_delete_with_failure( + object_dir, + rollback_dir, + volume, + path, + DeleteRollbackFailure { + stage: "delete_versions_test_after_stage", + error: DiskError::Unexpected, + namespace_owner: namespace_owner.clone(), + prepared: prepared_rollback.clone(), + }, + ) + .await); + } + return Err(restore_delete_rollback_after_error( + object_dir, + &xlpath, + Some(rollback_dir), + volume, + path, + DeleteRollbackFailure { + stage: "delete_versions_test_after_stage", + error: DiskError::Unexpected, + namespace_owner: namespace_owner.clone(), + prepared: prepared_rollback.clone(), + }, + &self.publication_root, + ) + .await); + } + } else if let Err(err) = self + .move_to_trash_with_prepared_target(&dir_path, true, false, namespace_owner.clone(), data_trash.remove(&dir)) + .await + && !(err == DiskError::FileNotFound || err == DiskError::VolumeNotFound) + { + return Err(err); } + // Removed shard descriptors must not remain readable through the cache. + self.io_backend.invalidate_cached_fds_under(volume, &format!("{path}/{dir}")); } // Remove xl.meta when no versions remain @@ -10483,6 +11530,9 @@ impl LocalDisk { ) .await); } + if let Some(transaction) = rollback_dir { + self.delete_recovery_barrier(volume, path, transaction, "APPLIED").await?; + } if reserved_version_delete && let Some(rollback_dir) = rollback_dir && let Err(err) = self @@ -10518,7 +11568,7 @@ impl LocalDisk { // Update xl.meta atomically: a concurrent reader or crash mid-write must // never observe a truncated xl.meta for versions that were not deleted. - let buf = match fm.marshal_msg() { + let buf = match recovery_target.map(Ok).unwrap_or_else(|| fm.marshal_msg()) { Ok(buf) => buf, Err(err) => { let err: DiskError = err.into(); @@ -10600,6 +11650,10 @@ impl LocalDisk { .await); } + if let Some(transaction) = rollback_dir { + self.delete_recovery_barrier(volume, path, transaction, "APPLIED").await?; + } + if reserved_version_delete && let Some(rollback_dir) = rollback_dir && let Err(err) = self @@ -10938,17 +11992,17 @@ impl LocalDisk { } } + let root = self.publication_root.clone(); tokio::task::spawn_blocking(move || { let _owner = owner; #[cfg(test)] run_owned_file_write_before_open(&path); - let mut file = std::fs::OpenOptions::new() - .create(true) - .write(true) - .truncate(true) - .open(&path) - .map_err(to_file_error)?; + let (mut file, parent_handle) = if sync == SyncMode::FileAndDir { + root.create_truncate_with_parent(&path).map_err(to_file_error)? + } else { + (root.create_truncate(&path).map_err(to_file_error)?, None) + }; std::io::Write::write_all(&mut file, buf.as_ref()).map_err(to_file_error)?; if sync != SyncMode::None { file.sync_data().map_err(to_file_error)?; @@ -10957,7 +12011,7 @@ impl LocalDisk { if sync == SyncMode::FileAndDir && let Some(parent) = path.parent() { - os::fsync_dir_std(parent).map_err(to_file_error)?; + os::fsync_directory_handle_std(parent, parent_handle.as_ref()).map_err(to_file_error)?; } } Ok::<_, std::io::Error>(()) @@ -10982,7 +12036,18 @@ impl LocalDisk { os::make_dir_all(parent, skip_parent).await?; } - let f = super::fs::open_file(path.as_ref(), mode).await.map_err(to_file_error)?; + let f = if mode == O_CREATE | O_WRONLY | O_TRUNC { + let path = path.as_ref().to_path_buf(); + let root = self.publication_root.clone(); + File::from_std( + tokio::task::spawn_blocking(move || root.create_truncate(&path)) + .await + .map_err(DiskError::from)? + .map_err(to_file_error)?, + ) + } else { + super::fs::open_file(path.as_ref(), mode).await.map_err(to_file_error)? + }; Ok(f) } @@ -11860,9 +12925,9 @@ fn file_meta_counts_toward_limit(meta: &FileMeta) -> bool { } // Filter std::io::ErrorKind::NotFound -async fn read_file_exists(path: impl AsRef) -> Result<(Bytes, Option)> { +async fn read_file_exists(path: impl AsRef, root: &os::PublicationRoot) -> Result<(Bytes, Option)> { let p = path.as_ref(); - let (data, meta) = match read_file_all(&p).await { + let (data, meta) = match read_file_all(&p, root).await { Ok((data, meta)) => (data, Some(meta)), Err(e) => { if e == Error::FileNotFound { @@ -11881,32 +12946,28 @@ async fn read_file_exists(path: impl AsRef) -> Result<(Bytes, Option) -> Result<(Bytes, Metadata)> { - let p = path.as_ref(); - let meta = read_file_metadata(&path).await?; - - let data = fs::read(&p) - .await - .inspect_err(|err| { - log_startup_disk_io_error("read_file_all", p, err); - }) - .map_err(to_file_error)?; - - Ok((data.into(), meta)) -} - -async fn read_file_metadata(p: impl AsRef) -> Result { - let path = p.as_ref(); - let meta = fs::metadata(path) - .await - .inspect_err(|err| { - if err.kind() != ErrorKind::NotFound { - log_startup_disk_io_error("read_file_metadata", path, err); - } - }) - .map_err(to_file_error)?; - - Ok(meta) +async fn read_file_all(path: impl AsRef, root: &os::PublicationRoot) -> Result<(Bytes, Metadata)> { + let path = path.as_ref().to_path_buf(); + let root = root.clone(); + tokio::task::spawn_blocking(move || { + let result = (|| { + let mut file = root.open_readonly(&path)?; + let metadata = file.metadata()?; + let mut data = Vec::new(); + std::io::Read::read_to_end(&mut file, &mut data)?; + Ok::<_, std::io::Error>((Bytes::from(data), metadata)) + })(); + result + .inspect_err(|err| { + if err.kind() != ErrorKind::NotFound { + log_startup_disk_io_error("read_file_all", &path, err); + } + }) + .map_err(to_file_error) + .map_err(DiskError::from) + }) + .await + .map_err(DiskError::from)? } fn skip_access_checks(p: impl AsRef) -> bool { @@ -11926,6 +12987,23 @@ fn skip_access_checks(p: impl AsRef) -> bool { false } +struct StagingCleanupLease { + io_prefix: PathBuf, + lease: Arc, +} + +// Use the configured root rather than per-handle /proc fd paths. A nested +// publication and cleanup of its top-level TMP entry must share one lease. +fn tmp_staging_mutation_path(root: &Path, key: &str) -> Result { + let (temporary, object) = build_local_disk_object_path(root, RUSTFS_META_TMP_BUCKET, key); + let object = normalize_path_components(object); + let relative = object.strip_prefix(&temporary).map_err(|_| DiskError::InvalidPath)?; + Ok(relative + .components() + .next() + .map_or_else(|| temporary.clone(), |first| temporary.join(first.as_os_str()))) +} + fn local_disk_object_path(root: &Path, bucket: &str, key: &str) -> Result { let (bucket_path, path) = build_local_disk_object_path(root, bucket, key); check_local_disk_valid_object_path(root, &bucket_path, &path)?; @@ -11988,13 +13066,19 @@ fn check_local_disk_valid_path(root: &Path, path: impl AsRef) -> Result<() #[cfg(target_os = "linux")] fn check_local_disk_valid_object_path_at(root: &Path, root_fd: &std::fs::File, bucket_path: &Path, path: &Path) -> Result<()> { + check_local_disk_object_path_components(root, bucket_path, path)?; + reject_local_disk_symlink_components_at(root, root_fd, &normalize_path_components(path)) +} + +#[cfg(target_os = "linux")] +fn check_local_disk_object_path_components(root: &Path, bucket_path: &Path, path: &Path) -> Result<()> { let bucket_path = normalize_path_components(bucket_path); let path = normalize_path_components(path); if !bucket_path.starts_with(root) || !path.starts_with(&bucket_path) { return Err(DiskError::InvalidPath); } - reject_local_disk_symlink_components_at(root, root_fd, &path) + Ok(()) } #[cfg(target_os = "linux")] @@ -12244,11 +13328,48 @@ impl LocalDisk { marker_paths = version_delete_marker_paths(&self.io_get_object_path(volume, object)?, rollback_dir).await?; &marker_paths }; - for pair in paths.chunks_exact(2) { - let reserved_path = &pair[0]; - let committed_path = &pair[1]; + self.commit_reserved_version_delete_marker_paths( + volume, + object, + rollback_dir, + namespace_owner, + paths, + prepared.and_then(|plan| plan.publication.clone()), + ) + .await + } + + async fn commit_reserved_version_delete_with_publication( + &self, + volume: &str, + object: &str, + rollback_dir: Uuid, + namespace_owner: Option>, + publication: Option>, + ) -> Result<()> { + let paths = version_delete_marker_paths(&self.io_get_object_path(volume, object)?, rollback_dir).await?; + self.commit_reserved_version_delete_marker_paths(volume, object, rollback_dir, namespace_owner, &paths, publication) + .await + } + + async fn commit_reserved_version_delete_marker_paths( + &self, + volume: &str, + object: &str, + rollback_dir: Uuid, + namespace_owner: Option>, + paths: &[PathBuf], + publication: Option>, + ) -> Result<()> { + let recovery_owned = + fs::try_exists(self.io_get_object_path(volume, &format!("{object}/{rollback_dir}/{}", delete_recovery::PREPARED))?) + .await + .map_err(to_file_error)?; + for paths in paths.chunks_exact(3) { + let reserved_path = &paths[0]; + let committed_path = &paths[if recovery_owned { 2 } else { 1 }]; let data_path = reserved_path.parent().ok_or(DiskError::InvalidPath)?; - let rename = if let Some(publication) = prepared.and_then(|plan| plan.publication.clone()) { + let rename = if let Some(publication) = publication.clone() { os::rename_with_lease(reserved_path.clone(), committed_path.clone(), publication).await } else { os::rename_with_namespace_owner(reserved_path, committed_path, namespace_owner.clone()).await @@ -12278,14 +13399,21 @@ impl LocalDisk { /// removed the object's metadata on this disk and committed, so the data /// it marked can no longer be read, healed or rolled back here. Finishing /// removes what `T`'s own cleanup pass would have removed. Reserved - /// markers, rollback backups without a committed marker, and object + /// transaction receipts also prove quorum commitment for inline objects. + /// Reserved markers, rollback backups without either proof, and object /// directories that hold `xl.meta` are left alone. - async fn finish_committed_delete_residue( + async fn finish_internal_delete_residue( &self, volume: &str, path: &str, namespace_owner: Option>, ) -> usize { + // RUSTFS_COMPAT_TODO(backlog#2807): retain legacy cleanup for internal + // maintenance until old system-bucket records drain. User-bucket + // residue requires the explicit set-wide recovery decision. + if !crate::bucket::utils::is_meta_bucketname(volume) { + return 0; + } let transactions = match self.committed_delete_transactions(volume, path).await { Ok(transactions) => transactions, Err(err) => { @@ -12295,7 +13423,7 @@ impl LocalDisk { subsystem = LOG_SUBSYSTEM_DISK_LOCAL, volume, path, - operation = "scan_committed_delete_residue", + operation = "scan_internal_delete_residue", error = ?err, "Disk local delete failed" ); @@ -12309,17 +13437,50 @@ impl LocalDisk { }; let mut finished = 0; for transaction in transactions { - // Unlike the delete's own pass, drop the rollback backup before the - // marked data: if this is cut short, the marker that proves the - // transaction is still in place for the next attempt. - let result = match self - .delete_unleased_with_namespace_owner(volume, &format!("{path}/{transaction}"), &opts, namespace_owner.clone()) - .await - { - Ok(()) | Err(DiskError::FileNotFound) => self - .finish_version_delete(volume, path, transaction, namespace_owner.clone()) - .await - .map(|_| ()), + let transaction_path = format!("{path}/{transaction}"); + let receipt = self + .io_get_object_path(volume, &transaction_path) + .map(|transaction_path| transaction_path.join(DELETE_QUORUM_COMMITTED_FILE)); + let result = match receipt { + Ok(receipt) => match fs::symlink_metadata(receipt).await { + Ok(metadata) if metadata.is_file() => { + match self + .finish_delete_transaction_with_namespace_owner(volume, path, transaction, namespace_owner.clone()) + .await + { + Ok(_) => { + self.delete_unleased_with_namespace_owner( + volume, + &transaction_path, + &DeleteOptions { + recursive: false, + ..opts.clone() + }, + namespace_owner.clone(), + None, + ) + .await + } + Err(err) => Err(err), + } + } + Ok(_) => Err(DiskError::FileCorrupt), + Err(err) if err.kind() == ErrorKind::NotFound => { + // Preserve the legacy marker-only cleanup policy without + // upgrading disk-local evidence into a quorum receipt. + match self + .delete_unleased_with_namespace_owner(volume, &transaction_path, &opts, namespace_owner.clone(), None) + .await + { + Ok(()) | Err(DiskError::FileNotFound) => self + .finish_version_delete(volume, path, transaction, false, namespace_owner.clone()) + .await + .map(|_| ()), + Err(err) => Err(err), + } + } + Err(err) => Err(to_file_error(err).into()), + }, Err(err) => Err(err), }; match result { @@ -12331,7 +13492,7 @@ impl LocalDisk { volume, path, transaction = %transaction, - operation = "finish_committed_delete_residue", + operation = "finish_internal_delete_residue", error = ?err, "Disk local delete failed" ), @@ -12393,7 +13554,22 @@ impl LocalDisk { Err(err) if err.kind() == ErrorKind::NotFound => continue, Err(err) => return Err(to_file_error(err).into()), }; + let mut has_quorum_receipt = false; + let mut safe_receipt_dir = true; while let Some(entry) = entries.next_entry().await.map_err(to_file_error)? { + let name = entry.file_name(); + let file_type = entry.file_type().await.map_err(to_file_error)?; + if !file_type.is_file() + || (name != STORAGE_FORMAT_FILE_BACKUP + && name != DELETE_MARKER_ROLLBACK_FILE + && name != DELETE_QUORUM_COMMITTED_FILE) + { + safe_receipt_dir = false; + } + if entry.file_name() == DELETE_QUORUM_COMMITTED_FILE { + has_quorum_receipt = file_type.is_file(); + continue; + } let Some(transaction) = entry .file_name() .to_str() @@ -12410,8 +13586,31 @@ impl LocalDisk { Err(err) => return Err(to_file_error(err).into()), } } + if has_quorum_receipt + && safe_receipt_dir + && let Some(transaction) = data_dir + .file_name() + .and_then(|name| name.to_str()) + .and_then(|name| Uuid::parse_str(name).ok()) + { + transactions.insert(transaction); + } } - Ok(transactions) + let mut legacy = BTreeSet::new(); + for transaction in transactions { + let rollback = object_path.join(transaction.to_string()); + let mut recovery_owned = false; + for name in [delete_recovery::PREPARED, delete_recovery::TARGET, delete_recovery::FINISHED] { + recovery_owned |= fs::try_exists(rollback.join(name)).await.map_err(to_file_error)?; + } + if let Ok(metadata) = fs::symlink_metadata(rollback.join(DELETE_QUORUM_COMMITTED_FILE)).await { + recovery_owned |= metadata.len() > 0; + } + if !recovery_owned { + legacy.insert(transaction); + } + } + Ok(legacy) } async fn finish_version_delete( @@ -12419,17 +13618,50 @@ impl LocalDisk { volume: &str, object: &str, rollback_dir: Uuid, + preserve_prefix_children: bool, namespace_owner: Option>, - ) -> Result { + ) -> Result<(bool, bool)> { + self.finish_version_delete_with_prepared_publication( + volume, + object, + rollback_dir, + preserve_prefix_children, + namespace_owner, + None, + ) + .await + } + + async fn finish_version_delete_with_prepared_publication( + &self, + volume: &str, + object: &str, + rollback_dir: Uuid, + preserve_prefix_children: bool, + namespace_owner: Option>, + publication: Option>, + ) -> Result<(bool, bool)> { let object_path = self.io_get_object_path(volume, object)?; let mut entries = match fs::read_dir(object_path).await { Ok(entries) => entries, - Err(err) if err.kind() == ErrorKind::NotFound => return Ok(false), + Err(err) if err.kind() == ErrorKind::NotFound => return Ok((false, true)), Err(err) => return Err(to_file_error(err).into()), }; - let marker_name = format!("{DELETE_DATA_DIR_MARKER_PREFIX}{rollback_dir}"); + let recovery_owned = preserve_prefix_children + && fs::try_exists( + self.io_get_object_path(volume, &format!("{object}/{rollback_dir}/{}", delete_recovery::PREPARED))?, + ) + .await + .map_err(to_file_error)?; + let prefix = if recovery_owned { + RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX + } else { + DELETE_DATA_DIR_MARKER_PREFIX + }; + let marker_name = format!("{prefix}{rollback_dir}"); let mut first_err = None; let mut found = false; + let mut deferred = false; while let Some(entry) = entries.next_entry().await.map_err(to_file_error)? { let Some(data_dir) = entry.file_name().to_str().and_then(|data_dir| Uuid::parse_str(data_dir).ok()) else { continue; @@ -12440,25 +13672,39 @@ impl LocalDisk { Err(err) if err.kind() == ErrorKind::NotFound => continue, Err(err) => return Err(to_file_error(err).into()), } - if let Err(err) = self - .delete_data_dir_with_namespace_owner( + let prepared = match publication.as_ref() { + Some(publication) => Some(PreparedPathDelete { + lease: os::try_acquire_namespace_mutation_lease_with_owner( + &self.io_get_object_path(volume, &format!("{object}/{data_dir}"))?, + publication.clone(), + )?, + trash: None, + encoded_trash: None, + }), + None => None, + }; + match self + .delete_data_dir_with_prepared_entry( volume, &format!("{object}/{data_dir}"), DeleteOptions { recursive: true, + old_data_dir: preserve_prefix_children.then_some(rollback_dir), ..Default::default() }, namespace_owner.clone(), + prepared, ) .await - && first_err.is_none() - && err != DiskError::FileNotFound - && err != DiskError::VolumeNotFound { - first_err = Some(err); + Ok(DataDirDeleteStatus::Deferred) => deferred = true, + Ok(DataDirDeleteStatus::Deleted) => {} + Err(err) if err == DiskError::FileNotFound || err == DiskError::VolumeNotFound => {} + Err(err) if first_err.is_none() => first_err = Some(err), + Err(_) => {} } } - first_err.map_or(Ok(found), Err) + first_err.map_or(Ok((found, !deferred)), Err) } async fn abort_reserved_version_delete_with_failure( @@ -12517,7 +13763,7 @@ impl LocalDisk { self.delete_path_prepared(&claim.key.volume, &claim.key.path, opts, owner.clone()) .await } else { - self.delete_unleased_with_namespace_owner(&claim.key.volume, &claim.key.path, &opts, None) + self.delete_unleased_with_namespace_owner(&claim.key.volume, &claim.key.path, &opts, None, None) .await }; if let Err(err) = &result { @@ -12579,6 +13825,79 @@ pub(crate) async fn batch_shard_pread(requests: Vec<(std::path::PathBuf, usize, #[async_trait::async_trait] impl DiskAPI for LocalDisk { + async fn supports_guarded_delete_versions(&self) -> Result<()> { + Ok(()) + } + async fn delete_versions_guarded_recoverable( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + ) -> Vec> { + self.delete_versions_guarded_with_namespace_owner(volume, versions, opts, None) + .await + } + async fn delete_version_recoverable( + &self, + volume: &str, + object: &str, + file_info: FileInfo, + force_del_marker: bool, + mut opts: DeleteOptions, + ) -> Result<()> { + super::validate_delete_recovery_options(&opts)?; + if object.starts_with('/') { + return Err(DiskError::InvalidPath); + } + opts.recoverable_delete = true; + self.delete_version(volume, object, file_info, force_del_marker, opts).await + } + async fn delete_versions_recoverable( + &self, + volume: &str, + versions: Vec, + mut opts: DeleteOptions, + ) -> Vec> { + if let Err(err) = super::validate_delete_recovery_options(&opts) { + return vec![Some(err); versions.len()]; + } + if versions.iter().any(|version| version.name.starts_with('/')) { + return vec![Some(DiskError::InvalidPath); versions.len()]; + } + opts.recoverable_delete = true; + self.delete_versions(volume, versions, opts).await + } + async fn has_delete_transaction_evidence(&self, volume: &str, object: &str) -> Result { + self.delete_recovery_has_evidence(volume, object).await + } + async fn discover_delete_transactions( + &self, + volume: &str, + object: Option<&str>, + cursor: Option, + limit: u32, + ) -> Result { + self.delete_recovery_discover(volume, object, cursor, limit).await + } + async fn inspect_delete_transaction( + &self, + volume: &str, + object: &str, + transaction: Uuid, + ) -> Result { + self.delete_recovery_inspect(volume, object, transaction).await + } + async fn settle_delete_transaction( + &self, + volume: &str, + object: &str, + transaction: Uuid, + expected_fingerprint: super::DeleteRecoveryDigest, + action: super::DeleteRecoveryAction, + ) -> Result { + self.delete_recovery_settle(volume, object, transaction, expected_fingerprint, action, None) + .await + } fn to_string(&self) -> String { self.root.to_string_lossy().to_string() } @@ -12729,9 +14048,9 @@ impl DiskAPI for LocalDisk { } } - let p = self.io_get_object_path(volume, path)?; + let p = self.io_get_object_open_path(volume, path)?; - let (data, _) = read_file_all(&p).await?; + let (data, _) = read_file_all(&p, &self.publication_root).await?; Ok(data) } @@ -13103,6 +14422,8 @@ impl DiskAPI for LocalDisk { // TODO(backlog): support io.writer cancellation and early termination in walk_dir #[tracing::instrument(level = "trace", skip_all)] async fn walk_dir(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> { + #[cfg(test)] + super::lifecycle_test_io::observe(&opts.bucket, None); self.wait_for_startup_cleanup().await; // Callers that do not pin a stall budget inherit the configured default, so @@ -13452,8 +14773,17 @@ impl DiskAPI for LocalDisk { entry.deleting = true; opts }; + // Release the reader token even under saturation. A rejected reclaim + // retains pending_delete (and its on-disk receipt) for a later retry. + // Do not wait for execution while a releasing reader may own locks. let claim = SnapshotLeaseDeleteClaim::new(&self.snapshot_leases, key); - let result = self.delete_path_prepared(volume, path, opts, owner.clone()).await; + let result = match super::cleanup_runtime::try_disk_execution() { + Ok(permit) => { + let _permit = permit; + self.delete_path_prepared(volume, path, opts, owner.clone()).await + } + Err(err) => Err(err.into()), + }; claim.finish(result.is_ok()); result } @@ -13656,6 +14986,8 @@ impl DiskAPI for LocalDisk { #[tracing::instrument(level = "trace", skip_all)] async fn read_xl(&self, volume: &str, path: &str, read_data: bool) -> Result { + #[cfg(test)] + super::lifecycle_test_io::observe(volume, Some(path)); crate::hp_guard!("LocalDisk::read_xl"); let file_path = self.io_get_object_path(volume, path)?; let file_dir = self.io_get_bucket_path(volume)?; @@ -13707,7 +15039,7 @@ impl DiskAPI for LocalDisk { }; // if req.metadata_only {} - match read_file_all(&fpath).await { + match read_file_all(&fpath, &self.publication_root).await { Ok((data, meta)) => { found += 1; @@ -13940,13 +15272,28 @@ async fn wait_for_startup_cleanup_signal( } #[tracing::instrument(level = "trace", skip_all)] +/// The disk root stays non-verbatim for `/`-joined IO, but the Win32 volume +/// APIs behind `rustfs_utils::os` only accept a path past the legacy limit in +/// extended-length form. Shorter roots are passed unchanged so the volume +/// identities those APIs report keep their conventional `C:\` form. +fn win32_volume_path(path: &Path) -> std::io::Result> { + // CreateDirectoryW's 248-unit limit, the lowest of the legacy path limits. + const LEGACY_MAX_PATH: usize = 248; + if path.to_string_lossy().encode_utf16().count() < LEGACY_MAX_PATH { + return Ok(std::borrow::Cow::Borrowed(path)); + } + rustfs_utils::extended_length_path(path) +} + async fn get_disk_info(drive_path: PathBuf) -> Result<(rustfs_utils::os::DiskInfo, bool)> { let drive_path = drive_path.to_string_lossy().to_string(); check_path_length(&drive_path)?; - let disk_info = get_info(&drive_path).inspect_err(|err| { - log_startup_disk_io_error("get_disk_info_stat", Path::new(&drive_path), err); - })?; + let disk_info = win32_volume_path(Path::new(&drive_path)) + .and_then(|volume_path| get_info(&volume_path)) + .inspect_err(|err| { + log_startup_disk_io_error("get_disk_info_stat", Path::new(&drive_path), err); + })?; let root_drive = if let Some(root_disk_threshold) = runtime_sources::root_disk_threshold_for_erasure_disk().await { if root_disk_threshold > 0 { disk_info.total <= root_disk_threshold @@ -14053,7 +15400,7 @@ mod test { } } - fn test_file_info(name: &str, version_id: Uuid, data_dir: Option, data: Option) -> FileInfo { + pub(super) fn test_file_info(name: &str, version_id: Uuid, data_dir: Option, data: Option) -> FileInfo { let size = data .as_ref() .map(|data| i64::try_from(data.len()).expect("test data length should fit i64")) @@ -14115,7 +15462,367 @@ mod test { assert!(!should_read_legacy_inline_part(&ec12, &config)); } - fn test_meta(fi: FileInfo) -> Vec { + fn guarded_delete_group(fi: FileInfo, expected_current: CurrentDeleteGeneration) -> GuardedFileInfoVersions { + GuardedFileInfoVersions { + group: FileInfoVersions { + name: fi.name.clone(), + versions: vec![fi], + ..Default::default() + }, + expected_current, + } + } + + #[tokio::test] + async fn guarded_delete_queued_before_replacement_rejects_stale_generation() { + let dir = tempfile::tempdir().expect("create disk directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse disk endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + ensure_test_volume(&disk, "bucket").await; + let object = "guarded-object"; + let object_dir = dir.path().join("bucket").join(object); + fs::create_dir_all(&object_dir).await.expect("create object directory"); + let mut original = test_file_info(object, Uuid::nil(), Some(Uuid::new_v4()), Some(Bytes::from_static(b"original"))); + original.metadata.insert("etag".into(), "etag-a".into()); + let expected = CurrentDeleteGeneration::from_file_info(&original).expect("capture original generation"); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), test_meta(original.clone())) + .await + .expect("write original metadata"); + let transaction = Uuid::new_v4(); + let owner = + os::acquire_metadata_mutation_lease(&disk.get_object_path("bucket", object).expect("object path"), None).await; + let mut deletion = Box::pin(disk.delete_versions_guarded_recoverable( + "bucket", + vec![guarded_delete_group(original.clone(), expected)], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + )); + assert!( + tokio::time::timeout(Duration::from_millis(50), deletion.as_mut()) + .await + .is_err(), + "guarded delete waits for metadata ownership" + ); + let mut replacement = original; + replacement.data_dir = Some(Uuid::new_v4()); + replacement.mod_time = replacement.mod_time.map(|time| time + time::Duration::seconds(1)); + replacement.metadata.insert("etag".into(), "etag-b".into()); + let newer = test_meta(replacement); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &newer) + .await + .expect("writer commits replacement while owning metadata"); + drop(owner); + assert_eq!(deletion.await, vec![Some(DiskError::ConditionalDeleteConflict)]); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("replacement metadata remains"), + newer + ); + assert!( + !object_dir.join(transaction.to_string()).exists(), + "rejected generation creates no recovery record" + ); + } + + #[tokio::test] + async fn guarded_delete_remote_only_matching_generation_accepts_missing_and_nil_data_directory() { + let dir = tempfile::tempdir().expect("create disk directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse disk endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + ensure_test_volume(&disk, "bucket").await; + for (case, data_dir, remote_version, remote_version_id) in [ + ("absent", None, None, None), + ("empty", Some(Uuid::nil()), Some(String::new()), None), + ("nil-version", None, None, Some(Uuid::nil())), + ] { + let object = format!("guarded-remote-only-{case}"); + let object_dir = dir.path().join("bucket").join(&object); + fs::create_dir_all(&object_dir) + .await + .expect("create remote-only object directory"); + let mut original = test_file_info(&object, Uuid::nil(), data_dir, None); + original.metadata.insert("etag".into(), "etag-remote".into()); + original.transition_status = rustfs_filemeta::TRANSITION_COMPLETE.to_owned(); + original.transition_tier = "WARM".into(); + original.transitioned_objname = "remote/object".into(); + original.transition_version = remote_version; + original.transition_version_id = remote_version_id; + let expected = CurrentDeleteGeneration::from_file_info(&original).expect("capture legacy remote-only generation"); + let before = test_meta(original.clone()); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &before) + .await + .expect("write remote-only metadata"); + let persisted = FileMeta::load(&before) + .expect("decode remote-only metadata") + .into_fileinfo("bucket", &object, "", false, false, false) + .expect("read remote-only generation"); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&persisted).expect("capture persisted remote generation"), + expected + ); + // Observed transition state belongs to the guard; a version-removal + // descriptor must not request another transition commit. + let mut deletion = FileInfo { + name: original.name, + version_id: original.version_id, + mod_time: original.mod_time, + ..Default::default() + }; + let free_version_id = Uuid::new_v4(); + deletion.set_tier_free_version_id(&free_version_id.to_string()); + let transaction = Uuid::new_v4(); + assert_eq!( + disk.delete_versions_guarded_recoverable( + "bucket", + vec![guarded_delete_group(deletion, expected)], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await, + vec![None], + "matching remote-only guard succeeds: {case}" + ); + let after = fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("durable tier cleanup metadata"); + let remaining = FileMeta::load(&after).expect("decode tier cleanup metadata"); + assert_eq!(remaining.versions.len(), 1, "only the tier cleanup owner remains: {case}"); + let listed = remaining + .get_file_info_versions("bucket", &object, false) + .expect("list live and free versions"); + assert!(listed.versions.is_empty(), "matching remote-only object version is removed: {case}"); + assert_eq!( + listed.free_versions.len(), + 1, + "deletion retains exactly one remote cleanup record: {case}" + ); + let cleanup = &listed.free_versions[0]; + assert!( + cleanup.deleted && cleanup.tier_free_version(), + "remaining metadata is a canonical tier cleanup marker: {case}" + ); + assert_eq!(cleanup.version_id, Some(free_version_id)); + assert_eq!(cleanup.transition_tier, "WARM"); + assert_eq!(cleanup.transitioned_objname, "remote/object"); + assert_eq!(cleanup.transition_version, None); + assert_eq!(cleanup.transition_version_id, None); + assert_eq!(cleanup.transition_version_state, rustfs_filemeta::TransitionVersionState::Unknown); + assert!( + matches!( + remaining.into_fileinfo("bucket", &object, "", false, false, false), + Err(rustfs_filemeta::Error::FileNotFound) + ), + "normal object reads no longer find the deleted version: {case}" + ); + assert_eq!( + fs::read(object_dir.join(transaction.to_string()).join(STORAGE_FORMAT_FILE_BACKUP)) + .await + .expect("recoverable metadata backup"), + before + ); + } + } + + #[tokio::test] + async fn guarded_delete_remote_tuple_changes_preserve_metadata_data_and_recovery_state() { + use rustfs_utils::http::metadata_compat::{SUFFIX_TRANSITION_TIER_DESTINATION_ID, insert_str}; + let dir = tempfile::tempdir().expect("create disk directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse disk endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + ensure_test_volume(&disk, "bucket").await; + for changed_field in [ + "backend", + "tier", + "object", + "version", + "version-state", + "status", + "local-to-remote", + ] { + let object = format!("guarded-remote-{changed_field}"); + let data_dir = Uuid::new_v4(); + let object_dir = dir.path().join("bucket").join(&object); + let data_path = object_dir.join(data_dir.to_string()).join("part.1"); + fs::create_dir_all(data_path.parent().expect("data directory")) + .await + .expect("create data directory"); + fs::write(&data_path, b"preserved-payload") + .await + .expect("write local data that must survive"); + let mut original = test_file_info(&object, Uuid::nil(), Some(data_dir), None); + original.size = 17; + original.parts[0].size = 17; + original.parts[0].actual_size = 17; + original.metadata.insert("etag".into(), "etag-same".into()); + original.transition_status = rustfs_filemeta::TRANSITION_COMPLETE.to_owned(); + original.transition_tier = "WARM".into(); + original.transitioned_objname = "remote/object-a".into(); + original.transition_version = Some("opaque/version-a==".into()); + original.transition_version_state = rustfs_filemeta::TransitionVersionState::Exact; + insert_str(&mut original.metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, "ab".repeat(32)); + let mut current = original.clone(); + match changed_field { + "backend" => insert_str(&mut current.metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, "cd".repeat(32)), + "tier" => current.transition_tier = "COLD".into(), + "object" => current.transitioned_objname = "remote/object-b".into(), + "version" => current.transition_version = Some("opaque/version-b==".into()), + "version-state" => current.transition_version_state = rustfs_filemeta::TransitionVersionState::Unknown, + "status" => current.transition_status = "pending".into(), + "local-to-remote" => { + original.transition_status.clear(); + original.transition_tier.clear(); + original.transitioned_objname.clear(); + original.transition_version = None; + original.transition_version_state = rustfs_filemeta::TransitionVersionState::Unknown; + rustfs_utils::http::metadata_compat::remove_str( + &mut original.metadata, + SUFFIX_TRANSITION_TIER_DESTINATION_ID, + ); + } + _ => unreachable!("test controls changed fields"), + } + let expected = CurrentDeleteGeneration::from_file_info(&original).expect("capture original remote generation"); + let actual = CurrentDeleteGeneration::from_file_info(¤t).expect("capture changed remote generation"); + assert_ne!(actual, expected, "remote identity change must alter the guard: {changed_field}"); + assert_eq!(current.version_id, original.version_id); + assert_eq!(current.data_dir, original.data_dir); + assert_eq!(current.mod_time, original.mod_time); + assert_eq!(current.get_etag(), original.get_etag()); + let current_metadata = test_meta(current); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), ¤t_metadata) + .await + .expect("publish changed remote identity"); + let transaction = Uuid::new_v4(); + let errors = disk + .delete_versions_guarded_recoverable( + "bucket", + vec![guarded_delete_group(original, expected)], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await; + assert_eq!( + errors, + vec![Some(DiskError::ConditionalDeleteConflict)], + "stale remote tuple is rejected: {changed_field}" + ); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("current metadata remains"), + current_metadata + ); + assert_eq!(fs::read(&data_path).await.expect("payload remains"), b"preserved-payload"); + assert!( + !object_dir.join(transaction.to_string()).exists(), + "stale remote tuple creates no recovery artifacts: {changed_field}" + ); + } + } + + #[tokio::test] + async fn guarded_delete_absence_guard_preserves_new_object() { + let dir = tempfile::tempdir().expect("create disk directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse disk endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + ensure_test_volume(&disk, "bucket").await; + let object = "guarded-object"; + let object_dir = dir.path().join("bucket").join(object); + fs::create_dir_all(&object_dir).await.expect("create object directory"); + let mut current = test_file_info(object, Uuid::nil(), Some(Uuid::new_v4()), Some(Bytes::from_static(b"current"))); + current.metadata.insert("etag".into(), "etag-current".into()); + let newer = test_meta(current.clone()); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &newer) + .await + .expect("write current metadata"); + let transaction = Uuid::new_v4(); + let errors = disk + .delete_versions_guarded_recoverable( + "bucket", + vec![guarded_delete_group(current, CurrentDeleteGeneration::Absent)], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await; + assert_eq!(errors, vec![Some(DiskError::ConditionalDeleteConflict)]); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("current metadata remains"), + newer + ); + assert!(!object_dir.join(transaction.to_string()).exists()); + } + + #[tokio::test] + async fn guarded_delete_recovery_rollback_preserves_replacement() { + let dir = tempfile::tempdir().expect("create disk directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse disk endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + ensure_test_volume(&disk, "bucket").await; + let object = "guarded-object"; + let object_dir = dir.path().join("bucket").join(object); + fs::create_dir_all(&object_dir).await.expect("create object directory"); + let mut original = test_file_info(object, Uuid::nil(), Some(Uuid::new_v4()), Some(Bytes::from_static(b"original"))); + original.metadata.insert("etag".into(), "etag-a".into()); + let expected = CurrentDeleteGeneration::from_file_info(&original).expect("capture original generation"); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), test_meta(original.clone())) + .await + .expect("write original metadata"); + let transaction = Uuid::new_v4(); + assert_eq!( + disk.delete_versions_guarded_recoverable( + "bucket", + vec![guarded_delete_group(original.clone(), expected)], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await, + vec![None] + ); + let applied = disk + .inspect_delete_transaction("bucket", object, transaction) + .await + .expect("inspect prepared delete"); + let mut replacement = original; + replacement.data_dir = Some(Uuid::new_v4()); + replacement.mod_time = replacement.mod_time.map(|time| time + time::Duration::seconds(1)); + replacement.metadata.insert("etag".into(), "etag-b".into()); + let newer = test_meta(replacement); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &newer) + .await + .expect("commit replacement after delete preparation"); + let error = disk + .settle_delete_transaction( + "bucket", + object, + transaction, + applied.fingerprint, + super::super::DeleteRecoveryAction::Rollback, + ) + .await + .expect_err("stale rollback cannot overwrite replacement"); + assert_eq!(error, DiskError::OutdatedXLMeta); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("replacement remains after rejected rollback"), + newer + ); + } + + pub(super) fn test_meta(fi: FileInfo) -> Vec { let mut meta = FileMeta::default(); meta.add_version(fi).expect("test metadata should accept file info"); meta.marshal_msg().expect("test metadata should encode") @@ -14379,7 +16086,7 @@ mod test { assert!(!child_path.exists(), "the exact locked directory should be removed"); } - async fn ensure_test_volume(disk: &LocalDisk, volume: &str) { + pub(super) async fn ensure_test_volume(disk: &LocalDisk, volume: &str) { match disk.make_volume(volume).await { Ok(()) | Err(DiskError::VolumeExists) => {} Err(err) => panic!("test volume should be available: {err:?}"), @@ -15533,10 +17240,15 @@ mod test { disk.wait_for_startup_cleanup().await; assert_eq!(disk.startup_cleanup_ready.load(Ordering::Acquire), 1); - LocalDisk::cleanup_stale_tmp_objects_with_expiry(dir.path().join("missing-root"), &publication_root, Duration::ZERO) - .await - .expect("missing tmp path should be a cleanup no-op"); - LocalDisk::cleanup_deleted_objects(dir.path().join("missing-root")) + LocalDisk::cleanup_stale_tmp_objects_with_expiry( + dir.path().join("missing-root"), + &publication_root, + Duration::ZERO, + &crate::disk::cleanup_runtime::GcBudget::default(), + ) + .await + .expect("missing tmp path should be a cleanup no-op"); + LocalDisk::cleanup_deleted_objects(dir.path().join("missing-root"), &crate::disk::cleanup_runtime::GcBudget::default()) .await .expect("missing trash path should be a cleanup no-op"); @@ -15549,9 +17261,14 @@ mod test { fs::create_dir_all(&trash_root).await.expect("trash dir should be created"); backdate_mtime(&stale_dir, Duration::from_secs(10)); - LocalDisk::cleanup_stale_tmp_objects_with_expiry(dir.path().to_path_buf(), &publication_root, Duration::ZERO) - .await - .expect("stale tmp directory should move to trash"); + LocalDisk::cleanup_stale_tmp_objects_with_expiry( + dir.path().to_path_buf(), + &publication_root, + Duration::ZERO, + &crate::disk::cleanup_runtime::GcBudget::default(), + ) + .await + .expect("stale tmp directory should move to trash"); assert!(!stale_dir.exists(), "stale tmp directory should be moved away"); assert!(live_file.exists(), "plain tmp files should be ignored by stale dir cleanup"); @@ -15561,7 +17278,7 @@ mod test { fs::create_dir_all(trash_root.join("trash-dir")) .await .expect("trash dir should be created"); - LocalDisk::cleanup_deleted_objects(dir.path().to_path_buf()) + LocalDisk::cleanup_deleted_objects(dir.path().to_path_buf(), &crate::disk::cleanup_runtime::GcBudget::default()) .await .expect("trash cleanup should remove files and directories"); assert!( @@ -20335,9 +22052,10 @@ mod test { fs::create_dir_all(object.join(rollback.to_string()).join(STORAGE_FORMAT_FILE_BACKUP)) .await .expect("backup persist obstacle"); - let expected = write_delete_rollback_file(&object, rollback, STORAGE_FORMAT_FILE_BACKUP, &original, None) - .await - .expect_err("actual backup syscall error"); + let expected = + write_delete_rollback_file(disk.io_root(), volume, &object, rollback, STORAGE_FORMAT_FILE_BACKUP, &original, None) + .await + .expect_err("actual backup syscall error"); let wrapper = LocalDiskWrapper::new(disk.clone(), false); let errors = wrapper .begin_legacy_file_write_with_namespace_owner(volume, None) @@ -20646,7 +22364,7 @@ mod test { for volume in [volume, RUSTFS_META_TMP_BUCKET, RUSTFS_META_TMP_DELETED_BUCKET] { ensure_test_volume(&disk, volume).await; } - let (mut group, rollback, original) = if bulk { + let (group, rollback, original) = if bulk { prepare_bulk_delete_fixture(&disk, volume, &object, case == "marker").await } else { let (fi, rollback, original) = prepare_legacy_delete_fixture(&disk, volume, &object, case).await; @@ -20661,14 +22379,6 @@ mod test { ) }; let fi = group.versions[0].clone(); - if bulk && case == "after-reserve" { - let mut invalid = fi.clone(); - invalid.parts[0].number = 0; - invalid - .validate_for_delete_operation() - .expect_err("later metadata operation must fail"); - group.versions.push(invalid); - } let object_dir = disk.io_get_object_path(volume, &object).expect("object path"); let metadata = object_dir.join(STORAGE_FORMAT_FILE); let rollback_path = object_dir.join(rollback.to_string()); @@ -20824,6 +22534,22 @@ mod test { .expect("missing marker intent"); fs::remove_file(&metadata).await.expect("missing old metadata"); } + let marker_paths: Vec<_> = fi + .data_dir + .into_iter() + .flat_map(|data| { + let data_path = object_dir.join(data.to_string()); + [RESERVED_DELETE_DATA_DIR_MARKER_PREFIX, DELETE_DATA_DIR_MARKER_PREFIX] + .map(|prefix| data_path.join(format!("{prefix}{rollback}"))) + }) + .collect(); + let marker_bytes: Vec<_> = marker_paths.iter().map(|path| std::fs::read(path).ok()).collect(); + if case == "no-backup" { + assert!( + marker_bytes.iter().any(Option::is_some), + "missing-backup fixture retains a reservation proof" + ); + } let opts = DeleteOptions { undo_write: true, undo_delete: true, @@ -20864,8 +22590,12 @@ mod test { fs::read(data_path.join("part.1")).await.expect("old shard restored"), b"old-complete-shard" ); - for prefix in [RESERVED_DELETE_DATA_DIR_MARKER_PREFIX, DELETE_DATA_DIR_MARKER_PREFIX] { - assert!(!data_path.join(format!("{prefix}{rollback}")).exists()); + for (marker, before) in marker_paths.iter().zip(&marker_bytes) { + if matches!(case, "no-backup" | "staged-no-backup") { + assert_eq!(&std::fs::read(marker).ok(), before, "failed restore retains existing marker evidence"); + } else { + assert!(!marker.exists(), "successful restore removes its marker"); + } } } assert_eq!(wrapper.metrics_snapshot().api_calls.get("delete_version"), Some(&1)); @@ -23847,6 +25577,306 @@ mod test { } } + #[cfg(unix)] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn cancelled_tmp_parent_pruning_retains_staging_lease_in_executor() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + + let root = tempfile::tempdir().expect("cancelled parent pruning fixture"); + let endpoint = Endpoint::try_from(root.path().to_str().expect("UTF-8 fixture path")).expect("endpoint"); + let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk")); + ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await; + disk.write_all(RUSTFS_META_TMP_BUCKET, "stage/child/payload", Bytes::from_static(b"old staging bytes")) + .await + .expect("seed nested staging prefix"); + let parent = disk + .io_get_object_path(RUSTFS_META_TMP_BUCKET, "stage") + .expect("staging parent path"); + let staging_key = tmp_staging_mutation_path(&disk.root, "stage").expect("shared staging key"); + let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); + let _hook = hooks::install_at(hooks::Stage::Remove, &parent, move || { + let _ = entered_tx.send(()); + let _ = release_rx.recv_timeout(Duration::from_secs(30)); + }); + let cleanup_disk = disk.clone(); + let mut cleanup = tokio::spawn(async move { + cleanup_disk + .delete( + RUSTFS_META_TMP_BUCKET, + "stage/child", + DeleteOptions { + recursive: true, + ..Default::default() + }, + ) + .await + }); + tokio::time::timeout(Duration::from_secs(10), async { + tokio::select! { + entered = entered_rx => entered.expect("physical parent rmdir pause"), + result = &mut cleanup => panic!("cleanup returned before parent pruning: {result:?}"), + } + }) + .await + .expect("cleanup must reach its already-leased parent"); + cleanup.abort(); + assert!(cleanup.await.expect_err("cleanup waiter cancelled").is_cancelled()); + let mut next_owner = + tokio::spawn(async move { os::acquire_namespace_mutation_lease_with_owner(&staging_key, None).await }); + assert!( + tokio::time::timeout(Duration::from_secs(1), &mut next_owner).await.is_err(), + "the physical rmdir must retain the staging lease after cancellation" + ); + drop(release_tx); + let _next_owner = tokio::time::timeout(Duration::from_secs(10), next_owner) + .await + .expect("staging lease released after physical rmdir") + .expect("next staging owner"); + assert!(!parent.exists(), "the empty staging parent must actually be pruned"); + assert!(disk.io_get_bucket_path(RUSTFS_META_TMP_BUCKET).expect("TMP path").is_dir()); + } + + #[test] + fn tmp_staging_keys_share_top_level_prefix_and_reject_escape() { + let root = tempfile::tempdir().expect("staging key fixture"); + let expected = root.path().join(RUSTFS_META_TMP_BUCKET).join("stage"); + for path in ["stage", "stage/child", "stage/./child", "stage/child/../next"] { + assert_eq!(tmp_staging_mutation_path(root.path(), path).expect("contained staging key"), expected); + } + assert!(matches!(tmp_staging_mutation_path(root.path(), "../escape"), Err(DiskError::InvalidPath))); + } + + #[cfg(unix)] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial_test::serial(dst_dir_fsync_group_commit)] + async fn inline_heal_owned_receiver_restores_metadata_when_tmp_cleanup_races_rollback() { + use crate::disk::{disk_store::LocalDiskWrapper, os::prepared_publication_test_hooks as hooks}; + + let _group_commit = os::set_dst_dir_fsync_group_commit_for_test(true); + let _mode = durability_mode_override::set(DurabilityMode::Strict); + for stale_gc in [false, true] { + let root = tempfile::tempdir().expect("owned receiver fixture directory"); + let endpoint = Endpoint::try_from(root.path().to_str().expect("UTF-8 fixture path")).expect("endpoint"); + let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk")); + let bucket = "inline-heal-owned-receiver"; + let object = "inline-heal-owned-receiver-tmp-cleanup-rollback-object"; + let staging = "inline-heal-owned-receiver-tmp-cleanup-stage"; + ensure_test_volume(&disk, bucket).await; + ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await; + let object_dir = disk.io_get_object_path(bucket, object).expect("physical object path"); + let metadata_path = object_dir.join(STORAGE_FORMAT_FILE); + fs::create_dir_all(&object_dir).await.expect("object directory"); + let version = Uuid::new_v4(); + let mut old = test_file_info(object, version, None, Some(Bytes::from_static(b"old owned receiver payload"))); + old.set_inline_data(); + let old_metadata = test_meta(old); + fs::write(&metadata_path, &old_metadata) + .await + .expect("seed acknowledged metadata"); + let staging_dir = disk + .io_get_object_path(RUSTFS_META_TMP_BUCKET, staging) + .expect("physical staging prefix"); + let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); + // Group fsync hooks use canonical paths, including on Linux mount-FD IO paths. + let fsync_hook_path = object_dir.canonicalize().expect("canonical group fsync path"); + let _hook = hooks::install_at(hooks::Stage::DirFsync, &fsync_hook_path, move || { + let _ = entered_tx.send(()); + // Dropping the sender also releases this executor when an assertion fails. + let _ = release_rx.recv_timeout(Duration::from_secs(30)); + }); + os::fsync_dir_recorder::set_grouped_failure(&object_dir, io::ErrorKind::PermissionDenied); + let mut repaired = test_file_info(object, version, None, Some(Bytes::from_static(b"new owned receiver payload"))); + repaired.set_inline_data(); + repaired.set_healing(); + let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + let owner = ctx.begin_namespace_commit(); + let owner_probe = Arc::downgrade(&owner); + let wrapper = LocalDiskWrapper::new(Arc::clone(&disk), false); + let mut receiver = tokio::spawn(async move { + wrapper + .rename_data_observed_with_guards( + RUSTFS_META_TMP_BUCKET, + staging, + &repaired, + bucket, + object, + crate::disk::RenameDataGuards { + external_guard: Some(owner), + ..Default::default() + }, + ) + .await + }); + tokio::time::timeout(Duration::from_secs(10), async { + tokio::select! { + entered = entered_rx => entered.expect("physical fsync pause"), + result = &mut receiver => match result { + Ok(observation) => panic!("receiver completed before the fsync pause; error: {:?}", observation.result.err()), + Err(error) => panic!("receiver task failed before the fsync pause: {error}"), + }, + } + }) + .await + .expect("owned receiver must reach post-publication fsync"); + assert_ne!(fs::read(&metadata_path).await.expect("published repaired metadata"), old_metadata); + assert_eq!( + fs::read(staging_dir.join(STORAGE_FORMAT_FILE_BACKUP)) + .await + .expect("active staging rollback backup"), + old_metadata + ); + // An incarnation-bound receiver drains even after its RPC waiter disappears. + receiver.abort(); + assert!(receiver.await.err().expect("RPC waiter cancellation").is_cancelled()); + assert!(ctx.namespace_commits_pending(), "the physical receiver must remain active"); + let cleanup_disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("reconnected disk handle")); + if stale_gc { + std::fs::File::open(&staging_dir) + .expect("staging directory") + .set_times(std::fs::FileTimes::new().set_modified(std::time::SystemTime::UNIX_EPOCH)) + .expect("make the active prefix eligible for stale GC"); + } + let mut cleanup = tokio::spawn(async move { + if stale_gc { + let mut entries = fs::read_dir(cleanup_disk.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?).await?; + while let Some(entry) = entries.next_entry().await? { + if entry.file_name() == staging { + return LocalDisk::cleanup_stale_tmp_entry( + entry, + cleanup_disk.io_root(), + &cleanup_disk.publication_root, + Duration::from_secs(1), + ) + .await; + } + } + panic!("active staging prefix must be discoverable by GC"); + } + cleanup_disk + .delete( + RUSTFS_META_TMP_BUCKET, + staging, + DeleteOptions { + recursive: true, + ..Default::default() + }, + ) + .await + }); + assert!( + tokio::time::timeout(Duration::from_secs(1), &mut cleanup).await.is_err(), + "request cleanup and stale GC must wait for the active receiver" + ); + drop(release_tx); + tokio::time::timeout(Duration::from_secs(10), async { + while owner_probe.upgrade().is_some() || ctx.namespace_commits_pending() { + tokio::task::yield_now().await; + } + }) + .await + .expect("physical receiver must finish its failed-fsync rollback"); + tokio::time::timeout(Duration::from_secs(10), cleanup) + .await + .expect("tmp cleanup must finish after physical rollback") + .expect("tmp cleanup task") + .expect("deferred tmp cleanup"); + assert_eq!( + fs::read(&metadata_path).await.expect("metadata after failed-fsync rollback"), + old_metadata, + "temporary-prefix cleanup must not destroy an active receiver's rollback source" + ); + } + } + + #[cfg(unix)] + #[tokio::test] + #[serial_test::serial(dst_dir_fsync_group_commit)] + async fn inline_heal_staged_backup_restores_metadata_after_commit_failure() { + let _group_commit = os::set_dst_dir_fsync_group_commit_for_test(true); + for mode in [DurabilityMode::None, DurabilityMode::Relaxed, DurabilityMode::Strict] { + let _mode = durability_mode_override::set(mode); + for version in [Uuid::nil(), Uuid::new_v4()] { + for old_inline in [true, false] { + let root = tempfile::tempdir().expect("rollback fixture directory"); + let endpoint = Endpoint::try_from(root.path().to_str().expect("UTF-8 fixture path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = "inline-heal-rollback"; + let object = "inline-heal-staged-backup-rollback-object"; + ensure_test_volume(&disk, bucket).await; + ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await; + let object_dir = disk.io_get_object_path(bucket, object).expect("physical object path"); + fs::create_dir_all(&object_dir).await.expect("object directory"); + let old_dir = Uuid::new_v4(); + let old = if old_inline { + test_file_info(object, version, None, Some(Bytes::from_static(b"old inline bytes"))) + } else { + fs::create_dir_all(object_dir.join(old_dir.to_string())) + .await + .expect("external directory"); + fs::write(object_dir.join(old_dir.to_string()).join("part.1"), b"x") + .await + .expect("old external shard"); + test_file_info(object, version, Some(old_dir), None) + }; + let old_metadata = test_meta(old); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &old_metadata) + .await + .expect("seed old metadata"); + if mode == DurabilityMode::Strict { + os::fsync_dir_recorder::set_grouped_failure(&object_dir, io::ErrorKind::PermissionDenied); + } else { + set_rename_data_fail_after_metadata_commit(object); + } + let mut new = test_file_info(object, version, None, Some(Bytes::from_static(b"repaired inline bytes"))); + new.set_healing(); + let error = disk + .rename_data(RUSTFS_META_TMP_BUCKET, "heal", new, bucket, object) + .await + .expect_err("injected post-publication failure"); + let expected_kind = if mode == DurabilityMode::Strict { + io::ErrorKind::PermissionDenied + } else { + io::ErrorKind::Other + }; + assert!( + matches!(error, DiskError::Io(ref error) if error.kind() == expected_kind), + "{mode:?}: {error:?}" + ); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("restored metadata"), + old_metadata + ); + if !old_inline { + assert_eq!( + fs::read(object_dir.join(old_dir.to_string()).join("part.1")) + .await + .expect("old shard"), + b"x" + ); + } + let staging = disk.io_get_object_path(RUSTFS_META_TMP_BUCKET, "heal").expect("staging path"); + assert!( + !staging.join(STORAGE_FORMAT_FILE_BACKUP).exists(), + "rollback must consume the staging backup" + ); + for entry in std::fs::read_dir(&object_dir).expect("object entries") { + let entry = entry.expect("object entry"); + if entry.file_type().expect("entry type").is_dir() { + assert!( + !entry.path().join(STORAGE_FORMAT_FILE_BACKUP).exists(), + "heal must not publish a live rollback backup" + ); + } + } + } + } + } + } + #[tokio::test] async fn windows_and_unix_rename_data_inline_post_commit_error_restores_old_metadata() { use tempfile::tempdir; @@ -24889,7 +26919,7 @@ mod test { .await .expect("existing metadata should be written"); - disk.delete_versions_internal(bucket, object, &[missing_fi, existing_fi], &DeleteOptions::default()) + disk.delete_versions_internal(bucket, object, &[missing_fi, existing_fi], &DeleteOptions::default(), None) .await .expect("missing non-deleted version should not abort deletion"); @@ -25046,6 +27076,7 @@ mod test { old_data_dir: Some(rollback_dir), ..Default::default() }, + None, ) .await .expect("missing inline data dir should not fail batch deletion"); @@ -25060,6 +27091,117 @@ mod test { assert!(!logs.contents().contains("reliable_rename failed")); } + // rustfs#8545: the rollback backup is staged through tempfile, whose Windows + // `persist` calls `MoveFileExW` with the raw path; past MAX_PATH every + // delete of such a key failed with 500 while PUT and GET still worked. + #[tokio::test(flavor = "current_thread")] + async fn windows_and_unix_delete_backs_up_metadata_beyond_max_path() { + use tempfile::tempdir; + + let dir = tempdir().expect("temp dir should be created"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created"); + let bucket = "bucket"; + ensure_test_volume(&disk, bucket).await; + + let deep_prefix = format!("dir/{}", "a".repeat(200)); + let version_id = Uuid::parse_str("37373737-1111-2222-3333-444444444444").expect("version id should parse"); + let data_dir = Uuid::parse_str("38383838-1111-2222-3333-444444444444").expect("data dir should parse"); + let rollback_dir = Uuid::parse_str("39393939-1111-2222-3333-444444444444").expect("rollback dir should parse"); + let opts = DeleteOptions { + old_data_dir: Some(rollback_dir), + ..Default::default() + }; + + for batch in [false, true] { + let object = format!("{deep_prefix}/{}", if batch { "batch" } else { "single" }); + let object_dir = dir.path().join(bucket).join(&object); + let backup = object_dir.join(rollback_dir.to_string()).join(STORAGE_FORMAT_FILE_BACKUP); + assert!( + backup.to_str().expect("temp dir should be utf8").encode_utf16().count() > 260, + "regression path must cross the traditional Windows MAX_PATH boundary: {backup:?}" + ); + fs::create_dir_all(&object_dir).await.expect("object dir should be created"); + let fi = test_file_info(&object, version_id, Some(data_dir), Some(Bytes::from_static(b"inline"))); + let meta = test_meta(fi.clone()); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &meta) + .await + .expect("inline metadata should be written"); + + if batch { + disk.delete_versions_internal(bucket, &object, &[fi], &opts, None) + .await + .expect("batch delete of a deep key should succeed"); + } else { + disk.delete_version(bucket, &object, fi, false, opts.clone()) + .await + .expect("delete of a deep key should succeed"); + } + + assert!(!object_dir.join(STORAGE_FORMAT_FILE).exists(), "{object}: metadata should be deleted"); + assert_eq!( + fs::read(&backup).await.expect("rollback backup should be published"), + meta, + "{object}: backup should hold the pre-delete metadata" + ); + } + } + + /// Opens a disk at `root` and drives metadata and data reads and writes + /// through it: they extend the root by joining `/`-separated relative paths + /// as strings, so they all depend on the root staying non-verbatim. + async fn assert_local_disk_serves_objects_at(root: &Path) { + let endpoint = Endpoint::try_from(root.to_str().expect("disk root should be utf8")).expect("endpoint should parse"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created"); + assert!( + !disk.root.to_string_lossy().starts_with(r"\\?\"), + "the disk root must not be verbatim: {:?}", + disk.root + ); + let bucket = "bucket"; + ensure_test_volume(&disk, bucket).await; + + let object = "dir/object"; + let version_id = Uuid::parse_str("41414141-1111-2222-3333-444444444444").expect("version id should parse"); + let data_dir = Uuid::parse_str("42424242-1111-2222-3333-444444444444").expect("data dir should parse"); + let fi = test_file_info(object, version_id, Some(data_dir), Some(Bytes::from_static(b"inline"))); + let meta = test_meta(fi); + disk.write_all(bucket, &format!("{object}/{STORAGE_FORMAT_FILE}"), Bytes::from(meta.clone())) + .await + .expect("metadata should be written below the disk root"); + let raw = disk + .read_xl(bucket, object, true) + .await + .expect("metadata should be read back"); + assert_eq!(raw.buf, meta); + + let payload = Bytes::from_static(b"part payload"); + disk.write_all(bucket, "dir/other/part.1", payload.clone()) + .await + .expect("data should be written below the disk root"); + assert_eq!( + disk.read_all(bucket, "dir/other/part.1") + .await + .expect("data should be read back"), + payload + ); + } + + // A disk root past MAX_PATH resolved to `\\?\C:\...`, where `/` is not a + // separator, so the first `/`-joined metadata read (pool.bin.identity) + // failed with ERROR_INVALID_NAME and startup never completed. + #[tokio::test] + async fn windows_and_unix_local_disk_root_beyond_max_path_serves_objects() { + let dir = tempfile::tempdir().expect("temp dir should be created"); + let mut root = dir.path().to_path_buf(); + while root.to_str().expect("temp dir should be utf8").encode_utf16().count() <= 270 { + root.push("r".repeat(100)); + } + fs::create_dir_all(&root).await.expect("long disk root should be created"); + + assert_local_disk_serves_objects_at(&root).await; + } + #[tokio::test(flavor = "current_thread")] async fn test_ignore_missing_source_helper_still_warns_on_real_error() { use tempfile::tempdir; @@ -25259,8 +27401,7 @@ mod test { } } - /// Leave the state a delete reaches once it committed on this disk but - /// before its cleanup pass: what a cancelled request used to strand. + /// A legacy disk-local mutation without a structured set decision. async fn write_committed_delete_residue(disk: &LocalDisk, bucket: &str, object: &str) -> CommittedDeleteResidue { let version_id = Uuid::new_v4(); let data_dir = Uuid::new_v4(); @@ -25300,6 +27441,56 @@ mod test { residue } + #[tokio::test] + async fn delete_recovery_user_bucket_apply_preserves_ambiguous_legacy_residue() { + for batch in [false, true] { + for recoverable in [false, true] { + let dir = tempfile::tempdir().expect("residue fixture directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 endpoint")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = "user-legacy-residue"; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + let original = fs::read(residue.backup()).await.expect("legacy metadata backup"); + let request = FileInfo { + name: object.to_owned(), + version_id: Some(Uuid::new_v4()), + ..Default::default() + }; + let opts = DeleteOptions { + old_data_dir: Some(Uuid::new_v4()), + recoverable_delete: recoverable, + delete_recovery_bucket_incarnation: Some(Uuid::new_v4()), + ..Default::default() + }; + if batch { + let errors = disk + .delete_versions( + bucket, + vec![FileInfoVersions { + name: object.to_owned(), + versions: vec![request], + ..Default::default() + }], + opts, + ) + .await; + assert!(matches!(errors.as_slice(), [Some(DiskError::FileNotFound)]), "{errors:?}"); + } else { + assert!(matches!( + disk.delete_version(bucket, object, request, false, opts).await, + Err(DiskError::FileVersionNotFound) + )); + } + assert_eq!(fs::read(residue.backup()).await.expect("preserved backup"), original); + assert_eq!(fs::read(residue.part()).await.expect("preserved shard"), b"stale"); + assert!(residue.marker().is_file(), "a missing local apply cannot certify historical deletion"); + assert_eq!(disk.finish_internal_delete_residue(bucket, object, None).await, 0); + } + } + } + fn residue_delete_request(object: &str) -> Vec { vec![FileInfoVersions { name: object.to_string(), @@ -25311,14 +27502,334 @@ mod test { }] } - // #6898: repeating the delete of a key whose earlier delete committed but - // never cleaned up reclaims the residue and still reports the key missing. + // This is a real process interruption of LocalDisk, not a server or + // power-loss test. The child stops at the rollback-backup unlink. + #[cfg(unix)] + #[tokio::test] + async fn quorum_delete_cleanup_process_crash_preserves_retry_evidence() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + use std::os::unix::process::ExitStatusExt; + + const CHILD_ROOT: &str = "RUSTFS_TEST_QUORUM_CLEANUP_ROOT"; + const CHILD_TRANSACTION: &str = "RUSTFS_TEST_QUORUM_CLEANUP_TRANSACTION"; + const CHILD_READY: &str = "RUSTFS_TEST_QUORUM_CLEANUP_READY"; + if let Some(root) = std::env::var_os(CHILD_ROOT) { + let endpoint = + Endpoint::try_from(Path::new(&root).to_str().expect("UTF-8 child disk path")).expect("parse child endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("open child disk"); + let transaction = + Uuid::parse_str(&std::env::var(CHILD_TRANSACTION).expect("child transaction")).expect("parse child transaction"); + let ready = std::env::var_os(CHILD_READY).expect("child readiness path"); + let rollback_path = format!("object/{transaction}"); + let backup = disk + .io_get_object_path("bucket", &rollback_path) + .expect("child transaction path") + .join(STORAGE_FORMAT_FILE_BACKUP); + let _hook = hooks::install_at(hooks::Stage::Remove, &backup, move || { + std::fs::write(ready, b"before-backup-unlink").expect("publish reached boundary"); + loop { + std::thread::park(); + } + }); + disk.delete( + "bucket", + &rollback_path, + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await + .expect("child cleanup should reach its boundary"); + panic!("child unexpectedly returned without being killed"); + } + + let dir = tempfile::tempdir().expect("create test disk"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + ensure_test_volume(&disk, "bucket").await; + let residue = write_committed_delete_residue(&disk, "bucket", "object").await; + let original_backup = fs::read(residue.backup()).await.expect("read original backup"); + let original_part = fs::read(residue.part()).await.expect("read original part"); + + for attempt in 0..2 { + let ready = dir.path().join(format!("cleanup-ready-{attempt}")); + let mut child = std::process::Command::new(std::env::current_exe().expect("current test binary")) + .args([ + "--exact", + "disk::local::test::quorum_delete_cleanup_process_crash_preserves_retry_evidence", + "--nocapture", + ]) + .env(CHILD_ROOT, dir.path()) + .env(CHILD_TRANSACTION, residue.transaction.to_string()) + .env(CHILD_READY, &ready) + .spawn() + .expect("spawn this test's cleanup process"); + let reached = tokio::time::timeout(Duration::from_secs(30), async { + while !ready.exists() { + if child.try_wait().expect("check child status").is_some() { + return false; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + true + }) + .await; + // Child is an owned process handle; no process-name matching or + // external PID is accepted by this test. + if child.try_wait().expect("check owned child").is_none() { + child.kill().expect("SIGKILL the owned cleanup child"); + } + let status = child.wait().expect("reap cleanup child"); + assert!(matches!(reached, Ok(true)), "child must reach the exact cleanup boundary"); + assert_eq!(status.signal(), Some(9), "child must have been interrupted by SIGKILL"); + println!( + "cleanup crash {attempt}: backup={} marker={} part={}", + residue.backup().exists(), + residue.marker().exists(), + residue.part().exists() + ); + assert_eq!(fs::read(residue.backup()).await.expect("backup at paused unlink"), original_backup); + assert_eq!(fs::read(residue.part()).await.expect("payload survives each crash"), original_part); + assert!(residue.marker().is_file(), "transaction marker survives each crash"); + assert!( + residue + .object_dir + .join(residue.transaction.to_string()) + .join(DELETE_QUORUM_COMMITTED_FILE) + .is_file(), + "quorum receipt survives each crash" + ); + } + + let reopened = LocalDisk::new(&endpoint, false) + .await + .expect("reopen disk after process interruption"); + reopened + .delete( + "bucket", + &format!("object/{}", residue.transaction), + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }, + ) + .await + .expect("retry the already committed coordinator transaction"); + assert_eq!(original_part, b"stale"); + let bucket_result = reopened.delete_volume("bucket", false).await; + println!("non-force bucket deletion after retry: {bucket_result:?}"); + assert!( + !residue.object_dir.exists(), + "repeated delete after two crashes must reclaim this known-successful transaction" + ); + bucket_result.expect("non-force bucket deletion after retry"); + } + + // Normal post-quorum cleanup must preserve its data-dir marker until the + // rollback backup is gone, including across an interrupted retry. + #[cfg(not(windows))] + #[tokio::test] + async fn quorum_delete_cleanup_removes_backup_before_data_and_can_retry() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + + let dir = tempfile::tempdir().expect("create test disk"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + let bucket = "bucket"; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + let data_path = residue.part(); + let rollback_path = format!("{object}/{}", residue.transaction); + let opts = DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }; + let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (release, release_rx) = std::sync::mpsc::channel::<()>(); + let _hook = hooks::install_at(hooks::Stage::Remove, &data_path, move || { + let _ = entered_tx.send(()); + let _ = release_rx.recv(); + }); + let mut cleanup = Box::pin(disk.delete(bucket, &rollback_path, opts.clone())); + tokio::time::timeout(Duration::from_secs(10), async { + tokio::select! { + entered = entered_rx => entered.expect("cleanup must reach data removal"), + result = cleanup.as_mut() => panic!("cleanup ended before data removal: {result:?}"), + } + }) + .await + .expect("cleanup reaches physical payload unlink"); + + let backup_gone = !residue.backup().exists(); + let marker_present = residue.marker().exists(); + let original_part = fs::read(residue.part()).await.expect("paused data must remain readable"); + drop(cleanup); + + // The in-flight unlink retains metadata ownership after cancellation. + // Retry must wait for that physical operation before removing proof. + let mut retry = Box::pin(disk.delete(bucket, &rollback_path, opts)); + assert!( + tokio::time::timeout(Duration::from_millis(100), retry.as_mut()) + .await + .is_err(), + "retry cannot bypass the parked cleanup's namespace ownership" + ); + assert!(residue.marker().is_file()); + assert!( + residue + .object_dir + .join(residue.transaction.to_string()) + .join(DELETE_QUORUM_COMMITTED_FILE) + .is_file() + ); + drop(release); + tokio::time::timeout(Duration::from_secs(10), retry) + .await + .expect("retry completes after the abandoned unlink releases ownership") + .expect("retry cleanup"); + + assert!( + backup_gone, + "post-quorum cleanup must remove the backup before touching its data-dir evidence" + ); + assert!(marker_present, "data removal must still have its transaction marker"); + assert_eq!(original_part, b"stale"); + assert!(!residue.object_dir.exists(), "interrupted cleanup and retry must leave no object residue"); + } + + #[cfg(not(windows))] + #[tokio::test] + async fn quorum_delete_cleanup_backup_error_preserves_data_and_marker() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + + let dir = tempfile::tempdir().expect("create test disk"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + let bucket = "bucket"; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + let backup_bytes = fs::read(residue.backup()).await.expect("read rollback evidence"); + let backup = residue.backup(); + let held_backup = residue.object_dir.join("held-backup"); + let blocked_backup = backup.clone(); + let retained_backup = held_backup.clone(); + // Replace only the classified backup inode with a directory so the + // real unlink fails, while retaining its exact bytes for retry. + let _hook = hooks::install_at(hooks::Stage::Remove, &backup, move || { + std::fs::rename(&blocked_backup, retained_backup).expect("retain rollback bytes"); + std::fs::create_dir(blocked_backup).expect("block the actual backup unlink"); + }); + let rollback_path = format!("{object}/{}", residue.transaction); + let opts = DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }; + disk.delete(bucket, &rollback_path, opts.clone()) + .await + .expect_err("backup removal failure must be returned"); + + assert!(backup.is_dir(), "the native backup unlink must have encountered the blocker"); + assert_eq!(fs::read(&held_backup).await.expect("backup bytes must survive"), backup_bytes); + assert!(residue.marker().exists(), "backup failure must retain the data-dir transaction marker"); + assert_eq!(fs::read(residue.part()).await.expect("payload must survive backup failure"), b"stale"); + + fs::remove_dir(&backup).await.expect("remove the one-shot unlink blocker"); + fs::rename(&held_backup, &backup) + .await + .expect("restore the unchanged backup inode"); + assert_eq!(fs::read(&backup).await.expect("canonical rollback backup"), backup_bytes); + disk.delete(bucket, &rollback_path, opts) + .await + .expect("retry cleanup after one-shot unlink failure"); + assert!(!residue.object_dir.exists(), "successful retry must reclaim the transaction"); + } + + #[tokio::test] + async fn quorum_delete_cleanup_preserves_other_generations_transactions_and_children() { + let dir = tempfile::tempdir().expect("create test disk"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + let bucket = "bucket"; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + let newer_data_dir = Uuid::new_v4(); + let newer_data = residue.object_dir.join(newer_data_dir.to_string()); + fs::create_dir_all(&newer_data).await.expect("create newer data directory"); + fs::write(newer_data.join("part.1"), b"newer payload") + .await + .expect("write newer payload"); + let metadata = test_meta(test_file_info(object, Uuid::new_v4(), Some(newer_data_dir), None)); + fs::write(residue.object_dir.join(STORAGE_FORMAT_FILE), &metadata) + .await + .expect("publish newer metadata before old transaction cleanup"); + let unrelated = Uuid::new_v4(); + let unrelated_backup = residue + .object_dir + .join(unrelated.to_string()) + .join(STORAGE_FORMAT_FILE_BACKUP); + fs::create_dir_all(unrelated_backup.parent().expect("backup parent")) + .await + .expect("create unrelated transaction"); + fs::write(&unrelated_backup, b"corrupt unproven backup") + .await + .expect("write ambiguous backup"); + let mismatched_marker = newer_data.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{unrelated}")); + fs::write(&mismatched_marker, []).await.expect("write unrelated marker"); + let child = residue.object_dir.join("child").join(STORAGE_FORMAT_FILE); + fs::create_dir_all(child.parent().expect("child parent")) + .await + .expect("create descendant"); + fs::write(&child, b"child metadata").await.expect("write descendant metadata"); + + disk.delete( + bucket, + &format!("{object}/{}", residue.transaction), + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }, + ) + .await + .expect("clean only the successful transaction"); + + assert!(!residue.backup().exists() && !residue.part().exists() && !residue.marker().exists()); + assert_eq!( + fs::read(residue.object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("read newer metadata"), + metadata + ); + assert_eq!(fs::read(newer_data.join("part.1")).await.expect("read newer payload"), b"newer payload"); + assert_eq!( + fs::read(&unrelated_backup).await.expect("read unproven backup"), + b"corrupt unproven backup" + ); + assert!(mismatched_marker.exists()); + assert_eq!(fs::read(child).await.expect("read descendant"), b"child metadata"); + } + + // Internal maintenance retains its legacy cleanup behavior while user + // buckets require the separate set-wide authorization boundary. #[tokio::test] async fn delete_versions_finishes_committed_delete_residue() { let dir = tempfile::tempdir().expect("temp dir should be created"); let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse"); let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created"); - let bucket = "bucket"; + let bucket = RUSTFS_META_BUCKET; let object = "prefix/object"; ensure_test_volume(&disk, bucket).await; let residue = write_committed_delete_residue(&disk, bucket, object).await; @@ -25338,7 +27849,7 @@ mod test { let dir = tempfile::tempdir().expect("temp dir should be created"); let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse"); let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created"); - let bucket = "bucket"; + let bucket = RUSTFS_META_BUCKET; let object = "object"; ensure_test_volume(&disk, bucket).await; let residue = write_committed_delete_residue(&disk, bucket, object).await; @@ -25358,14 +27869,13 @@ mod test { assert!(!residue.object_dir.exists(), "the committed residue must be reclaimed"); } - // The rollback backup may already be gone; the committed marker alone - // proves the delete. + // Internal maintenance can finish its legacy marker after backup cleanup. #[tokio::test] async fn delete_versions_finishes_committed_marker_without_rollback_backup() { let dir = tempfile::tempdir().expect("temp dir should be created"); let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse"); let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created"); - let bucket = "bucket"; + let bucket = RUSTFS_META_BUCKET; let object = "object"; ensure_test_volume(&disk, bucket).await; let residue = write_committed_delete_residue(&disk, bucket, object).await; @@ -25381,15 +27891,14 @@ mod test { assert!(!residue.object_dir.exists(), "the committed marker alone must be enough"); } - // Only a committed marker on this disk proves the delete. A reserved marker, - // a rollback backup alone, or live metadata beside the marker is left for - // the owning transaction, rollback, or heal. + // Internal maintenance preserves reserved markers, backup-only states and + // live metadata for the owning transaction, rollback or heal. #[tokio::test] async fn committed_delete_residue_finish_leaves_unproven_state_alone() { let dir = tempfile::tempdir().expect("temp dir should be created"); let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse"); let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created"); - let bucket = "bucket"; + let bucket = RUSTFS_META_BUCKET; ensure_test_volume(&disk, bucket).await; // A rollback backup without a committed marker, as an inline object leaves it. @@ -25418,7 +27927,7 @@ mod test { for object in ["backup-only", "reserved", "live"] { assert_eq!( - disk.finish_committed_delete_residue(bucket, object, None).await, + disk.finish_internal_delete_residue(bucket, object, None).await, 0, "{object} carries no committed proof" ); @@ -25445,7 +27954,7 @@ mod test { let dir = tempfile::tempdir().expect("temp dir should be created"); let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse"); let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created"); - let bucket = "bucket"; + let bucket = RUSTFS_META_BUCKET; let object = "logs"; ensure_test_volume(&disk, bucket).await; let residue = write_committed_delete_residue(&disk, bucket, object).await; @@ -25457,7 +27966,7 @@ mod test { .await .expect("nested metadata should be written"); - assert_eq!(disk.finish_committed_delete_residue(bucket, object, None).await, 0); + assert_eq!(disk.finish_internal_delete_residue(bucket, object, None).await, 0); assert!(residue.backup().exists() && residue.marker().exists() && residue.part().exists()); assert!(nested.exists()); } @@ -25473,7 +27982,7 @@ mod test { let dir = tempfile::tempdir().expect("temp dir should be created"); let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse"); let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created"); - let bucket = "bucket"; + let bucket = RUSTFS_META_BUCKET; let object = "object"; ensure_test_volume(&disk, bucket).await; let residue = write_committed_delete_residue(&disk, bucket, object).await; @@ -25491,7 +28000,7 @@ mod test { let _ = release_rx.recv(); }); - let mut finish = Box::pin(disk.finish_committed_delete_residue(bucket, object, None)); + let mut finish = Box::pin(disk.finish_internal_delete_residue(bucket, object, None)); tokio::time::timeout(Duration::from_secs(10), async { tokio::select! { entered = entered_rx => entered.expect("finish should reach the marked data dir"), @@ -25507,7 +28016,7 @@ mod test { // The abandoned rename is still parked; the next attempt recognizes the // residue by its marker and finishes it. - assert_eq!(disk.finish_committed_delete_residue(bucket, object, None).await, 1); + assert_eq!(disk.finish_internal_delete_residue(bucket, object, None).await, 1); assert!(!residue.object_dir.exists(), "the retried finish must reclaim the residue"); // Let the abandoned rename run into its missing source and wait until @@ -25529,7 +28038,7 @@ mod test { let dir = tempfile::tempdir().expect("temp dir should be created"); let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse"); let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created"); - let bucket = "bucket"; + let bucket = RUSTFS_META_BUCKET; let object = "object"; ensure_test_volume(&disk, bucket).await; let committed = write_committed_delete_residue(&disk, bucket, object).await; @@ -25548,12 +28057,620 @@ mod test { .await .expect("unproven backup should be written"); - assert_eq!(disk.finish_committed_delete_residue(bucket, object, None).await, 1); + assert_eq!(disk.finish_internal_delete_residue(bucket, object, None).await, 1); assert!(!committed.object_dir.join(committed.data_dir.to_string()).exists()); assert!(unproven_backup.exists(), "a backup without its own committed marker must remain"); assert_eq!(recorder.counter_value(METRIC_DISK_DELETE_RESIDUE_FINISHED_TOTAL, &[]), 1); } + #[tokio::test] + async fn delete_quorum_receipt_finishes_inline_backup_after_interrupted_cleanup() { + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = RUSTFS_META_BUCKET; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let object_dir = disk.io_get_object_path(bucket, object).expect("object path"); + let transaction = Uuid::new_v4(); + let rollback_path = object_dir.join(transaction.to_string()); + fs::create_dir_all(&rollback_path).await.expect("transaction directory"); + fs::write(rollback_path.join(STORAGE_FORMAT_FILE_BACKUP), b"inline metadata") + .await + .expect("inline rollback backup"); + // Backup alone remains ambiguous. Only the set-quorum receipt permits cleanup. + assert_eq!(disk.finish_internal_delete_residue(bucket, object, None).await, 0); + fs::write(rollback_path.join(DELETE_QUORUM_COMMITTED_FILE), b"") + .await + .expect("quorum receipt"); + assert_eq!(disk.finish_internal_delete_residue(bucket, object, None).await, 1); + assert!(!object_dir.exists(), "inline backup and receipt must be reclaimed"); + } + + #[cfg(not(windows))] + #[tokio::test] + async fn delete_quorum_cleanup_removes_backup_before_marked_data() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = "quorum-cleanup-order"; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + let data_path = residue.part(); + let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (release, release_rx) = std::sync::mpsc::channel::<()>(); + let _hook = hooks::install_at(hooks::Stage::Remove, &data_path, move || { + let _ = entered_tx.send(()); + let _ = release_rx.recv(); + }); + let transaction_path = format!("{object}/{}", residue.transaction); + let mut cleanup = Box::pin(disk.delete( + bucket, + &transaction_path, + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }, + )); + tokio::time::timeout(Duration::from_secs(10), async { + tokio::select! { + entered = entered_rx => entered.expect("data cleanup starts"), + result = cleanup.as_mut() => panic!("cleanup returned before data removal: {result:?}"), + } + }) + .await + .expect("cleanup reaches data removal"); + assert!(!residue.backup().exists(), "backup must disappear before data"); + assert!(residue.marker().exists(), "data proof must remain during removal"); + assert!( + residue + .object_dir + .join(residue.transaction.to_string()) + .join(DELETE_QUORUM_COMMITTED_FILE) + .is_file() + ); + drop(release); + tokio::time::timeout(Duration::from_secs(10), cleanup) + .await + .expect("cleanup completes") + .expect("cleanup succeeds"); + assert!(!residue.object_dir.exists(), "completed cleanup reclaims every transaction file"); + } + + #[cfg(not(windows))] + #[tokio::test] + async fn delete_rollback_keeps_markers_until_metadata_restore() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = "rollback-marker-order"; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + let backup = residue.backup(); + let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (release, release_rx) = std::sync::mpsc::channel::<()>(); + let _hook = hooks::install_at(hooks::Stage::Rename, &backup, move || { + let _ = entered_tx.send(()); + let _ = release_rx.recv(); + }); + let request = residue_delete_request(object); + let mut rollback = Box::pin(disk.delete_versions( + bucket, + request, + DeleteOptions { + undo_write: true, + undo_delete: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }, + )); + tokio::time::timeout(Duration::from_secs(10), async { + tokio::select! { + entered = entered_rx => entered.expect("metadata restore starts"), + result = rollback.as_mut() => panic!("rollback returned before restore: {result:?}"), + } + }) + .await + .expect("rollback reaches metadata restore"); + assert!(residue.marker().exists(), "rollback must retain its marker until metadata is restored"); + assert!(residue.backup().exists(), "backup must remain before restore rename"); + drop(release); + let errors = tokio::time::timeout(Duration::from_secs(10), rollback) + .await + .expect("rollback completes"); + assert!(errors.iter().all(Option::is_none), "{errors:?}"); + assert!(residue.object_dir.join(STORAGE_FORMAT_FILE).is_file()); + assert!(!residue.marker().exists()); + assert!(residue.part().exists()); + } + + #[tokio::test] + async fn delete_rollback_rejects_quorum_committed_transaction() { + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = "quorum-rollback-rejection"; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + fs::write( + residue + .object_dir + .join(residue.transaction.to_string()) + .join(DELETE_QUORUM_COMMITTED_FILE), + b"", + ) + .await + .expect("quorum receipt"); + let errors = disk + .delete_versions( + bucket, + residue_delete_request(object), + DeleteOptions { + undo_write: true, + undo_delete: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }, + ) + .await; + assert!(matches!(errors.as_slice(), [Some(DiskError::FileCorrupt)]), "{errors:?}"); + assert!( + !residue.object_dir.join(STORAGE_FORMAT_FILE).exists(), + "a committed delete must not resurrect metadata" + ); + assert!(residue.backup().exists() && residue.marker().exists()); + } + + #[cfg(not(windows))] + #[tokio::test] + async fn delete_quorum_receipt_sync_failure_preserves_rollback_backup() { + let _mode = durability_mode_override::set(DurabilityMode::Strict); + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = RUSTFS_META_BUCKET; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + let rollback_path = residue.object_dir.join(residue.transaction.to_string()); + os::fsync_dir_recorder::set_failure(&rollback_path, ErrorKind::Other); + let result = disk + .delete( + bucket, + &format!("{object}/{}", residue.transaction), + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }, + ) + .await; + assert!(result.is_err(), "failed receipt durability must stop cleanup"); + assert!(residue.backup().exists() && residue.part().exists() && residue.marker().exists()); + assert_eq!(disk.finish_internal_delete_residue(bucket, object, None).await, 1); + assert!(!residue.object_dir.exists(), "retry must complete the already-decided deletion"); + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_quorum_backup_removal_sync_failure_retains_receipt() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + let _mode = durability_mode_override::set(DurabilityMode::Strict); + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = RUSTFS_META_BUCKET; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + let transaction_path = residue.object_dir.join(residue.transaction.to_string()); + let receipt = transaction_path.join(DELETE_QUORUM_COMMITTED_FILE); + let _hook = hooks::install_at(hooks::Stage::Remove, &residue.backup(), move || { + // Arm after the receipt's initial fsync to fail the ordering fence + // between backup unlink and eventual receipt removal. + os::fsync_dir_recorder::set_failure(&transaction_path, ErrorKind::Other); + }); + let result = disk + .delete( + bucket, + &format!("{object}/{}", residue.transaction), + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }, + ) + .await; + assert!(result.is_err(), "backup unlink must become durable before removing commitment proof"); + assert!(receipt.is_file() && residue.part().is_file() && residue.marker().is_file()); + assert!(!residue.backup().exists()); + assert_eq!(disk.finish_internal_delete_residue(bucket, object, None).await, 1); + assert!(!residue.object_dir.exists()); + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_quorum_payload_removal_sync_failure_retains_markers_and_receipt() { + let _mode = durability_mode_override::set(DurabilityMode::Strict); + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = RUSTFS_META_BUCKET; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + os::fsync_dir_recorder::set_failure(&residue.object_dir.join(residue.data_dir.to_string()), ErrorKind::Other); + let receipt = residue + .object_dir + .join(residue.transaction.to_string()) + .join(DELETE_QUORUM_COMMITTED_FILE); + let result = disk + .delete( + bucket, + &format!("{object}/{}", residue.transaction), + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }, + ) + .await; + assert!(result.is_err(), "payload unlink sync failure must stop marker cleanup"); + assert!(receipt.is_file() && residue.marker().is_file()); + assert!(!residue.part().exists()); + assert_eq!(disk.finish_internal_delete_residue(bucket, object, None).await, 1); + assert!(!residue.object_dir.exists()); + } + + #[cfg(not(windows))] + #[tokio::test] + async fn delete_generic_uuid_backup_directory_does_not_publish_quorum_receipt() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = "generic-uuid-delete"; + ensure_test_volume(&disk, bucket).await; + let path = format!("object/{}", Uuid::new_v4()); + let directory = disk.io_get_object_path(bucket, &path).expect("UUID directory"); + fs::create_dir_all(&directory).await.expect("UUID directory"); + fs::write(directory.join("part.1"), b"data").await.expect("part"); + fs::write(directory.join(STORAGE_FORMAT_FILE_BACKUP), b"write transaction backup") + .await + .expect("non-delete rollback backup"); + let receipt = directory.join(DELETE_QUORUM_COMMITTED_FILE); + let (receipt_tx, receipt_rx) = tokio::sync::oneshot::channel(); + let _hook = hooks::install_at(hooks::Stage::Rename, &directory, move || { + let _ = receipt_tx.send(receipt.exists()); + }); + disk.delete( + bucket, + &path, + DeleteOptions { + recursive: true, + immediate: true, + ..Default::default() + }, + ) + .await + .expect("generic directory deletion"); + assert!( + !receipt_rx.await.expect("inspect directory before deletion"), + "ordinary UUID data is not a transaction" + ); + } + + #[tokio::test] + async fn delete_quorum_receipt_preserves_uuid_prefix_child() { + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = RUSTFS_META_BUCKET; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let transaction = Uuid::new_v4(); + let transaction_path = format!("{object}/{transaction}"); + let rollback = disk.io_get_object_path(bucket, &transaction_path).expect("transaction path"); + let child = rollback.join("child").join(STORAGE_FORMAT_FILE); + fs::create_dir_all(child.parent().expect("child directory")) + .await + .expect("child directory"); + fs::write(&child, b"live child metadata").await.expect("live child"); + fs::write(rollback.join(DELETE_QUORUM_COMMITTED_FILE), b"") + .await + .expect("receipt"); + fs::write(rollback.join(STORAGE_FORMAT_FILE_BACKUP), b"old backup") + .await + .expect("backup"); + assert_eq!(disk.finish_internal_delete_residue(bucket, object, None).await, 0); + let result = disk + .delete( + bucket, + &transaction_path, + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await; + assert!(matches!(result, Err(DiskError::FileCorrupt)), "{result:?}"); + assert_eq!(fs::read(&child).await.expect("preserved child"), b"live child metadata"); + assert!(rollback.join(STORAGE_FORMAT_FILE_BACKUP).is_file()); + } + + #[cfg(not(windows))] + #[tokio::test] + async fn delete_quorum_cleanup_preserves_child_created_after_classification() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = "receipt-prefix-race"; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + let child = residue + .object_dir + .join(residue.transaction.to_string()) + .join("child") + .join(STORAGE_FORMAT_FILE); + let published_child = child.clone(); + let _hook = hooks::install_at(hooks::Stage::Remove, &residue.part(), move || { + std::fs::create_dir_all(published_child.parent().expect("child directory")).expect("concurrent child directory"); + std::fs::write(&published_child, b"concurrent child").expect("concurrent child metadata"); + }); + let result = disk + .delete( + bucket, + &format!("{object}/{}", residue.transaction), + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }, + ) + .await; + result.expect("cleanup preserves a concurrent child and removes only deleted payload"); + assert_eq!(fs::read(child).await.expect("preserved concurrent child"), b"concurrent child"); + assert!(!residue.part().exists(), "the original deleted payload must still be reclaimed"); + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_rollback_backup_sync_failure_precedes_metadata_mutation() { + for mode in [DurabilityMode::Strict, DurabilityMode::None] { + let _mode = durability_mode_override::set(mode); + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = "backup-durability-gate"; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let root = disk.io_get_object_path(bucket, object).expect("object path"); + let data_dir = Uuid::new_v4(); + let data_path = root.join(data_dir.to_string()); + fs::create_dir_all(&data_path).await.expect("data directory"); + fs::write(data_path.join("part.1"), b"original payload") + .await + .expect("payload"); + let original = test_meta(test_file_info(object, Uuid::nil(), Some(data_dir), None)); + fs::write(root.join(STORAGE_FORMAT_FILE), &original) + .await + .expect("original metadata"); + let transaction = Uuid::new_v4(); + let rollback = root.join(transaction.to_string()); + os::fsync_dir_recorder::set_failure(&rollback, ErrorKind::Other); + let errors = disk + .delete_versions( + bucket, + residue_delete_request(object), + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await; + if mode == DurabilityMode::Strict { + assert!(errors.iter().any(Option::is_some), "backup sync failure must stop deletion"); + assert_eq!(fs::read(root.join(STORAGE_FORMAT_FILE)).await.expect("intact metadata"), original); + assert_eq!(fs::read(data_path.join("part.1")).await.expect("intact payload"), b"original payload"); + } else { + assert!(errors.iter().all(Option::is_none), "{errors:?}"); + assert!( + !os::fsync_dir_recorder::was_fsynced(&rollback), + "none mode must skip backup directory sync" + ); + assert!( + os::fsync_dir_recorder::take_failure(&rollback).is_some(), + "none mode must not consume a sync failure" + ); + assert!(!root.join(STORAGE_FORMAT_FILE).exists(), "none mode still applies the requested delete"); + } + } + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_rollback_restore_sync_failure_retains_markers() { + let _mode = durability_mode_override::set(DurabilityMode::Strict); + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = "restore-durability-gate"; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + os::fsync_dir_recorder::set_failure(&residue.object_dir, ErrorKind::Other); + let opts = DeleteOptions { + undo_write: true, + undo_delete: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }; + let errors = disk + .delete_versions(bucket, residue_delete_request(object), opts.clone()) + .await; + assert!(errors.iter().any(Option::is_some), "restore sync failure must remain visible"); + let bytes = fs::read(residue.object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("restored metadata is readable"); + assert_eq!(FileMeta::load(&bytes).expect("valid restored metadata").versions.len(), 1); + assert!(residue.marker().exists(), "failed durability must retain transaction evidence"); + assert!(residue.part().exists(), "rollback must retain payload"); + let errors = disk.delete_versions(bucket, residue_delete_request(object), opts).await; + assert!(errors.iter().all(Option::is_none), "retry must finish rollback: {errors:?}"); + assert!(!residue.marker().exists()); + assert!(residue.object_dir.join(STORAGE_FORMAT_FILE).is_file()); + } + + #[tokio::test] + async fn delete_quorum_receipt_survives_deferred_snapshot_cleanup() { + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = RUSTFS_META_BUCKET; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + let data_path = format!("{object}/{}", residue.data_dir); + let snapshot = disk + .acquire_snapshot_lease(bucket, &data_path) + .await + .expect("snapshot protects deleted payload"); + let receipt = residue + .object_dir + .join(residue.transaction.to_string()) + .join(DELETE_QUORUM_COMMITTED_FILE); + let result = disk + .delete( + bucket, + &format!("{object}/{}", residue.transaction), + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }, + ) + .await; + assert!(matches!(result, Err(DiskError::DiskOngoingReq)), "{result:?}"); + assert!(receipt.is_file() && residue.marker().is_file() && residue.part().is_file()); + assert!(!residue.backup().exists(), "committed backup is no longer needed"); + disk.release_snapshot_lease(bucket, &data_path, snapshot) + .await + .expect("snapshot release finishes physical cleanup"); + assert!(!residue.part().exists()); + assert!(receipt.is_file(), "receipt remains until transaction cleanup observes physical removal"); + assert_eq!(disk.finish_internal_delete_residue(bucket, object, None).await, 1); + assert!(!residue.object_dir.exists()); + } + + #[cfg(not(windows))] + #[tokio::test] + async fn delete_quorum_data_cleanup_preserves_concurrent_uuid_prefix_child() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = "quorum-data-prefix-race"; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + let child = residue + .object_dir + .join(residue.data_dir.to_string()) + .join("child") + .join(STORAGE_FORMAT_FILE); + let published = child.clone(); + let _hook = hooks::install_at(hooks::Stage::Remove, &residue.part(), move || { + std::fs::create_dir_all(published.parent().expect("child directory")).expect("concurrent child directory"); + std::fs::write(published, b"new child").expect("concurrent child metadata"); + }); + disk.delete( + bucket, + &format!("{object}/{}", residue.transaction), + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + }, + ) + .await + .expect("safe cleanup preserves new descendants"); + assert_eq!(fs::read(child).await.expect("preserved child"), b"new child"); + assert!(!residue.part().exists()); + assert!(!residue.marker().exists()); + } + + #[tokio::test] + async fn delete_quorum_snapshot_replay_preserves_live_uuid_prefix_child() { + let dir = tempfile::tempdir().expect("temporary directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = RUSTFS_META_BUCKET; + let object = "object"; + ensure_test_volume(&disk, bucket).await; + let residue = write_committed_delete_residue(&disk, bucket, object).await; + let data_path = format!("{object}/{}", residue.data_dir); + let snapshot = disk.acquire_snapshot_lease(bucket, &data_path).await.expect("snapshot lease"); + let receipt = residue + .object_dir + .join(residue.transaction.to_string()) + .join(DELETE_QUORUM_COMMITTED_FILE); + assert!(matches!( + disk.delete( + bucket, + &format!("{object}/{}", residue.transaction), + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(residue.transaction), + ..Default::default() + } + ) + .await, + Err(DiskError::DiskOngoingReq) + )); + let child = residue + .object_dir + .join(residue.data_dir.to_string()) + .join("child") + .join(STORAGE_FORMAT_FILE); + fs::create_dir_all(child.parent().expect("child directory")) + .await + .expect("live child directory"); + fs::write(&child, b"live child").await.expect("live child metadata"); + assert!( + disk.release_snapshot_lease(bucket, &data_path, snapshot).await.is_err(), + "replay must reject mixed live contents" + ); + assert_eq!(fs::read(&child).await.expect("preserved child"), b"live child"); + assert!( + receipt.is_file() && residue.part().is_file(), + "blocked replay retains payload and commitment evidence" + ); + fs::remove_file(&child).await.expect("remove test child metadata"); + fs::remove_dir(child.parent().expect("child directory")) + .await + .expect("remove test child directory"); + assert_eq!(disk.finish_internal_delete_residue(bucket, object, None).await, 1); + assert!(!residue.object_dir.exists()); + } + #[tokio::test] async fn test_delete_version_error_after_staging_restores_data_dir() { use tempfile::tempdir; @@ -25808,6 +28925,220 @@ mod test { assert!(!object_dir.join(STORAGE_FORMAT_FILE).exists()); } + #[tokio::test] + async fn test_windows_delete_version_long_path_crosses_max_path_boundary() { + use super::super::DeleteRecoveryAction; + use tempfile::tempdir; + + let dir = tempdir().expect("temp dir should be created"); + let base_len = dir.path().as_os_str().len().saturating_add(1); + let pad_len = 150usize.saturating_sub(base_len).max(16); + let root = dir.path().join("x".repeat(pad_len)); + fs::create_dir_all(&root).await.expect("long disk root should be created"); + let endpoint = Endpoint::try_from(root.to_str().expect("disk root should be utf8")).expect("endpoint should parse"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created"); + + let bucket = "test-bucket"; + ensure_test_volume(&disk, bucket).await; + ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await; + ensure_test_volume(&disk, RUSTFS_META_TMP_DELETED_BUCKET).await; + + let path_code_units = |path: &Path| -> usize { + #[cfg(windows)] + { + use std::os::windows::ffi::OsStrExt; + path.as_os_str().encode_wide().count() + } + #[cfg(not(windows))] + { + path.as_os_str().len() + } + }; + let probe_rollback_dir = Uuid::new_v4(); + let single_char_backup_len = path_code_units( + &disk + .get_object_path(bucket, "dir/a") + .expect("probe object path should resolve") + .join(probe_rollback_dir.to_string()) + .join(STORAGE_FORMAT_FILE_BACKUP), + ); + + for target_backup_len in [259usize, 260, 261, 300] { + let key_suffix_len = target_backup_len + .checked_sub(single_char_backup_len) + .and_then(|extra| extra.checked_add(1)) + .expect("resolved disk root must leave room for the target backup path length"); + let object = format!("dir/{}", "a".repeat(key_suffix_len)); + let physical_object_dir = disk + .get_object_path(bucket, &object) + .expect("physical long object path should resolve"); + let object_dir = disk + .io_get_object_path(bucket, &object) + .expect("io long object path should resolve"); + let rollback_dir = Uuid::new_v4(); + let physical_backup_file_path = physical_object_dir + .join(rollback_dir.to_string()) + .join(STORAGE_FORMAT_FILE_BACKUP); + assert_eq!( + path_code_units(&physical_backup_file_path), + target_backup_len, + "physical backup path must hit the exact target length: {physical_backup_file_path:?}" + ); + let backup_file_path = object_dir.join(rollback_dir.to_string()).join(STORAGE_FORMAT_FILE_BACKUP); + + let data_dir = Uuid::new_v4(); + let data_path = object_dir.join(data_dir.to_string()); + fs::create_dir_all(&data_path).await.expect("long data dir should be created"); + fs::write(data_path.join("part.1"), b"x") + .await + .expect("part data should be written"); + let fi = test_file_info(&object, Uuid::nil(), Some(data_dir), None); + let meta = test_meta(fi.clone()); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &meta) + .await + .expect("metadata should be written"); + + let read_fi = disk + .read_version( + bucket, + bucket, + &object, + "", + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("read_version should succeed on long path"); + assert_eq!(read_fi.name, object); + + disk.delete_version( + bucket, + &object, + fi.clone(), + false, + DeleteOptions { + old_data_dir: Some(rollback_dir), + ..Default::default() + }, + ) + .await + .expect("delete_version should stage rollback backup across MAX_PATH"); + assert!(backup_file_path.is_file(), "rollback backup must exist at {backup_file_path:?}"); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let mode = std::fs::metadata(&backup_file_path) + .expect("rollback backup metadata should be readable") + .permissions() + .mode() + & 0o777; + assert_eq!(mode, 0o600, "rollback backup must retain owner-only 0o600 permissions"); + } + + disk.undo_write_with_namespace_owner( + bucket, + &object, + fi.clone(), + DeleteOptions { + undo_write: true, + undo_delete: true, + old_data_dir: Some(rollback_dir), + ..Default::default() + }, + None, + ) + .await + .expect("undo_write should restore metadata and data across MAX_PATH"); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("restored metadata should be readable"), + meta + ); + assert!(!object_dir.join(rollback_dir.to_string()).exists()); + + let commit_rollback_dir = Uuid::new_v4(); + disk.delete_version( + bucket, + &object, + fi.clone(), + false, + DeleteOptions { + old_data_dir: Some(commit_rollback_dir), + ..Default::default() + }, + ) + .await + .expect("second delete_version should succeed across MAX_PATH"); + disk.delete_with_namespace_owner( + bucket, + &format!("{object}/{commit_rollback_dir}"), + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(commit_rollback_dir), + ..Default::default() + }, + None, + ) + .await + .expect("cleanup of committed delete transaction should succeed across MAX_PATH"); + assert!(!object_dir.join(STORAGE_FORMAT_FILE).exists()); + assert!(!object_dir.join(commit_rollback_dir.to_string()).exists()); + + let mut inline_fi = test_file_info(&object, Uuid::nil(), None, Some(Bytes::from_static(b"x"))); + inline_fi.set_inline_data(); + let inline_meta = test_meta(inline_fi.clone()); + fs::create_dir_all(&object_dir) + .await + .expect("object dir should be recreated for recoverable delete"); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &inline_meta) + .await + .expect("inline metadata should be written"); + + let tx = Uuid::new_v4(); + let mut delete_req = FileInfo { + name: object.clone(), + ..Default::default() + }; + delete_req.set_tier_free_version_id(&Uuid::new_v4().to_string()); + disk.delete_version_with_namespace_owner( + bucket, + &object, + delete_req, + false, + DeleteOptions { + old_data_dir: Some(tx), + recoverable_delete: true, + ..Default::default() + }, + None, + ) + .await + .expect("recoverable delete_version should prepare across MAX_PATH"); + + let inspected = disk + .delete_recovery_inspect(bucket, &object, tx) + .await + .expect("delete_recovery_inspect should succeed across MAX_PATH"); + let published = disk + .delete_recovery_settle(bucket, &object, tx, inspected.fingerprint, DeleteRecoveryAction::PublishCommit, None) + .await + .expect("PublishCommit should write quorum receipt across MAX_PATH"); + let finished = disk + .delete_recovery_settle(bucket, &object, tx, published.fingerprint, DeleteRecoveryAction::FinishCommit, None) + .await + .expect("FinishCommit should write finished markers across MAX_PATH"); + disk.delete_recovery_settle(bucket, &object, tx, finished.fingerprint, DeleteRecoveryAction::FinalizeCommit, None) + .await + .expect("FinalizeCommit should remove transaction directory across MAX_PATH"); + assert!(!object_dir.join(STORAGE_FORMAT_FILE).exists()); + assert!(!object_dir.join(tx.to_string()).exists()); + } + } + #[tokio::test] async fn test_rename_data_failure_before_metadata_commit_preserves_old_metadata() { use tempfile::tempdir; @@ -26047,9 +29378,14 @@ mod test { // Backdate after the write above: creating stale/data refreshes the // scanned tmp/stale directory's mtime. backdate_mtime(&tmp.join("stale"), Duration::from_secs(10)); - LocalDisk::cleanup_stale_tmp_objects_with_expiry(dir.path().to_path_buf(), &publication_root, Duration::ZERO) - .await - .expect("operation should succeed"); + LocalDisk::cleanup_stale_tmp_objects_with_expiry( + dir.path().to_path_buf(), + &publication_root, + Duration::ZERO, + &crate::disk::cleanup_runtime::GcBudget::default(), + ) + .await + .expect("operation should succeed"); assert!(!tmp.join("stale").exists()); assert!(trash.exists()); @@ -26076,9 +29412,14 @@ mod test { fs::write(&fresh_dir, b"temporary").await.expect("operation should succeed"); fs::write(®ular_file, b"keep").await.expect("operation should succeed"); - LocalDisk::cleanup_stale_tmp_objects_with_expiry(dir.path().to_path_buf(), &publication_root, Duration::from_secs(60)) - .await - .expect("operation should succeed"); + LocalDisk::cleanup_stale_tmp_objects_with_expiry( + dir.path().to_path_buf(), + &publication_root, + Duration::from_secs(60), + &crate::disk::cleanup_runtime::GcBudget::default(), + ) + .await + .expect("operation should succeed"); assert!(tmp.join("fresh").exists()); assert!(regular_file.exists()); @@ -26087,6 +29428,44 @@ mod test { assert!(entries.next_entry().await.expect("operation should succeed").is_none()); } + #[cfg(target_os = "linux")] + #[tokio::test] + async fn background_cleanup_keeps_mount_lease_after_disk_drop() { + let dir = tempfile::tempdir().unwrap(); + let endpoint = Endpoint::try_from(dir.path().to_str().unwrap()).unwrap(); + let disk = LocalDisk::new(&endpoint, false).await.unwrap(); + let root = disk.io_root.clone(); + let trash_entry = LocalDisk::meta_path(&root, RUSTFS_META_TMP_DELETED_BUCKET).join("detached-gc"); + fs::create_dir_all(&trash_entry).await.unwrap(); + fs::write(trash_entry.join("part.1"), b"garbage").await.unwrap(); + let live_file = root.join("keep"); + fs::write(&live_file, b"live").await.unwrap(); + let (exit, exited) = tokio::sync::broadcast::channel(1); + tokio::time::pause(); + let cleanup = tokio::spawn(LocalDisk::cleanup_deleted_objects_loop( + root.clone(), + disk.publication_root.clone(), + disk.mount_lease.clone(), + exited, + )); + tokio::task::yield_now().await; + drop(disk); + tokio::task::yield_now().await; + assert!(live_file.exists(), "the GC owner must retain the original /proc/self/fd root"); + tokio::time::advance(DELETED_OBJECTS_CLEANUP_INTERVAL + Duration::from_secs(1)).await; + tokio::time::resume(); + timeout(Duration::from_secs(20), async { + while trash_entry.exists() { + tokio::time::sleep(Duration::from_millis(5)).await; + } + }) + .await + .expect("detached cleanup must finish against the pinned mount"); + assert_eq!(fs::read(&live_file).await.unwrap(), b"live"); + exit.send(()).unwrap(); + timeout(Duration::from_secs(20), cleanup).await.unwrap().unwrap(); + } + #[tokio::test(start_paused = true)] async fn cleanup_loop_interval_does_not_tick_immediately() { let start_at = Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL; @@ -28824,6 +32203,9 @@ mod test { undo_delete: false, old_data_dir: None, expected_delete_marker: None, + delete_recovery_bucket_incarnation: None, + recoverable_delete: false, + orphan_data_dir: false, }; disk.delete("test-volume", "test-file.txt", delete_opts) .await @@ -29592,6 +32974,121 @@ mod test { ); } + #[tokio::test] + async fn legacy_special_cleanup_metadata_contention_with_queued_volume_write() { + use crate::disk::disk_store::{LegacyPathDeleteOperation, LocalDiskWrapper}; + for orphan in [false, true] { + for metadata_contention in [true, false] { + let root = tempfile::tempdir().expect("special cleanup lock-order fixture"); + let endpoint = Endpoint::try_from(root.path().to_str().expect("UTF-8 endpoint")).expect("local endpoint"); + let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk")); + let volume = "special-cleanup-queued"; + let object = "object"; + for name in [volume, RUSTFS_META_TMP_BUCKET, RUSTFS_META_TMP_DELETED_BUCKET] { + ensure_test_volume(&disk, name).await; + } + let object_path = disk.get_object_path(volume, object).expect("metadata lock identity"); + let io_object = disk.io_get_object_path(volume, object).expect("physical object path"); + fs::create_dir_all(&io_object).await.expect("real object directory"); + let mut fi = test_file_info(object, Uuid::new_v4(), None, Some(Bytes::from_static(b"live inline payload"))); + fi.set_inline_data(); + let original = test_meta(fi); + fs::write(io_object.join(STORAGE_FORMAT_FILE), &original) + .await + .expect("live metadata"); + let id = Uuid::new_v4(); + let path = format!("{object}/{id}"); + let candidate = io_object.join(id.to_string()); + fs::create_dir(&candidate).await.expect("exact cleanup candidate"); + let proof = candidate.join(if orphan { "part.1" } else { STORAGE_FORMAT_FILE_BACKUP }); + fs::write(&proof, b"preserved until admission") + .await + .expect("real cleanup proof or payload"); + let opts = if orphan { + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + } + } else { + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(id), + ..Default::default() + } + }; + let wrapper = LocalDiskWrapper::new(disk.clone(), false); + let admitted = wrapper + .begin_legacy_path_delete( + volume, + LegacyPathDeleteOperation::Delete, + None, + tokio::time::Instant::now() + Duration::from_secs(30), + ) + .await + .expect("A owns the real physical volume READ"); + let mut deletion = Box::pin(disk.delete_volume(volume, true)); + assert!( + matches!(futures::poll!(deletion.as_mut()), std::task::Poll::Pending), + "W queues behind A's actual READ" + ); + let contention: Arc = if metadata_contention { + os::acquire_metadata_mutation_lease(&object_path, None).await + } else { + os::acquire_namespace_mutation_lease_with_owner(&candidate, None).await + }; + let volume_lock = os::disk_volume_mutation_lock(&disk.root, volume); + let mut competing = Box::pin(async move { + let read = volume_lock.read_owned().await; + (contention, read) + }); + assert!( + matches!(futures::poll!(competing.as_mut()), std::task::Poll::Pending), + "B holds the selected M or namespace key and queues READ behind W" + ); + let result = timeout(Duration::from_secs(5), admitted.delete(&path, opts.clone())) + .await + .expect("A must try the held M or namespace instead of forming a lock cycle"); + assert!( + matches!(result, Err(DiskError::Io(error)) if error.kind() == ErrorKind::WouldBlock), + "exact admitted cleanup contention result" + ); + assert_eq!(fs::read(&proof).await.expect("cleanup has not started"), b"preserved until admission"); + assert_eq!( + fs::read(io_object.join(STORAGE_FORMAT_FILE)) + .await + .expect("unchanged live metadata"), + original + ); + drop(deletion); + drop( + timeout(Duration::from_secs(5), competing) + .await + .expect("B drains after W is cancelled"), + ); + wrapper + .begin_legacy_path_delete( + volume, + LegacyPathDeleteOperation::Delete, + None, + tokio::time::Instant::now() + Duration::from_secs(30), + ) + .await + .expect("uncontended retry admission") + .delete(&path, opts) + .await + .expect("exact cleanup retry succeeds"); + assert!(!candidate.exists(), "only the exact orphan or committed transaction is removed"); + assert_eq!( + fs::read(io_object.join(STORAGE_FORMAT_FILE)) + .await + .expect("live version survives exact cleanup"), + original + ); + } + } + } + #[tokio::test] async fn legacy_metadata_write_reuses_prepared_locks_with_queued_delete() { use crate::disk::disk_store::LocalDiskWrapper; @@ -30609,10 +34106,12 @@ mod test { #[tokio::test] async fn test_read_file_exists() { - let test_file = "./test_read_exists.txt"; + let directory = tempfile::tempdir().expect("test directory"); + let root = os::PublicationRoot::new(directory.path()).expect("publication root"); + let test_file = &directory.path().join("read_exists"); // Test non-existent file - let (data, metadata) = read_file_exists(test_file).await.expect("operation should succeed"); + let (data, metadata) = read_file_exists(test_file, &root).await.expect("operation should succeed"); assert!(data.is_empty()); assert!(metadata.is_none()); @@ -30620,7 +34119,7 @@ mod test { fs::write(test_file, b"test content").await.expect("operation should succeed"); // Test existing file - let (data, metadata) = read_file_exists(test_file).await.expect("operation should succeed"); + let (data, metadata) = read_file_exists(test_file, &root).await.expect("operation should succeed"); assert_eq!(data.as_ref(), b"test content"); assert!(metadata.is_some()); @@ -30630,14 +34129,16 @@ mod test { #[tokio::test] async fn test_read_file_all() { - let test_file = "./test_read_all.txt"; + let directory = tempfile::tempdir().expect("test directory"); + let root = os::PublicationRoot::new(directory.path()).expect("publication root"); + let test_file = &directory.path().join("read_all"); let test_content = b"test content for read_all"; // Create test file fs::write(test_file, test_content).await.expect("operation should succeed"); // Test reading file - let (data, metadata) = read_file_all(test_file).await.expect("operation should succeed"); + let (data, metadata) = read_file_all(test_file, &root).await.expect("operation should succeed"); assert_eq!(data.as_ref(), test_content); assert!(metadata.is_file()); assert_eq!(metadata.len(), test_content.len() as u64); @@ -30646,22 +34147,6 @@ mod test { let _ = fs::remove_file(test_file).await; } - #[tokio::test] - async fn test_read_file_metadata() { - let test_file = "./test_metadata.txt"; - - // Create test file - fs::write(test_file, b"test").await.expect("operation should succeed"); - - // Test reading metadata - let metadata = read_file_metadata(test_file).await.expect("operation should succeed"); - assert!(metadata.is_file()); - assert_eq!(metadata.len(), 4); // "test" is 4 bytes - - // Clean up - let _ = fs::remove_file(test_file).await; - } - #[test] fn test_is_root_path() { // Unix root path @@ -31025,6 +34510,22 @@ mod test { assert!(debug.contains("endpoint")); } + #[cfg(windows)] + #[tokio::test] + async fn windows_local_disk_opens_url_style_drive_endpoint() { + use tempfile::tempdir; + + let dir = tempdir().expect("temp dir should be created"); + let drive_path = dir.path().to_string_lossy().replace('\\', "/"); + let endpoint = + Endpoint::try_from(format!("http://127.0.0.1:9000/{drive_path}").as_str()).expect("URL endpoint should parse"); + assert_eq!(Path::new(&endpoint.get_file_path()), dir.path()); + + LocalDisk::new(&endpoint, false) + .await + .expect("a URL-style drive endpoint must open as a local disk"); + } + #[tokio::test] async fn std_backend_truncate_append_stream_and_full_read_restore_bytes() { use tempfile::tempdir; @@ -31197,6 +34698,303 @@ mod test { assert!(construction_source.is::()); } + async fn write_orphan_test_parent_metadata(disk: &LocalDisk, data_dir: Uuid) { + let mut info = FileInfo::new("object", 1, 1); + info.data_dir = Some(data_dir); + info.mod_time = Some(OffsetDateTime::now_utc()); + info.size = 1; + let mut metadata = FileMeta::default(); + metadata.add_version(info).expect("add parent version"); + disk.write_all("orphan", "object/xl.meta", metadata.marshal_msg().expect("encode parent metadata").into()) + .await + .expect("publish parent metadata"); + } + + #[tokio::test] + async fn orphan_data_dir_cleanup_preserves_protected_entries() { + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + for protected in [ + "xl.meta".to_owned(), + "unknown".to_owned(), + STORAGE_FORMAT_FILE_BACKUP.to_owned(), + DELETE_QUORUM_COMMITTED_FILE.to_owned(), + delete_recovery::PREPARED.to_owned(), + format!("{DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4()), + ] { + let path = format!("object/{}", Uuid::new_v4()); + disk.write_all("orphan", &format!("{path}/part.1"), Bytes::from_static(b"payload")) + .await + .expect("write payload"); + disk.write_all("orphan", &format!("{path}/{protected}"), Bytes::from_static(b"witness")) + .await + .expect("write protected entry"); + assert!(!disk.is_plain_orphan_data_dir("orphan", &path).await.expect("probe leaf")); + assert!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + } + ) + .await + .is_err() + ); + assert_eq!( + disk.read_all("orphan", &format!("{path}/part.1")) + .await + .expect("read payload"), + b"payload"[..] + ); + assert_eq!( + disk.read_all("orphan", &format!("{path}/{protected}")) + .await + .expect("read witness"), + b"witness"[..] + ); + } + } + + #[cfg(unix)] + #[tokio::test] + async fn orphan_data_dir_cleanup_rejects_symlinks() { + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + let path = format!("object/{}", Uuid::new_v4()); + disk.write_all("orphan", &format!("{path}/part.1"), Bytes::from_static(b"payload")) + .await + .expect("write payload"); + let directory = disk.io_get_object_path("orphan", &path).expect("resolve leaf"); + std::os::unix::fs::symlink(directory.join("part.1"), directory.join("part.2")).expect("create symlink"); + assert!(!disk.is_plain_orphan_data_dir("orphan", &path).await.expect("probe leaf")); + assert!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + } + ) + .await + .is_err() + ); + assert_eq!(std::fs::read(directory.join("part.1")).expect("read payload"), b"payload"); + assert!( + std::fs::symlink_metadata(directory.join("part.2")) + .expect("stat symlink") + .is_symlink() + ); + } + + #[cfg(not(windows))] + #[tokio::test] + async fn orphan_data_dir_cleanup_preserves_child_created_after_preflight() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + let path = format!("object/{}", Uuid::new_v4()); + disk.write_all("orphan", &format!("{path}/part.1"), Bytes::from_static(b"stale")) + .await + .expect("write orphan payload"); + let directory = disk.io_get_object_path("orphan", &path).expect("resolve leaf"); + let child = directory.join("child"); + let published_child = child.clone(); + let _hook = hooks::install_at(hooks::Stage::Remove, &directory.join("part.1"), move || { + std::fs::create_dir_all(&published_child).expect("publish concurrent child"); + std::fs::write(published_child.join("xl.meta"), b"child metadata").expect("publish child metadata"); + std::fs::write(published_child.join("part.1"), b"child payload").expect("publish child payload"); + }); + assert!(disk.is_plain_orphan_data_dir("orphan", &path).await.expect("probe leaf")); + assert!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + } + ) + .await + .is_err() + ); + assert_eq!(std::fs::read(child.join("xl.meta")).expect("read child metadata"), b"child metadata"); + assert_eq!(std::fs::read(child.join("part.1")).expect("read child payload"), b"child payload"); + assert!(!directory.join("part.1").exists(), "classified orphan part was removed without recursing"); + } + + #[tokio::test] + async fn orphan_data_dir_cleanup_rechecks_snapshot_deferred_leaf() { + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + let path = format!("object/{}", Uuid::new_v4()); + disk.write_all("orphan", &format!("{path}/part.1"), Bytes::from_static(b"payload")) + .await + .expect("write orphan payload"); + let token = disk.acquire_snapshot_lease("orphan", &path).await.expect("acquire snapshot"); + assert_eq!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + } + ) + .await + .expect("defer cleanup"), + DataDirDeleteStatus::Deferred + ); + disk.write_all("orphan", &format!("{path}/child/xl.meta"), Bytes::from_static(b"child metadata")) + .await + .expect("publish child while snapshot is retained"); + assert!(disk.release_snapshot_lease("orphan", &path, token).await.is_err()); + assert_eq!( + disk.read_all("orphan", &format!("{path}/part.1")) + .await + .expect("read protected part"), + b"payload"[..] + ); + assert_eq!( + disk.read_all("orphan", &format!("{path}/child/xl.meta")) + .await + .expect("read child metadata"), + b"child metadata"[..] + ); + } + + #[tokio::test] + async fn orphan_data_dir_cleanup_preserves_referenced_payload_after_snapshot_deferral() { + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + let data_dir = Uuid::new_v4(); + let path = format!("object/{data_dir}"); + disk.write_all("orphan", &format!("{path}/part.1"), Bytes::from_static(b"restored payload")) + .await + .expect("write orphan payload"); + let token = disk.acquire_snapshot_lease("orphan", &path).await.expect("acquire snapshot"); + assert_eq!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + } + ) + .await + .expect("defer cleanup"), + DataDirDeleteStatus::Deferred + ); + write_orphan_test_parent_metadata(&disk, data_dir).await; + assert!(disk.release_snapshot_lease("orphan", &path, token).await.is_err()); + assert_eq!( + disk.read_all("orphan", &format!("{path}/part.1")) + .await + .expect("read newly referenced payload"), + b"restored payload"[..] + ); + } + + #[tokio::test] + async fn orphan_data_dir_cleanup_preserves_journal_created_after_snapshot_deferral() { + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + let path = format!("object/{}", Uuid::new_v4()); + let part = format!("{path}/part.1"); + disk.write_all("orphan", &part, Bytes::from_static(b"rollback payload")) + .await + .expect("write orphan payload"); + let token = disk.acquire_snapshot_lease("orphan", &path).await.expect("acquire snapshot"); + assert_eq!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + }, + ) + .await + .expect("defer cleanup"), + DataDirDeleteStatus::Deferred + ); + let record = format!("object/{}/{}", Uuid::new_v4(), delete_recovery::PREPARED); + disk.write_all("orphan", &record, Bytes::from_static(b"malformed prepared record")) + .await + .expect("publish recovery record after cleanup was deferred"); + assert_eq!(disk.release_snapshot_lease("orphan", &path, token).await, Err(DiskError::DiskOngoingReq)); + assert_eq!( + disk.read_all("orphan", &part).await.expect("read retained rollback payload"), + b"rollback payload"[..] + ); + assert_eq!( + disk.read_all("orphan", &record).await.expect("read retained recovery record"), + b"malformed prepared record"[..] + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn orphan_data_dir_cleanup_propagates_parent_directory_sync_failure() { + let _mode = durability_mode_override::set(DurabilityMode::Strict); + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + let path = format!("object/{}", Uuid::new_v4()); + disk.write_all("orphan", &format!("{path}/part.1"), Bytes::from_static(b"stale")) + .await + .expect("write orphan payload"); + let parent = disk.io_get_object_path("orphan", "object").expect("resolve parent directory"); + os::fsync_dir_recorder::set_failure(&parent, ErrorKind::Other); + assert!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + } + ) + .await + .is_err(), + "a failed parent sync cannot report durable reclamation" + ); + assert!( + !disk + .io_get_object_path("orphan", &path) + .expect("resolve data directory") + .exists() + ); + assert!( + os::fsync_dir_recorder::take_failure(&parent).is_none(), + "the rmdir must attempt parent synchronization" + ); + } + #[tokio::test] async fn snapshot_leases_defer_data_dir_cleanup_until_last_release() { use tempfile::tempdir; @@ -31250,9 +35048,14 @@ mod test { Bytes::from_static(b"later") ); - disk.release_snapshot_lease(volume, &data_dir, renewed) - .await - .expect("renewed lease release should succeed"); + assert!( + disk.release_snapshot_lease_without_cleanup(volume, &data_dir, renewed).await, + "a non-final reader release must not dispatch cleanup" + ); + assert!( + !disk.release_snapshot_lease_without_cleanup(volume, &data_dir, second).await, + "the final reader must retain its token for owned deferred reclamation" + ); assert!( disk.read_all(volume, &first_part).await.is_ok(), "one remaining lease must keep the data directory" @@ -31339,6 +35142,44 @@ mod test { )); } + #[tokio::test] + async fn quota_mutation_fence_token_is_bound_to_resource_and_disk_process() { + let root_dir = tempfile::tempdir().expect("create quota fence disk"); + let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("parse disk endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("open initial disk process"); + let bucket = "quota-fence-volume"; + let object = "object"; + let fence_path = quota_mutation_fence_path(bucket, object); + let token = disk + .acquire_snapshot_lease(RUSTFS_META_BUCKET, &fence_path) + .await + .expect("acquire initial quota token"); + drop( + disk.claim_quota_mutation_fence(bucket, object, token) + .await + .expect("claim matching token"), + ); + assert!(disk.claim_quota_mutation_fence(bucket, "other", token).await.is_err()); + assert!(disk.claim_quota_mutation_fence("other-bucket", object, token).await.is_err()); + drop(disk); + + let reopened = LocalDisk::new(&endpoint, false).await.expect("reopen the same disk"); + let successor = reopened + .acquire_snapshot_lease(RUSTFS_META_BUCKET, &fence_path) + .await + .expect("acquire successor process token"); + assert!( + reopened.claim_quota_mutation_fence(bucket, object, token).await.is_err(), + "a token issued before restart must not authorize a later process" + ); + drop( + reopened + .claim_quota_mutation_fence(bucket, object, successor) + .await + .expect("claim successor token"), + ); + } + #[tokio::test] async fn quota_mutation_fence_revoke_waits_for_active_claim_and_rejects_late_claims() { use tempfile::tempdir; diff --git a/crates/ecstore/src/disk/local/commit.rs b/crates/ecstore/src/disk/local/commit.rs index 326f83326..8a3d09228 100644 --- a/crates/ecstore/src/disk/local/commit.rs +++ b/crates/ecstore/src/disk/local/commit.rs @@ -24,7 +24,7 @@ use super::{ LOG_SUBSYSTEM_DISK_LOCAL, LocalDisk, SyncMode, effective_durability, inline_metadata_rollback_dir, observe_old_current_size, remove_dir_all_if_exists, remove_dst_base_before_commit, remove_file_if_exists, rename_data_versions_signature, run_inline_preparation_before_backup, should_fail_after_metadata_commit, should_fail_before_old_metadata_backup, - should_fail_commit_rename, should_remove_staged_meta_before_commit, skip_access_checks, + should_fail_commit_rename, should_remove_staged_meta_before_commit, skip_access_checks, tmp_staging_mutation_path, }; #[cfg(test)] use super::{run_inline_before_file_sync_admission, run_owned_file_write_before_open, run_rename_data_after_first_publication}; @@ -47,6 +47,33 @@ use tokio::fs; use tracing::{info, warn}; use uuid::Uuid; +fn transitioned_old_current_source( + dst_meta_existed: bool, + dst_meta_unparsable: bool, + xlmeta: &FileMeta, + volume: &str, + path: &str, + version_id: Uuid, +) -> (bool, Option) { + if dst_meta_unparsable { + return (false, None); + } + if !dst_meta_existed { + return (true, None); + } + let Ok(old_current) = xlmeta.into_fileinfo(volume, path, "", false, false, true) else { + return (false, None); + }; + if old_current.version_id.unwrap_or_default() != version_id { + return (true, None); + } + if old_current.transition_status == rustfs_filemeta::TRANSITION_COMPLETE { + (true, Some(old_current)) + } else { + (true, None) + } +} + /// Hold later repair publications after admitting one baseline object. The /// fixture arms this on one replacement disk before rejoining the cluster. #[cfg(feature = "e2e-test-hooks")] @@ -214,7 +241,7 @@ pub(super) async fn lock_rename_commit_directories( publication_root: &os::PublicationRoot, mutation_lease: Arc, ) -> Result { - #[cfg(windows)] + #[cfg(any(windows, target_os = "linux"))] let result = { let source_parent = source_parent.to_path_buf(); let destination_parent = destination_parent.to_path_buf(); @@ -222,7 +249,7 @@ pub(super) async fn lock_rename_commit_directories( let publication_root = publication_root.clone(); os::run_blocking_namespace_operation(mutation_lease, move || { let result = os::prepare_rename_commit_guard(&source_parent, &destination_parent, &base_dir, &publication_root); - #[cfg(test)] + #[cfg(all(test, windows))] if result.is_ok() { run_destination_commit_directory_preparation(&destination_parent); } @@ -230,7 +257,7 @@ pub(super) async fn lock_rename_commit_directories( }) .await }; - #[cfg(not(windows))] + #[cfg(not(any(windows, target_os = "linux")))] let result = { let _ = mutation_lease; os::prepare_rename_commit_guard(source_parent, destination_parent, base_dir, publication_root) @@ -244,12 +271,12 @@ pub(super) async fn lock_rename_commit_directories( result.map_err(to_file_error).map_err(DiskError::from) } -async fn read_rename_destination_metadata( +async fn prepare_rename_destination_metadata( file_path: &Path, rename_commit_guard: &os::RenameCommitGuard, mutation_lease: Arc, ) -> Result> { - #[cfg(windows)] + #[cfg(any(windows, target_os = "linux"))] let result = { let file_path = file_path.to_path_buf(); let rename_commit_guard = rename_commit_guard.clone(); @@ -258,10 +285,10 @@ async fn read_rename_destination_metadata( }) .await }; - #[cfg(not(windows))] + #[cfg(not(any(windows, target_os = "linux")))] let _ = (rename_commit_guard, mutation_lease); - #[cfg(not(windows))] - let result = match super::super::fs::read_file(file_path).await { + #[cfg(not(any(windows, target_os = "linux")))] + let result = match fs::read(file_path).await { Ok(data) => Ok(Some(data)), Err(err) if err.kind() == ErrorKind::NotFound => Ok(None), Err(err) => Err(err), @@ -294,10 +321,12 @@ impl LocalDisk { async fn prepare_rename_data_admission( &self, fi: FileInfo, + source: (&str, &str), destination: (&str, &str), source_volume: Option<&str>, namespace_owner: Option>, ) -> Result { + let (src_volume, src_path) = source; let (dst_volume, dst_path) = destination; let mut fi = fi; // A non-force DeleteBucket must not remove a directory while a local @@ -326,6 +355,13 @@ impl LocalDisk { None => None, }; // Quota admission -> metadata RMW -> namespace/volume publication. + let no_inline = fi.data.is_none() && fi.size > 0; + let fi_healing = fi.is_healing(); + let staging_prefix = if fi_healing && !no_inline && src_volume == super::super::RUSTFS_META_TMP_BUCKET { + Some(tmp_staging_mutation_path(&self.root, src_path)?) + } else { + None + }; let metadata_lease = os::acquire_metadata_mutation_lease(&self.get_object_path(dst_volume, dst_path)?, namespace_owner).await; let mutation_lease = if let Some(source_volume) = source_volume { @@ -334,6 +370,7 @@ impl LocalDisk { source_volume, dst_volume, &destination_object_path, + staging_prefix.as_deref(), Some(metadata_lease), ) .await @@ -342,6 +379,7 @@ impl LocalDisk { &self.root, dst_volume, &destination_object_path, + staging_prefix.as_deref(), Some(metadata_lease), ) .await @@ -374,7 +412,7 @@ impl LocalDisk { #[cfg(not(target_os = "linux"))] let owner: Arc = Arc::new((permit, namespace_owner)); let prepared = self - .prepare_rename_data_admission(fi.clone(), destination, Some(source.0), Some(owner)) + .prepare_rename_data_admission(fi.clone(), source, destination, Some(source.0), Some(owner)) .await?; for volume in [Some(source.0), (source.0 != destination.0).then_some(destination.0)] .into_iter() @@ -435,9 +473,17 @@ impl LocalDisk { mutation_lease, } } else { - self.prepare_rename_data_admission(fi, (dst_volume, dst_path), None, state.namespace_owner.take()) - .await? + self.prepare_rename_data_admission( + fi, + (src_volume, src_path), + (dst_volume, dst_path), + None, + state.namespace_owner.take(), + ) + .await? }; + let no_inline = fi.data.is_none() && fi.size > 0; + let fi_healing = fi.is_healing(); // Snapshot the destination part paths before `fi` is consumed below. These // are the descriptors a reader may hold for the version this call is about // to replace (backlog#1145); readers build the identical string in @@ -493,7 +539,7 @@ impl LocalDisk { // xl.meta path let src_file_path = self.io_get_object_path(src_volume, format!("{}/{}", src_path, STORAGE_FORMAT_FILE).as_str())?; - let dst_file_path = self.io_get_object_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?; + let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?; // data_dir path let has_data_dir_path = { @@ -525,11 +571,6 @@ impl LocalDisk { check_path_length(src_file_path.to_string_lossy().to_string().as_str())?; check_path_length(dst_file_path.to_string_lossy().to_string().as_str())?; - let no_inline = fi.data.is_none() && fi.size > 0; - // Captured before `fi` is consumed by add_version; gates the stale - // destination purge below. - let fi_healing = fi.is_healing(); - // Resolved once for the whole commit so a concurrent configuration // change can never leave a single rename_data half-synced. The tier is // keyed on the destination volume: user data staged in scratch @@ -555,8 +596,7 @@ impl LocalDisk { .map_err(to_file_error)?; } // Acquire the common trees before reading destination metadata. On - // Windows this pins the object directory identity across metadata - // preparation, data publication, rollback backup, and final commit. + // Windows this pins the object directory identity across publication. let rename_commit_guard = lock_rename_commit_directories( src_file_parent, dst_file_parent, @@ -574,9 +614,12 @@ impl LocalDisk { source_volume_dir: &src_volume_dir, data_paths: has_data_dir_path.as_ref(), }; - let has_dst_buf = - read_rename_destination_metadata(commit.destination_metadata, &commit.directory_guard, commit.mutation_lease.clone()) - .await?; + let has_dst_buf = prepare_rename_destination_metadata( + commit.destination_metadata, + &commit.directory_guard, + commit.mutation_lease.clone(), + ) + .await?; if no_inline { // Non-inline: read xl.meta, parse, write, rename data dir, rename xl.meta @@ -610,6 +653,14 @@ impl LocalDisk { } let version_id = fi.version_id.unwrap_or_default(); + let (old_current_source_checked, old_current_source) = transitioned_old_current_source( + has_dst_buf.is_some(), + dst_meta_unparsable, + &xlmeta, + dst_volume, + dst_path, + version_id, + ); let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id)); let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok(); let rollback_data_dir = has_old_data_dir.or_else(|| { @@ -900,7 +951,9 @@ impl LocalDisk { && let Some(parent) = dst_file_path.parent() { let fsync_started = rustfs_io_metrics::put_stage_timer(); - if let Err(err) = os::fsync_dst_dir_group_commit(parent, Some(commit.mutation_lease.clone())).await { + if let Err(err) = + os::fsync_commit_directory(parent, &commit.directory_guard, commit.mutation_lease.clone(), None).await + { rustfs_io_metrics::record_put_object_stage_duration_from( rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC, fsync_started, @@ -999,6 +1052,8 @@ impl LocalDisk { cleanup_data_dir: has_old_data_dir, sign: version_signature, old_current_size, + old_current_source_checked, + old_current_source, }) } else { // Inline metadata preparation is blocking. The transaction lease is @@ -1012,6 +1067,8 @@ impl LocalDisk { None }; let dst_path_for_failpoint = dst_path.to_string(); + let dst_volume_for_source = dst_volume.to_string(); + let dst_path_for_source = dst_path.to_string(); #[cfg(windows)] let source_parent = src_file_parent.to_path_buf(); let rename_commit_guard_for_preparation = commit.directory_guard.clone(); @@ -1056,15 +1113,30 @@ impl LocalDisk { }; let version_id = fi.version_id.unwrap_or_default(); + let (old_current_source_checked, old_current_source) = transitioned_old_current_source( + has_dst_buf.is_some(), + dst_meta_unparsable, + &xlmeta, + &dst_volume_for_source, + &dst_path_for_source, + version_id, + ); let old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id)); - let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok(); - let rollback_data_dir = old_data_dir.or_else(|| { - if old_version_exists && has_dst_buf.is_some() { - Some(inline_metadata_rollback_dir(version_id, &xlmeta)) - } else { - None - } - }); + // Healing commits never run set-level post-return rollback; keep + // their intra-commit rollback backup in the staging directory so + // `//xl.meta.bkp` is never stranded. + let rollback_data_dir = if fi_healing { + None + } else { + let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok(); + old_data_dir.or_else(|| { + if old_version_exists && has_dst_buf.is_some() { + Some(inline_metadata_rollback_dir(version_id, &xlmeta)) + } else { + None + } + }) + }; let mut staged_rollback_path = None; if let Some(d) = old_data_dir.as_ref() { let _ = xlmeta.data.remove_two(version_id, *d); @@ -1080,7 +1152,7 @@ impl LocalDisk { prepared_metadata_source.write_all(&new_buf, sync)?; run_inline_preparation_before_backup(&dst_path_for_failpoint); if let Some(ref old_metadata) = has_dst_buf - && (rollback_data_dir.is_some() || sync || cfg!(test)) + && (rollback_data_dir.is_some() || fi_healing || sync || cfg!(test)) { #[cfg(windows)] let backup_path = { @@ -1105,6 +1177,8 @@ impl LocalDisk { old_data_dir, version_signature, old_current_size, + old_current_source_checked, + old_current_source, staged_rollback_path, has_dst_buf.is_none(), prepared_metadata_source, @@ -1124,6 +1198,8 @@ impl LocalDisk { cleanup_data_dir, version_signature, old_current_size, + old_current_source_checked, + old_current_source, mut local_rollback_path, destination_was_absent, prepared_metadata_source, @@ -1256,10 +1332,11 @@ impl LocalDisk { && let Some(dst_parent) = dst_file_path.parent() { let fsync_started = rustfs_io_metrics::put_stage_timer(); - if let Err(err) = os::fsync_dst_dir_group_commit_or_namespace_file_sync_limit( + if let Err(err) = os::fsync_commit_directory( dst_parent, + &commit.directory_guard, commit.mutation_lease.clone(), - admission, + Some(admission), ) .await { @@ -1371,6 +1448,8 @@ impl LocalDisk { cleanup_data_dir, sign: version_signature, old_current_size, + old_current_source_checked, + old_current_source, }) } } diff --git a/crates/ecstore/src/disk/local/delete_recovery.rs b/crates/ecstore/src/disk/local/delete_recovery.rs new file mode 100644 index 000000000..380d7a898 --- /dev/null +++ b/crates/ecstore/src/disk/local/delete_recovery.rs @@ -0,0 +1,4882 @@ +use super::*; +use crate::disk::{ + DELETE_RECOVERY_PROTOCOL_VERSION, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection, + DeleteRecoveryJournal, DeleteRecoveryPage, DeleteRecoveryState, DeleteTransactionId, +}; +use sha2::{Digest, Sha256}; + +pub(super) const PREPARED: &str = "delete-prepared"; +pub(super) const TARGET: &str = "delete-target"; +pub(super) const FINISHED: &str = "delete-commit-finished"; +const MAX_METADATA: u64 = 64 * 1024 * 1024; +struct DiscoverySession { + disk_root: PathBuf, + volume: String, + object: Option, + root: PathBuf, + _mount: std::fs::File, + frames: Vec<(String, std::fs::ReadDir)>, + touched: Instant, +} + +impl DiscoverySession { + /// Consume one bounded step, returning ownership only when traversal remains. + fn advance(mut self, limit: u32) -> std::io::Result<(Vec, Option)> { + if self.frames.is_empty() { + match std::fs::read_dir(&self.root) { + Ok(entries) => self.frames.push((String::new(), entries)), + Err(err) if err.kind() == ErrorKind::NotFound => { + return Ok((Vec::new(), None)); + } + Err(err) => return Err(err), + } + } + let mut transactions = BTreeSet::new(); + let mut visited = 0usize; + while !self.frames.is_empty() + && transactions.len() < usize::try_from(limit).map_err(|_| std::io::Error::from(ErrorKind::InvalidInput))? + && visited < 4096 + { + let (parent, entries) = self + .frames + .last_mut() + .ok_or_else(|| std::io::Error::from(ErrorKind::InvalidData))?; + let entry = match entries.next() { + Some(entry) => entry?, + None => { + self.frames.pop(); + continue; + } + }; + visited += 1; + let name = entry + .file_name() + .into_string() + .map_err(|_| std::io::Error::from(ErrorKind::InvalidData))?; + if !safe_path(&name, false) || name.contains('/') { + return Err(std::io::Error::from(ErrorKind::InvalidData)); + } + let relative = if parent.is_empty() { + name.clone() + } else { + format!("{parent}/{name}") + }; + let path = self.root.join(&relative); + reject_local_disk_symlink_components(&self.root, &path).map_err(std::io::Error::from)?; + let kind = match std::fs::symlink_metadata(&path) { + Ok(metadata) => metadata, + Err(err) if err.kind() == ErrorKind::NotFound => continue, + Err(err) => return Err(err), + }; + if !kind.is_dir() && !kind.is_file() { + return Err(std::io::Error::from(ErrorKind::InvalidData)); + } + if kind.is_file() { + let bound = match self.object.as_deref() { + Some(object) if !parent.is_empty() && !parent.contains('/') => Some((object, parent.as_str())), + Some(_) => None, + None => parent.rsplit_once('/'), + }; + if let Some((object, id)) = bound { + let transaction = + if [PREPARED, STORAGE_FORMAT_FILE_BACKUP, DELETE_QUORUM_COMMITTED_FILE].contains(&name.as_str()) { + Uuid::parse_str(id).ok().filter(|id| !id.is_nil()) + } else { + [ + DELETE_DATA_DIR_MARKER_PREFIX, + RESERVED_DELETE_DATA_DIR_MARKER_PREFIX, + RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX, + ] + .iter() + .find_map(|prefix| delete_transaction_marker_id(&name, prefix)) + }; + if let Some(transaction) = transaction { + let physical = match self.object.as_ref() { + Some(_) => self.root.clone(), + None => self.root.join(object), + }; + if name == STORAGE_FORMAT_FILE_BACKUP + && live_write_backup(&physical, transaction).map_err(std::io::Error::from)? + { + continue; + } + let proof_root = physical.join(transaction.to_string()); + let prepared = regular_bytes(&proof_root.join(PREPARED)).map_err(std::io::Error::from)?; + let receipt = + regular_bytes(&proof_root.join(DELETE_QUORUM_COMMITTED_FILE)).map_err(std::io::Error::from)?; + // Only a fully bound journal can name a logical + // key. Other records remain discoverable by their + // physical key for canonical inspection to reject. + let record = match prepared { + Some(bytes) => serde_json::from_slice::(&bytes).ok(), + None => receipt + .filter(|bytes| !bytes.is_empty()) + .and_then(|bytes| serde_json::from_slice::(&bytes).ok()) + .map(|receipt| receipt.journal), + }; + let logical = record + .filter(|journal| { + journal.protocol_version == DELETE_RECOVERY_PROTOCOL_VERSION + && journal.volume == self.volume + && journal.transaction == transaction + && safe_path(&journal.object, false) + && match self.object.as_ref() { + Some(scope) => scope == &journal.object, + None => build_local_disk_object_path(&self.root, "", &journal.object).1 == physical, + } + }) + .map_or_else(|| object.to_owned(), |journal| journal.object); + transactions.insert((logical, transaction)); + } + } + } else { + if self.object.is_some() && (!parent.is_empty() || !Uuid::parse_str(&name).is_ok_and(|id| !id.is_nil())) { + continue; + } + // A valid 1024-byte key can exceed 256 one-byte path + // components; allow its transaction/data directories too. + if self.frames.len() >= 1024 { + return Err(std::io::Error::from(ErrorKind::InvalidData)); + } + self.frames.push((relative, std::fs::read_dir(path)?)); + } + } + + let transactions = transactions + .into_iter() + .map(|(object, transaction)| DeleteTransactionId { object, transaction }) + .collect(); + let next = (!self.frames.is_empty()).then_some(self); + Ok((transactions, next)) + } + + async fn has_evidence(mut self, disk: &LocalDisk, path: &Path) -> Result { + loop { + let (transactions, next) = disk.advance_delete_discovery(path, self, 1).await?; + if !transactions.is_empty() { + return Ok(true); + } + let Some(mut next) = next else { + return Ok(false); + }; + next.touched = Instant::now(); + self = next; + } + } +} + +static DISCOVERY_SESSIONS: std::sync::LazyLock>> = + std::sync::LazyLock::new(|| ParkingLotMutex::new(HashMap::new())); + +#[derive(serde::Serialize, serde::Deserialize)] +#[serde(deny_unknown_fields)] +struct CommitReceipt { + journal: DeleteRecoveryJournal, + finished: bool, +} + +pub(super) struct DeletePreparationContext { + pub transaction: Uuid, + pub bucket_incarnation: Option, + pub namespace_owner: Option>, + pub prepared_publication: Option>, +} + +pub(super) fn digest(bytes: &[u8]) -> DeleteRecoveryDigest { + Sha256::digest(bytes).into() +} + +pub(in crate::disk) fn generation(mut canonical: FileInfo) -> Result { + canonical.data = None; + canonical.erasure.index = 0; + canonical.erasure.checksums.clear(); + canonical.is_latest = false; + canonical.num_versions = 0; + canonical.successor_mod_time = None; + rustfs_filemeta::shard_integrity::clear_integrity_metadata(&mut canonical.metadata); + // Only schema-declared string fields and the inline presence marker are + // aliases here. Unknown suffixes and opaque binary fields stay distinct. + use rustfs_utils::http::metadata_compat as compat; + for suffix in [ + compat::SUFFIX_INLINE_DATA, + compat::SUFFIX_COMPRESSION, + compat::SUFFIX_COMPRESSION_SIZE, + compat::SUFFIX_ACTUAL_SIZE, + compat::SUFFIX_ACTUAL_OBJECT_SIZE, + compat::SUFFIX_ACTUAL_OBJECT_SIZE_CAP, + compat::SUFFIX_TRANSITION_STATUS, + compat::SUFFIX_TRANSITIONED_OBJECTNAME, + compat::SUFFIX_TRANSITION_TIER, + compat::SUFFIX_TRANSITIONED_VERSION_STATE, + compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID, + compat::SUFFIX_PURGESTATUS, + compat::SUFFIX_REPLICA_STATUS, + compat::SUFFIX_REPLICATION_STATUS, + compat::SUFFIX_REPLICA_TIMESTAMP, + compat::SUFFIX_REPLICATION_TIMESTAMP, + compat::SUFFIX_TAGGING_TIMESTAMP, + compat::SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, + compat::SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, + ] { + if !compat::contains_key_str(&canonical.metadata, suffix) { + continue; + } + // The inline marker is a presence flag whose persisted value is empty. + // Compare both declared aliases explicitly before normalizing it; the + // general compatibility reader rejects empty strings by design. + let value = if suffix == compat::SUFFIX_INLINE_DATA { + let rustfs_key = compat::internal_key_rustfs(suffix); + let minio_key = format!("{}{suffix}", compat::MINIO_INTERNAL_PREFIX); + let mut value = None; + for (key, candidate) in &canonical.metadata { + if !key.eq_ignore_ascii_case(&rustfs_key) && !key.eq_ignore_ascii_case(&minio_key) { + continue; + } + if value.is_some_and(|current: &String| current != candidate) { + return Err(DiskError::FileCorrupt); + } + value = Some(candidate); + } + value.ok_or(DiskError::FileCorrupt)?.clone() + } else { + compat::get_consistent_str(&canonical.metadata, suffix) + .ok_or(DiskError::FileCorrupt)? + .to_owned() + }; + if value.is_empty() && suffix != compat::SUFFIX_INLINE_DATA { + continue; + } + compat::insert_str(&mut canonical.metadata, suffix, value); + } + fn ordered(value: serde_json::Value) -> serde_json::Value { + match value { + serde_json::Value::Object(map) => { + let sorted: std::collections::BTreeMap<_, _> = map.into_iter().collect(); + serde_json::Value::Object(sorted.into_iter().map(|(key, value)| (key, ordered(value))).collect()) + } + serde_json::Value::Array(values) => serde_json::Value::Array(values.into_iter().map(ordered).collect()), + other => other, + } + } + Ok(crate::disk::DeleteRecoveryVersionIdentity { + version_id: canonical.version_id, + mod_time: canonical.mod_time, + data_dir: canonical.data_dir, + deleted: canonical.deleted, + metadata_digest: digest(&serde_json::to_vec(&ordered(serde_json::to_value(canonical)?))?), + }) +} + +fn safe_path(path: &str, empty: bool) -> bool { + (empty && path.is_empty()) + || (!path.is_empty() && !path.starts_with('/') && crate::bucket::utils::is_valid_object_prefix(path)) +} + +fn regular_metadata_bytes(path: &Path) -> Result>> { + match std::fs::symlink_metadata(path) { + Ok(meta) if meta.is_file() => Ok(Some(std::fs::read(path).map_err(to_file_error)?)), + Ok(_) => Err(DiskError::FileCorrupt), + Err(err) if err.kind() == ErrorKind::NotFound => Ok(None), + Err(err) => Err(to_file_error(err).into()), + } +} + +fn regular_bytes(path: &Path) -> Result>> { + if std::fs::symlink_metadata(path).is_ok_and(|meta| meta.len() > MAX_METADATA) { + return Err(DiskError::FileCorrupt); + } + regular_metadata_bytes(path) +} + +/// A write/heal backup is not a delete transaction when the live metadata +/// still owns its directory and retains every backed-up generation. Any delete +/// evidence wins; uncertain backups remain discoverable and fail closed. +fn live_write_backup(root: &Path, transaction: Uuid) -> Result { + let rollback = root.join(transaction.to_string()); + for name in [ + PREPARED, + TARGET, + FINISHED, + DELETE_MARKER_ROLLBACK_FILE, + DELETE_QUORUM_COMMITTED_FILE, + ] { + if regular_bytes(&rollback.join(name))?.is_some() { + return Ok(false); + } + } + for prefix in [ + DELETE_DATA_DIR_MARKER_PREFIX, + RESERVED_DELETE_DATA_DIR_MARKER_PREFIX, + RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX, + ] { + let name = format!("{prefix}{transaction}"); + if regular_bytes(&root.join(&name))?.is_some() || regular_bytes(&rollback.join(name))?.is_some() { + return Ok(false); + } + } + let Some(current) = regular_metadata_bytes(&root.join(STORAGE_FORMAT_FILE))? else { + return Ok(false); + }; + let Some(backup) = regular_metadata_bytes(&rollback.join(STORAGE_FORMAT_FILE_BACKUP))? else { + return Ok(false); + }; + // Corrupt decoded metadata cannot prove that this is a live write backup. + // Keep it discoverable; filesystem and structural errors above still fail. + match (|| live_write_backup_matches(&FileMeta::load(¤t)?, &FileMeta::load(&backup)?, transaction))() { + Err(DiskError::FileCorrupt) => Ok(false), + result => result, + } +} + +fn live_write_backup_matches(current: &FileMeta, backup: &FileMeta, transaction: Uuid) -> Result { + let mut owns_directory = false; + let used_data_dirs: HashSet<_> = current.get_data_dirs()?.into_iter().flatten().collect(); + let mut live = HashMap::with_capacity(current.versions.len()); + for version in ¤t.versions { + let mut info = version.into_fileinfo("", "", true)?; + let id = info.version_id.unwrap_or_default(); + owns_directory |= + info.data_dir == Some(transaction) || inline_metadata_rollback_dir_with_used_dirs(id, &used_data_dirs) == transaction; + rustfs_utils::http::metadata_compat::remove_str(&mut info.metadata, rustfs_utils::http::metadata_compat::SUFFIX_HEALING); + if live.insert(id, generation(info)?).is_some() { + return Err(DiskError::FileCorrupt); + } + } + if !owns_directory || backup.versions.is_empty() { + return Ok(false); + } + let mut seen = HashSet::with_capacity(backup.versions.len()); + for version in &backup.versions { + let mut info = version.into_fileinfo("", "", true)?; + let id = info.version_id.unwrap_or_default(); + if !seen.insert(id) { + return Err(DiskError::FileCorrupt); + } + rustfs_utils::http::metadata_compat::remove_str(&mut info.metadata, rustfs_utils::http::metadata_compat::SUFFIX_HEALING); + if live.get(&id) != Some(&generation(info)?) { + return Ok(false); + } + } + Ok(true) +} + +fn journal_bytes(journal: &DeleteRecoveryJournal) -> Result> { + let bytes = serde_json::to_vec(journal)?; + // The same journal is carried by adoption requests and decision receipts. + if bytes.len() > usize::try_from(MAX_METADATA).map_err(|_| DiskError::FileCorrupt)? - 128 { + return Err(DiskError::FileCorrupt); + } + let projected = crate::disk::DeleteRecoveryRpcReply::Inspected(DeleteRecoveryInspection { + protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION, + id: DeleteTransactionId { + object: journal.object.clone(), + transaction: journal.transaction, + }, + state: DeleteRecoveryState::NewPrepared, + journal: Some(journal.clone()), + current_digest: journal.original_digest, + current_versions: journal.original_versions.clone(), + backup_digest: journal.original_digest, + rollback_data_complete: true, + commit_data_safe: true, + explicit_delete_safe: false, + fingerprint: [0; 32], + }); + if rmp_serde::to_vec_named(&projected)?.len() > rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES - 4096 { + return Err(DiskError::FileCorrupt); + } + Ok(bytes) +} + +fn regular_marker(path: &Path) -> Result { + match std::fs::symlink_metadata(path) { + Ok(meta) if meta.is_file() && meta.len() == 0 => Ok(true), + Ok(_) => Err(DiskError::FileCorrupt), + Err(err) if err.kind() == ErrorKind::NotFound => Ok(false), + Err(err) => Err(to_file_error(err).into()), + } +} + +fn complete_part_sources(live: &Path, staged: &Path, info: &FileInfo) -> Result { + let layout = info.validate(ValidationMode::RequireErasure)?.ok_or(DiskError::FileCorrupt)?; + for part in &info.parts { + let expected = u64::try_from(layout.shard_file_size(part.size).ok_or(DiskError::FileCorrupt)?) + .map_err(|_| DiskError::FileCorrupt)?; + let name = format!("part.{}", part.number); + if ![live, staged] + .iter() + .any(|root| std::fs::symlink_metadata(root.join(&name)).is_ok_and(|meta| meta.is_file() && meta.len() >= expected)) + { + return Ok(false); + } + if let Some(integrity) = &part.integrity { + let name = integrity.file_name(); + let length = u64::try_from(integrity.index_size()?).map_err(|_| DiskError::FileCorrupt)?; + if ![live, staged] + .iter() + .any(|root| std::fs::symlink_metadata(root.join(&name)).is_ok_and(|meta| meta.is_file() && meta.len() == length)) + { + return Ok(false); + } + } + } + Ok(true) +} + +async fn verify_recovery_integrity_sources( + live: &Path, + staged: &Path, + integrity: &rustfs_filemeta::shard_integrity::PartIntegrity, + coding_index: usize, + algorithm: HashAlgorithm, + owner: Option>, +) -> Result<()> { + let lease = os::acquire_namespace_mutation_lease_with_owner(live, owner).await; + let roots = [live.to_path_buf(), staged.to_path_buf()]; + let integrity = integrity.clone(); + os::run_blocking_namespace_operation(lease, move || { + Ok((|| { + integrity.validate()?; + if coding_index >= integrity.layout.shards() + || !matches!(algorithm, HashAlgorithm::HighwayHash256S | HashAlgorithm::HighwayHash256SLegacy) + { + return Err(DiskError::FileCorrupt); + } + let codec = coding::Erasure::try_new_with_options( + usize::from(integrity.layout.data), + usize::from(integrity.layout.parity), + usize::try_from(integrity.layout.block_size).map_err(|_| DiskError::FileCorrupt)?, + integrity.layout.legacy, + ) + .map_err(DiskError::from)?; + let shard_size = codec.shard_size(); + let part_size = + usize::try_from(codec.shard_file_size(i64::try_from(integrity.size).map_err(|_| DiskError::FileCorrupt)?)) + .map_err(|_| DiskError::FileCorrupt)?; + let encoded_size = u64::try_from(coding::bitrot_shard_file_size(part_size, shard_size, algorithm.clone())) + .map_err(|_| DiskError::FileCorrupt)?; + let expected = u64::try_from(integrity.index_size()?).map_err(|_| DiskError::FileCorrupt)?; + let header = integrity.index_header(); + let mut checked = false; + for root in &roots { + let path = root.join(integrity.file_name()); + match std::fs::symlink_metadata(&path) { + Ok(metadata) if metadata.is_file() && metadata.len() == expected => {} + Ok(_) => return Err(DiskError::FileCorrupt), + Err(err) if err.kind() == ErrorKind::NotFound => continue, + Err(err) => return Err(to_file_error(err).into()), + } + let mut file = std::fs::File::open(path).map_err(to_file_error)?; + let mut actual = [0u8; rustfs_filemeta::shard_integrity::INDEX_HEADER_SIZE]; + std::io::Read::read_exact(&mut file, &mut actual).map_err(to_file_error)?; + if actual != header { + return Err(DiskError::FileCorrupt); + } + // An interrupted restore can split payload and proof between + // live and staged roots. Verify every present payload copy + // against the declared proof, independently of its location. + let mut shards = Vec::new(); + for root in &roots { + let path = root.join(format!("part.{}", integrity.number)); + match std::fs::symlink_metadata(&path) { + Ok(metadata) if metadata.is_file() && metadata.len() == encoded_size => { + shards.push(std::fs::File::open(path).map_err(to_file_error)?); + } + Ok(_) => return Err(DiskError::FileCorrupt), + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + } + if shards.is_empty() { + return Err(DiskError::FileCorrupt); + } + let mut record = vec![0; integrity.record_size()]; + let mut payload = vec![0; shard_size]; + let mut bitrot_hash = vec![0; algorithm.size()]; + let mut remaining = part_size; + for stripe in 0..integrity.stripes { + std::io::Read::read_exact(&mut file, &mut record).map_err(to_file_error)?; + integrity.verify_record(stripe, &record)?; + let length = remaining.min(shard_size); + if length == 0 { + return Err(DiskError::FileCorrupt); + } + for shard in &mut shards { + std::io::Read::read_exact(shard, &mut bitrot_hash).map_err(to_file_error)?; + std::io::Read::read_exact(shard, &mut payload[..length]).map_err(to_file_error)?; + integrity.verify_shard(stripe, coding_index, &payload[..length], &record)?; + } + remaining -= length; + } + if remaining != 0 { + return Err(DiskError::FileCorrupt); + } + if integrity.stripes == 0 && integrity.root != integrity.root_digest(&integrity.padding_digest(0)) { + return Err(DiskError::FileCorrupt); + } + checked = true; + } + if !checked { + return Err(DiskError::FileCorrupt); + } + Ok(()) + })()) + }) + .await + .map_err(to_file_error)? +} + +struct ExplicitDeletePlan { + data_dirs: Vec, + proof_files: Vec, +} + +fn explicit_delete_plan( + root: &Path, + transaction: Uuid, + original: Option<&FileMeta>, + journal: Option<&DeleteRecoveryJournal>, + current_missing: bool, +) -> Result> { + if !current_missing + || journal + .is_some_and(|journal| journal.target_digest.is_some() || (journal.original_digest.is_some() && original.is_none())) + { + return Ok(None); + } + let mut referenced = HashSet::new(); + if let Some(metadata) = original { + if metadata.versions.len() != 1 || metadata.versions[0].header.version_type != rustfs_filemeta::VersionType::Object { + return Ok(None); + } + let info = metadata.versions[0].into_fileinfo("", "", false)?; + if info.version_id.is_some_and(|id| !id.is_nil()) + || info.deleted + || info.is_remote() + || info.metadata.keys().any(|key| { + let key = key.to_ascii_lowercase(); + key.starts_with("x-amz-object-lock-") + || rustfs_utils::http::metadata_compat::internal_key_strip_suffix_prefix(&key, "objectlock-").is_some() + }) + { + return Ok(None); + } + referenced.extend(metadata.get_data_dirs()?.into_iter().flatten().filter(|id| !id.is_nil())); + } + let rollback = root.join(transaction.to_string()); + let marker_names = [ + DELETE_DATA_DIR_MARKER_PREFIX, + RESERVED_DELETE_DATA_DIR_MARKER_PREFIX, + RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX, + ] + .map(|prefix| format!("{prefix}{transaction}")); + let mut data_dirs = Vec::new(); + let mut proof_files = Vec::new(); + let mut candidates = Vec::new(); + match std::fs::read_dir(&rollback) { + Ok(entries) => { + for entry in entries { + let entry = entry.map_err(to_file_error)?; + let kind = entry.file_type().map_err(to_file_error)?; + let name = entry.file_name().into_string().map_err(|_| DiskError::FileCorrupt)?; + if kind.is_dir() && Uuid::parse_str(&name).is_ok_and(|id| referenced.contains(&id)) { + candidates.push(entry.path()); + } else if kind.is_file() + && [ + STORAGE_FORMAT_FILE_BACKUP, + DELETE_MARKER_ROLLBACK_FILE, + DELETE_QUORUM_COMMITTED_FILE, + PREPARED, + TARGET, + FINISHED, + ] + .contains(&name.as_str()) + { + proof_files.push(entry.path()); + } else if kind.is_file() + && (marker_names.contains(&name) || (delete_recovery_part_file(&name) && referenced.contains(&transaction))) + { + candidates.push(rollback.clone()); + } else { + return Ok(None); + } + } + } + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + match std::fs::read_dir(root) { + Ok(entries) => { + for entry in entries { + let entry = entry.map_err(to_file_error)?; + let name = entry.file_name().into_string().map_err(|_| DiskError::FileCorrupt)?; + let Some(id) = Uuid::parse_str(&name).ok().filter(|id| !id.is_nil()) else { + continue; + }; + if !entry.file_type().map_err(to_file_error)?.is_dir() { + continue; + } + let own_marker = marker_names + .iter() + .any(|name| std::fs::symlink_metadata(entry.path().join(name)).is_ok_and(|metadata| metadata.is_file())); + if referenced.contains(&id) || own_marker { + candidates.push(entry.path()); + } + } + } + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + candidates.sort(); + candidates.dedup(); + for dir in candidates { + let mut has_parts = false; + for entry in std::fs::read_dir(&dir).map_err(to_file_error)? { + let entry = entry.map_err(to_file_error)?; + if !entry.file_type().map_err(to_file_error)?.is_file() { + return Ok(None); + } + let name = entry.file_name().into_string().map_err(|_| DiskError::FileCorrupt)?; + if delete_recovery_part_file(&name) { + has_parts = true; + } else if !(marker_names.contains(&name) + || (dir == rollback + && [ + STORAGE_FORMAT_FILE_BACKUP, + DELETE_MARKER_ROLLBACK_FILE, + DELETE_QUORUM_COMMITTED_FILE, + PREPARED, + TARGET, + FINISHED, + ] + .contains(&name.as_str()))) + { + return Ok(None); + } + } + if has_parts && original.is_none() { + return Ok(None); + } + data_dirs.push(dir); + } + data_dirs.sort_by_key(|path| std::cmp::Reverse(path.components().count())); + Ok(Some(ExplicitDeletePlan { data_dirs, proof_files })) +} + +fn inspection(root: &Path, volume: &str, object: &str, transaction: Uuid) -> Result { + let rollback = root.join(transaction.to_string()); + let before = regular_metadata_bytes(&rollback.join(STORAGE_FORMAT_FILE_BACKUP))?; + let current = regular_metadata_bytes(&root.join(STORAGE_FORMAT_FILE))?; + let prepared = regular_bytes(&rollback.join(PREPARED))?; + let target = regular_metadata_bytes(&rollback.join(TARGET))?; + let receipt_bytes = regular_bytes(&rollback.join(DELETE_QUORUM_COMMITTED_FILE))?; + let receipt: Option = receipt_bytes + .as_deref() + .filter(|bytes| !bytes.is_empty()) + .map(serde_json::from_slice) + .transpose() + .map_err(|_| DiskError::FileCorrupt)?; + let committed = receipt.is_some(); + let finished = regular_marker(&rollback.join(FINISHED))? || receipt.as_ref().is_some_and(|receipt| receipt.finished); + let journal: Option = prepared + .as_deref() + .map(serde_json::from_slice) + .transpose() + .map_err(|_| DiskError::FileCorrupt)? + .or_else(|| receipt.as_ref().map(|receipt| receipt.journal.clone())); + if receipt + .as_ref() + .is_some_and(|receipt| journal.as_ref() != Some(&receipt.journal)) + { + return Err(DiskError::FileCorrupt); + } + if let Some(journal) = &journal + && (journal.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION + || journal.volume != volume + || journal.object != object + || journal.transaction != transaction + || journal.transaction.is_nil() + || journal.data_dirs.iter().any(|id| id.is_nil() || *id == transaction)) + { + return Err(DiskError::FileCorrupt); + } + let backup_digest = before.as_deref().map(digest); + let current_digest = current.as_deref().map(digest); + let prepared_digest = prepared.as_deref().map(digest); + let target_digest = target.as_deref().map(digest); + let receipt_digest = receipt_bytes.as_deref().map(digest); + if let Some(journal) = &journal { + if (!finished + && journal.target_digest != target_digest + && !(target.is_none() + && !committed + && before.is_none() + && journal.original_digest.is_some() + && current_digest == journal.original_digest)) + || target_digest.is_some_and(|hash| journal.target_digest != Some(hash)) + || backup_digest.is_some_and(|hash| journal.original_digest != Some(hash)) + { + return Err(DiskError::FileCorrupt); + } + if let Some(bytes) = &target { + FileMeta::load(bytes)?; + } + } + let current_meta = current.as_deref().map(FileMeta::load).transpose()?; + let before_meta = before.as_deref().map(FileMeta::load).transpose()?; + let mut fingerprint = Sha256::new(); + for hash in [prepared_digest, target_digest, backup_digest, current_digest, receipt_digest] { + fingerprint.update([u8::from(hash.is_some())]); + if let Some(hash) = hash { + fingerprint.update(hash); + } + } + fingerprint.update([u8::from(committed), u8::from(finished)]); + let mut any_marker = false; + let mut descriptions = Vec::new(); + let entries = match std::fs::read_dir(root) { + Ok(entries) => Some(entries), + Err(err) if err.kind() == ErrorKind::NotFound => None, + Err(err) => return Err(to_file_error(err).into()), + }; + if let Some(entries) = entries { + for entry in entries { + let entry = entry.map_err(to_file_error)?; + let kind = entry.file_type().map_err(to_file_error)?; + if !kind.is_file() && !kind.is_dir() { + return Err(DiskError::InvalidPath); + } + let name = entry.file_name().into_string().map_err(|_| DiskError::FileCorrupt)?; + descriptions.push((name.clone(), kind.is_dir(), entry.metadata().map_err(to_file_error)?.len())); + if kind.is_dir() && Uuid::parse_str(&name).is_ok_and(|id| !id.is_nil()) { + for child in std::fs::read_dir(entry.path()).map_err(to_file_error)? { + let child = child.map_err(to_file_error)?; + let kind = child.file_type().map_err(to_file_error)?; + if !kind.is_dir() && !kind.is_file() { + return Err(DiskError::InvalidPath); + } + let child_name = child.file_name().into_string().map_err(|_| DiskError::FileCorrupt)?; + let metadata = child.metadata().map_err(to_file_error)?; + descriptions.push((format!("{name}/{child_name}"), kind.is_dir(), metadata.len())); + if kind.is_file() + && [ + DELETE_DATA_DIR_MARKER_PREFIX, + RESERVED_DELETE_DATA_DIR_MARKER_PREFIX, + RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX, + ] + .iter() + .any(|prefix| child_name == format!("{prefix}{transaction}")) + { + any_marker = true; + } + let staged_id = Uuid::parse_str(&child_name).ok(); + let known_staged = kind.is_dir() + && staged_id.is_some_and(|dir| { + journal.as_ref().is_some_and(|journal| journal.data_dirs.contains(&dir)) + || before_meta + .as_ref() + .is_some_and(|metadata| metadata.get_data_dirs().is_ok_and(|dirs| dirs.contains(&Some(dir)))) + }); + if name == transaction.to_string() + && !(kind.is_dir() && known_staged) + && !(journal.is_none() + && kind.is_file() + && (delete_recovery_part_file(&child_name) + || [DELETE_DATA_DIR_MARKER_PREFIX, RESERVED_DELETE_DATA_DIR_MARKER_PREFIX] + .iter() + .any(|prefix| child_name == format!("{prefix}{transaction}")))) + && (kind.is_dir() + || ![ + PREPARED, + TARGET, + FINISHED, + STORAGE_FORMAT_FILE_BACKUP, + DELETE_MARKER_ROLLBACK_FILE, + DELETE_QUORUM_COMMITTED_FILE, + ] + .contains(&child_name.as_str())) + { + return Err(DiskError::FileCorrupt); + } + if name == transaction.to_string() && kind.is_dir() { + for file in std::fs::read_dir(child.path()).map_err(to_file_error)? { + let file = file.map_err(to_file_error)?; + let kind = file.file_type().map_err(to_file_error)?; + let file_name = file.file_name().into_string().map_err(|_| DiskError::FileCorrupt)?; + if !kind.is_file() + || (!delete_recovery_part_file(&file_name) + && ![ + DELETE_DATA_DIR_MARKER_PREFIX, + RESERVED_DELETE_DATA_DIR_MARKER_PREFIX, + RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX, + ] + .iter() + .any(|prefix| file_name == format!("{prefix}{transaction}"))) + { + return Err(DiskError::FileCorrupt); + } + descriptions.push(( + format!("{name}/{child_name}/{file_name}"), + false, + file.metadata().map_err(to_file_error)?.len(), + )); + } + } + } + } + } + } + descriptions.sort(); + fingerprint.update(serde_json::to_vec(&descriptions)?); + let mut rollback_data_complete = false; + let mut commit_data_safe = false; + if let Some(journal) = &journal { + rollback_data_complete = (journal.original_digest == backup_digest || current_digest == journal.original_digest) + && (current_digest == journal.original_digest || current_digest == journal.target_digest); + if let Some(meta) = before_meta + .as_ref() + .or_else(|| current_meta.as_ref().filter(|_| current_digest == journal.original_digest)) + { + let inline: HashMap<&str, &[u8]> = if meta.data.as_slice().is_empty() { + HashMap::new() + } else { + if !meta.data.version_ok() { + return Err(DiskError::FileCorrupt); + } + rmp_serde::from_slice(meta.data.after_version())? + }; + for version in &meta.versions { + if version.header.version_type != rustfs_filemeta::VersionType::Object { + continue; + } + let mut info = version.into_fileinfo(volume, object, true)?; + info.hydrate_shard_integrity()?; + if info.is_remote() { + continue; + } + if info.inline_data() { + let id = info.version_id.unwrap_or_default(); + rollback_data_complete &= + inline.contains_key(id.to_string().as_str()) || (id.is_nil() && inline.contains_key("null")); + continue; + } + let Some(data_dir) = info.data_dir.filter(|id| !id.is_nil()) else { + rollback_data_complete = false; + continue; + }; + rollback_data_complete &= + complete_part_sources(&root.join(data_dir.to_string()), &rollback.join(data_dir.to_string()), &info)?; + } + } + commit_data_safe = current_digest == journal.original_digest || current_digest == journal.target_digest; + if current_digest != journal.original_digest && current_digest != journal.target_digest { + // A later certified delete can remove the remaining metadata of + // this committed version delete. Absence has no live references, + // but cannot itself authorize an undecided transaction. + commit_data_safe = (committed && current.is_none()) + || current_meta.as_ref().is_some_and(|meta| { + meta.get_data_dirs() + .is_ok_and(|dirs| !dirs.into_iter().flatten().any(|dir| journal.data_dirs.contains(&dir))) + && !meta.versions.iter().any(|version| { + version + .header + .version_id + .is_some_and(|id| !id.is_nil() && journal.deleted_versions.contains(&id)) + }) + }); + } + } + // In-place shard repair leaves a backup in the live data directory. + // Preserve it without interpreting it as an unfinished delete. + let live_write_backup = journal.is_none() + && !any_marker + && !regular_marker(&rollback.join(DELETE_MARKER_ROLLBACK_FILE))? + && prepared.is_none() + && target.is_none() + && receipt_bytes.is_none() + && !finished + && match (current_meta.as_ref(), before_meta.as_ref()) { + (Some(current), Some(backup)) => live_write_backup_matches(current, backup, transaction)?, + _ => false, + }; + let state = if live_write_backup { + DeleteRecoveryState::Absent + } else if journal.is_some() { + if committed { + if finished { + DeleteRecoveryState::CommitFinished + } else { + DeleteRecoveryState::QuorumCommitted + } + } else { + DeleteRecoveryState::NewPrepared + } + } else if before.is_some() || any_marker || receipt_bytes.is_some() || prepared.is_some() || target.is_some() { + DeleteRecoveryState::LegacyAmbiguous + } else { + DeleteRecoveryState::Absent + }; + let explicit_delete_safe = + explicit_delete_plan(root, transaction, before_meta.as_ref(), journal.as_ref(), current.is_none())?.is_some(); + Ok(DeleteRecoveryInspection { + protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION, + id: DeleteTransactionId { + object: object.to_owned(), + transaction, + }, + state, + journal, + current_digest, + backup_digest, + rollback_data_complete, + commit_data_safe, + explicit_delete_safe, + current_versions: current_meta + .as_ref() + .map(|meta| { + meta.versions + .iter() + .map(|version| generation(version.into_fileinfo(volume, object, true)?)) + .collect::>>() + }) + .transpose()? + .unwrap_or_default(), + fingerprint: fingerprint.finalize().into(), + }) +} + +impl LocalDisk { + async fn delete_recovery_predecessors( + &self, + volume: &str, + object: &str, + transaction: Uuid, + incarnation: Option, + owner: Option>, + ) -> Result> { + let root = self.io_get_object_path(volume, object)?; + let volume = volume.to_owned(); + let object = object.to_owned(); + let lease = os::acquire_namespace_mutation_lease_with_owner(&root, owner).await; + os::run_blocking_namespace_operation(lease, move || { + Ok((|| { + let entries = match std::fs::read_dir(&root) { + Ok(entries) => entries, + Err(err) if err.kind() == ErrorKind::NotFound => return Ok(Vec::new()), + Err(err) => return Err(to_file_error(err).into()), + }; + let mut predecessors = BTreeSet::new(); + let mut visited = 0usize; + for entry in entries { + visited += 1; + if visited > 65536 { + return Err(DiskError::DiskOngoingReq); + } + let entry = entry.map_err(to_file_error)?; + let Some(id) = entry + .file_name() + .to_str() + .and_then(|name| Uuid::parse_str(name).ok()) + .filter(|id| !id.is_nil() && *id != transaction) + else { + continue; + }; + if !entry.file_type().map_err(to_file_error)?.is_dir() { + return Err(DiskError::InvalidPath); + } + let prepared = regular_bytes(&entry.path().join(PREPARED))?; + let receipt = regular_bytes(&entry.path().join(DELETE_QUORUM_COMMITTED_FILE))?; + let mut records = Vec::new(); + if let Some(bytes) = prepared { + records + .push(serde_json::from_slice::(&bytes).map_err(|_| DiskError::FileCorrupt)?); + } + if let Some(bytes) = receipt.filter(|bytes| !bytes.is_empty()) { + records.push( + serde_json::from_slice::(&bytes) + .map_err(|_| DiskError::FileCorrupt)? + .journal, + ); + } + for record in records { + if record.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION + || record.volume != volume + || record.object != object + || record.transaction != id + || (incarnation.is_some() + && record.bucket_incarnation.is_some() + && record.bucket_incarnation != incarnation) + || record.predecessors.iter().any(|before| before.is_nil() || *before == id) + { + return Err(DiskError::FileCorrupt); + } + predecessors.insert(id); + } + } + Ok(predecessors.into_iter().collect()) + })()) + }) + .await + .map_err(to_file_error)? + } + /// Local maintenance may pause a page behind recovery backpressure. It + /// retains only its own bounded set of cursors; remote callers still have + /// the ordinary idle deadline and every resumed page rechecks mount identity. + pub(crate) fn renew_delete_recovery_cursor(&self, volume: &str, cursor: &DeleteRecoveryCursor) -> bool { + let mut sessions = DISCOVERY_SESSIONS.lock(); + let Some(session) = sessions.get_mut(&cursor.token) else { + return false; + }; + if session.disk_root != self.io_root() + || session.volume != volume + || session.object.is_some() + || session.touched.elapsed() > Duration::from_secs(300) + { + return false; + } + session.touched = Instant::now(); + true + } + + fn delete_discovery_path(&self, volume: &str, object: Option<&str>) -> Result { + if !safe_path(volume, false) || volume.contains('/') || object.is_some_and(|object| !safe_path(object, false)) { + return Err(DiskError::InvalidPath); + } + match object { + Some(object) => self.io_get_object_path(volume, object), + None => self.io_get_bucket_path(volume), + } + } + + fn new_delete_discovery_session(&self, volume: &str, object: Option<&str>, path: &Path) -> Result { + #[cfg(target_os = "linux")] + let (mount, root) = { + use std::os::fd::AsRawFd; + let mount = self.mount_lease.try_clone().map_err(to_file_error)?; + let root = PathBuf::from(format!("/proc/self/fd/{}/.", mount.as_raw_fd())).join(volume); + let root = match object { + Some(object) => root.join(object), + None => root, + }; + let _ = path; + (mount, root) + }; + #[cfg(windows)] + let (mount, root) = { + use std::os::windows::fs::OpenOptionsExt; + use windows_sys::Win32::Storage::FileSystem::{ + FILE_FLAG_BACKUP_SEMANTICS, FILE_READ_ATTRIBUTES, FILE_SHARE_READ, FILE_SHARE_WRITE, FILE_TRAVERSE, + }; + let mount = std::fs::OpenOptions::new() + .access_mode(FILE_TRAVERSE | FILE_READ_ATTRIBUTES) + .share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE) + .custom_flags(FILE_FLAG_BACKUP_SEMANTICS) + .open(self.io_root()) + .map_err(to_file_error)?; + (mount, path.to_owned()) + }; + #[cfg(all(not(target_os = "linux"), not(windows)))] + let (mount, root) = (std::fs::File::open(self.io_root()).map_err(to_file_error)?, path.to_owned()); + Ok(DiscoverySession { + disk_root: self.io_root().to_path_buf(), + volume: volume.to_owned(), + object: object.map(str::to_owned), + root, + _mount: mount, + frames: Vec::new(), + touched: Instant::now(), + }) + } + + fn validate_delete_discovery_owner(&self, session: &DiscoverySession) -> Result<()> { + if session.disk_root != self.io_root() || session.touched.elapsed() > Duration::from_secs(300) { + return Err(DiskError::InvalidPath); + } + #[cfg(unix)] + { + use std::os::unix::fs::MetadataExt; + let before = session._mount.metadata().map_err(to_file_error)?; + #[cfg(target_os = "linux")] + let current = self.mount_lease.metadata().map_err(to_file_error)?; + #[cfg(not(target_os = "linux"))] + let current = std::fs::metadata(self.io_root()).map_err(to_file_error)?; + if before.dev() != current.dev() || before.ino() != current.ino() { + return Err(DiskError::InvalidPath); + } + } + Ok(()) + } + + async fn advance_delete_discovery( + &self, + path: &Path, + session: DiscoverySession, + limit: u32, + ) -> Result<(Vec, Option)> { + let lease = os::acquire_namespace_mutation_lease_with_owner(path, None).await; + self.validate_delete_discovery_owner(&session)?; + os::run_blocking_namespace_operation(lease, move || session.advance(limit)) + .await + .map_err(|error| to_file_error(error).into()) + } + + pub(super) async fn delete_recovery_discover( + &self, + volume: &str, + object: Option<&str>, + cursor: Option, + limit: u32, + ) -> Result { + if !(1..=256).contains(&limit) { + return Err(DiskError::InvalidPath); + } + let path = self.delete_discovery_path(volume, object)?; + let (token, session) = if let Some(cursor) = cursor { + let session = DISCOVERY_SESSIONS + .lock() + .remove(&cursor.token) + .ok_or(DiskError::FileNotFound)?; + if session.volume != volume || session.object.as_deref() != object { + return Err(DiskError::InvalidPath); + } + (cursor.token, session) + } else { + (Uuid::new_v4(), self.new_delete_discovery_session(volume, object, &path)?) + }; + let (transactions, session) = self.advance_delete_discovery(&path, session, limit).await?; + let next = if let Some(mut session) = session { + session.touched = Instant::now(); + let mut sessions = DISCOVERY_SESSIONS.lock(); + sessions.retain(|_, session| session.touched.elapsed() <= Duration::from_secs(300)); + if sessions.len() >= 128 { + return Err(DiskError::Io(std::io::Error::from(ErrorKind::WouldBlock))); + } + sessions.insert(token, session); + Some(DeleteRecoveryCursor { token }) + } else { + None + }; + Ok(DeleteRecoveryPage { transactions, next }) + } + + /// Existence probes retain their traversal privately, including across cancellation. + pub(super) async fn delete_recovery_has_evidence(&self, volume: &str, object: &str) -> Result { + let path = self.delete_discovery_path(volume, Some(object))?; + self.new_delete_discovery_session(volume, Some(object), &path)? + .has_evidence(self, &path) + .await + } + + pub(super) async fn delete_recovery_inspect( + &self, + volume: &str, + object: &str, + transaction: Uuid, + ) -> Result { + if !safe_path(object, false) || transaction.is_nil() { + return Err(DiskError::InvalidPath); + } + let object_path = self.get_object_path(volume, object)?; + let owner: Option> = Some(os::acquire_metadata_mutation_lease(&object_path, None).await); + self.delete_recovery_inspect_owned(volume, object, transaction, owner).await + } + + pub(super) async fn delete_recovery_inspect_owned( + &self, + volume: &str, + object: &str, + transaction: Uuid, + owner: Option>, + ) -> Result { + let root = self.io_get_object_path(volume, object)?; + self.io_get_object_path(volume, &format!("{object}/{transaction}"))?; + let volume = volume.to_owned(); + let object = object.to_owned(); + let lease = os::acquire_namespace_mutation_lease_with_owner(&root, owner).await; + os::run_blocking_namespace_operation(lease, move || Ok(inspection(&root, &volume, &object, transaction))) + .await + .map_err(to_file_error)? + } + + async fn activate_delete_preparation( + &self, + journal: &DeleteRecoveryJournal, + original: Option<&[u8]>, + target: Option>, + owner: Option>, + prepared_publication: Option>, + ) -> Result>> { + let volume = journal.volume.as_str(); + let object = journal.object.as_str(); + let transaction = journal.transaction; + let journal = journal_bytes(journal)?; + self.wait_for_startup_cleanup().await; + let temporary = self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?; + let root = self.io_get_object_path(volume, object)?; + let destination = root.join(transaction.to_string()); + let original = original.map(<[u8]>::to_vec); + let sync = effective_durability(volume).syncs_commit_metadata(); + #[cfg(test)] + let record_destination = destination.clone(); + let staging_root = temporary.clone(); + let lease = match prepared_publication { + Some(publication) => publication, + None => os::acquire_namespace_mutation_lease_with_owner(&destination, owner.clone()).await, + }; + let (staged, target) = os::run_blocking_namespace_operation(lease.clone(), move || { + std::fs::create_dir_all(&staging_root)?; + let staged = Arc::new( + tempfile::Builder::new() + .prefix("delete-preparation-") + .tempdir_in(&staging_root)?, + ); + let missing_marker = original.is_none() && target.is_some(); + for (name, bytes) in [ + (STORAGE_FORMAT_FILE_BACKUP, original.as_deref()), + (TARGET, target.as_deref()), + (PREPARED, Some(journal.as_slice())), + (DELETE_MARKER_ROLLBACK_FILE, missing_marker.then_some(&[][..])), + ] { + let Some(bytes) = bytes else { + continue; + }; + #[cfg(test)] + run_owned_file_write_before_open(&record_destination.join(name)); + let mut file = std::fs::File::create(staged.path().join(name))?; + std::io::Write::write_all(&mut file, bytes)?; + if sync { + file.sync_all()?; + } + } + if sync { + os::fsync_dir_std(staged.path())?; + } + Ok((staged, target)) + }) + .await + .map_err(to_file_error)?; + self.delete_recovery_barrier(volume, object, transaction, "PREACTIVATION") + .await?; + match fs::symlink_metadata(&destination).await { + Ok(_) => return Err(DiskError::OutdatedXLMeta), + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + // The directory owner accompanies the blocking rename, so cancelling + // its waiter cannot delete staged files while activation is running. + // The fresh staging key cannot alias a forward-plan key. Its real + // lease retains the destination owner and TempDir through the syscall, + // without replacing the prepared publication's sibling guards. + let publication_owner: Arc = Arc::new((lease, staged.clone())); + let activation = os::try_acquire_namespace_mutation_lease_with_owner(staged.path(), publication_owner)?; + os::rename_all_with_lease( + staged.path(), + &destination, + &self.io_get_bucket_path(volume)?, + &self.publication_root, + activation, + ) + .await?; + if sync { + let bucket = self.io_get_bucket_path(volume)?; + let mut path = root.as_path(); + loop { + os::fsync_dir_with_owner(path, owner.clone()).await.map_err(to_file_error)?; + if path == bucket { + break; + } + path = path.parent().ok_or(DiskError::InvalidPath)?; + } + os::fsync_dir_with_owner(&temporary, owner.clone()) + .await + .map_err(to_file_error)?; + } + self.delete_recovery_barrier(volume, object, transaction, "PREPARED").await?; + Ok(target) + } + + pub(super) async fn prepare_delete_recovery( + &self, + volume: &str, + object: &str, + original: Option<&[u8]>, + requests: &[FileInfo], + authority: Option<&DeleteRecoveryJournal>, + context: DeletePreparationContext, + ) -> Result>> { + let DeletePreparationContext { + transaction, + bucket_incarnation: incarnation, + namespace_owner: owner, + prepared_publication, + } = context; + let mut plan = original.map(FileMeta::load).transpose()?.unwrap_or_else(FileMeta::new); + let mut deleted_headers: Vec<_> = plan + .versions + .iter() + .map(|version| (version.header.version_id, std::mem::discriminant(&version.header.version_type))) + .collect(); + let mut original_versions = Vec::new(); + for version in &plan.versions { + original_versions.push(generation(version.into_fileinfo(volume, object, true)?)?); + } + let data_dirs = if original.is_none() { + for request in requests { + plan.add_version(request.clone())?; + } + Vec::new() + } else { + plan.delete_versions(requests)?.into_iter().flatten().collect::>() + }; + if data_dirs.iter().any(|dir| *dir == transaction || dir.is_nil()) { + return Err(DiskError::FileCorrupt); + } + let target = if plan.versions.is_empty() { + None + } else { + Some(plan.marshal_msg()?) + }; + let predecessors = match authority { + Some(authority) => authority.predecessors.clone(), + None => { + self.delete_recovery_predecessors(volume, object, transaction, incarnation, owner.clone()) + .await? + } + }; + let surviving: HashSet<_> = plan + .versions + .iter() + .map(|version| (version.header.version_id, std::mem::discriminant(&version.header.version_type))) + .collect(); + deleted_headers.retain(|header| !surviving.contains(header)); + let deleted_versions = deleted_headers.into_iter().map(|(id, _)| id.unwrap_or_default()).collect(); + let requests = requests + .iter() + .map(|request| { + let mut sparse = request.clone(); + sparse.data = None; + sparse + }) + .collect(); + let mut journal = DeleteRecoveryJournal { + protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION, + volume: volume.to_owned(), + object: object.to_owned(), + transaction, + bucket_incarnation: incarnation, + original_digest: original.map(digest), + target_digest: target.as_deref().map(digest), + data_dirs, + deleted_versions, + requests, + original_versions, + predecessors, + }; + if let Some(authority) = authority { + if authority.volume != volume + || authority.object != object + || authority.transaction != transaction + || authority.bucket_incarnation != incarnation + { + return Err(DiskError::OutdatedXLMeta); + } + journal.original_versions.clone_from(&authority.original_versions); + journal.deleted_versions.clone_from(&authority.deleted_versions); + } + drop(plan); + self.activate_delete_preparation(&journal, original, target, owner, prepared_publication) + .await + } + + pub(crate) async fn delete_recovery_settle( + &self, + volume: &str, + object: &str, + transaction: Uuid, + expected_fingerprint: DeleteRecoveryDigest, + action: DeleteRecoveryAction, + namespace_owner: Option>, + ) -> Result { + if !safe_path(object, false) || transaction.is_nil() { + return Err(DiskError::InvalidPath); + } + let object_path = self.get_object_path(volume, object)?; + let metadata_owner = os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await; + let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await; + let owner: Option> = Some(Arc::new((metadata_owner, volume_guard))); + self.delete_recovery_settle_owned(volume, object, transaction, expected_fingerprint, action, owner, None) + .await + } + + pub(in crate::disk) async fn delete_recovery_settle_prepared( + &self, + volume: &str, + object: &str, + transaction: Uuid, + expected_fingerprint: DeleteRecoveryDigest, + action: DeleteRecoveryAction, + publication: Arc, + ) -> Result { + if !safe_path(object, false) || transaction.is_nil() { + return Err(DiskError::InvalidPath); + } + let owner: Option> = Some(publication.clone()); + self.delete_recovery_settle_owned(volume, object, transaction, expected_fingerprint, action, owner, Some(publication)) + .await + } + + pub(super) async fn rollback_prepared_delete( + &self, + volume: &str, + object: &str, + transaction: Uuid, + owner: Option>, + prepared_publication: Option>, + ) -> Result { + let prepared = self.io_get_object_path(volume, &format!("{object}/{transaction}/{PREPARED}"))?; + if regular_bytes(&prepared)?.is_none() { + return Ok(false); + } + let state = self + .delete_recovery_inspect_owned(volume, object, transaction, owner.clone()) + .await?; + self.delete_recovery_settle_owned( + volume, + object, + transaction, + state.fingerprint, + DeleteRecoveryAction::Rollback, + owner, + prepared_publication, + ) + .await?; + Ok(true) + } + + async fn delete_recovery_settle_owned( + &self, + volume: &str, + object: &str, + transaction: Uuid, + expected_fingerprint: DeleteRecoveryDigest, + action: DeleteRecoveryAction, + owner: Option>, + prepared_publication: Option>, + ) -> Result { + let observed = self + .delete_recovery_inspect_owned(volume, object, transaction, owner.clone()) + .await?; + if observed.fingerprint != expected_fingerprint { + return Err(DiskError::OutdatedXLMeta); + } + if let DeleteRecoveryAction::AdoptCommit { authority } = &action { + if authority.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION + || authority.volume != volume + || authority.object != object + || authority.transaction != transaction + { + return Err(DiskError::OutdatedXLMeta); + } + if observed.state == DeleteRecoveryState::NewPrepared { + let local = observed.journal.as_ref().ok_or(DiskError::FileCorrupt)?; + if observed.current_digest.is_some() || !local.matches_prepared_authority(authority) { + return Err(DiskError::OutdatedXLMeta); + } + // The coordinator supplies an inspected same-transaction + // receipt. Retain this drive's immutable hashes, data dirs and + // causal history when completing its partial publication. + let root = self.io_get_object_path(volume, object)?; + let receipt = CommitReceipt { + journal: local.clone(), + finished: false, + }; + write_delete_rollback_file_with_prepared_publication( + self.io_root(), + volume, + &root, + transaction, + DELETE_QUORUM_COMMITTED_FILE, + &serde_json::to_vec(&receipt)?, + owner.clone(), + prepared_publication.clone(), + ) + .await?; + self.delete_recovery_barrier(volume, object, transaction, "DECISION").await?; + return self.delete_recovery_inspect_owned(volume, object, transaction, owner).await; + } + if observed.state != DeleteRecoveryState::Absent { + return Err(DiskError::OutdatedXLMeta); + } + if observed.current_digest.is_none() { + // A replay witness inherits the original decision's causal + // history; observing later records cannot backdate new edges. + let mut witness = authority.as_ref().clone(); + witness.original_digest = None; + witness.target_digest = None; + witness.data_dirs.clear(); + let root = self.io_get_object_path(volume, object)?; + let _ = self + .activate_delete_preparation(&witness, None, None, owner.clone(), None) + .await?; + let receipt = CommitReceipt { + journal: witness, + finished: false, + }; + write_delete_rollback_file_with_prepared_publication( + self.io_root(), + volume, + &root, + transaction, + DELETE_QUORUM_COMMITTED_FILE, + &serde_json::to_vec(&receipt)?, + owner.clone(), + prepared_publication.clone(), + ) + .await?; + self.delete_recovery_barrier(volume, object, transaction, "DECISION").await?; + return self.delete_recovery_inspect_owned(volume, object, transaction, owner).await; + } + if !authority.matches_adoption_source(&observed.current_versions) { + return Err(DiskError::OutdatedXLMeta); + } + let root = self.io_get_object_path(volume, object)?; + let bytes = self.read_all(volume, &format!("{object}/{STORAGE_FORMAT_FILE}")).await?; + let _ = self + .prepare_delete_recovery( + volume, + object, + Some(&bytes), + &authority.requests, + Some(authority.as_ref()), + DeletePreparationContext { + transaction, + bucket_incarnation: authority.bucket_incarnation, + namespace_owner: owner.clone(), + prepared_publication: None, + }, + ) + .await?; + let adopted = self + .delete_recovery_inspect_owned(volume, object, transaction, owner.clone()) + .await?; + let adopted_journal = adopted.journal.ok_or(DiskError::FileCorrupt)?; + let receipt = CommitReceipt { + journal: adopted_journal, + finished: false, + }; + write_delete_rollback_file_with_prepared_publication( + self.io_root(), + volume, + &root, + transaction, + DELETE_QUORUM_COMMITTED_FILE, + &serde_json::to_vec(&receipt)?, + owner.clone(), + prepared_publication.clone(), + ) + .await?; + self.delete_recovery_barrier(volume, object, transaction, "DECISION").await?; + return self.delete_recovery_inspect_owned(volume, object, transaction, owner).await; + } + if observed.state == DeleteRecoveryState::Absent { + if action == DeleteRecoveryAction::PublishCommit { + if observed.current_digest.is_some() { + return Err(DiskError::OutdatedXLMeta); + } + self.prepare_delete_recovery( + volume, + object, + None, + &[], + None, + DeletePreparationContext { + transaction, + bucket_incarnation: None, + namespace_owner: owner.clone(), + prepared_publication: None, + }, + ) + .await?; + } else { + return Ok(observed); + } + } + let observed = self + .delete_recovery_inspect_owned(volume, object, transaction, owner.clone()) + .await?; + if action == DeleteRecoveryAction::DiscardLegacy { + if observed.state != DeleteRecoveryState::LegacyAmbiguous || !observed.explicit_delete_safe { + return Err(DiskError::FileCorrupt); + } + let root = self.io_get_object_path(volume, object)?; + let rollback = root.join(transaction.to_string()); + let plan_root = root.clone(); + let missing = observed.current_digest.is_none(); + let lease = os::acquire_namespace_mutation_lease_with_owner(&root, owner.clone()).await; + let plan = os::run_blocking_namespace_operation(lease, move || { + Ok((|| { + let before = + regular_metadata_bytes(&plan_root.join(transaction.to_string()).join(STORAGE_FORMAT_FILE_BACKUP))? + .as_deref() + .map(FileMeta::load) + .transpose()?; + explicit_delete_plan(&plan_root, transaction, before.as_ref(), None, missing)?.ok_or(DiskError::FileCorrupt) + })()) + }) + .await + .map_err(to_file_error)??; + for dir in plan.data_dirs { + let relative = dir.strip_prefix(&root).map_err(|_| DiskError::InvalidPath)?; + let data_dir = relative + .file_name() + .and_then(|name| name.to_str()) + .and_then(|name| Uuid::parse_str(name).ok()) + .ok_or(DiskError::FileCorrupt)?; + let parent = relative.parent().filter(|parent| !parent.as_os_str().is_empty()); + let staged_object = + parent.map_or_else(|| object.to_owned(), |parent| format!("{object}/{}", parent.to_string_lossy())); + self.reserve_version_delete_with_prepared_publication( + volume, + &staged_object, + data_dir, + transaction, + owner.clone(), + prepared_publication.clone(), + ) + .await?; + let reserved = dir.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}")); + let private = dir.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}")); + match async { + if let Some(publication) = &prepared_publication { + os::rename_with_lease(reserved.clone(), private.clone(), publication.clone()).await + } else { + os::rename_with_namespace_owner(&reserved, &private, owner.clone()).await + } + } + .await + { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + if effective_durability(volume).syncs_commit_metadata() { + os::fsync_dir_with_owner(&dir, owner.clone()).await.map_err(to_file_error)?; + } + if self + .delete_data_dir_with_namespace_owner( + volume, + &format!("{object}/{}", relative.to_string_lossy()), + DeleteOptions { + recursive: true, + old_data_dir: Some(transaction), + recoverable_delete: true, + ..Default::default() + }, + owner.clone(), + ) + .await? + == DataDirDeleteStatus::Deferred + { + return Err(DiskError::DiskOngoingReq); + } + } + for file in plan.proof_files { + match async { + if let Some(publication) = &prepared_publication { + os::remove_file_with_lease(file, publication.clone()).await + } else { + os::remove_file_with_owner(file, owner.clone()).await + } + } + .await + { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + } + if effective_durability(volume).syncs_commit_metadata() && fs::try_exists(&rollback).await.map_err(to_file_error)? { + os::fsync_dir_with_owner(&rollback, owner.clone()) + .await + .map_err(to_file_error)?; + } + self.delete_unleased_with_namespace_owner( + volume, + &format!("{object}/{transaction}"), + &DeleteOptions { + immediate: true, + ..Default::default() + }, + owner.clone(), + None, + ) + .await?; + return self.delete_recovery_inspect_owned(volume, object, transaction, owner).await; + } + let journal = observed.journal.as_ref().ok_or(DiskError::FileCorrupt)?; + let root = self.io_get_object_path(volume, object)?; + let rollback = root.join(transaction.to_string()); + match action { + DeleteRecoveryAction::PublishCommit => { + if !observed.commit_data_safe { + return Err(DiskError::OutdatedXLMeta); + } + if matches!(observed.state, DeleteRecoveryState::QuorumCommitted | DeleteRecoveryState::CommitFinished) { + return Ok(observed); + } + let receipt = CommitReceipt { + journal: journal.clone(), + finished: false, + }; + write_delete_rollback_file_with_prepared_publication( + self.io_root(), + volume, + &root, + transaction, + DELETE_QUORUM_COMMITTED_FILE, + &serde_json::to_vec(&receipt)?, + owner.clone(), + prepared_publication.clone(), + ) + .await?; + self.delete_recovery_barrier(volume, object, transaction, "DECISION").await?; + } + DeleteRecoveryAction::FinishCommit => { + if !matches!(observed.state, DeleteRecoveryState::QuorumCommitted | DeleteRecoveryState::CommitFinished) + || !observed.commit_data_safe + { + return Err(DiskError::OutdatedXLMeta); + } + if observed.state == DeleteRecoveryState::CommitFinished { + return Ok(observed); + } + for dir in &journal.data_dirs { + self.reserve_version_delete_with_prepared_publication( + volume, + object, + *dir, + transaction, + owner.clone(), + prepared_publication.clone(), + ) + .await?; + } + if observed.current_digest == journal.original_digest { + if journal.target_digest.is_some() { + let target = self.read_all(volume, &format!("{object}/{transaction}/{TARGET}")).await?; + self.write_all_meta_with_namespace_owner( + volume, + &format!("{object}/{STORAGE_FORMAT_FILE}"), + &target, + true, + owner.clone(), + prepared_publication.clone(), + ) + .await?; + } else { + self.delete_file_with_prepared_entry( + &self.io_get_bucket_path(volume)?, + &root.join(STORAGE_FORMAT_FILE), + false, + false, + owner.clone(), + prepared_publication.clone(), + None, + prepared_publication.as_ref().map(|publication| PreparedPathDelete { + lease: publication.clone(), + trash: None, + encoded_trash: None, + }), + None, + ) + .await?; + if effective_durability(volume).syncs_commit_metadata() { + os::fsync_dir_with_owner(&root, owner.clone()).await.map_err(to_file_error)?; + } + } + } + self.commit_reserved_version_delete_with_publication( + volume, + object, + transaction, + owner.clone(), + prepared_publication.clone(), + ) + .await?; + for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] { + match async { + if let Some(publication) = &prepared_publication { + os::remove_file_with_lease(rollback.join(name), publication.clone()).await + } else { + os::remove_file_with_owner(rollback.join(name), owner.clone()).await + } + } + .await + { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + } + if effective_durability(volume).syncs_commit_metadata() { + os::fsync_dir_with_owner(&rollback, owner.clone()) + .await + .map_err(to_file_error)?; + } + let (_, complete) = self + .finish_version_delete_with_prepared_publication( + volume, + object, + transaction, + true, + owner.clone(), + prepared_publication.clone(), + ) + .await?; + if !complete { + return Err(DiskError::DiskOngoingReq); + } + for dir in &journal.data_dirs { + let staged = rollback.join(dir.to_string()); + match fs::symlink_metadata(&staged).await { + Ok(metadata) if metadata.is_dir() => { + let staged_object = format!("{object}/{transaction}"); + self.reserve_version_delete_with_prepared_publication( + volume, + &staged_object, + *dir, + transaction, + owner.clone(), + prepared_publication.clone(), + ) + .await?; + let source = staged.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}")); + let destination = staged.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}")); + match async { + if let Some(publication) = &prepared_publication { + os::rename_with_lease(source.clone(), destination.clone(), publication.clone()).await + } else { + os::rename_with_namespace_owner(&source, &destination, owner.clone()).await + } + } + .await + { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + if effective_durability(volume).syncs_commit_metadata() { + os::fsync_dir_with_owner(&staged, owner.clone()) + .await + .map_err(to_file_error)?; + } + if self + .delete_data_dir_with_namespace_owner( + volume, + &format!("{staged_object}/{dir}"), + DeleteOptions { + recursive: true, + old_data_dir: Some(transaction), + ..Default::default() + }, + owner.clone(), + ) + .await? + == DataDirDeleteStatus::Deferred + { + return Err(DiskError::DiskOngoingReq); + } + } + Ok(_) => return Err(DiskError::FileCorrupt), + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + } + write_delete_rollback_file_with_prepared_publication( + self.io_root(), + volume, + &root, + transaction, + FINISHED, + &[], + owner.clone(), + prepared_publication.clone(), + ) + .await?; + let receipt = CommitReceipt { + journal: journal.clone(), + finished: true, + }; + write_delete_rollback_file_with_prepared_publication( + self.io_root(), + volume, + &root, + transaction, + DELETE_QUORUM_COMMITTED_FILE, + &serde_json::to_vec(&receipt)?, + owner.clone(), + prepared_publication.clone(), + ) + .await?; + } + DeleteRecoveryAction::FinalizeCommit => { + if observed.state != DeleteRecoveryState::CommitFinished { + return Err(DiskError::DiskOngoingReq); + } + for name in [TARGET, PREPARED, FINISHED] { + match async { + if let Some(publication) = &prepared_publication { + os::remove_file_with_lease(rollback.join(name), publication.clone()).await + } else { + os::remove_file_with_owner(rollback.join(name), owner.clone()).await + } + } + .await + { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + } + if effective_durability(volume).syncs_commit_metadata() { + os::fsync_dir_with_owner(&rollback, owner.clone()) + .await + .map_err(to_file_error)?; + } + async { + if let Some(publication) = &prepared_publication { + os::remove_file_with_lease(rollback.join(DELETE_QUORUM_COMMITTED_FILE), publication.clone()).await + } else { + os::remove_file_with_owner(rollback.join(DELETE_QUORUM_COMMITTED_FILE), owner.clone()).await + } + } + .await + .map_err(to_file_error)?; + self.delete_unleased_with_namespace_owner( + volume, + &format!("{object}/{transaction}"), + &DeleteOptions { + immediate: true, + ..Default::default() + }, + owner.clone(), + None, + ) + .await?; + } + DeleteRecoveryAction::Rollback => { + if observed.state != DeleteRecoveryState::NewPrepared || !observed.rollback_data_complete { + return Err(DiskError::FileCorrupt); + } + let source = if observed.backup_digest.is_some() { + format!("{object}/{transaction}/{STORAGE_FORMAT_FILE_BACKUP}") + } else { + format!("{object}/{STORAGE_FORMAT_FILE}") + }; + if journal.original_digest.is_some() { + let bytes = self.read_all(volume, &source).await?; + let metadata = FileMeta::load(&bytes)?; + for version in &metadata.versions { + if version.header.version_type != rustfs_filemeta::VersionType::Object { + continue; + } + let mut info = version.into_fileinfo(volume, object, true)?; + info.hydrate_shard_integrity()?; + if info.inline_data() || info.is_remote() { + continue; + } + let dir = info.data_dir.filter(|id| !id.is_nil()).ok_or(DiskError::FileCorrupt)?; + let codec = coding::Erasure::try_new_with_options( + info.erasure.data_blocks, + info.erasure.parity_blocks, + info.erasure.block_size, + info.uses_legacy_checksum, + ) + .map_err(DiskError::from)?; + let live = self.io_get_object_path(volume, &format!("{object}/{dir}"))?; + let staged = self.io_get_object_path(volume, &format!("{object}/{transaction}/{dir}"))?; + for part in &info.parts { + let name = format!("part.{}", part.number); + let checksum = info.erasure.get_checksum_info(part.number); + let algo = if info.uses_legacy_checksum && checksum.algorithm == HashAlgorithm::HighwayHash256S { + HashAlgorithm::HighwayHash256SLegacy + } else { + checksum.algorithm + }; + if let Some(integrity) = &part.integrity { + verify_recovery_integrity_sources( + &live, + &staged, + integrity, + info.erasure.index.checked_sub(1).ok_or(DiskError::FileCorrupt)?, + algo.clone(), + owner.clone(), + ) + .await?; + } + let mut checked = false; + for part_root in [&live, &staged] { + let path = part_root.join(&name); + match fs::symlink_metadata(&path).await { + Ok(metadata) if metadata.is_file() => {} + Ok(_) => return Err(DiskError::InvalidPath), + Err(err) if err.kind() == ErrorKind::NotFound => continue, + Err(err) => return Err(to_file_error(err).into()), + } + self.bitrot_verify( + &path, + usize::try_from( + codec.shard_file_size(i64::try_from(part.size).map_err(|_| DiskError::FileCorrupt)?), + ) + .map_err(|_| DiskError::FileCorrupt)?, + algo.clone(), + codec.shard_size(), + ) + .await?; + checked = true; + } + if !checked { + return Err(DiskError::FileCorrupt); + } + } + } + } + if journal.original_digest.is_none() && journal.target_digest.is_none() { + for name in [TARGET, PREPARED, FINISHED] { + match async { + if let Some(publication) = &prepared_publication { + os::remove_file_with_lease(rollback.join(name), publication.clone()).await + } else { + os::remove_file_with_owner(rollback.join(name), owner.clone()).await + } + } + .await + { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(to_file_error(err).into()), + } + } + self.delete_unleased_with_namespace_owner( + volume, + &format!("{object}/{transaction}"), + &DeleteOptions { + immediate: true, + ..Default::default() + }, + owner.clone(), + None, + ) + .await?; + } else { + restore_delete_rollback_with_prepared_publication( + volume, + &root, + &root.join(STORAGE_FORMAT_FILE), + transaction, + &self.publication_root, + owner.clone(), + prepared_publication.clone(), + ) + .await?; + } + } + DeleteRecoveryAction::AdoptCommit { .. } => return Err(DiskError::MethodNotAllowed), + DeleteRecoveryAction::DiscardLegacy => return Err(DiskError::MethodNotAllowed), + } + self.delete_recovery_inspect_owned(volume, object, transaction, owner).await + } + + pub(super) async fn delete_recovery_barrier(&self, volume: &str, object: &str, transaction: Uuid, stage: &str) -> Result<()> { + barrier(volume, object, transaction, stage, &self.root).await + } +} + +#[cfg(not(feature = "e2e-test-hooks"))] +async fn barrier(_volume: &str, _object: &str, _transaction: Uuid, _stage: &str, _drive: &Path) -> Result<()> { + Ok(()) +} + +#[cfg(feature = "e2e-test-hooks")] +#[derive(serde::Deserialize)] +#[serde(deny_unknown_fields)] +struct BarrierArm { + nonce: String, + bucket: String, + object: String, + stage: String, + #[serde(default)] + disk: Option, +} + +#[cfg(feature = "e2e-test-hooks")] +async fn barrier(volume: &str, object: &str, transaction: Uuid, stage: &str, drive: &Path) -> Result<()> { + let Some(directory) = std::env::var_os("RUSTFS_DELETE_RECOVERY_BARRIER_DIR").map(PathBuf::from) else { + return Ok(()); + }; + let bytes = match fs::read(directory.join("arm.json")).await { + Ok(bytes) => bytes, + Err(err) if err.kind() == ErrorKind::NotFound => return Ok(()), + Err(err) => return Err(to_file_error(err).into()), + }; + let arm: BarrierArm = serde_json::from_slice(&bytes)?; + if arm.bucket != volume + || arm.object != object + || arm.stage != stage + || arm.disk.as_deref().is_some_and(|disk| Path::new(disk) != drive) + { + return Ok(()); + } + if arm.nonce.is_empty() + || arm.nonce.len() > 64 + || !arm + .nonce + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || byte == b'-' || byte == b'_') + { + return Err(DiskError::InvalidPath); + } + let drive_hash = rustfs_utils::crypto::hex(digest(drive.to_string_lossy().as_bytes())); + let observation = serde_json::json!({"transaction": transaction, "bucket": volume, "object": object, "stage": stage, "pid": std::process::id(), "drive": drive}); + fs::write( + directory.join(format!("reached.{}.{}.{}.json", arm.nonce, std::process::id(), &drive_hash[..16])), + serde_json::to_vec(&observation)?, + ) + .await + .map_err(to_file_error)?; + let deadline = Instant::now() + Duration::from_secs(60); + while !fs::try_exists(directory.join(format!("release.{}", arm.nonce))) + .await + .map_err(to_file_error)? + { + if Instant::now() >= deadline { + return Err(DiskError::Timeout); + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + if fs::read(directory.join(format!("release.{}", arm.nonce))) + .await + .map_err(to_file_error)? + == b"io_error" + { + return Err(DiskError::Io(std::io::Error::other("injected delete recovery barrier failure"))); + } + Ok(()) +} + +pub(super) async fn restore_barrier(volume: &str, object_dir: &Path, transaction: Uuid) -> Result<()> { + #[cfg(feature = "e2e-test-hooks")] + if let Some(directory) = std::env::var_os("RUSTFS_DELETE_RECOVERY_BARRIER_DIR").map(PathBuf::from) { + let bytes = match fs::read(directory.join("arm.json")).await { + Ok(bytes) => bytes, + Err(err) if err.kind() == ErrorKind::NotFound => return Ok(()), + Err(err) => return Err(to_file_error(err).into()), + }; + let arm: BarrierArm = serde_json::from_slice(&bytes)?; + if arm.bucket == volume && arm.stage == "RESTORE" && object_dir.ends_with(Path::new(volume).join(&arm.object)) { + let suffix = Path::new(volume).join(&arm.object); + let canonical = fs::canonicalize(object_dir).await.map_err(to_file_error)?; + let mut drive = canonical; + for _ in suffix.components() { + drive.pop(); + } + return barrier(volume, &arm.object, transaction, "RESTORE", &drive).await; + } + } + let _ = (volume, object_dir, transaction); + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::disk::local::test::{ensure_test_volume, test_file_info, test_meta}; + use tokio::io::AsyncReadExt; + + #[test] + fn delete_recovery_native_sidecar_classifier_is_bounded() { + assert!(delete_recovery_part_file("part.1")); + assert!(delete_recovery_part_file("part.1.meta")); + assert!(delete_recovery_part_file(&format!("part.1.integrity.{}", Uuid::new_v4()))); + for name in [ + "part.0.meta", + "part.1.meta.extra", + "part.1.integrity.invalid", + "xl.meta", + "child/", + ] { + assert!(!delete_recovery_part_file(name), "unknown sidecar {name} must stay protected"); + } + assert!(!delete_recovery_part_file(&format!("part.1.integrity.{}", Uuid::nil()))); + assert!(!metadata_less_part_file("part.1.meta"), "the heal shard classifier remains unchanged"); + } + + #[tokio::test] + #[serial_test::serial(shard_integrity_rollout)] + async fn delete_recovery_multipart_rollback_moves_native_sidecars_and_preserves_proofs() { + use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks}; + use crate::object_api::{ObjectOptions, PutObjReader}; + use crate::storage_api_contracts::bucket::{BucketOperations, MakeBucketOptions}; + use crate::storage_api_contracts::multipart::{CompletePart, MultipartOperations}; + use crate::storage_api_contracts::object::ObjectIO; + use http::HeaderMap; + + temp_env::async_with_vars( + [ + (rustfs_config::ENV_SHARD_INTEGRITY_WRITE, Some("true")), + (rustfs_config::ENV_SHARD_INTEGRITY_FLEET_CONFIRMED, Some("true")), + ], + async { + for damage in [None, Some("missing"), Some("corrupt"), Some("payload")] { + let (dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let bucket = "recovery-multipart"; + let object = "object"; + store.make_bucket(bucket, &MakeBucketOptions::default()).await.unwrap(); + let upload = store + .new_multipart_upload(bucket, object, &ObjectOptions::default()) + .await + .unwrap(); + let payload = vec![63; 6 * 1024 * 1024]; + let mut reader = PutObjReader::from_vec(payload.clone()); + let part = store + .put_object_part(bucket, object, &upload.upload_id, 1, &mut reader, &ObjectOptions::default()) + .await + .unwrap(); + store + .clone() + .complete_multipart_upload( + bucket, + object, + &upload.upload_id, + vec![CompletePart { + part_num: part.part_num, + etag: part.etag, + ..Default::default() + }], + &ObjectOptions::default(), + ) + .await + .unwrap(); + let disks = store.pools[0].disk_set[0].disks.read().await.clone(); + let transaction = Uuid::new_v4(); + let mut originals = Vec::new(); + for (dir, disk) in dirs.iter().zip(&disks) { + let root = dir.path().join(bucket).join(object); + let original = fs::read(root.join(STORAGE_FORMAT_FILE)).await.unwrap(); + let metadata = FileMeta::load(&original).unwrap(); + let mut info = metadata.versions[0].into_fileinfo(bucket, object, true).unwrap(); + info.hydrate_shard_integrity().unwrap(); + let data = info.data_dir.unwrap(); + let proof_name = info.parts[0] + .integrity + .as_ref() + .expect("real protected MPU proof") + .file_name(); + assert!(root.join(data.to_string()).join("part.1.meta").is_file()); + disk.as_ref() + .unwrap() + .delete_version_recoverable( + bucket, + object, + FileInfo { + name: object.to_owned(), + ..Default::default() + }, + false, + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await + .unwrap(); + let staged = root.join(transaction.to_string()).join(data.to_string()); + fs::rename(root.join(data.to_string()), &staged).await.unwrap(); + originals.push((root, data, proof_name, original)); + } + let (root, data, proof, _) = &originals[0]; + let proof_path = root.join(transaction.to_string()).join(data.to_string()).join(proof); + if let Some(damage) = damage { + if damage == "missing" { + fs::remove_file(&proof_path).await.unwrap(); + } else if damage == "corrupt" { + let mut bytes = fs::read(&proof_path).await.unwrap(); + bytes[rustfs_filemeta::shard_integrity::INDEX_HEADER_SIZE] ^= 1; + fs::write(&proof_path, bytes).await.unwrap(); + } else { + let metadata = FileMeta::load(&originals[0].3).unwrap(); + let info = metadata.versions[0].into_fileinfo(bucket, object, true).unwrap(); + let codec = coding::Erasure::try_new_with_options( + info.erasure.data_blocks, + info.erasure.parity_blocks, + info.erasure.block_size, + info.uses_legacy_checksum, + ) + .unwrap(); + let algorithm = info.erasure.get_checksum_info(1).algorithm; + let path = proof_path.parent().unwrap().join("part.1"); + let mut bytes = fs::read(&path).await.unwrap(); + let hash_size = algorithm.size(); + let part_size = usize::try_from(codec.shard_file_size(info.parts[0].size as i64)).unwrap(); + let length = part_size.min(codec.shard_size()); + bytes[hash_size] ^= 1; + let rewritten = algorithm.hash_encode(&bytes[hash_size..hash_size + length]).as_ref().to_vec(); + bytes[..hash_size].copy_from_slice(&rewritten); + coding::bitrot_verify( + std::io::Cursor::new(bytes.clone()), + bytes.len(), + part_size, + algorithm, + codec.shard_size(), + ) + .await + .expect("recomputed self-described bitrot checksum passes the former rollback gate"); + fs::write(path, bytes).await.unwrap(); + } + let disk = disks[0].as_ref().unwrap(); + let observed = disk.inspect_delete_transaction(bucket, object, transaction).await.unwrap(); + let backup = fs::read(root.join(transaction.to_string()).join(STORAGE_FORMAT_FILE_BACKUP)) + .await + .unwrap(); + assert!(matches!( + disk.settle_delete_transaction( + bucket, + object, + transaction, + observed.fingerprint, + DeleteRecoveryAction::Rollback + ) + .await, + Err(DiskError::FileCorrupt) + )); + assert!(!root.join(STORAGE_FORMAT_FILE).exists()); + assert!(root.join(transaction.to_string()).join(PREPARED).is_file()); + assert_eq!( + fs::read(root.join(transaction.to_string()).join(STORAGE_FORMAT_FILE_BACKUP)) + .await + .unwrap(), + backup + ); + continue; + } + // Both directions of a partially restored native layout: + // proof still staged with live payload, and the inverse. + for (index, (root, data, proof, _)) in originals.iter().take(2).enumerate() { + let live = root.join(data.to_string()); + fs::create_dir(&live).await.unwrap(); + let name = if index == 0 { "part.1" } else { proof.as_str() }; + fs::rename(root.join(transaction.to_string()).join(data.to_string()).join(name), live.join(name)) + .await + .unwrap(); + } + for (disk, (root, data, proof, original)) in disks.iter().zip(originals) { + let disk = disk.as_ref().unwrap(); + let observed = disk.inspect_delete_transaction(bucket, object, transaction).await.unwrap(); + assert!(observed.rollback_data_complete); + disk.settle_delete_transaction( + bucket, + object, + transaction, + observed.fingerprint, + DeleteRecoveryAction::Rollback, + ) + .await + .expect("rollback restores native multipart sidecars"); + assert_eq!(fs::read(root.join(STORAGE_FORMAT_FILE)).await.unwrap(), original); + assert!(root.join(data.to_string()).join("part.1.meta").is_file()); + assert!(root.join(data.to_string()).join(proof).is_file()); + assert!(!root.join(transaction.to_string()).exists()); + } + let mut reader = store.pools[0].disk_set[0] + .get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .unwrap(); + let mut restored = Vec::new(); + reader.stream.read_to_end(&mut restored).await.unwrap(); + assert_eq!(restored, payload, "restored protected MPU remains readable through integrity checks"); + } + }, + ) + .await; + } + + async fn fixture() -> (tempfile::TempDir, LocalDisk, FileInfo, Vec) { + let temp = tempfile::tempdir().expect("temporary disk"); + let endpoint = Endpoint::try_from(temp.path().to_str().expect("UTF-8 disk path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + ensure_test_volume(&disk, "recovery").await; + let mut info = test_file_info("object", Uuid::nil(), None, Some(Bytes::from_static(b"complete original inline payload"))); + info.set_inline_data(); + let bytes = test_meta(info.clone()); + let path = disk.io_get_object_path("recovery", "object/xl.meta").expect("metadata path"); + fs::create_dir_all(path.parent().expect("object directory")) + .await + .expect("object directory"); + fs::write(path, &bytes).await.expect("original metadata"); + (temp, disk, info, bytes) + } + + #[tokio::test] + async fn delete_recovery_prepared_settle_reuses_admission_with_queued_volume_write() { + use crate::disk::disk_store::LocalDiskWrapper; + let (_temp, disk, _, original) = fixture().await; + let transaction = Uuid::new_v4(); + let state = apply(&disk, transaction).await; + let disk = Arc::new(disk); + let wrapper = LocalDiskWrapper::new(disk.clone(), false); + let admitted = wrapper + .begin_legacy_metadata_write("recovery", "object") + .await + .expect("physical admission before classification"); + let mut deletion = Box::pin(disk.delete_volume("recovery", true)); + assert!( + matches!(futures::poll!(deletion.as_mut()), std::task::Poll::Pending), + "volume WRITE queues behind the actual admitted READ" + ); + let restored = tokio::time::timeout( + Duration::from_secs(5), + admitted.settle_delete_transaction(transaction, state.fingerprint, DeleteRecoveryAction::Rollback), + ) + .await + .expect("settlement must reuse its metadata, namespace and volume owner") + .expect("prepared rollback succeeds"); + assert_eq!(restored.state, DeleteRecoveryState::Absent); + assert_eq!( + disk.read_all("recovery", "object/xl.meta") + .await + .expect("restored real metadata"), + original + ); + tokio::time::timeout(Duration::from_secs(5), deletion) + .await + .expect("queued WRITE drains after settlement") + .expect("old volume deletion"); + } + + #[tokio::test] + async fn delete_recovery_guarded_batch_reuses_admission_with_queued_volume_write() { + use crate::disk::disk_store::LocalDiskWrapper; + let (_temp, disk, mut info, _) = fixture().await; + info.data_dir = Some(Uuid::new_v4()); + info.metadata.insert( + "etag".to_owned(), + rustfs_utils::crypto::hex(HashAlgorithm::Md5.hash_encode(info.data.as_ref().expect("inline payload"))), + ); + let metadata_path = disk.io_get_object_path("recovery", "object/xl.meta").expect("metadata path"); + fs::write(&metadata_path, test_meta(info)) + .await + .expect("canonical inline metadata"); + let original = fs::read(&metadata_path).await.expect("actual canonical metadata bytes"); + let current = FileMeta::load(&original) + .expect("actual metadata") + .into_fileinfo("recovery", "object", "", false, false, false) + .expect("actual current generation"); + let expected_current = CurrentDeleteGeneration::from_file_info(¤t).expect("canonical current generation"); + let disk = Arc::new(disk); + let wrapper = LocalDiskWrapper::new(disk.clone(), false); + let admitted = wrapper + .begin_legacy_file_write("recovery") + .await + .expect("physical volume admission before classification"); + let mut deletion = Box::pin(disk.delete_volume("recovery", true)); + assert!( + matches!(futures::poll!(deletion.as_mut()), std::task::Poll::Pending), + "volume WRITE queues behind admitted batch READ" + ); + let transaction = Uuid::new_v4(); + let errors = tokio::time::timeout( + Duration::from_secs(5), + admitted.delete_versions_guarded( + vec![GuardedFileInfoVersions { + group: FileInfoVersions { + name: "object".to_owned(), + versions: vec![request()], + ..Default::default() + }, + expected_current, + }], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ), + ) + .await + .expect("guarded executor must not take a second volume READ"); + assert!(errors.iter().all(Option::is_none), "real guarded publication succeeds: {errors:?}"); + let state = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("durable guarded journal"); + assert_eq!(state.state, DeleteRecoveryState::NewPrepared); + assert_eq!(state.current_digest, state.journal.as_ref().expect("prepared journal").target_digest); + assert_eq!( + disk.read_all("recovery", &format!("object/{transaction}/{STORAGE_FORMAT_FILE_BACKUP}")) + .await + .expect("durable original backup"), + original + ); + tokio::time::timeout(Duration::from_secs(5), deletion) + .await + .expect("queued WRITE drains after guarded publication") + .expect("old volume deletion"); + } + + #[tokio::test] + async fn delete_recovery_live_write_backup_preserves_delete_evidence() { + let (_temp, disk, info, original) = fixture().await; + let root = disk.io_get_object_path("recovery", "object").unwrap(); + let meta = FileMeta::load(&original).unwrap(); + let transaction = inline_metadata_rollback_dir(Uuid::nil(), &meta); + let rollback = root.join(transaction.to_string()); + fs::create_dir_all(&rollback).await.unwrap(); + fs::write(rollback.join(STORAGE_FORMAT_FILE_BACKUP), &original).await.unwrap(); + let mut healing = info.clone(); + healing.set_healing(); + fs::write(root.join(STORAGE_FORMAT_FILE), test_meta(healing)).await.unwrap(); + assert!(live_write_backup(&root, transaction).unwrap()); + let page = disk + .delete_recovery_discover("recovery", Some("object"), None, 128) + .await + .unwrap(); + assert!(page.transactions.is_empty()); + assert_eq!(fs::read(rollback.join(STORAGE_FORMAT_FILE_BACKUP)).await.unwrap(), original); + fs::write(rollback.join(DELETE_QUORUM_COMMITTED_FILE), []).await.unwrap(); + assert!( + !live_write_backup(&root, transaction).unwrap(), + "even an empty legacy receipt is delete evidence" + ); + fs::remove_file(rollback.join(DELETE_QUORUM_COMMITTED_FILE)).await.unwrap(); + let mut changed_parts = info.clone(); + changed_parts.parts[0].actual_size += 1; + fs::write(root.join(STORAGE_FORMAT_FILE), test_meta(changed_parts)) + .await + .unwrap(); + assert!( + !matches!(live_write_backup(&root, transaction), Ok(true)), + "changed part metadata cannot hide recovery evidence" + ); + let mut changed = info; + changed.metadata.insert("x-amz-object-lock-legal-hold".into(), "ON".into()); + fs::write(root.join(STORAGE_FORMAT_FILE), test_meta(changed)).await.unwrap(); + assert!( + !live_write_backup(&root, transaction).unwrap(), + "protected metadata differences remain meaningful" + ); + fs::remove_file(root.join(STORAGE_FORMAT_FILE)).await.unwrap(); + assert!( + !live_write_backup(&root, transaction).unwrap(), + "hidden legacy deletion remains discoverable" + ); + } + + #[tokio::test] + async fn delete_recovery_live_external_backup_rejects_partial_version_delete() { + let (_temp, disk, mut info, _) = fixture().await; + let root = disk.io_get_object_path("recovery", "object").unwrap(); + let transaction = Uuid::new_v4(); + info.data = None; + rustfs_utils::http::metadata_compat::remove_str( + &mut info.metadata, + rustfs_utils::http::metadata_compat::SUFFIX_INLINE_DATA, + ); + info.data_dir = Some(transaction); + info.version_id = Some(Uuid::new_v4()); + let mut meta = FileMeta::new(); + meta.add_version(info.clone()).unwrap(); + let original = meta.marshal_msg().unwrap(); + let rollback = root.join(transaction.to_string()); + fs::create_dir_all(&rollback).await.unwrap(); + fs::write(rollback.join(STORAGE_FORMAT_FILE_BACKUP), &original).await.unwrap(); + info.set_healing(); + fs::write(root.join(STORAGE_FORMAT_FILE), test_meta(info.clone())) + .await + .unwrap(); + assert!(live_write_backup(&root, transaction).unwrap()); + let mut sibling = info.clone(); + sibling.version_id = Some(Uuid::new_v4()); + sibling.data_dir = Some(Uuid::new_v4()); + meta.add_version(sibling).unwrap(); + fs::write(rollback.join(STORAGE_FORMAT_FILE_BACKUP), meta.marshal_msg().unwrap()) + .await + .unwrap(); + assert!( + !live_write_backup(&root, transaction).unwrap(), + "a missing backed-up sibling retains delete recovery" + ); + assert_eq!( + disk.inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap() + .state, + DeleteRecoveryState::LegacyAmbiguous + ); + assert!( + disk.discover_delete_transactions("recovery", Some("object"), None, 128) + .await + .unwrap() + .transactions + .iter() + .any(|id| id.transaction == transaction) + ); + } + + fn request() -> FileInfo { + FileInfo { + name: "object".to_owned(), + version_id: Some(Uuid::nil()), + ..Default::default() + } + } + + #[test] + fn delete_recovery_generation_known_aliases_are_equal_unknowns_stay_distinct() { + use rustfs_utils::http::metadata_compat as compat; + let mut legacy = request(); + legacy + .metadata + .insert(format!("{}{}", compat::MINIO_INTERNAL_PREFIX, compat::SUFFIX_INLINE_DATA), String::new()); + legacy + .metadata + .insert(format!("{}{}", compat::MINIO_INTERNAL_PREFIX, compat::SUFFIX_COMPRESSION), "gzip".into()); + let mut modern = legacy.clone(); + compat::insert_str(&mut modern.metadata, compat::SUFFIX_INLINE_DATA, String::new()); + compat::insert_str(&mut modern.metadata, compat::SUFFIX_COMPRESSION, "gzip".into()); + assert_eq!( + generation(legacy.clone()).expect("legacy identity"), + generation(modern.clone()).expect("modern identity") + ); + modern.metadata.insert( + format!("{}{}", compat::RUSTFS_INTERNAL_PREFIX, compat::SUFFIX_COMPRESSION), + "different".into(), + ); + assert!(matches!(generation(modern), Err(DiskError::FileCorrupt))); + let mut unknown = legacy.clone(); + legacy + .metadata + .insert("x-minio-internal-unknown-extension".into(), "same".into()); + unknown + .metadata + .insert("x-rustfs-internal-unknown-extension".into(), "same".into()); + assert_ne!( + generation(legacy).expect("legacy unknown field"), + generation(unknown).expect("modern unknown field") + ); + } + + #[tokio::test] + async fn delete_recovery_inactive_preparation_artifacts_use_existing_tmp_cleanup() { + let (_temp, disk, _, original) = fixture().await; + let temporary = disk.io_get_bucket_path(RUSTFS_META_TMP_BUCKET).unwrap(); + fs::create_dir_all(temporary.join("delete-preparation-interrupted")) + .await + .unwrap(); + fs::write(temporary.join("delete-preparation-interrupted").join(TARGET), b"unfinished target") + .await + .unwrap(); + fs::create_dir_all(temporary.join(".delete-rollback-interrupted")) + .await + .unwrap(); + fs::write( + temporary.join(".delete-rollback-interrupted").join("metadata"), + b"unpublished replacement", + ) + .await + .unwrap(); + assert!( + disk.discover_delete_transactions("recovery", Some("object"), None, 16) + .await + .unwrap() + .transactions + .is_empty() + ); + let ready = Arc::new(std::sync::atomic::AtomicU32::new(0)); + let notify = Arc::new(Notify::new()); + LocalDisk::cleanup_tmp_on_startup(disk.io_root(), &disk.publication_root, ready.clone(), notify) + .await + .unwrap(); + tokio::time::timeout(Duration::from_secs(5), async { + while ready.load(Ordering::Acquire) == 0 { + tokio::task::yield_now().await; + } + }) + .await + .unwrap(); + assert!(!temporary.join("delete-preparation-interrupted").exists()); + assert!(!temporary.join(".delete-rollback-interrupted").exists()); + assert!(!disk.io_root().join(RUSTFS_META_TMP_OLD_BUCKET).exists()); + assert!(temporary.join(".trash").is_dir(), "startup recreates the active trash directory"); + assert_eq!(disk.read_all("recovery", "object/xl.meta").await.unwrap().as_ref(), original); + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_recovery_activation_is_complete_before_metadata_mutation_even_after_cancel() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + use futures::FutureExt; + let (_temp, disk, _, original) = fixture().await; + let disk = Arc::new(disk); + let transaction = Uuid::new_v4(); + let destination = disk.io_get_object_path("recovery", &format!("object/{transaction}")).unwrap(); + let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); + let hook = hooks::install_at(hooks::Stage::Rename, &destination, move || { + let _ = entered_tx.send(()); + let _ = release_rx.recv(); + }); + let deleting = disk.clone(); + let mut request = request(); + request.expire_restored = true; + let task = tokio::spawn(async move { + deleting + .delete_version_recoverable( + "recovery", + "object", + request, + false, + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await + }); + let observations = std::panic::AssertUnwindSafe(async { + tokio::time::timeout(Duration::from_secs(10), entered_rx) + .await + .unwrap() + .unwrap(); + assert!(!destination.exists()); + assert_eq!(disk.read_all("recovery", "object/xl.meta").await.unwrap().as_ref(), original); + task.abort(); + assert!(task.await.unwrap_err().is_cancelled()); + }) + .catch_unwind() + .await; + let _ = release_tx.send(()); + drop(hook); + tokio::time::timeout(Duration::from_secs(5), async { + while !destination.exists() { + tokio::task::yield_now().await; + } + }) + .await + .unwrap(); + for name in [PREPARED, TARGET, STORAGE_FORMAT_FILE_BACKUP] { + assert!(destination.join(name).is_file()); + } + let state = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + assert_eq!(state.state, DeleteRecoveryState::NewPrepared); + assert_eq!(state.current_digest, Some(digest(&original))); + if let Err(panic) = observations { + std::panic::resume_unwind(panic); + } + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_recovery_receiver_volume_guard_outlives_cancelled_restore() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + use futures::FutureExt; + let (_temp, disk, _, _) = fixture().await; + let disk = Arc::new(disk); + let transaction = Uuid::new_v4(); + let state = apply(&disk, transaction).await; + let metadata = disk.io_get_object_path("recovery", "object/xl.meta").unwrap(); + let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); + let hook = hooks::install_at(hooks::Stage::Rename, &metadata, move || { + let _ = entered_tx.send(()); + let _ = release_rx.recv(); + }); + let restoring = disk.clone(); + let task = tokio::spawn(async move { + restoring + .settle_delete_transaction("recovery", "object", transaction, state.fingerprint, DeleteRecoveryAction::Rollback) + .await + }); + let mut deletion = None; + let observations = std::panic::AssertUnwindSafe(async { + tokio::time::timeout(Duration::from_secs(10), entered_rx) + .await + .unwrap() + .unwrap(); + task.abort(); + assert!(task.await.unwrap_err().is_cancelled()); + let removing = disk.clone(); + deletion = Some(tokio::spawn(async move { removing.delete_volume("recovery", true).await })); + assert!( + tokio::time::timeout(Duration::from_millis(100), deletion.as_mut().unwrap()) + .await + .is_err() + ); + }) + .catch_unwind() + .await; + let _ = release_tx.send(()); + drop(hook); + if let Some(deletion) = deletion { + tokio::time::timeout(Duration::from_secs(5), deletion) + .await + .unwrap() + .unwrap() + .unwrap(); + } + disk.make_volume("recovery").await.unwrap(); + assert!(!metadata.exists(), "late restore must finish before volume recreation"); + if let Err(panic) = observations { + std::panic::resume_unwind(panic); + } + } + + async fn apply(disk: &LocalDisk, transaction: Uuid) -> DeleteRecoveryInspection { + disk.delete_version_recoverable( + "recovery", + "object", + request(), + false, + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await + .expect("prepared metadata application"); + let inspected = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("inspect applied transaction"); + assert_eq!( + inspected.current_digest, + inspected.journal.as_ref().unwrap().target_digest, + "planned target must match the unchanged mutation loop" + ); + inspected + } + + #[tokio::test] + async fn delete_recovery_uncommitted_inline_rolls_back_exact_bytes() { + let (_temp, disk, _info, original) = fixture().await; + let transaction = Uuid::new_v4(); + let observed = apply(&disk, transaction).await; + assert_eq!(observed.state, DeleteRecoveryState::NewPrepared); + assert!(observed.rollback_data_complete); + let restored = disk + .settle_delete_transaction("recovery", "object", transaction, observed.fingerprint, DeleteRecoveryAction::Rollback) + .await + .expect("rollback intact prepared metadata"); + assert_eq!(restored.state, DeleteRecoveryState::Absent); + assert_eq!( + disk.read_all("recovery", "object/xl.meta") + .await + .expect("readable original metadata") + .as_ref(), + original + ); + } + + #[tokio::test] + async fn delete_recovery_preparation_records_existing_decision_before_new_generation() { + let (_temp, disk, info, original) = fixture().await; + let first = Uuid::new_v4(); + let applied = apply(&disk, first).await; + disk.settle_delete_transaction("recovery", "object", first, applied.fingerprint, DeleteRecoveryAction::PublishCommit) + .await + .expect("persist first decision"); + let mut newer = info; + newer.mod_time = newer.mod_time.map(|time| time + time::Duration::seconds(1)); + let newer_bytes = test_meta(newer); + assert_ne!(newer_bytes, original); + fs::write(disk.io_get_object_path("recovery", "object/xl.meta").unwrap(), &newer_bytes) + .await + .unwrap(); + let second = Uuid::new_v4(); + let second_state = apply(&disk, second).await; + assert_eq!(second_state.journal.unwrap().predecessors, vec![first]); + } + + #[tokio::test] + async fn delete_recovery_large_intent_fails_before_metadata_application() { + let (_temp, disk, _info, original) = fixture().await; + let transaction = Uuid::new_v4(); + let mut large = request(); + large + .metadata + .insert("oversized-intent".to_owned(), "x".repeat(usize::try_from(MAX_METADATA).unwrap())); + assert!( + disk.delete_version_recoverable( + "recovery", + "object", + large, + false, + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + } + ) + .await + .is_err() + ); + assert_eq!(disk.read_all("recovery", "object/xl.meta").await.unwrap().as_ref(), original); + assert!( + !disk + .io_get_object_path("recovery", &format!("object/{transaction}/{PREPARED}")) + .unwrap() + .exists() + ); + } + + #[tokio::test] + async fn delete_recovery_batch_inline_matches_sequential_and_survives_settlement() { + for commit in [false, true] { + let (_temp, disk, _, _) = fixture().await; + let mut original = FileMeta::new(); + let mut requests = Vec::new(); + for id in 1_u128..=64 { + let mut info = test_file_info( + "object", + Uuid::from_u128(id), + Some(Uuid::from_u128(1000 + id)), + Some(Bytes::from(vec![u8::try_from(id).expect("small id"); 9 * 1024])), + ); + info.set_inline_data(); + original.add_version(info).expect("seed inline version"); + if id <= 48 { + requests.push(FileInfo { + name: "object".into(), + version_id: Some(Uuid::from_u128(id)), + ..Default::default() + }); + } + } + let original_bytes = original.marshal_msg().expect("encode original"); + let meta_path = disk.io_get_object_path("recovery", "object/xl.meta").expect("metadata path"); + fs::write(&meta_path, &original_bytes).await.expect("persist version chain"); + let mut expected = original; + for request in &requests { + if let Some(dir) = expected.delete_version(request).expect("sequential delete") { + expected + .data + .remove(vec![request.version_id.expect("explicit version"), dir]) + .expect("sequential inline removal"); + } + } + let transaction = Uuid::new_v4(); + let errors = disk + .delete_versions_recoverable( + "recovery", + vec![FileInfoVersions { + name: "object".into(), + volume: "recovery".into(), + versions: requests, + ..Default::default() + }], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await; + assert_eq!(errors, vec![None]); + let applied = disk.read_all("recovery", "object/xl.meta").await.expect("applied metadata"); + assert_eq!(applied.as_ref(), expected.marshal_msg().expect("expected metadata")); + let target = disk + .read_all("recovery", &format!("object/{transaction}/{TARGET}")) + .await + .expect("prepared target"); + assert_eq!(applied, target, "publication must use the exact prepared target"); + let current = FileMeta::load(&applied).expect("load survivors"); + assert_eq!(current.versions.len(), 16); + for id in 49_u128..=64 { + let fi = current + .into_fileinfo("recovery", "object", &Uuid::from_u128(id).to_string(), true, false, true) + .expect("surviving version"); + assert_eq!( + fi.data.expect("surviving inline body").as_ref(), + vec![u8::try_from(id).expect("small id"); 9 * 1024] + ); + } + let observed = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("inspect prepared batch"); + if commit { + let committed = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + observed.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .expect("publish commit"); + let finished = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + committed.fingerprint, + DeleteRecoveryAction::FinishCommit, + ) + .await + .expect("finish cleanup"); + assert_eq!(finished.state, DeleteRecoveryState::CommitFinished); + assert_eq!(fs::read(&meta_path).await.expect("committed survivors"), applied.as_ref()); + } else { + let restored = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + observed.fingerprint, + DeleteRecoveryAction::Rollback, + ) + .await + .expect("rollback batch"); + assert_eq!(restored.state, DeleteRecoveryState::Absent); + assert_eq!(fs::read(&meta_path).await.expect("restored metadata"), original_bytes); + } + } + } + + #[tokio::test] + async fn delete_recovery_metadata_updates_publish_exact_prepared_target_bytes() { + for batch in [false, true] { + for transition in [false, true] { + let (_temp, disk, mut info, _) = fixture().await; + info.metadata.insert("x-amz-meta-first".into(), "retained-a".into()); + info.metadata.insert("x-amz-meta-second".into(), "retained-b".into()); + info.metadata + .insert(rustfs_filemeta::metadata_keys::RESTORE.into(), "ongoing-request=\"false\"".into()); + let original = test_meta(info); + fs::write(disk.io_get_object_path("recovery", "object/xl.meta").unwrap(), &original) + .await + .unwrap(); + let transaction = Uuid::new_v4(); + let mut change = request(); + if transition { + change.transition_status = "complete".into(); + change.transition_tier = "test-tier".into(); + change.transitioned_objname = "remote/object".into(); + } else { + change.expire_restored = true; + } + let opts = DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }; + if batch { + let errors = disk + .delete_versions_recoverable( + "recovery", + vec![FileInfoVersions { + name: "object".into(), + volume: "recovery".into(), + versions: vec![change], + ..Default::default() + }], + opts, + ) + .await; + assert_eq!(errors, vec![None]); + } else { + disk.delete_version_recoverable("recovery", "object", change, false, opts) + .await + .unwrap(); + } + let observed = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + assert_eq!(observed.current_digest, observed.journal.unwrap().target_digest); + let applied = disk.read_all("recovery", "object/xl.meta").await.unwrap(); + let prepared = disk + .read_all("recovery", &format!("object/{transaction}/{TARGET}")) + .await + .unwrap(); + assert_eq!(applied, prepared, "batch={batch}, transition={transition}"); + assert_ne!(applied.as_ref(), original); + } + } + } + + async fn legacy_fixture(disk: &LocalDisk, original: &[u8], transaction: Uuid) -> PathBuf { + let root = disk.io_get_object_path("recovery", "object").unwrap(); + fs::remove_file(root.join(STORAGE_FORMAT_FILE)).await.unwrap(); + fs::create_dir(root.join(transaction.to_string())).await.unwrap(); + fs::write(root.join(transaction.to_string()).join(STORAGE_FORMAT_FILE_BACKUP), original) + .await + .unwrap(); + root + } + + #[tokio::test] + async fn delete_recovery_live_repair_backup_preserves_legacy_delete_guards() { + for shape in ["live", "rollback-marker", "missing-current", "foreign-backup"] { + let (_temp, disk, mut info, _) = fixture().await; + let transaction = Uuid::new_v4(); + info.data_dir = Some(transaction); + let root = disk.io_get_object_path("recovery", "object").unwrap(); + let current = test_meta(info.clone()); + fs::write(root.join(STORAGE_FORMAT_FILE), ¤t).await.unwrap(); + let rollback = root.join(transaction.to_string()); + fs::create_dir(&rollback).await.unwrap(); + if shape == "foreign-backup" { + info.data_dir = Some(Uuid::new_v4()); + } + let backup = test_meta(info); + fs::write(rollback.join(STORAGE_FORMAT_FILE_BACKUP), &backup).await.unwrap(); + if shape == "rollback-marker" { + fs::write(rollback.join(DELETE_MARKER_ROLLBACK_FILE), []).await.unwrap(); + } else if shape == "missing-current" { + fs::remove_file(root.join(STORAGE_FORMAT_FILE)).await.unwrap(); + } + let inspection = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + assert_eq!( + inspection.state, + if shape == "live" { + DeleteRecoveryState::Absent + } else { + DeleteRecoveryState::LegacyAmbiguous + }, + "{shape}" + ); + assert_eq!(fs::read(rollback.join(STORAGE_FORMAT_FILE_BACKUP)).await.unwrap(), backup); + if shape != "missing-current" { + assert_eq!(fs::read(root.join(STORAGE_FORMAT_FILE)).await.unwrap(), current); + } + } + } + + #[tokio::test] + async fn delete_recovery_explicit_legacy_discard_accepts_old_empty_receipt() { + let (_temp, disk, _info, original) = fixture().await; + let transaction = Uuid::new_v4(); + let root = legacy_fixture(&disk, &original, transaction).await; + fs::write(root.join(transaction.to_string()).join(DELETE_QUORUM_COMMITTED_FILE), []) + .await + .unwrap(); + let state = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + assert_eq!(state.state, DeleteRecoveryState::LegacyAmbiguous); + assert!(state.explicit_delete_safe); + let cleared = disk + .settle_delete_transaction("recovery", "object", transaction, state.fingerprint, DeleteRecoveryAction::DiscardLegacy) + .await + .unwrap(); + assert_eq!(cleared.state, DeleteRecoveryState::Absent); + assert!(!root.join(transaction.to_string()).exists()); + } + + #[tokio::test] + async fn delete_recovery_prepared_explicit_delete_checks_original_header() { + for protected in ["none", "version", "lock", "internal-lock"] { + let (_temp, disk, mut info, _) = fixture().await; + match protected { + "version" => info.version_id = Some(Uuid::new_v4()), + "lock" => { + info.metadata.insert("x-amz-object-lock-mode".into(), "COMPLIANCE".into()); + } + "internal-lock" => rustfs_utils::http::metadata_compat::insert_str( + &mut info.metadata, + rustfs_utils::http::metadata_compat::SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, + "known-lock".to_owned(), + ), + _ => {} + } + fs::write(disk.io_get_object_path("recovery", "object/xl.meta").unwrap(), test_meta(info.clone())) + .await + .unwrap(); + let transaction = Uuid::new_v4(); + disk.delete_version_recoverable( + "recovery", + "object", + FileInfo { + version_id: info.version_id, + ..request() + }, + false, + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await + .unwrap(); + let state = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + assert_eq!(state.state, DeleteRecoveryState::NewPrepared); + assert_eq!(state.explicit_delete_safe, protected == "none", "original header {protected}"); + } + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_recovery_cancelled_restore_retains_actual_key_and_bucket_guards() { + use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks}; + use crate::disk::os::prepared_publication_test_hooks as hooks; + use crate::object_api::{ObjectOptions, PutObjReader}; + use crate::storage_api_contracts::bucket::{BucketOperations, MakeBucketOptions}; + use crate::storage_api_contracts::object::ObjectIO; + use futures::FutureExt; + use rustfs_heal_contracts::heal_channel::HealOpts; + + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let bucket = "recover-owned-timeout"; + store.make_bucket(bucket, &MakeBucketOptions::default()).await.unwrap(); + store + .put_object(bucket, "object", &mut PutObjReader::from_vec(vec![61; 4096]), &ObjectOptions::default()) + .await + .unwrap(); + let disks = store.pools[0].disk_set[0].disk_inventory().await; + let transaction = Uuid::new_v4(); + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.unwrap(); + for disk in disks.iter().flatten() { + disk.delete_version_recoverable( + bucket, + "object", + request(), + false, + DeleteOptions { + old_data_dir: Some(transaction), + delete_recovery_bucket_incarnation: Some(incarnation), + ..Default::default() + }, + ) + .await + .unwrap(); + } + let metadata = disks[0] + .as_ref() + .unwrap() + .get_object_path_for_io_if_local(bucket, "object/xl.meta") + .unwrap() + .unwrap(); + let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); + let hook = hooks::install_at(hooks::Stage::Rename, &metadata, move || { + let _ = entered_tx.send(()); + let _ = release_rx.recv(); + }); + let before = store.ctx.namespace_commit_generation(); + let recovering = store.clone(); + let mut recovery = + tokio::spawn(async move { recovering.recover_delete_transactions(bucket, &HealOpts::default()).await }); + let mut writer = None; + let mut bucket_writer = None; + let observations = std::panic::AssertUnwindSafe(async { + tokio::time::timeout(Duration::from_secs(10), entered_rx) + .await + .unwrap() + .unwrap(); + assert!(tokio::time::timeout(Duration::from_millis(100), &mut recovery).await.is_err()); + assert!(store.ctx.namespace_commits_pending()); + assert_eq!(store.ctx.namespace_commit_generation(), before + 1); + recovery.abort(); + assert!(recovery.await.unwrap_err().is_cancelled()); + assert!(store.ctx.namespace_commits_pending(), "late restore must retain namespace accounting"); + let writing = store.clone(); + writer = Some(tokio::spawn(async move { + writing + .put_object(bucket, "object", &mut PutObjReader::from_vec(vec![62; 4096]), &ObjectOptions::default()) + .await + })); + assert!( + tokio::time::timeout(Duration::from_millis(100), writer.as_mut().unwrap()) + .await + .is_err(), + "same-key PUT must wait for the late restore" + ); + let removing = store.clone(); + bucket_writer = Some(tokio::spawn(async move { removing.acquire_bucket_lifecycle_write_lock(bucket).await })); + assert!( + tokio::time::timeout(Duration::from_millis(100), bucket_writer.as_mut().unwrap()) + .await + .is_err(), + "bucket incarnation mutation must wait for the late restore" + ); + }) + .catch_unwind() + .await; + let _ = release_tx.send(()); + drop(hook); + if let Some(writer) = writer { + tokio::time::timeout(Duration::from_secs(10), writer) + .await + .unwrap() + .unwrap() + .unwrap(); + } + if let Some(writer) = bucket_writer { + drop( + tokio::time::timeout(Duration::from_secs(10), writer) + .await + .unwrap() + .unwrap() + .unwrap(), + ); + } + if let Err(panic) = observations { + std::panic::resume_unwind(panic); + } + } + + #[tokio::test] + async fn delete_recovery_explicit_legacy_discard_preserves_versions_locks_and_live_children() { + for protected in ["version", "lock", "internal-lock", "child", "unknown"] { + let (_temp, disk, mut info, original) = fixture().await; + let transaction = Uuid::new_v4(); + let root = legacy_fixture(&disk, &original, transaction).await; + let rollback = root.join(transaction.to_string()); + match protected { + "version" => { + info.version_id = Some(Uuid::new_v4()); + } + "lock" => { + info.metadata.insert("X-Amz-Object-Lock-Legal-Hold".into(), "ON".into()); + } + "internal-lock" => { + rustfs_utils::http::metadata_compat::insert_str( + &mut info.metadata, + rustfs_utils::http::metadata_compat::SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, + "old-lock-state".to_owned(), + ); + } + "child" => { + fs::create_dir(rollback.join("live-child")).await.unwrap(); + } + _ => { + fs::write(rollback.join("unknown"), b"preserve").await.unwrap(); + } + } + fs::write(rollback.join(STORAGE_FORMAT_FILE_BACKUP), test_meta(info)) + .await + .unwrap(); + match disk.inspect_delete_transaction("recovery", "object", transaction).await { + Ok(state) => { + assert!(!state.explicit_delete_safe, "protected shape {protected}"); + assert!( + disk.settle_delete_transaction( + "recovery", + "object", + transaction, + state.fingerprint, + DeleteRecoveryAction::DiscardLegacy + ) + .await + .is_err() + ); + } + Err(error) => assert!(matches!(error, DiskError::FileCorrupt | DiskError::InvalidPath)), + } + assert!(rollback.join(STORAGE_FORMAT_FILE_BACKUP).exists()); + } + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_recovery_explicit_legacy_discard_rejects_selected_symlink() { + let (temp, disk, _info, original) = fixture().await; + let transaction = Uuid::new_v4(); + let root = legacy_fixture(&disk, &original, transaction).await; + std::os::unix::fs::symlink(temp.path(), root.join(transaction.to_string()).join("linked")).unwrap(); + assert_eq!( + disk.inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap_err(), + DiskError::InvalidPath + ); + assert!(root.join(transaction.to_string()).join(STORAGE_FORMAT_FILE_BACKUP).exists()); + } + + #[tokio::test] + async fn delete_recovery_commit_retains_proof_until_explicit_finalization() { + let (_temp, disk, _info, _original) = fixture().await; + let transaction = Uuid::new_v4(); + let applied = apply(&disk, transaction).await; + let committed = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + applied.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .expect("durable set decision"); + let finished = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + committed.fingerprint, + DeleteRecoveryAction::FinishCommit, + ) + .await + .expect("physical cleanup"); + assert_eq!(finished.state, DeleteRecoveryState::CommitFinished); + assert_eq!( + disk.finish_internal_delete_residue("recovery", "object", None).await, + 0, + "legacy retry must not GC new decision proof" + ); + let receipt = disk + .io_get_object_path("recovery", &format!("object/{transaction}/{DELETE_QUORUM_COMMITTED_FILE}")) + .expect("receipt path"); + assert!(receipt.is_file()); + let settled = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + finished.fingerprint, + DeleteRecoveryAction::FinalizeCommit, + ) + .await + .expect("finalize after coordinator convergence proof"); + assert_eq!(settled.state, DeleteRecoveryState::Absent); + assert!(!disk.io_get_object_path("recovery", "object").expect("object path").exists()); + } + + #[tokio::test] + async fn delete_recovery_reclaims_legacy_reservation_after_commit_and_snapshot_release() { + for batch in [false, true] { + for snapshot in [false, true] { + let (temp, disk, _, _) = fixture().await; + let data_dir = Uuid::new_v4(); + let transaction = Uuid::new_v4(); + let root = disk.io_get_object_path("recovery", "object").expect("object path"); + let data = root.join(data_dir.to_string()); + fs::create_dir(&data).await.expect("external data directory"); + fs::write(data.join("part.1"), b"x").await.expect("external payload"); + let reservation = data.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4())); + fs::write(&reservation, []).await.expect("interrupted legacy reservation"); + fs::write( + root.join(STORAGE_FORMAT_FILE), + test_meta(test_file_info("object", Uuid::nil(), Some(data_dir), None)), + ) + .await + .expect("live metadata still owns reserved data"); + let path = format!("object/{data_dir}"); + let lease = if snapshot { + Some(disk.acquire_snapshot_lease("recovery", &path).await.expect("active reader")) + } else { + None + }; + let options = DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }; + if batch { + let errors = disk + .delete_versions_recoverable( + "recovery", + vec![FileInfoVersions { + name: "object".into(), + volume: "recovery".into(), + versions: vec![request()], + ..Default::default() + }], + options, + ) + .await; + assert_eq!(errors, vec![None]); + } else { + disk.delete_version_recoverable("recovery", "object", request(), false, options) + .await + .expect("apply single delete"); + } + let applied = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("applied"); + let committed = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + applied.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .expect("publish certified decision"); + let finish = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + committed.fingerprint, + DeleteRecoveryAction::FinishCommit, + ) + .await; + if let Some(lease) = lease { + assert_eq!(finish.expect_err("snapshot retains payload"), DiskError::DiskOngoingReq); + assert_eq!(fs::read(data.join("part.1")).await.expect("snapshot payload"), b"x"); + assert!(reservation.is_file()); + disk.release_snapshot_lease("recovery", &path, lease) + .await + .expect("release replays safe cleanup"); + } else { + assert_eq!( + finish.expect("old reservation must not block commit").state, + DeleteRecoveryState::CommitFinished + ); + } + assert!(!data.exists(), "batch={batch}, snapshot={snapshot}"); + drop(disk); + let endpoint = Endpoint::try_from(temp.path().to_str().expect("UTF-8 disk path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("reopen after cleanup"); + let observed = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("replay inspection"); + let finished = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + observed.fingerprint, + DeleteRecoveryAction::FinishCommit, + ) + .await + .expect("idempotent finish after restart"); + disk.settle_delete_transaction( + "recovery", + "object", + transaction, + finished.fingerprint, + DeleteRecoveryAction::FinalizeCommit, + ) + .await + .expect("finalize converged transaction"); + assert!(!root.exists(), "committed deletion leaves no physical residue"); + } + } + } + + #[tokio::test] + async fn delete_recovery_legacy_reservation_never_authorizes_unrelated_data() { + for shape in [ + "no-proof", + "foreign-commit", + "nil", + "invalid", + "directory", + "child", + "metadata", + ] { + let (_temp, disk, _, _) = fixture().await; + let transaction = Uuid::new_v4(); + let data_dir = Uuid::new_v4(); + let path = format!("object/{data_dir}"); + let data = disk.io_get_object_path("recovery", &path).expect("data path"); + fs::create_dir(&data).await.expect("data directory"); + fs::write(data.join("part.1"), b"preserve").await.expect("payload"); + let old = format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4()); + fs::write(data.join(&old), []).await.expect("old reservation"); + if shape != "no-proof" { + fs::write(data.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}")), []) + .await + .expect("current proof"); + } + match shape { + "foreign-commit" => fs::write(data.join(format!("{DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4())), []) + .await + .expect("foreign proof"), + "nil" => fs::write(data.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::nil())), []) + .await + .expect("nil reservation"), + "invalid" => fs::write(data.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}invalid")), []) + .await + .expect("malformed reservation"), + "directory" => { + fs::remove_file(data.join(&old)).await.expect("replace marker"); + fs::create_dir(data.join(&old)).await.expect("marker is not a regular file"); + } + "child" => fs::create_dir(data.join("child")).await.expect("UUID-prefix descendant"), + "metadata" => fs::write(data.join(STORAGE_FORMAT_FILE), b"child metadata") + .await + .expect("UUID-prefix object"), + _ => {} + } + let error = disk + .delete_unleased( + "recovery", + &path, + &DeleteOptions { + recursive: true, + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await + .expect_err("unproven or non-regular entries must prevent all cleanup"); + assert_eq!(error, DiskError::FileCorrupt, "shape={shape}"); + assert_eq!( + fs::read(data.join("part.1")).await.expect("payload survives"), + b"preserve", + "shape={shape}" + ); + assert!(data.join(old).exists(), "reservation survives failed preflight"); + } + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_recovery_legacy_reservation_rejects_symlink() { + let (temp, disk, _, _) = fixture().await; + let transaction = Uuid::new_v4(); + let path = format!("object/{}", Uuid::new_v4()); + let data = disk.io_get_object_path("recovery", &path).expect("data path"); + fs::create_dir(&data).await.expect("data directory"); + let external = temp.path().join("external"); + fs::write(&external, b"external").await.expect("external payload"); + fs::write(data.join("part.1"), b"preserve").await.expect("payload"); + fs::write(data.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}")), []) + .await + .expect("current proof"); + std::os::unix::fs::symlink( + &external, + data.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4())), + ) + .expect("symlink reservation"); + assert_eq!( + disk.delete_unleased( + "recovery", + &path, + &DeleteOptions { + recursive: true, + old_data_dir: Some(transaction), + ..Default::default() + } + ) + .await + .expect_err("non-regular reservation must fail closed"), + DiskError::FileCorrupt + ); + assert_eq!(fs::read(data.join("part.1")).await.expect("payload survives"), b"preserve"); + assert_eq!(fs::read(external).await.expect("symlink target survives"), b"external"); + } + + #[tokio::test] + async fn delete_recovery_fingerprint_rejects_new_generation() { + let (_temp, disk, mut info, _original) = fixture().await; + let transaction = Uuid::new_v4(); + let applied = apply(&disk, transaction).await; + info.mod_time = Some(OffsetDateTime::now_utc()); + info.data = Some(Bytes::from_static(b"new generation")); + let newer = test_meta(info); + fs::write(disk.io_get_object_path("recovery", "object/xl.meta").expect("metadata path"), &newer) + .await + .expect("concurrent new generation"); + assert!(matches!( + disk.settle_delete_transaction( + "recovery", + "object", + transaction, + applied.fingerprint, + DeleteRecoveryAction::PublishCommit + ) + .await, + Err(DiskError::OutdatedXLMeta) + )); + assert_eq!( + disk.read_all("recovery", "object/xl.meta") + .await + .expect("new generation remains") + .as_ref(), + newer + ); + } + + #[tokio::test] + async fn delete_recovery_untouched_peer_adopts_only_matching_generation() { + let (_temp, source, info, original) = fixture().await; + let (_peer_temp, peer, _peer_info, _) = fixture().await; + fs::write(peer.io_get_object_path("recovery", "object/xl.meta").expect("peer metadata"), &original) + .await + .expect("same original generation"); + let transaction = Uuid::new_v4(); + let applied = apply(&source, transaction).await; + let committed = source + .settle_delete_transaction( + "recovery", + "object", + transaction, + applied.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .expect("source durable decision"); + let authority = committed.journal.expect("source intent"); + let untouched = peer + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("untouched peer"); + let adopted = peer + .settle_delete_transaction( + "recovery", + "object", + transaction, + untouched.fingerprint, + DeleteRecoveryAction::AdoptCommit { + authority: Box::new(authority.clone()), + }, + ) + .await + .expect("generation-bound adoption"); + assert_eq!(adopted.state, DeleteRecoveryState::QuorumCommitted); + let finished = peer + .settle_delete_transaction("recovery", "object", transaction, adopted.fingerprint, DeleteRecoveryAction::FinishCommit) + .await + .expect("converge untouched peer"); + assert_eq!(finished.state, DeleteRecoveryState::CommitFinished); + assert!(finished.current_digest.is_none()); + let (_foreign_temp, foreign, _, _) = fixture().await; + let mut replacement = info; + replacement.mod_time = Some(OffsetDateTime::now_utc()); + let newer = test_meta(replacement); + fs::write( + foreign + .io_get_object_path("recovery", "object/xl.meta") + .expect("foreign metadata"), + &newer, + ) + .await + .expect("foreign generation"); + let observed = foreign + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("inspect foreign generation"); + assert!(matches!( + foreign + .settle_delete_transaction( + "recovery", + "object", + transaction, + observed.fingerprint, + DeleteRecoveryAction::AdoptCommit { + authority: Box::new(authority) + } + ) + .await, + Err(DiskError::OutdatedXLMeta) + )); + assert_eq!( + foreign + .read_all("recovery", "object/xl.meta") + .await + .expect("protected foreign generation") + .as_ref(), + newer + ); + } + + #[tokio::test] + async fn delete_recovery_adoption_preserves_unrelated_new_version() { + let (_source_temp, source, original_info, original) = fixture().await; + let (_peer_temp, peer, _, _) = fixture().await; + let transaction = Uuid::new_v4(); + let applied = apply(&source, transaction).await; + let committed = source + .settle_delete_transaction( + "recovery", + "object", + transaction, + applied.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .unwrap(); + let mut unrelated = original_info; + unrelated.version_id = Some(Uuid::new_v4()); + unrelated.mod_time = unrelated.mod_time.map(|time| time + time::Duration::seconds(1)); + let mut peer_meta = FileMeta::load(&original).unwrap(); + peer_meta.add_version(unrelated.clone()).unwrap(); + fs::write( + peer.io_get_object_path("recovery", "object/xl.meta").unwrap(), + peer_meta.marshal_msg().unwrap(), + ) + .await + .unwrap(); + let observed = peer + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + let adopted = peer + .settle_delete_transaction( + "recovery", + "object", + transaction, + observed.fingerprint, + DeleteRecoveryAction::AdoptCommit { + authority: Box::new(committed.journal.unwrap()), + }, + ) + .await + .unwrap(); + peer.settle_delete_transaction( + "recovery", + "object", + transaction, + adopted.fingerprint, + DeleteRecoveryAction::FinishCommit, + ) + .await + .unwrap(); + let remaining = FileMeta::load(&peer.read_all("recovery", "object/xl.meta").await.unwrap()).unwrap(); + assert_eq!(remaining.versions.len(), 1); + assert_eq!(remaining.versions[0].header.version_id, unrelated.version_id); + } + + #[tokio::test] + async fn delete_recovery_absent_replay_witness_cannot_acquire_future_predecessors() { + let (_source_temp, source, _, _) = fixture().await; + let (_peer_temp, peer, _, _) = fixture().await; + let transaction = Uuid::new_v4(); + let source_state = apply(&source, transaction).await; + let committed = source + .settle_delete_transaction( + "recovery", + "object", + transaction, + source_state.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .unwrap(); + let authority = committed.journal.unwrap(); + assert!(authority.predecessors.is_empty()); + let future = Uuid::new_v4(); + apply(&peer, future).await; + let absent = peer + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + assert_eq!(absent.state, DeleteRecoveryState::Absent); + assert!(absent.current_digest.is_none()); + let adopted = peer + .settle_delete_transaction( + "recovery", + "object", + transaction, + absent.fingerprint, + DeleteRecoveryAction::AdoptCommit { + authority: Box::new(authority.clone()), + }, + ) + .await + .unwrap(); + let witness = adopted.journal.unwrap(); + assert_eq!(witness.predecessors, authority.predecessors); + assert!(!witness.predecessors.contains(&future)); + assert_eq!(witness.original_versions, authority.original_versions); + assert_eq!(witness.deleted_versions, authority.deleted_versions); + assert_eq!(witness.bucket_incarnation, authority.bucket_incarnation); + assert!(witness.original_digest.is_none() && witness.target_digest.is_none() && witness.data_dirs.is_empty()); + } + + #[tokio::test] + async fn delete_recovery_raw_discovery_keeps_malformed_records_and_later_keys() { + let (temp, disk, _, _) = fixture().await; + let transaction = Uuid::new_v4(); + apply(&disk, transaction).await; + let bad = temp + .path() + .join("recovery/object") + .join(transaction.to_string()) + .join(PREPARED); + fs::write(&bad, b"{invalid journal").await.expect("corrupt prepared record"); + let legacy = Uuid::new_v4(); + let later = temp.path().join("recovery/later").join(legacy.to_string()); + fs::create_dir_all(&later).await.expect("another hidden key"); + fs::write(later.join(STORAGE_FORMAT_FILE_BACKUP), b"legacy backup") + .await + .expect("later transaction"); + let mut cursor = None; + let mut found = BTreeSet::new(); + loop { + let page = disk + .discover_delete_transactions("recovery", None, cursor, 1) + .await + .expect("bounded discovery continues"); + found.extend(page.transactions.into_iter().map(|id| (id.object, id.transaction))); + cursor = page.next; + if let Some(cursor) = cursor.as_ref() { + assert!(!disk.renew_delete_recovery_cursor("different-bucket", cursor)); + assert!(disk.renew_delete_recovery_cursor("recovery", cursor)); + } + if cursor.is_none() { + break; + } + } + assert!(found.contains(&("object".to_owned(), transaction))); + assert!(found.contains(&("later".to_owned(), legacy))); + assert_eq!( + disk.inspect_delete_transaction("recovery", "object", transaction) + .await + .expect_err("inspection still rejects corruption"), + DiskError::FileCorrupt + ); + assert_eq!(fs::read(bad).await.expect("corrupt evidence retained"), b"{invalid journal"); + assert_eq!( + fs::read(later.join(STORAGE_FORMAT_FILE_BACKUP)) + .await + .expect("legacy evidence retained"), + b"legacy backup" + ); + } + + async fn prepare_discovery_key( + disk: &LocalDisk, + object: &str, + transaction: Uuid, + metadata: &[u8], + ) -> DeleteRecoveryInspection { + let root = disk.io_get_object_path("recovery", object).expect("discovery fixture path"); + fs::create_dir_all(&root).await.expect("discovery fixture directory"); + fs::write(root.join(STORAGE_FORMAT_FILE), metadata) + .await + .expect("discovery fixture metadata"); + let mut request = request(); + request.name = object.to_owned(); + disk.delete_version_recoverable( + "recovery", + object, + request, + false, + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await + .expect("prepare discoverable transaction"); + disk.inspect_delete_transaction("recovery", object, transaction) + .await + .expect("healthy prepared transaction") + } + + async fn discover_all_keys(disk: &LocalDisk, object: Option<&str>) -> BTreeSet<(String, Uuid)> { + let mut cursor = None; + let mut found = BTreeSet::new(); + for _ in 0..64 { + let page = disk + .discover_delete_transactions("recovery", object, cursor, 1) + .await + .expect("record corruption must not stop bounded discovery"); + found.extend(page.transactions.into_iter().map(|id| (id.object, id.transaction))); + cursor = page.next; + if cursor.is_none() { + return found; + } + } + panic!("discovery did not finish the small fixture within its page bound"); + } + + #[tokio::test] + async fn delete_recovery_raw_discovery_isolates_corrupt_record_bindings() { + for record_name in [PREPARED, DELETE_QUORUM_COMMITTED_FILE] { + for damage in ["protocol", "volume", "transaction", "object", "unsafe_object", "invalid_json"] { + let (temp, disk, _, original) = fixture().await; + let transaction = Uuid::new_v4(); + let prepared = apply(&disk, transaction).await; + let root = temp.path().join("recovery/object").join(transaction.to_string()); + if record_name == DELETE_QUORUM_COMMITTED_FILE { + disk.settle_delete_transaction( + "recovery", + "object", + transaction, + prepared.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .expect("write receipt fixture"); + fs::remove_file(root.join(PREPARED)) + .await + .expect("discover the receipt without its prepared journal"); + } + let later = Uuid::new_v4(); + let later_before = prepare_discovery_key(&disk, "later", later, &original).await; + let path = root.join(record_name); + let mut record: serde_json::Value = + serde_json::from_slice(&fs::read(&path).await.expect("original record")).expect("valid fixture record"); + let journal = if record_name == PREPARED { + &mut record + } else { + record.get_mut("journal").expect("receipt journal") + }; + match damage { + "protocol" => journal["protocol_version"] = serde_json::json!(DELETE_RECOVERY_PROTOCOL_VERSION + 1), + "volume" => journal["volume"] = serde_json::json!("another-bucket"), + "transaction" => journal["transaction"] = serde_json::json!(Uuid::new_v4()), + "object" => journal["object"] = serde_json::json!("another-key"), + "unsafe_object" => journal["object"] = serde_json::json!("../another-key"), + "invalid_json" => {} + _ => unreachable!("fixture damage is exhaustive"), + } + let bytes = if damage == "invalid_json" { + b"{invalid record".to_vec() + } else { + serde_json::to_vec(&record).expect("encode damaged record") + }; + fs::write(&path, &bytes).await.expect("write corrupt record"); + + let found = discover_all_keys(&disk, None).await; + assert_eq!( + found, + BTreeSet::from([("object".to_owned(), transaction), ("later".to_owned(), later)]), + "{record_name}: {damage}" + ); + assert_eq!( + discover_all_keys(&disk, Some("object")).await, + BTreeSet::from([("object".to_owned(), transaction)]) + ); + assert_eq!( + disk.inspect_delete_transaction("recovery", "object", transaction) + .await + .expect_err("canonical inspection must reject corrupt record bindings"), + DiskError::FileCorrupt, + "{record_name}: {damage}" + ); + assert_eq!(fs::read(path).await.expect("retained corrupt evidence"), bytes); + assert_eq!( + disk.inspect_delete_transaction("recovery", "later", later) + .await + .expect("later transaction remains healthy") + .fingerprint, + later_before.fingerprint, + "discovery does not mutate a later healthy transaction" + ); + } + } + } + + #[tokio::test] + async fn delete_recovery_raw_discovery_keeps_corrupt_write_backups() { + for corrupt_current in [false, true] { + let (temp, disk, _, original) = fixture().await; + let transaction = Uuid::new_v4(); + let root = temp.path().join("recovery/object"); + let backup = root.join(transaction.to_string()).join(STORAGE_FORMAT_FILE_BACKUP); + fs::create_dir_all(backup.parent().expect("rollback directory")) + .await + .expect("rollback directory"); + fs::write(&backup, &original).await.expect("valid write backup"); + let damaged = if corrupt_current { + root.join(STORAGE_FORMAT_FILE) + } else { + backup + }; + fs::write(&damaged, b"corrupt metadata") + .await + .expect("record-local metadata corruption"); + let later = Uuid::new_v4(); + prepare_discovery_key(&disk, "later", later, &original).await; + + assert!(!live_write_backup(&root, transaction).expect("corrupt decoded metadata is not a proven live backup")); + assert_eq!( + discover_all_keys(&disk, None).await, + BTreeSet::from([("object".to_owned(), transaction), ("later".to_owned(), later)]) + ); + assert_eq!( + disk.inspect_delete_transaction("recovery", "object", transaction) + .await + .expect_err("inspection rejects corrupt backup metadata"), + DiskError::FileCorrupt + ); + assert_eq!(fs::read(damaged).await.expect("corrupt metadata remains untouched"), b"corrupt metadata"); + } + } + + #[tokio::test] + async fn delete_recovery_raw_discovery_preserves_bound_directory_keys_and_exact_scope() { + let (_temp, disk, _, original) = fixture().await; + let directory_transaction = Uuid::new_v4(); + prepare_discovery_key(&disk, "directory/", directory_transaction, &original).await; + let literal_transaction = Uuid::new_v4(); + prepare_discovery_key(&disk, "literal__XLDIR__", literal_transaction, &original).await; + + assert_eq!( + discover_all_keys(&disk, None).await, + BTreeSet::from([ + ("directory/".to_owned(), directory_transaction), + ("literal__XLDIR__".to_owned(), literal_transaction), + ]) + ); + assert_eq!( + discover_all_keys(&disk, Some("directory/")).await, + BTreeSet::from([("directory/".to_owned(), directory_transaction)]) + ); + assert_eq!( + discover_all_keys(&disk, Some("directory")).await, + BTreeSet::from([("directory".to_owned(), directory_transaction)]), + "a journal cannot redirect an exact object scope to a trailing-slash alias" + ); + assert_eq!( + disk.inspect_delete_transaction("recovery", "directory", directory_transaction) + .await + .expect_err("the alternate scope is not journal authority"), + DiskError::FileCorrupt + ); + assert_eq!( + disk.inspect_delete_transaction("recovery", "directory/", directory_transaction) + .await + .expect("the bound trailing-slash object remains valid") + .state, + DeleteRecoveryState::NewPrepared + ); + + let literal_record = disk + .io_get_object_path("recovery", "literal__XLDIR__") + .expect("literal key path") + .join(literal_transaction.to_string()) + .join(PREPARED); + let mut record: DeleteRecoveryJournal = + serde_json::from_slice(&fs::read(&literal_record).await.expect("literal journal")).expect("valid literal journal"); + record.object = "literal/".to_owned(); + let damaged = serde_json::to_vec(&record).expect("encode invalid literal binding"); + fs::write(&literal_record, &damaged) + .await + .expect("inject invalid literal binding"); + let found = discover_all_keys(&disk, None).await; + assert!(found.contains(&("literal__XLDIR__".to_owned(), literal_transaction))); + assert!( + !found.contains(&("literal/".to_owned(), literal_transaction)), + "physical suffixes are never decoded into authority" + ); + assert_eq!( + disk.inspect_delete_transaction("recovery", "literal__XLDIR__", literal_transaction) + .await + .expect_err("the physical fallback cannot certify an invalid logical binding"), + DiskError::FileCorrupt + ); + assert_eq!(fs::read(literal_record).await.expect("retained invalid literal binding"), damaged); + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_recovery_raw_discovery_keeps_structural_backup_errors_fatal() { + let (temp, disk, _, original) = fixture().await; + let transaction = Uuid::new_v4(); + let root = temp.path().join("recovery/object"); + let rollback = root.join(transaction.to_string()); + fs::create_dir_all(&rollback).await.expect("backup directory"); + fs::write(rollback.join(STORAGE_FORMAT_FILE_BACKUP), &original) + .await + .expect("valid backup metadata"); + let outside = tempfile::tempdir().expect("external target"); + let sentinel = outside.path().join("sentinel"); + fs::write(&sentinel, b"external bytes").await.expect("external sentinel"); + std::os::unix::fs::symlink(&sentinel, rollback.join(TARGET)).expect("structurally invalid delete target"); + + assert!( + live_write_backup(&root, transaction).is_err(), + "only decoded record corruption is isolated" + ); + assert!( + disk.discover_delete_transactions("recovery", Some("object"), None, 128) + .await + .is_err(), + "a structural error must terminate discovery" + ); + assert_eq!(fs::read(sentinel).await.expect("unchanged external sentinel"), b"external bytes"); + assert_eq!( + fs::read(rollback.join(STORAGE_FORMAT_FILE_BACKUP)) + .await + .expect("unchanged backup"), + original + ); + } + + #[tokio::test] + async fn delete_recovery_evidence_probes_do_not_register_public_cursors() { + let (_temp, disk, _, _) = fixture().await; + apply(&disk, Uuid::new_v4()).await; + for _ in 0..256 { + assert!( + disk.delete_recovery_has_evidence("recovery", "object") + .await + .expect("positive private probe") + ); + } + assert!( + DISCOVERY_SESSIONS + .lock() + .values() + .all(|session| session.disk_root != disk.io_root()), + "early positive probes must not consume any public cursor slots" + ); + } + + #[tokio::test] + async fn delete_recovery_evidence_probe_advances_empty_bounded_pages() { + let (_temp, disk, _, _) = fixture().await; + let path = disk.delete_discovery_path("recovery", Some("object")).expect("probe scope"); + let filler_id = Uuid::new_v4().to_string(); + let filler = path.join(&filler_id); + fs::create_dir_all(&filler).await.expect("non-evidence directory"); + for index in 0..5000 { + fs::write(filler.join(format!("part-{index:04}")), []) + .await + .expect("non-evidence entry"); + } + let session = disk + .new_delete_discovery_session("recovery", Some("object"), &path) + .expect("private session"); + let (first, next) = disk + .advance_delete_discovery(&path, session, 1) + .await + .expect("bounded first step"); + assert!(first.is_empty()); + assert!(next.is_some(), "an empty 4096-entry step must retain real continuation state"); + drop(next); + assert!( + !disk + .delete_recovery_has_evidence("recovery", "object") + .await + .expect("negative probe reaches EOF") + ); + + let transaction = Uuid::new_v4(); + let proof = path.join(transaction.to_string()); + fs::create_dir_all(&proof).await.expect("later evidence directory"); + fs::write(proof.join(PREPARED), b"{invalid journal") + .await + .expect("discoverable record-local corruption"); + let mut session = disk + .new_delete_discovery_session("recovery", Some("object"), &path) + .expect("private late-evidence session"); + // Seed a valid DFS continuation so this assertion is independent of + // filesystem directory ordering: the large empty frame runs first. + session.frames = vec![ + (transaction.to_string(), std::fs::read_dir(&proof).expect("later evidence frame")), + (filler_id, std::fs::read_dir(&filler).expect("empty intermediate frame")), + ]; + assert!( + session + .has_evidence(&disk, &path) + .await + .expect("late evidence survives an empty intermediate page") + ); + assert!( + DISCOVERY_SESSIONS + .lock() + .values() + .all(|session| session.disk_root != disk.io_root()), + "private continuation never enters the public cursor table" + ); + } + + #[tokio::test] + async fn delete_recovery_cancelled_evidence_probe_drops_private_session() { + let (_temp, disk, _, _) = fixture().await; + apply(&disk, Uuid::new_v4()).await; + let path = disk.delete_discovery_path("recovery", Some("object")).expect("probe scope"); + let lease = os::acquire_namespace_mutation_lease_with_owner(&path, None).await; + let mut probe = Box::pin(disk.delete_recovery_has_evidence("recovery", "object")); + assert!( + futures::poll!(probe.as_mut()).is_pending(), + "the private session waits for the held namespace lease" + ); + drop(probe); + assert!( + DISCOVERY_SESSIONS + .lock() + .values() + .all(|session| session.disk_root != disk.io_root()), + "cancelling before a page runs must not retain a global cursor" + ); + drop(lease); + assert!( + disk.delete_recovery_has_evidence("recovery", "object") + .await + .expect("later probes remain live") + ); + assert!( + DISCOVERY_SESSIONS + .lock() + .values() + .all(|session| session.disk_root != disk.io_root()) + ); + } + + #[tokio::test] + async fn delete_recovery_raw_discovery_paginates_and_binds_exact_key_scope() { + let (temp, disk, _info, _original) = fixture().await; + for index in 0..512 { + let path = temp + .path() + .join("recovery") + .join(format!("key-{index:04}")) + .join(Uuid::new_v4().to_string()); + fs::create_dir_all(&path).await.expect("hidden transaction directory"); + fs::write(path.join(STORAGE_FORMAT_FILE_BACKUP), b"legacy backup") + .await + .expect("hidden backup"); + } + let mut cursor = None; + let mut found = BTreeSet::new(); + loop { + let page = disk + .discover_delete_transactions("recovery", None, cursor, 13) + .await + .expect("bounded raw discovery"); + found.extend(page.transactions.into_iter().map(|id| id.object)); + cursor = page.next; + if cursor.is_none() { + break; + } + } + assert_eq!(found.len(), 512, "every hidden legacy transaction must be discovered across pages"); + let scoped = disk + .discover_delete_transactions("recovery", Some("key-0000"), None, 1) + .await + .expect("shallow exact-key discovery"); + assert_eq!(scoped.transactions.len(), 1); + assert_eq!(scoped.transactions[0].object, "key-0000"); + if let Some(cursor) = scoped.next { + assert!( + disk.discover_delete_transactions("recovery", Some("key-0001"), Some(cursor), 1) + .await + .is_err(), + "a cursor cannot cross object scopes" + ); + } + } + + #[tokio::test] + async fn delete_recovery_gc_interruption_replays_finished_receipt_without_journal() { + let (_temp, disk, _info, _) = fixture().await; + let transaction = Uuid::new_v4(); + let applied = apply(&disk, transaction).await; + let committed = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + applied.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .expect("record decision"); + disk.settle_delete_transaction( + "recovery", + "object", + transaction, + committed.fingerprint, + DeleteRecoveryAction::FinishCommit, + ) + .await + .expect("finish physical cleanup"); + let root = disk + .io_get_object_path("recovery", &format!("object/{transaction}")) + .expect("transaction root"); + fs::remove_file(root.join(PREPARED)) + .await + .expect("simulate partially completed proof GC"); + fs::remove_file(root.join(FINISHED)) + .await + .expect("simulate removed completion marker"); + let observed = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("receipt preserves completed intent"); + assert_eq!(observed.state, DeleteRecoveryState::CommitFinished); + let repeated = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + observed.fingerprint, + DeleteRecoveryAction::FinishCommit, + ) + .await + .expect("finish is idempotent after interrupted GC"); + let finalized = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + repeated.fingerprint, + DeleteRecoveryAction::FinalizeCommit, + ) + .await + .expect("finish proof GC"); + assert_eq!(finalized.state, DeleteRecoveryState::Absent); + } + + #[tokio::test] + async fn delete_recovery_completed_restore_without_backup_can_finish_rollback() { + let (_temp, disk, _info, original) = fixture().await; + let transaction = Uuid::new_v4(); + apply(&disk, transaction).await; + let root = disk.io_get_object_path("recovery", "object").expect("object root"); + fs::rename( + root.join(transaction.to_string()).join(STORAGE_FORMAT_FILE_BACKUP), + root.join(STORAGE_FORMAT_FILE), + ) + .await + .expect("simulate crash after rollback metadata rename"); + let observed = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("already restored generation"); + assert_eq!(observed.state, DeleteRecoveryState::NewPrepared); + assert!( + observed.rollback_data_complete, + "restored original replaces consumed backup as rollback evidence" + ); + disk.settle_delete_transaction("recovery", "object", transaction, observed.fingerprint, DeleteRecoveryAction::Rollback) + .await + .expect("finish interrupted rollback"); + assert_eq!( + disk.read_all("recovery", "object/xl.meta") + .await + .expect("original remains readable") + .as_ref(), + original + ); + } + + #[tokio::test] + async fn delete_recovery_completed_restore_without_target_can_finish_versioned_rollback() { + let (_temp, disk, info, original) = fixture().await; + let mut remaining = info; + remaining.version_id = Some(Uuid::new_v4()); + remaining.mod_time = remaining.mod_time.map(|time| time + time::Duration::seconds(1)); + let mut metadata = FileMeta::load(&original).unwrap(); + metadata.add_version(remaining).unwrap(); + let original = metadata.marshal_msg().unwrap(); + let root = disk.io_get_object_path("recovery", "object").unwrap(); + fs::write(root.join(STORAGE_FORMAT_FILE), &original).await.unwrap(); + let transaction = Uuid::new_v4(); + let observed = apply(&disk, transaction).await; + assert!(observed.journal.as_ref().unwrap().target_digest.is_some()); + let rollback = root.join(transaction.to_string()); + let target = fs::read(rollback.join(TARGET)).await.unwrap(); + fs::remove_file(rollback.join(TARGET)).await.unwrap(); + assert!(matches!( + disk.inspect_delete_transaction("recovery", "object", transaction).await, + Err(DiskError::FileCorrupt) + )); + fs::write(rollback.join(TARGET), target).await.unwrap(); + fs::rename(rollback.join(STORAGE_FORMAT_FILE_BACKUP), root.join(STORAGE_FORMAT_FILE)) + .await + .unwrap(); + fs::remove_file(rollback.join(TARGET)).await.unwrap(); + let receipt_path = rollback.join(DELETE_QUORUM_COMMITTED_FILE); + fs::write( + &receipt_path, + serde_json::to_vec(&CommitReceipt { + journal: observed.journal.unwrap(), + finished: false, + }) + .unwrap(), + ) + .await + .unwrap(); + assert!(matches!( + disk.inspect_delete_transaction("recovery", "object", transaction).await, + Err(DiskError::FileCorrupt) + )); + fs::remove_file(receipt_path).await.unwrap(); + let retry = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + assert_eq!(retry.state, DeleteRecoveryState::NewPrepared); + assert!(retry.rollback_data_complete); + disk.settle_delete_transaction("recovery", "object", transaction, retry.fingerprint, DeleteRecoveryAction::Rollback) + .await + .expect("finish cleanup of an exactly restored versioned generation"); + assert_eq!(fs::read(root.join(STORAGE_FORMAT_FILE)).await.unwrap(), original); + assert!(!rollback.exists()); + } + + #[tokio::test] + async fn delete_recovery_absent_remaining_metadata_requires_existing_commit_receipt() { + let (_temp, disk, info, original) = fixture().await; + let mut remaining = info; + remaining.version_id = Some(Uuid::new_v4()); + remaining.mod_time = remaining.mod_time.map(|time| time + time::Duration::seconds(1)); + let mut metadata = FileMeta::load(&original).unwrap(); + metadata.add_version(remaining).unwrap(); + let root = disk.io_get_object_path("recovery", "object").unwrap(); + fs::write(root.join(STORAGE_FORMAT_FILE), metadata.marshal_msg().unwrap()) + .await + .unwrap(); + let transaction = Uuid::new_v4(); + let applied = apply(&disk, transaction).await; + assert!(applied.journal.as_ref().unwrap().target_digest.is_some()); + let rollback = root.join(transaction.to_string()); + fs::remove_file(root.join(STORAGE_FORMAT_FILE)).await.unwrap(); + let undecided = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + assert_eq!(undecided.state, DeleteRecoveryState::NewPrepared); + assert!( + !undecided.commit_data_safe, + "absence cannot authorize a target-Some undecided transaction" + ); + assert!(matches!( + disk.settle_delete_transaction( + "recovery", + "object", + transaction, + undecided.fingerprint, + DeleteRecoveryAction::PublishCommit + ) + .await, + Err(DiskError::OutdatedXLMeta) + )); + assert!(rollback.join(PREPARED).is_file()); + assert!(rollback.join(STORAGE_FORMAT_FILE_BACKUP).is_file()); + assert!(!rollback.join(DELETE_QUORUM_COMMITTED_FILE).exists()); + fs::copy(rollback.join(TARGET), root.join(STORAGE_FORMAT_FILE)).await.unwrap(); + let applied = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + disk.settle_delete_transaction( + "recovery", + "object", + transaction, + applied.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .unwrap(); + fs::remove_file(root.join(STORAGE_FORMAT_FILE)).await.unwrap(); + let certified = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + assert_eq!(certified.state, DeleteRecoveryState::QuorumCommitted); + assert!(certified.commit_data_safe); + let finished = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + certified.fingerprint, + DeleteRecoveryAction::FinishCommit, + ) + .await + .expect("certified obsolete version cleanup has no live metadata references"); + assert_eq!(finished.state, DeleteRecoveryState::CommitFinished); + assert!( + !root.join(STORAGE_FORMAT_FILE).exists(), + "cleanup must never republish the old remaining version" + ); + assert!( + rollback.join(DELETE_QUORUM_COMMITTED_FILE).is_file(), + "set-wide GC still owns the final witness" + ); + } + + #[tokio::test] + async fn delete_recovery_partial_publication_adopts_same_intent_without_replacing_local_journal() { + let (_temp, disk, mut original, _) = fixture().await; + let (_peer_temp, peer, _, _) = fixture().await; + let mut remaining = original.clone(); + remaining.version_id = Some(Uuid::new_v4()); + remaining.mod_time = remaining.mod_time.map(|time| time + time::Duration::seconds(1)); + for (index, target) in [(1, &disk), (2, &peer)] { + original.erasure.index = index; + remaining.erasure.index = index; + let mut metadata = FileMeta::load(&test_meta(original.clone())).unwrap(); + metadata.add_version(remaining.clone()).unwrap(); + fs::write( + target.io_get_object_path("recovery", "object/xl.meta").unwrap(), + metadata.marshal_msg().unwrap(), + ) + .await + .unwrap(); + } + let transaction = Uuid::new_v4(); + let local = apply(&disk, transaction).await; + let root = disk.io_get_object_path("recovery", "object").unwrap(); + let rollback = root.join(transaction.to_string()); + let prepared = fs::read(rollback.join(PREPARED)).await.unwrap(); + let target = fs::read(rollback.join(TARGET)).await.unwrap(); + fs::remove_file(root.join(STORAGE_FORMAT_FILE)).await.unwrap(); + let applied = apply(&peer, transaction).await; + let committed = peer + .settle_delete_transaction( + "recovery", + "object", + transaction, + applied.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .unwrap(); + let authority = committed.journal.unwrap(); + assert_ne!( + local.journal.as_ref().unwrap().original_digest, + authority.original_digest, + "each drive owns distinct raw metadata" + ); + let mut invalid = Vec::new(); + let mut changed = authority.clone(); + changed.bucket_incarnation = Some(Uuid::new_v4()); + invalid.push(changed); + let mut changed = authority.clone(); + changed.deleted_versions.push(Uuid::new_v4()); + invalid.push(changed); + let mut changed = authority.clone(); + changed.requests[0].version_id = Some(Uuid::new_v4()); + invalid.push(changed); + let mut changed = authority.clone(); + changed + .original_versions + .iter_mut() + .find(|version| version.version_id.unwrap_or_default().is_nil()) + .unwrap() + .metadata_digest[0] ^= 1; + invalid.push(changed); + for authority in invalid { + let observed = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + assert!(matches!( + disk.settle_delete_transaction( + "recovery", + "object", + transaction, + observed.fingerprint, + DeleteRecoveryAction::AdoptCommit { + authority: Box::new(authority) + } + ) + .await, + Err(DiskError::OutdatedXLMeta) + )); + assert!(!rollback.join(DELETE_QUORUM_COMMITTED_FILE).exists()); + } + let observed = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + assert!(!observed.commit_data_safe); + let adopted = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + observed.fingerprint, + DeleteRecoveryAction::AdoptCommit { + authority: Box::new(authority), + }, + ) + .await + .expect("same-T certified intent completes partial publication"); + assert_eq!( + adopted.journal, local.journal, + "receipt preserves local hashes and data-directory ownership" + ); + assert_eq!(fs::read(rollback.join(PREPARED)).await.unwrap(), prepared); + assert_eq!(fs::read(rollback.join(TARGET)).await.unwrap(), target); + disk.settle_delete_transaction( + "recovery", + "object", + transaction, + adopted.fingerprint, + DeleteRecoveryAction::FinishCommit, + ) + .await + .unwrap(); + assert!(!root.join(STORAGE_FORMAT_FILE).exists()); + assert!(rollback.join(DELETE_QUORUM_COMMITTED_FILE).is_file()); + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_recovery_rollback_marker_sync_failure_retains_journal() { + for mode in [DurabilityMode::Strict, DurabilityMode::None] { + let _mode = durability_mode_override::set(mode); + let (_temp, disk, _info, original) = fixture().await; + let transaction = Uuid::new_v4(); + apply(&disk, transaction).await; + let root = disk.io_get_object_path("recovery", "object").unwrap(); + let data = root.join(Uuid::new_v4().to_string()); + fs::create_dir(&data).await.unwrap(); + let marker = data.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}")); + fs::write(&marker, []).await.unwrap(); + let prepared = root.join(transaction.to_string()).join(PREPARED); + let observed = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + os::fsync_dir_recorder::set_failure(&data, ErrorKind::Other); + let result = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + observed.fingerprint, + DeleteRecoveryAction::Rollback, + ) + .await; + assert_eq!(fs::read(root.join(STORAGE_FORMAT_FILE)).await.unwrap(), original); + assert!(!marker.exists(), "the unlink precedes the durability barrier"); + if mode == DurabilityMode::Strict { + assert!(result.is_err(), "a failed marker-directory sync must stop proof removal"); + assert!(prepared.exists(), "rollback evidence survives the sync failure"); + let retry = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .unwrap(); + disk.settle_delete_transaction( + "recovery", + "object", + transaction, + retry.fingerprint, + DeleteRecoveryAction::Rollback, + ) + .await + .expect("retry syncs the directory even though its marker is already absent"); + assert!(os::fsync_dir_recorder::was_fsynced(&data)); + } else { + result.expect("none mode does not add a marker durability barrier"); + assert!(!os::fsync_dir_recorder::was_fsynced(&data)); + assert!(os::fsync_dir_recorder::take_failure(&data).is_some()); + } + assert!(!prepared.exists()); + } + } + + #[tokio::test] + async fn delete_recovery_validates_staged_parts_before_restoring_metadata() { + use crate::erasure::coding::BitrotWriter; + let (_temp, disk, _info, _) = fixture().await; + let data_dir = Uuid::new_v4(); + let payload = Bytes::from_static(b"complete staged external payload"); + let mut info = test_file_info("object", Uuid::nil(), Some(data_dir), None); + info.size = i64::try_from(payload.len()).expect("payload length"); + info.erasure.block_size = payload.len(); + info.parts[0].size = payload.len(); + info.parts[0].actual_size = info.size; + let mut second = info.parts[0].clone(); + second.number = 2; + info.parts.push(second); + info.size *= 2; + let original = test_meta(info.clone()); + let root = disk.io_get_object_path("recovery", "object").expect("object root"); + fs::write(root.join(STORAGE_FORMAT_FILE), &original) + .await + .expect("external original metadata"); + let mut writer = BitrotWriter::new( + std::io::Cursor::new(Vec::new()), + info.erasure.shard_size(), + HashAlgorithm::HighwayHash256S, + ); + assert_eq!(writer.write(&payload).await.expect("encode intact external shard"), payload.len()); + writer.shutdown().await.expect("flush shard encoding"); + let encoded = writer.into_inner().into_inner(); + fs::create_dir_all(root.join(data_dir.to_string())) + .await + .expect("external data directory"); + fs::write(root.join(data_dir.to_string()).join("part.1"), &encoded) + .await + .expect("intact external part"); + fs::write(root.join(data_dir.to_string()).join("part.2"), &encoded) + .await + .unwrap(); + let transaction = Uuid::new_v4(); + let _ = disk + .prepare_delete_recovery( + "recovery", + "object", + Some(&original), + &[request()], + None, + DeletePreparationContext { + transaction, + bucket_incarnation: None, + namespace_owner: None, + prepared_publication: None, + }, + ) + .await + .expect("prepared intent"); + fs::rename( + root.join(data_dir.to_string()), + root.join(transaction.to_string()).join(data_dir.to_string()), + ) + .await + .expect("simulate staged legacy-compatible data"); + fs::create_dir(root.join(data_dir.to_string())).await.unwrap(); + fs::rename( + root.join(transaction.to_string()).join(data_dir.to_string()).join("part.1"), + root.join(data_dir.to_string()).join("part.1"), + ) + .await + .unwrap(); + fs::remove_file(root.join(STORAGE_FORMAT_FILE)) + .await + .expect("simulate applied metadata removal"); + let observed = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("inspect complete staged data"); + assert!(observed.rollback_data_complete); + disk.settle_delete_transaction("recovery", "object", transaction, observed.fingerprint, DeleteRecoveryAction::Rollback) + .await + .expect("verified staged rollback"); + assert_eq!( + disk.read_all("recovery", "object/xl.meta") + .await + .expect("restored metadata") + .as_ref(), + original + ); + assert_eq!( + disk.read_all("recovery", &format!("object/{data_dir}/part.1")) + .await + .expect("restored shard") + .as_ref(), + encoded + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_recovery_raw_discovery_rejects_symlink_and_forged_cursor() { + use std::os::unix::fs::symlink; + let (temp, disk, _info, _) = fixture().await; + let outside = tempfile::tempdir().expect("outside sentinel"); + fs::write(outside.path().join("sentinel"), b"outside bytes") + .await + .expect("outside sentinel"); + symlink(outside.path(), temp.path().join("recovery").join("untrusted-prefix")).expect("untrusted symlink"); + assert!(disk.discover_delete_transactions("recovery", None, None, 16).await.is_err()); + assert!( + disk.discover_delete_transactions("recovery", None, Some(DeleteRecoveryCursor { token: Uuid::new_v4() }), 16) + .await + .is_err() + ); + assert_eq!( + fs::read(outside.path().join("sentinel")) + .await + .expect("outside remains intact"), + b"outside bytes" + ); + } +} diff --git a/crates/ecstore/src/disk/local/replacement_lease.rs b/crates/ecstore/src/disk/local/replacement_lease.rs index 2a94654d7..3864c490f 100644 --- a/crates/ecstore/src/disk/local/replacement_lease.rs +++ b/crates/ecstore/src/disk/local/replacement_lease.rs @@ -26,6 +26,42 @@ pub struct ReplacementExecutionLease { // file while it is held would allow a second owner to lock a different inode. const EXECUTION_LOCK_FILE: &str = ".rustfs-replacement.lock"; +/// A compatibility retry may only admit a disk that has not received data. +/// The caller retains the mount descriptor and the replacement executor lease. +pub(crate) async fn is_blank_target(root: PathBuf) -> Result { + tokio::task::spawn_blocking(move || { + let mut directories = vec![(root, 0usize)]; + let mut entries_left = 64usize; + while let Some((directory, depth)) = directories.pop() { + for entry in std::fs::read_dir(directory)? { + let entry = entry?; + let Some(remaining) = entries_left.checked_sub(1) else { + return Ok(false); + }; + entries_left = remaining; + let kind = entry.file_type()?; + let name = entry.file_name(); + if depth == 0 && name == EXECUTION_LOCK_FILE && kind.is_file() { + if entry.metadata()?.len() != 0 { + return Ok(false); + } + } else if depth == 1 && name == crate::disk::HEALING_MARKER_PATH && kind.is_file() { + if entry.metadata()?.len() > 256 { + return Ok(false); + } + } else if kind.is_dir() && depth < 4 && (depth > 0 || name == crate::disk::RUSTFS_META_BUCKET) { + directories.push((entry.path(), depth + 1)); + } else { + return Ok(false); + } + } + } + Ok(true) + }) + .await + .map_err(DiskError::from)? +} + pub(super) async fn acquire(root: PathBuf) -> Result> { #[cfg(unix)] { @@ -56,6 +92,53 @@ pub(super) async fn acquire(root: PathBuf) -> Result String { use sha2::{Digest, Sha256}; @@ -135,7 +138,7 @@ pub(crate) fn quota_mutation_fence_path(bucket: &str, object: &str) -> String { ) } -pub(crate) fn is_quota_mutation_fence_path(path: &str) -> bool { +pub fn is_quota_mutation_fence_path(path: &str) -> bool { path.strip_prefix(QUOTA_MUTATION_FENCE_PREFIX) .is_some_and(|digest| digest.len() == 64 && digest.bytes().all(|byte| byte.is_ascii_hexdigit())) } @@ -346,6 +349,113 @@ impl Disk { } } + /// A read-only probe; the physical delete repeats the same leaf checks. + pub async fn is_plain_orphan_data_dir(&self, volume: &str, path: &str) -> Result { + match self { + Self::Local(disk) => disk.get_disk().is_plain_orphan_data_dir(volume, path).await, + Self::Remote(disk) => disk.is_plain_orphan_data_dir(volume, path).await, + } + } + + pub(crate) async fn delete_versions_with_namespace_owner( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + namespace_owner: Option>, + ) -> Vec> { + match self { + Self::Local(disk) => { + disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner) + .await + } + Self::Remote(disk) => { + let _owner = namespace_owner; + disk.delete_versions(volume, versions, opts).await + } + } + } + + pub(crate) async fn delete_versions_recoverable_with_namespace_owner( + &self, + volume: &str, + versions: Vec, + mut opts: DeleteOptions, + namespace_owner: Option>, + ) -> Vec> { + if let Err(error) = validate_delete_recovery_options(&opts) { + return vec![Some(error); versions.len()]; + } + if versions.iter().any(|version| version.name.starts_with('/')) { + return vec![Some(DiskError::InvalidPath); versions.len()]; + } + opts.recoverable_delete = true; + match self { + Self::Local(disk) => { + disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner) + .await + } + Self::Remote(disk) => { + let _owner = namespace_owner; + disk.delete_versions_recoverable(volume, versions, opts).await + } + } + } + + pub(crate) async fn settle_delete_transaction_with_namespace_owner( + &self, + volume: &str, + object: &str, + transaction: Uuid, + expected_fingerprint: DeleteRecoveryDigest, + action: DeleteRecoveryAction, + namespace_owner: Option>, + ) -> Result { + match self { + Self::Local(disk) => { + disk.settle_delete_transaction_with_namespace_owner( + volume, + object, + transaction, + expected_fingerprint, + action, + namespace_owner, + ) + .await + } + Self::Remote(disk) => { + let _owner = namespace_owner; + disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action) + .await + } + } + } + + pub(crate) async fn delete_version_recoverable_with_namespace_owner( + &self, + volume: &str, + path: &str, + fi: FileInfo, + force_del_marker: bool, + mut opts: DeleteOptions, + namespace_owner: Option>, + ) -> Result<()> { + validate_delete_recovery_options(&opts)?; + if path.starts_with('/') { + return Err(DiskError::InvalidPath); + } + opts.recoverable_delete = true; + match self { + Self::Local(disk) => { + disk.delete_version_with_namespace_owner(volume, path, fi, force_del_marker, opts, namespace_owner) + .await + } + Self::Remote(disk) => { + disk.delete_version_recoverable(volume, path, fi, force_del_marker, opts) + .await + } + } + } pub(crate) async fn set_disk_id_state(&self, id: Option) -> Result<()> { match self { Disk::Local(local_disk) => { @@ -363,6 +473,25 @@ impl Disk { } } + pub(crate) async fn delete_versions_guarded_recoverable_with_namespace_owner( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + namespace_owner: Option>, + ) -> Vec> { + match self { + Self::Local(disk) => { + disk.delete_versions_guarded_with_namespace_owner(volume, versions, opts, namespace_owner) + .await + } + Self::Remote(disk) => { + let _owner = namespace_owner; + disk.delete_versions_guarded_recoverable(volume, versions, opts).await + } + } + } + pub(crate) async fn cached_disk_id(&self) -> Option { match self { Disk::Local(local_disk) => local_disk.get_current_disk_id().await, @@ -373,6 +502,91 @@ impl Disk { #[async_trait::async_trait] impl DiskAPI for Disk { + async fn supports_guarded_delete_versions(&self) -> Result<()> { + match self { + Self::Local(disk) => disk.supports_guarded_delete_versions().await, + Self::Remote(disk) => disk.supports_guarded_delete_versions().await, + } + } + async fn delete_versions_guarded_recoverable( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + ) -> Vec> { + self.delete_versions_guarded_recoverable_with_namespace_owner(volume, versions, opts, None) + .await + } + async fn delete_version_recoverable( + &self, + volume: &str, + object: &str, + file_info: FileInfo, + force_del_marker: bool, + opts: DeleteOptions, + ) -> Result<()> { + self.delete_version_recoverable_with_namespace_owner(volume, object, file_info, force_del_marker, opts, None) + .await + } + async fn delete_versions_recoverable( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + ) -> Vec> { + match self { + Self::Local(disk) => disk.delete_versions_recoverable(volume, versions, opts).await, + Self::Remote(disk) => disk.delete_versions_recoverable(volume, versions, opts).await, + } + } + async fn has_delete_transaction_evidence(&self, volume: &str, object: &str) -> Result { + match self { + Self::Local(disk) => disk.has_delete_transaction_evidence(volume, object).await, + Self::Remote(disk) => disk.has_delete_transaction_evidence(volume, object).await, + } + } + async fn discover_delete_transactions( + &self, + volume: &str, + object: Option<&str>, + cursor: Option, + limit: u32, + ) -> Result { + match self { + Self::Local(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await, + Self::Remote(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await, + } + } + async fn inspect_delete_transaction( + &self, + volume: &str, + object: &str, + transaction: Uuid, + ) -> Result { + match self { + Self::Local(disk) => disk.inspect_delete_transaction(volume, object, transaction).await, + Self::Remote(disk) => disk.inspect_delete_transaction(volume, object, transaction).await, + } + } + async fn settle_delete_transaction( + &self, + volume: &str, + object: &str, + transaction: Uuid, + expected_fingerprint: DeleteRecoveryDigest, + action: DeleteRecoveryAction, + ) -> Result { + match self { + Self::Local(disk) => { + disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action) + .await + } + Self::Remote(disk) => { + disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action) + .await + } + } + } fn to_string(&self) -> String { match self { Disk::Local(local_disk) => local_disk.to_string(), @@ -574,6 +788,7 @@ impl DiskAPI for Disk { } #[tracing::instrument(level = "trace", skip_all)] + #[track_caller] async fn read_version( &self, _org_volume: &str, @@ -582,10 +797,20 @@ impl DiskAPI for Disk { version_id: &str, opts: &ReadOptions, ) -> Result { - match self { - Disk::Local(local_disk) => local_disk.read_version(_org_volume, volume, path, version_id, opts).await, - Disk::Remote(remote_disk) => remote_disk.read_version(_org_volume, volume, path, version_id, opts).await, + let operation = async { + match self { + Disk::Local(local_disk) => local_disk.read_version(_org_volume, volume, path, version_id, opts).await, + Disk::Remote(remote_disk) => remote_disk.read_version(_org_volume, volume, path, version_id, opts).await, + } + }; + #[cfg(feature = "diagnostic-read-version-caller")] + if crate::cluster::rpc::runtime_sources::current_read_version_request_caller().is_none() { + let caller = crate::cluster::rpc::runtime_sources::read_version_request_caller_from_location( + std::panic::Location::caller().file(), + ); + return crate::cluster::rpc::runtime_sources::with_read_version_request_caller(caller, operation).await; } + operation.await } #[tracing::instrument(level = "trace", skip_all)] @@ -1197,6 +1422,14 @@ impl Disk { Self::Remote(_) => Err(DiskError::other("replacement execution requires a local target")), } } + + pub(crate) async fn is_blank_replacement_target(&self) -> Result { + let Some(root) = self.replacement_mount_lease_root() else { + return Ok(false); + }; + let blank = local::is_blank_replacement_target(root).await?; + Ok(blank && self.has_replacement_mount_lease()) + } } pub async fn new_disk(ep: &Endpoint, opt: &DiskOption) -> Result { @@ -1282,6 +1515,7 @@ pub trait DiskAPI: Debug + Send + Sync + 'static { } async fn write_metadata(&self, org_volume: &str, volume: &str, path: &str, fi: FileInfo) -> Result<()>; async fn update_metadata(&self, volume: &str, path: &str, fi: FileInfo, opts: &UpdateMetadataOpts) -> Result<()>; + #[track_caller] async fn read_version( &self, org_volume: &str, @@ -1367,6 +1601,68 @@ pub trait DiskAPI: Debug + Send + Sync + 'static { Err(DiskError::MethodNotAllowed) } async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()>; + /// Exact-key existence owns its traversal state; unsupported peers cannot + /// establish the absence of recovery obligations. + async fn has_delete_transaction_evidence(&self, _volume: &str, _object: &str) -> Result { + Err(DiskError::MethodNotAllowed) + } + async fn discover_delete_transactions( + &self, + _volume: &str, + _object: Option<&str>, + _cursor: Option, + _limit: u32, + ) -> Result { + Err(DiskError::MethodNotAllowed) + } + /// A read-only capability check; unsupported implementations must fail closed. + async fn supports_guarded_delete_versions(&self) -> Result<()> { + Err(DiskError::GuardedDeleteUnsupported) + } + async fn delete_versions_guarded_recoverable( + &self, + _volume: &str, + versions: Vec, + _opts: DeleteOptions, + ) -> Vec> { + vec![Some(DiskError::GuardedDeleteUnsupported); versions.len()] + } + async fn delete_version_recoverable( + &self, + _volume: &str, + _object: &str, + _file_info: FileInfo, + _force_del_marker: bool, + _opts: DeleteOptions, + ) -> Result<()> { + Err(DiskError::MethodNotAllowed) + } + async fn delete_versions_recoverable( + &self, + _volume: &str, + versions: Vec, + _opts: DeleteOptions, + ) -> Vec> { + vec![Some(DiskError::MethodNotAllowed); versions.len()] + } + async fn inspect_delete_transaction( + &self, + _volume: &str, + _object: &str, + _transaction: Uuid, + ) -> Result { + Err(DiskError::MethodNotAllowed) + } + async fn settle_delete_transaction( + &self, + _volume: &str, + _object: &str, + _transaction: Uuid, + _expected_fingerprint: DeleteRecoveryDigest, + _action: DeleteRecoveryAction, + ) -> Result { + Err(DiskError::MethodNotAllowed) + } // VerifyFile async fn verify_file(&self, volume: &str, path: &str, fi: &FileInfo) -> Result; // CheckParts @@ -1405,13 +1701,15 @@ where { validate_batch_read_version_item_count(req.items.len())?; let parallelism = batch_read_version_server_parallelism(); + #[cfg(feature = "diagnostic-read-version-caller")] + let item_count = req.items.len(); let mut responses = stream::iter(req.items.into_iter().enumerate()) .map(|(index, item)| async move { - match disk - .read_version(&item.org_volume, &item.volume, &item.path, &item.version_id, &req.opts) - .await - { + let read = disk.read_version(&item.org_volume, &item.volume, &item.path, &item.version_id, &req.opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::cluster::rpc::runtime_sources::with_batch_read_version_item_context(index, item_count, read); + match read.await { Ok(file_info) => BatchReadVersionResp { index, path: item.path, @@ -1561,6 +1859,117 @@ pub struct FileInfoVersions { pub free_versions: Vec, } +/// The durable remote location and restore state of the observed generation. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CurrentDeleteTransition { + pub status: String, + pub tier: String, + pub remote_object: String, + pub remote_version: Option, + pub version_state: rustfs_filemeta::TransitionVersionState, + pub destination_id: Option<[u8; 32]>, + pub remote_only: bool, +} + +/// The current object generation observed before a guarded batch delete. +/// This transient value is never persisted in object metadata. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub enum CurrentDeleteGeneration { + Absent, + Present { + version_id: Option, + data_dir: Option, + mod_time: OffsetDateTime, + delete_marker: bool, + etag: String, + #[serde(default)] + transition: Option>, + }, +} + +impl CurrentDeleteGeneration { + pub fn from_file_info(fi: &FileInfo) -> Result { + let mod_time = fi.mod_time.ok_or(DiskError::FileCorrupt)?; + let etag = fi.get_etag().unwrap_or_default(); + let data_dir = fi.data_dir.filter(|id| !id.is_nil()); + let remote_only = fi.is_remote(); + if !fi.deleted && (etag.is_empty() || (data_dir.is_none() && !remote_only)) { + return Err(DiskError::FileCorrupt); + } + if remote_only && (fi.transition_tier.is_empty() || fi.transitioned_objname.is_empty()) { + return Err(DiskError::FileCorrupt); + } + let destination_id = + crate::services::tier::tier::tier_destination_id_from_metadata(&fi.metadata).map_err(|_| DiskError::FileCorrupt)?; + let remote_version = match fi.transition_version.as_deref() { + Some(value) => (!value.is_empty() && !Uuid::parse_str(value).is_ok_and(|id| id.is_nil())).then(|| value.to_owned()), + None => fi.transition_version_id.filter(|id| !id.is_nil()).map(|id| id.to_string()), + }; + let transition = (!fi.transition_status.is_empty() + || !fi.transition_tier.is_empty() + || !fi.transitioned_objname.is_empty() + || remote_version.is_some() + || fi.transition_version_state != rustfs_filemeta::TransitionVersionState::Unknown + || destination_id.is_some()) + .then(|| { + Box::new(CurrentDeleteTransition { + status: fi.transition_status.clone(), + tier: fi.transition_tier.clone(), + remote_object: fi.transitioned_objname.clone(), + remote_version, + version_state: fi.transition_version_state, + destination_id, + remote_only, + }) + }); + Ok(Self::Present { + version_id: fi.version_id.filter(|id| !id.is_nil()), + data_dir, + mod_time, + delete_marker: fi.deleted, + etag, + transition, + }) + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct GuardedFileInfoVersions { + pub group: FileInfoVersions, + pub expected_current: CurrentDeleteGeneration, +} + +#[derive(Debug, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct GuardedDeleteVersionsPayload { + pub versions: Vec, + pub opts: DeleteOptions, +} + +impl GuardedDeleteVersionsPayload { + pub fn validate(&self) -> Result<()> { + validate_delete_recovery_options(&self.opts)?; + if self.versions.is_empty() + || self.versions.len() > 1000 + || self + .versions + .iter() + .try_fold(0_usize, |count, version| count.checked_add(version.group.versions.len())) + .is_none_or(|count| count > 1000) + || self + .versions + .iter() + .any(|version| version.group.name.starts_with('/') || version.group.versions.is_empty()) + { + return Err(DiskError::InvalidPath); + } + Ok(()) + } +} + impl FileInfoVersions { pub fn find_version_index(&self, v: &str) -> Option { if v.is_empty() { @@ -1689,6 +2098,14 @@ pub struct RenameDataResp { /// "cannot vote", never as `Absent`. #[serde(default)] pub old_current_size: Option, + /// Whether this disk decoded enough of the overwritten current version to + /// decide if a transitioned source has to be returned. + #[serde(default)] + pub old_current_source_checked: bool, + /// Transitioned old-current source used by PUT free-version cleanup receipt. + /// Empty when the old current is not transitioned or no source was known. + #[serde(default)] + pub old_current_source: Option, } #[derive(Debug, Clone, Default, Serialize, Deserialize)] @@ -1703,6 +2120,282 @@ pub struct DeleteOptions { /// Remote calls carrying it must use DeleteRetiredMarker, never DeleteVersion. #[serde(default, skip_serializing_if = "Option::is_none")] pub expected_delete_marker: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub delete_recovery_bucket_incarnation: Option, + #[serde(default)] + pub recoverable_delete: bool, + /// Reclaim only plain erasure-part leaves, preserving every other entry. + /// Keep `recursive` false so an older peer cannot widen this operation. + #[serde(default)] + pub orphan_data_dir: bool, +} + +pub const DELETE_RECOVERY_PROTOCOL_VERSION: u32 = 1; +pub(crate) fn get_delete_recovery_timeout() -> Result { + let value = std::env::var("RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS"); + match value { + Ok(value) => parse_delete_recovery_timeout(Some(&value)), + Err(std::env::VarError::NotPresent) => parse_delete_recovery_timeout(None), + Err(std::env::VarError::NotUnicode(_)) => Err(DiskError::other("invalid delete recovery timeout configuration")), + } +} + +fn parse_delete_recovery_timeout(value: Option<&str>) -> Result { + let seconds = match value { + None => 900, + Some(value) => value + .parse::() + .ok() + .filter(|seconds| (30..=86400).contains(seconds)) + .ok_or_else(|| DiskError::other("delete recovery timeout must be an integer between 30 and 86400 seconds"))?, + }; + Ok(std::time::Duration::from_secs(seconds)) +} + +pub(crate) fn validate_delete_recovery_options(opts: &DeleteOptions) -> Result<()> { + if opts.old_data_dir.is_none_or(|id| id.is_nil()) + || opts.undo_write + || opts.undo_delete + || opts.recursive + || opts.immediate + || opts.orphan_data_dir + || opts.expected_delete_marker.is_some() + || opts.delete_recovery_bucket_incarnation.is_some_and(|id| id.is_nil()) + { + return Err(DiskError::FileCorrupt); + } + Ok(()) +} +pub type DeleteRecoveryDigest = [u8; 32]; + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct DeleteTransactionId { + pub object: String, + pub transaction: Uuid, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct DeleteRecoveryCursor { + pub token: Uuid, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct DeleteRecoveryPage { + pub transactions: Vec, + pub next: Option, +} + +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct DeleteRecoveryJournal { + pub protocol_version: u32, + pub volume: String, + pub object: String, + pub transaction: Uuid, + pub bucket_incarnation: Option, + pub original_digest: Option, + pub target_digest: Option, + pub data_dirs: Vec, + pub deleted_versions: Vec, + pub requests: Vec, + pub original_versions: Vec, + #[serde(default)] + pub predecessors: Vec, +} + +impl DeleteRecoveryJournal { + /// Checks source generations only; the coordinator must supply a certified + /// decision and the receiver must retain its namespace/incarnation fences. + pub(crate) fn matches_adoption_source(&self, current: &[DeleteRecoveryVersionIdentity]) -> bool { + if self.requests.is_empty() { + return false; + } + let mut originals = std::collections::HashMap::with_capacity(self.original_versions.len()); + for version in &self.original_versions { + originals.entry(version.version_id.unwrap_or_default()).or_insert(version); + } + let mut locals = std::collections::HashMap::with_capacity(current.len()); + for version in current { + locals.entry(version.version_id.unwrap_or_default()).or_insert(version); + } + self.requests.iter().all(|request| { + if request.deleted && request.mark_deleted { + return current.first() == self.original_versions.first(); + } + let id = request.version_id.unwrap_or_default(); + let Some(original) = originals.get(&id) else { + return false; + }; + match locals.get(&id) { + Some(local) => local == original, + None => !id.is_nil(), + } + }) + } + + /// Matches immutable local preparation intent against an independently + /// inspected same-transaction commit receipt. This match is not authority + /// by itself; the coordinator must supply the certified receipt. + pub(crate) fn matches_prepared_authority(&self, authority: &Self) -> bool { + if self.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION + || authority.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION + || self.transaction.is_nil() + || self.transaction != authority.transaction + || self.volume != authority.volume + || self.object != authority.object + || self.bucket_incarnation != authority.bucket_incarnation + || self.bucket_incarnation.is_some_and(|id| id.is_nil()) + || self.requests.is_empty() + || self.requests != authority.requests + || self.deleted_versions.iter().collect::>() + != authority.deleted_versions.iter().collect::>() + { + return false; + } + let sources: std::collections::HashMap<_, _> = self + .original_versions + .iter() + .map(|version| (version.version_id.unwrap_or_default(), version)) + .collect(); + let certified: std::collections::HashMap<_, _> = authority + .original_versions + .iter() + .map(|version| (version.version_id.unwrap_or_default(), version)) + .collect(); + if sources.len() != self.original_versions.len() || certified.len() != authority.original_versions.len() { + return false; + } + self.requests.iter().all(|request| { + let (local, remote) = if request.deleted && request.mark_deleted { + (self.original_versions.first(), authority.original_versions.first()) + } else { + let id = request.version_id.unwrap_or_default(); + (sources.get(&id).copied(), certified.get(&id).copied()) + }; + match (local, remote) { + (Some(local), Some(remote)) => local == remote, + (None, None) => self.original_digest.is_none() && authority.original_digest.is_none(), + _ => false, + } + }) + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct DeleteRecoveryVersionIdentity { + pub version_id: Option, + pub mod_time: Option, + pub data_dir: Option, + pub deleted: bool, + pub metadata_digest: DeleteRecoveryDigest, +} + +impl DeleteRecoveryVersionIdentity { + pub(crate) fn from_file_info(info: FileInfo) -> error::Result { + local::delete_recovery::generation(info) + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum DeleteRecoveryState { + Absent, + NewPrepared, + QuorumCommitted, + CommitFinished, + LegacyAmbiguous, +} + +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct DeleteRecoveryInspection { + pub protocol_version: u32, + pub id: DeleteTransactionId, + pub state: DeleteRecoveryState, + pub journal: Option, + pub current_digest: Option, + pub current_versions: Vec, + pub backup_digest: Option, + pub rollback_data_complete: bool, + pub commit_data_safe: bool, + pub explicit_delete_safe: bool, + pub fingerprint: DeleteRecoveryDigest, +} + +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub enum DeleteRecoveryAction { + PublishCommit, + FinishCommit, + FinalizeCommit, + Rollback, + DiscardLegacy, + AdoptCommit { authority: Box }, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub enum DeleteRecoveryRpcOperation { + InspectOrphanDataDir { + path: String, + }, + Discover { + object: Option, + cursor: Option, + limit: u32, + }, + Inspect { + object: String, + transaction: Uuid, + }, + Settle { + object: String, + transaction: Uuid, + expected_fingerprint: DeleteRecoveryDigest, + action: DeleteRecoveryAction, + }, + ApplyVersion { + object: String, + file_info: Box, + force_del_marker: bool, + opts: DeleteOptions, + }, + ApplyVersions { + versions: Vec, + opts: DeleteOptions, + }, + Exists { + object: String, + }, +} + +impl DeleteRecoveryRpcOperation { + /// Full metadata application and adoption share the existing mutation envelope. + pub fn max_wire_payload_bytes(&self) -> usize { + match self { + Self::ApplyVersion { .. } + | Self::ApplyVersions { .. } + | Self::Settle { + action: DeleteRecoveryAction::AdoptCommit { .. }, + .. + } => rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES, + _ => rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES, + } + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub enum DeleteRecoveryRpcReply { + OrphanDataDir { plain: bool }, + Discovered(DeleteRecoveryPage), + Inspected(DeleteRecoveryInspection), + Settled(DeleteRecoveryInspection), + AppliedVersion, + AppliedVersions(Vec>), + Evidence { present: bool }, } #[derive(Debug, Clone, Serialize, Deserialize)] @@ -1818,6 +2511,243 @@ pub fn count_part_not_success(part_errs: &[usize]) -> usize { #[cfg(test)] mod tests { use super::*; + + #[test] + fn guarded_delete_generation_boxing_preserves_legacy_wire_bytes() { + #[derive(Serialize)] + struct LegacyPresent { + version_id: Option, + data_dir: Option, + mod_time: OffsetDateTime, + delete_marker: bool, + etag: String, + transition: Option, + } + + assert!(std::mem::size_of::() < std::mem::size_of::()); + let mut fi = FileInfo { + mod_time: Some(OffsetDateTime::UNIX_EPOCH), + data_dir: Some(Uuid::from_u128(1)), + ..Default::default() + }; + fi.metadata.insert("etag".into(), "opaque-etag".into()); + let plain = CurrentDeleteGeneration::from_file_info(&fi).expect("local generation"); + fi.transition_status = rustfs_filemeta::TRANSITION_COMPLETE.to_owned(); + fi.transition_tier = "WARM".into(); + fi.transitioned_objname = "remote/object".into(); + fi.transition_version = Some("remote-version".into()); + let remote = CurrentDeleteGeneration::from_file_info(&fi).expect("transition generation"); + for current in [CurrentDeleteGeneration::Absent, plain, remote] { + let (legacy_messagepack, legacy_json) = match ¤t { + CurrentDeleteGeneration::Absent => ( + rmp_serde::to_vec_named("Absent").expect("legacy absent MessagePack"), + serde_json::to_vec("Absent").expect("legacy absent JSON"), + ), + CurrentDeleteGeneration::Present { + version_id, + data_dir, + mod_time, + delete_marker, + etag, + transition, + } => { + let legacy = std::collections::BTreeMap::from([( + "Present", + LegacyPresent { + version_id: *version_id, + data_dir: *data_dir, + mod_time: *mod_time, + delete_marker: *delete_marker, + etag: etag.clone(), + transition: transition.as_deref().cloned(), + }, + )]); + ( + rmp_serde::to_vec_named(&legacy).expect("legacy inline MessagePack"), + serde_json::to_vec(&legacy).expect("legacy inline JSON"), + ) + } + }; + assert_eq!(rmp_serde::to_vec_named(¤t).expect("current MessagePack"), legacy_messagepack); + assert_eq!(serde_json::to_vec(¤t).expect("current JSON"), legacy_json); + assert_eq!( + rustfs_protos::decode_delete_recovery_payload::(&legacy_messagepack) + .expect("existing payload remains readable"), + current + ); + } + } + + #[test] + fn guarded_delete_generation_requires_reliable_current_identity() { + let mut fi = FileInfo::default(); + assert_eq!(CurrentDeleteGeneration::from_file_info(&fi), Err(DiskError::FileCorrupt)); + fi.mod_time = Some(OffsetDateTime::UNIX_EPOCH); + fi.data_dir = Some(Uuid::new_v4()); + assert_eq!(CurrentDeleteGeneration::from_file_info(&fi), Err(DiskError::FileCorrupt)); + fi.metadata.insert("etag".into(), "opaque-etag".into()); + fi.version_id = Some(Uuid::nil()); + assert!(matches!( + CurrentDeleteGeneration::from_file_info(&fi), + Ok(CurrentDeleteGeneration::Present { version_id: None, .. }) + )); + fi.data_dir = Some(Uuid::nil()); + assert_eq!(CurrentDeleteGeneration::from_file_info(&fi), Err(DiskError::FileCorrupt)); + } + + #[test] + fn guarded_delete_generation_accepts_remote_only_and_legacy_unversioned_identity() { + let mut fi = FileInfo { + mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1)), + transition_status: rustfs_filemeta::TRANSITION_COMPLETE.to_owned(), + transition_tier: "WARM".into(), + transitioned_objname: "remote/object".into(), + ..Default::default() + }; + fi.metadata.insert("etag".into(), "opaque-etag".into()); + for version in [None, Some(String::new()), Some(Uuid::nil().to_string())] { + fi.transition_version = version; + for data_dir in [None, Some(Uuid::nil())] { + fi.data_dir = data_dir; + assert!( + matches!( + CurrentDeleteGeneration::from_file_info(&fi), + Ok(CurrentDeleteGeneration::Present { + data_dir: None, + transition: Some(transition), + .. + }) if transition.remote_version.is_none() + && transition.version_state == rustfs_filemeta::TransitionVersionState::Unknown + && transition.destination_id.is_none() + && transition.remote_only + ), + "legacy unversioned remote metadata remains usable" + ); + } + } + fi.transition_version = None; + fi.transition_version_id = Some(Uuid::nil()); + CurrentDeleteGeneration::from_file_info(&fi).expect("nil legacy remote UUID means no remote version"); + fi.transition_status.clear(); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&fi), + Err(DiskError::FileCorrupt), + "local payload metadata still requires a data directory" + ); + fi.transition_status = rustfs_filemeta::TRANSITION_COMPLETE.to_owned(); + fi.transition_tier.clear(); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&fi), + Err(DiskError::FileCorrupt), + "remote-only identity requires its tier" + ); + fi.transition_tier = "WARM".into(); + fi.transitioned_objname.clear(); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&fi), + Err(DiskError::FileCorrupt), + "remote-only identity requires its remote object" + ); + } + + #[test] + fn guarded_delete_remote_transition_tuple_survives_rpc_codec() { + use rustfs_utils::http::metadata_compat::{SUFFIX_TRANSITION_TIER_DESTINATION_ID, insert_str}; + let mut fi = FileInfo { + name: "object".into(), + mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1)), + transition_status: rustfs_filemeta::TRANSITION_COMPLETE.to_owned(), + transition_tier: "WARM".into(), + transitioned_objname: "opaque/object+name".into(), + transition_version: Some("opaque/version+id==".into()), + transition_version_state: rustfs_filemeta::TransitionVersionState::Exact, + ..Default::default() + }; + fi.metadata.insert("etag".into(), "opaque-etag".into()); + insert_str(&mut fi.metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, "ab".repeat(32)); + let generation = CurrentDeleteGeneration::from_file_info(&fi).expect("capture complete remote identity"); + let payload = GuardedDeleteVersionsPayload { + versions: vec![GuardedFileInfoVersions { + group: FileInfoVersions { + name: "object".into(), + versions: vec![fi.clone()], + ..Default::default() + }, + expected_current: generation.clone(), + }], + opts: DeleteOptions { + old_data_dir: Some(Uuid::new_v4()), + ..Default::default() + }, + }; + let encoded = rmp_serde::to_vec_named(&payload).expect("encode remote guarded request"); + let decoded: GuardedDeleteVersionsPayload = + rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode remote guarded request"); + assert_eq!(decoded.versions[0].expected_current, generation); + assert!( + matches!(&generation, CurrentDeleteGeneration::Present { transition: Some(transition), .. } + if transition.status == rustfs_filemeta::TRANSITION_COMPLETE + && transition.tier == "WARM" + && transition.remote_object == "opaque/object+name" + && transition.remote_version.as_deref() == Some("opaque/version+id==") + && transition.version_state == rustfs_filemeta::TransitionVersionState::Exact + && transition.destination_id == Some([0xab; 32]) + && transition.remote_only) + ); + insert_str(&mut fi.metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, format!("a{}", "€".repeat(21))); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&fi), + Err(DiskError::FileCorrupt), + "64-byte non-ASCII backend identities are rejected without panicking" + ); + insert_str(&mut fi.metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, "ab".repeat(32)); + fi.metadata + .insert(format!("x-minio-internal-{SUFFIX_TRANSITION_TIER_DESTINATION_ID}"), "cd".repeat(32)); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&fi), + Err(DiskError::FileCorrupt), + "conflicting backend identities fail closed" + ); + } + + #[test] + fn guarded_delete_payload_rejects_malformed_or_unbounded_batches() { + let group = GuardedFileInfoVersions { + group: FileInfoVersions { + name: "object".into(), + versions: vec![FileInfo::default()], + ..Default::default() + }, + expected_current: CurrentDeleteGeneration::Absent, + }; + let mut payload = GuardedDeleteVersionsPayload { + versions: vec![group], + opts: DeleteOptions { + old_data_dir: Some(Uuid::new_v4()), + ..Default::default() + }, + }; + payload.validate().expect("valid guarded batch"); + let encoded = rmp_serde::to_vec_named(&payload).expect("serialize guarded batch"); + let decoded: GuardedDeleteVersionsPayload = + rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode guarded batch"); + decoded.validate().expect("round-trip guarded batch remains valid"); + let mut trailing = encoded; + trailing.push(0xc0); + assert!(rustfs_protos::decode_delete_recovery_payload::(&trailing).is_err()); + let mut unknown = serde_json::to_value(&payload).expect("serialize guarded batch to map"); + unknown + .as_object_mut() + .expect("payload is a map") + .insert("unrecognized".into(), true.into()); + assert!(serde_json::from_value::(unknown).is_err()); + payload.versions[0].group.versions.resize_with(1001, FileInfo::default); + assert_eq!(payload.validate(), Err(DiskError::InvalidPath)); + payload.versions[0].group.versions.truncate(1); + payload.opts.undo_delete = true; + assert_eq!(payload.validate(), Err(DiskError::FileCorrupt)); + } + use endpoint::Endpoint; use local::LocalDisk; use tokio::fs; @@ -1993,6 +2923,9 @@ mod tests { undo_delete: false, old_data_dir: Some(Uuid::new_v4()), expected_delete_marker: None, + delete_recovery_bucket_incarnation: None, + recoverable_delete: false, + orphan_data_dir: false, }; assert!(opts.recursive); @@ -2165,6 +3098,8 @@ mod tests { cleanup_data_dir: Some(uuid), sign: Some(signature.clone()), old_current_size: Some(OldCurrentSize::Present(42)), + old_current_source_checked: false, + old_current_source: None, }; assert_eq!(resp.old_data_dir, Some(uuid)); @@ -2185,6 +3120,8 @@ mod tests { cleanup_data_dir: Some(Uuid::new_v4()), sign: Some(vec![0x01, 0x02, 0x03]), old_current_size, + old_current_source_checked: false, + old_current_source: None, }; let encoded = rmp_serde::encode::to_vec_named(&resp).expect("named msgpack should encode"); @@ -2222,6 +3159,8 @@ mod tests { assert_eq!(decoded.cleanup_data_dir, None); assert_eq!(decoded.sign, legacy.sign); assert_eq!(decoded.old_current_size, None); + assert!(!decoded.old_current_source_checked); + assert!(decoded.old_current_source.is_none()); } /// Test constants @@ -2417,3 +3356,349 @@ mod tests { } } } + +#[cfg(test)] +mod delete_recovery_timeout_tests { + use super::*; + #[test] + fn recovery_timeout_defaults_and_validates_range() { + assert_eq!( + parse_delete_recovery_timeout(None).expect("default timeout"), + std::time::Duration::from_secs(900) + ); + for seconds in ["30", "900", "86400"] { + assert!(parse_delete_recovery_timeout(Some(seconds)).is_ok()); + } + for value in ["", "invalid", "0", "2", "29", "86401", "-1", "18446744073709551616"] { + assert!(parse_delete_recovery_timeout(Some(value)).is_err()); + } + } +} + +#[cfg(test)] +mod delete_recovery_wire_limit_tests { + use super::*; + + #[test] + fn boxed_current_delete_transition_preserves_existing_wire_payload() { + #[derive(Debug, PartialEq, Eq, Serialize, Deserialize)] + #[serde(deny_unknown_fields)] + enum PriorGeneration { + Present { + version_id: Option, + data_dir: Option, + mod_time: OffsetDateTime, + delete_marker: bool, + etag: String, + #[serde(default)] + transition: Option, + }, + } + + let version_id = Some(Uuid::from_u128(1)); + let data_dir = Some(Uuid::from_u128(2)); + let mod_time = OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1); + for transition in [ + None, + Some(CurrentDeleteTransition { + status: rustfs_filemeta::TRANSITION_COMPLETE.to_owned(), + tier: "WARM".into(), + remote_object: "opaque/object+name".into(), + remote_version: Some("opaque/version+id==".into()), + version_state: rustfs_filemeta::TransitionVersionState::Exact, + destination_id: Some([0xab; 32]), + remote_only: true, + }), + ] { + let prior = PriorGeneration::Present { + version_id, + data_dir, + mod_time, + delete_marker: false, + etag: "opaque-etag".into(), + transition: transition.clone(), + }; + let generation = CurrentDeleteGeneration::Present { + version_id, + data_dir, + mod_time, + delete_marker: false, + etag: "opaque-etag".into(), + transition: transition.map(Box::new), + }; + let prior_encoded = rmp_serde::to_vec_named(&prior).expect("encode the previous unboxed generation shape"); + let encoded = rmp_serde::to_vec_named(&generation).expect("encode boxed transition generation"); + assert_eq!(encoded, prior_encoded); + let decoded: CurrentDeleteGeneration = + rustfs_protos::decode_delete_recovery_payload(&prior_encoded).expect("decode the previous generation payload"); + assert_eq!(decoded, generation); + let prior_decoded: PriorGeneration = + rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode boxed transition with the previous shape"); + assert_eq!(prior_decoded, prior); + if matches!(generation, CurrentDeleteGeneration::Present { transition: None, .. }) { + #[derive(Serialize)] + enum GenerationWithoutTransition { + Present { + version_id: Option, + data_dir: Option, + mod_time: OffsetDateTime, + delete_marker: bool, + etag: String, + }, + } + let legacy = GenerationWithoutTransition::Present { + version_id, + data_dir, + mod_time, + delete_marker: false, + etag: "opaque-etag".into(), + }; + let legacy_encoded = rmp_serde::to_vec_named(&legacy).expect("encode a generation without the transition field"); + let legacy_decoded: CurrentDeleteGeneration = + rustfs_protos::decode_delete_recovery_payload(&legacy_encoded).expect("default an omitted transition field"); + assert_eq!(legacy_decoded, generation); + } + } + } + + #[test] + fn boxed_apply_version_preserves_existing_wire_payload() { + #[derive(Serialize)] + enum PriorOperation<'a> { + ApplyVersion { + object: &'a str, + file_info: &'a FileInfo, + force_del_marker: bool, + opts: &'a DeleteOptions, + }, + } + let file_info = FileInfo { + name: "object".into(), + version_id: Some(Uuid::new_v4()), + metadata: std::collections::HashMap::from([("metadata".into(), "value".into())]), + ..Default::default() + }; + let opts = DeleteOptions { + old_data_dir: Some(Uuid::new_v4()), + recoverable_delete: true, + delete_recovery_bucket_incarnation: Some(Uuid::new_v4()), + ..Default::default() + }; + let prior = PriorOperation::ApplyVersion { + object: "object", + file_info: &file_info, + force_del_marker: true, + opts: &opts, + }; + let encoded = rmp_serde::to_vec_named(&prior).expect("encode the previous unboxed request shape"); + let operation = DeleteRecoveryRpcOperation::ApplyVersion { + object: "object".into(), + file_info: Box::new(file_info.clone()), + force_del_marker: true, + opts: opts.clone(), + }; + assert_eq!(rmp_serde::to_vec_named(&operation).expect("encode boxed request"), encoded); + assert_eq!( + serde_json::to_value(&operation).expect("boxed request JSON"), + serde_json::to_value(prior).expect("previous request JSON") + ); + let decoded: DeleteRecoveryRpcOperation = + rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode the previous payload"); + assert_eq!(rmp_serde::to_vec_named(&decoded).expect("re-encode decoded prior request"), encoded); + } + #[test] + fn adoption_and_inspection_reply_support_large_version_journals() { + let transaction = Uuid::new_v4(); + let request = FileInfo { + metadata: std::collections::HashMap::from([("metadata".into(), "x".repeat(2048))]), + ..Default::default() + }; + let journal = DeleteRecoveryJournal { + protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION, + volume: "bucket".into(), + object: "object".into(), + transaction, + bucket_incarnation: None, + original_digest: None, + target_digest: None, + data_dirs: Vec::new(), + deleted_versions: Vec::new(), + requests: vec![request; 1000], + original_versions: Vec::new(), + predecessors: Vec::new(), + }; + let operation = DeleteRecoveryRpcOperation::Settle { + object: "object".into(), + transaction, + expected_fingerprint: [0; 32], + action: DeleteRecoveryAction::AdoptCommit { + authority: Box::new(journal.clone()), + }, + }; + let encoded = rmp_serde::to_vec_named(&operation).expect("encode large adoption"); + assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES); + assert!(encoded.len() < operation.max_wire_payload_bytes()); + let decoded: DeleteRecoveryRpcOperation = + rustfs_protos::decode_delete_recovery_payload(&encoded).expect("large adoption can decode"); + assert_eq!(decoded.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES); + let reply = DeleteRecoveryRpcReply::Inspected(DeleteRecoveryInspection { + protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION, + id: DeleteTransactionId { + object: "object".into(), + transaction, + }, + state: DeleteRecoveryState::QuorumCommitted, + journal: Some(journal), + current_digest: None, + current_versions: Vec::new(), + backup_digest: None, + rollback_data_complete: false, + commit_data_safe: true, + explicit_delete_safe: false, + fingerprint: [0; 32], + }); + let encoded = rmp_serde::to_vec_named(&reply).expect("encode large inspection"); + assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES); + assert!(rustfs_protos::decode_delete_recovery_payload::(&encoded).is_ok()); + let inspect = DeleteRecoveryRpcOperation::Inspect { + object: "object".into(), + transaction, + }; + assert_eq!(inspect.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES); + } +} + +#[cfg(test)] +mod delete_recovery_adoption_tests { + use super::*; + + #[test] + fn source_matching_preserves_non_null_absence_and_rejects_foreign_or_missing_null() { + let id = Uuid::new_v4(); + let source = DeleteRecoveryVersionIdentity { + version_id: Some(id), + mod_time: None, + data_dir: None, + deleted: false, + metadata_digest: [1; 32], + }; + let mut journal = DeleteRecoveryJournal { + protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION, + volume: "bucket".into(), + object: "object".into(), + transaction: Uuid::new_v4(), + bucket_incarnation: None, + original_digest: None, + target_digest: None, + data_dirs: Vec::new(), + deleted_versions: vec![id], + requests: vec![FileInfo { + version_id: Some(id), + ..Default::default() + }], + original_versions: vec![source.clone()], + predecessors: Vec::new(), + }; + assert!(journal.matches_adoption_source(&[]), "certified non-null removal can already be absent"); + assert!(journal.matches_adoption_source(std::slice::from_ref(&source))); + let mut foreign = source.clone(); + foreign.metadata_digest = [2; 32]; + assert!(!journal.matches_adoption_source(std::slice::from_ref(&foreign))); + journal.original_versions.push(foreign.clone()); + assert!( + journal.matches_adoption_source(&[source.clone(), foreign.clone()]), + "retain original first-match semantics" + ); + assert!(!journal.matches_adoption_source(&[foreign, source])); + journal.requests[0].version_id = None; + journal.original_versions[0].version_id = None; + assert!(!journal.matches_adoption_source(&[]), "null absence does not establish its generation"); + assert!(journal.matches_adoption_source(std::slice::from_ref(&journal.original_versions[0]))); + journal.requests.clear(); + assert!(!journal.matches_adoption_source(&journal.original_versions)); + } +} + +#[cfg(test)] +mod orphan_data_dir_protocol_tests { + use super::*; + + #[test] + fn delete_recovery_exists_roundtrips_without_cursor_ownership() { + let operation = DeleteRecoveryRpcOperation::Exists { + object: "object".to_owned(), + }; + assert_eq!( + operation.max_wire_payload_bytes(), + rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES + ); + let encoded = rmp_serde::to_vec_named(&operation).expect("encode exact-key existence probe"); + assert!(matches!( + rustfs_protos::decode_delete_recovery_payload::(&encoded).expect("decode existence probe"), + DeleteRecoveryRpcOperation::Exists { object } if object == "object" + )); + for present in [false, true] { + let encoded = rmp_serde::to_vec_named(&DeleteRecoveryRpcReply::Evidence { present }).expect("encode evidence reply"); + assert!(matches!( + rustfs_protos::decode_delete_recovery_payload::(&encoded).expect("decode evidence reply"), + DeleteRecoveryRpcReply::Evidence { present: decoded } if decoded == present + )); + } + #[derive(Deserialize)] + enum OlderOperation { + Discover { + _object: Option, + _cursor: Option, + _limit: u32, + }, + } + assert!( + rustfs_protos::decode_delete_recovery_payload::(&encoded).is_err(), + "an older peer must reject existence probing instead of reporting absence" + ); + } + + #[test] + fn orphan_probe_roundtrips_without_mutation_options() { + let path = format!("object/{}", Uuid::new_v4()); + let operation = DeleteRecoveryRpcOperation::InspectOrphanDataDir { path: path.clone() }; + assert_eq!( + operation.max_wire_payload_bytes(), + rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES + ); + let encoded = rmp_serde::to_vec_named(&operation).expect("encode orphan probe"); + let decoded: DeleteRecoveryRpcOperation = rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode probe"); + assert!(matches!(decoded, DeleteRecoveryRpcOperation::InspectOrphanDataDir { path: decoded } if decoded == path)); + let reply = rmp_serde::to_vec_named(&DeleteRecoveryRpcReply::OrphanDataDir { plain: true }).expect("encode probe reply"); + assert!(matches!( + rustfs_protos::decode_delete_recovery_payload::(&reply).expect("decode probe reply"), + DeleteRecoveryRpcReply::OrphanDataDir { plain: true } + )); + #[derive(Deserialize)] + enum OlderOperation { + Inspect { _object: String, _transaction: Uuid }, + } + assert!( + rustfs_protos::decode_delete_recovery_payload::(&encoded).is_err(), + "older peers must reject the additive probe" + ); + } + + #[test] + fn orphan_delete_is_nonrecursive_when_older_peer_ignores_new_option() { + #[derive(Deserialize)] + struct OlderDeleteOptions { + recursive: bool, + immediate: bool, + } + let opts = DeleteOptions { + orphan_data_dir: true, + immediate: true, + ..Default::default() + }; + let encoded = serde_json::to_vec(&opts).expect("encode leaf deletion"); + let old: OlderDeleteOptions = serde_json::from_slice(&encoded).expect("older options ignore additive field"); + assert!(!old.recursive, "ignoring the new option must never broaden deletion"); + assert!(old.immediate); + } +} diff --git a/crates/ecstore/src/disk/os.rs b/crates/ecstore/src/disk/os.rs index 806f885ba..4e308b103 100644 --- a/crates/ecstore/src/disk/os.rs +++ b/crates/ecstore/src/disk/os.rs @@ -558,18 +558,25 @@ pub(crate) mod prepare_rename_test_hooks { /// Fsync a directory so recently created or renamed entries survive power loss. /// No-op on non-Unix platforms where directories cannot be opened for syncing. pub fn fsync_dir_std(dir: impl AsRef) -> io::Result<()> { + fsync_directory_handle_std(dir.as_ref(), None) +} + +pub(crate) fn fsync_directory_handle_std(dir: &Path, opened: Option<&std::fs::File>) -> io::Result<()> { #[cfg(test)] - fsync_dir_recorder::record(dir.as_ref()); + fsync_dir_recorder::record(dir); #[cfg(unix)] { #[cfg(test)] - if let Some(kind) = fsync_dir_recorder::take_failure(dir.as_ref()) { + if let Some(kind) = fsync_dir_recorder::take_failure(dir) { return Err(io::Error::from(kind)); } - std::fs::File::open(dir.as_ref())?.sync_all()?; + match opened { + Some(file) => file.sync_all()?, + None => std::fs::File::open(dir)?.sync_all()?, + } } #[cfg(not(unix))] - let _ = dir; + let _ = (dir, opened); Ok(()) } @@ -742,6 +749,8 @@ fn file_fdatasync_group_commit_wait() -> Duration { #[derive(Clone, Eq, Hash, PartialEq)] struct DstDirFsyncGroupKey { canonical_path: PathBuf, + // A batch must execute in the same resource class as its producers. + isolated: bool, #[cfg(unix)] dev: u64, #[cfg(unix)] @@ -758,13 +767,17 @@ impl DstDirFsyncGroupKey { use std::os::unix::fs::MetadataExt; Ok(Self { canonical_path, + isolated: FSYNC_ON_CURRENT_RUNTIME.get(), dev: metadata.dev(), ino: metadata.ino(), }) } #[cfg(not(unix))] { - Ok(Self { canonical_path }) + Ok(Self { + canonical_path, + isolated: FSYNC_ON_CURRENT_RUNTIME.get(), + }) } } } @@ -801,6 +814,17 @@ impl OpenedDstDirFsyncGroup { Ok(Self { key, dir }) } } + + #[cfg(target_os = "linux")] + fn from_file(dir: &Path, file: Arc) -> io::Result { + let key = DstDirFsyncGroupKey::from_metadata(dir.canonicalize()?, file.metadata()?)?; + Ok(Self { + #[cfg(test)] + dir: key.canonical_path.clone(), + key, + dir_file: file, + }) + } } struct DstDirFsyncWaiter { @@ -1074,19 +1098,45 @@ fn run_dst_dir_fsync_group_worker(group: Arc) -> impl std::fut } } +#[cfg(any(test, not(target_os = "linux")))] async fn fsync_dst_dir_group_commit_with_enabled( dir: impl AsRef, enabled: bool, namespace_owner: Option>, +) -> io::Result<()> { + fsync_dst_dir_group_commit_opened(dir.as_ref(), enabled, namespace_owner, None).await +} + +async fn fsync_dst_dir_group_commit_opened( + dir: &Path, + enabled: bool, + namespace_owner: Option>, + opened: Option>, ) -> io::Result<()> { if !enabled { - return fsync_dir_with_owner(dir.as_ref(), namespace_owner).await; + if let Some(file) = opened { + let dir = dir.to_path_buf(); + return fsync_spawn_blocking(move || { + let _owner = namespace_owner; + fsync_directory_handle_std(&dir, Some(&file)) + }) + .await?; + } + return fsync_dir_with_owner(dir, namespace_owner).await; } - let dir = dir.as_ref().to_path_buf(); - let opened = tokio::task::spawn_blocking(move || OpenedDstDirFsyncGroup::open(&dir)) - .await - .map_err(|err| io::Error::other(format!("blocking dst dir group open failed: {err}")))??; + let dir = dir.to_path_buf(); + let opened = tokio::task::spawn_blocking(move || { + #[cfg(target_os = "linux")] + if let Some(file) = opened { + return OpenedDstDirFsyncGroup::from_file(&dir, file); + } + #[cfg(not(target_os = "linux"))] + let _ = opened; + OpenedDstDirFsyncGroup::open(&dir) + }) + .await + .map_err(|err| io::Error::other(format!("blocking dst dir group open failed: {err}")))??; let (result_rx, worker) = DST_DIR_FSYNC_GROUP_COMMIT.enqueue_opened(opened, namespace_owner)?; if let Some(group) = worker { tokio::spawn(run_dst_dir_fsync_group_worker(group)); @@ -1099,6 +1149,49 @@ async fn fsync_dst_dir_group_commit_with_enabled( } } +/// Reuse the published directory without bypassing group commit or sync admission. +pub(crate) async fn fsync_commit_directory( + dir: &Path, + guard: &RenameCommitGuard, + lease: Arc, + admission: Option<&FileSyncAdmission>, +) -> io::Result<()> { + #[cfg(target_os = "linux")] + { + if dir != guard.directories.destination_path { + return Err(io::Error::new(io::ErrorKind::InvalidInput, "sync directory does not match commit guard")); + } + let file = guard + .directories + .destination + .get() + .ok_or_else(|| io::Error::other("commit directory was not published"))? + .file + .clone(); + if !dst_dir_fsync_group_commit_enabled() + && let Some(admission) = admission + { + let dir = dir.to_path_buf(); + return run_blocking_namespace_file_sync_operation(lease, admission, move || { + #[cfg(test)] + fsync_dir_recorder::record_limited(&dir); + fsync_directory_handle_std(&dir, Some(&file)) + }) + .await; + } + fsync_dst_dir_group_commit_opened(dir, dst_dir_fsync_group_commit_enabled(), Some(lease), Some(file)).await + } + #[cfg(not(target_os = "linux"))] + { + let _ = guard; + match admission { + Some(admission) => fsync_dst_dir_group_commit_or_namespace_file_sync_limit(dir, lease, admission).await, + None => fsync_dst_dir_group_commit(dir, Some(lease)).await, + } + } +} + +#[cfg(not(target_os = "linux"))] pub(crate) async fn fsync_dst_dir_group_commit( dir: impl AsRef, namespace_owner: Option>, @@ -1106,6 +1199,7 @@ pub(crate) async fn fsync_dst_dir_group_commit( fsync_dst_dir_group_commit_with_enabled(dir, dst_dir_fsync_group_commit_enabled(), namespace_owner).await } +#[cfg(not(target_os = "linux"))] pub(crate) async fn fsync_dst_dir_group_commit_or_namespace_file_sync_limit( dir: impl AsRef, lease: Arc, @@ -1133,7 +1227,7 @@ fn clear_dst_dir_fsync_group_commit_for_test() { DST_DIR_FSYNC_GROUP_COMMIT.clear_for_test(); } -type FileFdatasyncGroupKey = usize; +type FileFdatasyncGroupKey = (usize, bool); struct FileFdatasyncWaiter { files: Vec, @@ -1205,7 +1299,7 @@ impl FileFdatasyncGroupCommit { )); } let (result_tx, result_rx) = oneshot::channel(); - let key = Arc::as_ptr(&disk_permits) as FileFdatasyncGroupKey; + let key = (Arc::as_ptr(&disk_permits) as usize, FSYNC_ON_CURRENT_RUNTIME.get()); let mut registry = self.inner.lock(); registry.groups.retain(|_, group| group.disk_permits.strong_count() > 0); if registry.total_waiters >= MAX_FILE_FDATASYNC_WAITERS { @@ -1443,9 +1537,26 @@ static FSYNC_RUNTIME: LazyLock> = LazyLock::new( } }); +thread_local! { + static FSYNC_ON_CURRENT_RUNTIME: std::cell::Cell = const { std::cell::Cell::new(false) }; +} + +/// Called only when starting threads belonging to an isolated I/O runtime. +/// Async and blocking threads use the same runtime's blocking budget for fsync. +pub(crate) fn use_current_runtime_for_fsync() { + FSYNC_ON_CURRENT_RUNTIME.set(true); +} + +pub(super) fn is_isolated_io_thread() -> bool { + FSYNC_ON_CURRENT_RUNTIME.get() +} + /// Spawn a blocking task on the fsync-dedicated runtime if configured, /// otherwise fall back to the main tokio blocking pool. fn fsync_spawn_blocking(f: impl FnOnce() -> T + Send + 'static) -> tokio::task::JoinHandle { + if FSYNC_ON_CURRENT_RUNTIME.get() { + return tokio::task::spawn_blocking(f); + } match FSYNC_RUNTIME.as_ref() { Some(rt) => rt.spawn_blocking(f), None => tokio::task::spawn_blocking(f), @@ -1572,6 +1683,7 @@ pub(crate) async fn acquire_metadata_mutation_lease( /// cancelled while a blocking filesystem call is still running. pub(crate) struct NamespaceMutationLease { _namespace_guard: OwnedMutexGuard<()>, + _staging_guard: Option>, _volume_guard: Option>, _secondary_volume_guard: Option>, external_guard: Mutex>>, @@ -1595,6 +1707,7 @@ pub(in crate::disk) async fn acquire_namespace_mutation_lease_with_owner( ) -> Arc { Arc::new(NamespaceMutationLease { _namespace_guard: disk_namespace_mutation_lock(path).lock_owned().await, + _staging_guard: None, _volume_guard: None, _secondary_volume_guard: None, external_guard: Mutex::new(None), @@ -1613,6 +1726,7 @@ pub(in crate::disk) fn try_acquire_namespace_mutation_lease_with_owner( .map_err(|_| DiskError::Io(io::Error::from(io::ErrorKind::WouldBlock)))?; Ok(Arc::new(NamespaceMutationLease { _namespace_guard: guard, + _staging_guard: None, _volume_guard: None, _secondary_volume_guard: None, external_guard: Mutex::new(None), @@ -1627,19 +1741,34 @@ pub(crate) async fn acquire_rename_data_mutation_lease( volume: &str, destination_object: &Path, ) -> Arc { - acquire_rename_data_mutation_lease_with_owner(root, volume, destination_object, None).await + acquire_rename_data_mutation_lease_with_owner(root, volume, destination_object, None, None).await } pub(crate) async fn acquire_rename_data_mutation_lease_with_owner( root: &Path, volume: &str, destination_object: &Path, + staging_prefix: Option<&Path>, namespace_owner: Option>, ) -> Arc { - let namespace_guard = disk_namespace_mutation_lock(destination_object).lock_owned().await; + // Metadata ownership precedes namespace paths in lexical order, then volume + // ownership. Deduplicate identical keys so TMP-to-TMP publication cannot self-lock. + let destination_lock = disk_namespace_mutation_lock(destination_object); + let (namespace_guard, staging_guard) = match staging_prefix { + Some(source) if source < destination_object => { + let staging = disk_namespace_mutation_lock(source).lock_owned().await; + (destination_lock.lock_owned().await, Some(staging)) + } + Some(source) if source > destination_object => { + let destination = destination_lock.lock_owned().await; + (destination, Some(disk_namespace_mutation_lock(source).lock_owned().await)) + } + _ => (destination_lock.lock_owned().await, None), + }; let volume_guard = disk_volume_mutation_lock(root, volume).read_owned().await; Arc::new(NamespaceMutationLease { _namespace_guard: namespace_guard, + _staging_guard: staging_guard, _volume_guard: Some(volume_guard), _secondary_volume_guard: None, external_guard: Mutex::new(None), @@ -1654,8 +1783,13 @@ pub(in crate::disk) async fn acquire_rename_data_mutation_lease_for_volumes( source_volume: &str, destination_volume: &str, destination_object: &Path, + staging_prefix: Option<&Path>, namespace_owner: Option>, ) -> Arc { + let staging_guard = match staging_prefix { + Some(source) if source < destination_object => Some(disk_namespace_mutation_lock(source).lock_owned().await), + _ => None, + }; let namespace = disk_namespace_mutation_lock(destination_object).lock_owned(); #[cfg(all(test, not(windows)))] let namespace_guard = { @@ -1675,6 +1809,10 @@ pub(in crate::disk) async fn acquire_rename_data_mutation_lease_for_volumes( }; #[cfg(not(all(test, not(windows))))] let namespace_guard = namespace.await; + let staging_guard = match staging_prefix { + Some(source) if source > destination_object => Some(disk_namespace_mutation_lock(source).lock_owned().await), + _ => staging_guard, + }; let source_key = root.join(source_volume); let destination_key = root.join(destination_volume); let (first, second) = if source_key <= destination_key { @@ -1690,6 +1828,7 @@ pub(in crate::disk) async fn acquire_rename_data_mutation_lease_for_volumes( }; Arc::new(NamespaceMutationLease { _namespace_guard: namespace_guard, + _staging_guard: staging_guard, _volume_guard: Some(first_guard), _secondary_volume_guard: second_guard, external_guard: Mutex::new(None), @@ -2251,9 +2390,19 @@ pub(crate) async fn remove_dir_with_owner( if namespace_owner.is_none() { return tokio::fs::remove_dir(path).await; } + let lease = acquire_namespace_mutation_lease_with_owner(path.as_ref(), namespace_owner).await; + remove_dir_with_lease(path, lease).await +} + +/// Remove a directory whose namespace is already leased by the caller. +pub(crate) async fn remove_dir_with_lease(path: impl AsRef, lease: Arc) -> io::Result<()> { let path = path.as_ref().to_path_buf(); - let lease = acquire_namespace_mutation_lease_with_owner(&path, namespace_owner).await; - run_blocking_namespace_operation(lease, move || std::fs::remove_dir(path)).await + run_blocking_namespace_operation(lease, move || { + #[cfg(all(any(test, feature = "test-util"), not(windows)))] + prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Remove, &path); + std::fs::remove_dir(path) + }) + .await } /// Preserve raw rename semantics while retaining a counted owner in the syscall. @@ -2456,6 +2605,20 @@ pub(crate) fn create_prepared_rename_source_with_commit_guard( #[cfg(not(windows))] { let _ = (dst_file_path, commit_guard); + #[cfg(target_os = "linux")] + let source = { + use rustix::fs::{Mode, OFlags}; + let directories = &commit_guard.directories; + let name = directories.file_name(src_file_path, true)?; + directories.file_name(dst_file_path, false)?; + open_linux_relative( + &directories.source.file, + Path::new(name), + OFlags::CREATE | OFlags::TRUNC | OFlags::WRONLY, + Mode::from_raw_mode(0o666), + )? + }; + #[cfg(not(target_os = "linux"))] let source = std::fs::OpenOptions::new() .create(true) .write(true) @@ -2476,18 +2639,39 @@ pub(crate) fn create_prepared_rename_source_with_commit_guard( } } -#[cfg(windows)] +#[cfg(any(windows, target_os = "linux"))] pub(crate) fn read_destination_file_with_commit_guard( file_path: &Path, commit_guard: &RenameCommitGuard, ) -> io::Result>> { - if file_path.parent() != Some(commit_guard.destination_parent.as_path()) { - return Err(io::Error::new( - io::ErrorKind::InvalidInput, - "destination file parent does not match its commit guard", - )); + #[cfg(target_os = "linux")] + { + use rustix::fs::{Mode, OFlags}; + let directories = &commit_guard.directories; + let name = directories.file_name(file_path, false)?; + let Some(parent) = directories.destination.get() else { + return Ok(None); + }; + match open_linux_relative(&parent.file, Path::new(name), OFlags::RDONLY, Mode::empty()) { + Ok(mut file) => { + let mut data = Vec::new(); + std::io::Read::read_to_end(&mut file, &mut data)?; + Ok(Some(data)) + } + Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(None), + Err(err) => Err(err), + } + } + #[cfg(windows)] + { + if file_path.parent() != Some(commit_guard.destination_parent.as_path()) { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "destination file parent does not match its commit guard", + )); + } + read_windows_relative_file(file_path, &commit_guard.destination_parent_guard) } - read_windows_relative_file(file_path, &commit_guard.destination_parent_guard) } #[cfg(windows)] @@ -2525,23 +2709,40 @@ pub(crate) async fn rename_all_with_prepared_source( dst_file_path: impl AsRef, base_dir: impl AsRef, publication_root: &PublicationRoot, - _commit_guard: &RenameCommitGuard, + commit_guard: &RenameCommitGuard, lease: Arc, ) -> Result<()> { let src_file_path = src_file_path.as_ref().to_path_buf(); let dst_file_path = dst_file_path.as_ref().to_path_buf(); let base_dir = base_dir.as_ref().to_path_buf(); + #[cfg(not(target_os = "linux"))] let publication_root = publication_root.clone(); + #[cfg(target_os = "linux")] + let _ = publication_root; + let commit_guard = commit_guard.clone(); let operation = { let src_file_path = src_file_path.clone(); let dst_file_path = dst_file_path.clone(); + #[cfg(not(target_os = "linux"))] let base_dir = base_dir.clone(); move || { - validate_prepared_rename_source(&prepared_source, &src_file_path)?; - let preparation = prepare_rename_with_retry(&src_file_path, &dst_file_path, &base_dir, &publication_root)?; - #[cfg(any(test, feature = "test-util"))] - prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::PreparedRename, &dst_file_path); - rename_prepared(&src_file_path, &dst_file_path, &preparation) + #[cfg(target_os = "linux")] + { + #[cfg(any(test, feature = "test-util"))] + prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::PreparedRename, &dst_file_path); + commit_guard + .directories + .rename(&src_file_path, &dst_file_path, Some(&prepared_source)) + } + #[cfg(not(target_os = "linux"))] + { + let _ = commit_guard; + validate_prepared_rename_source(&prepared_source, &src_file_path)?; + let preparation = prepare_rename_with_retry(&src_file_path, &dst_file_path, &base_dir, &publication_root)?; + #[cfg(any(test, feature = "test-util"))] + prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::PreparedRename, &dst_file_path); + rename_prepared(&src_file_path, &dst_file_path, &preparation) + } } }; let result = run_blocking_namespace_operation(lease, operation).await; @@ -2558,10 +2759,35 @@ pub(crate) async fn rename_all_with_commit_guard( dst_file_path: impl AsRef, base_dir: impl AsRef, publication_root: &PublicationRoot, - _commit_guard: &RenameCommitGuard, + commit_guard: &RenameCommitGuard, lease: Arc, ) -> Result<()> { - rename_all_with_lease(src_file_path, dst_file_path, base_dir, publication_root, lease).await + #[cfg(target_os = "linux")] + { + let source = src_file_path.as_ref().to_path_buf(); + let destination = dst_file_path.as_ref().to_path_buf(); + let guard = commit_guard.clone(); + let _ = (base_dir, publication_root); + run_blocking_namespace_operation(lease, move || { + guard.directories.prepare_destination()?; + #[cfg(test)] + prepared_publication_test_hooks::run_rename_destination(&source, &destination); + #[cfg(any(test, feature = "test-util"))] + { + prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, &source); + prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, &destination); + } + guard.directories.rename(&source, &destination, None) + }) + .await + .map_err(to_file_error) + .map_err(DiskError::from) + } + #[cfg(not(target_os = "linux"))] + { + let _ = commit_guard; + rename_all_with_lease(src_file_path, dst_file_path, base_dir, publication_root, lease).await + } } #[tracing::instrument(level = "debug", skip_all)] @@ -2801,7 +3027,7 @@ async fn reliable_rename_inner_with_lease( result } -#[cfg(not(windows))] +#[cfg(not(any(windows, target_os = "linux")))] fn validate_prepared_rename_source(prepared_source: &PreparedRenameSource, src_file_path: &Path) -> io::Result<()> { if prepared_source.path != src_file_path { return Err(io::Error::new( @@ -3001,6 +3227,8 @@ pub(crate) async fn fsync_dir_with_namespace_file_sync_limit( struct RenamePreparation { parent_guard: Option, + #[cfg(target_os = "linux")] + publication_root: PublicationRoot, #[cfg(windows)] _source_parent_guard: ExistingBaseDirectoryGuard, #[cfg(windows)] @@ -3009,15 +3237,32 @@ struct RenamePreparation { #[cfg(not(windows))] fn prepare_rename_with_retry( - src_file_path: &Path, + _src_file_path: &Path, dst_file_path: &Path, base_dir: &Path, publication_root: &PublicationRoot, ) -> io::Result { - let prune_budget = prepare_prune_budget(dst_file_path, base_dir); + let parent_guard = dst_file_path + .parent() + .map(|parent| prepare_destination_parent_with_retry(parent, base_dir, publication_root)) + .transpose()?; + Ok(RenamePreparation { + parent_guard, + #[cfg(target_os = "linux")] + publication_root: publication_root.clone(), + }) +} + +#[cfg(not(windows))] +fn prepare_destination_parent_with_retry( + parent: &Path, + base_dir: &Path, + publication_root: &PublicationRoot, +) -> io::Result { + let prune_budget = parent_prune_budget(parent, base_dir); let mut attempt = 0; loop { - match prepare_rename(src_file_path, dst_file_path, base_dir, publication_root) { + match mkdir_all_below_existing_base_std(parent, base_dir, publication_root) { Ok(preparation) => return Ok(preparation), Err(err) if should_retry_prepare(&err, attempt, prune_budget) => { attempt += 1; @@ -3107,20 +3352,6 @@ fn prepare_rename_with_retry( }) } -#[cfg(not(windows))] -fn prepare_rename( - _src_file_path: &Path, - dst_file_path: &Path, - base_dir: &Path, - publication_root: &PublicationRoot, -) -> io::Result { - let parent_guard = dst_file_path - .parent() - .map(|parent| mkdir_all_below_existing_base_std(parent, base_dir, publication_root)) - .transpose()?; - Ok(RenamePreparation { parent_guard }) -} - /// Publish a prepared rename. The retry budget starts fresh here: preparation /// keeps its own counter, so a chain rebuilt after a concurrent prune must not /// cost the rename its one retry. @@ -3136,9 +3367,23 @@ fn rename_prepared(_src_file_path: &Path, dst_file_path: &Path, preparation: &Re #[cfg(not(windows))] { + // A missing source parent is not a destination-prune race. Open it once + // outside both retry loops, retaining it through any rename retry. + #[cfg(target_os = "linux")] + let source_parent = preparation.publication_root.open_directory( + _src_file_path + .parent() + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "missing rename source parent"))?, + )?; let mut attempt = 0; loop { - let rename_result = rename_into_existing_parent(_src_file_path, dst_file_path, preparation.parent_guard.as_ref()); + let rename_result = rename_into_existing_parent( + _src_file_path, + dst_file_path, + preparation.parent_guard.as_ref(), + #[cfg(target_os = "linux")] + &source_parent, + ); match rename_result { Ok(()) => return Ok(()), Err(err) if should_retry_rename(&err, attempt) => { @@ -3177,8 +3422,9 @@ fn rename_into_existing_parent( src_file_path: &Path, dst_file_path: &Path, parent_guard: Option<&ExistingBaseDirectoryGuard>, + #[cfg(target_os = "linux")] source_parent: &std::fs::File, ) -> io::Result<()> { - use rustix::fs::{Mode, OFlags, open, renameat}; + use rustix::fs::renameat; let Some(parent_guard) = parent_guard else { let rename_started = rustfs_io_metrics::put_stage_timer(); @@ -3189,6 +3435,7 @@ fn rename_into_existing_parent( ); return result; }; + #[cfg(not(target_os = "linux"))] let src_parent = src_file_path .parent() .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "rename source must have a parent directory"))?; @@ -3198,10 +3445,11 @@ fn rename_into_existing_parent( let dst_name = dst_file_path .file_name() .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "rename destination must have a file name"))?; - let src_parent = open( + #[cfg(not(target_os = "linux"))] + let src_parent = rustix::fs::open( src_parent, - OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, - Mode::empty(), + rustix::fs::OFlags::RDONLY | rustix::fs::OFlags::DIRECTORY | rustix::fs::OFlags::NOFOLLOW | rustix::fs::OFlags::CLOEXEC, + rustix::fs::Mode::empty(), ) .map_err(io::Error::from)?; let dst_parent = parent_guard @@ -3209,6 +3457,9 @@ fn rename_into_existing_parent( .ok_or_else(|| io::Error::other("rename destination parent guard is empty"))?; let rename_started = rustfs_io_metrics::put_stage_timer(); + #[cfg(target_os = "linux")] + let result = renameat(source_parent, src_name, dst_parent, dst_name).map_err(io::Error::from); + #[cfg(not(target_os = "linux"))] let result = renameat(&src_parent, src_name, dst_parent, dst_name).map_err(io::Error::from); rustfs_io_metrics::record_put_object_stage_duration_from( rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_RENAME_SYSCALL, @@ -3391,16 +3642,18 @@ struct WindowsDirectoryHandle { /// Stable root for namespace-changing disk operations. /// -/// Windows opens the configured endpoint once and keeps that directory identity -/// pinned for the lifetime of the disk. Publication then resolves every source -/// and destination component relative to this handle instead of re-entering the -/// mutable pathname namespace. Other platforms retain the path so callers use a -/// uniform API while their existing `openat`/`renameat` guards remain unchanged. +/// Linux and Windows pin the configured directory for the lifetime of the disk. +/// Linux also exposes a procfs alias for operations not yet using relative IO; +/// both access forms share one root handle and therefore one mount identity. #[derive(Clone)] pub(crate) struct PublicationRoot { - path: PathBuf, + path: Arc, + #[cfg(target_os = "linux")] + directory: Arc, + #[cfg(target_os = "linux")] + io_path: Arc, #[cfg(windows)] - configured_path: PathBuf, + configured_path: Arc, #[cfg(windows)] directory: WindowsDirectoryHandle, } @@ -3414,15 +3667,34 @@ impl PublicationRoot { #[cfg(windows)] let (resolved_path, directory) = open_windows_publication_root(path)?; + #[cfg(target_os = "linux")] + let directory = { + use rustix::fs::{Mode, OFlags, open}; + Arc::new(std::fs::File::from(open( + path, + OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + )?)) + }; + #[cfg(target_os = "linux")] + let io_path = { + use std::os::fd::AsRawFd; + Arc::new(PathBuf::from(format!("/proc/self/fd/{}/.", directory.as_raw_fd()))) + }; + Ok(Self { #[cfg(not(windows))] - path: path.to_path_buf(), + path: Arc::new(path.to_path_buf()), #[cfg(windows)] - path: resolved_path, + path: Arc::new(resolved_path), #[cfg(windows)] - configured_path: path.to_path_buf(), + configured_path: Arc::new(path.to_path_buf()), #[cfg(windows)] directory, + #[cfg(target_os = "linux")] + directory, + #[cfg(target_os = "linux")] + io_path, }) } @@ -3430,15 +3702,170 @@ impl PublicationRoot { &self.path } + #[cfg(target_os = "linux")] + pub(crate) fn directory(&self) -> &Arc { + &self.directory + } + + #[cfg(target_os = "linux")] + pub(crate) fn io_path(&self) -> &Path { + &self.io_path + } + + pub(crate) fn open_readonly(&self, path: &Path) -> io::Result { + #[cfg(target_os = "linux")] + return self.open_relative(path, rustix::fs::OFlags::RDONLY, rustix::fs::Mode::empty()); + #[cfg(not(target_os = "linux"))] + std::fs::File::open(path) + } + + pub(crate) fn create_truncate(&self, path: &Path) -> io::Result { + #[cfg(target_os = "linux")] + { + use rustix::fs::{Mode, OFlags}; + self.open_relative(path, OFlags::CREATE | OFlags::WRONLY | OFlags::TRUNC, Mode::from_raw_mode(0o666)) + } + #[cfg(not(target_os = "linux"))] + std::fs::OpenOptions::new().create(true).write(true).truncate(true).open(path) + } + + pub(crate) fn create_truncate_with_parent(&self, path: &Path) -> io::Result<(std::fs::File, Option)> { + #[cfg(target_os = "linux")] + { + use rustix::fs::{Mode, OFlags}; + let parent = self.open_directory( + path.parent() + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "missing file parent"))?, + )?; + let name = path + .file_name() + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "missing file name"))?; + let file = open_linux_relative( + &parent, + Path::new(name), + OFlags::CREATE | OFlags::WRONLY | OFlags::TRUNC, + Mode::from_raw_mode(0o666), + )?; + Ok((file, Some(parent))) + } + #[cfg(not(target_os = "linux"))] + Ok((self.create_truncate(path)?, None)) + } + + #[cfg(target_os = "linux")] + fn open_relative(&self, path: &Path, flags: rustix::fs::OFlags, mode: rustix::fs::Mode) -> io::Result { + let relative = path + .strip_prefix(self.io_path.as_path()) + .or_else(|_| path.strip_prefix(self.path.as_path())) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "path must remain below its publication root"))?; + open_linux_relative(&self.directory, relative, flags, mode) + } + + #[cfg(target_os = "linux")] + fn open_directory(&self, path: &Path) -> io::Result { + self.open_relative( + path, + rustix::fs::OFlags::RDONLY | rustix::fs::OFlags::DIRECTORY, + rustix::fs::Mode::empty(), + ) + } + #[cfg(windows)] fn relative_path<'a>(&self, path: &'a Path) -> io::Result<&'a Path> { // The configured path only derives a suffix; traversal stays rooted at the pinned directory handle. - path.strip_prefix(&self.path) - .or_else(|_| path.strip_prefix(&self.configured_path)) + path.strip_prefix(self.path.as_path()) + .or_else(|_| path.strip_prefix(self.configured_path.as_path())) .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "path must remain below its publication root")) } } +/// Open the actual IO handle with the same containment policy as path validation. +/// Older kernels retain fd-relative traversal; only ENOSYS enables the fallback. +#[cfg(target_os = "linux")] +fn open_linux_relative( + directory: &std::fs::File, + relative: &Path, + flags: rustix::fs::OFlags, + mode: rustix::fs::Mode, +) -> io::Result { + use rustix::fs::{OFlags, ResolveFlags, openat2}; + if relative + .components() + .any(|component| !matches!(component, Component::Normal(_) | Component::CurDir)) + { + return Err(io::Error::new(io::ErrorKind::InvalidInput, "invalid relative disk path")); + } + let relative = if relative.as_os_str().is_empty() { + Path::new(".") + } else { + relative + }; + let flags = flags | OFlags::CLOEXEC | OFlags::NOFOLLOW; + match openat2(directory, relative, flags, mode, ResolveFlags::BENEATH | ResolveFlags::NO_SYMLINKS) { + Ok(fd) => Ok(fd.into()), + Err(rustix::io::Errno::NOSYS) => open_linux_relative_legacy(directory, relative, flags, mode), + Err(rustix::io::Errno::LOOP | rustix::io::Errno::XDEV) => Err(DiskError::InvalidPath.into()), + Err(err) => Err(err.into()), + } +} + +#[cfg(target_os = "linux")] +fn open_linux_relative_legacy( + directory: &std::fs::File, + relative: &Path, + flags: rustix::fs::OFlags, + mode: rustix::fs::Mode, +) -> io::Result { + use rustix::fs::OFlags; + + let mut parent = None; + let mut components = relative + .components() + .filter_map(|component| match component { + Component::Normal(name) => Some(name), + _ => None, + }) + .peekable(); + while let Some(name) = components.next() { + let current = parent.as_ref().unwrap_or(directory); + if components.peek().is_none() { + return open_linux_relative_component(current, name, flags, mode); + } + parent = Some(open_linux_relative_component( + current, + name, + OFlags::PATH | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + rustix::fs::Mode::empty(), + )?); + } + open_linux_relative_component(directory, std::ffi::OsStr::new("."), flags, mode) +} + +#[cfg(target_os = "linux")] +fn open_linux_relative_component( + directory: &std::fs::File, + name: &std::ffi::OsStr, + flags: rustix::fs::OFlags, + mode: rustix::fs::Mode, +) -> io::Result { + use rustix::fs::{AtFlags, FileType, openat, statat}; + use rustix::io::Errno; + match openat(directory, name, flags, mode) { + Ok(fd) => Ok(fd.into()), + Err(Errno::LOOP) => Err(DiskError::InvalidPath.into()), + // O_DIRECTORY | O_NOFOLLOW reports ENOTDIR for an intermediate link. + // Inspect only this failure so ordinary non-directory errors retain + // their classification without adding a stat to successful opens. + Err(Errno::NOTDIR) + if statat(directory, name, AtFlags::SYMLINK_NOFOLLOW) + .is_ok_and(|metadata| FileType::from_raw_mode(metadata.st_mode) == FileType::Symlink) => + { + Err(DiskError::InvalidPath.into()) + } + Err(err) => Err(err.into()), + } +} + #[cfg(windows)] #[derive(Clone)] pub(crate) struct ExistingBaseDirectoryGuard { @@ -3476,6 +3903,8 @@ pub(crate) type ExistingBaseDirectoryGuard = (); #[derive(Clone)] pub(crate) struct RenameCommitGuard { + #[cfg(target_os = "linux")] + directories: Arc, #[cfg(windows)] source_parent: PathBuf, #[cfg(windows)] @@ -3486,6 +3915,108 @@ pub(crate) struct RenameCommitGuard { destination_parent_guard: ExistingBaseDirectoryGuard, } +#[cfg(target_os = "linux")] +struct LinuxCommitDirectory { + file: Arc, + device: u64, + inode: u64, +} + +#[cfg(target_os = "linux")] +impl LinuxCommitDirectory { + fn new(file: std::fs::File) -> io::Result { + use std::os::unix::fs::MetadataExt; + let metadata = file.metadata()?; + Ok(Self { + file: Arc::new(file), + device: metadata.dev(), + inode: metadata.ino(), + }) + } +} + +#[cfg(target_os = "linux")] +struct LinuxCommitDirectories { + root: PublicationRoot, + source_path: PathBuf, + destination_path: PathBuf, + source: LinuxCommitDirectory, + destination: std::sync::OnceLock, + destination_base: PathBuf, +} + +#[cfg(target_os = "linux")] +impl LinuxCommitDirectories { + fn prepare_destination(&self) -> io::Result<&LinuxCommitDirectory> { + if let Some(directory) = self.destination.get() { + return Ok(directory); + } + let mut parents = prepare_destination_parent_with_retry(&self.destination_path, &self.destination_base, &self.root)?; + let directory = std::fs::File::from( + parents + .pop() + .ok_or_else(|| io::Error::other("missing destination directory guard"))?, + ); + let directory = LinuxCommitDirectory::new(directory)?; + Ok(self.destination.get_or_init(|| directory)) + } + + fn file_name<'a>(&self, path: &'a Path, source: bool) -> io::Result<&'a std::ffi::OsStr> { + let parent = if source { &self.source_path } else { &self.destination_path }; + if path.parent() != Some(parent) { + return Err(io::Error::new(io::ErrorKind::InvalidInput, "file parent does not match its commit guard")); + } + path.file_name() + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "missing file name")) + } + + fn validate(&self) -> io::Result<()> { + use std::os::unix::fs::MetadataExt; + // A pinned directory may have been unlinked or replaced. Do not report a + // successful publication into a detached object tree. + for (path, directory) in [ + (&self.source_path, &self.source), + (&self.destination_path, self.prepare_destination()?), + ] { + let current = self.root.open_directory(path)?.metadata()?; + if (current.dev(), current.ino()) != (directory.device, directory.inode) { + return Err(io::Error::new(io::ErrorKind::InvalidData, "commit directory identity changed")); + } + } + Ok(()) + } + + fn rename(&self, source: &Path, destination: &Path, prepared: Option<&PreparedRenameSource>) -> io::Result<()> { + let source_name = self.file_name(source, true)?; + let destination_name = self.file_name(destination, false)?; + let destination_parent = self.prepare_destination()?; + self.validate()?; + let mut attempt = 0; + loop { + if let Some(prepared) = prepared { + let metadata = rustix::fs::statat(&*self.source.file, source_name, rustix::fs::AtFlags::SYMLINK_NOFOLLOW)?; + if prepared.path != source || metadata.st_dev != prepared.device || metadata.st_ino != prepared.inode { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "rename source identity changed while publication was prepared", + )); + } + } + let started = rustfs_io_metrics::put_stage_timer(); + let result = rustix::fs::renameat(&*self.source.file, source_name, &*destination_parent.file, destination_name) + .map_err(io::Error::from); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_RENAME_SYSCALL, + started, + ); + match result { + Err(err) if should_retry_rename(&err, attempt) => attempt += 1, + result => return result, + } + } + } +} + pub(crate) struct RenameDestinationPathGuard { #[cfg(windows)] directory: PathBuf, @@ -3685,6 +4216,29 @@ pub(crate) fn prepare_rename_commit_guard( destination_base: &Path, publication_root: &PublicationRoot, ) -> io::Result { + #[cfg(target_os = "linux")] + { + existing_base_directory_suffix(destination_parent, destination_base)?; + let source = LinuxCommitDirectory::new(publication_root.open_directory(source_parent)?)?; + let destination = std::sync::OnceLock::new(); + match publication_root.open_directory(destination_parent) { + Ok(directory) => { + let _ = destination.set(LinuxCommitDirectory::new(directory)?); + } + Err(err) if err.kind() == io::ErrorKind::NotFound => {} + Err(err) => return Err(err), + } + Ok(RenameCommitGuard { + directories: Arc::new(LinuxCommitDirectories { + root: publication_root.clone(), + source_path: source_parent.to_path_buf(), + destination_path: destination_parent.to_path_buf(), + source, + destination, + destination_base: destination_base.to_path_buf(), + }), + }) + } #[cfg(windows)] { // A same-directory rename must use one shared-write parent handle: @@ -3713,7 +4267,7 @@ pub(crate) fn prepare_rename_commit_guard( }) } - #[cfg(not(windows))] + #[cfg(not(any(windows, target_os = "linux")))] { let _ = (source_parent, destination_parent, destination_base, publication_root); Ok(RenameCommitGuard {}) @@ -3832,7 +4386,10 @@ fn windows_final_path_with_flags(handle: &winapi_util::Handle, flags: u32) -> io .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "Windows final path length exceeds usize"))?, ); let path = PathBuf::from(OsString::from_wide(&buffer)); - Ok(rustfs_utils::simplified(&path).to_path_buf()) + // Keep the root in the non-verbatim form: callers extend it by joining + // `/`-separated relative paths, which a verbatim (`\\?\`) root would + // turn into ERROR_INVALID_NAME for UNC shares and roots past MAX_PATH. + Ok(rustfs_utils::non_verbatim(&path).into_owned()) } #[cfg(windows)] @@ -4511,11 +5068,7 @@ fn windows_rename_source_is_allowed(attributes: u32, reparse_tag: u32) -> bool { attributes & FILE_ATTRIBUTE_REPARSE_POINT == 0 || reparse_tag == IO_REPARSE_TAG_DEDUP } -pub(crate) fn mkdir_all_below_existing_base_std( - dir_path: &Path, - base_dir: &Path, - publication_root: &PublicationRoot, -) -> io::Result { +fn existing_base_directory_suffix<'a>(dir_path: &'a Path, base_dir: &Path) -> io::Result<&'a Path> { let relative = dir_path .strip_prefix(base_dir) .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "rename destination must remain below its base directory"))?; @@ -4528,15 +5081,31 @@ pub(crate) fn mkdir_all_below_existing_base_std( } } + Ok(relative) +} + +pub(crate) fn mkdir_all_below_existing_base_std( + dir_path: &Path, + base_dir: &Path, + publication_root: &PublicationRoot, +) -> io::Result { + let relative = existing_base_directory_suffix(dir_path, base_dir)?; + #[cfg(unix)] { - let _ = publication_root; - use rustix::fs::{Mode, OFlags, mkdirat, open, openat}; + use rustix::fs::{Mode, OFlags, mkdirat, openat}; use rustix::io::Errno; let flags = OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC; let mode = Mode::RWXU | Mode::RWXG | Mode::RWXO; - let mut parents = vec![open(base_dir, flags, Mode::empty()).map_err(io::Error::from)?]; + #[cfg(target_os = "linux")] + let base = publication_root.open_directory(base_dir)?.into(); + #[cfg(not(target_os = "linux"))] + let base = { + let _ = publication_root; + rustix::fs::open(base_dir, flags, Mode::empty()).map_err(io::Error::from)? + }; + let mut parents = vec![base]; #[cfg(test)] let mut walked_path = base_dir.to_path_buf(); @@ -4656,10 +5225,18 @@ fn should_retry_rename(err: &io::Error, attempt: usize) -> bool { /// steal an attempt by making that same upward progress. A destination whose /// parent *is* the base gets a budget of zero, keeping `NotFound` immediately /// terminal for speculative cleanup renames. +#[cfg(any(windows, test))] fn prepare_prune_budget(dst_file_path: &Path, base_dir: &Path) -> usize { dst_file_path .parent() - .and_then(|parent| parent.strip_prefix(base_dir).ok()) + .map(|parent| parent_prune_budget(parent, base_dir)) + .unwrap_or(0) +} + +fn parent_prune_budget(parent: &Path, base_dir: &Path) -> usize { + parent + .strip_prefix(base_dir) + .ok() .map(|relative| relative.components().count()) .unwrap_or(0) } @@ -4808,6 +5385,107 @@ mod tests { use tempfile::tempdir; use tracing_subscriber::fmt::MakeWriter; + #[tokio::test] + async fn rename_staging_leases_are_ordered_and_deduplicated() { + let root = tempdir().expect("namespace fixture"); + let early = root.path().join("a"); + let late = root.path().join("b"); + let blocker = acquire_namespace_mutation_lease_with_owner(&late, None).await; + let first_root = root.path().to_path_buf(); + let first_early = early.clone(); + let first_late = late.clone(); + let first = tokio::spawn(async move { + drop( + acquire_rename_data_mutation_lease_with_owner(&first_root, "bucket", &first_late, Some(&first_early), None).await, + ); + }); + tokio::time::timeout(Duration::from_secs(10), async { + while disk_namespace_mutation_lock(&early).try_lock_owned().is_ok() { + tokio::task::yield_now().await; + } + }) + .await + .expect("source must acquire the earlier key before waiting for destination"); + let second_root = root.path().to_path_buf(); + let second_early = early.clone(); + let second_late = late.clone(); + let second = tokio::spawn(async move { + drop( + acquire_rename_data_mutation_lease_with_owner(&second_root, "bucket", &second_early, Some(&second_late), None) + .await, + ); + }); + drop(blocker); + tokio::time::timeout(Duration::from_secs(10), async { + first.await.expect("first publication"); + second.await.expect("reverse publication"); + }) + .await + .expect("reverse source/destination order must not deadlock"); + drop( + tokio::time::timeout( + Duration::from_secs(10), + acquire_rename_data_mutation_lease_with_owner(root.path(), "bucket", &early, Some(&early), None), + ) + .await + .expect("identical source and destination key must be acquired once"), + ); + } + + #[test] + fn fsync_groups_do_not_mix_cleanup_and_foreground_workers() { + let dir = tempdir().unwrap(); + let dst_groups = Arc::new(DstDirFsyncGroupCommit::default()); + let file_groups = Arc::new(FileFdatasyncGroupCommit::default()); + let disk_permits = Arc::new(Semaphore::new(1)); + let (_, dst_worker) = dst_groups.enqueue_for_test(dir.path()).unwrap(); + let (_, file_worker) = file_groups + .enqueue(disk_permits.clone(), vec![dir.path().join("one")]) + .unwrap(); + assert!(dst_worker.is_some()); + assert!(file_worker.is_some()); + let cleanup_dst_groups = dst_groups.clone(); + let cleanup_file_groups = file_groups.clone(); + std::thread::spawn(move || { + use_current_runtime_for_fsync(); + let (_, dst_worker) = cleanup_dst_groups.enqueue_for_test(dir.path()).unwrap(); + let (_, file_worker) = cleanup_file_groups + .enqueue(disk_permits, vec![dir.path().join("two")]) + .unwrap(); + assert!(dst_worker.is_some(), "cleanup must not join a foreground directory worker"); + assert!(file_worker.is_some(), "cleanup must not join a foreground file worker"); + }) + .join() + .unwrap(); + assert_eq!(dst_groups.inner.lock().groups.len(), 2); + assert_eq!(file_groups.inner.lock().groups.len(), 2); + } + + #[tokio::test] + async fn isolated_runtime_keeps_fsync_on_its_own_blocking_pool() { + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(1) + .max_blocking_threads(1) + .thread_name("isolated-fsync-test") + .on_thread_start(use_current_runtime_for_fsync) + .enable_all() + .build() + .unwrap(); + let result = runtime + .spawn(async { + fsync_spawn_blocking(|| { + assert_eq!(std::thread::current().name(), Some("isolated-fsync-test")); + }) + .await + .unwrap(); + }) + .await; + // A runtime cannot be dropped from another runtime's async context. + runtime.shutdown_background(); + result.unwrap(); + assert!(!FSYNC_ON_CURRENT_RUNTIME.get(), "the calling runtime must retain normal fsync routing"); + } + fn file_sync_limiter() -> Arc { Arc::new(Semaphore::new(MAX_PARALLEL_FILE_SYNCS)) } @@ -4817,7 +5495,11 @@ mod tests { .first() .expect("test publication root requires at least one path") .to_path_buf(); - while !paths.iter().all(|path| path.starts_with(&common)) { + // The disk root must exist independently of a missing or symlinked + // rename base; those are inputs whose rejection the tests exercise. + while !paths.iter().all(|path| path.starts_with(&common)) + || !std::fs::symlink_metadata(&common).is_ok_and(|metadata| metadata.is_dir()) + { assert!(common.pop(), "test paths must share an absolute root"); } PublicationRoot::new(&common).expect("test publication root should open") @@ -8061,3 +8743,172 @@ mod tests { assert_eq!(err.kind(), io::ErrorKind::NotFound); } } + +#[cfg(all(test, target_os = "linux"))] +mod fd_relative_tests { + use super::*; + use std::io::{Read, Write}; + + #[tokio::test] + async fn guarded_first_publication_recovers_from_a_concurrent_parent_prune() { + let temp = tempfile::tempdir().expect("temporary disk"); + let root = PublicationRoot::new(temp.path()).expect("pinned root"); + let source = temp.path().join("tmp"); + let base = temp.path().join("bucket"); + let shared = base.join("prefix"); + let destination = shared.join("object"); + std::fs::create_dir(&source).expect("staging directory"); + std::fs::create_dir_all(&shared).expect("shared prefix"); + let guard = prepare_rename_commit_guard(&source, &destination, &base, &root).expect("commit directories"); + let src = source.join("xl.meta"); + let dst = destination.join("xl.meta"); + let mut staged = create_prepared_rename_source_with_commit_guard(&src, &dst, &guard).expect("staged metadata"); + staged.write_all(b"metadata", true).expect("write metadata"); + let pruned = shared.clone(); + prepare_rename_test_hooks::queue_after_component_opened(&shared, move || { + std::fs::remove_dir(pruned).expect("prune empty prefix"); + }); + let lease = acquire_namespace_mutation_lease(&dst).await; + rename_all_with_prepared_source(staged, &src, &dst, &base, &root, &guard, lease) + .await + .expect("publication must rebuild the pruned parent"); + assert_eq!(std::fs::read(dst).expect("published metadata"), b"metadata"); + assert!(!src.exists()); + } + + #[test] + fn legacy_relative_open_rejects_symlinks_and_opens_existing_components() { + use rustix::fs::{Mode, OFlags}; + let temp = tempfile::tempdir().expect("temporary disk"); + std::fs::create_dir(temp.path().join("parent")).expect("parent directory"); + std::fs::write(temp.path().join("parent/data"), b"data").expect("file data"); + std::os::unix::fs::symlink("parent", temp.path().join("link")).expect("directory link"); + std::os::unix::fs::symlink("data", temp.path().join("parent/link")).expect("file link"); + let root = PublicationRoot::new(temp.path()).expect("pinned root"); + let flags = OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC; + let mut bytes = Vec::new(); + open_linux_relative_legacy(root.directory(), Path::new("parent/data"), flags, Mode::empty()) + .expect("legacy open") + .read_to_end(&mut bytes) + .expect("legacy read"); + assert_eq!(bytes, b"data"); + for relative in ["link/data", "parent/link"] { + assert!(open_linux_relative_legacy(root.directory(), Path::new(relative), flags, Mode::empty()).is_err()); + } + } + + #[test] + fn root_relative_io_retains_mount_identity_after_path_replacement() { + let temp = tempfile::tempdir().expect("temporary disk"); + let disk = temp.path().join("disk"); + std::fs::create_dir(&disk).expect("disk root"); + std::fs::write(disk.join("data"), b"original").expect("original data"); + let root = PublicationRoot::new(&disk).expect("pinned root"); + let original = temp.path().join("original"); + std::fs::rename(&disk, &original).expect("move root"); + std::fs::create_dir(&disk).expect("replacement root"); + std::fs::write(disk.join("data"), b"replacement").expect("replacement data"); + for path in [disk.join("data"), root.io_path().join("data")] { + let mut actual = Vec::new(); + root.open_readonly(&path) + .expect("pinned read") + .read_to_end(&mut actual) + .expect("read bytes"); + assert_eq!(actual, b"original"); + } + let (mut file, parent) = root.create_truncate_with_parent(&disk.join("new")).expect("pinned creation"); + file.write_all(b"new").expect("write bytes"); + file.sync_data().expect("file sync"); + fsync_directory_handle_std(&disk, parent.as_ref()).expect("pinned parent sync"); + assert_eq!(std::fs::read(original.join("new")).expect("original disk file"), b"new"); + assert!(!disk.join("new").exists()); + } + + #[test] + fn root_relative_io_rejects_links_and_parent_escape_without_truncation() { + let temp = tempfile::tempdir().expect("temporary disk"); + let disk = temp.path().join("disk"); + std::fs::create_dir(&disk).expect("disk root"); + let outside = temp.path().join("outside"); + std::fs::write(&outside, b"preserved").expect("outside data"); + std::os::unix::fs::symlink(&outside, disk.join("link")).expect("file link"); + std::os::unix::fs::symlink(temp.path(), disk.join("parent")).expect("directory link"); + let root = PublicationRoot::new(&disk).expect("pinned root"); + for path in [disk.join("link"), disk.join("parent/outside"), disk.join("../outside")] { + assert!(root.open_readonly(&path).is_err(), "unsafe read: {path:?}"); + assert!(root.create_truncate(&path).is_err(), "unsafe write: {path:?}"); + } + assert_eq!(std::fs::read(&outside).expect("outside remains readable"), b"preserved"); + } + + #[tokio::test] + async fn guarded_publication_reuses_parents_and_syncs_without_changing_bytes() { + for grouped in [false, true] { + let _grouping = set_dst_dir_fsync_group_commit_for_test(grouped); + let temp = tempfile::tempdir().expect("temporary disk"); + let root = PublicationRoot::new(temp.path()).expect("pinned root"); + let source = temp.path().join("tmp"); + let destination = temp.path().join("bucket/object"); + let base = temp.path().join("bucket"); + std::fs::create_dir(&source).expect("staging directory"); + std::fs::create_dir(&base).expect("bucket directory"); + let guard = prepare_rename_commit_guard(&source, &destination, &base, &root).expect("commit directories"); + assert!(!destination.exists(), "preparation must not create the object directory"); + let src = source.join("xl.meta"); + let dst = destination.join("xl.meta"); + assert!( + read_destination_file_with_commit_guard(&dst, &guard) + .expect("absent read") + .is_none() + ); + let mut staged = create_prepared_rename_source_with_commit_guard(&src, &dst, &guard).expect("staged metadata"); + staged.write_all(b"metadata", true).expect("write metadata"); + let lease = acquire_namespace_mutation_lease(&dst).await; + rename_all_with_prepared_source(staged, &src, &dst, &base, &root, &guard, lease.clone()) + .await + .expect("publish"); + fsync_commit_directory(&destination, &guard, lease, None) + .await + .expect("directory sync"); + assert_eq!( + read_destination_file_with_commit_guard(&dst, &guard).expect("committed read"), + Some(b"metadata".to_vec()) + ); + assert!(!src.exists()); + assert_eq!(std::fs::read_dir(&destination).expect("object directory").count(), 1); + } + } + + #[tokio::test] + async fn guarded_publication_rejects_replaced_directory_and_source() { + for replace_directory in [false, true] { + let temp = tempfile::tempdir().expect("temporary disk"); + let root = PublicationRoot::new(temp.path()).expect("pinned root"); + let source = temp.path().join("tmp"); + let destination = temp.path().join("object"); + std::fs::create_dir(&source).expect("staging directory"); + std::fs::create_dir(&destination).expect("existing object directory"); + let guard = prepare_rename_commit_guard(&source, &destination, temp.path(), &root).expect("commit directories"); + let src = source.join("xl.meta"); + let dst = destination.join("xl.meta"); + std::fs::write(&dst, b"old").expect("old metadata"); + let mut staged = create_prepared_rename_source_with_commit_guard(&src, &dst, &guard).expect("staged metadata"); + staged.write_all(b"new", false).expect("new metadata"); + if replace_directory { + std::fs::rename(&destination, temp.path().join("detached")).expect("replace destination"); + std::fs::create_dir(&destination).expect("replacement directory"); + std::fs::write(&dst, b"old").expect("replacement metadata"); + } else { + std::fs::rename(&src, source.join("detached")).expect("replace staged file"); + std::fs::write(&src, b"unrelated").expect("replacement file"); + } + let lease = acquire_namespace_mutation_lease(&dst).await; + assert!( + rename_all_with_prepared_source(staged, &src, &dst, temp.path(), &root, &guard, lease) + .await + .is_err() + ); + assert_eq!(std::fs::read(&dst).expect("old metadata survives"), b"old"); + } + } +} diff --git a/crates/ecstore/src/error/conversion_roundtrip_tests.rs b/crates/ecstore/src/error/conversion_roundtrip_tests.rs index 9a01d16a2..9a1db49cb 100644 --- a/crates/ecstore/src/error/conversion_roundtrip_tests.rs +++ b/crates/ecstore/src/error/conversion_roundtrip_tests.rs @@ -34,6 +34,19 @@ use crate::disk::error_reduce::{OBJECT_OP_IGNORED_ERRS, reduce_errs}; use crate::error::StorageError; use rustfs_protos::proto_gen::node_service::Error as WireError; +#[test] +fn conditional_delete_errors_preserve_disk_storage_and_wire_identity() { + for disk_error in [DiskError::ConditionalDeleteConflict, DiskError::GuardedDeleteUnsupported] { + let storage: StorageError = disk_error.clone().into(); + let raw_code = storage.to_u32(); + assert_eq!(StorageError::from_u32(raw_code), Some(storage.clone())); + assert_eq!(storage.narrow_to_disk().expect("guard errors must narrow"), disk_error); + let wire: WireError = disk_error.clone().into(); + let decoded: DiskError = wire.into(); + assert_eq!(decoded, disk_error, "guard errors must survive peer RPC"); + } +} + /// The variants heal / replication / quorum repair match on, and that must /// survive the DiskError → StorageError → DiskError loop unchanged. fn heal_matched_disk_variants() -> Vec { diff --git a/crates/ecstore/src/error/mod.rs b/crates/ecstore/src/error/mod.rs index aba4fc411..bc1fe8226 100644 --- a/crates/ecstore/src/error/mod.rs +++ b/crates/ecstore/src/error/mod.rs @@ -290,6 +290,10 @@ pub enum StorageError { ConfigNotFound, #[error("Precondition failed")] PreconditionFailed, + #[error("Object generation changed during conditional deletion")] + ConditionalDeleteConflict, + #[error("The storage peer does not support guarded deletion")] + GuardedDeleteUnsupported, #[error("Not modified")] NotModified, #[error("Invalid range specified: {0}")] @@ -469,6 +473,8 @@ impl From for StorageError { DiskError::SourceStalled => StorageError::SourceStalled, DiskError::Timeout => StorageError::Timeout, DiskError::InvalidPath => StorageError::InvalidPath, + DiskError::ConditionalDeleteConflict => StorageError::ConditionalDeleteConflict, + DiskError::GuardedDeleteUnsupported => StorageError::GuardedDeleteUnsupported, DiskError::RemoteClientUnavailable(detail) => StorageError::RemoteClientUnavailable(detail), } } @@ -512,6 +518,8 @@ impl StorageError { StorageError::SourceStalled => DiskError::SourceStalled, StorageError::Timeout => DiskError::Timeout, StorageError::InvalidPath => DiskError::InvalidPath, + StorageError::ConditionalDeleteConflict => DiskError::ConditionalDeleteConflict, + StorageError::GuardedDeleteUnsupported => DiskError::GuardedDeleteUnsupported, StorageError::ErasureReadQuorum => DiskError::ErasureReadQuorum, StorageError::ErasureWriteQuorum => DiskError::ErasureWriteQuorum, StorageError::TooManyOpenFiles => DiskError::TooManyOpenFiles, @@ -758,6 +766,8 @@ impl Clone for StorageError { StorageError::InsufficientReadQuorum(a, b) => StorageError::InsufficientReadQuorum(a.clone(), b.clone()), StorageError::InsufficientWriteQuorum(a, b) => StorageError::InsufficientWriteQuorum(a.clone(), b.clone()), StorageError::PreconditionFailed => StorageError::PreconditionFailed, + StorageError::ConditionalDeleteConflict => StorageError::ConditionalDeleteConflict, + StorageError::GuardedDeleteUnsupported => StorageError::GuardedDeleteUnsupported, StorageError::NotModified => StorageError::NotModified, StorageError::InvalidPartNumber(a) => StorageError::InvalidPartNumber(*a), StorageError::InvalidRangeSpec(a) => StorageError::InvalidRangeSpec(a.clone()), @@ -871,6 +881,8 @@ impl StorageError { StorageError::InsufficientReadQuorum(_, _) => StorageErrorCode::InsufficientReadQuorum, StorageError::InsufficientWriteQuorum(_, _) => StorageErrorCode::InsufficientWriteQuorum, StorageError::PreconditionFailed => StorageErrorCode::PreconditionFailed, + StorageError::ConditionalDeleteConflict => StorageErrorCode::ConditionalDeleteConflict, + StorageError::GuardedDeleteUnsupported => StorageErrorCode::GuardedDeleteUnsupported, StorageError::EntityTooSmall(_, _, _) => StorageErrorCode::EntityTooSmall, StorageError::EntityTooLarge(_, _) => StorageErrorCode::EntityTooLarge, StorageError::InvalidRangeSpec(_) => StorageErrorCode::InvalidRangeSpec, @@ -992,6 +1004,8 @@ impl StorageError { Some(StorageError::InsufficientWriteQuorum(Default::default(), Default::default())) } StorageErrorCode::PreconditionFailed => Some(StorageError::PreconditionFailed), + StorageErrorCode::ConditionalDeleteConflict => Some(StorageError::ConditionalDeleteConflict), + StorageErrorCode::GuardedDeleteUnsupported => Some(StorageError::GuardedDeleteUnsupported), StorageErrorCode::EntityTooSmall => { Some(StorageError::EntityTooSmall(Default::default(), Default::default(), Default::default())) } diff --git a/crates/ecstore/src/io_support/bitrot.rs b/crates/ecstore/src/io_support/bitrot.rs index 9d4615f50..081dc963b 100644 --- a/crates/ecstore/src/io_support/bitrot.rs +++ b/crates/ecstore/src/io_support/bitrot.rs @@ -174,10 +174,14 @@ async fn open_reader_source( metrics_path: Option<&'static str>, ) -> disk::error::Result> { if let Some(data) = inline_data { + #[cfg(test)] + crate::set_disk::disk_call_counters::record_shard_open(path, true); let mut reader = Cursor::new(data); reader.set_position(u64::try_from(offset).map_err(|_| DiskError::FileCorrupt)?); Ok(Some(ShardReader::InMemory(reader))) } else if let Some(disk) = disk { + #[cfg(test)] + crate::set_disk::disk_call_counters::record_shard_open(path, false); open_disk_reader(disk, bucket, path, offset, length, use_mmap_read, metrics_path) .await .map(Some) diff --git a/crates/ecstore/src/layout/endpoint.rs b/crates/ecstore/src/layout/endpoint.rs index 5e9ca0c58..f93e68808 100644 --- a/crates/ecstore/src/layout/endpoint.rs +++ b/crates/ecstore/src/layout/endpoint.rs @@ -101,23 +101,30 @@ impl TryFrom<&str> for Endpoint { let path = Path::new(&path).absolutize()?; #[cfg(windows)] - let path = if has_leading_slash_windows_drive(&path) { + let path = match strip_windows_drive_url_path(&path) { // Url::path() exposes file-like Windows paths as `/C:/...`. // Strip only that synthetic leading slash; plain URL paths // such as `/export1` must stay URL paths, not become // relative paths under the current drive. - Path::new(&path[1..]).absolutize()?.to_string_lossy().into_owned() - } else { - path + Some(drive_path) => Path::new(drive_path).absolutize()?.to_string_lossy().into_owned(), + None => path, }; #[cfg(windows)] let path = Path::new(&path); debug!("endpoint try_from: path={}", path.display()); - if path.parent().is_none() || path.as_os_str().is_empty() { + // A Windows drive root such as `C:\` has no parent, but it is + // accepted for parity with the path-style `C:\` endpoint. + #[cfg(windows)] + let is_root = path.parent().is_none() && !is_windows_drive_root(path); + #[cfg(not(windows))] + let is_root = path.parent().is_none(); + if is_root || path.as_os_str().is_empty() { return Err(Error::other("empty or root path is not supported in URL endpoint")); } + #[cfg(windows)] + reject_windows_unc_path(path)?; match path.to_str() { Some(v) => url.set_path(v), @@ -230,20 +237,63 @@ impl Endpoint { std::borrow::Cow::Borrowed(path) } }; + // Url::path() always starts with `/`, so every scheme stores a Windows + // drive path as `/C:/...`. Restore the drive path for `file` and + // `http(s)` endpoints alike; Windows would otherwise resolve `/C:/data` + // as an invalid `C:` component under the current drive. #[cfg(windows)] - if self.url.scheme() == "file" { - let stripped: &str = decoded.strip_prefix('/').unwrap_or(&decoded); - debug!("get_file_path windows: path={}", stripped); - return stripped.to_string(); + if let Some(drive_path) = strip_windows_drive_url_path(&decoded) { + debug!("get_file_path windows: path={}", drive_path); + return drive_path.to_string(); } decoded.into_owned() } } -#[cfg(windows)] -fn has_leading_slash_windows_drive(path: &str) -> bool { +/// Returns `C:/...` for a URL path shaped like `/C:/...`, otherwise `None`. +/// +/// The separator after the colon is required: `/C:foo` is a plain URL path, +/// and `/export1` must never be reinterpreted as a drive path. +#[cfg_attr(not(windows), allow(dead_code))] +fn strip_windows_drive_url_path(path: &str) -> Option<&str> { let bytes = path.as_bytes(); - bytes.len() >= 4 && bytes[0] == b'/' && bytes[1].is_ascii_alphabetic() && bytes[2] == b':' && bytes[3] == b'/' + if bytes.len() >= 4 && bytes[0] == b'/' && bytes[1].is_ascii_alphabetic() && bytes[2] == b':' && bytes[3] == b'/' { + Some(&path[1..]) + } else { + None + } +} + +#[cfg(windows)] +fn is_windows_drive_root(path: &Path) -> bool { + use std::path::{Component, Prefix}; + + let mut components = path.components(); + matches!( + components.next(), + Some(Component::Prefix(prefix)) if matches!(prefix.kind(), Prefix::Disk(_) | Prefix::VerbatimDisk(_)) + ) && matches!(components.next(), Some(Component::RootDir)) + && components.next().is_none() +} + +/// UNC shares are rejected instead of supported: `Url::from_file_path` turns +/// `\\nas\share\d1` into `file://nas/share/d1`, so the share server would be +/// read as the endpoint host and the share name dropped from the drive path, +/// and a network share does not provide the local-volume semantics (atomic +/// rename, fsync, volume identity) that erasure-coded drives rely on. +#[cfg(windows)] +fn reject_windows_unc_path(path: &Path) -> Result<()> { + use std::path::{Component, Prefix}; + + if let Some(Component::Prefix(prefix)) = path.components().next() + && matches!(prefix.kind(), Prefix::UNC(..) | Prefix::VerbatimUNC(..)) + { + return Err(Error::other( + "UNC network share paths are not supported as drive endpoints; use a local volume path such as \ +D:\\rustfs\\data1, or http://host:9000/D:/rustfs/data1 for URL endpoints", + )); + } + Ok(()) } /// parse a file path into a URL. @@ -261,6 +311,9 @@ fn url_parse_from_file_path(value: &str) -> Result { Err(err) => return Err(Error::other(format!("absolute path failed: {err}"))), }; + #[cfg(windows)] + reject_windows_unc_path(&file_path)?; + match Url::from_file_path(file_path) { Ok(url) => Ok(url), Err(_) => Err(Error::other("Convert a file path into an URL failed")), @@ -538,17 +591,110 @@ mod test { assert_eq!(url_endpoint.get_file_path(), "/path/to/data"); } + #[test] + fn strip_windows_drive_url_path_requires_drive_separator() { + assert_eq!(strip_windows_drive_url_path("/C:/data1"), Some("C:/data1")); + assert_eq!(strip_windows_drive_url_path("/d:/rustfs/data1"), Some("d:/rustfs/data1")); + assert_eq!(strip_windows_drive_url_path("/C:/"), Some("C:/")); + assert_eq!(strip_windows_drive_url_path("/C:foo"), None); + assert_eq!(strip_windows_drive_url_path("/C:"), None); + assert_eq!(strip_windows_drive_url_path("/export1"), None); + assert_eq!(strip_windows_drive_url_path("/1:/data"), None); + assert_eq!(strip_windows_drive_url_path("C:/data1"), None); + } + + #[cfg(not(windows))] + #[test] + fn url_endpoint_drive_shaped_path_stays_a_url_path_off_windows() { + let endpoint = Endpoint::try_from("http://host:9000/C:/data1").unwrap(); + assert_eq!(endpoint.get_file_path(), "/C:/data1"); + } + #[cfg(windows)] #[test] - fn test_windows_url_drive_path_requires_separator_after_colon() { - let drive_path_endpoint = Endpoint::try_from("http://host/C:/data").unwrap(); - assert_eq!(drive_path_endpoint.get_type(), EndpointType::Url); - assert!(has_leading_slash_windows_drive(Url::parse("http://host/C:/data").unwrap().path())); + fn windows_url_endpoint_file_path_restores_drive_path() { + for (arg, expected) in [ + ("http://host:9000/C:/data1", "C:/data1"), + ("https://host:9000/d:/rustfs/data4", "d:/rustfs/data4"), + ("http://host:9000/C:\\data1", "C:/data1"), + ("http://host:9000/C:/rustfs/../data1", "C:/data1"), + ("http://host:9000/C:/Program%20Files/rustfs", "C:/Program Files/rustfs"), + ] { + let endpoint = Endpoint::try_from(arg).unwrap_or_else(|err| panic!("{arg}: {err}")); + assert_eq!(endpoint.get_type(), EndpointType::Url, "{arg}"); + assert_eq!(endpoint.get_file_path(), expected, "{arg}"); + assert!(Path::new(&endpoint.get_file_path()).is_absolute(), "{arg}"); + } + } - let url_path_endpoint = Endpoint::try_from("http://host/C:foo").unwrap(); - assert_eq!(url_path_endpoint.get_type(), EndpointType::Url); - assert!(!has_leading_slash_windows_drive(Url::parse("http://host/C:foo").unwrap().path())); - assert_eq!(url_path_endpoint.get_file_path(), "/C:foo"); + #[cfg(windows)] + #[test] + fn windows_url_endpoint_keeps_url_identity_after_drive_path_parse() { + let endpoint = Endpoint::try_from("http://host:9000/C:/data1").unwrap(); + assert_eq!(endpoint.url.as_str(), "http://host:9000/C:/data1"); + assert_eq!(endpoint.to_string(), "http://host:9000/C:/data1"); + assert_eq!(endpoint.host_port(), "host:9000"); + } + + #[cfg(windows)] + #[test] + fn windows_url_endpoint_keeps_plain_url_paths() { + for (arg, expected) in [("http://host:9000/export1", "/export1"), ("http://host:9000/C:foo", "/C:foo")] { + let endpoint = Endpoint::try_from(arg).unwrap(); + assert_eq!(endpoint.get_type(), EndpointType::Url, "{arg}"); + assert_eq!(endpoint.get_file_path(), expected, "{arg}"); + } + } + + #[cfg(windows)] + #[test] + fn windows_url_and_path_endpoints_share_drive_file_path() { + let path_endpoint = Endpoint::try_from("C:\\rustfs\\data1").unwrap(); + let url_endpoint = Endpoint::try_from("http://host:9000/C:/rustfs/data1").unwrap(); + let verbatim_endpoint = Endpoint::try_from("\\\\?\\C:\\rustfs\\data1").unwrap(); + + assert_eq!(path_endpoint.get_type(), EndpointType::Path); + assert_eq!(path_endpoint.get_file_path(), "C:/rustfs/data1"); + assert_eq!(url_endpoint.get_file_path(), path_endpoint.get_file_path()); + assert_eq!(verbatim_endpoint.get_file_path(), path_endpoint.get_file_path()); + } + + #[cfg(windows)] + #[test] + fn windows_url_endpoint_resolves_existing_drive_directory() { + let dir = tempfile::tempdir().unwrap(); + let drive_path = dir.path().to_string_lossy().replace('\\', "/"); + let endpoint = Endpoint::try_from(format!("http://127.0.0.1:9000/{drive_path}").as_str()).unwrap(); + + let file_path = endpoint.get_file_path(); + assert_eq!(Path::new(&file_path), dir.path()); + rustfs_utils::canonicalize(&file_path) + .unwrap_or_else(|err| panic!("URL endpoint path '{file_path}' must resolve to the temp directory: {err}")); + } + + #[cfg(windows)] + #[test] + fn windows_drive_root_endpoints_are_accepted_in_both_forms() { + let path_endpoint = Endpoint::try_from("D:\\").unwrap(); + assert_eq!(path_endpoint.get_file_path(), "D:/"); + + let url_endpoint = Endpoint::try_from("http://host:9000/D:/").unwrap(); + assert_eq!(url_endpoint.get_type(), EndpointType::Url); + assert_eq!(url_endpoint.get_file_path(), "D:/"); + } + + #[cfg(windows)] + #[test] + fn windows_unc_endpoints_are_rejected() { + for arg in [ + "\\\\nas\\share\\d1", + "//nas/share/d1", + "\\\\?\\UNC\\nas\\share\\d1", + "http://host:9000//nas/share/d1", + ] { + let err = Endpoint::try_from(arg).expect_err(arg); + assert!(err.to_string().contains("UNC network share paths are not supported"), "{arg}: {err}"); + } } #[test] diff --git a/crates/ecstore/src/layout/endpoints.rs b/crates/ecstore/src/layout/endpoints.rs index d6f17d9b3..95abdc22a 100644 --- a/crates/ecstore/src/layout/endpoints.rs +++ b/crates/ecstore/src/layout/endpoints.rs @@ -1729,11 +1729,21 @@ fn validate_local_physical_disk_independence(pools: &[Endpoints]) -> Result<()> Err(Error::other(format!( "local erasure endpoints must use distinct physical disks; detected shared devices [{details}]. \ -validation diagnostics: [{diagnostics_summary}]. \ +validation diagnostics: [{diagnostics_summary}]. {SHARED_DEVICE_PLATFORM_HINT}\ Set {ENV_UNSAFE_BYPASS_DISK_CHECK}=true only for local testing or CI to bypass this safety check" ))) } +/// Windows reports the volume mount point (for example `C:\`) as the device +/// identity, so every directory on one drive letter or mounted volume counts +/// as the same physical disk. +#[cfg(windows)] +const SHARED_DEVICE_PLATFORM_HINT: &str = "On Windows the device identity is the volume mount point, so directories on the same \ +drive letter or mounted volume (for example C:\\data1 and C:\\data2) count as one disk; place each endpoint on its own \ +volume. "; +#[cfg(not(windows))] +const SHARED_DEVICE_PLATFORM_HINT: &str = ""; + fn validate_local_cross_device_mounts(local_paths: &[String]) -> Result<()> { rustfs_utils::os::check_cross_device_mounts(local_paths) .map_err(|err| Error::other(format!("local endpoint cross-device mount validation failed: {err}"))) @@ -3581,6 +3591,38 @@ mod test { } } + #[cfg(windows)] + #[serial] + #[tokio::test] + async fn windows_url_endpoints_on_one_volume_resolve_and_report_shared_volume() { + async_with_vars([(ENV_UNSAFE_BYPASS_DISK_CHECK, None::<&str>), (ENV_MINIO_CI, None::<&str>)], async { + let dir = tempfile::tempdir().unwrap(); + let mut endpoints = Vec::new(); + for name in ["disk1", "disk2"] { + let disk = dir.path().join(name); + std::fs::create_dir_all(&disk).unwrap(); + let drive_path = disk.to_string_lossy().replace('\\', "/"); + let mut endpoint = Endpoint::try_from(format!("http://127.0.0.1:9000/{drive_path}").as_str()).unwrap(); + endpoint.is_local = true; + endpoints.push(endpoint); + } + + let err = validate_local_physical_disk_independence(&[Endpoints::from(endpoints)]).unwrap_err(); + + // Before the drive-path fix, `/C:/...` failed canonicalization here + // with "failed to resolve local endpoint path". + let err_text = err.to_string(); + assert!( + !err_text.contains("failed to resolve local endpoint path"), + "unexpected error: {err_text}" + ); + assert!(err_text.contains("distinct physical disks"), "unexpected error: {err_text}"); + assert!(err_text.contains("same drive letter or mounted volume"), "unexpected error: {err_text}"); + assert!(err_text.contains(ENV_UNSAFE_BYPASS_DISK_CHECK), "unexpected error: {err_text}"); + }) + .await; + } + #[cfg(target_os = "linux")] #[serial] #[tokio::test] diff --git a/crates/ecstore/src/object_api/mod.rs b/crates/ecstore/src/object_api/mod.rs index 008005547..75b9db65c 100644 --- a/crates/ecstore/src/object_api/mod.rs +++ b/crates/ecstore/src/object_api/mod.rs @@ -102,10 +102,14 @@ mod body_cache_hook; mod encryption; mod hook_slot; mod object_mutation_hook; +mod read_condition; mod readers; mod types; pub(crate) mod write_commit_context; +pub use read_condition::{GetObjectReadCondition, GetObjectReadMetadata, with_get_object_read_condition}; +pub(crate) use read_condition::{get_object_read_condition_is_active, get_object_read_condition_is_terminal}; + #[cfg(test)] mod persisted_metadata_keys_tests; diff --git a/crates/ecstore/src/object_api/read_condition.rs b/crates/ecstore/src/object_api/read_condition.rs new file mode 100644 index 000000000..18754e55c --- /dev/null +++ b/crates/ecstore/src/object_api/read_condition.rs @@ -0,0 +1,113 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use rustfs_filemeta::{FileInfo, ObjectPartInfo}; +use std::future::Future; +use std::sync::Arc; +use time::OffsetDateTime; + +/// Borrowed fields needed to decide whether an object read needs a body. +pub struct GetObjectReadMetadata<'a> { + pub bucket: &'a str, + pub object: &'a str, + pub etag: Option<&'a str>, + pub mod_time: Option, + pub parts: &'a [ObjectPartInfo], + pub delete_marker: bool, +} + +impl<'a> GetObjectReadMetadata<'a> { + pub(crate) fn from_file_info(bucket: &'a str, object: &'a str, info: &'a FileInfo) -> Self { + Self { + bucket, + object, + etag: info.metadata.get("etag").map(String::as_str), + mod_time: info.mod_time, + parts: &info.parts, + delete_marker: info.deleted, + } + } +} + +/// A pure decision on quorum-validated metadata. Authorization belongs to the caller. +pub trait GetObjectReadCondition: Send + Sync { + fn is_terminal(&self, metadata: GetObjectReadMetadata<'_>) -> bool; +} + +tokio::task_local! { + static READ_CONDITION: Arc; +} + +/// Applies a condition to this reader call, without inheriting it into shared producers. +pub async fn with_get_object_read_condition(condition: Arc, read: F) -> F::Output { + READ_CONDITION.scope(condition, read).await +} + +pub(crate) fn get_object_read_condition_is_terminal(metadata: GetObjectReadMetadata<'_>) -> bool { + READ_CONDITION + .try_with(|condition| condition.is_terminal(metadata)) + .unwrap_or(false) +} + +pub(crate) fn get_object_read_condition_is_active() -> bool { + READ_CONDITION.try_with(|_| ()).is_ok() +} + +#[cfg(test)] +mod tests { + use super::*; + + struct Stop; + impl GetObjectReadCondition for Stop { + fn is_terminal(&self, _: GetObjectReadMetadata<'_>) -> bool { + true + } + } + fn metadata() -> GetObjectReadMetadata<'static> { + GetObjectReadMetadata { + bucket: "bucket", + object: "object", + etag: Some("etag"), + mod_time: None, + parts: &[], + delete_marker: false, + } + } + + #[tokio::test] + async fn read_condition_scope_does_not_escape_or_enter_shared_producers() { + assert!(!get_object_read_condition_is_terminal(metadata())); + with_get_object_read_condition(Arc::new(Stop), async { + assert!(get_object_read_condition_is_terminal(metadata())); + assert!( + !tokio::spawn(async { get_object_read_condition_is_terminal(metadata()) }) + .await + .expect("shared producer") + ); + }) + .await; + assert!(!get_object_read_condition_is_terminal(metadata())); + } + + #[tokio::test] + async fn dropping_a_pending_read_removes_its_condition_scope() { + let mut read = Box::pin(with_get_object_read_condition(Arc::new(Stop), async { + assert!(get_object_read_condition_is_terminal(metadata())); + std::future::pending::<()>().await; + })); + assert!(futures::poll!(&mut read).is_pending()); + drop(read); + assert!(!get_object_read_condition_is_terminal(metadata())); + } +} diff --git a/crates/ecstore/src/object_api/readers.rs b/crates/ecstore/src/object_api/readers.rs index 1ee72b227..67154621d 100644 --- a/crates/ecstore/src/object_api/readers.rs +++ b/crates/ecstore/src/object_api/readers.rs @@ -528,6 +528,14 @@ pub struct GetObjectReader { } impl GetObjectReader { + pub(crate) fn condition_stopped(object_info: ObjectInfo) -> Self { + Self { + stream: Box::new(ConditionStoppedBody), + object_info, + buffered_body: None, + body_source: GetObjectBodySource::Unprobed, + } + } /// Builds a fully materialized reader from a cache-coordinated body. pub fn from_cache_body(mut object_info: ObjectInfo, body: Bytes) -> Result { object_info.size = i64::try_from(body.len()).map_err(|_| Error::other("cached GET body length exceeds i64::MAX"))?; @@ -552,6 +560,17 @@ impl GetObjectReader { } } +struct ConditionStoppedBody; + +impl AsyncRead for ConditionStoppedBody { + fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll> { + if buf.remaining() == 0 { + return Poll::Ready(Ok(())); + } + Poll::Ready(Err(std::io::Error::other("object read condition must be handled before body access"))) + } +} + enum ReadTransform { // Written but never read by production code: the enclosing struct already // carries the same pair as `storage_offset`/`storage_length`. They survive diff --git a/crates/ecstore/src/object_api/types.rs b/crates/ecstore/src/object_api/types.rs index ea98f5dcc..dfd37cc35 100644 --- a/crates/ecstore/src/object_api/types.rs +++ b/crates/ecstore/src/object_api/types.rs @@ -152,7 +152,7 @@ pub struct ObjectLockConfigSnapshot { config_revision: Option, state: crate::bucket::metadata_sys::ObjectLockConfigState, lifecycle_fence: NamespaceLockFence, - _lifecycle_guard: Option, + _lifecycle_guard: Option>, metadata_transaction_guard: Option>, } @@ -210,11 +210,36 @@ impl ObjectLockConfigSnapshot { config_revision: Some(config_revision), state, lifecycle_fence, - _lifecycle_guard: Some(lifecycle_guard), + _lifecycle_guard: Some(Arc::new(lifecycle_guard)), metadata_transaction_guard: Some(Arc::new(metadata_transaction_guard)), } } + pub(crate) fn for_store_bucket_under_lifecycle_owner( + store_id: Uuid, + bucket: &str, + bucket_incarnation_id: Uuid, + config_revision: OffsetDateTime, + state: crate::bucket::metadata_sys::ObjectLockConfigState, + lifecycle_guard: Arc, + metadata_transaction_guard: Arc, + ) -> Self { + let mut lifecycle_fence = NamespaceLockFence::new(); + if let Some(signal) = lifecycle_guard.lock_lost_signal() { + lifecycle_fence.add_signal(signal); + } + Self { + store_id: Some(store_id), + bucket: Some(bucket.to_owned()), + bucket_incarnation_id: Some(bucket_incarnation_id), + config_revision: Some(config_revision), + state, + lifecycle_fence, + _lifecycle_guard: Some(lifecycle_guard), + metadata_transaction_guard: Some(metadata_transaction_guard), + } + } + pub(crate) fn for_store_bucket_under_lifecycle_fence( store_id: Uuid, bucket: &str, @@ -912,6 +937,10 @@ pub enum WriteCompletion { /// Drain the rename fan-out before returning. Minority failures still heal /// after a successful quorum commit; this does not require every disk to succeed. TailDrained, + /// Drain the rename fan-out and return system-metadata repair to the + /// caller's existing deployment/transaction owner. This does not grant + /// write or heal authority and does not change the quorum requirement. + TailDrainedWithOwnerRepair, } /// Storage-owned write mode inherited by physical rewrites. This selects the @@ -936,6 +965,10 @@ pub struct ObjectOptions { /// Lifecycle-only staged purge request checked under the object write lock. #[doc(hidden)] pub lifecycle_delete_all: Option, + /// Policy observed by lifecycle evaluation, checked under the metadata + /// transaction lock before logical deletion. Not persisted or sent over RPC. + #[doc(hidden)] + pub lifecycle_expiry_policy: Option, #[doc(hidden)] pub lifecycle_delete_all_journal: Option>>, /// Whole-operation authorization created only by consuming a validated @@ -955,6 +988,15 @@ pub struct ObjectOptions { /// Ordinary ECStore PUT target bound to its bucket's immutable artifact namespace. #[doc(hidden)] pub artifact_put_target: Option>, + /// In-process delete ownership for late filesystem completion. Never + /// serialized or copied into object metadata. + #[doc(hidden)] + pub delete_mutation_owner: Option>, + /// Pool-local generations for admitted current-object conditional actions. + /// Ordinary actions execute in a separate phase without this transient map. + /// This must never be populated from client metadata. + #[doc(hidden)] + pub conditional_delete_generations: Option>>, /// Internal read-only inspection must not enqueue metadata or payload repairs. pub suppress_read_repair: bool, /// Control-plane writers that immediately read or CAS the same namespace @@ -1022,6 +1064,9 @@ pub struct ObjectOptions { /// Actual generation owner retained by storage commits and their physical tails. #[doc(hidden)] pub bucket_incarnation_fence: Option>, + /// Retains the actual lifecycle guard for an owned multipart commit. A + /// lock-loss fence alone does not keep a cancelled caller's lock alive. + pub bucket_lifecycle_lock_owner: Option>, pub replication_request: bool, /// True when the inbound request carried the /// `{x-rustfs-,x-minio-}source-proxy-request` header family with the @@ -1178,8 +1223,11 @@ impl std::fmt::Debug for ObjectOptions { .field("http_preconditions", &self.http_preconditions.is_some()) .field("delete_replication", &self.delete_replication.is_some()) .field("delete_replication_config_snapshot", &self.delete_replication_config_snapshot) + .field("delete_mutation_owner", &self.delete_mutation_owner.is_some()) + .field("conditional_delete_generations", &self.conditional_delete_generations.is_some()) .field("namespace_lock_fence", &self.namespace_lock_fence.is_some()) .field("bucket_lifecycle_lock_fence", &self.bucket_lifecycle_lock_fence.is_some()) + .field("bucket_lifecycle_lock_owner", &self.bucket_lifecycle_lock_owner.is_some()) .field("replication_request", &self.replication_request) .field("proxy_request", &self.proxy_request) .field("proxy_header_set", &self.proxy_header_set) diff --git a/crates/ecstore/src/object_api/write_commit_context.rs b/crates/ecstore/src/object_api/write_commit_context.rs index 79edbec09..fb3ec3be2 100644 --- a/crates/ecstore/src/object_api/write_commit_context.rs +++ b/crates/ecstore/src/object_api/write_commit_context.rs @@ -14,6 +14,7 @@ use super::{ObjectOptions, WriteCompletion}; use crate::error::{Error, Result}; +use rustfs_utils::path::object_name_disk_identity; use std::sync::Arc; use std::time::Duration; @@ -53,7 +54,7 @@ impl WriteCommitGuard { pub(crate) fn from_guard(guard: Arc, bucket: &str, object: &str) -> Self { Self { bucket: bucket.into(), - object: object.into(), + object: object_name_disk_identity(object).as_ref().into(), guards: Arc::new(vec![guard]), } } @@ -68,8 +69,10 @@ impl WriteCommitGuard { self.guards.iter().any(|guard| guard.is_lock_lost() || guard.is_released()) } + /// Namespace locks key the disk identity of the name (see `new_ns_lock`), + /// so a guard proves every spelling that resolves to the same file. pub(crate) fn matches(&self, bucket: &str, object: &str) -> bool { - self.bucket.as_ref() == bucket && self.object.as_ref() == object + self.bucket.as_ref() == bucket && self.object.as_ref() == object_name_disk_identity(object).as_ref() } } diff --git a/crates/ecstore/src/runtime/instance.rs b/crates/ecstore/src/runtime/instance.rs index 93ccdcbd3..879708402 100644 --- a/crates/ecstore/src/runtime/instance.rs +++ b/crates/ecstore/src/runtime/instance.rs @@ -49,6 +49,7 @@ use crate::layout::endpoints::{EndpointServerPools, SetupType}; use crate::object_api::ObjectEncryptionResolver; use crate::services::event_notification::EventNotifier; use crate::services::tier::tier::TierConfigMgr; +use crate::store::delete_recovery_runtime::DeleteRecoveryHints; use rustfs_lock::{GlobalLockManager, get_global_lock_manager}; use s3s::region::Region; use std::collections::{HashMap, HashSet}; @@ -240,6 +241,7 @@ pub struct InstanceContext { namespace_commit_generation: AtomicU64, /// Resolves object-encryption material at the application boundary. object_encryption_resolver: OnceLock>, + delete_recovery_hints: DeleteRecoveryHints, tier_delete_journal_recovery_stores: std::sync::Mutex>, #[cfg(test)] suppress_tier_delete_journal_recovery: bool, @@ -294,6 +296,7 @@ impl InstanceContext { namespace_commits: AtomicU64::new(0), namespace_commit_generation: AtomicU64::new(0), object_encryption_resolver: OnceLock::new(), + delete_recovery_hints: DeleteRecoveryHints::default(), tier_delete_journal_recovery_stores: std::sync::Mutex::new(HashSet::new()), #[cfg(test)] suppress_tier_delete_journal_recovery: false, @@ -736,6 +739,10 @@ impl InstanceContext { self.background_cancel_token.get_or_init(|| token).clone() } + pub(crate) fn delete_recovery_hints(&self) -> &DeleteRecoveryHints { + &self.delete_recovery_hints + } + pub(crate) fn mark_tier_delete_journal_recovery_started(&self, store_id: Uuid) -> bool { #[cfg(test)] if self.suppress_tier_delete_journal_recovery { diff --git a/crates/ecstore/src/runtime/sources.rs b/crates/ecstore/src/runtime/sources.rs index f121cffe5..b7db108bf 100644 --- a/crates/ecstore/src/runtime/sources.rs +++ b/crates/ecstore/src/runtime/sources.rs @@ -50,6 +50,7 @@ use s3s::dto::BucketLifecycleConfiguration; use s3s::region::Region; use tokio::sync::{OwnedRwLockReadGuard, RwLock}; use tokio_util::sync::CancellationToken; +#[cfg(test)] use tonic::transport::Channel; use uuid::Uuid; @@ -251,10 +252,6 @@ pub(crate) async fn root_disk_threshold_for_erasure_disk() -> Option { } } -pub(crate) async fn cached_node_channel(addr: &str) -> Option { - rustfs_common::cached_connection(addr).await -} - #[cfg(test)] pub(crate) async fn cache_test_node_channel(addr: String, channel: Channel) { rustfs_common::cache_connection(addr, channel).await; diff --git a/crates/ecstore/src/services/metrics_realtime.rs b/crates/ecstore/src/services/metrics_realtime.rs index 41d2843c3..532f8f53c 100644 --- a/crates/ecstore/src/services/metrics_realtime.rs +++ b/crates/ecstore/src/services/metrics_realtime.rs @@ -231,6 +231,9 @@ fn to_madmin_scanner_metrics(metrics: rustfs_scanner_metrics::metrics::ScannerMe deferred_total: metrics.usage_freshness.deferred_total, last_deferred_unix_secs: metrics.usage_freshness.last_deferred_unix_secs, last_deferred_reason: metrics.usage_freshness.last_deferred_reason, + guardrail_veto_total: metrics.usage_freshness.guardrail_veto_total, + last_guardrail_veto_unix_secs: metrics.usage_freshness.last_guardrail_veto_unix_secs, + last_guardrail_veto_reason: metrics.usage_freshness.last_guardrail_veto_reason, }, maintenance_control: MadminScannerMaintenanceControlSnapshot { primary_control: metrics.maintenance_control.primary_control, @@ -886,10 +889,16 @@ mod test { deferred_total: 15, last_deferred_unix_secs: 16, last_deferred_reason: "data_movement".to_string(), + guardrail_veto_total: 4, + last_guardrail_veto_unix_secs: 17, + last_guardrail_veto_reason: "publication_blocked@".to_string(), }, ..Default::default() }); + assert_eq!(scanner.usage_freshness.guardrail_veto_total, 4); + assert_eq!(scanner.usage_freshness.last_guardrail_veto_unix_secs, 17); + assert_eq!(scanner.usage_freshness.last_guardrail_veto_reason, "publication_blocked@"); assert_eq!(scanner.usage_freshness.dirty_pending_buckets, 3); assert_eq!(scanner.usage_freshness.last_dirty_mark_unix_secs, 10); assert_eq!(scanner.usage_freshness.last_dirty_clear_unix_secs, 11); diff --git a/crates/ecstore/src/services/notification_sys.rs b/crates/ecstore/src/services/notification_sys.rs index 655caa47e..7db0ccba6 100644 --- a/crates/ecstore/src/services/notification_sys.rs +++ b/crates/ecstore/src/services/notification_sys.rs @@ -157,6 +157,18 @@ struct PeerAdminCache { last_server_success: Option, } +fn scanner_publication_lease_acquisition_error(host: &str, observational_only: bool, error: Error) -> Error { + let purpose = if observational_only { + "observational" + } else { + "authoritative" + }; + // Retry classification depends on the original error suffix. + Error::other(format!( + "scanner publication lease acquisition failed: peer={host}, purpose={purpose}: {error}" + )) +} + #[derive(Default)] struct TierConfigReloadWorkers { peers: HashMap, @@ -546,6 +558,17 @@ pub fn acquire_cross_pool_fence_fleet_proof() -> Option Option<&'static str> { + let state = cross_pool_fence_fleet_proof_slot() + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if state.topology_conflict { + return None; + } + REMOTE_VERSION_STATE_PROBE_TOPOLOGY.get().map(String::as_str) +} + pub fn cross_pool_fence_fleet_proof_matches(proof: &CrossPoolFenceFleetProofToken) -> bool { fleet_capability_proof_matches(cross_pool_fence_fleet_proof_slot(), &proof.0) } @@ -970,8 +993,8 @@ pub(crate) fn install_cross_pool_fence_fleet_proof_for_test() { } } -#[cfg(test)] -pub(crate) struct CrossPoolFenceFleetProofGuard { +#[cfg(any(test, feature = "test-util"))] +pub struct CrossPoolFenceFleetProofGuard { previous_proof: Option, previous_topology_conflict: bool, previous_journal_proof: Option, @@ -980,7 +1003,7 @@ pub(crate) struct CrossPoolFenceFleetProofGuard { previous_decommission_topology_conflict: bool, } -#[cfg(test)] +#[cfg(any(test, feature = "test-util"))] impl Drop for CrossPoolFenceFleetProofGuard { fn drop(&mut self) { let mut state = cross_pool_fence_fleet_proof_slot() @@ -1020,8 +1043,19 @@ impl Drop for CrossPoolFenceFleetProofGuard { /// Temporarily revoke the test proof so activation paths can exercise their /// fail-closed behavior without changing the process-wide topology binding. +#[cfg(any(test, feature = "test-util"))] +pub fn without_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceFleetProofGuard { + revoke_cross_pool_fence_fleet_proof_for_test(true) +} + +/// Simulate peer unavailability while retaining the configured topology identity. #[cfg(test)] -pub(crate) fn without_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceFleetProofGuard { +pub(crate) fn without_live_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceFleetProofGuard { + revoke_cross_pool_fence_fleet_proof_for_test(false) +} + +#[cfg(any(test, feature = "test-util"))] +fn revoke_cross_pool_fence_fleet_proof_for_test(topology_conflict: bool) -> CrossPoolFenceFleetProofGuard { let mut state = cross_pool_fence_fleet_proof_slot() .write() .unwrap_or_else(std::sync::PoisonError::into_inner); @@ -1045,21 +1079,21 @@ pub(crate) fn without_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceF state.draining_generation = Some(proof.generation); } } - state.topology_conflict = true; + state.topology_conflict |= topology_conflict; if let Some(proof) = journal_state.proof.take() { proof.generation.revoke(); if !proof.generation.is_drained() { journal_state.draining_generation = Some(proof.generation); } } - journal_state.topology_conflict = true; + journal_state.topology_conflict |= topology_conflict; if let Some(proof) = decommission_state.proof.take() { proof.generation.revoke(); if !proof.generation.is_drained() { decommission_state.draining_generation = Some(proof.generation); } } - decommission_state.topology_conflict = true; + decommission_state.topology_conflict |= topology_conflict; guard } @@ -2932,7 +2966,7 @@ impl NotificationSys { Ok(lease) => grants.push(ScannerPublicationLeaseGrant { host, lease }), Err(err) => { let _ = self.release_scanner_publication_leases(grants).await; - return Err(Error::other(format!("scanner publication lease acquisition failed: {err}"))); + return Err(scanner_publication_lease_acquisition_error(&host, observational_only, err)); } } } @@ -3931,6 +3965,22 @@ mod tests { use crate::bucket::lifecycle::tier_last_day_stats::LastDayTierStats; use rustfs_data_usage::TierStats; + #[test] + fn scanner_lease_acquisition_error_identifies_peer_and_preserves_retry_suffix() { + for (observational_only, purpose) in [(false, "authoritative"), (true, "observational")] { + let reason = "scanner publication lease is blocked by pending namespace commit"; + let error = scanner_publication_lease_acquisition_error( + "peer-a.example.invalid:9000", + observational_only, + Error::other(reason), + ) + .to_string(); + assert!(error.contains("peer-a.example.invalid:9000"), "missing peer: {error}"); + assert!(error.contains(purpose), "missing purpose: {error}"); + assert!(error.ends_with(reason), "retry classification must retain the original suffix"); + } + } + fn ring(total_size: u64) -> LastDayTierStats { let mut stats = LastDayTierStats::default(); stats.add_stats(TierStats { diff --git a/crates/ecstore/src/services/rebalance/control.rs b/crates/ecstore/src/services/rebalance/control.rs index a8fef5fd8..0ee302157 100644 --- a/crates/ecstore/src/services/rebalance/control.rs +++ b/crates/ecstore/src/services/rebalance/control.rs @@ -18,8 +18,8 @@ use super::{ encode_rebalance_stop_propagation_record, }; use crate::core::pools::{ - PoolMeta, PoolRebalanceActivationFence, acquire_pool_activation_fleet_proof, acquire_pool_rebalance_activation_locks, - pool_meta_has_active_decommission, + PoolMeta, PoolRebalanceActivationFence, PoolRebalanceWorkerFence, acquire_pool_activation_fleet_proof, + acquire_pool_rebalance_activation_locks, acquire_pool_rebalance_worker_locks, pool_meta_has_active_decommission, }; use crate::error::{Error, Result}; use crate::object_api::ObjectOptions; @@ -77,10 +77,33 @@ fn ensure_rebalance_activation_pool_meta_allowed(meta: &PoolMeta) -> Result<()> } pub(super) enum RebalanceWorkerActivationFence { - Ready(Box), + Ready(Box), NotStartedTerminal, } +async fn persisted_rebalance_requires_worker_activation(pool: Arc, expected_id: &str) -> Result +where + S: EcstoreObjectIO, +{ + let mut persisted = RebalanceMeta::new(); + persisted + .load_with_opts( + pool, + ObjectOptions { + no_lock: true, + ..Default::default() + }, + ) + .await?; + if persisted.id != expected_id { + return Err(Error::other(format!( + "rebalance metadata changed before worker activation: expected {expected_id}, found {}", + persisted.id + ))); + } + Ok(crate::services::rebalance::rebalance_requires_worker_activation(&persisted)) +} + pub(super) struct RebalanceRunGuard { _local_guard: tokio::sync::OwnedRwLockReadGuard<()>, persisted_guard: rustfs_lock::NamespaceLockGuard, @@ -577,48 +600,58 @@ impl ECStore { where S: EcstoreObjectIO + StorageNamespaceLocking, { - // Lock order: pool_meta_save_gate -> rebalance.bin -> pool.bin. + // Lock order: pool_meta_save_gate -> rebalance.bin -> pool.bin. These + // shared fences admit a peer without waiting for the run's own entries. let mut pool_meta_guard = self.pool_meta_save_gate.lock().await; pool_meta_guard.ensure_write_safe("rebalance worker activation")?; - // Classify the durable rebalance record while holding both namespace - // fences. A terminal record is a no-op and must not depend on the - // notification subsystem having published a fleet proof yet. - let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?; + let mut worker_fence = acquire_pool_rebalance_worker_locks(pool.clone()).await?; let pool_meta = self - .load_runtime_pool_meta_under_activation_fence(&mut pool_meta_guard, &activation_fence, "rebalance worker activation") + .load_runtime_pool_meta_under_worker_fence(&mut pool_meta_guard, &worker_fence, "rebalance worker activation") .await?; ensure_rebalance_activation_pool_meta_allowed(&pool_meta)?; - - let mut persisted = RebalanceMeta::new(); - persisted - .load_with_opts( - pool, - ObjectOptions { - no_lock: true, - ..Default::default() - }, - ) - .await?; - if persisted.id != expected_id { - return Err(Error::other(format!( - "rebalance metadata changed before worker activation: expected {expected_id}, found {}", - persisted.id - ))); - } - - activation_fence.ensure_held()?; - if !crate::services::rebalance::rebalance_requires_worker_activation(&persisted) { + let needs_activation = persisted_rebalance_requires_worker_activation(pool, expected_id).await?; + worker_fence.ensure_held()?; + if !needs_activation { return Ok(RebalanceWorkerActivationFence::NotStartedTerminal); } - // Active worker admission still requires the fail-closed fleet proof. - // Attach it immediately before the final fence validation so expiry or - // topology changes are checked again at every later commit boundary. + let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?; + worker_fence.set_fleet_proof(fleet_proof); + worker_fence.ensure_held()?; + Ok(RebalanceWorkerActivationFence::Ready(Box::new(worker_fence))) + } + + pub(super) async fn fence_rebalance_worker_metadata_update( + &self, + pool: Arc, + expected_id: &str, + ) -> Result>> + where + S: EcstoreObjectIO + StorageNamespaceLocking, + { + // The admission read fences must already be dropped. Reacquire in the + // original order and revalidate every durable boundary across that gap. + let mut pool_meta_guard = self.pool_meta_save_gate.lock().await; + pool_meta_guard.ensure_write_safe("rebalance worker metadata update")?; + let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?; + let pool_meta = self + .load_runtime_pool_meta_under_activation_fence( + &mut pool_meta_guard, + &activation_fence, + "rebalance worker metadata update", + ) + .await?; + ensure_rebalance_activation_pool_meta_allowed(&pool_meta)?; + let needs_activation = persisted_rebalance_requires_worker_activation(pool, expected_id).await?; + activation_fence.ensure_held()?; + if !needs_activation { + return Ok(None); + } + let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?; activation_fence.set_fleet_proof(fleet_proof); activation_fence.ensure_held()?; - - Ok(RebalanceWorkerActivationFence::Ready(Box::new(activation_fence))) + Ok(Some(Box::new(activation_fence))) } #[tracing::instrument(skip_all)] @@ -1816,6 +1849,107 @@ mod tests { assert_activation_locks_released(&store).await; } + #[tokio::test] + #[serial_test::serial] + async fn rebalance_init_missing_fleet_proof_preserves_metadata_and_rechecks_competing_run() { + let previous_id = "completed-before-capability-refresh"; + let terminal = RebalanceMeta { + id: previous_id.to_string(), + stopped_at: Some(OffsetDateTime::now_utc()), + pool_stats: vec![RebalanceStats::default(); 2], + ..Default::default() + }; + let (_temp_dirs, store, competing_store) = crate::services::rebalance::test_two_pool_stores(Some(terminal.clone())).await; + let disk_stats = vec![ + DiskStat { + total_space: 100, + available_space: 0, + }, + DiskStat { + total_space: 100, + available_space: 100, + }, + ]; + let initial_epoch = store.ctx.data_movement_operation_epoch(); + let proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test(); + set_rebalance_disk_stats_override_for_test(store.id, disk_stats.clone()); + let err = store + .init_rebalance_start(vec!["waiting-request".to_string()]) + .await + .expect_err("initialization must fail closed while the live proof is missing"); + assert!( + err.to_string() + .contains("pool activation requires a live fleet capability proof") + ); + assert!( + crate::core::pools::is_pool_activation_fleet_proof_error(&err), + "the admin handler retries init only on this error shape: {err}" + ); + assert_eq!(store.ctx.data_movement_operation_epoch(), initial_epoch); + { + let local = store.rebalance_meta.read().await; + let local = local.as_ref().expect("failed initialization must preserve the previous run"); + assert_eq!(local.id, previous_id); + assert_eq!(local.stopped_at, terminal.stopped_at); + assert!(local.cancel.is_none(), "failed initialization must not start a worker"); + } + let mut persisted = RebalanceMeta::new(); + persisted + .load(store.pools[0].clone()) + .await + .expect("failed initialization must leave durable metadata readable"); + assert_eq!(persisted.id, previous_id); + assert_eq!(persisted.stopped_at, terminal.stopped_at); + + drop(proof_guard); + set_rebalance_disk_stats_override_for_test(competing_store.id, disk_stats.clone()); + let winning_id = competing_store + .init_rebalance_start(vec!["winning-request".to_string()]) + .await + .expect("a ready request must be able to initialize one run after the failed attempt"); + assert_ne!(winning_id, previous_id); + + // The waiting request still holds its old local snapshot. Its retry must + // recheck durable state under the activation locks before publishing. + set_rebalance_disk_stats_override_for_test(store.id, disk_stats); + let remote_conflict = store + .init_rebalance_start(vec!["waiting-request".to_string()]) + .await + .expect_err("the waiting request must not replace a durable competing run"); + assert!( + matches!( + crate::data_movement::data_movement_stage_source(&remote_conflict), + Some(Error::RebalanceAlreadyRunning) + ), + "the save-stage error must retain the typed competing-run conflict: {remote_conflict:?}" + ); + let local_conflict = competing_store + .init_rebalance_start(vec!["another-request".to_string()]) + .await + .expect_err("a local retry must not replace its active run"); + assert!( + matches!(local_conflict, Error::RebalanceAlreadyRunning), + "the local admission check must report the competing-run conflict: {local_conflict:?}" + ); + persisted + .load(store.pools[0].clone()) + .await + .expect("the competing run must remain durable after both rejected retries"); + assert_eq!(persisted.id, winning_id); + assert_eq!(persisted.pool_stats[0].buckets, vec!["winning-request".to_string()]); + assert_eq!( + store + .rebalance_meta + .read() + .await + .as_ref() + .expect("old local run must survive") + .id, + previous_id + ); + assert_eq!(store.ctx.data_movement_operation_epoch(), initial_epoch + 1); + } + #[tokio::test] #[serial_test::serial] async fn rebalance_worker_skips_terminal_metadata_without_fleet_proof() { @@ -1874,6 +2008,128 @@ mod tests { ); } + async fn assert_worker_write_upgrade_revalidates_durable_state(change: &str) { + let rebalance_id = "rebalance-worker-write-upgrade"; + let active = RebalanceMeta { + id: rebalance_id.to_string(), + percent_free_goal: 0.5, + pool_stats: vec![ + RebalanceStats { + participating: true, + init_capacity: 100, + info: RebalanceInfo { + status: RebalStatus::Started, + ..Default::default() + }, + ..Default::default() + }, + RebalanceStats::default(), + ], + ..Default::default() + }; + let (_temp_dirs, store, peer) = + crate::services::rebalance::test_two_pool_stores_with_isolated_node_contexts(Some(active.clone())).await; + let arrived = Arc::new(tokio::sync::Notify::new()); + let release = Arc::new(tokio::sync::Notify::new()); + let worker = Arc::clone(&store); + let mut tasks = tokio::task::JoinSet::new(); + tasks.spawn( + super::super::runtime::REBALANCE_WORKER_WRITE_UPGRADE_BARRIER + .scope((Arc::clone(&arrived), Arc::clone(&release)), async move { + worker.start_rebalance_for_id(rebalance_id).await + }), + ); + tokio::time::timeout(std::time::Duration::from_secs(30), arrived.notified()) + .await + .expect("empty-queue activation must release its read fences before requesting write authority"); + + let mut changed = active.clone(); + match change { + "stopped" => { + changed.stopped_at = Some(OffsetDateTime::now_utc()); + changed.pool_stats[0].info.status = RebalStatus::Stopped; + } + "replaced" => changed.id = "replacement-rebalance-run".to_string(), + "decommission" => { + let mut pool_meta = peer.pool_meta.read().await.clone(); + pool_meta.pools[0].decommission = Some(crate::core::pools::PoolDecommissionInfo { + start_time: Some(OffsetDateTime::now_utc()), + ..Default::default() + }); + pool_meta + .save(peer.pools.clone()) + .await + .expect("a competing pool metadata writer must acquire the released pool fence"); + } + _ => panic!("unknown durable state transition"), + } + changed + .save_with_opts( + peer.pools[0].clone(), + ObjectOptions { + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("a competing run writer must acquire the released rebalance fence"); + release.notify_one(); + let result = tokio::time::timeout(std::time::Duration::from_secs(30), tasks.join_next()) + .await + .expect("activation must finish after the durable transition") + .expect("activation task must be tracked") + .expect("activation must not panic"); + match change { + "stopped" => result.expect("a durably stopped run must become an activation no-op"), + "replaced" => assert!( + result + .expect_err("a replaced run must reject stale activation") + .to_string() + .contains("expected"), + ), + "decommission" => assert!(matches!(result, Err(Error::DecommissionAlreadyRunning))), + _ => unreachable!(), + } + let mut persisted = RebalanceMeta::new(); + persisted + .load(peer.pools[0].clone()) + .await + .expect("the competing durable state must remain readable"); + assert_eq!(persisted.id, changed.id, "activation must not overwrite the run identity"); + assert_eq!(persisted.stopped_at, changed.stopped_at, "activation must not overwrite the stop marker"); + assert_eq!(persisted.pool_stats[0].info.status, changed.pool_stats[0].info.status); + assert!( + store + .rebalance_meta + .read() + .await + .as_ref() + .expect("local metadata must remain present") + .cancel + .is_none(), + "rejected activation must not install a worker token", + ); + assert_activation_locks_released(&store).await; + } + + #[tokio::test] + #[serial_test::serial] + async fn rebalance_worker_write_upgrade_rechecks_stop() { + assert_worker_write_upgrade_revalidates_durable_state("stopped").await; + } + + #[tokio::test] + #[serial_test::serial] + async fn rebalance_worker_write_upgrade_rechecks_run_identity() { + assert_worker_write_upgrade_revalidates_durable_state("replaced").await; + } + + #[tokio::test] + #[serial_test::serial] + async fn rebalance_worker_write_upgrade_rechecks_decommission() { + assert_worker_write_upgrade_revalidates_durable_state("decommission").await; + } + #[tokio::test] #[serial_test::serial] async fn rebalance_activation_adopts_commit_after_post_save_fence_loss() { diff --git a/crates/ecstore/src/services/rebalance/entry.rs b/crates/ecstore/src/services/rebalance/entry.rs index 6a9987707..09cbdc4f4 100644 --- a/crates/ecstore/src/services/rebalance/entry.rs +++ b/crates/ecstore/src/services/rebalance/entry.rs @@ -50,7 +50,7 @@ fn ensure_rebalance_entry_active(cancel: &CancellationToken) -> Result<()> { Ok(()) } -#[cfg(test)] +#[cfg(any(test, feature = "test-util"))] tokio::task_local! { static REBALANCE_ENTRY_RUN_FENCE_BARRIER: (Arc, Arc); } @@ -268,7 +268,7 @@ impl ECStore { self.rebalance_run_guard(rebalance_id.as_ref(), "rebalance entry") }) .await?; - #[cfg(test)] + #[cfg(any(test, feature = "test-util"))] if let Ok((arrived, release)) = REBALANCE_ENTRY_RUN_FENCE_BARRIER.try_with(Clone::clone) { arrived.notify_one(); release.notified().await; @@ -836,7 +836,8 @@ pub mod test_util { _temp_dirs: Vec, store: Arc, cancel: CancellationToken, - barrier: crate::data_movement::SourceCleanupDeleteBarrier, + entry_arrived: Arc, + entry_release: Arc, stop_probe: super::super::control::RebalanceStopWaitProbe, entry_task: Option>>, } @@ -897,31 +898,36 @@ pub mod test_util { cached: Some(file_meta), reusable: false, }; - let barrier = crate::data_movement::SourceCleanupDeleteBarrier::install(bucket, object); + let entry_arrived = Arc::new(tokio::sync::Notify::new()); + let entry_release = Arc::new(tokio::sync::Notify::new()); let entry_store = Arc::clone(&store); let entry_set = Arc::clone(&set); let entry_cancel = cancel.clone(); - let entry_task = tokio::spawn(async move { - entry_store - .rebalance_entry( - RebalanceEntryTarget { - bucket: bucket.to_string(), - pool_index: 0, - }, - entry, - entry_set, - Arc::new(RebalanceBucketConfigs::default()), - Arc::from(rebalance_id), - entry_cancel, - ) - .await - }); + let entry_task = tokio::spawn(REBALANCE_ENTRY_RUN_FENCE_BARRIER.scope( + (Arc::clone(&entry_arrived), Arc::clone(&entry_release)), + async move { + entry_store + .rebalance_entry( + RebalanceEntryTarget { + bucket: bucket.to_string(), + pool_index: 0, + }, + entry, + entry_set, + Arc::new(RebalanceBucketConfigs::default()), + Arc::from(rebalance_id), + entry_cancel, + ) + .await + }, + )); Self { _temp_dirs, store, cancel, - barrier, + entry_arrived, + entry_release, stop_probe: super::super::control::RebalanceStopWaitProbe::install(rebalance_id), entry_task: Some(entry_task), } @@ -932,7 +938,9 @@ pub mod test_util { } pub async fn wait_until_entry_paused(&self) { - self.barrier.wait_until_paused().await; + tokio::time::timeout(std::time::Duration::from_secs(30), self.entry_arrived.notified()) + .await + .expect("the entry should acquire its run fence before pausing"); } pub async fn wait_until_admission_cancelled(&self) { @@ -946,7 +954,7 @@ pub mod test_util { } pub fn release_entry(&self) { - self.barrier.release(); + self.entry_release.notify_one(); } pub async fn assert_entry_cancelled(&mut self) { @@ -964,7 +972,7 @@ pub mod test_util { impl Drop for PausedRebalanceEntryTestFixture { fn drop(&mut self) { - self.barrier.release(); + self.entry_release.notify_one(); if let Some(task) = self.entry_task.take() { task.abort(); } @@ -1432,7 +1440,7 @@ mod tests { #[tokio::test] #[serial_test::serial] - async fn real_rebalance_entry_progresses_while_peer_activation_waits_for_run_fence() { + async fn real_rebalance_peer_activates_while_entry_retains_run_fence() { const REBALANCE_ID: &str = "rebalance-peer-activation-lock-order"; let (_temp_dirs, store, peer) = crate::services::rebalance::test_two_pool_stores_with_isolated_node_contexts(Some( active_rebalance_meta(REBALANCE_ID), @@ -1448,7 +1456,7 @@ mod tests { let bucket = crate::disk::RUSTFS_META_BUCKET; let object = "rebalance-peer-activation-object"; let version_id = uuid::Uuid::new_v4(); - let payload = b"entry must drain before peer activation takes the pool fence".repeat(1024); + let payload = b"peer activation must not wait for this entry to drain".repeat(1024); let source_set = store.pools[0].get_disks_by_key(object); let target_set = store.pools[1].get_disks_by_key(object); let opts = ObjectOptions { @@ -1508,7 +1516,12 @@ mod tests { ); tokio::time::timeout(StdDuration::from_secs(30), attempted.notified()) .await - .expect("peer activation must attempt the persisted rebalance write fence"); + .expect("peer activation must attempt the persisted rebalance fence"); + tokio::time::timeout(StdDuration::from_secs(30), activation_result) + .await + .expect("peer activation must finish while the entry remains paused") + .expect("peer activation result should be sent") + .expect("joining the same run must not require its active entry to release the run fence"); release.notify_one(); tokio::time::timeout(StdDuration::from_secs(30), async { @@ -1523,10 +1536,6 @@ mod tests { }) .await .expect("entry and peer activation must both make progress"); - activation_result - .await - .expect("peer activation result should be sent") - .expect("peer activation must not time out behind the entry it blocks"); let mut reader = target_set .get_object_reader(bucket, object, None, HeaderMap::new(), &opts) diff --git a/crates/ecstore/src/services/rebalance/meta.rs b/crates/ecstore/src/services/rebalance/meta.rs index 8983f40cd..54f5efe8e 100644 --- a/crates/ecstore/src/services/rebalance/meta.rs +++ b/crates/ecstore/src/services/rebalance/meta.rs @@ -71,6 +71,31 @@ impl From for RebalStatus { } impl RebalanceMeta { + /// A historical run has no authority over the current deployment's pool + /// indices. Stop timestamps alone do not prove that its workers stopped. + pub(crate) fn has_terminal_topology(&self) -> bool { + !self.id.is_empty() + && !self.pool_stats.is_empty() + && self.pool_stats.iter().all(|pool| { + if pool.info.stopping + || !matches!(pool.info.status, RebalStatus::Completed | RebalStatus::Stopped | RebalStatus::Failed) + && (pool.participating || pool.info.status != RebalStatus::None) + { + return false; + } + let Some(message) = pool.info.last_error.as_deref() else { + return true; + }; + if !is_rebalance_stop_propagation_error(Some(message)) { + return true; + } + decode_rebalance_stop_propagation_record(message).is_some_and(|record| { + !record.has_failures() + && (record.stop_attempt_at.or(self.stopped_at).is_none() || record.terminal_reload_attempt_at.is_some()) + }) + }) + } + pub fn new() -> Self { Self::default() } diff --git a/crates/ecstore/src/services/rebalance/runtime.rs b/crates/ecstore/src/services/rebalance/runtime.rs index 39302d592..abead2485 100644 --- a/crates/ecstore/src/services/rebalance/runtime.rs +++ b/crates/ecstore/src/services/rebalance/runtime.rs @@ -27,6 +27,11 @@ use tokio::time::{Duration, Instant}; use tokio_util::sync::CancellationToken; use tracing::{debug, error, info, warn}; +#[cfg(test)] +tokio::task_local! { + pub(super) static REBALANCE_WORKER_WRITE_UPGRADE_BARRIER: (Arc, Arc); +} + pub(super) fn should_fail_repeated_rebalance_bucket_defer( deferred_buckets: &mut HashSet, bucket: &str, @@ -213,6 +218,22 @@ impl ECStore { } if must_persist { + // Shared admission grants no write authority. Release it before + // taking the exclusive metadata fences, then recheck stop/run/pool + // state before publishing the locally staged terminal candidate. + drop(activation_fence); + #[cfg(test)] + if let Ok((arrived, release)) = REBALANCE_WORKER_WRITE_UPGRADE_BARRIER.try_with(Clone::clone) { + arrived.notify_one(); + release.notified().await; + } + let Some(activation_fence) = self + .fence_rebalance_worker_metadata_update(pool.clone(), expected_id.as_ref()) + .await? + else { + cancel_tx.cancel(); + return Ok(false); + }; let save_result = resolve_rebalance_meta_save_result( self.save_rebalance_meta_under_activation_fence( pool, @@ -242,13 +263,14 @@ impl ECStore { cancel_tx.cancel(); return Err(err); } + } else { + if let Err(err) = activation_fence.ensure_held() { + let mut rebalance_meta = self.rebalance_meta.write().await; + rollback_local_rebalance_worker_activation(rebalance_meta.as_mut(), expected_id.as_ref(), &rx); + return Err(err); + } + drop(activation_fence); } - if !must_persist && let Err(err) = activation_fence.ensure_held() { - let mut rebalance_meta = self.rebalance_meta.write().await; - rollback_local_rebalance_worker_activation(rebalance_meta.as_mut(), expected_id.as_ref(), &rx); - return Err(err); - } - drop(activation_fence); if activation_outcome != RebalanceLocalActivationOutcome::Started { return Ok(must_persist); diff --git a/crates/ecstore/src/services/tier/tier.rs b/crates/ecstore/src/services/tier/tier.rs index 8d80f37e9..9c0f84907 100644 --- a/crates/ecstore/src/services/tier/tier.rs +++ b/crates/ecstore/src/services/tier/tier.rs @@ -2270,7 +2270,7 @@ pub(crate) fn tier_destination_id_from_metadata(metadata: &HashMap(handle: &Arc>, api: Arc) -> io::Result<()> + where + S: EcstoreObjectIO + + EcstoreObjectOperations + + NamespaceLocking + + ListOperations>, + { + // Periodic and committed-mutation refreshes call this shared entry directly. + let reload = Self::reload_handle_inner(handle, api); + #[cfg(feature = "diagnostic-read-version-caller")] + let reload = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::TierConfigReload, + reload, + ); + reload.await + } + + async fn reload_handle_inner(handle: &Arc>, api: Arc) -> io::Result<()> where S: EcstoreObjectIO + EcstoreObjectOperations @@ -12533,6 +12550,51 @@ mod tests { assert!(reloaded.iter().any(|intent| intent.mutation_id == prepared_id)); } + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn diagnostic_reload_shared_entry_preserves_source_and_api() { + use crate::api::diagnostic::{ReadVersionEntrySource, current_read_version_entry_source, with_read_version_entry}; + use rustfs_io_metrics::s3_http_metrics::{ + S3HttpRequestGuard, current_s3_http_operation_context, with_isolated_s3_http_operation_context, + }; + + let store = Arc::new(CasConfigStore::default()); + empty_mgr() + .save_tiering_config_if_current(store.clone(), None) + .await + .expect("empty tier config fixture should persist"); + let mut request = S3HttpRequestGuard::new("GET"); + let get = request.in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::GetObject); + current_s3_http_operation_context().expect("GetObject context should be captured") + }); + + for (earlier, api) in [(None, None), (Some(ReadVersionEntrySource::ConfigRead), Some(get))] { + store.reload_contexts.lock().expect("context recorder").clear(); + let handle = TierConfigMgr::new(); + with_isolated_s3_http_operation_context(api, async { + let reload = TierConfigMgr::reload_handle_with(&handle, store.clone()); + if let Some(source) = earlier { + with_read_version_entry(source, reload).await + } else { + reload.await + } + }) + .await + .expect("shared reload entry should load config and scan intents"); + + let observed = store.reload_contexts.lock().expect("context recorder"); + assert!(observed.iter().any(|(boundary, _, _)| *boundary == "reader")); + assert!(observed.iter().any(|(boundary, _, _)| *boundary == "list")); + let expected = earlier.unwrap_or(ReadVersionEntrySource::TierConfigReload); + assert!(observed.iter().all(|(_, source, operation)| { + *source == Some(expected) && *operation == api.map(|context| context.operation_name()) + })); + assert_eq!(current_read_version_entry_source(), None); + assert_eq!(current_s3_http_operation_context(), None); + } + } + #[tokio::test] async fn reload_handle_keeps_committed_block_until_publish_swaps_generation() { let store = Arc::new(CasConfigStore::default()); @@ -14725,8 +14787,17 @@ mod tests { release: tokio::sync::Notify, } + #[cfg(feature = "diagnostic-read-version-caller")] + type ReloadDiagnosticObservation = ( + &'static str, + Option, + Option<&'static str>, + ); + #[derive(Debug)] struct CasConfigStore { + #[cfg(feature = "diagnostic-read-version-caller")] + reload_contexts: Mutex>, objects: tokio::sync::Mutex, String)>>, legacy_state: tokio::sync::Mutex>>, if_match_race_rewrite: tokio::sync::Mutex)>>, @@ -14750,6 +14821,8 @@ mod tests { impl Default for CasConfigStore { fn default() -> Self { Self { + #[cfg(feature = "diagnostic-read-version-caller")] + reload_contexts: Mutex::new(Vec::new()), objects: tokio::sync::Mutex::new(HashMap::new()), legacy_state: tokio::sync::Mutex::new(None), if_match_race_rewrite: tokio::sync::Mutex::new(std::collections::VecDeque::new()), @@ -14904,6 +14977,12 @@ mod tests { _headers: Self::HeaderMap, _opts: &Self::ObjectOptions, ) -> Result { + #[cfg(feature = "diagnostic-read-version-caller")] + self.reload_contexts.lock().expect("context recorder").push(( + "reader", + crate::api::diagnostic::current_read_version_entry_source(), + rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(), + )); let legacy_config_path = tier_config_path(TIER_CONFIG_LEGACY_FILE); let (data, etag) = if object == legacy_config_path { let state = self.legacy_state.lock().await; @@ -15259,6 +15338,12 @@ mod tests { _start_after: Option, _incl_deleted: bool, ) -> Result { + #[cfg(feature = "diagnostic-read-version-caller")] + self.reload_contexts.lock().expect("context recorder").push(( + "list", + crate::api::diagnostic::current_read_version_entry_source(), + rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(), + )); if matches!( prefix, crate::services::tier::tier_mutation_intent::TIER_MUTATION_INTENT_RECORD_PREFIX diff --git a/crates/ecstore/src/set_disk/conditional_read_bench.rs b/crates/ecstore/src/set_disk/conditional_read_bench.rs new file mode 100644 index 000000000..b218d4786 --- /dev/null +++ b/crates/ecstore/src/set_disk/conditional_read_bench.rs @@ -0,0 +1,63 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Test-util-only controls for the paired read benchmark's dedicated bucket. + +use std::sync::atomic::{AtomicBool, Ordering}; + +static ACTIVE: AtomicBool = AtomicBool::new(false); + +/// Keeps metadata off the cache in the benchmark's isolated bucket so GET +/// metadata reads exercise the configured slow disk. Body-cache policy +/// remains intact. +#[must_use] +pub struct ConditionalReadBenchmarkMetadataGuard { + _private: (), +} + +impl ConditionalReadBenchmarkMetadataGuard { + pub fn acquire() -> Self { + assert!( + ACTIVE + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .is_ok() + ); + Self { _private: () } + } +} + +impl Drop for ConditionalReadBenchmarkMetadataGuard { + fn drop(&mut self) { + ACTIVE.store(false, Ordering::Release); + } +} + +pub(super) fn cache_bypass_applies(bucket: &str) -> bool { + bucket == "conditional-bench" && ACTIVE.load(Ordering::Acquire) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn benchmark_metadata_profile_is_scoped_to_its_bucket_and_owner() { + assert!(!cache_bypass_applies("conditional-bench")); + let owner = ConditionalReadBenchmarkMetadataGuard::acquire(); + assert!(cache_bypass_applies("conditional-bench")); + assert!(!cache_bypass_applies("other-bucket")); + drop(owner); + assert!(!cache_bypass_applies("conditional-bench")); + } +} diff --git a/crates/ecstore/src/set_disk/core/io_primitives.rs b/crates/ecstore/src/set_disk/core/io_primitives.rs index 9976d113e..e44298432 100644 --- a/crates/ecstore/src/set_disk/core/io_primitives.rs +++ b/crates/ecstore/src/set_disk/core/io_primitives.rs @@ -43,10 +43,10 @@ use super::super::{ EVENT_SET_DISK_RENAME_TAIL_DRAIN_FAILED, EVENT_SET_DISK_WRITE, Error, FileInfo, FileMeta, FileMetaShallowVersion, GetCodecStreamingFallbackReason, GetObjectMetadataCacheEntry, HTTPPreconditions, HashAlgorithm, HealAdmissionResult, HealChannelPriority, HealRequestSource, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, MultipartWriteQuorumContext, - OBJECT_OP_IGNORED_ERRS, ObjectOptions, ObjectPartInfo, OffsetDateTime, RUSTFS_META_BUCKET, RUSTFS_META_MULTIPART_BUCKET, - RawFileInfo, ReadMultipleReq, ReadMultipleResp, Result, SLASH_SEPARATOR, STORAGE_FORMAT_FILE, SetDisks, SnapshotLeaseToken, - StorageError, UpdateMetadataOpts, Uuid, capacity_scope_from_disks, current_dirty_generation, debug, disk, - file_info_is_valid_for_metadata, info, is_object_dangling, issue3031_diag_enabled, join_all, join_errs, + OBJECT_OP_IGNORED_ERRS, ObjectInfo, ObjectOptions, ObjectPartInfo, OffsetDateTime, RUSTFS_META_BUCKET, + RUSTFS_META_MULTIPART_BUCKET, RawFileInfo, ReadMultipleReq, ReadMultipleResp, Result, SLASH_SEPARATOR, STORAGE_FORMAT_FILE, + SetDisks, SnapshotLeaseToken, StorageError, UpdateMetadataOpts, Uuid, capacity_scope_from_disks, current_dirty_generation, + debug, disk, file_info_is_valid_for_metadata, info, is_object_dangling, issue3031_diag_enabled, join_all, join_errs, log_multipart_write_quorum_failure, merge_file_meta_versions, path_join_buf, record_global_dirty_scope, reduce_read_quorum_errs, reduce_write_quorum_errs, send_heal_request_with_admission, should_prevent_write, to_object_err, warn, @@ -145,7 +145,7 @@ use std::{ time::{Duration, Instant}, }; use tokio::io::{AsyncRead, ReadBuf}; -use tokio::sync::RwLock; +use tokio::sync::{RwLock, oneshot}; use tokio::task::JoinSet; pub(in crate::set_disk) const EVENT_SET_DISK_READ: &str = "set_disk_read"; @@ -1260,6 +1260,10 @@ async fn try_create_bitrot_readers_via_batch_pread( } let requests: Vec<_> = batch_items.iter().map(|(_, p, off, len)| (p.clone(), *off, *len)).collect(); + #[cfg(test)] + for (disk_index, _, _, _) in &batch_items { + disk_call_counters::record(object, disk_call_counters::KIND_DISK_SHARD_OPEN, *disk_index); + } let batch_results = batch_shard_pread(requests).await; let mut setup = BitrotReaderSetup::new(disks.len()); @@ -2447,6 +2451,17 @@ impl RenameConvergence { } } +struct RenameDataObservations<'a> { + disks: &'a [Option], + file_infos: &'a [FileInfo], + disk_versions: &'a [Option>], + errs: &'a [Option], + cleanup_data_dirs: &'a [Option], + old_current_sizes: &'a [Option], + old_current_source_checked: &'a [bool], + old_current_sources: &'a [Option], +} + pub(in crate::set_disk) struct RenameDataCommit { pub(in crate::set_disk) online_disks: Vec>, /// Disks whose capacity must be marked for this commit. Under early ACK, @@ -2456,6 +2471,9 @@ pub(in crate::set_disk) struct RenameDataCommit { pub(in crate::set_disk) data_dir: Option, pub(in crate::set_disk) cleanup_disks: Vec>, pub(in crate::set_disk) old_current_size: Option, + pub(in crate::set_disk) old_current_source_checked: bool, + pub(in crate::set_disk) old_current_source_all_successful_checked: bool, + pub(in crate::set_disk) old_current_source: Option, pub(in crate::set_disk) committed_file_info: FileInfo, pub(in crate::set_disk) tail_drain: Option>>, } @@ -2471,6 +2489,500 @@ pub(in crate::set_disk) struct RenameTailOutcome { pub(in crate::set_disk) cleanup: Vec, } +const RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP"; +const RENAME_TAIL_CLEANUP_DEFER_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ENABLE"; +const RENAME_TAIL_CLEANUP_DEFER_QUEUE_CAPACITY_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_QUEUE_CAPACITY"; +const RENAME_TAIL_CLEANUP_DEFER_WORKERS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKERS"; +const RENAME_TAIL_CLEANUP_DEFER_WORKER_DELAY_MS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKER_DELAY_MS"; +const RENAME_TAIL_CLEANUP_DEFER_BATCH_WINDOW_MS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_BATCH_WINDOW_MS"; +const RENAME_TAIL_CLEANUP_DEFER_BATCH_MAX_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_BATCH_MAX"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_ENABLE_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_ENABLE"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_MIN_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_MIN"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_LOW_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_LOW"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_HIGH_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_HIGH"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_LOW_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_LOW"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_HIGH_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_HIGH"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_LOW_MS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_LOW_MS"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_HIGH_MS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_HIGH_MS"; +const RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER"; +const RENAME_FOREGROUND_ACTIVE_LANES_ENV: &str = "RUSTFS_PUT_RENAME_FOREGROUND_ACTIVE_LANES"; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_QUEUE_CAPACITY: usize = 1024; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_WORKERS: usize = 2; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_BATCH_MAX: usize = 1; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_LOW: usize = 128; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_HIGH: usize = 512; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_LOW_MS: u64 = 500; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_HIGH_MS: u64 = 2_000; +const DEFAULT_RENAME_FOREGROUND_ACTIVE_LANES: usize = 1024; + +fn rename_tail_cleanup_counterfactual_skip_enabled() -> bool { + static ENABLED: OnceLock = OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP_ENV).ok().as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_tail_cleanup_defer_enabled() -> bool { + static ENABLED: OnceLock = OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(RENAME_TAIL_CLEANUP_DEFER_ENV).ok().as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_tail_cleanup_defer_queue_capacity() -> usize { + static CAPACITY: OnceLock = OnceLock::new(); + *CAPACITY.get_or_init(|| { + rustfs_utils::get_env_usize( + RENAME_TAIL_CLEANUP_DEFER_QUEUE_CAPACITY_ENV, + DEFAULT_RENAME_TAIL_CLEANUP_DEFER_QUEUE_CAPACITY, + ) + .max(1) + }) +} + +fn rename_tail_cleanup_defer_workers() -> usize { + static WORKERS: OnceLock = OnceLock::new(); + *WORKERS.get_or_init(|| { + rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_DEFER_WORKERS_ENV, DEFAULT_RENAME_TAIL_CLEANUP_DEFER_WORKERS).max(1) + }) +} + +fn rename_tail_cleanup_defer_worker_delay() -> Duration { + static DELAY: OnceLock = OnceLock::new(); + *DELAY.get_or_init(|| Duration::from_millis(rustfs_utils::get_env_u64(RENAME_TAIL_CLEANUP_DEFER_WORKER_DELAY_MS_ENV, 0))) +} + +fn rename_tail_cleanup_defer_batch_window() -> Duration { + static WINDOW: OnceLock = OnceLock::new(); + *WINDOW.get_or_init(|| Duration::from_millis(rustfs_utils::get_env_u64(RENAME_TAIL_CLEANUP_DEFER_BATCH_WINDOW_MS_ENV, 0))) +} + +fn rename_tail_cleanup_defer_batch_max() -> usize { + static BATCH_MAX: OnceLock = OnceLock::new(); + *BATCH_MAX.get_or_init(|| { + rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_DEFER_BATCH_MAX_ENV, DEFAULT_RENAME_TAIL_CLEANUP_DEFER_BATCH_MAX).max(1) + }) +} + +fn rename_tail_cleanup_defer_adaptive_batch_enabled() -> bool { + static ENABLED: OnceLock = OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_ENABLE_ENV) + .ok() + .as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_tail_cleanup_defer_adaptive_batch_min() -> usize { + static BATCH_MIN: OnceLock = OnceLock::new(); + *BATCH_MIN.get_or_init(|| rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_MIN_ENV, 1).max(1)) +} + +fn rename_tail_cleanup_defer_adaptive_batch_low() -> usize { + static BATCH_LOW: OnceLock = OnceLock::new(); + *BATCH_LOW.get_or_init(|| { + rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_LOW_ENV, rename_tail_cleanup_defer_batch_max()) + .max(1) + }) +} + +fn rename_tail_cleanup_defer_adaptive_batch_high() -> usize { + static BATCH_HIGH: OnceLock = OnceLock::new(); + *BATCH_HIGH.get_or_init(|| { + rustfs_utils::get_env_usize( + RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_HIGH_ENV, + rename_tail_cleanup_defer_adaptive_batch_low().max(rename_tail_cleanup_defer_batch_max()), + ) + .max(1) + }) +} + +fn rename_tail_cleanup_defer_adaptive_queue_low() -> usize { + static QUEUE_LOW: OnceLock = OnceLock::new(); + *QUEUE_LOW.get_or_init(|| { + rustfs_utils::get_env_usize( + RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_LOW_ENV, + DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_LOW, + ) + }) +} + +fn rename_tail_cleanup_defer_adaptive_queue_high() -> usize { + static QUEUE_HIGH: OnceLock = OnceLock::new(); + *QUEUE_HIGH.get_or_init(|| { + rustfs_utils::get_env_usize( + RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_HIGH_ENV, + DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_HIGH, + ) + }) +} + +fn rename_tail_cleanup_defer_adaptive_age_low() -> Duration { + static AGE_LOW: OnceLock = OnceLock::new(); + *AGE_LOW.get_or_init(|| { + Duration::from_millis(rustfs_utils::get_env_u64( + RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_LOW_MS_ENV, + DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_LOW_MS, + )) + }) +} + +fn rename_tail_cleanup_defer_adaptive_age_high() -> Duration { + static AGE_HIGH: OnceLock = OnceLock::new(); + *AGE_HIGH.get_or_init(|| { + Duration::from_millis(rustfs_utils::get_env_u64( + RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_HIGH_MS_ENV, + DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_HIGH_MS, + )) + }) +} + +fn rename_tail_cleanup_defer_batch_target(queued_depth: usize, oldest_age: Option) -> usize { + let fixed = rename_tail_cleanup_defer_batch_max(); + if !rename_tail_cleanup_defer_adaptive_batch_enabled() { + return fixed; + } + + let min = rename_tail_cleanup_defer_adaptive_batch_min(); + let low = rename_tail_cleanup_defer_adaptive_batch_low().max(min); + let high = rename_tail_cleanup_defer_adaptive_batch_high().max(low); + let queue_low = rename_tail_cleanup_defer_adaptive_queue_low(); + let queue_high = rename_tail_cleanup_defer_adaptive_queue_high().max(queue_low); + let age_low = rename_tail_cleanup_defer_adaptive_age_low(); + let age_high = rename_tail_cleanup_defer_adaptive_age_high().max(age_low); + + let age = oldest_age.unwrap_or_default(); + if queued_depth >= queue_high || age >= age_high { + high + } else if queued_depth >= queue_low || age >= age_low { + low + } else { + min + } +} + +fn rename_tail_cleanup_defer_hold_worker_enabled() -> bool { + static ENABLED: OnceLock = OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER_ENV).ok().as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_foreground_active_lanes() -> &'static Vec { + static LANES: OnceLock> = OnceLock::new(); + LANES.get_or_init(|| { + let lanes = + rustfs_utils::get_env_usize(RENAME_FOREGROUND_ACTIVE_LANES_ENV, DEFAULT_RENAME_FOREGROUND_ACTIVE_LANES).max(1); + (0..lanes).map(|_| AtomicUsize::new(0)).collect() + }) +} + +pub(in crate::set_disk) fn rename_foreground_active_count(disk_index: usize) -> usize { + let lanes = rename_foreground_active_lanes(); + lanes[disk_index % lanes.len()].load(Ordering::Relaxed) +} + +struct RenameForegroundActiveGuard { + disk_index: usize, +} + +impl Drop for RenameForegroundActiveGuard { + fn drop(&mut self) { + let lanes = rename_foreground_active_lanes(); + lanes[self.disk_index % lanes.len()].fetch_sub(1, Ordering::Relaxed); + } +} + +fn rename_foreground_active_guard(disk_index: usize) -> RenameForegroundActiveGuard { + let lanes = rename_foreground_active_lanes(); + let active = lanes[disk_index % lanes.len()].fetch_add(1, Ordering::Relaxed) + 1; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_FOREGROUND_ACTIVE, + active as f64, + ); + RenameForegroundActiveGuard { disk_index } +} + +type RenameTailCleanupFuture = Pin + Send + 'static>>; + +struct RenameTailCleanupJob { + enqueued_at: Instant, + targets: usize, + future: RenameTailCleanupFuture, +} + +struct RenameTailCleanupQueue { + sender: tokio::sync::mpsc::Sender, + queued: Arc, + oldest_enqueued_at: Arc>>, +} + +async fn run_rename_tail_cleanup_job(job: RenameTailCleanupJob) { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_START_DELAY, + job.enqueued_at.elapsed().as_secs_f64() * 1000.0, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_TARGETS, + job.targets as f64, + ); + let worker_delay = rename_tail_cleanup_defer_worker_delay(); + if !worker_delay.is_zero() { + let delay_started = rustfs_io_metrics::put_stage_timer(); + tokio::time::sleep(worker_delay).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_DELAY, + delay_started, + ); + } + let cleanup_started = rustfs_io_metrics::put_stage_timer(); + job.future.await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_DURATION, + cleanup_started, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_COMPLETED, + 1.0, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_COMPLETED_LAG, + job.enqueued_at.elapsed().as_secs_f64() * 1000.0, + ); +} + +impl RenameTailCleanupQueue { + fn new() -> Self { + let (sender, receiver) = tokio::sync::mpsc::channel::(rename_tail_cleanup_defer_queue_capacity()); + let queued = Arc::new(AtomicUsize::new(0)); + let active_workers = Arc::new(AtomicUsize::new(0)); + let oldest_enqueued_at = Arc::new(tokio::sync::Mutex::new(None::)); + let receiver = Arc::new(tokio::sync::Mutex::new(receiver)); + if !rename_tail_cleanup_defer_hold_worker_enabled() { + for _ in 0..rename_tail_cleanup_defer_workers() { + let receiver = Arc::clone(&receiver); + let queued = Arc::clone(&queued); + let active_workers = Arc::clone(&active_workers); + let oldest_enqueued_at = Arc::clone(&oldest_enqueued_at); + tokio::spawn(async move { + loop { + let queued_before_poll = queued.load(Ordering::Relaxed); + let worker_poll_started = rustfs_io_metrics::put_stage_timer(); + let Some(job) = receiver.lock().await.recv().await else { + return; + }; + if queued_before_poll == 0 { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_IDLE_WAIT, + worker_poll_started, + ); + } else { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_POLL_WAIT, + worker_poll_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_JOB_WAIT, + worker_poll_started, + ); + } + let mut batch = vec![job]; + let batch_window = rename_tail_cleanup_defer_batch_window(); + let queued_snapshot = queued.load(Ordering::Relaxed); + let oldest_snapshot = *oldest_enqueued_at.lock().await; + let batch_max = rename_tail_cleanup_defer_batch_target( + queued_snapshot, + oldest_snapshot.map(|oldest| oldest.elapsed()), + ); + if batch_max > 1 && !batch_window.is_zero() { + let window_started = rustfs_io_metrics::put_stage_timer(); + tokio::time::sleep(batch_window).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BATCH_WINDOW, + window_started, + ); + } + while batch.len() < batch_max { + let next = { + let mut locked = receiver.lock().await; + locked.try_recv().ok() + }; + let Some(next) = next else { + break; + }; + batch.push(next); + } + let queued_after_dequeue = queued.fetch_sub(batch.len(), Ordering::Relaxed).saturating_sub(batch.len()); + if queued_after_dequeue == 0 { + *oldest_enqueued_at.lock().await = None; + } + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH, + queued_after_dequeue as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH_MAX_SAMPLE, + queued_after_dequeue as f64, + ); + if let Some(oldest) = *oldest_enqueued_at.lock().await { + let oldest_age_ms = oldest.elapsed().as_secs_f64() * 1000.0; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE, + oldest_age_ms, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE_MAX_SAMPLE, + oldest_age_ms, + ); + } + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BATCH_SIZE, + batch.len() as f64, + ); + if batch.len() > 1 { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BURST_BATCH_SIZE, + batch.len() as f64, + ); + } + let active = active_workers.fetch_add(1, Ordering::Relaxed) + 1; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_ACTIVE, + active as f64, + ); + for job in batch { + run_rename_tail_cleanup_job(job).await; + } + active_workers.fetch_sub(1, Ordering::Relaxed); + } + }); + } + } + Self { + sender, + queued, + oldest_enqueued_at, + } + } + + async fn try_enqueue(&self, job: RenameTailCleanupJob) -> std::result::Result<(), RenameTailCleanupJob> { + let queue_push_started = rustfs_io_metrics::put_stage_timer(); + { + let mut oldest = self.oldest_enqueued_at.lock().await; + if oldest.is_none() { + *oldest = Some(job.enqueued_at); + } + } + let queued_after_enqueue = self.queued.fetch_add(1, Ordering::Relaxed) + 1; + if rename_tail_cleanup_defer_hold_worker_enabled() { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_ENQUEUED, + 1.0, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER, + 1.0, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH, + queued_after_enqueue as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH_MAX_SAMPLE, + queued_after_enqueue as f64, + ); + if let Some(oldest) = *self.oldest_enqueued_at.lock().await { + let oldest_age_ms = oldest.elapsed().as_secs_f64() * 1000.0; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE, + oldest_age_ms, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE_MAX_SAMPLE, + oldest_age_ms, + ); + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_PUSH, + queue_push_started, + ); + return Ok(()); + } + match self.sender.try_send(job) { + Ok(()) => { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_ENQUEUED, + 1.0, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH, + queued_after_enqueue as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH_MAX_SAMPLE, + queued_after_enqueue as f64, + ); + if let Some(oldest) = *self.oldest_enqueued_at.lock().await { + let oldest_age_ms = oldest.elapsed().as_secs_f64() * 1000.0; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE, + oldest_age_ms, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE_MAX_SAMPLE, + oldest_age_ms, + ); + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_PUSH, + queue_push_started, + ); + Ok(()) + } + Err(err) => { + self.queued.fetch_sub(1, Ordering::Relaxed); + if self.queued.load(Ordering::Relaxed) == 0 { + *self.oldest_enqueued_at.lock().await = None; + } + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_FALLBACK_SYNC, + 1.0, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH, + self.queued.load(Ordering::Relaxed) as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH_MAX_SAMPLE, + self.queued.load(Ordering::Relaxed) as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_PUSH, + queue_push_started, + ); + Err(err.into_inner()) + } + } + } +} + +fn rename_tail_cleanup_queue() -> &'static RenameTailCleanupQueue { + static QUEUE: OnceLock = OnceLock::new(); + QUEUE.get_or_init(RenameTailCleanupQueue::new) +} + const EVENT_SET_DISK_RENAME_ROLLBACK: &str = "set_disk_rename_rollback"; #[derive(Clone, Copy)] @@ -2565,10 +3077,9 @@ async fn inspect_incomplete_rename_rollback( request.recreate_missing = Some(false); request.update_parity = Some(false); request.recursive = Some(false); - match tokio::time::timeout(Duration::from_secs(1), submitter(request)).await { - Ok(result) => result, - Err(_) => ReadRepairAdmissionOutcome::Failed("rollback inspection admission timed out".to_string()), - } + tokio::time::timeout(Duration::from_secs(1), submitter(request)) + .await + .unwrap_or_else(|_| ReadRepairAdmissionOutcome::Failed("rollback inspection admission timed out".to_string())) } fn rename_rollback_task_outcome( @@ -2865,21 +3376,24 @@ pub(in crate::set_disk) async fn finish_rename_tail_heal< SubmitFuture, >( tail_drain: tokio::task::JoinHandle>, - guard_release: tokio::sync::oneshot::Receiver, + guard_release: oneshot::Receiver, guards: Guards, request: rustfs_heal_contracts::heal_channel::HealChannelRequest, finalize: Finalize, cleanup: Cleanup, submit: Submit, ) where - Guards: Send, + Guards: Send + 'static, Finalize: FnOnce() -> FinalizeFuture + Send, FinalizeFuture: Future + Send, Cleanup: FnOnce(Guards, Vec) -> CleanupFuture + Send, CleanupFuture: Future + Send, + Cleanup: 'static, + CleanupFuture: 'static, Submit: FnOnce(rustfs_heal_contracts::heal_channel::HealChannelRequest) -> SubmitFuture + Send, SubmitFuture: Future + Send, { + let tail_async_started = rustfs_io_metrics::put_stage_timer(); let (needs_heal, tail_cleanup, tail_complete) = match tail_drain.await { Ok(Some(outcome)) => (outcome.convergence.needs_heal(), outcome.cleanup, true), Ok(None) => { @@ -2908,16 +3422,80 @@ pub(in crate::set_disk) async fn finish_rename_tail_heal< (true, Vec::new(), false) } }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_ASYNC_WAIT, + tail_async_started, + ); // A dropped sender means the request continuation was cancelled. Cleanup // must still run; only the explicit `false` used by the hard-crash harness // suppresses all in-process continuation work. + let guard_release_wait_started = rustfs_io_metrics::put_stage_timer(); if matches!(guard_release.await, Ok(false)) { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_GUARD_RELEASE_WAIT, + guard_release_wait_started, + ); drop(guards); return; } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_GUARD_RELEASE_WAIT, + guard_release_wait_started, + ); + let finalize_started = rustfs_io_metrics::put_stage_timer(); finalize().await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_FINALIZE, + finalize_started, + ); if tail_complete { - cleanup(guards, tail_cleanup).await; + let cleanup_submit_started = rustfs_io_metrics::put_stage_timer(); + if rename_tail_cleanup_counterfactual_skip_enabled() { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP, + 1.0, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_SUBMIT, + cleanup_submit_started, + ); + drop(guards); + } else if rename_tail_cleanup_defer_enabled() { + let targets = tail_cleanup.len(); + let job = RenameTailCleanupJob { + enqueued_at: Instant::now(), + targets, + future: Box::pin(cleanup(guards, tail_cleanup)), + }; + if let Err(job) = rename_tail_cleanup_queue().try_enqueue(job).await { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_SUBMIT, + cleanup_submit_started, + ); + let cleanup_started = rustfs_io_metrics::put_stage_timer(); + job.future.await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP, + cleanup_started, + ); + } else { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_SUBMIT, + cleanup_submit_started, + ); + } + } else { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_SUBMIT, + cleanup_submit_started, + ); + let cleanup_started = rustfs_io_metrics::put_stage_timer(); + cleanup(guards, tail_cleanup).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP, + cleanup_started, + ); + } } else { // The outer coordinator may fail while a disk rename still owns the // staging source. Preserve it for later healing/GC instead of racing @@ -2925,7 +3503,12 @@ pub(in crate::set_disk) async fn finish_rename_tail_heal< drop(guards); } if needs_heal { + let heal_submit_started = rustfs_io_metrics::put_stage_timer(); submit(request).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_HEAL_SUBMIT, + heal_submit_started, + ); } } @@ -2979,8 +3562,8 @@ impl SetDisks { bucket: &str, object: &str, write_quorum: usize, - ) -> crate::error::Result<(Vec>, Vec>)> { - let fence_path = crate::disk::quota_mutation_fence_path(bucket, object); + ) -> Result<(Vec>, Vec>)> { + let fence_path = disk::quota_mutation_fence_path(bucket, object); let results = join_all(disks.iter().map(|disk| { let disk = disk.clone(); let fence_path = fence_path.clone(); @@ -3022,8 +3605,8 @@ impl SetDisks { bucket: &str, object: &str, write_quorum: usize, - ) -> crate::error::Result<()> { - let fence_path = crate::disk::quota_mutation_fence_path(bucket, object); + ) -> Result<()> { + let fence_path = disk::quota_mutation_fence_path(bucket, object); let results = join_all(disks.iter().zip(tokens).filter_map(|(disk, token)| { let disk = disk.as_ref()?.clone(); let token = (*token)?; @@ -3045,18 +3628,36 @@ impl SetDisks { } } + #[allow(clippy::too_many_arguments)] fn rename_data_commit_from_observations( - disks: &[Option], - file_infos: &[FileInfo], - disk_versions: &[Option>], - errs: &[Option], - cleanup_data_dirs: &[Option], - old_current_sizes: &[Option], + observations: RenameDataObservations<'_>, write_quorum: usize, ) -> disk::error::Result { + let RenameDataObservations { + disks, + file_infos, + disk_versions, + errs, + cleanup_data_dirs, + old_current_sizes, + old_current_source_checked, + old_current_sources, + } = observations; let data_dir = Self::reduce_common_data_dir(cleanup_data_dirs, write_quorum); let convergence = Self::classify_rename_convergence(disk_versions, errs); let old_current_size = Self::reduce_common_old_current_size(old_current_sizes, write_quorum); + let old_current_source_quorum_checked = + old_current_source_checked.iter().filter(|checked| **checked).count() >= write_quorum; + let old_current_source_all_successful_checked = errs + .iter() + .enumerate() + .filter(|(_, err)| err.is_none()) + .all(|(idx, _)| old_current_source_checked.get(idx).copied().unwrap_or(false)); + let old_current_source = if old_current_source_quorum_checked { + Self::reduce_common_old_current_source(old_current_sources, write_quorum) + } else { + None + }; let online_disks = Self::eval_disks(disks, errs); let committed_slot = online_disks.iter().position(Option::is_some).ok_or(DiskError::Unexpected)?; let committed_file_info = file_infos.get(committed_slot).cloned().ok_or(DiskError::Unexpected)?; @@ -3084,6 +3685,9 @@ impl SetDisks { data_dir, cleanup_disks, old_current_size, + old_current_source_checked: old_current_source_quorum_checked, + old_current_source_all_successful_checked, + old_current_source, committed_file_info, tail_drain: None, }) @@ -3202,9 +3806,10 @@ impl SetDisks { let src_object = Arc::new(src_object.to_string()); let dst_bucket = Arc::new(dst_bucket.to_string()); let dst_object = Arc::new(dst_object.to_string()); - let (commit_tx, commit_rx) = tokio::sync::oneshot::channel(); + let (commit_tx, commit_rx) = oneshot::channel(); let coordinator_failure_receipt = rollback_receipt.clone(); + let early_ack_spawn_to_quorum_wait_started = rustfs_io_metrics::put_stage_timer(); let tail_drain = tokio::spawn({ let fanout_src_bucket = src_bucket.clone(); let fanout_src_object = src_object.clone(); @@ -3237,6 +3842,11 @@ impl SetDisks { let Some(disk) = disk else { return Err(DiskError::DiskNotFound); }; + let disk_wait_location_stage = if disk.is_local() { + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_LOCAL + } else { + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_REMOTE + }; let is_delete_marker = file_info.is_canonical_delete_marker(); // Clone FileInfo and set erasure.index for this disk @@ -3256,6 +3866,7 @@ impl SetDisks { let disk_wait_started = rustfs_io_metrics::put_stage_timer(); dispatch_state = RenameDispatchState::MayHavePublished; + let _foreground_active_guard = rename_foreground_active_guard(i); let observed = disk .rename_data_borrowed_with_fence_observed( &src_bucket, @@ -3283,6 +3894,7 @@ impl SetDisks { rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT, duration_ms, ); + rustfs_io_metrics::record_put_object_stage_duration(disk_wait_location_stage, duration_ms); let position = if result.is_ok() { let rank = successful_rename_completion_rank .as_ref() @@ -3321,9 +3933,12 @@ impl SetDisks { let mut data_dirs = vec![None; disk_count]; let mut cleanup_data_dirs = vec![None; disk_count]; let mut old_current_sizes = vec![None; disk_count]; + let mut old_current_source_checked = vec![false; disk_count]; + let mut old_current_sources = vec![None; disk_count]; let mut capacity_scope_generation = None; let mut cleanup_at_snapshot = vec![false; disk_count]; + let mut tail_after_quorum_started = None; while let Some(joined) = tasks.join_next().await { results_seen += 1; match joined { @@ -3333,6 +3948,8 @@ impl SetDisks { cleanup_data_dirs[idx] = res.cleanup_data_dir; disk_versions[idx] = res.sign; old_current_sizes[idx] = res.old_current_size; + old_current_source_checked[idx] = res.old_current_source_checked; + old_current_sources[idx] = res.old_current_source; errs[idx] = None; success_count += 1; } @@ -3352,12 +3969,16 @@ impl SetDisks { if !sent_commit && success_count >= write_quorum { let snapshot_commit = Self::rename_data_commit_from_observations( - &coordinator_disks, - &file_infos, - &disk_versions, - &errs, - &cleanup_data_dirs, - &old_current_sizes, + RenameDataObservations { + disks: &coordinator_disks, + file_infos: &file_infos, + disk_versions: &disk_versions, + errs: &errs, + cleanup_data_dirs: &cleanup_data_dirs, + old_current_sizes: &old_current_sizes, + old_current_source_checked: &old_current_source_checked, + old_current_sources: &old_current_sources, + }, write_quorum, ); if let Ok(commit) = snapshot_commit.as_ref() { @@ -3368,8 +3989,13 @@ impl SetDisks { let _ = commit_tx.send(snapshot_commit); } sent_commit = true; + tail_after_quorum_started = rustfs_io_metrics::put_stage_timer(); } } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_AFTER_QUORUM_WAIT, + tail_after_quorum_started, + ); #[cfg(test)] rollback_fault_injection::after_fanout(&fanout_dst_object); @@ -3489,6 +4115,10 @@ impl SetDisks { } }); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_EARLY_ACK_SPAWN_TO_QUORUM_WAIT, + early_ack_spawn_to_quorum_wait_started, + ); let quorum_wait_started = rustfs_io_metrics::put_stage_timer(); let commit = match commit_rx.await { Ok(commit) => commit, @@ -3501,8 +4131,13 @@ impl SetDisks { rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_QUORUM_WAIT, quorum_wait_started, ); + let attach_tail_owner_started = rustfs_io_metrics::put_stage_timer(); commit.map(|mut commit| { commit.tail_drain = Some(tail_drain); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_EARLY_ACK_ATTACH_TAIL_OWNER, + attach_tail_owner_started, + ); commit }) } @@ -3626,6 +4261,11 @@ impl SetDisks { let Some(disk) = disk else { return Err(DiskError::DiskNotFound); }; + let disk_wait_location_stage = if disk.is_local() { + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_LOCAL + } else { + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_REMOTE + }; let is_delete_marker = file_info.is_canonical_delete_marker(); let mut local_file_info; @@ -3659,6 +4299,7 @@ impl SetDisks { let disk_wait_started = rustfs_io_metrics::put_stage_timer(); dispatch_state = RenameDispatchState::MayHavePublished; + let _foreground_active_guard = rename_foreground_active_guard(i); let observed = disk .rename_data_borrowed_with_fence_observed( &src_bucket, @@ -3686,6 +4327,7 @@ impl SetDisks { rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT, duration_ms, ); + rustfs_io_metrics::record_put_object_stage_duration(disk_wait_location_stage, duration_ms); let position = if result.is_ok() { let rank = successful_rename_completion_rank .as_ref() @@ -3721,9 +4363,16 @@ impl SetDisks { let mut data_dirs = vec![None; disk_count]; let mut cleanup_data_dirs = vec![None; disk_count]; let mut old_current_sizes = vec![None; disk_count]; + let mut old_current_source_checked = vec![false; disk_count]; + let mut old_current_sources = vec![None; disk_count]; let quorum_wait_started = rustfs_io_metrics::put_stage_timer(); + let sync_full_fanout_started = rustfs_io_metrics::put_stage_timer(); let fanout_result = fanout.await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_SYNC_FULL_FANOUT_WAIT, + sync_full_fanout_started, + ); rustfs_io_metrics::record_put_object_stage_duration_from( rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_QUORUM_WAIT, quorum_wait_started, @@ -3764,6 +4413,8 @@ impl SetDisks { cleanup_data_dirs[idx] = res.cleanup_data_dir; disk_versions[idx].clone_from(&res.sign); old_current_sizes[idx] = res.old_current_size; + old_current_source_checked[idx] = res.old_current_source_checked; + old_current_sources[idx] = res.old_current_source.clone(); errs.push(None); } Ok(Err(e)) => { @@ -3900,6 +4551,18 @@ impl SetDisks { let data_dir = Self::reduce_common_data_dir(&cleanup_data_dirs, write_quorum); let convergence = Self::classify_rename_convergence(&disk_versions, &errs); let old_current_size = Self::reduce_common_old_current_size(&old_current_sizes, write_quorum); + let old_current_source_quorum_checked = + old_current_source_checked.iter().filter(|checked| **checked).count() >= write_quorum; + let old_current_source_all_successful_checked = errs + .iter() + .enumerate() + .filter(|(_, err)| err.is_none()) + .all(|(idx, _)| old_current_source_checked.get(idx).copied().unwrap_or(false)); + let old_current_source = if old_current_source_quorum_checked { + Self::reduce_common_old_current_source(&old_current_sources, write_quorum) + } else { + None + }; let online_disks = Self::eval_disks(disks, &errs); let committed_slot = online_disks.iter().position(Option::is_some).ok_or(DiskError::Unexpected)?; let committed_file_info = std::mem::take(&mut file_infos[committed_slot]); @@ -3927,11 +4590,34 @@ impl SetDisks { data_dir, cleanup_disks, old_current_size, + old_current_source_checked: old_current_source_quorum_checked, + old_current_source_all_successful_checked, + old_current_source, committed_file_info, tail_drain: None, }) } + pub(in crate::set_disk) fn reduce_common_old_current_source( + old_current_sources: &[Option], + write_quorum: usize, + ) -> Option { + let mut counts: HashMap<[u8; 32], (usize, FileInfo)> = HashMap::new(); + + for source in old_current_sources.iter().flatten() { + let entry = counts + .entry(Self::file_info_quorum_hash(source)) + .or_insert_with(|| (0, source.clone())); + entry.0 += 1; + } + + counts + .into_values() + .filter(|(count, _)| *count >= write_quorum) + .max_by_key(|(count, _)| *count) + .map(|(_, source)| source) + } + /// rustfs/backlog#1009: reduce the per-disk observations of the /// destination's previous current version to one set-level value, mirroring /// `reduce_common_data_dir`: the observation reported by at least @@ -4997,6 +5683,7 @@ impl SetDisks { fi.set_tier_free_version_id(&Uuid::new_v4().to_string()); let disks = self.get_disks_internal().await; + self.invalidate_get_object_metadata_cache(bucket, object).await; let mut futures = Vec::with_capacity(disks.len()); for (disk_index, disk_op) in disks.iter().enumerate() { @@ -5077,6 +5764,9 @@ impl SetDisks { } else { false }; + if absent { + self.invalidate_get_object_metadata_cache(bucket, object).await; + } Ok((m, absent)) } @@ -5525,12 +6215,10 @@ impl SetDisks { /// * The set of referenced data dirs is the UNION of `get_data_dirs()` across /// every online disk's `xl.meta`, so a dir named by *any* replica is kept. /// * If a disk holds the object directory but its `xl.meta` is missing or - /// unparsable, the object is treated as degraded and unmarked data dirs are - /// never removed. A data dir carrying a committed delete-transaction marker - /// remains reclaimable after a downgrade/re-upgrade cleanup interruption. - /// * Only subdirectories whose names parse as a UUID are ever considered; - /// removal is non-recursive-safe via a recursive delete of the full stray - /// data-dir path only. + /// unparsable, leave its data to quorum-aware recovery and healing. + /// * Any outstanding delete transaction defers all candidates to recovery. + /// * UUID-shaped S3 prefixes are protected: only regular part files and native + /// sidecars qualify. The final mutation repeats this check and never recurses. /// /// Returns the number of stray data directories removed across the set. This /// is best-effort maintenance: individual delete failures are logged and @@ -5545,13 +6233,16 @@ impl SetDisks { async fn reclaim_orphan_data_dirs_inner(&self, bucket: &str, object: &str, dry_run: bool) -> disk::error::Result { let disks = self.get_disks_internal().await; + if disks.len() != self.set_drive_count || disks.iter().any(Option::is_none) { + // An unavailable participant can still reference an apparent orphan. + return Ok(0); + } // Phase 1 (read-only): build the referenced-data-dir union and record the // physical UUID subdirectories present on each disk. Abort on any degraded // copy so a healable object is never stripped of a referenced data dir. let mut referenced: HashSet = HashSet::new(); - let mut per_disk_dirs: Vec<(usize, Vec<(Uuid, bool)>)> = Vec::new(); - let mut healthy_metas = 0usize; + let mut per_disk_dirs: Vec<(usize, Vec)> = Vec::new(); for (i, disk) in disks.iter().enumerate() { let Some(disk) = disk else { continue }; @@ -5586,22 +6277,6 @@ impl SetDisks { // to the orphan-dir / dangling-object heal paths. continue; } - let mut committed = Vec::with_capacity(physical.len()); - for dir in physical { - let data_dir = format!("{object}/{dir}"); - let committed_delete = disk.list_dir("", bucket, &data_dir, 0).await.is_ok_and(|entries| { - entries.iter().any(|entry| { - entry - .strip_prefix(DELETE_DATA_DIR_MARKER_PREFIX) - .is_some_and(|transaction| Uuid::parse_str(transaction).is_ok()) - }) - }); - committed.push((dir, committed_delete)); - } - if committed.iter().all(|(_, committed_delete)| *committed_delete) { - per_disk_dirs.push((i, committed)); - continue; - } warn!( target: "rustfs_ecstore::set_disk", bucket, object, @@ -5636,21 +6311,39 @@ impl SetDisks { } } - healthy_metas += 1; if !physical.is_empty() { - per_disk_dirs.push((i, physical.into_iter().map(|dir| (dir, false)).collect())); + per_disk_dirs.push((i, physical)); } } - // Phase 2: delete every physical data dir not referenced by the union. + if !per_disk_dirs + .iter() + .any(|(_, dirs)| dirs.iter().any(|dir| !referenced.contains(dir))) + { + return Ok(0); + } + // A neighboring journal can still need an unreferenced payload for + // rollback. Objects whose physical dirs are all referenced pay no + // recovery-discovery overhead. + for disk in disks.iter().flatten() { + match disk.has_delete_transaction_evidence(bucket, object).await { + Ok(false) => {} + _ => return Ok(0), + } + } + + // Phase 2: classify candidates consistently for dry-run and reclamation. let mut removed = 0usize; for (i, physical) in per_disk_dirs { let Some(disk) = disks[i].as_ref() else { continue }; - for (dir, committed_delete) in physical { - if referenced.contains(&dir) || (healthy_metas == 0 && !committed_delete) { + for dir in physical { + if referenced.contains(&dir) { continue; } let stray = format!("{object}/{dir}"); + if !disk.is_plain_orphan_data_dir(bucket, &stray).await.unwrap_or(false) { + continue; + } if dry_run { removed += 1; debug!( @@ -5666,18 +6359,18 @@ impl SetDisks { continue; } match disk - .delete( + .delete_data_dir( bucket, &stray, DeleteOptions { - recursive: true, + orphan_data_dir: true, immediate: true, ..Default::default() }, ) .await { - Ok(()) => { + Ok(DataDirDeleteStatus::Deleted) => { removed += 1; info!( target: "rustfs_ecstore::set_disk", @@ -5685,7 +6378,8 @@ impl SetDisks { "reclaim_orphan_data_dirs: removed orphaned data directory" ); } - Err(DiskError::FileNotFound | DiskError::VolumeNotFound) => {} + Ok(DataDirDeleteStatus::Deferred) + | Err(DiskError::FileNotFound | DiskError::VolumeNotFound | DiskError::DiskOngoingReq) => {} Err(err) => { warn!( target: "rustfs_ecstore::set_disk", @@ -5718,6 +6412,45 @@ impl SetDisks { } } + pub(crate) fn evaluate_write_precondition( + oi: Result, + http_preconditions: &HTTPPreconditions, + bucket: &str, + object: &str, + opts: &ObjectOptions, + ) -> Option { + match oi { + Ok(oi) => { + // Ordinary writes may proceed past a top-level delete marker; + // data movement and guarded internal writes must preserve it. + if oi.delete_marker { + if opts.data_movement || opts.preserve_delete_marker { + return Some(StorageError::PreconditionFailed); + } + return Self::write_precondition_lookup_error( + StorageError::ObjectNotFound(bucket.to_string(), object.to_string()), + http_preconditions, + bucket, + object, + ); + } + let if_none_match = http_preconditions.if_none_match_value().map(str::to_owned); + let if_match = http_preconditions.if_match_value().map(str::to_owned); + if should_prevent_write(&oi, if_none_match, if_match) + && !crate::data_movement::can_replace_stale_data_movement_target(&oi, opts) + { + return Some(StorageError::PreconditionFailed); + } + } + + Err(error) => { + return Self::write_precondition_lookup_error(error, http_preconditions, bucket, object); + } + } + + None + } + pub(in crate::set_disk) async fn check_write_precondition( &self, bucket: &str, @@ -5733,30 +6466,19 @@ impl SetDisks { // - If no_lock is false, we must have obtained the lock out side of this function // - If no_lock is true, we should not obtain locks lookup_opts.no_lock = true; - let oi = self.get_object_info(bucket, object, &lookup_opts).await; - - match oi { - Ok(oi) => { - // Ordinary writes may proceed past a top-level delete marker; - // data movement and guarded internal writes must preserve it. - if oi.delete_marker { - return (opts.data_movement || opts.preserve_delete_marker).then_some(StorageError::PreconditionFailed); - } - let if_none_match = http_preconditions.if_none_match_value().map(str::to_owned); - let if_match = http_preconditions.if_match_value().map(str::to_owned); - if should_prevent_write(&oi, if_none_match, if_match) - && !crate::data_movement::can_replace_stale_data_movement_target(&oi, opts) - { - return Some(StorageError::PreconditionFailed); - } - } - - Err(error) => { - return Self::write_precondition_lookup_error(error, &http_preconditions, bucket, object); - } - } - - None + let lookup = self.get_object_info(bucket, object, &lookup_opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let lookup = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::WritePreconditionRead, + async { + crate::cluster::rpc::runtime_sources::observe_read_version_context_boundary( + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::WritePreconditionLookup, + ); + lookup.await + }, + ); + let oi = lookup.await; + Self::evaluate_write_precondition(oi, &http_preconditions, bucket, object, opts) } } @@ -6088,10 +6810,20 @@ pub(crate) mod disk_call_counters { pub const KIND_READ_VERSION: &str = "read_version"; pub const KIND_BATCH_READ_VERSION: &str = "batch_read_version"; pub const KIND_METADATA_SLOWTAIL_FAULT: &str = "metadata_slowtail_fault"; + pub const KIND_INLINE_SHARD_OPEN: &str = "inline_shard_open"; + pub const KIND_DISK_SHARD_OPEN: &str = "disk_shard_open"; + /// Body verification, direct shard assembly, and erasure decoder entry points. + pub const KIND_BODY_DECODE: &str = "body_decode"; /// Registry key: (object, kind, disk_index). type CountKey = (String, String, usize); + #[cfg(feature = "diagnostic-read-version-caller")] + type ReadVersionContext = ( + Option<&'static str>, + Option, + ); + #[derive(Default)] struct Registry { /// Object names with an active observing scope. Only these accumulate, @@ -6099,6 +6831,8 @@ pub(crate) mod disk_call_counters { observed: HashSet, /// (object, kind, disk_index) -> call count. counts: HashMap, + #[cfg(feature = "diagnostic-read-version-caller")] + read_version_contexts: HashMap<(String, usize), Vec>, } fn registry() -> &'static Mutex { @@ -6116,6 +6850,39 @@ pub(crate) mod disk_call_counters { *reg.counts .entry((object.to_string(), kind.to_string(), disk_index)) .or_insert(0) += 1; + #[cfg(feature = "diagnostic-read-version-caller")] + if kind == KIND_READ_VERSION { + let context = ( + rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(), + crate::cluster::rpc::runtime_sources::current_read_version_request_caller(), + ); + reg.read_version_contexts + .entry((object.to_string(), disk_index)) + .or_default() + .push(context); + } + } + + /// Matches shard paths to the observed object's namespace path. This also + /// records deferred opens running in spawned decoder tasks. + pub(crate) fn record_shard_open(path: &str, inline: bool) { + let mut reg = registry().lock().expect("disk call-counter registry poisoned"); + let objects: Vec<_> = reg + .observed + .iter() + .filter(|object| { + path == object.as_str() || path.strip_prefix(object.as_str()).is_some_and(|rest| rest.starts_with('/')) + }) + .cloned() + .collect(); + let kind = if inline { + KIND_INLINE_SHARD_OPEN + } else { + KIND_DISK_SHARD_OPEN + }; + for object in objects { + *reg.counts.entry((object, kind.to_owned(), 0)).or_insert(0) += 1; + } } /// RAII scope that observes call counts for a single `object`. Counts recorded @@ -6136,6 +6903,15 @@ pub(crate) mod disk_call_counters { } impl CallCounterScope { + #[cfg(feature = "diagnostic-read-version-caller")] + pub fn read_version_contexts_for_disk(&self, disk_index: usize) -> Vec { + let reg = registry().lock().expect("disk call-counter registry poisoned"); + reg.read_version_contexts + .get(&(self.object.clone(), disk_index)) + .cloned() + .unwrap_or_default() + } + /// Total number of `kind` calls across all disks for the observed object. pub fn total(&self, kind: &str) -> u64 { let reg = registry().lock().expect("disk call-counter registry poisoned"); @@ -6161,6 +6937,8 @@ pub(crate) mod disk_call_counters { let mut reg = registry().lock().expect("disk call-counter registry poisoned"); reg.observed.remove(&self.object); reg.counts.retain(|(obj, _, _), _| obj != &self.object); + #[cfg(feature = "diagnostic-read-version-caller")] + reg.read_version_contexts.retain(|(obj, _), _| obj != &self.object); } } } @@ -6546,9 +7324,9 @@ mod tests { ); scope.try_begin().expect("delete scope should enter flight"); let scope_guard = crate::object_api::ScannerPublicationCommitScopeGuard::new(scope.clone()); - let (started_tx, started_rx) = tokio::sync::oneshot::channel(); - let (release_tx, release_rx) = tokio::sync::oneshot::channel(); - let (finished_tx, finished_rx) = tokio::sync::oneshot::channel(); + let (started_tx, started_rx) = oneshot::channel(); + let (release_tx, release_rx) = oneshot::channel(); + let (finished_tx, finished_rx) = oneshot::channel(); let waiter = tokio::spawn(run_scanner_publication_delete_owner(Some(scope.clone()), move || async move { started_tx.send(()).expect("delete owner should start"); @@ -7302,6 +8080,290 @@ mod tests { drop(dirs); } + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_coalescer_restores_each_item_after_delayed_and_threshold_flush() { + use crate::cluster::rpc::runtime_sources::{ + ReadVersionDiagnosticContext, with_optional_read_version_request_caller, with_read_version_diagnostic_context, + with_read_version_entry, + }; + use crate::set_disk::core::metadata_read::enqueue_coalesced_read_version; + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + use rustfs_io_metrics::internode_metrics::{ + ReadVersionContextBoundary, ReadVersionEntrySource, ReadVersionRequestCaller, + }; + use rustfs_io_metrics::s3_http_metrics::{ + S3HttpRequestGuard, current_s3_http_operation_context, with_isolated_s3_http_operation_context, + }; + let operation = |op| { + let mut guard = S3HttpRequestGuard::new("GET"); + guard.in_scope(|| { + rustfs_io_metrics::record_s3_op(op); + current_s3_http_operation_context().expect("operation should be captured") + }) + }; + let get = operation(rustfs_s3_ops::S3Operation::GetObject); + let list = operation(rustfs_s3_ops::S3Operation::ListObjectsV2); + let foreign = operation(rustfs_s3_ops::S3Operation::DeleteObject); + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + for (items, delay) in [(4, "2000"), (crate::disk::BATCH_READ_VERSION_MAX_ITEMS, "10000")] { + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + let bucket = "diagnostic-coalescer-context"; + let (_dirs, disks) = call_counter_local_disks(bucket, 1).await; + let disk = disks[0].clone().expect("test disk should exist"); + for case in 0..4 { + install_metadata_fanout_fileinfo(&disks, bucket, &format!("context-{case}"), None).await; + } + let mut contexts = Vec::new(); + for (api, caller, source) in [ + (Some(get), Some(ReadVersionRequestCaller::Object), ReadVersionEntrySource::ObjectRead), + (None, None, ReadVersionEntrySource::ConfigRead), + ( + Some(list), + Some(ReadVersionRequestCaller::ListObjects), + ReadVersionEntrySource::ListObjects, + ), + (None, Some(ReadVersionRequestCaller::Heal), ReadVersionEntrySource::LifecycleRecovery), + ] { + contexts.push( + with_isolated_s3_http_operation_context( + api, + with_optional_read_version_request_caller( + caller, + with_read_version_entry(source, async { ReadVersionDiagnosticContext::capture() }), + ), + ) + .await, + ); + } + let opts = ReadOptions::default(); + let foreign_context = with_isolated_s3_http_operation_context( + Some(foreign), + with_optional_read_version_request_caller( + Some(ReadVersionRequestCaller::Multipart), + with_read_version_entry(ReadVersionEntrySource::ObjectRead, async { + ReadVersionDiagnosticContext::capture() + }), + ), + ) + .await; + temp_env::async_with_vars([("RUSTFS_GET_METADATA_READ_VERSION_COALESCE_DELAY_MICROS", Some(delay))], async { + with_read_version_diagnostic_context(foreign_context, ReadVersionContextBoundary::Entry, async { + let reads = (0..items).map(|index| { + let case = index % contexts.len(); + let item = BatchReadVersionItem { + org_volume: String::new(), + volume: bucket.to_owned(), + path: format!("context-{case}"), + version_id: String::new(), + }; + with_read_version_diagnostic_context( + contexts[case], + ReadVersionContextBoundary::EarlyStop, + enqueue_coalesced_read_version(disk.clone(), item, &opts), + ) + }); + let results = futures::future::join_all(reads).await; + for (index, result) in results.into_iter().enumerate() { + assert_eq!(result.expect("fallback read should succeed").name, format!("context-{}", index % 4)); + } + assert_eq!( + current_s3_http_operation_context(), + Some(foreign), + "flush must restore its caller's ambient scope" + ); + }) + .await; + }) + .await; + assert_eq!(current_s3_http_operation_context(), None); + }) + }); + let rows = snapshots.snapshot().into_vec(); + let item_label = items.to_string(); + assert!( + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_get_metadata_read_version_coalescer_total" + && labels.get("event") == Some(&"attempted_batch") + && labels.get("item_count") == Some(&item_label.as_str()) + && matches!(value, DebugValue::Counter(1)) + }), + "all requests should flush in exactly one batch" + ); + for (source, api, caller) in [ + ("object_read", "s3:GetObject", "object"), + ("config_read", "unattributed", "unattributed"), + ("list_objects", "s3:ListObjectsV2", "list_objects"), + ("lifecycle_recovery", "unattributed", "heal"), + ] { + assert!(rows.iter().any(|(key, _, _, value)| { + let labels = key.key().labels().map(|label| (label.key(), label.value())).collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("boundary") == Some(&"unary_fallback") + && labels.get("entry_source") == Some(&source) + && labels.get("parent_api_operation") == Some(&api) + && labels.get("child_api_operation") == Some(&api) + && labels.get("parent_caller") == Some(&caller) + && labels.get("child_caller") == Some(&caller) + && matches!(value, DebugValue::Counter(count) if *count == u64::try_from(items / 4).expect("small item count")) + }), "each item must retain its own API, caller, and entry source"); + } + } + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn metadata_full_wait_preserves_external_request_diagnostic_context() { + use crate::cluster::rpc::runtime_sources::with_read_version_request_caller; + use rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller; + use rustfs_io_metrics::s3_http_metrics::{ + S3HttpRequestGuard, current_s3_http_operation_context, with_s3_http_operation_context, + }; + + const DISKS: usize = 4; + let bucket = "full-wait-external-context-bucket"; + let object = "full-wait-external-context-object"; + let (_dirs, disks) = call_counter_local_disks(bucket, DISKS).await; + install_metadata_fanout_fileinfo(&disks, bucket, object, None).await; + let calls = disk_call_counters::observe(object); + let mut request = S3HttpRequestGuard::new("GET"); + let operation = request + .in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::GetObject); + current_s3_http_operation_context() + }) + .expect("external request operation should be captured"); + let (metadata, errors, _) = with_s3_http_operation_context( + Some(operation), + with_read_version_request_caller( + ReadVersionRequestCaller::Object, + SetDisks::read_all_fileinfo_observed(&disks, "", bucket, object, "", false, false, false, false, 2), + ), + ) + .await + .expect("full-wait metadata read should complete"); + + assert_eq!(metadata.iter().filter(|info| info.name == object).count(), DISKS); + assert!(errors.iter().all(Option::is_none)); + for disk_index in 0..DISKS { + assert_eq!( + calls.read_version_contexts_for_disk(disk_index), + vec![(Some("s3:GetObject"), Some(ReadVersionRequestCaller::Object))], + "every full-wait child must retain the external request context" + ); + } + assert_eq!(current_s3_http_operation_context(), None); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn metadata_full_wait_keeps_internal_reads_without_s3_context() { + use crate::cluster::rpc::runtime_sources::with_optional_read_version_request_caller; + use rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller; + + const DISKS: usize = 4; + let bucket = "full-wait-internal-context-bucket"; + let (_dirs, disks) = call_counter_local_disks(bucket, DISKS).await; + for (case, caller) in [None, Some(ReadVersionRequestCaller::Heal)].into_iter().enumerate() { + let object = format!("full-wait-internal-context-object-{case}"); + install_metadata_fanout_fileinfo(&disks, bucket, &object, None).await; + let calls = disk_call_counters::observe(&object); + let (metadata, errors, _) = with_optional_read_version_request_caller( + caller, + SetDisks::read_all_fileinfo_observed(&disks, "", bucket, &object, "", false, false, false, false, 2), + ) + .await + .expect("internal full-wait metadata read should complete"); + + assert_eq!(metadata.iter().filter(|info| info.name == object).count(), DISKS); + assert!(errors.iter().all(Option::is_none)); + for disk_index in 0..DISKS { + assert_eq!( + calls.read_version_contexts_for_disk(disk_index), + vec![(None, caller)], + "internal reads must retain their caller without acquiring an external request context" + ); + } + } + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn metadata_early_stop_preserves_external_and_internal_diagnostic_context() { + use crate::cluster::rpc::runtime_sources::with_optional_read_version_request_caller; + use rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller; + use rustfs_io_metrics::s3_http_metrics::{ + S3HttpRequestGuard, current_s3_http_operation_context, with_s3_http_operation_context, + }; + + const DISKS: usize = 4; + let bucket = "early-stop-diagnostic-context-bucket"; + let (_dirs, disks) = call_counter_local_disks(bucket, DISKS).await; + let mut request = S3HttpRequestGuard::new("GET"); + let operation = request + .in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::GetObject); + current_s3_http_operation_context() + }) + .expect("external request operation should be captured"); + + temp_env::async_with_vars( + [ + ("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")), + ], + async { + for (case, api, caller) in [ + (0, Some(operation), Some(ReadVersionRequestCaller::Object)), + (1, None, None), + (2, None, Some(ReadVersionRequestCaller::Heal)), + ] { + let object = object_with_initial_data_shards(bucket, &format!("early-stop-context-object-{case}"), 2, 3); + install_inline_metadata_fanout_fileinfo(&disks, bucket, &object, b"context probe", |_| {}).await; + let calls = disk_call_counters::observe(&object); + let (metadata, errors, _) = with_s3_http_operation_context( + api, + with_optional_read_version_request_caller( + caller, + SetDisks::read_all_fileinfo_observed(&disks, "", bucket, &object, "", true, false, false, true, 2), + ), + ) + .await + .expect("early-stop metadata read should complete"); + + assert!(metadata.iter().filter(|info| info.name == object).count() >= 2); + assert!(errors.iter().all(Option::is_none)); + assert!(calls.total(disk_call_counters::KIND_READ_VERSION) >= 2); + let expected_api = api.map(|context| context.operation_name()); + for disk_index in 0..DISKS { + for context in calls.read_version_contexts_for_disk(disk_index) { + assert_eq!( + context, + (expected_api, caller), + "every polled early-stop child must retain its parent's diagnostic context" + ); + } + } + } + }, + ) + .await; + assert_eq!(current_s3_http_operation_context(), None); + } + #[test] fn batch_read_version_response_mapping_preserves_index_and_errors() { let expected_items = vec![ @@ -9006,7 +10068,7 @@ mod tests { #[allow(unreachable_code)] None }); - let (release, released) = tokio::sync::oneshot::channel::(); + let (release, released) = oneshot::channel::(); drop(release); finish_rename_tail_heal( @@ -10294,7 +11356,7 @@ mod tests { info.size = 11; info.parts.clear(); info.add_object_part(1, "new-etag".to_string(), 11, None, 11, None, None); - let crate::disk::Disk::Local(local) = disk.as_ref() else { + let disk::Disk::Local(local) = disk.as_ref() else { panic!("physical publication fixture requires local disks"); }; // Linux IO paths use a mount FD, which is also the namespace lock key. @@ -10302,7 +11364,7 @@ mod tests { .get_disk() .get_object_path_for_io(bucket, object) .expect("the publication path must resolve through the disk's mount lease"); - let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (entered_tx, entered_rx) = oneshot::channel(); let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); hooks.push(os::prepared_publication_test_hooks::install( &destination.join(STORAGE_FORMAT_FILE), @@ -10488,11 +11550,11 @@ mod tests { info.add_object_part(1, "new-etag".to_string(), 11, None, 11, None, None); } let disk = disks[3].as_ref().expect("tail disk"); - let crate::disk::Disk::Local(local) = disk.as_ref() else { + let disk::Disk::Local(local) = disk.as_ref() else { panic!("local fixture"); }; let destination = local.get_disk().get_object_path_for_io(bucket, object).expect("tail IO path"); - let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (entered_tx, entered_rx) = oneshot::channel(); let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); let _hook = os::prepared_publication_test_hooks::install(&destination.join(STORAGE_FORMAT_FILE), move || { let _ = entered_tx.send(()); @@ -10504,7 +11566,7 @@ mod tests { let receipt = RenameRollbackReceipt::default(); let caller_receipt = receipt.clone(); let caller_disks = disks.clone(); - let (ack_tx, ack_rx) = tokio::sync::oneshot::channel(); + let (ack_tx, ack_rx) = oneshot::channel(); let caller = tokio::spawn(async move { let commit = SetDisks::rename_data_owned_with_fence( &caller_disks, @@ -12093,6 +13155,9 @@ mod tests { let bucket = "dangling-invalid-meta-bucket"; let object = "object"; let set = io_primitives_test_set(vec![None, None, None, None], 1).await; + let cached_generation = set + .get_object_metadata_cache_generation(bucket, object) + .expect("local cache generation should be enabled"); let mut data_errs_by_part = HashMap::new(); data_errs_by_part.insert( 1, @@ -12127,6 +13192,10 @@ mod tests { .expect("invalid metadata should be cleanable when part results prove the object is dangling"); assert!(!deleted.is_valid()); + assert!( + !set.is_get_object_metadata_cache_generation_current(cached_generation), + "dangling cleanup must retire locally cached metadata before deletion" + ); } #[tokio::test] diff --git a/crates/ecstore/src/set_disk/core/metadata_read.rs b/crates/ecstore/src/set_disk/core/metadata_read.rs index ab1700b51..e3d318589 100644 --- a/crates/ecstore/src/set_disk/core/metadata_read.rs +++ b/crates/ecstore/src/set_disk/core/metadata_read.rs @@ -66,6 +66,7 @@ use tokio::task::JoinSet; pub(in crate::set_disk) struct MetadataReadResult { pub(in crate::set_disk) slots: Vec, pub(in crate::set_disk) diagnostics: MetadataFanoutDiagnostics, + pub(in crate::set_disk) condition_stopped: bool, } impl MetadataReadResult { @@ -170,6 +171,23 @@ fn read_version_coalescing_delay() -> Duration { struct CoalescedReadVersionRequest { item: BatchReadVersionItem, tx: oneshot::Sender>, + #[cfg(feature = "diagnostic-read-version-caller")] + context: crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext, +} + +impl CoalescedReadVersionRequest { + fn new(item: BatchReadVersionItem, tx: oneshot::Sender>) -> Self { + #[cfg(feature = "diagnostic-read-version-caller")] + let context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture(); + #[cfg(feature = "diagnostic-read-version-caller")] + context.observe(rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::CoalescerEnqueue); + Self { + item, + tx, + #[cfg(feature = "diagnostic-read-version-caller")] + context, + } + } } #[derive(Clone, Debug, Eq, PartialEq)] @@ -245,26 +263,49 @@ async fn read_version_via_coalescer( allow_coalescing: bool, ) -> disk::error::Result { if !allow_coalescing || !read_version_coalescing_enabled() { + #[cfg(feature = "diagnostic-read-version-caller")] + return crate::cluster::rpc::runtime_sources::with_read_version_request_caller_if_absent( + rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::MetadataRead, + disk.read_version(org_bucket, bucket, object, version_id, opts), + ) + .await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] return disk.read_version(org_bucket, bucket, object, version_id, opts).await; } if !matches!(disk.as_ref(), Disk::Remote(_)) { record_read_version_coalescer_event("bypass_non_remote", 1); + #[cfg(feature = "diagnostic-read-version-caller")] + return crate::cluster::rpc::runtime_sources::with_read_version_request_caller_if_absent( + rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::MetadataRead, + disk.read_version(org_bucket, bucket, object, version_id, opts), + ) + .await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] return disk.read_version(org_bucket, bucket, object, version_id, opts).await; } - let (tx, rx) = oneshot::channel(); let item = BatchReadVersionItem { org_volume: org_bucket.to_string(), volume: bucket.to_string(), path: object.to_string(), version_id: version_id.to_string(), }; + enqueue_coalesced_read_version(disk, item, opts).await +} + +pub(super) async fn enqueue_coalesced_read_version( + disk: DiskStore, + item: BatchReadVersionItem, + opts: &ReadOptions, +) -> disk::error::Result { + let (tx, rx) = oneshot::channel(); let lane_key = ReadVersionCoalescerKey::new(&disk, opts); + let request = CoalescedReadVersionRequest::new(item, tx); let pending = { let mut coalescer = read_version_coalescer().lock().await; let lane = coalescer.lanes.entry(lane_key).or_default(); let schedule_delayed_flush = lane.is_empty(); - lane.push(CoalescedReadVersionRequest { item, tx }); + lane.push(request); if lane.len() >= BATCH_READ_VERSION_MAX_ITEMS { coalescer.lanes.remove(&lane_key) } else if schedule_delayed_flush { @@ -325,20 +366,24 @@ async fn flush_read_version_coalescer_pending( let mut senders = Vec::with_capacity(pending.len()); let mut items = Vec::with_capacity(pending.len()); + #[cfg(feature = "diagnostic-read-version-caller")] + let mut contexts = Vec::with_capacity(pending.len()); for request in pending { senders.push(request.tx); items.push(request.item); + #[cfg(feature = "diagnostic-read-version-caller")] + contexts.push(request.context); } let expected_items = items.iter().map(ExpectedBatchReadVersionItem::from).collect::>(); record_read_version_coalescer_event("attempted_batch", items.len()); - let result = - match tokio::time::timeout(get_drive_metadata_timeout(), disk.batch_read_version(BatchReadVersionReq { items, opts })) - .await - { - Ok(result) => result, - Err(_) => Err(DiskError::Timeout), - }; + let batch = disk.batch_read_version(BatchReadVersionReq { items, opts }); + #[cfg(feature = "diagnostic-read-version-caller")] + let batch = crate::cluster::rpc::runtime_sources::with_read_version_batch_contexts(contexts, batch); + let result = match tokio::time::timeout(get_drive_metadata_timeout(), batch).await { + Ok(result) => result, + Err(_) => Err(DiskError::Timeout), + }; match result { Ok(responses) => { let map_started = batch_read_version_stage_timer(); @@ -575,6 +620,7 @@ impl MetadataFanoutDiagnostics { pub(in crate::set_disk) fn record_quorum_candidate_latency(&self, path: &'static str, read_quorum: usize) { if let Some(latency) = self.quorum_candidate_latency(read_quorum) { + rustfs_io_metrics::record_get_object_metadata_quorum_result(path, "reached"); rustfs_io_metrics::record_get_object_quorum_reached_latency(path, latency.as_secs_f64()); } } @@ -655,6 +701,8 @@ async fn inline_body_miss_reason_from_observations<'a>( return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY); }; + #[cfg(test)] + crate::set_disk::disk_call_counters::record(object, crate::set_disk::disk_call_counters::KIND_BODY_DECODE, 0); match try_read_inline_data_shards_direct(&mut readers, erasure.data_shards, read_length, object_size).await { Some(body) if body.len() == object_size => None, _ => Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY), @@ -1012,6 +1060,10 @@ impl SetDisks { let object: Arc = Arc::from(object); let version_id: Arc = Arc::from(version_id); let slowtail_fault = get_metadata_slowtail_fault_request(bucket.as_ref(), object.as_ref(), read_data); + #[cfg(feature = "diagnostic-read-version-caller")] + let diagnostic_context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::MetadataRead, + ); let futures = disks.iter().enumerate().map(|(disk_index, disk)| { let disk = disk.clone(); let task_opts = opts; @@ -1020,21 +1072,35 @@ impl SetDisks { let object = object.clone(); let version_id = version_id.clone(); let slowtail_fault = slowtail_fault.clone(); + #[cfg(feature = "diagnostic-read-version-caller")] + #[allow(clippy::redundant_locals)] + let diagnostic_context = diagnostic_context; AbortOnDropJoinHandle(tokio::spawn(async move { - let response_start = observe.then(Instant::now); - let result = if let Some(disk) = disk { - Self::record_read_version_call(&object, disk_index); - if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(disk_index)) { - Self::record_metadata_slowtail_fault(&object, disk_index); - tokio::time::sleep(delay).await; - } - read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing) - .await - } else { - Err(DiskError::DiskNotFound) + let read = async move { + let response_start = observe.then(Instant::now); + let result = if let Some(disk) = disk { + Self::record_read_version_call(&object, disk_index); + if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(disk_index)) { + Self::record_metadata_slowtail_fault(&object, disk_index); + tokio::time::sleep(delay).await; + } + read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing) + .await + } else { + Err(DiskError::DiskNotFound) + }; + let elapsed = response_start.map(|start| start.elapsed()); + (result, elapsed) }; - let elapsed = response_start.map(|start| start.elapsed()); - (result, elapsed) + #[cfg(feature = "diagnostic-read-version-caller")] + return crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context( + diagnostic_context, + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::FullWait, + read, + ) + .await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] + read.await })) }); @@ -1078,7 +1144,11 @@ impl SetDisks { 0, ); } - Ok(MetadataReadResult { slots, diagnostics }) + Ok(MetadataReadResult { + slots, + diagnostics, + condition_stopped: false, + }) } #[allow(clippy::too_many_arguments)] @@ -1121,10 +1191,15 @@ impl SetDisks { let mut next_fanout_index = 0usize; let mut scheduled_count = 0usize; let mut force_full_wait = false; + let read_condition_active = read_data && crate::object_api::get_object_read_condition_is_active(); let mut final_miss_reason_override = None; let mut non_inline_candidate_eligible = None; let mut single_pending_hedge_deadline = None; let slowtail_fault = get_metadata_slowtail_fault_request(bucket.as_ref(), object.as_ref(), read_data); + #[cfg(feature = "diagnostic-read-version-caller")] + let diagnostic_context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::MetadataRead, + ); let spawn_read_version = |join_set: &mut JoinSet<(usize, disk::error::Result, Duration)>, task_ids: &mut [Option], index: usize, @@ -1135,27 +1210,45 @@ impl SetDisks { let object = object.clone(); let version_id = version_id.clone(); let slowtail_fault = slowtail_fault.clone(); + #[cfg(feature = "diagnostic-read-version-caller")] + #[allow(clippy::redundant_locals)] + let diagnostic_context = diagnostic_context; let task = join_set.spawn(async move { - let response_start = Instant::now(); - let result = if let Some(disk) = disk { - #[cfg(test)] - let _fanout_task_guard = rename_fanout_barrier::task_guard(&object); - Self::record_read_version_call(&object, index); - #[cfg(test)] - Self::read_version_fanout_barrier(&object, index).await; - if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(index)) { - Self::record_metadata_slowtail_fault(&object, index); + let read = async move { + let response_start = Instant::now(); + let result = if let Some(disk) = disk { #[cfg(test)] - rename_fanout_barrier::checkpoint(&object, index, rename_fanout_barrier::PHASE_METADATA_SLOWTAIL_FAULT) + let _fanout_task_guard = rename_fanout_barrier::task_guard(&object); + Self::record_read_version_call(&object, index); + #[cfg(test)] + Self::read_version_fanout_barrier(&object, index).await; + if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(index)) { + Self::record_metadata_slowtail_fault(&object, index); + #[cfg(test)] + rename_fanout_barrier::checkpoint( + &object, + index, + rename_fanout_barrier::PHASE_METADATA_SLOWTAIL_FAULT, + ) .await; - tokio::time::sleep(delay).await; - } - read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing) - .await - } else { - Err(DiskError::DiskNotFound) + tokio::time::sleep(delay).await; + } + read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing) + .await + } else { + Err(DiskError::DiskNotFound) + }; + (index, result, response_start.elapsed()) }; - (index, result, response_start.elapsed()) + #[cfg(feature = "diagnostic-read-version-caller")] + return crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context( + diagnostic_context, + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::EarlyStop, + read, + ) + .await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] + read.await }); task_ids[index] = Some(task.id()); }; @@ -1258,12 +1351,28 @@ impl SetDisks { } } - if !force_full_wait + if (!force_full_wait || read_condition_active) && let Some(decision) = accumulator .early_stop_decision() .or_else(|| accumulator.version_early_stop_decision()) { - let should_return_early = if read_data { + let condition_stopped = read_condition_active + && decision.reason == crate::diagnostics::get::GET_METADATA_EARLY_STOP_REASON_VALID_QUORUM + && accumulator.candidate.as_ref().is_some_and(|candidate| { + !candidate.deleted + && crate::object_api::get_object_read_condition_is_terminal( + crate::object_api::GetObjectReadMetadata::from_file_info( + bucket.as_ref(), + object.as_ref(), + candidate, + ), + ) + }); + let should_return_early = if condition_stopped { + true + } else if force_full_wait { + false + } else if read_data { match accumulator.candidate.as_ref() { Some(_candidate) if non_inline_candidate_eligible == Some(true) => { accumulator.candidate_has_read_reserve() @@ -1332,7 +1441,11 @@ impl SetDisks { cancelled_count, ); let diagnostics = MetadataFanoutDiagnostics::new(fanout_start.elapsed(), observations); - return Ok(MetadataReadResult { slots, diagnostics }); + return Ok(MetadataReadResult { + slots, + diagnostics, + condition_stopped, + }); } } @@ -1410,7 +1523,11 @@ impl SetDisks { rustfs_io_metrics::record_get_object_metadata_early_stop_saved_responses(metrics_path, 0); rustfs_io_metrics::record_get_object_metadata_fanout_lifecycle(metrics_path, scheduled_count, scheduled_count, 0); let diagnostics = MetadataFanoutDiagnostics::new(fanout_start.elapsed(), observations); - Ok(MetadataReadResult { slots, diagnostics }) + Ok(MetadataReadResult { + slots, + diagnostics, + condition_stopped: false, + }) } /// Test-only seam that records one per-disk `read_version` metadata RPC for diff --git a/crates/ecstore/src/set_disk/metadata.rs b/crates/ecstore/src/set_disk/metadata.rs index 36e3bf6d6..6be934391 100644 --- a/crates/ecstore/src/set_disk/metadata.rs +++ b/crates/ecstore/src/set_disk/metadata.rs @@ -35,6 +35,58 @@ struct FileInfoIdentityGroup { } impl SetDisks { + /// Read one key's complete version stack without enumerating its parent. + pub(crate) async fn read_object_versions_for_lifecycle( + &self, + bucket: &str, + object: &str, + ) -> disk::error::Result> { + let disks = self.get_disks_internal().await; + if disks.len() != self.set_drive_count || disks.is_empty() { + return Err(DiskError::ErasureReadQuorum); + } + let disk_object = rustfs_utils::path::encode_dir_object(object); + let (mut raw, mut errors) = Self::read_all_raw_file_info(&disks, bucket, &disk_object, false).await; + for (metadata, error) in raw.iter_mut().zip(&mut errors) { + if metadata.as_ref().is_some_and(|metadata| metadata.buf.is_empty()) { + *metadata = None; + *error = Some(DiskError::FileCorrupt); + } + } + // An unavailable disk is not evidence that a version group is absent. + // Unlike a walk, an exact read must also decide the all-missing case. + if raw.iter().all(Option::is_none) { + return if errors.iter().all(|error| { + matches!( + error, + Some(DiskError::FileNotFound | DiskError::FileVersionNotFound | DiskError::VolumeNotFound) + ) + }) { + Ok(None) + } else { + Err(DiskError::ErasureReadQuorum) + }; + } + let entries = rustfs_filemeta::MetaCacheEntries( + raw.into_iter() + .map(|raw| { + raw.map(|raw| rustfs_filemeta::MetaCacheEntry { + name: object.to_owned(), + metadata: raw.buf, + cached: None, + reusable: false, + }) + }) + .collect(), + ); + // Reuse the listing's per-version authority, including the object's + // actual erasure layout. Never turn unreadable/stale metadata into a + // singleton made from the preceding write's ObjectInfo. + Self::resolve_listed_versions(bucket, entries, &errors, self.set_drive_count, self.default_parity_count)? + .map(Some) + .ok_or(DiskError::ErasureReadQuorum) + } + /// Resolve each listed version using the same metadata authority as an exact read. pub(crate) fn resolve_listed_versions( bucket: &str, diff --git a/crates/ecstore/src/set_disk/mod.rs b/crates/ecstore/src/set_disk/mod.rs index 17c7c3222..fcbea50b4 100644 --- a/crates/ecstore/src/set_disk/mod.rs +++ b/crates/ecstore/src/set_disk/mod.rs @@ -160,13 +160,14 @@ use rustfs_utils::http::{ use rustfs_utils::{ HashAlgorithm, crypto::hex, - path::{SLASH_SEPARATOR, encode_dir_object, encode_dir_object_ref, has_suffix, path_join_buf}, + path::{SLASH_SEPARATOR, encode_dir_object, encode_dir_object_ref, has_suffix, object_name_disk_identity, path_join_buf}, }; use sha2::Sha256; +use std::borrow::Cow; use std::hash::{BuildHasher, Hash, Hasher}; use std::mem::{self}; use std::pin::Pin; -use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::{Arc, OnceLock}; use std::task::{Context, Poll}; use std::time::{Instant, SystemTime, UNIX_EPOCH}; @@ -410,6 +411,12 @@ impl Drop for ObjectLockDiagGuard { } let hold = self.acquired_at.elapsed(); + if self.op == "put_object_commit" && self.mode == "write" && rustfs_io_metrics::put_stage_metrics_enabled() { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD, + hold.as_secs_f64() * 1000.0, + ); + } record_object_lock_diag_hold_duration(self.op, self.mode, hold); let threshold = get_object_lock_diag_slow_hold_threshold(); if hold >= threshold { @@ -793,6 +800,11 @@ const DEFAULT_RUSTFS_GET_METADATA_TWO_PHASE_READ_PLAN_ENABLE: bool = false; const ENV_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT: &str = "RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT"; const DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT: bool = true; +#[cfg(feature = "test-util")] +mod conditional_read_bench; +#[cfg(feature = "test-util")] +pub use conditional_read_bench::ConditionalReadBenchmarkMetadataGuard; + const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS"; const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DISKS: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DISKS"; const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_BUCKET: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_BUCKET"; @@ -874,6 +886,7 @@ mod ctx; mod metadata; mod ops; pub(crate) use ops::bucket::BucketInfoQuorum; +pub(crate) use ops::delete_recovery::{DeleteRecoveryContext, DeleteRecoverySummary}; pub(crate) use ops::heal::AbsenceProofRequest; #[cfg(test)] pub(crate) use ops::heal::DanglingDeleteFailure; @@ -920,16 +933,22 @@ pub use ops::heal_walk::HealWalkVersion; pub(in crate::set_disk) struct GetObjectFileInfo { owned: Option, shared: Option>, + condition_stopped: bool, + late_metadata_fanout: Option, } struct OwnedGetObjectFileInfo { fi: FileInfo, parts_metadata: Vec, online_disks: Vec>, - late_metadata_fanout_disks: Option>>, } -type OwnedGetObjectFileInfoParts = (FileInfo, Vec, Vec>, Option>>); +enum LateMetadataFanout { + CapturedDisks(Vec>), + RefreshCurrentSet, +} + +type OwnedGetObjectFileInfoParts = (FileInfo, Vec, Vec>, Option); impl GetObjectFileInfo { fn owned(fi: FileInfo, parts_metadata: Vec, online_disks: Vec>) -> Self { @@ -938,9 +957,10 @@ impl GetObjectFileInfo { fi, parts_metadata, online_disks, - late_metadata_fanout_disks: None, }), shared: None, + condition_stopped: false, + late_metadata_fanout: None, } } @@ -950,24 +970,27 @@ impl GetObjectFileInfo { online_disks: Vec>, late_metadata_fanout_disks: Vec>, ) -> Self { - Self { - owned: Some(OwnedGetObjectFileInfo { - fi, - parts_metadata, - online_disks, - late_metadata_fanout_disks: Some(late_metadata_fanout_disks), - }), - shared: None, - } + let mut snapshot = Self::owned(fi, parts_metadata, online_disks); + snapshot.late_metadata_fanout = Some(LateMetadataFanout::CapturedDisks(late_metadata_fanout_disks)); + snapshot } fn shared(entry: Arc) -> Self { + let condition_stopped = entry.capability == GetObjectMetadataCacheCapability::ConditionOnly; Self { owned: None, shared: Some(entry), + condition_stopped, + late_metadata_fanout: None, } } + fn shared_with_late_metadata_refresh(entry: Arc) -> Self { + let mut snapshot = Self::shared(entry); + snapshot.late_metadata_fanout = Some(LateMetadataFanout::RefreshCurrentSet); + snapshot + } + fn fi(&self) -> &FileInfo { match (&self.owned, &self.shared) { (Some(snapshot), None) => &snapshot.fi, @@ -993,9 +1016,7 @@ impl GetObjectFileInfo { } fn has_late_metadata_fanout(&self) -> bool { - self.owned - .as_ref() - .is_some_and(|snapshot| snapshot.late_metadata_fanout_disks.is_some()) + self.late_metadata_fanout.is_some() } fn into_owned(self) -> (FileInfo, Vec, Vec>) { @@ -1004,16 +1025,22 @@ impl GetObjectFileInfo { } fn into_owned_with_late_metadata_fanout(self) -> OwnedGetObjectFileInfoParts { - match (self.owned, self.shared) { - (Some(snapshot), None) => ( - snapshot.fi, - snapshot.parts_metadata, - snapshot.online_disks, - snapshot.late_metadata_fanout_disks, - ), + let Self { + owned, + shared, + late_metadata_fanout, + .. + } = self; + match (owned, shared) { + (Some(snapshot), None) => (snapshot.fi, snapshot.parts_metadata, snapshot.online_disks, late_metadata_fanout), (None, Some(entry)) => match Arc::try_unwrap(entry) { - Ok(entry) => (entry.fi, entry.parts_metadata, entry.online_disks, None), - Err(entry) => (entry.fi.clone(), entry.parts_metadata.clone(), entry.online_disks.clone(), None), + Ok(entry) => (entry.fi, entry.parts_metadata, entry.online_disks, late_metadata_fanout), + Err(entry) => ( + entry.fi.clone(), + entry.parts_metadata.clone(), + entry.online_disks.clone(), + late_metadata_fanout, + ), }, _ => unreachable!("GET metadata snapshot representation must be exclusive"), } @@ -1450,6 +1477,248 @@ mod prepared_get_object_metadata_tests { assert!(take_prepared_get_object_metadata().is_none()); } + struct StopReadForEtag(String); + impl crate::object_api::GetObjectReadCondition for StopReadForEtag { + fn is_terminal(&self, metadata: crate::object_api::GetObjectReadMetadata<'_>) -> bool { + !metadata.delete_marker && metadata.etag == Some(self.0.as_str()) + } + } + + #[tokio::test] + #[serial_test::serial(body_cache_hook)] + async fn read_condition_stops_on_quorum_before_slow_disk_and_body_verification() { + use crate::object_api::with_get_object_read_condition; + for size in [4096, 1_300_000] { + let (dirs, set) = make_local_set_disks(4, 2).await; + let bucket = "read-condition-quorum"; + let object = object_with_initial_data_shards(bucket, &format!("read-condition-{size}")); + let opts = ObjectOptions { + no_lock: true, + ..Default::default() + }; + set.make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let mut put = PutObjReader::from_vec(vec![b'v'; size]); + let info = set.put_object(bucket, &object, &mut put, &opts).await.expect("write object"); + for dir in &dirs { + let object_path = dir.path().join(bucket).join(&object); + if size == 4096 { + let path = object_path.join("xl.meta"); + let mut bytes = tokio::fs::read(&path).await.expect("read inline metadata"); + *bytes.last_mut().expect("inline payload") ^= 1; + tokio::fs::write(path, bytes).await.expect("damage inline data"); + } else { + tokio::fs::remove_dir_all(object_path.join(info.data_dir.expect("data directory").to_string())) + .await + .expect("remove shards"); + } + } + let slow = bounded_initial_parity_disk_index(bucket, &object); + let barrier = rename_fanout_barrier::arm(&object, slow, rename_fanout_barrier::PHASE_READ_VERSION); + let calls = disk_call_counters::observe(&object); + let read_opts = ObjectOptions { + metadata_cache_safe: true, + ..opts + }; + let followup_opts = read_opts.clone(); + let condition = Arc::new(StopReadForEtag(info.etag.clone().expect("committed ETag"))); + let set_for_read = Arc::clone(&set); + let object_for_read = object.clone(); + let mut read = tokio::spawn(async move { + with_get_object_read_condition( + condition, + set_for_read.get_object_reader(bucket, &object_for_read, None, HeaderMap::new(), &read_opts), + ) + .await + }); + tokio::time::timeout(READ_VERSION_BARRIER_GUARD, barrier.wait_until_paused()) + .await + .expect("slow metadata read is pending"); + let mut reader = tokio::time::timeout(READ_VERSION_BARRIER_GUARD, &mut read) + .await + .expect("condition must not wait for the slow disk or decode damaged data") + .expect("reader task") + .expect("metadata-only result"); + assert_eq!(reader.object_info.etag, info.etag); + assert_eq!( + calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN), + 0, + "damaged inline data was never opened" + ); + assert_eq!( + calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), + 0, + "missing shards were never opened" + ); + assert_eq!( + calls.total(disk_call_counters::KIND_BODY_DECODE), + 0, + "the body verifier and decoder were never entered" + ); + let mut body = Vec::new(); + assert!( + reader.stream.read_to_end(&mut body).await.is_err(), + "terminal metadata cannot be served as a successful body" + ); + assert!(body.is_empty()); + let first_calls = calls.total(disk_call_counters::KIND_READ_VERSION); + drop(reader); + barrier.release(); + let snapshot = set + .get_object_fileinfo_for_get_object_reader(bucket, &object, &followup_opts, true, true) + .await + .expect("fresh metadata after terminal scope"); + assert!(!snapshot.condition_stopped); + assert!( + calls.total(disk_call_counters::KIND_READ_VERSION) > first_calls, + "an incomplete terminal snapshot must not satisfy the next metadata read from cache" + ); + } + } + + #[tokio::test] + #[serial_test::serial(body_cache_hook)] + async fn read_condition_skips_shard_open_and_decode_with_live_positive_controls() { + for size in [4096, 1_300_000] { + let (_dirs, set) = make_local_set_disks(4, 2).await; + let bucket = "read-condition-body-counters"; + let object = object_with_initial_data_shards(bucket, &format!("body-counters-{size}")); + let opts = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let payload: Vec<_> = (0..size) + .map(|index| u8::try_from(index % 251).expect("pattern byte")) + .collect(); + set.make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let mut put = PutObjReader::from_vec(payload.clone()); + let info = set.put_object(bucket, &object, &mut put, &opts).await.expect("write object"); + let calls = disk_call_counters::observe(&object); + let reader = crate::object_api::with_get_object_read_condition( + Arc::new(StopReadForEtag(info.etag.expect("committed ETag"))), + set.get_object_reader(bucket, &object, None, HeaderMap::new(), &opts), + ) + .await + .expect("terminal metadata reader"); + drop(reader); + assert!(calls.total(disk_call_counters::KIND_READ_VERSION) > 0, "metadata really was read"); + assert_eq!( + calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN), + 0, + "no inline shard source opened" + ); + assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 0, "no disk shard source opened"); + assert_eq!( + calls.total(disk_call_counters::KIND_BODY_DECODE), + 0, + "no body verifier or decoder entered" + ); + + let mut reader = crate::object_api::with_get_object_read_condition( + Arc::new(StopReadForEtag("different-etag".to_owned())), + set.get_object_reader(bucket, &object, None, HeaderMap::new(), &opts), + ) + .await + .expect("unmatched condition opens the intact object"); + let mut body = Vec::new(); + reader.stream.read_to_end(&mut body).await.expect("positive-control body"); + assert_eq!(body, payload); + let source_kind = if size == 4096 { + disk_call_counters::KIND_INLINE_SHARD_OPEN + } else { + disk_call_counters::KIND_DISK_SHARD_OPEN + }; + assert!( + calls.total(source_kind) > 0, + "the same source counter observes the positive control: size={size}, inline={}, disk={}, decode={}", + calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN), + calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), + calls.total(disk_call_counters::KIND_BODY_DECODE), + ); + assert!( + calls.total(disk_call_counters::KIND_BODY_DECODE) > 0, + "the same decode counter observes the positive control" + ); + } + } + + #[tokio::test] + #[serial_test::serial(body_cache_hook)] + async fn read_condition_rechecks_selected_metadata_before_body_access() { + struct Changing(std::sync::atomic::AtomicUsize); + impl crate::object_api::GetObjectReadCondition for Changing { + fn is_terminal(&self, _: crate::object_api::GetObjectReadMetadata<'_>) -> bool { + self.0.fetch_add(1, std::sync::atomic::Ordering::Relaxed) == 0 + } + } + let (_dirs, set) = make_local_set_disks(4, 2).await; + let bucket = "read-condition-consistency"; + let object = object_with_initial_data_shards(bucket, "read-condition-consistency"); + let opts = ObjectOptions { + no_lock: true, + ..Default::default() + }; + set.make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let mut put = PutObjReader::from_vec(vec![b'x'; 4096]); + set.put_object(bucket, &object, &mut put, &opts).await.expect("write object"); + let result = crate::object_api::with_get_object_read_condition( + Arc::new(Changing(std::sync::atomic::AtomicUsize::new(0))), + set.get_object_reader(bucket, &object, None, HeaderMap::new(), &opts), + ) + .await; + assert!(result.is_err(), "an unverified terminal snapshot must never become a 200 reader"); + } + + #[tokio::test] + #[serial_test::serial(body_cache_hook)] + async fn read_condition_miss_preserves_inline_early_stop_and_exact_body() { + let (_dirs, set) = make_local_set_disks(4, 2).await; + let bucket = "read-condition-miss"; + let object = object_with_initial_data_shards(bucket, "read-condition-miss"); + let opts = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let payload: Vec = (0..8192) + .map(|index| u8::try_from(index % 251).expect("pattern byte")) + .collect(); + set.make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let mut put = PutObjReader::from_vec(payload.clone()); + set.put_object(bucket, &object, &mut put, &opts) + .await + .expect("write inline object"); + let slow = bounded_initial_parity_disk_index(bucket, &object); + let barrier = rename_fanout_barrier::arm(&object, slow, rename_fanout_barrier::PHASE_READ_VERSION); + let set_for_read = Arc::clone(&set); + let object_for_read = object.clone(); + let mut read = tokio::spawn(async move { + crate::object_api::with_get_object_read_condition( + Arc::new(StopReadForEtag("other".to_owned())), + set_for_read.get_object_reader(bucket, &object_for_read, None, HeaderMap::new(), &opts), + ) + .await + }); + tokio::time::timeout(READ_VERSION_BARRIER_GUARD, barrier.wait_until_paused()) + .await + .expect("slow metadata read is pending"); + let mut reader = tokio::time::timeout(READ_VERSION_BARRIER_GUARD, &mut read) + .await + .expect("condition miss preserves the normal early stop") + .expect("reader task") + .expect("reader"); + let mut bytes = Vec::new(); + reader.stream.read_to_end(&mut bytes).await.expect("normal body"); + assert_eq!(bytes, payload); + barrier.release(); + } + #[test] fn cache_hit_consumers_release_snapshot_at_legacy_ownership() { let fi = FileInfo { @@ -1458,6 +1727,7 @@ mod prepared_get_object_metadata_tests { }; let cached = Arc::new(GetObjectMetadataCacheEntry { created_at: Instant::now(), + capability: GetObjectMetadataCacheCapability::Full, parts_metadata: vec![fi.clone()], fi, online_disks: vec![None], @@ -1466,11 +1736,11 @@ mod prepared_get_object_metadata_tests { let snapshot = GetObjectFileInfo::shared(Arc::clone(&cached)); assert!(snapshot.has_valid_representation()); - assert!(std::mem::size_of::() >= std::mem::size_of::()); - assert!( - std::mem::size_of::() - <= std::mem::size_of::() + 2 * std::mem::size_of::() - ); + // Keep the original payload budget after moving the captured disks out + // of OwnedGetObjectFileInfo. + let owned_payload_size = std::mem::size_of::<(OwnedGetObjectFileInfo, Option>>)>(); + assert!(std::mem::size_of::() >= owned_payload_size); + assert!(std::mem::size_of::() <= owned_payload_size + 2 * std::mem::size_of::()); assert_eq!(Arc::strong_count(&cached), 2, "a cache hit must add one snapshot reference"); assert_eq!(snapshot.fi().name, "object"); assert_eq!(snapshot.parts_metadata().len(), 1); @@ -4390,6 +4660,9 @@ pub struct SetDisks { /// writes skip the global registry mutex (backlog#1315). `Arc` so clones of /// a set share one generation marker. capacity_dirty_generation: Arc, + /// Per-set proof that successful rename_data peers return the old-current + /// source capability needed before PUT may skip its pre-rename lookup. + tier_free_version_rename_data_source_capable: Arc, /// Orphan prefixes whose last purge scan met data that can never be /// purged by listing (residue without a committed marker, an in-flight /// write), keyed by `bucket/prefix` with the time of that scan. Empty @@ -4645,6 +4918,14 @@ struct GetObjectMetadataCacheKey { hash: u64, } +#[derive(Clone, Debug)] +struct GetObjectMetadataCacheLookupKey<'a> { + bucket: &'a str, + object: Cow<'a, str>, + generation: u64, + hash: u64, +} + #[derive(Clone, Copy, Debug, Eq, PartialEq)] struct GetObjectMetadataCacheGeneration { index: usize, @@ -4726,13 +5007,40 @@ impl GetObjectMetadataCacheKey { fn new(bucket: &str, object: &str, generation: GetObjectMetadataCacheGeneration) -> Self { Self { bucket: Arc::from(bucket), - object: Arc::from(object), + object: Arc::from(object_name_disk_identity(object).as_ref()), generation: generation.value, hash: generation.hash, } } } +impl<'a> GetObjectMetadataCacheLookupKey<'a> { + fn new(bucket: &'a str, object: &'a str, generation: GetObjectMetadataCacheGeneration) -> Self { + Self { + bucket, + object: object_name_disk_identity(object), + generation: generation.value, + hash: generation.hash, + } + } +} + +impl Hash for GetObjectMetadataCacheLookupKey<'_> { + fn hash(&self, state: &mut H) { + self.hash.hash(state); + self.generation.hash(state); + } +} + +impl moka::Equivalent for GetObjectMetadataCacheLookupKey<'_> { + fn equivalent(&self, key: &GetObjectMetadataCacheKey) -> bool { + self.bucket == key.bucket.as_ref() + && self.object.as_ref() == key.object.as_ref() + && self.generation == key.generation + && self.hash == key.hash + } +} + impl Hash for GetObjectMetadataCacheKey { fn hash(&self, state: &mut H) { self.hash.hash(state); @@ -4740,6 +5048,15 @@ impl Hash for GetObjectMetadataCacheKey { } } +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum GetObjectMetadataCacheCapability { + Full, + // A quorum-selected layout can serve body reads, but is not a mutation layout. + QuorumSelected, + // Quorum-validated metadata used only to evaluate a terminal read condition. + ConditionOnly, +} + #[derive(Debug)] struct GetObjectMetadataCacheEntry { #[allow(dead_code)] // Kept for debugging; moka handles TTL internally @@ -4748,6 +5065,7 @@ struct GetObjectMetadataCacheEntry { parts_metadata: Vec, online_disks: Vec>, read_quorum: usize, + capability: GetObjectMetadataCacheCapability, } #[derive(Clone, Debug)] @@ -4768,6 +5086,10 @@ impl DiskHealthEntry { } } +#[derive(Debug, thiserror::Error)] +#[error("committed system metadata write requires repair by its owning writer")] +pub(crate) struct SystemMetadataRepairRequired; + impl SetDisks { pub(in crate::set_disk) async fn persist_delete_marker_purge( &self, @@ -4878,7 +5200,7 @@ impl SetDisks { state = "mrf_durable_admission_failed", bucket, object, version_id, pool_index = self.pool_index, set_index = self.set_index, error = %error, - "Partial write repair falling back to the heal channel" + "Partial write repair responsibility could not be persisted" ); false } @@ -4888,7 +5210,17 @@ impl SetDisks { pub(in crate::set_disk) async fn submit_rename_tail_heal( &self, mut request: rustfs_heal_contracts::heal_channel::HealChannelRequest, - ) { + completion: crate::object_api::WriteCompletion, + ) -> Result<()> { + // System metadata repair remains with the writer's deployment and + // transaction fence; a user-bucket MRF authority cannot authorize it. + if crate::bucket::utils::is_meta_bucketname(&request.bucket) { + return if completion == crate::object_api::WriteCompletion::TailDrainedWithOwnerRepair { + Err(Error::other(SystemMetadataRepairRequired)) + } else { + Ok(()) + }; + } if let Some(object) = request.object_prefix.as_deref() && self .persist_partial_write( @@ -4899,10 +5231,10 @@ impl SetDisks { ) .await { - return; + return Ok(()); } if request.expected_bucket_incarnation_id.is_none() { - return; + return Ok(()); } request.source = rustfs_heal_contracts::heal_channel::HealRequestSource::Mrf; #[cfg(test)] @@ -4914,11 +5246,12 @@ impl SetDisks { .clone(); if let Some(capture) = capture { let _ = capture.send(request); - return; + return Ok(()); } } let _ = rustfs_heal_contracts::heal_channel::send_heal_request(request).await; + Ok(()) } #[cfg(test)] @@ -4960,7 +5293,9 @@ impl SetDisks { fn get_object_metadata_cache_hash(&self, bucket: &str, object: &str) -> u64 { let mut hasher = self.get_object_metadata_cache_hash_builder.build_hasher(); bucket.hash(&mut hasher); - object.hash(&mut hasher); + // Every spelling of one xl.meta (Windows `a\b` vs `a/b`) shares one + // cache identity, so a mutation through any of them retires the rest. + object_name_disk_identity(object).hash(&mut hasher); hasher.finish() } @@ -5278,6 +5613,7 @@ impl SetDisks { ctx, capacity_scope_cache: Arc::new(std::sync::RwLock::new(CapacityScopeCache::default())), capacity_dirty_generation: Arc::new(AtomicU64::new(u64::MAX)), + tier_free_version_rename_data_source_capable: Arc::new(AtomicBool::new(false)), orphan_purge_backoff: Arc::new(std::sync::Mutex::new(HashMap::new())), #[cfg(test)] storage_class_config_override: Arc::new(std::sync::RwLock::new(None)), @@ -5292,6 +5628,15 @@ impl SetDisks { &self.ctx } + pub(in crate::set_disk) fn tier_free_version_rename_data_source_capable(&self) -> bool { + self.tier_free_version_rename_data_source_capable.load(Ordering::Acquire) + } + + pub(in crate::set_disk) fn observe_tier_free_version_rename_data_source_capability(&self, capable: bool) { + self.tier_free_version_rename_data_source_capable + .store(capable, Ordering::Release); + } + #[cfg(test)] pub(crate) fn set_instance_ctx_for_test(&mut self, ctx: Arc) { self.local_lock_manager = ctx.lock_manager(); @@ -7698,6 +8043,21 @@ pub fn e_tag_matches(etag: &str, condition: &str) -> bool { canonicalize_etag(etag) == canonicalize_etag(condition) } +fn weak_etag_matches(etag: &str, condition: &str) -> bool { + let condition = condition.trim(); + if condition == "*" { + return true; + } + fn strip_weak(value: &str) -> &str { + let trimmed = value.trim(); + trimmed + .strip_prefix("W/") + .or_else(|| trimmed.strip_prefix("w/")) + .unwrap_or(trimmed) + } + canonicalize_etag(strip_weak(etag)) == canonicalize_etag(strip_weak(condition)) +} + pub fn should_prevent_write(oi: &ObjectInfo, if_none_match: Option, if_match: Option) -> bool { let if_none_match = if_none_match .as_deref() @@ -7708,7 +8068,7 @@ pub fn should_prevent_write(oi: &ObjectInfo, if_none_match: Option, if_m match &oi.etag { Some(etag) => { if let Some(if_none_match) = if_none_match - && e_tag_matches(etag, if_none_match) + && weak_etag_matches(etag, if_none_match) { return true; } @@ -8064,6 +8424,78 @@ mod tests { assert_eq!(Arc::strong_count(&set.set_lock_namespace), before); } + /// Spellings that a Windows disk resolves to the xl.meta of `a/b`. + const BACKSLASH_SPELLINGS_OF_A_B: [&str; 4] = ["a\\b", "a\\\\b", "\\a\\b", "a\\b\\"]; + + async fn ns_write_lock_free_while_held(set: &SetDisks, held: &str, contender: &str) -> bool { + let _held = set + .new_ns_lock("bucket", held) + .await + .expect("namespace lock should be created") + .get_write_lock(Duration::from_secs(1)) + .await + .expect("holder write lock should be acquired"); + set.new_ns_lock("bucket", contender) + .await + .expect("namespace lock should be created") + .get_write_lock(Duration::from_millis(100)) + .await + .is_ok() + } + + async fn ns_lock_test_sets() -> [Arc; 2] { + let local_ctx = Arc::new(InstanceContext::new()); + local_ctx.update_erasure_type(SetupType::Erasure).await; + let dist_ctx = Arc::new(InstanceContext::new()); + dist_ctx.update_erasure_type(SetupType::DistErasure).await; + let manager = Arc::new(rustfs_lock::GlobalLockManager::new()); + let locker: Arc = Arc::new(LocalClient::with_manager(manager)); + [ + make_test_set_disks_with_ctx(Vec::new(), local_ctx).await, + make_test_set_disks_with_ctx(vec![locker], dist_ctx).await, + ] + } + + #[cfg(not(windows))] + #[tokio::test(flavor = "multi_thread")] + async fn new_ns_lock_keeps_backslash_and_slash_keys_distinct_off_windows() { + for set in ns_lock_test_sets().await { + assert!( + !ns_write_lock_free_while_held(&set, "a/b", "a/b").await, + "baseline: one key must conflict" + ); + for spelling in BACKSLASH_SPELLINGS_OF_A_B { + assert!( + ns_write_lock_free_while_held(&set, "a/b", spelling).await, + "off Windows {spelling:?} is a different object and must not wait on `a/b`" + ); + } + } + } + + #[cfg(windows)] + #[tokio::test(flavor = "multi_thread")] + async fn windows_new_ns_lock_serializes_backslash_spellings_of_one_object() { + // Local and distributed lock domains: the distributed key is folded by + // the requester before it reaches the lock client. + for set in ns_lock_test_sets().await { + for spelling in BACKSLASH_SPELLINGS_OF_A_B { + assert!( + !ns_write_lock_free_while_held(&set, "a/b", spelling).await, + "{spelling:?} addresses the xl.meta of `a/b` and must wait on its lock" + ); + assert!( + !ns_write_lock_free_while_held(&set, spelling, "a/b").await, + "`a/b` must wait on the lock held through {spelling:?}" + ); + } + assert!( + ns_write_lock_free_while_held(&set, "a/b", "a/b/c").await, + "folding must not merge distinct objects" + ); + } + } + fn put_object_commit_namespace_lock_wait_sample_count(snapshotter: &metrics_util::debugging::Snapshotter) -> usize { snapshotter .snapshot() @@ -10389,6 +10821,13 @@ mod tests { .expect("orphan part should be written"); let set = make_set_disks_with(vec![Some(disk)]).await; + assert_eq!( + set.dry_run_reclaim_orphan_data_dirs("bucket", "obj") + .await + .expect("dry-run orphan cleanup"), + 1 + ); + assert!(object_dir.join(orphan.to_string()).exists(), "dry run must retain the orphan payload"); let removed = set .reclaim_orphan_data_dirs("bucket", "obj") .await @@ -10400,6 +10839,67 @@ mod tests { assert!(object_dir.join(STORAGE_FORMAT_FILE).exists(), "metadata must be preserved"); } + #[tokio::test] + async fn reclaim_orphan_data_dirs_drains_empty_discovery_pages() { + let (dir, disk) = make_single_local_disk().await; + let live = Uuid::new_v4(); + let orphan = Uuid::new_v4(); + let object_dir = dir.path().join("bucket/obj"); + write_object_meta_with_data_dirs(&object_dir, "bucket", "obj", &[live]).await; + let live_dir = object_dir.join(live.to_string()); + let orphan_dir = object_dir.join(orphan.to_string()); + fs::create_dir_all(&live_dir).await.expect("create live multipart directory"); + fs::create_dir_all(&orphan_dir).await.expect("create orphan directory"); + fs::write(orphan_dir.join("part.1"), b"stale") + .await + .expect("write orphan payload"); + let parts_dir = live_dir.clone(); + tokio::task::spawn_blocking(move || { + for part in 1..=4200 { + std::fs::write(parts_dir.join(format!("part.{part}")), b"live").expect("write live multipart entry"); + } + }) + .await + .expect("create multipart fixture"); + + let first = disk + .discover_delete_transactions("bucket", Some("obj"), None, 1) + .await + .expect("read first discovery page"); + assert!(first.transactions.is_empty()); + assert!(first.next.is_some(), "ordinary multipart entries must exhaust the first scan budget"); + let last = disk + .discover_delete_transactions("bucket", Some("obj"), first.next, 1) + .await + .expect("finish fixture discovery"); + assert!( + last.transactions.is_empty() && last.next.is_none(), + "the complete inventory has no transaction evidence" + ); + + let set = make_set_disks_with(vec![Some(disk)]).await; + assert_eq!( + set.dry_run_reclaim_orphan_data_dirs("bucket", "obj") + .await + .expect("dry-run paginated orphan cleanup"), + 1 + ); + assert!(orphan_dir.exists(), "dry-run must retain the orphan"); + assert_eq!( + set.reclaim_orphan_data_dirs("bucket", "obj") + .await + .expect("reclaim after complete discovery"), + 1 + ); + assert!(!orphan_dir.exists(), "both the set preflight and final disk guard must drain empty pages"); + assert_eq!( + fs::read(live_dir.join("part.4200")) + .await + .expect("read referenced multipart tail"), + b"live" + ); + } + async fn recv_abandoned_parts_trace( trace: &mut rustfs_common::trace_bus::TraceSubscription, bucket: &str, @@ -10613,7 +11113,7 @@ mod tests { } #[tokio::test] - async fn reclaim_orphan_data_dirs_recovers_committed_delete_marker_without_meta() { + async fn reclaim_orphan_data_dirs_defers_delete_marker_without_meta() { let (dir, disk) = make_single_local_disk().await; let stale = Uuid::new_v4(); let transaction = Uuid::new_v4(); @@ -10638,8 +11138,105 @@ mod tests { .await .expect("upgrade reclaim should succeed"); - assert_eq!(removed, 1, "the committed delete residue should be reclaimed"); - assert!(!stale_dir.exists(), "the committed stale data dir should be removed"); + assert_eq!(removed, 0, "a local marker is not a set-level commit decision"); + assert_eq!(fs::read(stale_dir.join("part.1")).await.expect("read retained payload"), b"stale"); + } + + #[tokio::test] + async fn reclaim_orphan_data_dirs_preserves_uuid_prefix_children() { + let (dir, disk) = make_single_local_disk().await; + let live = Uuid::new_v4(); + let prefix = Uuid::new_v4(); + let child_data = Uuid::new_v4(); + let object_dir = dir.path().join("bucket/obj"); + write_object_meta_with_data_dirs(&object_dir, "bucket", "obj", &[live]).await; + let child_dir = object_dir.join(prefix.to_string()).join("child"); + write_object_meta_with_data_dirs(&child_dir, "bucket", &format!("obj/{prefix}/child"), &[child_data]).await; + let data_dir = child_dir.join(child_data.to_string()); + fs::create_dir_all(&data_dir).await.expect("create child payload directory"); + fs::write(data_dir.join("part.1"), b"live child payload") + .await + .expect("write child payload"); + let parent_meta = fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("read parent metadata"); + let child_meta = fs::read(child_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("read child metadata"); + let set = make_set_disks_with(vec![Some(disk)]).await; + assert_eq!(set.dry_run_reclaim_orphan_data_dirs("bucket", "obj").await.expect("dry run"), 0); + assert_eq!(set.reclaim_orphan_data_dirs("bucket", "obj").await.expect("reclaim"), 0); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("read parent metadata"), + parent_meta + ); + assert_eq!( + fs::read(child_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("read child metadata"), + child_meta + ); + assert_eq!( + fs::read(data_dir.join("part.1")).await.expect("read child payload"), + b"live child payload" + ); + } + + #[tokio::test] + async fn reclaim_orphan_data_dirs_waits_for_complete_participant_inventory() { + let (dir, disk) = make_single_local_disk().await; + let live = Uuid::new_v4(); + let orphan = Uuid::new_v4(); + let object_dir = dir.path().join("bucket/obj"); + write_object_meta_with_data_dirs(&object_dir, "bucket", "obj", &[live]).await; + let payload_dir = object_dir.join(orphan.to_string()); + fs::create_dir_all(&payload_dir).await.expect("create orphan directory"); + fs::write(payload_dir.join("part.1"), b"potentially referenced payload") + .await + .expect("write payload"); + let set = make_set_disks_with(vec![Some(disk), None]).await; + assert_eq!(set.dry_run_reclaim_orphan_data_dirs("bucket", "obj").await.expect("dry run"), 0); + assert_eq!(set.reclaim_orphan_data_dirs("bucket", "obj").await.expect("reclaim"), 0); + assert_eq!( + fs::read(payload_dir.join("part.1")).await.expect("read retained payload"), + b"potentially referenced payload" + ); + } + + #[tokio::test] + async fn reclaim_orphan_data_dirs_defers_payload_referenced_by_neighboring_journal() { + let (dir, disk) = make_single_local_disk().await; + let live = Uuid::new_v4(); + let old = Uuid::new_v4(); + let transaction = Uuid::new_v4(); + let object_dir = dir.path().join("bucket/obj"); + write_object_meta_with_data_dirs(&object_dir, "bucket", "obj", &[live]).await; + let payload_dir = object_dir.join(old.to_string()); + fs::create_dir_all(&payload_dir).await.expect("create old payload directory"); + fs::write(payload_dir.join("part.1"), b"rollback payload") + .await + .expect("write old payload"); + let transaction_dir = object_dir.join(transaction.to_string()); + write_object_meta_with_data_dirs(&transaction_dir, "bucket", "obj", &[old]).await; + fs::rename( + transaction_dir.join(STORAGE_FORMAT_FILE), + transaction_dir.join(STORAGE_FORMAT_FILE_BACKUP), + ) + .await + .expect("retain rollback metadata"); + let set = make_set_disks_with(vec![Some(disk)]).await; + assert_eq!(set.dry_run_reclaim_orphan_data_dirs("bucket", "obj").await.expect("dry run"), 0); + assert_eq!(set.reclaim_orphan_data_dirs("bucket", "obj").await.expect("reclaim"), 0); + assert_eq!( + fs::read(payload_dir.join("part.1")).await.expect("read rollback payload"), + b"rollback payload" + ); + assert!( + transaction_dir.join(STORAGE_FORMAT_FILE_BACKUP).exists(), + "canonical recovery owns the journal" + ); } // Cross-replica union: a data dir referenced by ANOTHER disk's xl.meta must be @@ -13488,6 +14085,14 @@ mod tests { let if_none_match = Some(String::new()); let if_match = Some(" ".to_string()); assert!(!should_prevent_write(&oi, if_none_match, if_match)); + + let oi = ObjectInfo { + etag: Some("abc".to_string()), + ..Default::default() + }; + assert!(should_prevent_write(&oi, Some("W/\"abc\"".to_string()), None)); + assert!(!should_prevent_write(&oi, Some("W/\"def\"".to_string()), None)); + assert!(should_prevent_write(&oi, None, Some("W/\"abc\"".to_string()))); } #[test] @@ -16055,6 +16660,34 @@ mod tests { Some(StorageError::PreconditionFailed), "data movement must not replace a target delete marker" ); + let ordinary_create_only = ObjectOptions { + versioned: true, + http_preconditions: Some(HTTPPreconditions { + if_none_match: Some("*".to_string()), + ..Default::default() + }), + ..Default::default() + }; + assert_eq!( + set_disks + .check_write_precondition(bucket, object, &ordinary_create_only) + .await, + None, + "ordinary If-None-Match: * must proceed past a top-level delete marker" + ); + let ordinary_if_match = ObjectOptions { + versioned: true, + http_preconditions: Some(HTTPPreconditions { + if_match: Some("*".to_string()), + ..Default::default() + }), + ..Default::default() + }; + assert_eq!( + set_disks.check_write_precondition(bucket, object, &ordinary_if_match).await, + Some(StorageError::ObjectNotFound(bucket.to_string(), object.to_string())), + "ordinary If-Match must treat a top-level delete marker as absent" + ); let err = match set_disks .get_object_reader(bucket, object, None, HeaderMap::new(), &opts) .await diff --git a/crates/ecstore/src/set_disk/ops/bucket.rs b/crates/ecstore/src/set_disk/ops/bucket.rs index d1dc1770a..d64128e5e 100644 --- a/crates/ecstore/src/set_disk/ops/bucket.rs +++ b/crates/ecstore/src/set_disk/ops/bucket.rs @@ -31,6 +31,7 @@ use crate::disk::DiskAPI; pub(crate) enum BucketInfoQuorum { Read, Write, + HealCandidates, } impl SetDisks { @@ -104,7 +105,9 @@ impl SetDisks { // class. A namespace read must intersect that majority; object // readers still enforce the persisted layout's data-shard quorum. BucketInfoQuorum::Read => reduce_read_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, disk_count.div_ceil(2).max(1)), - BucketInfoQuorum::Write => reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, disk_count / 2 + 1), + BucketInfoQuorum::Write | BucketInfoQuorum::HealCandidates => { + reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, disk_count / 2 + 1) + } }; if let Some(err) = error { return Err(err.into()); @@ -123,8 +126,42 @@ impl SetDisks { } pub(crate) async fn list_bucket_for_scanner(&self, _opts: &BucketOptions) -> Result<(Vec, bool)> { + self.list_bucket_with_quorum(BucketInfoQuorum::Write).await + } + + pub(crate) async fn bucket_needs_reconstruction(&self, bucket: &str) -> Result { + let results = join_all(self.disk_inventory().await.into_iter().map(|disk| async move { + match disk { + Some(disk) => disk.stat_volume(bucket).await, + None => Err(DiskError::DiskNotFound), + } + })) + .await; + let mut present = false; + let mut missing = false; + for result in results { + match result { + Ok(_) => present = true, + Err(DiskError::VolumeNotFound) => missing = true, + Err(error) => return Err(error.into()), + } + } + if !present { + return Err(Error::VolumeNotFound); + } + Ok(missing) + } + + pub(crate) async fn list_bucket_with_quorum(&self, quorum: BucketInfoQuorum) -> Result<(Vec, bool)> { let disks = self.disk_inventory().await; let write_quorum = (disks.len() / 2) + 1; + // Discovery uses the same namespace read boundary as stat_bucket. + // It grants no authority to mutate buckets or accept object reads. + let required = match quorum { + BucketInfoQuorum::Read => disks.len().div_ceil(2).max(1), + BucketInfoQuorum::Write => write_quorum, + BucketInfoQuorum::HealCandidates => disks.len().max(1), + }; let mut futures = Vec::with_capacity(disks.len()); for disk in disks { @@ -153,7 +190,13 @@ impl SetDisks { } } - if let Some(err) = reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, write_quorum) { + let error = match quorum { + BucketInfoQuorum::Read => reduce_read_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, required), + BucketInfoQuorum::Write | BucketInfoQuorum::HealCandidates => { + reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, required) + } + }; + if let Some(err) = error { return Err(err.into()); } @@ -176,10 +219,17 @@ impl SetDisks { } } - topology_complete &= counts.values().all(|(count, _)| *count >= write_quorum); + let minimum_presence = if matches!(quorum, BucketInfoQuorum::HealCandidates) { + // Candidate presence is not mutation authority. Every disk must answer, + // and the heal owner validates the committed generation before recreation. + 1 + } else { + topology_complete &= counts.values().all(|(count, _)| *count >= write_quorum); + required + }; let mut buckets = counts .into_values() - .filter_map(|(count, bucket)| (count >= write_quorum).then_some(bucket)) + .filter_map(|(count, bucket)| (count >= minimum_presence).then_some(bucket)) .collect::>(); buckets.sort_by(|left, right| left.name.cmp(&right.name)); Ok((buckets, topology_complete)) diff --git a/crates/ecstore/src/set_disk/ops/delete_recovery.rs b/crates/ecstore/src/set_disk/ops/delete_recovery.rs new file mode 100644 index 000000000..403244d80 --- /dev/null +++ b/crates/ecstore/src/set_disk/ops/delete_recovery.rs @@ -0,0 +1,2354 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Set-wide delete decisions and replay. Decisions remain until every +//! participant and related transaction for the exact key converges. + +use crate::disk::error::{DiskError, Result, RpcStatusError}; +use crate::disk::{ + DELETE_RECOVERY_PROTOCOL_VERSION, DeleteRecoveryAction, DeleteRecoveryInspection, DeleteRecoveryState, DeleteTransactionId, + DiskAPI, DiskStore, +}; +use crate::set_disk::SetDisks; +use futures::future::join_all; +use std::collections::{BTreeMap, BTreeSet, HashSet}; +use std::sync::Arc; +use uuid::Uuid; + +const DISCOVERY_PAGE_SIZE: u32 = 128; +const MAX_DISCOVERED_TRANSACTIONS: usize = 1_000_000; + +#[derive(Debug, Default, Clone, PartialEq, Eq)] +pub(crate) struct DeleteRecoverySummary { + pub discovered: usize, + pub committed: usize, + pub rolled_back: usize, + pub blocked: usize, + pub pending: usize, + pub failed: usize, + pub reclaimed: usize, + /// Observed removals for this exact key, not aggregated across prefixes. + pub removed_versions: HashSet, + /// Read-only evidence for ordinary missing-key S3 semantics. This never + /// authorizes settlement while a configured participant is unavailable. + pub no_transactions_at_quorum: bool, +} + +#[derive(Clone)] +pub(crate) struct DeleteRecoveryContext<'a> { + pub incarnation: Option, + pub dry_run: bool, + pub explicit_delete: bool, + /// Only the locked heal path may repair a live marker-creation source. + pub repair_marker_source: bool, + pub namespace_owner: Option>, + pub recovery_ticket: Option, + pub can_commit: &'a (dyn Fn() -> bool + Sync), +} + +impl<'a> DeleteRecoveryContext<'a> { + pub(crate) fn new(incarnation: Option, can_commit: &'a (dyn Fn() -> bool + Sync)) -> Self { + Self { + incarnation, + dry_run: false, + explicit_delete: false, + repair_marker_source: false, + namespace_owner: None, + recovery_ticket: None, + can_commit, + } + } +} + +impl DeleteRecoverySummary { + pub(crate) fn add(&mut self, other: Self) { + self.discovered += other.discovered; + self.committed += other.committed; + self.rolled_back += other.rolled_back; + self.blocked += other.blocked; + self.pending += other.pending; + self.failed += other.failed; + self.reclaimed += other.reclaimed; + } +} + +pub(in crate::set_disk) fn unsupported_recovery(error: &DiskError) -> bool { + matches!(error, DiskError::UnsupportedDisk) + || matches!(error, DiskError::Io(source) if source.get_ref() + .and_then(|source| source.downcast_ref::()) + .is_some_and(|status| status.status().code() == tonic::Code::Unimplemented)) +} + +/// Set authority needs generation identities, not the potentially large +/// per-version apply requests carried by returning-drive adoption. +struct DecisionEvidence { + transaction: Uuid, + deleted_versions: HashSet, + original_versions: HashSet, + predecessors: HashSet, + target_absent: bool, +} + +fn receipt(state: DeleteRecoveryState) -> bool { + matches!(state, DeleteRecoveryState::QuorumCommitted | DeleteRecoveryState::CommitFinished) +} + +fn validate(inspection: &DeleteRecoveryInspection, object: &str, transaction: Uuid) -> Result<()> { + if inspection.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION + || inspection.id.object != object + || inspection.id.transaction != transaction + { + return Err(DiskError::FileCorrupt); + } + Ok(()) +} + +async fn inspect( + disks: &[Option], + bucket: &str, + object: &str, + transaction: Uuid, +) -> Vec> { + join_all(disks.iter().map(|disk| async move { + let state = disk + .as_ref() + .ok_or(DiskError::DiskNotFound)? + .inspect_delete_transaction(bucket, object, transaction) + .await?; + validate(&state, object, transaction)?; + Ok(state) + })) + .await +} + +async fn discover(disk: &DiskStore, bucket: &str, object: Option<&str>) -> Result> { + let mut cursor = None; + let mut ids = BTreeSet::new(); + loop { + let page = disk + .discover_delete_transactions(bucket, object, cursor, DISCOVERY_PAGE_SIZE) + .await?; + if page.transactions.len() > usize::try_from(DISCOVERY_PAGE_SIZE).map_err(|_| DiskError::FileCorrupt)? { + return Err(DiskError::FileCorrupt); + } + for id in page.transactions { + if id.transaction.is_nil() || object.is_some_and(|object| object != id.object) { + return Err(DiskError::FileCorrupt); + } + ids.insert((id.object, id.transaction)); + if ids.len() > MAX_DISCOVERED_TRANSACTIONS { + return Err(DiskError::other("delete recovery transaction limit exceeded")); + } + } + cursor = page.next; + if cursor.is_none() { + break; + } + } + Ok(ids + .into_iter() + .map(|(object, transaction)| DeleteTransactionId { object, transaction }) + .collect()) +} + +async fn settle( + disk: &DiskStore, + bucket: &str, + state: &DeleteRecoveryInspection, + action: DeleteRecoveryAction, + can_commit: &(impl Fn() -> bool + Sync + ?Sized), + namespace_owner: Option<&Arc>, +) -> Result { + if !can_commit() { + return Err(DiskError::other("delete recovery namespace fence was lost")); + } + let result = disk + .settle_delete_transaction_with_namespace_owner( + bucket, + &state.id.object, + state.id.transaction, + state.fingerprint, + action, + namespace_owner.cloned(), + ) + .await; + let next = result?; + validate(&next, &state.id.object, state.id.transaction)?; + Ok(next) +} + +/// Each phase fans out across independent drives, and awaits all started work +/// before evaluating the next decision or cleanup barrier. +async fn settle_group( + disks: &[Option], + bucket: &str, + copies: &mut [DeleteRecoveryInspection], + action: impl Fn(&DeleteRecoveryInspection) -> Option + Sync, + can_commit: &(impl Fn() -> bool + Sync + ?Sized), + namespace_owner: Option<&Arc>, + removed_versions: &mut HashSet, +) -> Result { + let results = join_all(disks.iter().zip(copies.iter()).map(|(disk, copy)| { + let action = action(copy); + async move { + match action { + Some(action) => settle( + disk.as_ref().ok_or(DiskError::DiskNotFound)?, + bucket, + copy, + action, + can_commit, + namespace_owner, + ) + .await + .map(Some), + None => Ok(None), + } + } + })) + .await; + let mut pending = false; + for (copy, result) in copies.iter_mut().zip(results) { + match result { + Ok(Some(next)) => { + if copy.current_digest != next.current_digest { + let retained: HashSet<_> = next + .current_versions + .iter() + .map(|v| v.version_id.unwrap_or_default()) + .collect(); + removed_versions.extend( + copy.current_versions + .iter() + .map(|v| v.version_id.unwrap_or_default()) + .filter(|id| !retained.contains(id)), + ); + } + *copy = next; + } + Ok(None) => {} + Err(DiskError::DiskOngoingReq) => pending = true, + Err(error) => return Err(error), + } + } + Ok(pending) +} + +/// A later decision may require settling its prepared predecessor before an +/// untouched participant can adopt that decision. Ordering provides no commit +/// authority; reject contradictory causal records before any settlement. +fn causal_transaction_order(states: &BTreeMap>) -> Result> { + let mut indegrees: BTreeMap<_, usize> = states.keys().map(|id| (*id, 0)).collect(); + let mut successors: BTreeMap> = BTreeMap::new(); + for (transaction, copies) in states { + for predecessor in copies + .iter() + .filter_map(|copy| copy.journal.as_ref()) + .flat_map(|journal| &journal.predecessors) + { + if states.contains_key(predecessor) && successors.entry(*predecessor).or_default().insert(*transaction) { + let degree = indegrees.get_mut(transaction).ok_or(DiskError::FileCorrupt)?; + *degree = degree.checked_add(1).ok_or(DiskError::FileCorrupt)?; + } + } + } + let mut ready: BTreeSet<_> = indegrees + .iter() + .filter_map(|(id, degree)| (*degree == 0).then_some(*id)) + .collect(); + let mut order = Vec::with_capacity(states.len()); + while let Some(transaction) = ready.pop_first() { + order.push(transaction); + if let Some(children) = successors.get(&transaction) { + for child in children { + let degree = indegrees.get_mut(child).ok_or(DiskError::FileCorrupt)?; + *degree = degree.checked_sub(1).ok_or(DiskError::FileCorrupt)?; + if *degree == 0 { + ready.insert(*child); + } + } + } + } + if order.len() != states.len() { + return Err(DiskError::FileCorrupt); + } + Ok(order) +} + +impl SetDisks { + /// Publication follows the ordinary metadata quorum. Once a publication is + /// attempted, response loss must never trigger blind rollback. + pub(in crate::set_disk) async fn publish_delete_commit( + &self, + bucket: &str, + object: &str, + transaction: Uuid, + disks: &[Option], + write_quorum: usize, + mut context: DeleteRecoveryContext<'_>, + ) -> Result<()> { + if context.namespace_owner.is_none() { + context.namespace_owner = Some(self.ctx.begin_namespace_commit()); + } + let can_commit = context.can_commit; + let states = inspect(disks, bucket, object, transaction).await; + let untouched_absence = |state: &Result| matches!(state, Ok(state) if state.state == DeleteRecoveryState::Absent && state.current_digest.is_none()); + if states.iter().filter(|state| untouched_absence(state)).count() >= write_quorum + && states.iter().all(|state| { + untouched_absence(state) || matches!(state, Err(error) if unavailable(error) && !unsupported_recovery(error)) + }) + { + // An ordinary missing-key no-op requires no recovery decision. + // Do not manufacture witness transactions while a drive is offline. + if let Some(ticket) = context.recovery_ticket.as_ref() { + self.ctx.delete_recovery_hints().ack(ticket); + } + return Ok(()); + } + let published = join_all(disks.iter().zip(&states).map(|(disk, state)| async { + let disk = disk.as_ref().ok_or(DiskError::DiskNotFound)?; + let state = state.as_ref().map_err(Clone::clone)?; + if state.state == DeleteRecoveryState::LegacyAmbiguous { + return Err(DiskError::FileCorrupt); + } + settle( + disk, + bucket, + state, + DeleteRecoveryAction::PublishCommit, + can_commit, + context.namespace_owner.as_ref(), + ) + .await + })) + .await; + if published + .iter() + .filter(|state| matches!(state, Ok(state) if receipt(state.state))) + .count() + < write_quorum + { + return Err(DiskError::ErasureWriteQuorum); + } + // Physical cleanup is retryable after the certified decision. An + // unavailable participant retains the decision for its eventual return. + let ticket = context.recovery_ticket.clone(); + if let Ok(summary) = self.recover_delete_key(bucket, object, context).await + && summary.pending == 0 + && summary.blocked == 0 + && summary.failed == 0 + && let Some(ticket) = ticket.as_ref() + { + self.ctx.delete_recovery_hints().ack(ticket); + } + Ok(()) + } + + pub(crate) async fn recover_delete_bucket( + &self, + bucket: &str, + context: DeleteRecoveryContext<'_>, + ) -> Result { + self.recover_delete_prefix(bucket, "", context).await + } + + pub(crate) async fn recover_delete_prefix( + &self, + bucket: &str, + prefix: &str, + context: DeleteRecoveryContext<'_>, + ) -> Result { + let disks = self.disk_inventory().await; + if disks.len() != self.set_drive_count || disks.iter().any(Option::is_none) { + return Ok(DeleteRecoverySummary { + pending: 1, + ..Default::default() + }); + } + let pages = join_all(disks.iter().flatten().map(|disk| discover(disk, bucket, None))).await; + let mut keys = BTreeSet::new(); + let mut summary = DeleteRecoverySummary::default(); + for page in pages { + match page { + Ok(ids) => { + keys.extend( + ids.into_iter() + .map(|id| id.object) + .filter(|object| object.starts_with(prefix)), + ); + } + Err(error) if unavailable(&error) => { + summary.pending += 1; + } + Err(DiskError::FileCorrupt) => summary.failed += 1, + Err(error) => return Err(error), + } + } + // Every key requires complete participation. Avoid taking each key's + // lock and repeating discovery while this sweep already lacks a peer. + if summary.pending > 0 { + return Ok(summary); + } + // Lock order: caller's bucket lifecycle fence, then one exact key. + for object in keys { + match self.recover_delete_object(bucket, &object, context.clone()).await { + Ok(result) => summary.add(result), + Err(DiskError::FileCorrupt) => summary.failed += 1, + Err(error) => return Err(error), + } + } + Ok(summary) + } + + pub(crate) async fn recover_delete_object( + &self, + bucket: &str, + object: &str, + mut context: DeleteRecoveryContext<'_>, + ) -> Result { + // Lock order: caller's bucket lifecycle fence, then one exact key. + let guard = Arc::new( + self.acquire_write_lock_diag("recover_delete", bucket, object) + .await + .map_err(DiskError::other)?, + ); + if !context.dry_run { + let namespace_owner = context + .namespace_owner + .clone() + .unwrap_or_else(|| self.ctx.begin_namespace_commit()); + context.namespace_owner = Some(Arc::new((namespace_owner, guard.clone()))); + } + let can_commit = context.can_commit; + let fence = || can_commit() && !guard.is_lock_lost(); + context.can_commit = &fence; + self.recover_delete_key(bucket, object, context).await + } + + /// Caller retains the exact key's write lock and its lifecycle owner. + pub(in crate::set_disk) async fn recover_delete_key( + &self, + bucket: &str, + object: &str, + context: DeleteRecoveryContext<'_>, + ) -> Result { + let DeleteRecoveryContext { + incarnation, + dry_run, + explicit_delete, + repair_marker_source, + namespace_owner, + can_commit, + recovery_ticket: _, + } = context; + let disks = self.disk_inventory().await; + let complete_inventory = disks.len() == self.set_drive_count && disks.iter().all(Option::is_some); + let pages = join_all(disks.iter().flatten().map(|disk| discover(disk, bucket, Some(object)))).await; + let mut transactions = BTreeSet::new(); + let mut inspected = 0; + let mut offline = !complete_inventory; + let mut unsupported = false; + for page in pages { + match page { + Ok(ids) => { + inspected += 1; + transactions.extend(ids.into_iter().map(|id| id.transaction)); + } + Err(error) if unavailable(&error) => { + offline = true; + unsupported |= unsupported_recovery(&error); + } + Err(error) => return Err(error), + } + } + if offline { + return Ok(DeleteRecoverySummary { + discovered: transactions.len(), + pending: 1, + no_transactions_at_quorum: transactions.is_empty() && !unsupported && inspected > self.set_drive_count / 2, + ..Default::default() + }); + } + let mut summary = DeleteRecoverySummary { + discovered: transactions.len(), + ..Default::default() + }; + let mut all_states = BTreeMap::new(); + for transaction in transactions { + let mut complete = Vec::with_capacity(disks.len()); + for state in inspect(&disks, bucket, object, transaction).await { + match state { + Ok(state) => { + if let Some(journal) = &state.journal { + let witness = journal.original_digest.is_none() + && journal.target_digest.is_none() + && journal.requests.is_empty(); + if incarnation.is_some_and(|expected| journal.bucket_incarnation != Some(expected) && !witness) { + summary.blocked += 1; + return Ok(summary); + } + } + complete.push(state); + } + Err(error) if unavailable(&error) => { + summary.pending += 1; + return Ok(summary); + } + Err(error) => return Err(error), + } + } + all_states.insert(transaction, complete); + } + // Validate causal authority before prerequisite repair can publish metadata. + let transaction_order = causal_transaction_order(&all_states)?; + let prerequisite = if repair_marker_source && !dry_run { + let mut active = all_states + .iter() + .filter(|(_, copies)| copies.iter().any(|copy| copy.state != DeleteRecoveryState::Absent)); + let first = active.next(); + if active.next().is_none() { + first.and_then(|(&transaction, copies)| { + marker_source_to_repair(copies, incarnation).map(|source| (transaction, source)) + }) + } else { + None + } + } else { + None + }; + if let Some((transaction, source)) = prerequisite { + Box::pin(self.heal_delete_recovery_source(bucket, object, &source, can_commit, namespace_owner.as_ref())).await?; + let refreshed = inspect(&disks, bucket, object, transaction) + .await + .into_iter() + .collect::>>()?; + if !can_commit() { + return Err(DiskError::DiskOngoingReq); + } + if refreshed.iter().any(|copy| { + copy.journal + .as_ref() + .is_some_and(|journal| journal.bucket_incarnation != incarnation) + }) { + return Err(DiskError::OutdatedXLMeta); + } + all_states.insert(transaction, refreshed); + } + // Legacy snapshots contain no set decision. A fresh authorized removal + // applies only after the entire key is absent and every raw shape is safe. + let legacy_only = explicit_delete + && all_states.values().flatten().all(|copy| { + copy.current_digest.is_none() + && matches!(copy.state, DeleteRecoveryState::Absent | DeleteRecoveryState::LegacyAmbiguous) + && (copy.state != DeleteRecoveryState::LegacyAmbiguous || copy.explicit_delete_safe) + }); + // This is a fresh authorized delete of an absent unversioned key, + // rather than evidence about the outcome of a historical transaction. + let fresh_absence_decision = explicit_delete + && all_states.values().flatten().all(|copy| { + copy.current_digest.is_none() + && match copy.state { + DeleteRecoveryState::Absent => true, + DeleteRecoveryState::NewPrepared => copy.explicit_delete_safe, + DeleteRecoveryState::QuorumCommitted | DeleteRecoveryState::CommitFinished => { + copy.journal.as_ref().is_some_and(|journal| journal.target_digest.is_none()) + } + DeleteRecoveryState::LegacyAmbiguous => false, + } + }); + // Receipt cleanup alone does not mutate xl.meta. Returning peers and + // undecided preparations still need fences before and after recovery. + let metadata_may_change = all_states.values().any(|copies| { + !copies.iter().all(|copy| copy.state == DeleteRecoveryState::Absent) + && copies.iter().any(|copy| { + copy.state == DeleteRecoveryState::NewPrepared + || (copy.state == DeleteRecoveryState::Absent && copy.current_digest.is_some()) + || copy + .journal + .as_ref() + .is_some_and(|journal| copy.current_digest != journal.target_digest) + }) + }); + let namespace_owner = if dry_run { + None + } else { + Some(namespace_owner.unwrap_or_else(|| self.ctx.begin_namespace_commit())) + }; + if !dry_run && metadata_may_change { + self.invalidate_get_object_metadata_cache(bucket, object).await; + } + let authorities = all_states + .values() + .flatten() + .filter(|state| receipt(state.state)) + .filter_map(|state| state.journal.as_ref()) + .map(|journal| DecisionEvidence { + transaction: journal.transaction, + deleted_versions: journal.deleted_versions.iter().copied().collect(), + original_versions: journal.original_versions.iter().cloned().collect(), + predecessors: journal.predecessors.iter().copied().collect(), + target_absent: journal.target_digest.is_none(), + }) + .collect::>(); + let mut unresolved = false; + let mut mutated = false; + for transaction in transaction_order { + let copies = all_states.get_mut(&transaction).ok_or(DiskError::FileCorrupt)?; + if mutated { + // A sibling may have restored metadata or reclaimed shared + // evidence. Refresh fingerprints before acting on this group. + *copies = inspect(&disks, bucket, object, transaction) + .await + .into_iter() + .collect::>>()?; + } + if copies.iter().all(|state| state.state == DeleteRecoveryState::Absent) { + continue; + } + if copies.iter().any(|state| state.state == DeleteRecoveryState::LegacyAmbiguous) { + if legacy_only { + if dry_run { + summary.reclaimed += 1; + continue; + } + mutated = true; + if settle_group( + &disks, + bucket, + copies, + |copy| { + (copy.state == DeleteRecoveryState::LegacyAmbiguous).then_some(DeleteRecoveryAction::DiscardLegacy) + }, + can_commit, + namespace_owner.as_ref(), + &mut summary.removed_versions, + ) + .await? + { + summary.pending += 1; + unresolved = true; + } else { + summary.reclaimed += 1; + } + } else { + summary.blocked += 1; + unresolved = true; + } + continue; + } + let authority = copies + .iter() + .filter(|state| receipt(state.state)) + .filter_map(|state| state.journal.as_ref()) + .max_by_key(|journal| !journal.requests.is_empty()) + .cloned(); + let prepared = copies.iter().find_map(|state| state.journal.as_ref()); + let covered = prepared + .filter(|journal| !journal.deleted_versions.is_empty()) + .is_some_and(|journal| { + authorities.iter().any(|authority| { + journal.deleted_versions.iter().all(|version| { + authority.deleted_versions.contains(version) + && (!version.is_nil() + || journal + .original_versions + .iter() + .filter(|source| source.version_id.is_none_or(|id| id.is_nil())) + .all(|source| authority.original_versions.contains(source)) + || (authority.predecessors.contains(&journal.transaction) + && journal.target_digest.is_none() + && authority.target_absent + && copies.iter().all(|copy| { + copy.current_digest.is_none() || copy.current_versions == journal.original_versions + }))) + }) + }) + }); + let original_quorum = prepared.is_some_and(|journal| { + copies + .iter() + .filter(|copy| copy.current_versions == journal.original_versions) + .count() + > self.set_drive_count / 2 + }); + let all_decisions_precede = !authorities.is_empty() + && authorities.iter().all(|authority| { + copies + .iter() + .filter_map(|copy| copy.journal.as_ref()) + .any(|journal| journal.predecessors.contains(&authority.transaction)) + }); + if authority.is_some() || covered || fresh_absence_decision { + let certified_same_transaction = authority.is_some(); + let reference = authority + .or_else(|| copies.iter().find_map(|state| state.journal.clone())) + .ok_or(DiskError::FileCorrupt)?; + if dry_run { + if copies.iter().any(|copy| { + if certified_same_transaction + && copy.state == DeleteRecoveryState::NewPrepared + && copy.current_digest.is_none() + && !copy.commit_data_safe + { + copy.journal + .as_ref() + .is_none_or(|local| !local.matches_prepared_authority(&reference)) + } else if copy.state == DeleteRecoveryState::Absent { + copy.current_digest.is_some() && !reference.matches_adoption_source(©.current_versions) + } else { + !copy.commit_data_safe + } + }) { + summary.blocked += 1; + } else { + summary.committed += 1; + } + continue; + } + mutated = true; + let mut pending = settle_group( + &disks, + bucket, + copies, + |copy| { + Some( + if copy.state == DeleteRecoveryState::Absent + || (certified_same_transaction + && copy.state == DeleteRecoveryState::NewPrepared + && copy.current_digest.is_none() + && !copy.commit_data_safe) + { + DeleteRecoveryAction::AdoptCommit { + authority: Box::new(reference.clone()), + } + } else { + DeleteRecoveryAction::PublishCommit + }, + ) + }, + can_commit, + namespace_owner.as_ref(), + &mut summary.removed_versions, + ) + .await?; + if !pending { + pending = settle_group( + &disks, + bucket, + copies, + |_| Some(DeleteRecoveryAction::FinishCommit), + can_commit, + namespace_owner.as_ref(), + &mut summary.removed_versions, + ) + .await?; + } + if pending { + summary.pending += 1; + unresolved = true; + } else { + summary.committed += 1; + } + } else { + // A sibling decision or foreign generation can supersede this + // snapshot. Never guess by restoring the whole old backup. + if (!authorities.is_empty() && !original_quorum && !all_decisions_precede) + || copies + .iter() + .any(|state| state.state != DeleteRecoveryState::Absent && !state.rollback_data_complete) + { + summary.blocked += 1; + unresolved = true; + continue; + } + let originals = copies + .iter() + .filter_map(|state| state.journal.as_ref()) + .flat_map(|journal| journal.original_versions.iter()) + .collect::>(); + if copies.iter().any(|state| { + state.state == DeleteRecoveryState::Absent + && state.current_versions.iter().any(|version| !originals.contains(version)) + }) { + summary.blocked += 1; + unresolved = true; + continue; + } + if dry_run { + summary.rolled_back += 1; + continue; + } + mutated = true; + if settle_group( + &disks, + bucket, + copies, + |copy| (copy.state == DeleteRecoveryState::NewPrepared).then_some(DeleteRecoveryAction::Rollback), + can_commit, + namespace_owner.as_ref(), + &mut summary.removed_versions, + ) + .await? + { + summary.pending += 1; + unresolved = true; + } else { + summary.rolled_back += 1; + } + } + } + if !unresolved && !dry_run { + // GC is a key-wide barrier: every predecessor must have settled + // before the last surviving commit witness can disappear. + let mut settled = Vec::with_capacity(all_states.len()); + for transaction in all_states.keys() { + let fresh = inspect(&disks, bucket, object, *transaction).await; + if fresh.iter().any(|state| !matches!(state, Ok(state) if matches!(state.state, DeleteRecoveryState::Absent | DeleteRecoveryState::CommitFinished))) { + summary.pending += 1; + return Ok(summary); + } + settled.push(fresh.into_iter().collect::>>()?); + } + for (index, mut copies) in settled.into_iter().enumerate() { + if index > 0 { + copies = inspect(&disks, bucket, object, copies[0].id.transaction) + .await + .into_iter() + .collect::>>()?; + if copies + .iter() + .any(|copy| !matches!(copy.state, DeleteRecoveryState::Absent | DeleteRecoveryState::CommitFinished)) + { + summary.pending += 1; + return Ok(summary); + } + } + if settle_group( + &disks, + bucket, + &mut copies, + |copy| (copy.state == DeleteRecoveryState::CommitFinished).then_some(DeleteRecoveryAction::FinalizeCommit), + can_commit, + namespace_owner.as_ref(), + &mut summary.removed_versions, + ) + .await? + { + summary.pending += 1; + } + } + } + + if !dry_run && metadata_may_change { + self.invalidate_get_object_metadata_cache(bucket, object).await; + } + Ok(summary) + } +} + +// A lagging member may have the complete older suffix of a marker creation's +// live source. Repair that source before applying the original strict matcher. +fn marker_source_to_repair( + copies: &[DeleteRecoveryInspection], + incarnation: Option, +) -> Option { + let expected = incarnation.filter(|id| !id.is_nil())?; + let authority = copies + .iter() + .filter(|copy| receipt(copy.state)) + .find_map(|copy| copy.journal.as_ref())?; + let source = authority.original_versions.first()?; + let request = authority.requests.first()?; + if authority.bucket_incarnation != Some(expected) + || authority.requests.len() != 1 + || !request.deleted + || !request.mark_deleted + || request.version_id.is_none_or(|id| id.is_nil()) + || authority + .original_versions + .iter() + .any(|version| version.version_id == request.version_id) + || !authority.deleted_versions.is_empty() + || !authority.data_dirs.is_empty() + || authority.target_digest.is_none() + || source.version_id.is_none_or(|id| id.is_nil()) + || source.deleted + { + return None; + } + let mut missing_source = false; + for copy in copies { + match copy.state { + DeleteRecoveryState::QuorumCommitted | DeleteRecoveryState::CommitFinished => { + let journal = copy.journal.as_ref()?; + if journal.bucket_incarnation != Some(expected) + || journal.original_versions != authority.original_versions + || !journal.matches_prepared_authority(authority) + || copy.current_digest != journal.target_digest + { + return None; + } + } + DeleteRecoveryState::Absent => { + if copy.journal.is_some() + || copy.current_versions.is_empty() + || !authority.original_versions.ends_with(©.current_versions) + { + return None; + } + missing_source |= copy.current_versions.first() != Some(source); + } + _ => return None, + } + } + missing_source.then(|| source.clone()) +} + +fn unavailable(error: &DiskError) -> bool { + if unsupported_recovery(error) { + return true; + } + match error { + DiskError::DiskNotFound + | DiskError::FaultyDisk + | DiskError::DiskOngoingReq + | DiskError::Timeout + | DiskError::RemoteClientUnavailable(_) => true, + DiskError::Io(source) => { + if let Some(status) = crate::cluster::rpc::client::embedded_tonic_status(source) { + return crate::cluster::rpc::client::is_network_like_status(status); + } + matches!( + source.kind(), + std::io::ErrorKind::TimedOut + | std::io::ErrorKind::ConnectionRefused + | std::io::ErrorKind::ConnectionReset + | std::io::ErrorKind::BrokenPipe + | std::io::ErrorKind::NotConnected + | std::io::ErrorKind::ConnectionAborted + | std::io::ErrorKind::UnexpectedEof + ) + } + _ => false, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::bucket::metadata_sys::{self, init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks}; + use crate::disk::{DeleteOptions, ReadOptions, STORAGE_FORMAT_FILE}; + use crate::object_api::{ObjectOptions, PutObjReader}; + use crate::storage_api_contracts::bucket::{BucketOperations, DeleteBucketOptions, MakeBucketOptions}; + use crate::storage_api_contracts::object::{ObjectIO, ObjectOperations}; + use crate::store::ECStore; + use rustfs_filemeta::FileInfo; + use rustfs_heal_contracts::heal_channel::HealOpts; + use std::sync::Arc; + use tokio::io::AsyncReadExt; + + async fn fixture(bucket: &str) -> (Vec, Arc, Vec) { + let (dirs, store) = isolated_store_over_temp_disks().await; + // Manual recovery assertions need sole ownership of the seeded journals. + store.background_cancel_token().expect("fixture background token").cancel(); + tokio::task::yield_now().await; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create authoritative recovery bucket"); + let disks = store.pools[0].disk_set[0] + .disk_inventory() + .await + .into_iter() + .map(|disk| disk.expect("fixture drive is online")) + .collect(); + (dirs, store, disks) + } + + async fn put(store: &Arc, bucket: &str, body: &[u8], versioned: bool) -> Option { + let mut reader = PutObjReader::from_vec(body.to_vec()); + store + .put_object( + bucket, + "object", + &mut reader, + &ObjectOptions { + versioned, + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("put recovery source") + .version_id + } + + async fn applied(store: &Arc, disks: &[DiskStore], bucket: &str, version: Option, count: usize) -> Uuid { + applied_at(store, disks, bucket, version, count, Uuid::new_v4()).await + } + + async fn applied_at( + store: &Arc, + disks: &[DiskStore], + bucket: &str, + version: Option, + count: usize, + transaction: Uuid, + ) -> Uuid { + let incarnation = store + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("source bucket identity"); + for disk in disks.iter().take(count) { + disk.delete_version_recoverable( + bucket, + "object", + FileInfo { + name: "object".to_string(), + version_id: version, + ..Default::default() + }, + false, + DeleteOptions { + old_data_dir: Some(transaction), + delete_recovery_bucket_incarnation: Some(incarnation), + ..Default::default() + }, + ) + .await + .expect("apply prepared delete without publishing a set decision"); + } + transaction + } + + async fn publish(disk: &DiskStore, bucket: &str, transaction: Uuid) { + let state = disk + .inspect_delete_transaction(bucket, "object", transaction) + .await + .expect("inspect prepared source"); + disk.settle_delete_transaction(bucket, "object", transaction, state.fingerprint, DeleteRecoveryAction::PublishCommit) + .await + .expect("persist a decision after the simulated metadata quorum"); + } + + async fn body(store: &Arc, bucket: &str, version: Option) -> Vec { + let opts = ObjectOptions { + version_id: version.map(|version| version.to_string()), + ..Default::default() + }; + let mut reader = store + .get_object_reader(bucket, "object", None, http::HeaderMap::new(), &opts) + .await + .expect("open recovered object"); + let mut body = Vec::new(); + reader + .stream + .read_to_end(&mut body) + .await + .expect("read exact recovered bytes"); + body + } + + #[tokio::test] + async fn delete_recovery_set_rolls_back_undecided_inline_bytes() { + let bucket = "recover-undecided"; + let (_dirs, store, disks) = fixture(bucket).await; + let original = vec![21; 4096]; + put(&store, bucket, &original, false).await; + applied(&store, &disks, bucket, None, disks.len()).await; + let report = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("recover undisputed prepared delete"); + assert!( + report.detail.contains("rolled_back=1"), + "actual rollback must be reported: {}", + report.detail + ); + assert_eq!(body(&store, bucket, None).await, original); + } + + #[tokio::test] + async fn delete_recovery_corrupt_bucket_does_not_prevent_other_bucket_rollback() { + let bad_bucket = "recover-corrupt-bucket"; + let good_bucket = "recover-good-bucket"; + let (dirs, store, disks) = fixture(bad_bucket).await; + store + .make_bucket(good_bucket, &MakeBucketOptions::default()) + .await + .expect("create healthy bucket"); + put(&store, bad_bucket, &[61; 4096], false).await; + let bad_transaction = applied(&store, &disks, bad_bucket, None, disks.len()).await; + let good_bytes = vec![62; 4096]; + put(&store, good_bucket, &good_bytes, false).await; + applied(&store, &disks, good_bucket, None, disks.len()).await; + for dir in &dirs { + std::fs::write( + dir.path() + .join(bad_bucket) + .join("object") + .join(bad_transaction.to_string()) + .join("delete-prepared"), + b"{broken", + ) + .expect("inject malformed preparation journal"); + } + + let failed = store + .recover_delete_transactions(bad_bucket, &HealOpts::default()) + .await + .expect("bucket corruption is an explicit partial recovery result"); + assert_eq!(failed.delete_recovery.expect("recovery status").failed, 1); + let error = store + .recover_delete_transactions_at_incarnation(bad_bucket, Some("object"), None, &HealOpts::default()) + .await + .expect_err("exact object recovery must retain typed corruption"); + assert!(matches!(error, crate::error::StorageError::FileCorrupt)); + let recovered = store + .recover_delete_transactions(good_bucket, &HealOpts::default()) + .await + .expect("another bucket must remain recoverable"); + assert!(recovered.detail.contains("rolled_back=1")); + assert!(!recovered.delete_recovery.expect("healthy recovery status").has_unresolved()); + assert_eq!(body(&store, good_bucket, None).await, good_bytes); + } + + #[tokio::test] + async fn delete_recovery_corrupt_key_does_not_prevent_other_key_rollback() { + let bucket = "recover-corrupt-key"; + let (dirs, store, disks) = fixture(bucket).await; + let healthy_bytes = vec![63; 4096]; + put(&store, bucket, &healthy_bytes, false).await; + applied(&store, &disks, bucket, None, disks.len()).await; + let mut reader = PutObjReader::from_vec(vec![64; 4096]); + store + .put_object( + bucket, + "a-corrupt", + &mut reader, + &ObjectOptions { + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("write corrupt-key fixture"); + let transaction = Uuid::new_v4(); + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("bucket identity"); + for disk in &disks { + disk.delete_version_recoverable( + bucket, + "a-corrupt", + FileInfo { + name: "a-corrupt".to_owned(), + ..Default::default() + }, + false, + DeleteOptions { + old_data_dir: Some(transaction), + delete_recovery_bucket_incarnation: Some(incarnation), + ..Default::default() + }, + ) + .await + .expect("prepare corrupt-key fixture"); + } + let damaged = dirs[0] + .path() + .join(bucket) + .join("a-corrupt") + .join(transaction.to_string()) + .join(crate::disk::STORAGE_FORMAT_FILE_BACKUP); + std::fs::write(&damaged, b"corrupt metadata").expect("damage one backup while retaining discoverable journal"); + + let report = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("per-key corruption must not abort the remaining prefix"); + + assert_eq!(report.delete_recovery.expect("recovery status").failed, 1); + assert!(report.detail.contains("rolled_back=1")); + assert_eq!(std::fs::read(damaged).expect("corrupt proof is retained"), b"corrupt metadata"); + assert_eq!(body(&store, bucket, None).await, healthy_bytes); + } + + #[tokio::test] + async fn delete_recovery_offline_bucket_reports_pending_without_hiding_exact_object_error() { + let bucket = "recover-offline-status"; + let (_dirs, store, _disks) = fixture(bucket).await; + store.pools[0].disk_set[0].disks.write().await[3] = None; + + let result = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("offline recovery must expose a pending summary even with no discovered transactions"); + let status = result.delete_recovery.expect("offline status is retained"); + assert_eq!(status.pending, 1); + assert_eq!(status.failed, 0); + assert!(status.has_unresolved()); + let error = store + .recover_delete_transactions_at_incarnation(bucket, Some("object"), None, &HealOpts::default()) + .await + .expect_err("exact-object workers require a typed retry error"); + assert!(matches!(error, crate::error::StorageError::ErasureReadQuorum)); + } + + fn snapshot(root: &std::path::Path) -> BTreeMap> { + let mut files = BTreeMap::new(); + let mut pending = vec![root.to_path_buf()]; + while let Some(directory) = pending.pop() { + for entry in std::fs::read_dir(directory).unwrap() { + let path = entry.unwrap().path(); + if path.is_dir() { + pending.push(path); + } else { + files.insert(path.strip_prefix(root).unwrap().to_path_buf(), std::fs::read(path).unwrap()); + } + } + } + files + } + + #[tokio::test] + async fn delete_recovery_marker_source_heal_preserves_foreign_history_and_damaged_data() { + for shape in ["live-backup", "foreign", "partial-history", "damaged-data", "causal-cycle"] { + let bucket = "recover-marker-source-guards"; + let (dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + store + .make_bucket( + bucket, + &MakeBucketOptions { + versioning_enabled: true, + ..Default::default() + }, + ) + .await + .unwrap(); + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.unwrap(); + let set = &store.pools[0].disk_set[0]; + let disks: Vec<_> = set.disk_inventory().await.into_iter().map(Option::unwrap).collect(); + let original = vec![71; 1024 * 1024]; + let oldest = put(&store, bucket, &original, true).await.unwrap(); + let first = put(&store, bucket, &original, true).await.unwrap(); + set.disks.write().await[3] = None; + let payload = vec![72; 1024 * 1024]; + let source = put(&store, bucket, &payload, true).await.unwrap(); + store + .delete_object( + bucket, + "object", + ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect("publish marker while one member lacks the source version"); + set.disks.write().await[3] = Some(disks[3].clone()); + let mut first_info = disks[3] + .read_version("", bucket, "object", &first.to_string(), &ReadOptions::default()) + .await + .unwrap(); + match shape { + "live-backup" => { + let root = dirs[3].path().join(bucket).join("object"); + let backup = root.join(first_info.data_dir.unwrap().to_string()).join("xl.meta.bkp"); + tokio::fs::write(backup, tokio::fs::read(root.join("xl.meta")).await.unwrap()) + .await + .unwrap(); + } + "foreign" => { + first_info.version_id = Some(Uuid::new_v4()); + first_info.mod_time = Some(time::OffsetDateTime::now_utc()); + disks[3].write_metadata("", bucket, "object", first_info).await.unwrap(); + } + "partial-history" => { + disks[3] + .delete_version( + bucket, + "object", + FileInfo { + name: "object".into(), + version_id: Some(oldest), + ..Default::default() + }, + false, + DeleteOptions::default(), + ) + .await + .unwrap(); + } + "damaged-data" => { + // Three metadata sources meet write quorum, but corrupting + // two shards leaves fewer than EC2+2's two required readers. + for (index, disk) in disks.iter().take(2).enumerate() { + let info = disk + .read_version("", bucket, "object", &source.to_string(), &ReadOptions::default()) + .await + .unwrap(); + let part = dirs[index] + .path() + .join(bucket) + .join("object") + .join(info.data_dir.unwrap().to_string()) + .join("part.1"); + let mut bytes = tokio::fs::read(&part).await.unwrap(); + *bytes.last_mut().unwrap() ^= 1; + tokio::fs::write(part, bytes).await.unwrap(); + } + } + "causal-cycle" => { + for dir in dirs.iter().take(3) { + let root = dir.path().join(bucket).join("object"); + let mut found = 0; + for entry in std::fs::read_dir(root).unwrap() { + let entry = entry.unwrap(); + let prepared = entry.path().join("delete-prepared"); + if !prepared.is_file() { + continue; + } + let transaction = entry.file_name().to_str().unwrap().to_owned(); + let mut journal: serde_json::Value = + serde_json::from_slice(&std::fs::read(&prepared).unwrap()).unwrap(); + journal["predecessors"] = serde_json::json!([transaction]); + std::fs::write(prepared, serde_json::to_vec(&journal).unwrap()).unwrap(); + let receipt = entry.path().join(crate::disk::local::DELETE_QUORUM_COMMITTED_FILE); + if receipt.is_file() { + let mut value: serde_json::Value = + serde_json::from_slice(&std::fs::read(&receipt).unwrap()).unwrap(); + value["journal"] = journal; + std::fs::write(receipt, serde_json::to_vec(&value).unwrap()).unwrap(); + } + found += 1; + } + assert_eq!(found, 1, "one active marker journal must exercise prerequisite rejection"); + } + } + _ => unreachable!(), + } + let before: Vec<_> = dirs + .iter() + .map(|dir| snapshot(&dir.path().join(bucket).join("object"))) + .collect(); + let result = store + .heal_object_at_incarnation( + bucket, + "object", + &source.to_string(), + incarnation, + &HealOpts { + scan_mode: rustfs_heal_contracts::heal_channel::HealScanMode::Deep, + ..Default::default() + }, + ) + .await; + if shape == "live-backup" { + assert!(result.unwrap().error.is_none(), "plain live backups cannot block source repair"); + assert_eq!(body(&store, bucket, Some(source)).await, payload); + assert_eq!(body(&store, bucket, Some(first)).await, original); + } else { + assert!(result.is_err() || result.unwrap().error.is_some(), "{shape} must fail safely"); + for (dir, expected) in dirs.iter().zip(before) { + assert_eq!(snapshot(&dir.path().join(bucket).join("object")), expected, "{shape}"); + } + } + } + } + + #[tokio::test] + async fn delete_recovery_admin_prefix_excludes_other_keys_and_restores_hidden_match() { + let bucket = "recover-prefix-scope"; + let (dirs, store, disks) = fixture(bucket).await; + let original = vec![27; 4096]; + put(&store, bucket, &original, false).await; + let transaction = applied(&store, &disks, bucket, None, disks.len()).await; + let outside = store + .recover_delete_prefix_at_incarnation(bucket, "other/", None, &HealOpts::default()) + .await + .expect("inspect unrelated prefix"); + assert!(outside.is_none()); + for dir in &dirs { + assert!( + dir.path() + .join(bucket) + .join("object") + .join(transaction.to_string()) + .join("xl.meta.bkp") + .exists() + ); + } + let matching = store + .recover_delete_prefix_at_incarnation(bucket, "obj", None, &HealOpts::default()) + .await + .expect("recover matching hidden key") + .expect("report matching transaction"); + assert!(matching.detail.contains("rolled_back=1")); + assert_eq!(body(&store, bucket, None).await, original); + } + + #[tokio::test] + async fn delete_recovery_set_commits_receipt_and_unblocks_bucket() { + let bucket = "recover-committed"; + let (_dirs, store, disks) = fixture(bucket).await; + put(&store, bucket, &vec![22; 1024 * 1024], false).await; + let transaction = applied(&store, &disks, bucket, None, disks.len()).await; + publish(&disks[0], bucket, transaction).await; + let report = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("complete certified delete"); + assert!( + report.detail.contains("committed=1"), + "certified replay must be reported: {}", + report.detail + ); + assert!( + store + .get_object_info(bucket, "object", &ObjectOptions::default()) + .await + .is_err() + ); + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("non-force bucket deletion must succeed after proof GC"); + } + + #[tokio::test] + async fn delete_recovery_set_handles_prepared_subset() { + let bucket = "recover-subset"; + let (_dirs, store, disks) = fixture(bucket).await; + let original = vec![23; 4096]; + put(&store, bucket, &original, false).await; + applied(&store, &disks, bucket, None, 1).await; + let report = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("untouched peers remain valid evidence"); + assert!(report.detail.contains("rolled_back=1")); + assert_eq!(body(&store, bucket, None).await, original); + } + + #[tokio::test] + async fn delete_recovery_set_adopts_untouched_returning_peer() { + let bucket = "recover-untouched"; + let (_dirs, store, disks) = fixture(bucket).await; + put(&store, bucket, &vec![24; 1024 * 1024], false).await; + let transaction = applied(&store, &disks, bucket, None, 3).await; + for disk in disks.iter().take(3) { + publish(disk, bucket, transaction).await; + } + let report = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("matching untouched generation must converge"); + assert!(report.detail.contains("committed=1")); + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("returning old replica must not leave bucket residue"); + } + + #[tokio::test] + async fn delete_recovery_tracks_only_observed_removed_version() { + let bucket = "recover-actual-removal"; + let (_dirs, store, disks) = fixture(bucket).await; + let target = put(&store, bucket, &[41; 4096], true).await.unwrap(); + let retained = put(&store, bucket, &[42; 4096], true).await.unwrap(); + let transaction = applied(&store, &disks, bucket, Some(target), 3).await; + publish(&disks[0], bucket, transaction).await; + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.unwrap(); + let set = &store.pools[0].disk_set[0]; + let report = set + .recover_delete_object(bucket, "object", DeleteRecoveryContext::new(Some(incarnation), &|| true)) + .await + .unwrap(); + assert_eq!(report.pending, 0); + assert_eq!(report.blocked, 0); + assert_eq!(report.removed_versions, HashSet::from([target])); + assert_eq!(body(&store, bucket, Some(retained)).await, vec![42; 4096]); + let replay = set + .recover_delete_object(bucket, "object", DeleteRecoveryContext::new(Some(incarnation), &|| true)) + .await + .unwrap(); + assert!(replay.removed_versions.is_empty(), "receipt replay cannot claim a second removal"); + } + + #[tokio::test] + async fn delete_recovery_set_retains_decisions_until_offline_peer_returns() { + let bucket = "recover-offline"; + let (dirs, store, disks) = fixture(bucket).await; + put(&store, bucket, &vec![25; 4096], false).await; + let transaction = applied(&store, &disks, bucket, None, 4).await; + let set = &store.pools[0].disk_set[0]; + set.disks.write().await[3] = None; + let available = set.disk_inventory().await; + set.publish_delete_commit(bucket, "object", transaction, &available, 3, DeleteRecoveryContext::new(None, &|| true)) + .await + .expect("commit quorum remains available"); + for dir in dirs.iter().take(3) { + assert!( + dir.path() + .join(bucket) + .join("object") + .join(transaction.to_string()) + .join(crate::disk::local::DELETE_QUORUM_COMMITTED_FILE) + .exists(), + "decision must survive an unavailable participant" + ); + } + set.disks.write().await[3] = Some(disks[3].clone()); + store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("returning prepared peer must roll forward"); + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("proof collection follows complete convergence"); + } + + #[tokio::test] + async fn delete_recovery_set_never_resurrects_prepared_old_null_after_new_delete() { + let bucket = "recover-null-history"; + let (_dirs, store, disks) = fixture(bucket).await; + put(&store, bucket, &vec![26; 4096], false).await; + applied(&store, &disks, bucket, None, 4).await; + let set = &store.pools[0].disk_set[0]; + set.disks.write().await[3] = None; + put(&store, bucket, &vec![27; 4096], false).await; + store + .delete_object(bucket, "object", ObjectOptions::default()) + .await + .expect("later visible generation can commit with quorum"); + set.disks.write().await[3] = Some(disks[3].clone()); + store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("later deletion must supersede hidden old null residue"); + assert!( + store + .get_object_info(bucket, "object", &ObjectOptions::default()) + .await + .is_err(), + "old bytes must never become visible again" + ); + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("both generations' transactions converge"); + } + + #[tokio::test] + async fn delete_recovery_set_missing_version_does_not_ack_hidden_rollback() { + let bucket = "recover-version-history"; + let (_dirs, store, disks) = fixture(bucket).await; + metadata_sys::update_in(&store.ctx, bucket, crate::bucket::metadata::BUCKET_VERSIONING_CONFIG, + br#"Enabled"#.to_vec()) + .await.expect("enable versioning"); + let first = put(&store, bucket, &vec![28; 4096], true).await.expect("first version"); + let survivor = vec![29; 4096]; + let second = put(&store, bucket, &survivor, true).await.expect("second version"); + applied(&store, &disks, bucket, Some(first), 4).await; + let set = &store.pools[0].disk_set[0]; + set.disks.write().await[3] = None; + let opts = ObjectOptions { + version_id: Some(first.to_string()), + versioned: true, + ..Default::default() + }; + assert!( + store.delete_object(bucket, "object", opts.clone()).await.is_err(), + "hidden prepared version cannot receive an idempotent success while a peer is unavailable" + ); + set.disks.write().await[3] = Some(disks[3].clone()); + store + .delete_object(bucket, "object", opts) + .await + .expect("recover then execute the currently authorized exact-version delete"); + assert!( + store + .get_object_info( + bucket, + "object", + &ObjectOptions { + version_id: Some(first.to_string()), + ..Default::default() + } + ) + .await + .is_err() + ); + assert_eq!(body(&store, bucket, Some(second)).await, survivor); + } + + #[tokio::test] + async fn delete_recovery_set_dry_run_preserves_every_file() { + let bucket = "recover-dry-run"; + let (dirs, store, disks) = fixture(bucket).await; + put(&store, bucket, &vec![30; 4096], false).await; + let transaction = applied(&store, &disks, bucket, None, 4).await; + let before = disks[0] + .inspect_delete_transaction(bucket, "object", transaction) + .await + .expect("before inspection"); + let report = store + .recover_delete_transactions( + bucket, + &HealOpts { + dry_run: true, + ..Default::default() + }, + ) + .await + .expect("dry-run report"); + assert!(report.detail.contains("dry run")); + let after = disks[0] + .inspect_delete_transaction(bucket, "object", transaction) + .await + .expect("after inspection"); + assert_eq!(before.fingerprint, after.fingerprint, "dry run may not modify recovery evidence"); + for dir in dirs { + assert!(!dir.path().join(bucket).join("object").join(STORAGE_FORMAT_FILE).exists()); + } + } + #[tokio::test] + async fn delete_recovery_set_admin_object_restores_hidden_metadata() { + let bucket = "recover-admin-object"; + let (_dirs, store, disks) = fixture(bucket).await; + let payload = vec![41; 4096]; + put(&store, bucket, &payload, false).await; + applied(&store, &disks, bucket, None, 4).await; + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("bucket identity"); + let result = store + .heal_object_at_incarnation(bucket, "object", "", incarnation, &HealOpts::default()) + .await + .expect("admin object recovery"); + assert!(result.error.is_none(), "recovered object should heal cleanly: {:?}", result.error); + assert_eq!(body(&store, bucket, None).await, payload); + } + + #[tokio::test] + async fn delete_recovery_current_version_delete_preserves_legacy_on_missing_participant() { + use crate::storage_api_contracts::object::ObjectToDelete; + for batch in [false, true] { + let bucket = "recover-current-with-legacy"; + let (dirs, store, disks) = fixture(bucket).await; + let original = vec![44; 1024 * 1024]; + let old_version = put(&store, bucket, &original, true).await.expect("historical version"); + let legacy = Uuid::new_v4(); + disks[3] + .delete_version( + bucket, + "object", + FileInfo { + name: "object".into(), + version_id: Some(old_version), + ..Default::default() + }, + false, + DeleteOptions { + old_data_dir: Some(legacy), + ..Default::default() + }, + ) + .await + .expect("historical disk-local mutation without a set decision"); + let backup = dirs[3] + .path() + .join(bucket) + .join("object") + .join(legacy.to_string()) + .join("xl.meta.bkp"); + let before = tokio::fs::read(&backup).await.expect("historical backup"); + let set = &store.pools[0].disk_set[0]; + set.disks.write().await[3] = None; + let current = put(&store, bucket, &vec![45; 4096], true) + .await + .expect("current version at quorum"); + set.disks.write().await[3] = Some(disks[3].clone()); + if batch { + let (_, errors) = store + .delete_objects( + bucket, + vec![ObjectToDelete { + object_name: "object".into(), + version_id: Some(current), + ..Default::default() + }], + ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await; + assert!(errors.iter().all(Option::is_none), "current deletion at quorum: {errors:?}"); + } else { + store + .delete_object( + bucket, + "object", + ObjectOptions { + versioned: true, + version_id: Some(current.to_string()), + ..Default::default() + }, + ) + .await + .expect("current deletion at quorum"); + } + assert_eq!(tokio::fs::read(&backup).await.expect("legacy backup retained"), before); + let state = disks[3] + .inspect_delete_transaction(bucket, "object", legacy) + .await + .expect("legacy inspection"); + assert_eq!(state.state, DeleteRecoveryState::LegacyAmbiguous); + assert_eq!(body(&store, bucket, Some(old_version)).await, original); + } + } + + #[tokio::test] + async fn delete_recovery_dry_run_matches_adoption_of_already_absent_non_null_version() { + let bucket = "recover-preview-non-null"; + let (_dirs, store, disks) = fixture(bucket).await; + let first = put(&store, bucket, &vec![46; 4096], true).await.expect("first version"); + let remaining = vec![47; 4096]; + let second = put(&store, bucket, &remaining, true).await.expect("remaining version"); + disks[3] + .delete_version( + bucket, + "object", + FileInfo { + name: "object".into(), + version_id: Some(first), + ..Default::default() + }, + false, + DeleteOptions::default(), + ) + .await + .expect("returning peer already lacks the explicit version"); + let transaction = applied(&store, &disks, bucket, Some(first), 3).await; + publish(&disks[0], bucket, transaction).await; + let preview = store + .recover_delete_transactions( + bucket, + &HealOpts { + dry_run: true, + ..Default::default() + }, + ) + .await + .expect("preview certified recovery"); + assert!( + preview.detail.contains("committed=1") && preview.detail.contains("blocked=0"), + "{}", + preview.detail + ); + let actual = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("recover certified removal"); + assert!( + actual.detail.contains("committed=1") && actual.detail.contains("blocked=0"), + "{}", + actual.detail + ); + assert_eq!(body(&store, bucket, Some(second)).await, remaining); + assert!( + store + .get_object_info( + bucket, + "object", + &ObjectOptions { + version_id: Some(first.to_string()), + ..Default::default() + } + ) + .await + .is_err() + ); + } + + #[tokio::test] + async fn delete_recovery_set_legacy_requires_explicit_removal() { + let bucket = "recover-legacy-remove"; + let (dirs, store, disks) = fixture(bucket).await; + put(&store, bucket, &vec![42; 4096], false).await; + let transaction = Uuid::new_v4(); + for disk in &disks { + disk.delete_version( + bucket, + "object", + FileInfo { + name: "object".to_owned(), + ..Default::default() + }, + false, + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await + .expect("create undecided historical delete shape"); + } + let report = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("automatic report"); + assert!(report.detail.contains("blocked=1")); + for dir in &dirs { + assert!( + dir.path() + .join(bucket) + .join("object") + .join(transaction.to_string()) + .join("xl.meta.bkp") + .exists() + ); + } + let dry = store + .recover_delete_transactions( + bucket, + &HealOpts { + dry_run: true, + remove: true, + ..Default::default() + }, + ) + .await + .expect("authorized-removal dry run"); + assert!(dry.detail.contains("reclaimed=1")); + for dir in &dirs { + assert!(dir.path().join(bucket).join("object").exists()); + } + let report = store + .recover_delete_transactions( + bucket, + &HealOpts { + remove: true, + ..Default::default() + }, + ) + .await + .expect("explicit admin removal"); + assert!(report.detail.contains("reclaimed=1")); + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("non-force bucket deletion"); + } + + #[tokio::test] + async fn delete_recovery_set_old_null_decision_cannot_commit_new_undecided_generation() { + let bucket = "recover-inverse-null-history"; + let (_dirs, store, disks) = fixture(bucket).await; + put(&store, bucket, &vec![51; 4096], false).await; + let old_transaction = applied(&store, &disks, bucket, None, 4).await; + let set = &store.pools[0].disk_set[0]; + set.disks.write().await[3] = None; + set.publish_delete_commit( + bucket, + "object", + old_transaction, + &set.disk_inventory().await, + 3, + DeleteRecoveryContext::new(None, &|| true), + ) + .await + .expect("old generation decision while peer offline"); + let survivor = vec![52; 4096]; + put(&store, bucket, &survivor, false).await; + let new_transaction = applied(&store, &disks, bucket, None, 3).await; + set.disks.write().await[3] = Some(disks[3].clone()); + let report = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("causal recovery across old receipt"); + assert!(report.detail.contains("rolled_back=1")); + assert_eq!( + body(&store, bucket, None).await, + survivor, + "an old null-version receipt cannot authorize deleting a later generation" + ); + for disk in &disks { + assert_eq!( + disk.inspect_delete_transaction(bucket, "object", new_transaction) + .await + .expect("new transaction settled") + .state, + DeleteRecoveryState::Absent + ); + } + store + .delete_object(bucket, "object", ObjectOptions::default()) + .await + .expect("delete recovered generation explicitly"); + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("all proof is eventually collected"); + } + + #[tokio::test] + async fn delete_recovery_set_ordinary_missing_single_and_batch_keep_quorum_semantics() { + let bucket = "recover-missing-quorum"; + let (_dirs, store, disks) = fixture(bucket).await; + let set = &store.pools[0].disk_set[0]; + set.disks.write().await[3] = None; + let error = store + .delete_object(bucket, "single-missing", ObjectOptions::default()) + .await + .expect_err("ordinary missing storage result"); + assert!( + crate::error::is_err_object_not_found(&error), + "S3 may translate missing key into idempotent204; not a bucket/quorum error: {error}" + ); + let objects = ["batch-one", "batch-two"] + .into_iter() + .map(|name| crate::storage_api_contracts::object::ObjectToDelete { + object_name: name.to_owned(), + ..Default::default() + }) + .collect(); + let (deleted, errors) = store.delete_objects(bucket, objects, ObjectOptions::default()).await; + assert!( + errors.iter().all(Option::is_none), + "ordinary batch no-op must succeed at available quorum: {errors:?}" + ); + assert_eq!(deleted.len(), 2); + for disk in disks.iter().take(3) { + for key in ["single-missing", "batch-one", "batch-two"] { + assert!( + disk.discover_delete_transactions(bucket, Some(key), None, 128) + .await + .expect("no-op leaves no recovery journals") + .transactions + .is_empty() + ); + } + } + set.disks.write().await[3] = Some(disks[3].clone()); + let payload = vec![53; 4096]; + put(&store, bucket, &payload, false).await; + assert_eq!(body(&store, bucket, None).await, payload); + } + #[tokio::test] + async fn delete_recovery_set_fresh_delete_resolves_unknown_null_predecessor() { + let bucket = "recover-unknown-null-history"; + let (_dirs, store, disks) = fixture(bucket).await; + put(&store, bucket, &vec![54; 4096], false).await; + let transaction = Uuid::new_v4(); + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("bucket identity"); + disks[3] + .delete_version_recoverable( + bucket, + "object", + FileInfo { + name: "object".to_owned(), + ..Default::default() + }, + false, + DeleteOptions { + old_data_dir: Some(transaction), + delete_recovery_bucket_incarnation: Some(incarnation), + ..Default::default() + }, + ) + .await + .expect("only future-offline peer prepared old delete"); + let set = &store.pools[0].disk_set[0]; + set.disks.write().await[3] = None; + put(&store, bucket, &vec![55; 4096], false).await; + store + .delete_object(bucket, "object", ObjectOptions::default()) + .await + .expect("later generation deleted at quorum"); + set.disks.write().await[3] = Some(disks[3].clone()); + let report = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("unknown causal history is reported"); + assert!( + report.detail.contains("blocked=1"), + "automatic recovery cannot invent causality: {}", + report.detail + ); + store + .delete_object(bucket, "object", ObjectOptions::default()) + .await + .expect("fresh authorized delete certifies current absence"); + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("fresh intent drains all proven unprotected null history"); + } + #[test] + fn delete_recovery_network_errors_do_not_mask_application_corruption() { + assert!(unavailable(&DiskError::RemoteClientUnavailable( + "transport error: connection refused".to_owned() + ))); + assert!(unavailable(&DiskError::from(tonic::Status::unavailable("peer disconnected")))); + assert!( + unavailable(&DiskError::FaultyDisk), + "a drive disabled by health cannot vote in recovery inspection" + ); + assert!(!unavailable(&DiskError::FileCorrupt)); + assert!(!unavailable(&DiskError::other("invalid journal containing connection refused"))); + assert!(!unavailable(&DiskError::from(tonic::Status::internal("connection refused")))); + assert!(!unavailable(&DiskError::other("bucket unavailable-logs has corrupt metadata"))); + } + + #[tokio::test] + async fn delete_recovery_chained_committed_versions_finish_after_current_metadata_disappears() { + let bucket = "recover-chained-versions"; + let (dirs, store, disks) = fixture(bucket).await; + let first = put(&store, bucket, &vec![61; 1024 * 1024], true) + .await + .expect("first version"); + let second = put(&store, bucket, &vec![62; 4096], true).await.expect("second version"); + let set = &store.pools[0].disk_set[0]; + let returning_key = dirs[3].path().join(bucket).join("object"); + let before_outage = snapshot(&returning_key); + // A missing slot can be renewed by the endpoint monitor. Keep its + // original identity attached while rejecting I/O until explicit return. + disks[3].close().await.expect("stop returning drive's health monitor"); + disks[3].force_offline_for_test(); + set.connect_disks().await; + assert!(Arc::ptr_eq(set.disks.read().await[3].as_ref().unwrap(), &disks[3])); + assert_eq!(disks[3].runtime_state(), crate::disk::health_state::RuntimeDriveHealthState::Offline); + assert!(matches!(discover(&disks[3], bucket, Some("object")).await, Err(DiskError::FaultyDisk))); + for version in [first, second] { + store + .delete_object( + bucket, + "object", + ObjectOptions { + versioned: true, + version_id: Some(version.to_string()), + ..Default::default() + }, + ) + .await + .expect("commit each explicit version at quorum while peer is absent"); + } + assert!( + store + .get_object_info(bucket, "object", &ObjectOptions::default()) + .await + .is_err() + ); + assert_eq!(snapshot(&returning_key), before_outage, "offline peer retains both deleted versions"); + assert_eq!(disks[3].runtime_state(), crate::disk::health_state::RuntimeDriveHealthState::Offline); + disks[3].reset_health_for_store_init_retry(); + let report = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("older certified target may be superseded by a later certified absence"); + assert!( + report.detail.contains("committed=2"), + "both retained decisions must converge: {}", + report.detail + ); + for version in [first, second] { + assert!( + store + .get_object_info( + bucket, + "object", + &ObjectOptions { + version_id: Some(version.to_string()), + ..Default::default() + } + ) + .await + .is_err(), + "certified cleanup must not resurrect either deleted version" + ); + } + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("no retained version decision may keep the empty bucket nonempty"); + } + + #[tokio::test] + async fn delete_recovery_partial_receipt_converges_after_later_version_absence() { + let bucket = "recover-partial-chained-versions"; + let (dirs, store, disks) = fixture(bucket).await; + let first = put(&store, bucket, &vec![63; 1024 * 1024], true) + .await + .expect("first version"); + let second = put(&store, bucket, &vec![64; 4096], true).await.expect("second version"); + let transaction = applied(&store, &disks, bucket, Some(first), 4).await; + // A crash after the first durable decision leaves other copies prepared. + publish(&disks[0], bucket, transaction).await; + let later = applied(&store, &disks, bucket, Some(second), 4).await; + publish(&disks[0], bucket, later).await; + let dry = store + .recover_delete_transactions( + bucket, + &HealOpts { + dry_run: true, + ..Default::default() + }, + ) + .await + .expect("dry-run uses the same certified prepared-copy authority"); + assert!( + dry.detail.contains("committed=2") && dry.detail.contains("blocked=0"), + "accurate dry-run: {}", + dry.detail + ); + for dir in &dirs { + for id in [transaction, later] { + assert!( + dir.path() + .join(bucket) + .join("object") + .join(id.to_string()) + .join("delete-prepared") + .exists(), + "dry-run must retain every preparation record" + ); + } + } + store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("same-transaction receipt certifies each prepared absent copy"); + for dir in &dirs { + assert!(!dir.path().join(bucket).join("object").exists(), "all related proof must converge"); + } + for version in [first, second] { + assert!( + store + .get_object_info( + bucket, + "object", + &ObjectOptions { + version_id: Some(version.to_string()), + ..Default::default() + } + ) + .await + .is_err(), + "no erased version may be recreated by receipt adoption" + ); + } + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("non-force bucket removal"); + } + + #[tokio::test] + async fn delete_recovery_causal_cycle_rejects_before_mutation_and_retains_proof() { + let bucket = "recover-causal-cycle"; + let (dirs, store, disks) = fixture(bucket).await; + put(&store, bucket, &vec![58; 4096], false).await; + let first = applied(&store, &disks, bucket, None, 4).await; + put(&store, bucket, &vec![59; 4096], false).await; + let second = applied(&store, &disks, bucket, None, 4).await; + let mut evidence = Vec::new(); + for (dir, disk) in dirs.iter().zip(&disks) { + let root = dir.path().join(bucket).join("object"); + let mut journal = disk + .inspect_delete_transaction(bucket, "object", first) + .await + .expect("inspect first preparation") + .journal + .expect("original journal"); + journal.predecessors.push(second); + let path = root.join(first.to_string()).join("delete-prepared"); + std::fs::write(&path, serde_json::to_vec(&journal).expect("encode cyclic fixture")).expect("write cyclic edge"); + for transaction in [first, second] { + for file in ["delete-prepared", "xl.meta.bkp"] { + let path = root.join(transaction.to_string()).join(file); + evidence.push((path.clone(), std::fs::read(path).expect("capture untouched proof"))); + } + } + } + let error = store + .recover_delete_transactions_at_incarnation(bucket, Some("object"), None, &HealOpts::default()) + .await + .expect_err("cyclic history cannot settle"); + assert!( + matches!(error, crate::error::StorageError::FileCorrupt), + "typed corruption must reject cyclic evidence: {error:?}" + ); + let report = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("a bucket report must retain its corrupt-key status"); + assert_eq!(report.delete_recovery.expect("corrupt-key status").failed, 1); + for (path, bytes) in evidence { + assert_eq!(std::fs::read(path).expect("all proof retained"), bytes); + } + for dir in &dirs { + assert!( + !dir.path().join(bucket).join("object").join(STORAGE_FORMAT_FILE).exists(), + "cycle rejection must not restore or publish metadata" + ); + } + } + + #[tokio::test] + async fn delete_recovery_set_later_null_decision_covers_unapplied_original_peer() { + let bucket = "recover-unapplied-null-history"; + let (_dirs, store, disks) = fixture(bucket).await; + put(&store, bucket, &vec![56; 1024 * 1024], false).await; + // Every generated v4 UUID sorts before MAX, so the former UUID order + // deterministically tried the newer decision before its predecessor. + applied_at(&store, &disks, bucket, None, 3, Uuid::from_u128(u128::MAX)).await; + let set = &store.pools[0].disk_set[0]; + set.disks.write().await[3] = None; + put(&store, bucket, &vec![57; 1024 * 1024], false).await; + store + .delete_object(bucket, "object", ObjectOptions::default()) + .await + .expect("later null decision"); + set.disks.write().await[3] = Some(disks[3].clone()); + store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("causal older original can be removed"); + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("no stale old body or receipts remain"); + } + + #[tokio::test] + async fn heal_mrf_inline_overwrite_does_not_strand_rollback_backup() { + use crate::disk::STORAGE_FORMAT_FILE_BACKUP; + use crate::disk::local::inline_metadata_rollback_dir; + use crate::object_api::ShardIntegrityWriteMode; + use rustfs_filemeta::FileMeta; + use rustfs_heal_contracts::heal_channel::{HealScanMode, LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL}; + + for incoming_inline in [true, false] { + let bucket = if incoming_inline { + "heal-inline-rollback" + } else { + "heal-inline-to-external-rollback" + }; + let (dirs, store, disks) = fixture(bucket).await; + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("bucket identity"); + let synthetic = inline_metadata_rollback_dir(Uuid::nil(), &FileMeta::new()); + let object = "object"; + let child = format!("{object}/{synthetic}"); + let child_body = b"child bytes must survive parent repair".to_vec(); + let write_options = ObjectOptions { + shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Legacy), + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }; + for (key, body) in [(child.as_str(), child_body.as_slice()), (object, b"inline-gen-1".as_slice())] { + store + .put_object(bucket, key, &mut PutObjReader::from_vec(body.to_vec()), &write_options) + .await + .expect("seed initial inline objects"); + } + let read_options = ReadOptions { + read_data: true, + ..Default::default() + }; + let old_child = disks[3] + .read_version("", bucket, &child, "", &read_options) + .await + .expect("physical child before repair"); + let set = &store.pools[0].disk_set[0]; + set.disks.write().await[3] = None; + let payload = if incoming_inline { + b"inline-gen-2-updated".to_vec() + } else { + vec![0x53; 1024 * 1024] + }; + store + .put_object(bucket, object, &mut PutObjReader::from_vec(payload.clone()), &write_options) + .await + .expect("degraded overwrite with a real stale target"); + set.disks.write().await[3] = Some(disks[3].clone()); + let donor = disks[0] + .read_version("", bucket, object, "", &read_options) + .await + .expect("current donor generation"); + assert_eq!(donor.data.is_some(), incoming_inline, "fixture must use the intended commit branch"); + let heal_opts = HealOpts { + scan_mode: HealScanMode::Normal, + ..Default::default() + }; + let first = store + .heal_mrf_object_at_incarnation(bucket, object, "", incarnation, &heal_opts) + .await + .expect("first MRF repair"); + assert!(first.error.is_none(), "first MRF repair: {:?}", first.error); + assert_eq!(first.item.drives_healed(), Some(1)); + assert!(!first.item.repair_verified); + assert_eq!(first.item.detail, LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL); + let target = disks[3] + .read_version("", bucket, object, "", &read_options) + .await + .expect("repaired target generation"); + assert_eq!(target.mod_time, donor.mod_time); + assert_eq!(target.data_dir, donor.data_dir); + assert_eq!(target.size, donor.size); + assert_eq!(target.version_id, donor.version_id); + let child_after = disks[3] + .read_version("", bucket, &child, "", &read_options) + .await + .expect("physical child survives parent cleanup"); + assert_eq!(child_after.data, old_child.data); + assert!(child_after.data.is_some(), "child fixture must carry its physical inline bytes"); + let child_dir = dirs[3].path().join(bucket).join(object).join(synthetic.to_string()); + assert!(!child_dir.join(STORAGE_FORMAT_FILE_BACKUP).exists()); + assert!(child_dir.join(STORAGE_FORMAT_FILE).exists()); + let second = store + .heal_mrf_object_at_incarnation(bucket, object, "", incarnation, &heal_opts) + .await + .expect("follow-up MRF check must not return SlowDown"); + assert!(second.error.is_none(), "second MRF check: {:?}", second.error); + assert_eq!(second.item.drives_healed(), Some(0)); + assert_eq!(second.item.detail, LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL); + + // EC2+2: force the repaired shard to participate in the exact read quorum. + let inventory = set.disks.read().await.clone(); + *set.disks.write().await = vec![None, None, Some(disks[2].clone()), Some(disks[3].clone())]; + for (key, expected) in [(object, payload), (child.as_str(), child_body)] { + let mut reader = set + .get_object_reader( + bucket, + key, + None, + Default::default(), + &ObjectOptions { + no_lock: true, + suppress_read_repair: true, + ..Default::default() + }, + ) + .await + .expect("exact-quorum GET using repaired target"); + let mut body = Vec::new(); + reader.read_to_end(&mut body).await.expect("read complete parent/child body"); + assert_eq!(body, expected, "exact-quorum body for {key}"); + } + *set.disks.write().await = inventory; + } + } +} diff --git a/crates/ecstore/src/set_disk/ops/heal.rs b/crates/ecstore/src/set_disk/ops/heal.rs index e166a50b7..818a578f2 100644 --- a/crates/ecstore/src/set_disk/ops/heal.rs +++ b/crates/ecstore/src/set_disk/ops/heal.rs @@ -25,11 +25,15 @@ use super::super::{ use crate::bucket::retirement::MarkerRetirementContext; use crate::diagnostics::object_lock::ObjectLockAttempt; use crate::set_disk::{ObjectLockDiagGuard, is_object_lock_diag_enabled}; +use std::sync::Arc; #[derive(Clone, Copy)] struct ExplicitVersionHeal<'a> { opts: &'a HealOpts, allow_regeneration: bool, + preserve_unrepairable: bool, + commit_fence: Option<&'a (dyn Fn() -> bool + Sync)>, + namespace_owner: Option<&'a Arc>, retirement: Option<&'a MarkerRetirementContext<'a>>, } @@ -63,6 +67,14 @@ pub(crate) struct AbsenceProofRequest<'a> { pub allow_unversioned: bool, } +fn heal_rollback_backup_dir(response: &disk::RenameDataResp) -> Option { + // A dedicated cleanup directory identifies a current response whose absent + // rollback is intentional. Older peers report only old_data_dir. + response + .rollback_data_dir + .or_else(|| response.old_data_dir.filter(|_| response.cleanup_data_dir.is_none())) +} + fn is_unavailable_heal_rpc(error: &DiskError) -> bool { let DiskError::Io(error) = error else { return false; @@ -99,6 +111,7 @@ static HEAL_RENAME_FAILURES: std::sync::Mutex> = st #[derive(Clone, Copy, PartialEq, Eq)] enum ReadRepairPausePhase { Data, + PartStaging(usize), Commit, } @@ -250,7 +263,16 @@ fn should_fail_heal_rename(_bucket: &str, _object: &str, _disk_index: usize) -> } #[cfg(test)] -static HEAL_WRITER_FAILURES: std::sync::Mutex> = std::sync::Mutex::new(Vec::new()); +struct HealWriterFailure { + bucket: String, + object: String, + part_number: Option, + disk_index: usize, + error: DiskError, +} + +#[cfg(test)] +static HEAL_WRITER_FAILURES: std::sync::Mutex> = std::sync::Mutex::new(Vec::new()); #[cfg(test)] struct HealWriterFailureScope { @@ -260,21 +282,23 @@ struct HealWriterFailureScope { #[cfg(test)] impl HealWriterFailureScope { - fn install(bucket: &str, object: &str, disk_indexes: &[usize], error: DiskError) -> Self { + fn install(bucket: &str, object: &str, part_number: Option, disk_indexes: &[usize], error: DiskError) -> Self { let mut failures = HEAL_WRITER_FAILURES .lock() .expect("heal writer failure registry should not poison"); assert!( - !failures.iter().any(|(registered_bucket, registered_object, _, _)| { - registered_bucket == bucket && registered_object == object - }), + !failures + .iter() + .any(|failure| failure.bucket == bucket && failure.object == object), "heal writer failures must be installed once per object" ); - failures.extend( - disk_indexes - .iter() - .map(|index| (bucket.to_string(), object.to_string(), *index, error.clone())), - ); + failures.extend(disk_indexes.iter().map(|index| HealWriterFailure { + bucket: bucket.to_string(), + object: object.to_string(), + part_number, + disk_index: *index, + error: error.clone(), + })); Self { bucket: bucket.to_string(), object: object.to_string(), @@ -288,25 +312,28 @@ impl Drop for HealWriterFailureScope { HEAL_WRITER_FAILURES .lock() .expect("heal writer failure registry should not poison") - .retain(|(bucket, object, _, _)| bucket != &self.bucket || object != &self.object); + .retain(|failure| failure.bucket != self.bucket || failure.object != self.object); } } #[cfg(test)] -fn injected_heal_writer_error(bucket: &str, object: &str, disk_index: usize) -> Option { +fn injected_heal_writer_error(bucket: &str, object: &str, part_number: usize, disk_index: usize) -> Option { let mut failures = HEAL_WRITER_FAILURES .lock() .expect("heal writer failure registry should not poison"); failures .iter() - .position(|(registered_bucket, registered_object, registered_index, _)| { - registered_bucket == bucket && registered_object == object && *registered_index == disk_index + .position(|failure| { + failure.bucket == bucket + && failure.object == object + && failure.disk_index == disk_index + && failure.part_number.is_none_or(|part| part == part_number) }) - .map(|position| failures.swap_remove(position).3) + .map(|position| failures.swap_remove(position).error) } #[cfg(not(test))] -fn injected_heal_writer_error(_bucket: &str, _object: &str, _disk_index: usize) -> Option { +fn injected_heal_writer_error(_bucket: &str, _object: &str, _part_number: usize, _disk_index: usize) -> Option { None } @@ -518,30 +545,64 @@ fn is_retryable_heal_writer_error(error: &DiskError) -> bool { ) || error.is_retryable_internode_write_failure() } -fn warn_heal_writer_failures( - bucket: &str, - object: &str, - version_id: &str, - writer_failure_count: usize, - result: &'static str, - first_failure: &(usize, usize, String), -) { - let (first_part_number, first_disk_index, first_error) = first_failure; - warn!( - event = EVENT_SET_DISK_HEAL, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_SET_DISK, - bucket, - object, - version_id, - writer_failure_count, - first_part_number, - first_disk_index, - error = %first_error, - result, - state = "writer_unavailable", - "Set disk object heal writer failures" - ); +struct HealWriterFailures { + target_count: usize, + count: usize, + first: Option<(usize, usize, String)>, + error: Option, +} + +impl HealWriterFailures { + fn new(target_count: usize) -> Self { + Self { + target_count, + count: 0, + first: None, + error: None, + } + } + + fn record(&mut self, part_number: usize, disk_index: usize, error: DiskError) { + self.count += 1; + if self.first.is_none() { + self.first = Some((part_number, disk_index, heal_writer_error_summary(&error))); + } + // A permanent failure must not be hidden behind an earlier transient + // target failure when the remaining targets also become unavailable. + if self + .error + .as_ref() + .is_none_or(|first| is_retryable_heal_writer_error(first) && !is_retryable_heal_writer_error(&error)) + { + self.error = Some(error); + } + } + + fn into_error(self) -> DiskError { + self.error.unwrap_or(DiskError::ErasureWriteQuorum) + } + + fn warn(&self, bucket: &str, object: &str, version_id: &str, result: &'static str) { + let Some((first_part_number, first_disk_index, first_error)) = &self.first else { + return; + }; + warn!( + event = EVENT_SET_DISK_HEAL, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_SET_DISK, + state = "writer_unavailable", + result, + bucket, + object, + version_id, + target_count = self.target_count, + writer_failure_count = self.count, + first_part_number, + first_disk_index, + error = %first_error, + "Set disk object heal writer failures" + ); + } } // Destructive distributed repair needs its own receiver-side drain protocol. @@ -731,6 +792,9 @@ impl SetDisks { ExplicitVersionHeal { opts, allow_regeneration: true, + preserve_unrepairable: false, + commit_fence: None, + namespace_owner: None, retirement: None, }, &mut None, @@ -756,8 +820,15 @@ impl SetDisks { false, true, self.default_parity_count, + ); + #[cfg(feature = "diagnostic-read-version-caller")] + let metadata_read = crate::cluster::rpc::runtime_sources::with_read_version_request_caller_if_absent( + rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::Heal, + metadata_read, ) .await?; + #[cfg(not(feature = "diagnostic-read-version-caller"))] + let metadata_read = metadata_read.await?; let (parts_metadata, errs, _) = metadata_read.into_legacy(); let (read_quorum, _) = Self::object_quorum_from_meta(&parts_metadata, &errs, self.default_parity_count)?; let read_quorum = usize::try_from(read_quorum).map_err(|_| DiskError::ErasureReadQuorum)?; @@ -868,6 +939,9 @@ impl SetDisks { let ExplicitVersionHeal { opts, allow_regeneration: allow_explicit_version_regen, + preserve_unrepairable, + commit_fence, + namespace_owner, retirement, } = run; trace!( @@ -886,6 +960,9 @@ impl SetDisks { if let Some(scope) = &bucket_heal_scope { scope.check()?; } + if commit_fence.is_some_and(|check| !check()) { + return Err(DiskError::DiskOngoingReq); + } let mut result = HealResultItem { heal_item_type: HealItemType::Object.to_string(), @@ -927,10 +1004,10 @@ impl SetDisks { // Bound, not `_`: this guard must live to the end of the scope. A bare // `_` would drop it here and release the namespace write lock. let _write_lock_guard = if !opts.no_lock && !read_repair_uses_shared_lock { - Some( + Some(Arc::new( self.acquire_heal_object_lock(bucket, object, version_id, HealObjectLockKind::Write, "heal_object") .await?, - ) + )) } else { None }; @@ -1212,7 +1289,7 @@ impl SetDisks { // return here and never reach the post-heal reclaim tail // below. Sweep the strays on this path too (issues #3231, // #3191). Skipped on dry-run, like every mutating step. - if !opts.dry_run { + if !opts.dry_run && !preserve_unrepairable { self.reclaim_orphan_data_dirs_best_effort(bucket, object).await; } return Ok((result, None)); @@ -1292,7 +1369,7 @@ impl SetDisks { ); } - if read_repair_uses_shared_lock { + if read_repair_uses_shared_lock || preserve_unrepairable { return Ok((result, Some(cannot_heal_err))); } @@ -1491,6 +1568,9 @@ impl SetDisks { if let Some(scope) = &bucket_heal_scope { scope.check()?; } + if commit_fence.is_some_and(|check| !check()) { + return Err(DiskError::DiskOngoingReq); + } let disk = outdated_disk.as_ref().expect("checked above").clone(); let endpoint = disk.endpoint().to_string(); match disk.make_volume(bucket).await { @@ -1526,6 +1606,9 @@ impl SetDisks { if let Some(scope) = &bucket_heal_scope { scope.check()?; } + if commit_fence.is_some_and(|check| !check()) { + return Err(DiskError::DiskOngoingReq); + } if disks_to_heal_count == 0 { return Ok((result, volume_creation_error.or(Some(DiskError::ErasureWriteQuorum)))); } @@ -1561,196 +1644,169 @@ impl SetDisks { let dst_data_dir = data_dir; if !latest_meta.deleted && !latest_meta.is_remote() { - let erasure_info = latest_meta.erasure.clone(); - let mut writer_failure_count = 0usize; - let mut first_writer_failure = None; - let mut first_writer_error = None; - let mut all_writer_failures_retryable = true; - let mut writer_failure_warned = false; + let stage_result: disk::error::Result> = async { + let erasure_info = latest_meta.erasure.clone(); + let mut writer_failures = HealWriterFailures::new(disks_to_heal_count); - for (part_index, part) in latest_meta.parts.iter().enumerate() { - use crate::io_support::shard_integrity::{PartProofReader, PreparedIntegrity, ShardVerifier}; - let proof = part - .integrity - .as_ref() - .map(|part| { - PartProofReader::new(part.clone(), &parts_metadata, &latest_disks, bucket, object) - }) - .transpose() - .map_err(DiskError::from)?; - // This reader covers the whole encoded shard. A range - // offset can exclude the final encoded byte when the - // logical part ends inside a data block, allowing a - // one-byte tail truncation to evade deep Heal. - let till_offset = usize::try_from(erasure.shard_file_size(part.size as i64)) - .map_err(|_| DiskError::FileCorrupt)?; - let use_mmap_read = object_mmap_read_enabled(); + for (part_index, part) in latest_meta.parts.iter().enumerate() { + use crate::io_support::shard_integrity::{PartProofReader, PreparedIntegrity, ShardVerifier}; + let proof = part + .integrity + .as_ref() + .map(|part| { + PartProofReader::new(part.clone(), &parts_metadata, &latest_disks, bucket, object) + }) + .transpose()?; + // Cover the complete encoded shard, including a logical + // part's partial final data block, when checking truncation. + let till_offset = usize::try_from(erasure.shard_file_size(part.size as i64)) + .map_err(|_| DiskError::FileCorrupt)?; + let use_mmap_read = object_mmap_read_enabled(); - let mut readers = Vec::with_capacity(latest_disks.len()); - let mut writers = Vec::with_capacity(out_dated_disks.len()); - // let mut errors = Vec::with_capacity(out_dated_disks.len()); + let mut writers = Vec::with_capacity(out_dated_disks.len()); - let mut prefer = vec![false; latest_disks.len()]; - for (index, disk) in latest_disks.iter().enumerate() { + // Preserve the committed layout: recomputing inline-ness here + // (with a hardcoded unversioned threshold) makes healed replicas + // diverge from healthy ones in quorum identity, so heal would + // flag them forever. + let is_inline_buffer = latest_meta.inline_data(); + // create writers for all disk positions, but only for outdated disks + for (index, disk_op) in out_dated_disks.iter().enumerate() { + if let Some(outdated_disk) = disk_op { + let writer_result = if let Some(error) = + injected_heal_writer_error(bucket, object, part.number, index) + { + Err(error) + } else { + create_bitrot_writer( + is_inline_buffer, + Some(outdated_disk), + RUSTFS_META_TMP_BUCKET, + &path_join_buf(&[ + &tmp_id.to_string(), + &dst_data_dir.to_string(), + &format!("part.{}", part.number), + ]), + erasure.shard_file_size(part.size as i64), + erasure.shard_size(), + HashAlgorithm::HighwayHash256S, + ) + .await + }; + let writer = match writer_result { + Ok(writer) => writer, + Err(err) => { + writer_failures.record(part.number, index, err); + writers.push(None); + continue; + } + }; + let mut writer = writer; + if let Some(proof) = &proof { + let verifier = ShardVerifier::new(std::sync::Arc::clone(proof), index, 0, None)?; + writer.set_integrity(verifier); + } + writers.push(Some(writer)); + } else { + writers.push(None); + } + } + + // Heal each part. erasure.Heal() will write the healed + // part to .rustfs/tmp/uuid/ which needs to be renamed + // later to the final location. + if writers.iter().all(Option::is_none) { + writer_failures.warn(bucket, object, version_id, "all_targets_unavailable"); + // No target can accept this part. Preserve the setup + // cause before opening source readers or reconstructing. + return Ok(Some(writer_failures.into_error())); + } + #[cfg(test)] + pause_read_repair(bucket, object, ReadRepairPausePhase::PartStaging(part.number)).await; + + let mut readers = Vec::with_capacity(latest_disks.len()); + + let mut prefer = vec![false; latest_disks.len()]; let this_part_errs = Self::shuffle_check_parts(&data_errs_by_part[&part_index], &erasure_info.distribution); - if this_part_errs[index] != CHECK_PART_SUCCESS { - trace!( - event = EVENT_SET_DISK_HEAL, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_SET_DISK, - part_number = part.number, - disk_index = index, - part_status = this_part_errs[index], - state = "source_shard_skipped", - "Set disk source shard skipped" - ); - readers.push(None); - continue; - } - - if let (Some(disk), Some(metadata)) = (disk, ©_parts_metadata[index]) { - let checksum_info = metadata.erasure.get_checksum_info(part.number); - let checksum_algo = if metadata.uses_legacy_checksum - && checksum_info.algorithm == HashAlgorithm::HighwayHash256S - { - HashAlgorithm::HighwayHash256SLegacy - } else { - checksum_info.algorithm - }; - - match create_bitrot_reader( - metadata.data.as_deref(), - Some(disk), - bucket, - &path_join_buf(&[object, &src_data_dir, &format!("part.{}", part.number)]), - 0, - till_offset, - erasure.shard_size(), - checksum_algo.clone(), - false, - use_mmap_read, - ) - .await - { - Ok(Some(mut reader)) => { - if let Some(proof) = &proof { - reader - .set_integrity( - ShardVerifier::new(std::sync::Arc::clone(proof), index, 0, None) - .map_err(DiskError::from)?, - ) - .map_err(DiskError::from)?; - } - readers.push(Some(reader)); - } - Ok(None) => { - readers.push(None); - continue; - } - Err(_e) => { - readers.push(None); - continue; - } + for (index, disk) in latest_disks.iter().enumerate() { + if this_part_errs[index] != CHECK_PART_SUCCESS { + trace!( + event = EVENT_SET_DISK_HEAL, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_SET_DISK, + part_number = part.number, + disk_index = index, + part_status = this_part_errs[index], + state = "source_shard_skipped", + "Set disk source shard skipped" + ); + readers.push(None); + continue; } - prefer[index] = disk.host_name().is_empty(); - } else { - readers.push(None); - // errors.push(Some(DiskError::DiskNotFound)); - } - } + if let (Some(disk), Some(metadata)) = (disk, ©_parts_metadata[index]) { + let checksum_info = metadata.erasure.get_checksum_info(part.number); + let checksum_algo = if metadata.uses_legacy_checksum + && checksum_info.algorithm == HashAlgorithm::HighwayHash256S + { + HashAlgorithm::HighwayHash256SLegacy + } else { + checksum_info.algorithm + }; - // Preserve the committed layout: recomputing inline-ness here - // (with a hardcoded unversioned threshold) makes healed replicas - // diverge from healthy ones in quorum identity, so heal would - // flag them forever. - let is_inline_buffer = latest_meta.inline_data(); - // create writers for all disk positions, but only for outdated disks - for (index, disk_op) in out_dated_disks.iter().enumerate() { - if let Some(outdated_disk) = disk_op { - let writer_result = if let Some(error) = injected_heal_writer_error(bucket, object, index) - { - Err(error) - } else { - create_bitrot_writer( - is_inline_buffer, - Some(outdated_disk), - RUSTFS_META_TMP_BUCKET, - &path_join_buf(&[ - &tmp_id.to_string(), - &dst_data_dir.to_string(), - &format!("part.{}", part.number), - ]), - erasure.shard_file_size(part.size as i64), + match create_bitrot_reader( + metadata.data.as_deref(), + Some(disk), + bucket, + &path_join_buf(&[object, &src_data_dir, &format!("part.{}", part.number)]), + 0, + till_offset, erasure.shard_size(), - HashAlgorithm::HighwayHash256S, + checksum_algo.clone(), + false, + use_mmap_read, ) .await - }; - let writer = match writer_result { - Ok(writer) => writer, - Err(err) => { - writer_failure_count += 1; - all_writer_failures_retryable &= is_retryable_heal_writer_error(&err); - if first_writer_failure.is_none() { - first_writer_failure = - Some((part.number, index, heal_writer_error_summary(&err))); - first_writer_error = Some(err); + { + Ok(Some(mut reader)) => { + if let Some(proof) = &proof { + let verifier = + ShardVerifier::new(std::sync::Arc::clone(proof), index, 0, None)?; + reader.set_integrity(verifier)?; + } + readers.push(Some(reader)); + } + Ok(None) => { + readers.push(None); + continue; + } + Err(_e) => { + readers.push(None); + continue; } - writers.push(None); - continue; } - }; - let mut writer = writer; - if let Some(proof) = &proof { - writer.set_integrity( - ShardVerifier::new(std::sync::Arc::clone(proof), index, 0, None) - .map_err(DiskError::from)?, - ); - } - writers.push(Some(writer)); - } else { - writers.push(None); - } - } - // Heal each part. erasure.Heal() will write the healed - // part to .rustfs/tmp/uuid/ which needs to be renamed - // later to the final location. - if writer_failure_count > 0 - && writers.iter().all(Option::is_none) - && let Some(first_failure) = first_writer_failure.as_ref() - { - warn_heal_writer_failures( - bucket, - object, - version_id, - writer_failure_count, - "all_targets_unavailable", - first_failure, - ); - writer_failure_warned = true; - } - let heal_part = erasure.heal(&mut writers, readers, part.size, &prefer); - let heal_part_result = if read_repair_uses_shared_lock { - match tokio::time::timeout(READ_REPAIR_DATA_PHASE_TIMEOUT, heal_part).await { - Ok(result) => result, - Err(_) => Err(DiskError::Timeout), + prefer[index] = disk.host_name().is_empty(); + } else { + readers.push(None); + // errors.push(Some(DiskError::DiskNotFound)); + } } - } else { - heal_part.await - }; - if let Err(e) = heal_part_result { - // Don't leak the partially-written healed shards in - // .rustfs/tmp when heal fails midway (backlog#799 B20). - let _ = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_id).await; - return Err(e); - } - // close_bitrot_writers(&mut writers).await?; - if !is_inline_buffer && let Some(proof) = &proof { - let before = out_dated_disks.iter().filter(|disk| disk.is_some()).count(); - let proof_result = async { + + let heal_part = erasure.heal(&mut writers, readers, part.size, &prefer); + let heal_part_result = if read_repair_uses_shared_lock { + match tokio::time::timeout(READ_REPAIR_DATA_PHASE_TIMEOUT, heal_part).await { + Ok(result) => result, + Err(_) => Err(DiskError::Timeout), + } + } else { + heal_part.await + }; + heal_part_result?; + // close_bitrot_writers(&mut writers).await?; + if !is_inline_buffer && let Some(proof) = &proof { + let before = out_dated_disks.iter().filter(|disk| disk.is_some()).count(); PreparedIntegrity::copy_from(proof) .await? .write( @@ -1759,89 +1815,71 @@ impl SetDisks { RUSTFS_META_TMP_BUCKET, &format!("{tmp_id}/{dst_data_dir}"), ) - .await - } - .await; - if let Err(error) = proof_result { - let _ = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_id).await; - return Err(error.into()); - } - disks_to_heal_count -= before - out_dated_disks.iter().filter(|disk| disk.is_some()).count(); - } - - for (index, disk_op) in out_dated_disks.iter_mut().enumerate() { - if disk_op.is_none() { - continue; + .await?; + disks_to_heal_count -= + before - out_dated_disks.iter().filter(|disk| disk.is_some()).count(); } - if writers[index].is_none() { - *disk_op = None; - disks_to_heal_count -= 1; - continue; - } - - parts_metadata[index].data_dir = Some(dst_data_dir); - parts_metadata[index].add_object_part( - part.number, - part.etag.clone(), - part.size, - part.mod_time, - part.actual_size, - part.index.clone(), - part.checksums.clone(), - ); - if let Some(updated) = parts_metadata[index] - .parts - .iter_mut() - .find(|updated| updated.number == part.number) - { - updated.integrity.clone_from(&part.integrity); - } - if is_inline_buffer { - if let Some(writer) = writers[index].take() { - // if let Some(w) = writer.as_any().downcast_ref::() { - // parts_metadata[index].data = Some(w.inline_data().to_vec()); - // } - parts_metadata[index].data = - Some(writer.into_inline_data().map(Bytes::from).unwrap_or_default()); + for (index, disk_op) in out_dated_disks.iter_mut().enumerate() { + if disk_op.is_none() { + continue; } - parts_metadata[index].set_inline_data(); - } else { - parts_metadata[index].data = None; - } - } - if disks_to_heal_count == 0 { - if !writer_failure_warned && let Some(first_failure) = first_writer_failure.as_ref() { - warn_heal_writer_failures( - bucket, - object, - version_id, - writer_failure_count, - "all_targets_unavailable", - first_failure, + if writers[index].is_none() { + *disk_op = None; + disks_to_heal_count -= 1; + continue; + } + + parts_metadata[index].data_dir = Some(dst_data_dir); + parts_metadata[index].add_object_part( + part.number, + part.etag.clone(), + part.size, + part.mod_time, + part.actual_size, + part.index.clone(), + part.checksums.clone(), ); + if let Some(updated) = parts_metadata[index] + .parts + .iter_mut() + .find(|updated| updated.number == part.number) + { + updated.integrity.clone_from(&part.integrity); + } + if is_inline_buffer { + if let Some(writer) = writers[index].take() { + // if let Some(w) = writer.as_any().downcast_ref::() { + // parts_metadata[index].data = Some(w.inline_data().to_vec()); + // } + parts_metadata[index].data = + Some(writer.into_inline_data().map(Bytes::from).unwrap_or_default()); + } + parts_metadata[index].set_inline_data(); + } else { + parts_metadata[index].data = None; + } } - // Clean up healed shards written to .rustfs/tmp before bailing (B20). - let _ = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_id).await; - let error = if all_writer_failures_retryable { - first_writer_error.take().unwrap_or(DiskError::FaultyDisk) - } else { - DiskError::other(format!("all drives had write errors, unable to heal {bucket}/{object}")) - }; - return Ok((result, Some(error))); - } - } - if !writer_failure_warned && let Some(first_failure) = first_writer_failure.as_ref() { - warn_heal_writer_failures( - bucket, - object, - version_id, - writer_failure_count, - "partial_targets_unavailable", - first_failure, - ); + if disks_to_heal_count == 0 { + writer_failures.warn(bucket, object, version_id, "all_targets_unavailable"); + return Ok(Some(writer_failures.into_error())); + } + } + + writer_failures.warn(bucket, object, version_id, "partial_targets_unavailable"); + Ok(None) + } + .await; + match stage_result { + Ok(None) => {} + failure => { + // Release readers and writers before removing all parts + // staged under this heal's exclusively owned prefix. + let _ = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_id).await; + return failure.map(|error| (result, error)); + } } } let _read_repair_write_lock_guard = if let Some(expected_fingerprint) = read_repair_commit_fingerprint { @@ -1856,7 +1894,7 @@ impl SetDisks { ) .await { - Ok(Some(guard)) => Some(guard), + Ok(Some(guard)) => Some(Arc::new(guard)), Ok(None) => { let _ = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_id).await; return Ok((result, Some(DiskError::Timeout))); @@ -1877,11 +1915,16 @@ impl SetDisks { let mut rename_attempts = 0usize; let mut rename_successes = 0usize; let mut healed_disks = vec![None; out_dated_disks.len()]; + let mut backup_cleanup_owner: Option> = None; for (index, outdated_disk) in out_dated_disks.iter().enumerate() { if let Some(disk) = outdated_disk { if let Some(scope) = &bucket_heal_scope { scope.check()?; } + if commit_fence.is_some_and(|check| !check()) { + let _ = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_id).await; + return Err(DiskError::DiskOngoingReq); + } rename_attempts += 1; // record the index of the updated disks parts_metadata[index].erasure.index = index + 1; @@ -1901,60 +1944,115 @@ impl SetDisks { .await }; - if let Err(err) = &rename_result { - warn!( - event = EVENT_HEAL_OBJECT_RENAME, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_HEAL, - bucket, - object, - version_id, - disk_index = index, - endpoint = %disk.endpoint(), - tmp_id, - result = "failed", - error = %err, - "Heal object rename failed" - ); - } else { - rename_successes += 1; - healed_disks[index] = Some(disk.clone()); - if parts_metadata[index].is_remote() { - check_artifact_repair_delete(bucket, object).await?; - let rm_data_dir = - parts_metadata[index].data_dir.expect("operation should succeed").to_string(); - - let d_path = Path::new(&encode_dir_object(object)).join(rm_data_dir); - - if let Err(e) = disk - .delete( - bucket, - d_path.to_str().expect("operation should succeed"), - DeleteOptions { - immediate: true, - recursive: true, - ..Default::default() - }, - ) - .await - { - // The healed shard has already been renamed into place; a - // failure cleaning up the old remote data dir must not abort - // the heal and leak the tmp shards (backlog#799 B20). - warn!( - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_HEAL, - bucket, - object, - error = %e, - "Heal remote data-dir cleanup failed" - ); - } + match rename_result { + Err(err) => { + warn!( + event = EVENT_HEAL_OBJECT_RENAME, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_HEAL, + result = "failed", + bucket, + object, + version_id, + disk_index = index, + endpoint = %disk.endpoint(), + tmp_id, + error = %err, + "Heal object rename failed" + ); } + Ok(rename_response) => { + rename_successes += 1; + healed_disks[index] = Some(disk.clone()); + // Heal can reuse a live data directory, and a synthetic + // rollback directory can also contain a child key. + // Retire only this publication's backup while fenced. + if let Some(rollback_dir) = heal_rollback_backup_dir(&rename_response) + && check_artifact_repair_delete(bucket, object).await.is_ok() + && bucket_heal_scope.as_ref().is_none_or(|scope| scope.check().is_ok()) + && commit_fence.is_none_or(|check| check()) + && !_write_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) + && !_read_repair_write_lock_guard + .as_ref() + .is_some_and(|guard| guard.is_lock_lost()) + { + let backup_path = format!( + "{}/{rollback_dir}/{}", + rustfs_utils::path::encode_dir_object_ref(object), + disk::STORAGE_FORMAT_FILE_BACKUP, + ); + // The syscall can outlive this future; retain the exact-key + // locks until physical cleanup has finished. + let cleanup_owner = backup_cleanup_owner.get_or_insert_with(|| { + Arc::new(( + namespace_owner.cloned(), + _write_lock_guard.clone(), + _read_repair_write_lock_guard.clone(), + bucket_heal_scope.clone(), + self.ctx.begin_namespace_commit(), + )) + }); + match disk + .delete_with_namespace_owner( + bucket, + &backup_path, + DeleteOptions::default(), + Some(cleanup_owner.clone()), + ) + .await + { + Ok(()) | Err(DiskError::FileNotFound | DiskError::VolumeNotFound) => {} + Err(err) => warn!( + event = EVENT_HEAL_OBJECT_RENAME, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_HEAL, + result = "cleanup_failed", + bucket, + object, + version_id, + disk_index = index, + error = %err, + "Heal rollback backup cleanup failed" + ), + } + } + if parts_metadata[index].is_remote() { + check_artifact_repair_delete(bucket, object).await?; + let rm_data_dir = + parts_metadata[index].data_dir.expect("operation should succeed").to_string(); - for (i, v) in result.before.drives.iter().enumerate() { - if v.endpoint == disk.endpoint().to_string() { - result.after.drives[i].state = DriveState::Ok.to_string(); + let d_path = Path::new(&encode_dir_object(object)).join(rm_data_dir); + + if let Err(e) = disk + .delete( + bucket, + d_path.to_str().expect("operation should succeed"), + DeleteOptions { + immediate: true, + recursive: true, + ..Default::default() + }, + ) + .await + { + // The healed shard has already been renamed into place; a + // failure cleaning up the old remote data dir must not abort + // the heal and leak the tmp shards (backlog#799 B20). + warn!( + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_HEAL, + bucket, + object, + error = %e, + "Heal remote data-dir cleanup failed" + ); + } + } + + for (i, v) in result.before.drives.iter().enumerate() { + if v.endpoint == disk.endpoint().to_string() { + result.after.drives[i].state = DriveState::Ok.to_string(); + } } } } @@ -2014,7 +2112,9 @@ impl SetDisks { // by pre-#3510 unversioned overwrites, which the dangling paths // above never touch (issues #3231, #3191). Best effort — a // failure must not fail the heal. - self.reclaim_orphan_data_dirs_best_effort(bucket, object).await; + if !preserve_unrepairable { + self.reclaim_orphan_data_dirs_best_effort(bucket, object).await; + } Ok((result, None)) } @@ -2080,7 +2180,7 @@ impl SetDisks { .await; } - if opts.dry_run { + if opts.dry_run || preserve_unrepairable { return Ok(( self.default_heal_result(FileInfo::default(), &errs, bucket, object, version_id) .await, @@ -3371,6 +3471,76 @@ impl crate::storage_api_contracts::heal::HealOperations for SetDisks { } impl SetDisks { + /// The recovery coordinator retains the key write lock and lifecycle fence. + /// Only current live metadata can supply this exact source; backups and + /// dangling cleanup cannot authorize prerequisite repair. + pub(in crate::set_disk) async fn heal_delete_recovery_source( + &self, + bucket: &str, + object: &str, + source: &crate::disk::DeleteRecoveryVersionIdentity, + can_commit: &(dyn Fn() -> bool + Sync), + namespace_owner: Option<&Arc>, + ) -> disk::error::Result<()> { + if !can_commit() { + return Err(DiskError::DiskOngoingReq); + } + let version = source.version_id.ok_or(DiskError::OutdatedXLMeta)?.to_string(); + let disks = self.get_disks_internal().await; + let (metadata, errors, _) = Self::read_metadata(&disks, "", bucket, object, &version, true, false, false) + .await? + .into_legacy(); + let (read_quorum, write_quorum) = Self::object_quorum_from_meta(&metadata, &errors, self.default_parity_count)?; + let read_quorum = usize::try_from(read_quorum).map_err(|_| DiskError::ErasureReadQuorum)?; + let write_quorum = usize::try_from(write_quorum).map_err(|_| DiskError::ErasureWriteQuorum)?; + let live_sources = metadata + .iter() + .zip(&errors) + .filter(|(info, error)| { + error.is_none() + && !info.deleted + && !info.is_remote() + && crate::disk::DeleteRecoveryVersionIdentity::from_file_info((*info).clone()) + .is_ok_and(|identity| identity == *source) + }) + .count(); + if live_sources < write_quorum || !can_commit() { + return Err(DiskError::OutdatedXLMeta); + } + let (_, mod_time, etag) = Self::list_online_disks(&disks, &metadata, &errors, read_quorum); + let selected = Self::pick_valid_fileinfo(&metadata, mod_time, etag, read_quorum)?; + if crate::disk::DeleteRecoveryVersionIdentity::from_file_info(selected)? != *source { + return Err(DiskError::OutdatedXLMeta); + } + let opts = HealOpts { + no_lock: true, + scan_mode: HealScanMode::Deep, + ..Default::default() + }; + let (_, error) = Box::pin(self.heal_object_with_explicit_version_regen( + bucket, + object, + &version, + ExplicitVersionHeal { + opts: &opts, + allow_regeneration: false, + preserve_unrepairable: true, + commit_fence: Some(can_commit), + namespace_owner, + retirement: None, + }, + &mut None, + )) + .await?; + if let Some(error) = error { + return Err(error); + } + if !can_commit() { + return Err(DiskError::DiskOngoingReq); + } + Ok(()) + } + pub(crate) async fn heal_object_with_absence( &self, bucket: &str, @@ -3418,11 +3588,11 @@ impl SetDisks { .new_ns_lock(bucket, object) .await .map_err(|e| e.narrow_to_disk().unwrap_or_else(DiskError::other))?; - Some(ns_lock.get_write_lock(get_lock_acquire_timeout()).await.map_err(|e| { + Some(Arc::new(ns_lock.get_write_lock(get_lock_acquire_timeout()).await.map_err(|e| { self.map_namespace_lock_error(bucket, object, "write", e) .narrow_to_disk() .unwrap_or_else(DiskError::other) - })?) + })?)) } else { None }; @@ -3432,6 +3602,73 @@ impl SetDisks { return Ok((result, err.map(|e| e.into()))); } + let mut recovery_removed_version = false; + if !opts.read_repair + && _write_lock_guard.is_some() + && let Some(scope) = crate::store::bucket_heal_scope_for_object(bucket, object) + && !crate::bucket::utils::is_meta_bucketname(bucket) + { + let explicit_delete = opts.remove + && version_id.is_empty() + && crate::bucket::metadata_sys::has_authoritative_never_versioned_state_in(&self.ctx, bucket).await? + && matches!( + crate::bucket::metadata_sys::get_object_lock_config_state_in(&self.ctx, bucket).await?, + crate::bucket::metadata_sys::ObjectLockConfigState::ConfirmedAbsent + ); + let recovery = self + .recover_delete_key( + bucket, + object, + crate::set_disk::DeleteRecoveryContext { + dry_run: opts.dry_run, + explicit_delete, + repair_marker_source: true, + namespace_owner: (!opts.dry_run).then(|| { + Arc::new((scope.clone(), self.ctx.begin_namespace_commit(), _write_lock_guard.clone())) + as Arc + }), + ..crate::set_disk::DeleteRecoveryContext::new(Some(scope.incarnation), &|| { + scope.check().is_ok() && !_write_lock_guard.as_deref().is_some_and(|guard| guard.is_lock_lost()) + }) + }, + ) + .await?; + recovery_removed_version = if version_id.is_empty() { + !recovery.removed_versions.is_empty() + } else { + let target = if version_id == "null" { + Some(Uuid::nil()) + } else { + Uuid::parse_str(version_id).ok() + }; + target.is_some_and(|id| recovery.removed_versions.contains(&id)) + }; + if opts.dry_run && recovery.discovered > 0 { + return Ok(( + HealResultItem { + heal_item_type: HealItemType::Object.to_string(), + bucket: bucket.to_owned(), + object: object.to_owned(), + version_id: version_id.to_owned(), + detail: format!( + "delete recovery (dry run): discovered={}, committed={}, rolled_back={}, reclaimed={}, blocked={}, pending={}", + recovery.discovered, + recovery.committed, + recovery.rolled_back, + recovery.reclaimed, + recovery.blocked, + recovery.pending + ), + ..Default::default() + }, + None, + )); + } + if recovery.pending > 0 || recovery.blocked > 0 { + return Err(StorageError::SlowDown); + } + } + // The inner heal and missing-object report read the registry again; // release this snapshot guard before a topology writer can queue between reads. let disks = self.get_disks_internal().await; @@ -3483,12 +3720,12 @@ impl SetDisks { && errs .iter() .all(|err| matches!(err, Some(DiskError::FileNotFound | DiskError::FileVersionNotFound))) - && !_write_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) + && !_write_lock_guard.as_deref().is_some_and(|guard| guard.is_lock_lost()) { *absence = Some(HealedObjectAbsence { pool_index: self.pool_index, set_index: self.set_index, - removed: false, + removed: recovery_removed_version, }); } return Ok((result, Some(err))); @@ -3498,6 +3735,8 @@ impl SetDisks { // Pass no_lock=true since we already obtained write lock (or are already called with no_lock=true) let mut inner_opts = *opts; inner_opts.no_lock = true; + let commit_fence = || !_write_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()); + let namespace_owner = _write_lock_guard.as_ref().map(|guard| guard.clone() as Arc); let (result, err) = self .heal_object_with_explicit_version_regen( bucket, @@ -3506,6 +3745,9 @@ impl SetDisks { ExplicitVersionHeal { opts: &inner_opts, allow_regeneration: true, + preserve_unrepairable: false, + commit_fence: Some(&commit_fence), + namespace_owner: namespace_owner.as_ref(), retirement: proof.retirement, }, absence, @@ -3526,6 +3768,9 @@ impl SetDisks { ExplicitVersionHeal { opts: &inner_opts, allow_regeneration: true, + preserve_unrepairable: false, + commit_fence: Some(&commit_fence), + namespace_owner: namespace_owner.as_ref(), retirement: proof.retirement, }, absence, @@ -3536,7 +3781,7 @@ impl SetDisks { result, err.map(Into::into), absence, - _write_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()), + _write_lock_guard.as_deref().is_some_and(|guard| guard.is_lock_lost()), bucket, object, )); @@ -3548,7 +3793,7 @@ impl SetDisks { result, err.map(Into::into), absence, - _write_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()), + _write_lock_guard.as_deref().is_some_and(|guard| guard.is_lock_lost()), bucket, object, )) @@ -3578,10 +3823,11 @@ impl SetDisks { #[cfg(test)] mod heal_result_report_tests { use super::{ - AbsenceProofRequest, DanglingCheckPartsFailure, DanglingDeleteFailure, DanglingDeleteSafety, ReadRepairCommitFingerprint, - ReadRepairPauseScope, SetDisks, heal_writer_error_summary, is_retryable_heal_writer_error, + AbsenceProofRequest, DanglingCheckPartsFailure, DanglingDeleteFailure, DanglingDeleteSafety, ExplicitVersionHeal, + HealWriterFailures, ReadRepairCommitFingerprint, ReadRepairPauseScope, SetDisks, heal_writer_error_summary, + is_retryable_heal_writer_error, }; - use super::{HEAL_RENAME_INCOMPLETE, HealRenameFailureScope, HealWriterFailureScope}; + use super::{HEAL_RENAME_INCOMPLETE, HealRenameFailureScope, HealWriterFailureScope, heal_rollback_backup_dir}; use crate::disk::endpoint::Endpoint; use crate::disk::error::DiskError; use crate::disk::format::FormatV3; @@ -3835,10 +4081,17 @@ mod heal_result_report_tests { #[tokio::test(flavor = "current_thread")] #[serial_test::serial] async fn heal_writer_failures_emit_one_aggregate_warning_per_object() { - for (case, failed_target_count, expected_result, expect_error) in [ - ("partial", 1usize, "partial_targets_unavailable", false), - ("all", 2usize, "all_targets_unavailable", true), + for (case, target_count, failed_target_count, injected_error) in [ + ("partial", 2usize, 1usize, DiskError::DiskFull), + ("all-full", 2, 2, DiskError::DiskFull), + ("all-timeout", 2, 2, DiskError::Timeout), + ("single-faulty", 1, 1, DiskError::FaultyDisk), ] { + let expected_result = if target_count == failed_target_count { + "all_targets_unavailable" + } else { + "partial_targets_unavailable" + }; let (temp_dirs, disks, set) = hermetic_set_disks_isolated(4).await; let bucket = format!("heal-writer-{case}"); let object = "object.bin"; @@ -3870,11 +4123,13 @@ mod heal_result_report_tests { .read_version("", &bucket, object, "", &ReadOptions::default()) .await .expect("source metadata should be readable"); - let mut target_slots = [source.erasure.distribution[0] - 1, source.erasure.distribution[1] - 1]; + let mut target_slots = (0..target_count) + .map(|index| source.erasure.distribution[index] - 1) + .collect::>(); target_slots.sort_unstable(); - for index in [0, 1] { - remove_current_object_part(&temp_dirs[index], &bucket, object) + for dir in temp_dirs.iter().take(target_count) { + remove_current_object_part(dir, &bucket, object) .await .expect("target shard should be removed before heal"); } @@ -3888,7 +4143,9 @@ mod heal_result_report_tests { .without_time() .finish(); let subscriber_guard = tracing::subscriber::set_default(subscriber); - let failure_scope = HealWriterFailureScope::install(&bucket, object, failed_slots, DiskError::DiskFull); + let tmp_before = non_trash_tmp_entries(&temp_dirs).await; + let injected_summary = heal_writer_error_summary(&injected_error); + let failure_scope = HealWriterFailureScope::install(&bucket, object, None, failed_slots, injected_error.clone()); let heal_outcome = set .heal_object( @@ -3905,12 +4162,22 @@ mod heal_result_report_tests { drop(failure_scope); drop(subscriber_guard); + let (_, error) = heal_outcome.expect("writer setup failures must retain a heal report and their typed error"); assert_eq!( - heal_outcome.is_err(), - expect_error, - "{case}: aggregate heal result should match writer outcomes" + error, + (target_count == failed_target_count).then_some(injected_error), + "{case}: partial repair succeeds and all-target failure retains its cause" + ); + assert_eq!( + non_trash_tmp_entries(&temp_dirs).await, + tmp_before, + "{case}: heal must clean temporary shards" ); let output = logs.contents(); + assert!( + !output.contains("Erasure encode write quorum unavailable"), + "{case}: unavailable heal targets must not fabricate an all-disk encode failure: {output}" + ); assert_eq!( output.matches("Set disk object heal writer failures").count(), 1, @@ -3933,12 +4200,172 @@ mod heal_result_report_tests { "{case}: warning must distinguish partial from all-target failure: {output}" ); assert!( - output.contains("error=drive path full"), + output.contains(&format!("error={injected_summary}")), "{case}: warning must preserve a redacted failure reason: {output}" ); } } + #[test] + fn heal_writer_failure_aggregation_retains_permanent_cause() { + for errors in [ + [DiskError::Timeout, DiskError::DiskFull], + [DiskError::DiskFull, DiskError::Timeout], + ] { + let mut failures = HealWriterFailures::new(2); + for (index, error) in errors.into_iter().enumerate() { + failures.record(1, index, error); + } + assert_eq!(failures.count, 2); + assert_eq!(failures.into_error(), DiskError::DiskFull); + } + } + + #[tokio::test] + #[serial_test::serial] + async fn heal_second_part_failures_clean_staged_parts_and_retry() { + use crate::set_disk::CompletePart; + use crate::storage_api_contracts::multipart::MultipartOperations as _; + use tokio::io::AsyncReadExt as _; + + let (dirs, disks, set) = hermetic_set_disks_isolated(4).await; + let bucket = "heal-writer-second-part"; + let object = "multipart.bin"; + for disk in &disks { + disk.make_volume(bucket).await.expect("fixture bucket"); + } + let options = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let upload = set + .new_multipart_upload(bucket, object, &options) + .await + .expect("multipart upload"); + let payloads = [vec![0x35; 5 * 1024 * 1024], vec![0x69; 1024 * 1024]]; + let mut completed = Vec::new(); + for (index, payload) in payloads.iter().enumerate() { + let part = set + .put_object_part( + bucket, + object, + &upload.upload_id, + index + 1, + &mut PutObjReader::from_vec(payload.clone()), + &options, + ) + .await + .expect("multipart part"); + completed.push(CompletePart { + part_num: part.part_num, + etag: part.etag, + ..Default::default() + }); + } + set.clone() + .complete_multipart_upload(bucket, object, &upload.upload_id, completed, &options) + .await + .expect("completed multipart object"); + let source = disks[2] + .read_version("", bucket, object, "", &ReadOptions::default()) + .await + .expect("healthy source metadata"); + let data_dir = source.data_dir.expect("multipart data directory").to_string(); + let target_slots = [source.erasure.distribution[0] - 1, source.erasure.distribution[1] - 1]; + for dir in &dirs[..2] { + tokio::fs::remove_dir_all(dir.path().join(bucket).join(object).join(&data_dir)) + .await + .expect("remove target shards"); + } + let tmp_before = non_trash_tmp_entries(&dirs).await; + let scope = HealWriterFailureScope::install(bucket, object, Some(2), &target_slots, DiskError::Timeout); + let heal_options = HealOpts { + no_lock: true, + scan_mode: HealScanMode::Deep, + ..Default::default() + }; + let (_, error) = set + .heal_object(bucket, object, "", &heal_options) + .await + .expect("failed heal report"); + drop(scope); + assert_eq!(error, Some(DiskError::Timeout)); + assert_eq!( + non_trash_tmp_entries(&dirs).await, + tmp_before, + "first-part staging must be cleaned after the second-part setup failure" + ); + + let source_path = dirs[2].path().join(bucket).join(object).join(&data_dir).join("part.2"); + let source_bytes = tokio::fs::read(&source_path).await.expect("committed source shard"); + let mut metadata_before = Vec::new(); + for dir in &dirs { + metadata_before.push( + tokio::fs::read(dir.path().join(bucket).join(object).join("xl.meta")) + .await + .expect("committed metadata before reconstruction failure"), + ); + } + let (pause_scope, arrived, release) = + ReadRepairPauseScope::install_at(bucket, object, super::ReadRepairPausePhase::PartStaging(2)); + let lose_source = async { + arrived.notified().await; + assert_ne!( + non_trash_tmp_entries(&dirs).await, + tmp_before, + "the first part must already be staged before the source loss" + ); + tokio::fs::remove_file(&source_path) + .await + .expect("lose one remaining source shard"); + release.notify_one(); + }; + let (outcome, ()) = tokio::time::timeout(std::time::Duration::from_secs(10), async { + tokio::join!(set.heal_object(bucket, object, "", &heal_options), lose_source) + }) + .await + .expect("heal must report source loss without waiting indefinitely"); + drop(pause_scope); + assert_eq!( + outcome.expect_err("source loss after writer admission must preserve the outer failure"), + DiskError::ErasureReadQuorum + ); + assert_eq!( + non_trash_tmp_entries(&dirs).await, + tmp_before, + "reconstruction failure must clean every staged part" + ); + for (dir, expected) in dirs.iter().zip(metadata_before) { + assert_eq!( + tokio::fs::read(dir.path().join(bucket).join(object).join("xl.meta")) + .await + .expect("metadata after reconstruction failure"), + expected, + "a failed data stage must not publish replacement metadata" + ); + } + tokio::fs::write(&source_path, source_bytes) + .await + .expect("restore the committed source shard"); + + let (_, error) = set + .heal_object(bucket, object, "", &heal_options) + .await + .expect("retry after target recovery"); + assert!(error.is_none(), "recovered targets should heal both parts: {error:?}"); + let mut reader = set + .get_object_reader(bucket, object, None, Default::default(), &options) + .await + .expect("read repaired multipart object"); + let mut actual = Vec::new(); + reader + .stream + .read_to_end(&mut actual) + .await + .expect("complete repaired object body"); + assert_eq!(actual, payloads.concat()); + } + async fn real_disk() -> (TempDir, Endpoint, DiskStore) { let dir = tempfile::tempdir().expect("tempdir should be created"); let endpoint = @@ -4180,6 +4607,11 @@ mod heal_result_report_tests { expected.to_string(), "{case}: after.drives must reflect the actual rename outcome at index {index}" ); + let data_path = temp_dirs[index].path().join(&bucket).join(object).join(data_dir.to_string()); + assert!( + !data_path.join(crate::disk::STORAGE_FORMAT_FILE_BACKUP).exists(), + "{case}: heal must retire the backup even when it shares the live data directory" + ); assert_eq!( temp_dirs[index] .path() @@ -5123,6 +5555,331 @@ mod heal_result_report_tests { } } + #[test] + fn heal_rollback_backup_selection_preserves_legacy_peer_semantics() { + let a = Uuid::from_u128(1); + let b = Uuid::from_u128(2); + for (old, rollback, cleanup, expected) in [ + (None, None, None, None), + (Some(a), None, Some(a), None), + (Some(a), Some(a), Some(a), Some(a)), + (Some(a), Some(b), Some(a), Some(b)), + (Some(a), None, None, Some(a)), + ] { + let response = crate::disk::RenameDataResp { + old_data_dir: old, + rollback_data_dir: rollback, + cleanup_data_dir: cleanup, + ..Default::default() + }; + assert_eq!(heal_rollback_backup_dir(&response), expected); + } + let legacy: crate::disk::RenameDataResp = + serde_json::from_str(r#"{"old_data_dir":"00000000-0000-0000-0000-000000000001","sign":null}"#) + .expect("older peer response without rollback or cleanup fields"); + assert_eq!(heal_rollback_backup_dir(&legacy), Some(a)); + } + + #[tokio::test] + async fn non_inline_heal_preserves_backup_after_publication_fence_loss() { + let (dirs, disks, set) = hermetic_set_disks_isolated(4).await; + let bucket = "heal-non-inline-fence"; + let object = "object"; + for disk in &disks { + disk.make_volume(bucket).await.expect("bucket volume"); + } + let options = ObjectOptions { + no_lock: true, + ..Default::default() + }; + set.put_object(bucket, object, &mut PutObjReader::from_vec(vec![0x11; 1024 * 1024]), &options) + .await + .expect("seed external generation"); + let old = disks[0] + .read_version("", bucket, object, "", &ReadOptions::default()) + .await + .expect("old target"); + assert!(old.data.is_none()); + let old_dir = old.data_dir.expect("external old data directory"); + let metadata_path = dirs[0].path().join(bucket).join(object).join(STORAGE_FORMAT_FILE); + let old_metadata = tokio::fs::read(&metadata_path).await.expect("save stale metadata"); + let payload = vec![0x22; 1024 * 1024]; + set.put_object(bucket, object, &mut PutObjReader::from_vec(payload.clone()), &options) + .await + .expect("write external current generation"); + let current = disks[0] + .read_version("", bucket, object, "", &ReadOptions::default()) + .await + .expect("current target"); + assert!(current.data.is_none()); + let part = format!("{object}/{}/part.1", current.data_dir.expect("current external directory")); + let expected_shard = disks[0].read_all(bucket, &part).await.expect("current encoded shard"); + tokio::fs::write(&metadata_path, old_metadata) + .await + .expect("restore stale metadata"); + let backup = metadata_path + .parent() + .expect("object directory") + .join(old_dir.to_string()) + .join(crate::disk::STORAGE_FORMAT_FILE_BACKUP); + let can_commit = || !backup.exists(); + let opts = HealOpts { + no_lock: true, + scan_mode: HealScanMode::Deep, + ..Default::default() + }; + let (_, error) = set + .heal_object_with_explicit_version_regen( + bucket, + object, + "", + ExplicitVersionHeal { + opts: &opts, + allow_regeneration: true, + preserve_unrepairable: false, + commit_fence: Some(&can_commit), + namespace_owner: None, + retirement: None, + }, + &mut None, + ) + .await + .expect("report committed external repair"); + assert!( + error.is_none(), + "post-publication fence loss must preserve the committed target: {error:?}" + ); + assert!(backup.exists(), "lost fence must retain the actual rollback witness"); + assert_eq!(disks[0].read_all(bucket, &part).await.expect("repaired physical shard"), expected_shard); + let mut reader = set + .get_object_reader(bucket, object, None, Default::default(), &options) + .await + .expect("committed object remains readable"); + let mut body = Vec::new(); + tokio::io::copy(&mut reader, &mut body).await.expect("complete repaired GET"); + assert_eq!(body, payload); + } + + #[cfg(not(windows))] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn cancelled_heal_backup_cleanup_retains_key_owner_until_unlink_finishes() { + use crate::disk::os::prepared_publication_test_hooks::{Stage, install_at}; + use std::time::Duration; + + for mode in ["outer", "inner", "read-repair"] { + let (dirs, disks, set) = hermetic_set_disks_isolated(4).await; + let bucket = format!("heal-cleanup-cancel-{mode}"); + let object = "object"; + for disk in &disks { + disk.make_volume(&bucket).await.expect("create bucket volume"); + } + let options = ObjectOptions { + no_lock: true, + ..Default::default() + }; + set.put_object(&bucket, object, &mut PutObjReader::from_vec(vec![0x11; 1024 * 1024]), &options) + .await + .expect("write old generation on every disk"); + let metadata_path = dirs[0].path().join(&bucket).join(object).join(STORAGE_FORMAT_FILE); + let old_metadata = tokio::fs::read(&metadata_path).await.expect("save stale replica"); + let old = disks[0] + .read_version("", &bucket, object, "", &ReadOptions::default()) + .await + .expect("read old non-inline target"); + assert!(old.data.is_none(), "cancellation fixture must use non-inline publication"); + let rollback_dir = old.data_dir.expect("old external data directory"); + let payload = vec![0x22; 1024 * 1024]; + set.put_object(&bucket, object, &mut PutObjReader::from_vec(payload.clone()), &options) + .await + .expect("write current generation on every disk"); + tokio::fs::write(&metadata_path, old_metadata) + .await + .expect("restore stale target"); + let backup_path = disks[0] + .get_object_path_for_io_if_local( + &bucket, + &format!("{object}/{rollback_dir}/{}", crate::disk::STORAGE_FORMAT_FILE_BACKUP), + ) + .expect("fixture disk is local") + .expect("physical backup path"); + let (arrived_tx, arrived_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel(); + let _hook = install_at(Stage::Remove, &backup_path, move || { + let _ = arrived_tx.send(()); + let _ = release_rx.recv_timeout(Duration::from_secs(30)); + }); + let task_set = set.clone(); + let task_bucket = bucket.clone(); + let heal = tokio::spawn(async move { + let opts = HealOpts { + read_repair: mode == "read-repair", + scan_mode: HealScanMode::Deep, + ..Default::default() + }; + if mode == "outer" { + let (_, error) = task_set + .heal_object_with_retirement( + &task_bucket, + object, + "", + &opts, + &mut None, + AbsenceProofRequest { + retirement: None, + allow_unversioned: false, + }, + ) + .await + .expect("production wrapper heal should reach cleanup"); + assert!(error.is_none(), "production wrapper heal error: {error:?}"); + } else { + let (_, error) = task_set + .heal_object(&task_bucket, object, "", &opts) + .await + .expect("inner/read-repair heal should reach cleanup"); + assert!(error.is_none(), "inner/read-repair heal error: {error:?}"); + } + }); + let arrived = tokio::time::timeout(Duration::from_secs(10), arrived_rx).await; + if !matches!(arrived, Ok(Ok(()))) { + heal.abort(); + let outcome = heal.await; + let _ = release_tx.send(()); + panic!("{mode}: cleanup must reach the owned physical unlink hook; task: {outcome:?}"); + } + heal.abort(); + assert!(heal.await.expect_err("heal must be cancelled at the unlink").is_cancelled()); + assert!( + set.ctx.namespace_commits_pending(), + "{mode}: physical cleanup must retain namespace ownership" + ); + let namespace = set.new_ns_lock(&bucket, object).await.expect("successor namespace"); + let mut successor = Box::pin(namespace.get_write_lock(Duration::from_secs(30))); + assert!( + tokio::time::timeout(Duration::from_secs(1), &mut successor).await.is_err(), + "{mode}: a successor must not enter while the old unlink is pending" + ); + release_tx.send(()).expect("release physical cleanup"); + let successor = tokio::time::timeout(Duration::from_secs(10), successor) + .await + .expect("physical completion should release the key lock") + .expect("successor should acquire the released lock"); + assert!(!backup_path.exists(), "{mode}: owned unlink must finish before releasing the key"); + drop(successor); + let mut reader = set + .get_object_reader(&bucket, object, None, Default::default(), &options) + .await + .expect("committed object remains readable after cancellation"); + let mut body = Vec::new(); + tokio::io::copy(&mut reader, &mut body) + .await + .expect("read full committed body"); + assert_eq!(body, payload, "{mode}: cancellation must preserve committed data"); + } + } + + #[tokio::test] + async fn heal_reclaims_published_inline_backup_and_preserves_child_key() { + let (dirs, disks, set) = hermetic_set_disks_isolated(4).await; + let bucket = "heal-inline-rollback"; + let object = "nested/object"; + for disk in &disks { + disk.make_volume(bucket).await.expect("create bucket volume"); + } + let options = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let rollback_dir = crate::disk::local::inline_metadata_rollback_dir(Uuid::nil(), &rustfs_filemeta::FileMeta::new()); + let child = format!("{object}/{rollback_dir}"); + let child_payload = b"a child key must survive parent repair".to_vec(); + set.put_object(bucket, &child, &mut PutObjReader::from_vec(child_payload.clone()), &options) + .await + .expect("write child key in the synthetic rollback directory"); + set.put_object(bucket, object, &mut PutObjReader::from_vec(vec![0x11; 592]), &options) + .await + .expect("write old inline generation on every disk"); + let metadata_path = dirs[0].path().join(bucket).join(object).join(STORAGE_FORMAT_FILE); + let old_metadata = tokio::fs::read(&metadata_path).await.expect("save stale target metadata"); + let payload = vec![0x22; 592]; + set.put_object(bucket, object, &mut PutObjReader::from_vec(payload.clone()), &options) + .await + .expect("write current inline generation on every disk"); + let read_options = ReadOptions { + read_data: true, + ..Default::default() + }; + let expected = disks[0] + .read_version("", bucket, object, "", &read_options) + .await + .expect("read current target generation"); + assert!(expected.data.is_some(), "fixture must exercise inline publication"); + tokio::fs::write(&metadata_path, &old_metadata) + .await + .expect("restore one stale replica without a pre-existing delete transaction"); + let backup_path = metadata_path + .parent() + .expect("object directory") + .join(rollback_dir.to_string()) + .join(crate::disk::STORAGE_FORMAT_FILE_BACKUP); + let opts = HealOpts { + no_lock: true, + scan_mode: HealScanMode::Deep, + ..Default::default() + }; + let (_, error) = set + .heal_object(bucket, object, "", &HealOpts { dry_run: true, ..opts }) + .await + .expect("dry-run stale inline repair"); + assert!(error.is_none(), "dry-run error: {error:?}"); + assert_eq!(tokio::fs::read(&metadata_path).await.expect("dry-run target"), old_metadata); + assert!(!backup_path.exists(), "dry-run must not publish a backup"); + + let failure_scope = HealRenameFailureScope::install(bucket, object, &[expected.erasure.distribution[0] - 1]); + let (_, error) = set + .heal_object(bucket, object, "", &opts) + .await + .expect("report failed publication"); + drop(failure_scope); + assert!(error.is_some(), "failed rename must remain retryable"); + assert_eq!(tokio::fs::read(&metadata_path).await.expect("failed target"), old_metadata); + assert!(!backup_path.exists(), "failed publication must not reclaim or manufacture a backup"); + + let (result, error) = set + .heal_object(bucket, object, "", &opts) + .await + .expect("repair stale inline replica"); + assert!(error.is_none(), "repair error: {error:?}"); + assert_eq!(result.drives_healed(), Some(1)); + let repaired = disks[0] + .read_version("", bucket, object, "", &read_options) + .await + .expect("read repaired physical target"); + assert_eq!(repaired.mod_time, expected.mod_time); + assert_eq!(repaired.data_dir, expected.data_dir); + assert_eq!(repaired.data, expected.data); + assert!(!backup_path.exists(), "successful heal must retire its old-generation backup"); + let evidence = disks[0] + .discover_delete_transactions(bucket, Some(object), None, 128) + .await + .expect("discover delete evidence without treating a child key as a transaction"); + assert!(evidence.transactions.is_empty(), "successful repair must leave no delete blocker"); + for (key, expected_body) in [(object, payload), (child.as_str(), child_payload)] { + let mut reader = set + .get_object_reader(bucket, key, None, Default::default(), &options) + .await + .expect("read repaired parent and preserved child"); + let mut body = Vec::new(); + tokio::io::copy(&mut reader, &mut body) + .await + .expect("read complete object body"); + assert_eq!(body, expected_body, "object {key}"); + } + let (healthy, error) = set.heal_object(bucket, object, "", &opts).await.expect("repeat repair"); + assert!(error.is_none(), "repeat repair error: {error:?}"); + assert_eq!(healthy.drives_healed(), Some(0)); + } + #[tokio::test] async fn deep_heal_inline_bitrot_rebuilds_physical_data_and_parity_shards() { use crate::storage_api_contracts::range::HTTPRangeSpec; @@ -5140,12 +5897,13 @@ mod heal_result_report_tests { read_data: true, ..Default::default() }; - let object_options = ObjectOptions { + let mut object_options = ObjectOptions { no_lock: true, ..Default::default() }; for parity in [false, true] { + object_options.versioned = parity; for corrupt_prefix in [false, true] { let object = format!("parity-{parity}-prefix-{corrupt_prefix}.bin"); set.put_object( @@ -5224,6 +5982,19 @@ mod heal_result_report_tests { assert_eq!(repaired.data, original_meta.data, "heal must restore the exact encoded shard"); assert_eq!(repaired.version_id, original_meta.version_id); assert_eq!(repaired.data_dir, original_meta.data_dir); + let rollback_dir = crate::disk::local::inline_metadata_rollback_dir( + repaired.version_id.unwrap_or_default(), + &rustfs_filemeta::FileMeta::new(), + ); + assert!( + !metadata_path + .parent() + .expect("object directory") + .join(rollback_dir.to_string()) + .join(crate::disk::STORAGE_FORMAT_FILE_BACKUP) + .exists(), + "inline bitrot repair must not retain a rollback backup" + ); } for (range, expected) in [ diff --git a/crates/ecstore/src/set_disk/ops/heal/shard_integrity_rollout_tests.rs b/crates/ecstore/src/set_disk/ops/heal/shard_integrity_rollout_tests.rs index 637e80a31..a03192772 100644 --- a/crates/ecstore/src/set_disk/ops/heal/shard_integrity_rollout_tests.rs +++ b/crates/ecstore/src/set_disk/ops/heal/shard_integrity_rollout_tests.rs @@ -15,7 +15,9 @@ use crate::disk::{DiskAPI as _, ReadOptions}; use crate::object_api::{ObjectOptions, PutObjReader, ShardIntegrityWriteMode}; use crate::set_disk::CompletePart; -use crate::set_disk::ops::object::hermetic_set_disks_support::hermetic_set_disks_for_pool_with_default_parity_isolated; +use crate::set_disk::ops::object::hermetic_set_disks_support::{ + hermetic_set_disks_for_pool_with_default_parity_isolated, lifecycle_owned_put_options, +}; use crate::storage_api_contracts::multipart::MultipartOperations as _; use crate::storage_api_contracts::object::{ObjectIO as _, ObjectOperations as _}; use rustfs_heal_contracts::heal_channel::{HealOpts, HealScanMode}; @@ -39,18 +41,19 @@ async fn remote_metadata_heal_does_not_certify_unread_tier_payload() { for disk in &disks { disk.make_volume(bucket).await.expect("fixture bucket"); } - set.put_object( + let opts = lifecycle_owned_put_options( + &set, bucket, - object, - &mut PutObjReader::from_vec(vec![0x31; 1024 * 1024 + 17]), - &ObjectOptions { + ObjectOptions { no_lock: true, shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Protected), ..Default::default() }, ) - .await - .expect("protected source"); + .await; + set.put_object(bucket, object, &mut PutObjReader::from_vec(vec![0x31; 1024 * 1024 + 17]), &opts) + .await + .expect("protected source"); for (disk, dir) in disks.iter().zip(&dirs) { let mut info = disk .read_version("", bucket, object, "", &ReadOptions::default()) @@ -94,10 +97,15 @@ async fn rollout_requires_both_flags_for_new_puts() { for disk in &disks { disk.make_volume(bucket).await.expect("fixture bucket"); } - let opts = ObjectOptions { - no_lock: true, - ..Default::default() - }; + let opts = lifecycle_owned_put_options( + &set, + bucket, + ObjectOptions { + no_lock: true, + ..Default::default() + }, + ) + .await; for (requested, confirmed, protected) in [ (false, false, false), @@ -159,12 +167,17 @@ async fn legacy_shard_repair_and_explicit_version_metadata_recovery_remain_avail for disk in &disks { disk.make_volume(bucket).await.expect("fixture bucket"); } - let opts = ObjectOptions { - no_lock: true, - versioned: true, - shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Legacy), - ..Default::default() - }; + let opts = lifecycle_owned_put_options( + &set, + bucket, + ObjectOptions { + no_lock: true, + versioned: true, + shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Legacy), + ..Default::default() + }, + ) + .await; let expected = vec![0x67; 1024 * 1024 + 37]; for case in ["missing-part", "bad-bitrot", "missing-version-metadata"] { let object = case; @@ -272,10 +285,15 @@ async fn multipart_mode_survives_rollout_switch_changes() { for disk in &disks { disk.make_volume(bucket).await.expect("fixture bucket"); } - let opts = ObjectOptions { - no_lock: true, - ..Default::default() - }; + let opts = lifecycle_owned_put_options( + &set, + bucket, + ObjectOptions { + no_lock: true, + ..Default::default() + }, + ) + .await; for protected in [false, true] { let object = format!("upload-{protected}"); let upload = @@ -354,11 +372,16 @@ async fn physical_copy_inherits_source_mode_instead_of_current_switch() { }; let object = format!("source-{protected}"); let expected = vec![0x39; 1024 * 1024 + 11]; - let seed_opts = ObjectOptions { - no_lock: true, - shard_integrity_write_mode: Some(mode), - ..Default::default() - }; + let seed_opts = lifecycle_owned_put_options( + &set, + bucket, + ObjectOptions { + no_lock: true, + shard_integrity_write_mode: Some(mode), + ..Default::default() + }, + ) + .await; let mut source = set .put_object(bucket, &object, &mut PutObjReader::from_vec(expected.clone()), &seed_opts) .await @@ -366,8 +389,8 @@ async fn physical_copy_inherits_source_mode_instead_of_current_switch() { source.metadata_only = false; source.put_object_reader = Some(PutObjReader::from_vec(expected.clone())); let opts = ObjectOptions { - no_lock: true, - ..Default::default() + shard_integrity_write_mode: None, + ..seed_opts.clone() }; let copied = temp_env::async_with_vars( rollout_vars(!protected, !protected), diff --git a/crates/ecstore/src/set_disk/ops/locking.rs b/crates/ecstore/src/set_disk/ops/locking.rs index 6c3fc112f..df3f8400e 100644 --- a/crates/ecstore/src/set_disk/ops/locking.rs +++ b/crates/ecstore/src/set_disk/ops/locking.rs @@ -28,6 +28,7 @@ use crate::disk::health_state::DriveMembershipSnapshot; use crate::disk::{DiskAPI, new_disk}; use crate::runtime::sources as runtime_sources; use rand::prelude::SliceRandom; +use rustfs_utils::path::object_name_disk_identity; #[cfg(test)] use uuid::Uuid; @@ -63,9 +64,12 @@ impl crate::storage_api_contracts::namespace::NamespaceLocking for SetDisks { NamespaceLock::with_local_manager_shared(self.set_lock_namespace.clone(), self.local_lock_manager.clone()) }; + // Lock the file the disk resolves, not the spelling: on Windows `a\b` + // and `a/b` share one xl.meta, so they must share one lock. The key is + // folded by the requesting node only; lock servers compare it verbatim. let resource = ObjectKey { bucket: Arc::from(bucket), - object: Arc::from(object), + object: Arc::from(object_name_disk_identity(object).as_ref()), version: None, }; diff --git a/crates/ecstore/src/set_disk/ops/mod.rs b/crates/ecstore/src/set_disk/ops/mod.rs index 4cea63793..5dfaa66b5 100644 --- a/crates/ecstore/src/set_disk/ops/mod.rs +++ b/crates/ecstore/src/set_disk/ops/mod.rs @@ -19,6 +19,7 @@ pub(crate) mod bitrot_self_verify; pub(crate) mod bucket; +pub(crate) mod delete_recovery; pub(crate) mod heal; pub(crate) mod heal_walk; pub(crate) mod list; diff --git a/crates/ecstore/src/set_disk/ops/multipart.rs b/crates/ecstore/src/set_disk/ops/multipart.rs index af09429d7..bc0d5f00e 100644 --- a/crates/ecstore/src/set_disk/ops/multipart.rs +++ b/crates/ecstore/src/set_disk/ops/multipart.rs @@ -63,7 +63,6 @@ use crate::crash_inject::{self, CrashPoint}; use crate::disk::DiskAPI; #[cfg(test)] use crate::disk::DiskOption; -#[cfg(test)] use crate::disk::STORAGE_FORMAT_FILE; #[cfg(test)] use crate::disk::new_disk; @@ -231,6 +230,60 @@ pub(crate) struct StaleMultipartCleanupGuard { lock_guard: ObjectLockDiagGuard, } +/// Exact-key recovery is admitted only after every disk proves that the upload +/// metadata is absent. Keep the checked disk generation and upload lock through +/// deletion; a quorum miss is not proof that an upload is an orphan. +/// Final object data, completion replay identity, and old-data cleanup receipts +/// live under the destination bucket/object, never this staging path. This also +/// permits removing an exact orphan from a previous bucket incarnation without +/// claiming that its UploadID belongs to the current incarnation. +pub(crate) struct OrphanMultipartCleanupGuard { + set: Arc, + disks: Vec>, + upload_path: String, + lock_guard: Arc, + exists: bool, +} + +impl OrphanMultipartCleanupGuard { + pub(crate) async fn delete(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result { + fence_commit_on_lock_loss(Some(&self.lock_guard), "orphan_multipart_cleanup", &self.upload_path)?; + ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?; + if !self.exists { + return Ok(false); + } + + let results = join_all(self.disks.iter().map(|disk| async { + let Some(disk) = disk else { + return Err(DiskError::DiskNotFound); + }; + match disk + .delete( + RUSTFS_META_MULTIPART_BUCKET, + &self.upload_path, + crate::disk::DeleteOptions { + recursive: true, + ..Default::default() + }, + ) + .await + { + Err(DiskError::FileNotFound | DiskError::PathNotFound | DiskError::VolumeNotFound) => Ok(()), + result => result, + } + })) + .await; + self.set.record_capacity_scope_if_needed(None, &self.disks); + let errors = results.into_iter().map(|result| result.err()).collect::>(); + if let Some(err) = reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, self.disks.len()) { + return Err(to_object_err(err.into(), vec![bucket, object])); + } + fence_commit_on_lock_loss(Some(&self.lock_guard), "orphan_multipart_cleanup", &self.upload_path)?; + ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?; + Ok(true) + } +} + pub(crate) struct DataMovementMultipartAbortGuard { upload_path: String, write_quorum: Option, @@ -557,6 +610,34 @@ fn multipart_bucket_incarnation_matches(metadata: &HashMap, expe matches!(multipart_bucket_incarnation_id(metadata), Ok(Some(actual)) if actual == expected) } +fn orphan_multipart_upload_id_is_canonical(upload_id: &str) -> bool { + let Ok(decoded) = base64_simd::URL_SAFE_NO_PAD.decode_to_vec(upload_id.as_bytes()) else { + return false; + }; + let Ok(decoded) = std::str::from_utf8(&decoded) else { + return false; + }; + let Some((_, raw)) = decoded.split_once('.') else { + return false; + }; + let Some((uuid, timestamp)) = raw.split_once('x') else { + return false; + }; + let Ok(id) = Uuid::parse_str(uuid) else { + return false; + }; + if id.is_nil() || id.to_string() != uuid { + return false; + } + let Ok(timestamp_ns) = timestamp.parse::() else { + return false; + }; + if timestamp_ns.to_string() != timestamp { + return false; + } + OffsetDateTime::from_unix_timestamp_nanos(timestamp_ns).is_ok() +} + fn validate_multipart_bucket_incarnation( metadata: &HashMap, bucket: &str, @@ -1246,6 +1327,114 @@ impl SetDisks { }) } + pub(crate) async fn lock_orphan_multipart_cleanup( + self: &Arc, + bucket: &str, + object: &str, + upload_id: &str, + opts: &ObjectOptions, + held_guards: &[OrphanMultipartCleanupGuard], + ) -> Result { + let invalid_upload = || StorageError::InvalidUploadID(bucket.to_owned(), object.to_owned(), upload_id.to_owned()); + if opts.data_movement + || opts.no_lock + || self.local_lock_manager.is_disabled() + || !orphan_multipart_upload_id_is_canonical(upload_id) + { + return Err(invalid_upload()); + } + // This path is derived from the caller's exact bucket/key and a canonical + // UUID/timestamp upload identity, never from the shared MPU directory list. + let upload_path = Self::get_upload_id_dir(bucket, object, upload_id); + // Pool labels do not partition the raw ObjectKey sent to lock clients. + // Reuse the actual owner only for this exact path and proven backend; + // independent distributed domains still need their own guards. + let distributed = self.ctx.is_dist_erasure().await; + let mut lock_guard = None; + for held in held_guards { + if held.upload_path != upload_path { + continue; + } + let shared_domain = match (distributed, held.set.ctx.is_dist_erasure().await) { + (false, false) => self.shares_namespace_lock_domain(&held.set).await, + (true, true) => { + let same = super::super::same_distributed_lock_domain(&self.lockers, &held.set.lockers); + if !same + && self + .lockers + .iter() + .any(|client| held.set.lockers.iter().any(|held_client| Arc::ptr_eq(client, held_client))) + { + // A guard for only part of another quorum is not proof + // of ownership there, and acquiring it again may wait + // on ourselves. Preserve all staging in this topology. + return Err(Error::other("orphan multipart cleanup cannot cover partially overlapping lock domains")); + } + same + } + _ => false, + }; + if shared_domain { + lock_guard = Some(Arc::clone(&held.lock_guard)); + break; + } + } + let lock_guard = match lock_guard { + Some(guard) => guard, + None => Arc::new( + self.acquire_write_lock_diag("orphan_multipart_cleanup", RUSTFS_META_MULTIPART_BUCKET, &upload_path) + .await?, + ), + }; + fence_commit_on_lock_loss(Some(&lock_guard), "orphan_multipart_cleanup", &upload_path)?; + ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?; + let disks = self.disks.read().await.clone(); + if disks.len() != self.set_drive_count || disks.is_empty() { + return Err(Error::ErasureReadQuorum); + } + // Raw full fan-out cannot quorum-short-circuit or interpret an empty + // version list as absence of the physical metadata file. + let (_, errors) = Self::read_all_raw_file_info(&disks, RUSTFS_META_MULTIPART_BUCKET, &upload_path, false).await; + if errors.len() != disks.len() { + return Err(Error::ErasureReadQuorum); + } + for error in errors { + match error { + Some(DiskError::FileNotFound | DiskError::VolumeNotFound) => {} + // Even one valid metadata copy may still be recoverable. Never + // convert a partial, corrupt, or unreachable upload into debris. + None => return Err(invalid_upload()), + Some(error) => return Err(to_object_err(error.into(), vec![bucket, object, upload_id])), + } + } + let mut exists = false; + for disk in &disks { + let Some(disk) = disk else { + return Err(Error::ErasureReadQuorum); + }; + match disk.list_dir(bucket, RUSTFS_META_MULTIPART_BUCKET, &upload_path, -1).await { + Ok(entries) => { + // Raw metadata reads report a directory named xl.meta as + // FileNotFound. Its physical presence is unknown/corrupt + // metadata, not proof that this staging can be deleted. + if entries.iter().any(|entry| entry.trim_end_matches('/') == STORAGE_FORMAT_FILE) { + return Err(to_object_err(DiskError::FileCorrupt.into(), vec![bucket, object, upload_id])); + } + exists = true; + } + Err(DiskError::FileNotFound | DiskError::VolumeNotFound) => {} + Err(error) => return Err(to_object_err(error.into(), vec![bucket, object, upload_id])), + } + } + Ok(OrphanMultipartCleanupGuard { + set: self.clone(), + disks, + upload_path, + lock_guard, + exists, + }) + } + #[allow(clippy::too_many_arguments)] pub(crate) async fn list_multipart_uploads_for_incarnation( &self, @@ -1678,6 +1867,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { drop(admission_guard); } + let mut commit_owns_cleanup = false; let result: Result = async { let erasure = Arc::new(coding::Erasure::try_new(fi.erasure.data_blocks, fi.erasure.parity_blocks, fi.erasure.block_size) @@ -1881,120 +2071,171 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { .await?; (Some(upload_guard), Some(part_guard)) }; - let (commit_fi, _) = self - .check_upload_id_exists_with_opts(bucket, object, upload_id, multipart_size_limit.is_some(), opts) - .await?; - let commit_size_limit = multipart_size_limit_from_metadata(&commit_fi.metadata)?; - if commit_size_limit != multipart_size_limit { - return Err(Error::InvalidArgument( - "multipart upload".to_string(), - SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(), - "size limit metadata changed or is missing".to_string(), - )); - } - ensure_data_movement_upload_access(&commit_fi, bucket, object, upload_id, opts)?; - ensure_multipart_bucket_incarnation( - &self.ctx, - &commit_fi, - bucket, - object, - upload_id, - opts.expected_bucket_incarnation_id, - ) - .await?; - #[cfg(test)] - pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockLost).await; - fence_commit_on_lock_loss(_upload_commit_guard.as_ref(), "put_object_part_commit", &upload_id_path)?; - fence_commit_on_lock_loss(_part_commit_guard.as_ref(), "put_object_part_commit", &part_lock_path)?; - if opts - .namespace_lock_fence - .as_ref() - .is_some_and(NamespaceLockFence::is_lock_lost) - { - return Err(StorageError::NamespaceLockQuorumUnavailable { - mode: "put_object_part_outer_lock", - bucket: bucket.to_string(), - object: object.to_string(), - required: 1, - achieved: 0, - }); - } - ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?; - - if let Some(limit) = commit_size_limit { - let current_size = self - .current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &commit_fi, part_id) - .await?; - let candidate_size = u64::try_from(actual_size).map_err(|_| Error::PartMissingOrCorrupt)?; - admitted_multipart_size(current_size, candidate_size, limit)?; - } - - let decommission_capacity_guard = if let Some(store) = opts.decommission_capacity_admission.as_ref() { - Some( - store - .acquire_external_decommission_capacity_fence( - &[self.pool_index], - DecommissionCapacityAdmission::ExistingMultipart, - ) - .await?, - ) - } else { - None - }; - #[cfg(test)] - pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartAfterCapacityAdmission).await; - if decommission_capacity_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) { - return Err(StorageError::NamespaceLockQuorumUnavailable { - mode: "put_object_part_decommission_capacity", - bucket: bucket.to_string(), - object: object.to_string(), - required: 1, - achieved: 0, - }); - } - let rename_result = self - .rename_part( - &shuffle_disks, - RUSTFS_META_TMP_BUCKET, - &tmp_part_path, - RUSTFS_META_MULTIPART_BUCKET, - &part_path, - part_info_buff.into(), - write_quorum, - Some(MultipartWriteQuorumContext { - stage: MULTIPART_WRITE_QUORUM_RENAME_PART, + // Only guards owned by this operation may outlive its caller. Internal + // data-movement and borrowed-lock callers retain their synchronous + // lifetime; a fence by itself cannot transfer the outer lock lease. + let detach_commit_owner = _upload_commit_guard.is_some() + && opts.namespace_lock_fence.is_none() + && (opts.bucket_lifecycle_lock_fence.is_none() || opts.bucket_lifecycle_lock_owner.is_some()); + let commit_set = self.clone(); + let commit_bucket = bucket.to_owned(); + let commit_object = object.to_owned(); + let commit_upload_id = upload_id.to_owned(); + let commit_upload_id_path = upload_id_path.clone(); + let commit_tmp_part = tmp_part.clone(); + let commit_opts = opts.clone(); + commit_owns_cleanup = true; + // The body is already staged. Keep every commit guard through the + // prepare, publish, settlement/rollback and temporary cleanup I/O, + // even when the HTTP request stops awaiting this task. + let commit = async move { + let _upload_commit_guard = _upload_commit_guard; + let _part_commit_guard = _part_commit_guard; + let _capped_staging_guard = _capped_staging_guard; + let bucket = commit_bucket.as_str(); + let object = commit_object.as_str(); + let upload_id = commit_upload_id.as_str(); + let upload_id_path = commit_upload_id_path; + let opts = &commit_opts; + let result: Result = async { + let (commit_fi, _) = commit_set + .check_upload_id_exists_with_opts(bucket, object, upload_id, multipart_size_limit.is_some(), opts) + .await?; + let commit_size_limit = multipart_size_limit_from_metadata(&commit_fi.metadata)?; + if commit_size_limit != multipart_size_limit { + return Err(Error::InvalidArgument( + "multipart upload".to_string(), + SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(), + "size limit metadata changed or is missing".to_string(), + )); + } + ensure_data_movement_upload_access(&commit_fi, bucket, object, upload_id, opts)?; + ensure_multipart_bucket_incarnation( + &commit_set.ctx, + &commit_fi, bucket, object, upload_id, - part_number: Some(part_id), - }), - ) + opts.expected_bucket_incarnation_id, + ) + .await?; + #[cfg(test)] + pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockLost).await; + fence_commit_on_lock_loss(_upload_commit_guard.as_ref(), "put_object_part_commit", &upload_id_path)?; + fence_commit_on_lock_loss(_part_commit_guard.as_ref(), "put_object_part_commit", &part_lock_path)?; + if opts + .namespace_lock_fence + .as_ref() + .is_some_and(NamespaceLockFence::is_lock_lost) + { + return Err(StorageError::NamespaceLockQuorumUnavailable { + mode: "put_object_part_outer_lock", + bucket: bucket.to_string(), + object: object.to_string(), + required: 1, + achieved: 0, + }); + } + ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?; + + if let Some(limit) = commit_size_limit { + let current_size = commit_set + .current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &commit_fi, part_id) + .await?; + let candidate_size = u64::try_from(actual_size).map_err(|_| Error::PartMissingOrCorrupt)?; + admitted_multipart_size(current_size, candidate_size, limit)?; + } + + let decommission_capacity_guard = if let Some(store) = opts.decommission_capacity_admission.as_ref() { + Some( + store + .acquire_external_decommission_capacity_fence( + &[commit_set.pool_index], + DecommissionCapacityAdmission::ExistingMultipart, + ) + .await?, + ) + } else { + None + }; + #[cfg(test)] + pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartAfterCapacityAdmission).await; + if decommission_capacity_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) { + return Err(StorageError::NamespaceLockQuorumUnavailable { + mode: "put_object_part_decommission_capacity", + bucket: bucket.to_string(), + object: object.to_string(), + required: 1, + achieved: 0, + }); + } + let rename_result = commit_set + .rename_part( + &shuffle_disks, + RUSTFS_META_TMP_BUCKET, + &tmp_part_path, + RUSTFS_META_MULTIPART_BUCKET, + &part_path, + part_info_buff.into(), + write_quorum, + Some(MultipartWriteQuorumContext { + stage: MULTIPART_WRITE_QUORUM_RENAME_PART, + bucket, + object, + upload_id, + part_number: Some(part_id), + }), + ) + .await; + drop(decommission_capacity_guard); + let _ = rename_result?; + #[cfg(test)] + observe_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockLost); + + #[cfg(test)] + pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartAfterRename).await; + + let ret: PartInfo = PartInfo { + etag: Some(etag.clone()), + part_num: part_id, + last_mod: Some(OffsetDateTime::now_utc()), + size: w_size, + actual_size, + }; + + // error!("put_object_part ret {:?}", &ret); + + Ok(ret) + } .await; - drop(decommission_capacity_guard); - let _ = rename_result?; - #[cfg(test)] - observe_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockLost); - - #[cfg(test)] - pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartAfterRename).await; - drop(_part_commit_guard); - drop(_upload_commit_guard); - - let ret: PartInfo = PartInfo { - etag: Some(etag.clone()), - part_num: part_id, - last_mod: Some(OffsetDateTime::now_utc()), - size: w_size, - actual_size, + if result.is_err() + && let Err(err) = commit_set.delete_all(RUSTFS_META_TMP_BUCKET, &commit_tmp_part).await + { + warn!( + event = EVENT_SET_DISK_MULTIPART, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_SET_DISK, + result = "cleanup_failed", + bucket, + object, + error = ?err, + "failed to clean multipart commit staging" + ); + } + result }; - - // error!("put_object_part ret {:?}", &ret); - - Ok(ret) + if detach_commit_owner { + tokio::spawn(commit) + .await + .map_err(|err| Error::other_with_context("put_object_part commit task failed", err))? + } else { + commit.await + } } .await; - if result.is_err() + if !commit_owns_cleanup + && result.is_err() && let Err(err) = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_part).await { warn!(tmp_part = %tmp_part, error = ?err, "failed to cleanup multipart temporary data"); @@ -3385,7 +3626,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { #[cfg(any(test, feature = "test-util"))] pause_multipart_commit(&commit_bucket, &commit_object, MultipartCommitPause::BeforeQuotaRename).await; if quota_reservation.is_lock_lost() - || !quota_reservation.capability_proof_matches() || write_context.is_lock_lost() || _object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) || _decommission_object_lock_guard @@ -3440,7 +3680,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { verify_object_transaction_epoch_fence(&commit_set, &commit_bucket, &commit_object, expected).await?; } if quota_reservation.is_lock_lost() - || !quota_reservation.capability_proof_matches() || write_context.is_lock_lost() || _object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) || _decommission_object_lock_guard @@ -4240,6 +4479,300 @@ mod tests { } } + #[test] + fn orphan_multipart_identity_requires_canonical_upload_id() { + let created = OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(10); + let id = Uuid::new_v4(); + let timestamp = (created + time::Duration::seconds(1)).unix_timestamp_nanos(); + let encode = |raw: &str| runtime_sources::deployment_upload_id(raw); + let upload_id = encode(&format!("{id}x{timestamp}")); + assert!(orphan_multipart_upload_id_is_canonical(&upload_id)); + for raw in [ + format!("{}x{timestamp}", Uuid::nil()), + format!("{}x{timestamp}", id.simple()), + format!("{id}x0{timestamp}"), + format!("{id}x+{timestamp}"), + format!("{id}x{timestamp}/../another-upload"), + format!("../{id}x{timestamp}"), + format!("{id}x{}", i128::MAX), + ] { + assert!( + !orphan_multipart_upload_id_is_canonical(&encode(&raw)), + "noncanonical upload identity must be rejected: {raw}" + ); + } + assert!(!orphan_multipart_upload_id_is_canonical("not-base64")); + } + + #[tokio::test] + async fn orphan_multipart_cleanup_requires_every_disk_to_prove_metadata_absence() { + let (_temp_dirs, disks, set) = hermetic_set_disks(4).await; + let bucket = "orphan-multipart-proof"; + let object = "object"; + make_bucket_on_all(&disks, bucket).await; + let upload = set + .new_multipart_upload(bucket, object, &ObjectOptions::default()) + .await + .expect("create upload"); + let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload.upload_id); + let metadata_path = format!("{upload_path}/{STORAGE_FORMAT_FILE}"); + let leftover_path = format!("{upload_path}/part.late"); + for disk in &disks { + disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &leftover_path, Bytes::from_static(b"late part")) + .await + .expect("write late part fixture"); + } + for disk in &disks[..3] { + disk.delete(RUSTFS_META_MULTIPART_BUCKET, &metadata_path, Default::default()) + .await + .expect("remove majority metadata"); + } + let opts = ObjectOptions::default(); + assert!( + set.lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[]) + .await + .is_err(), + "one surviving metadata copy must block orphan recovery" + ); + disks[3] + .write_all(RUSTFS_META_MULTIPART_BUCKET, &metadata_path, Bytes::from_static(b"corrupt metadata")) + .await + .expect("corrupt final metadata copy"); + assert!( + set.lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[]) + .await + .is_err(), + "corrupt metadata must not be treated as absence" + ); + disks[3] + .delete(RUSTFS_META_MULTIPART_BUCKET, &metadata_path, Default::default()) + .await + .expect("remove final metadata copy"); + set.disks.write().await[3] = None; + assert!( + set.lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[]) + .await + .is_err(), + "an unreachable disk must not be hidden by absence quorum" + ); + set.disks.write().await[3] = Some(disks[3].clone()); + for disk in &disks { + assert_eq!( + disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &leftover_path) + .await + .expect("failed preflight must preserve every part"), + Bytes::from_static(b"late part") + ); + } + + let guard = set + .lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[]) + .await + .expect("all disks now prove metadata absence"); + assert!(guard.delete(bucket, object, &opts).await.expect("recover exact staging")); + drop(guard); + for disk in &disks { + assert!(matches!( + disk.list_dir(bucket, RUSTFS_META_MULTIPART_BUCKET, &upload_path, -1).await, + Err(DiskError::FileNotFound | DiskError::VolumeNotFound) + )); + } + let guard = set + .lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[]) + .await + .expect("already absent staging remains provably absent"); + assert!( + !guard + .delete(bucket, object, &opts) + .await + .expect("absence does not report recovery") + ); + } + + #[tokio::test] + async fn orphan_multipart_cleanup_rejects_metadata_directory_on_one_disk() { + let (temp_dirs, disks, set) = hermetic_set_disks(4).await; + let bucket = "orphan-multipart-directory"; + let object = "object"; + make_bucket_on_all(&disks, bucket).await; + let upload = set + .new_multipart_upload(bucket, object, &ObjectOptions::default()) + .await + .expect("create upload"); + let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload.upload_id); + let metadata_path = format!("{upload_path}/{STORAGE_FORMAT_FILE}"); + let leftover_path = format!("{upload_path}/part.late"); + for disk in &disks { + disk.delete(RUSTFS_META_MULTIPART_BUCKET, &metadata_path, Default::default()) + .await + .expect("remove upload metadata"); + disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &leftover_path, Bytes::from_static(b"late part")) + .await + .expect("write leftover part"); + } + let metadata_directory = temp_dirs[3].path().join(RUSTFS_META_MULTIPART_BUCKET).join(&metadata_path); + tokio::fs::create_dir_all(&metadata_directory) + .await + .expect("replace one missing metadata file with a directory"); + let (_, errors) = + SetDisks::read_all_raw_file_info(&set.disks.read().await, RUSTFS_META_MULTIPART_BUCKET, &upload_path, false).await; + assert!( + errors.iter().all(|error| matches!(error, Some(DiskError::FileNotFound))), + "raw metadata reads alone must reproduce the directory-as-absence case: {errors:?}" + ); + assert!( + set.lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &ObjectOptions::default(), &[]) + .await + .is_err(), + "one physical metadata directory must block cleanup on every disk" + ); + assert!(metadata_directory.is_dir(), "failed preflight must preserve the metadata directory"); + for disk in &disks { + assert_eq!( + disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &leftover_path) + .await + .expect("failed preflight must preserve every leftover part"), + Bytes::from_static(b"late part") + ); + } + } + + #[derive(Clone, Copy, PartialEq, Eq)] + enum OrphanMultipartLockDomainCase { + Shared, + Independent, + PartiallyOverlapping, + } + + async fn check_orphan_multipart_distributed_lock_owners(case: OrphanMultipartLockDomainCase) { + let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + ctx.update_erasure_type(SetupType::DistErasure).await; + let first_client: Arc = + Arc::new(LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::new()))); + let second_client: Arc = + Arc::new(LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::new()))); + let second_clients = match case { + OrphanMultipartLockDomainCase::Shared => vec![first_client.clone()], + OrphanMultipartLockDomainCase::Independent => vec![second_client], + OrphanMultipartLockDomainCase::PartiallyOverlapping => vec![first_client.clone(), second_client], + }; + let (first_dirs, first_disks, first_set) = + hermetic_set_disks_with_lockers_and_ctx(4, 0, 2, vec![first_client], ctx.clone()).await; + let (second_dirs, second_disks, second_set) = hermetic_set_disks_with_lockers_and_ctx(4, 1, 2, second_clients, ctx).await; + let bucket = "orphan-multipart-dist-locks"; + let object = "object"; + make_bucket_on_all(&first_disks, bucket).await; + make_bucket_on_all(&second_disks, bucket).await; + let upload_id = runtime_sources::deployment_upload_id(&format!( + "{}x{}", + Uuid::new_v4(), + OffsetDateTime::now_utc().unix_timestamp_nanos() + )); + let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id); + for directory in first_dirs.iter().chain(second_dirs.iter()) { + let staging = directory.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&upload_path); + tokio::fs::create_dir_all(&staging) + .await + .expect("create exact orphan staging"); + tokio::fs::write(staging.join("part.late"), b"late part") + .await + .expect("write orphan part"); + } + let opts = ObjectOptions::default(); + let first_guard = first_set + .lock_orphan_multipart_cleanup(bucket, object, &upload_id, &opts, &[]) + .await + .expect("first domain has an orphan"); + let second_result = tokio::time::timeout( + Duration::from_secs(10), + second_set.lock_orphan_multipart_cleanup(bucket, object, &upload_id, &opts, std::slice::from_ref(&first_guard)), + ) + .await + .expect("lock-domain admission must not wait on a guard it already owns"); + if case == OrphanMultipartLockDomainCase::PartiallyOverlapping { + let error = match second_result { + Ok(_) => panic!("one existing quorum cannot authorize a partially overlapping domain"), + Err(error) => error, + }; + assert!(error.to_string().contains("partially overlapping lock domains")); + for directory in first_dirs.iter().chain(second_dirs.iter()) { + assert!( + directory + .path() + .join(RUSTFS_META_MULTIPART_BUCKET) + .join(&upload_path) + .join("part.late") + .exists(), + "uncertain lock coverage must preserve every physical part" + ); + } + return; + } + let second_guard = second_result.expect("the second pool has a proven lock domain"); + assert_eq!( + Arc::ptr_eq(&first_guard.lock_guard, &second_guard.lock_guard), + case == OrphanMultipartLockDomainCase::Shared, + "only the same exact backend may share a real upload guard across pool labels" + ); + let second_namespace = second_set + .new_ns_lock(RUSTFS_META_MULTIPART_BUCKET, &upload_path) + .await + .unwrap(); + let mut second_writer = Box::pin(second_namespace.get_write_lock(Duration::from_secs(10))); + assert!(futures::poll!(&mut second_writer).is_pending()); + if case == OrphanMultipartLockDomainCase::Shared { + drop(first_guard); + assert!( + futures::poll!(&mut second_writer).is_pending(), + "the second pool must retain the shared real owner after the first guard drops" + ); + drop(second_guard); + drop( + tokio::time::timeout(Duration::from_secs(10), second_writer) + .await + .expect("the shared backend should release after the last owner") + .expect("shared-domain writer resumes"), + ); + return; + } + let first_namespace = first_set + .new_ns_lock(RUSTFS_META_MULTIPART_BUCKET, &upload_path) + .await + .unwrap(); + let mut first_writer = Box::pin(first_namespace.get_write_lock(Duration::from_secs(10))); + assert!(futures::poll!(&mut first_writer).is_pending()); + drop(first_guard); + drop( + tokio::time::timeout(Duration::from_secs(10), first_writer) + .await + .expect("first domain should release independently") + .expect("first writer resumes"), + ); + assert!(futures::poll!(&mut second_writer).is_pending(), "the second real owner must remain held"); + drop(second_guard); + drop( + tokio::time::timeout(Duration::from_secs(10), second_writer) + .await + .expect("second domain should release with its owner") + .expect("second writer resumes"), + ); + } + + #[tokio::test] + async fn orphan_multipart_cleanup_keeps_separate_distributed_lock_owners() { + check_orphan_multipart_distributed_lock_owners(OrphanMultipartLockDomainCase::Independent).await; + } + + #[tokio::test] + async fn orphan_multipart_cleanup_reuses_shared_distributed_owner_across_pools() { + check_orphan_multipart_distributed_lock_owners(OrphanMultipartLockDomainCase::Shared).await; + } + + #[tokio::test] + async fn orphan_multipart_cleanup_rejects_partially_overlapping_distributed_domains() { + check_orphan_multipart_distributed_lock_owners(OrphanMultipartLockDomainCase::PartiallyOverlapping).await; + } + async fn stage_upload_with_create_opts( set_disks: &Arc, bucket: &str, @@ -5502,11 +6035,18 @@ mod tests { } async fn assert_complete_first_linearizes(bucket: &'static str, object: &'static str, create_opts: ObjectOptions) { + use crate::storage_api_contracts::bucket::BucketOperations as _; + let manager = Arc::new(rustfs_lock::GlobalLockManager::new()); let signaling = Arc::new(SignalingLockClient::new(Arc::new(LocalClient::with_manager(manager)))); let lockers: Vec> = vec![signaling.clone()]; - let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await; - make_bucket_on_all(&disk_stores, bucket).await; + let (_temp_dirs, _disk_stores, set_disks) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await; + crate::bucket::metadata_sys::object_store_in(&set_disks.ctx) + .await + .expect("multipart fixture metadata owner") + .make_bucket(bucket, &Default::default()) + .await + .expect("multipart fixture bucket has authoritative quota metadata"); let (upload_id, parts) = stage_upload_with_create_opts(&set_disks, bucket, object, &[0x47; 4096], &create_opts).await; let upload_id_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id); signaling.set_target(rustfs_lock::ObjectKey::new(RUSTFS_META_MULTIPART_BUCKET, upload_id_path)); @@ -5553,11 +6093,18 @@ mod tests { } async fn assert_abort_first_linearizes(bucket: &'static str, object: &'static str, create_opts: ObjectOptions) { + use crate::storage_api_contracts::bucket::BucketOperations as _; + let manager = Arc::new(rustfs_lock::GlobalLockManager::new()); let signaling = Arc::new(SignalingLockClient::new(Arc::new(LocalClient::with_manager(manager)))); let lockers: Vec> = vec![signaling.clone()]; - let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await; - make_bucket_on_all(&disk_stores, bucket).await; + let (_temp_dirs, _disk_stores, set_disks) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await; + crate::bucket::metadata_sys::object_store_in(&set_disks.ctx) + .await + .expect("multipart fixture metadata owner") + .make_bucket(bucket, &Default::default()) + .await + .expect("multipart fixture bucket has authoritative quota metadata"); let (upload_id, parts) = stage_upload_with_create_opts(&set_disks, bucket, object, &[0x48; 4096], &create_opts).await; let upload_id_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id); signaling.set_target(rustfs_lock::ObjectKey::new(RUSTFS_META_MULTIPART_BUCKET, upload_id_path.clone())); @@ -6746,6 +7293,76 @@ mod tests { .expect("abort should delete the upload after UploadPart releases the lock"); } + #[tokio::test(flavor = "multi_thread")] + #[serial] + async fn cancelled_put_object_part_keeps_upload_lock_through_commit() { + for pause in [ + MultipartCommitPause::PutPartAfterCapacityAdmission, + MultipartCommitPause::PutPartAfterRename, + ] { + let manager = Arc::new(rustfs_lock::GlobalLockManager::new()); + let signaling = Arc::new(SignalingLockClient::new(Arc::new(LocalClient::with_manager(manager)))); + let lockers: Vec> = vec![signaling.clone()]; + let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await; + let bucket = "multipart-cancelled-part-commit-bucket"; + let object = "object"; + make_bucket_on_all(&disk_stores, bucket).await; + let upload = set_disks + .new_multipart_upload(bucket, object, &ObjectOptions::default()) + .await + .expect("multipart upload should be created"); + let upload_id = upload.upload_id; + let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id); + signaling.set_target(rustfs_lock::ObjectKey::new(RUSTFS_META_MULTIPART_BUCKET, upload_path.clone())); + let _setup_type_guard = SetupTypeGuard::switch_to(&set_disks.ctx, SetupType::DistErasure).await; + let barrier = MultipartCommitBarrier::install(bucket, object, pause); + + let put_store = set_disks.clone(); + let put_upload_id = upload_id.clone(); + let put = tokio::spawn(async move { + let mut reader = PutObjReader::from_vec(vec![0x45; 4096]); + put_store + .put_object_part(bucket, object, &put_upload_id, 1, &mut reader, &ObjectOptions::default()) + .await + }); + barrier.wait_until_paused().await; + put.abort(); + assert!( + put.await + .expect_err("the request waiter should remain cancellable") + .is_cancelled() + ); + + let abort_store = set_disks.clone(); + let abort_upload_id = upload_id.clone(); + let mut abort = tokio::spawn(async move { + abort_store + .abort_multipart_upload(bucket, object, &abort_upload_id, &ObjectOptions::default()) + .await + }); + signaling.wait_for_attempts(2).await; + assert!( + tokio::time::timeout(Duration::from_millis(100), &mut abort).await.is_err(), + "Abort must not acquire the upload lock while a cancelled request's commit is paused at {pause:?}" + ); + barrier.release(); + tokio::time::timeout(Duration::from_secs(10), abort) + .await + .expect("Abort should finish after the commit drains") + .expect("Abort task should not panic") + .expect("Abort should delete the committed part staging"); + for disk in &disk_stores { + assert!( + matches!( + disk.list_dir(bucket, RUSTFS_META_MULTIPART_BUCKET, &upload_path, -1).await, + Err(DiskError::FileNotFound) + ), + "the upload directory must be absent after the commit and Abort finish" + ); + } + } + } + #[tokio::test(flavor = "multi_thread")] #[serial] async fn put_object_part_different_part_numbers_commit_concurrently() { @@ -10261,6 +10878,18 @@ mod tests { "the post-commit crash must leave the upload listable for reclamation" ); + let cleanup = set_disks + .lock_orphan_multipart_cleanup(bucket, object, &u_new, &ObjectOptions::default(), &[]) + .await + .expect("the committed upload has no staging metadata on any disk"); + assert!( + cleanup + .delete(bucket, object, &ObjectOptions::default()) + .await + .expect("recover completed upload staging") + ); + drop(cleanup); + // The commit rename already consumed the upload's xl.meta, so the // retry cannot read staging. It must still return the committed // object: AWS CompleteMultipartUpload is idempotent for the same @@ -10279,21 +10908,6 @@ mod tests { !upload_is_listed(&set_disks, bucket, object, &u_new).await, "the idempotent retry must reclaim staging left behind by the crash" ); - - // Reclaim the leftover exactly as the production tail does: delete_all - // on the upload path (abort_multipart_upload cannot — the upload's - // metadata is gone, so it too answers InvalidUploadID). - let upload_dir = SetDisks::get_upload_id_dir(bucket, object, &u_new); - set_disks - .delete_all(RUSTFS_META_MULTIPART_BUCKET, &upload_dir) - .await - .expect("sweeping the leftover upload staging should succeed"); - assert!( - !upload_is_listed(&set_disks, bucket, object, &u_new).await, - "reclaiming the upload must drop it from the listing" - ); - let (body_after, _) = read_object(&set_disks, bucket, object).await; - assert_eq!(body_after, new, "reclaiming the leftover upload must not disturb the committed object"); } #[tokio::test] @@ -10340,6 +10954,18 @@ mod tests { .expect("the failed post-commit completion should release its object guard"), ); + let cleanup = set_disks + .lock_orphan_multipart_cleanup(bucket, object, &u_new, &ObjectOptions::default(), &[]) + .await + .expect("committed staging has no upload metadata"); + assert!( + cleanup + .delete(bucket, object, &ObjectOptions::default()) + .await + .expect("orphan recovery must only remove upload staging") + ); + drop(cleanup); + let (body, _) = read_object(&set_disks, bucket, object).await; assert_eq!(body, new, "the committed replacement must remain readable after the crash"); let mut receipts = 0; diff --git a/crates/ecstore/src/set_disk/ops/object.rs b/crates/ecstore/src/set_disk/ops/object.rs index e84c08524..bb39dc070 100644 --- a/crates/ecstore/src/set_disk/ops/object.rs +++ b/crates/ecstore/src/set_disk/ops/object.rs @@ -20,6 +20,7 @@ //! SetDisks core (io_primitives) via inherent calls. use crate::core::pools::DecommissionCapacityAdmission; +use crate::disk::cleanup_runtime::{self, OLD_DATA_CLEANUP_RECEIPT_FILE}; use rustfs_common::mrf_channel::MrfDeleteMarkerPurge; use rustfs_filemeta::metadata_keys; @@ -36,9 +37,9 @@ use super::super::{ GET_STAGE_LOCK_ACQUIRE, GET_STAGE_METADATA, GET_STAGE_OBJECT_INFO, GET_STAGE_PATH_DECISION, GET_STAGE_READER_SETUP, GenericError, GetCodecStreamingDecision, GetCodecStreamingFallbackReason, GetDirectMemoryDecision, GetObjectReader, HTTPRangeSpec, HashAlgorithm, HashMap, HashReader, HashSet, HeaderMap, HealChannelPriority, InstanceContext, Instant, - LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, OBJECT_OP_IGNORED_ERRS, ObjectApiError, ObjectInfo, ObjectKey, - ObjectLockConfigSnapshot, ObjectLockConfigState, ObjectOptions, ObjectReader, ObjectToDelete, OffsetDateTime, Ordering, Pin, - PutObjReader, RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET, ReadPathPlan, ReaderImpl, ReplicateDecision, + LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, LateMetadataFanout, OBJECT_OP_IGNORED_ERRS, ObjectApiError, ObjectInfo, + ObjectKey, ObjectLockConfigSnapshot, ObjectLockConfigState, ObjectOptions, ObjectReader, ObjectToDelete, OffsetDateTime, + Ordering, Pin, PutObjReader, RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET, ReadPathPlan, ReaderImpl, ReplicateDecision, ReplicationObjectBridge, Result, SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS, SLASH_SEPARATOR, SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_RESTORE_OPERATION_ID, SUFFIX_RESTORE_WORKER_LOCK, SetDisks, SmallWritePath, StorageError, TRANSITION_COMPLETE, UpdateMetadataOpts, Uuid, WriteLayout, adaptive_duplex_buffer_size, @@ -64,6 +65,420 @@ use super::super::{ take_prepared_get_object_metadata, to_object_err, try_read_inline_data_shards_direct, warn, }; use super::bitrot_self_verify::{BitrotSelfVerifyTarget, drop_failed_writer_disks, verify_written_bitrot_shards}; +use super::delete_recovery::DeleteRecoveryContext; + +#[derive(Default)] +struct DeleteCommitContext<'a> { + incarnation: Option, + options: Option<&'a ObjectOptions>, + guard: Option>, +} + +impl<'a> DeleteCommitContext<'a> { + fn request(options: &'a ObjectOptions, guard: Option<&Arc>) -> Self { + Self { + incarnation: options.expected_bucket_incarnation_id, + options: Some(options), + guard: guard.cloned(), + } + } + + fn can_commit(&self, bucket: &str, object: &str) -> bool { + self.options.is_none_or(|options| { + ensure_delete_commit_locks_held(self.guard.as_deref(), bucket, object, options).is_ok() + && options + .scanner_publication_commit_scope + .as_ref() + .is_none_or(|scope| scope.can_commit()) + }) + } +} + +const TIER_FREE_VERSION_SOURCE_LOOKUP_COUNTERFACTUAL_SKIP_ENV: &str = + "RUSTFS_PUT_TIER_FREE_VERSION_SOURCE_LOOKUP_COUNTERFACTUAL_SKIP"; +const TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA_ENV: &str = "RUSTFS_PUT_TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA"; +const TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA_ASSUME_HOMOGENEOUS_ENV: &str = + "RUSTFS_PUT_TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA_ASSUME_HOMOGENEOUS"; +const RENAME_TAIL_CLEANUP_PER_DISK_LIMIT_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_PER_DISK_LIMIT"; +const RENAME_TAIL_CLEANUP_PER_DISK_LANES_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_PER_DISK_LANES"; +const RENAME_TAIL_CLEANUP_LANE_ENABLE_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_ENABLE"; +const RENAME_TAIL_CLEANUP_LANE_CAPACITY_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_CAPACITY"; +const RENAME_TAIL_CLEANUP_LANE_BATCH_MAX_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_BATCH_MAX"; +const RENAME_TAIL_CLEANUP_LANE_BATCH_WINDOW_MS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_BATCH_WINDOW_MS"; +const RENAME_TAIL_CLEANUP_LANE_ACTIVE_WINDOW_MS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_ACTIVE_WINDOW_MS"; +const RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP"; +const DEFAULT_RENAME_TAIL_CLEANUP_PER_DISK_LANES: usize = 1024; +const DEFAULT_RENAME_TAIL_CLEANUP_LANE_CAPACITY: usize = 1024; +const DEFAULT_RENAME_TAIL_CLEANUP_LANE_BATCH_MAX: usize = 1; + +fn tier_free_version_source_lookup_counterfactual_skip_enabled() -> bool { + static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(TIER_FREE_VERSION_SOURCE_LOOKUP_COUNTERFACTUAL_SKIP_ENV) + .ok() + .as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_tail_cleanup_zero_target_tmp_delete_skip_enabled() -> bool { + static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP_ENV) + .ok() + .as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn tier_free_version_source_reuse_rename_data_enabled() -> bool { + static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA_ENV).ok().as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn tier_free_version_source_reuse_rename_data_assume_homogeneous_enabled() -> bool { + static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA_ASSUME_HOMOGENEOUS_ENV) + .ok() + .as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_tail_cleanup_per_disk_limit() -> usize { + static LIMIT: std::sync::OnceLock = std::sync::OnceLock::new(); + *LIMIT.get_or_init(|| rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_PER_DISK_LIMIT_ENV, 0)) +} + +fn rename_tail_cleanup_per_disk_lanes() -> usize { + static LANES: std::sync::OnceLock = std::sync::OnceLock::new(); + *LANES.get_or_init(|| { + rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_PER_DISK_LANES_ENV, DEFAULT_RENAME_TAIL_CLEANUP_PER_DISK_LANES).max(1) + }) +} + +fn rename_tail_cleanup_foreground_yield() -> Duration { + static YIELD: std::sync::OnceLock = std::sync::OnceLock::new(); + *YIELD + .get_or_init(|| Duration::from_millis(rustfs_utils::get_env_u64("RUSTFS_PUT_RENAME_TAIL_CLEANUP_FOREGROUND_YIELD_MS", 0))) +} + +fn rename_tail_cleanup_foreground_active_yield() -> Duration { + static YIELD: std::sync::OnceLock = std::sync::OnceLock::new(); + *YIELD.get_or_init(|| { + Duration::from_millis(rustfs_utils::get_env_u64("RUSTFS_PUT_RENAME_TAIL_CLEANUP_FOREGROUND_ACTIVE_YIELD_MS", 0)) + }) +} + +fn rename_tail_cleanup_lane_enabled() -> bool { + static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(RENAME_TAIL_CLEANUP_LANE_ENABLE_ENV).ok().as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_tail_cleanup_lane_capacity() -> usize { + static CAPACITY: std::sync::OnceLock = std::sync::OnceLock::new(); + *CAPACITY.get_or_init(|| { + rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_LANE_CAPACITY_ENV, DEFAULT_RENAME_TAIL_CLEANUP_LANE_CAPACITY).max(1) + }) +} + +fn rename_tail_cleanup_lane_batch_max() -> usize { + static BATCH_MAX: std::sync::OnceLock = std::sync::OnceLock::new(); + *BATCH_MAX.get_or_init(|| { + rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_LANE_BATCH_MAX_ENV, DEFAULT_RENAME_TAIL_CLEANUP_LANE_BATCH_MAX).max(1) + }) +} + +fn rename_tail_cleanup_lane_batch_window() -> Duration { + static WINDOW: std::sync::OnceLock = std::sync::OnceLock::new(); + *WINDOW.get_or_init(|| Duration::from_millis(rustfs_utils::get_env_u64(RENAME_TAIL_CLEANUP_LANE_BATCH_WINDOW_MS_ENV, 0))) +} + +fn rename_tail_cleanup_lane_active_window() -> Duration { + static WINDOW: std::sync::OnceLock = std::sync::OnceLock::new(); + *WINDOW.get_or_init(|| Duration::from_millis(rustfs_utils::get_env_u64(RENAME_TAIL_CLEANUP_LANE_ACTIVE_WINDOW_MS_ENV, 0))) +} + +async fn rename_tail_cleanup_per_disk_permit(disk_index: usize) -> Option { + let limit = rename_tail_cleanup_per_disk_limit(); + if limit == 0 { + return None; + } + static LANES: std::sync::OnceLock>> = std::sync::OnceLock::new(); + let lanes = LANES.get_or_init(|| { + (0..rename_tail_cleanup_per_disk_lanes()) + .map(|_| Arc::new(tokio::sync::Semaphore::new(limit))) + .collect() + }); + let semaphore = Arc::clone(&lanes[disk_index % lanes.len()]); + let wait_started = rustfs_io_metrics::put_stage_timer(); + let permit = semaphore.clone().acquire_owned().await.ok(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_PER_DISK_PERMIT_WAIT, + wait_started, + ); + if let Some(semaphore_permit) = permit { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_PER_DISK_INFLIGHT, + limit.saturating_sub(semaphore.available_permits()) as f64, + ); + Some(semaphore_permit) + } else { + None + } +} + +type RenameTailCleanupLaneFuture = Pin + Send + 'static>>; + +struct RenameTailCleanupLaneJob { + disk_index: usize, + future: RenameTailCleanupLaneFuture, + done: tokio::sync::oneshot::Sender<()>, +} + +struct RenameTailCleanupLane { + sender: tokio::sync::mpsc::Sender, + queued: Arc, + oldest_enqueued_at: Arc>>, +} + +struct RenameTailCleanupLaneQueue { + lanes: Vec, +} + +async fn run_rename_tail_cleanup_lane_job(job: RenameTailCleanupLaneJob) { + let service_started = rustfs_io_metrics::put_stage_timer(); + job.future.await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_SERVICE_TIME, + service_started, + ); + let _ = job.done.send(()); +} + +impl RenameTailCleanupLaneQueue { + fn new() -> Self { + let lanes = (0..rename_tail_cleanup_per_disk_lanes()) + .map(|_| { + let (sender, mut receiver) = + tokio::sync::mpsc::channel::(rename_tail_cleanup_lane_capacity()); + let queued = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let active_workers = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let oldest_enqueued_at = Arc::new(tokio::sync::Mutex::new(None::)); + let worker_queued = Arc::clone(&queued); + let worker_active = Arc::clone(&active_workers); + let worker_oldest_enqueued_at = Arc::clone(&oldest_enqueued_at); + tokio::spawn(async move { + loop { + let queued_before_poll = worker_queued.load(Ordering::Relaxed); + let worker_poll_started = rustfs_io_metrics::put_stage_timer(); + let Some(job) = receiver.recv().await else { + return; + }; + if queued_before_poll == 0 { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_IDLE_WAIT, + worker_poll_started, + ); + } else { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_POLL_WAIT, + worker_poll_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_JOB_WAIT, + worker_poll_started, + ); + } + let mut batch = vec![job]; + let batch_window = rename_tail_cleanup_lane_batch_window(); + let batch_max = rename_tail_cleanup_lane_batch_max(); + if batch_max > 1 && !batch_window.is_zero() { + tokio::time::sleep(batch_window).await; + } + while batch.len() < batch_max { + let Some(next) = receiver.try_recv().ok() else { + break; + }; + batch.push(next); + } + let queued_after_dequeue = worker_queued + .fetch_sub(batch.len(), Ordering::Relaxed) + .saturating_sub(batch.len()); + if queued_after_dequeue == 0 { + *worker_oldest_enqueued_at.lock().await = None; + } + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH, + queued_after_dequeue as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH_MAX_SAMPLE, + queued_after_dequeue as f64, + ); + if let Some(oldest) = *worker_oldest_enqueued_at.lock().await { + let oldest_age_ms = oldest.elapsed().as_secs_f64() * 1000.0; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE, + oldest_age_ms, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE_MAX_SAMPLE, + oldest_age_ms, + ); + } + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_BATCH_SIZE, + batch.len() as f64, + ); + if batch.len() > 1 { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_BURST_BATCH_SIZE, + batch.len() as f64, + ); + } + let active = worker_active.fetch_add(1, Ordering::Relaxed) + 1; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_ACTIVE, + active as f64, + ); + let active_window = rename_tail_cleanup_lane_active_window(); + if !active_window.is_zero() { + let mut deferred = Vec::new(); + let mut ready = Vec::new(); + for job in batch { + let active = rename_foreground_active_count(job.disk_index); + if active > 0 { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_FOREGROUND_ACTIVE_HIT, + active as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_REQUEUE, + 1.0, + ); + deferred.push(job); + } else { + ready.push(job); + } + } + for job in ready { + run_rename_tail_cleanup_lane_job(job).await; + } + if !deferred.is_empty() { + tokio::time::sleep(active_window).await; + for job in deferred { + run_rename_tail_cleanup_lane_job(job).await; + } + } + } else { + for job in batch { + run_rename_tail_cleanup_lane_job(job).await; + } + } + worker_active.fetch_sub(1, Ordering::Relaxed); + } + }); + RenameTailCleanupLane { + sender, + queued, + oldest_enqueued_at, + } + }) + .collect(); + Self { lanes } + } + + async fn enqueue( + &self, + disk_index: usize, + future: RenameTailCleanupLaneFuture, + ) -> Option> { + let lane = &self.lanes[disk_index % self.lanes.len()]; + let enqueued_at = Instant::now(); + let queue_push_started = rustfs_io_metrics::put_stage_timer(); + { + let mut oldest = lane.oldest_enqueued_at.lock().await; + if oldest.is_none() { + *oldest = Some(enqueued_at); + } + } + let queued_after_enqueue = lane.queued.fetch_add(1, Ordering::Relaxed) + 1; + let (done, done_rx) = tokio::sync::oneshot::channel(); + let job = RenameTailCleanupLaneJob { + disk_index, + future, + done, + }; + match lane.sender.try_send(job) { + Ok(()) => { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH, + queued_after_enqueue as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH_MAX_SAMPLE, + queued_after_enqueue as f64, + ); + if let Some(oldest) = *lane.oldest_enqueued_at.lock().await { + let oldest_age_ms = oldest.elapsed().as_secs_f64() * 1000.0; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE, + oldest_age_ms, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE_MAX_SAMPLE, + oldest_age_ms, + ); + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_PUSH, + queue_push_started, + ); + Some(done_rx) + } + Err(err) => { + lane.queued.fetch_sub(1, Ordering::Relaxed); + if lane.queued.load(Ordering::Relaxed) == 0 { + *lane.oldest_enqueued_at.lock().await = None; + } + let job = err.into_inner(); + run_rename_tail_cleanup_lane_job(job).await; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH_MAX_SAMPLE, + lane.queued.load(Ordering::Relaxed) as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_PUSH, + queue_push_started, + ); + None + } + } + } +} + +fn rename_tail_cleanup_lane_queue() -> &'static RenameTailCleanupLaneQueue { + static QUEUE: std::sync::OnceLock = std::sync::OnceLock::new(); + QUEUE.get_or_init(RenameTailCleanupLaneQueue::new) +} fn delete_marker_purge_candidate( file_info: &FileInfo, @@ -330,7 +745,9 @@ use crate::object_api::{ }; use crate::services::notification_sys::RemoteVersionStateFleetProofToken; use crate::services::tier::tier::{TierConfigMgr, TierDestinationId, TierOperationLease, tier_destination_id_from_metadata}; -use crate::set_disk::core::io_primitives::{RenameRollbackReceipt, RenameTailCleanup, finish_rename_tail_heal}; +use crate::set_disk::core::io_primitives::{ + RenameRollbackReceipt, RenameTailCleanup, finish_rename_tail_heal, rename_foreground_active_count, +}; #[cfg(test)] use crate::storage_api_contracts::namespace::NamespaceLocking; #[cfg(test)] @@ -338,7 +755,7 @@ use crate::storage_api_contracts::object::HTTPPreconditions; use crate::store::ECStore; use crate::store::utils::clean_metadata; use crate::store::{RemoteTuplePublicationCommitGuard, RemoteTuplePublicationFence}; -use futures::FutureExt as _; +use futures::{FutureExt as _, StreamExt as _}; use http::HeaderValue; #[cfg(test)] use rustfs_filemeta::FileMeta; @@ -347,6 +764,8 @@ use rustfs_filemeta::ObjectPartInfo; use rustfs_utils::path::decode_dir_object; #[cfg(test)] use rustfs_utils::path::encode_dir_object; +use rustfs_utils::path::object_name_disk_identity; +use std::borrow::Cow; use std::future::Future; use std::task::{Context, Poll}; use std::time::Duration; @@ -500,7 +919,6 @@ async fn acquire_single_tier_delete_lease(opts: &ObjectOptions, source: &ObjectI Ok(Some(lease)) } -const OLD_DATA_CLEANUP_RECEIPT_FILE: &str = ".rustfs-old-data-cleanup-receipt.json"; const SCANNER_PUBLICATION_LEASE_FENCE_MAX_BYTES: usize = 64 * 1024; const SCANNER_PUBLICATION_LEASE_FENCE_MAX_ENTRIES: usize = 256; @@ -981,6 +1399,42 @@ mod tier_free_version_receipt_eligibility_tests { } } +// Own only completed staging, before the commit owner can rename or roll back +// shards. Borrow the request state; clone it only if cancellation needs cleanup. +struct PutObjectStagingCleanup<'a> { + set: &'a SetDisks, + tmp_dir: &'a str, + reservation: &'a Option, + runtime: Option, +} + +impl Drop for PutObjectStagingCleanup<'_> { + fn drop(&mut self) { + let Some(runtime) = self.runtime.take() else { + return; + }; + let set = self.set.clone(); + let tmp_dir = self.tmp_dir.to_owned(); + let reservation = self.reservation.clone(); + // A request future may be dropped outside its polling runtime. Cleanup + // remains best-effort on runtime shutdown, like the existing PUT tail. + let _enter = runtime.enter(); + cleanup_runtime::spawn(reservation, async move { + if let Err(err) = set.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_dir).await { + warn!( + event = EVENT_SET_DISK_WRITE, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_SET_DISK, + state = "cancelled_staging_cleanup_failed", + tmp_dir = %tmp_dir, + error = ?err, + "Cancelled PUT staging cleanup failed" + ); + } + }); + } +} + struct PutObjectCommitCancellation { token: CancellationToken, armed: bool, @@ -2481,6 +2935,22 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks { return Err(to_object_err(Error::MethodNotAllowed, vec![bucket, object])); } + let condition_terminal = + crate::object_api::get_object_read_condition_is_terminal(crate::object_api::GetObjectReadMetadata { + bucket, + object, + etag: object_info.etag.as_deref(), + mod_time: object_info.mod_time, + parts: &object_info.parts, + delete_marker: false, + }); + if snapshot.condition_stopped && !condition_terminal { + return Err(Error::other("conditional metadata snapshot no longer satisfies the read condition")); + } + if condition_terminal { + return Ok(GetObjectReader::condition_stopped(object_info)); + } + // if object_info.size == 0 { // let empty_rd: Box = Box::new(Bytes::new()); @@ -2580,6 +3050,8 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks { // Decode directly let decode_stage_start = rustfs_io_metrics::get_stage_metrics_enabled().then(Instant::now); + #[cfg(test)] + crate::set_disk::disk_call_counters::record(object, crate::set_disk::disk_call_counters::KIND_BODY_DECODE, 0); if let Some(body) = try_read_inline_data_shards_direct(&mut readers, data_shards, read_length, object_size).await { if let Some(decode_stage_start) = decode_stage_start { @@ -2649,6 +3121,8 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks { let decode_stage_start = rustfs_io_metrics::get_stage_metrics_enabled().then(Instant::now); let mut output = Cursor::new(Vec::with_capacity(object_size)); + #[cfg(test)] + crate::set_disk::disk_call_counters::record(object, crate::set_disk::disk_call_counters::KIND_BODY_DECODE, 0); let (written, err) = erasure.decode(&mut output, readers, 0, object_size, object_size).await; if let Some(e) = err { return Err(to_object_err(e.into(), vec![bucket, object])); @@ -2761,9 +3235,9 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks { let object_size = usize::try_from(object_info.size) .map_err(|_| to_object_err(Error::other("two-phase GET object size is invalid"), vec![bucket, object]))?; let mut output = Vec::with_capacity(object_size); - let (fi, files, disks, late_metadata_fanout_disks) = snapshot.into_owned_with_late_metadata_fanout(); + let (fi, files, disks, late_metadata_fanout) = snapshot.into_owned_with_late_metadata_fanout(); let expected_identity = super::super::read::LateMetadataIdentity::from_file_info(&fi); - let late_metadata_fanout_disks = late_metadata_fanout_disks.ok_or_else(|| { + let late_metadata_fanout = late_metadata_fanout.ok_or_else(|| { to_object_err(Error::other("two-phase GET fallback context is missing"), vec![bucket, object]) })?; let initial_result = Self::get_object_with_fileinfo( @@ -2788,6 +3262,10 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks { ) .await; if prepare_late_materialized_retry(&initial_result, &mut output, object_size) { + let late_metadata_fanout_disks = match late_metadata_fanout { + LateMetadataFanout::CapturedDisks(disks) => disks, + LateMetadataFanout::RefreshCurrentSet => self.get_disks_for_data_read().await, + }; #[cfg(test)] super::super::rename_fanout_barrier::checkpoint( object, @@ -3307,7 +3785,96 @@ pub(in crate::set_disk) fn merge_replication_metadata_lww( changed } +struct CommittedPutOutcome { + // Namespace publication succeeded even if its writer must retain repair + // responsibility. Completion errors must not select rollback cleanup. + completion: Result<(ObjectInfo, Option)>, +} + impl SetDisks { + /// Failure cleanup stays awaited, but shares the same execution budget as + /// successful PUT continuations when isolation is enabled. + async fn delete_put_tmp_with_budget(&self, tmp_dir: &str, reservation: &Option) -> Result<()> { + if reservation.is_none() { + return self.delete_all(RUSTFS_META_TMP_BUCKET, tmp_dir).await; + } + let set = self.clone(); + let tmp_dir = tmp_dir.to_owned(); + cleanup_runtime::spawn(reservation.clone(), async move { set.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_dir).await }) + .await + .map_err(Error::other)? + } + + async fn reclaim_missing_delete_residue( + &self, + bucket: &str, + object: &str, + opts: &ObjectOptions, + lock_guard: Option<&Arc>, + ) -> Result { + if is_meta_bucketname(bucket) + || opts.version_id.is_some() + || opts.versioned + || opts.version_suspended + || opts.http_preconditions.is_some() + || opts.data_movement + || opts.transition.expire_restored + || opts.replication_request + || opts.lifecycle_delete_all.is_some() + || opts.delete_prefix + || opts.expected_current_version_id.is_some() + || !metadata_sys::has_authoritative_never_versioned_state_in(&self.ctx, bucket) + .await + .unwrap_or(false) + { + return Ok(false); + } + let lock_state = match opts.object_lock_config_snapshot.as_ref() { + Some(snapshot) => Some(snapshot.state().clone()), + None => metadata_sys::get_object_lock_config_state_in(&self.ctx, bucket).await.ok(), + }; + if !matches!(lock_state, Some(ObjectLockConfigState::ConfirmedAbsent)) { + return Ok(false); + } + if let Some(expected) = opts.expected_bucket_incarnation_id { + self.validate_bucket_incarnation(bucket, expected, opts).await?; + } + ensure_delete_commit_locks_held(lock_guard.map(Arc::as_ref), bucket, object, opts)?; + begin_scanner_publication_delete_mutation(opts.scanner_publication_commit_scope.as_ref())?; + let result = self + .recover_delete_key( + bucket, + object, + DeleteRecoveryContext { + explicit_delete: true, + namespace_owner: Some(Arc::new(( + self.ctx.begin_namespace_commit(), + lock_guard.cloned(), + opts.object_lock_config_snapshot.clone(), + opts.scanner_publication_commit_scope.clone(), + opts.write_lock_context.clone(), + opts.delete_mutation_owner.clone(), + ))), + ..DeleteRecoveryContext::new(opts.expected_bucket_incarnation_id, &|| { + ensure_delete_commit_locks_held(lock_guard.map(Arc::as_ref), bucket, object, opts).is_ok() + && opts + .scanner_publication_commit_scope + .as_ref() + .is_none_or(|scope| scope.can_commit()) + }) + }, + ) + .await; + ensure_delete_commit_locks_held(lock_guard.map(Arc::as_ref), bucket, object, opts)?; + let recovered = result.map_err(|err| to_object_err(err.into(), vec![bucket, object]))?; + if recovered.pending > 0 && !recovered.no_transactions_at_quorum { + return Err(StorageError::SlowDown); + } + if recovered.blocked > 0 { + return Err(StorageError::SlowDown); + } + Ok(recovered.reclaimed > 0 || recovered.committed > 0) + } pub(in crate::set_disk) async fn cleanup_rename_tail( &self, targets: Vec, @@ -3316,26 +3883,107 @@ impl SetDisks { committed_data_dir: Option, epoch: Option, ) { + if rename_tail_cleanup_lane_enabled() { + let bucket = bucket.to_string(); + let object = object.to_string(); + let mut completions = Vec::with_capacity(targets.len()); + for target in targets { + let cleanup_set = self.clone(); + let cleanup_bucket = bucket.clone(); + let cleanup_object = object.clone(); + let disk_index = target.disk_index; + let future = Box::pin(async move { + cleanup_set + .cleanup_rename_tail_target(target, &cleanup_bucket, &cleanup_object, committed_data_dir, epoch) + .await; + }); + if let Some(done) = rename_tail_cleanup_lane_queue().enqueue(disk_index, future).await { + completions.push(done); + } + } + for done in completions { + let _ = done.await; + } + return; + } + for target in targets { - let mut disks = vec![None; self.set_drive_count]; - disks[target.disk_index] = Some(target.disk); - self.persist_old_data_cleanup_receipts(&disks, bucket, object, target.old_data_dir, committed_data_dir, epoch) - .await; - let cleanup = self - .commit_rename_data_dir_and_mark_capacity( - &disks, - bucket, - object, - &target.old_data_dir.to_string(), - &committed_data_dir.unwrap_or_default().to_string(), - 1, - ) - .await; - self.report_old_data_dir_cleanup(bucket, object, &target.old_data_dir.to_string(), &cleanup) + self.cleanup_rename_tail_target(target, bucket, object, committed_data_dir, epoch) .await; } } + async fn cleanup_rename_tail_target( + &self, + target: RenameTailCleanup, + bucket: &str, + object: &str, + committed_data_dir: Option, + epoch: Option, + ) { + let foreground_yield = rename_tail_cleanup_foreground_yield(); + if !foreground_yield.is_zero() { + let yield_started = rustfs_io_metrics::put_stage_timer(); + tokio::time::sleep(foreground_yield).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_FOREGROUND_YIELD, + yield_started, + ); + } + let active_yield = rename_tail_cleanup_foreground_active_yield(); + if !active_yield.is_zero() { + let active = rename_foreground_active_count(target.disk_index); + if active > 0 { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFERRED_FOREGROUND_ACTIVE, + active as f64, + ); + let yield_started = rustfs_io_metrics::put_stage_timer(); + tokio::time::sleep(active_yield).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_FOREGROUND_YIELD, + yield_started, + ); + } + } + let _cleanup_per_disk_permit = rename_tail_cleanup_per_disk_permit(target.disk_index).await; + let mut disks = vec![None; self.set_drive_count]; + disks[target.disk_index] = Some(target.disk); + let receipt_started = rustfs_io_metrics::put_stage_timer(); + self.persist_old_data_cleanup_receipts(&disks, bucket, object, target.old_data_dir, committed_data_dir, epoch) + .await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_RECEIPT, + receipt_started, + ); + let commit_capacity_started = rustfs_io_metrics::put_stage_timer(); + let cleanup = self + .commit_rename_data_dir_and_mark_capacity( + &disks, + bucket, + object, + &target.old_data_dir.to_string(), + &committed_data_dir.unwrap_or_default().to_string(), + 1, + ) + .await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_COMMIT_CAPACITY, + commit_capacity_started, + ); + let report_started = rustfs_io_metrics::put_stage_timer(); + self.report_old_data_dir_cleanup(bucket, object, &target.old_data_dir.to_string(), &cleanup) + .await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_REPORT, + report_started, + ); + } + + fn needs_old_data_cleanup_receipt(old_data_dir: Uuid, committed_data_dir: Option, epoch: Option) -> bool { + epoch.is_some() && committed_data_dir != Some(old_data_dir) + } + pub(in crate::set_disk) async fn persist_old_data_cleanup_receipts( &self, disks: &[Option], @@ -3345,10 +3993,10 @@ impl SetDisks { committed_data_dir: Option, epoch: Option, ) { - let Some(epoch) = epoch else { return }; - if committed_data_dir == Some(old_data_dir) { + if !Self::needs_old_data_cleanup_receipt(old_data_dir, committed_data_dir, epoch) { return; } + let Some(epoch) = epoch else { return }; let receipt = OldDataCleanupReceipt::new(epoch, old_data_dir, committed_data_dir); let Ok(encoded) = receipt.encode() else { return; @@ -3445,7 +4093,28 @@ impl SetDisks { } async fn validate_bucket_incarnation(&self, bucket: &str, expected: Uuid, opts: &ObjectOptions) -> Result<()> { - let current = metadata_sys::get_bucket_incarnation_id_for_options_in(&self.ctx, bucket, opts).await?; + let mut metadata_opts = None; + if opts + .object_lock_config_snapshot + .as_ref() + .is_some_and(|snapshot| snapshot.is_unbound()) + { + let store = metadata_sys::object_store_in(&self.ctx).await?; + if opts + .bucket_incarnation_fence + .as_ref() + .is_some_and(|owner| owner.matches(&store, bucket, expected)) + { + // An unbound snapshot carries policy, not metadata ownership. + // Keep the actual lifecycle owner for this incarnation read. + let mut options = opts.clone(); + options.object_lock_config_snapshot = None; + metadata_opts = Some(options); + } + } + let current = + metadata_sys::get_bucket_incarnation_id_for_options_in(&self.ctx, bucket, metadata_opts.as_ref().unwrap_or(opts)) + .await?; if current != expected { return Err(StorageError::BucketNotFound(bucket.to_string())); } @@ -3537,6 +4206,23 @@ impl SetDisks { .scanner_publication_commit_scope .clone() .map(ScannerPublicationCommitScopeGuard::new); + // Reserve before staging. Some callers already hold external locks, + // so admission must not wait for another PUT to release its budget. + // Commit/cleanup owners retain clones across cancellation and early ACK. + if opts + .put_object_cancellation + .as_ref() + .is_some_and(CancellationToken::is_cancelled) + { + return Err(StorageError::OperationCanceled); + } + let cleanup_reservation = cleanup_runtime::reserve().map_err(|err| { + if err.kind() == std::io::ErrorKind::WouldBlock { + StorageError::SlowDown + } else { + Error::other(err) + } + })?; let storage_class_config = self.storage_class_config_snapshot(); self.invalidate_get_object_metadata_cache(bucket, object).await; @@ -3655,8 +4341,20 @@ impl SetDisks { (None, None, None) }; let mut tmp_cleanup_owned = false; + let mut staging_cleanup = PutObjectStagingCleanup { + set: self, + tmp_dir: &tmp_dir, + reservation: &cleanup_reservation, + runtime: None, + }; let rollback_receipt = RenameRollbackReceipt::default(); + let operation_first_poll_delay_started = rustfs_io_metrics::put_stage_timer(); let operation = async { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_OPERATION_FIRST_POLL_DELAY, + operation_first_poll_delay_started, + ); + let operation_body_to_commit_submit_started = rustfs_io_metrics::put_stage_timer(); let erasure = Arc::new(erasure_from_file_info(&fi, false)?); let put_object_size = known_put_object_storage_size(data.size()); @@ -4013,6 +4711,12 @@ impl SetDisks { drop(writers); // drop writers to close all files, this is to prevent FileAccessDenied errors when renaming data + // File creation, writes and integrity sidecars must finish before + // cancellation can remove staging without racing pending writers. + if !direct_inline_commit { + staging_cleanup.runtime = Some(tokio::runtime::Handle::current()); + } + if parity_blocks == 0 { let written_size = i64::try_from(w_size).map_err(|_| Error::other("put_object written size overflows i64"))?; let logical_shard_size = usize::try_from(erasure.shard_file_size(written_size)) @@ -4108,6 +4812,8 @@ impl SetDisks { } #[cfg(any(test, feature = "test-util"))] pause_put_object_commit(bucket, object, PutObjectCommitPause::AfterNamespace).await; + let namespace_lock_acquired_to_commit_ready_started = rustfs_io_metrics::put_stage_timer(); + let commit_ready_precondition_and_timestamp_started = rustfs_io_metrics::put_stage_timer(); if opts.http_preconditions.is_some() && let Some(err) = self.check_write_precondition(bucket, object, opts).await @@ -4128,8 +4834,13 @@ impl SetDisks { } } } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_PRECONDITION_AND_TIMESTAMP, + commit_ready_precondition_and_timestamp_started, + ); if let Some(expected) = opts.expected_current_version_id.as_deref() { + let expected_version_lookup_started = rustfs_io_metrics::put_stage_timer(); let current = self .get_object_info( bucket, @@ -4152,11 +4863,16 @@ impl SetDisks { if current.version_id.map(|version| version.to_string()).as_deref() != Some(expected) { return Err(StorageError::PreconditionFailed); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_EXPECTED_VERSION_LOOKUP, + expected_version_lookup_started, + ); } if let Some(version_id) = opts.version_id.as_deref() && !is_meta_bucketname(bucket) { + let explicit_version_lookup_started = rustfs_io_metrics::put_stage_timer(); let current = self .get_object_info( bucket, @@ -4209,13 +4925,23 @@ impl SetDisks { Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => {} Err(err) => return Err(err), } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_EXPLICIT_VERSION_LOOKUP, + explicit_version_lookup_started, + ); } + let restore_verify_started = rustfs_io_metrics::put_stage_timer(); self.require_current_restore_operation_id(bucket, object, opts, expected_restore_operation_id, "put_object_commit") .await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_RESTORE_VERIFY, + restore_verify_started, + ); // Fence every commit-time read before entering rename_data. Once // rename_data returns Ok the write is durable and must not be aborted. + let fence_verify_started = rustfs_io_metrics::put_stage_timer(); if object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) || publication_commit_guard .as_ref() @@ -4259,6 +4985,7 @@ impl SetDisks { }); } + let decommission_capacity_fence_started = rustfs_io_metrics::put_stage_timer(); if decommission_capacity_guard.is_none() && let Some(store) = opts.decommission_capacity_admission.as_ref() { @@ -4268,6 +4995,10 @@ impl SetDisks { .await?, ); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_DECOMMISSION_CAPACITY_FENCE, + decommission_capacity_fence_started, + ); // The object namespace is acquired above, after the input stream // has been fully staged. Only then admit the local publication @@ -4285,21 +5016,36 @@ impl SetDisks { achieved: 0, }); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_FENCE_VERIFY, + fence_verify_started, + ); let transaction_fencing_proof = object_transaction_fencing_fleet_proof(); if object_transaction_fencing_requested() && transaction_fencing_proof.is_none() { return Err(Error::other("object transaction fencing requires a live fleet capability proof")); } + let transaction_epoch_fence_read_started = rustfs_io_metrics::put_stage_timer(); let transaction_epoch_fence = if transaction_fencing_proof.is_some() { Some(read_object_transaction_epoch_fence(self, bucket, object).await?) } else { None }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_TRANSACTION_EPOCH_FENCE_READ, + transaction_epoch_fence_read_started, + ); + let quota_begin_started = rustfs_io_metrics::put_stage_timer(); let quota_context = reservation::begin(&self.ctx, bucket, object, opts, self.pool_index, self.set_index).await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_BEGIN, + quota_begin_started, + ); let quota_mutation_fence = quota_context.is_enforced() || opts.quota_admission.is_some(); let mut replication_quota_size = None; + let metadata_adjust_started = rustfs_io_metrics::put_stage_timer(); if opts.replication_request { if quota_context.is_enforced() && opts.preserve_ciphertext { return Err(Error::PartMissingOrCorrupt); @@ -4363,7 +5109,12 @@ impl SetDisks { } } } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_METADATA_ADJUST, + metadata_adjust_started, + ); + let quota_reserve_started = rustfs_io_metrics::put_stage_timer(); let (quota_old_size, quota_new_size) = if quota_context.is_enforced() { let new_size = match replication_quota_size { Some(size) => size, @@ -4381,8 +5132,13 @@ impl SetDisks { (0, 0) }; let quota_reservation = quota_context.reserve(quota_old_size, quota_new_size).await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_RESERVE, + quota_reserve_started, + ); let (commit_disks, quota_fence_tokens) = if quota_mutation_fence { - match Self::prepare_quota_mutation_fences(&shuffle_disks, bucket, object, write_quorum).await { + let quota_mutation_fence_prepare_started = rustfs_io_metrics::put_stage_timer(); + let prepared = match Self::prepare_quota_mutation_fences(&shuffle_disks, bucket, object, write_quorum).await { Ok((disks, tokens)) => { for (metadata, token) in parts_metadatas.iter_mut().zip(tokens.iter().copied()) { if let Some(token) = token { @@ -4399,10 +5155,16 @@ impl SetDisks { quota_reservation.abort().await; return Err(err); } - } + }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_MUTATION_FENCE_PREPARE, + quota_mutation_fence_prepare_started, + ); + prepared } else { (shuffle_disks.clone(), vec![None; shuffle_disks.len()]) }; + let commit_context_prepare_started = rustfs_io_metrics::put_stage_timer(); let transaction_epoch = transaction_epoch_fence.map(|_| assign_object_transaction_epoch(&commit_disks, &mut parts_metadatas)); @@ -4427,7 +5189,8 @@ impl SetDisks { || commit_object_lock_guard.is_some() || commit_decommission_object_lock_guard.is_some() || commit_publication_guard.is_some(); - let commit_allows_early_ack = write_context.allows_early_ack(opts.write_completion) + let commit_write_completion = opts.write_completion; + let commit_allows_early_ack = write_context.allows_early_ack(commit_write_completion) && !(opts.data_movement && opts.has_decommission_capacity_reservation()) && commit_owns_namespace_guard && commit_scanner_publication_scope.is_none(); @@ -4452,10 +5215,29 @@ impl SetDisks { let commit_tier_free_version_receipt_sink = opts.tier_free_version_receipt_sink.clone(); let commit_skip_free_version = opts.skip_free_version; let request_cancellation = operation_cancellation.clone(); + let commit_cleanup_reservation = cleanup_reservation.clone(); tmp_cleanup_owned = true; + // No await separates this handoff from starting the commit owner. + // Its rollback rules, not request Drop, now decide what to retain. + staging_cleanup.runtime = None; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_CONTEXT_PREPARE, + commit_context_prepare_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_ACQUIRED_TO_COMMIT_READY, + namespace_lock_acquired_to_commit_ready_started, + ); - let commit = move |cancellation: Option| async move { + let commit = move |commit_submit_to_closure_enter_started: Option, + cancellation: Option| async move { let write_context = write_context; + let commit_closure_total_started = rustfs_io_metrics::put_stage_timer(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_SUBMIT_TO_CLOSURE_ENTER, + commit_submit_to_closure_enter_started, + ); + let commit_closure_enter_to_first_await_started = rustfs_io_metrics::put_stage_timer(); let mut _object_lock_guard = commit_object_lock_guard; let mut _decommission_object_lock_guard = commit_decommission_object_lock_guard; let mut _publication_guard = commit_publication_guard; @@ -4465,14 +5247,21 @@ impl SetDisks { let mut _decommission_capacity_guard = commit_decommission_capacity_guard; let mut quota_reservation = quota_reservation; let rename_stage_start = Instant::now(); + let commit_before_rename_started = rustfs_io_metrics::put_stage_timer(); + let lock_held_pre_rename_prepare_started = rustfs_io_metrics::put_stage_timer(); let pre_rename = async { #[cfg(any(test, feature = "test-util"))] pause_put_object_commit(&commit_bucket, &commit_object, PutObjectCommitPause::AfterQuotaReservation).await; + let quota_mark_commit_started = rustfs_io_metrics::put_stage_timer(); quota_reservation.mark_commit_started().await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_QUOTA_MARK_COMMIT_STARTED, + quota_mark_commit_started, + ); #[cfg(any(test, feature = "test-util"))] pause_put_object_commit(&commit_bucket, &commit_object, PutObjectCommitPause::BeforeQuotaRename).await; + let first_fence_check_started = rustfs_io_metrics::put_stage_timer(); if quota_reservation.is_lock_lost() - || !quota_reservation.capability_proof_matches() || write_context.is_lock_lost() || _artifact_put_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) || _object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) @@ -4501,6 +5290,10 @@ impl SetDisks { achieved: 0, }); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_FIRST_FENCE_CHECK, + first_fence_check_started, + ); let restore_opts = ObjectOptions { version_id: commit_version_id.clone(), versioned: commit_versioned, @@ -4508,6 +5301,7 @@ impl SetDisks { no_lock: true, ..Default::default() }; + let restore_verify_started = rustfs_io_metrics::put_stage_timer(); commit_set .require_current_restore_operation_id( &commit_bucket, @@ -4517,6 +5311,10 @@ impl SetDisks { "put_object_quota_reservation", ) .await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_RESTORE_VERIFY, + restore_verify_started, + ); if let Some(proof) = transaction_fencing_proof.as_ref() && !object_transaction_fencing_fleet_proof_matches(proof) { @@ -4530,10 +5328,15 @@ impl SetDisks { PutObjectCommitPause::BeforeTransactionEpochVerify, ) .await; + let transaction_fence_verify_started = rustfs_io_metrics::put_stage_timer(); verify_object_transaction_epoch_fence(&commit_set, &commit_bucket, &commit_object, expected).await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_TRANSACTION_FENCE_VERIFY, + transaction_fence_verify_started, + ); } + let second_fence_check_started = rustfs_io_metrics::put_stage_timer(); if quota_reservation.is_lock_lost() - || !quota_reservation.capability_proof_matches() || write_context.is_lock_lost() || _artifact_put_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) || _object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) @@ -4562,11 +5365,20 @@ impl SetDisks { achieved: 0, }); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_SECOND_FENCE_CHECK, + second_fence_check_started, + ); Ok(()) }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_ENTER_TO_FIRST_AWAIT, + commit_closure_enter_to_first_await_started, + ); + let pre_rename_await_total_started = rustfs_io_metrics::put_stage_timer(); let mut pre_rename_result = if cancellation.is_some() || request_cancellation.is_some() { tokio::select! { - biased; + biased; _ = wait_for_put_object_commit_cancellation(cancellation.as_ref(), request_cancellation.as_ref()) => { Err(StorageError::OperationCanceled) }, @@ -4575,6 +5387,12 @@ impl SetDisks { } else { pre_rename.await }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_AWAIT_TOTAL, + pre_rename_await_total_started, + ); + let pre_rename_post_await_to_rename_started = rustfs_io_metrics::put_stage_timer(); + let scanner_scope_check_started = rustfs_io_metrics::put_stage_timer(); if pre_rename_result.is_ok() && let Some(scope) = commit_scanner_publication_scope.as_ref() && let Err(err) = scope.try_begin() @@ -4589,6 +5407,10 @@ impl SetDisks { let _ = scope.mark_indeterminate(); pre_rename_result = Err(StorageError::OperationCanceled); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_SCANNER_SCOPE_CHECK, + scanner_scope_check_started, + ); if let Err(err) = pre_rename_result { SetDisks::abort_quota_reservation_after_fence( quota_reservation, @@ -4600,7 +5422,10 @@ impl SetDisks { quota_mutation_fence, ) .await; - if let Err(cleanup_err) = commit_set.delete_all(RUSTFS_META_TMP_BUCKET, &commit_tmp_dir).await { + if let Err(cleanup_err) = commit_set + .delete_put_tmp_with_budget(&commit_tmp_dir, &commit_cleanup_reservation) + .await + { warn!(tmp_dir = %commit_tmp_dir, error = ?cleanup_err, "failed to cleanup put_object temporary data"); } else if issue3031_diag_enabled() { warn!( @@ -4613,43 +5438,76 @@ impl SetDisks { } return Err(err); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_PRE_RENAME_PREPARE, + lock_held_pre_rename_prepare_started, + ); - let put_tier_free_version_source = - if commit_put_tier_free_version_id.is_some() && commit_tier_free_version_receipt_sink.is_some() { - match commit_set - .get_object_info( - &commit_bucket, - &commit_object, - &ObjectOptions { - no_lock: true, - metadata_cache_safe: false, - versioned: commit_versioned, - version_suspended: commit_version_suspended, - ..Default::default() - }, - ) - .await - { - Ok(source) => Some(source), - Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => None, - Err(err) => { - debug!( - event = EVENT_LIFECYCLE_TRANSITIONED_DELETE_CLEANUP_OWNER, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_SET_DISK, - bucket = %commit_bucket, - object = %commit_object, - error = ?err, - "Skipped opportunistic tier free-version receipt source capture" - ); - None - } + let tier_free_version_source_lookup_started = rustfs_io_metrics::put_stage_timer(); + let reuse_rename_data_source_for_tier_free_version = tier_free_version_source_reuse_rename_data_enabled() + && (commit_set.tier_free_version_rename_data_source_capable() + || tier_free_version_source_reuse_rename_data_assume_homogeneous_enabled()); + let mut put_tier_free_version_source = if commit_put_tier_free_version_id.is_some() + && commit_tier_free_version_receipt_sink.is_some() + && !tier_free_version_source_lookup_counterfactual_skip_enabled() + && !reuse_rename_data_source_for_tier_free_version + { + #[cfg(feature = "diagnostic-read-version-caller")] + crate::cluster::rpc::runtime_sources::observe_read_version_context_boundary( + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::TierFreeVersionSourceLookup, + ); + match commit_set + .get_object_info( + &commit_bucket, + &commit_object, + &ObjectOptions { + no_lock: true, + metadata_cache_safe: false, + versioned: commit_versioned, + version_suspended: commit_version_suspended, + ..Default::default() + }, + ) + .await + { + Ok(source) => Some(source), + Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => None, + Err(err) => { + debug!( + event = EVENT_LIFECYCLE_TRANSITIONED_DELETE_CLEANUP_OWNER, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_SET_DISK, + bucket = %commit_bucket, + object = %commit_object, + error = ?err, + "Skipped opportunistic tier free-version receipt source capture" + ); + None } - } else { - None - }; + } + } else { + None + }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_TIER_FREE_VERSION_SOURCE_LOOKUP, + tier_free_version_source_lookup_started, + ); + let assign_rename_indexes_started = rustfs_io_metrics::put_stage_timer(); Self::assign_rename_data_indexes(&mut parts_metadatas); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_ASSIGN_RENAME_INDEXES, + assign_rename_indexes_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_POST_AWAIT_TO_RENAME, + pre_rename_post_await_to_rename_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_BEFORE_RENAME, + commit_before_rename_started, + ); + let lock_held_rename_started = rustfs_io_metrics::put_stage_timer(); let mut rename_result = SetDisks::rename_data_owned_with_fence( &commit_disks, (RUSTFS_META_TMP_BUCKET, commit_tmp_dir.as_str()), @@ -4676,6 +5534,12 @@ impl SetDisks { })), ) .await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_RENAME, + lock_held_rename_started, + ); + let commit_after_rename_to_return_started = rustfs_io_metrics::put_stage_timer(); + let lock_held_rename_result_to_tail_handoff_started = rustfs_io_metrics::put_stage_timer(); if let Some(scope) = commit_scanner_publication_scope.as_ref() { if rename_result.is_ok() { let _ = scope.mark_committed(); @@ -4690,7 +5554,15 @@ impl SetDisks { let mut rename_guard_release = None; let mut needs_immediate_heal = false; let mut tail_owns_tmp_cleanup = false; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_RENAME_RESULT_TO_TAIL_HANDOFF, + lock_held_rename_result_to_tail_handoff_started, + ); + let lock_held_tail_handoff_started = rustfs_io_metrics::put_stage_timer(); if let Ok(rename_commit) = rename_result.as_mut() { + commit_set.observe_tier_free_version_rename_data_source_capability( + rename_commit.tail_drain.is_none() && rename_commit.old_current_source_all_successful_checked, + ); commit_set.record_capacity_scope_if_needed(commit_capacity_scope_token, &rename_commit.capacity_disks); // Install the tail watcher before any post-commit await. The // latch keeps namespace guards through their prior handoff point. @@ -4727,6 +5599,7 @@ impl SetDisks { let fence_object = commit_object.clone(); let (guard_release_tx, guard_release_rx) = tokio::sync::oneshot::channel(); rename_guard_release = Some(guard_release_tx); + let cleanup_reservation = commit_cleanup_reservation.clone(); tokio::spawn(finish_rename_tail_heal( rename_tail_drain, guard_release_rx, @@ -4763,38 +5636,74 @@ impl SetDisks { decommission_object_lock_guard, decommission_capacity_guard, ), - targets| async move { + targets| { drop(object_lock_guard); drop(write_context); drop(publication_guard); - cleanup_set - .cleanup_rename_tail( - targets, - &cleanup_bucket, - &cleanup_object, - committed_data_dir, - transaction_epoch, - ) - .await; - if let Err(err) = cleanup_set.delete_all(RUSTFS_META_TMP_BUCKET, &cleanup_tmp_dir).await { - warn!(tmp_dir = %cleanup_tmp_dir, error = ?err, "failed to cleanup put_object temporary data"); - } else if issue3031_diag_enabled() { - warn!( - target: "rustfs_ecstore::set_disk", - tmp_dir = %cleanup_tmp_dir, - "issue3031_put_object_tmp_cleanup_done" + cleanup_runtime::run(cleanup_reservation, async move { + let zero_target_cleanup = targets.is_empty(); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_COUNT, + targets.len() as f64, ); - } - drop(decommission_object_lock_guard); - drop(decommission_capacity_guard); - drop(artifact_put_guard); - drop(bucket_lifecycle_guard); - drop(lifecycle_snapshot); + if zero_target_cleanup { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_ZERO_TARGET, + 1.0, + ); + } + cleanup_set + .cleanup_rename_tail( + targets, + &cleanup_bucket, + &cleanup_object, + committed_data_dir, + transaction_epoch, + ) + .await; + let tmp_delete_started = rustfs_io_metrics::put_stage_timer(); + if zero_target_cleanup && rename_tail_cleanup_zero_target_tmp_delete_skip_enabled() { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TMP_DELETE_COUNTERFACTUAL_SKIP, + 1.0, + ); + } else if let Err(err) = + cleanup_set.delete_all(RUSTFS_META_TMP_BUCKET, &cleanup_tmp_dir).await + { + warn!(tmp_dir = %cleanup_tmp_dir, error = ?err, "failed to cleanup put_object temporary data"); + } else if issue3031_diag_enabled() { + warn!( + target: "rustfs_ecstore::set_disk", + tmp_dir = %cleanup_tmp_dir, + "issue3031_put_object_tmp_cleanup_done" + ); + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TMP_DELETE, + tmp_delete_started, + ); + let guard_drop_started = rustfs_io_metrics::put_stage_timer(); + drop(decommission_object_lock_guard); + drop(decommission_capacity_guard); + drop(artifact_put_guard); + drop(bucket_lifecycle_guard); + drop(lifecycle_snapshot); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DECOMMISSION_GUARD_DROP, + guard_drop_started, + ); + }) + }, + move |request| async move { + let _ = heal_set.submit_rename_tail_heal(request, commit_write_completion).await; }, - |request| async move { heal_set.submit_rename_tail_heal(request).await }, )); } } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_TAIL_HANDOFF, + lock_held_tail_handoff_started, + ); if !tail_owns_tmp_cleanup { drop(_decommission_capacity_guard.take()); } @@ -4803,6 +5712,11 @@ impl SetDisks { pause_put_object_commit(&commit_bucket, &commit_object, PutObjectCommitPause::AfterRenameHandoff).await; } if quota_mutation_fence && !tail_owns_tmp_cleanup { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE_TRIGGERED, + 1.0, + ); + let quota_fence_release_started = rustfs_io_metrics::put_stage_timer(); let _ = SetDisks::release_quota_mutation_fences( &commit_disks, "a_fence_tokens, @@ -4811,9 +5725,23 @@ impl SetDisks { write_quorum, ) .await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE, + quota_fence_release_started, + ); + } else { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE_SKIPPED, + 1.0, + ); } if rename_result.is_ok() { + let quota_commit_started = rustfs_io_metrics::put_stage_timer(); quota_reservation.commit().await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_COMMIT, + quota_commit_started, + ); } let rename_commit = match rename_result { Ok(commit) => commit, @@ -4823,7 +5751,10 @@ impl SetDisks { // rollback backup for recovery; cleanup is unsafe. return Err(err.into()); } - if let Err(cleanup_err) = commit_set.delete_all(RUSTFS_META_TMP_BUCKET, &commit_tmp_dir).await { + if let Err(cleanup_err) = commit_set + .delete_put_tmp_with_budget(&commit_tmp_dir, &commit_cleanup_reservation) + .await + { warn!(tmp_dir = %commit_tmp_dir, error = ?cleanup_err, "failed to cleanup put_object temporary data"); } else if issue3031_diag_enabled() { warn!( @@ -4841,7 +5772,17 @@ impl SetDisks { let op_old_dir = rename_commit.data_dir; let cleanup_disks = rename_commit.cleanup_disks; let old_current_size = rename_commit.old_current_size; + if put_tier_free_version_source.is_none() + && reuse_rename_data_source_for_tier_free_version + && rename_commit.old_current_source_checked + { + put_tier_free_version_source = rename_commit + .old_current_source + .as_ref() + .map(|source| build_get_object_info(source, &commit_bucket, &commit_object, commit_versioned)); + } let mut fi = rename_commit.committed_file_info; + let lock_held_post_rename_started = rustfs_io_metrics::put_stage_timer(); if let (Some(source), Some(free_version_id)) = (put_tier_free_version_source.as_ref(), commit_put_tier_free_version_id) && transitioned_delete_publishes_free_version(source, &fi, commit_skip_free_version) @@ -4856,7 +5797,12 @@ impl SetDisks { ); } - if needs_immediate_heal { + let repair_result = if needs_immediate_heal { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT_TRIGGERED, + 1.0, + ); + let heal_submit_started = rustfs_io_metrics::put_stage_timer(); let mut request = rustfs_heal_contracts::heal_channel::create_heal_request_with_options( commit_bucket.clone(), Some(commit_object.clone()), @@ -4869,29 +5815,86 @@ impl SetDisks { .or_else(|| commit_version_suspended.then(Uuid::nil)) .map(|version_id| version_id.to_string()); request.expected_bucket_incarnation_id = source_bucket_incarnation_id; - commit_set.submit_rename_tail_heal(request).await; - } + let result = commit_set.submit_rename_tail_heal(request, commit_write_completion).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT, + heal_submit_started, + ); + result + } else { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT_SKIPPED, + 1.0, + ); + Ok(()) + }; let rename_stage_elapsed = rename_stage_start.elapsed(); let rename_stage_ms = rename_stage_elapsed.as_millis() as u64; - if let Some(old_dir) = op_old_dir { - commit_set - .persist_old_data_cleanup_receipts( - &cleanup_disks, - &commit_bucket, - &commit_object, - old_dir, - committed_data_dir, - transaction_epoch, - ) + if let Some(old_dir) = op_old_dir + && Self::needs_old_data_cleanup_receipt(old_dir, committed_data_dir, transaction_epoch) + { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT_TRIGGERED, + 1.0, + ); + let cleanup_receipt_started = rustfs_io_metrics::put_stage_timer(); + if commit_cleanup_reservation.is_none() { + commit_set + .persist_old_data_cleanup_receipts( + &cleanup_disks, + &commit_bucket, + &commit_object, + old_dir, + committed_data_dir, + transaction_epoch, + ) + .await; + } else { + let cleanup_set = commit_set.clone(); + let receipt_disks = cleanup_disks.clone(); + let receipt_bucket = commit_bucket.clone(); + let receipt_object = commit_object.clone(); + cleanup_runtime::run(commit_cleanup_reservation.clone(), async move { + cleanup_set + .persist_old_data_cleanup_receipts( + &receipt_disks, + &receipt_bucket, + &receipt_object, + old_dir, + committed_data_dir, + transaction_epoch, + ) + .await; + }) .await; + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT, + cleanup_receipt_started, + ); + } else { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT_SKIPPED, + 1.0, + ); } + let metadata_invalidate_started = rustfs_io_metrics::put_stage_timer(); commit_set .invalidate_get_object_metadata_cache(&commit_bucket, &commit_object) .await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_METADATA_INVALIDATE, + metadata_invalidate_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_POST_RENAME, + lock_held_post_rename_started, + ); + let lock_held_guard_release_started = rustfs_io_metrics::put_stage_timer(); if let Some(release) = rename_guard_release.take() { let _ = release.send(true); } @@ -4900,6 +5903,10 @@ impl SetDisks { drop(_object_lock_guard.take()); drop(_publication_guard.take()); drop(_bucket_lifecycle_guard.take()); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_GUARD_RELEASE, + lock_held_guard_release_started, + ); rustfs_io_metrics::record_put_object_stage_duration("set_disk_rename", duration_millis_f64(rename_stage_elapsed)); if (rename_stage_ms as u128) >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS { @@ -4927,16 +5934,41 @@ impl SetDisks { // here must not negate an already-committed, durable write, so we // deliberately do NOT `?`-propagate it into a 503. On residue the // report path emits the leak metric and enqueues a heal. - let cleanup = commit_set - .commit_rename_data_dir_and_mark_capacity( - &cleanup_disks, - &commit_bucket, - &commit_object, - &old_dir.to_string(), - &committed_dir, - write_quorum, - ) + if commit_cleanup_reservation.is_none() { + let cleanup = commit_set + .commit_rename_data_dir_and_mark_capacity( + &cleanup_disks, + &commit_bucket, + &commit_object, + &old_dir.to_string(), + &committed_dir, + write_quorum, + ) + .await; + commit_set + .report_old_data_dir_cleanup(&commit_bucket, &commit_object, &old_dir.to_string(), &cleanup) + .await; + } else { + let cleanup_set = commit_set.clone(); + let cleanup_bucket = commit_bucket.clone(); + let cleanup_object = commit_object.clone(); + cleanup_runtime::run(commit_cleanup_reservation.clone(), async move { + let cleanup = cleanup_set + .commit_rename_data_dir_and_mark_capacity( + &cleanup_disks, + &cleanup_bucket, + &cleanup_object, + &old_dir.to_string(), + &committed_dir, + write_quorum, + ) + .await; + cleanup_set + .report_old_data_dir_cleanup(&cleanup_bucket, &cleanup_object, &old_dir.to_string(), &cleanup) + .await; + }) .await; + } let cleanup_elapsed = cleanup_stage_start.elapsed(); let cleanup_ms = cleanup_elapsed.as_millis() as u64; cleanup_stage_ms = Some(cleanup_ms); @@ -4944,9 +5976,6 @@ impl SetDisks { "set_disk_old_data_cleanup", duration_millis_f64(cleanup_elapsed), ); - commit_set - .report_old_data_dir_cleanup(&commit_bucket, &commit_object, &old_dir.to_string(), &cleanup) - .await; if (cleanup_ms as u128) >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS { warn!( event = EVENT_SET_DISK_COMMIT_TAIL_SLOW, @@ -5008,6 +6037,7 @@ impl SetDisks { event = EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY, component = LOG_COMPONENT_ECSTORE, subsystem = LOG_SUBSYSTEM_SET_DISK, + result = if repair_result.is_ok() { "success" } else { "repair_required" }, bucket = %commit_bucket, object = %commit_object, write_quorum, @@ -5018,7 +6048,6 @@ impl SetDisks { cleanup_ms = cleanup_stage_ms.unwrap_or_default(), cleanup_present = cleanup_stage_ms.is_some(), commit_tail_ms = total_commit_tail_ms as u64, - result = "success", "SetDisk put_object stage summary" ); } @@ -5026,7 +6055,7 @@ impl SetDisks { if !tail_owns_tmp_cleanup { let cleanup_set = commit_set.clone(); let cleanup_tmp_dir = commit_tmp_dir.clone(); - tokio::spawn(async move { + cleanup_runtime::spawn(commit_cleanup_reservation.clone(), async move { if let Err(err) = cleanup_set.delete_all(RUSTFS_META_TMP_BUCKET, &cleanup_tmp_dir).await { warn!(tmp_dir = %cleanup_tmp_dir, error = ?err, "failed to cleanup put_object temporary data"); } else if issue3031_diag_enabled() { @@ -5039,15 +6068,38 @@ impl SetDisks { }); } - Ok(( - ObjectInfo::from_file_info(&fi, &commit_bucket, &commit_object, commit_is_versioned), - old_current_size, - )) + let completion = repair_result.map(|()| { + let object_info_build_started = rustfs_io_metrics::put_stage_timer(); + let object_info = ObjectInfo::from_file_info(&fi, &commit_bucket, &commit_object, commit_is_versioned); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_OBJECT_INFO_BUILD, + object_info_build_started, + ); + let post_object_info_to_return_started = rustfs_io_metrics::put_stage_timer(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_POST_OBJECT_INFO_TO_RETURN, + post_object_info_to_return_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_AFTER_RENAME_TO_RETURN, + commit_after_rename_to_return_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_TOTAL, + commit_closure_total_started, + ); + (object_info, old_current_size) + }); + Ok(CommittedPutOutcome { completion }) }; if let Some(handoff) = commit_cancellation_handoff_tx.take() { let _ = handoff.send(()); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_OPERATION_BODY_TO_COMMIT_SUBMIT, + operation_body_to_commit_submit_started, + ); if detach_commit_owner { if let Some(scope_outcome_guard) = scope_outcome_guard.as_mut() { // The spawned commit closure owns the scope clone and is @@ -5056,17 +6108,53 @@ impl SetDisks { } let mut cancellation = PutObjectCommitCancellation::new(); let child_token = cancellation.child_token(); - let result = tokio::spawn(async move { Box::pin(commit(Some(child_token))).await }) - .await - .map_err(|err| Error::other(format!("put_object commit task failed: {err}")))?; + let commit_task_wait_started = rustfs_io_metrics::put_stage_timer(); + let commit_submit_to_closure_enter_started = rustfs_io_metrics::put_stage_timer(); + #[cfg(feature = "diagnostic-read-version-caller")] + let diagnostic_context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::PutCommitOwner, + ); + let result = tokio::spawn(async move { + let commit = Box::pin(commit(commit_submit_to_closure_enter_started, Some(child_token))); + #[cfg(feature = "diagnostic-read-version-caller")] + return crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context( + diagnostic_context, + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::PutCommitOwner, + commit, + ) + .await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] + commit.await + }) + .await + .map_err(|err| Error::other(format!("put_object commit task failed: {err}")))?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_TASK_WAIT, + commit_task_wait_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_FUTURE_TOTAL, + commit_task_wait_started, + ); cancellation.disarm(); result } else { - Box::pin(commit(None)).await + let commit_task_wait_started = rustfs_io_metrics::put_stage_timer(); + let commit_submit_to_closure_enter_started = rustfs_io_metrics::put_stage_timer(); + let result = Box::pin(commit(commit_submit_to_closure_enter_started, None)).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_TASK_WAIT, + commit_task_wait_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_FUTURE_TOTAL, + commit_task_wait_started, + ); + result } }; - let result: Result<(ObjectInfo, Option)> = if let Some((cancellation, mut handoff_rx)) = cancellation_wait - { + let operation_select_wait_started = rustfs_io_metrics::put_stage_timer(); + let result: Result = if let Some((cancellation, mut handoff_rx)) = cancellation_wait { tokio::pin!(operation); tokio::select! { biased; @@ -5077,27 +6165,22 @@ impl SetDisks { } else { operation.await }; - - if issue3031_diag_enabled() - && let Err(err) = &result - { - let stage_hint = if err.to_string().contains("not enough disks to write") { - "writer_setup_or_quorum" - } else { - "unknown" - }; - warn!( - event = EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_SET_DISK, - bucket = %bucket, - object = %object, - result = "error", - stage_hint, - error = %err, - "SetDisk put_object stage summary" - ); - } + // Normal completion keeps the existing cleanup/error semantics below. + staging_cleanup.runtime = None; + drop(staging_cleanup); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_OPERATION_SELECT_WAIT, + operation_select_wait_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_OPERATION_AWAIT_TOTAL, + operation_select_wait_started, + ); + let operation_result_return_started = rustfs_io_metrics::put_stage_timer(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_OPERATION_RESULT_RETURN, + operation_result_return_started, + ); if issue3031_diag_enabled() { warn!( @@ -5124,7 +6207,7 @@ impl SetDisks { // entry is reclaimed by cleanup_stale_tmp_objects (24h expiry, // 5-minute background loop). let set_disks = self.clone(); - tokio::spawn(async move { + cleanup_runtime::spawn(cleanup_reservation.clone(), async move { if let Err(err) = set_disks.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_dir).await { warn!(tmp_dir = %tmp_dir, error = ?err, "failed to cleanup put_object temporary data"); } else if issue3031_diag_enabled() { @@ -5139,7 +6222,7 @@ impl SetDisks { // Failure path (quorum loss / rollback): keep the cleanup inline so // a failed PUT never returns while its tmp shards are still on disk // (state-residue hardening tracked by backlog#864 / backlog#898). - if let Err(err) = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_dir).await { + if let Err(err) = self.delete_put_tmp_with_budget(&tmp_dir, &cleanup_reservation).await { warn!(tmp_dir = %tmp_dir, error = ?err, "failed to cleanup put_object temporary data"); } else if issue3031_diag_enabled() { warn!( @@ -5152,6 +6235,29 @@ impl SetDisks { } } + let result = result.and_then(|committed| committed.completion); + + if issue3031_diag_enabled() + && let Err(err) = &result + { + let stage_hint = if err.to_string().contains("not enough disks to write") { + "writer_setup_or_quorum" + } else { + "unknown" + }; + warn!( + event = EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_SET_DISK, + result = "error", + bucket = %bucket, + object = %object, + stage_hint, + error = %err, + "SetDisk put_object stage summary" + ); + } + result } } @@ -7703,7 +8809,7 @@ impl SetDisks { &delete_request, false, None, - opts.expected_bucket_incarnation_id, + DeleteCommitContext::request(opts, None), ) .await?; if let Some((_, deleted_object)) = replication_delete { @@ -7777,27 +8883,6 @@ impl SetDisks { Ok(ObjectInfo::from_file_info(&fi, bucket, object, opts.versioned || opts.version_suspended)) } - #[tracing::instrument(skip(self, delete_marker_purge))] - async fn delete_object_version_with_purge( - &self, - bucket: &str, - object: &str, - fi: &FileInfo, - force_del_marker: bool, - delete_marker_purge: Option, - source_bucket_incarnation_id: Option, - ) -> Result<()> { - self.delete_object_version_with_purge_owned( - bucket, - object, - fi, - force_del_marker, - delete_marker_purge, - (source_bucket_incarnation_id, None), - ) - .await - } - async fn delete_object_version_with_purge_owned( &self, bucket: &str, @@ -7807,18 +8892,76 @@ impl SetDisks { delete_marker_purge: Option, publication: (Option, Option>), ) -> Result<()> { - let (source_bucket_incarnation_id, lifecycle_owner) = publication; + let opts = ObjectOptions { + expected_bucket_incarnation_id: publication.0, + delete_mutation_owner: publication.1, + ..Default::default() + }; + self.delete_object_version_with_purge( + bucket, + object, + fi, + force_del_marker, + delete_marker_purge, + DeleteCommitContext::request(&opts, None), + ) + .await + } + + #[tracing::instrument(skip(self, delete_marker_purge, commit_context))] + async fn delete_object_version_with_purge( + &self, + bucket: &str, + object: &str, + fi: &FileInfo, + force_del_marker: bool, + delete_marker_purge: Option, + commit_context: DeleteCommitContext<'_>, + ) -> Result<()> { + let source_bucket_incarnation_id = commit_context.incarnation; let transported = delete_file_info_with_replication_transport_metadata(fi); let fi = &transported; let disks = self.disk_inventory().await; - let namespace_owner = (!is_meta_bucketname(bucket)) - .then(|| Arc::new((self.ctx.begin_namespace_commit(), lifecycle_owner)) as Arc); + let namespace_owner = (!is_meta_bucketname(bucket)).then(|| { + Arc::new(( + self.ctx.begin_namespace_commit(), + commit_context.guard.clone(), + commit_context + .options + .and_then(|options| options.object_lock_config_snapshot.clone()), + commit_context + .options + .and_then(|options| options.bucket_incarnation_fence.clone()), + commit_context + .options + .and_then(|options| options.scanner_publication_commit_scope.clone()), + commit_context.options.and_then(|options| options.write_lock_context.clone()), + commit_context + .options + .and_then(|options| options.delete_mutation_owner.clone()), + )) as Arc + }); // Quorum is a property of the configured set, not of the current // online snapshot. A decommission/offline refresh may temporarily // shorten the snapshot; deriving quorum from it would turn a // quorum-minus-one delete into an apparent success. let write_quorum = self.set_drive_count / 2 + 1; let rollback_dir = Uuid::new_v4(); + // Register before preparation: cancellation and response loss can + // leave a durable journal even when publication never returns. + let recovery_ticket = (!is_meta_bucketname(bucket)) + .then(|| { + self.ctx + .delete_recovery_hints() + .register(crate::store::delete_recovery_runtime::DeleteRecoveryKey { + pool: self.pool_index, + set: self.set_index, + bucket: bucket.to_owned(), + object: object.to_owned(), + incarnation: source_bucket_incarnation_id, + }) + }) + .flatten(); let mut futures = Vec::with_capacity(disks.len()); let mut errs = Vec::with_capacity(disks.len()); @@ -7827,20 +8970,33 @@ impl SetDisks { let disk_namespace_owner = namespace_owner.clone().map(|owner| owner as Arc); futures.push(async move { if let Some(disk) = disk { - match disk - .delete_version_with_namespace_owner( + let options = DeleteOptions { + old_data_dir: Some(rollback_dir), + delete_recovery_bucket_incarnation: source_bucket_incarnation_id, + ..Default::default() + }; + let result = if is_meta_bucketname(bucket) { + disk.delete_version_with_namespace_owner( bucket, object, fi.clone(), force_del_marker, - DeleteOptions { - old_data_dir: Some(rollback_dir), - ..Default::default() - }, + options, disk_namespace_owner, ) .await - { + } else { + disk.delete_version_recoverable_with_namespace_owner( + bucket, + object, + fi.clone(), + force_del_marker, + options, + disk_namespace_owner, + ) + .await + }; + match result { Ok(r) => Ok(r), Err(e) => Err(e), } @@ -7862,12 +9018,35 @@ impl SetDisks { } } - let quorum_result = resolve_tiered_decommission_write_quorum_result(&errs, write_quorum, bucket, object); + let mut quorum_result = resolve_tiered_decommission_write_quorum_result(&errs, write_quorum, bucket, object); + if !is_meta_bucketname(bucket) && errs.iter().flatten().any(super::delete_recovery::unsupported_recovery) { + quorum_result = Err(DiskError::UnsupportedDisk.into()); + } let should_rollback = quorum_result.is_err(); + let structured_cleanup = !is_meta_bucketname(bucket) && !should_rollback; + if structured_cleanup { + self.publish_delete_commit( + bucket, + object, + rollback_dir, + &disks, + write_quorum, + DeleteRecoveryContext { + recovery_ticket, + namespace_owner: namespace_owner.clone().map(|owner| owner as Arc), + ..DeleteRecoveryContext::new(source_bucket_incarnation_id, &|| commit_context.can_commit(bucket, object)) + }, + ) + .await + .map_err(StorageError::from)?; + } #[cfg(test)] pause_delete_before_cleanup(bucket).await; let mut rollback_futures = Vec::new(); for (index, err) in errs.iter().enumerate() { + if structured_cleanup { + break; + } // backlog#1158: when rolling back, fan the idempotent undo out to every // online disk (each self-decides from its staged backup: restore if the // rollback dir is present, no-op otherwise). This covers a disk that @@ -7923,6 +9102,7 @@ impl SetDisks { DeleteOptions { recursive: true, immediate: true, + old_data_dir: Some(rollback_dir), ..Default::default() }, disk_namespace_owner, @@ -8013,6 +9193,41 @@ impl SetDisks { } } +impl SetDisks { + /// Read current metadata without cache admission or read-side repair. The + /// outer batch owner already holds the sorted namespace write-lock set. + pub(crate) async fn conditional_delete_current_snapshot( + &self, + bucket: &str, + object: &str, + opts: &ObjectOptions, + ) -> Result<(ObjectInfo, crate::disk::CurrentDeleteGeneration)> { + let snapshot = self.get_object_fileinfo(bucket, object, opts, false, false).await?; + let file_info = snapshot.fi(); + let generation = crate::disk::CurrentDeleteGeneration::from_file_info(file_info) + .map_err(|error| to_object_err(error.into(), vec![bucket, object]))?; + Ok(( + ObjectInfo::from_file_info(file_info, bucket, object, opts.versioned || opts.version_suspended), + generation, + )) + } + + /// Mixed-version peers must reject the operation before any group mutates. + pub(crate) async fn preflight_guarded_delete_versions(&self) -> Result<()> { + let disks = self.disks.read().await.clone(); + guarded_delete_capability_barrier(disks.iter().flatten().map(|disk| disk.supports_guarded_delete_versions())).await + } +} + +async fn guarded_delete_capability_barrier(probes: impl IntoIterator) -> Result<()> +where + F: std::future::Future>, +{ + // Capability probes are read-only, so a failure can cancel the remaining + // probes. Admission still requires every probe to complete successfully. + futures::future::try_join_all(probes).await.map(|_| ()).map_err(Error::from) +} + #[async_trait::async_trait] impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { type Error = Error; @@ -8306,16 +9521,23 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } else { opts.expected_bucket_incarnation_id }; - let owner: Arc = Arc::new((opts.bucket_incarnation_fence, opts.object_lock_config_snapshot)); - self.delete_object_version_with_purge_owned( - bucket, - object, - fi, - force_del_marker, - None, - (source_bucket_incarnation_id, Some(owner)), - ) - .await + self.invalidate_get_object_metadata_cache(bucket, object).await; + let result = self + .delete_object_version_with_purge( + bucket, + object, + fi, + force_del_marker, + None, + DeleteCommitContext { + incarnation: source_bucket_incarnation_id, + options: Some(&opts), + ..Default::default() + }, + ) + .await; + self.invalidate_get_object_metadata_cache(bucket, object).await; + result } #[tracing::instrument(skip(self, objects, opts))] @@ -8347,6 +9569,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { ); } }; + let current_generations = opts.conditional_delete_generations.as_deref(); let mut del_objects = vec![DeletedObject::default(); objects.len()]; let mut accounting = vec![None; objects.len()]; let delete_config_snapshot = opts @@ -8358,14 +9581,31 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { self.invalidate_get_object_metadata_cache(bucket, &object.object_name).await; } - let mut del_errs = (0..objects.len()).map(|_| None).collect::>(); + let mut del_errs = objects + .iter() + .map(|object| { + object.delete_condition.as_ref().map(|_| { + StorageError::InvalidArgument( + bucket.to_owned(), + object.object_name.clone(), + "conditional delete was not admitted by the store preflight".to_owned(), + ) + }) + }) + .collect::>(); // Acquire locks in batch mode (best effort, matching previous behavior) let mut batch = rustfs_lock::BatchLockRequest::new(self.locker_owner.as_str()).with_all_or_nothing(false); + // Keyed by the same disk identity `new_ns_lock` uses, so aliases such + // as `a\b` and `a/b` on Windows share one lock (and one entry here). + let lock_names: Vec> = objects + .iter() + .map(|dobj| object_name_disk_identity(&dobj.object_name)) + .collect(); let mut unique_objects: HashSet = HashSet::new(); - for dobj in &objects { - if unique_objects.insert(dobj.object_name.clone()) { - batch = batch.add_write_lock(ObjectKey::new(bucket, dobj.object_name.clone())); + for lock_name in &lock_names { + if unique_objects.insert(lock_name.to_string()) { + batch = batch.add_write_lock(ObjectKey::new(bucket, lock_name.to_string())); } } let unique_lock_count = batch.requests.len(); @@ -8418,8 +9658,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } // Mark failures for objects that could not be locked - for (i, dobj) in objects.iter().enumerate() { - if let Some(err) = failed_map.get(&(bucket.to_string(), dobj.object_name.clone())) { + for (i, lock_name) in lock_names.iter().enumerate() { + if let Some(err) = failed_map.get(&(bucket.to_string(), lock_name.to_string())) { del_errs[i] = Some(Error::other(err.to_string())); } } @@ -8434,7 +9674,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { Err(err) => { let message = err.to_string(); for (index, item) in del_errs.iter_mut().enumerate() { - if locked_objects.contains(&objects[index].object_name) { + if locked_objects.contains(lock_names[index].as_ref()) { *item = Some(Error::other(message.clone())); } } @@ -8447,12 +9687,23 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { let mut tier_reference_leases: Vec<(usize, String, Option)> = Vec::new(); let mut tier_free_version_receipt_candidates: HashMap = HashMap::new(); let mut delete_marker_purge_candidates = vec![None; objects.len()]; + let mut missing_delete_retry_indices = Vec::new(); + let mut deferred_recovery_indices = Vec::new(); for (i, dobj) in objects.iter().enumerate() { if del_errs[i].is_some() { continue; } + let guarded = current_generations.is_some_and(|generations| generations.contains_key(&dobj.object_name)); + if guarded && dobj.version_id.is_some() { + del_errs[i] = Some(StorageError::InvalidArgument( + bucket.to_owned(), + dobj.object_name.clone(), + "current-object generation guards cannot protect explicit-version deletes".to_owned(), + )); + continue; + } let replication_object_name = decode_dir_object(&dobj.object_name); let explicit_null_version = is_explicit_null_version(dobj.version_id); let version_id = delete_file_info_version_id(dobj.version_id); @@ -8466,6 +9717,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { object_lock_delete: opts.object_lock_delete.clone(), object_lock_config_snapshot: Some(Arc::clone(&object_lock_config)), no_lock: true, + suppress_read_repair: guarded, ..Default::default() }; // A missing bucket Object Lock configuration does not prove that @@ -8477,7 +9729,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { let marker_delete = dobj.version_id.is_none() || dobj.synthetic_version_id; let replication_needs_source = replicate_delete && (!marker_delete || delete_config_snapshot.active_delete_marker_rules_require_tags(&replication_object_name)); - let (goi, authoritative_file_info, gerr) = if object_lock_check_required + let (mut goi, mut authoritative_file_info, mut gerr) = if object_lock_check_required || replication_needs_source || opts.tier_delete_journal_api.is_some() || dobj.expected_identity.is_some() @@ -8490,9 +9742,76 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } else { (ObjectInfo::default(), FileInfo::default(), None) }; + if gerr + .as_ref() + .is_some_and(|error| is_err_object_not_found(error) || is_err_version_not_found(error)) + && !is_meta_bucketname(bucket) + && !opts.data_movement + && !opts.replication_request + && opts.http_preconditions.is_none() + && !guarded + { + let recovery = self + .recover_delete_key( + bucket, + &dobj.object_name, + DeleteRecoveryContext { + dry_run: !opts.no_lock, + namespace_owner: opts.delete_mutation_owner.clone(), + ..DeleteRecoveryContext::new(opts.expected_bucket_incarnation_id, &|| { + ensure_delete_commit_locks_held(None, bucket, &dobj.object_name, &opts).is_ok() + }) + }, + ) + .await; + match recovery { + Ok(recovery) if !opts.no_lock && recovery.discovered > 0 && recovery.pending == 0 => { + deferred_recovery_indices.push(i); + continue; + } + Ok(recovery) + if (recovery.pending > 0 + && !(recovery.no_transactions_at_quorum + && dobj.version_id.is_none() + && !versioned + && !version_suspended)) + || (recovery.blocked > 0 && (dobj.version_id.is_some() || versioned || version_suspended)) => + { + del_errs[i] = Some(if recovery.pending > 0 { + StorageError::InsufficientWriteQuorum(bucket.to_owned(), dobj.object_name.clone()) + } else { + StorageError::SlowDown + }); + continue; + } + Ok(recovery) if recovery.rolled_back > 0 || recovery.committed > 0 => { + let (fresh, info, _, error) = self + .get_object_info_fileinfo_and_quorum(bucket, &dobj.object_name, &check_opts) + .await; + goi = fresh; + authoritative_file_info = info; + gerr = error; + } + Ok(_) => {} + Err(error) => { + del_errs[i] = Some(to_object_err(error.into(), vec![bucket, &dobj.object_name])); + continue; + } + } + } let source_missing = gerr .as_ref() .is_some_and(|err| is_err_object_not_found(err) || is_err_version_not_found(err)); + if source_missing + && dobj.version_id.is_none() + && dobj.expected_identity.is_none() + && !versioned + && !version_suspended + && opts.http_preconditions.is_none() + && !guarded + { + missing_delete_retry_indices.push(i); + } // A pool-local miss keeps the existing idempotent batch-delete // semantics. Exact-version deletes fan out to every pool, and a // retry may also arrive after the intended generation is gone. @@ -8645,20 +9964,6 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { vr.set_delete_marker_incarnation(incarnation); } - let v = { - if vers_map.contains_key(&dobj.object_name) { - let val = vers_map.get_mut(&dobj.object_name).unwrap(); - val.versions.push(vr.clone()); - val.clone() - } else { - FileInfoVersions { - name: vr.name.clone(), - versions: vec![vr.clone()], - ..Default::default() - } - } - }; - if vr.deleted || removed_delete_marker { del_objects[i] = DeletedObject { delete_marker: true, @@ -8712,8 +10017,15 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } // Only add to vers_map if we hold the lock - if locked_objects.contains(&dobj.object_name) { - vers_map.insert(&dobj.object_name, v); + if locked_objects.contains(lock_names[i].as_ref()) { + vers_map + .entry(&dobj.object_name) + .or_insert_with(|| FileInfoVersions { + name: vr.name.clone(), + ..Default::default() + }) + .versions + .push(vr); } } @@ -8730,10 +10042,17 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } let rollback_dir = Uuid::new_v4(); + let namespace_owner = (!is_meta_bucketname(bucket)).then(|| { + Arc::new(( + self.ctx.begin_namespace_commit(), + opts.delete_mutation_owner.clone(), + opts.object_lock_config_snapshot.clone(), + opts.scanner_publication_commit_scope.clone(), + opts.write_lock_context.clone(), + )) as Arc + }); - let disks = self.disks.read().await; - - let disks = disks.clone(); + let disks = self.disks.read().await.clone(); if opts .namespace_lock_fence @@ -8796,12 +10115,31 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { fi_vers.versions.retain(|fi| del_errs[fi.idx].is_none()); } vers.retain(|fi_vers| !fi_vers.versions.is_empty()); + let recovery_tickets: Vec<_> = vers + .iter() + .map(|group| { + (!is_meta_bucketname(bucket)) + .then(|| { + self.ctx + .delete_recovery_hints() + .register(crate::store::delete_recovery_runtime::DeleteRecoveryKey { + pool: self.pool_index, + set: self.set_index, + bucket: bucket.to_owned(), + object: group.name.clone(), + incarnation: opts.expected_bucket_incarnation_id, + }) + }) + .flatten() + }) + .collect(); let mut futures = Vec::with_capacity(disks.len()); let lock_lost_during_commit = Arc::new(std::sync::atomic::AtomicBool::new(false)); // let mut errors = Vec::with_capacity(disks.len()); for disk in disks.iter() { + let disk_namespace_owner = namespace_owner.clone(); let vers = vers.clone(); let namespace_lock_fence = opts.namespace_lock_fence.clone(); let lock_lost_during_commit = Arc::clone(&lock_lost_during_commit); @@ -8811,15 +10149,66 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { return (0..vers.len()).map(|_| Some(DiskError::DiskOngoingReq)).collect(); } if let Some(disk) = disk { - disk.delete_versions( - bucket, - vers, - DeleteOptions { - old_data_dir: Some(rollback_dir), - ..Default::default() - }, - ) - .await + let options = DeleteOptions { + old_data_dir: Some(rollback_dir), + delete_recovery_bucket_incarnation: opts.expected_bucket_incarnation_id, + ..Default::default() + }; + if current_generations + .is_some_and(|generations| vers.iter().any(|group| generations.contains_key(&group.name))) + { + let mut guarded = Vec::new(); + let mut ordinary = Vec::new(); + let mut guarded_indices = Vec::new(); + let mut ordinary_indices = Vec::new(); + let count = vers.len(); + for (index, group) in vers.into_iter().enumerate() { + if let Some(generation) = current_generations.and_then(|generations| generations.get(&group.name)) { + guarded_indices.push(index); + guarded.push(crate::disk::GuardedFileInfoVersions { + group, + expected_current: generation.clone(), + }); + } else { + ordinary_indices.push(index); + ordinary.push(group); + } + } + let mut results = vec![None; count]; + let guarded_results = disk + .delete_versions_guarded_recoverable_with_namespace_owner( + bucket, + guarded, + options.clone(), + disk_namespace_owner.clone(), + ) + .await; + for (index, error) in guarded_indices.into_iter().zip(guarded_results) { + results[index] = error; + } + if !ordinary.is_empty() { + let errors = if is_meta_bucketname(bucket) { + disk.delete_versions(bucket, ordinary, options).await + } else { + disk.delete_versions_recoverable_with_namespace_owner( + bucket, + ordinary, + options, + disk_namespace_owner, + ) + .await + }; + for (index, error) in ordinary_indices.into_iter().zip(errors) { + results[index] = error; + } + } + results + } else if is_meta_bucketname(bucket) { + disk.delete_versions(bucket, vers, options).await + } else { + disk.delete_versions_recoverable_with_namespace_owner(bucket, vers, options, disk_namespace_owner) + .await + } } else { let mut errs = Vec::with_capacity(vers.len()); for _ in 0..vers.len() { @@ -8855,7 +10244,17 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } } - let mut has_err = reduce_write_quorum_errs(&disk_err, OBJECT_OP_IGNORED_ERRS, disks.len() / 2 + 1); + let mut has_err = reduce_write_quorum_errs(&disk_err, OBJECT_OP_IGNORED_ERRS, self.set_drive_count / 2 + 1); + if disk_err + .iter() + .flatten() + .any(|error| matches!(error, DiskError::GuardedDeleteUnsupported)) + { + has_err = Some(DiskError::GuardedDeleteUnsupported); + } + if !is_meta_bucketname(bucket) && disk_err.iter().flatten().any(super::delete_recovery::unsupported_recovery) { + has_err = Some(DiskError::UnsupportedDisk); + } if let Some(err) = has_err.clone() { let er = err.into(); if (is_err_object_not_found(&er) || is_err_version_not_found(&er)) && !del_objects[obj_idx].delete_marker { @@ -8899,10 +10298,69 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { #[cfg(test)] pause_delete_before_cleanup(bucket).await; + let mut published_groups = HashSet::new(); + if !is_meta_bucketname(bucket) { + let groups = vers.iter().enumerate().filter_map(|(index, group)| { + group + .versions + .iter() + .all(|version| del_errs[version.idx].is_none()) + .then_some(index) + }); + // Mark all attempted groups before starting publication. A lost + // response can hide a persisted decision and must never roll back. + let groups = groups.collect::>(); + published_groups.extend(groups.iter().map(|index| vers[*index].name.clone())); + let publications = futures::stream::iter(groups.into_iter().map(|index: usize| { + let group = &vers[index]; + let recovery_ticket = recovery_tickets[index].clone(); + let namespace_owner = namespace_owner.clone(); + let disks = &disks; + let opts = &opts; + async move { + let can_commit = || { + ensure_delete_commit_locks_held(None, bucket, &group.name, opts).is_ok() + && opts + .scanner_publication_commit_scope + .as_ref() + .is_none_or(|scope| scope.can_commit()) + }; + let result = self + .publish_delete_commit( + bucket, + &group.name, + rollback_dir, + disks, + self.set_drive_count / 2 + 1, + DeleteRecoveryContext { + recovery_ticket, + namespace_owner, + ..DeleteRecoveryContext::new(opts.expected_bucket_incarnation_id, &can_commit) + }, + ) + .await; + (index, result) + } + })) + .buffer_unordered(16) + .collect::>() + .await; + for (index, result) in publications { + let group = &vers[index]; + if let Err(error) = result { + for version in &group.versions { + del_errs[version.idx] = Some(to_object_err(error.clone().into(), vec![bucket, &group.name])); + } + } + } + } let mut rollback_futures = Vec::new(); let committed_receipt_indices = committed_tier_free_version_receipt_indices(&vers, &del_errs, &tier_free_version_receipt_candidates); for fi_vers in &vers { + if published_groups.contains(&fi_vers.name) { + continue; + } // delete_versions commits one xl.meta per object group, so rollback must use the same boundary. let should_rollback = fi_vers.versions.iter().any(|fi| del_errs[fi.idx].is_some()); for (disk_idx, disk) in disks.iter().enumerate() { @@ -8922,10 +10380,11 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { let bucket = bucket.to_string(); let object = fi_vers.name.clone(); let versions = fi_vers.clone(); + let disk_namespace_owner = namespace_owner.clone(); rollback_futures.push(async move { if should_rollback { let errs = disk - .delete_versions( + .delete_versions_with_namespace_owner( &bucket, vec![versions], DeleteOptions { @@ -8934,6 +10393,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { old_data_dir: Some(rollback_dir), ..Default::default() }, + disk_namespace_owner, ) .await; if let Some(err) = errs.into_iter().flatten().next() { @@ -8954,6 +10414,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { DeleteOptions { recursive: true, immediate: true, + old_data_dir: Some(rollback_dir), ..Default::default() }, ) @@ -8990,12 +10451,70 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { ); } + // A new explicit DELETE authorizes removal of the exact missing key's + // legacy residue; it does not infer whether the interrupted delete committed. + for index in missing_delete_retry_indices { + if del_errs[index].is_none() { + let object = &objects[index].object_name; + let result = match ensure_delete_commit_locks_held(None, bucket, object, &opts) { + Ok(()) => self.reclaim_missing_delete_residue(bucket, object, &opts, None).await, + Err(err) => Err(err), + }; + if let Err(err) = result { + del_errs[index] = Some(err); + } + } + } + // TODO(backlog): support partial object deletion for multi-part objects if dist_erasure { self.release_dist_delete_object_locks_batch(dist_batch_lock_ids).await; } + // Cold missing-key replay uses the same batch contract after the fast + // batch lock is released; renewable key ownership survives settlement. + for index in deferred_recovery_indices { + let object = &objects[index]; + let explicit_delete = object.version_id.is_none() + && !object.synthetic_version_id + && !opts.versioned + && !opts.version_suspended + && opts.http_preconditions.is_none() + && metadata_sys::has_authoritative_never_versioned_state_in(&self.ctx, bucket) + .await + .unwrap_or(false) + && matches!(object_lock_config.state(), ObjectLockConfigState::ConfirmedAbsent); + let recovery = self + .recover_delete_object( + bucket, + &object.object_name, + DeleteRecoveryContext { + explicit_delete, + namespace_owner: Some(Arc::new(( + self.ctx.begin_namespace_commit(), + opts.delete_mutation_owner.clone(), + opts.object_lock_config_snapshot.clone(), + ))), + ..DeleteRecoveryContext::new(opts.expected_bucket_incarnation_id, &|| { + ensure_delete_commit_locks_held(None, bucket, &object.object_name, &opts).is_ok() + }) + }, + ) + .await; + match recovery { + Ok(recovery) if recovery.pending == 0 && recovery.blocked == 0 => { + let (mut deleted, mut errors, mut receipts) = + Box::pin(self.delete_objects_with_accounting(bucket, vec![object.clone()], opts.clone())).await; + del_objects[index] = deleted.remove(0); + del_errs[index] = errors.remove(0); + accounting[index] = receipts.remove(0); + } + Ok(_) => del_errs[index] = Some(StorageError::SlowDown), + Err(error) => del_errs[index] = Some(to_object_err(error.into(), vec![bucket, &object.object_name])), + } + } + let purge_submissions = delete_marker_purge_candidates .into_iter() .enumerate() @@ -9054,7 +10573,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { // Guard lock for single object delete let _lock_guard = if (!opts.delete_prefix || opts.delete_prefix_object) && !opts.no_lock { - Some(self.acquire_write_lock_diag("delete_object", bucket, object).await?) + Some(Arc::new(self.acquire_write_lock_diag("delete_object", bucket, object).await?)) } else { None }; @@ -9124,7 +10643,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } }; for version in plan { - ensure_delete_commit_locks_held(_lock_guard.as_ref(), bucket, object, &opts)?; + ensure_delete_commit_locks_held(_lock_guard.as_deref(), bucket, object, &opts)?; // The lease closes the gap between the durable // reference proof and publishing either a committed // v6 journal or the legacy free-version fallback. @@ -9158,7 +10677,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { &delete_request, false, None, - opts.expected_bucket_incarnation_id, + DeleteCommitContext::request(&opts, _lock_guard.as_ref()), ) .await?; if let Some((_, deleted_object)) = replication_delete { @@ -9167,7 +10686,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } } else { for version in &versions.versions { - ensure_delete_commit_locks_held(_lock_guard.as_ref(), bucket, object, &opts)?; + ensure_delete_commit_locks_held(_lock_guard.as_deref(), bucket, object, &opts)?; let mut delete_request = FileInfo { name: object.to_string(), version_id: version.version_id, @@ -9181,12 +10700,12 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { &delete_request, false, None, - opts.expected_bucket_incarnation_id, + DeleteCommitContext::request(&opts, _lock_guard.as_ref()), ) .await?; } for version in &versions.free_versions { - ensure_delete_commit_locks_held(_lock_guard.as_ref(), bucket, object, &opts)?; + ensure_delete_commit_locks_held(_lock_guard.as_deref(), bucket, object, &opts)?; let mut delete_request = FileInfo { name: object.to_string(), version_id: version.version_id, @@ -9201,7 +10720,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { &delete_request, false, None, - opts.expected_bucket_incarnation_id, + DeleteCommitContext::request(&opts, _lock_guard.as_ref()), ) .await?; } @@ -9217,7 +10736,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { self.validate_bucket_incarnation(bucket, expected_incarnation_id, &opts) .await?; } - ensure_delete_commit_locks_held(_lock_guard.as_ref(), bucket, object, &opts)?; + ensure_delete_commit_locks_held(_lock_guard.as_deref(), bucket, object, &opts)?; begin_scanner_publication_delete_mutation(scanner_publication_commit_scope.as_ref())?; self.delete_prefix_with_scanner_publication_lease( bucket, @@ -9268,14 +10787,73 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { let mut version_found = true; // delete_object_version below derives its own majority quorum from the // disk array, so the object-derived quorum here is unused. - let (mut goi, authoritative_file_info, _write_quorum, gerr) = + let (mut goi, mut authoritative_file_info, _write_quorum, mut gerr) = self.get_object_info_fileinfo_and_quorum(bucket, object, &opts).await; + if gerr + .as_ref() + .is_some_and(|error| is_err_object_not_found(error) || is_err_version_not_found(error)) + && !is_meta_bucketname(bucket) + && !opts.data_movement + && !opts.replication_request + && opts.http_preconditions.is_none() + { + if let Some(expected) = opts.expected_bucket_incarnation_id { + self.validate_bucket_incarnation(bucket, expected, &opts).await?; + } + let recovery = self + .recover_delete_key( + bucket, + object, + DeleteRecoveryContext { + namespace_owner: Some(Arc::new(( + self.ctx.begin_namespace_commit(), + _lock_guard.clone(), + opts.object_lock_config_snapshot.clone(), + opts.scanner_publication_commit_scope.clone(), + opts.write_lock_context.clone(), + opts.delete_mutation_owner.clone(), + ))), + ..DeleteRecoveryContext::new(opts.expected_bucket_incarnation_id, &|| { + ensure_delete_commit_locks_held(_lock_guard.as_deref(), bucket, object, &opts).is_ok() + }) + }, + ) + .await?; + if (recovery.pending > 0 + && !(recovery.no_transactions_at_quorum + && opts.version_id.is_none() + && !opts.versioned + && !opts.version_suspended)) + || (recovery.blocked > 0 && (opts.version_id.is_some() || opts.versioned || opts.version_suspended)) + { + return Err(if recovery.pending > 0 { + StorageError::InsufficientWriteQuorum(bucket.to_owned(), object.to_owned()) + } else { + StorageError::SlowDown + }); + } + if recovery.rolled_back > 0 || recovery.committed > 0 { + (goi, authoritative_file_info, _, gerr) = self.get_object_info_fileinfo_and_quorum(bucket, object, &opts).await; + } + } if let Some(err) = &gerr && goi.name.is_empty() { if should_force_delete_marker_for_missing_version(&opts) { version_found = false; } else { + if is_err_object_not_found(err) { + ensure_delete_commit_locks_held(_lock_guard.as_deref(), bucket, object, &opts)?; + if self + .reclaim_missing_delete_residue(bucket, object, &opts, _lock_guard.as_ref()) + .await? + { + if let Some(scope) = scanner_publication_commit_scope.as_ref() { + let _ = scope.mark_committed(); + } + return Ok(ObjectInfo::default()); + } + } return Err(err.clone()); } } @@ -9309,11 +10887,18 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { expire_restored: true, ..Default::default() }; - ensure_delete_commit_locks_held(_lock_guard.as_ref(), bucket, object, &opts)?; + ensure_delete_commit_locks_held(_lock_guard.as_deref(), bucket, object, &opts)?; begin_scanner_publication_delete_mutation(scanner_publication_commit_scope.as_ref())?; - self.delete_object_version_with_purge(bucket, object, &dfi, false, None, opts.expected_bucket_incarnation_id) - .await - .map_err(|e| to_object_err(e, vec![bucket, object]))?; + self.delete_object_version_with_purge( + bucket, + object, + &dfi, + false, + None, + DeleteCommitContext::request(&opts, _lock_guard.as_ref()), + ) + .await + .map_err(|e| to_object_err(e, vec![bucket, object]))?; self.invalidate_get_object_metadata_cache(bucket, object).await; if let Some(scope) = scanner_publication_commit_scope.as_ref() { let _ = scope.mark_committed(); @@ -9393,7 +10978,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { None }; - ensure_delete_commit_locks_held(_lock_guard.as_ref(), bucket, object, &opts)?; + ensure_delete_commit_locks_held(_lock_guard.as_deref(), bucket, object, &opts)?; begin_scanner_publication_delete_mutation(scanner_publication_commit_scope.as_ref())?; let _tier_delete_lease = acquire_single_tier_delete_lease(&opts, &goi).await?; if opts.skip_free_version { @@ -9405,7 +10990,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { &fi, should_force_delete_marker_for_missing_version(&opts), delete_marker_purge.clone(), - opts.expected_bucket_incarnation_id, + DeleteCommitContext::request(&opts, _lock_guard.as_ref()), ) .await .map_err(|e| to_object_err(e, vec![bucket, object]))?; @@ -9452,7 +11037,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { dfi.set_delete_marker_incarnation(incarnation); } - ensure_delete_commit_locks_held(_lock_guard.as_ref(), bucket, object, &opts)?; + ensure_delete_commit_locks_held(_lock_guard.as_deref(), bucket, object, &opts)?; begin_scanner_publication_delete_mutation(scanner_publication_commit_scope.as_ref())?; let _tier_delete_lease = acquire_single_tier_delete_lease(&opts, &goi).await?; if opts.skip_free_version { @@ -9464,7 +11049,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { &dfi, opts.delete_marker, delete_marker_purge, - opts.expected_bucket_incarnation_id, + DeleteCommitContext::request(&opts, _lock_guard.as_ref()), ) .await .map_err(|e| to_object_err(e, vec![bucket, object]))?; @@ -9517,10 +11102,18 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { // Use the same full xl.meta read path as GetObject metadata resolution. // This avoids HEAD/GetObject metadata visibility skew immediately after // PutObject/CompleteMultipartUpload. - let snapshot = self - .get_object_fileinfo(bucket, object, opts, true, false) - .await - .map_err(|e| to_object_err(e, vec![bucket, object]))?; + let read = self.get_object_fileinfo(bucket, object, opts, true, false); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ObjectInfoRead, + async { + crate::cluster::rpc::runtime_sources::observe_read_version_context_boundary( + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::ObjectInfoRead, + ); + read.await + }, + ); + let snapshot = read.await.map_err(|e| to_object_err(e, vec![bucket, object]))?; let oi = ObjectInfo::from_file_info(snapshot.fi(), bucket, object, opts.versioned || opts.version_suspended); @@ -10095,7 +11688,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { pause_transition_transaction_at(bucket, object, TransitionTransactionKillPoint::CommitFenceBeforeLocalCommit).await; upload_cleanup.disarm(); if let Err(err) = self - .delete_object_version_with_purge(bucket, object, &fi, false, None, opts.expected_bucket_incarnation_id) + .delete_object_version_with_purge(bucket, object, &fi, false, None, DeleteCommitContext::request(opts, None)) .await { warn!( @@ -10863,6 +12456,232 @@ mod object_encryption_resolver_wiring_tests { } } + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_put_owner_preserves_parent_context_at_tier_free_version_lookup() { + use crate::cluster::rpc::runtime_sources::{current_read_version_entry_source, with_read_version_entry}; + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + use rustfs_io_metrics::internode_metrics::ReadVersionEntrySource; + use rustfs_io_metrics::s3_http_metrics::{S3HttpRequestGuard, current_s3_http_operation_context}; + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let (_dirs, disks, set) = runtime.block_on(crate::set_disk::hermetic_set_disks_isolated(4)); + let bucket = "diagnostic-put-owner"; + runtime.block_on(async { + for disk in &disks { + disk.make_volume(bucket).await.expect("make test bucket volume"); + } + for index in 0..4 { + set.put_object( + bucket, + &format!("object-{index}"), + &mut PutObjReader::from_vec(b"old body".to_vec()), + &ObjectOptions::default(), + ) + .await + .expect("seed source object"); + } + }); + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let mut request = S3HttpRequestGuard::new("PUT"); + let put = request.in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::PutObject); + current_s3_http_operation_context().expect("captured PutObject") + }); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + let mut index = 0; + for earlier in [None, Some(ReadVersionEntrySource::ScannerConfigWrite)] { + for api in [None, Some(put)] { + let object = format!("object-{index}"); + index += 1; + let write = async { + let mut reader = PutObjReader::from_vec(b"new body".to_vec()); + let info = set + .put_object( + bucket, + &object, + &mut reader, + &ObjectOptions { + tier_free_version_receipt_sink: Some(crate::object_api::TierFreeVersionReceiptSink::new()), + ..Default::default() + }, + ) + .await + .expect("replace source object"); + assert_eq!(info.size, 8); + }; + rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(api, async { + match earlier { + Some(source) => with_read_version_entry(source, write).await, + None => write.await, + } + }) + .await; + assert_eq!(current_read_version_entry_source(), None); + assert_eq!(current_s3_http_operation_context(), None); + } + } + }); + }); + let rows = snapshots.snapshot().into_vec(); + for source in ["scanner_config_write", "put_commit_owner"] { + for boundary in ["put_commit_owner", "tier_free_version_source_lookup"] { + for (api, state) in [("unattributed", "absent"), ("s3:PutObject", "preserved")] { + assert!( + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("entry_source") == Some(&source) + && labels.get("boundary") == Some(&boundary) + && labels.get("parent_api_operation") == Some(&api) + && labels.get("child_api_operation") == Some(&api) + && labels.get("api_context") == Some(&state) + && matches!(value, DebugValue::Counter(count) if *count > 0) + }), + "actual detached owner and lookup must retain context: {source}/{boundary}/{api}" + ); + } + } + } + runtime.block_on(async { + for index in 0..4 { + let mut reader = set + .get_object_reader(bucket, &format!("object-{index}"), None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("committed object remains readable"); + let mut body = Vec::new(); + tokio::io::AsyncReadExt::read_to_end(&mut reader.stream, &mut body) + .await + .expect("read committed body"); + assert_eq!(body, b"new body"); + } + }); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_write_precondition_and_object_info_keep_origin_and_api() { + use crate::cluster::rpc::runtime_sources::{current_read_version_entry_source, with_read_version_entry}; + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + use rustfs_io_metrics::internode_metrics::ReadVersionEntrySource; + use rustfs_io_metrics::s3_http_metrics::{S3HttpRequestGuard, current_s3_http_operation_context}; + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let mut guard = S3HttpRequestGuard::new("PUT"); + let put = guard.in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::PutObject); + current_s3_http_operation_context().expect("captured PutObject") + }); + let (_dirs, disks, set) = runtime.block_on(crate::set_disk::hermetic_set_disks_isolated(4)); + let bucket = "diagnostic-precondition"; + let object = "object"; + let original = runtime.block_on(async { + for disk in &disks { + disk.make_volume(bucket).await.expect("create test bucket volume"); + } + set.put_object(bucket, object, &mut PutObjReader::from_vec(b"body".to_vec()), &ObjectOptions::default()) + .await + .expect("seed object") + }); + let etag = original.etag.expect("seed ETag"); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + for earlier in [None, Some(ReadVersionEntrySource::ConfigWrite)] { + for api in [None, Some(put)] { + let read = async { + assert_eq!( + set.get_object_info(bucket, object, &ObjectOptions::default()) + .await + .expect("read object info") + .etag, + Some(etag.clone()) + ); + for (expected, condition) in [(false, etag.clone()), (true, "other-etag".to_string())] { + let result = set + .check_write_precondition( + bucket, + object, + &ObjectOptions { + http_preconditions: Some(HTTPPreconditions { + if_match: Some(condition), + ..Default::default() + }), + ..Default::default() + }, + ) + .await; + assert_eq!(result, expected.then_some(Error::PreconditionFailed)); + assert_eq!(current_s3_http_operation_context(), api); + } + }; + rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(api, async { + match earlier { + Some(source) => with_read_version_entry(source, read).await, + None => read.await, + } + }) + .await; + assert_eq!(current_read_version_entry_source(), None); + assert_eq!(current_s3_http_operation_context(), None); + } + } + }); + }); + let rows = snapshots.snapshot().into_vec(); + for (source, boundary) in [ + ("write_precondition_read", "write_precondition_lookup"), + ("write_precondition_read", "object_info_read"), + ("write_precondition_read", "full_wait"), + ("object_info_read", "object_info_read"), + ("config_write", "write_precondition_lookup"), + ("config_write", "object_info_read"), + ("config_write", "full_wait"), + ] { + for (api, state) in [("unattributed", "absent"), ("s3:PutObject", "preserved")] { + assert!( + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("entry_source") == Some(&source) + && labels.get("boundary") == Some(&boundary) + && labels.get("parent_api_operation") == Some(&api) + && labels.get("child_api_operation") == Some(&api) + && labels.get("api_context") == Some(&state) + && matches!(value, DebugValue::Counter(count) if *count > 0) + }), + "actual lookup must preserve origin and API: {source}/{boundary}/{api}" + ); + } + } + assert_eq!( + runtime + .block_on(set.get_object_info(bucket, object, &ObjectOptions::default())) + .expect("object retained") + .etag, + Some(etag) + ); + } + #[tokio::test] async fn get_object_reader_forwards_instance_resolver() { let resolver = Arc::new(CountingResolver { @@ -12942,36 +14761,36 @@ mod get_object_downstream_close_accounting_tests { .enable_all() .build() .expect("current-thread runtime should build"); + let (_temp_dirs, _disk_stores, set_disks, bucket, object, payload, options) = runtime.block_on(async { + let (_temp_dirs, _disk_stores, set_disks) = hermetic_set_disks(4).await; + let bucket = "get-downstream-close-accounting"; + let object = "object.bin"; + let payload = b"downstream-close-accounting-".repeat(10_000); + let options = ObjectOptions { + no_lock: true, + ..Default::default() + }; + + set_disks + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("bucket should be created"); + let owned_options = + hermetic_set_disks_support::lifecycle_owned_put_options(&set_disks, bucket, options.clone()).await; + let mut put_reader = PutObjReader::from_vec(payload.clone()); + set_disks + .put_object(bucket, object, &mut put_reader, &owned_options) + .await + .expect("object should be written"); + + (_temp_dirs, _disk_stores, set_disks, bucket, object, payload, options) + }); let recorder = CapturingRecorder::default(); let previous_gate = rustfs_io_metrics::get_stage_metrics_enabled(); rustfs_io_metrics::set_get_stage_metrics_enabled(true); let (decode_failures, emit_failures, legacy_fanout, internal_fanout) = metrics::with_local_recorder(&recorder, || { runtime.block_on(async { - let (_temp_dirs, _disk_stores, set_disks) = hermetic_set_disks(4).await; - let bucket = "get-downstream-close-accounting"; - let object = "object.bin"; - let payload = b"downstream-close-accounting-".repeat(10_000); - let options = ObjectOptions { - no_lock: true, - ..Default::default() - }; - - set_disks - .make_bucket(bucket, &MakeBucketOptions::default()) - .await - .expect("bucket should be created"); - let mut put_reader = PutObjReader::from_vec(payload.clone()); - set_disks - .put_object( - bucket, - object, - &mut put_reader, - &hermetic_set_disks_support::lifecycle_owned_put_options(&set_disks, bucket, options.clone()).await, - ) - .await - .expect("object should be written"); - let range = Some(HTTPRangeSpec { is_suffix_length: false, start: 0, @@ -14095,6 +15914,25 @@ mod metadata_mutation_generation_tests { assert_retired(&set_disks, &delete_key).await; drop(delete_probe); + let raw_object = "raw-delete-object"; + let (_, raw_key) = put_and_prime(&set_disks, bucket, raw_object, b"raw DELETE body").await; + let raw_probe = MetadataCacheInvalidationProbe::install(bucket, raw_object); + set_disks + .delete_object_version( + bucket, + raw_object, + &FileInfo { + name: raw_object.to_owned(), + ..Default::default() + }, + false, + ) + .await + .expect("raw DELETE should succeed"); + assert_eq!(raw_probe.count(), 2, "raw DELETE must fence the cache before and after mutation"); + assert_retired(&set_disks, &raw_key).await; + drop(raw_probe); + let copy_object = "copy-object"; let (mut copy_info, copy_key) = put_and_prime(&set_disks, bucket, copy_object, b"COPY body").await; copy_info.metadata_only = true; @@ -15212,6 +17050,7 @@ mod transition_commit_failure_tests { cache_key.clone(), Arc::new(GetObjectMetadataCacheEntry { created_at: Instant::now(), + capability: crate::set_disk::GetObjectMetadataCacheCapability::Full, fi, parts_metadata, online_disks, @@ -17553,6 +19392,44 @@ mod transition_upload_integrity_tests { /// Plain objects must keep streaming the remote bytes through untouched: /// their plan is `Plain`, so the tiered read stays byte-identical. + #[tokio::test] + #[serial_test::serial] + async fn transitioned_conditional_read_skips_remote_get_with_live_positive_control() { + struct MatchingEtag(String); + impl crate::object_api::GetObjectReadCondition for MatchingEtag { + fn is_terminal(&self, metadata: crate::object_api::GetObjectReadMetadata<'_>) -> bool { + !metadata.delete_marker && metadata.etag == Some(self.0.as_str()) + } + } + let (_dirs, disks, set) = hermetic_set_disks(4).await; + let bucket = "transitioned-condition-body-get"; + let object = "object.bin"; + let payload = b"conditional remote reads need metadata only".repeat(256); + let original = write_source(&set, &disks, bucket, object, &payload).await; + let tier = format!("CONDITION{}", &Uuid::new_v4().simple().to_string()[..8]).to_uppercase(); + let backend = register_mock_tier(&runtime_sources::global_tier_config_mgr(), &tier).await; + set.transition_object(bucket, object, &transition_options(&original, tier)) + .await + .expect("transition object"); + assert_eq!(backend.get_count().await, 0, "transition setup has not fetched the remote body"); + let opts = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let reader = crate::object_api::with_get_object_read_condition( + Arc::new(MatchingEtag(original.etag.expect("source ETag"))), + set.get_object_reader(bucket, object, None, HeaderMap::new(), &opts), + ) + .await + .expect("terminal transitioned metadata"); + assert!(reader.object_info.is_remote(), "the condition was evaluated on transitioned metadata"); + drop(reader); + assert_eq!(backend.get_count().await, 0, "a matching condition never calls the warm backend GET"); + let (body, _) = read_transitioned(&set, bucket, object, None, &opts).await; + assert_eq!(body, payload); + assert_eq!(backend.get_count().await, 1, "the same backend counter observes a normal remote GET"); + } + #[tokio::test] #[serial_test::serial] async fn transitioned_plain_object_get_is_unchanged() { @@ -19233,6 +21110,73 @@ mod put_object_tmp_cleanup_tests { ); } + #[tokio::test] + async fn put_cleanup_admission_rejects_before_staging_with_an_external_lock() { + const CHILD: &str = "RUSTFS_CLEANUP_ADMISSION_TEST_CHILD"; + if std::env::var_os(CHILD).is_none() { + let output = tokio::task::spawn_blocking(|| { + std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "set_disk::ops::object::put_object_tmp_cleanup_tests::put_cleanup_admission_rejects_before_staging_with_an_external_lock", + "--nocapture", + ]) + .env(CHILD, "1") + .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true") + .env("RUSTFS_PUT_RENAME_TAIL_CLEANUP_MAX_PENDING", "1") + .env("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKERS", "1") + .env("RUSTFS_CLEANUP_CPUS", "none") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP") + .output() + .unwrap() + }) + .await + .unwrap(); + assert!( + output.status.success(), + "{}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed")); + return; + } + let (dirs, disks, set) = hermetic_set_disks(4).await; + let bucket = "cleanup-admission-bucket"; + let object = "externally-locked"; + for disk in &disks { + disk.make_volume(bucket).await.unwrap(); + } + let occupied = cleanup_runtime::reserve().unwrap().expect("isolation enabled in child"); + let external_lock = set.acquire_write_lock_diag("admission_test", bucket, object).await.unwrap(); + let mut reader = PutObjReader::from_vec(vec![7u8; TEST_OBJECT_SIZE]); + let opts = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let result = tokio::time::timeout(Duration::from_secs(20), set.put_object(bucket, object, &mut reader, &opts)) + .await + .expect("admission must not wait while the caller owns locks"); + assert!(matches!(result, Err(StorageError::SlowDown))); + assert!(non_trash_tmp_entries(&dirs).await.is_empty(), "rejection must not stage shards"); + drop(external_lock); + drop(occupied); + let mut reader = PutObjReader::from_vec(vec![9u8; TEST_OBJECT_SIZE]); + set.put_object(bucket, object, &mut reader, &ObjectOptions::default()) + .await + .unwrap(); + wait_for_tmp_workspace_to_drain(&dirs, "admitted PUT must eventually clean staging").await; + let mut actual = set + .get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .unwrap(); + let mut body = Vec::new(); + actual.stream.read_to_end(&mut body).await.unwrap(); + assert_eq!(body, vec![9u8; TEST_OBJECT_SIZE]); + } + #[tokio::test] async fn put_object_success_eventually_cleans_tmp_workspace() { let (temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await; @@ -20541,6 +22485,294 @@ mod put_object_tmp_cleanup_tests { .await; } + #[tokio::test] + async fn system_metadata_repair_error_does_not_wait_for_committed_tmp_cleanup() { + const CHILD: &str = "RUSTFS_METADATA_REPAIR_CLEANUP_TEST_CHILD"; + if std::env::var_os(CHILD).is_none() { + let output = tokio::task::spawn_blocking(|| { + std::process::Command::new(std::env::current_exe().expect("test executable must be available")) + .args([ + "--exact", + "set_disk::ops::object::put_object_tmp_cleanup_tests::system_metadata_repair_error_does_not_wait_for_committed_tmp_cleanup", + "--nocapture", + ]) + .env(CHILD, "1") + .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true") + .env("RUSTFS_PUT_RENAME_TAIL_CLEANUP_MAX_PENDING", "2") + .env("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKERS", "1") + .env("RUSTFS_CLEANUP_CPUS", "none") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP") + .output() + .expect("the isolated cleanup regression child must run") + }) + .await + .expect("cleanup regression child launcher must join"); + assert!( + output.status.success(), + "cleanup regression failed:\n{}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed")); + return; + } + + let (dirs, _disks, set) = hermetic_set_disks(4).await; + let occupied = cleanup_runtime::reserve() + .expect("the cleanup blocker must be admitted") + .expect("the child must enable cleanup isolation"); + let (ready, started) = tokio::sync::oneshot::channel(); + let (release, released) = tokio::sync::oneshot::channel(); + let blocker = cleanup_runtime::spawn(Some(occupied), async move { + ready.send(()).expect("the fixture must observe the cleanup worker"); + let _ = released.await; + }); + tokio::time::timeout(Duration::from_secs(30), started) + .await + .expect("the cleanup blocker must acquire the execution worker") + .expect("the blocker readiness sender must stay alive"); + + let object = "pool.bin.identity-cleanup-owner"; + let body = b"committed metadata with pending repair".to_vec(); + let fault = rename_fault_injection::fail_rename_on(object, &[0]); + let response = tokio::time::timeout( + Duration::from_secs(30), + crate::config::com::save_config_with_opts( + Arc::clone(&set), + object, + body.clone(), + &ObjectOptions { + max_parity: true, + write_completion: WriteCompletion::TailDrainedWithOwnerRepair, + ..Default::default() + }, + ), + ) + .await; + release.send(()).expect("the cleanup blocker must stay alive until released"); + tokio::time::timeout(Duration::from_secs(30), blocker) + .await + .expect("the released cleanup blocker must finish") + .expect("the cleanup blocker task must join"); + let error = response + .expect("repair-required response must not wait for the committed temporary cleanup owner") + .expect_err("the committed minority failure must reach the metadata writer"); + assert!( + matches!(error, Error::Io(ref inner) if inner.get_ref().is_some_and(|source| source.is::())) + ); + drop(fault); + wait_for_tmp_workspace_to_drain(&dirs, "the retained cleanup owner must eventually remove staged metadata").await; + let committed = crate::config::com::read_config(Arc::clone(&set), object) + .await + .expect("the repair-required metadata quorum must remain readable"); + assert_eq!(committed, body); + } + + #[tokio::test] + async fn rename_tail_heal_keeps_system_authority_separate_from_user_incarnation() { + let (_dirs, _disks, set) = hermetic_set_disks(4).await; + let mut heals = set.capture_test_rename_tail_heals(); + let incarnation = Uuid::new_v4(); + for bucket in [RUSTFS_META_BUCKET, ".minio.sys"] { + for expected in [None, Some(incarnation)] { + let mut request = rustfs_heal_contracts::heal_channel::create_heal_request_with_options( + bucket.to_string(), + Some("pool.bin.identity".to_string()), + false, + Some(HealChannelPriority::Normal), + Some(0), + Some(0), + ); + request.expected_bucket_incarnation_id = expected; + let error = set + .submit_rename_tail_heal(request, WriteCompletion::TailDrainedWithOwnerRepair) + .await + .expect_err("system repair must remain with its writer"); + assert!( + matches!(error, Error::Io(ref inner) if inner.get_ref().is_some_and(|source| source.is::())) + ); + assert!( + heals.try_recv().is_err(), + "even a supplied user incarnation cannot authorize system metadata heal" + ); + } + } + let mut request = rustfs_heal_contracts::heal_channel::create_heal_request_with_options( + "ordinary-bucket".to_string(), + Some("ordinary-object".to_string()), + false, + Some(HealChannelPriority::Normal), + Some(0), + Some(0), + ); + request.expected_bucket_incarnation_id = Some(incarnation); + set.submit_rename_tail_heal(request, WriteCompletion::Quorum) + .await + .expect("ordinary incarnation-bound fallback must remain available"); + let admitted = heals + .try_recv() + .expect("the ordinary fallback must actually reach the channel"); + assert_eq!(admitted.expected_bucket_incarnation_id, Some(incarnation)); + assert_eq!(admitted.source, rustfs_heal_contracts::heal_channel::HealRequestSource::Mrf); + assert_eq!(admitted.object_prefix.as_deref(), Some("ordinary-object")); + } + + #[tokio::test] + #[serial_test::serial(capacity_dirty_scope)] + async fn system_metadata_partial_write_respects_completion_contract() { + temp_env::async_with_vars([(ENV_RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE, Some("true"))], async { + for completion in [ + WriteCompletion::Quorum, + WriteCompletion::TailDrained, + WriteCompletion::TailDrainedWithOwnerRepair, + ] { + let (dirs, disks, set) = hermetic_set_disks(4).await; + let object = format!("pool.bin.identity-{completion:?}"); + let original = b"original deployment identity".to_vec(); + let fixture_options = ObjectOptions { + max_parity: true, + write_completion: WriteCompletion::TailDrained, + ..Default::default() + }; + crate::config::com::save_config_with_opts(Arc::clone(&set), &object, original.clone(), &fixture_options) + .await + .expect("healthy system metadata write must succeed without a bucket incarnation"); + let original_info = set + .get_object_info(RUSTFS_META_BUCKET, &object, &ObjectOptions::default()) + .await + .expect("original metadata must be readable before the partial overwrite"); + let original_etag = original_info.etag.expect("original metadata must have an ETag"); + let updated = b"updated deployment identity".to_vec(); + let mut heals = set.capture_test_rename_tail_heals(); + let tasks = rename_fanout_barrier::observe_tasks(&object); + let barrier = rename_fanout_barrier::arm(&object, 0, rename_fanout_barrier::PHASE_RENAME); + let fault = rename_fault_injection::fail_rename_on(&object, &[0]); + let writer = Arc::clone(&set); + let write_object = object.clone(); + let write_body = updated.clone(); + let write = tokio::spawn(async move { + crate::config::com::save_config_with_opts( + writer, + &write_object, + write_body, + &ObjectOptions { + max_parity: true, + write_completion: completion, + ..Default::default() + }, + ) + .await + }); + tokio::time::timeout(Duration::from_secs(30), barrier.wait_until_paused()) + .await + .expect("the metadata rename tail must reach the failure barrier"); + tokio::time::timeout(Duration::from_secs(30), async { + loop { + let mut updated_copies = 0; + for disk in &disks { + let info = disk + .read_version("", RUSTFS_META_BUCKET, &object, "", &ReadOptions::default()) + .await + .expect("each disk must retain either the original or updated metadata"); + if info.metadata.get("etag").is_some_and(|etag| etag != &original_etag) { + updated_copies += 1; + } + } + if updated_copies == 3 { + break; + } + tokio::task::yield_now().await; + } + }) + .await + .expect("the metadata overwrite must reach quorum while the last disk is paused"); + if completion != WriteCompletion::Quorum { + assert!( + !write.is_finished(), + "drained completion must wait for the rename tail before acknowledging" + ); + } + barrier.release(); + let result = tokio::time::timeout(Duration::from_secs(30), write) + .await + .expect("the metadata tail must drain") + .expect("the metadata writer task must join"); + if completion == WriteCompletion::TailDrainedWithOwnerRepair { + let error = result.expect_err("the pool metadata owner must receive repair responsibility"); + let Error::Io(error) = error else { + panic!("system metadata repair must retain its typed error"); + }; + assert!( + error + .get_ref() + .is_some_and(|source| source.is::()), + "the config writer must retain the exact repair-required cause: {error}" + ); + } else { + result.expect("ordinary system metadata must preserve quorum-success semantics"); + } + tokio::time::timeout(Duration::from_secs(30), async { + while tasks.running() != 0 { + tokio::task::yield_now().await; + } + }) + .await + .expect("the committed rename tail must eventually finish for every completion mode"); + assert_eq!(tasks.running(), 0, "repair failure must be returned after all rename tasks drain"); + assert!(heals.try_recv().is_err(), "system metadata must never enqueue an unfenced MRF heal"); + let mut stale_copies = 0; + let mut committed_etags = Vec::new(); + for disk in &disks { + let info = disk + .read_version("", RUSTFS_META_BUCKET, &object, "", &ReadOptions::default()) + .await + .expect("every physical disk must retain its metadata after the partial rename"); + let etag = info.metadata.get("etag").expect("each metadata copy must retain an ETag"); + if etag == &original_etag { + stale_copies += 1; + } else { + committed_etags.push(etag.clone()); + } + } + assert_eq!(stale_copies, 1, "exactly the failed rename must retain the old metadata"); + assert_eq!(committed_etags.len(), 3, "the new metadata must retain its write quorum"); + assert!(committed_etags.iter().all(|etag| etag == &committed_etags[0])); + let committed = + tokio::time::timeout(Duration::from_secs(30), crate::config::com::read_config(Arc::clone(&set), &object)) + .await + .expect("the committed metadata read must finish after the rename tail drains") + .expect("a repair-required result must not roll back the committed metadata quorum"); + assert_eq!(committed, updated, "the read must see the new bytes after cache invalidation"); + tokio::time::timeout(Duration::from_secs(30), async { + while !non_trash_tmp_entries(&dirs).await.is_empty() { + tokio::task::yield_now().await; + } + }) + .await + .expect("repair-required results must still schedule committed temporary-data cleanup"); + drop(fault); + drop(barrier); + tokio::time::timeout( + Duration::from_secs(30), + crate::config::com::save_config_with_opts(Arc::clone(&set), &object, updated.clone(), &fixture_options), + ) + .await + .expect("the reconciliation must finish after the fault and barrier are disarmed") + .expect("the owning writer must be able to reconcile after the failed disk recovers"); + for disk in &disks { + let info = disk + .read_version("", RUSTFS_META_BUCKET, &object, "", &ReadOptions::default()) + .await + .expect("the reconciled metadata must exist on every disk"); + assert_ne!(info.metadata.get("etag"), Some(&original_etag)); + } + } + }) + .await; + } + #[tokio::test] #[serial_test::serial(capacity_dirty_scope)] async fn tail_drained_put_preserves_quorum_success_without_unfenced_fallback_heal() { @@ -20816,6 +23048,42 @@ mod put_object_tmp_cleanup_tests { )); } + async fn borrowed_write_context_accepts(set: &SetDisks, bucket: &str, held: &str, target: &str) -> bool { + let guard = crate::object_api::WriteCommitGuard::acquire( + &set.new_ns_lock(bucket, held).await.expect("namespace wrapper"), + get_lock_acquire_timeout(), + ) + .await + .expect("outer write lock"); + let mut opts = ObjectOptions::default(); + opts.add_write_commit_guard(&guard); + matches!( + WriteCommitContext::from_options(&opts, bucket, target, false), + Ok(WriteCommitContext::Borrowed(_)) + ) + } + + #[cfg(not(windows))] + #[tokio::test] + async fn borrowed_write_context_keeps_backslash_spelling_distinct_off_windows() { + let (_dirs, _disks, set) = hermetic_set_disks(4).await; + assert!(borrowed_write_context_accepts(&set, "bucket", "a/b", "a/b").await); + assert!(!borrowed_write_context_accepts(&set, "bucket", "a/b", "a\\b").await); + assert!(!borrowed_write_context_accepts(&set, "bucket", "a\\b", "a/b").await); + } + + #[cfg(windows)] + #[tokio::test] + async fn windows_borrowed_write_context_accepts_backslash_spelling_of_held_object() { + // The guard records the folded lock key; a self-copy through `a\b` + // must still prove the namespace it commits into. + let (_dirs, _disks, set) = hermetic_set_disks(4).await; + assert!(borrowed_write_context_accepts(&set, "bucket", "a/b", "a\\b").await); + assert!(borrowed_write_context_accepts(&set, "bucket", "a\\b", "a\\b").await); + assert!(borrowed_write_context_accepts(&set, "bucket", "a\\b\\", "a/b").await); + assert!(!borrowed_write_context_accepts(&set, "bucket", "a/b", "a/b/c").await); + } + #[tokio::test] #[serial_test::serial(capacity_dirty_scope)] async fn borrowed_tail_drained_put_keeps_owner_after_caller_cancel() { @@ -22160,6 +24428,387 @@ mod delete_objects_lock_gating_tests { .expect("plain object should be written"); } + #[tokio::test] + #[serial_test::serial(shard_integrity_rollout)] + async fn conditional_delete_snapshot_preserves_multipart_etag() { + use crate::storage_api_contracts::multipart::MultipartOperations as _; + let (_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await; + let bucket = "conditional-multipart-etag"; + let object = "multipart.bin"; + for disk in &disk_stores { + disk.make_volume(bucket).await.expect("create bucket"); + } + let options = ObjectOptions::default(); + let upload = set_disks + .new_multipart_upload(bucket, object, &options) + .await + .expect("create multipart upload"); + let part = set_disks + .put_object_part( + bucket, + object, + &upload.upload_id, + 1, + &mut PutObjReader::from_vec(vec![11; 4097]), + &options, + ) + .await + .expect("upload the final part"); + let committed = set_disks + .clone() + .complete_multipart_upload( + bucket, + object, + &upload.upload_id, + vec![CompletePart { + part_num: part.part_num, + etag: part.etag, + ..Default::default() + }], + &options, + ) + .await + .expect("complete multipart object"); + let (snapshot, generation) = set_disks + .conditional_delete_current_snapshot( + bucket, + object, + &ObjectOptions { + no_lock: true, + metadata_cache_safe: false, + suppress_read_repair: true, + ..Default::default() + }, + ) + .await + .expect("read current multipart snapshot"); + let etag = committed.etag.as_deref().expect("multipart ETag"); + assert!(etag.ends_with("-1")); + assert_eq!(snapshot.etag, committed.etag); + let condition = crate::storage_api_contracts::object::DeleteObjectCondition::parse(&format!("\"{etag}\"")) + .expect("quoted response ETag"); + assert!(condition.matches(snapshot.etag.as_deref().expect("snapshot ETag"))); + assert!(matches!(generation, crate::disk::CurrentDeleteGeneration::Present { etag: stored, .. } if stored == etag)); + } + + #[tokio::test] + async fn guarded_delete_drive_capability_barrier_waits_for_success_and_cancels_on_error() { + use std::sync::atomic::{AtomicBool, AtomicUsize}; + use tokio_util::sync::CancellationToken; + + struct ProbeLifetime { + active: Arc, + dropped: Arc, + } + impl Drop for ProbeLifetime { + fn drop(&mut self) { + self.active.fetch_sub(1, Ordering::SeqCst); + self.dropped.fetch_add(1, Ordering::SeqCst); + } + } + + for unsupported in [false, true] { + let entered = Arc::new(AtomicUsize::new(0)); + let active = Arc::new(AtomicUsize::new(0)); + let dropped = Arc::new(AtomicUsize::new(0)); + let all_entered = Arc::new(tokio::sync::Notify::new()); + let completed = Arc::new(AtomicUsize::new(0)); + let two_completed = Arc::new(tokio::sync::Notify::new()); + let releases: Vec<_> = (0..3).map(|_| CancellationToken::new()).collect(); + let mutation_started = Arc::new(AtomicBool::new(false)); + let probes: Vec<_> = releases + .iter() + .enumerate() + .map(|(index, release)| { + let entered = Arc::clone(&entered); + let active = Arc::clone(&active); + let dropped = Arc::clone(&dropped); + let all_entered = Arc::clone(&all_entered); + let completed = Arc::clone(&completed); + let two_completed = Arc::clone(&two_completed); + let release = release.clone(); + async move { + active.fetch_add(1, Ordering::SeqCst); + let _lifetime = ProbeLifetime { active, dropped }; + if entered.fetch_add(1, Ordering::SeqCst) + 1 == 3 { + all_entered.notify_one(); + } + release.cancelled().await; + if completed.fetch_add(1, Ordering::SeqCst) + 1 == 2 { + two_completed.notify_one(); + } + if unsupported && index == 0 { + Err(DiskError::GuardedDeleteUnsupported) + } else { + Ok(()) + } + } + }) + .collect(); + let mutated = Arc::clone(&mutation_started); + let operation = tokio::spawn(async move { + guarded_delete_capability_barrier(probes).await?; + mutated.store(true, Ordering::SeqCst); + Ok::<(), StorageError>(()) + }); + tokio::time::timeout(std::time::Duration::from_secs(5), all_entered.notified()) + .await + .expect("all drive probes start concurrently"); + assert_eq!(entered.load(Ordering::SeqCst), 3); + assert_eq!(active.load(Ordering::SeqCst), 3); + assert!(!mutation_started.load(Ordering::SeqCst)); + releases[0].cancel(); + if !unsupported { + releases[1].cancel(); + tokio::time::timeout(std::time::Duration::from_secs(5), two_completed.notified()) + .await + .expect("both released probes complete"); + assert_eq!(completed.load(Ordering::SeqCst), 2); + assert!(!operation.is_finished(), "the last successful probe is required before admission"); + assert!(!mutation_started.load(Ordering::SeqCst), "unknown capability prevents mutation"); + releases[2].cancel(); + } + let result = tokio::time::timeout(std::time::Duration::from_secs(5), operation) + .await + .expect("a failed probe does not wait for unreleased peers") + .expect("probe task joins"); + assert_eq!(active.load(Ordering::SeqCst), 0, "no probe lifetime leaks after the barrier returns"); + assert_eq!(dropped.load(Ordering::SeqCst), 3, "completed and canceled probes release their resources"); + if unsupported { + assert_eq!(result, Err(StorageError::GuardedDeleteUnsupported)); + assert_eq!(completed.load(Ordering::SeqCst), 1, "unfinished peers are canceled rather than drained"); + assert!( + !releases[1].is_cancelled() && !releases[2].is_cancelled(), + "the error returns while other peers remain blocked" + ); + assert!(!mutation_started.load(Ordering::SeqCst), "unsupported capability prevents mutation"); + } else { + assert!(result.is_ok()); + assert_eq!(completed.load(Ordering::SeqCst), 3); + assert!(mutation_started.load(Ordering::SeqCst), "all successful probes admit mutation"); + } + } + } + + #[tokio::test] + async fn conditional_batch_delete_preserves_write_quorum_with_one_faulty_local_drive() { + use crate::disk::{CurrentDeleteGeneration, GuardedFileInfoVersions}; + let (_dirs, disks, set) = hermetic_set_disks(4).await; + let bucket = "guarded-faulty-quorum"; + let object = "object"; + for disk in &disks { + disk.make_volume(bucket).await.expect("create fixture volume"); + } + put_plain_object(&set, bucket, object).await; + let disk_info = disks[0] + .read_version(bucket, bucket, object, "", &crate::disk::ReadOptions::default()) + .await + .expect("read authoritative disk metadata"); + let (_, generation) = set + .conditional_delete_current_snapshot( + bucket, + object, + &ObjectOptions { + no_lock: true, + metadata_cache_safe: false, + suppress_read_repair: true, + ..Default::default() + }, + ) + .await + .expect("capture guarded generation"); + assert_eq!( + generation, + CurrentDeleteGeneration::from_file_info(&disk_info).expect("disk and set share a generation") + ); + let metadata_path = disks[0] + .path() + .join(bucket) + .join(object) + .join(crate::disk::STORAGE_FORMAT_FILE); + let before = tokio::fs::read(&metadata_path) + .await + .expect("read faulty drive's original metadata"); + disks[0].force_offline_for_test(); + set.preflight_guarded_delete_versions() + .await + .expect("local capability is known independently of drive health"); + let rejected_transaction = Uuid::new_v4(); + let rejected = disks[0] + .delete_versions_guarded_recoverable( + bucket, + vec![GuardedFileInfoVersions { + group: FileInfoVersions { + name: object.to_owned(), + versions: vec![FileInfo { + name: object.to_owned(), + version_id: disk_info.version_id, + ..Default::default() + }], + ..Default::default() + }, + expected_current: generation.clone(), + }], + DeleteOptions { + old_data_dir: Some(rejected_transaction), + ..Default::default() + }, + ) + .await; + assert_eq!( + rejected, + vec![Some(DiskError::FaultyDisk)], + "a known capability does not allow mutation on a faulty drive" + ); + let (_, errors) = set + .delete_objects( + bucket, + vec![ObjectToDelete { + object_name: object.to_owned(), + ..Default::default() + }], + ObjectOptions { + conditional_delete_generations: Some(Arc::new(HashMap::from([(object.to_owned(), generation)]))), + object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new( + ObjectLockConfigState::ConfirmedAbsent, + ))), + ..Default::default() + }, + ) + .await; + assert!( + errors[0].is_none(), + "three healthy drives satisfy the configured write quorum: {errors:?}" + ); + assert_eq!(tokio::fs::read(&metadata_path).await.expect("faulty drive is not mutated"), before); + assert!( + !metadata_path + .parent() + .expect("object directory") + .join(rejected_transaction.to_string()) + .exists() + ); + for disk in &disks[1..] { + assert!( + !disk + .path() + .join(bucket) + .join(object) + .join(crate::disk::STORAGE_FORMAT_FILE) + .exists(), + "healthy quorum drives apply the guarded deletion" + ); + } + assert!( + set.get_object_info(bucket, object, &ObjectOptions::default()) + .await + .err() + .is_some_and(|error| is_err_object_not_found(&error)), + "the healthy read quorum observes the completed delete" + ); + } + + #[tokio::test] + async fn conditional_batch_delete_rejects_unadmitted_conditions() { + let (_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await; + let bucket = "conditional-unadmitted"; + for disk in &disk_stores { + disk.make_volume(bucket).await.expect("create bucket"); + } + put_plain_object(&set_disks, bucket, "protected").await; + let (_, errors) = set_disks + .delete_objects( + bucket, + vec![ObjectToDelete { + object_name: "protected".to_owned(), + delete_condition: Some(crate::storage_api_contracts::object::DeleteObjectCondition::Exists), + ..Default::default() + }], + ObjectOptions { + object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new( + ObjectLockConfigState::ConfirmedAbsent, + ))), + ..Default::default() + }, + ) + .await; + assert!(matches!(&errors[0], Some(StorageError::InvalidArgument(_, _, _)))); + set_disks + .get_object_info(bucket, "protected", &ObjectOptions::default()) + .await + .expect("unadmitted condition must not delete"); + } + + #[tokio::test] + async fn conditional_batch_delete_disk_generation_rejects_overwrite_in_mixed_batch() { + let (_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await; + let bucket = "conditional-generation"; + for disk in &disk_stores { + disk.make_volume(bucket).await.expect("create bucket"); + } + put_plain_object(&set_disks, bucket, "protected").await; + put_plain_object(&set_disks, bucket, "ordinary").await; + let (_, generation) = set_disks + .conditional_delete_current_snapshot( + bucket, + "protected", + &ObjectOptions { + no_lock: true, + metadata_cache_safe: false, + suppress_read_repair: true, + ..Default::default() + }, + ) + .await + .expect("snapshot original generation"); + let current = set_disks + .put_object( + bucket, + "protected", + &mut PutObjReader::from_vec(b"new-current".to_vec()), + &ObjectOptions::default(), + ) + .await + .expect("overwrite after snapshot"); + let (_, errors) = set_disks + .delete_objects( + bucket, + vec![ + ObjectToDelete { + object_name: "protected".to_owned(), + ..Default::default() + }, + ObjectToDelete { + object_name: "ordinary".to_owned(), + ..Default::default() + }, + ], + ObjectOptions { + conditional_delete_generations: Some(Arc::new(HashMap::from([("protected".to_owned(), generation)]))), + object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new( + ObjectLockConfigState::ConfirmedAbsent, + ))), + ..Default::default() + }, + ) + .await; + assert_eq!(errors[0], Some(StorageError::ConditionalDeleteConflict)); + assert!(errors[1].is_none(), "ordinary group still succeeds: {errors:?}"); + let after = set_disks + .get_object_info(bucket, "protected", &ObjectOptions::default()) + .await + .expect("new data survives stale delete"); + assert_eq!(after.data_dir, current.data_dir); + assert_eq!(after.etag, current.etag); + assert!( + set_disks + .get_object_info(bucket, "ordinary", &ObjectOptions::default()) + .await + .err() + .is_some_and(|error| is_err_object_not_found(&error)) + ); + } + #[tokio::test] async fn delete_objects_reports_mixed_results_per_key() { let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await; @@ -22808,7 +25457,7 @@ mod delete_objects_lock_gating_tests { } put_plain_object(&set_disks, bucket, "obj-a").await; - let _outer_guard = set_disks + let outer_guard = set_disks .new_ns_lock(bucket, "obj-a") .await .expect("namespace lock should be created") @@ -22821,25 +25470,122 @@ mod delete_objects_lock_gating_tests { ..Default::default() }]; - let (deleted, errs) = tokio::time::timeout( - Duration::from_secs(1), - set_disks.delete_objects( - bucket, - objects, - ObjectOptions { - no_lock: true, - object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new( - ObjectLockConfigState::ConfirmedAbsent, - ))), - ..Default::default() - }, - ), - ) - .await - .expect("no_lock batch delete path must not wait for the outer lock"); + let lock_manager = set_disks + .local_lock_manager + .as_fast_lock_manager() + .expect("isolated fixture should use the local lock manager"); + let lock_metrics = lock_manager.get_metrics().shard_metrics; + assert_eq!(lock_manager.current_lock_counts(), (1, 0)); + let mut delete = Box::pin(set_disks.delete_objects( + bucket, + objects, + ObjectOptions { + no_lock: true, + object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new( + ObjectLockConfigState::ConfirmedAbsent, + ))), + ..Default::default() + }, + )); + // Observe lock attempts on every poll: durable disk work can take more + // than a second on CI without waiting for the namespace lock at all. + let deletion = futures::future::poll_fn(|cx| { + let result = std::future::Future::poll(delete.as_mut(), cx); + assert_eq!( + lock_manager.current_lock_counts(), + (1, 0), + "no_lock batch delete must preserve the outer holder without adding lock waiters" + ); + let current = lock_manager.get_metrics().shard_metrics; + assert_eq!( + (current.total_acquisitions(), current.timeouts), + (lock_metrics.total_acquisitions(), lock_metrics.timeouts), + "no_lock batch delete must not acquire or time out on another namespace lock" + ); + result + }); + let (deleted, errs) = tokio::time::timeout(Duration::from_secs(30), deletion) + .await + .expect("no_lock batch delete disk work should complete"); assert!(errs[0].is_none(), "no_lock batch delete should not fail with a lock error: {:?}", errs[0]); assert!(deleted[0].found, "existing object must still be deleted"); + for disk in &disk_stores { + assert_eq!( + disk.read_version("", bucket, "obj-a", "", &crate::disk::ReadOptions::default()) + .await + .expect_err("every drive must confirm the deleted version is absent"), + DiskError::FileNotFound + ); + } + drop(outer_guard); + } + + #[cfg(windows)] + #[tokio::test] + async fn windows_delete_objects_batch_waits_on_backslash_alias_lock() { + let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await; + let bucket = "batch-alias-lock-bucket"; + for disk in &disk_stores { + disk.make_volume(bucket).await.expect("bucket volume should be created"); + } + put_plain_object(&set_disks, bucket, "a/b").await; + + let outer_guard = set_disks + .new_ns_lock(bucket, "a/b") + .await + .expect("namespace lock should be created") + .get_write_lock(Duration::from_secs(1)) + .await + .expect("outer write lock should be acquired"); + let lock_manager = set_disks + .local_lock_manager + .as_fast_lock_manager() + .expect("isolated fixture should use the local lock manager"); + assert_eq!(lock_manager.current_lock_counts(), (1, 0)); + + let objects = vec![ObjectToDelete { + object_name: "a\\b".to_string(), + ..Default::default() + }]; + let mut delete = Box::pin(set_disks.delete_objects( + bucket, + objects, + ObjectOptions { + object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new( + ObjectLockConfigState::ConfirmedAbsent, + ))), + ..Default::default() + }, + )); + // `a\b` is the xl.meta of `a/b`: the batch must queue behind the outer + // holder instead of taking a second exclusive lock beside it. + let blocked = tokio::time::timeout( + Duration::from_millis(300), + futures::future::poll_fn(|cx| { + let result = std::future::Future::poll(delete.as_mut(), cx); + // Holders only: the queued batch shows up as a waiter. + assert_eq!( + lock_manager.current_lock_counts().0, + 1, + "batch delete through `a\\b` must not lock beside the `a/b` holder" + ); + result + }), + ) + .await; + assert!(blocked.is_err(), "batch delete through `a\\b` must wait for the `a/b` holder"); + + drop(outer_guard); + let (deleted, errs) = tokio::time::timeout(Duration::from_secs(30), delete) + .await + .expect("batch delete should finish once the holder releases"); + assert!(errs[0].is_none(), "batch delete should succeed after the holder releases: {:?}", errs[0]); + assert!(deleted[0].found, "`a\\b` must delete the object stored as `a/b`"); + set_disks + .get_object_info(bucket, "a/b", &ObjectOptions::default()) + .await + .expect_err("`a/b` must be gone after deleting `a\\b`"); } } @@ -23333,9 +26079,19 @@ mod marker_purge_receipt_tests { .acquire_write_lock_diag("marker_purge_receipt_test", bucket, object) .await .expect("own the deleted object namespace"); - set.delete_object_version_with_purge(bucket, object, &marker, true, None, Some(incarnation)) - .await - .expect("seed the real marker on every disk"); + set.delete_object_version_with_purge( + bucket, + object, + &marker, + true, + None, + DeleteCommitContext { + incarnation: Some(incarnation), + ..Default::default() + }, + ) + .await + .expect("seed the real marker on every disk"); let marker = disks[0] .read_version("", bucket, object, &version.to_string(), &ReadOptions::default()) .await @@ -23353,7 +26109,17 @@ mod marker_purge_receipt_tests { set.disks.write().await[0] = None; tokio::time::timeout( Duration::from_secs(10), - set.delete_object_version_with_purge(bucket, object, &request, false, Some(purge.clone()), Some(incarnation)), + set.delete_object_version_with_purge( + bucket, + object, + &request, + false, + Some(purge.clone()), + DeleteCommitContext { + incarnation: Some(incarnation), + ..Default::default() + }, + ), ) .await .expect("the receipt must not reacquire the deleted object's owner") @@ -23537,6 +26303,88 @@ mod single_delete_namespace_owner_tests { } } + #[cfg(not(windows))] + #[tokio::test] + async fn quorum_delete_cleanup_coordinator_binds_single_and_batch() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + + for batch in [false, true] { + let (_dirs, disks, set) = hermetic_set_disks_isolated(4).await; + let bucket = "quorum-cleanup-order"; + let object = "object"; + let version = Uuid::new_v4(); + for disk in &disks { + disk.make_volume(bucket).await.expect("create fixture bucket"); + } + seed_version(&set, bucket, object, version, &vec![0x51; 256 * 1024]).await; + let (observed, observations) = std::sync::mpsc::channel(); + let mut hooks = Vec::new(); + for disk in &disks { + let fi = disk + .read_version("", bucket, object, &version.to_string(), &ReadOptions::default()) + .await + .expect("read seeded version"); + assert!(!fi.inline_data(), "fixture must reach physical data cleanup"); + let crate::disk::Disk::Local(local) = disk.as_ref() else { + panic!("local fixture required"); + }; + let object_dir = local.get_disk().get_object_path_for_io(bucket, object).expect("object path"); + let data_dir = object_dir.join(fi.data_dir.expect("seeded data directory").to_string()); + let observed = observed.clone(); + hooks.push(hooks::install_at(hooks::Stage::Remove, &data_dir.join("part.1"), move || { + let backup_present = std::fs::read_dir(&object_dir) + .expect("object exists before payload unlink") + .any(|entry| { + entry + .expect("read transaction entry") + .path() + .join(crate::disk::STORAGE_FORMAT_FILE_BACKUP) + .exists() + }); + observed.send(backup_present).expect("record actual cleanup order"); + })); + } + if batch { + let (_, errors) = set + .delete_objects( + bucket, + vec![ObjectToDelete { + object_name: object.to_string(), + version_id: Some(version), + ..Default::default() + }], + ObjectOptions { + object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new( + ObjectLockConfigState::ConfirmedAbsent, + ))), + ..Default::default() + }, + ) + .await; + assert!(errors.iter().all(Option::is_none), "batch delete must commit: {errors:?}"); + } else { + set.delete_object_version( + bucket, + object, + &FileInfo { + name: object.to_string(), + version_id: Some(version), + ..Default::default() + }, + false, + ) + .await + .expect("single delete must commit"); + } + let observations = observations.try_iter().collect::>(); + assert_eq!(observations.len(), 4, "all four disks must reach cleanup: batch={batch}"); + assert!( + observations.iter().all(|backup_present| !backup_present), + "the coordinator must bind post-quorum cleanup before any payload unlink: batch={batch}" + ); + } + } + #[tokio::test] #[serial_test::serial] async fn internal_metadata_mutations_do_not_reenter_pool_metadata() { @@ -23635,6 +26483,54 @@ mod single_delete_namespace_owner_tests { ); } + #[tokio::test] + #[serial_test::serial(capacity_dirty_scope)] + async fn batch_delete_uses_one_namespace_owner_across_object_groups() { + let (dirs, disks, set) = hermetic_set_disks_isolated(4).await; + let bucket = "batch-delete-namespace"; + for disk in &disks { + disk.make_volume(bucket).await.expect("fixture bucket"); + } + let mut objects = Vec::new(); + for key in ["inline", "external"] { + let version = Uuid::new_v4(); + let body = vec![0x41; if key == "inline" { 4096 } else { 256 * 1024 }]; + seed_version(&set, bucket, key, version, &body).await; + objects.push(ObjectToDelete { + object_name: key.to_owned(), + version_id: Some(version), + ..Default::default() + }); + } + let before = set.ctx.namespace_commit_generation(); + assert!(!set.ctx.namespace_commits_pending()); + let (_, errors, _) = set + .delete_objects_with_accounting( + bucket, + objects, + ObjectOptions { + versioned: true, + object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new( + ObjectLockConfigState::ConfirmedAbsent, + ))), + ..Default::default() + }, + ) + .await; + assert!(errors.iter().all(Option::is_none), "batch groups must commit: {errors:?}"); + for dir in &dirs { + for key in ["inline", "external"] { + assert!(!dir.path().join(bucket).join(key).exists(), "all group evidence must converge"); + } + } + assert!(!set.ctx.namespace_commits_pending()); + assert_eq!( + set.ctx.namespace_commit_generation(), + before + 2, + "all batch groups share one complete owner lifetime" + ); + } + #[cfg(not(windows))] async fn assert_single_delete_physical_owner(case: &'static str) { use crate::disk::os::prepared_publication_test_hooks as hooks; @@ -23713,9 +26609,14 @@ mod single_delete_namespace_owner_tests { for (index, path) in metadata_paths.iter().enumerate() { let tx = entered_tx.clone(); let (release, rx) = std::sync::mpsc::channel::<()>(); - if last || cleanup { - let source = if cleanup { &cleanup_sources[index] } else { path }; - destination_guards.push(hooks::observe_rename_destination(source, move |destination| { + if cleanup { + let source = cleanup_sources[index].clone(); + guards.push(hooks::install_at(hooks::Stage::Remove, &source.join("part.1"), move || { + let _ = tx.send((index, source)); + let _ = rx.recv(); + })); + } else if last { + destination_guards.push(hooks::observe_rename_destination(path, move |destination| { let _ = tx.send((index, destination.to_path_buf())); let _ = rx.recv(); })); @@ -23788,6 +26689,12 @@ mod single_delete_namespace_owner_tests { } } } + if cleanup { + for (index, source) in cleanup_sources.iter().enumerate() { + assert_eq!(std::fs::read(source.join("part.1")).expect("paused original shard"), cleanup_parts[index], + "the paused physical cleanup must still own the exact original shard"); + } + } if rollback { tokio::time::pause(); tokio::time::advance(Duration::from_secs(61)).await; @@ -23864,11 +26771,7 @@ mod single_delete_namespace_owner_tests { if cleanup { assert!(!path.exists(), "metadata must remain deleted after cleanup cancellation"); assert!(!cleanup_sources[index].exists(), "physical cleanup must remove the shard directory"); - assert_eq!( - std::fs::read(physical_keys[&index].join("part.1")).expect("actual trashed shard"), - cleanup_parts[index], - "trash must contain the exact original shard" - ); + assert!(!physical_keys[&index].exists(), "the owned physical cleanup must remove its original shard directory"); continue; } if last { @@ -23978,3 +26881,446 @@ mod single_delete_namespace_owner_tests { assert_single_delete_physical_owner("rollback").await; } } + +#[cfg(test)] +mod cleanup_receipt_admission_tests { + use super::*; + + #[test] + fn receipt_admission_skips_only_work_that_the_writer_would_skip() { + let old = Uuid::new_v4(); + let current = Uuid::new_v4(); + let epoch = Uuid::new_v4(); + assert!(!SetDisks::needs_old_data_cleanup_receipt(old, Some(current), None)); + assert!(!SetDisks::needs_old_data_cleanup_receipt(old, Some(old), Some(epoch))); + assert!(SetDisks::needs_old_data_cleanup_receipt(old, Some(current), Some(epoch))); + assert!(SetDisks::needs_old_data_cleanup_receipt(old, None, Some(epoch))); + } +} + +#[cfg(test)] +mod crash_delete_retry_tests { + use super::*; + use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks}; + use crate::disk::STORAGE_FORMAT_FILE_BACKUP; + use crate::storage_api_contracts::bucket::{BucketOperations, DeleteBucketOptions, MakeBucketOptions}; + + async fn stage_interrupted_delete(store: &Arc, dirs: &[tempfile::TempDir], bucket: &str, object: &str, size: usize) { + let mut reader = PutObjReader::from_vec(vec![37; size]); + store + .put_object( + bucket, + object, + &mut reader, + &ObjectOptions { + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("write the object before simulating an interrupted delete"); + let transaction = Uuid::new_v4(); + for dir in dirs { + let root = dir.path().join(bucket).join(object); + let xl = root.join(STORAGE_FORMAT_FILE); + let bytes = tokio::fs::read(&xl).await.expect("read original object metadata"); + let mut metadata = FileMeta::default(); + metadata.unmarshal_msg(&bytes).expect("decode original object metadata"); + let info = metadata.versions[0] + .into_fileinfo(bucket, object, false) + .expect("read original version"); + let backup = root.join(transaction.to_string()); + tokio::fs::create_dir_all(&backup) + .await + .expect("create transaction directory"); + tokio::fs::write(backup.join(STORAGE_FORMAT_FILE_BACKUP), bytes) + .await + .expect("stage intact rollback backup"); + if let Some(data_dir) = info.data_dir { + let parts = root.join(data_dir.to_string()); + if parts.is_dir() { + tokio::fs::write( + parts.join(format!("{}{}", crate::disk::local::RESERVED_DELETE_DATA_DIR_MARKER_PREFIX, transaction)), + [], + ) + .await + .expect("reserve external deletion before its local commit"); + } + } + tokio::fs::remove_file(&xl) + .await + .expect("simulate crash after removing metadata"); + } + } + + #[tokio::test] + async fn crash_delete_retry_single_reclaims_inline_backup_only() { + let (dirs, store) = isolated_store_over_temp_disks().await; + let bucket = "crash-delete-single"; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create authoritative bucket"); + stage_interrupted_delete(&store, &dirs, bucket, "inline", 4096).await; + store + .delete_object(bucket, "inline", ObjectOptions::default()) + .await + .expect("retry should reclaim the exact missing key"); + for dir in &dirs { + assert!( + !dir.path().join(bucket).join("inline").exists(), + "inline backup residue must be removed on every disk" + ); + } + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("the empty bucket must be deletable without force"); + } + + #[tokio::test] + async fn crash_delete_retry_batch_reclaims_inline_and_reserved_external() { + let (dirs, store) = isolated_store_over_temp_disks().await; + let bucket = "crash-delete-batch"; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create authoritative bucket"); + stage_interrupted_delete(&store, &dirs, bucket, "inline", 4096).await; + stage_interrupted_delete(&store, &dirs, bucket, "external", 1024 * 1024).await; + let objects = ["inline", "external"] + .into_iter() + .map(|name| ObjectToDelete { + object_name: name.to_string(), + ..Default::default() + }) + .collect(); + let (_, errors) = store.delete_objects(bucket, objects, ObjectOptions::default()).await; + assert!( + errors.iter().all(Option::is_none), + "retry must complete for both crash shapes: {errors:?}" + ); + for dir in &dirs { + for object in ["inline", "external"] { + assert!(!dir.path().join(bucket).join(object).exists(), "retry must remove {object} on every disk"); + } + } + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("batch retry must unblock non-force bucket deletion"); + } + + #[tokio::test] + async fn crash_delete_retry_preserves_conditional_and_versioned_residue() { + let (dirs, store) = isolated_store_over_temp_disks().await; + let bucket = "crash-delete-gates"; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create authoritative bucket"); + stage_interrupted_delete(&store, &dirs, bucket, "inline", 4096).await; + for opts in [ + ObjectOptions { + http_preconditions: Some(HTTPPreconditions { + if_match: Some("old-etag".to_string()), + ..Default::default() + }), + ..Default::default() + }, + ObjectOptions { + version_id: Some(Uuid::new_v4().to_string()), + ..Default::default() + }, + ] { + let _ = store.delete_object(bucket, "inline", opts).await; + for dir in &dirs { + assert!( + dir.path().join(bucket).join("inline").exists(), + "conditional/version-specific deletes must retain ambiguous backups" + ); + } + } + } + #[tokio::test] + async fn crash_delete_retry_preserves_suspended_bucket_null_backup() { + let (dirs, store) = isolated_store_over_temp_disks().await; + let bucket = "crash-delete-suspended"; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create authoritative bucket"); + stage_interrupted_delete(&store, &dirs, bucket, "inline", 4096).await; + metadata_sys::update_in(&store.ctx, bucket, crate::bucket::metadata::BUCKET_VERSIONING_CONFIG, + b"Suspended".to_vec()) + .await.expect("persist suspended versioning state"); + let _ = store.delete_object(bucket, "inline", ObjectOptions::default()).await; + for dir in &dirs { + assert!( + dir.path().join(bucket).join("inline").exists(), + "authoritative suspended state must protect a null-version backup" + ); + } + } + + #[tokio::test] + async fn crash_delete_retry_rejects_stale_bucket_incarnation_at_set_boundary() { + let (dirs, store) = isolated_store_over_temp_disks().await; + let bucket = "crash-delete-incarnation"; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create authoritative bucket"); + stage_interrupted_delete(&store, &dirs, bucket, "inline", 4096).await; + let error = store.pools[0].disk_set[0] + .delete_object( + bucket, + "inline", + ObjectOptions { + expected_bucket_incarnation_id: Some(Uuid::new_v4()), + ..Default::default() + }, + ) + .await + .expect_err("the set must reject a stale authorized bucket identity"); + assert!( + matches!(error, StorageError::BucketNotFound(_)), + "identity mismatch must retain its typed error: {error:?}" + ); + for dir in &dirs { + assert!( + dir.path().join(bucket).join("inline").exists(), + "a stale request must leave every backup intact" + ); + } + } + #[tokio::test] + #[serial_test::serial(shard_integrity_rollout)] + async fn batch_delete_completed_multipart_reclaims_native_part_sidecars() { + assert_completed_multipart_delete_reclaims_sidecars(false).await; + } + + #[tokio::test] + #[serial_test::serial(shard_integrity_rollout)] + async fn batch_delete_completed_multipart_reclaims_protected_part_sidecars() { + assert_completed_multipart_delete_reclaims_sidecars(true).await; + } + + async fn assert_completed_multipart_delete_reclaims_sidecars(protected: bool) { + temp_env::async_with_vars( + [ + (rustfs_config::ENV_SHARD_INTEGRITY_WRITE, Some(if protected { "true" } else { "false" })), + ( + rustfs_config::ENV_SHARD_INTEGRITY_FLEET_CONFIRMED, + Some(if protected { "true" } else { "false" }), + ), + ], + assert_completed_multipart_delete_reclaims_sidecars_inner(protected), + ) + .await; + } + + async fn assert_completed_multipart_delete_reclaims_sidecars_inner(protected: bool) { + let (dirs, store) = isolated_store_over_temp_disks().await; + let bucket = "multipart-delete-cleanup"; + let object = "completed-object"; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let upload = store + .new_multipart_upload(bucket, object, &ObjectOptions::default()) + .await + .expect("create native multipart upload"); + let mut reader = PutObjReader::from_vec(vec![37; 6 * 1024 * 1024]); + let part = store + .put_object_part(bucket, object, &upload.upload_id, 1, &mut reader, &ObjectOptions::default()) + .await + .expect("write native part"); + store + .clone() + .complete_multipart_upload( + bucket, + object, + &upload.upload_id, + vec![CompletePart { + part_num: part.part_num, + etag: part.etag, + ..Default::default() + }], + &ObjectOptions::default(), + ) + .await + .expect("complete native multipart upload"); + for dir in &dirs { + let bytes = tokio::fs::read(dir.path().join(bucket).join(object).join(STORAGE_FORMAT_FILE)) + .await + .expect("completed metadata"); + let mut metadata = FileMeta::default(); + metadata.unmarshal_msg(&bytes).expect("decode completed metadata"); + let mut info = metadata.versions[0] + .into_fileinfo(bucket, object, true) + .expect("completed version"); + info.hydrate_shard_integrity() + .expect("decode native completed integrity descriptor"); + let path = dir + .path() + .join(bucket) + .join(object) + .join(info.data_dir.expect("external multipart directory").to_string()); + assert!(path.join("part.1.meta").exists(), "real multipart completion retains its native sidecar"); + let integrity = info.parts[0].integrity.as_ref(); + assert_eq!(integrity.is_some(), protected, "completion must retain the selected native format"); + if let Some(integrity) = integrity { + assert!( + path.join(integrity.file_name()).exists(), + "real protected completion retains its integrity proof" + ); + } + } + let (_, errors) = store + .delete_objects( + bucket, + vec![ObjectToDelete { + object_name: object.to_string(), + ..Default::default() + }], + ObjectOptions::default(), + ) + .await; + assert!(errors.iter().all(Option::is_none), "native batch delete must succeed: {errors:?}"); + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("successful multipart delete must permit ordinary bucket removal"); + for dir in &dirs { + assert!( + !dir.path().join(bucket).exists(), + "bucket and completed part sidecars must be removed on every disk" + ); + } + } +} + +#[cfg(test)] +mod delete_recovery_inventory_tests { + use super::*; + use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks}; + use crate::storage_api_contracts::bucket::{BucketOperations, MakeBucketOptions}; + use std::future::Future; + use std::task::Poll; + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial_test::serial] + async fn batch_delete_completes_with_queued_disk_renewal() { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let set = store.pools[0].disk_set[0].clone(); + let disks = set.disk_inventory().await.into_iter().flatten().collect::>(); + let bucket = "batch-delete-renew"; + let object = "existing-inline-version"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create authoritative real bucket"); + store + .put_object( + bucket, + object, + &mut PutObjReader::from_vec(vec![0x73; 4096]), + &ObjectOptions { + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("seed actual version"); + let endpoint = set.set_endpoints[0].clone(); + let ctx = crate::runtime::global::current_ctx(); + let was_dist = ctx.is_dist_erasure().await; + if was_dist { + ctx.update_erasure_type(crate::layout::endpoints::SetupType::Erasure).await; + } + let map = ctx.local_disk_map(); + let previous = map.read().await.get(&endpoint.to_string()).cloned(); + + let barrier = DeleteCleanupBarrier::install(bucket); + let delete_store = store.clone(); + let mut deletion = tokio::spawn(async move { + delete_store + .delete_objects( + bucket, + vec![ObjectToDelete { + object_name: object.to_owned(), + ..Default::default() + }], + ObjectOptions::default(), + ) + .await + }); + barrier.wait_until_paused().await; + let extra_gate = set.disks.read().await; + let mut renewal = Box::pin(set.renew_disk(&endpoint)); + tokio::time::timeout( + Duration::from_secs(5), + futures::future::poll_fn(|cx| { + assert!( + std::pin::pin!(tokio::task::unconstrained(renewal.as_mut())) + .poll(cx) + .is_pending(), + "real renewal must reach inventory write" + ); + if set.disks.try_read().is_err() { + Poll::Ready(()) + } else { + Poll::Pending + } + }), + ) + .await + .expect("queue real renewal writer"); + let connected = map + .read() + .await + .get(&endpoint.to_string()) + .cloned() + .flatten() + .expect("renewal registered new handle"); + assert!(!Arc::ptr_eq(&connected, &disks[0])); + drop(extra_gate); + barrier.release(); + let outcome = tokio::time::timeout(Duration::from_secs(3), async { tokio::join!(renewal.as_mut(), &mut deletion) }).await; + if outcome.is_err() { + deletion.abort(); + } + drop(renewal); + if outcome.is_err() { + let _ = deletion.await; + } + { + let mut map = map.write().await; + match previous { + Some(old) => { + map.insert(endpoint.to_string(), old); + } + None => { + map.remove(&endpoint.to_string()); + } + } + } + if was_dist { + ctx.update_erasure_type(crate::layout::endpoints::SetupType::DistErasure) + .await; + } + let (_, result) = + outcome.expect("batch DELETE and real queued disk renewal must finish without nested inventory-read deadlock"); + let (_, errors) = result.expect("delete task must finish"); + assert!(errors.iter().all(Option::is_none), "DELETE result must remain successful: {errors:?}"); + } +} diff --git a/crates/ecstore/src/set_disk/read.rs b/crates/ecstore/src/set_disk/read.rs index c91a16db0..e5a96d57c 100644 --- a/crates/ecstore/src/set_disk/read.rs +++ b/crates/ecstore/src/set_disk/read.rs @@ -14,18 +14,20 @@ use super::{ Arc, Bytes, DiskError, DiskOption, DiskStore, Endpoint, ErasureCache, Error, FileInfo, GetCodecStreamingFallbackReason, - GetObjectFileInfo, GetObjectMetadataCacheEntry, GetObjectMetadataCacheGeneration, GetObjectMetadataCacheKey, - GetObjectReadPolicy, HashAlgorithm, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, OBJECT_OP_IGNORED_ERRS, ObjectInfo, - ObjectOptions, RUSTFS_META_BUCKET, ReadOptions, Result, SetDisks, StorageError, adaptive_duplex_buffer_size, - build_get_codec_streaming_decode_engine, build_inline_bitrot_readers_from_refs, collect_inline_data_shard_fileinfos_by_index, - debug, error, get_codec_streaming_metrics_path, get_codec_streaming_multipart_max_parts, get_object_read_policy, - is_codec_streaming_multipart_enabled, is_multipart_reader_setup_prefetch_enabled, object_fits_single_block, - reduce_read_quorum_errs, to_object_err, try_read_inline_data_shards_direct, warn, + GetObjectFileInfo, GetObjectMetadataCacheCapability, GetObjectMetadataCacheEntry, GetObjectMetadataCacheGeneration, + GetObjectMetadataCacheKey, GetObjectMetadataCacheLookupKey, GetObjectReadPolicy, HashAlgorithm, LOG_COMPONENT_ECSTORE, + LOG_SUBSYSTEM_SET_DISK, OBJECT_OP_IGNORED_ERRS, ObjectInfo, ObjectOptions, RUSTFS_META_BUCKET, ReadOptions, Result, SetDisks, + StorageError, adaptive_duplex_buffer_size, build_get_codec_streaming_decode_engine, build_inline_bitrot_readers_from_refs, + collect_inline_data_shard_fileinfos_by_index, debug, error, get_codec_streaming_metrics_path, + get_codec_streaming_multipart_max_parts, get_object_read_policy, is_codec_streaming_multipart_enabled, + is_multipart_reader_setup_prefetch_enabled, object_fits_single_block, reduce_read_quorum_errs, to_object_err, + try_read_inline_data_shards_direct, warn, }; use crate::diagnostics::get::{ GET_DIRECT_MEMORY_SUBPATH_DISK_DATA_BLOCKS, GET_DIRECT_MEMORY_SUBPATH_INLINE_BUFFERED, GET_METADATA_CACHE_DECISION_HIT, GET_METADATA_CACHE_DECISION_MISS, GET_METADATA_CACHE_DECISION_REJECT, GET_METADATA_CACHE_DECISION_SKIP, - GET_METADATA_CACHE_REASON_DATA_MOVEMENT, GET_METADATA_CACHE_REASON_DELETE_MARKER, GET_METADATA_CACHE_REASON_DIST_ERASURE, + GET_METADATA_CACHE_REASON_BODY_LAYOUT_REQUIRED, GET_METADATA_CACHE_REASON_DATA_MOVEMENT, + GET_METADATA_CACHE_REASON_DELETE_MARKER, GET_METADATA_CACHE_REASON_DIST_ERASURE, GET_METADATA_CACHE_REASON_INCL_FREE_VERSIONS, GET_METADATA_CACHE_REASON_INSUFFICIENT_CACHED_QUORUM, GET_METADATA_CACHE_REASON_META_BUCKET, GET_METADATA_CACHE_REASON_NO_LOCK, GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED, GET_METADATA_CACHE_REASON_NOT_READ_DATA, GET_METADATA_CACHE_REASON_PART_CHECKSUMS, GET_METADATA_CACHE_REASON_PART_NUMBER, @@ -206,6 +208,16 @@ use tokio::time::timeout; #[cfg(test)] use uuid::Uuid; +fn metadata_quorum_outcome(error: &StorageError) -> &'static str { + if error.is_not_found() { + "not_found" + } else if error.is_quorum_error() { + "not_reached" + } else { + "unavailable" + } +} + pub(super) struct GetObjectDownstreamWriter { inner: W, } @@ -246,6 +258,12 @@ impl AsyncWrite for GetObjectDownstreamWriter { } } +#[derive(Clone, Copy, PartialEq, Eq)] +enum MetadataReadPurpose { + FullLayout, + GetObjectReader, +} + impl SetDisks { /// A startup quorum can leave remote slots unregistered even after their /// peers become ready. Refresh those slots before taking the GET metadata @@ -392,7 +410,7 @@ impl SetDisks { let Some(generation) = self.get_object_metadata_cache_generation(bucket, object) else { return MetadataCacheLookup::Miss; }; - let key = GetObjectMetadataCacheKey::new(bucket, object, generation); + let key = GetObjectMetadataCacheLookupKey::new(bucket, object, generation); // moka handles TTL expiry automatically; no is_fresh() check needed let Some(entry) = self.get_object_metadata_cache.get(&key).await else { return MetadataCacheLookup::Miss; @@ -413,11 +431,12 @@ impl SetDisks { &self, identity: (&str, &str), generation: Option, - fi: &FileInfo, + selected: (&FileInfo, GetObjectMetadataCacheCapability), parts_metadata: &[FileInfo], online_disks: &[Option], read_quorum: usize, ) { + let (fi, capability) = selected; if fi.deleted || !fi.has_valid_erasure_geometry() { return; } @@ -440,6 +459,7 @@ impl SetDisks { parts_metadata: parts_metadata.to_vec(), online_disks: online_disks.to_vec(), read_quorum, + capability, }); self.insert_get_object_metadata_cache_entry_after_insert(key, generation, entry, || {}) .await; @@ -452,7 +472,25 @@ impl SetDisks { entry: Arc, after_insert: impl FnOnce(), ) { - self.get_object_metadata_cache.insert(key.clone(), entry).await; + // Every publisher shares Moka's per-key compute lock. A partial snapshot + // cannot downgrade a full layout or extend its original TTL. + self.get_object_metadata_cache + .entry(key.clone()) + .and_compute_with(|current| { + std::future::ready( + if !self.is_get_object_metadata_cache_generation_current(generation) + || (entry.capability != GetObjectMetadataCacheCapability::Full + && current + .as_ref() + .is_some_and(|existing| existing.value().capability == GetObjectMetadataCacheCapability::Full)) + { + moka::ops::compute::Op::Nop + } else { + moka::ops::compute::Op::Put(entry) + }, + ) + }) + .await; after_insert(); if self.is_get_object_metadata_cache_generation_current(generation) { return; @@ -481,6 +519,10 @@ impl SetDisks { let object: Arc = Arc::from(object); let version_id: Arc = Arc::from(version_id); let opts = *opts; + #[cfg(feature = "diagnostic-read-version-caller")] + let api_operation = rustfs_io_metrics::get_stage_metrics_enabled() + .then(rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_context) + .flatten(); let processor = runtime_sources::batch_processors().read_processor(); let tasks: Vec<_> = disks @@ -492,8 +534,28 @@ impl SetDisks { let object = object.clone(); let version_id = version_id.clone(); let task_opts = opts; + #[cfg(feature = "diagnostic-read-version-caller")] + #[allow(clippy::redundant_locals)] + let api_operation = api_operation; - async move { disk.read_version(&bucket, &bucket, &object, &version_id, &task_opts).await } + async move { + let read_version = + async move { disk.read_version(&bucket, &bucket, &object, &version_id, &task_opts).await }; + #[cfg(feature = "diagnostic-read-version-caller")] + let read_version = crate::cluster::rpc::runtime_sources::with_read_version_request_caller_if_absent( + rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::SetDiskRead, + read_version, + ); + #[cfg(feature = "diagnostic-read-version-caller")] + if let Some(api_operation) = api_operation { + rustfs_io_metrics::s3_http_metrics::with_s3_http_operation_context(Some(api_operation), read_version) + .await + } else { + read_version.await + } + #[cfg(not(feature = "diagnostic-read-version-caller"))] + read_version.await + } }) }) .collect(); @@ -514,8 +576,15 @@ impl SetDisks { read_data: bool, caller_allows_early_stop: bool, ) -> Result { - self.get_object_fileinfo_gated_inner(bucket, object, opts, read_data, caller_allows_early_stop, false) - .await + self.get_object_fileinfo_gated_inner( + bucket, + object, + opts, + read_data, + caller_allows_early_stop, + MetadataReadPurpose::FullLayout, + ) + .await } #[tracing::instrument(level = "debug", skip(self))] @@ -528,15 +597,13 @@ impl SetDisks { read_data: bool, caller_allows_early_stop: bool, ) -> Result { - let allow_read_version_coalescing = !crate::bucket::utils::is_meta_bucketname(bucket) - && crate::runtime::global::get_metadata_read_version_coalescing_service_ready(); self.get_object_fileinfo_gated_inner( bucket, object, opts, read_data, caller_allows_early_stop, - allow_read_version_coalescing, + MetadataReadPurpose::GetObjectReader, ) .await } @@ -554,7 +621,7 @@ impl SetDisks { read_data: bool, allow_early_stop: bool, ) -> Result { - self.get_object_fileinfo_gated_inner(bucket, object, opts, read_data, allow_early_stop, false) + self.get_object_fileinfo_gated_inner(bucket, object, opts, read_data, allow_early_stop, MetadataReadPurpose::FullLayout) .await } @@ -566,9 +633,14 @@ impl SetDisks { opts: &ObjectOptions, read_data: bool, allow_early_stop: bool, - allow_read_version_coalescing: bool, + purpose: MetadataReadPurpose, ) -> Result { let vid = opts.version_id.clone().unwrap_or_default(); + let metadata_metrics_path = if crate::bucket::utils::is_meta_bucketname(bucket) { + GET_OBJECT_PATH_INTERNAL_META + } else { + GET_OBJECT_PATH_LEGACY_DUPLEX + }; let stage_metrics_enabled = rustfs_io_metrics::get_stage_metrics_enabled(); let metadata_cache_lookup_start = get_stage_timer_if_enabled(stage_metrics_enabled); @@ -582,7 +654,19 @@ impl SetDisks { ); } else if vid.is_empty() { match self.lookup_cached_get_object_fileinfo(bucket, object).await { - MetadataCacheLookup::Hit(cached) => { + MetadataCacheLookup::Hit(cached) + if cached.capability == GetObjectMetadataCacheCapability::Full + || (cached.capability == GetObjectMetadataCacheCapability::QuorumSelected + && read_data + && allow_early_stop) + || (purpose == MetadataReadPurpose::GetObjectReader + && cached.capability == GetObjectMetadataCacheCapability::ConditionOnly + && allow_early_stop + && crate::object_api::get_object_read_condition_is_terminal( + crate::object_api::GetObjectReadMetadata::from_file_info(bucket, object, &cached.fi), + )) => + { + rustfs_io_metrics::record_get_object_metadata_quorum_result(metadata_metrics_path, "reached"); rustfs_io_metrics::record_get_object_metadata_cache_decision( GET_OBJECT_PATH_SET_DISK, GET_METADATA_CACHE_DECISION_HIT, @@ -593,7 +677,24 @@ impl SetDisks { GET_STAGE_METADATA_CACHE_LOOKUP, metadata_cache_lookup_start, ); - return Ok(GetObjectFileInfo::shared(cached)); + let snapshot = if cached.capability == GetObjectMetadataCacheCapability::QuorumSelected + && non_inline_data_read_early_stop_allowed(read_data, bucket, object) + && late_materialization_candidate_is_safe(&cached.fi) + { + GetObjectFileInfo::shared_with_late_metadata_refresh(cached) + } else { + GetObjectFileInfo::shared(cached) + }; + return Ok(snapshot); + } + MetadataCacheLookup::Hit(_) => { + // Keep the conditional entry for other requests; this read + // needs a body or a complete mutation target set. + rustfs_io_metrics::record_get_object_metadata_cache_decision( + GET_OBJECT_PATH_SET_DISK, + GET_METADATA_CACHE_DECISION_REJECT, + GET_METADATA_CACHE_REASON_BODY_LAYOUT_REQUIRED, + ); } MetadataCacheLookup::Miss => { rustfs_io_metrics::record_get_object_metadata_cache_decision( @@ -631,41 +732,59 @@ impl SetDisks { // read_metadata_observed (see read_all_fileinfo_early_stop in // core/metadata_read.rs); unsafe requests and callers that opt out // (allow_early_stop=false) fall back to full-wait. - let metadata_read = if allow_read_version_coalescing { - Self::read_metadata_for_get_object( - &disks, - "", - bucket, - object, - vid.as_str(), - read_data, - opts.incl_free_versions, - allow_early_stop, - self.default_parity_count, - ) - .await? - } else { - Self::read_metadata_observed( - &disks, - "", - bucket, - object, - vid.as_str(), - read_data, - false, - opts.incl_free_versions, - allow_early_stop, - self.default_parity_count, - ) - .await? + let allow_read_version_coalescing = purpose == MetadataReadPurpose::GetObjectReader + && !crate::bucket::utils::is_meta_bucketname(bucket) + && crate::runtime::global::get_metadata_read_version_coalescing_service_ready(); + let metadata_read = async { + if allow_read_version_coalescing { + Self::read_metadata_for_get_object( + &disks, + "", + bucket, + object, + vid.as_str(), + read_data, + opts.incl_free_versions, + allow_early_stop, + self.default_parity_count, + ) + .await + } else { + Self::read_metadata_observed( + &disks, + "", + bucket, + object, + vid.as_str(), + read_data, + false, + opts.incl_free_versions, + allow_early_stop, + self.default_parity_count, + ) + .await + } + }; + #[cfg(feature = "diagnostic-read-version-caller")] + let metadata_read_result = crate::cluster::rpc::runtime_sources::with_read_version_caller_and_entry_if_absent( + rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::Object, + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ObjectRead, + metadata_read, + ) + .await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] + let metadata_read_result = metadata_read.await; + let metadata_read = match metadata_read_result { + Ok(metadata_read) => metadata_read, + Err(err) => { + let err = StorageError::from(err); + rustfs_io_metrics::record_get_object_metadata_quorum_result(metadata_metrics_path, metadata_quorum_outcome(&err)); + return Err(err); + } }; let metadata_fanout_complete = metadata_read.is_complete(); + let condition_stopped = metadata_read.condition_stopped; let (mut parts_metadata, errs, metadata_fanout_diagnostics) = metadata_read.into_legacy(); - let metadata_metrics_path = if crate::bucket::utils::is_meta_bucketname(bucket) { - GET_OBJECT_PATH_INTERNAL_META - } else { - GET_OBJECT_PATH_LEGACY_DUPLEX - }; metadata_fanout_diagnostics.record(metadata_metrics_path); // warn!("get_object_fileinfo parts_metadata {:?}", &parts_metadata); // warn!("get_object_fileinfo {}/{} errs {:?}", bucket, object, &errs); @@ -678,6 +797,7 @@ impl SetDisks { { Ok(v) => v, Err(e) => { + rustfs_io_metrics::record_get_object_metadata_quorum_result(metadata_metrics_path, metadata_quorum_outcome(&e)); // error!("Self::object_quorum_from_meta: {:?}, bucket: {}, object: {}", &e, bucket, object); record_get_stage_duration_if_enabled( GET_OBJECT_PATH_SET_DISK, @@ -692,6 +812,7 @@ impl SetDisks { let write_quorum = usize::try_from(write_quorum) .map_err(|_| to_object_err(DiskError::ErasureWriteQuorum.into(), vec![bucket, object]))?; if let Some(err) = reduce_read_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, read_quorum) { + rustfs_io_metrics::record_get_object_metadata_quorum_result(metadata_metrics_path, "not_reached"); error!("reduce_read_quorum_errs: {:?}, bucket: {}, object: {}", &err, bucket, object); record_get_stage_duration_if_enabled( GET_OBJECT_PATH_SET_DISK, @@ -702,7 +823,17 @@ impl SetDisks { } let (op_online_disks, mut fi, fileinfo_selection_quorum) = - Self::select_valid_fileinfo(&disks, &parts_metadata, &errs, vid.as_str(), read_quorum, write_quorum)?; + match Self::select_valid_fileinfo(&disks, &parts_metadata, &errs, vid.as_str(), read_quorum, write_quorum) { + Ok(selection) => selection, + Err(err) => { + let err = StorageError::from(err); + rustfs_io_metrics::record_get_object_metadata_quorum_result( + metadata_metrics_path, + metadata_quorum_outcome(&err), + ); + return Err(err); + } + }; let include_part_checksums = opts.include_part_checksums || opts.part_number.is_some() || opts.data_movement || opts.raw_data_movement_read; if include_part_checksums { @@ -725,13 +856,31 @@ impl SetDisks { "metadata_read_error", ) .await; - } else if use_metadata_cache && metadata_fanout_complete { + } else if use_metadata_cache + && (metadata_fanout_complete + || condition_stopped + || (read_data + && allow_early_stop + && metadata_fanout_diagnostics.valid_responses() >= read_quorum + && errs.iter().all(Option::is_none) + && fileinfo_selection_quorum >= read_quorum + && op_online_disks.iter().filter(|disk| disk.is_some()).count() >= read_quorum)) + { #[cfg(test)] metadata_cache_tests::wait_before_metadata_cache_publish(bucket, object).await; self.cache_get_object_fileinfo( (bucket, object), metadata_cache_generation, - &fi, + ( + &fi, + if metadata_fanout_complete { + GetObjectMetadataCacheCapability::Full + } else if condition_stopped { + GetObjectMetadataCacheCapability::ConditionOnly + } else { + GetObjectMetadataCacheCapability::QuorumSelected + }, + ), &parts_metadata, &op_online_disks, read_quorum, @@ -743,20 +892,17 @@ impl SetDisks { // let online_disks: Vec> = op_online_disks.iter().filter(|v| v.is_some()).cloned().collect(); - if !metadata_fanout_complete + let mut snapshot = if !metadata_fanout_complete && allow_early_stop && non_inline_data_read_early_stop_allowed(read_data, bucket, object) && late_materialization_candidate_is_safe(&fi) { - Ok(GetObjectFileInfo::owned_with_late_metadata_fanout( - fi, - parts_metadata, - op_online_disks, - disks, - )) + GetObjectFileInfo::owned_with_late_metadata_fanout(fi, parts_metadata, op_online_disks, disks) } else { - Ok(GetObjectFileInfo::owned(fi, parts_metadata, op_online_disks)) - } + GetObjectFileInfo::owned(fi, parts_metadata, op_online_disks) + }; + snapshot.condition_stopped = condition_stopped; + Ok(snapshot) } #[hotpath::measure(impl_type = "SetDisks")] @@ -886,6 +1032,8 @@ impl SetDisks { ); let decode_stage_start = Instant::now(); + #[cfg(test)] + crate::set_disk::disk_call_counters::record(object, crate::set_disk::disk_call_counters::KIND_BODY_DECODE, 0); let body = try_read_inline_data_shards_direct(&mut readers, erasure.data_shards, read_length, object_size).await; let decode_elapsed = decode_stage_start.elapsed(); rustfs_io_metrics::record_get_object_decode_duration(decode_elapsed.as_secs_f64()); @@ -943,6 +1091,8 @@ impl SetDisks { } let decode_stage_start = Instant::now(); + #[cfg(test)] + crate::set_disk::disk_call_counters::record(object, crate::set_disk::disk_call_counters::KIND_BODY_DECODE, 0); let body = try_read_inline_data_shards_direct(&mut reader_setup.readers, erasure.data_shards, read_length, object_size).await; let decode_elapsed = decode_stage_start.elapsed(); @@ -1361,6 +1511,8 @@ impl SetDisks { let readers = reader_setup.readers; let deferred_stripe_handles = reader_setup.deferred_stripe_handles; let deferred_reopeners = reader_setup.deferred_reopeners; + #[cfg(test)] + crate::set_disk::disk_call_counters::record(object, crate::set_disk::disk_call_counters::KIND_BODY_DECODE, 0); let coding::decode::DecodeOutcome { written, error: err, @@ -1916,6 +2068,8 @@ impl SetDisks { .with_deferred_parity_handles(deferred_stripe_handles) .with_deferred_parity_reopeners(deferred_reopeners) .with_repair_evidence(Arc::clone(&repair)); + #[cfg(test)] + crate::set_disk::disk_call_counters::record(object, crate::set_disk::disk_call_counters::KIND_BODY_DECODE, 0); let engine = build_get_codec_streaming_decode_engine(erasure.clone())?; let reader = if single_inflight { coding::decode_reader::ErasureDecodeReader::new_single_inflight_with_metrics_path( @@ -2535,6 +2689,10 @@ impl Drop for LegacyPerPartDecodeReader { } fn get_object_metadata_cache_request_bypass_reason(bucket: &str, opts: &ObjectOptions, read_data: bool) -> Option<&'static str> { + #[cfg(feature = "test-util")] + if super::conditional_read_bench::cache_bypass_applies(bucket) { + return Some("benchmark"); + } if !read_data { return Some(GET_METADATA_CACHE_REASON_NOT_READ_DATA); } @@ -2581,7 +2739,9 @@ fn is_get_object_metadata_cache_request_eligible(bucket: &str, opts: &ObjectOpti #[cfg(test)] mod metadata_cache_tests { use super::*; + use crate::set_disk::core::io_primitives::disk_call_counters; use rustfs_heal_contracts::heal_channel::HealAdmissionDropReason; + use rustfs_utils::path::object_name_disk_identity; use serial_test::serial; use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::{Mutex, OnceLock}; @@ -3519,8 +3679,15 @@ mod metadata_cache_tests { let online_disks = Vec::new(); let generation = set.get_object_metadata_cache_generation("bucket", "object"); - set.cache_get_object_fileinfo(("bucket", "object"), generation, &fi, &parts_metadata, &online_disks, 0) - .await; + set.cache_get_object_fileinfo( + ("bucket", "object"), + generation, + (&fi, GetObjectMetadataCacheCapability::Full), + &parts_metadata, + &online_disks, + 0, + ) + .await; let cached = set .cached_get_object_fileinfo("bucket", "object") @@ -3539,8 +3706,15 @@ mod metadata_cache_tests { let parts_metadata = vec![fi.clone()]; let online_disks = Vec::new(); let generation = set.get_object_metadata_cache_generation("bucket", "object"); - set.cache_get_object_fileinfo(("bucket", "object"), generation, &fi, &parts_metadata, &online_disks, 0) - .await; + set.cache_get_object_fileinfo( + ("bucket", "object"), + generation, + (&fi, GetObjectMetadataCacheCapability::Full), + &parts_metadata, + &online_disks, + 0, + ) + .await; let cached = set .cached_get_object_fileinfo("bucket", "object") .await @@ -3803,8 +3977,15 @@ mod metadata_cache_tests { let mut deleted = valid_test_fileinfo("deleted-object"); deleted.deleted = true; let generation = set.get_object_metadata_cache_generation("bucket", "deleted-object"); - set.cache_get_object_fileinfo(("bucket", "deleted-object"), generation, &deleted, &[deleted.clone()], &[], 0) - .await; + set.cache_get_object_fileinfo( + ("bucket", "deleted-object"), + generation, + (&deleted, GetObjectMetadataCacheCapability::Full), + &[deleted.clone()], + &[], + 0, + ) + .await; assert!( set.cached_get_object_fileinfo("bucket", "deleted-object").await.is_none(), "deleted metadata must not be cached" @@ -3812,8 +3993,15 @@ mod metadata_cache_tests { let invalid = FileInfo::default(); let generation = set.get_object_metadata_cache_generation("bucket", "invalid-object"); - set.cache_get_object_fileinfo(("bucket", "invalid-object"), generation, &invalid, std::slice::from_ref(&invalid), &[], 0) - .await; + set.cache_get_object_fileinfo( + ("bucket", "invalid-object"), + generation, + (&invalid, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&invalid), + &[], + 0, + ) + .await; assert!( set.cached_get_object_fileinfo("bucket", "invalid-object").await.is_none(), "invalid metadata must not be cached" @@ -3835,6 +4023,7 @@ mod metadata_cache_tests { ), Arc::new(GetObjectMetadataCacheEntry { created_at: Instant::now(), + capability: GetObjectMetadataCacheCapability::Full, fi: fi.clone(), parts_metadata: vec![fi], online_disks: vec![None], @@ -3855,8 +4044,15 @@ mod metadata_cache_tests { let fi = valid_test_fileinfo("object"); let generation = set.get_object_metadata_cache_generation("bucket", "object"); - set.cache_get_object_fileinfo(("bucket", "object"), generation, &fi, std::slice::from_ref(&fi), &[], 0) - .await; + set.cache_get_object_fileinfo( + ("bucket", "object"), + generation, + (&fi, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&fi), + &[], + 0, + ) + .await; assert!( set.cached_get_object_fileinfo("bucket", "object").await.is_some(), @@ -3881,8 +4077,15 @@ mod metadata_cache_tests { .get_object_metadata_cache_generation("bucket", "object") .expect("metadata cache generation should be active"); let retired_key = GetObjectMetadataCacheKey::new("bucket", "object", generation); - set.cache_get_object_fileinfo(("bucket", "object"), Some(generation), &fi, std::slice::from_ref(&fi), &[], 0) - .await; + set.cache_get_object_fileinfo( + ("bucket", "object"), + Some(generation), + (&fi, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&fi), + &[], + 0, + ) + .await; set.get_object_metadata_cache.run_pending_tasks().await; assert!(set.cached_get_object_fileinfo("bucket", "object").await.is_some()); assert_eq!(set.get_object_metadata_cache.entry_count(), 1); @@ -3896,8 +4099,65 @@ mod metadata_cache_tests { assert_eq!(set.get_object_metadata_cache.entry_count(), 0); } + async fn cache_valid_test_fileinfo(set: &SetDisks, object: &str) { + let fi = valid_test_fileinfo(object); + let generation = set.get_object_metadata_cache_generation("bucket", object); + set.cache_get_object_fileinfo( + ("bucket", object), + generation, + (&fi, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&fi), + &[], + 0, + ) + .await; + assert!(set.cached_get_object_fileinfo("bucket", object).await.is_some()); + } + + #[cfg(not(windows))] #[tokio::test] - #[serial(metadata_cache_publish_barrier)] + async fn metadata_cache_keeps_backslash_and_slash_keys_distinct_off_windows() { + assert_eq!(object_name_disk_identity("a\\b"), "a\\b"); + let set = new_metadata_cache_test_set().await; + cache_valid_test_fileinfo(&set, "a/b").await; + cache_valid_test_fileinfo(&set, "a\\b").await; + + set.invalidate_get_object_metadata_cache("bucket", "a\\b").await; + + assert!( + set.cached_get_object_fileinfo("bucket", "a/b").await.is_some(), + "off Windows `a\\b` is a different object and must not retire `a/b`" + ); + assert!(set.cached_get_object_fileinfo("bucket", "a\\b").await.is_none()); + } + + #[cfg(windows)] + #[tokio::test] + async fn windows_metadata_cache_shares_backslash_and_slash_spellings() { + assert_eq!(object_name_disk_identity("a\\b"), "a/b"); + let set = new_metadata_cache_test_set().await; + cache_valid_test_fileinfo(&set, "a/b").await; + assert!( + set.cached_get_object_fileinfo("bucket", "a\\b").await.is_some(), + "both spellings address the same xl.meta and must share one cache entry" + ); + + set.invalidate_get_object_metadata_cache("bucket", "a\\b").await; + assert!( + set.cached_get_object_fileinfo("bucket", "a/b").await.is_none(), + "a mutation through `a\\b` must retire metadata cached for `a/b`" + ); + + cache_valid_test_fileinfo(&set, "a\\b").await; + set.invalidate_get_object_metadata_cache("bucket", "a/b").await; + assert!( + set.cached_get_object_fileinfo("bucket", "a\\b").await.is_none(), + "a mutation through `a/b` must retire metadata cached for `a\\b`" + ); + } + + #[tokio::test] + #[serial(metadata_cache_early_stop)] async fn metadata_cache_production_fanout_cannot_publish_after_invalidation() { // Isolated context: an ambient DistErasure window (another test's // SetupTypeGuard) would bypass metadata-cache publication entirely @@ -3951,14 +4211,1065 @@ mod metadata_cache_tests { ); } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial(metadata_cache_early_stop)] + async fn metadata_cache_publishes_quorum_selected_early_stop_result() { + use tokio::io::AsyncReadExt; + + let isolated_ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + isolated_ctx + .update_erasure_type(crate::layout::endpoints::SetupType::Erasure) + .await; + let (dirs, set) = crate::ecstore_validation_blackbox::make_local_set_disks_with_ctx(4, 2, isolated_ctx).await; + let bucket = "metadata-cache-early-stop"; + let object = "objects/repeated-key"; + let disks = set.disks.read().await.clone(); + for disk in disks.iter().flatten() { + disk.make_volume(bucket) + .await + .expect("test bucket should be created on each disk"); + } + let expected_body = vec![0x5a; 1024 * 1024]; + set.put_object( + bucket, + object, + &mut PutObjReader::from_vec(expected_body.clone()), + &ObjectOptions::default(), + ) + .await + .expect("test object should be written before enabling the slow-tail fault"); + + let calls = disk_call_counters::observe(object); + temp_env::async_with_vars( + [ + ("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_TWO_PHASE_READ_PLAN_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("false")), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS", Some("500")), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DISKS", Some("3")), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_BUCKET", Some(bucket)), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_OBJECT_PREFIX", Some("objects/")), + ], + async { + let first = tokio::time::timeout( + Duration::from_millis(400), + set.get_object_fileinfo(bucket, object, &ObjectOptions::default(), true, true), + ) + .await + .expect("first metadata read should stop before the delayed peer") + .expect("read quorum should provide valid object metadata"); + assert!(first.owned.is_some(), "the first lookup should resolve through disk metadata"); + assert_eq!(calls.total(disk_call_counters::KIND_READ_VERSION), 4); + assert_eq!(calls.total(disk_call_counters::KIND_METADATA_SLOWTAIL_FAULT), 1); + + let second = set + .get_object_fileinfo(bucket, object, &ObjectOptions::default(), true, true) + .await + .expect("second lookup should use the quorum-published metadata cache entry"); + assert!(second.shared_entry().is_some(), "the repeated GET should hit the published cache entry"); + assert_eq!(calls.total(disk_call_counters::KIND_READ_VERSION), 4); + + let cached = second.shared_entry().expect("second lookup should share cached metadata"); + assert_eq!(cached.fi.name, object); + let cached_online_count = cached.online_disks.iter().filter(|disk| disk.is_some()).count(); + assert_eq!(cached_online_count, 3, "the warm cache should contain three of four disks"); + assert_eq!(cached.capability, GetObjectMetadataCacheCapability::QuorumSelected); + + let data_dir = cached + .fi + .data_dir + .expect("quorum cache retains the object data directory") + .to_string(); + let selected_disk_indices = cached + .online_disks + .iter() + .enumerate() + .filter_map(|(index, disk)| disk.as_ref().map(|_| index)) + .take(2) + .collect::>(); + assert_eq!(selected_disk_indices.len(), 2, "the warmed cache should name enough selected shards"); + for index in selected_disk_indices { + tokio::fs::remove_file(dirs[index].path().join(bucket).join(object).join(&data_dir).join("part.1")) + .await + .expect("simulate loss of two cached selected shards"); + } + + let mut reader = set + .get_object_reader(bucket, object, None, http::HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("a cached partial layout must recover through omitted disks"); + let mut body = Vec::new(); + reader + .stream + .read_to_end(&mut body) + .await + .expect("the GET should decode after refreshing the omitted disks"); + assert_eq!(body, expected_body, "recovered body must be byte-exact"); + }, + ) + .await; + + let full_layout = set + .get_object_fileinfo_gated(bucket, object, &ObjectOptions::default(), false, false) + .await + .expect("read-before-write consumers must fetch a complete layout"); + assert!( + full_layout.shared_entry().is_none(), + "a quorum-selected partial layout cannot satisfy mutation consumers" + ); + assert!( + calls.total(disk_call_counters::KIND_READ_VERSION) > 4, + "mutation consumers must independently fan out after a quorum-only cache hit" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial(metadata_cache_early_stop)] + async fn metadata_cache_early_stop_publication_is_fenced_by_invalidation() { + let isolated_ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + isolated_ctx + .update_erasure_type(crate::layout::endpoints::SetupType::Erasure) + .await; + let (_dirs, set) = crate::ecstore_validation_blackbox::make_local_set_disks_with_ctx(4, 2, isolated_ctx).await; + let bucket = "metadata-cache-early-stop-fence"; + let object = "objects/repeated-key"; + let disks = set.disks.read().await.clone(); + for disk in disks.iter().flatten() { + disk.make_volume(bucket) + .await + .expect("test bucket should be created on each disk"); + } + set.put_object( + bucket, + object, + &mut PutObjReader::from_vec(vec![0x5a; 1024 * 1024]), + &ObjectOptions::default(), + ) + .await + .expect("test object should be written before enabling the slow-tail fault"); + + temp_env::async_with_vars( + [ + ("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_TWO_PHASE_READ_PLAN_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("false")), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS", Some("500")), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DISKS", Some("3")), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_BUCKET", Some(bucket)), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_OBJECT_PREFIX", Some("objects/")), + ], + async { + let barrier = MetadataCachePublishBarrier::install(bucket, object); + let stale_generation = set + .get_object_metadata_cache_generation(bucket, object) + .expect("metadata cache generation should be active"); + let reader_set = Arc::clone(&set); + let read = tokio::spawn(async move { + reader_set + .get_object_fileinfo(bucket, object, &ObjectOptions::default(), true, true) + .await + }); + barrier.wait_until_paused().await; + set.invalidate_get_object_metadata_cache(bucket, object).await; + barrier.release(); + + let snapshot = read + .await + .expect("early-stop metadata read should not panic") + .expect("read quorum should resolve the selected FileInfo"); + assert!(snapshot.owned.is_some()); + assert!( + set.get_object_metadata_cache + .get(&GetObjectMetadataCacheKey::new(bucket, object, stale_generation)) + .await + .is_none(), + "early-stop publication must not retain an entry under the retired generation" + ); + assert!( + set.cached_get_object_fileinfo(bucket, object).await.is_none(), + "invalidation must reject the quorum-selected early-stop publication" + ); + }, + ) + .await; + } + + #[tokio::test] + #[serial(metadata_cache_publish_barrier, body_cache_hook)] + async fn conditional_inline_hedge_caches_only_terminal_metadata() { + use crate::object_api::{GetObjectReadCondition, GetObjectReadMetadata, WriteCompletion, with_get_object_read_condition}; + use crate::set_disk::core::io_primitives::{disk_call_counters, rename_fanout_barrier}; + use crate::set_disk::core::metadata_read::bounded_metadata_fanout_order; + use crate::storage_api_contracts::bucket::{BucketOperations, MakeBucketOptions}; + use http::HeaderMap; + use tokio::io::AsyncReadExt; + + struct MatchingEtag { + bucket: &'static str, + object: &'static str, + etag: String, + reached: Notify, + } + impl GetObjectReadCondition for MatchingEtag { + fn is_terminal(&self, metadata: GetObjectReadMetadata<'_>) -> bool { + let matched = metadata.bucket == self.bucket + && metadata.object == self.object + && !metadata.delete_marker + && metadata.etag == Some(self.etag.as_str()); + if matched { + // Observe the real metadata decision without changing its result. + self.reached.notify_one(); + } + matched + } + } + + temp_env::async_with_vars( + [ + (super::super::ENV_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE, Some("true")), + (super::super::ENV_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT, Some("true")), + ], + async { + let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + ctx.update_erasure_type(crate::layout::endpoints::SetupType::Erasure).await; + let (_dirs, set) = crate::ecstore_validation_blackbox::make_local_set_disks_with_ctx(4, 2, ctx).await; + let bucket = "conditional-inline-cache-hedge"; + let object = "conditional-inline-cache-hedge.bin"; + let payload: Vec<_> = (0..4096) + .map(|index| u8::try_from(index % 251).expect("pattern byte")) + .collect(); + set.make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("real bucket"); + let info = set + .put_object( + bucket, + object, + &mut PutObjReader::from_vec(payload.clone()), + &ObjectOptions { + write_completion: WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("commit all four real inline shards"); + let etag = info.etag.expect("committed ETag"); + let opts = ObjectOptions::default(); + // This metadata-only inspection bypasses publication and discovers the actual layout. + let snapshot = set + .get_object_fileinfo(bucket, object, &opts, false, false) + .await + .expect("actual metadata"); + let distribution = snapshot.fi().erasure.distribution.clone(); + assert_eq!(snapshot.fi().erasure.data_blocks, 2); + assert_eq!(snapshot.fi().erasure.parity_blocks, 2); + assert!(snapshot.fi().inline_data()); + let d1 = distribution.iter().position(|block| *block == 1).expect("D1"); + let d2 = distribution.iter().position(|block| *block == 2).expect("D2"); + let order = bounded_metadata_fanout_order(bucket, object, 4, 2); + // Parity slots retain disk order, not their erasure block-number order. + let p1 = order[2]; + let p2 = order[3]; + assert_eq!(&order[..2], &[d1, d2]); + assert!(distribution[p1] > 2 && distribution[p2] > 2); + drop(snapshot); + assert!(set.cached_get_object_fileinfo(bucket, object).await.is_none(), "cold real object"); + assert!( + set.get_object_metadata_cache_bypass_reason(bucket, &opts, true) + .await + .is_none() + ); + + let calls = disk_call_counters::observe(object); + let data = rename_fanout_barrier::arm(object, d2, rename_fanout_barrier::PHASE_READ_VERSION); + let parity = rename_fanout_barrier::arm(object, p2, rename_fanout_barrier::PHASE_READ_VERSION); + let condition = Arc::new(MatchingEtag { + bucket, + object, + etag: etag.clone(), + reached: Notify::new(), + }); + let reader_set = Arc::clone(&set); + let reader_condition = Arc::clone(&condition); + let mut read = tokio::spawn(async move { + with_get_object_read_condition( + reader_condition, + reader_set.get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default()), + ) + .await + }); + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + let completed = tokio::select! { + _ = data.wait_until_paused_before(deadline) => { + parity.wait_until_paused_before(deadline).await; + assert!(data.is_paused(), "D2 is still unavailable when the spare P2 is scheduled"); + assert_eq!(calls.for_disk(disk_call_counters::KIND_READ_VERSION, d1), 1); + assert_eq!(calls.for_disk(disk_call_counters::KIND_READ_VERSION, p1), 1); + parity.release(); + tokio::time::timeout_at(deadline, condition.reached.notified()).await + .expect("actual matching decision on the first three completed responses"); + // The 304 must finish while D2 remains blocked; waiting + // for all four disks would hide the cache starvation. + let result = tokio::time::timeout_at(deadline, &mut read).await.expect("conditional reader deadline"); + data.release(); + result + } + _ = &mut read => panic!("cold read must cross the real hedge barriers"), + }; + let mut reader = completed.expect("reader task").expect("conditional metadata reader"); + assert_eq!(reader.object_info.etag.as_deref(), Some(etag.as_str())); + let mut forbidden_body = Vec::new(); + assert!( + reader.stream.read_to_end(&mut forbidden_body).await.is_err(), + "terminal read is not a successful body" + ); + assert!(forbidden_body.is_empty()); + drop(reader); + assert_eq!(calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN), 0); + assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 0); + assert_eq!( + calls.total(disk_call_counters::KIND_BODY_DECODE), + 0, + "304 metadata setup never verifies or decodes a body" + ); + drop(data); + drop(parity); + let cached = set + .cached_get_object_fileinfo(bucket, object) + .await + .expect("a terminal quorum must remain reusable for another condition"); + assert_eq!(cached.capability, GetObjectMetadataCacheCapability::ConditionOnly); + assert_eq!(cached.online_disks.iter().flatten().count(), 3); + assert_eq!( + cached + .parts_metadata + .iter() + .filter(|metadata| !metadata.name.is_empty()) + .count(), + 3 + ); + + let before = calls.total(disk_call_counters::KIND_READ_VERSION); + let mut hit = with_get_object_read_condition( + Arc::new(MatchingEtag { + bucket, + object, + etag: etag.clone(), + reached: Notify::new(), + }), + set.get_object_reader(bucket, object, None, HeaderMap::new(), &opts), + ) + .await + .expect("warm conditional metadata"); + assert!(hit.stream.read_to_end(&mut Vec::new()).await.is_err()); + drop(hit); + assert_eq!( + calls.total(disk_call_counters::KIND_READ_VERSION), + before, + "warm conditions reuse metadata without fanout" + ); + let mut ordinary = set + .get_object_reader(bucket, object, None, HeaderMap::new(), &opts) + .await + .expect("ordinary warm GET"); + let mut actual = Vec::new(); + ordinary.stream.read_to_end(&mut actual).await.expect("ordinary body"); + assert_eq!(actual, payload); + assert!( + calls.total(disk_call_counters::KIND_READ_VERSION) > before, + "conditional cached layout cannot supply an ordinary body" + ); + assert!( + calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN) > 0, + "same source counter sees the ordinary positive control" + ); + assert!( + calls.total(disk_call_counters::KIND_BODY_DECODE) > 0, + "same decoder counter sees the ordinary positive control" + ); + + // Restore the proven conditional quorum for each nonterminal + // consumer, so an earlier ordinary read cannot mask a leak. + for range in [ + None, + Some(HTTPRangeSpec { + is_suffix_length: false, + start: 1, + end: 16, + }), + ] { + set.invalidate_get_object_metadata_cache(bucket, object).await; + set.cache_get_object_fileinfo( + (bucket, object), + set.get_object_metadata_cache_generation(bucket, object), + (&cached.fi, GetObjectMetadataCacheCapability::ConditionOnly), + &cached.parts_metadata, + &cached.online_disks, + cached.read_quorum, + ) + .await; + let before = calls.total(disk_call_counters::KIND_READ_VERSION); + let mut reader = with_get_object_read_condition( + Arc::new(MatchingEtag { + bucket, + object, + etag: "different-etag".to_owned(), + reached: Notify::new(), + }), + set.get_object_reader(bucket, object, range.clone(), HeaderMap::new(), &opts), + ) + .await + .expect("nonterminal condition must read its own body layout"); + let mut actual = Vec::new(); + reader + .stream + .read_to_end(&mut actual) + .await + .expect("complete nonterminal body"); + assert_eq!( + actual.as_slice(), + if range.is_some() { + &payload[1..17] + } else { + payload.as_slice() + } + ); + assert!(calls.total(disk_call_counters::KIND_READ_VERSION) > before); + } + + set.invalidate_get_object_metadata_cache(bucket, object).await; + set.cache_get_object_fileinfo( + (bucket, object), + set.get_object_metadata_cache_generation(bucket, object), + (&cached.fi, GetObjectMetadataCacheCapability::ConditionOnly), + &cached.parts_metadata, + &cached.online_disks, + cached.read_quorum, + ) + .await; + let before = calls.total(disk_call_counters::KIND_READ_VERSION); + let full = with_get_object_read_condition( + Arc::new(MatchingEtag { + bucket, + object, + etag, + reached: Notify::new(), + }), + set.get_object_fileinfo_gated(bucket, object, &opts, true, false), + ) + .await + .expect("full-layout caller under a matching condition"); + assert!(calls.total(disk_call_counters::KIND_READ_VERSION) > before); + assert_eq!(full.online_disks().iter().flatten().count(), 4); + assert_eq!( + set.cached_get_object_fileinfo(bucket, object) + .await + .expect("complete publication") + .capability, + GetObjectMetadataCacheCapability::Full + ); + }, + ) + .await; + } + + fn cache_capability_race_entry( + object: &str, + capability: GetObjectMetadataCacheCapability, + etag: &str, + ) -> Arc { + let mut fi = valid_test_fileinfo(object); + fi.metadata.insert("etag".to_string(), etag.to_string()); + Arc::new(GetObjectMetadataCacheEntry { + created_at: Instant::now(), + parts_metadata: vec![fi.clone()], + fi, + online_disks: Vec::new(), + read_quorum: 0, + capability, + }) + } + + #[tokio::test] + async fn metadata_cache_capability_publication_preserves_full_in_both_orders() { + use GetObjectMetadataCacheCapability::{ConditionOnly, Full}; + + for capabilities in [[Full, ConditionOnly], [ConditionOnly, Full]] { + let set = new_metadata_cache_test_set().await; + let generation = set + .get_object_metadata_cache_generation("bucket", "object") + .expect("active cache generation"); + let key = GetObjectMetadataCacheKey::new("bucket", "object", generation); + let full = cache_capability_race_entry("object", Full, "full-layout"); + let partial = cache_capability_race_entry("object", ConditionOnly, "partial-layout"); + for capability in capabilities { + let incoming = if capability == Full { &full } else { &partial }; + set.insert_get_object_metadata_cache_entry_after_insert(key.clone(), generation, Arc::clone(incoming), || {}) + .await; + } + let cached = set.get_object_metadata_cache.get(&key).await.expect("published cache entry"); + assert_eq!(cached.capability, Full); + assert!(Arc::ptr_eq(&cached, &full), "the complete layout must survive either publication order"); + } + } + + #[tokio::test] + async fn metadata_cache_capability_publisher_waits_for_real_compute_in_both_orders() { + use GetObjectMetadataCacheCapability::{ConditionOnly, Full}; + use moka::ops::compute::Op; + + for held_capability in [Full, ConditionOnly] { + let set = new_metadata_cache_test_set().await; + let generation = set + .get_object_metadata_cache_generation("bucket", "object") + .expect("active cache generation"); + let key = GetObjectMetadataCacheKey::new("bucket", "object", generation); + let full = cache_capability_race_entry("object", Full, "full-layout"); + let partial = cache_capability_race_entry("object", ConditionOnly, "partial-layout"); + let (held_entry, incoming) = if held_capability == Full { + (Arc::clone(&full), Arc::clone(&partial)) + } else { + (Arc::clone(&partial), Arc::clone(&full)) + }; + let release = Arc::new(Notify::new()); + let holder_release = Arc::clone(&release); + // Hold the actual Moka compute waiter across an await. Polling the + // production publisher below proves it queues before any cache write. + let mut holder = Box::pin( + set.get_object_metadata_cache + .entry(key.clone()) + .and_compute_with(|current| async move { + assert!(current.is_none(), "the holder must see an empty key"); + holder_release.notified().await; + Op::Put(held_entry) + }), + ); + assert!(futures::poll!(holder.as_mut()).is_pending()); + let hook_calls = AtomicUsize::new(0); + let mut publisher = + Box::pin( + set.insert_get_object_metadata_cache_entry_after_insert(key.clone(), generation, incoming, || { + hook_calls.fetch_add(1, Ordering::AcqRel); + }), + ); + assert!( + futures::poll!(publisher.as_mut()).is_pending(), + "all production publishers must share Moka's compute waiter" + ); + assert!( + set.get_object_metadata_cache.get(&key).await.is_none(), + "a plain insert must not bypass the holder and expose an entry" + ); + assert_eq!(hook_calls.load(Ordering::Acquire), 0); + release.notify_one(); + holder.await; + publisher.await; + assert_eq!(hook_calls.load(Ordering::Acquire), 1); + let cached = set.get_object_metadata_cache.get(&key).await.expect("published cache entry"); + assert_eq!(cached.capability, Full); + assert!(Arc::ptr_eq(&cached, &full), "a queued conditional publisher must not downgrade Full"); + } + } + + #[tokio::test] + async fn metadata_cache_waiting_stale_compute_cannot_touch_new_generation() { + use GetObjectMetadataCacheCapability::{ConditionOnly, Full}; + use moka::ops::compute::Op; + + let set = new_metadata_cache_test_set().await; + let old_generation = set + .get_object_metadata_cache_generation("bucket", "object") + .expect("active old generation"); + let old_key = GetObjectMetadataCacheKey::new("bucket", "object", old_generation); + let release = Arc::new(Notify::new()); + let holder_release = Arc::clone(&release); + let mut holder = Box::pin( + set.get_object_metadata_cache + .entry(old_key.clone()) + .and_compute_with(|current| async move { + assert!(current.is_none()); + holder_release.notified().await; + Op::Nop + }), + ); + assert!(futures::poll!(holder.as_mut()).is_pending()); + let hook_calls = AtomicUsize::new(0); + let old_partial = cache_capability_race_entry("object", ConditionOnly, "retired-layout"); + let mut old_publisher = Box::pin(set.insert_get_object_metadata_cache_entry_after_insert( + old_key.clone(), + old_generation, + old_partial, + || { + hook_calls.fetch_add(1, Ordering::AcqRel); + }, + )); + assert!(futures::poll!(old_publisher.as_mut()).is_pending()); + assert_eq!(hook_calls.load(Ordering::Acquire), 0); + + set.invalidate_get_object_metadata_cache("bucket", "object").await; + let new_generation = set + .get_object_metadata_cache_generation("bucket", "object") + .expect("active new generation"); + assert_eq!(new_generation.value, old_generation.value + 1); + let new_key = GetObjectMetadataCacheKey::new("bucket", "object", new_generation); + assert_ne!(old_key, new_key); + let new_full = cache_capability_race_entry("object", Full, "current-layout"); + tokio::time::timeout( + Duration::from_secs(1), + set.insert_get_object_metadata_cache_entry_after_insert( + new_key.clone(), + new_generation, + Arc::clone(&new_full), + || {}, + ), + ) + .await + .expect("a retired-generation waiter must not block the new key"); + release.notify_one(); + holder.await; + old_publisher.await; + assert_eq!(hook_calls.load(Ordering::Acquire), 1, "a stale Nop still runs the post-compute hook"); + assert!(set.get_object_metadata_cache.get(&old_key).await.is_none()); + let cached = set + .get_object_metadata_cache + .get(&new_key) + .await + .expect("new generation must survive"); + assert!( + Arc::ptr_eq(&cached, &new_full), + "old publication and its cleanup must not replace or remove the new key" + ); + } + + #[tokio::test] + async fn metadata_cache_put_and_nop_hooks_recheck_generation_exactly_once() { + use GetObjectMetadataCacheCapability::{ConditionOnly, Full}; + + for incoming_capability in [Full, ConditionOnly] { + let set = new_metadata_cache_test_set().await; + let generation = set + .get_object_metadata_cache_generation("bucket", "object") + .expect("active cache generation"); + let key = GetObjectMetadataCacheKey::new("bucket", "object", generation); + let existing = cache_capability_race_entry("object", Full, "existing-full"); + set.insert_get_object_metadata_cache_entry_after_insert(key.clone(), generation, existing, || {}) + .await; + let incoming = cache_capability_race_entry("object", incoming_capability, "incoming"); + let hook_calls = AtomicUsize::new(0); + set.insert_get_object_metadata_cache_entry_after_insert(key.clone(), generation, incoming, || { + assert_eq!(hook_calls.fetch_add(1, Ordering::AcqRel), 0, "hook must run once"); + // The existing production seam invalidates after either a Put or + // Full-preserving Nop; both must still perform the final fence read. + set.get_object_metadata_cache_generations[generation.index].fetch_add(1, Ordering::AcqRel); + }) + .await; + assert_eq!(hook_calls.load(Ordering::Acquire), 1); + assert!( + set.get_object_metadata_cache.get(&key).await.is_none(), + "post-compute generation change must withdraw the retired key for both Put and Nop" + ); + } + } + + #[tokio::test] + async fn metadata_cache_conditional_nop_preserves_actual_moka_expiration() { + use GetObjectMetadataCacheCapability::{ConditionOnly, Full}; + + let set = new_metadata_cache_test_set().await; + let generation = set + .get_object_metadata_cache_generation("bucket", "object") + .expect("active cache generation"); + let key = GetObjectMetadataCacheKey::new("bucket", "object", generation); + let full = cache_capability_race_entry("object", Full, "original-full"); + set.insert_get_object_metadata_cache_entry_after_insert(key.clone(), generation, Arc::clone(&full), || {}) + .await; + let original_insert_finished = Instant::now(); + + // Moka uses its own real clock, so Tokio paused-time advancement would not + // establish this property. Use the actual configured TTL and expiry reads. + tokio::time::sleep(GET_OBJECT_METADATA_CACHE_TTL * 3 / 4).await; + assert!( + set.get_object_metadata_cache.get(&key).await.is_some(), + "the retained entry must be live immediately before the Nop" + ); + let partial = cache_capability_race_entry("object", ConditionOnly, "partial-layout"); + set.insert_get_object_metadata_cache_entry_after_insert(key.clone(), generation, partial, || {}) + .await; + let retained = set.get_object_metadata_cache.get(&key).await.expect("Full must remain live"); + assert!(Arc::ptr_eq(&retained, &full)); + + // A distinct fresh Put is the positive control for the renewed deadline. + let fresh_generation = set + .get_object_metadata_cache_generation("bucket", "fresh-object") + .expect("active positive-control generation"); + let fresh_key = GetObjectMetadataCacheKey::new("bucket", "fresh-object", fresh_generation); + let fresh = cache_capability_race_entry("fresh-object", Full, "fresh-full"); + set.insert_get_object_metadata_cache_entry_after_insert(fresh_key.clone(), fresh_generation, fresh, || {}) + .await; + let renewed_insert_finished = Instant::now(); + let original_expired_by = original_insert_finished + GET_OBJECT_METADATA_CACHE_TTL + Duration::from_millis(200); + tokio::time::sleep_until(tokio::time::Instant::from_std(original_expired_by)).await; + assert!( + renewed_insert_finished.elapsed() < GET_OBJECT_METADATA_CACHE_TTL, + "the measurement must end before a renewed TTL could expire" + ); + assert!( + set.get_object_metadata_cache.get(&fresh_key).await.is_some(), + "a genuinely fresh Put must still be live at the same observation time" + ); + assert!( + set.get_object_metadata_cache.get(&key).await.is_none(), + "Nop must preserve Moka's original expiration, not merely the entry's created_at" + ); + } + + struct CacheBoundaryCondition { + bucket: &'static str, + object: &'static str, + etag: String, + calls: AtomicUsize, + terminal_calls: usize, + } + + impl crate::object_api::GetObjectReadCondition for CacheBoundaryCondition { + fn is_terminal(&self, metadata: crate::object_api::GetObjectReadMetadata<'_>) -> bool { + if metadata.bucket != self.bucket + || metadata.object != self.object + || metadata.delete_marker + || metadata.etag != Some(self.etag.as_str()) + { + return false; + } + self.calls.fetch_add(1, Ordering::AcqRel) < self.terminal_calls + } + } + + async fn condition_only_cache_boundary_fixture( + bucket: &'static str, + object: &'static str, + ) -> (Vec, Arc, Arc) { + use crate::object_api::WriteCompletion; + use crate::storage_api_contracts::bucket::{BucketOperations, MakeBucketOptions}; + + let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + ctx.update_erasure_type(crate::layout::endpoints::SetupType::Erasure).await; + let (dirs, set) = crate::ecstore_validation_blackbox::make_local_set_disks_with_ctx(4, 2, ctx).await; + set.make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("real bucket"); + let committed = set + .put_object( + bucket, + object, + &mut PutObjReader::from_vec(vec![17; 4096]), + &ObjectOptions { + write_completion: WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("commit all four shards"); + let full = set + .get_object_fileinfo(bucket, object, &ObjectOptions::default(), true, false) + .await + .expect("complete real metadata"); + assert_eq!(full.online_disks().iter().flatten().count(), 4); + let mut metadata = full.parts_metadata().to_vec(); + let mut disks = full.online_disks().to_vec(); + let omitted = metadata + .iter() + .position(|fi| fi.erasure.index == 2) + .expect("actual second data disk"); + metadata[omitted] = FileInfo::default(); + disks[omitted] = None; + // This seam prepares a real, independently validated D1/P1/P2 quorum. + // The hedge test separately exercises its production publication timing. + let errs: Vec> = (0..4).map(|_| None).collect(); + let (online_disks, fi, _) = SetDisks::select_valid_fileinfo(&disks, &metadata, &errs, "", 2, 3) + .expect("three real members satisfy latest-generation selection"); + assert_eq!(online_disks.iter().flatten().count(), 3); + assert_eq!(fi.metadata.get("etag"), committed.etag.as_ref()); + set.invalidate_get_object_metadata_cache(bucket, object).await; + set.cache_get_object_fileinfo( + (bucket, object), + set.get_object_metadata_cache_generation(bucket, object), + (&fi, GetObjectMetadataCacheCapability::ConditionOnly), + &metadata, + &online_disks, + 2, + ) + .await; + let cached = set + .cached_get_object_fileinfo(bucket, object) + .await + .expect("prepared conditional quorum"); + assert_eq!(cached.capability, GetObjectMetadataCacheCapability::ConditionOnly); + (dirs, set, cached) + } + + #[tokio::test] + #[serial(metadata_cache_publish_barrier, body_cache_hook)] + async fn condition_only_cache_final_predicate_flip_fails_before_body() { + use crate::object_api::with_get_object_read_condition; + use crate::set_disk::core::io_primitives::disk_call_counters; + + let bucket = "conditional-cache-final-flip"; + let object = "conditional-cache-final-flip.bin"; + let (_dirs, set, cached) = condition_only_cache_boundary_fixture(bucket, object).await; + let condition = Arc::new(CacheBoundaryCondition { + bucket, + object, + etag: cached.fi.metadata["etag"].clone(), + calls: AtomicUsize::new(0), + // Deliberately violate purity to attack the final fail-closed guard. + terminal_calls: 1, + }); + let calls = disk_call_counters::observe(object); + let result = with_get_object_read_condition( + condition.clone(), + set.get_object_reader(bucket, object, None, http::HeaderMap::new(), &ObjectOptions::default()), + ) + .await; + let err = result + .err() + .expect("a partial snapshot cannot become a body reader after predicate flip"); + assert!( + err.to_string() + .contains("conditional metadata snapshot no longer satisfies the read condition") + ); + assert_eq!(condition.calls.load(Ordering::Acquire), 2, "lookup and final selected-FI check both ran"); + for kind in [ + disk_call_counters::KIND_READ_VERSION, + disk_call_counters::KIND_INLINE_SHARD_OPEN, + disk_call_counters::KIND_DISK_SHARD_OPEN, + disk_call_counters::KIND_BODY_DECODE, + ] { + assert_eq!(calls.total(kind), 0, "rejected cached snapshot cannot materialize any body"); + } + } + + #[tokio::test] + #[serial(metadata_cache_publish_barrier, body_cache_hook)] + async fn condition_only_cache_scope_and_full_layout_boundaries() { + use crate::object_api::with_get_object_read_condition; + use crate::set_disk::core::io_primitives::disk_call_counters; + + let bucket = "conditional-cache-scope-boundary"; + let object = "conditional-cache-scope-boundary.bin"; + let (_dirs, set, cached) = condition_only_cache_boundary_fixture(bucket, object).await; + for mode in 0..3 { + set.invalidate_get_object_metadata_cache(bucket, object).await; + set.cache_get_object_fileinfo( + (bucket, object), + set.get_object_metadata_cache_generation(bucket, object), + (&cached.fi, GetObjectMetadataCacheCapability::ConditionOnly), + &cached.parts_metadata, + &cached.online_disks, + cached.read_quorum, + ) + .await; + let condition = Arc::new(CacheBoundaryCondition { + bucket, + object, + etag: cached.fi.metadata["etag"].clone(), + calls: AtomicUsize::new(0), + terminal_calls: usize::MAX, + }); + let calls = disk_call_counters::observe(object); + let snapshot = with_get_object_read_condition(condition.clone(), async { + assert!(crate::object_api::get_object_read_condition_is_active()); + match mode { + 0 => { + let producer_set = set.clone(); + tokio::spawn(Box::pin(async move { + assert!( + !crate::object_api::get_object_read_condition_is_active(), + "spawned producers inherit no caller condition" + ); + producer_set + .get_object_fileinfo_for_get_object_reader(bucket, object, &ObjectOptions::default(), true, true) + .await + })) + .await + .expect("producer task") + .expect("producer reads its own layout") + } + 1 => set + .get_object_fileinfo(bucket, object, &ObjectOptions::default(), true, false) + .await + .expect("generic complete-layout consumer"), + _ => set + .get_object_fileinfo_for_get_object_reader( + bucket, + object, + &ObjectOptions { + no_lock: true, + metadata_cache_safe: false, + ..Default::default() + }, + true, + true, + ) + .await + .expect("unprotected NoLock reads bypass even a terminal cached condition"), + } + }) + .await; + assert!(snapshot.shared.is_none(), "mode {mode} cannot consume a conditional cached layout"); + assert!( + calls.total(disk_call_counters::KIND_READ_VERSION) > 0, + "mode {mode} must independently read metadata" + ); + if mode < 2 { + assert!(!snapshot.condition_stopped); + assert_eq!(condition.calls.load(Ordering::Acquire), 0); + } + } + } + + #[tokio::test] + async fn condition_only_cache_rejects_invalid_deleted_and_insufficient_quorum() { + let set = new_metadata_cache_test_set().await; + let mut deleted = valid_test_fileinfo("conditional-deleted"); + deleted.deleted = true; + let invalid = FileInfo::default(); + for (object, fi) in [("conditional-deleted", deleted), ("conditional-invalid", invalid)] { + set.cache_get_object_fileinfo( + ("bucket", object), + set.get_object_metadata_cache_generation("bucket", object), + (&fi, GetObjectMetadataCacheCapability::ConditionOnly), + std::slice::from_ref(&fi), + &[], + 0, + ) + .await; + assert!(matches!( + set.lookup_cached_get_object_fileinfo("bucket", object).await, + MetadataCacheLookup::Miss + )); + } + for object in ["conditional-deleted", "conditional-invalid"] { + let condition = Arc::new(CacheBoundaryCondition { + bucket: "bucket", + object, + etag: "etag-1".to_owned(), + calls: AtomicUsize::new(0), + terminal_calls: usize::MAX, + }); + let result = crate::object_api::with_get_object_read_condition( + condition.clone(), + set.get_object_fileinfo_for_get_object_reader("bucket", object, &ObjectOptions::default(), true, true), + ) + .await; + assert!(result.is_err(), "invalid or deleted cache metadata cannot terminate the read"); + assert_eq!(condition.calls.load(Ordering::Acquire), 0); + } + let object = "conditional-insufficient"; + let fi = valid_test_fileinfo(object); + let generation = set + .get_object_metadata_cache_generation("bucket", object) + .expect("generation"); + set.get_object_metadata_cache + .insert( + GetObjectMetadataCacheKey::new("bucket", object, generation), + Arc::new(GetObjectMetadataCacheEntry { + created_at: Instant::now(), + capability: GetObjectMetadataCacheCapability::ConditionOnly, + fi: fi.clone(), + parts_metadata: vec![fi], + online_disks: vec![None], + read_quorum: 1, + }), + ) + .await; + assert!(matches!( + set.lookup_cached_get_object_fileinfo("bucket", object).await, + MetadataCacheLookup::RejectedInsufficientQuorum + )); + // With no disks, a rejected entry must produce an actual metadata error, + // even when a caller's predicate would accept the poisoned cached ETag. + let condition = Arc::new(CacheBoundaryCondition { + bucket: "bucket", + object, + etag: "etag-1".to_owned(), + calls: AtomicUsize::new(0), + terminal_calls: usize::MAX, + }); + let result = crate::object_api::with_get_object_read_condition( + condition.clone(), + set.get_object_fileinfo_for_get_object_reader("bucket", object, &ObjectOptions::default(), true, true), + ) + .await; + assert!(result.is_err(), "insufficient cached quorum cannot yield terminal metadata"); + assert_eq!(condition.calls.load(Ordering::Acquire), 0); + } + + #[tokio::test] + #[serial(metadata_cache_publish_barrier, body_cache_hook)] + async fn condition_only_cache_generation_rechecked_after_lookup() { + let bucket = "conditional-cache-generation-boundary"; + let object = "conditional-cache-generation-boundary.bin"; + let (_dirs, set, _) = condition_only_cache_boundary_fixture(bucket, object).await; + let generation = set + .get_object_metadata_cache_generation(bucket, object) + .expect("current generation"); + let key = GetObjectMetadataCacheKey::new(bucket, object, generation); + let condition = Arc::new(CacheBoundaryCondition { + bucket, + object, + etag: set + .cached_get_object_fileinfo(bucket, object) + .await + .expect("cached quorum") + .fi + .metadata["etag"] + .clone(), + calls: AtomicUsize::new(0), + terminal_calls: usize::MAX, + }); + let lookup = crate::object_api::with_get_object_read_condition(condition.clone(), async { + set.lookup_cached_get_object_fileinfo_after_get(bucket, object, || { + set.get_object_metadata_cache_generations[generation.index].fetch_add(1, Ordering::AcqRel); + }) + .await + }) + .await; + assert!( + matches!(lookup, MetadataCacheLookup::Miss), + "a concurrent invalidation rejects old conditional metadata" + ); + assert!( + set.get_object_metadata_cache.get(&key).await.is_none(), + "retired partial entry is withdrawn" + ); + assert_eq!( + condition.calls.load(Ordering::Acquire), + 0, + "generation validation precedes conditional use" + ); + assert!(matches!( + set.lookup_cached_get_object_fileinfo(bucket, object).await, + MetadataCacheLookup::Miss + )); + } + #[tokio::test] async fn metadata_cache_lookup_rechecks_generation_after_get() { let set = new_metadata_cache_test_set().await; let fi = valid_test_fileinfo("object"); let generation = set.get_object_metadata_cache_generation("bucket", "object"); let generation_index = generation.expect("metadata cache generation should be active").index; - set.cache_get_object_fileinfo(("bucket", "object"), generation, &fi, std::slice::from_ref(&fi), &[], 0) - .await; + set.cache_get_object_fileinfo( + ("bucket", "object"), + generation, + (&fi, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&fi), + &[], + 0, + ) + .await; let lookup = set .lookup_cached_get_object_fileinfo_after_get("bucket", "object", || { @@ -3983,6 +5294,7 @@ mod metadata_cache_tests { let key = GetObjectMetadataCacheKey::new("bucket", "object", generation); let entry = Arc::new(GetObjectMetadataCacheEntry { created_at: Instant::now(), + capability: GetObjectMetadataCacheCapability::Full, fi: fi.clone(), parts_metadata: vec![fi], online_disks: Vec::new(), @@ -4022,8 +5334,15 @@ mod metadata_cache_tests { assert_eq!(first.value, initial.value + 1); assert_eq!(second.value, first.value + 1); - set.cache_get_object_fileinfo(("bucket", "object"), Some(first), &fi, std::slice::from_ref(&fi), &[], 0) - .await; + set.cache_get_object_fileinfo( + ("bucket", "object"), + Some(first), + (&fi, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&fi), + &[], + 0, + ) + .await; assert!(set.cached_get_object_fileinfo("bucket", "object").await.is_none()); } @@ -4062,10 +5381,24 @@ mod metadata_cache_tests { let first_generation = first.get_object_metadata_cache_generation("bucket", "object"); let second_generation = second.get_object_metadata_cache_generation("bucket", "object"); first - .cache_get_object_fileinfo(("bucket", "object"), first_generation, &fi, std::slice::from_ref(&fi), &[], 0) + .cache_get_object_fileinfo( + ("bucket", "object"), + first_generation, + (&fi, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&fi), + &[], + 0, + ) .await; second - .cache_get_object_fileinfo(("bucket", "object"), second_generation, &fi, std::slice::from_ref(&fi), &[], 0) + .cache_get_object_fileinfo( + ("bucket", "object"), + second_generation, + (&fi, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&fi), + &[], + 0, + ) .await; first.invalidate_get_object_metadata_cache("bucket", "object").await; @@ -4082,17 +5415,21 @@ mod metadata_cache_tests { .get_object_metadata_cache_generation("bucket-a", "object-a") .expect("metadata cache generation should be active"); let first_key = GetObjectMetadataCacheKey::new("bucket-a", "object-a", generation); + let first_lookup = GetObjectMetadataCacheLookupKey::new("bucket-a", "object-a", generation); let second_key = GetObjectMetadataCacheKey { bucket: Arc::from("bucket-b"), object: Arc::from("object-b"), generation: generation.value, hash: generation.hash, }; + let second_lookup = GetObjectMetadataCacheLookupKey::new("bucket-b", "object-b", generation); + let unrelated_lookup = GetObjectMetadataCacheLookupKey::new("bucket-c", "object-c", generation); let first_fi = valid_test_fileinfo("object-a"); let second_fi = valid_test_fileinfo("object-b"); let entry = |fi: FileInfo| { Arc::new(GetObjectMetadataCacheEntry { created_at: Instant::now(), + capability: GetObjectMetadataCacheCapability::Full, parts_metadata: vec![fi.clone()], fi, online_disks: Vec::new(), @@ -4107,7 +5444,7 @@ mod metadata_cache_tests { assert_eq!( set.get_object_metadata_cache - .get(&first_key) + .get(&first_lookup) .await .expect("first colliding entry should remain addressable") .fi @@ -4116,13 +5453,14 @@ mod metadata_cache_tests { ); assert_eq!( set.get_object_metadata_cache - .get(&second_key) + .get(&second_lookup) .await .expect("second colliding entry should remain addressable") .fi .name, "object-b" ); + assert!(set.get_object_metadata_cache.get(&unrelated_lookup).await.is_none()); } #[tokio::test] @@ -4140,8 +5478,15 @@ mod metadata_cache_tests { set.invalidate_get_object_metadata_cache("bucket", "object").await; assert_eq!(set.get_object_metadata_cache_generation("bucket", "object"), None); - set.cache_get_object_fileinfo(("bucket", "object"), last_generation, &fi, std::slice::from_ref(&fi), &[], 0) - .await; + set.cache_get_object_fileinfo( + ("bucket", "object"), + last_generation, + (&fi, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&fi), + &[], + 0, + ) + .await; assert!(set.cached_get_object_fileinfo("bucket", "object").await.is_none()); set.invalidate_get_object_metadata_cache("bucket", "object").await; assert_eq!( @@ -4159,8 +5504,15 @@ mod metadata_cache_tests { set.invalidate_all_get_object_metadata_cache(); - set.cache_get_object_fileinfo(("bucket", "object"), stale_generation, &fi, std::slice::from_ref(&fi), &[], 0) - .await; + set.cache_get_object_fileinfo( + ("bucket", "object"), + stale_generation, + (&fi, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&fi), + &[], + 0, + ) + .await; assert!(set.cached_get_object_fileinfo("bucket", "object").await.is_none()); } @@ -4174,8 +5526,15 @@ mod metadata_cache_tests { .get_object_metadata_cache_generation("bucket", object) .expect("metadata cache generation should be active"); retired_keys.push(GetObjectMetadataCacheKey::new("bucket", object, generation)); - set.cache_get_object_fileinfo(("bucket", object), Some(generation), &fi, std::slice::from_ref(&fi), &[], 0) - .await; + set.cache_get_object_fileinfo( + ("bucket", object), + Some(generation), + (&fi, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&fi), + &[], + 0, + ) + .await; } set.get_object_metadata_cache.run_pending_tasks().await; assert_eq!(set.get_object_metadata_cache.entry_count(), 3); @@ -4200,8 +5559,15 @@ mod metadata_cache_tests { .get_object_metadata_cache_generation("bucket", "object") .expect("metadata cache generation should be active"); let retired_key = GetObjectMetadataCacheKey::new("bucket", "object", generation); - set.cache_get_object_fileinfo(("bucket", "object"), Some(generation), &fi, std::slice::from_ref(&fi), &[], 0) - .await; + set.cache_get_object_fileinfo( + ("bucket", "object"), + Some(generation), + (&fi, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&fi), + &[], + 0, + ) + .await; for fence in set.get_object_metadata_cache_generations.iter() { fence.store(u64::MAX, Ordering::Release); } @@ -4228,8 +5594,15 @@ mod metadata_cache_tests { let fresh_fi = valid_test_fileinfo("fresh-object"); let generation = set.get_object_metadata_cache_generation("bucket", "fresh-object"); - set.cache_get_object_fileinfo(("bucket", "fresh-object"), generation, &fresh_fi, std::slice::from_ref(&fresh_fi), &[], 0) - .await; + set.cache_get_object_fileinfo( + ("bucket", "fresh-object"), + generation, + (&fresh_fi, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&fresh_fi), + &[], + 0, + ) + .await; assert!( set.cached_get_object_fileinfo("bucket", "fresh-object").await.is_some(), @@ -4254,6 +5627,16 @@ mod tests { const CODEC_STREAMING_TEST_OBJECT: &str = "object"; static CAPTURED_READ_REPAIR_CALLS: AtomicUsize = AtomicUsize::new(0); + #[test] + fn metadata_quorum_outcome_uses_bounded_error_categories() { + assert_eq!(metadata_quorum_outcome(&StorageError::ErasureReadQuorum), "not_reached"); + assert_eq!(metadata_quorum_outcome(&StorageError::FileNotFound), "not_found"); + assert_eq!( + metadata_quorum_outcome(&StorageError::Io(std::io::Error::other("disk unavailable"))), + "unavailable" + ); + } + fn capture_read_repair_submitter( _request: rustfs_heal_contracts::heal_channel::HealChannelRequest, ) -> ReadRepairAdmissionFuture { diff --git a/crates/ecstore/src/storage_api_contracts/mod.rs b/crates/ecstore/src/storage_api_contracts/mod.rs index a03018934..715f090b4 100644 --- a/crates/ecstore/src/storage_api_contracts/mod.rs +++ b/crates/ecstore/src/storage_api_contracts/mod.rs @@ -64,6 +64,8 @@ pub(crate) mod namespace { pub(crate) mod object { use super::{Debug, Error, FileInfo, GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader}; use crate::storage_api_contracts::range::HTTPRangeSpec; + #[cfg(test)] + pub(crate) use rustfs_storage_api::object::DeleteObjectCondition; pub(crate) use rustfs_storage_api::{ DeleteAccounting, DeletedObject, HTTPPreconditions, ObjectIO, ObjectLockDeleteOptions, ObjectLockRetentionOptions, ObjectOperations, ObjectPreconditionError, ObjectPreconditionPart, ObjectPreconditionState, ObjectToDelete, diff --git a/crates/ecstore/src/store/bucket.rs b/crates/ecstore/src/store/bucket.rs index 3792e5fd9..6d835cb6e 100644 --- a/crates/ecstore/src/store/bucket.rs +++ b/crates/ecstore/src/store/bucket.rs @@ -20,6 +20,7 @@ use crate::bucket::{ use crate::error::is_err_bucket_not_found; use crate::runtime::sources as runtime_sources; use crate::set_disk::{BucketInfoQuorum, get_lock_acquire_timeout}; +use crate::storage_api_contracts::admin::DiskSetSelector; use crate::storage_api_contracts::bucket::{BUCKET_LIFECYCLE_LOCK_OBJECT, SRBucketDeleteOp}; use crate::storage_api_contracts::namespace::NamespaceLocking as _; use futures::stream::{self, StreamExt}; @@ -1172,7 +1173,7 @@ impl ECStore { let mut scoped_results = futures::future::join_all(self.bucket_sets().map(|(pool_index, set_index, set)| async move { let result = match quorum { BucketInfoQuorum::Read => set.stat_bucket_with_quorum(bucket, quorum).await, - BucketInfoQuorum::Write => set.get_bucket_info(bucket, opts).await, + BucketInfoQuorum::Write | BucketInfoQuorum::HealCandidates => set.get_bucket_info(bucket, opts).await, }; (pool_index, set_index, result) })) @@ -1252,34 +1253,70 @@ impl ECStore { #[instrument(skip(self))] pub(super) async fn handle_list_bucket(&self, opts: &BucketOptions) -> Result> { // TODO(backlog): support cached bucket listing via opts.cached - Ok(self.list_bucket_from_sets(opts, usize::MAX).await?.buckets) + Ok(self + .list_bucket_from_sets(opts, usize::MAX, BucketInfoQuorum::Write, None) + .await? + .buckets) } pub async fn list_bucket_for_scanner(&self, opts: &BucketOptions) -> Result { - self.list_bucket_from_sets(opts, SCANNER_BUCKET_LIST_SET_CONCURRENCY).await + self.list_bucket_from_sets(opts, SCANNER_BUCKET_LIST_SET_CONCURRENCY, BucketInfoQuorum::Write, None) + .await + } + + /// Discover candidates for metadata initialization at each set's namespace + /// read quorum. Metadata loading still validates persisted configuration; + /// this listing must not be used as a complete scanner or mutation view. + pub async fn list_buckets_for_metadata_startup(&self) -> Result> { + Ok(self + .list_bucket_from_sets( + &BucketOptions { + no_metadata: true, + ..Default::default() + }, + SCANNER_BUCKET_LIST_SET_CONCURRENCY, + BucketInfoQuorum::Read, + None, + ) + .await? + .buckets) + } + + /// Preserve namespace completeness for administrator traversal. A scoped + /// request must not depend on unrelated sets being available. + pub async fn list_buckets_for_admin_heal( + &self, + selector: Option, + ) -> Result { + self.list_bucket_from_sets( + &BucketOptions { + no_metadata: true, + ..Default::default() + }, + SCANNER_BUCKET_LIST_SET_CONCURRENCY, + BucketInfoQuorum::HealCandidates, + selector, + ) + .await } async fn list_bucket_from_sets( &self, opts: &BucketOptions, max_concurrency: usize, + quorum: BucketInfoQuorum, + selector: Option, ) -> Result { - let set_count = self.pools.iter().map(|pool| pool.disk_set.len()).sum(); + let sets = self + .bucket_sets() + .filter(|(pool, set, _)| selector.is_none_or(|scope| scope.pool_idx == *pool && scope.set_idx == *set)) + .collect::>(); + let set_count = sets.len(); let concurrency = bucket_list_set_concurrency(set_count, max_concurrency); - let deleted = opts.deleted; - let cached = opts.cached; - let no_metadata = opts.no_metadata; - let mut set_listings = stream::iter(self.bucket_sets().map(move |(pool_index, set_index, set)| { - let opts = BucketOptions { - deleted, - cached, - no_metadata, - }; - async move { - set.list_bucket_for_scanner(&opts) - .await - .map(|(buckets, complete)| (pool_index, set_index, buckets, complete)) - } + let mut set_listings = stream::iter(sets.into_iter().map(move |(pool_index, set_index, set)| async move { + set.list_bucket_with_quorum(quorum) + .await + .map(|(buckets, complete)| (pool_index, set_index, buckets, complete)) })) .buffer_unordered(concurrency); let mut topology_complete = set_count != 0; @@ -2245,6 +2282,64 @@ mod tests { } } + #[tokio::test] + async fn admin_bucket_discovery_preserves_completeness_and_scope() { + use crate::storage_api_contracts::admin::DiskSetSelector; + let (temp, store) = setup_bucket_quorum_test_env(&[2, 2], None).await; + let empty = store + .list_buckets_for_admin_heal(None) + .await + .expect("empty namespace enumeration"); + assert!(empty.buckets.is_empty()); + assert!(empty.topology_complete); + for disk in 0..2 { + tokio::fs::create_dir(temp.path().join(format!("pool0-disk{disk}/healthy-bucket"))) + .await + .expect("healthy bucket"); + } + tokio::fs::create_dir(temp.path().join("pool1-disk0/partial-bucket")) + .await + .expect("single surviving bucket directory"); + let global = store + .list_buckets_for_admin_heal(None) + .await + .expect("online disk enumeration"); + assert!(global.topology_complete, "every disk answered the candidate inventory"); + assert_eq!( + global.buckets.iter().map(|b| b.name.as_str()).collect::>(), + ["healthy-bucket", "partial-bucket"] + ); + let healthy = store + .list_buckets_for_admin_heal(Some(DiskSetSelector::new(0, 0))) + .await + .expect("healthy scope"); + assert!(healthy.topology_complete); + assert_eq!(healthy.buckets.len(), 1); + assert_eq!(healthy.set_buckets.len(), 1); + let partial = store + .list_buckets_for_admin_heal(Some(DiskSetSelector::new(1, 0))) + .await + .expect("partial scope"); + assert!(partial.topology_complete); + assert_eq!(partial.buckets[0].name, "partial-bucket"); + let invalid = store + .list_buckets_for_admin_heal(Some(DiskSetSelector::new(0, 1))) + .await + .expect("unconfigured scope"); + assert!(!invalid.topology_complete); + assert!(invalid.buckets.is_empty()); + let offline = take_set_disks_offline(&store, &store.pools[1].disk_set[0], &[0]).await; + assert!(store.list_buckets_for_admin_heal(None).await.is_err(), "quorum errors must remain errors"); + assert!( + store + .list_buckets_for_admin_heal(Some(DiskSetSelector::new(0, 0))) + .await + .expect("unrelated offline pool") + .topology_complete + ); + restore_set_disks(&store, &store.pools[1].disk_set[0], offline).await; + } + #[test] fn should_not_override_when_metadata_created_is_unix_epoch() { assert!(!should_override_created_from_metadata(OffsetDateTime::UNIX_EPOCH)); @@ -3564,6 +3659,79 @@ mod tests { assert_eq!(info_err, StorageError::ErasureReadQuorum); } + #[tokio::test] + #[serial] + async fn metadata_startup_listing_uses_each_sets_read_quorum() { + for drives_per_pool in [vec![2], vec![3], vec![4], vec![6], vec![8], vec![4, 6]] { + let (_temp_dir, store) = setup_bucket_quorum_test_env(&drives_per_pool, None).await; + let bucket = "metadata-startup-read-quorum"; + for pool in &store.pools { + pool.disk_set[0] + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("healthy set should create the startup candidate"); + } + let mut offline = Vec::new(); + for (pool_index, &width) in drives_per_pool.iter().enumerate() { + let set = &store.pools[pool_index].disk_set[0]; + let indexes = (width.div_ceil(2)..width).collect::>(); + offline.push(take_set_disks_offline(&store, set, &indexes).await); + } + let listing = store + .list_buckets_for_metadata_startup() + .await + .expect("startup must discover buckets at every set's read quorum"); + assert_eq!(listing.len(), 1); + assert_eq!(listing[0].name, bucket); + if drives_per_pool.iter().any(|width| width % 2 == 0) { + let error = store + .list_bucket_for_scanner(&BucketOptions::default()) + .await + .expect_err("startup discovery must not weaken scanner quorum"); + assert_eq!(error, StorageError::ErasureWriteQuorum); + let error = store + .list_bucket(&BucketOptions { + no_metadata: true, + ..Default::default() + }) + .await + .expect_err("ordinary listing must retain its existing quorum"); + assert_eq!(error, StorageError::ErasureWriteQuorum); + } + // A healthy earlier pool must not hide read-quorum loss in a later pool. + let last_pool = store.pools.len() - 1; + let set = &store.pools[last_pool].disk_set[0]; + let extra = take_set_disks_offline(&store, set, &[0]).await; + let error = store + .list_buckets_for_metadata_startup() + .await + .expect_err("every set must retain read quorum for startup discovery"); + assert_eq!(error, StorageError::ErasureReadQuorum); + restore_set_disks(&store, set, extra).await; + for (pool_index, disks) in offline.into_iter().enumerate() { + restore_set_disks(&store, &store.pools[pool_index].disk_set[0], disks).await; + } + } + } + + #[tokio::test] + #[serial] + async fn metadata_startup_listing_does_not_accept_single_disk_bucket_remnants() { + let (temp_dir, store) = setup_bucket_quorum_test_env(&[4], None).await; + let bucket = "incomplete-startup-bucket"; + tokio::fs::create_dir(temp_dir.path().join("pool0-disk0").join(bucket)) + .await + .expect("create one uncommitted directory remnant"); + let set = &store.pools[0].disk_set[0]; + let offline = take_set_disks_offline(&store, set, &[2, 3]).await; + let listing = store + .list_buckets_for_metadata_startup() + .await + .expect("two responsive disks satisfy discovery quorum"); + assert!(listing.iter().all(|info| info.name != bucket)); + restore_set_disks(&store, set, offline).await; + } + #[tokio::test] #[serial] async fn scanner_bucket_listing_unions_every_erasure_set() { diff --git a/crates/ecstore/src/store/delete_recovery_runtime.rs b/crates/ecstore/src/store/delete_recovery_runtime.rs new file mode 100644 index 000000000..6256a71a9 --- /dev/null +++ b/crates/ecstore/src/store/delete_recovery_runtime.rs @@ -0,0 +1,1290 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Bounded scheduling over the existing on-disk delete journals and receipts. +//! Hints are expendable: startup, returning drives and overflow retain a raw +//! discovery obligation. Only the canonical recovery path decides mutations. + +use super::ECStore; +use crate::bucket::utils::is_meta_bucketname; +use crate::disk::{DeleteRecoveryCursor, DiskAPI, DiskStore}; +use rustfs_heal_contracts::heal_channel::HealOpts; +use std::collections::{HashMap, HashSet, VecDeque}; +use std::sync::{ + Arc, Mutex, + atomic::{AtomicBool, Ordering}, +}; +use std::time::Duration; +use tokio::time::{Instant, MissedTickBehavior}; +use tokio_util::sync::CancellationToken; +use uuid::Uuid; + +const HINT_CAPACITY: usize = 1_024; +const DISCOVERY_PAGE_SIZE: u32 = 128; +const PAGES_PER_PASS: usize = 8; +const MAX_ACTIVE_DISCOVERY_CURSORS: usize = 32; +const LOG_COMPONENT_ECSTORE: &str = "ecstore"; +const LOG_SUBSYSTEM_DELETE_RECOVERY: &str = "delete_recovery"; +const EVENT_DELETE_RECOVERY_STATUS: &str = "delete_recovery_status"; +const KEYS_PER_PASS: usize = 16; +const INTERVAL: Duration = Duration::from_secs(1); +const INVENTORY_INTERVAL: Duration = Duration::from_secs(30); +const DISCOVERY_BUDGET: Duration = Duration::from_secs(5); +const ANTI_ENTROPY_INTERVAL: Duration = Duration::from_secs(60 * 60); +const PASS_START_BUDGET: Duration = Duration::from_secs(30); + +#[derive(Clone, Debug, Eq, Hash, PartialEq)] +pub(crate) struct DeleteRecoveryKey { + pub(crate) pool: usize, + pub(crate) set: usize, + pub(crate) bucket: String, + pub(crate) object: String, + pub(crate) incarnation: Option, +} + +#[derive(Clone, Debug)] +pub(crate) struct DeleteRecoveryTicket { + key: Arc, + generation: Uuid, + admission: Option>, +} + +#[derive(Debug)] +struct AdmissionGuard { + state: Arc>, + key: Arc, + settled: AtomicBool, +} + +impl Drop for AdmissionGuard { + fn drop(&mut self) { + if !self.settled.load(Ordering::Acquire) { + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if !state.pending.contains_key(&self.key) { + // A scan can finish before an evicted request persists its + // journal. Re-arm after that request finishes or is canceled. + state.rescan = Some(Uuid::new_v4()); + } + } + } +} + +#[derive(Debug)] +struct HintState { + pending: HashMap, (Uuid, Uuid, bool)>, + order: VecDeque, + rescan: Option, + started: HashSet, + #[cfg(test)] + compaction_visits: usize, +} + +impl HintState { + fn compact_order(&mut self, capacity: usize) { + // Twice the pending capacity guarantees at least capacity completed + // entries between compactions, even with a full retry backlog. This + // amortizes FIFO maintenance across successful delete requests. + // The queue generation preserves FIFO position across coalescing. + if self.order.len() < capacity.saturating_mul(2) { + return; + } + #[cfg(test)] + { + self.compaction_visits += self.order.len(); + } + let Self { pending, order, .. } = self; + order.retain(|ticket| pending.get(&ticket.key).is_some_and(|entry| entry.1 == ticket.generation)); + } +} + +pub(crate) struct DeleteRecoveryHints { + state: Arc>, + capacity: usize, +} + +impl Default for DeleteRecoveryHints { + fn default() -> Self { + Self::with_capacity(HINT_CAPACITY) + } +} + +impl DeleteRecoveryHints { + fn with_capacity(capacity: usize) -> Self { + Self { + state: Arc::new(Mutex::new(HintState { + pending: HashMap::new(), + order: VecDeque::new(), + rescan: Some(Uuid::new_v4()), + started: HashSet::new(), + #[cfg(test)] + compaction_visits: 0, + })), + capacity, + } + } + + // The queue mutex is held only for in-memory operations and is always + // released before disk, namespace, bucket lifecycle or movement locks. + pub(crate) fn register(&self, key: DeleteRecoveryKey) -> Option { + let key = Arc::new(key); + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let generation = Uuid::new_v4(); + if let Some((current, _, _)) = state.pending.get_mut(&key) { + *current = generation; + } else { + if state.pending.len() >= self.capacity { + // Durable evidence remains the authority. Rotate the oldest + // hint so permanently blocked keys cannot exclude later raw + // discoveries forever; the next scan revisits evicted keys. + while let Some(oldest) = state.order.pop_front() { + if state + .pending + .get(&oldest.key) + .is_some_and(|entry| entry.1 == oldest.generation) + { + state.pending.remove(&oldest.key); + break; + } + } + state.rescan = Some(Uuid::new_v4()); + } + state.compact_order(self.capacity); + state.pending.insert(key.clone(), (generation, generation, false)); + state.order.push_back(DeleteRecoveryTicket { + key: key.clone(), + generation, + admission: None, + }); + } + Some(DeleteRecoveryTicket { + admission: Some(Arc::new(AdmissionGuard { + state: self.state.clone(), + key: key.clone(), + settled: AtomicBool::new(false), + })), + key, + generation, + }) + } + + fn register_discovered(&self, key: DeleteRecoveryKey) -> bool { + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if state.pending.contains_key(&key) { + return true; + } + if state.pending.len() >= self.capacity { + // Only previously attempted work may yield its slot. Unattempted + // keys apply backpressure so a fast scan cannot overwrite them. + let retried = state.order.iter().position(|ticket| { + state + .pending + .get(&ticket.key) + .is_some_and(|entry| entry.1 == ticket.generation && entry.2) + }); + let Some(index) = retried else { return false }; + if let Some(oldest) = state.order.remove(index) { + state.pending.remove(&oldest.key); + } + // This durable key has already been attempted. The hourly sweep + // revisits it; immediate rescans would loop on permanent failures. + } + state.compact_order(self.capacity); + let key = Arc::new(key); + let generation = Uuid::new_v4(); + state.pending.insert(key.clone(), (generation, generation, false)); + state.order.push_back(DeleteRecoveryTicket { + key, + generation, + admission: None, + }); + true + } + + pub(crate) fn ack(&self, ticket: &DeleteRecoveryTicket) { + if let Some(admission) = &ticket.admission { + admission.settled.store(true, Ordering::Release); + } + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if state + .pending + .get(&ticket.key) + .is_some_and(|entry| entry.0 == ticket.generation) + { + state.pending.remove(&ticket.key); + } + } + + fn take(&self, limit: usize) -> Vec { + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let mut tickets = Vec::with_capacity(limit.min(state.pending.len())); + while tickets.len() < limit { + let Some(ticket) = state.order.pop_front() else { break }; + if let Some(&(generation, slot, _)) = state.pending.get(&ticket.key) + && slot == ticket.generation + { + state.pending.remove(&ticket.key); + tickets.push(DeleteRecoveryTicket { + key: ticket.key, + generation, + admission: None, + }); + } + } + tickets + } + + fn retry(&self, ticket: DeleteRecoveryTicket) { + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if state.pending.contains_key(&ticket.key) { + return; + } + state.compact_order(self.capacity); + if state.pending.len() >= self.capacity { + // Previously attempted work yields to admitted keys until the + // periodic sweep. Foreground admission retains its own rescan guard. + return; + } + state + .pending + .insert(ticket.key.clone(), (ticket.generation, ticket.generation, true)); + state.order.push_back(ticket); + } + + fn pending_len(&self) -> usize { + self.state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .pending + .len() + } + + fn request_rescan(&self) { + self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner).rescan = Some(Uuid::new_v4()); + } + + fn rescan(&self) -> Option { + self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner).rescan + } + + fn ack_rescan(&self, generation: Uuid) { + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if state.rescan == Some(generation) { + state.rescan = None; + } + } + + fn start(&self, store: Uuid) -> bool { + self.state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .started + .insert(store) + } +} + +#[derive(Clone)] +struct LocalSource { + pool: usize, + set: usize, + disk: DiskStore, +} + +struct BucketScan { + bucket: String, + incarnation: Option, + cursor: Option, + pending: VecDeque, +} + +struct DiskScan { + source: LocalSource, + buckets: Option>, +} + +fn admit_discovered_keys(hints: &DeleteRecoveryHints, source: &LocalSource, bucket: &mut BucketScan) { + while let Some(object) = bucket.pending.front() { + if !hints.register_discovered(DeleteRecoveryKey { + pool: source.pool, + set: source.set, + bucket: bucket.bucket.clone(), + object: object.clone(), + incarnation: bucket.incarnation, + }) { + break; + } + bucket.pending.pop_front(); + } +} + +#[derive(Default)] +struct RecoveryPass { + sources: Vec, + next_inventory: Option, + next_sweep: Option, + scans: VecDeque, + scan_generation: Option, + deferred: VecDeque<(Instant, DiskScan)>, +} + +impl RecoveryPass { + async fn refresh_inventory(&mut self, store: &Arc) { + if self.next_inventory.is_some_and(|next| Instant::now() < next) { + return; + } + let mut sources = Vec::new(); + for pool in &store.pools { + for set in &pool.disk_set { + for disk in set + .disk_inventory() + .await + .into_iter() + .flatten() + .filter(|disk| disk.is_local()) + { + sources.push(LocalSource { + pool: set.pool_index, + set: set.set_index, + disk, + }); + } + } + } + let current_source = |source: &LocalSource| { + sources.iter().any(|current| { + current.pool == source.pool && current.set == source.set && Arc::ptr_eq(¤t.disk, &source.disk) + }) + }; + if sources.len() != self.sources.len() || self.sources.iter().any(|source| !current_source(source)) { + store.ctx.delete_recovery_hints().request_rescan(); + self.scans.retain(|scan| current_source(&scan.source)); + self.deferred.retain(|(_, scan)| current_source(&scan.source)); + } + self.sources = sources; + self.next_inventory = Some(Instant::now() + INVENTORY_INTERVAL); + } + + fn defer(&mut self, mut scan: DiskScan) { + if let Some(buckets) = scan.buckets.as_mut() { + for bucket in buckets { + bucket.cursor = None; + bucket.incarnation = None; + } + } + let same_scope = |other: &DiskScan| { + scan.source.pool == other.source.pool + && scan.source.set == other.source.set + && scan.source.disk.to_string() == other.source.disk.to_string() + && scan + .buckets + .as_ref() + .and_then(|buckets| buckets.front()) + .map(|bucket| &bucket.bucket) + == other + .buckets + .as_ref() + .and_then(|buckets| buckets.front()) + .map(|bucket| &bucket.bucket) + }; + if !self.deferred.iter().any(|(_, pending)| same_scope(pending)) { + self.deferred.push_back((Instant::now() + INVENTORY_INTERVAL, scan)); + } + } + + async fn discover_page(&mut self, store: &Arc, deadline: Instant) { + let hints = store.ctx.delete_recovery_hints(); + let active_cursor = |scan: &DiskScan| { + scan.buckets + .as_ref() + .and_then(|buckets| buckets.front()) + .is_some_and(|bucket| bucket.cursor.is_some() || !bucket.pending.is_empty()) + }; + if self.scans.iter().filter(|scan| active_cursor(scan)).count() >= MAX_ACTIVE_DISCOVERY_CURSORS { + // Keep room for exact-key recovery and other discovery callers in + // the disk API's bounded session table. Finish existing cursors + // before starting another bucket; queued sources retain their place. + if let Some(index) = self.scans.iter().position(active_cursor) { + self.scans.rotate_left(index); + } + } + let Some(mut scan) = self.scans.pop_front() else { return }; + if scan.buckets.is_none() { + match tokio::time::timeout_at(deadline, scan.source.disk.list_volumes()).await { + Ok(Ok(volumes)) => { + scan.buckets = Some( + volumes + .into_iter() + .filter(|volume| !is_meta_bucketname(&volume.name)) + .map(|volume| BucketScan { + bucket: volume.name, + incarnation: None, + cursor: None, + pending: VecDeque::new(), + }) + .collect(), + ); + } + _ => { + self.defer(scan); + return; + } + } + } else if let Some(buckets) = scan.buckets.as_mut() + && let Some(mut bucket) = buckets.pop_front() + { + let had_pending = !bucket.pending.is_empty(); + admit_discovered_keys(hints, &scan.source, &mut bucket); + if !bucket.pending.is_empty() || (had_pending && bucket.cursor.is_none()) { + if !bucket.pending.is_empty() { + buckets.push_front(bucket); + } + if !buckets.is_empty() { + self.scans.push_back(scan); + } + return; + } + if bucket.incarnation.is_none() { + match tokio::time::timeout_at(deadline, store.bucket_incarnation_id_from_disk(&bucket.bucket)).await { + Ok(Ok(incarnation)) => bucket.incarnation = Some(incarnation), + _ => { + self.defer(DiskScan { + source: scan.source.clone(), + buckets: Some(VecDeque::from([bucket])), + }); + if !buckets.is_empty() { + self.scans.push_back(scan); + } + return; + } + } + } + match tokio::time::timeout_at( + deadline, + scan.source + .disk + .discover_delete_transactions(&bucket.bucket, None, bucket.cursor.take(), DISCOVERY_PAGE_SIZE), + ) + .await + { + Ok(Ok(page)) => { + bucket.pending = page.transactions.into_iter().map(|transaction| transaction.object).collect(); + bucket.cursor = page.next; + admit_discovered_keys(hints, &scan.source, &mut bucket); + if bucket.cursor.is_some() || !bucket.pending.is_empty() { + // Keep one bounded page until its keys are admitted; + // unopened buckets cannot displace its continuation. + buckets.push_front(bucket); + } + } + _ => { + // Retry only this source, never a full healthy tree + // because another disk is offline or a cursor expired. + self.defer(DiskScan { + source: scan.source.clone(), + buckets: Some(VecDeque::from([bucket])), + }); + } + } + } + if scan.buckets.as_ref().is_some_and(|buckets| !buckets.is_empty()) { + self.scans.push_back(scan); + } + } + + async fn discover(&mut self, store: &Arc, cancel: &CancellationToken) { + let hints = store.ctx.delete_recovery_hints(); + // Renew only this worker's bounded local cursors before any await. + // Backpressure and a slow canonical rollback must not erase progress. + for scan in &mut self.scans { + if let Some(bucket) = scan.buckets.as_mut().and_then(|buckets| buckets.front_mut()) + && let Some(cursor) = bucket.cursor.as_ref() + { + let renewed = match scan.source.disk.as_ref() { + crate::disk::Disk::Local(local) => local.get_disk().renew_delete_recovery_cursor(&bucket.bucket, cursor), + crate::disk::Disk::Remote(_) => false, + }; + if !renewed { + bucket.cursor = None; + hints.request_rescan(); + } + } + } + let discovery_deadline = Instant::now() + DISCOVERY_BUDGET; + tokio::select! { + _ = cancel.cancelled() => return, + result = tokio::time::timeout_at(discovery_deadline, self.refresh_inventory(store)) => { + if result.is_err() { hints.request_rescan(); } + } + } + if self.scan_generation.is_none() { + if self.next_sweep.is_some_and(|next| Instant::now() >= next) { + hints.request_rescan(); + self.next_sweep = None; + } + if let Some(generation) = hints.rescan() { + self.scan_generation = Some(generation); + self.scans = self + .sources + .iter() + .cloned() + .map(|source| DiskScan { source, buckets: None }) + .collect(); + } + } + for _ in 0..self.deferred.len() { + let Some((ready, scan)) = self.deferred.pop_front() else { break }; + if ready <= Instant::now() { + self.scans.push_back(scan); + } else { + self.deferred.push_back((ready, scan)); + } + } + for _ in 0..PAGES_PER_PASS { + if self.scans.is_empty() || Instant::now() >= discovery_deadline { + break; + } + tokio::select! { + _ = cancel.cancelled() => return, + _ = self.discover_page(store, discovery_deadline) => {} + } + } + if self.scans.is_empty() + && let Some(generation) = self.scan_generation.take() + { + hints.ack_rescan(generation); + self.next_sweep = Some(Instant::now() + ANTI_ENTROPY_INTERVAL); + } + } + + #[cfg(test)] + async fn run(&mut self, store: &Arc, cancel: &CancellationToken) { + self.discover(store, cancel).await; + recover_hints(store, cancel).await; + } +} + +#[derive(Default, PartialEq, Eq)] +struct RecoveryFailures { + pending: usize, + blocked: usize, + corrupt: usize, + failed: usize, +} + +async fn recover_hints(store: &Arc, cancel: &CancellationToken) -> RecoveryFailures { + let hints = store.ctx.delete_recovery_hints(); + let deadline = Instant::now() + PASS_START_BUDGET; + let mut failures = RecoveryFailures::default(); + // Remove only the selected generation before awaiting. A concurrent + // DELETE can enqueue the same key independently; neither successful + // recovery nor retry of this ticket acknowledges that newer hint. + let attempts = hints.pending_len().min(KEYS_PER_PASS); + for _ in 0..attempts { + if Instant::now() >= deadline || cancel.is_cancelled() { + break; + } + // Only the operation about to start leaves the queue. A slow first + // key must not turn untouched work into evictable retry entries. + let Some(ticket) = hints.take(1).pop() else { break }; + let key = &ticket.key; + let opts = HealOpts { + pool: Some(key.pool), + set: Some(key.set), + ..Default::default() + }; + // The pass deadline limits starting work, not an admitted rollback. + // Canonical disk/RPC recovery owns its configured large-shard timeout. + let outcome = tokio::select! { + _ = cancel.cancelled() => None, + result = store.recover_delete_transactions_at_incarnation( + &key.bucket, Some(&key.object), key.incarnation, &opts, + ) => Some(result), + }; + if !matches!(outcome, Some(Ok(_))) { + match outcome { + Some(Err(crate::error::StorageError::ErasureReadQuorum)) => failures.pending += 1, + Some(Err(crate::error::StorageError::SlowDown)) => failures.blocked += 1, + Some(Err(crate::error::StorageError::FileCorrupt)) => failures.corrupt += 1, + Some(Err(_)) => failures.failed += 1, + _ => {} + } + hints.retry(ticket); + } + } + failures +} + +pub(super) fn start_delete_recovery(store: Arc, cancel: CancellationToken) { + if !store.ctx.delete_recovery_hints().start(store.id) { + return; + } + tokio::spawn(async move { + let discovery = async { + let mut interval = tokio::time::interval(INTERVAL); + interval.set_missed_tick_behavior(MissedTickBehavior::Delay); + let mut pass = RecoveryPass::default(); + let mut previous_deferred = 0; + loop { + tokio::select! { + biased; + _ = cancel.cancelled() => return, + _ = interval.tick() => {}, + } + pass.discover(&store, &cancel).await; + let deferred = pass.deferred.len(); + if deferred != previous_deferred { + tracing::warn!( + event = EVENT_DELETE_RECOVERY_STATUS, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DELETE_RECOVERY, + state = "discovery_retry_changed", + deferred_sources = deferred, + pending_keys = store.ctx.delete_recovery_hints().pending_len(), + "Delete recovery discovery retry state changed" + ); + previous_deferred = deferred; + } + } + }; + let recovery = async { + let mut interval = tokio::time::interval(INTERVAL); + interval.set_missed_tick_behavior(MissedTickBehavior::Delay); + let mut previous = RecoveryFailures::default(); + loop { + tokio::select! { + biased; + _ = cancel.cancelled() => return, + _ = interval.tick() => {}, + } + let failures = recover_hints(&store, &cancel).await; + if failures != previous { + tracing::warn!( + event = EVENT_DELETE_RECOVERY_STATUS, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DELETE_RECOVERY, + state = "recovery_retry_changed", + pending = failures.pending, + blocked = failures.blocked, + corrupt = failures.corrupt, + failed = failures.failed, + pending_keys = store.ctx.delete_recovery_hints().pending_len(), + "Delete recovery settlement retry state changed" + ); + previous = failures; + } + } + }; + // Long canonical verification uses its configured deadline while raw + // discovery continues to advance bounded, expiring disk cursors. + tokio::join!(discovery, recovery); + }); +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks}; + use crate::disk::{DeleteOptions, DeleteRecoveryAction, STORAGE_FORMAT_FILE}; + use crate::object_api::{ObjectOptions, PutObjReader}; + use crate::storage_api_contracts::bucket::{BucketOperations, MakeBucketOptions}; + use crate::storage_api_contracts::object::{ObjectIO, ObjectOperations}; + use rustfs_filemeta::FileInfo; + + fn key(object: &str) -> DeleteRecoveryKey { + DeleteRecoveryKey { + pool: 0, + set: 0, + bucket: "bucket".to_owned(), + object: object.to_owned(), + incarnation: Some(Uuid::from_u128(1)), + } + } + + #[test] + fn delete_recovery_hints_coalesce_without_losing_fifo_or_new_generations() { + let hints = DeleteRecoveryHints::with_capacity(2); + let first = hints.register(key("a")).expect("first hint"); + hints.register(key("b")).expect("second hint"); + let newer = hints.register(key("a")).expect("coalesced hint"); + hints.ack(&first); + let taken = hints.take(1); + assert_eq!(taken.len(), 1); + assert_eq!(taken[0].key.object, "a", "coalescing must preserve fairness"); + assert_eq!(taken[0].generation, newer.generation); + let latest = hints.register(key("a")).expect("hint during recovery"); + hints.ack(&taken[0]); + hints.retry(taken.into_iter().next().expect("selected ticket")); + let pending = hints.take(2); + assert_eq!(pending[0].key.object, "b"); + assert_eq!(pending[1].generation, latest.generation, "old recovery cannot replace a newer hint"); + } + + #[test] + fn delete_recovery_overflow_rearms_after_preparation_and_preserves_new_scan_obligations() { + let hints = DeleteRecoveryHints::with_capacity(1); + let overflow = hints.register(key("occupied")).expect("retain request lifetime"); + hints.register(key("later")).expect("evict before preparation"); + let scan = hints.rescan().expect("overflow requests discovery"); + hints.ack_rescan(scan); + assert!(hints.rescan().is_none()); + drop(overflow); + let next = hints.rescan().expect("completion after discovery must rearm overflow"); + hints.request_rescan(); + hints.ack_rescan(next); + assert!(hints.rescan().is_some(), "old scan cannot clear a newer obligation"); + let completed = hints.register(key("completed")).expect("overflow ticket"); + hints.ack(&completed); + let final_scan = hints.rescan().expect("queued scan"); + hints.ack_rescan(final_scan); + drop(completed); + assert!(hints.rescan().is_none(), "synchronous settlement needs no new scan"); + } + + #[test] + fn delete_recovery_hints_bound_tombstones_and_retry_after_drain() { + let hints = DeleteRecoveryHints::with_capacity(2); + for _ in 0..100 { + let ticket = hints.register(key("same")).expect("register reusable key"); + hints.ack(&ticket); + } + let _a = hints.register(key("a")).expect("a"); + hints.register(key("b")).expect("b"); + assert!(hints.state.lock().expect("hint state").order.len() <= 2); + let mut tickets = hints.take(2); + assert_eq!(tickets.len(), 2); + hints.retry(tickets.remove(0)); + assert_eq!(hints.take(2).len(), 1, "failed work survives an empty queue"); + } + + #[test] + fn delete_recovery_hint_compaction_stays_amortized_near_capacity() { + let capacity = HINT_CAPACITY; + let completed = capacity * 4; + let hints = DeleteRecoveryHints::with_capacity(capacity); + let blocked: Vec<_> = (0..capacity - 1) + .map(|index| hints.register(key(&format!("blocked-{index}"))).expect("blocked hint")) + .collect(); + hints.ack_rescan(hints.rescan().expect("startup discovery")); + for _ in 0..completed { + let ticket = hints.register(key("completed")).expect("healthy delete"); + hints.ack(&ticket); + let state = hints.state.lock().expect("hint state"); + assert_eq!(state.pending.len(), capacity - 1, "healthy completion cannot evict blocked work"); + assert!(state.order.len() <= capacity * 2, "tombstones remain bounded"); + } + let visits = hints.state.lock().expect("hint state").compaction_visits; + eprintln!("compaction visited {visits} tickets for {completed} successful deletes near capacity"); + assert!( + visits <= (completed + capacity) * 2, + "queue maintenance must be amortized, not capacity per request" + ); + assert!(hints.rescan().is_none(), "successful deletes must not cause false overflow"); + let remaining = hints.take(capacity); + assert_eq!(remaining.len(), blocked.len()); + for (actual, expected) in remaining.iter().zip(&blocked) { + assert_eq!(actual.generation, expected.generation, "compaction preserves FIFO and generations"); + } + } + + #[test] + fn delete_recovery_saturated_blocked_hints_cannot_exclude_later_discovery() { + let hints = DeleteRecoveryHints::with_capacity(2); + hints.register(key("blocked-a")).expect("first blocked key"); + hints.register(key("blocked-b")).expect("second blocked key"); + for ticket in hints.take(2) { + hints.retry(ticket); + } + hints + .register(key("healthy-later")) + .expect("later page can replace oldest hint"); + let next = hints.take(2); + assert_eq!(next.len(), 2); + assert!(next.iter().any(|ticket| ticket.key.object == "healthy-later")); + assert!(hints.rescan().is_some(), "evicted durable work remains a discovery obligation"); + } + + async fn fixture(bucket: &str) -> (Vec, Arc, Vec) { + let (dirs, store) = isolated_store_over_temp_disks().await; + // These tests drive discovery and recovery explicitly. Stop the loops + // started by ECStore::init before adding work or advancing virtual time. + store.background_cancel_token().expect("fixture background token").cancel(); + tokio::task::yield_now().await; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create recovery bucket"); + let mut reader = PutObjReader::from_vec(vec![19; 4096]); + store + .put_object( + bucket, + "object", + &mut reader, + &ObjectOptions { + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("write recovery source"); + let disks = store.pools[0].disk_set[0] + .disk_inventory() + .await + .into_iter() + .map(|disk| disk.expect("online disk")) + .collect(); + (dirs, store, disks) + } + + async fn prepare(store: &Arc, disks: &[DiskStore], bucket: &str) -> Uuid { + let transaction = Uuid::new_v4(); + let incarnation = store + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("bucket incarnation"); + for disk in disks { + disk.delete_version_recoverable( + bucket, + "object", + FileInfo { + name: "object".to_owned(), + ..Default::default() + }, + false, + DeleteOptions { + old_data_dir: Some(transaction), + delete_recovery_bucket_incarnation: Some(incarnation), + ..Default::default() + }, + ) + .await + .expect("persist preparation"); + } + transaction + } + + async fn publish_one(disk: &DiskStore, bucket: &str, transaction: Uuid) { + let inspection = disk + .inspect_delete_transaction(bucket, "object", transaction) + .await + .expect("inspect preparation"); + disk.settle_delete_transaction( + bucket, + "object", + transaction, + inspection.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .expect("persist receipt with lost response"); + } + + fn hint_only_pass(store: &Arc) -> RecoveryPass { + let hints = store.ctx.delete_recovery_hints(); + if let Some(scan) = hints.rescan() { + hints.ack_rescan(scan); + } + RecoveryPass { + next_inventory: Some(Instant::now() + INVENTORY_INTERVAL), + ..Default::default() + } + } + + async fn register(store: &Arc, bucket: &str) { + store.ctx.delete_recovery_hints().register(DeleteRecoveryKey { + bucket: bucket.to_owned(), + object: "object".to_owned(), + pool: 0, + set: 0, + incarnation: Some( + store + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("source incarnation"), + ), + }); + } + + #[tokio::test] + async fn delete_recovery_clean_startup_discovers_hidden_preparation() { + let bucket = "runtime-clean-start"; + let (_dirs, store, disks) = fixture(bucket).await; + prepare(&store, &disks, bucket).await; + assert!( + disks[0] + .read_all(bucket, &format!("object/{STORAGE_FORMAT_FILE}")) + .await + .is_err() + ); + let mut pass = RecoveryPass::default(); + for _ in 0..8 { + pass.run(&store, &CancellationToken::new()).await; + } + assert!( + disks[0] + .read_all(bucket, &format!("object/{STORAGE_FORMAT_FILE}")) + .await + .is_ok(), + "startup must restore an uncommitted hidden object without a client DELETE or dirty shutdown" + ); + assert!( + store.ctx.delete_recovery_hints().rescan().is_none(), + "a successful scan is not repeated each tick" + ); + } + + #[tokio::test] + async fn delete_recovery_partial_publication_converges_without_another_delete() { + let bucket = "runtime-lost-response"; + let (_dirs, store, disks) = fixture(bucket).await; + let mut pass = hint_only_pass(&store); + register(&store, bucket).await; + let transaction = prepare(&store, &disks, bucket).await; + publish_one(&disks[0], bucket, transaction).await; + pass.run(&store, &CancellationToken::new()).await; + for disk in &disks { + let page = disk + .discover_delete_transactions(bucket, Some("object"), None, 128) + .await + .expect("inspect cleanup"); + assert!( + page.transactions.is_empty(), + "partial publication must converge through canonical recovery" + ); + } + assert!(store.ctx.delete_recovery_hints().take(1).is_empty()); + } + + #[tokio::test] + async fn delete_recovery_offline_participant_retains_work_until_return() { + let bucket = "runtime-offline-return"; + let (_dirs, store, disks) = fixture(bucket).await; + let mut pass = hint_only_pass(&store); + register(&store, bucket).await; + let transaction = prepare(&store, &disks, bucket).await; + publish_one(&disks[0], bucket, transaction).await; + let set = &store.pools[0].disk_set[0]; + let offline = set.disks.write().await[3].take(); + pass.run(&store, &CancellationToken::new()).await; + assert_eq!( + store + .ctx + .delete_recovery_hints() + .state + .lock() + .expect("pending state") + .pending + .len(), + 1 + ); + set.disks.write().await[3] = offline; + pass.run(&store, &CancellationToken::new()).await; + assert!( + store.ctx.delete_recovery_hints().take(1).is_empty(), + "returning participant releases retained work" + ); + assert!( + disks[0] + .discover_delete_transactions(bucket, Some("object"), None, 128) + .await + .expect("reclaimed journals") + .transactions + .is_empty() + ); + } + + #[tokio::test] + async fn delete_recovery_raw_discovery_is_bounded_and_rotates_disks_and_buckets() { + let bucket = "runtime-page-budget"; + let (_dirs, store, _disks) = fixture(bucket).await; + for index in 0..8 { + store + .make_bucket(&format!("runtime-other-{index}"), &MakeBucketOptions::default()) + .await + .expect("create scan bucket"); + } + let mut pass = RecoveryPass::default(); + pass.run(&store, &CancellationToken::new()).await; + assert_eq!(pass.scans.len(), 4, "one pass cannot consume all bucket pages"); + let remaining: Vec<_> = pass + .scans + .iter() + .map(|scan| scan.buckets.as_ref().expect("initialized disk").len()) + .collect(); + assert!( + remaining.iter().all(|count| *count == 8), + "each disk receives one page fairly after volume discovery" + ); + for _ in 0..8 { + pass.run(&store, &CancellationToken::new()).await; + } + assert!(pass.scans.is_empty()); + assert!(store.ctx.delete_recovery_hints().rescan().is_none()); + } + #[tokio::test] + async fn delete_recovery_blocked_capacity_yields_until_periodic_sweep() { + let (_dirs, store, _disks) = fixture("runtime-blocked-sweep").await; + let hints = store.ctx.delete_recovery_hints(); + hints.request_rescan(); + let scanning = hints.rescan().expect("startup discovery obligation"); + for index in 0..HINT_CAPACITY { + assert!(hints.register_discovered(key(&format!("blocked-{index}")))); + } + for ticket in hints.take(KEYS_PER_PASS) { + hints.retry(ticket); + } + for index in 0..KEYS_PER_PASS { + assert!( + hints.register_discovered(key(&format!("later-page-{index}"))), + "attempted failures yield to later discovery" + ); + } + let in_flight = hints.take(1).pop().expect("one admitted recovery attempt"); + assert!(hints.register_discovered(key("arrived-during-recovery"))); + hints.retry(in_flight); + let mut pass = RecoveryPass { + next_inventory: Some(Instant::now() + ANTI_ENTROPY_INTERVAL), + scan_generation: Some(scanning), + ..Default::default() + }; + pass.discover(&store, &CancellationToken::new()).await; + assert!( + hints.rescan().is_none(), + "attempted failures cannot invalidate the completed scan generation" + ); + let scheduled = pass.next_sweep.expect("periodic discovery remains scheduled"); + assert!(scheduled >= Instant::now() + ANTI_ENTROPY_INTERVAL - Duration::from_secs(1)); + pass.discover(&store, &CancellationToken::new()).await; + assert!(pass.scan_generation.is_none()); + assert_eq!(pass.next_sweep, Some(scheduled), "the next tick must wait for the hourly sweep"); + hints + .register(key("foreground-new-request")) + .expect("foreground overflow remains tracked"); + let foreground = hints + .rescan() + .expect("foreground eviction still requests immediate discovery"); + assert_ne!(foreground, scanning); + hints.ack_rescan(scanning); + assert_eq!(hints.rescan(), Some(foreground), "an older scan cannot clear a foreground obligation"); + } + + #[tokio::test] + async fn delete_recovery_slow_first_key_preserves_unattempted_backpressure() { + let bucket = "runtime-slow-first"; + let (_dirs, store, _disks) = fixture(bucket).await; + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("bucket identity"); + let hints = store.ctx.delete_recovery_hints(); + for index in 0..HINT_CAPACITY { + hints.register(DeleteRecoveryKey { + pool: 0, + set: 0, + bucket: bucket.to_owned(), + object: format!("waiting-{index}"), + incarnation: Some(incarnation), + }); + } + let gate = store.ctx.data_movement_operation_gate(); + let held = gate.write().await; + tokio::time::pause(); + let worker_store = store.clone(); + let worker = tokio::spawn(async move { recover_hints(&worker_store, &CancellationToken::new()).await }); + for _ in 0..100 { + if hints.pending_len() != HINT_CAPACITY { + break; + } + tokio::task::yield_now().await; + } + assert_eq!(hints.pending_len(), HINT_CAPACITY - 1, "only the admitted key leaves the queue"); + assert!(hints.register_discovered(key("newly-discovered")), "one in-flight key releases one slot"); + assert!(!hints.register_discovered(key("must-wait")), "untouched keys must apply backpressure"); + tokio::time::advance(PASS_START_BUDGET + Duration::from_secs(1)).await; + drop(held); + worker.await.expect("finish the admitted operation"); + tokio::time::resume(); + let state = hints.state.lock().expect("retained pending work"); + for index in 1..HINT_CAPACITY { + let queued = state + .pending + .iter() + .find(|(key, _)| key.object == format!("waiting-{index}")) + .expect("unattempted key remains queued") + .1; + assert!(!queued.2, "budget exhaustion must not grant discovery permission to evict untouched work"); + } + } + + #[test] + fn delete_recovery_backpressure_eventually_attempts_every_healthy_key() { + for total in [4_096, 16_384] { + let hints = DeleteRecoveryHints::with_capacity(HINT_CAPACITY); + let mut recovered = HashSet::new(); + let mut next = 0usize; + let healthy = total - total / 1_024 * KEYS_PER_PASS; + for _ in 0..(total / KEYS_PER_PASS * 4) { + for _ in 0..(PAGES_PER_PASS * usize::try_from(DISCOVERY_PAGE_SIZE).expect("page size")) { + if !recovered.contains(&next) && !hints.register_discovered(key(&next.to_string())) { + break; + } + next = (next + 1) % total; + } + for ticket in hints.take(KEYS_PER_PASS) { + let index: usize = ticket.key.object.parse().expect("numeric test key"); + if index % 1_024 < KEYS_PER_PASS { + hints.retry(ticket); + } else { + recovered.insert(index); + } + } + assert!(hints.pending_len() <= HINT_CAPACITY); + if recovered.len() == healthy { + break; + } + } + assert_eq!( + recovered.len(), + healthy, + "permanent failures at every page-group boundary cannot starve later keys" + ); + } + } + + #[tokio::test] + async fn delete_recovery_real_single_and_batch_delete_register_and_ack_cleanup() { + for batch in [false, true] { + for held_snapshot in [false, true] { + let bucket = "runtime-real-delete"; + let (_dirs, store, disks) = fixture(bucket).await; + let mut reader = PutObjReader::from_vec(vec![37; 1024 * 1024]); + store + .put_object( + bucket, + "object", + &mut reader, + &ObjectOptions { + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("external source for snapshot cleanup"); + let _pass = hint_only_pass(&store); + let mut leases = Vec::new(); + if held_snapshot { + for disk in &disks { + let info = disk + .read_version("", bucket, "object", "", &crate::disk::ReadOptions::default()) + .await + .expect("source data directory"); + let path = format!("object/{}", info.data_dir.expect("external payload")); + let lease = disk.acquire_snapshot_lease(bucket, &path).await.expect("protect cleanup"); + leases.push((disk.clone(), path, lease)); + } + } + if batch { + let (_, errors) = store + .delete_objects( + bucket, + vec![crate::storage_api_contracts::object::ObjectToDelete { + object_name: "object".to_owned(), + ..Default::default() + }], + ObjectOptions::default(), + ) + .await; + assert!(errors.iter().all(Option::is_none), "logical batch delete succeeds: {errors:?}"); + } else { + store + .delete_object(bucket, "object", ObjectOptions::default()) + .await + .expect("logical single delete succeeds"); + } + assert_eq!( + store.ctx.delete_recovery_hints().pending_len(), + usize::from(held_snapshot), + "the real preparation hook retains cleanup; synchronous settlement acknowledges its ticket" + ); + if held_snapshot { + for (disk, path, lease) in leases { + disk.release_snapshot_lease(bucket, &path, lease) + .await + .expect("release deferred payload"); + } + recover_hints(&store, &CancellationToken::new()).await; + assert_eq!( + store.ctx.delete_recovery_hints().pending_len(), + 0, + "maintenance finishes the real DELETE without manual registration" + ); + } + for disk in &disks { + assert!( + disk.discover_delete_transactions(bucket, Some("object"), None, 128) + .await + .expect("no residual delete journal") + .transactions + .is_empty() + ); + } + } + } + } + + #[tokio::test] + async fn delete_recovery_continues_current_bucket_before_opening_later_buckets() { + let bucket = "runtime-cursor-progress"; + let (dirs, store, disks) = fixture(bucket).await; + let other = "runtime-after-large"; + store + .make_bucket(other, &MakeBucketOptions::default()) + .await + .expect("next bucket"); + for index in 0..4_130 { + std::fs::write(dirs[0].path().join(bucket).join(format!("filler-{index}")), []).expect("raw discovery visit budget"); + } + let mut pass = RecoveryPass::default(); + pass.scans.push_back(DiskScan { + source: LocalSource { + pool: 0, + set: 0, + disk: disks[0].clone(), + }, + buckets: Some(VecDeque::from([bucket, other].map(|bucket| BucketScan { + bucket: bucket.to_owned(), + incarnation: None, + cursor: None, + pending: VecDeque::new(), + }))), + }); + pass.discover_page(&store, Instant::now() + Duration::from_secs(30)).await; + let first = pass.scans[0] + .buckets + .as_ref() + .expect("pending buckets") + .front() + .expect("current bucket"); + assert_eq!(first.bucket, bucket); + assert!(first.cursor.is_some(), "raw entry budget must leave a continuation"); + pass.discover_page(&store, Instant::now() + Duration::from_secs(30)).await; + assert_eq!( + pass.scans[0] + .buckets + .as_ref() + .expect("remaining bucket") + .front() + .expect("next bucket") + .bucket, + other, + "the second page completes instead of cycling through unopened buckets" + ); + } +} diff --git a/crates/ecstore/src/store/heal.rs b/crates/ecstore/src/store/heal.rs index 1cc52b417..3175c17f0 100644 --- a/crates/ecstore/src/store/heal.rs +++ b/crates/ecstore/src/store/heal.rs @@ -121,6 +121,47 @@ enum HealFormatPoolSkip { Retryable, } +#[derive(Debug)] +enum HealFormatRebalance { + Missing, + Current(RebalanceMeta), + Historical, +} + +impl HealFormatRebalance { + fn current(&self) -> Option<&RebalanceMeta> { + match self { + Self::Current(meta) => Some(meta), + Self::Missing | Self::Historical => None, + } + } +} + +#[derive(Debug, thiserror::Error)] +#[error( + "format repair blocked: rebalance metadata describes {stored_pools} pools but the deployment has {current_pools}; the historical run is not confirmed terminal" +)] +struct HealFormatRebalanceTopologyMismatch { + stored_pools: usize, + current_pools: usize, +} + +fn classify_heal_format_rebalance(meta: Option, current_pools: usize) -> Result { + let Some(meta) = meta else { + return Ok(HealFormatRebalance::Missing); + }; + if meta.pool_stats.len() == current_pools { + return Ok(HealFormatRebalance::Current(meta)); + } + if meta.has_terminal_topology() { + return Ok(HealFormatRebalance::Historical); + } + Err(Error::other(HealFormatRebalanceTopologyMismatch { + stored_pools: meta.pool_stats.len(), + current_pools, + })) +} + fn classify_heal_format_pool( pool_idx: usize, pool_cmd_line: &str, @@ -274,13 +315,159 @@ impl ECStore { .map_err(|error| Error::other_with_context("bucket heal owner task failed", error))? } + /// Recover hidden delete journals through the lifecycle-owned heal boundary. + pub async fn recover_delete_transactions(self: &Arc, bucket: &str, opts: &HealOpts) -> Result { + Ok(self + .recover_delete_transactions_at_incarnation(bucket, None, None, opts) + .await? + .unwrap_or_else(|| HealResultItem { + heal_item_type: HealItemType::Bucket.to_string(), + bucket: bucket.to_owned(), + detail: "delete recovery: discovered=0".to_owned(), + ..Default::default() + })) + } + + pub async fn recover_delete_transactions_at_incarnation( + self: &Arc, + bucket: &str, + object: Option<&str>, + expected: Option, + opts: &HealOpts, + ) -> Result> { + self.recover_delete_transactions_in_scope(bucket, object, None, expected, opts) + .await + } + + pub async fn recover_delete_prefix_at_incarnation( + self: &Arc, + bucket: &str, + prefix: &str, + expected: Option, + opts: &HealOpts, + ) -> Result> { + self.recover_delete_transactions_in_scope(bucket, None, Some(prefix), expected, opts) + .await + } + + async fn recover_delete_transactions_in_scope( + self: &Arc, + bucket: &str, + object: Option<&str>, + prefix: Option<&str>, + expected: Option, + opts: &HealOpts, + ) -> Result> { + if opts.no_lock && !opts.dry_run { + return Err(StorageError::InvalidArgument( + "heal".to_string(), + "no_lock".to_string(), + "delete recovery requires namespace locking".to_string(), + )); + } + let expected = match expected { + Some(expected) => expected, + None => self.bucket_incarnation_id_from_disk(bucket).await?, + }; + let object = object.map(str::to_owned); + let prefix = prefix.map(str::to_owned); + self.run_bucket_heal_at_incarnation(bucket, expected, opts, |store, bucket, opts| async move { + let scope = bucket_heal_scope(&bucket).ok_or(StorageError::PreconditionFailed)?; + let explicit_delete = opts.remove + && metadata_sys::has_authoritative_never_versioned_state_in(&store.ctx, &bucket).await? + && matches!( + metadata_sys::get_object_lock_config_state_in(&store.ctx, &bucket).await?, + metadata_sys::ObjectLockConfigState::ConfirmedAbsent + ); + let movement_gate = store.ctx.data_movement_operation_gate(); + let _movement_guard = movement_gate.read().await; + let can_commit = || scope.check().is_ok(); + let recovery_context = crate::set_disk::DeleteRecoveryContext { + dry_run: opts.dry_run, + explicit_delete, + namespace_owner: (!opts.dry_run) + .then(|| Arc::new((scope.clone(), store.ctx.begin_namespace_commit())) as Arc), + ..crate::set_disk::DeleteRecoveryContext::new(Some(scope.incarnation), &can_commit) + }; + let mut summary = crate::set_disk::DeleteRecoverySummary::default(); + for pool in store.get_pools_for_heal_object(&opts)? { + if store.is_suspended(pool.pool_idx).await || store.is_pool_rebalancing(pool.pool_idx).await { + return Err(StorageError::SlowDown); + } + for (set_index, set) in pool.disk_set.iter().enumerate() { + if opts.set.is_some_and(|requested| requested != set_index) { + continue; + } + if let Some(object) = &object { + if !Arc::ptr_eq(set, &pool.get_disks_by_key(object)) { + continue; + } + summary.add(set.recover_delete_object(&bucket, object, recovery_context.clone()).await?); + } else if let Some(prefix) = &prefix { + summary.add(set.recover_delete_prefix(&bucket, prefix, recovery_context.clone()).await?); + } else { + summary.add(set.recover_delete_bucket(&bucket, recovery_context.clone()).await?); + } + } + } + if object.is_some() && summary.pending > 0 { + return Err(StorageError::ErasureReadQuorum); + } + if object.is_some() && !opts.dry_run && summary.blocked > 0 { + return Err(StorageError::SlowDown); + } + let delete_recovery = rustfs_madmin::heal_commands::DeleteRecoveryStatus { + pending: summary.pending, + blocked: summary.blocked, + failed: summary.failed, + }; + if summary.discovered == 0 && !delete_recovery.has_unresolved() { + return Ok(None); + } + Ok(Some(HealResultItem { + heal_item_type: if object.is_some() { + HealItemType::Object + } else { + HealItemType::Bucket + } + .to_string(), + object: object.or(prefix).unwrap_or_default(), + bucket, + detail: format!( + "delete recovery{}: discovered={}, committed={}, rolled_back={}, reclaimed={}, blocked={}, pending={}, failed={}", + if opts.dry_run { " (dry run)" } else { "" }, + summary.discovered, + summary.committed, + summary.rolled_back, + summary.reclaimed, + summary.blocked, + summary.pending, + summary.failed + ), + delete_recovery: Some(delete_recovery), + ..Default::default() + })) + }) + .await + } + pub async fn heal_bucket_at_incarnation( self: &Arc, bucket: &str, expected: uuid::Uuid, opts: &HealOpts, ) -> Result { - self.run_bucket_heal_at_incarnation(bucket, expected, opts, |store, bucket, opts| async move { + self.run_bucket_heal_at_incarnation(bucket, expected, opts, move |store, bucket, mut opts| async move { + if !opts.recreate && !opts.remove { + let mut missing = false; + for (pool, set) in store.heal_erasure_set_scopes(&opts).await? { + missing |= store.pools[pool].get_disks(set).bucket_needs_reconstruction(&bucket).await?; + } + if missing { + crate::bucket::metadata_sys::validate_bucket_reconstruction_in(&store.ctx, &bucket, expected).await?; + opts.recreate = true; + } + } store.heal_bucket(&bucket, &opts).await }) .await @@ -345,7 +532,7 @@ impl ECStore { NamespaceLockGuard, NamespaceLockGuard, PoolMeta, - Option, + HealFormatRebalance, )> { let metadata_pool = self .pools @@ -401,12 +588,9 @@ impl ECStore { Err(err) => return Err(err), }; - if rebalance_meta - .as_ref() - .is_some_and(|meta| meta.pool_stats.len() != self.pools.len()) - { - return Err(heal_format_fence_lost_error()); - } + // Keep historical status on disk, but never use its obsolete indices + // to gate the current pools. The current pool identity remains fenced. + let rebalance_meta = classify_heal_format_rebalance(rebalance_meta, self.pools.len())?; write_state.ensure_write_safe("heal format fence failed")?; if pool_guard.is_lock_lost() || rebalance_guard.is_lock_lost() { @@ -609,7 +793,8 @@ impl ECStore { first_error.get_or_insert(heal_format_fence_lost_error()); break; } - if let Some(skip) = classify_heal_format_pool(pool_idx, &pool.endpoints.cmd_line, &pool_meta, rebalance_meta.as_ref()) + if let Some(skip) = + classify_heal_format_pool(pool_idx, &pool.endpoints.cmd_line, &pool_meta, rebalance_meta.current()) { if matches!(skip, HealFormatPoolSkip::Completed) { count_completed += 1; @@ -683,6 +868,74 @@ impl ECStore { Ok((r, None)) } + /// Read-only admission for one legacy pre-format recovery handoff. The + /// caller holds the target execution leases and validates their owners. + pub async fn replacement_preformat_retry_is_safe( + &self, + pool_index: usize, + set_index: usize, + targets: &[String], + ) -> Result { + let pool = self + .pools + .get(pool_index) + .ok_or_else(|| invalid_heal_pool_index(pool_index, self.pools.len()))?; + let Some(set) = pool.disk_set.get(set_index) else { + return Err(StorageError::PreconditionFailed); + }; + let (write_state, pool_guard, rebalance_guard, pool_meta, rebalance_meta) = self.acquire_heal_format_fence().await?; + if !matches!(rebalance_meta, HealFormatRebalance::Historical) + || classify_heal_format_pool(pool_index, &pool.endpoints.cmd_line, &pool_meta, None).is_some() + || targets.is_empty() + { + return Ok(false); + } + + let mut slots = BTreeSet::new(); + let mut target_disks = Vec::with_capacity(targets.len()); + for target in targets { + let Some((slot, endpoint)) = set + .set_endpoints + .iter() + .enumerate() + .find(|(_, endpoint)| endpoint.to_string() == *target) + else { + return Ok(false); + }; + if !endpoint.is_local || !slots.insert(slot) { + return Ok(false); + } + // Open the current mounted target; a cached disk handle can still + // refer to the previous disk at the same endpoint. + let target_disk = disk::new_disk( + endpoint, + &disk::DiskOption { + cleanup: false, + health_check: false, + }, + ) + .await?; + if !target_disk.is_blank_replacement_target().await? { + return Ok(false); + } + target_disks.push(target_disk); + } + + let (_, error) = set + .heal_replacement_format_with_fence(true, targets, || pool_guard.is_lock_lost() || rebalance_guard.is_lock_lost()) + .await?; + write_state.ensure_write_safe("pre-format recovery admission fence failed")?; + if pool_guard.is_lock_lost() || rebalance_guard.is_lock_lost() { + return Err(heal_format_fence_lost_error()); + } + for target_disk in target_disks { + if !target_disk.is_blank_replacement_target().await? { + return Ok(false); + } + } + Ok(error.is_none()) + } + #[instrument(skip(self, targets), fields(pool_index, set_index, target_count = targets.len()))] pub async fn heal_replacement_format( &self, @@ -704,7 +957,8 @@ impl ECStore { })?; let (mut write_state, pool_guard, rebalance_guard, pool_meta, rebalance_meta) = self.acquire_heal_format_fence().await?; - if let Some(skip) = classify_heal_format_pool(pool_index, &pool.endpoints.cmd_line, &pool_meta, rebalance_meta.as_ref()) { + if let Some(skip) = classify_heal_format_pool(pool_index, &pool.endpoints.cmd_line, &pool_meta, rebalance_meta.current()) + { return Ok((HealResultItem::default(), Some(heal_format_pool_skip_error(skip)))); } @@ -1360,6 +1614,178 @@ mod tests { .expect("minimal pool should build") } + #[tokio::test] + #[serial_test::serial] + async fn admin_bucket_recovery_recreates_missing_directories_at_current_incarnation() { + let (root, store, shutdown) = multi_pool_heal_store().await; + let bucket = "admin-bucket-recovery"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create a committed bucket generation"); + let incarnation = store + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("persisted generation"); + for disk in 1..4 { + tokio::fs::remove_dir(root.path().join(format!("pool0-disk{disk}/{bucket}"))) + .await + .expect("remove only an empty fixture bucket directory"); + } + store + .heal_bucket_at_incarnation(bucket, incarnation, &HealOpts::default()) + .await + .expect("default administrator recovery should use committed generation authority"); + for disk in 0..4 { + assert!(root.path().join(format!("pool0-disk{disk}/{bucket}")).is_dir()); + } + assert_eq!( + store + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("recovered generation"), + incarnation + ); + shutdown.cancel(); + } + + #[tokio::test] + #[serial_test::serial] + async fn admin_bucket_recovery_rejects_missing_corrupt_uncommitted_and_retired_authority() { + let (root, store, shutdown) = multi_pool_heal_store().await; + for state in ["missing", "corrupt", "uncommitted", "mismatched", "retired"] { + let bucket = format!("recovery-authority-{state}"); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create fixture generation"); + let incarnation = store + .bucket_incarnation_id_from_disk(&bucket) + .await + .expect("fixture identity"); + let (mut metadata, persisted) = metadata_sys::get_config_from_disk_with_presence_in(&store.ctx, &bucket) + .await + .expect("committed metadata"); + assert!(persisted && metadata.bucket_creation_committed); + let path = metadata.save_file_path(); + match state { + "missing" => delete_config(store.clone(), &path).await.expect("remove metadata authority"), + "corrupt" => save_config(store.clone(), &path, b"corrupt bucket metadata".to_vec()) + .await + .expect("corrupt authority"), + "uncommitted" | "mismatched" => { + if state == "uncommitted" { + metadata.bucket_creation_committed = false; + } else { + metadata.bucket_incarnation_id = Uuid::new_v4(); + } + metadata + .save_with_store_committed(store.clone()) + .await + .expect("inject inconsistent metadata authority"); + } + "retired" => { + // A returning disk may retain a directory after retirement was committed. + crate::bucket::retirement::commit_retirement(store.clone(), &bucket, incarnation, &ObjectOptions::default()) + .await + .expect("persist retirement evidence"); + } + _ => unreachable!("fixed fixture states"), + } + for disk in 1..4 { + tokio::fs::remove_dir(root.path().join(format!("pool0-disk{disk}/{bucket}"))) + .await + .expect("lose fixture directories"); + } + assert!( + store + .heal_bucket_at_incarnation(&bucket, incarnation, &HealOpts::default()) + .await + .is_err(), + "must reject {state}" + ); + for disk in 1..4 { + assert!( + !root.path().join(format!("pool0-disk{disk}/{bucket}")).exists(), + "{state} cannot authorize reconstruction" + ); + } + } + shutdown.cancel(); + } + + #[tokio::test] + #[serial_test::serial] + async fn admin_bucket_recovery_rejects_offline_disks_and_fully_missing_sets() { + for offline in [false, true] { + let (root, store, shutdown) = multi_pool_heal_store().await; + let bucket = "recovery-unproven-set"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("fixture bucket"); + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("fixture identity"); + let first_missing = usize::from(offline); + for disk in first_missing..4 { + tokio::fs::remove_dir(root.path().join(format!("pool0-disk{disk}/{bucket}"))) + .await + .expect("remove fixture directories"); + } + if offline { + let disk = store.pools[0].disk_set[0].disks.read().await[0] + .clone() + .expect("online fixture disk"); + disk.close().await.expect("stop disk monitoring"); + disk.force_offline_for_test(); + } + assert!( + store + .heal_bucket_at_incarnation(bucket, incarnation, &HealOpts::default()) + .await + .is_err() + ); + for disk in first_missing..4 { + assert!(!root.path().join(format!("pool0-disk{disk}/{bucket}")).exists()); + } + shutdown.cancel(); + } + } + + #[tokio::test] + #[serial_test::serial] + async fn admin_bucket_recovery_dry_run_preserves_missing_directories() { + let (root, store, shutdown) = multi_pool_heal_store().await; + let bucket = "recovery-dry-run"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("fixture bucket"); + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("fixture identity"); + for disk in 1..4 { + tokio::fs::remove_dir(root.path().join(format!("pool0-disk{disk}/{bucket}"))) + .await + .expect("remove fixture directories"); + } + store + .heal_bucket_at_incarnation( + bucket, + incarnation, + &HealOpts { + dry_run: true, + ..Default::default() + }, + ) + .await + .expect("dry run can inspect eligible recovery"); + for disk in 1..4 { + assert!( + !root.path().join(format!("pool0-disk{disk}/{bucket}")).exists(), + "dry run must not reconstruct directories" + ); + } + shutdown.cancel(); + } + #[tokio::test] #[serial_test::serial] async fn bucket_incarnation_heal_preserves_successor_shards_and_allows_fresh_repair() { @@ -1856,9 +2282,13 @@ mod tests { } async fn multi_pool_heal_store() -> (tempfile::TempDir, Arc, CancellationToken) { + heal_store_with_pool_count(2).await + } + + async fn heal_store_with_pool_count(pool_count: usize) -> (tempfile::TempDir, Arc, CancellationToken) { let temp_dir = tempfile::tempdir().expect("multi-pool heal test directory should be created"); let mut pool_endpoints = Vec::new(); - for pool_index in 0..2 { + for pool_index in 0..pool_count { let mut endpoints = Vec::new(); for disk_index in 0..4 { let disk_path = temp_dir.path().join(format!("pool{pool_index}-disk{disk_index}")); @@ -2175,6 +2605,224 @@ mod tests { ); } + fn completed_two_pool_rebalance() -> RebalanceMeta { + RebalanceMeta { + id: Uuid::new_v4().to_string(), + pool_stats: vec![ + RebalanceStats { + participating: true, + info: RebalanceInfo { + status: RebalStatus::Completed, + ..Default::default() + }, + ..Default::default() + }, + RebalanceStats::default(), + ], + ..Default::default() + } + } + + #[test] + fn historical_rebalance_topology_requires_every_pool_to_be_terminal() { + use crate::services::rebalance::{RebalanceStopPropagationRecord, encode_rebalance_stop_propagation_record}; + + assert!(matches!( + classify_heal_format_rebalance(Some(completed_two_pool_rebalance()), 1), + Ok(HealFormatRebalance::Historical) + )); + // Expansion has the same historical-index problem as removal. + assert!(matches!( + classify_heal_format_rebalance(Some(completed_two_pool_rebalance()), 3), + Ok(HealFormatRebalance::Historical) + )); + assert!(matches!(classify_heal_format_rebalance(None, 1), Ok(HealFormatRebalance::Missing))); + for status in [RebalStatus::Completed, RebalStatus::Stopped, RebalStatus::Failed] { + let mut meta = completed_two_pool_rebalance(); + meta.pool_stats[0].info.status = status; + assert!(meta.has_terminal_topology(), "terminal status {status:?}"); + } + + let mut invalid = Vec::new(); + for index in 0..2 { + let mut active = completed_two_pool_rebalance(); + active.stopped_at = Some(OffsetDateTime::UNIX_EPOCH); + active.pool_stats[index].info.status = RebalStatus::Started; + invalid.push(active); + let mut stopping = completed_two_pool_rebalance(); + stopping.pool_stats[index].info.stopping = true; + invalid.push(stopping); + } + let mut unknown_participant = completed_two_pool_rebalance(); + unknown_participant.pool_stats[0].info.status = RebalStatus::None; + invalid.push(unknown_participant); + let mut no_id = completed_two_pool_rebalance(); + no_id.id.clear(); + invalid.push(no_id); + invalid.push(RebalanceMeta::default()); + for record in [ + RebalanceStopPropagationRecord { + stop_attempt_at: Some(OffsetDateTime::UNIX_EPOCH), + ..Default::default() + }, + RebalanceStopPropagationRecord { + stop_failures: vec!["peer".to_string()], + ..Default::default() + }, + RebalanceStopPropagationRecord { + terminal_reload_attempt_at: Some(OffsetDateTime::UNIX_EPOCH), + terminal_reload_failures: vec!["peer".to_string()], + ..Default::default() + }, + ] { + let mut meta = completed_two_pool_rebalance(); + meta.pool_stats[1].info.last_error = Some(encode_rebalance_stop_propagation_record(&record)); + invalid.push(meta); + } + let mut malformed = completed_two_pool_rebalance(); + malformed.pool_stats[0].info.last_error = Some("rebalance stop propagation incomplete: invalid".to_string()); + invalid.push(malformed); + for meta in invalid { + let error = classify_heal_format_rebalance(Some(meta), 1).expect_err("uncertain history must block repair"); + assert!(error.to_string().contains("not confirmed terminal")); + assert!(!error.to_string().contains("request rate")); + } + let mut acknowledged = completed_two_pool_rebalance(); + acknowledged.pool_stats[0].info.last_error = + Some(encode_rebalance_stop_propagation_record(&RebalanceStopPropagationRecord { + stop_attempt_at: Some(OffsetDateTime::UNIX_EPOCH), + terminal_reload_attempt_at: Some(OffsetDateTime::UNIX_EPOCH), + ..Default::default() + })); + assert!(acknowledged.has_terminal_topology()); + } + + #[tokio::test] + #[serial_test::serial] + async fn replacement_format_after_historical_pool_removal() { + let (temp_dir, store, shutdown) = heal_store_with_pool_count(1).await; + let mut meta = completed_two_pool_rebalance(); + meta.save(store.pools[0].clone()) + .await + .expect("persist historical two-pool run"); + let (before, _) = read_config_no_lock_preserve_empty_with_metadata(store.pools[0].clone(), REBAL_META_NAME) + .await + .expect("read historical run before repair"); + let target = remove_heal_test_format(&temp_dir, &store, 0, 3).await; + + // Even a worker in a removed pool must fence every current format write. + meta.pool_stats[1].participating = true; + meta.pool_stats[1].info.status = RebalStatus::Started; + meta.save(store.pools[0].clone()) + .await + .expect("persist active historical pool"); + let err = store + .heal_replacement_format(false, 0, 0, std::slice::from_ref(&target)) + .await + .expect_err("removed active source must block format repair"); + assert!(err.to_string().contains("2 pools but the deployment has 1")); + assert_heal_test_format_missing(&temp_dir, 0, 3).await; + meta.pool_stats[1] = RebalanceStats::default(); + meta.save(store.pools[0].clone()) + .await + .expect("persist terminal history again"); + + let (result, error) = store + .heal_replacement_format(false, 0, 0, std::slice::from_ref(&target)) + .await + .expect("terminal historical topology must allow target format repair"); + assert!(error.is_none(), "target repair: {error:?}"); + assert!(result.after.drives.iter().all(|drive| !drive.uuid.is_empty())); + let disk = store.pools[0].disk_set[0].disks.read().await[3] + .clone() + .expect("replacement disk"); + let repaired = load_format_erasure(&disk, true).await.expect("read repaired target format"); + assert!(!repaired.erasure.this.is_nil()); + + remove_heal_test_format(&temp_dir, &store, 0, 3).await; + let (_, error) = store + .handle_heal_format(false) + .await + .expect("full format heal uses the same historical fence"); + assert!(error.is_none(), "full format repair: {error:?}"); + assert!(heal_test_format_path(&temp_dir, 0, 3).exists()); + let (after, _) = read_config_no_lock_preserve_empty_with_metadata(store.pools[0].clone(), REBAL_META_NAME) + .await + .expect("historical status remains readable"); + assert_eq!(before, after, "format repair must not rewrite historical rebalance status"); + shutdown.cancel(); + } + + #[tokio::test] + #[serial_test::serial] + async fn preformat_retry_requires_historical_topology_and_a_blank_target() { + let (temp_dir, store, shutdown) = heal_store_with_pool_count(1).await; + completed_two_pool_rebalance() + .save(store.pools[0].clone()) + .await + .expect("persist historical two-pool run"); + let target = remove_heal_test_format(&temp_dir, &store, 0, 3).await; + assert!( + !store + .replacement_preformat_retry_is_safe(0, 0, std::slice::from_ref(&target)) + .await + .expect("inspect target with old metadata"), + "missing format alone must not admit a disk containing old metadata" + ); + let target_path = temp_dir.path().join("pool0-disk3"); + for entry in std::fs::read_dir(&target_path).expect("list target fixture") { + let entry = entry.expect("target fixture entry"); + if entry.file_type().expect("fixture entry type").is_dir() { + std::fs::remove_dir_all(entry.path()).expect("clear replacement fixture directory"); + } else { + std::fs::remove_file(entry.path()).expect("clear replacement fixture file"); + } + } + let target_disk = store.pools[0].disk_set[0].disks.read().await[3] + .clone() + .expect("replacement disk"); + if !cfg!(target_os = "linux") { + assert!(target_disk.acquire_replacement_execution_lease().await.is_err()); + assert!( + !store + .replacement_preformat_retry_is_safe(0, 0, &[target]) + .await + .expect("platforms without pinned mount identity must reject compatibility recovery") + ); + shutdown.cancel(); + return; + } + let _lease = target_disk + .acquire_replacement_execution_lease() + .await + .expect("own replacement executor"); + assert!( + store + .replacement_preformat_retry_is_safe(0, 0, std::slice::from_ref(&target)) + .await + .expect("inspect blank target") + ); + assert_heal_test_format_missing(&temp_dir, 0, 3).await; + std::fs::create_dir(target_path.join("user-bucket")).expect("simulate residual user bucket"); + assert!( + !store + .replacement_preformat_retry_is_safe(0, 0, std::slice::from_ref(&target)) + .await + .expect("inspect nonblank target") + ); + std::fs::remove_dir(target_path.join("user-bucket")).expect("remove fixture bucket"); + delete_config(store.pools[0].clone(), REBAL_META_NAME) + .await + .expect("remove historical run"); + assert!( + !store + .replacement_preformat_retry_is_safe(0, 0, &[target]) + .await + .expect("ordinary replacement has no compatibility authority") + ); + shutdown.cancel(); + } + #[tokio::test] #[serial_test::serial] async fn replacement_format_heal_coexists_with_capacity_read_fence() { @@ -3467,7 +4115,6 @@ mod tests { let mut recoverable_temp_dirs = Vec::new(); let mut recoverable_endpoints = Vec::new(); let mut recoverable_disks = Vec::new(); - let mut unformatted_disk = None; for disk_index in 0..3 { let temp_dir = tempfile::tempdir().expect("temporary disk root should be created"); let mut endpoint = Endpoint::try_from(temp_dir.path().to_str().expect("temporary path should be UTF-8")) @@ -3484,15 +4131,11 @@ mod tests { ) .await .expect("temporary disk should open"); - if disk_index < 2 { - let mut disk_format = recoverable_format.clone(); - disk_format.erasure.this = recoverable_format.erasure.sets[0][disk_index]; - save_format_file(&Some(disk.clone()), &Some(disk_format)) - .await - .expect("recoverable format should be written"); - } else { - unformatted_disk = Some(disk.clone()); - } + let mut disk_format = recoverable_format.clone(); + disk_format.erasure.this = recoverable_format.erasure.sets[0][disk_index]; + save_format_file(&Some(disk.clone()), &Some(disk_format)) + .await + .expect("recoverable format should be written before pool metadata"); recoverable_temp_dirs.push(temp_dir); recoverable_endpoints.push(endpoint); recoverable_disks.push(Some(disk)); @@ -3541,6 +4184,36 @@ mod tests { .expect("pool metadata should be persisted before format heal"); *store.pool_meta_save_gate.lock().await = PoolMetaWriteState::default(); + let replacement_endpoint = store.pools[1].endpoints.endpoints.as_ref()[2].clone(); + tokio::fs::remove_file( + recoverable_temp_dirs[2] + .path() + .join(crate::disk::RUSTFS_META_BUCKET) + .join(FORMAT_CONFIG_FILE), + ) + .await + .expect("the target format must be removed only after durable pool metadata exists"); + let replacement_disk = new_disk( + &replacement_endpoint, + &DiskOption { + cleanup: false, + health_check: false, + }, + ) + .await + .expect("the unformatted replacement disk must reopen"); + assert!(matches!( + load_format_erasure(&replacement_disk, true).await, + Err(crate::disk::error::DiskError::UnformattedDisk) + )); + let replacement_set = &store.pools[1].disk_set[0]; + let replacement_slot = replacement_set + .set_endpoints + .iter() + .position(|endpoint| endpoint == &replacement_endpoint) + .expect("the replacement must remain in its original set"); + replacement_set.disks.write().await[replacement_slot] = Some(replacement_disk.clone()); + let (result, err) = store .handle_heal_format(false) .await @@ -3550,7 +4223,7 @@ mod tests { "foreign format majority must not be downgraded to a successful heal: {err:?}" ); assert_eq!(result.disk_count, 3, "the recoverable pool should still be inspected"); - let healed = load_format_erasure(&unformatted_disk.expect("the unformatted disk handle should be retained"), true) + let healed = load_format_erasure(&replacement_disk, true) .await .expect("the later pool should be healed despite the first pool error"); assert_eq!(healed.erasure.this, recoverable_format.erasure.sets[0][2]); diff --git a/crates/ecstore/src/store/init.rs b/crates/ecstore/src/store/init.rs index 4f72b8787..f130ec1d1 100644 --- a/crates/ecstore/src/store/init.rs +++ b/crates/ecstore/src/store/init.rs @@ -108,6 +108,10 @@ fn should_retry_format_load(err: &Error) -> bool { ) } +fn format_load_retry_exhausted(err: &Error, retries: u32) -> bool { + retries >= 10 && !matches!(err, Error::ErasureReadQuorum | Error::FirstDiskWait | Error::NotFirstDisk) +} + fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_resume_required: bool) -> bool { rebalance_resume_required && !decommission_running } @@ -484,7 +488,7 @@ impl ECStore { check_disk_fatal_errs(&errs)?; let loaded_format = { - let mut times = 0; + let mut times = 0_u32; let mut interval = 1; loop { match init_format::connect_load_init_formats_with_instance_ctx( @@ -500,13 +504,13 @@ impl ECStore { Ok(fm) => break Ok(fm), Err(e) if !should_retry_format_load(&e) => break Err(e), // Wrap the final error if we are giving up - Err(e) if times >= 10 => { + Err(e) if format_load_retry_exhausted(&e, times) => { break Err(Error::other(format!("store init failed to load formats after {times} retries: {e}"))); } // Retrying so just drop the error Err(_) => {} } - times += 1; + times = times.saturating_add(1); if interval < 16 { interval *= 2; } @@ -519,6 +523,12 @@ impl ECStore { "Retrying storage format load" ); select! { + _ = ctx.cancelled() => { + return Err(Error::other(std::io::Error::new( + std::io::ErrorKind::Interrupted, + "storage format initialization cancelled", + ))); + } _ = tokio::signal::ctrl_c() => { info!( event = EVENT_STORE_FORMAT_RETRY, @@ -821,6 +831,8 @@ impl ECStore { } }); + super::delete_recovery_runtime::start_delete_recovery(self.clone(), rx.clone()); + runtime_sources::init_bucket_monitor_for_current_endpoints(); crate::bucket::bucket_target_sys::BucketTargetSys::get().start_heartbeat(); @@ -880,6 +892,7 @@ mod tests { run_local_decommission_watchdog, save_validated_pool_meta_for_startup, should_auto_start_rebalance_after_init, should_defer_rebalance_auto_start, should_retry_format_load, wait_for_local_decommission_resume_delay, }; + use crate::bucket::metadata::BUCKET_LIFECYCLE_CONFIG; use crate::core::pools::PoolMetaBootstrapAuthority; #[cfg(feature = "test-util")] use crate::disk::DiskAPI; @@ -941,7 +954,7 @@ mod tests { }, validate_durable_ilm_record, }, - bucket::metadata::{BUCKET_LIFECYCLE_CONFIG, BUCKET_VERSIONING_CONFIG}, + bucket::metadata::BUCKET_VERSIONING_CONFIG, config::com, core::pools::DecomBucketInfo, data_movement::SourceCleanupDeleteBarrier, @@ -1097,6 +1110,9 @@ mod tests { revision: AtomicUsize, pool_meta_write_attempts: AtomicUsize, fail_pool_meta_write_at: AtomicUsize, + identity_repair_failures: AtomicUsize, + identity_write_attempts: AtomicUsize, + identity_unconverged: AtomicBool, pool_meta_written_versions: Mutex>, objects: Mutex, String)>>, } @@ -1117,6 +1133,9 @@ mod tests { revision: AtomicUsize::new(0), pool_meta_write_attempts: AtomicUsize::new(0), fail_pool_meta_write_at: AtomicUsize::new(0), + identity_repair_failures: AtomicUsize::new(0), + identity_write_attempts: AtomicUsize::new(0), + identity_unconverged: AtomicBool::new(false), pool_meta_written_versions: Mutex::new(Vec::new()), objects: Mutex::new(objects), } @@ -1133,6 +1152,9 @@ mod tests { revision: AtomicUsize::new(0), pool_meta_write_attempts: AtomicUsize::new(0), fail_pool_meta_write_at: AtomicUsize::new(0), + identity_repair_failures: AtomicUsize::new(0), + identity_write_attempts: AtomicUsize::new(0), + identity_unconverged: AtomicBool::new(false), pool_meta_written_versions: Mutex::new(Vec::new()), objects: Mutex::new(HashMap::new()), } @@ -1247,6 +1269,17 @@ mod tests { .lock() .unwrap_or_else(std::sync::PoisonError::into_inner) .insert(object.to_string(), (payload, etag.clone())); + if object == POOL_META_IDENTITY_NAME { + self.identity_write_attempts.fetch_add(1, Ordering::SeqCst); + let partial = self + .identity_repair_failures + .try_update(Ordering::SeqCst, Ordering::SeqCst, |remaining| remaining.checked_sub(1)) + .is_ok(); + self.identity_unconverged.store(partial, Ordering::SeqCst); + if partial { + return Err(Error::other(crate::set_disk::SystemMetadataRepairRequired)); + } + } Ok(self.object_info(bucket, object, size, etag)) } } @@ -1753,6 +1786,220 @@ mod tests { assert!(startup_object(&pool0, POOL_META_NAME).is_none()); } + #[tokio::test] + async fn test_four_pool_expansion_repairs_missing_identity_disk_without_data_change() { + use crate::disk::{DiskAPI, RUSTFS_META_BUCKET, ReadOptions}; + use crate::set_disk::hermetic_set_disks_isolated; + + let mut dirs = Vec::new(); + let mut disks = Vec::new(); + let mut pools = Vec::new(); + for _ in 0..4 { + let (pool_dirs, pool_disks, pool) = hermetic_set_disks_isolated(4).await; + dirs.push(pool_dirs); + disks.push(pool_disks); + pools.push(pool); + } + let deployment_id = Uuid::new_v4(); + let mut founder = PoolMetaWriteState::for_startup(deployment_id, true); + establish_pool_meta_bootstrap_identity_if_proven(pools[..3].to_vec(), &mut founder, true) + .await + .expect("seed three healthy pool identities"); + let (_, replicas) = load_pool_meta_for_startup(pools[..3].to_vec(), &mut founder) + .await + .expect("load the original deployment"); + persist_pool_meta_for_startup_if_safe( + &init_test_pool_meta_with_pools(3), + pools[..3].to_vec(), + replicas, + &mut founder, + true, + true, + ) + .await + .expect("commit three healthy pools"); + let identity = crate::config::com::read_config_no_lock(pools[0].clone(), POOL_META_IDENTITY_NAME) + .await + .expect("read the initialized deployment identity"); + let bucket = "identity-repair-data"; + for disk in &disks[0] { + disk.make_volume(bucket).await.expect("create the data bucket"); + } + let original = vec![0x73; 256 * 1024]; + pools[0] + .put_object( + bucket, + "retained", + &mut PutObjReader::from_vec(original.clone()), + &ObjectOptions { + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("seed user bytes before expansion"); + + // Model the startup snapshot containing a missing disk handle, while + // the other three disks can commit and serve the new pool identity. + let missing = pools[3].disks.write().await[0].take().expect("healthy fixture disk"); + let startup = tokio::spawn({ + let pools = pools.clone(); + async move { + let mut owner = PoolMetaWriteState::for_startup_with_pool_bootstrap_authorities( + deployment_id, + vec![PoolMetaBootstrapAuthority::None; 4], + true, + ); + let (_, replicas) = load_pool_meta_for_startup(pools.clone(), &mut owner) + .await + .expect("load the three-pool predecessor"); + let committed = persist_pool_meta_for_startup_if_safe( + &init_test_pool_meta_with_pools(4), + pools, + replicas, + &mut owner, + true, + true, + ) + .await + .expect("startup must recover the partial identity and commit expansion"); + owner + .ensure_write_safe("expanded startup") + .expect("repaired startup remains writable"); + committed + } + }); + tokio::time::timeout(std::time::Duration::from_secs(30), async { + loop { + let mut committed = 0; + for disk in &disks[3][1..] { + if disk + .read_version("", RUSTFS_META_BUCKET, POOL_META_IDENTITY_NAME, "", &ReadOptions::default()) + .await + .is_ok() + { + committed += 1; + } + } + if committed == 3 { + break; + } + assert!(!startup.is_finished(), "startup must reach the partial-commit barrier"); + tokio::task::yield_now().await; + } + }) + .await + .expect("three disks must physically commit the identity"); + assert!( + disks[3][0] + .read_version("", RUSTFS_META_BUCKET, POOL_META_IDENTITY_NAME, "", &ReadOptions::default()) + .await + .is_err() + ); + assert_eq!( + crate::config::com::read_config_no_lock(pools[3].clone(), POOL_META_IDENTITY_NAME) + .await + .expect("quorum can already read the unhealed identity"), + identity + ); + assert!(!startup.is_finished(), "quorum readback must not acknowledge unrepaired identity"); + pools[3].disks.write().await[0] = Some(missing); + let committed = tokio::time::timeout(std::time::Duration::from_secs(30), startup) + .await + .expect("startup must finish after reconnect") + .expect("startup task must join"); + assert_eq!(committed.pools.len(), 4); + let metadata = crate::config::com::read_config_no_lock(pools[0].clone(), POOL_META_NAME) + .await + .expect("read committed expanded topology"); + for (pool, pool_disks) in pools.iter().zip(&disks) { + for (object, payload) in [(POOL_META_IDENTITY_NAME, &identity), (POOL_META_NAME, &metadata)] { + assert_eq!( + crate::config::com::read_config_no_lock(pool.clone(), object) + .await + .expect("read every pool"), + *payload + ); + let reference = pool_disks[0] + .read_version("", RUSTFS_META_BUCKET, object, "", &ReadOptions::default()) + .await + .expect("metadata exists on the first disk"); + for disk in &pool_disks[1..] { + let actual = disk + .read_version("", RUSTFS_META_BUCKET, object, "", &ReadOptions::default()) + .await + .expect("metadata must exist on every physical disk"); + assert_eq!(actual.metadata.get("etag"), reference.metadata.get("etag")); + assert_eq!(actual.mod_time, reference.mod_time); + } + } + } + let mut reader = pools[0] + .get_object_reader(bucket, "retained", None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("user data remains readable after expansion"); + let mut actual = Vec::new(); + reader.stream.read_to_end(&mut actual).await.expect("read all retained bytes"); + assert_eq!(actual, original); + } + + #[tokio::test(start_paused = true)] + async fn test_four_pool_expansion_repairs_identity_before_completing_startup() { + let deployment_id = Uuid::new_v4(); + let mut pools = (0..3) + .map(|_| Arc::new(StartupPoolMetaStorage::new(Vec::new()))) + .collect::>(); + let mut founder = PoolMetaWriteState::for_startup(deployment_id, true); + establish_pool_meta_bootstrap_identity_if_proven(pools.clone(), &mut founder, true) + .await + .expect("seed the three-pool deployment identity"); + let (_, replicas) = load_pool_meta_for_startup(pools.clone(), &mut founder) + .await + .expect("load the fresh deployment"); + persist_pool_meta_for_startup_if_safe( + &init_test_pool_meta_with_pools(3), + pools.clone(), + replicas, + &mut founder, + true, + true, + ) + .await + .expect("commit the three-pool deployment"); + let identity = startup_object(&pools[0], POOL_META_IDENTITY_NAME).expect("initialized identity"); + let new_pool = Arc::new(StartupPoolMetaStorage::new(Vec::new())); + new_pool.identity_repair_failures.store(1, Ordering::SeqCst); + pools.push(new_pool.clone()); + let mut owner = PoolMetaWriteState::for_startup_with_pool_bootstrap_authorities( + deployment_id, + vec![PoolMetaBootstrapAuthority::None; 4], + true, + ); + let (_, replicas) = load_pool_meta_for_startup(pools.clone(), &mut owner) + .await + .expect("read the committed predecessor during expansion"); + let committed = persist_pool_meta_for_startup_if_safe( + &init_test_pool_meta_with_pools(4), + pools.clone(), + replicas, + &mut owner, + true, + true, + ) + .await + .expect("the owning startup must repair its partial identity write before publishing expansion"); + assert_eq!(committed.pools.len(), 4); + assert_eq!(new_pool.identity_write_attempts.load(Ordering::SeqCst), 2); + assert!(!new_pool.identity_unconverged.load(Ordering::SeqCst)); + owner + .ensure_write_safe("expanded startup") + .expect("verified repair must not latch the write gate"); + for pool in &pools { + assert_eq!(startup_object(pool, POOL_META_IDENTITY_NAME).as_ref(), Some(&identity)); + assert_eq!(startup_object(pool, POOL_META_NAME), startup_object(&pools[0], POOL_META_NAME)); + } + } + #[tokio::test] async fn test_fresh_pool_joining_an_initialized_deployment_never_reopens_bootstrap() { let deployment_id = Uuid::new_v4(); @@ -2216,6 +2463,23 @@ mod tests { assert_eq!(attempts.load(Ordering::SeqCst), 2); } + #[test] + fn format_peer_wait_does_not_exhaust_the_retry_budget() { + for retries in [0, 9, 10, u32::MAX] { + for error in [ + StorageError::ErasureReadQuorum, + StorageError::FirstDiskWait, + StorageError::NotFirstDisk, + ] { + assert!( + !super::format_load_retry_exhausted(&error, retries), + "peer availability must remain retryable" + ); + } + assert_eq!(super::format_load_retry_exhausted(&StorageError::Timeout, retries), retries >= 10); + } + } + #[test] fn test_should_retry_format_load_rejects_permanent_corruption() { assert!(!should_retry_format_load(&StorageError::CorruptedFormat)); @@ -2702,6 +2966,106 @@ mod tests { (instance_ctx, store, shutdown) } + /// CopyObject destinations and CreateMultipartUpload create keys, so they + /// must reject exactly the keys PutObject rejects, with the same error. + /// The copy path reaches the pools' put_object directly and used to admit + /// keys PutObject refused (e.g. over 1024 bytes, or Windows-unaddressable). + #[tokio::test] + #[serial_test::serial(storage_class_env)] + async fn copy_destination_and_multipart_create_share_put_object_admission() { + let temp_dir = tempfile::tempdir().expect("create key-admission store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "key-admission", &[4])).await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + + let bucket = format!("key-admission-{}", Uuid::new_v4()); + let source_object = "source.bin"; + let payload = b"key admission source".to_vec(); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("key-admission bucket should be created"); + store + .put_object( + &bucket, + source_object, + &mut PutObjReader::from_vec(payload.clone()), + &ObjectOptions::default(), + ) + .await + .expect("source object should be written"); + let source_info = store + .get_object_info(&bucket, source_object, &ObjectOptions::default()) + .await + .expect("source metadata should be readable"); + + // Every segment fits NAME_MAX, but the key exceeds S3's 1024 bytes. + let too_long = vec!["s".repeat(210); 5].join("/"); + let mut rejected = vec![too_long]; + if cfg!(target_os = "windows") { + rejected.extend(["a:b", "a\\b", "NUL.txt", "tab\tname"].map(str::to_owned)); + } + for dst_object in &rejected { + let put_err = store + .put_object( + &bucket, + dst_object, + &mut PutObjReader::from_vec(payload.clone()), + &ObjectOptions::default(), + ) + .await + .expect_err("PutObject must reject the key"); + let mut copy_source = source_info.clone(); + copy_source.put_object_reader = Some(PutObjReader::from_vec(payload.clone())); + let copy_err = store + .copy_object( + &bucket, + source_object, + &bucket, + dst_object, + &mut copy_source, + &ObjectOptions::default(), + &ObjectOptions::default(), + ) + .await + .expect_err("CopyObject must reject the destination key"); + assert_eq!(format!("{copy_err:?}"), format!("{put_err:?}"), "{dst_object:?}"); + let multipart_err = store + .new_multipart_upload(&bucket, dst_object, &ObjectOptions::default()) + .await + .expect_err("CreateMultipartUpload must reject the key"); + assert_eq!(format!("{multipart_err:?}"), format!("{put_err:?}"), "{dst_object:?}"); + } + + // A segment over NAME_MAX used to be admitted by CreateMultipartUpload + // and only fail at completion with ENAMETOOLONG (rustfs#5785). + let oversized_segment = "m".repeat(256); + assert!(matches!( + store + .new_multipart_upload(&bucket, &oversized_segment, &ObjectOptions::default()) + .await, + Err(StorageError::ObjectNameInvalid(_, _)) + )); + + // In-place copies rewrite an existing key and keep the copy-only check. + let mut in_place = source_info.clone(); + in_place.metadata_only = true; + store + .copy_object( + &bucket, + source_object, + &bucket, + source_object, + &mut in_place, + &ObjectOptions::default(), + &ObjectOptions::default(), + ) + .await + .expect("in-place copy of an existing key should succeed"); + + shutdown.cancel(); + } + #[cfg(feature = "test-util")] #[tokio::test] #[serial_test::serial(storage_class_env)] @@ -4151,6 +4515,178 @@ mod tests { OfflineTestDisks { disks } } + #[cfg(feature = "test-util")] + async fn durable_quota_writes_with_one_offline_disk(sharded: bool) { + use crate::bucket::quota::BucketQuota; + use crate::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause}; + + let temp_dir = tempfile::tempdir().expect("create degraded quota store dir"); + let (ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "degraded-quota", &[4])).await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let bucket = format!("degraded-quota-{}", Uuid::new_v4()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create degraded quota bucket"); + let quota = if sharded { + BucketQuota::new_sharded(16_384) + } else { + BucketQuota::new(Some(16_384)) + }; + crate::bucket::metadata_sys::update_in( + &ctx, + &bucket, + crate::bucket::metadata::BUCKET_QUOTA_CONFIG_FILE, + serde_json::to_vec("a).expect("encode durable quota"), + ) + .await + .expect("configure durable quota before losing a disk"); + + let set = &store.pools[0].disk_set[0]; + assert_eq!(set.disks.read().await.len(), 4); + let offline_disk = force_set_disk_range_offline_for_test(set, 0..1).await; + let _without_proof = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test(); + assert!(crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_none()); + + let payload = vec![0x51; 4096]; + let mut reader = PutObjReader::from_vec(payload.clone()); + store + .put_object(&bucket, "put", &mut reader, &ObjectOptions::default()) + .await + .expect("three healthy disks must admit a quota PUT without fleet proof"); + + let upload = store + .new_multipart_upload(&bucket, "multipart", &ObjectOptions::default()) + .await + .expect("create multipart upload with one disk offline"); + let mut reader = PutObjReader::from_vec(payload.clone()); + let part = store + .put_object_part(&bucket, "multipart", &upload.upload_id, 1, &mut reader, &ObjectOptions::default()) + .await + .expect("stage multipart part with one disk offline"); + Arc::clone(&store) + .complete_multipart_upload( + &bucket, + "multipart", + &upload.upload_id, + vec![crate::storage_api_contracts::multipart::CompletePart { + part_num: part.part_num, + etag: part.etag, + ..Default::default() + }], + &ObjectOptions::default(), + ) + .await + .expect("three healthy disks must complete a quota multipart upload without fleet proof"); + + let source = store + .get_object_reader(&bucket, "put", None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("read degraded copy source"); + let mut copy_info = source.object_info.clone(); + let size = copy_info.get_actual_size().expect("copy source logical size"); + copy_info.put_object_reader = Some(PutObjReader::new( + rustfs_rio::HashReader::from_stream(source.stream, size, size, None, None, false) + .expect("construct copy source reader"), + )); + store + .copy_object( + &bucket, + "put", + &bucket, + "copy", + &mut copy_info, + &ObjectOptions::default(), + &ObjectOptions::default(), + ) + .await + .expect("three healthy disks must copy into a quota bucket without fleet proof"); + + // Hold the last available credit before publication so the competing + // writer must account for an outstanding reservation, not just objects. + let barrier = PutObjectCommitBarrier::install(&bucket, "reserved", PutObjectCommitPause::BeforeQuotaRename); + let first_store = Arc::clone(&store); + let first_bucket = bucket.clone(); + let first = tokio::spawn(async move { + let mut reader = PutObjReader::from_vec(vec![0x52; 4096]); + first_store + .put_object(&first_bucket, "reserved", &mut reader, &ObjectOptions::default()) + .await + }); + tokio::time::timeout(Duration::from_secs(30), barrier.wait_until_paused()) + .await + .expect("the last within-limit writer must reach its commit barrier"); + let mut reader = PutObjReader::from_vec(vec![0x53; 4096]); + let denied = store + .put_object(&bucket, "over-limit", &mut reader, &ObjectOptions::default()) + .await + .expect_err("an outstanding reservation must prevent concurrent quota oversubscription"); + assert!(matches!(denied, StorageError::QuotaExceeded { limit: 16_384, .. })); + barrier.release(); + first + .await + .expect("reserved PUT task should not panic") + .expect("the reserved writer must commit while one disk is offline"); + + for object in ["put", "multipart", "copy", "reserved"] { + let mut reader = store + .get_object_reader(&bucket, object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("degraded quota writes must be readable"); + let mut body = Vec::new(); + reader + .stream + .read_to_end(&mut body) + .await + .expect("read committed quota object"); + let expected = if object == "reserved" { + vec![0x52; 4096] + } else { + payload.clone() + }; + assert_eq!(body, expected, "committed body for {object}"); + } + store + .get_object_info(&bucket, "over-limit", &ObjectOptions::default()) + .await + .expect_err("the over-limit writer must not publish an object"); + + let second_offline_disk = force_set_disk_range_offline_for_test(set, 1..2).await; + let mut reader = PutObjReader::from_vec(vec![0x54; 4096]); + let denied = store + .put_object(&bucket, "put", &mut reader, &ObjectOptions::default()) + .await + .expect_err("a zero-growth overwrite must still fail when the actual write quorum is lost"); + assert!( + matches!( + denied, + StorageError::ErasureWriteQuorum + | StorageError::ErasureReadQuorum + | StorageError::InsufficientWriteQuorum(_, _) + | StorageError::InsufficientReadQuorum(_, _) + ), + "loss of actual disk quorum must return a storage quorum error, got {denied:?}" + ); + shutdown.cancel(); + drop(second_offline_disk); + drop(offline_disk); + } + + #[cfg(feature = "test-util")] + #[test] + #[serial_test::serial(storage_class_env)] + fn durable_quota_v1_writes_with_one_offline_disk_and_no_fleet_proof() { + run_large_stack_async_test("degraded-quota-v1", || durable_quota_writes_with_one_offline_disk(false)); + } + + #[cfg(feature = "test-util")] + #[test] + #[serial_test::serial(storage_class_env)] + fn durable_quota_v2_writes_with_one_offline_disk_and_no_fleet_proof() { + run_large_stack_async_test("degraded-quota-v2", || durable_quota_writes_with_one_offline_disk(true)); + } + fn active_rebalance_meta_for_pool(pool_count: usize, active_pool_idx: usize) -> RebalanceMeta { let now = OffsetDateTime::now_utc(); let mut pool_stats = vec![RebalanceStats::default(); pool_count]; @@ -5018,6 +5554,339 @@ mod tests { shutdown.cancel(); } + #[tokio::test] + #[serial_test::serial(storage_class_env)] + async fn repeated_abort_recovers_exact_orphan_before_decommission_drains() { + use crate::storage_api_contracts::namespace::NamespaceLocking as _; + use sha2::Digest; + + let temp_dir = tempfile::tempdir().expect("create orphan recovery store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "orphan-multipart-drain", &[4, 4])).await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let bucket = format!("orphan-multipart-{}", uuid::Uuid::new_v4()); + let object = "exact-object.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let incarnation = store.bucket_incarnation_id(&bucket).await.expect("bucket incarnation"); + let lifecycle_guard = store.acquire_bucket_lifecycle_read_lock(&bucket).await.expect("bucket guard"); + let mut upload_opts = ObjectOptions { + expected_bucket_incarnation_id: Some(incarnation), + ..Default::default() + }; + upload_opts.add_bucket_lifecycle_lock_guard(&lifecycle_guard); + let upload = store.pools[0] + .new_multipart_upload(&bucket, object, &upload_opts) + .await + .expect("create source upload"); + drop(lifecycle_guard); + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await + .expect("initial abort"); + + let upload_sha = + hex_simd::encode_to_string(sha2::Sha256::digest(format!("{bucket}/{object}").as_bytes()), hex_simd::AsciiCase::Lower); + let upload_uuid = crate::runtime::sources::upload_uuid_suffix(&upload.upload_id); + let staging_dir = |pool_index: usize, disk_index: usize| { + temp_dir + .path() + .join(format!("pool{pool_index}/set0/disk{disk_index}")) + .join(crate::disk::RUSTFS_META_MULTIPART_BUCKET) + .join(&upload_sha) + .join(&upload_uuid) + }; + for disk_index in 0..3 { + let path = staging_dir(0, disk_index); + tokio::fs::create_dir_all(&path).await.expect("create late part directory"); + tokio::fs::write(path.join("part.1"), b"late part after abort") + .await + .expect("write late part fixture"); + } + mark_test_pool_decommissioning(&store, 0).await; + assert!( + store + .list_multipart_uploads(&bucket, "", None, None, None, 100) + .await + .expect("logical upload listing") + .uploads + .is_empty() + ); + assert!(store.ensure_decommission_multipart_uploads_drained_for_test(0).await.is_err()); + assert!( + store + .abort_multipart_upload(&bucket, "different-key", &upload.upload_id, &ObjectOptions::default()) + .await + .is_err(), + "the same UploadID for a different key must not recover this path" + ); + + // A minority metadata copy in another pool blocks recovery everywhere. + let conflicting_metadata = staging_dir(1, 0).join(crate::disk::STORAGE_FORMAT_FILE); + tokio::fs::create_dir_all(conflicting_metadata.parent().expect("metadata parent")) + .await + .expect("create conflicting pool staging"); + tokio::fs::write(&conflicting_metadata, b"unreadable metadata") + .await + .expect("write conflicting metadata"); + assert!( + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await + .is_err(), + "no pool may be cleaned until every pool proves absence" + ); + for disk_index in 0..3 { + assert!(staging_dir(0, disk_index).join("part.1").exists()); + } + tokio::fs::remove_file(conflicting_metadata) + .await + .expect("remove conflicting fixture"); + let (paused, release) = crate::store::multipart::pause_orphan_multipart_cleanup_for_test(&bucket); + let recovery_store = store.clone(); + let recovery_bucket = bucket.clone(); + let recovery_upload = upload.upload_id.clone(); + let mut recovery = tokio::spawn(async move { + recovery_store + .abort_multipart_upload(&recovery_bucket, object, &recovery_upload, &ObjectOptions::default()) + .await + }); + tokio::time::timeout(Duration::from_secs(10), async { + tokio::select! { + paused = paused => paused.expect("cleanup owner announces arrival"), + result = &mut recovery => panic!("recovery returned before physical cleanup: {result:?}"), + } + }) + .await + .expect("recovery should reach physical cleanup"); + recovery.abort(); + assert!(recovery.await.expect_err("request waiter is cancellable").is_cancelled()); + let upload_lock = store.pools[0] + .get_disks_by_key(object) + .new_ns_lock(crate::disk::RUSTFS_META_MULTIPART_BUCKET, &format!("{upload_sha}/{upload_uuid}")) + .await + .expect("same upload namespace"); + let mut upload_writer = Box::pin(upload_lock.get_write_lock(Duration::from_secs(10))); + let mut bucket_writer = Box::pin(store.acquire_bucket_lifecycle_write_lock(&bucket)); + assert!( + futures::poll!(&mut upload_writer).is_pending(), + "cancelled recovery must retain the upload lock" + ); + assert!( + futures::poll!(&mut bucket_writer).is_pending(), + "cancelled recovery must retain the real bucket guard" + ); + release.send(()).expect("allow physical cleanup to finish"); + let (upload_guard, bucket_guard) = + tokio::time::timeout(Duration::from_secs(10), async { tokio::join!(upload_writer, bucket_writer) }) + .await + .expect("both guards should release after physical cleanup"); + drop(upload_guard.expect("upload writer resumes")); + drop(bucket_guard.expect("bucket writer resumes")); + for pool_index in 0..2 { + for disk_index in 0..4 { + assert!(!staging_dir(pool_index, disk_index).exists(), "every checked staging path must be absent"); + } + } + store + .ensure_decommission_multipart_uploads_drained_for_test(0) + .await + .expect("physical decommission gate should open after recovery"); + assert!( + matches!( + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await, + Err(StorageError::InvalidUploadID(_, _, _)) + ), + "an absent upload must not claim to have recovered anything" + ); + shutdown.cancel(); + } + + #[tokio::test] + #[serial_test::serial(storage_class_env)] + async fn repeated_abort_keeps_no_such_upload_while_a_disk_is_offline() { + use sha2::Digest; + + let temp_dir = tempfile::tempdir().expect("create offline-disk recovery store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "orphan-multipart-offline", &[4, 4])).await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let bucket = format!("orphan-multipart-{}", uuid::Uuid::new_v4()); + let object = "offline-object.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let incarnation = store.bucket_incarnation_id(&bucket).await.expect("bucket incarnation"); + let lifecycle_guard = store.acquire_bucket_lifecycle_read_lock(&bucket).await.expect("bucket guard"); + let mut upload_opts = ObjectOptions { + expected_bucket_incarnation_id: Some(incarnation), + ..Default::default() + }; + upload_opts.add_bucket_lifecycle_lock_guard(&lifecycle_guard); + let upload = store.pools[0] + .new_multipart_upload(&bucket, object, &upload_opts) + .await + .expect("create source upload"); + drop(lifecycle_guard); + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await + .expect("initial abort"); + + let upload_sha = + hex_simd::encode_to_string(sha2::Sha256::digest(format!("{bucket}/{object}").as_bytes()), hex_simd::AsciiCase::Lower); + let upload_uuid = crate::runtime::sources::upload_uuid_suffix(&upload.upload_id); + let staging_dir = |disk_index: usize| { + temp_dir + .path() + .join(format!("pool0/set0/disk{disk_index}")) + .join(crate::disk::RUSTFS_META_MULTIPART_BUCKET) + .join(&upload_sha) + .join(&upload_uuid) + }; + for disk_index in 0..3 { + let path = staging_dir(disk_index); + tokio::fs::create_dir_all(&path).await.expect("create late part directory"); + tokio::fs::write(path.join("part.1"), b"late part after abort") + .await + .expect("write late part fixture"); + } + + // One offline disk means absence cannot be proven. The idempotent Abort + // must keep answering NoSuchUpload instead of surfacing a disk error, + // and must not delete anything. + let set = store.pools[0].get_disks_by_key(object); + let offline = set.disks.write().await[3].take(); + assert!(offline.is_some(), "fixture disk must have been online"); + assert!( + matches!( + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await, + Err(StorageError::InvalidUploadID(_, _, _)) + ), + "unprovable absence must keep NoSuchUpload while a disk is offline" + ); + for disk_index in 0..3 { + assert!( + staging_dir(disk_index).join("part.1").exists(), + "no staging may be deleted while a disk is offline" + ); + } + + set.disks.write().await[3] = offline; + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await + .expect("recovery resumes once every disk is observable"); + for disk_index in 0..4 { + assert!(!staging_dir(disk_index).exists(), "recovered staging must be absent"); + } + shutdown.cancel(); + } + + #[tokio::test] + #[serial_test::serial(storage_class_env)] + async fn orphan_multipart_recovery_preserves_current_bucket_incarnation() { + use sha2::Digest; + + let temp_dir = tempfile::tempdir().expect("create bucket lifetime recovery store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "orphan-bucket-lifetime", &[4])).await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let bucket = format!("orphan-lifetime-{}", uuid::Uuid::new_v4()); + let object = "object.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create original bucket"); + let original_incarnation = store.bucket_incarnation_id(&bucket).await.expect("original incarnation"); + let upload = store + .new_multipart_upload(&bucket, object, &ObjectOptions::default()) + .await + .expect("create original upload"); + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await + .expect("abort original upload"); + store + .delete_bucket(&bucket, &crate::storage_api_contracts::bucket::DeleteBucketOptions::default()) + .await + .expect("delete original bucket"); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("recreate bucket"); + assert_ne!(store.bucket_incarnation_id(&bucket).await.expect("new incarnation"), original_incarnation); + let mut reader = PutObjReader::from_vec(b"current bucket object".to_vec()); + store + .put_object(&bucket, object, &mut reader, &ObjectOptions::default()) + .await + .expect("write current bucket object"); + let current_upload = store + .new_multipart_upload(&bucket, object, &ObjectOptions::default()) + .await + .expect("create current incarnation upload for the same key"); + + let upload_sha = + hex_simd::encode_to_string(sha2::Sha256::digest(format!("{bucket}/{object}").as_bytes()), hex_simd::AsciiCase::Lower); + let upload_uuid = crate::runtime::sources::upload_uuid_suffix(&upload.upload_id); + let leftover = temp_dir + .path() + .join("pool0/set0/disk0") + .join(crate::disk::RUSTFS_META_MULTIPART_BUCKET) + .join(upload_sha) + .join(upload_uuid) + .join("part.1"); + tokio::fs::create_dir_all(leftover.parent().expect("staging parent")) + .await + .expect("create old incarnation staging"); + tokio::fs::write(&leftover, b"previous bucket lifetime") + .await + .expect("write previous incarnation fixture"); + for opts in [ + ObjectOptions { + expected_bucket_incarnation_id: Some(original_incarnation), + ..Default::default() + }, + ObjectOptions { + expected_bucket_incarnation_id: Some(uuid::Uuid::nil()), + ..Default::default() + }, + ] { + assert!( + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &opts) + .await + .is_err(), + "a stale or nil current-bucket identity must fail closed" + ); + assert!(leftover.exists(), "failed bucket lifetime proof must preserve the path"); + } + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await + .expect("exact metadata-less staging from an old incarnation is unreferenced"); + assert!(!leftover.exists()); + store + .get_multipart_info(&bucket, object, ¤t_upload.upload_id, &ObjectOptions::default()) + .await + .expect("recovery must preserve the current incarnation's active upload"); + let mut reader = store + .get_object_reader(&bucket, object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("current object must remain readable"); + let mut body = Vec::new(); + reader.stream.read_to_end(&mut body).await.expect("read current object"); + assert_eq!(body, b"current bucket object"); + shutdown.cancel(); + } + #[tokio::test] #[serial_test::serial(storage_class_env)] async fn active_multipart_upload_routes_before_faulted_suspended_source() { @@ -5169,6 +6038,179 @@ mod tests { shutdown.cancel(); } + #[test] + #[serial_test::serial(storage_class_env)] + fn decommission_listing_accepts_committed_inline_layout_change() { + // Keep the native migration scenario off the small debug test stack. + std::thread::Builder::new() + .stack_size(16 * 1024 * 1024) + .spawn(|| { + let runtime = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .worker_threads(2) + .build() + .expect("build migration listing runtime"); + runtime.block_on(async move { + use crate::config::storageclass::{ + INLINE_BLOCK_ENV, OPTIMIZE_ENV, RRS_ENV, STANDARD_ENV, lookup_config_for_pools, + }; + + let temp_dir = tempfile::tempdir().expect("create migration listing store"); + let (_ctx, store, shutdown) = Box::pin(without_storage_class_env(build_isolated_test_store( + temp_dir.path(), + "decommission-inline-listing", + &[3, 9], + ))) + .await; + let _shutdown_guard = shutdown.drop_guard(); + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let bucket = "decommission-inline-listing"; + let object = "object.bin"; + let payload = vec![0x5a; 64 * 1024]; + Box::pin(store.make_bucket(bucket, &MakeBucketOptions::default())) + .await + .expect("create migration listing bucket"); + + // A source written with inline storage disabled remains valid after + // the destination starts using the default inline policy. + let external_config = temp_env::with_vars( + [ + (STANDARD_ENV, None), + (RRS_ENV, None), + (OPTIMIZE_ENV, None), + (INLINE_BLOCK_ENV, Some("0")), + ], + || lookup_config_for_pools(&KVS::new(), &[3, 9]), + ) + .expect("external storage policy"); + for set in &store.pools[1].disk_set { + set.set_test_storage_class_config(external_config.clone()); + } + let source = Box::pin(store.pools[1].put_object( + bucket, + object, + &mut PutObjReader::from_vec(payload.clone()), + &ObjectOptions::default(), + )) + .await + .expect("write external source"); + assert!(!source.inlined, "fixture must use external source shards"); + store.pool_meta.write().await.pools[1].decommission = Some(PoolDecommissionInfo { + start_time: Some(OffsetDateTime::now_utc()), + ..Default::default() + }); + + let reader = Box::pin(store.pools[1].get_object_reader( + bucket, + object, + None, + HeaderMap::new(), + &ObjectOptions { + no_lock: true, + data_movement: true, + raw_data_movement_read: true, + ..Default::default() + }, + )) + .await + .expect("read migration source"); + let incarnation = store.bucket_incarnation_id(bucket).await.expect("bucket incarnation"); + Box::pin(crate::data_movement::migrate_decommission_object( + store.clone(), + 1, + bucket.to_string(), + reader, + Some(incarnation), + "test_decommission_inline_listing", + None, + )) + .await + .expect("commit migrated target while retaining source"); + + for pool_index in [0, 1] { + let mut reader = Box::pin(store.pools[pool_index].get_object_reader( + bucket, + object, + None, + HeaderMap::new(), + &ObjectOptions::default(), + )) + .await + .expect("both physical copies must remain readable"); + assert_eq!(reader.object_info.inlined, pool_index == 0); + assert_eq!(reader.object_info.etag, source.etag); + assert_eq!(reader.object_info.mod_time, source.mod_time); + let mut actual = Vec::new(); + reader + .stream + .read_to_end(&mut actual) + .await + .expect("read complete object body"); + assert_eq!(actual, payload); + } + + Box::pin(store.pools[0].put_object( + bucket, + "z-sentinel", + &mut PutObjReader::from_vec(b"sentinel".to_vec()), + &ObjectOptions::default(), + )) + .await + .expect("write pagination sentinel"); + let listing = Box::pin( + store + .clone() + .inner_list_objects_v2(bucket, "", None, None, 1, false, None, false), + ) + .await + .expect("listing must merge committed source and target copies"); + assert!(listing.is_truncated); + assert_eq!(listing.objects.len(), 1); + assert_eq!(listing.objects[0].name, object); + assert_eq!(listing.objects[0].etag, source.etag); + assert_eq!(listing.objects[0].size, i64::try_from(payload.len()).expect("payload size")); + let next = Box::pin(store.clone().inner_list_objects_v2( + bucket, + "", + listing.next_continuation_token, + None, + 1, + false, + None, + false, + )) + .await + .expect("continue after the merged object"); + assert!(!next.is_truncated); + assert_eq!(next.objects.len(), 1); + assert_eq!(next.objects[0].name, "z-sentinel"); + + Box::pin(store.pools[1].delete_object(bucket, object, ObjectOptions::default())) + .await + .expect("remove the migrated source copy"); + let after_cleanup = Box::pin( + store + .clone() + .inner_list_objects_v2(bucket, "", None, None, 100, false, None, false), + ) + .await + .expect("listing must remain complete after source cleanup"); + assert!(!after_cleanup.is_truncated); + assert_eq!( + after_cleanup + .objects + .iter() + .map(|info| info.name.as_str()) + .collect::>(), + [object, "z-sentinel"] + ); + }); + }) + .expect("spawn migration listing test") + .join() + .expect("migration listing test must complete"); + } + #[test] #[serial_test::serial(storage_class_env)] fn data_movement_conflicts_and_multipart_retries_converge_safely() { @@ -9582,8 +10624,18 @@ mod tests { #[cfg(feature = "test-util")] async fn decommission_migrates_and_verifies_registered_durable_ilm_records_scenario() { let temp_dir = tempfile::tempdir().expect("create temp store dir"); - let (ctx, store, _shutdown) = - without_storage_class_env(build_isolated_test_store(temp_dir.path(), "durable-ilm-decommission", &[4, 4])).await; + // The explicit recovery below must be the only journal consumer, + // including when receipt validation outlasts the background interval. + let mut instance_ctx = crate::runtime::instance::InstanceContext::new(); + instance_ctx.suppress_tier_delete_journal_recovery_for_test(); + let (ctx, store, _shutdown) = without_storage_class_env(build_isolated_test_store_with_layout( + temp_dir.path(), + "durable-ilm-decommission", + &[(1, 4), (1, 4)], + CancellationToken::new(), + Some(Arc::new(instance_ctx)), + )) + .await; let tier_name = "DECOMMISSION-ILM"; let backend = register_transition_reconcile_test_tier(&ctx.tier_config_mgr(), tier_name).await; @@ -10055,6 +11107,16 @@ mod tests { .await .expect("manual task receipt should restore after the corruption check"); + assert!(matches!( + com::read_config(store.pools[0].clone(), &tier_path).await, + Err(Error::ConfigNotFound) + )); + assert_eq!( + com::read_config(store.pools[1].clone(), &tier_path) + .await + .expect("manual recovery must still observe the exact migrated tier journal"), + tier_bytes + ); let tier_stats = recover_tier_delete_journal_entries(store.clone(), 100, None) .await .expect("tier journal recovery should consume the migrated record before completion"); @@ -10811,8 +11873,19 @@ mod tests { const JOURNAL_COUNT: usize = 65; let temp_dir = tempfile::tempdir().expect("create bounded rollback store dir"); - let (ctx, store, _shutdown) = - without_storage_class_env(build_isolated_test_store(temp_dir.path(), "bounded-dispatch-rollback", &[4])).await; + // The test's worker must be the only one to roll back the seeded manifest: if + // the setup outlasts TIER_DELETE_JOURNAL_RECOVERY_INTERVAL, a background pass + // rolls it back before the slow-delete hook is installed. + let mut instance_ctx = crate::runtime::instance::InstanceContext::new(); + instance_ctx.suppress_tier_delete_journal_recovery_for_test(); + let (ctx, store, _shutdown) = without_storage_class_env(build_isolated_test_store_with_layout( + temp_dir.path(), + "bounded-dispatch-rollback", + &[(1, 4)], + CancellationToken::new(), + Some(Arc::new(instance_ctx)), + )) + .await; crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; let bucket = "bounded-dispatch-rollback-bucket"; store @@ -10968,8 +12041,19 @@ mod tests { #[serial_test::serial(storage_class_env)] async fn dispatch_manifest_page_timeout_detaches_one_deduplicated_worker_until_convergence() { let temp_dir = tempfile::tempdir().expect("create detached rollback store dir"); - let (ctx, store, _shutdown) = - without_storage_class_env(build_isolated_test_store(temp_dir.path(), "detached-dispatch-rollback", &[4])).await; + // The test's own passes must be the only ones to process the seeded manifest: + // if the setup outlasts TIER_DELETE_JOURNAL_RECOVERY_INTERVAL, a background + // pass rolls it back before the slow-delete hook is installed. + let mut instance_ctx = crate::runtime::instance::InstanceContext::new(); + instance_ctx.suppress_tier_delete_journal_recovery_for_test(); + let (ctx, store, _shutdown) = without_storage_class_env(build_isolated_test_store_with_layout( + temp_dir.path(), + "detached-dispatch-rollback", + &[(1, 4)], + CancellationToken::new(), + Some(Arc::new(instance_ctx)), + )) + .await; crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; let bucket = "detached-dispatch-rollback-bucket"; store @@ -11037,9 +12121,20 @@ mod tests { const WORKER_LIMIT: usize = 4; let temp_dir = tempfile::tempdir().expect("create bounded detached rollback store dir"); - let (ctx, store, shutdown) = - without_storage_class_env(build_isolated_test_store(temp_dir.path(), "bounded-detached-dispatch-rollback", &[4])) - .await; + // The test's passes must be the only ones to process the seeded manifests: if + // the setup outlasts TIER_DELETE_JOURNAL_RECOVERY_INTERVAL, a background pass + // rolls them back before the pause hook is installed. The restarted store + // below keeps background recovery, which its half of the test relies on. + let mut instance_ctx = crate::runtime::instance::InstanceContext::new(); + instance_ctx.suppress_tier_delete_journal_recovery_for_test(); + let (ctx, store, shutdown) = without_storage_class_env(build_isolated_test_store_with_layout( + temp_dir.path(), + "bounded-detached-dispatch-rollback", + &[(1, 4)], + CancellationToken::new(), + Some(Arc::new(instance_ctx)), + )) + .await; crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; let tier_name = "BOUNDED-DETACHED-DISPATCH-ROLLBACK"; let _backend = register_mock_tier(&ctx.tier_config_mgr(), tier_name).await; @@ -16430,11 +17525,10 @@ mod tests { .delete_object_with_tier_delete_journal(bucket, object, ObjectOptions::default()) .await .expect_err("one post-commit error plus one offline disk must miss delete quorum"); - assert!(matches!(err, StorageError::Io(_)), "unexpected local delete failure: {err:?}"); - assert!( - err.to_string() - .contains("Storage resources are insufficient for the write operation"), - "the caller must observe the wrapped quorum failure: {err}" + assert_eq!( + err, + StorageError::InsufficientWriteQuorum(bucket.to_owned(), object.to_owned()), + "the caller must observe the typed quorum failure with its object identity" ); set.disks.write().await[1] = Some(saved_disk); @@ -17361,6 +18455,19 @@ mod tests { #[tokio::test] #[serial_test::serial(storage_class_env)] async fn delete_holds_bucket_incarnation_sentinel_through_object_commit() { + // The recreation waits for the delete's sentinel on its lifecycle write lock + // from before `barrier.release()`, so that lock wait also has to cover the + // delete's commit. Raise the lock acquire timeout from the 5 s default to give + // the commit room on a starved runner; a kept sentinel still fails the test, + // after 30 s instead of 5 s. + temp_env::async_with_vars( + [(rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT, Some("30"))], + delete_holds_bucket_incarnation_sentinel_through_object_commit_body(), + ) + .await; + } + + async fn delete_holds_bucket_incarnation_sentinel_through_object_commit_body() { let temp = tempfile::tempdir().expect("create temp store dir"); let (_ctx, store, _shutdown) = without_storage_class_env(build_isolated_test_store(temp.path(), "delete-bucket-incarnation", &[4])).await; @@ -17433,13 +18540,19 @@ mod tests { "bucket delete and same-name recreation must wait for the in-flight object delete" ); barrier.release(); - delete + // The recreation's lock wait started before the release; give the delete as long. + tokio::time::timeout(Duration::from_secs(30), delete) .await + .expect("delete should finish once the barrier is released") .expect("delete task should join") .expect("old-generation delete should complete"); - tokio::time::timeout(Duration::from_secs(10), recreate) + // If the delete kept the sentinel, `delete_bucket` would fail on its lifecycle + // write lock after the lock acquire timeout (30 s here). The rest of the + // recreation is fsync-bound IO on four disks, so this bound, above the lock + // timeout, only stops a hang. + tokio::time::timeout(Duration::from_secs(120), recreate) .await - .expect("bucket recreation should finish after delete releases the sentinel") + .expect("bucket recreation should not hang after delete releases the sentinel") .expect("bucket recreation task should join") .expect("bucket recreation should succeed"); @@ -17492,6 +18605,19 @@ mod tests { #[tokio::test] #[serial_test::serial(storage_class_env)] async fn delete_objects_holds_bucket_incarnation_sentinel_through_commit() { + // The recreation waits for the delete's sentinel on its lifecycle write lock + // from before `barrier.release()`, so that lock wait also has to cover the + // delete's commit. Raise the lock acquire timeout from the 5 s default to give + // the commit room on a starved runner; a kept sentinel still fails the test, + // after 30 s instead of 5 s. + temp_env::async_with_vars( + [(rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT, Some("30"))], + delete_objects_holds_bucket_incarnation_sentinel_through_commit_body(), + ) + .await; + } + + async fn delete_objects_holds_bucket_incarnation_sentinel_through_commit_body() { let temp = tempfile::tempdir().expect("create temp store dir"); let (_ctx, store, _shutdown) = without_storage_class_env(build_isolated_test_store(temp.path(), "batch-delete-bucket-incarnation", &[4])).await; @@ -17553,14 +18679,22 @@ mod tests { "bucket recreation must wait for the in-flight batch delete" ); barrier.release(); - let (_deleted, errors) = delete.await.expect("batch delete task should join"); + // The recreation's lock wait started before the release; give the delete as long. + let (_deleted, errors) = tokio::time::timeout(Duration::from_secs(30), delete) + .await + .expect("batch delete should finish once the barrier is released") + .expect("batch delete task should join"); assert!( errors.iter().all(Option::is_none), "old-generation batch delete should complete: {errors:?}" ); - tokio::time::timeout(Duration::from_secs(10), recreate) + // If the delete kept the sentinel, `delete_bucket` would fail on its lifecycle + // write lock after the lock acquire timeout (30 s here). The rest of the + // recreation is fsync-bound IO on four disks, so this bound, above the lock + // timeout, only stops a hang. + tokio::time::timeout(Duration::from_secs(120), recreate) .await - .expect("bucket recreation should finish after batch delete releases the sentinel") + .expect("bucket recreation should not hang after batch delete releases the sentinel") .expect("bucket recreation task should join") .expect("bucket recreation should succeed"); store @@ -17774,6 +18908,15 @@ mod tests { .put_object(&bucket, object, &mut reader, &ObjectOptions::default()) .await .expect("put expired object"); + let lifecycle_xml = br#"expireEnabled0"#; + crate::bucket::metadata_sys::update_in(&store.ctx, &bucket, BUCKET_LIFECYCLE_CONFIG, lifecycle_xml.to_vec()) + .await + .expect("install lifecycle policy"); + let expiry_policy = crate::bucket::lifecycle::get_expiry_configs(&store, &bucket) + .await + .unwrap() + .lifecycle + .map(Into::into); let bucket_incarnation_id = store.bucket_incarnation_id(&bucket).await.expect("read bucket incarnation"); let barrier = crate::store::object::DeleteAfterObjectLockSnapshotBarrier::install(&bucket); @@ -17784,6 +18927,7 @@ mod tests { &object_info, &crate::bucket::lifecycle::lifecycle::Event { action: crate::bucket::lifecycle::lifecycle::IlmAction::DeleteAction, + expiry_policy, ..Default::default() }, &crate::bucket::lifecycle::bucket_lifecycle_audit::LcEventSrc::Scanner, @@ -18464,11 +19608,21 @@ mod tests { .await .expect("put replacement object"); + let lifecycle_xml = br#"expireEnabled0"#; + crate::bucket::metadata_sys::update_in(&store.ctx, &bucket, BUCKET_LIFECYCLE_CONFIG, lifecycle_xml.to_vec()) + .await + .expect("install lifecycle policy"); + let expiry_policy = crate::bucket::lifecycle::get_expiry_configs(&store, &bucket) + .await + .unwrap() + .lifecycle + .map(Into::into); let applied = crate::bucket::lifecycle::bucket_lifecycle_ops::apply_expiry_on_non_transitioned_objects( store.clone(), &old_object, &crate::bucket::lifecycle::lifecycle::Event { action: crate::bucket::lifecycle::lifecycle::IlmAction::DeleteAction, + expiry_policy, ..Default::default() }, &crate::bucket::lifecycle::bucket_lifecycle_audit::LcEventSrc::Scanner, @@ -18486,6 +19640,70 @@ mod tests { assert_eq!(body, b"new-generation"); } + #[tokio::test(flavor = "multi_thread")] + #[serial_test::serial] + #[serial_test::serial(storage_class_env)] + async fn cancelled_ecstore_put_part_keeps_bucket_lifecycle_lock_through_commit() { + for pause in [ + crate::set_disk::MultipartCommitPause::PutPartAfterCapacityAdmission, + crate::set_disk::MultipartCommitPause::PutPartAfterRename, + ] { + let temp_dir = tempfile::tempdir().expect("create cancelled multipart lifecycle store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "cancelled-multipart-lifecycle", &[4, 4])) + .await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let bucket = format!("cancelled-part-lifecycle-{}", Uuid::new_v4()); + let object = "part.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket for multipart lifecycle cancellation"); + let upload = store + .new_multipart_upload(&bucket, object, &ObjectOptions::default()) + .await + .expect("create upload through the owning ECStore"); + let barrier = crate::set_disk::MultipartCommitBarrier::install(&bucket, object, pause); + let put_store = Arc::clone(&store); + let put_bucket = bucket.clone(); + let put_upload_id = upload.upload_id.clone(); + let put = tokio::spawn(async move { + let mut reader = PutObjReader::from_vec(vec![0x45; 4096]); + put_store + .put_object_part(&put_bucket, object, &put_upload_id, 1, &mut reader, &ObjectOptions::default()) + .await + }); + barrier.wait_until_paused().await; + put.abort(); + assert!( + put.await + .expect_err("the ECStore request waiter must remain cancellable") + .is_cancelled() + ); + + let mut writer = Box::pin(store.acquire_bucket_lifecycle_write_lock(&bucket)); + assert!( + tokio::time::timeout(Duration::from_millis(100), &mut writer).await.is_err(), + "canceling UploadPart at {pause:?} must not release its bucket lifecycle owner before the commit drains" + ); + barrier.release(); + let lifecycle_guard = tokio::time::timeout(Duration::from_secs(10), writer) + .await + .expect("the bucket lifecycle writer must resume after the part commit drains") + .expect("acquire the bucket lifecycle write lock after cancellation"); + drop(lifecycle_guard); + + let parts = store + .list_object_parts(&bucket, object, &upload.upload_id, None, 100, &ObjectOptions::default()) + .await + .expect("the cancelled request's owned part commit must finish successfully"); + assert_eq!(parts.parts.len(), 1, "the commit must publish exactly the uploaded part at {pause:?}"); + assert_eq!(parts.parts[0].part_num, 1); + assert_eq!(parts.parts[0].size, 4096); + shutdown.cancel(); + } + } + #[tokio::test] #[serial_test::serial(storage_class_env)] async fn pre_upgrade_multipart_upload_can_complete_and_abort_in_the_same_bucket_lifetime() { @@ -18959,10 +20177,13 @@ mod tests { } #[cfg(feature = "test-util")] - #[tokio::test] + #[test] #[serial_test::serial(storage_class_env)] - async fn legacy_recovery_disposition_removes_only_local_journals_and_replays() { - Box::pin(legacy_recovery_disposition_removes_only_local_journals_and_replays_case()).await; + fn legacy_recovery_disposition_removes_only_local_journals_and_replays() { + run_large_stack_async_test( + "legacy-recovery-disposition", + legacy_recovery_disposition_removes_only_local_journals_and_replays_case, + ); } #[cfg(feature = "test-util")] @@ -20042,6 +21263,11 @@ mod tests { let lifecycle_event = crate::bucket::lifecycle::lifecycle::Event { action: rustfs_scanner_metrics::metrics::IlmAction::DeleteAllVersionsAction, rule_id: "delete-all-versions".to_string(), + expiry_policy: crate::bucket::lifecycle::get_expiry_configs(&store, bucket) + .await + .unwrap() + .lifecycle + .map(Into::into), ..Default::default() }; let legacy_applied = @@ -22038,7 +23264,6 @@ mod tests { .make_bucket(bucket, &MakeBucketOptions::default()) .await .expect("kill-point bucket should be created"); - let _compaction_proof = install_transition_transaction_compaction_fleet_proof_for_test("object-transaction-fencing-test"); for (index, point, expected_state, expected_revision, expected_committed, expected_recovered) in [ ( @@ -22082,10 +23307,14 @@ mod tests { .await .expect("kill-point source should be written"); let remote_before = backend.object_count().await; + // Earlier cancellation/recovery cases must not consume this + // transaction's 30-second homogeneous-fleet proof window. + let _compaction_proof = + install_transition_transaction_compaction_fleet_proof_for_test("object-transaction-fencing-test"); let barrier = TransitionTransactionKillPointBarrier::install(bucket, &object, point); let transition_store = store.clone(); let transition_object = object.clone(); - let transition = tokio::spawn(async move { + let mut transition = tokio::spawn(async move { transition_store .transition_object( bucket, @@ -22104,7 +23333,12 @@ mod tests { ) .await }); - barrier.wait_until_paused().await; + tokio::select! { + _ = barrier.wait_until_paused() => {} + result = &mut transition => { + panic!("transition exited before reaching {point:?}: {result:?}"); + } + } transition.abort(); assert!( transition diff --git a/crates/ecstore/src/store/init/tests/artifact_receiver_tests.rs b/crates/ecstore/src/store/init/tests/artifact_receiver_tests.rs index efc256e3a..3bcb0a07a 100644 --- a/crates/ecstore/src/store/init/tests/artifact_receiver_tests.rs +++ b/crates/ecstore/src/store/init/tests/artifact_receiver_tests.rs @@ -906,3 +906,156 @@ async fn repair_physical_tail_case(metadata: bool, retirement: bool) { ); shutdown.cancel(); } + +#[test] +#[serial_test::serial(storage_class_env)] +fn artifact_recovery_receiver_reclassifies_after_waiting_across_bucket_recreation() { + use crate::disk::{DeleteOptions, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, error::DiskError, os}; + use crate::storage_api_contracts::bucket::DeleteBucketOptions; + use std::{future::Future as _, task::Poll}; + + run_large_stack_async_test("artifact-recovery-recreated", || async { + let root = tempfile::tempdir().expect("recreated recovery receiver root"); + let (ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(root.path(), "artifact-recovery-recreated", &[4])).await; + metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let ordinary = bucket(&store, BUCKET, false).await; + let (disk, reference, _) = disk(&store).await; + let attempt = prefix(); + let control = format!("{attempt}control"); + let object = format!("{attempt}late-marker"); + let marker = |object: &str| FileInfo { + volume: BUCKET.to_owned(), + name: object.to_owned(), + version_id: Some(Uuid::new_v4()), + deleted: true, + mark_deleted: true, + mod_time: Some(time::OffsetDateTime::now_utc()), + ..Default::default() + }; + let operation = |object: &str, fi: FileInfo| DeleteRecoveryRpcOperation::ApplyVersion { + object: object.to_owned(), + file_info: Box::new(fi), + // Missing keys require explicit marker creation in the recovery protocol. + force_del_marker: true, + opts: DeleteOptions { + old_data_dir: Some(Uuid::new_v4()), + ..Default::default() + }, + }; + let control_marker = marker(&control); + assert!(matches!( + tokio::time::timeout( + DEADLINE, + store.recover_local_delete_transaction(&reference, BUCKET, operation(&control, control_marker.clone())) + ) + .await + .expect("bounded ordinary marker publication") + .expect("ordinary recovery really creates a marker"), + DeleteRecoveryRpcReply::AppliedVersion + )); + let read = disk + .read_version(BUCKET, BUCKET, &control, "", &ReadOptions::default()) + .await + .expect("read actual ordinary control marker"); + assert!(read.deleted); + assert_eq!(read.version_id, control_marker.version_id); + tokio::time::timeout(DEADLINE, async { + while ctx.namespace_commits_pending() { + tokio::task::yield_now().await; + } + }) + .await + .expect("control physical owner drains before measuring admission"); + + // This existing mutex precedes volume admission. It holds no volume + // READ, so a real DeleteBucket and managed recreation can pass the waiter. + let blocked = os::acquire_metadata_mutation_lease(&disk.path().join(BUCKET).join(&object), None).await; + let mut mutation = + Box::pin(store.recover_local_delete_transaction(&reference, BUCKET, operation(&object, marker(&object)))); + tokio::time::timeout( + DEADLINE, + std::future::poll_fn(|cx| { + assert!( + mutation.as_mut().poll(cx).is_pending(), + "receiver must really wait for metadata admission" + ); + if ctx.namespace_commits_pending() { + Poll::Ready(()) + } else { + Poll::Pending + } + }), + ) + .await + .expect("receiver reached its real admitted metadata wait"); + tokio::time::timeout( + DEADLINE, + store.delete_bucket( + BUCKET, + &DeleteBucketOptions { + force: true, + ..Default::default() + }, + ), + ) + .await + .expect("bucket deletion is not blocked by a signal-only observation") + .expect("delete the ordinary generation while receiver waits"); + let managed = tokio::time::timeout(DEADLINE, store.clone().make_artifact_bucket(BUCKET, "outputs/")) + .await + .expect("bounded managed recreation") + .expect("commit real new managed authority"); + assert_ne!(managed, ordinary); + let disks = store.pools[0].disk_set[0] + .disk_inventory() + .await + .into_iter() + .flatten() + .collect::>(); + assert_eq!(disks.len(), 4); + let census = || { + let mut pending = disks + .iter() + .flat_map(|disk| [disk.path().join(BUCKET), disk.path().join(".rustfs.sys/buckets").join(BUCKET)]) + .collect::>(); + let mut entries = Vec::new(); + while let Some(path) = pending.pop() { + if !path.exists() { + continue; + } + if path.is_dir() { + entries.push((path.clone(), None)); + pending.extend( + std::fs::read_dir(path) + .expect("census directory") + .map(|entry| entry.expect("census entry").path()), + ); + } else { + entries.push((path.clone(), Some(std::fs::read(path).expect("census file")))); + } + } + entries.sort(); + entries + }; + let before = census(); + drop(blocked); + let result = tokio::time::timeout(DEADLINE, mutation) + .await + .expect("bounded resumed receiver"); + assert_eq!( + result.expect_err("admitted receiver must classify the new managed generation"), + DiskError::FileAccessDenied + ); + assert_eq!(census(), before, "every shard, version, metadata and recovery path remains unchanged"); + assert_eq!( + store + .bucket_incarnation_id_from_disk(BUCKET) + .await + .expect("retained managed authority"), + managed + ); + assert!(!ctx.namespace_commits_pending(), "rejected admission releases its actual owner"); + shutdown.cancel(); + }); +} diff --git a/crates/ecstore/src/store/init_format.rs b/crates/ecstore/src/store/init_format.rs index 9bfb62c60..c5f786848 100644 --- a/crates/ecstore/src/store/init_format.rs +++ b/crates/ecstore/src/store/init_format.rs @@ -134,6 +134,18 @@ pub(crate) async fn connect_load_init_formats_with_instance_ctx( { return Err(Error::CorruptedFormat); } + if matches!(format_quorum, Some(Err(Error::ErasureReadQuorum))) + && let Some((format, members)) = select_existing_format_read_quorum(&formats, &errs, set_drive_count) + { + retain_format_quorum_members(instance_ctx, disks, &format, &members, set_drive_count).await?; + // Loading an existing format grants no authority to initialize pool + // metadata or migrate formats. The persisted storage classes and each + // object's EC metadata still determine service and operation quorums. + return Ok(LoadedFormat { + format, + pool_meta_bootstrap_authority: PoolMetaBootstrapAuthority::None, + }); + } let resumable_partial_migration = formats_present > 0 && errs.iter().any(|error| matches!(error, Some(DiskError::UnformattedDisk))) && format_quorum.as_ref().is_some_and(Result::is_err) @@ -578,6 +590,43 @@ pub fn get_format_erasure_in_quorum(formats: &[Option], slot_offset: u select_format_erasure_in_quorum(formats, slot_offset).map(|(format, _)| format) } +/// Load a consistent existing layout with enough members per set to read the +/// max-parity system metadata containing the persisted storage classes. This +/// is not object-read admission: STANDARD, RRS and historical objects can have +/// different EC layouts, which are validated after this bootstrap read. +/// Format migration and healing still use strict-majority election. +fn select_existing_format_read_quorum( + formats: &[Option], + errors: &[Option], + set_drive_count: usize, +) -> Option<(FormatV3, Vec)> { + if set_drive_count == 0 || formats.is_empty() || formats.len() != errors.len() { + return None; + } + let metadata_parity = set_drive_count / 2; + let metadata_read_quorum = set_drive_count - metadata_parity; + let reference = formats.iter().flatten().next()?; + if !formats_match_reference_slots(formats, reference, 0) + || check_format_erasure_value_for_topology(reference, formats.len(), set_drive_count).is_err() + || formats.iter().zip(errors).any(|(format, error)| match (format, error) { + (Some(_), None) => false, + (None, Some(DiskError::DiskNotFound)) => false, + (None, Some(error)) => !is_network_like_disk_error(error), + _ => true, + }) + || formats + .chunks(set_drive_count) + .any(|set| set.iter().flatten().count() < metadata_read_quorum) + { + return None; + } + + let mut format = reference.clone(); + format.erasure.this = Uuid::nil(); + format.disk_info = None; + Some((format, formats.iter().map(Option::is_some).collect())) +} + pub(crate) fn select_format_erasure_in_quorum(formats: &[Option], slot_offset: usize) -> Result<(FormatV3, Vec)> { let mut candidates = HashMap::new(); let formats_present = formats.iter().flatten().count(); @@ -1120,6 +1169,190 @@ mod tests { assert!(!formats_match_reference_slots(&formats, &reference, 0)); } + fn half_pool_formats() -> (Vec>, Vec>) { + let reference = FormatV3::new(6, 4); + let formats = (0..24) + .map(|slot| { + (slot % 4 < 2).then(|| { + let mut format = reference.clone(); + format.erasure.this = reference.erasure.sets[slot / 4][slot % 4]; + format + }) + }) + .collect::>(); + let errors = formats + .iter() + .map(|format| format.is_none().then_some(DiskError::DiskNotFound)) + .collect(); + (formats, errors) + } + + #[test] + fn existing_format_read_quorum_follows_each_sets_metadata_layout() { + // These are system-metadata EC boundaries, independent of user-object + // storage classes and of how many drives are hosted by each node. + for (set_width, read_quorum) in [(1, 1), (2, 1), (3, 2), (4, 2), (6, 3), (8, 4), (12, 6), (16, 8)] { + for set_count in [1, 2, 5] { + let reference = FormatV3::new(set_count, set_width); + let mut formats = (0..set_count * set_width) + .map(|slot| { + let set = slot / set_width; + let drive = slot % set_width; + // Rotate surviving slots between sets so admission + // cannot depend on the first drive or node being up. + ((drive + set) % set_width < read_quorum).then(|| { + let mut format = reference.clone(); + format.erasure.this = reference.erasure.sets[set][drive]; + format + }) + }) + .collect::>(); + let mut errors = formats + .iter() + .map(|format| format.is_none().then_some(DiskError::DiskNotFound)) + .collect::>(); + let (loaded, members) = select_existing_format_read_quorum(&formats, &errors, set_width) + .unwrap_or_else(|| panic!("metadata read quorum must load {set_count} sets of {set_width} drives")); + assert_eq!(loaded.shared_identity(), reference.shared_identity()); + assert!(loaded.erasure.this.is_nil()); + assert_eq!(members.iter().filter(|member| **member).count(), set_count * read_quorum); + + // More members in other sets cannot compensate for one set + // falling below its own metadata reconstruction boundary. + for slot in 0..(set_count - 1) * set_width { + let mut format = reference.clone(); + format.erasure.this = reference.erasure.sets[slot / set_width][slot % set_width]; + formats[slot] = Some(format); + errors[slot] = None; + } + let lost_slot = formats + .iter() + .rposition(Option::is_some) + .expect("at least one surviving drive"); + formats[lost_slot] = None; + errors[lost_slot] = Some(DiskError::DiskNotFound); + assert!( + select_existing_format_read_quorum(&formats, &errors, set_width).is_none(), + "one set below read quorum must block {set_count} sets of {set_width} drives" + ); + } + } + } + + #[test] + fn existing_format_read_quorum_preserves_election_and_set_boundaries() { + let (mut formats, mut errors) = half_pool_formats(); + assert!(select_existing_format_read_quorum(&[], &[], 0).is_none()); + assert!(select_existing_format_read_quorum(&formats, &errors[..23], 4).is_none()); + assert!(select_existing_format_read_quorum(&formats, &errors, 0).is_none()); + assert!(select_existing_format_read_quorum(&formats, &errors, 6).is_none()); + assert!(select_existing_format_read_quorum(&formats, &errors, 4).is_some()); + assert!(matches!(get_format_erasure_in_quorum(&formats, 0), Err(Error::ErasureReadQuorum))); + + let mut extra = formats[0].as_ref().expect("seed format").clone(); + extra.erasure.this = extra.erasure.sets[0][2]; + formats[2] = Some(extra); + errors[2] = None; + assert!(get_format_erasure_in_quorum(&formats, 0).is_ok(), "13 votes must establish a majority"); + formats[5] = None; + errors[5] = Some(DiskError::DiskNotFound); + assert!( + select_existing_format_read_quorum(&formats, &errors, 4).is_none(), + "12 votes cannot compensate for a set with only one readable drive" + ); + formats[2] = None; + errors[2] = Some(DiskError::DiskNotFound); + assert!(select_existing_format_read_quorum(&formats, &errors, 4).is_none(), "11 votes must wait"); + } + + #[test] + fn existing_format_read_quorum_rejects_conflicts_invalid_slots_and_blank_disks() { + for failure in [ + "conflict", + "wrong_slot", + "nil", + "duplicate", + "blank", + "corrupt", + "access_denied", + "tie", + ] { + let (mut formats, mut errors) = half_pool_formats(); + match failure { + "conflict" => formats[1].as_mut().expect("second format").id = Uuid::new_v4(), + "wrong_slot" => { + let first_id = formats[0].as_ref().expect("first format").erasure.this; + formats[1].as_mut().expect("second format").erasure.this = first_id; + } + "nil" => { + for format in formats.iter_mut().flatten() { + format.id = Uuid::nil(); + } + } + "duplicate" => { + for format in formats.iter_mut().flatten() { + format.erasure.sets[5][3] = format.erasure.sets[0][0]; + } + } + "blank" => errors[2] = Some(DiskError::UnformattedDisk), + "corrupt" => errors[2] = Some(DiskError::CorruptedFormat), + "access_denied" => errors[2] = Some(DiskError::FileAccessDenied), + "tie" => { + let other = FormatV3::new(6, 4); + for slot in 0..24 { + if formats[slot].is_none() { + let mut format = other.clone(); + format.erasure.this = other.erasure.sets[slot / 4][slot % 4]; + formats[slot] = Some(format); + errors[slot] = None; + } + } + } + _ => unreachable!(), + } + assert!( + select_existing_format_read_quorum(&formats, &errors, 4).is_none(), + "{failure} must not authorize degraded startup" + ); + } + } + + #[tokio::test] + async fn existing_format_load_accepts_each_half_of_four_by_six_without_format_writes() { + let (_root, all_disks) = local_disks_with_set_width(24, 4).await; + let reference = FormatV3::new(6, 4); + for (slot, disk) in all_disks.iter().enumerate() { + let mut format = reference.clone(); + format.erasure.this = reference.erasure.sets[slot / 4][slot % 4]; + save_format_file(disk, &Some(format)).await.expect("persist existing format"); + } + let before = format_bytes(&all_disks).await; + for first in 0..4 { + for second in first + 1..4 { + let mut disks = all_disks.clone(); + for (slot, disk) in disks.iter_mut().enumerate() { + if slot % 4 != first && slot % 4 != second { + *disk = None; + } + } + let loaded = connect_load_init_formats_with_instance_ctx( + &Arc::new(InstanceContext::new()), + first == 0, + &mut disks, + 6, + 4, + None, + ) + .await + .expect("two nodes must load the existing format"); + assert_eq!(loaded.format.shared_identity(), reference.shared_identity()); + assert!(matches!(loaded.pool_meta_bootstrap_authority, PoolMetaBootstrapAuthority::None)); + assert_eq!(disks.iter().flatten().count(), 12); + assert_eq!(format_bytes(&all_disks).await, before, "read startup must never rewrite formats"); + } + } + } + #[tokio::test] async fn existing_format_load_succeeds_with_a_strict_majority() { let (_temp_dir, mut disks) = two_local_disks_with_missing_third().await; diff --git a/crates/ecstore/src/store/list_objects.rs b/crates/ecstore/src/store/list_objects.rs index 13721bd4e..9c737318b 100644 --- a/crates/ecstore/src/store/list_objects.rs +++ b/crates/ecstore/src/store/list_objects.rs @@ -24,7 +24,7 @@ use crate::bucket::lifecycle::bucket_lifecycle_ops::free_version_remote_tuple_ma use crate::bucket::metadata_sys::{ get_versioning_config, has_authoritative_never_versioned_state, has_authoritative_never_versioned_state_in, }; -use crate::bucket::utils::check_list_objs_args; +use crate::bucket::utils::{check_list_objs_args, has_bad_path_component}; use crate::bucket::versioning::VersioningApi; use crate::cache_value::metacache_set::{ FallbackClaimTracker, ListPathRawOptions, list_path_raw_with_claim_tracker, list_path_raw_with_partial_result, @@ -56,7 +56,7 @@ use rustfs_filemeta::{ }; use rustfs_io_metrics::{ LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED, LIST_OBJECTS_GATHER_OUTCOME_LIMIT_REACHED, LIST_OBJECTS_SOURCE_WALKER, - ListObjectsGatherObservation, ListObjectsIndexPageObservation, + ListObjectsGatherCaller, ListObjectsGatherObservation, ListObjectsIndexPageObservation, }; use rustfs_utils::path::{self, SLASH_SEPARATOR, base_dir_from_prefix}; use std::cmp::Reverse; @@ -109,6 +109,7 @@ fn recursive_walk_list_path_options(bucket: String, prefix: &str, opts: &WalkOpt incl_deleted: true, recursive: true, versioned: true, + caller: ListObjectsGatherCaller::S3ListVersions, walkdir_timeout: opts.walkdir_timeout, walkdir_stall_timeout: opts.walkdir_stall_timeout, ..Default::default() @@ -274,6 +275,7 @@ fn log_list_path_finished( #[derive(Debug, Default, Clone)] pub struct ListPathOptions { + pub caller: ListObjectsGatherCaller, pub id: Option, // Bucket of the listing. @@ -4112,10 +4114,17 @@ impl ECStore { max_keys: i32, incl_deleted: bool, ) -> Result { + // Stored keys cannot contain repeated separators. Keep the query literal + // and avoid walking or cleaning up an aliased filesystem path. + if prefix.contains("//") && !has_bad_path_component(prefix) && !prefix.contains('\0') { + check_list_objs_args(bucket, "", &marker)?; + return Ok(ListObjectsInfo::default()); + } let max_keys = normalize_max_keys(max_keys); let effective_max_keys = if max_keys <= 0 { 0 } else { max_keys_plus_one(max_keys, true) }; let mut opts = ListPathOptions { bucket: bucket.to_owned(), + caller: ListObjectsGatherCaller::S3List, prefix: prefix.to_owned(), separator: delimiter.clone(), // Always request max_keys + 1 to detect if there are more results @@ -4261,6 +4270,79 @@ impl ECStore { .await } + /// Capture one logical key's version group for write-triggered expiry. + pub(crate) async fn get_object_versions_for_lifecycle( + &self, + bucket: &str, + object: &str, + opts: &ObjectOptions, + bucket_incarnation_id: Uuid, + ) -> Result> { + crate::bucket::utils::check_object_args(bucket, object)?; + let mut opts = opts.clone(); + // Lock order matches writes: bucket lifecycle, metadata validation, + // then object namespace. Borrow fences supplied by an outer operation. + let bucket_guard = if opts.bucket_lifecycle_lock_fence.is_none() { + let guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?; + opts.add_bucket_lifecycle_lock_guard(&guard); + Some(guard) + } else { + None + }; + let incarnation = crate::bucket::metadata_sys::get_bucket_incarnation_id_for_options_in(&self.ctx, bucket, &opts).await?; + if incarnation != bucket_incarnation_id { + return Err(Error::BucketNotFound(bucket.to_owned())); + } + let metadata_sys = crate::bucket::metadata_sys::bucket_metadata_sys_of(&self.ctx)?; + let metadata_sys = metadata_sys.read().await.clone(); + let (versioning, _) = metadata_sys.get_versioning_config(bucket).await?; + let disk_object = path::encode_dir_object(object); + let object_guard = self + .acquire_object_read_lock_if_needed("lifecycle_versions", bucket, &disk_object, &mut opts) + .await?; + + // As with version listings, include every physical set: migration can + // leave a key's history in more than its current hash destination. + let reads = self + .pools + .iter() + .flat_map(|pool| &pool.disk_set) + .map(|set| set.read_object_versions_for_lifecycle(bucket, object)); + let mut entries = Vec::new(); + for result in join_all(reads).await { + if let Some(entry) = result.map_err(|error| to_object_err(error.into(), vec![bucket, object]))? { + entries.push(entry); + } + } + if self.pools.is_empty() + || bucket_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) + || opts + .bucket_lifecycle_lock_fence + .as_ref() + .is_some_and(|fence| fence.is_lock_lost()) + || opts.namespace_lock_fence.as_ref().is_some_and(|fence| fence.is_lock_lost()) + || object_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) + { + return Err(Error::PreconditionFailed); + } + let mut entries = entries.into_iter(); + let Some(mut entry) = entries.next() else { + return Ok(Vec::new()); + }; + if entries.len() > 0 { + merge_object_entry_versions(&mut entry, entries)?; + } + // This projection excludes tier free-version bookkeeping but retains + // pending version purges for the evaluator's replication safeguards. + let versions = entry.file_info_versions(bucket)?; + let versioned = versioning.versioned(object); + Ok(versions + .versions + .iter() + .map(|version| ObjectInfo::from_file_info(version, bucket, object, versioned)) + .collect()) + } + async fn inner_list_object_versions_with_projection( self: Arc, input: ListObjectVersionsInput<'_>, @@ -4289,6 +4371,11 @@ impl ECStore { let effective_max_keys = list_versions_scan_limit(max_keys, has_version_marker); let mut opts = ListPathOptions { bucket: bucket.to_owned(), + caller: if include_version_purge { + ListObjectsGatherCaller::Lifecycle + } else { + ListObjectsGatherCaller::S3ListVersions + }, prefix: prefix.to_owned(), separator: delimiter.clone(), limit: effective_max_keys, @@ -4993,11 +5080,45 @@ where let _cancel_guard = cancel.clone().drop_guard(); let (result_tx, mut result_rx) = mpsc::channel(1); let mut workers = JoinSet::new(); - workers.spawn(async move { producer.await.map(|()| None) }.instrument(tracing::Span::current())); - let gather_cancel = cancel.clone(); - workers.spawn( - async move { gather_results(gather_cancel, opts, recv, result_tx).await.map(Some) }.instrument(tracing::Span::current()), + #[cfg(feature = "diagnostic-read-version-caller")] + let mut diagnostic_context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ListObjects, ); + #[cfg(feature = "diagnostic-read-version-caller")] + let request_caller = diagnostic_context.caller.or_else(|| { + use rustfs_io_metrics::ListObjectsGatherCaller; + use rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller; + + match opts.caller { + ListObjectsGatherCaller::S3List | ListObjectsGatherCaller::S3ListVersions => { + Some(ReadVersionRequestCaller::ListObjects) + } + ListObjectsGatherCaller::Lifecycle => Some(ReadVersionRequestCaller::Lifecycle), + ListObjectsGatherCaller::Other => Some(ReadVersionRequestCaller::Other), + ListObjectsGatherCaller::Unattributed => None, + } + }); + #[cfg(feature = "diagnostic-read-version-caller")] + { + diagnostic_context.caller = request_caller; + } + let producer = async move { producer.await.map(|()| None) }; + #[cfg(feature = "diagnostic-read-version-caller")] + let producer = crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context( + diagnostic_context, + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::ListProducer, + producer, + ); + workers.spawn(producer.instrument(tracing::Span::current())); + let gather_cancel = cancel.clone(); + let gather = async move { gather_results(gather_cancel, opts, recv, result_tx).await.map(Some) }; + #[cfg(feature = "diagnostic-read-version-caller")] + let gather = crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context( + diagnostic_context, + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::ListGather, + gather, + ); + workers.spawn(gather.instrument(tracing::Span::current())); let mut failure = None; let mut stopping = false; @@ -5133,6 +5254,7 @@ async fn gather_results( let duration_ms = started.elapsed().as_secs_f64() * 1000.0; rustfs_io_metrics::record_list_objects_gather(ListObjectsGatherObservation { source: LIST_OBJECTS_SOURCE_WALKER, + caller: opts.caller, outcome: LIST_OBJECTS_GATHER_OUTCOME_LIMIT_REACHED, limit: opts.limit, scanned_entries, @@ -5195,6 +5317,7 @@ async fn gather_results( let duration_ms = started.elapsed().as_secs_f64() * 1000.0; rustfs_io_metrics::record_list_objects_gather(ListObjectsGatherObservation { source: LIST_OBJECTS_SOURCE_WALKER, + caller: opts.caller, outcome: LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED, limit: opts.limit, scanned_entries, @@ -5326,6 +5449,23 @@ async fn send_or_cancel(rx: &CancellationToken, out_channel: &Sender, validation: &'static str) -> Error { + const EVENT_LISTING_MERGE_REJECTED: &str = "listing_merge_rejected"; + const LOG_COMPONENT_ECSTORE: &str = "ecstore"; + const LOG_SUBSYSTEM_LISTING: &str = "listing"; + error!( + event = EVENT_LISTING_MERGE_REJECTED, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_LISTING, + result = "rejected", + object, + version_id = ?version_id, + validation, + "Listing version merge rejected metadata" + ); + Error::FileCorrupt +} + /// Each input has already been resolved inside its own erasure set. This is a /// union of version histories, never a quorum vote between unrelated pools. fn merge_object_entry_versions(first: &mut MetaCacheEntry, others: impl Iterator) -> Result<()> { @@ -5334,23 +5474,26 @@ fn merge_object_entry_versions(first: &mut MetaCacheEntry, others: impl Iterator for mut entry in std::iter::once(std::mem::take(first)).chain(others) { let meta = match entry.cached.take() { Some(meta) => meta, - None => FileMeta::load(&entry.metadata).map_err(|_| Error::FileCorrupt)?, + None => FileMeta::load(&entry.metadata).map_err(|_| listing_merge_corruption(&name, None, "metadata_decode"))?, }; if meta.versions.is_empty() { - return Err(Error::FileCorrupt); + return Err(listing_merge_corruption(&name, None, "empty_version_stack")); } for version in meta.versions { - let parsed = version.parse_version_meta().map_err(|_| Error::FileCorrupt)?; + let corrupt = |validation| listing_merge_corruption(&name, version.header.version_id, validation); + let parsed = version.parse_version_meta().map_err(|_| corrupt("version_decode"))?; if !parsed.valid() || parsed.version_type != version.header.version_type { - return Err(Error::FileCorrupt); + return Err(corrupt("version_type_or_validity")); } - let fi = parsed.into_fileinfo("", &name, true).map_err(|_| Error::FileCorrupt)?; + let fi = parsed + .into_fileinfo("", &name, true) + .map_err(|_| corrupt("file_info_decode"))?; let version_id = fi.version_id.filter(|id| !id.is_nil()); if version_id != version.header.version_id.filter(|id| !id.is_nil()) || fi.mod_time != version.header.mod_time || fi.tier_free_version() != version.header.free_version() { - return Err(Error::FileCorrupt); + return Err(corrupt("version_header_mismatch")); } let info = ObjectInfo::from_file_info(&fi, "", &name, true); let identity = (version_id, version.header.free_version()); @@ -5374,7 +5517,7 @@ fn merge_object_entry_versions(first: &mut MetaCacheEntry, others: impl Iterator crate::data_movement::is_equivalent_data_movement_object_identity(&info, previous_info, true, true) }; if !equivalent { - return Err(Error::FileCorrupt); + return Err(corrupt("version_identity_conflict")); } // Equivalent migrated copies can have different coding or // data directories. Choose a stable representation without @@ -7229,6 +7372,9 @@ fn calc_common_counter(infos: &[DiskInfo], read_quorum: usize) -> u64 { #[cfg(test)] mod differential_tests; +#[cfg(test)] +mod lifecycle_exact_tests; + #[cfg(test)] mod test { use super::{ @@ -7458,7 +7604,7 @@ mod test { } } - fn test_pending_version_purge_meta_entry(bucket: &str, name: &str) -> (MetaCacheEntry, Uuid) { + pub(super) fn test_pending_version_purge_meta_entry(bucket: &str, name: &str) -> (MetaCacheEntry, Uuid) { let version_id = Uuid::from_u128(0x1234_5678_9abc_def0_1234_5678_9abc_def0); let data_dir = Uuid::from_u128(0xabcd_ef12_3456_7890_abcd_ef12_3456_7890); let mod_time = time::OffsetDateTime::from_unix_timestamp(1_705_312_300).expect("valid timestamp"); @@ -9221,7 +9367,7 @@ mod test { }; let snapshot = PersistentListMetadataObject::from_object_info(&object); assert!(snapshot.to_object_info("bucket").etag.is_none()); - let mut old_snapshot = snapshot.clone(); + let mut old_snapshot = snapshot; old_snapshot.etag = object.etag.clone(); let old_contents = format!( "# rustfs-listobjects-key-only-v2\n{}\n", @@ -12117,6 +12263,82 @@ mod test { assert_eq!(versions.versions[0].metadata.get("etag").map(String::as_str), Some("same-etag")); } + #[tokio::test] + async fn merge_entry_channels_data_movement_accepts_inline_layout_changes() { + let time = time::OffsetDateTime::from_unix_timestamp(1_705_312_300).expect("valid timestamp"); + let external = test_object_meta_entry_with_erasure_versions("key", &[(time, "same-etag", 2, 1)]); + let inline = rewrite_test_version( + test_object_meta_entry_with_erasure_versions("key", &[(time, "same-etag", 5, 4)]), + |version| { + rustfs_utils::http::insert_bytes( + &mut version.object.as_mut().expect("object version").meta_sys, + rustfs_utils::http::SUFFIX_INLINE_DATA, + b"true".to_vec(), + ); + }, + ); + let legacy_inline = rewrite_test_version(inline.clone(), |version| { + version + .object + .as_mut() + .expect("object version") + .meta_sys + .remove(&rustfs_utils::http::internal_key_rustfs(rustfs_utils::http::SUFFIX_INLINE_DATA)); + }); + let copies = [external, legacy_inline, inline]; + let mut expected = None; + for order in [[0, 1, 2], [0, 2, 1], [1, 0, 2], [1, 2, 0], [2, 0, 1], [2, 1, 0]] { + let merged = merge_test_object_entries(order.map(|index| copies[index].clone()).to_vec()) + .await + .expect("physical inline placement and legacy aliases must not change object identity"); + assert_eq!( + merged + .file_info_versions("bucket") + .expect("decode merged history") + .versions + .len(), + 1 + ); + if let Some(expected) = &expected { + assert_eq!(&merged.metadata, expected, "representation must not depend on pool order"); + } else { + expected = Some(merged.metadata); + } + } + } + + #[test] + fn data_movement_listing_conflict_logs_identity_without_metadata_values() { + let logs = CapturedLogs::default(); + let subscriber = tracing_subscriber::fmt() + .with_writer(logs.clone()) + .with_ansi(false) + .without_time() + .finish(); + let _guard = tracing::subscriber::set_default(subscriber); + let time = time::OffsetDateTime::from_unix_timestamp(1_705_312_300).expect("valid timestamp"); + let mut original = test_object_meta_entry_with_erasure_versions("failing-key", &[(time, "same-etag", 2, 1)]); + let conflicting = rewrite_test_version(original.clone(), |version| { + version + .object + .as_mut() + .expect("object version") + .meta_user + .insert("x-amz-meta-private".to_string(), "sensitive-value-must-not-be-logged".to_string()); + }); + assert_eq!( + super::merge_object_entry_versions(&mut original, std::iter::once(conflicting)) + .expect_err("conflicting user metadata must still fail closed"), + StorageError::FileCorrupt + ); + let output = logs.contents(); + assert!(output.contains("listing_merge_rejected")); + assert!(output.contains("failing-key")); + assert!(output.contains("00000000-0000-0000-0000-000000000001")); + assert!(output.contains("version_identity_conflict")); + assert!(!output.contains("sensitive-value-must-not-be-logged")); + } + #[tokio::test] async fn merge_entry_channels_defers_free_version_while_same_remote_source_is_live() { let live = test_transitioned_meta_entry("key", "remote/shared", false); @@ -12294,6 +12516,44 @@ mod test { ); } + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn list_path_producer_worker_preserves_s3_and_caller_context() { + let mut request = rustfs_io_metrics::s3_http_metrics::S3HttpRequestGuard::new("GET"); + let api_operation = request + .in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::ListObjectsV2); + rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_context() + }) + .expect("dispatched list operation should be captured"); + request.response(200); + + let (sender, recv) = mpsc::channel(1); + let opts = ListPathOptions { + caller: rustfs_io_metrics::ListObjectsGatherCaller::S3List, + ..Default::default() + }; + let result = rustfs_io_metrics::s3_http_metrics::with_s3_http_operation_context(Some(api_operation), async move { + collect_list_path_results(CancellationToken::new(), opts, recv, async move { + assert_eq!( + rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(), + Some("s3:ListObjectsV2") + ); + assert_eq!( + crate::cluster::rpc::runtime_sources::current_read_version_request_caller(), + Some(rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::ListObjects) + ); + drop(sender); + Ok(()) + }) + .await + }) + .await; + + assert!(result.err.is_some(), "the empty page channel remains an error"); + assert_eq!(rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(), None); + } + #[tokio::test] async fn list_path_workers_preserve_error_after_input_closes() { let (sender, recv) = mpsc::channel(1); diff --git a/crates/ecstore/src/store/list_objects/lifecycle_exact_tests.rs b/crates/ecstore/src/store/list_objects/lifecycle_exact_tests.rs new file mode 100644 index 000000000..91cf0c0a5 --- /dev/null +++ b/crates/ecstore/src/store/list_objects/lifecycle_exact_tests.rs @@ -0,0 +1,498 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use crate::bucket::lifecycle::core::IlmAction; +use crate::bucket::lifecycle::evaluator::Evaluator; +use crate::bucket::metadata::{BUCKET_LIFECYCLE_CONFIG, BUCKET_VERSIONING_CONFIG}; +use crate::bucket::metadata_sys; +use crate::disk::lifecycle_test_io::Observer; +use crate::layout::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints}; +use crate::object_api::PutObjReader; +use crate::runtime::instance::InstanceContext; +use crate::storage_api_contracts::bucket::{BucketOperations, MakeBucketOptions}; +use crate::storage_api_contracts::multipart::{CompletePart, MultipartOperations}; +use crate::storage_api_contracts::object::ObjectIO; +use rustfs_filemeta::{FileInfo, ObjectPartInfo}; +use tempfile::TempDir; +use time::OffsetDateTime; + +async fn fixture(pool_count: usize) -> (Vec, Arc, String, Uuid) { + let mut dirs = Vec::new(); + let mut pools = Vec::new(); + for pool_idx in 0..pool_count { + let mut endpoints = Vec::new(); + for disk_idx in 0..4 { + let dir = tempfile::tempdir().expect("disk directory"); + let mut endpoint = + crate::disk::endpoint::Endpoint::try_from(dir.path().to_str().expect("disk path")).expect("disk endpoint"); + endpoint.set_pool_index(pool_idx); + endpoint.set_set_index(0); + endpoint.set_disk_index(disk_idx); + dirs.push(dir); + endpoints.push(endpoint); + } + pools.push(PoolEndpoints { + legacy: false, + set_count: 1, + drives_per_set: 4, + endpoints: Endpoints::from(endpoints), + cmd_line: format!("lifecycle-exact-pool-{pool_idx}"), + platform: "test".into(), + }); + } + let endpoints = EndpointServerPools(pools); + let ctx = Arc::new(InstanceContext::new()); + crate::store::init_local_disks_with_instance_ctx(&ctx, endpoints.clone()) + .await + .expect("initialize disks"); + let store = + ECStore::new_with_instance_ctx("127.0.0.1:0".parse().expect("test address"), endpoints, CancellationToken::new(), ctx) + .await + .expect("initialize store"); + metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let bucket = format!("lifecycle-exact-{}", Uuid::new_v4().simple()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let incarnation = metadata_sys::get_bucket_incarnation_id_in(&store.ctx, &bucket) + .await + .expect("bucket incarnation"); + (dirs, store, bucket, incarnation) +} + +fn version(object: &str, id: Option, timestamp: i64) -> FileInfo { + let mut fi = FileInfo::new(object, 2, 2); + fi.name = object.to_owned(); + fi.version_id = id; + fi.versioned = id.is_some(); + fi.data_dir = Some(Uuid::from_u128(0x1234)); + fi.mod_time = Some(OffsetDateTime::from_unix_timestamp(timestamp).expect("version timestamp")); + fi.size = 1; + fi.parts = vec![ObjectPartInfo { + number: 1, + size: 1, + actual_size: 1, + ..Default::default() + }]; + fi.metadata.insert("etag".into(), timestamp.to_string()); + fi +} + +async fn seed(dirs: &[TempDir], bucket: &str, object: &str, versions: &[FileInfo]) { + for (index, dir) in dirs.iter().enumerate() { + let mut meta = FileMeta::new(); + for version in versions { + let mut fi = version.clone(); + fi.volume = bucket.to_owned(); + if !fi.is_canonical_delete_marker() { + fi.erasure.index = index % 4 + 1; + } + fi.validate_for_metadata_read().expect("valid fixture version"); + meta.add_version(fi).expect("encode version"); + } + let path = dir.path().join(bucket).join(path::encode_dir_object(object)); + tokio::fs::create_dir_all(&path).await.expect("object directory"); + tokio::fs::write(path.join(crate::disk::STORAGE_FORMAT_FILE), meta.marshal_msg().expect("encode metadata")) + .await + .expect("persist fixture metadata"); + } +} + +async fn configure_versioning(store: &ECStore, bucket: &str, state: &str) { + if state == "Unversioned" { + return; + } + metadata_sys::update_in( + &store.ctx, + bucket, + BUCKET_VERSIONING_CONFIG, + format!("{state}").into_bytes(), + ) + .await + .expect("persist versioning configuration"); +} + +#[tokio::test] +async fn lifecycle_exact_matches_listing_for_complete_version_groups() { + let (dirs, store, bucket, incarnation) = fixture(1).await; + for state in ["Unversioned", "Enabled", "Suspended"] { + configure_versioning(&store, &bucket, state).await; + for object in ["abc", "nested/abc", "directory/"] { + let mut versions = vec![version(object, None, 1_700_000_000)]; + if state != "Unversioned" { + versions.push(version(object, Some(Uuid::from_u128(1)), 1_700_000_001)); + versions.push(FileInfo { + name: object.into(), + version_id: Some(Uuid::from_u128(2)), + deleted: true, + mod_time: Some(OffsetDateTime::from_unix_timestamp(1_700_000_002).expect("marker timestamp")), + ..Default::default() + }); + } + seed(&dirs, &bucket, object, &versions).await; + let sibling = format!("{object}-sibling"); + seed(&dirs, &bucket, &sibling, &[version(&sibling, None, 1_700_000_003)]).await; + let observed = Observer::new(&bucket); + let exact = store + .get_object_versions_for_lifecycle(&bucket, object, &ObjectOptions::default(), incarnation) + .await + .unwrap_or_else(|error| panic!("read {state} versions for {object}: {error}")); + let counts = observed.snapshot(); + assert_eq!(counts.walks, 0, "exact metadata reads must never enumerate a directory"); + assert_eq!(counts.reads.len(), 4, "one exact metadata read per physical drive"); + assert!(counts.reads.iter().all(|key| key == &path::encode_dir_object(object))); + let mut listed = store + .clone() + .list_object_versions_for_lifecycle(&bucket, object, None, None, None, 1000) + .await + .expect("reference lifecycle listing") + .objects + .into_iter() + .filter(|info| info.name == object) + .collect::>(); + // The legacy list projection uses ambient bucket versioning. This + // fixture has an isolated store; restore its known null-version + // representation before comparing the version-group semantics. + if state != "Unversioned" { + for info in &mut listed { + if info.version_id.is_none() { + info.version_id = Some(Uuid::nil()); + } + } + assert!(exact.iter().any(|info| info.version_id == Some(Uuid::nil()))); + } + assert!(observed.snapshot().walks > 0, "the old prefix-listing path must exercise the observer"); + assert_eq!(exact.len(), versions.len(), "{state}: {object}"); + assert_eq!(exact.len(), listed.len()); + for (exact, listed) in exact.iter().zip(&listed) { + assert_eq!( + ( + exact.version_id, + exact.is_latest, + exact.delete_marker, + exact.num_versions, + exact.successor_mod_time + ), + ( + listed.version_id, + listed.is_latest, + listed.delete_marker, + listed.num_versions, + listed.successor_mod_time + ) + ); + assert_eq!(exact.user_defined, listed.user_defined); + assert_eq!(exact.version_purge_status, listed.version_purge_status); + } + let policy = crate::bucket::utils::deserialize( + br#"historyEnabled3011"#, + ).expect("reference policy"); + let evaluator = Evaluator::new(Arc::new(policy)); + let to_opts = |objects: &[ObjectInfo]| { + objects + .iter() + .map(crate::bucket::lifecycle::core::object_opts_from_object_info) + .collect::>() + }; + let exact_events = evaluator.eval(&to_opts(&exact)).await.expect("evaluate exact group"); + let listed_events = evaluator.eval(&to_opts(&listed)).await.expect("evaluate listed group"); + for (exact, listed) in exact_events.iter().zip(listed_events) { + assert_eq!((exact.action, exact.due, &exact.rule_id), (listed.action, listed.due, &listed.rule_id)); + } + } + } +} + +#[tokio::test] +async fn lifecycle_exact_merges_pools_and_more_than_one_page_of_versions() { + let (dirs, store, bucket, incarnation) = fixture(2).await; + configure_versioning(&store, &bucket, "Enabled").await; + let object = "history"; + let versions = (0..1002) + .map(|index| { + version( + object, + Some(Uuid::from_u128(index + 1)), + 1_700_000_000 + i64::try_from(index).expect("timestamp index"), + ) + }) + .collect::>(); + seed(&dirs[..4], &bucket, object, &versions[..601]).await; + seed(&dirs[4..], &bucket, object, &versions[600..]).await; + let observed = Observer::new(&bucket); + let exact = store + .get_object_versions_for_lifecycle(&bucket, object, &ObjectOptions::default(), incarnation) + .await + .expect("merge independently authoritative pools"); + assert_eq!(exact.len(), 1002, "overlapping migration copies must be deduplicated without pagination"); + assert_eq!(exact[0].version_id, Some(Uuid::from_u128(1002))); + assert!(exact[0].is_latest); + assert!(exact.iter().all(|info| info.num_versions == 1002)); + for pair in exact.windows(2) { + assert!(!pair[1].is_latest); + assert_eq!(pair[1].successor_mod_time, pair[0].mod_time); + } + assert_eq!(observed.snapshot().walks, 0); + assert_eq!(observed.snapshot().reads.len(), 8, "read count depends on drives, not version pages"); + let mut conflict = versions[600].clone(); + conflict.metadata.insert("etag".into(), "conflicting-content".into()); + seed(&dirs[4..], &bucket, object, &[conflict]).await; + assert!( + store + .get_object_versions_for_lifecycle(&bucket, object, &ObjectOptions::default(), incarnation) + .await + .is_err(), + "conflicting cross-pool copies cannot authorize expiry" + ); +} + +#[tokio::test] +async fn lifecycle_exact_rejects_uncertain_metadata_and_stale_bucket_identity() { + let (dirs, store, bucket, incarnation) = fixture(1).await; + let object = "probe"; + let opts = ObjectOptions::default(); + assert!( + store + .get_object_versions_for_lifecycle(&bucket, object, &opts, incarnation) + .await + .expect("confirmed absence") + .is_empty() + ); + seed(&dirs, &bucket, object, &[version(object, None, 1_700_000_000)]).await; + assert!( + store + .get_object_versions_for_lifecycle(&bucket, object, &opts, Uuid::new_v4()) + .await + .is_err() + ); + for dir in &dirs { + tokio::fs::write( + dir.path() + .join(bucket.as_str()) + .join(object) + .join(crate::disk::STORAGE_FORMAT_FILE), + b"corrupt", + ) + .await + .expect("corrupt fixture metadata"); + } + assert!( + store + .get_object_versions_for_lifecycle(&bucket, object, &opts, incarnation) + .await + .is_err(), + "corruption is not absence" + ); + seed(&dirs, &bucket, object, &[version(object, None, 1_700_000_000)]).await; + let set = &store.pools[0].disk_set[0]; + let saved = set.disks.read().await.clone(); + *set.disks.write().await = vec![saved[0].clone(), saved[1].clone(), None, None]; + assert_eq!( + store + .get_object_versions_for_lifecycle(&bucket, object, &opts, incarnation) + .await + .expect("exactly the object's read quorum") + .len(), + 1 + ); + *set.disks.write().await = vec![saved[0].clone(), None, None, None]; + assert!( + store + .get_object_versions_for_lifecycle(&bucket, object, &opts, incarnation) + .await + .is_err(), + "a minority cannot authorize expiry" + ); + assert!( + store + .get_object_versions_for_lifecycle(&bucket, "missing", &opts, incarnation) + .await + .is_err(), + "missing on one drive does not prove absence" + ); + *set.disks.write().await = saved; + let lost = ObjectOptions { + no_lock: true, + namespace_lock_fence: Some(crate::object_api::NamespaceLockFence::lost_for_test()), + ..Default::default() + }; + assert!( + store + .get_object_versions_for_lifecycle(&bucket, object, &lost, incarnation) + .await + .is_err(), + "an inherited lost object fence must reject the snapshot" + ); +} + +#[tokio::test] +async fn lifecycle_exact_write_hooks_never_walk_parent_directories() { + let (dirs, store, bucket, _) = fixture(1).await; + metadata_sys::update_in(&store.ctx, &bucket, BUCKET_LIFECYCLE_CONFIG, + br#"expireEnabled30"#.to_vec()) + .await.expect("configure expiration"); + for index in 0..64 { + let key = format!("probe-{index}"); + seed(&dirs, &bucket, &key, &[version(&key, None, 1_700_000_000)]).await; + } + for state in ["Unversioned", "Enabled", "Suspended"] { + configure_versioning(&store, &bucket, state).await; + let opts = ObjectOptions { + versioned: state == "Enabled", + version_suspended: state == "Suspended", + ..Default::default() + }; + let observer = Observer::new(&bucket); + let mut source = store + .put_object(&bucket, "probe", &mut PutObjReader::from_vec(b"payload".to_vec()), &opts) + .await + .expect("PUT with lifecycle"); + source.put_object_reader = Some(PutObjReader::from_vec(b"payload".to_vec())); + let copied = store + .copy_object(&bucket, "probe", &bucket, "copy", &mut source, &ObjectOptions::default(), &opts) + .await + .expect("COPY with lifecycle"); + assert_eq!(copied.size, source.size); + let upload = store + .new_multipart_upload(&bucket, "multipart", &opts) + .await + .expect("init multipart"); + let part = store + .put_object_part( + &bucket, + "multipart", + &upload.upload_id, + 1, + &mut PutObjReader::from_vec(b"part".to_vec()), + &opts, + ) + .await + .expect("upload part"); + let complete = store + .clone() + .complete_multipart_upload( + &bucket, + "multipart", + &upload.upload_id, + vec![CompletePart { + part_num: 1, + etag: part.etag, + checksum_crc32: None, + checksum_crc32c: None, + checksum_sha1: None, + checksum_sha256: None, + checksum_crc64nvme: None, + }], + &opts, + ) + .await + .expect("complete multipart with lifecycle"); + assert_eq!(complete.size, 4); + let counts = observer.snapshot(); + assert_eq!(counts.walks, 0, "{state}: successful writes must not run the version walker"); + for key in ["probe", "copy", "multipart"] { + assert!( + counts.reads.iter().filter(|read| read.as_str() == key).count() >= 4, + "{state}: {key} must run exact expiry evaluation" + ); + } + assert!( + !counts.reads.iter().any(|key| key.starts_with("probe-")), + "prefix siblings must never enter evaluation" + ); + } +} + +#[tokio::test] +async fn lifecycle_exact_keeps_pending_purges_visible_to_expiry_guards() { + let (dirs, store, bucket, incarnation) = fixture(1).await; + configure_versioning(&store, &bucket, "Enabled").await; + let object = "pending"; + let (entry, _) = super::test::test_pending_version_purge_meta_entry(&bucket, object); + for dir in &dirs { + let path = dir.path().join(&bucket).join(object); + tokio::fs::create_dir_all(&path).await.expect("object directory"); + tokio::fs::write(path.join(crate::disk::STORAGE_FORMAT_FILE), &entry.metadata) + .await + .expect("persist pending purge"); + } + let exact = store + .get_object_versions_for_lifecycle(&bucket, object, &ObjectOptions::default(), incarnation) + .await + .expect("read pending purge"); + assert_eq!(exact.len(), 1); + assert_eq!(exact[0].num_versions, 1); + assert_eq!(exact[0].version_purge_status, crate::bucket::replication::VersionPurgeStatusType::Pending); + let policy = crate::bucket::utils::deserialize( + br#"expireEnabled1"#, + ).expect("expiration policy"); + let opts = crate::bucket::lifecycle::core::object_opts_from_object_info(&exact[0]); + let events = Evaluator::new(Arc::new(policy)) + .eval(&[opts]) + .await + .expect("evaluate pending purge"); + assert_eq!(events[0].action, IlmAction::NoneAction); +} + +#[tokio::test] +async fn lifecycle_exact_io_is_independent_of_parent_directory_width() { + let (dirs, store, bucket, incarnation) = fixture(1).await; + let object = "probe"; + seed(&dirs, &bucket, object, &[version(object, None, 1_700_000_000)]).await; + let mut previous = 0; + for width in [64, 1024, 8192] { + let roots = dirs.iter().map(|dir| dir.path().join(&bucket)).collect::>(); + tokio::task::spawn_blocking(move || { + for root in roots { + for index in previous..width { + std::fs::create_dir(root.join(format!("sibling-{index:05}"))).expect("parent-directory entry"); + } + } + }) + .await + .expect("create wide directory"); + previous = width; + let mut exact_times = Vec::new(); + let mut listing_times = Vec::new(); + for _ in 0..5 { + let observer = Observer::new(&bucket); + let started = std::time::Instant::now(); + let exact = store + .get_object_versions_for_lifecycle(&bucket, object, &ObjectOptions::default(), incarnation) + .await + .expect("exact lifecycle read"); + exact_times.push(started.elapsed().as_micros()); + assert_eq!(exact.len(), 1); + assert_eq!(observer.snapshot().walks, 0); + assert_eq!(observer.snapshot().reads.len(), 4, "directory width {width}"); + let started = std::time::Instant::now(); + let listed = store + .clone() + .list_object_versions_for_lifecycle(&bucket, object, None, None, None, 1000) + .await + .expect("reference prefix scan"); + listing_times.push(started.elapsed().as_micros()); + assert_eq!(listed.objects.len(), 1); + assert!(observer.snapshot().walks > 0); + } + exact_times.sort_unstable(); + listing_times.sort_unstable(); + eprintln!( + "lifecycle directory width={width}: exact median={}us max={}us; prefix median={}us max={}us", + exact_times[2], exact_times[4], listing_times[2], listing_times[4] + ); + } +} diff --git a/crates/ecstore/src/store/mod.rs b/crates/ecstore/src/store/mod.rs index 8408fb750..3ecfbd726 100644 --- a/crates/ecstore/src/store/mod.rs +++ b/crates/ecstore/src/store/mod.rs @@ -27,6 +27,7 @@ use crate::bucket::utils::check_get_obj_args; use crate::bucket::utils::check_list_multipart_args; use crate::bucket::utils::check_list_parts_args; use crate::bucket::utils::check_new_multipart_args; +use crate::bucket::utils::check_new_object_args; use crate::bucket::utils::check_object_args; use crate::bucket::utils::check_put_object_args; use crate::bucket::utils::check_put_object_part_args; @@ -428,6 +429,7 @@ mod bucket; mod bucket_fence; pub(crate) use bucket::await_bucket_namespace_operation; pub use bucket_fence::BucketIncarnationFenceGuard; +pub(crate) mod delete_recovery_runtime; mod heal; pub(crate) use heal::{BucketHealScope, bucket_heal_scope, bucket_heal_scope_for_object}; pub use heal::{HealObjectAbsenceProof, HealObjectStorageResult}; @@ -438,7 +440,7 @@ pub(crate) mod init_format; pub(crate) mod list_objects; mod multipart; mod object; -#[cfg(feature = "test-util")] +#[cfg(any(test, feature = "test-util"))] pub use object::DeleteAfterObjectLockSnapshotBarrier; pub(crate) use object::{ DecommissionFixedReadAnchor, ObjectLockDiagGuard, RemoteTuplePublicationCommitGuard, RemoteTuplePublicationFence, @@ -1363,7 +1365,13 @@ impl crate::storage_api_contracts::object::ObjectIO for ECStore { h: HeaderMap, opts: &ObjectOptions, ) -> Result { - self.handle_get_object_reader(bucket, object, range, h, opts).await + let read = self.handle_get_object_reader(bucket, object, range, h, opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::StoreObjectReader, + read, + ); + read.await } #[instrument(level = "debug", skip(self, data))] async fn put_object(&self, bucket: &str, object: &str, data: &mut PutObjReader, opts: &ObjectOptions) -> Result { diff --git a/crates/ecstore/src/store/multipart.rs b/crates/ecstore/src/store/multipart.rs index 2a887356b..73f0427aa 100644 --- a/crates/ecstore/src/store/multipart.rs +++ b/crates/ecstore/src/store/multipart.rs @@ -32,6 +32,42 @@ fn data_movement_multipart_discovery_counts() -> &'static std::sync::Mutex, tokio::sync::oneshot::Receiver<()>); + +#[cfg(test)] +static ORPHAN_MULTIPART_CLEANUP_PAUSES: std::sync::OnceLock< + std::sync::Mutex>, +> = std::sync::OnceLock::new(); + +#[cfg(test)] +pub(crate) fn pause_orphan_multipart_cleanup_for_test( + bucket: &str, +) -> (tokio::sync::oneshot::Receiver<()>, tokio::sync::oneshot::Sender<()>) { + let (arrived_tx, arrived_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = tokio::sync::oneshot::channel(); + let previous = ORPHAN_MULTIPART_CLEANUP_PAUSES + .get_or_init(Default::default) + .lock() + .expect("orphan cleanup pause mutex should not poison") + .insert(bucket.to_owned(), (arrived_tx, release_rx)); + assert!(previous.is_none(), "each bucket has at most one orphan cleanup pause"); + (arrived_rx, release_tx) +} + +#[cfg(test)] +async fn pause_orphan_multipart_cleanup(bucket: &str) { + let pause = ORPHAN_MULTIPART_CLEANUP_PAUSES + .get_or_init(Default::default) + .lock() + .expect("orphan cleanup pause mutex should not poison") + .remove(bucket); + if let Some((arrived, release)) = pause { + let _ = arrived.send(()); + let _ = release.await; + } +} + fn decommission_multipart_target_clear_pending(opts: &ObjectOptions, target: Option<&ObjectInfo>) -> Result { let expected_mod_time = opts.mod_time.ok_or_else(|| Error::DecommissionCapacityBlocked { message: "multipart cleanup cannot prove exact target absence without a modification time".to_string(), @@ -520,7 +556,7 @@ impl ECStore { opts: &ObjectOptions, mutation_fence: Option<&ObjectLockDiagGuard>, ) -> Result<(MultipartUploadResult, usize, Option)> { - check_new_multipart_args(bucket, object)?; + check_new_object_args(bucket, object, opts.data_movement)?; let (mut opts, _bucket_lifecycle_guard) = self.prepare_unsupported_artifact_write(bucket, opts).await?; if self.single_pool() { @@ -529,6 +565,12 @@ impl ECStore { let result = self.pools[0].new_multipart_upload(bucket, object, &opts).await?; return Ok((result, 0, opts.expected_bucket_incarnation_id)); } + if opts.http_preconditions.is_some() + && !opts.data_movement + && let Some(err) = self.check_write_precondition(bucket, object, &opts).await + { + return Err(err); + } let capacity_owner = DecommissionCapacityOwner::from_options(&opts); if opts.data_movement && (opts.version_id.is_some() || capacity_owner.is_some()) { @@ -750,19 +792,12 @@ impl ECStore { opts: &ObjectOptions, ) -> Result<()> { check_abort_multipart_args(bucket, object, upload_id)?; - let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?; - let opts = &opts; - - // TODO(backlog): defer DeleteUploadID to background for faster abort response - - if self.single_pool() { - return self.pools[0].abort_multipart_upload(bucket, object, upload_id, opts).await; - } + let (opts, bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?; for pool_idx in self.existing_multipart_pool_order().await { let pool = &self.pools[pool_idx]; - let err = match pool.abort_multipart_upload(bucket, object, upload_id, opts).await { + let err = match pool.abort_multipart_upload(bucket, object, upload_id, &opts).await { Ok(_) => return Ok(()), Err(err) => { // @@ -775,7 +810,92 @@ impl ECStore { } } - Err(StorageError::InvalidUploadID(bucket.to_owned(), object.to_owned(), upload_id.to_owned())) + self.recover_orphan_multipart_upload(bucket, object, upload_id, opts, bucket_lifecycle_guard) + .await + } + + async fn recover_orphan_multipart_upload( + &self, + bucket: &str, + object: &str, + upload_id: &str, + opts: ObjectOptions, + bucket_lifecycle_guard: Option, + ) -> Result<()> { + let invalid_upload = || StorageError::InvalidUploadID(bucket.to_owned(), object.to_owned(), upload_id.to_owned()); + if self.ctx.lock_manager().is_disabled() { + return Err(invalid_upload()); + } + let Some(bucket_lifecycle_guard) = bucket_lifecycle_guard else { + // A signal borrowed from an enclosing operation cannot keep its lock + // alive when the request waiter disappears during physical cleanup. + return Err(invalid_upload()); + }; + let Some(expected_incarnation) = opts.expected_bucket_incarnation_id.filter(|id| !id.is_nil()) else { + return Err(invalid_upload()); + }; + // Recovery is a side path of an idempotent Abort. Until the first + // deletion is issued, anything that stops staging absence from being + // proven (an offline disk, lock contention, partially overlapping lock + // domains, unreadable bucket metadata) keeps the upload reported as + // gone, exactly as before recovery existed; only failures inside the + // deletion task are request failures, because a retry can finish them. + let (metadata, persisted) = + match crate::bucket::metadata_sys::get_config_from_disk_with_presence_in(&self.ctx, bucket).await { + Ok(loaded) => loaded, + Err(error) => { + debug!(bucket, object, upload_id, %error, "orphan multipart cleanup skipped: bucket metadata unavailable"); + return Err(invalid_upload()); + } + }; + if !persisted + || !metadata.bucket_creation_committed + || !metadata.bucket_incarnation_sidecar + || metadata.bucket_incarnation_id != expected_incarnation + { + return Err(invalid_upload()); + } + + // Lock order: bucket lifecycle, then upload locks in pool-index order. + // Prove metadata absence across every pool before deleting any staging. + let mut cleanup_guards = Vec::with_capacity(self.pools.len()); + for pool in &self.pools { + let guard = match pool + .get_disks_by_key(object) + .lock_orphan_multipart_cleanup(bucket, object, upload_id, &opts, &cleanup_guards) + .await + { + Ok(guard) => guard, + Err(error) => { + if !is_err_invalid_upload_id(&error) { + debug!(bucket, object, upload_id, %error, "orphan multipart cleanup skipped: staging absence not proven"); + } + return Err(invalid_upload()); + } + }; + cleanup_guards.push(guard); + } + let bucket = bucket.to_owned(); + let object = object.to_owned(); + let upload_id = upload_id.to_owned(); + tokio::spawn(async move { + // Own both the real bucket guard and all upload guards until every + // disk deletion settles, including after cancellation of the waiter. + let _bucket_lifecycle_guard = bucket_lifecycle_guard; + #[cfg(test)] + pause_orphan_multipart_cleanup(&bucket).await; + let mut recovered = false; + for guard in &cleanup_guards { + recovered |= guard.delete(&bucket, &object, &opts).await?; + } + if recovered { + Ok(()) + } else { + Err(StorageError::InvalidUploadID(bucket, object, upload_id)) + } + }) + .await + .map_err(|error| Error::other_with_context("orphan multipart cleanup task failed", error))? } pub(crate) async fn abort_multipart_upload_for_data_movement( @@ -970,6 +1090,15 @@ impl ECStore { Ok(()) } + async fn check_write_precondition(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Option { + let mut lookup_opts = opts.clone(); + let http_preconditions = lookup_opts.http_preconditions?; + lookup_opts.http_preconditions = None; + lookup_opts.no_lock = true; + let oi = self.get_object_info_for_delete(bucket, object, &lookup_opts).await; + crate::set_disk::SetDisks::evaluate_write_precondition(oi, &http_preconditions, bucket, object, opts) + } + #[instrument(skip(self))] pub(super) async fn handle_complete_multipart_upload( self: Arc, @@ -992,6 +1121,12 @@ impl ECStore { .complete_multipart_upload(bucket, object, upload_id, uploaded_parts, opts) .await; } + if opts.http_preconditions.is_some() + && !opts.data_movement + && let Some(err) = self.check_write_precondition(bucket, object, opts).await + { + return Err(err); + } match self.multipart_upload_pool_idx(bucket, object, upload_id, opts).await { Ok(pool_idx) => { diff --git a/crates/ecstore/src/store/object.rs b/crates/ecstore/src/store/object.rs index 2012d43c5..fa54bba80 100644 --- a/crates/ecstore/src/store/object.rs +++ b/crates/ecstore/src/store/object.rs @@ -12,6 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +use super::rebalance::support::{LatestObjectInfoCandidate, resolve_latest_object_info_candidates_with_pool_state}; use super::*; use crate::bucket::lifecycle::{ bucket_lifecycle_ops::{enqueue_committed_free_versions, eval_action_from_lifecycle}, @@ -29,7 +30,7 @@ use crate::bucket::lifecycle::{ }, }; use crate::bucket::metadata_sys::{ - acquire_bucket_metadata_transaction_read_lock_in, get_cached_bucket_incarnation_id_in, + acquire_bucket_metadata_transaction_read_lock_in, get_cached_bucket_incarnation_id_in, get_delete_config_from_disk_in, get_object_lock_config_and_incarnation_from_disk_in, }; use crate::bucket::object_lock::objectlock_sys::{ @@ -39,7 +40,7 @@ use crate::bucket::replication::{DeleteReplicationConfigSnapshot, ReplicationObj use crate::bucket::versioning::VersioningApi; use crate::core::pools::{DecommissionCapacityAdmission, DecommissionCapacityOwner, ensure_decommission_capacity_mutation_id}; use crate::diagnostics::object_lock::ObjectLockAttempt; -use crate::disk::OldCurrentSize; +use crate::disk::{CurrentDeleteGeneration, OldCurrentSize}; use crate::object_api::{ NamespaceLockFence, ObjectLockConfigSnapshot, ScannerPublicationCommitScopeGuard, ScannerPublicationCommitState, TierFreeVersionReceiptSink, @@ -58,13 +59,14 @@ use crate::storage_api_contracts::{ namespace::NamespaceLocking as _, object::{DeleteAccounting, ObjectIO as _, ObjectOperations as _}, }; -use futures::StreamExt as _; +use futures::{StreamExt as _, TryStreamExt as _}; use parking_lot::Mutex as ParkingMutex; use rustfs_filemeta::ObjectPartInfo; use rustfs_io_metrics::{ record_object_lock_diag_acquire_duration, record_object_lock_diag_hold_duration, record_object_lock_diag_slow_acquire, record_object_lock_diag_slow_hold, }; +use std::collections::HashSet; use std::{ fmt, pin::Pin, @@ -1993,6 +1995,10 @@ fn batch_delete_creates_latest_marker(object: &ObjectToDelete, delete_config_sna versioned || version_suspended } +fn batch_delete_action_key(object: &ObjectToDelete) -> (&str, Option) { + (object.object_name.as_str(), object.version_id) +} + fn batch_delete_targets_pool(creates_latest_marker: bool, marker_target_pool_idx: Option, pool_idx: usize) -> bool { !creates_latest_marker || marker_target_pool_idx == Some(pool_idx) } @@ -2484,6 +2490,250 @@ fn sorted_unique_delete_object_names(objects: &[ObjectToDelete]) -> Vec<&str> { object_names } +#[cfg(test)] +static CONDITIONAL_DELETE_POST_ADMISSION_REBALANCE: std::sync::OnceLock>> = + std::sync::OnceLock::new(); + +#[cfg(test)] +struct ConditionalDeletePostAdmissionOverwrite { + pool_idx: usize, + object: String, + payload: Vec, + versioned: bool, +} + +#[cfg(test)] +static CONDITIONAL_DELETE_POST_ADMISSION_OVERWRITE: std::sync::OnceLock< + std::sync::Mutex>, +> = std::sync::OnceLock::new(); + +#[cfg(test)] +async fn inject_conditional_delete_post_admission_mutations(store: &ECStore, bucket: &str) { + let pool_idx = CONDITIONAL_DELETE_POST_ADMISSION_REBALANCE + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("conditional rebalance injection must not poison") + .remove(bucket); + if let Some(pool_idx) = pool_idx { + let mut pool_stats = vec![crate::services::rebalance::RebalanceStats::default(); store.pools.len()]; + pool_stats[pool_idx].participating = true; + pool_stats[pool_idx].info.status = crate::services::rebalance::RebalStatus::Started; + *store.rebalance_meta.write().await = Some(crate::services::rebalance::RebalanceMeta { + pool_stats, + ..Default::default() + }); + } + let overwrite = CONDITIONAL_DELETE_POST_ADMISSION_OVERWRITE + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("conditional overwrite injection must not poison") + .remove(bucket); + if let Some(overwrite) = overwrite { + // Model a writer in a different lock domain after admission. The + // destructive path must still compare metadata under the disk lease. + store.pools[overwrite.pool_idx] + .put_object( + bucket, + &overwrite.object, + &mut PutObjReader::from_vec(overwrite.payload), + &ObjectOptions { + no_lock: true, + versioned: overwrite.versioned, + ..Default::default() + }, + ) + .await + .expect("post-admission generation overwrite"); + } +} + +const CONDITIONAL_DELETE_SNAPSHOT_CONCURRENCY: usize = 16; +const CONDITIONAL_DELETE_CAPABILITY_SET_CONCURRENCY: usize = 16; + +#[cfg(test)] +struct ConditionalCapabilitySetProbeGateState { + started: std::sync::atomic::AtomicUsize, + active: std::sync::atomic::AtomicUsize, + max_active: std::sync::atomic::AtomicUsize, + reject_first: AtomicBool, + changed: tokio::sync::Notify, + release: tokio::sync::Semaphore, +} + +#[cfg(test)] +struct ConditionalCapabilitySetProbeGate { + bucket: String, + state: Arc, + abort_task: Option, +} + +#[cfg(test)] +static CONDITIONAL_CAPABILITY_SET_PROBE_GATES: std::sync::OnceLock< + std::sync::Mutex>>, +> = std::sync::OnceLock::new(); + +#[cfg(test)] +impl ConditionalCapabilitySetProbeGate { + fn install(bucket: &str, reject_first: bool) -> Self { + let state = Arc::new(ConditionalCapabilitySetProbeGateState { + started: std::sync::atomic::AtomicUsize::new(0), + active: std::sync::atomic::AtomicUsize::new(0), + max_active: std::sync::atomic::AtomicUsize::new(0), + reject_first: AtomicBool::new(reject_first), + changed: tokio::sync::Notify::new(), + release: tokio::sync::Semaphore::new(0), + }); + let previous = CONDITIONAL_CAPABILITY_SET_PROBE_GATES + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("conditional capability probe gates must not poison") + .insert(bucket.to_owned(), Arc::clone(&state)); + assert!(previous.is_none(), "one probe gate per test bucket"); + Self { + bucket: bucket.to_owned(), + state, + abort_task: None, + } + } + + async fn wait_started(&self, count: usize) { + tokio::time::timeout(Duration::from_secs(30), async { + loop { + let changed = self.state.changed.notified(); + if self.state.started.load(Ordering::Acquire) >= count { + break; + } + changed.await; + } + }) + .await + .expect("bounded set probes must reach the requested admission count"); + } +} + +#[cfg(test)] +impl Drop for ConditionalCapabilitySetProbeGate { + fn drop(&mut self) { + if let Some(task) = &self.abort_task { + task.abort(); + } + self.state.release.close(); + let mut gates = CONDITIONAL_CAPABILITY_SET_PROBE_GATES + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("conditional capability probe gates must not poison"); + if gates.get(&self.bucket).is_some_and(|state| Arc::ptr_eq(state, &self.state)) { + gates.remove(&self.bucket); + } + } +} + +#[cfg(test)] +struct ConditionalCapabilitySetProbeActivity(Arc); + +#[cfg(test)] +impl Drop for ConditionalCapabilitySetProbeActivity { + fn drop(&mut self) { + self.0.active.fetch_sub(1, Ordering::AcqRel); + } +} + +#[cfg(test)] +async fn await_conditional_capability_set_probe_gate(bucket: &str) -> Result<()> { + let state = CONDITIONAL_CAPABILITY_SET_PROBE_GATES + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("conditional capability probe gates must not poison") + .get(bucket) + .cloned(); + let Some(state) = state else { + return Ok(()); + }; + let active = state.active.fetch_add(1, Ordering::AcqRel) + 1; + let _activity = ConditionalCapabilitySetProbeActivity(Arc::clone(&state)); + state.max_active.fetch_max(active, Ordering::AcqRel); + state.started.fetch_add(1, Ordering::AcqRel); + state.changed.notify_one(); + state.release.acquire().await.map_err(|_| Error::OperationCanceled)?.forget(); + if state.reject_first.swap(false, Ordering::AcqRel) { + return Err(StorageError::GuardedDeleteUnsupported); + } + Ok(()) +} + +struct BatchDeletePhasePlan<'a> { + indices: &'a [usize], + pool_generations: HashMap>, + marker_targets: &'a HashMap, +} + +struct ConditionalDeleteSnapshot { + current: ObjectInfo, + current_pool_idx: usize, + pool_generations: HashMap, +} + +fn resolve_conditional_delete_snapshot( + bucket: &str, + object: &str, + results: Vec<(usize, Result<(ObjectInfo, CurrentDeleteGeneration)>)>, + suspended_pools: &[bool], + opts: &ObjectOptions, +) -> Result { + let mut candidates = Vec::with_capacity(results.len()); + let mut pool_generations = HashMap::with_capacity(results.len()); + for (idx, result) in results { + match result { + Ok((info, generation)) => { + pool_generations.insert(idx, generation); + candidates.push(LatestObjectInfoCandidate { + info: Some(info), + idx, + err: None, + }); + } + Err(error) if is_err_object_not_found(&error) || is_err_version_not_found(&error) => { + pool_generations.insert(idx, CurrentDeleteGeneration::Absent); + candidates.push(LatestObjectInfoCandidate { + info: None, + idx, + err: Some(error), + }); + } + // A readable old copy cannot establish the condition while a + // different pool might hold a newer, unreadable generation. + Err(error) => return Err(error), + } + } + let (current, current_pool_idx) = + resolve_latest_object_info_candidates_with_pool_state(candidates, suspended_pools, bucket, object, opts)?; + if current.delete_marker { + return Err(StorageError::ObjectNotFound(bucket.to_owned(), object.to_owned())); + } + Ok(ConditionalDeleteSnapshot { + current, + current_pool_idx, + pool_generations, + }) +} + +fn check_batch_delete_condition(object: &ObjectToDelete, snapshot: &ConditionalDeleteSnapshot, bucket: &str) -> Result<()> { + let Some(condition) = object.delete_condition.as_ref() else { + return Ok(()); + }; + if object.version_id.is_some() || condition.validate().is_err() { + return Err(StorageError::InvalidArgument( + bucket.to_owned(), + object.object_name.clone(), + "invalid current-object delete condition".to_owned(), + )); + } + if !condition.matches(snapshot.current.etag.as_deref().unwrap_or_default()) { + return Err(StorageError::PreconditionFailed); + } + Ok(()) +} + /// A CopyObject branch that writes a fresh local data stream owns new bytes and /// must not inherit the source version's remote-tier ownership record. FileMeta /// exposes internal transition fields through `ObjectInfo::user_defined` for @@ -2794,6 +3044,40 @@ impl RemoteTuplePublicationFence { } impl ECStore { + async fn conditional_delete_snapshot( + &self, + bucket: &str, + object: &str, + opts: &ObjectOptions, + ) -> Result { + let mut lookup_opts = opts.clone(); + lookup_opts.no_lock = true; + lookup_opts.metadata_cache_safe = false; + lookup_opts.suppress_read_repair = true; + lookup_opts.http_preconditions = None; + lookup_opts.version_id = None; + lookup_opts.conditional_delete_generations = None; + let results = join_all(self.pools.iter().map(|pool| { + let lookup_opts = &lookup_opts; + async move { + ( + pool.pool_idx, + pool.get_disks_by_key(object) + .conditional_delete_current_snapshot(bucket, object, lookup_opts) + .await, + ) + } + })) + .await; + let suspended_pools = { + let pool_meta = self.pool_meta.read().await; + (0..self.pools.len()) + .map(|idx| pool_meta.is_suspended(idx)) + .collect::>() + }; + resolve_conditional_delete_snapshot(bucket, object, results, &suspended_pools, &lookup_opts) + } + /// Acquire the pre-copy half of a data-movement publication fence. Legacy /// remote objects without a destination identity pin the named tier; an /// ordinary local source takes the zero-tier-manager-work `None` lease @@ -3173,7 +3457,7 @@ impl ECStore { Ok(guards) } - async fn acquire_object_read_lock_if_needed( + pub(super) async fn acquire_object_read_lock_if_needed( &self, op: &'static str, bucket: &str, @@ -4278,7 +4562,7 @@ impl ECStore { } async fn prepare_put_object(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<(String, ObjectOptions)> { - check_put_object_args(bucket, object)?; + check_new_object_args(bucket, object, opts.data_movement)?; let object = encode_dir_object(object); let mut opts = opts.clone(); @@ -4509,7 +4793,17 @@ impl ECStore { dst_opts: &ObjectOptions, ) -> Result { check_copy_obj_args(src_bucket, src_object)?; - check_copy_obj_args(dst_bucket, dst_object)?; + if src_bucket == dst_bucket && src_object == dst_object { + // In-place copies (metadata replace, restore) rewrite a key that + // already exists, so they must keep working for keys that new + // writes would no longer accept. + check_copy_obj_args(dst_bucket, dst_object)?; + } else { + // A copy onto another key creates that key: it must be admitted + // exactly like PutObject, since the copy reaches the pools' own + // put_object without passing through prepare_put_object. + check_new_object_args(dst_bucket, dst_object, dst_opts.data_movement)?; + } let src_object = encode_dir_object(src_object); let dst_object = encode_dir_object(dst_object); @@ -4866,6 +5160,9 @@ impl ECStore { } else { None }; + let guards = Arc::new(guards); + let mut opts = opts; + opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guards.clone()))); let store = Arc::clone(self); let bucket = bucket.to_owned(); let object = object.to_owned(); @@ -4877,7 +5174,7 @@ impl ECStore { if result.is_ok() { list_objects::observe_list_objects_mutation(&store, &bucket).await; } - // Keep: the only use of `guards` here, it moves them into the task so the locks outlive a dropped caller. + // Transient options also retain these guards through late disk I/O. drop(guards); result }) @@ -4958,7 +5255,7 @@ impl ECStore { let mut opts = opts; let receipt_sink = install_tier_free_version_receipt_sink(&mut opts); let result = self - .handle_delete_object_with_journal_inner(bucket, object, opts, tier_journal_api, lifecycle_owner) + .handle_delete_object_with_journal_inner_boxed(bucket, object, opts, tier_journal_api, lifecycle_owner) .await; enqueue_recorded_tier_free_versions(self, receipt_sink).await; result @@ -4966,6 +5263,17 @@ impl ECStore { .await } + fn handle_delete_object_with_journal_inner_boxed<'a>( + &'a self, + bucket: &'a str, + object: &'a str, + opts: ObjectOptions, + tier_journal_api: Option>, + lifecycle_owner: Option, + ) -> impl std::future::Future> + Send + 'a { + Box::pin(self.handle_delete_object_with_journal_inner(bucket, object, opts, tier_journal_api, lifecycle_owner)) + } + async fn handle_delete_object_with_journal_inner( &self, bucket: &str, @@ -4985,13 +5293,13 @@ impl ECStore { let mut bucket_lifecycle_guard = if is_meta_bucketname(bucket) { None } else if let Some(owner) = lifecycle_owner { - Some(owner) + Some(Arc::new(owner)) } else if opts.delete_prefix && opts.bucket_lifecycle_lock_fence.is_some() { return Err(Error::other("recursive delete requires its owned bucket lifecycle guard")); } else if opts.delete_prefix { - Some(self.acquire_bucket_lifecycle_write_lock(bucket).await?) + Some(Arc::new(self.acquire_bucket_lifecycle_write_lock(bucket).await?)) } else { - Some(self.acquire_bucket_lifecycle_read_lock(bucket).await?) + Some(Arc::new(self.acquire_bucket_lifecycle_read_lock(bucket).await?)) }; if !is_meta_bucketname(bucket) { let metadata = metadata_sys::get_bucket_incarnation_metadata_in(&self.ctx, bucket).await?; @@ -5019,6 +5327,7 @@ impl ECStore { opts.tier_delete_journal_api = tier_journal_api.clone(); if let Some(guard) = bucket_lifecycle_guard.as_ref() { opts.add_bucket_lifecycle_lock_guard(guard); + opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guard.clone()))); } if !is_meta_bucketname(bucket) { @@ -5034,8 +5343,8 @@ impl ECStore { } let current_bucket_incarnation_id = if let Some(metadata_guard) = _object_lock_metadata_guard { let (state, incarnation_id, config_revision) = - get_object_lock_config_and_incarnation_from_disk_in(&self.ctx, bucket).await?; - opts.object_lock_config_snapshot = Some(Arc::new(ObjectLockConfigSnapshot::for_guarded_store_bucket( + get_delete_config_from_disk_in(&self.ctx, bucket, None, opts.lifecycle_expiry_policy.as_ref()).await?; + opts.object_lock_config_snapshot = Some(Arc::new(ObjectLockConfigSnapshot::for_store_bucket_under_lifecycle_owner( self.id, bucket, incarnation_id, @@ -5044,7 +5353,7 @@ impl ECStore { bucket_lifecycle_guard .take() .ok_or_else(|| Error::other("delete requires its owned bucket lifecycle guard"))?, - metadata_guard, + Arc::new(metadata_guard), ))); Some(incarnation_id) } else { @@ -5271,6 +5580,40 @@ impl ECStore { return Ok(obj); } Err(err) => { + if (is_err_object_not_found(&err) || is_err_version_not_found(&err)) + && (opts.version_id.is_some() || (!opts.versioned && !opts.version_suspended)) + && opts.http_preconditions.is_none() + && !opts.replication_request + { + let mut reclaimed = None; + for pool in &self.pools { + if self.is_suspended(pool.pool_idx).await || self.is_pool_rebalancing(pool.pool_idx).await { + return Err(StorageError::SlowDown); + } + let pool_idx = pool.pool_idx; + let pool = pool.clone(); + match self + .run_external_decommission_capacity_object_delete( + pool_idx, + bucket, + object, + object, + opts.clone(), + |opts| async move { pool.delete_object(bucket, object, opts).await }, + ) + .await + { + Ok(result) => reclaimed = Some(result), + Err(missing) if is_err_object_not_found(&missing) || is_err_version_not_found(&missing) => {} + Err(error) => return Err(error), + } + } + if let Some(mut result) = reclaimed { + result.bucket = bucket.to_owned(); + result.name = decode_dir_object(object); + return Ok(result); + } + } // A folder key (`prefix/`) with no object metadata may still exist on // disk as an orphan empty-directory tree (issue #4189): listings show // it as a common prefix, but no regular delete path can remove it. @@ -5434,150 +5777,53 @@ impl ECStore { .await } - async fn handle_delete_objects_with_journal_and_accounting_inner( + /// Each phase owns separate disk transactions, receipts, and rollback. + /// The caller retains the batch namespace locks across both phases. + async fn run_batch_delete_phase( &self, bucket: &str, - objects: Vec, - opts: ObjectOptions, - tier_journal_api: Option>, + objects: &[ObjectToDelete], + opts: &ObjectOptions, + plan: BatchDeletePhasePlan<'_>, ) -> (Vec, Vec>, Vec>) { - // encode object name - let objects: Vec = objects - .iter() - .map(|v| { - let mut v = v.clone(); - v.object_name = encode_dir_object(v.object_name.as_str()); - v - }) - .collect(); - - // Default return value + let BatchDeletePhasePlan { + indices, + mut pool_generations, + marker_targets: conditional_marker_targets, + } = plan; + let mut active = vec![false; objects.len()]; + for idx in indices { + active[*idx] = true; + } let mut del_objects = vec![DeletedObject::default(); objects.len()]; + let mut del_errs = vec![None; objects.len()]; let mut accounting = vec![None; objects.len()]; - - let mut del_errs = Vec::with_capacity(objects.len()); - for _ in 0..objects.len() { - del_errs.push(None) - } - - let mut opts = opts; - opts.tier_delete_journal_api = tier_journal_api; - let mut bucket_lifecycle_guard = if is_meta_bucketname(bucket) { - None - } else { - match self.acquire_bucket_lifecycle_read_lock(bucket).await { - Ok(guard) => Some(guard), - Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), - } - }; - if let Some(guard) = bucket_lifecycle_guard.as_ref() { - opts.add_bucket_lifecycle_lock_guard(guard); - } - // Classify under the lifecycle owner before any per-object delete - // can stage metadata or enter its rollback path. - if !is_meta_bucketname(bucket) { - let metadata = match metadata_sys::get_bucket_incarnation_metadata_in(&self.ctx, bucket).await { - Ok(metadata) => metadata, - Err(error) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), error), - }; - if let Err(error) = crate::bucket::artifact::reject_unsupported_artifact_write(&metadata) { - return return_batch_delete_lock_error_with_accounting(objects.as_slice(), error); - } - } - if opts.delete_replication_config_snapshot.is_none() { - match ReplicationObjectBridge::delete_request_config_in(&self.ctx, bucket).await { - Ok(snapshot) => opts.delete_replication_config_snapshot = Some(Arc::new(snapshot)), - Err(err) => { - let message = err.to_string(); - let errors = (0..objects.len()).map(|_| Some(Error::other(message.clone()))).collect(); - return (del_objects, errors, accounting); - } - } - } - if !is_meta_bucketname(bucket) - && let Err(err) = get_cached_bucket_incarnation_id_in(&self.ctx, bucket).await - { - return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err); - } - let _object_lock_metadata_guard = if is_meta_bucketname(bucket) { - None - } else { - Some(match acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await { - Ok(guard) => guard, - Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), - }) - }; - if let Some(guard) = _object_lock_metadata_guard.as_ref() { - opts.add_namespace_lock_guard(guard); - } - let current_bucket_incarnation_id = if let Some(metadata_guard) = _object_lock_metadata_guard { - let (state, incarnation_id, config_revision) = - match get_object_lock_config_and_incarnation_from_disk_in(&self.ctx, bucket).await { - Ok(snapshot) => snapshot, - Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), - }; - opts.object_lock_config_snapshot = Some(Arc::new(ObjectLockConfigSnapshot::for_guarded_store_bucket( - self.id, - bucket, - incarnation_id, - config_revision, - state, - match bucket_lifecycle_guard.take() { - Some(guard) => guard, - None => { - return return_batch_delete_lock_error_with_accounting( - objects.as_slice(), - Error::other("delete requires its owned bucket lifecycle guard"), - ); - } - }, - metadata_guard, - ))); - Some(incarnation_id) - } else { - None - }; - if let (Some(expected), Some(current)) = (opts.expected_bucket_incarnation_id, current_bucket_incarnation_id) - && expected != current - { - return return_batch_delete_lock_error_with_accounting( - objects.as_slice(), - StorageError::BucketNotFound(bucket.to_string()), - ); - } - if opts.expected_bucket_incarnation_id.is_none() { - opts.expected_bucket_incarnation_id = current_bucket_incarnation_id; - } - #[cfg(any(test, feature = "test-util"))] - if current_bucket_incarnation_id.is_some() { - pause_delete_after_object_lock_snapshot(bucket).await; - } - let _object_lock_guards = match self.acquire_delete_objects_write_locks(bucket, &objects, &mut opts).await { - Ok(guards) => guards, - Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), - }; - #[cfg(any(test, feature = "test-util"))] - if !_object_lock_guards.is_empty() { - notify_delete_namespace_acquired(bucket); - } - let delete_config_snapshot = opts .delete_replication_config_snapshot .as_deref() .expect("batch delete replication config snapshot should be loaded"); let latest_marker_objects = objects .iter() - .map(|object| batch_delete_creates_latest_marker(object, delete_config_snapshot)) + .enumerate() + .map(|(idx, object)| { + active[idx] && del_errs[idx].is_none() && batch_delete_creates_latest_marker(object, delete_config_snapshot) + }) .collect::>(); - let marker_target_results = join_all(objects.iter().zip(&latest_marker_objects).map( - |(object, creates_marker)| async move { + let marker_target_results = join_all(objects.iter().zip(&latest_marker_objects).map(|(object, creates_marker)| { + let admitted_target = conditional_marker_targets.get(&object.object_name).copied(); + async move { if *creates_marker { - Some(self.get_pool_idx_no_lock(bucket, &object.object_name, 0).await) + // Routing is part of condition admission. A rebalance + // start cannot replace the validated current owner. + Some(match admitted_target { + Some(pool_idx) => Ok(pool_idx), + None => self.get_pool_idx_no_lock(bucket, &object.object_name, 0).await, + }) } else { None } - }, - )) + } + })) .await; let mut marker_target_pool_indices = Vec::with_capacity(objects.len()); for (idx, target_result) in marker_target_results.into_iter().enumerate() { @@ -5603,7 +5849,13 @@ impl ECStore { .await { Ok(fence) => Some(fence), - Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), + Err(err) => { + let (mut deleted, errors, receipts) = return_batch_delete_lock_error_with_accounting(objects, err); + for (result, object) in deleted.iter_mut().zip(objects) { + result.object_name.clone_from(&object.object_name); + } + return (deleted, errors, receipts); + } } } else { None @@ -5611,15 +5863,22 @@ impl ECStore { let mut futures = Vec::with_capacity(self.pools.len()); for pool in self.pools.iter() { - if self.is_pool_rebalancing(pool.pool_idx).await { + let rebalancing = self.is_pool_rebalancing(pool.pool_idx).await; + let guarded_generations = pool_generations.get(&pool.pool_idx); + if rebalancing && guarded_generations.is_none() { continue; } let (object_indices, pool_objects): (Vec<_>, Vec<_>) = objects .iter() .enumerate() - .filter(|(idx, _)| { - batch_delete_targets_pool(latest_marker_objects[*idx], marker_target_pool_indices[*idx], pool.pool_idx) + .filter(|(idx, object)| { + active[*idx] && del_errs[*idx].is_none() + // A newly rebalancing pool still owes every admitted + // conditional key its guarded mutation. Ordinary keys + // retain the existing rebalance skip behavior. + && (!rebalancing || guarded_generations.is_some_and(|generations| generations.contains_key(&object.object_name))) + && batch_delete_targets_pool(latest_marker_objects[*idx], marker_target_pool_indices[*idx], pool.pool_idx) }) .map(|(idx, object)| (idx, object.clone())) .unzip(); @@ -5627,7 +5886,8 @@ impl ECStore { continue; } - let pool_opts = opts.clone(); + let mut pool_opts = (*opts).clone(); + pool_opts.conditional_delete_generations = pool_generations.remove(&pool.pool_idx).map(Arc::new); futures.push(async move { #[cfg(test)] let pool_object_names = pool_objects @@ -5675,6 +5935,363 @@ impl ECStore { } } + (del_objects, del_errs, accounting) + } + + async fn handle_delete_objects_with_journal_and_accounting_inner( + &self, + bucket: &str, + objects: Vec, + opts: ObjectOptions, + tier_journal_api: Option>, + ) -> (Vec, Vec>, Vec>) { + // encode object name + let mut objects: Vec = objects + .iter() + .map(|v| { + let mut v = v.clone(); + v.object_name = encode_dir_object(v.object_name.as_str()); + v + }) + .collect(); + + // Default return value + let mut del_objects = vec![DeletedObject::default(); objects.len()]; + let mut accounting = vec![None; objects.len()]; + + let mut del_errs = Vec::with_capacity(objects.len()); + for _ in 0..objects.len() { + del_errs.push(None) + } + + let mut opts = opts; + opts.conditional_delete_generations = None; + opts.tier_delete_journal_api = tier_journal_api; + let _bucket_lifecycle_guard = if is_meta_bucketname(bucket) { + None + } else { + match self.acquire_bucket_lifecycle_read_lock(bucket).await { + Ok(guard) => Some(Arc::new(guard)), + Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), + } + }; + if let Some(guard) = _bucket_lifecycle_guard.as_ref() { + opts.add_bucket_lifecycle_lock_guard(guard); + opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guard.clone()))); + } + if !is_meta_bucketname(bucket) { + let metadata = match metadata_sys::get_bucket_incarnation_metadata_in(&self.ctx, bucket).await { + Ok(metadata) => metadata, + Err(error) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), error), + }; + if let Err(error) = crate::bucket::artifact::reject_unsupported_artifact_write(&metadata) { + return return_batch_delete_lock_error_with_accounting(objects.as_slice(), error); + } + } + if opts.delete_replication_config_snapshot.is_none() { + match ReplicationObjectBridge::delete_request_config_in(&self.ctx, bucket).await { + Ok(snapshot) => opts.delete_replication_config_snapshot = Some(Arc::new(snapshot)), + Err(err) => { + let message = err.to_string(); + let errors = (0..objects.len()).map(|_| Some(Error::other(message.clone()))).collect(); + return (del_objects, errors, accounting); + } + } + } + if !is_meta_bucketname(bucket) + && let Err(err) = get_cached_bucket_incarnation_id_in(&self.ctx, bucket).await + { + return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err); + } + let _object_lock_metadata_guard = if is_meta_bucketname(bucket) { + None + } else { + Some(Arc::new( + match acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await { + Ok(guard) => guard, + Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), + }, + )) + }; + if let Some(guard) = _object_lock_metadata_guard.as_ref() { + opts.add_namespace_lock_guard(guard); + } + let current_bucket_incarnation_id = if let Some(metadata_guard) = _object_lock_metadata_guard.as_ref() { + let (state, incarnation_id, config_revision) = + match get_delete_config_from_disk_in(&self.ctx, bucket, None, opts.lifecycle_expiry_policy.as_ref()).await { + Ok(snapshot) => snapshot, + Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), + }; + opts.object_lock_config_snapshot = Some(Arc::new(ObjectLockConfigSnapshot::for_store_bucket_under_lifecycle_owner( + self.id, + bucket, + incarnation_id, + config_revision, + state, + match _bucket_lifecycle_guard.clone() { + Some(owner) => owner, + None => { + return return_batch_delete_lock_error_with_accounting( + objects.as_slice(), + Error::other("delete requires its owned bucket lifecycle guard"), + ); + } + }, + Arc::clone(metadata_guard), + ))); + Some(incarnation_id) + } else { + None + }; + if let (Some(expected), Some(current)) = (opts.expected_bucket_incarnation_id, current_bucket_incarnation_id) + && expected != current + { + return return_batch_delete_lock_error_with_accounting( + objects.as_slice(), + StorageError::BucketNotFound(bucket.to_string()), + ); + } + if opts.expected_bucket_incarnation_id.is_none() { + opts.expected_bucket_incarnation_id = current_bucket_incarnation_id; + } + #[cfg(any(test, feature = "test-util"))] + if current_bucket_incarnation_id.is_some() { + pause_delete_after_object_lock_snapshot(bucket).await; + } + let _object_lock_guards = Arc::new(match self.acquire_delete_objects_write_locks(bucket, &objects, &mut opts).await { + Ok(guards) => guards, + Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), + }); + opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), _object_lock_guards.clone()))); + #[cfg(any(test, feature = "test-util"))] + if !_object_lock_guards.is_empty() { + notify_delete_namespace_acquired(bucket); + } + + let requested_conditions = objects + .iter() + .map(|object| object.delete_condition.is_some()) + .collect::>(); + let has_conditions = requested_conditions.iter().any(|condition| *condition); + let mut pool_generations: HashMap> = HashMap::new(); + let mut conditional_marker_targets = HashMap::new(); + let mut representatives: Vec = (0..objects.len()).collect(); + let mut admitted_conditions = vec![false; objects.len()]; + if has_conditions { + let mut rebalancing = false; + for pool in &self.pools { + rebalancing |= self.is_pool_rebalancing(pool.pool_idx).await; + } + let mut conditional_names = Vec::with_capacity(objects.len()); + for (idx, object) in objects.iter().enumerate() { + if let Some(condition) = object.delete_condition.as_ref() { + if self.ctx.lock_manager().is_disabled() || object.version_id.is_some() || condition.validate().is_err() { + del_errs[idx] = Some(StorageError::InvalidArgument( + bucket.to_owned(), + object.object_name.clone(), + "conditional batch delete requires a valid current-object condition and namespace locking".to_owned(), + )); + } else if rebalancing { + del_errs[idx] = Some(StorageError::SlowDown); + } else { + conditional_names.push(object.object_name.clone()); + } + } + } + conditional_names.sort_unstable(); + conditional_names.dedup(); + let snapshots = futures::stream::iter(conditional_names.into_iter().map(|object| { + let opts = &opts; + async move { + let snapshot = self.conditional_delete_snapshot(bucket, &object, opts).await; + (object, snapshot) + } + })) + .buffer_unordered(CONDITIONAL_DELETE_SNAPSHOT_CONCURRENCY) + .collect::>() + .await; + + for (idx, object) in objects.iter_mut().enumerate() { + if object.delete_condition.is_none() || del_errs[idx].is_some() { + continue; + } + let snapshot = snapshots + .get(&object.object_name) + .expect("each valid conditional key was included in the bounded snapshot pass"); + let result = match snapshot { + Ok(snapshot) => check_batch_delete_condition(object, snapshot, bucket), + Err(error) => Err(error.clone()), + }; + if let Err(error) = result { + del_errs[idx] = Some(error); + continue; + } + let snapshot = snapshot.as_ref().expect("condition admission requires a resolved snapshot"); + if !conditional_marker_targets.contains_key(&object.object_name) { + conditional_marker_targets.insert(object.object_name.clone(), snapshot.current_pool_idx); + } + for (pool_idx, generation) in &snapshot.pool_generations { + let generations = pool_generations.entry(*pool_idx).or_default(); + if !generations.contains_key(&object.object_name) { + generations.insert(object.object_name.clone(), generation.clone()); + } + } + // The condition has been consumed once, against the global + // current object; disks receive only their local generation. + admitted_conditions[idx] = true; + object.delete_condition = None; + } + + // All relevant peers are checked before marker selection, capacity + // admission, or any destructive work, even in a mixed batch. + let mut probe_sets = Vec::new(); + for (pool_idx, generations) in &pool_generations { + let pool = &self.pools[*pool_idx]; + let mut checked_sets = HashSet::new(); + for object in generations.keys() { + let set = pool.get_disks_by_key(object); + if checked_sets.insert(set.set_index) { + probe_sets.push(set); + } + } + } + let preflight = futures::stream::iter(probe_sets.into_iter().map(|set| async move { + #[cfg(test)] + await_conditional_capability_set_probe_gate(bucket).await?; + set.preflight_guarded_delete_versions().await + })) + .buffer_unordered(CONDITIONAL_DELETE_CAPABILITY_SET_CONCURRENCY) + .try_collect::>() + .await; + // Every selected set must succeed before conditional mutation. + // These probes are read-only: the first error can safely cancel + // pending work instead of holding namespace locks for its timeout. + if let Err(error) = preflight { + for (idx, admitted) in admitted_conditions.iter().enumerate() { + if *admitted { + del_errs[idx] = Some(error.clone()); + } + } + pool_generations.clear(); + conditional_marker_targets.clear(); + } + } + + #[cfg(test)] + inject_conditional_delete_post_admission_mutations(self, bucket).await; + + if !has_conditions { + let indices = (0..objects.len()).collect::>(); + (del_objects, del_errs, accounting) = self + .run_batch_delete_phase( + bucket, + &objects, + &opts, + BatchDeletePhasePlan { + indices: &indices, + pool_generations, + marker_targets: &conditional_marker_targets, + }, + ) + .await; + } else { + let mut conditional_actions = HashMap::with_capacity(objects.len()); + let mut conditional_indices = Vec::with_capacity(objects.len()); + for (idx, object) in objects.iter().enumerate() { + if admitted_conditions[idx] && del_errs[idx].is_none() { + let representative = *conditional_actions.entry(batch_delete_action_key(object)).or_insert(idx); + representatives[idx] = representative; + if representative == idx { + conditional_indices.push(idx); + } + } + } + if !conditional_indices.is_empty() { + let (deleted, mut errors, mut receipts) = self + .run_batch_delete_phase( + bucket, + &objects, + &opts, + BatchDeletePhasePlan { + indices: &conditional_indices, + pool_generations, + marker_targets: &conditional_marker_targets, + }, + ) + .await; + for idx in &conditional_indices { + del_objects[*idx] = deleted[*idx].clone(); + del_errs[*idx] = errors[*idx].take(); + accounting[*idx] = receipts[*idx].take(); + } + } + let successful_conditional_actions = conditional_actions + .into_iter() + .filter(|(_, idx)| del_errs[*idx].is_none()) + .collect::>(); + let mut ordinary_actions = HashMap::with_capacity(objects.len()); + let mut ordinary_indices = Vec::with_capacity(objects.len()); + for (idx, object) in objects.iter().enumerate() { + if requested_conditions[idx] || del_errs[idx].is_some() { + continue; + } + let action = batch_delete_action_key(object); + if object.expected_identity.is_none() + && let Some(representative) = successful_conditional_actions.get(&action) + { + // Only a committed conditional action can satisfy its + // unconditional alias. A CAS failure grants no deletion. + representatives[idx] = *representative; + continue; + } + let representative = *ordinary_actions.entry(action).or_insert(idx); + representatives[idx] = representative; + if representative == idx { + ordinary_indices.push(idx); + } + } + if !ordinary_indices.is_empty() { + // The guarded phase has finished all rollback/receipt work. + // These actions receive neither its CAS nor its transaction. + let (deleted, mut errors, mut receipts) = self + .run_batch_delete_phase( + bucket, + &objects, + &opts, + BatchDeletePhasePlan { + indices: &ordinary_indices, + pool_generations: HashMap::new(), + marker_targets: &HashMap::new(), + }, + ) + .await; + for idx in ordinary_indices { + del_objects[idx] = deleted[idx].clone(); + del_errs[idx] = errors[idx].take(); + accounting[idx] = receipts[idx].take(); + } + } + } + + if has_conditions { + for (idx, representative) in representatives.into_iter().enumerate() { + if representative != idx { + del_objects[idx] = del_objects[representative].clone(); + del_errs[idx] = del_errs[representative].clone(); + accounting[idx] = None; + } + } + let mut first_successes = HashMap::with_capacity(objects.len()); + for (idx, object) in objects.iter().enumerate() { + if del_errs[idx].is_none() { + let first = *first_successes.entry(batch_delete_action_key(object)).or_insert(idx); + if first != idx && accounting[idx].is_some() { + accounting[first] = accounting[idx].take(); + } + } + } + } + #[cfg(test)] for (idx, object) in objects.iter().enumerate() { if del_errs[idx].is_none() && del_objects[idx].delete_marker { @@ -8755,6 +9372,99 @@ mod tests { assert_eq!(sorted_unique_delete_object_names(&objects), vec!["alpha", "beta"]); } + fn conditional_pool_fixture(etag: &str, timestamp: i64) -> (ObjectInfo, CurrentDeleteGeneration) { + let mod_time = OffsetDateTime::from_unix_timestamp(timestamp).expect("fixture timestamp"); + let data_dir = Some(Uuid::from_u128(u128::try_from(timestamp).expect("positive fixture timestamp"))); + let info = ObjectInfo { + etag: Some(etag.to_owned()), + mod_time: Some(mod_time), + data_dir, + ..Default::default() + }; + let generation = CurrentDeleteGeneration::Present { + version_id: None, + data_dir, + mod_time, + delete_marker: false, + etag: etag.to_owned(), + transition: None, + }; + (info, generation) + } + + #[test] + fn conditional_delete_snapshot_rejects_unknown_pool_even_with_readable_old_copy() { + let error = resolve_conditional_delete_snapshot( + "bucket", + "key", + vec![ + (0, Ok(conditional_pool_fixture("old", 1))), + (1, Err(StorageError::ErasureReadQuorum)), + ], + &[false, false], + &ObjectOptions::default(), + ) + .err() + .expect("unknown pool must veto admission"); + assert_eq!(error, StorageError::ErasureReadQuorum); + } + + #[test] + fn conditional_delete_snapshot_keeps_each_pool_generation_and_explicit_absence() { + let old = conditional_pool_fixture("old", 1); + let new = conditional_pool_fixture("new", 2); + let old_generation = old.1.clone(); + let new_generation = new.1.clone(); + let snapshot = resolve_conditional_delete_snapshot( + "bucket", + "key", + vec![ + (0, Ok(old)), + (1, Ok(new)), + (2, Err(StorageError::ObjectNotFound("bucket".to_owned(), "key".to_owned()))), + ], + &[true, false, false], + &ObjectOptions::default(), + ) + .expect("all pools are known"); + assert_eq!(snapshot.current.etag.as_deref(), Some("new")); + assert_eq!(snapshot.current_pool_idx, 1); + assert_eq!(snapshot.pool_generations.get(&0), Some(&old_generation)); + assert_eq!(snapshot.pool_generations.get(&1), Some(&new_generation)); + assert_eq!(snapshot.pool_generations.get(&2), Some(&CurrentDeleteGeneration::Absent)); + let conditions = ["old", "new", "*", "new"]; + let results = conditions.map(|etag| { + check_batch_delete_condition( + &ObjectToDelete { + object_name: "key".to_owned(), + delete_condition: Some( + crate::storage_api_contracts::object::DeleteObjectCondition::parse(etag).expect("valid condition"), + ), + ..Default::default() + }, + &snapshot, + "bucket", + ) + }); + assert_eq!(results[0], Err(StorageError::PreconditionFailed)); + assert!(results[1..].iter().all(Result::is_ok), "duplicate conditions use one current snapshot"); + } + + #[test] + fn conditional_delete_snapshot_rejects_current_marker_and_missing_object() { + let (mut marker, generation) = conditional_pool_fixture("marker", 2); + marker.delete_marker = true; + for results in [ + vec![(0, Ok((marker, generation)))], + vec![(0, Err(StorageError::ObjectNotFound("bucket".to_owned(), "key".to_owned())))], + ] { + let error = resolve_conditional_delete_snapshot("bucket", "key", results, &[false], &ObjectOptions::default()) + .err() + .expect("missing live object must fail"); + assert!(is_err_object_not_found(&error)); + } + } + async fn new_read_lock_test_store() -> ECStore { let format = FormatV3::new(1, 2); let endpoints = vec![ @@ -9123,6 +9833,10 @@ mod tests { } async fn multipool_version_test_store(bucket: &str) -> (Vec, Arc) { + multipool_delete_test_store(bucket, true).await + } + + async fn multipool_delete_test_store(bucket: &str, versioned: bool) -> (Vec, Arc) { let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); let (mut dirs, first_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await; let (second_dirs, second_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await; @@ -9133,15 +9847,587 @@ mod tests { .handle_make_bucket( bucket, &MakeBucketOptions { - versioning_enabled: true, + versioning_enabled: versioned, ..Default::default() }, ) .await - .expect("create the versioned bucket in both pools"); + .expect("create the bucket in both pools"); (dirs, store) } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_set_probes_are_bounded_before_mutation() { + assert_conditional_set_probe_admission(false).await; + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_set_probe_error_cancels_pending_probes() { + assert_conditional_set_probe_admission(true).await; + } + + async fn assert_conditional_set_probe_admission(reject_first: bool) { + use crate::storage_api_contracts::object::DeleteObjectCondition; + let bucket = if reject_first { + "conditional-capability-set-rejected" + } else { + "conditional-capability-set-supported" + }; + let protected = "protected.bin"; + let ordinary = "ordinary.bin"; + let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + let mut dirs = Vec::new(); + let mut sets = Vec::new(); + for _ in 0..17 { + let (set_dirs, set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await; + dirs.extend(set_dirs); + sets.push(set); + } + let store = Arc::new(new_prepared_reader_test_store_with_ctx(&sets, ctx).await); + crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + store + .handle_make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create all pool buckets"); + let protected_before = store.pools[0] + .put_object( + bucket, + protected, + &mut PutObjReader::from_vec(b"protected data".to_vec()), + &ObjectOptions::default(), + ) + .await + .expect("write conditional target"); + let ordinary_before = store.pools[0] + .put_object( + bucket, + ordinary, + &mut PutObjReader::from_vec(b"ordinary data".to_vec()), + &ObjectOptions::default(), + ) + .await + .expect("write ordinary control"); + let mut gate = ConditionalCapabilitySetProbeGate::install(bucket, reject_first); + let delete_store = Arc::clone(&store); + let condition_etag = protected_before.etag.clone().expect("conditional target ETag"); + let delete = tokio::spawn(async move { + delete_store + .handle_delete_objects( + bucket, + vec![ + ObjectToDelete { + object_name: protected.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(condition_etag)), + ..Default::default() + }, + ObjectToDelete { + object_name: ordinary.to_owned(), + ..Default::default() + }, + ], + ObjectOptions::default(), + ) + .await + }); + gate.abort_task = Some(delete.abort_handle()); + gate.wait_started(16).await; + assert_eq!(gate.state.started.load(Ordering::Acquire), 16, "the seventeenth probe must remain queued"); + assert_eq!(gate.state.active.load(Ordering::Acquire), 16); + let read_options = ObjectOptions { + no_lock: true, + metadata_cache_safe: false, + suppress_read_repair: true, + ..Default::default() + }; + for (key, before) in [(protected, &protected_before), (ordinary, &ordinary_before)] { + let current = store.pools[0] + .get_object_info(bucket, key, &read_options) + .await + .expect("read disk metadata under the retained outer write lock"); + assert_eq!(current.data_dir, before.data_dir, "pending probes cannot mutate {key}"); + assert_eq!(current.etag, before.etag); + } + gate.state.release.add_permits(1); + if !reject_first { + gate.wait_started(17).await; + assert!(!delete.is_finished(), "one successful set cannot bypass still-running capability probes"); + for (key, before) in [(protected, &protected_before), (ordinary, &ordinary_before)] { + let current = store.pools[0] + .get_object_info(bucket, key, &read_options) + .await + .expect("read metadata while the final set remains paused"); + assert_eq!(current.data_dir, before.data_dir, "all sets must approve before mutating {key}"); + assert_eq!(current.etag, before.etag); + } + gate.state.release.add_permits(16); + } + let (_, errors) = tokio::time::timeout(Duration::from_secs(30), delete) + .await + .expect("capability admission must finish without releasing failed peers") + .expect("delete task joins"); + assert_eq!(gate.state.max_active.load(Ordering::Acquire), 16, "capability fanout must be bounded"); + assert_eq!( + gate.state.active.load(Ordering::Acquire), + 0, + "completed or canceled probes must release their activity" + ); + assert_eq!( + gate.state.started.load(Ordering::Acquire), + if reject_first { 16 } else { 17 }, + "a rejected capability must cancel queued set probes" + ); + if reject_first { + assert_eq!(errors[0], Some(StorageError::GuardedDeleteUnsupported)); + let after = store + .get_object_info(bucket, protected, &ObjectOptions::default()) + .await + .expect("rejected condition preserves data"); + assert_eq!(after.data_dir, protected_before.data_dir); + assert_eq!(after.etag, protected_before.etag); + } else { + assert!(errors[0].is_none(), "all supported sets authorize conditional deletion: {errors:?}"); + assert!( + store + .get_object_info(bucket, protected, &ObjectOptions::default()) + .await + .err() + .is_some_and(|error| is_err_object_not_found(&error)) + ); + } + assert!( + errors[1].is_none(), + "ordinary deletion remains independent of capability admission: {errors:?}" + ); + assert!( + store + .get_object_info(bucket, ordinary, &ObjectOptions::default()) + .await + .err() + .is_some_and(|error| is_err_object_not_found(&error)) + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_conflict_preserves_unconditional_same_key_actions() { + use crate::storage_api_contracts::object::DeleteObjectCondition; + for delete_current in [false, true] { + let bucket = if delete_current { + "conditional-conflict-current" + } else { + "conditional-conflict-history" + }; + let object = "phase-conflict.bin"; + let (_dirs, store) = multipool_version_test_store(bucket).await; + let options = ObjectOptions { + versioned: true, + ..Default::default() + }; + let history = store.pools[1] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"history".to_vec()), &options) + .await + .expect("write unconditional historical target"); + let current = store.pools[1] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"admitted".to_vec()), &options) + .await + .expect("write admitted current generation"); + CONDITIONAL_DELETE_POST_ADMISSION_OVERWRITE + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("overwrite injection lock") + .insert( + bucket.to_owned(), + ConditionalDeletePostAdmissionOverwrite { + pool_idx: 1, + object: object.to_owned(), + payload: b"replacement".to_vec(), + versioned: true, + }, + ); + let conditional = ObjectToDelete { + object_name: object.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(current.etag.clone().expect("current ETag"))), + ..Default::default() + }; + let (deleted, errors) = store + .handle_delete_objects( + bucket, + vec![ + conditional.clone(), + ObjectToDelete { + object_name: object.to_owned(), + version_id: if delete_current { None } else { history.version_id }, + ..Default::default() + }, + conditional, + ], + options, + ) + .await; + assert_eq!(errors[0], Some(StorageError::ConditionalDeleteConflict)); + assert_eq!(errors[2], Some(StorageError::ConditionalDeleteConflict)); + assert!( + errors[1].is_none(), + "unconditional action must not inherit the current-object CAS: {errors:?}" + ); + let versions = store + .clone() + .inner_list_object_versions(bucket, object, None, None, None, 10) + .await + .expect("list independent transaction results"); + if delete_current { + let marker = deleted[1] + .delete_marker_version_id + .expect("ordinary current delete creates its own marker"); + assert_eq!( + versions.objects.iter().filter(|entry| entry.delete_marker).count(), + 1, + "a rejected guarded transaction cannot leave or roll back the ordinary marker" + ); + assert!( + versions + .objects + .iter() + .any(|entry| entry.delete_marker && entry.version_id == Some(marker)) + ); + } else { + assert_eq!(deleted[1].version_id, history.version_id); + assert!(!versions.objects.iter().any(|entry| entry.version_id == history.version_id)); + assert!(versions.objects.iter().all(|entry| !entry.delete_marker)); + let mut reader = store + .handle_get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("replacement survives the failed guarded action"); + let mut body = Vec::new(); + reader + .stream + .read_to_end(&mut body) + .await + .expect("read surviving replacement"); + assert_eq!(body, b"replacement"); + } + } + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_success_reuses_unconditional_current_alias_once() { + use crate::storage_api_contracts::object::DeleteObjectCondition; + for versioned in [false, true] { + let bucket = if versioned { + "conditional-alias-marker" + } else { + "conditional-alias-plain" + }; + let object = "phase-alias.bin"; + let (_dirs, store) = multipool_delete_test_store(bucket, versioned).await; + let options = ObjectOptions { + versioned, + ..Default::default() + }; + let current = store.pools[1] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"one action".to_vec()), &options) + .await + .expect("write current alias target"); + let conditional = ObjectToDelete { + object_name: object.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(current.etag.clone().expect("current ETag"))), + ..Default::default() + }; + let (deleted, errors, accounting) = store + .handle_delete_objects_with_journal_and_accounting( + bucket, + vec![ + ObjectToDelete { + object_name: object.to_owned(), + ..Default::default() + }, + conditional.clone(), + conditional, + ], + options, + None, + ) + .await; + assert!(errors.iter().all(Option::is_none), "all aliases succeed: {errors:?}"); + if versioned { + let marker = deleted[0].delete_marker_version_id.expect("one physical marker"); + assert!(deleted.iter().all(|entry| entry.delete_marker_version_id == Some(marker))); + let versions = store + .clone() + .inner_list_object_versions(bucket, object, None, None, None, 10) + .await + .expect("list reused marker"); + assert_eq!(versions.objects.iter().filter(|entry| entry.delete_marker).count(), 1); + } else { + assert!(accounting[0].is_some(), "the first successful input owns the physical accounting"); + assert!(accounting[1..].iter().all(Option::is_none)); + assert!( + store + .get_object_info(bucket, object, &ObjectOptions::default()) + .await + .err() + .is_some_and(|error| is_err_object_not_found(&error)) + ); + } + } + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_directory_marker_keeps_synthetic_null_delete_distinct() { + use crate::storage_api_contracts::object::DeleteObjectCondition; + let bucket = "conditional-directory-targets"; + let object = "dir/"; + let (_dirs, store) = multipool_version_test_store(bucket).await; + // Directory PUTs use the null data version even in an enabled bucket. + let current = store + .put_object( + bucket, + object, + &mut PutObjReader::from_vec(b"directory payload".to_vec()), + &ObjectOptions::default(), + ) + .await + .expect("write the null directory data version"); + let before = store + .clone() + .inner_list_object_versions(bucket, object, None, None, None, 10) + .await + .expect("list the null data fixture"); + assert_eq!(before.objects.len(), 1); + assert!(before.objects[0].version_id.is_none_or(|id| id.is_nil()) && !before.objects[0].delete_marker); + let (deleted, errors) = store + .handle_delete_objects( + bucket, + vec![ + ObjectToDelete { + object_name: object.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(current.etag.expect("directory ETag"))), + ..Default::default() + }, + ObjectToDelete { + object_name: object.to_owned(), + version_id: Some(Uuid::nil()), + synthetic_version_id: true, + ..Default::default() + }, + ], + ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await; + assert!(errors.iter().all(Option::is_none), "both distinct targets must succeed: {errors:?}"); + let marker = deleted[0] + .delete_marker_version_id + .expect("conditional current delete creates a marker"); + assert!(!marker.is_nil()); + assert!(deleted[0].delete_marker); + assert!( + !deleted[1].delete_marker, + "the synthetic request deletes null data, not the created marker" + ); + assert_eq!(deleted[1].version_id, Some(Uuid::nil())); + let after = store + .clone() + .inner_list_object_versions(bucket, object, None, None, None, 10) + .await + .expect("list both action results"); + assert_eq!(after.objects.len(), 1, "one UUID marker remains and null data must be removed"); + assert!(after.objects[0].delete_marker); + assert_eq!(after.objects[0].version_id, Some(marker)); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_freezes_routes_when_rebalance_starts_after_admission() { + use crate::storage_api_contracts::object::DeleteObjectCondition; + for versioned in [false, true] { + let bucket = if versioned { + "conditional-route-versioned" + } else { + "conditional-route-plain" + }; + let object = "conditional-routed.bin"; + let ordinary = "ordinary-routed.bin"; + let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + let (mut dirs, first_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await; + let (second_dirs, second_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await; + dirs.extend(second_dirs); + let store = Arc::new(new_prepared_reader_test_store_with_ctx(&[first_set, second_set], ctx).await); + crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + store + .handle_make_bucket( + bucket, + &MakeBucketOptions { + versioning_enabled: versioned, + ..Default::default() + }, + ) + .await + .expect("create bucket in both pools"); + let options = ObjectOptions { + versioned, + ..Default::default() + }; + store.pools[0] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"old".to_vec()), &options) + .await + .expect("write an older copy to the other pool"); + let current = store.pools[1] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"current".to_vec()), &options) + .await + .expect("write the current copy to the future rebalance pool"); + let ordinary_before = store.pools[1] + .put_object(bucket, ordinary, &mut PutObjReader::from_vec(b"ordinary".to_vec()), &options) + .await + .expect("write the unconditional control key to the same pool"); + CONDITIONAL_DELETE_POST_ADMISSION_REBALANCE + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("rebalance injection lock") + .insert(bucket.to_owned(), 1); + let (deleted, errors) = store + .handle_delete_objects( + bucket, + vec![ + ObjectToDelete { + object_name: object.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(current.etag.clone().expect("current ETag"))), + ..Default::default() + }, + ObjectToDelete { + object_name: ordinary.to_owned(), + version_id: ordinary_before.version_id, + ..Default::default() + }, + ], + options, + ) + .await; + assert!( + store.is_pool_rebalancing(1).await, + "rebalance must have started in the admission-to-dispatch window" + ); + assert!(errors[0].is_none(), "the frozen conditional action must commit: {errors:?}"); + *store.rebalance_meta.write().await = None; + let ordinary_after = store.pools[1] + .get_object_info(bucket, ordinary, &ObjectOptions::default()) + .await + .expect("unconditional key must retain the existing rebalance skip behavior"); + assert_eq!(ordinary_after.data_dir, ordinary_before.data_dir); + if versioned { + let marker = deleted[0].delete_marker_version_id.expect("conditional marker result"); + for pool in &store.pools { + let versions = pool + .clone() + .inner_list_object_versions(bucket, object, None, None, None, 10) + .await + .expect("list the frozen marker destination"); + assert_eq!( + versions + .objects + .iter() + .any(|entry| entry.delete_marker && entry.version_id == Some(marker)), + pool.pool_idx == 1, + "the marker must stay with the admitted current owner" + ); + } + } else { + for pool in &store.pools { + assert!( + pool.get_object_info(bucket, object, &ObjectOptions::default()) + .await + .err() + .is_some_and(|error| is_err_object_not_found(&error)), + "the current copy and stale copy must both be deleted" + ); + } + } + } + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_mixed_duplicates_preserve_historical_actions() { + use crate::storage_api_contracts::object::DeleteObjectCondition; + let bucket = "conditional-batch-duplicates"; + let object = "conditional-history.bin"; + let (_dirs, store) = multipool_version_test_store(bucket).await; + let options = ObjectOptions { + versioned: true, + ..Default::default() + }; + let old = store.pools[0] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"old".to_vec()), &options) + .await + .expect("write older pool-local copy"); + let current = store.pools[1] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"new".to_vec()), &options) + .await + .expect("write current pool-local copy"); + let mut objects = vec![ObjectToDelete { + object_name: object.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(old.etag.clone().expect("old etag"))), + ..Default::default() + }]; + objects.extend((0..997).map(|_| ObjectToDelete { + object_name: object.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(current.etag.clone().expect("current etag"))), + ..Default::default() + })); + objects.push(ObjectToDelete { + object_name: object.to_owned(), + version_id: old.version_id, + ..Default::default() + }); + objects.push(ObjectToDelete { + object_name: "missing".to_owned(), + delete_condition: Some(DeleteObjectCondition::Exists), + ..Default::default() + }); + let calls = disk_call_counters::observe(object); + let (deleted, errors, accounting) = store + .handle_delete_objects_with_journal_and_accounting(bucket, objects, options, None) + .await; + assert!( + calls.total(disk_call_counters::KIND_READ_VERSION) <= 64, + "997 identical conditions must share one metadata snapshot and mutation plan" + ); + assert_eq!(errors[0], Some(StorageError::PreconditionFailed)); + assert!( + errors[1..=998].iter().all(Option::is_none), + "accepted conditions and historical delete: {errors:?}" + ); + assert!(errors[999].as_ref().is_some_and(is_err_object_not_found)); + let marker_id = deleted[1].delete_marker_version_id.expect("one created marker"); + assert!( + deleted[1..=997] + .iter() + .all(|entry| entry.delete_marker && entry.delete_marker_version_id == Some(marker_id)) + ); + assert!( + accounting[2..=997].iter().all(Option::is_none), + "duplicate actions cannot duplicate accounting" + ); + let versions = store + .clone() + .inner_list_object_versions(bucket, object, None, None, None, 10) + .await + .expect("list remaining versions"); + assert_eq!(versions.objects.len(), 2, "one marker and the current data version must remain"); + assert!( + versions + .objects + .iter() + .any(|entry| entry.version_id == current.version_id && !entry.delete_marker) + ); + assert!( + versions + .objects + .iter() + .any(|entry| entry.version_id == Some(marker_id) && entry.delete_marker) + ); + assert!(!versions.objects.iter().any(|entry| entry.version_id == old.version_id)); + } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn batch_delete_rejects_dot_segments_and_keeps_valid_result_positions() { let bucket = "batch-delete-key-validation"; @@ -9261,10 +10547,34 @@ mod tests { .expect("read original marker"); assert_eq!(info.delete_marker_incarnation(), Some(old_incarnation)); } - // Taking four actual members offline preserves their existing xl.meta. - // All deletion/recreation work below goes through the normal store API. - for slot in 12..16 { - set.disks.write().await[slot] = None; + // Save each stale member's inline versions before converging the old + // bucket's delete receipts, then reintroduce them after recreation. + let mut stale_metadata = Vec::new(); + for disk in before.iter().skip(12).flatten() { + for version in &historical { + let info = disk + .read_version( + "", + bucket, + "history.bin", + &version.to_string(), + &crate::disk::ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("read stale inline data version"); + assert!(info.inline_data() && info.data.is_some(), "fixture payload must be preserved in xl.meta"); + } + stale_metadata.push(( + disk.read_all(bucket, "history.bin/xl.meta").await.unwrap(), + disk.read_all(bucket, &format!("{object}/xl.meta")).await.unwrap(), + )); + } + for disk in before.iter().skip(12).flatten() { + disk.close().await.expect("stop stale member health monitoring"); + disk.force_offline_for_test(); } for version in historical { store @@ -9292,6 +10602,31 @@ mod tests { ) .await .expect("delete exact old marker with four drives offline"); + let pending = store + .handle_delete_bucket(bucket, &crate::storage_api_contracts::bucket::DeleteBucketOptions::default()) + .await + .expect_err("offline participants require retained delete receipts"); + assert!(matches!( + pending, + StorageError::BucketNotEmpty(_) | StorageError::BucketNotEmptyWithDetails { .. } + )); + assert!( + !crate::bucket::retirement::is_retired(store.clone(), bucket, old_incarnation) + .await + .unwrap() + ); + for disk in before.iter().skip(12).flatten() { + disk.reset_health_for_store_init_retry(); + } + store + .recover_delete_transactions_at_incarnation( + bucket, + None, + Some(old_incarnation), + &rustfs_heal_contracts::heal_channel::HealOpts::default(), + ) + .await + .expect("converge old bucket deletes on all participants"); store .handle_delete_bucket(bucket, &crate::storage_api_contracts::bucket::DeleteBucketOptions::default()) .await @@ -9301,6 +10636,10 @@ mod tests { .await .expect("durable retirement") ); + for disk in before.iter().skip(12).flatten() { + disk.close().await.expect("stop returning member health monitoring"); + disk.force_offline_for_test(); + } store .handle_make_bucket(bucket, &create) .await @@ -9326,7 +10665,16 @@ mod tests { .expect("write new generation"); new_versions.push((key, info.version_id.unwrap(), payload)); } - *set.disks.write().await = before.clone(); + for (slot, (history, marker)) in (12..16).zip(stale_metadata) { + for (key, bytes) in [("history.bin", history), (object, marker)] { + let root = dirs[slot].path().join(bucket).join(key); + tokio::fs::create_dir_all(&root).await.unwrap(); + tokio::fs::write(root.join("xl.meta"), bytes).await.unwrap(); + } + } + for disk in before.iter().skip(12).flatten() { + disk.reset_health_for_store_init_retry(); + } let heal = rustfs_heal_contracts::heal_channel::HealOpts { remove: true, scan_mode: rustfs_heal_contracts::heal_channel::HealScanMode::Deep, @@ -9410,14 +10758,16 @@ mod tests { .unwrap(); } } - set.disks.write().await[15] = None; + let offline = before[15].as_ref().expect("cleanup target must exist"); + offline.close().await.expect("stop cleanup target health monitoring"); + offline.force_offline_for_test(); let incomplete = store .heal_object_with_proof(bucket, object, &marker.to_string(), &heal) .await .unwrap(); assert!(incomplete.error.as_ref().is_some_and(Error::is_retired_marker_deferred)); assert!(incomplete.absence.is_none(), "offline targets cannot be omitted from a cleanup receipt"); - set.disks.write().await[15] = before[15].clone(); + offline.reset_health_for_store_init_retry(); let failure = crate::set_disk::DanglingDeleteFailure::install(bucket, object, 15, crate::disk::error::DiskError::FaultyDisk); let partial = store @@ -9536,9 +10886,13 @@ mod tests { .expect("bucket incarnation"); let set = &store.pools[0].disk_set[0]; let before = set.disks.read().await.clone(); - for slot in 12..16 { - set.disks.write().await[slot] = None; + // Keep the handles attached so reconnect probes inherit the forced + // outage instead of recreating healthy handles for these local paths. + for disk in before.iter().skip(12).flatten() { + disk.close().await.expect("stop stale member health monitoring"); + disk.force_offline_for_test(); } + set.connect_disks().await; store .delete_object( bucket, @@ -9551,8 +10905,6 @@ mod tests { ) .await .expect("acknowledged delete should remove the marker from the online quorum"); - *set.disks.write().await = before.clone(); - for disk in before.iter().take(12).flatten() { assert!(matches!( disk.read_version("", bucket, object, &marker.to_string(), &crate::disk::ReadOptions::default()) @@ -9561,6 +10913,8 @@ mod tests { )); } for disk in before.iter().skip(12).flatten() { + assert_eq!(disk.runtime_state(), crate::disk::health_state::RuntimeDriveHealthState::Offline); + disk.reset_health_for_store_init_retry(); assert!( disk.read_version("", bucket, object, &marker.to_string(), &crate::disk::ReadOptions::default()) .await @@ -10834,10 +12188,8 @@ mod tests { #[tokio::test] #[serial_test::serial(body_cache_hook)] async fn legacy_reader_reuses_selected_pool_metadata() { - let (_first_dirs, first_set) = make_local_set_disks(4, 2).await; - let (_second_dirs, second_set) = make_local_set_disks(4, 2).await; - let store = new_prepared_reader_test_store(&[first_set, second_set]).await; let bucket = "legacy-reader-second-pool"; + let (_dirs, store) = multipool_delete_test_store(bucket, false).await; let object = "object.bin"; let payload = b"legacy-reader-second-pool-payload-".repeat(40_000); let opts = ObjectOptions { @@ -10845,16 +12197,25 @@ mod tests { ..Default::default() }; - for pool in &store.pools { - pool.make_bucket(bucket, &MakeBucketOptions::default()) + { + let incarnation = crate::bucket::metadata_sys::get_bucket_incarnation_metadata_in(&store.ctx, bucket) .await - .expect("bucket should be created in each pool"); + .expect("fixture bucket metadata") + .bucket_incarnation_id; + let owner = Arc::new( + store + .acquire_bucket_incarnation_fence(bucket, incarnation) + .await + .expect("fixture lifecycle owner"), + ); + let mut put_opts = opts.clone(); + owner.attach_owner(&mut put_opts); + let mut put_reader = PutObjReader::from_vec(payload.clone()); + store.pools[1] + .put_object(bucket, object, &mut put_reader, &put_opts) + .await + .expect("object should be written only to the second pool"); } - let mut put_reader = PutObjReader::from_vec(payload.clone()); - store.pools[1] - .put_object(bucket, object, &mut put_reader, &opts) - .await - .expect("object should be written only to the second pool"); clear_get_object_body_cache_hook(); let hook = Arc::new(CountingMissHook { @@ -11561,3 +12922,160 @@ mod tests { .await; } } + +#[cfg(test)] +mod missing_key_delete_quorum_tests { + use super::*; + use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks}; + use crate::storage_api_contracts::bucket::{BucketOperations, BucketOptions, MakeBucketOptions}; + + async fn degraded_store(bucket: &str) -> (Vec, Arc) { + let (dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create authoritative never-versioned bucket"); + store.pools[0].disk_set[0].disks.write().await[0] = None; + let info = store + .get_bucket_info(bucket, &BucketOptions::default()) + .await + .expect("existing bucket must remain readable at quorum"); + assert_eq!(info.name, bucket); + let lookup = store + .get_object_info_for_delete(bucket, "never-existed", &ObjectOptions::default()) + .await; + assert!( + lookup.as_ref().is_err_and(is_err_object_not_found), + "remaining disks must establish absence through production lookup: {lookup:?}" + ); + (dirs, store) + } + + #[tokio::test] + async fn single_delete_missing_key_preserves_quorum_noop() { + let (_dirs, store) = degraded_store("missing-key-quorum-single").await; + let result = store + .delete_object_with_tier_delete_journal("missing-key-quorum-single", "never-existed", ObjectOptions::default()) + .await; + assert!( + result.as_ref().is_err_and(is_err_object_not_found), + "single missing-key delete must preserve its idempotent not-found outcome at quorum: {result:?}" + ); + } + + #[tokio::test] + async fn batch_delete_missing_key_preserves_quorum_noop() { + let (_dirs, store) = degraded_store("missing-key-quorum-batch").await; + let (_, errors, _) = store + .delete_objects_with_tier_delete_journal_and_accounting( + "missing-key-quorum-batch", + vec![ObjectToDelete { + object_name: "never-existed".to_owned(), + ..Default::default() + }], + ObjectOptions::default(), + ) + .await; + assert_eq!(errors.len(), 1, "batch DELETE must return one result per key"); + assert!( + errors.iter().all(Option::is_none), + "batch missing-key delete must preserve its idempotent success at quorum: {errors:?}" + ); + } + + #[cfg(not(windows))] + #[tokio::test] + async fn batch_delete_timeout_retains_physical_key_guard() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + use crate::disk::{DiskAPI, ReadOptions}; + use crate::storage_api_contracts::object::ObjectIO; + temp_env::async_with_vars([("RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS", Some("30"))], async { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let bucket = "review-batch-late-owner"; + let key = "object"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("fixture bucket"); + let mut reader = PutObjReader::from_vec(vec![0x51; 1024 * 1024]); + store + .put_object( + bucket, + key, + &mut reader, + &ObjectOptions { + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("seed real external object"); + let disks = store.pools[0].disk_set[0].disk_inventory().await; + let disk = disks[0].as_ref().expect("online drive"); + let info = disk + .read_version("", bucket, key, "", &ReadOptions::default()) + .await + .expect("seed metadata"); + assert!(!info.inline_data()); + let crate::disk::Disk::Local(local) = disk.as_ref() else { + panic!("local disk"); + }; + let part = local + .get_disk() + .get_object_path_for_io(bucket, key) + .expect("leased key") + .join(info.data_dir.expect("external data directory").to_string()) + .join("part.1"); + let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (release, release_rx) = std::sync::mpsc::channel::<()>(); + let hook = hooks::install_at(hooks::Stage::Remove, &part, move || { + let _ = entered_tx.send(()); + let _ = release_rx.recv(); + }); + let deleting = store.clone(); + let delete = tokio::spawn(async move { + deleting + .delete_objects_with_tier_delete_journal_and_accounting( + bucket, + vec![ObjectToDelete { + object_name: key.to_owned(), + ..Default::default() + }], + ObjectOptions::default(), + ) + .await + }); + tokio::time::timeout(Duration::from_secs(10), entered_rx) + .await + .expect("physical cleanup entry") + .expect("entry hook"); + assert!(part.exists(), "payload unlink has not run"); + let (_, errors, _) = tokio::time::timeout(Duration::from_secs(40), delete) + .await + .expect("real settlement timeout must return") + .expect("delete task"); + assert!(errors.iter().all(Option::is_none), "certified batch response: {errors:?}"); + assert!( + store.ctx.namespace_commits_pending(), + "background physical operation must retain its owner" + ); + let ns = store.handle_new_ns_lock(bucket, key).await.expect("same production key lock"); + let probe = ns.get_write_lock(Duration::from_millis(300)).await; + let acquired_while_pending = probe.is_ok(); + drop(probe); + drop(release); + drop(hook); + assert!( + !acquired_while_pending, + "batch released the exact-key guard while physical cleanup was still paused after its real recovery timeout" + ); + let _released = ns + .get_write_lock(Duration::from_secs(10)) + .await + .expect("physical cleanup must eventually release the exact-key guard"); + }) + .await; + } +} diff --git a/crates/ecstore/src/store/peer.rs b/crates/ecstore/src/store/peer.rs index 56b15720b..b91e78bef 100644 --- a/crates/ecstore/src/store/peer.rs +++ b/crates/ecstore/src/store/peer.rs @@ -16,11 +16,17 @@ use super::*; use crate::bucket::utils::has_bad_path_component; use crate::disk::error::{DiskError, Result as DiskResult}; use crate::disk::{ - DeleteOptions, Disk, LegacyFileWrite, LegacyMetadataWrite, LegacyPartMutation, LegacyPartOperation, LegacyPathDelete, - LegacyPathDeleteOperation, LegacyRenameData, LegacyRenameFile, RenameDataGuards, RenameDataResp, + DeleteOptions, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, Disk, GuardedDeleteVersionsPayload, LegacyFileWrite, + LegacyMetadataWrite, LegacyPartMutation, LegacyPartOperation, LegacyPathDelete, LegacyPathDeleteOperation, LegacyRenameData, + LegacyRenameFile, RenameDataGuards, RenameDataResp, }; use crate::runtime::instance::{InstanceContext, NamespaceCommitGuard}; use crate::runtime::sources as runtime_sources; +use rustfs_io_metrics::{ + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_ADMIT, PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOCAL_RENAME, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOOKUP, PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_OVERHEAD, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_TOTAL, +}; use tracing::{debug, error}; const LOG_COMPONENT_ECSTORE: &str = "ecstore"; @@ -36,6 +42,26 @@ pub struct BootstrapLocalTarget { } impl BootstrapLocalTarget { + pub async fn recover_local_delete_transaction( + &self, + disk_ref: &str, + volume: &str, + operation: DeleteRecoveryRpcOperation, + ) -> DiskResult { + validate_bootstrap_volume(volume)?; + recover_local_delete_transaction_with_ctx(&self.ctx, disk_ref, volume, operation, None).await + } + + pub async fn guarded_local_delete_versions( + &self, + disk_ref: &str, + volume: &str, + payload: GuardedDeleteVersionsPayload, + ) -> DiskResult>> { + validate_bootstrap_volume(volume)?; + guarded_local_delete_versions_with_ctx(&self.ctx, disk_ref, volume, payload, None).await + } + pub async fn begin_legacy_local_file_write( &self, disk: &DiskStore, @@ -202,6 +228,24 @@ fn validate_artifact_repair_target(target: (&str, &str), fi: &FileInfo, disk: Uu } impl ECStore { + pub async fn recover_local_delete_transaction( + &self, + disk_ref: &str, + volume: &str, + operation: DeleteRecoveryRpcOperation, + ) -> DiskResult { + recover_local_delete_transaction_with_ctx(&self.ctx, disk_ref, volume, operation, Some(self)).await + } + + pub async fn guarded_local_delete_versions( + &self, + disk_ref: &str, + volume: &str, + payload: GuardedDeleteVersionsPayload, + ) -> DiskResult>> { + guarded_local_delete_versions_with_ctx(&self.ctx, disk_ref, volume, payload, Some(self)).await + } + pub async fn begin_legacy_local_file_write( &self, disk: &DiskStore, @@ -496,6 +540,13 @@ impl ECStore { Ok(expected_disk) } + /// Resolve a quota lease against this request's captured storage instance. + pub async fn quota_mutation_fence_disk(&self, disk_ref: &str) -> DiskResult { + let (disk, id) = local_disk_candidate(&self.ctx, disk_ref).await?; + let _owner = admit_local_disk(&self.ctx, &disk, id, false).await?; + Ok(disk) + } + pub async fn write_local_metadata_at_incarnation( self: &Arc, disk_ref: &str, @@ -692,6 +743,104 @@ impl ECStore { } } +async fn recover_local_delete_transaction_with_ctx( + ctx: &Arc, + disk_ref: &str, + volume: &str, + operation: DeleteRecoveryRpcOperation, + authority: Option<&ECStore>, +) -> DiskResult { + match operation { + DeleteRecoveryRpcOperation::ApplyVersion { + object, + file_info, + force_del_marker, + mut opts, + } => { + crate::disk::validate_delete_recovery_options(&opts)?; + if object.starts_with('/') { + return Err(DiskError::InvalidPath); + } + let write = begin_legacy_local_metadata_mutation_with_ctx(ctx, disk_ref, volume, &object).await?; + // Classify only after this exact physical volume is admitted. The + // capability retains the same owner through publication and its tail. + if let Some(store) = authority { + store.reject_legacy_artifact_mutation(volume).await?; + } + opts.recoverable_delete = true; + write.stage_delete_version(*file_info, force_del_marker, opts).await?; + Ok(DeleteRecoveryRpcReply::AppliedVersion) + } + DeleteRecoveryRpcOperation::ApplyVersions { versions, mut opts } => { + crate::disk::validate_delete_recovery_options(&opts)?; + if versions.iter().any(|version| version.name.starts_with('/')) { + return Err(DiskError::InvalidPath); + } + let write = begin_legacy_local_delete_versions_with_ctx(ctx, disk_ref, volume).await?; + if let Some(store) = authority { + store.reject_legacy_artifact_mutation(volume).await?; + } + opts.recoverable_delete = true; + let errors = write.delete_versions(versions, opts).await; + Ok(DeleteRecoveryRpcReply::AppliedVersions( + errors.into_iter().map(|error| error.map(|error| error.to_u32())).collect(), + )) + } + DeleteRecoveryRpcOperation::Settle { + object, + transaction, + expected_fingerprint, + action, + } => { + let write = begin_legacy_local_metadata_mutation_with_ctx(ctx, disk_ref, volume, &object).await?; + if let Some(store) = authority { + store.reject_legacy_artifact_mutation(volume).await?; + } + write + .settle_delete_transaction(transaction, expected_fingerprint, action) + .await + .map(DeleteRecoveryRpcReply::Settled) + } + operation => { + let disk = local_mutation_disk(ctx, disk_ref).await?; + match operation { + DeleteRecoveryRpcOperation::Exists { object } => disk + .has_delete_transaction_evidence(volume, &object) + .await + .map(|present| DeleteRecoveryRpcReply::Evidence { present }), + DeleteRecoveryRpcOperation::InspectOrphanDataDir { path } => disk + .is_plain_orphan_data_dir(volume, &path) + .await + .map(|plain| DeleteRecoveryRpcReply::OrphanDataDir { plain }), + DeleteRecoveryRpcOperation::Discover { object, cursor, limit } => disk + .discover_delete_transactions(volume, object.as_deref(), cursor, limit) + .await + .map(DeleteRecoveryRpcReply::Discovered), + DeleteRecoveryRpcOperation::Inspect { object, transaction } => disk + .inspect_delete_transaction(volume, &object, transaction) + .await + .map(DeleteRecoveryRpcReply::Inspected), + _ => Err(DiskError::Unexpected), + } + } + } +} + +async fn guarded_local_delete_versions_with_ctx( + ctx: &Arc, + disk_ref: &str, + volume: &str, + payload: GuardedDeleteVersionsPayload, + authority: Option<&ECStore>, +) -> DiskResult>> { + payload.validate()?; + let write = begin_legacy_local_delete_versions_with_ctx(ctx, disk_ref, volume).await?; + if let Some(store) = authority { + store.reject_legacy_artifact_mutation(volume).await?; + } + Ok(write.delete_versions_guarded(payload.versions, payload.opts).await) +} + async fn local_mutation_disk(ctx: &Arc, disk_ref: &str) -> DiskResult { let (disk, id) = local_disk_candidate(ctx, disk_ref).await?; admit_local_disk(ctx, &disk, id, false).await?; @@ -777,15 +926,38 @@ async fn rename_local_data_with_ctx( destination: (&str, &str), mut guards: RenameDataGuards, ) -> DiskResult { + let handler_started = rustfs_io_metrics::put_stage_timer(); + let lookup_started = rustfs_io_metrics::put_stage_timer(); let (disk, disk_id) = local_disk_candidate(ctx, disk_ref).await?; + let lookup_ms = lookup_started.map(|started| started.elapsed().as_secs_f64() * 1000.0); + if let Some(duration_ms) = lookup_ms { + rustfs_io_metrics::record_put_object_stage_duration(PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOOKUP, duration_ms); + } let mutates_namespace = !is_meta_bucketname(source.0) || !is_meta_bucketname(destination.0); + let admit_started = rustfs_io_metrics::put_stage_timer(); let owner = admit_local_disk(ctx, &disk, disk_id, mutates_namespace).await?; + let admit_ms = admit_started.map(|started| started.elapsed().as_secs_f64() * 1000.0); + if let Some(duration_ms) = admit_ms { + rustfs_io_metrics::record_put_object_stage_duration(PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_ADMIT, duration_ms); + } guards.namespace_owner = owner.as_ref().map(|owner| owner.clone() as Arc); + let local_rename_started = rustfs_io_metrics::put_stage_timer(); let result = disk .rename_data_borrowed_with_fence_observed(source.0, source.1, fi, destination.0, destination.1, guards) .await .result; + let local_rename_ms = local_rename_started.map(|started| started.elapsed().as_secs_f64() * 1000.0); + if let Some(duration_ms) = local_rename_ms { + rustfs_io_metrics::record_put_object_stage_duration(PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOCAL_RENAME, duration_ms); + } drop(owner); + if let Some(handler_started) = handler_started { + let handler_ms = handler_started.elapsed().as_secs_f64() * 1000.0; + rustfs_io_metrics::record_put_object_stage_duration(PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_TOTAL, handler_ms); + let child_ms = lookup_ms.unwrap_or(0.0) + admit_ms.unwrap_or(0.0) + local_rename_ms.unwrap_or(0.0); + let overhead_ms = (handler_ms - child_ms).max(0.0); + rustfs_io_metrics::record_put_object_stage_duration(PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_OVERHEAD, overhead_ms); + } result } @@ -2521,7 +2693,9 @@ mod tests { #[serial_test::serial] async fn target_ordinary_timeout_keeps_its_physical_namespace_owner() { use crate::disk::os::prepared_publication_test_hooks as hooks; - temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, Some("1"))], async { + // Allow real fixture I/O to finish under load. Expire the operation + // with virtual time only after its physical publication is paused. + temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, Some("60"))], async { let ctx = Arc::new(InstanceContext::new()); let store = super::super::tests::build_store_with_ctx(ctx.clone()); let root = tempfile::tempdir().expect("root"); @@ -2555,7 +2729,7 @@ mod tests { .await .expect("bounded entry"); tokio::time::pause(); - tokio::time::advance(std::time::Duration::from_secs(2)).await; + tokio::time::advance(std::time::Duration::from_secs(61)).await; tokio::time::resume(); let result = tokio::time::timeout(std::time::Duration::from_secs(5), &mut rename) .await diff --git a/crates/ecstore/src/store/rebalance.rs b/crates/ecstore/src/store/rebalance.rs index e55cc5fdc..6a0b69ebb 100644 --- a/crates/ecstore/src/store/rebalance.rs +++ b/crates/ecstore/src/store/rebalance.rs @@ -125,9 +125,8 @@ async fn pause_prepared_pool_read_before_refetch(object: &str) { #[cfg(test)] use support::resolve_latest_object_info_candidates; use support::{ - LatestObjectInfoCandidate, PoolErr, PoolObjInfo, RebalanceDeletePoolResult, pool_lookup_not_found_error, - rebalance_disk_set_lookup_error, resolve_latest_object_info_candidates_with_pool_state, - resolve_rebalance_delete_from_all_pools_result, resolve_rebalance_delete_from_all_pools_results, + LatestObjectInfoCandidate, PoolErr, PoolObjInfo, pool_lookup_not_found_error, rebalance_disk_set_lookup_error, + resolve_latest_object_info_candidates_with_pool_state, resolve_rebalance_delete_from_all_pools_results, resolve_store_rebalance_pool_meta_reload_result, validate_prepared_pool_refetch_identity, }; @@ -996,42 +995,27 @@ impl ECStore { let mut results = Vec::with_capacity(errs.len()); - for pe in errs.iter() { - if let Some(err) = &pe.err - && err == &StorageError::ErasureWriteQuorum - { - if let Some(idx) = pe.index { - results.push(RebalanceDeletePoolResult { - pool_idx: idx, - result: Err(StorageError::ErasureWriteQuorum), - }); - } + for pe in &errs { + let Some(idx) = pe.index else { continue }; + if pe.err.as_ref() == Some(&StorageError::ErasureWriteQuorum) { + results.push(Err(StorageError::ErasureWriteQuorum)); continue; } - - if let Some(idx) = pe.index { - let pool = self.pools[idx].clone(); - results.push(RebalanceDeletePoolResult { - pool_idx: idx, - result: self - .run_external_decommission_capacity_object_delete( - idx, - bucket, - object, - object, - opts.clone(), - |opts| async move { pool.delete_object(bucket, object, opts).await }, - ) - .await, - }); - } + let pool = self.pools[idx].clone(); + results.push( + self.run_external_decommission_capacity_object_delete( + idx, + bucket, + object, + object, + opts.clone(), + |opts| async move { pool.delete_object(bucket, object, opts).await }, + ) + .await, + ); } - resolve_rebalance_delete_from_all_pools_result( - resolve_rebalance_delete_from_all_pools_results(results, bucket, object), - bucket, - object, - ) + resolve_rebalance_delete_from_all_pools_results(results, bucket, object) } /// Peer reload entry: refreshes in-memory pool metadata from the shared @@ -2460,56 +2444,22 @@ mod tests { assert!(err_message.contains(&Error::SlowDown.to_string())); } - #[test] - fn resolve_rebalance_delete_from_all_pools_result_passthrough_ok() { - let info = ObjectInfo { - bucket: "bucket".to_string(), - name: "object".to_string(), - ..Default::default() - }; - - let resolved = resolve_rebalance_delete_from_all_pools_result(Ok(info.clone()), "bucket", "object") - .expect("successful rebalance delete should pass through"); - - assert_eq!(resolved.bucket, info.bucket); - assert_eq!(resolved.name, info.name); - } - - #[test] - fn resolve_rebalance_delete_from_all_pools_result_wraps_object_context() { - let err = resolve_rebalance_delete_from_all_pools_result(Err(Error::SlowDown), "bucket", "object") - .expect_err("failed rebalance delete should be wrapped"); - let rendered = err.to_string(); - - assert!(rendered.contains("failed to delete rebalance source object bucket/object"), "{rendered}"); - assert!(rendered.contains(&Error::SlowDown.to_string()), "{rendered}"); - } - #[test] fn resolve_rebalance_delete_from_all_pools_results_fails_on_later_pool_error() { let err = resolve_rebalance_delete_from_all_pools_results( vec![ - RebalanceDeletePoolResult { - pool_idx: 0, - result: Ok(ObjectInfo { - bucket: "bucket".to_string(), - name: "object".to_string(), - ..Default::default() - }), - }, - RebalanceDeletePoolResult { - pool_idx: 1, - result: Err(Error::SlowDown), - }, + Ok(ObjectInfo { + bucket: "bucket".to_string(), + name: "object".to_string(), + ..Default::default() + }), + Err(Error::SlowDown), ], "bucket", "object", ) .expect_err("non-ignorable errors from later pools must not be hidden"); - let rendered = err.to_string(); - - assert!(rendered.contains("pool 1 delete failed for bucket/object"), "{rendered}"); - assert!(rendered.contains(&Error::SlowDown.to_string()), "{rendered}"); + assert_eq!(err, Error::SlowDown); } #[test] @@ -2522,14 +2472,8 @@ mod tests { let resolved = resolve_rebalance_delete_from_all_pools_results( vec![ - RebalanceDeletePoolResult { - pool_idx: 0, - result: Ok(info.clone()), - }, - RebalanceDeletePoolResult { - pool_idx: 1, - result: Err(Error::ObjectNotFound("bucket".to_string(), "object".to_string())), - }, + Ok(info.clone()), + Err(Error::ObjectNotFound("bucket".to_string(), "object".to_string())), ], "bucket", "object", @@ -2550,14 +2494,8 @@ mod tests { let resolved = resolve_rebalance_delete_from_all_pools_results( vec![ - RebalanceDeletePoolResult { - pool_idx: 0, - result: Err(Error::ObjectNotFound("bucket".to_string(), "object".to_string())), - }, - RebalanceDeletePoolResult { - pool_idx: 1, - result: Ok(info.clone()), - }, + Err(Error::ObjectNotFound("bucket".to_string(), "object".to_string())), + Ok(info.clone()), ], "bucket", "object", @@ -2572,50 +2510,35 @@ mod tests { fn resolve_rebalance_delete_from_all_pools_results_fails_when_all_results_are_ignored_errors() { let err = resolve_rebalance_delete_from_all_pools_results( vec![ - RebalanceDeletePoolResult { - pool_idx: 0, - result: Err(Error::ObjectNotFound("bucket".to_string(), "object".to_string())), - }, - RebalanceDeletePoolResult { - pool_idx: 1, - result: Err(Error::VersionNotFound("bucket".to_string(), "object".to_string(), "vid-1".to_string())), - }, + Err(Error::ObjectNotFound("bucket".to_string(), "object".to_string())), + Err(Error::VersionNotFound("bucket".to_string(), "object".to_string(), "vid-1".to_string())), ], "bucket", "object", ) .expect_err("all ignored errors without any successful delete should still fail"); - let rendered = err.to_string(); - - assert!(rendered.contains("pool 1 delete failed for bucket/object"), "{rendered}"); - assert!(rendered.contains("Version not found"), "{rendered}"); + assert_eq!( + err, + Error::VersionNotFound("bucket".to_string(), "object".to_string(), "vid-1".to_string()) + ); } #[test] fn resolve_rebalance_delete_from_all_pools_results_fails_on_write_quorum_even_with_success() { let err = resolve_rebalance_delete_from_all_pools_results( vec![ - RebalanceDeletePoolResult { - pool_idx: 0, - result: Ok(ObjectInfo { - bucket: "bucket".to_string(), - name: "object".to_string(), - ..Default::default() - }), - }, - RebalanceDeletePoolResult { - pool_idx: 1, - result: Err(Error::ErasureWriteQuorum), - }, + Ok(ObjectInfo { + bucket: "bucket".to_string(), + name: "object".to_string(), + ..Default::default() + }), + Err(Error::ErasureWriteQuorum), ], "bucket", "object", ) .expect_err("write quorum failures must fail the aggregate delete"); - let rendered = err.to_string(); - - assert!(rendered.contains("pool 1 delete failed for bucket/object"), "{rendered}"); - assert!(rendered.contains(&Error::ErasureWriteQuorum.to_string()), "{rendered}"); + assert_eq!(err, Error::ErasureWriteQuorum); } #[test] diff --git a/crates/ecstore/src/store/rebalance/support.rs b/crates/ecstore/src/store/rebalance/support.rs index 376ac05b9..13be3844c 100644 --- a/crates/ecstore/src/store/rebalance/support.rs +++ b/crates/ecstore/src/store/rebalance/support.rs @@ -46,15 +46,10 @@ impl Clone for PoolObjInfo { } } -pub(super) struct LatestObjectInfoCandidate { - pub(super) info: Option, - pub(super) idx: usize, - pub(super) err: Option, -} - -pub(super) struct RebalanceDeletePoolResult { - pub(super) pool_idx: usize, - pub(super) result: Result, +pub(in crate::store) struct LatestObjectInfoCandidate { + pub(in crate::store) info: Option, + pub(in crate::store) idx: usize, + pub(in crate::store) err: Option, } pub(super) fn pool_lookup_not_found_error(bucket: &str, object: &str, opts: &ObjectOptions) -> Error { @@ -71,53 +66,30 @@ pub(super) fn resolve_store_rebalance_pool_meta_reload_result(result: Result< result.map_err(|err| Error::other(format!("store rebalance pool meta reload failed during {stage}: {err}"))) } -pub(super) fn resolve_rebalance_delete_from_all_pools_result( - result: Result, - bucket: &str, - object: &str, -) -> Result { - result.map_err(|err| { - if matches!(&err, Error::PreconditionFailed | Error::PrefixAccessDenied(_, _)) { - err - } else { - Error::other(format!("failed to delete rebalance source object {bucket}/{object}: {err}")) - } - }) -} - fn is_ignorable_rebalance_delete_error(err: &Error) -> bool { is_err_object_not_found(err) || is_err_version_not_found(err) } -fn rebalance_delete_pool_error(pool_idx: usize, bucket: &str, object: &str, err: Error) -> Error { - if matches!(&err, Error::PreconditionFailed | Error::PrefixAccessDenied(_, _)) { - err - } else { - Error::other(format!("pool {pool_idx} delete failed for {bucket}/{object}: {err}")) - } -} - pub(super) fn resolve_rebalance_delete_from_all_pools_results( - results: Vec, + results: Vec>, bucket: &str, object: &str, ) -> Result { let mut deleted = None; let mut ignored_error = None; - for pool_result in results { - let pool_idx = pool_result.pool_idx; - match pool_result.result { + for result in results { + match result { Ok(info) => { if deleted.is_none() { deleted = Some(info); } } Err(err) if is_ignorable_rebalance_delete_error(&err) => { - ignored_error = Some((pool_idx, err)); + ignored_error = Some(err); } Err(err) => { - return Err(rebalance_delete_pool_error(pool_idx, bucket, object, err)); + return Err(err); } } } @@ -126,8 +98,8 @@ pub(super) fn resolve_rebalance_delete_from_all_pools_results( return Ok(info); } - if let Some((pool_idx, err)) = ignored_error { - return Err(rebalance_delete_pool_error(pool_idx, bucket, object, err)); + if let Some(err) = ignored_error { + return Err(err); } Err(Error::other(format!( @@ -295,7 +267,7 @@ pub(super) fn resolve_latest_object_info_candidates( resolve_latest_object_info_candidates_with_pool_state(candidates, &[], bucket, object, opts) } -pub(super) fn resolve_latest_object_info_candidates_with_pool_state( +pub(in crate::store) fn resolve_latest_object_info_candidates_with_pool_state( candidates: Vec, suspended_pools: &[bool], bucket: &str, @@ -377,52 +349,56 @@ mod tests { } #[test] - fn rebalance_delete_result_preserves_precondition_failed() { - let err = resolve_rebalance_delete_from_all_pools_result(Err(Error::PreconditionFailed), "bucket", "object") - .expect_err("precondition failure should remain structured"); - - assert_eq!(err, Error::PreconditionFailed); + fn rebalance_delete_aggregation_preserves_typed_failures() { + for expected in [ + Error::UnsupportedDisk, + Error::GuardedDeleteUnsupported, + Error::ErasureReadQuorum, + Error::ErasureWriteQuorum, + Error::SlowDown, + Error::PreconditionFailed, + Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned()), + Error::FileCorrupt, + ] { + for results in [ + vec![Err(expected.clone())], + vec![Ok(ObjectInfo::default()), Err(expected.clone())], + vec![Err(expected.clone()), Ok(ObjectInfo::default())], + vec![ + Err(Error::ObjectNotFound("bucket".to_owned(), "object".to_owned())), + Err(expected.clone()), + ], + ] { + let actual = resolve_rebalance_delete_from_all_pools_results(results, "bucket", "object") + .expect_err("a pool failure must retain its S3 error classification"); + assert_eq!(actual, expected); + } + } } #[test] - fn rebalance_delete_result_preserves_prefix_access_denied() { - let err = resolve_rebalance_delete_from_all_pools_result( - Err(Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned())), - "bucket", - "object", - ) - .expect_err("prefix access denial should remain structured"); - - assert_eq!(err, Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned())); + fn rebalance_delete_aggregation_preserves_missing_identity() { + for expected in [ + Error::ObjectNotFound("bucket".to_owned(), "object".to_owned()), + Error::VersionNotFound("bucket".to_owned(), "object".to_owned(), "version".to_owned()), + ] { + let actual = resolve_rebalance_delete_from_all_pools_results(vec![Err(expected.clone())], "bucket", "object") + .expect_err("all-missing deletes must preserve their typed object or version identity"); + assert_eq!(actual, expected); + } } #[test] - fn rebalance_delete_pool_result_preserves_precondition_failed() { - let err = resolve_rebalance_delete_from_all_pools_results( - vec![RebalanceDeletePoolResult { - pool_idx: 0, - result: Err(Error::PreconditionFailed), - }], - "bucket", - "object", - ) - .expect_err("precondition failure should remain structured"); - - assert_eq!(err, Error::PreconditionFailed); - } - - #[test] - fn rebalance_delete_pool_result_preserves_prefix_access_denied() { - let err = resolve_rebalance_delete_from_all_pools_results( - vec![RebalanceDeletePoolResult { - pool_idx: 0, - result: Err(Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned())), - }], - "bucket", - "object", - ) - .expect_err("prefix access denial should remain structured"); - - assert_eq!(err, Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned())); + fn rebalance_delete_aggregation_rejects_no_attempted_pools() { + let error = resolve_rebalance_delete_from_all_pools_results(Vec::new(), "bucket", "object") + .expect_err("an empty attempt set cannot report a successful delete"); + let Error::Io(error) = error else { + panic!("an empty attempt set must remain an internal error"); + }; + assert_eq!(error.kind(), std::io::ErrorKind::Other); + assert_eq!( + error.to_string(), + "failed to delete rebalance source object bucket/object: no pools were attempted" + ); } } diff --git a/crates/ecstore/tests/cleanup_isolation_test.rs b/crates/ecstore/tests/cleanup_isolation_test.rs new file mode 100644 index 000000000..663681445 --- /dev/null +++ b/crates/ecstore/tests/cleanup_isolation_test.rs @@ -0,0 +1,174 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![cfg(target_os = "linux")] + +use bytes::Bytes; +mod storage_api; + +use std::time::Duration; +use storage_api::cleanup_isolation::{DeleteOptions, DiskAPI, DiskError, DiskOption, Endpoint, new_disk}; + +#[tokio::test] +async fn cleanup_disk_and_periodic_gc_use_affinity_without_changing_foreground() { + const CHILD: &str = "RUSTFS_CLEANUP_LINUX_TEST_CHILD"; + let foreground = rustix::thread::sched_getaffinity(None).unwrap(); + let cpu = (0..rustix::thread::CpuSet::MAX_CPU) + .find(|cpu| foreground.is_set(*cpu)) + .expect("the test process must have an allowed CPU"); + if std::env::var_os(CHILD).is_none() { + for mode in ["custom", "default", "none", "restricted"] { + if mode == "default" && (!foreground.is_set(1) || !foreground.is_set(2)) { + continue; + } + let output = tokio::task::spawn_blocking(move || { + let mut command = std::process::Command::new(std::env::current_exe().unwrap()); + command + .args([ + "--exact", + "cleanup_disk_and_periodic_gc_use_affinity_without_changing_foreground", + "--nocapture", + ]) + .env(CHILD, mode) + .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true") + .env("RUSTFS_PUT_RENAME_TAIL_CLEANUP_MAX_PENDING", "4") + .env("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKERS", "1") + .env("RUSTFS_CLEANUP_DISK_MAX_PENDING", "4") + .env("RUSTFS_CLEANUP_DISK_WORKERS", "1") + .env("RUSTFS_CLEANUP_GC_WORKERS", "1") + .env("RUSTFS_CLEANUP_BLOCKING_THREADS", "1") + .env("RUSTFS_CLEANUP_ASYNC_THREADS", "1") + .env_remove("RUSTFS_CLEANUP_CPUS") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP"); + match mode { + "custom" => { + command.env("RUSTFS_CLEANUP_CPUS", cpu.to_string()); + } + "none" => { + command.env("RUSTFS_CLEANUP_CPUS", "none"); + } + _ => {} + } + command.output().unwrap() + }) + .await + .unwrap(); + assert!( + output.status.success(), + "{}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed")); + } + return; + } + + let mode = std::env::var(CHILD).unwrap(); + if mode == "restricted" { + let mut restricted = rustix::thread::CpuSet::new(); + restricted.set(cpu); + rustix::thread::sched_setaffinity(None, &restricted).expect("restrict initialization to one allowed CPU"); + } + let dir = tempfile::tempdir().unwrap(); + let disk = new_disk(&Endpoint::try_from(dir.path().to_str().unwrap()).unwrap(), &DiskOption::default()) + .await + .unwrap(); + let bucket = "cleanup-affinity"; + disk.make_volume(bucket).await.unwrap(); + disk.write_all(bucket, "old/part.1", Bytes::from_static(b"garbage")) + .await + .unwrap(); + let receipt = disk + .write_all(bucket, "old/.rustfs-old-data-cleanup-receipt.json", Bytes::from_static(b"receipt")) + .await; + if mode == "restricted" { + assert!( + receipt + .expect_err("unavailable default CPUs must reject cleanup") + .to_string() + .contains("allowed CPU set") + ); + assert_eq!(disk.read_all(bucket, "old/part.1").await.unwrap(), Bytes::from_static(b"garbage")); + return; + } + receipt.unwrap(); + let expected_mask = match mode.as_str() { + "default" => "1-2".to_owned(), + "none" => std::fs::read_to_string("/proc/thread-self/status") + .unwrap() + .lines() + .find_map(|line| line.strip_prefix("Cpus_allowed_list:")) + .unwrap() + .trim() + .to_owned(), + _ => cpu.to_string(), + }; + disk.write_all(bucket, "live/part.1", Bytes::from_static(b"keep")) + .await + .unwrap(); + disk.delete( + bucket, + "old", + DeleteOptions { + recursive: true, + ..Default::default() + }, + ) + .await + .unwrap(); + assert!(matches!(disk.read_all(bucket, "old/part.1").await, Err(DiskError::FileNotFound))); + + // One async worker and one lazily started blocking worker. Inspect the + // real process threads, including the threads executing filesystem work. + let mut cleanup_threads = 0; + for entry in std::fs::read_dir("/proc/self/task").unwrap() { + let task = entry.unwrap().path(); + let Ok(name) = std::fs::read_to_string(task.join("comm")) else { + continue; + }; + if name.trim() != "rustfs-cleanup" { + continue; + } + cleanup_threads += 1; + let status = std::fs::read_to_string(task.join("status")).unwrap(); + let mask = status + .lines() + .find_map(|line| line.strip_prefix("Cpus_allowed_list:")) + .unwrap() + .trim(); + assert_eq!(mask, expected_mask); + } + assert_eq!(cleanup_threads, 2, "async and blocking workers must both use the configured runtime"); + assert_eq!(rustix::thread::sched_getaffinity(None).unwrap(), foreground); + + let trash = dir.path().join(".rustfs.sys/tmp/.trash"); + assert!(std::fs::read_dir(&trash).unwrap().next().is_some()); + // Advance the foreground timer that dispatches the five-minute periodic + // scan; the dedicated runtime then performs the actual physical removal. + tokio::time::pause(); + tokio::time::advance(Duration::from_secs(301)).await; + tokio::time::resume(); + tokio::time::timeout(Duration::from_secs(20), async { + while std::fs::read_dir(&trash).unwrap().next().is_some() { + tokio::time::sleep(Duration::from_millis(5)).await; + } + }) + .await + .expect("periodic GC must physically remove trash"); + assert_eq!(disk.read_all(bucket, "live/part.1").await.unwrap(), Bytes::from_static(b"keep")); + assert_eq!(rustix::thread::sched_getaffinity(None).unwrap(), foreground); +} diff --git a/crates/ecstore/tests/fd_relative_io_test.rs b/crates/ecstore/tests/fd_relative_io_test.rs new file mode 100644 index 000000000..fa19deb52 --- /dev/null +++ b/crates/ecstore/tests/fd_relative_io_test.rs @@ -0,0 +1,283 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![cfg(all(target_os = "linux", feature = "test-util"))] + +mod storage_api; + +use bytes::Bytes; +use rustfs_filemeta::{FileInfo, ObjectPartInfo}; +use std::{path::Path, time::Duration}; +use storage_api::fd_relative::{ + DiskAPI, DiskError, DiskOption, DiskStore, Endpoint, LocalPublicationPause, LocalPublicationStage, ReadOptions, new_disk, +}; +use time::OffsetDateTime; +use uuid::Uuid; + +const RUSTFS_META_TMP_BUCKET: &str = ".rustfs.sys/tmp"; + +async fn disk_at(path: &Path) -> DiskStore { + new_disk( + &Endpoint::try_from(path.to_str().expect("UTF-8 test path")).expect("endpoint"), + &DiskOption::default(), + ) + .await + .expect("local disk") +} + +fn metadata(data: Bytes, inline: bool, directory: Uuid) -> FileInfo { + let mut fi = FileInfo::new("object", 1, 0); + fi.erasure.index = 1; + fi.version_id = Some(Uuid::nil()); + fi.data_dir = Some(directory); + fi.size = i64::try_from(data.len()).expect("test data size"); + fi.parts = vec![ObjectPartInfo { + number: 1, + size: data.len(), + actual_size: fi.size, + ..Default::default() + }]; + fi.mod_time = Some(OffsetDateTime::now_utc()); + fi.data = inline.then_some(data); + if inline { + fi.set_inline_data(); + } + fi +} + +async fn put(disk: &DiskStore, staging: &str, data: Bytes, inline: bool) -> Uuid { + let directory = Uuid::new_v4(); + if !inline { + disk.write_all(RUSTFS_META_TMP_BUCKET, &format!("{staging}/{directory}/part.1"), data.clone()) + .await + .expect("stage shard"); + } + disk.rename_data(RUSTFS_META_TMP_BUCKET, staging, metadata(data, inline, directory), "bucket", "object") + .await + .expect("commit object"); + directory +} + +#[tokio::test] +async fn pinned_root_reads_and_writes_original_disk_after_path_replacement() { + let temp = tempfile::tempdir().expect("test directory"); + let root = temp.path().join("disk"); + std::fs::create_dir(&root).expect("disk root"); + let disk = disk_at(&root).await; + disk.make_volume("bucket").await.expect("bucket"); + disk.write_all("bucket", "metadata", Bytes::from_static(b"original")) + .await + .expect("initial write"); + let moved = temp.path().join("original"); + std::fs::rename(&root, &moved).expect("move mount path"); + std::fs::create_dir_all(root.join("bucket")).expect("replacement disk"); + std::fs::write(root.join("bucket/metadata"), b"replacement").expect("replacement sentinel"); + assert_eq!( + disk.read_all("bucket", "metadata").await.expect("pinned read"), + Bytes::from_static(b"original") + ); + disk.write_all("bucket", "metadata", Bytes::from_static(b"updated")) + .await + .expect("pinned write"); + assert_eq!(std::fs::read(moved.join("bucket/metadata")).expect("original disk"), b"updated"); + assert_eq!(std::fs::read(root.join("bucket/metadata")).expect("replacement disk"), b"replacement"); +} + +#[tokio::test] +async fn relative_reads_and_writes_reject_symlink_escape() { + let temp = tempfile::tempdir().expect("test directory"); + let disk = disk_at(temp.path()).await; + disk.make_volume("bucket").await.expect("bucket"); + let outside = tempfile::tempdir().expect("outside directory"); + std::fs::write(outside.path().join("sentinel"), b"preserved").expect("sentinel"); + std::os::unix::fs::symlink(outside.path(), temp.path().join("bucket/link")).expect("directory link"); + std::os::unix::fs::symlink(outside.path().join("sentinel"), temp.path().join("bucket/file-link")).expect("file link"); + for path in ["link/sentinel", "file-link"] { + assert!(matches!(disk.read_all("bucket", path).await, Err(DiskError::InvalidPath))); + assert!(matches!( + disk.write_all("bucket", path, Bytes::from_static(b"unsafe")).await, + Err(DiskError::InvalidPath) + )); + } + assert_eq!(std::fs::read(outside.path().join("sentinel")).expect("outside bytes"), b"preserved"); +} + +#[tokio::test] +async fn inline_and_non_inline_overwrites_remain_readable_after_reopen() { + for inline in [true, false] { + let temp = tempfile::tempdir().expect("test directory"); + let disk = disk_at(temp.path()).await; + disk.make_volume("bucket").await.expect("bucket"); + put(&disk, "first", Bytes::from_static(b"old bytes"), inline).await; + let directory = put(&disk, "second", Bytes::from_static(b"new bytes"), inline).await; + drop(disk); + let disk = disk_at(temp.path()).await; + let fi = disk + .read_version( + "bucket", + "bucket", + "object", + "", + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("committed metadata after reopen"); + assert_eq!(fi.data_dir, Some(directory)); + if inline { + assert_eq!(fi.data, Some(Bytes::from_static(b"new bytes"))); + } else { + assert_eq!( + disk.read_all("bucket", &format!("object/{directory}/part.1")) + .await + .expect("shard bytes"), + Bytes::from_static(b"new bytes") + ); + } + } +} + +#[tokio::test] +async fn rename_rejects_symlinked_destination_metadata() { + for inline in [true, false] { + let temp = tempfile::tempdir().expect("disk directory"); + let outside = tempfile::tempdir().expect("outside directory"); + let disk = disk_at(temp.path()).await; + disk.make_volume("bucket").await.expect("bucket"); + put(&disk, "seed", Bytes::from_static(b"old object"), inline).await; + let destination = temp.path().join("bucket/object/xl.meta"); + let original = std::fs::read(&destination).expect("valid existing metadata"); + let external_metadata = outside.path().join("xl.meta"); + std::fs::write(&external_metadata, &original).expect("external valid metadata"); + std::fs::remove_file(&destination).expect("replace destination metadata"); + std::os::unix::fs::symlink(&external_metadata, &destination).expect("metadata escape link"); + + let directory = Uuid::new_v4(); + let data = Bytes::from_static(b"replacement"); + if !inline { + disk.write_all(RUSTFS_META_TMP_BUCKET, &format!("replacement/{directory}/part.1"), data.clone()) + .await + .expect("stage non-inline data"); + } + let result = disk + .rename_data( + RUSTFS_META_TMP_BUCKET, + "replacement", + metadata(data, inline, directory), + "bucket", + "object", + ) + .await; + assert!( + matches!(result, Err(DiskError::InvalidPath)), + "symlinked metadata must be rejected: {result:?}" + ); + assert_eq!(std::fs::read_link(&destination).expect("link must remain"), external_metadata); + assert_eq!(std::fs::read(&external_metadata).expect("external metadata must survive"), original); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn prepared_publication_rejects_source_or_directory_replacement() { + for replace_directory in [false, true] { + let temp = tempfile::tempdir().expect("test directory"); + let disk = disk_at(temp.path()).await; + disk.make_volume("bucket").await.expect("bucket"); + put(&disk, "first", Bytes::from_static(b"old bytes"), true).await; + let old = std::fs::read(temp.path().join("bucket/object/xl.meta")).expect("old metadata"); + let mut pause = LocalPublicationPause::install(&disk, "bucket", "object/xl.meta", LocalPublicationStage::PreparedRename) + .expect("publication pause"); + let writer = disk.clone(); + let task = tokio::spawn(async move { + writer + .rename_data( + RUSTFS_META_TMP_BUCKET, + "second", + metadata(Bytes::from_static(b"new bytes"), true, Uuid::new_v4()), + "bucket", + "object", + ) + .await + }); + tokio::time::timeout(Duration::from_secs(30), pause.entered()) + .await + .expect("publication reached pause") + .expect("pause entered"); + if replace_directory { + std::fs::rename(temp.path().join("bucket/object"), temp.path().join("detached")).expect("detach object"); + std::fs::create_dir(temp.path().join("bucket/object")).expect("replacement directory"); + std::fs::write(temp.path().join("bucket/object/xl.meta"), &old).expect("replacement metadata"); + } else { + let src = temp.path().join(RUSTFS_META_TMP_BUCKET).join("second/xl.meta"); + std::fs::rename(&src, src.with_extension("detached")).expect("detach staged metadata"); + std::fs::write(src, b"unrelated").expect("replacement staged file"); + } + drop(pause); + assert!(task.await.expect("writer task").is_err()); + assert_eq!( + std::fs::read(temp.path().join("bucket/object/xl.meta")).expect("old metadata survives"), + old + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn cancelled_waiter_keeps_publication_serialized_until_physical_completion() { + let temp = tempfile::tempdir().expect("test directory"); + let disk = disk_at(temp.path()).await; + disk.make_volume("bucket").await.expect("bucket"); + put(&disk, "first", Bytes::from_static(b"old bytes"), true).await; + let mut pause = LocalPublicationPause::install(&disk, "bucket", "object/xl.meta", LocalPublicationStage::PreparedRename) + .expect("publication pause"); + let writer = disk.clone(); + let pending = tokio::spawn(async move { put(&writer, "shared-stage", Bytes::from_static(b"cancelled waiter"), true).await }); + tokio::time::timeout(Duration::from_secs(30), pause.entered()) + .await + .expect("publication reached pause") + .expect("pause entered"); + pending.abort(); + assert!(pending.await.expect_err("waiter aborted").is_cancelled()); + let writer = disk.clone(); + let (started_tx, started_rx) = tokio::sync::oneshot::channel(); + let mut retry = tokio::spawn(async move { + let _ = started_tx.send(()); + put(&writer, "shared-stage", Bytes::from_static(b"latest bytes"), true).await + }); + started_rx.await.expect("retry started"); + assert!( + tokio::time::timeout(Duration::from_millis(200), &mut retry).await.is_err(), + "a retry must not publish while the cancelled writer still owns physical IO" + ); + drop(pause); + tokio::time::timeout(Duration::from_secs(30), retry) + .await + .expect("retry completed") + .expect("retry task"); + let fi = disk + .read_version( + "bucket", + "bucket", + "object", + "", + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("latest metadata"); + assert_eq!(fi.data, Some(Bytes::from_static(b"latest bytes"))); +} diff --git a/crates/ecstore/tests/inline_heal_staging_test.rs b/crates/ecstore/tests/inline_heal_staging_test.rs new file mode 100644 index 000000000..95867d93e --- /dev/null +++ b/crates/ecstore/tests/inline_heal_staging_test.rs @@ -0,0 +1,226 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![cfg(all(feature = "test-util", not(windows)))] + +mod storage_api; + +use bytes::Bytes; +use rustfs_filemeta::{FileInfo, ObjectPartInfo}; +use std::{process::Command, time::Duration}; +use storage_api::inline_heal_staging::{ + DeleteOptions, DiskAPI, DiskError, DiskOption, Endpoint, LocalPublicationPause, LocalPublicationStage, ReadOptions, new_disk, +}; +use time::OffsetDateTime; +use uuid::Uuid; + +const TMP_BUCKET: &str = ".rustfs.sys/tmp"; +const CHILD: &str = "RUSTFS_INLINE_HEAL_STAGING_CHILD"; + +fn metadata(object: &str, version: Uuid, data_dir: Uuid, data: Bytes, inline: bool) -> FileInfo { + let mut info = FileInfo::new(object, 1, 0); + info.erasure.index = 1; + info.version_id = Some(version); + info.data_dir = Some(data_dir); + info.size = i64::try_from(data.len()).expect("fixture size"); + info.parts = vec![ObjectPartInfo { + number: 1, + size: data.len(), + actual_size: info.size, + ..Default::default() + }]; + info.mod_time = Some(OffsetDateTime::now_utc()); + if inline { + info.data = Some(data); + info.set_inline_data(); + } + info +} + +#[test] +fn inline_heal_stages_backup_in_production_durability_modes() { + if std::env::var_os(CHILD).is_none() { + // Durability is cached process-wide; use fresh production-library processes. + for mode in ["none", "relaxed"] { + let result = Command::new(std::env::current_exe().expect("integration test binary")) + .args([ + "--exact", + "inline_heal_stages_backup_in_production_durability_modes", + "--nocapture", + ]) + .env(CHILD, "1") + .env("RUSTFS_DURABILITY_MODE", mode) + .status() + .expect("run production-library durability fixture"); + assert!(result.success(), "inline staging failed in {mode} mode"); + } + return; + } + tokio::runtime::Builder::new_multi_thread() + .worker_threads(2) + .enable_all() + .build() + .expect("fixture runtime") + .block_on(async { + for version in [Uuid::nil(), Uuid::new_v4()] { + for old_inline in [true, false] { + let root = tempfile::tempdir().expect("fixture directory"); + let endpoint = Endpoint::try_from(root.path().to_str().expect("UTF-8 fixture path")).expect("endpoint"); + let disk = new_disk(&endpoint, &DiskOption::default()).await.expect("local disk"); + disk.make_volume("bucket").await.expect("bucket"); + match disk.make_volume(TMP_BUCKET).await { + Ok(()) | Err(DiskError::VolumeExists) => {} + Err(error) => panic!("temporary bucket: {error:?}"), + } + let old_dir = Uuid::new_v4(); + let old_body = Bytes::from_static(b"old bytes must remain recoverable"); + if !old_inline { + disk.write_all(TMP_BUCKET, &format!("seed/{old_dir}/part.1"), old_body.clone()) + .await + .expect("stage external source"); + } + disk.rename_data( + TMP_BUCKET, + "seed", + metadata("object", version, old_dir, old_body, old_inline), + "bucket", + "object", + ) + .await + .expect("seed old version"); + let object_dir = root.path().join("bucket/object"); + let old_metadata = tokio::fs::read(object_dir.join("xl.meta")).await.expect("old metadata"); + let new_body = Bytes::from_static(b"repaired inline bytes"); + let mut repaired = metadata("object", version, Uuid::new_v4(), new_body.clone(), true); + repaired.set_healing(); + let mut pause = + LocalPublicationPause::install(&disk, "bucket", "object/xl.meta", LocalPublicationStage::PreparedRename) + .expect("pause physical metadata publication"); + let task_disk = disk.clone(); + let task = + tokio::spawn( + async move { task_disk.rename_data(TMP_BUCKET, "heal", repaired, "bucket", "object").await }, + ); + tokio::time::timeout(Duration::from_secs(10), pause.entered()) + .await + .expect("publication must reach its physical boundary") + .expect("publication pause"); + let backup = root.path().join(TMP_BUCKET).join("heal/xl.meta.bkp"); + assert_eq!( + tokio::fs::read(&backup) + .await + .expect("production heal must stage rollback bytes"), + old_metadata + ); + assert_eq!( + tokio::fs::read(object_dir.join("xl.meta")) + .await + .expect("unpublished metadata"), + old_metadata + ); + drop(pause); + let response = tokio::time::timeout(Duration::from_secs(10), task) + .await + .expect("publication must finish") + .expect("publication task") + .expect("healing commit"); + assert!(response.rollback_data_dir.is_none(), "inline heal must not publish a live backup"); + assert_eq!(response.cleanup_data_dir.is_some(), !old_inline); + assert!(!backup.exists(), "committed staging backup must be retired"); + for entry in std::fs::read_dir(&object_dir).expect("object directory") { + let entry = entry.expect("object entry"); + if entry.file_type().expect("entry type").is_dir() { + assert!(!entry.path().join("xl.meta.bkp").exists(), "no live rollback backup may be published"); + } + } + let current = disk + .read_version( + "", + "bucket", + "object", + &version.to_string(), + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("read healed physical target"); + assert_eq!(current.data, Some(new_body)); + assert_eq!(current.version_id.unwrap_or_default(), version); + } + } + }); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn recursive_tmp_cleanup_prunes_nested_stage_without_relocking() { + for key in ["stage", "stage/child", "stage/deep/child"] { + for layout in ["absent", "empty-parent", "present"] { + for has_sibling in [false, true] { + let root = tempfile::tempdir().expect("nested cleanup fixture"); + let endpoint = Endpoint::try_from(root.path().to_str().expect("UTF-8 fixture path")).expect("endpoint"); + let disk = new_disk(&endpoint, &DiskOption::default()).await.expect("local disk"); + let temporary = root.path().join(TMP_BUCKET); + let target = temporary.join(key); + tokio::fs::create_dir_all(&temporary).await.expect("TMP bucket"); + if layout != "absent" { + tokio::fs::create_dir_all(target.parent().expect("target parent")) + .await + .expect("prepare cleanup ancestors"); + } + if layout == "present" { + disk.write_all(TMP_BUCKET, &format!("{key}/payload"), Bytes::from_static(b"removed child")) + .await + .expect("seed cleanup target"); + } + let sibling = if key == "stage" { "other/payload" } else { "stage/sibling" }; + if has_sibling { + disk.write_all(TMP_BUCKET, sibling, Bytes::from_static(b"preserved sibling")) + .await + .expect("seed sibling"); + } + for _ in 0..2 { + tokio::time::timeout( + Duration::from_secs(10), + disk.delete( + TMP_BUCKET, + key, + DeleteOptions { + recursive: true, + ..Default::default() + }, + ), + ) + .await + .expect("nested TMP cleanup must not reacquire its staging lease") + .expect("recursive TMP cleanup"); + } + assert!(!target.exists(), "cleanup target must be gone: {key}"); + assert!(temporary.is_dir(), "TMP bucket must remain"); + if has_sibling { + assert_eq!( + tokio::fs::read(temporary.join(sibling)).await.expect("sibling survives"), + b"preserved sibling" + ); + } + assert_eq!( + temporary.join("stage").exists(), + has_sibling && key != "stage", + "empty staging ancestors must be pruned: {key}" + ); + } + } + } +} diff --git a/crates/ecstore/tests/list_prefix_test.rs b/crates/ecstore/tests/list_prefix_test.rs new file mode 100644 index 000000000..214f12eb3 --- /dev/null +++ b/crates/ecstore/tests/list_prefix_test.rs @@ -0,0 +1,82 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![cfg(feature = "test-util")] + +mod storage_api; + +use std::sync::Arc; +use storage_api::contract_compat::StorageListOperations; +use storage_api::metadata_lock::{ + BucketOperations, Error, MakeBucketOptions, ObjectIO, ObjectOptions, PutObjReader, init_bucket_metadata_sys, + isolated_store_over_temp_disks, +}; + +#[tokio::test] +async fn double_slash_list_prefix_is_literal_and_empty() { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let bucket = "list-double-slash-prefix"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let key = "foo/foo/base/backup"; + let mut data = PutObjReader::from_vec(b"backup".to_vec()); + store + .put_object(bucket, key, &mut data, &ObjectOptions::default()) + .await + .expect("write control object"); + + for prefix in ["foo//foo/base/", "zzz//yyy", "foo///", "//foo/", "/foo/"] { + for delimiter in [None, Some("/".to_owned())] { + for max in [0, 1, 100] { + for marker in [None, Some(key.to_owned())] { + let page = store + .clone() + .list_objects_v2(bucket, prefix, marker.clone(), delimiter.clone(), max, false, None, false) + .await + .expect("a literal prefix with no stored matches is a valid list request"); + assert!(page.objects.is_empty(), "{prefix} must not alias {key}"); + assert!(page.prefixes.is_empty(), "{prefix} must not publish normalized prefixes"); + assert!(!page.is_truncated); + assert!(page.next_continuation_token.is_none()); + assert_eq!(page.continuation_token, marker); + } + } + } + } + let oversized = format!("foo//{}", "x".repeat(32 << 10)); + for prefix in ["foo//../secret", "foo//./secret", "foo//bad\0name", &oversized] { + let err = store + .clone() + .list_objects_v2(bucket, prefix, None, None, 100, false, None, false) + .await + .expect_err("invalid prefixes must still be rejected"); + assert!(matches!(err, Error::ObjectNameInvalid(_, _)), "{err:?}"); + } + let err = store + .clone() + .list_objects_v2("invalid_bucket", "foo//bar", None, None, 100, false, None, false) + .await + .expect_err("invalid buckets must still be rejected"); + assert!(matches!(err, Error::BucketNameInvalid(_)), "{err:?}"); + + let control = store + .clone() + .list_objects_v2(bucket, "foo/", None, None, 100, false, None, false) + .await + .expect("normal prefix still lists the control object"); + assert_eq!(control.objects.iter().map(|object| object.name.as_str()).collect::>(), [key]); +} diff --git a/crates/ecstore/tests/metadata_read_lock_reuse_test.rs b/crates/ecstore/tests/metadata_read_lock_reuse_test.rs index 6433bf0cd..c37949477 100644 --- a/crates/ecstore/tests/metadata_read_lock_reuse_test.rs +++ b/crates/ecstore/tests/metadata_read_lock_reuse_test.rs @@ -18,6 +18,7 @@ mod storage_api; use std::sync::Arc; use std::time::Duration; +use storage_api::contract_compat::ECStore; use storage_api::metadata_lock::{ BucketOperations, CompletePart, Error, MakeBucketOptions, MultipartOperations, NamespaceLocking, ObjectIO, ObjectOperations, ObjectOptions, PutObjReader, PutObjectCommitBarrier, PutObjectCommitPause, init_bucket_metadata_sys, @@ -170,6 +171,100 @@ async fn reused_metadata_lock_still_enforces_quota() { .expect("acquire writer"); } +#[tokio::test] +async fn authoritative_quota_read_does_not_ignore_later_quota() { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let bucket = "authoritative-quota-quota-set"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + + let mut first = PutObjReader::from_vec(b"cache no quota".to_vec()); + store + .put_object(bucket, "warm", &mut first, &ObjectOptions::default()) + .await + .expect("warm no-quota metadata cache"); + store + .update_bucket_metadata_config(bucket, "quota.json", br#"{"quota":1}"#.to_vec()) + .await + .expect("set quota after no-quota cache was observed"); + + let mut opts = quota_snapshot_options(&store, bucket).await; + assert!(opts.set_quota_admission(0, 1)); + let mut data = PutObjReader::from_vec(b"too large".to_vec()); + let error = store + .put_object(bucket, "object", &mut data, &opts) + .await + .expect_err("quota set after a no-quota cache hit must still reject growth"); + assert!(matches!(error, Error::QuotaExceeded { limit: 1, .. }), "{error:?}"); +} + +#[tokio::test] +async fn authoritative_quota_read_allows_write_after_quota_is_cleared() { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let bucket = "authoritative-quota-quota-cleared"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + store + .update_bucket_metadata_config(bucket, "quota.json", br#"{"quota":1}"#.to_vec()) + .await + .expect("set quota"); + store + .update_bucket_metadata_config(bucket, "quota.json", Vec::new()) + .await + .expect("clear quota"); + + let mut data = PutObjReader::from_vec(b"allowed after clear".to_vec()); + store + .put_object(bucket, "object", &mut data, &ObjectOptions::default()) + .await + .expect("authoritatively cleared quota should allow writes"); + + let mut reader = store + .get_object_reader(bucket, "object", None, Default::default(), &ObjectOptions::default()) + .await + .expect("read object written after quota clear"); + let mut bytes = Vec::new(); + reader.stream.read_to_end(&mut bytes).await.expect("read body"); + assert_eq!(bytes, b"allowed after clear"); +} + +#[tokio::test] +async fn authoritative_quota_read_fails_closed_on_invalid_quota_json() { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let bucket = "authoritative-quota-invalid-quota"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + store + .update_bucket_metadata_config(bucket, "quota.json", b"{not-json".to_vec()) + .await + .expect("persist invalid quota bytes for fail-closed read"); + + let mut data = PutObjReader::from_vec(b"must fail".to_vec()); + store + .put_object(bucket, "object", &mut data, &ObjectOptions::default()) + .await + .expect_err("invalid quota JSON must not degrade to no-quota"); +} + +async fn quota_snapshot_options(store: &Arc, bucket: &str) -> ObjectOptions { + ObjectOptions { + versioned: true, + version_id: Some(Uuid::new_v4().to_string()), + expected_bucket_incarnation_id: Some(store.bucket_incarnation_id(bucket).await.expect("load incarnation")), + object_lock_config_snapshot: Some(store.object_lock_config_snapshot(bucket).await.expect("capture snapshot")), + ..Default::default() + } +} + #[tokio::test] async fn cancelled_put_releases_shared_metadata_guard() { let (_dirs, store) = isolated_store_over_temp_disks().await; diff --git a/crates/ecstore/tests/put_object_cancellation_test.rs b/crates/ecstore/tests/put_object_cancellation_test.rs new file mode 100644 index 000000000..bee170ca4 --- /dev/null +++ b/crates/ecstore/tests/put_object_cancellation_test.rs @@ -0,0 +1,133 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![cfg(feature = "test-util")] + +mod storage_api; + +use std::{path::PathBuf, time::Duration}; +use storage_api::metadata_lock::{ + ObjectIO, ObjectOptions, PutObjReader, PutObjectCommitBarrier, PutObjectCommitPause, isolated_store_over_temp_disks, +}; +use tempfile::TempDir; +use tokio::{ + fs, + io::AsyncReadExt, + time::{Instant, timeout}, +}; + +async fn staged_workspaces(dirs: &[TempDir]) -> Vec { + let mut staged = Vec::new(); + for dir in dirs { + let mut entries = fs::read_dir(dir.path().join(".rustfs.sys/tmp")) + .await + .expect("read temporary namespace"); + while let Some(entry) = entries.next_entry().await.expect("read temporary entry") { + if entry.file_name() != ".trash" { + staged.push(entry.path()); + } + } + } + staged +} + +async fn wait_for_staging_cleanup(dirs: &[TempDir]) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + let staged = staged_workspaces(dirs).await; + if staged.is_empty() { + return; + } + assert!(Instant::now() < deadline, "cancelled PUT left staging: {staged:?}"); + tokio::time::sleep(Duration::from_millis(25)).await; + } +} + +async fn checkpoint_timeout_at(pause: PutObjectCommitPause) { + let (dirs, store) = isolated_store_over_temp_disks().await; + let bucket = ".rustfs.sys"; + let object = "buckets/scanner-timeout/.usage-cache.bin.bkp"; + let old_body = vec![0x31; 1 << 20]; + let opts = ObjectOptions { + max_parity: true, + ..Default::default() + }; + store + .put_object(bucket, object, &mut PutObjReader::from_vec(old_body.clone()), &opts) + .await + .expect("seed previous checkpoint"); + wait_for_staging_cleanup(&dirs).await; + + let barrier = PutObjectCommitBarrier::install(bucket, object, pause); + let mut reader = PutObjReader::from_vec(vec![0x32; 1 << 20]); + let mut put = Box::pin(store.put_object(bucket, object, &mut reader, &opts)); + tokio::select! { + result = &mut put => panic!("replacement completed before the barrier: {result:?}"), + paused = timeout(Duration::from_secs(10), barrier.wait_until_paused()) => { + paused.expect("replacement must reach its commit barrier"); + } + } + let staged = staged_workspaces(&dirs).await; + assert_eq!(staged.len(), 4, "replacement must stage on every disk before cancellation"); + for workspace in staged { + let mut entries = fs::read_dir(workspace).await.expect("read staged workspace"); + let data_dir = entries + .next_entry() + .await + .expect("read staged directory") + .expect("a real shard"); + assert!( + fs::metadata(data_dir.path().join("part.1")) + .await + .expect("real staged payload") + .len() + > 0 + ); + } + // Hold the deterministic barrier so the deadline, not disk speed, drops + // the save future. No detached owner may finish and mask missing cleanup. + timeout(Duration::from_secs(1), put) + .await + .expect_err("held replacement must time out"); + // delete_all hands staging to .trash for ordinary background GC; it does + // not promise immediate space reclamation when the request is cancelled. + wait_for_staging_cleanup(&dirs).await; + drop(barrier); + let mut reader = store + .get_object_reader(bucket, object, None, Default::default(), &ObjectOptions::default()) + .await + .expect("previous checkpoint remains readable"); + let mut actual = Vec::new(); + reader + .stream + .read_to_end(&mut actual) + .await + .expect("read previous checkpoint"); + assert_eq!(actual, old_body, "timeout must neither publish replacement nor delete old data"); +} + +#[tokio::test] +async fn checkpoint_timeout_before_namespace_cleans_staging() { + checkpoint_timeout_at(PutObjectCommitPause::BeforeNamespace).await; +} + +#[tokio::test] +async fn checkpoint_timeout_before_commit_owner_cleans_staging() { + checkpoint_timeout_at(PutObjectCommitPause::AfterNamespace).await; +} + +#[tokio::test] +async fn checkpoint_timeout_after_commit_owner_cleans_staging() { + checkpoint_timeout_at(PutObjectCommitPause::AfterQuotaReservation).await; +} diff --git a/crates/ecstore/tests/storage_api.rs b/crates/ecstore/tests/storage_api.rs index acfd5c85e..c14769b3f 100644 --- a/crates/ecstore/tests/storage_api.rs +++ b/crates/ecstore/tests/storage_api.rs @@ -61,6 +61,25 @@ pub(crate) mod minio_generated_read { }; } +pub(crate) mod cleanup_isolation { + pub(crate) use super::{DiskAPI, DiskOption, Endpoint, new_disk}; + pub(crate) use rustfs_ecstore::api::disk::{DeleteOptions, DiskError}; +} + +#[cfg(all(feature = "test-util", not(windows)))] +pub(crate) mod inline_heal_staging { + pub(crate) use super::{DiskAPI, DiskOption, Endpoint, new_disk}; + pub(crate) use rustfs_ecstore::api::disk::{ + DeleteOptions, DiskError, LocalPublicationPause, LocalPublicationStage, ReadOptions, + }; +} + +#[cfg(all(feature = "test-util", target_os = "linux"))] +pub(crate) mod fd_relative { + pub(crate) use super::{DiskAPI, DiskOption, DiskStore, Endpoint, new_disk}; + pub(crate) use rustfs_ecstore::api::disk::{DiskError, LocalPublicationPause, LocalPublicationStage, ReadOptions}; +} + pub(crate) mod pinned_erasure_fixtures { pub(crate) use super::storage_contracts::{BucketOperations, HTTPRangeSpec, MakeBucketOptions, ObjectIO}; pub(crate) use super::{ECStore, Endpoint, Erasure, Error, ObjectOptions}; diff --git a/crates/filemeta/benches/xl_meta_bench.rs b/crates/filemeta/benches/xl_meta_bench.rs index ee46865a4..dc093e21d 100644 --- a/crates/filemeta/benches/xl_meta_bench.rs +++ b/crates/filemeta/benches/xl_meta_bench.rs @@ -12,9 +12,12 @@ // See the License for the specific language governing permissions and // limitations under the License. -use criterion::{Criterion, Throughput, criterion_group, criterion_main}; -use rustfs_filemeta::{FileMeta, MetaCacheEntry, test_data::*}; +use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main}; +use rustfs_filemeta::{FileInfo, FileMeta, MetaCacheEntry, ObjectPartInfo, test_data::*}; use std::hint::black_box; +use std::time::Duration; +use time::OffsetDateTime; +use uuid::Uuid; fn bench_create_real_xlmeta(c: &mut Criterion) { c.bench_function("create_real_xlmeta", |b| b.iter(|| black_box(create_real_xlmeta().unwrap()))); @@ -134,6 +137,73 @@ fn bench_list_page_to_fileinfo(c: &mut Criterion) { group.finish(); } +fn bench_delete_inline_versions(c: &mut Criterion) { + let mut group = c.benchmark_group("delete_inline_versions"); + group.sample_size(10); + group.warm_up_time(Duration::from_millis(500)); + group.measurement_time(Duration::from_secs(2)); + for count in [1000_u32, 2000, 10000] { + let mut meta = FileMeta::new(); + let mut inline = vec![1]; + rmp::encode::write_map_len(&mut inline, count).expect("inline map header"); + let payload = vec![42; 9 * 1024]; + let mut requests = Vec::new(); + for id in 1..=count { + let version = Uuid::from_u128(u128::from(id)); + let mut fi = FileInfo::new("object", 1, 0); + fi.version_id = Some(version); + fi.data_dir = Some(Uuid::from_u128(u128::from(count) + u128::from(id))); + fi.mod_time = Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1_700_000_000 + i64::from(id))); + fi.erasure.index = 1; + fi.size = i64::try_from(payload.len()).expect("payload length"); + fi.parts.push(ObjectPartInfo { + number: 1, + size: payload.len(), + actual_size: fi.size, + ..Default::default() + }); + fi.set_inline_data(); + meta.add_version(fi).expect("seed version metadata"); + rmp::encode::write_str(&mut inline, &version.to_string()).expect("inline key"); + rmp::encode::write_bin(&mut inline, &payload).expect("inline body"); + if id <= 1000 { + requests.push(FileInfo { + version_id: Some(version), + ..Default::default() + }); + } + } + meta.data.update(&inline); + group.throughput(Throughput::Elements(u64::try_from(requests.len()).expect("batch size"))); + group.bench_with_input(BenchmarkId::new("batch", count), &count, |b, _| { + b.iter_batched( + || meta.clone(), + |mut meta| black_box(meta.delete_versions(black_box(&requests)).expect("batch delete")), + BatchSize::LargeInput, + ); + }); + if count <= 2000 { + group.bench_with_input(BenchmarkId::new("sequential", count), &count, |b, _| { + b.iter_batched( + || meta.clone(), + |mut meta| { + for request in black_box(&requests) { + if let Some(dir) = meta.delete_version(request).expect("sequential delete") { + meta.data + .remove(vec![request.version_id.expect("explicit version"), dir]) + .expect("remove inline body"); + } + } + black_box(meta); + }, + BatchSize::LargeInput, + ); + }); + } + } + group.finish(); +} + criterion_group!( benches, bench_create_real_xlmeta, @@ -147,7 +217,8 @@ criterion_group!( bench_version_stats, bench_validate_integrity, bench_into_fileinfo_realistic, - bench_list_page_to_fileinfo + bench_list_page_to_fileinfo, + bench_delete_inline_versions ); criterion_main!(benches); diff --git a/crates/filemeta/src/filemeta.rs b/crates/filemeta/src/filemeta.rs index 860c00707..a3ebd1a94 100644 --- a/crates/filemeta/src/filemeta.rs +++ b/crates/filemeta/src/filemeta.rs @@ -347,21 +347,25 @@ impl FileMeta { self.versions.sort_by(cmp_shallow_versions_for_order); } - fn find_inline_data_for_version(&self, version_id: Option) -> Result>> { + fn find_inline_data_in_slice(inline_data: &[u8], version_id: Option) -> Result>> { let key = data_key_for_version(version_id); - if let Some(data) = self.data.find(key.as_str())? { + if let Some(data) = InlineData::find_in_slice(inline_data, key.as_str())? { return Ok(Some(data)); } if let Some(legacy_key) = legacy_data_key_for_version(version_id) && legacy_key != key { - return self.data.find(legacy_key.as_str()); + return InlineData::find_in_slice(inline_data, legacy_key.as_str()); } Ok(None) } + fn find_inline_data_for_version(&self, version_id: Option) -> Result>> { + Self::find_inline_data_in_slice(self.data.as_slice(), version_id) + } + // Find version pub fn find_version(&self, vid: Option) -> Result<(usize, FileMetaVersion)> { let vid = vid.unwrap_or_default(); @@ -696,9 +700,55 @@ impl FileMeta { // Ok(()) } + /// Apply a disk deletion batch while compacting inline payloads only once. + /// Missing non-marker requests retain the disk layer's idempotent semantics. + pub fn delete_versions(&mut self, requests: &[FileInfo]) -> Result>> { + if requests.is_empty() { + return Ok(Vec::new()); + } + let mut inline_keys = self.data.key_set()?; + let mut removed_keys = std::collections::HashSet::new(); + let mut data_dirs = Vec::with_capacity(requests.len()); + let mut failure = None; + for request in requests { + // Later requests must observe earlier inline removals even though + // the payload buffer is not rebuilt until the batch is complete. + let has_inline_data = self.data.version_ok() + && (inline_keys.contains(data_key_for_version(request.version_id).as_bytes()) + || legacy_data_key_for_version(request.version_id).is_some_and(|key| inline_keys.contains(key.as_bytes()))); + match self.delete_version_inner(request, Some(has_inline_data)) { + Ok(data_dir) => { + if let Some(dir) = data_dir { + for key in [request.version_id.unwrap_or_default(), dir] { + let key = key.to_string().into_bytes(); + if inline_keys.remove(&key) { + removed_keys.insert(key); + } + } + } + data_dirs.push(data_dir); + } + Err(Error::FileNotFound | Error::FileVersionNotFound) if !request.deleted => data_dirs.push(None), + Err(err) => { + failure = Some(err); + break; + } + } + } + self.data.remove_keys(&removed_keys)?; + if let Some(err) = failure { + return Err(err); + } + Ok(data_dirs) + } + // delete_version deletes version, returns data_dir #[tracing::instrument(level = "debug", skip(self))] pub fn delete_version(&mut self, fi: &FileInfo) -> Result> { + self.delete_version_inner(fi, None) + } + + fn delete_version_inner(&mut self, fi: &FileInfo, has_inline_data: Option) -> Result> { fi.validate_for_delete_operation()?; let vid = Some(fi.version_id.unwrap_or(Uuid::nil())); @@ -930,7 +980,7 @@ impl FileMeta { return Err(e); } - if self.shared_data_dir_count(obj_version_id, obj_data_dir) > 0 { + if self.shared_data_dir_count_with_inline_data(obj_version_id, obj_data_dir, has_inline_data) > 0 { return Ok(None); } @@ -987,6 +1037,74 @@ impl FileMeta { version_id: &str, opts: FileInfoDecodeOptions, ) -> Result { + Self::select_and_decode_fileinfo( + self.versions.len(), + self.versions.iter().map(|ver| Ok((ver.header.clone(), ver.meta.as_slice()))), + self.data.as_slice(), + volume, + path, + version_id, + opts, + ) + } + + fn decode_fileinfo_from_buf( + buf: &[u8], + volume: &str, + path: &str, + version_id: &str, + opts: FileInfoDecodeOptions, + ) -> Result { + let vid = if version_id.is_empty() { + None + } else { + Some(Uuid::parse_str(version_id)?) + }; + + let (meta, inline_buf) = Self::parse_validated_xl_envelope(buf)?; + let (versions_len, header_ver, meta) = if meta.is_empty() { + (0, 0, meta) + } else { + let (versions_len, header_ver, _meta_ver, meta) = Self::decode_validated_xl_headers(meta)?; + (versions_len, header_ver, meta) + }; + + if versions_len == 0 { + return Ok(FileInfo { + volume: volume.to_owned(), + name: path.to_owned(), + version_id: vid, + is_latest: true, + deleted: true, + mod_time: Some(OffsetDateTime::from_unix_timestamp(1)?), + ..Default::default() + }); + } + + let mut cur = Cursor::new(meta); + Self::select_and_decode_fileinfo( + versions_len, + (0..versions_len).map(|_| Self::read_next_shallow_slice(meta, &mut cur, header_ver)), + inline_buf, + volume, + path, + version_id, + opts, + ) + } + + fn select_and_decode_fileinfo<'a, I>( + versions_len: usize, + versions: I, + inline_data: &[u8], + volume: &str, + path: &str, + version_id: &str, + opts: FileInfoDecodeOptions, + ) -> Result + where + I: IntoIterator>, + { let vid = { if !version_id.is_empty() { Uuid::parse_str(version_id)? @@ -1006,20 +1124,20 @@ impl FileMeta { let mut is_latest = true; let mut succ_mod_time = None; - let mut non_free_versions = self.versions.len(); + let mut non_free_versions = versions_len; let mut found = false; let mut found_free_version = None; - let mut found_fi = None; + let mut found_fi: Option> = None; - for ver in self.versions.iter() { - let header = &ver.header; + for item in versions { + let (header, ver_meta) = item?; if header.free_version() { non_free_versions -= 1; if opts.include_free_versions && found_free_version.is_none() - && let Ok(found_free_fi) = ver.parse_version_meta() + && let Ok(found_free_fi) = FileMetaVersion::try_from(ver_meta) && found_free_fi.version_type != VersionType::Invalid { // Graceful degradation: the free-version replication-accounting record @@ -1028,8 +1146,12 @@ impl FileMeta { // Known side effect: if a disk holds only free versions and they are // corrupt, `into_fileinfo` falls through to `FileNotFound` (not // `FileCorrupt`), so that disk is not enqueued for heal. - match found_free_fi.to_fileinfo_with_part_checksums(volume, path, opts.all_parts, opts.include_part_checksums) - { + match found_free_fi.into_fileinfo_with_part_checksums( + volume, + path, + opts.all_parts, + opts.include_part_checksums, + ) { Ok(mut free_fi) => { free_fi.is_latest = true; found_free_version = Some(free_fi); @@ -1057,18 +1179,23 @@ impl FileMeta { found = true; - let mut fi = ver.to_fileinfo_with_part_checksums(volume, path, opts.all_parts, opts.include_part_checksums)?; - fi.is_latest = is_latest; + found_fi = Some( + FileMetaVersion::try_from(ver_meta) + .and_then(|v| v.into_fileinfo_with_part_checksums(volume, path, opts.all_parts, opts.include_part_checksums)) + .and_then(|mut fi| { + fi.is_latest = is_latest; - if let Some(_d) = succ_mod_time { - fi.successor_mod_time = succ_mod_time; - } + if let Some(_d) = succ_mod_time { + fi.successor_mod_time = succ_mod_time; + } - if opts.read_data && fi.inline_data() { - fi.data = self.find_inline_data_for_version(fi.version_id)?.map(bytes::Bytes::from); - } + if opts.read_data && fi.inline_data() { + fi.data = Self::find_inline_data_in_slice(inline_data, fi.version_id)?.map(bytes::Bytes::from); + } - found_fi = Some(fi); + Ok(fi) + }), + ); } if !found { @@ -1085,7 +1212,8 @@ impl FileMeta { } } - if let Some(mut fi) = found_fi { + if let Some(fi_res) = found_fi { + let mut fi = fi_res?; fi.num_versions = non_free_versions; return Ok(fi); diff --git a/crates/filemeta/src/filemeta/codec.rs b/crates/filemeta/src/filemeta/codec.rs index 851785ddb..0cebc19ea 100644 --- a/crates/filemeta/src/filemeta/codec.rs +++ b/crates/filemeta/src/filemeta/codec.rs @@ -130,10 +130,8 @@ impl FileMeta { Ok((bin_len, remaining)) } - #[hotpath::measure(impl_type = "FileMeta")] - pub fn unmarshal_msg(&mut self, buf: &[u8]) -> Result { - let i = buf.len() as u64; - + #[inline] + pub(super) fn parse_validated_xl_envelope(buf: &[u8]) -> Result<(&[u8], &[u8])> { // check version, buf = buf[8..] let (buf, _, _) = Self::check_xl2_v1(buf)?; @@ -188,89 +186,116 @@ impl FileMeta { } if !buf.is_empty() { - self.data.update(buf); - self.data.validate().map_err(|e| { + InlineData::validate_slice(buf).map_err(|e| { error!("data validation failed: {}", e); e })?; } + Ok((meta, buf)) + } + + #[inline] + pub(super) fn read_next_shallow_slice<'a>( + meta: &'a [u8], + cur: &mut Cursor<&'a [u8]>, + header_ver: u8, + ) -> Result<(FileMetaVersionHeader, &'a [u8])> { + let bin_len = rmp::decode::read_bin_len(cur).map_err(|e| { + error!("failed to read binary length for version header: {}", e); + Error::other(format!("failed to read binary length for version header: {e}")) + })? as usize; + + let start = cur.position() as usize; + let remaining = meta.len().saturating_sub(start); + if bin_len > remaining { + error!("corrupt xl.meta: version header length {} exceeds remaining {} bytes", bin_len, remaining); + return Err(Error::other("corrupt xl.meta: version header length exceeds metadata size")); + } + let end = start + bin_len; + + let mut header = FileMetaVersionHeader::default(); + header.unmarshal_v(header_ver, &meta[start..end]).map_err(|e| { + error!("failed to unmarshal version header: {}", e); + e + })?; + cur.set_position(end as u64); + + let bin_len = rmp::decode::read_bin_len(cur).map_err(|e| { + error!("failed to read binary length for version metadata: {}", e); + Error::other(format!("failed to read binary length for version metadata: {e}")) + })? as usize; + + let start = cur.position() as usize; + let remaining = meta.len().saturating_sub(start); + if bin_len > remaining { + error!( + "corrupt xl.meta: version metadata length {} exceeds remaining {} bytes", + bin_len, remaining + ); + return Err(Error::other("corrupt xl.meta: version metadata length exceeds metadata size")); + } + let end = start + bin_len; + cur.set_position(end as u64); + + Ok((header, &meta[start..end])) + } + + #[hotpath::measure(impl_type = "FileMeta")] + pub fn unmarshal_msg(&mut self, buf: &[u8]) -> Result { + let i = buf.len() as u64; + let (meta, inline_buf) = Self::parse_validated_xl_envelope(buf)?; + + if !inline_buf.is_empty() { + self.data.update(inline_buf); + } + // Parse meta if !meta.is_empty() { - let (versions_len, header_ver, meta_ver, meta) = Self::decode_xl_headers(meta).map_err(|e| { - error!("failed to decode XL headers: {}", e); - e - })?; - - // let (_, meta) = meta.split_at(read_size as usize); + let (versions_len, header_ver, meta_ver, meta) = Self::decode_validated_xl_headers(meta)?; self.meta_ver = meta_ver; - - // `versions_len` is decoded from a potentially corrupted buffer. - // Every version contributes at least two msgpack bin headers to - // `meta`, so a count larger than the remaining bytes is corrupt; - // reject it before sizing any allocation from it (rustfs/rustfs#2715). - if versions_len > meta.len() { - error!( - "corrupt xl.meta: version count {} exceeds remaining metadata size {}", - versions_len, - meta.len() - ); - return Err(Error::other(format!( - "corrupt xl.meta: version count {versions_len} exceeds metadata size {}", - meta.len() - ))); - } - self.versions = Vec::with_capacity(versions_len); let mut cur: Cursor<&[u8]> = Cursor::new(meta); for _ in 0..versions_len { - let bin_len = rmp::decode::read_bin_len(&mut cur).map_err(|e| { - error!("failed to read binary length for version header: {}", e); - Error::other(format!("failed to read binary length for version header: {e}")) - })? as usize; - - let remaining = meta.len().saturating_sub(cur.position() as usize); - if bin_len > remaining { - error!("corrupt xl.meta: version header length {} exceeds remaining {} bytes", bin_len, remaining); - return Err(Error::other("corrupt xl.meta: version header length exceeds metadata size")); - } - let mut header_buf = vec![0u8; bin_len]; - - cur.read_exact(&mut header_buf)?; - - let mut ver = FileMetaShallowVersion::default(); - ver.header.unmarshal_v(header_ver, &header_buf).map_err(|e| { - error!("failed to unmarshal version header: {}", e); - e - })?; - - let bin_len = rmp::decode::read_bin_len(&mut cur).map_err(|e| { - error!("failed to read binary length for version metadata: {}", e); - Error::other(format!("failed to read binary length for version metadata: {e}")) - })? as usize; - - let remaining = meta.len().saturating_sub(cur.position() as usize); - if bin_len > remaining { - error!( - "corrupt xl.meta: version metadata length {} exceeds remaining {} bytes", - bin_len, remaining - ); - return Err(Error::other("corrupt xl.meta: version metadata length exceeds metadata size")); - } - let mut ver_meta_buf = vec![0u8; bin_len]; - cur.read_exact(&mut ver_meta_buf)?; - - ver.meta.extend_from_slice(&ver_meta_buf); - - self.versions.push(ver); + let (header, ver_meta) = Self::read_next_shallow_slice(meta, &mut cur, header_ver)?; + self.versions.push(FileMetaShallowVersion { + header, + meta: ver_meta.to_vec(), + }); } } Ok(i) } + #[inline] + pub(super) fn decode_validated_xl_headers(meta: &[u8]) -> Result<(usize, u8, u8, &[u8])> { + let (versions_len, header_ver, meta_ver, meta) = Self::decode_xl_headers(meta).map_err(|e| { + error!("failed to decode XL headers: {}", e); + e + })?; + + // `versions_len` is decoded from a potentially corrupted buffer. + // Every version contributes at least two msgpack bin headers to + // `meta`, so a count larger than the remaining bytes is corrupt; + // reject it before sizing any allocation from it (rustfs/rustfs#2715). + if versions_len > meta.len() { + error!( + "corrupt xl.meta: version count {} exceeds remaining metadata size {}", + versions_len, + meta.len() + ); + return Err(Error::other(format!( + "corrupt xl.meta: version count {versions_len} exceeds metadata size {}", + meta.len() + ))); + } + + Ok((versions_len, header_ver, meta_ver, meta)) + } + // decode_xl_headers parses meta header, returns (versions count, xl_header_version, xl_meta_version, read data length) fn decode_xl_headers(buf: &[u8]) -> Result<(usize, u8, u8, &[u8])> { let mut cur = Cursor::new(buf); @@ -353,7 +378,8 @@ impl FileMeta { #[hotpath::measure(impl_type = "FileMeta")] pub fn marshal_msg(&self) -> Result> { - let mut wr = Vec::new(); + let mut wr = + Vec::with_capacity(24 + self.versions.iter().map(|v| 48 + v.meta.len()).sum::() + self.data.as_slice().len()); // header wr.write_all(XL_FILE_HEADER.as_slice())?; @@ -376,9 +402,12 @@ impl FileMeta { rmp::encode::write_sint(&mut wr, self.versions.len() as i64)?; + let mut hbuf = [0u8; 48]; for ver in self.versions.iter() { - let hmsg = ver.header.marshal_msg()?; - rmp::encode::write_bin(&mut wr, &hmsg)?; + let mut hcur = Cursor::new(&mut hbuf[..]); + ver.header.encode_to(&mut hcur)?; + let hlen = hcur.position() as usize; + rmp::encode::write_bin(&mut wr, &hbuf[..hlen])?; rmp::encode::write_bin(&mut wr, &ver.meta)?; } diff --git a/crates/filemeta/src/filemeta/inline_data.rs b/crates/filemeta/src/filemeta/inline_data.rs index e1154e5c3..286f49775 100644 --- a/crates/filemeta/src/filemeta/inline_data.rs +++ b/crates/filemeta/src/filemeta/inline_data.rs @@ -114,14 +114,24 @@ impl FileMeta { /// Count shared data directories pub fn shared_data_dir_count(&self, version_id: Option, data_dir: Option) -> usize { + self.shared_data_dir_count_with_inline_data(version_id, data_dir, None) + } + + pub(super) fn shared_data_dir_count_with_inline_data( + &self, + version_id: Option, + data_dir: Option, + has_inline_data: Option, + ) -> usize { let vid = version_id.unwrap_or_default(); let Some(data_dir) = data_dir else { return 0; }; - if self.data.entries().unwrap_or_default() > 0 - && self.find_inline_data_for_version(version_id).unwrap_or_default().is_some() - { + if has_inline_data.unwrap_or_else(|| { + self.data.entries().unwrap_or_default() > 0 + && self.find_inline_data_for_version(version_id).unwrap_or_default().is_some() + }) { return 0; } @@ -297,4 +307,191 @@ mod tests { let got = meta.find_unshared_data_dir_for_version(Some(target_version)); assert_eq!(got, None); } + + fn sequential_delete(meta: &mut FileMeta, requests: &[FileInfo]) -> Result>> { + let mut dirs = Vec::new(); + for request in requests { + match meta.delete_version(request) { + Ok(dir) => { + if let Some(dir) = dir { + meta.data.remove(vec![request.version_id.unwrap_or_default(), dir])?; + } + dirs.push(dir); + } + Err(Error::FileNotFound | Error::FileVersionNotFound) if !request.deleted => dirs.push(None), + Err(err) => return Err(err), + } + } + Ok(dirs) + } + + fn assert_batch_matches_sequential(original: &FileMeta, requests: &[FileInfo]) { + let mut expected = original.clone(); + let expected_dirs = sequential_delete(&mut expected, requests); + let mut actual = original.clone(); + assert_eq!(actual.delete_versions(requests), expected_dirs); + let decode = |meta: &FileMeta| { + meta.versions + .iter() + .map(|version| version.parse_version_meta().expect("decode retained version")) + .collect::>() + }; + assert_eq!(decode(&actual), decode(&expected)); + assert_eq!(actual.data, expected.data, "retained inline bytes and ordering must be identical"); + } + + #[test] + fn delete_versions_matches_sequential_mixed_version_subsets() { + let mut original = FileMeta::new(); + let mut requests = Vec::new(); + for id in 0_u128..8 { + let version = Uuid::from_u128(id); + // Versions 2 and 3 share physical data. Version 4's data-dir key + // aliases version 5's inline key, so later decisions must see its removal. + let dir = Uuid::from_u128(match id { + 2 | 3 | 5 => 100, + 4 => 5, + _ => 200 + id, + }); + let mut fi = make_plain_file_info(version, dir); + fi.mod_time = + Some(OffsetDateTime::UNIX_EPOCH + Duration::seconds(1_700_000_000 + i64::try_from(id).expect("small id"))); + if !matches!(id, 2 | 3) { + fi.data = Some(Bytes::from(vec![u8::try_from(id).expect("small id"); 97])); + fi.set_inline_data(); + } + original.add_version(fi).expect("seed mixed version chain"); + requests.push(FileInfo { + version_id: Some(version), + ..Default::default() + }); + } + for mask in 0_u32..256 { + let selected: Vec<_> = requests + .iter() + .enumerate() + .filter(|(i, _)| mask & (1 << i) != 0) + .map(|(_, fi)| fi.clone()) + .collect(); + assert_batch_matches_sequential(&original, &selected); + let reversed: Vec<_> = selected.into_iter().rev().collect(); + assert_batch_matches_sequential(&original, &reversed); + } + let mut duplicate_and_missing = requests.clone(); + duplicate_and_missing.extend(requests.iter().rev().cloned()); + duplicate_and_missing.push(FileInfo { + version_id: Some(Uuid::from_u128(999)), + ..Default::default() + }); + assert_batch_matches_sequential(&original, &duplicate_and_missing); + } + + #[test] + fn delete_versions_preserves_pending_purge_and_stops_at_invalid_request() { + let mut original = FileMeta::new(); + let mut fi = make_plain_file_info(Uuid::from_u128(1), Uuid::from_u128(101)); + fi.data = Some(Bytes::from_static(b"retained until replication completes")); + fi.set_inline_data(); + original.add_version(fi).expect("seed inline version"); + let pending = FileInfo { + version_id: Some(Uuid::from_u128(1)), + mark_deleted: true, + replication_state_internal: Some(ReplicationState { + version_purge_status_internal: Some("target=PENDING;".into()), + purge_targets: version_purge_statuses_map("target=PENDING;"), + ..Default::default() + }), + ..Default::default() + }; + assert_batch_matches_sequential(&original, std::slice::from_ref(&pending)); + let completed = FileInfo { + version_id: pending.version_id, + replication_state_internal: Some(ReplicationState { + version_purge_status_internal: Some("target=COMPLETE;".into()), + purge_targets: version_purge_statuses_map("target=COMPLETE;"), + ..Default::default() + }), + ..Default::default() + }; + assert_batch_matches_sequential(&original, &[pending, completed]); + let valid = FileInfo { + version_id: Some(Uuid::from_u128(1)), + ..Default::default() + }; + let invalid = FileInfo { + parts: vec![ + ObjectPartInfo { + number: 1, + ..Default::default() + }; + 2 + ], + ..Default::default() + }; + assert!(invalid.validate_for_delete_operation().is_err()); + assert_batch_matches_sequential(&original, &[valid, invalid]); + } + + #[test] + fn delete_versions_rejects_corrupt_inline_before_mutating_versions() { + let mut meta = FileMeta::new(); + meta.add_version(make_plain_file_info(Uuid::from_u128(1), Uuid::from_u128(101))) + .expect("seed version"); + for bytes in [ + vec![1, 0x81, 0xa1, b'x', 0xc4, 255], + vec![2, 0x80], + vec![0, 0x80], + vec![1, 0x81, 0xa1, 0xff, 0xc4, 0], + vec![1, 0x81, 0xdb, 1, 0, 0, 1], + vec![1, 0x81, 0xa1, b'x', 0xc6, 0xff, 0xff, 0xff, 0xff], + ] { + meta.data.update(&bytes); + let before = meta.clone(); + assert!( + meta.delete_versions(&[]) + .expect("empty batch does not inspect metadata") + .is_empty() + ); + assert!( + meta.delete_versions(&[FileInfo { + version_id: Some(Uuid::from_u128(1)), + ..Default::default() + }]) + .is_err() + ); + assert_eq!(meta, before); + } + } + + #[test] + fn delete_versions_preserves_both_null_inline_key_encodings() { + for legacy in [false, true] { + let mut original = FileMeta::new(); + let mut fi = make_plain_null_version_file_info(Uuid::from_u128(100)); + fi.data = Some(Bytes::from_static(b"null version body")); + fi.set_inline_data(); + original.add_version(fi).expect("seed null version"); + if legacy { + let value = original + .data + .find(NULL_VERSION_ID) + .expect("find null body") + .expect("null body"); + original.data.remove_key(NULL_VERSION_ID).expect("remove modern null key"); + original + .data + .replace(&Uuid::nil().to_string(), value) + .expect("write legacy null key"); + } + for version_id in [None, Some(Uuid::nil())] { + assert_batch_matches_sequential( + &original, + &[FileInfo { + version_id, + ..Default::default() + }], + ); + } + } + } } diff --git a/crates/filemeta/src/filemeta/msgp_decode.rs b/crates/filemeta/src/filemeta/msgp_decode.rs index b99e6a7ed..1ebb58755 100644 --- a/crates/filemeta/src/filemeta/msgp_decode.rs +++ b/crates/filemeta/src/filemeta/msgp_decode.rs @@ -17,45 +17,14 @@ use rmp::Marker; use std::io::Read; /// Maximum accepted length for a single length-prefixed msgpack element -/// (map key, string, binary blob, ext payload, or a whole serialized -/// xl.meta record inside a metacache stream). +/// (map key, string, binary blob, or ext payload) inside xl.meta. /// -/// Legitimate elements are far smaller: object keys are at most a few KiB -/// and even an xl.meta record fully populated up to the 10,000-version -/// object cap serializes to a few MiB (metacache streams carry xl.meta -/// without inline data), so 16 MiB leaves ample headroom. A larger value -/// means the length prefix itself is corrupt; decoding must fail with an -/// error instead of attempting a huge allocation that aborts the whole -/// process (see rustfs/rustfs#2715). +/// This is not a limit on the aggregate xl.meta carried by a metacache +/// stream: many versions and their inline data can exceed it. Bound each +/// individual value before allocation so corrupted prefixes return an error +/// instead of attempting a huge allocation (see rustfs/rustfs#2715). pub(crate) const MAX_MSGP_ELEMENT_SIZE: usize = 16 << 20; -/// Reads exactly `len` bytes into a fresh buffer, treating `len` as -/// untrusted input: lengths above [`MAX_MSGP_ELEMENT_SIZE`] and allocation -/// failures surface as decode errors instead of aborting the process. -pub(crate) fn read_exact_vec(rd: &mut R, len: usize) -> Result> { - if len > MAX_MSGP_ELEMENT_SIZE { - return Err(Error::other(format!( - "corrupt msgpack element: length {len} exceeds the {MAX_MSGP_ELEMENT_SIZE} byte limit" - ))); - } - let mut buf = Vec::new(); - buf.try_reserve_exact(len) - .map_err(|e| Error::other(format!("msgpack element allocation of {len} bytes failed: {e}")))?; - buf.resize(len, 0); - rd.read_exact(&mut buf).map_err(Error::from)?; - Ok(buf) -} - -/// Bounds a decoded collection count when it is used only as a -/// pre-allocation hint. The decode loop still consumes exactly the decoded -/// number of elements (a corrupt count fails with an EOF decode error once -/// the input runs out); this merely keeps the speculative reservation from -/// aborting the process on an absurd count. -pub(crate) fn prealloc_hint(len: usize) -> usize { - const MAX_PREALLOC_ITEMS: usize = 4096; - len.min(MAX_PREALLOC_ITEMS) -} - /// Reader that prepends a single byte to the stream. Used when we've read the marker /// and need to pass it to a decoder that expects to read the marker itself. pub(crate) struct PrependByteReader<'a, R> { @@ -77,7 +46,72 @@ impl Read for PrependByteReader<'_, R> { } } +/// Reads a nil marker (`0xc0`) or a MessagePack map length using `rmp::decode::read_map_len` +/// semantics so error variants match `read_map_len` while leaving `rd` unwrapped afterward. +#[inline] +pub(crate) fn read_nil_or_submap_len(rd: &mut R) -> Result> { + let mut first = [0u8; 1]; + rd.read_exact(&mut first).map_err(Error::from)?; + match first[0] { + 0xc0 => Ok(None), + 0x80..=0x8f => Ok(Some((first[0] & 0x0f) as usize)), + b => { + let mut prepend = PrependByteReader { + byte: Some(b), + inner: rd, + }; + let len = rmp::decode::read_map_len(&mut prepend)?; + Ok(Some(len as usize)) + } + } +} + +/// Reads a MessagePack string key into a 16-byte stack buffer when it fits, +/// falling back to `heap` only for longer keys. +#[inline] +pub(crate) fn read_small_str<'a, R: Read>(rd: &mut R, inline: &'a mut [u8; 16], heap: &'a mut Vec) -> Result<&'a str> { + let len = usize::try_from(rmp::decode::read_str_len(rd)?).map_err(Error::other)?; + let bytes = if len <= inline.len() { + rd.read_exact(&mut inline[..len]).map_err(Error::from)?; + &inline[..len] + } else { + *heap = read_exact_vec(rd, len)?; + heap.as_slice() + }; + std::str::from_utf8(bytes).map_err(|e| Error::FromUtf8(e.to_string())) +} + +/// Reads exactly `len` bytes into a fresh buffer, treating `len` as +/// untrusted input: lengths above [`MAX_MSGP_ELEMENT_SIZE`] and allocation +/// failures surface as decode errors instead of aborting the process. +#[inline] +pub(crate) fn read_exact_vec(rd: &mut R, len: usize) -> Result> { + if len > MAX_MSGP_ELEMENT_SIZE { + return Err(Error::other(format!( + "corrupt msgpack element: length {len} exceeds the {MAX_MSGP_ELEMENT_SIZE} byte limit" + ))); + } + let mut buf = Vec::new(); + buf.try_reserve_exact(len) + .map_err(|e| Error::other(format!("msgpack element allocation of {len} bytes failed: {e}")))?; + buf.resize(len, 0); + rd.read_exact(&mut buf).map_err(Error::from)?; + Ok(buf) +} + +/// Bounds a decoded collection count when it is used only as a +/// pre-allocation hint. The decode loop still consumes exactly the decoded +/// number of elements (a corrupt count fails with an EOF decode error once +/// the input runs out); this merely keeps the speculative reservation from +/// aborting the process on an absurd count. +#[inline] +pub(crate) fn prealloc_hint(len: usize) -> usize { + const MAX_PREALLOC_ITEMS: usize = 4096; + len.min(MAX_PREALLOC_ITEMS) +} + /// Read MessagePack nil or array length. Returns None for nil, Some(len) for array. +#[inline] pub(crate) fn read_nil_or_array_len(rd: &mut R) -> Result> { let mut buf = [0u8; 1]; rd.read_exact(&mut buf).map_err(Error::from)?; @@ -100,6 +134,7 @@ pub(crate) fn read_nil_or_array_len(rd: &mut R) -> Result } /// Read MessagePack nil or map length. Returns None for nil, Some(len) for map. +#[inline] pub(crate) fn read_nil_or_map_len(rd: &mut R) -> Result> { let mut buf = [0u8; 1]; rd.read_exact(&mut buf).map_err(Error::from)?; diff --git a/crates/filemeta/src/filemeta/version.rs b/crates/filemeta/src/filemeta/version.rs index 87beb3d80..e5384d5cd 100644 --- a/crates/filemeta/src/filemeta/version.rs +++ b/crates/filemeta/src/filemeta/version.rs @@ -23,7 +23,8 @@ //! legacy (rmp_serde) format. `try_from` falls back to rmp_serde when hand-written decode fails. use super::msgp_decode::{ - PrependByteReader, prealloc_hint, read_exact_vec, read_nil_or_array_len, read_nil_or_map_len, skip_msgp_value, + prealloc_hint, read_exact_vec, read_nil_or_array_len, read_nil_or_map_len, read_nil_or_submap_len, read_small_str, + skip_msgp_value, }; use super::*; use crate::{ChecksumInfo, TransitionVersionState}; @@ -85,6 +86,21 @@ fn fold_signature(crc: u64) -> [u8; 4] { ((crc ^ (crc >> 32)) as u32).to_le_bytes() } +struct Xxh64Writer(xxhash_rust::xxh64::Xxh64); + +impl std::io::Write for Xxh64Writer { + #[inline] + fn write(&mut self, buf: &[u8]) -> std::io::Result { + self.0.update(buf); + Ok(buf.len()) + } + + #[inline] + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + fn read_msgp_string(rd: &mut R) -> Result { let len = rmp::decode::read_str_len(rd)? as usize; let buf = read_exact_vec(rd, len)?; @@ -588,18 +604,8 @@ impl FileMetaShallowVersion { } pub fn into_fileinfo(&self, volume: &str, path: &str, all_parts: bool) -> Result { - self.parse_version_meta()?.into_fileinfo(volume, path, all_parts) - } - - pub(super) fn to_fileinfo_with_part_checksums( - &self, - volume: &str, - path: &str, - all_parts: bool, - include_part_checksums: bool, - ) -> Result { self.parse_version_meta()? - .to_fileinfo_with_part_checksums(volume, path, all_parts, include_part_checksums) + .into_fileinfo_with_part_checksums(volume, path, all_parts, true) } } @@ -652,15 +658,15 @@ impl FileMetaVersion { let mut cur = std::io::Cursor::new(buf); let mut fields = rmp::decode::read_map_len(&mut cur)?; let mut version_type = VersionType::Invalid; + let mut inline_key = [0u8; MSGP_OBJECT_KEY_STACK_CAP]; + let mut heap_key = Vec::new(); while fields > 0 { fields -= 1; - let key_len = rmp::decode::read_str_len(&mut cur)? as usize; - let key_buf = read_exact_vec(&mut cur, key_len)?; - let key = String::from_utf8(key_buf)?; + let key = read_small_str(&mut cur, &mut inline_key, &mut heap_key)?; - match key.as_str() { + match key { "Type" => { let v: i64 = rmp::decode::read_int(&mut cur)?; version_type = VersionType::from_u8(v as u8); @@ -671,39 +677,29 @@ impl FileMetaVersion { continue; } - let mut first = [0u8; 1]; - cur.read_exact(&mut first)?; - if first[0] == 0xc0 { + let Some(mut obj_fields) = read_nil_or_submap_len(&mut cur)? else { return Ok(None); - } - - let mut prepend = PrependByteReader { - byte: Some(first[0]), - inner: &mut cur, }; - let mut obj_fields = rmp::decode::read_map_len(&mut prepend)?; let mut data_dir: Option = None; while obj_fields > 0 { obj_fields -= 1; - let obj_key_len = rmp::decode::read_str_len(&mut prepend)? as usize; - let obj_key_buf = read_exact_vec(&mut prepend, obj_key_len)?; - let obj_key = String::from_utf8(obj_key_buf)?; + let obj_key = read_small_str(&mut cur, &mut inline_key, &mut heap_key)?; if obj_key == "DDir" { - let bin_len = rmp::decode::read_bin_len(&mut prepend)? as usize; + let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize; if bin_len != 16 { return Err(Error::other(format!("DDir must be 16 bytes, got {bin_len}"))); } let mut raw = [0u8; 16]; - prepend.read_exact(&mut raw)?; + cur.read_exact(&mut raw)?; let id = Uuid::from_bytes(raw); data_dir = if id.is_nil() { None } else { Some(id) }; break; } - skip_msgp_value(&mut prepend)?; + skip_msgp_value(&mut cur)?; } return Ok(data_dir); @@ -781,62 +777,44 @@ impl FileMetaVersion { pub fn decode_from(&mut self, rd: &mut R) -> Result<()> { let mut fields = rmp::decode::read_map_len(rd)?; *self = FileMetaVersion::default(); + let mut inline_key = [0u8; MSGP_OBJECT_KEY_STACK_CAP]; + let mut heap_key = Vec::new(); while fields > 0 { fields -= 1; - let key_len = rmp::decode::read_str_len(rd)?; - let key_buf = read_exact_vec(rd, key_len as usize)?; - let key = String::from_utf8(key_buf)?; + let key = read_small_str(rd, &mut inline_key, &mut heap_key)?; - match key.as_str() { + match key { "Type" => { let v: i64 = rmp::decode::read_int(rd)?; self.version_type = VersionType::from_u8(v as u8); } "V1Obj" => { - let mut buf = [0u8; 1]; - rd.read_exact(&mut buf).map_err(Error::from)?; - if buf[0] == 0xc0 { - self.legacy_object = None; - } else { - let mut prepend = PrependByteReader { - byte: Some(buf[0]), - inner: rd, - }; + if let Some(obj_fields) = read_nil_or_submap_len(rd)? { let mut obj = MetaObjectV1::default(); - obj.decode_from(&mut prepend)?; + obj.decode_fields(rd, obj_fields)?; self.legacy_object = Some(obj); + } else { + self.legacy_object = None; } } "V2Obj" => { - let mut buf = [0u8; 1]; - rd.read_exact(&mut buf).map_err(Error::from)?; - if buf[0] == 0xc0 { - self.object = None; - } else { - let mut prepend = PrependByteReader { - byte: Some(buf[0]), - inner: rd, - }; + if let Some(obj_fields) = read_nil_or_submap_len(rd)? { let mut obj = MetaObject::default(); - obj.decode_from(&mut prepend)?; + obj.decode_fields(rd, obj_fields)?; self.object = Some(obj); + } else { + self.object = None; } } "DelObj" => { - let mut buf = [0u8; 1]; - rd.read_exact(&mut buf).map_err(Error::from)?; - if buf[0] == 0xc0 { - self.delete_marker = None; - } else { - let mut prepend = PrependByteReader { - byte: Some(buf[0]), - inner: rd, - }; + if let Some(dm_fields) = read_nil_or_submap_len(rd)? { let mut dm = MetaDeleteMarker::default(); - dm.decode_from(&mut prepend)?; + dm.decode_fields(rd, dm_fields)?; self.delete_marker = Some(dm); + } else { + self.delete_marker = None; } } "v" => { @@ -908,7 +886,7 @@ impl FileMetaVersion { } pub fn marshal_msg(&self) -> Result> { - let mut wr = Vec::new(); + let mut wr = Vec::with_capacity(256); self.encode_to(&mut wr)?; Ok(wr) } @@ -918,13 +896,33 @@ impl FileMetaVersion { } pub fn header(&self) -> FileMetaVersionHeader { - FileMetaVersionHeader::from(self.clone()) + FileMetaVersionHeader::from(self) } pub fn into_fileinfo(&self, volume: &str, path: &str, all_parts: bool) -> Result { self.to_fileinfo_with_part_checksums(volume, path, all_parts, true) } + pub(super) fn into_fileinfo_with_part_checksums( + self, + volume: &str, + path: &str, + all_parts: bool, + include_part_checksums: bool, + ) -> Result { + if self.version_type == VersionType::Object { + let mut fi = self.object.unwrap_or_default().into_fileinfo_with_part_checksums( + volume, + path, + all_parts, + include_part_checksums, + )?; + fi.uses_legacy_checksum = self.uses_legacy_checksum; + return Ok(fi); + } + self.to_fileinfo_with_part_checksums(volume, path, all_parts, include_part_checksums) + } + pub(super) fn to_fileinfo_with_part_checksums( &self, volume: &str, @@ -1215,27 +1213,31 @@ impl FileMetaVersionHeader { } pub fn marshal_msg(&self) -> Result> { - let mut wr = Vec::new(); + let mut wr = Vec::with_capacity(42); + self.encode_to(&mut wr)?; + Ok(wr) + } + pub fn encode_to(&self, wr: &mut W) -> Result<()> { // array len 7 - rmp::encode::write_array_len(&mut wr, 7)?; + rmp::encode::write_array_len(wr, 7)?; // version_id - rmp::encode::write_bin(&mut wr, self.version_id.unwrap_or_default().as_bytes())?; + rmp::encode::write_bin(wr, self.version_id.unwrap_or_default().as_bytes())?; // mod_time - rmp::encode::write_i64(&mut wr, self.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH).unix_timestamp_nanos() as i64)?; + rmp::encode::write_i64(wr, self.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH).unix_timestamp_nanos() as i64)?; // signature - rmp::encode::write_bin(&mut wr, self.signature.as_slice())?; + rmp::encode::write_bin(wr, self.signature.as_slice())?; // version_type - rmp::encode::write_uint8(&mut wr, self.version_type.to_u8())?; + rmp::encode::write_uint8(wr, self.version_type.to_u8())?; // flags - rmp::encode::write_uint8(&mut wr, self.flags)?; + rmp::encode::write_uint8(wr, self.flags)?; // ec_n - rmp::encode::write_uint8(&mut wr, self.ec_n)?; + rmp::encode::write_uint8(wr, self.ec_n)?; // ec_m - rmp::encode::write_uint8(&mut wr, self.ec_m)?; + rmp::encode::write_uint8(wr, self.ec_m)?; - Ok(wr) + Ok(()) } pub fn unmarshal_v(&mut self, version: u8, buf: &[u8]) -> Result { @@ -1396,8 +1398,8 @@ impl Ord for FileMetaVersionHeader { } } -impl From for FileMetaVersionHeader { - fn from(value: FileMetaVersion) -> Self { +impl From<&FileMetaVersion> for FileMetaVersionHeader { + fn from(value: &FileMetaVersion) -> Self { let flags = { let mut f: u8 = 0; if value.free_version() { @@ -1424,7 +1426,7 @@ impl From for FileMetaVersionHeader { version_id: value.get_version_id(), mod_time: value.get_mod_time(), signature: value.get_signature(), - version_type: value.version_type, + version_type: value.version_type.clone(), flags, ec_n, ec_m, @@ -1432,6 +1434,12 @@ impl From for FileMetaVersionHeader { } } +impl From for FileMetaVersionHeader { + fn from(value: FileMetaVersion) -> Self { + Self::from(&value) + } +} + #[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq)] // Because of custom message_pack, field order must be guaranteed pub struct MetaObject { @@ -1605,8 +1613,7 @@ impl MetaObjectV1 { && crate::fileinfo::is_valid_distribution(&self.erasure.distribution, data_blocks + parity_blocks) } - fn decode_from(&mut self, rd: &mut R) -> Result<()> { - let mut fields = rmp::decode::read_map_len(rd)?; + fn decode_fields(&mut self, rd: &mut R, mut fields: usize) -> Result<()> { *self = Self::default(); while fields > 0 { @@ -2049,7 +2056,7 @@ impl MetaObject { // marshal_msg custom messagepack naming consistent with go pub fn marshal_msg(&self) -> Result> { - let mut wr = Vec::new(); + let mut wr = Vec::with_capacity(256); self.encode_to(&mut wr)?; Ok(wr) } @@ -2058,32 +2065,30 @@ impl MetaObject { /// `xlMetaV2Object.Signature`. See [`FileMetaVersion::get_signature`] for /// why divergence detection requires covering every body field. pub fn get_signature(&self) -> [u8; 4] { - let mut c = self.clone(); - - // Zero fields that legitimately vary per disk within an erasure set. - c.erasure_index = 0; - - // Treat an all-empty PartETags vector the same as an absent one, so two - // disks that encode `[]` vs `["", ""]` do not falsely diverge. - if c.part_etags.iter().all(String::is_empty) { - c.part_etags.clear(); - } - // Fold maps in with an order-independent hash: msgpack map order is not // stable across disks, so they must not be part of the marshaled body. - let mut crc = hash_deterministic_string(&c.meta_user); - crc ^= hash_deterministic_bytes(&c.meta_sys); - c.meta_sys.clear(); - c.meta_user.clear(); + let mut crc = hash_deterministic_string(&self.meta_user); + crc ^= hash_deterministic_bytes(&self.meta_sys); - if let Ok(bytes) = c.marshal_msg() { - crc ^= xxhash_rust::xxh64::xxh64(&bytes, XXHASH_SEED); + // Zero fields that legitimately vary per disk within an erasure set, and + // treat an all-empty PartETags vector the same as an absent one so two + // disks that encode `[]` vs `["", ""]` do not falsely diverge. + let mut hasher = Xxh64Writer(xxhash_rust::xxh64::Xxh64::new(XXHASH_SEED)); + let include_part_etags = !self.part_etags.iter().all(String::is_empty); + if self.encode_to_with_options(&mut hasher, 0, include_part_etags, false).is_ok() { + crc ^= hasher.0.finish(); } fold_signature(crc) } - pub fn encode_to(&self, wr: &mut W) -> Result<()> { + fn encode_to_with_options( + &self, + wr: &mut W, + erasure_index: usize, + include_part_etags: bool, + include_maps: bool, + ) -> Result<()> { // Variable map size: omit PartIdx when empty let mut map_len = 18u32; if self.part_indices.is_empty() { @@ -2117,7 +2122,7 @@ impl MetaObject { // EcIndex rmp::encode::write_str(wr, "EcIndex")?; - rmp::encode::write_sint(wr, self.erasure_index as i64)?; + rmp::encode::write_sint(wr, erasure_index as i64)?; // EcDist rmp::encode::write_str(wr, "EcDist")?; @@ -2139,7 +2144,7 @@ impl MetaObject { // PartETags (write nil when empty) rmp::encode::write_str(wr, "PartETags")?; - if self.part_etags.is_empty() { + if !include_part_etags || self.part_etags.is_empty() { rmp::encode::write_nil(wr)?; } else { rmp::encode::write_array_len(wr, self.part_etags.len() as u32)?; @@ -2186,7 +2191,7 @@ impl MetaObject { // MetaSys (write nil when empty) rmp::encode::write_str(wr, "MetaSys")?; - if self.meta_sys.is_empty() { + if !include_maps || self.meta_sys.is_empty() { rmp::encode::write_nil(wr)?; } else { rmp::encode::write_map_len(wr, self.meta_sys.len() as u32)?; @@ -2198,7 +2203,7 @@ impl MetaObject { // MetaUsr (write nil when empty) rmp::encode::write_str(wr, "MetaUsr")?; - if self.meta_user.is_empty() { + if !include_maps || self.meta_user.is_empty() { rmp::encode::write_nil(wr)?; } else { rmp::encode::write_map_len(wr, self.meta_user.len() as u32)?; @@ -2211,44 +2216,28 @@ impl MetaObject { Ok(()) } + pub fn encode_to(&self, wr: &mut W) -> Result<()> { + self.encode_to_with_options(wr, self.erasure_index, !self.part_etags.is_empty(), true) + } + pub fn decode_from(&mut self, rd: &mut R) -> Result<()> { - let mut fields = rmp::decode::read_map_len(rd).map_err(|e| { + let fields = rmp::decode::read_map_len(rd).map_err(|e| { tracing::error!(error = %e, "decode_from: read_map_len failed"); e - })?; + })? as usize; + self.decode_fields(rd, fields) + } + + fn decode_fields(&mut self, rd: &mut R, mut fields: usize) -> Result<()> { *self = MetaObject::default(); + let mut inline_key = [0u8; MSGP_OBJECT_KEY_STACK_CAP]; + let mut heap_key = Vec::new(); while fields > 0 { fields -= 1; - let key_len = rmp::decode::read_str_len(rd).map_err(|e| { - tracing::error!(error = %e, "decode_from: read_str_len key failed"); - e - })?; - let key_len = usize::try_from(key_len).map_err(|e| { - tracing::error!(error = %e, "decode_from: key length conversion failed"); - Error::other(e) - })?; - - let mut inline_key = [0u8; MSGP_OBJECT_KEY_STACK_CAP]; - let heap_key; - let key_buf = if key_len <= MSGP_OBJECT_KEY_STACK_CAP { - rd.read_exact(&mut inline_key[..key_len]).map_err(|e| { - tracing::error!(error = %e, "decode_from: read key_buf failed"); - Error::from(e) - })?; - &inline_key[..key_len] - } else { - heap_key = read_exact_vec(rd, key_len).map_err(|e| { - tracing::error!(error = %e, "decode_from: read key_buf failed"); - e - })?; - heap_key.as_slice() - }; - - let key = std::str::from_utf8(key_buf).map_err(|e| { - tracing::error!(error = %e, "decode_from: from_utf8 key failed"); - Error::FromUtf8(e.to_string()) + let key = read_small_str(rd, &mut inline_key, &mut heap_key).inspect_err(|e| { + tracing::error!(error = %e, "decode_from: read key failed"); })?; match key { @@ -2506,7 +2495,7 @@ impl MetaObject { Some(n) => n, }; self.meta_user.clear(); - self.meta_user.reserve(prealloc_hint(len)); + self.meta_user.reserve(prealloc_hint(len.saturating_add(self.meta_sys.len()))); for _ in 0..len { let k_len = rmp::decode::read_str_len(rd).map_err(|e| { tracing::error!(error = %e, "decode_from: read_str_len MetaUsr key failed"); @@ -2561,9 +2550,40 @@ impl MetaObject { path: &str, all_parts: bool, include_part_checksums: bool, + ) -> Result { + Self { + version_id: self.version_id, + data_dir: self.data_dir, + erasure_algorithm: self.erasure_algorithm.clone(), + erasure_m: self.erasure_m, + erasure_n: self.erasure_n, + erasure_block_size: self.erasure_block_size, + erasure_index: self.erasure_index, + erasure_dist: self.erasure_dist.clone(), + bitrot_checksum_algo: self.bitrot_checksum_algo.clone(), + part_numbers: self.part_numbers.clone(), + part_etags: if all_parts { self.part_etags.clone() } else { Vec::new() }, + part_sizes: self.part_sizes.clone(), + part_actual_sizes: self.part_actual_sizes.clone(), + part_indices: if all_parts { self.part_indices.clone() } else { Vec::new() }, + size: self.size, + mod_time: self.mod_time, + meta_sys: self.meta_sys.clone(), + meta_user: self.meta_user.clone(), + } + .into_fileinfo_with_part_checksums(volume, path, all_parts, include_part_checksums) + } + + fn into_fileinfo_with_part_checksums( + mut self, + volume: &str, + path: &str, + all_parts: bool, + include_part_checksums: bool, ) -> Result { let version_id = self.version_id.filter(|&vid| !vid.is_nil()); - let derived_metadata = DerivedInternalMetadata::from_meta_sys(&self.meta_sys)?; + let meta_sys = std::mem::take(&mut self.meta_sys); + let derived_metadata = DerivedInternalMetadata::from_meta_sys(&meta_sys)?; let parts = if all_parts { let n = self.part_numbers.len(); @@ -2578,74 +2598,39 @@ impl MetaObject { if n != 0 && (self.part_sizes.len() != n || self.part_actual_sizes.len() != n) { return Err(Error::FileCorrupt); } - let mut parts = vec![ObjectPartInfo::default(); n]; - - for (i, part) in parts.iter_mut().enumerate() { - part.number = self.part_numbers[i]; - part.size = self.part_sizes[i]; - part.actual_size = self.part_actual_sizes[i]; + let mut etags = (self.part_etags.len() == n).then(|| self.part_etags.into_iter()); + let mut indices = (self.part_indices.len() == n).then(|| self.part_indices.into_iter()); + let mut parts = Vec::with_capacity(n); + for i in 0..n { // etag/index stay soft-guarded: they are recomputable / optional, an empty // PartETags is legitimate MinIO interop, and a length mismatch there does // not corrupt returned data. - if self.part_etags.len() == n { - part.etag = self.part_etags[i].clone(); - } - - if self.part_indices.len() == n { - part.index = if self.part_indices[i].is_empty() { - None - } else { - Some(self.part_indices[i].clone()) - }; - } + parts.push(ObjectPartInfo { + number: self.part_numbers[i], + size: self.part_sizes[i], + actual_size: self.part_actual_sizes[i], + etag: etags.as_mut().and_then(Iterator::next).unwrap_or_default(), + index: indices.as_mut().and_then(Iterator::next).filter(|idx| !idx.is_empty()), + ..Default::default() + }); } parts } else { Vec::new() }; - let mut metadata = HashMap::with_capacity(self.meta_user.len() + self.meta_sys.len()); - for (k, v) in &self.meta_user { - if k == AMZ_META_UNENCRYPTED_CONTENT_LENGTH || k == AMZ_META_UNENCRYPTED_CONTENT_MD5 { - continue; - } - - if k == metadata_keys::STORAGE_CLASS && v == "STANDARD" { - continue; - } - - metadata.insert(k.to_owned(), v.to_owned()); - } - - for (k, v) in &self.meta_sys { - if has_internal_suffix(k, SUFFIX_TIER_FV_ID) || has_internal_suffix(k, SUFFIX_TIER_FV_MARKER) { - continue; - } - - if k.eq_ignore_ascii_case(metadata_keys::STORAGE_CLASS) && v == b"STANDARD" { - continue; - } - - if is_internal_key(k) { - metadata.insert(k.to_owned(), String::from_utf8(v.to_owned()).unwrap_or_default()); - } - } - - let replication_state_internal = get_internal_replication_state(&metadata); - - let mut deleted = false; - - if let Some(v) = replication_state_internal.as_ref() { - if !v.composite_version_purge_status().is_empty() { - deleted = true; - } - - let st = v.composite_replication_status(); - if !st.is_empty() { - metadata.insert(metadata_keys::REPLICATION_STATUS.to_string(), st.to_string()); - } + let mut metadata = self.meta_user; + metadata.remove(AMZ_META_UNENCRYPTED_CONTENT_LENGTH); + metadata.remove(AMZ_META_UNENCRYPTED_CONTENT_MD5); + if metadata.get(metadata_keys::STORAGE_CLASS).is_some_and(|v| v == "STANDARD") { + metadata.remove(metadata_keys::STORAGE_CLASS); } + let user_has_internal = metadata.keys().any(|k| is_internal_key(k)); + let mut has_replication_candidate = user_has_internal; + let mut has_part_checksums = user_has_internal || derived_metadata.part_checksums.is_some(); + let mut has_upload_integrity = user_has_internal; + let mut has_shard_or_inline_integrity = user_has_internal; let checksum = derived_metadata .checksum @@ -2658,7 +2643,7 @@ impl MetaObject { parity_blocks: self.erasure_n, block_size: self.erasure_block_size, index: self.erasure_index, - distribution: self.erasure_dist.iter().map(|&v| v as usize).collect(), + distribution: self.erasure_dist.into_iter().map(usize::from).collect(), ..Default::default() }; @@ -2677,11 +2662,6 @@ impl MetaObject { if derived_metadata.transitioned_version_state.is_some() { validate_transition_version_state(transition_version_state, transition_version.as_deref())?; } - for (key, value) in &self.meta_sys { - if is_transition_version_metadata_key(key) { - metadata.insert(key.to_owned(), transition_version_metadata_value(value, transition_version.as_deref())); - } - } let transition_version_id = transition_version.as_deref().and_then(|value| Uuid::parse_str(value).ok()); let transition_tier = derived_metadata .transition_tier @@ -2689,6 +2669,68 @@ impl MetaObject { .map(|v| String::from_utf8_lossy(v).to_string()) .unwrap_or_default(); + metadata.reserve(meta_sys.len()); + for (k, v) in meta_sys { + if has_internal_suffix(&k, SUFFIX_TIER_FV_ID) || has_internal_suffix(&k, SUFFIX_TIER_FV_MARKER) { + continue; + } + let Some(suffix) = strip_internal_prefix_preserving_case(&k) else { + continue; + }; + if suffix.eq_ignore_ascii_case(SUFFIX_TRANSITIONED_VERSION_ID) { + let val = transition_version_metadata_value(&v, transition_version.as_deref()); + metadata.insert(k, val); + continue; + } + if !k.is_ascii() { + has_replication_candidate = true; + has_part_checksums = true; + has_upload_integrity = true; + has_shard_or_inline_integrity = true; + } else { + match suffix.as_bytes().first().map(u8::to_ascii_lowercase) { + Some(b'p') => { + if suffix.eq_ignore_ascii_case(SUFFIX_PURGESTATUS) { + has_replication_candidate = true; + } else if suffix.eq_ignore_ascii_case(SUFFIX_PART_CHECKSUMS) { + has_part_checksums = true; + } + } + Some(b'r') => has_replication_candidate = true, + Some(b's') => { + if suffix.eq_ignore_ascii_case(crate::shard_integrity::SUFFIX_UPLOAD_INTEGRITY) { + has_upload_integrity = true; + } else if suffix.eq_ignore_ascii_case(crate::shard_integrity::SUFFIX_SHARD_INTEGRITY) + || suffix.eq_ignore_ascii_case(crate::shard_integrity::SUFFIX_INLINE_INTEGRITY) + { + has_shard_or_inline_integrity = true; + } + } + _ => {} + } + } + metadata.insert(k, String::from_utf8(v).unwrap_or_default()); + } + + let replication_state_internal = if has_replication_candidate { + get_internal_replication_state(&metadata) + } else { + None + }; + + let mut deleted = false; + + if let Some(v) = replication_state_internal.as_ref() { + if !v.composite_version_purge_status().is_empty() { + deleted = true; + } + + let st = v.composite_replication_status(); + if !st.is_empty() { + metadata.insert(metadata_keys::REPLICATION_STATUS.to_string(), st.to_string()); + } + } + let mut file_info = FileInfo { version_id, erasure, @@ -2710,15 +2752,18 @@ impl MetaObject { transition_tier, ..Default::default() }; - if all_parts && include_part_checksums { + if all_parts && include_part_checksums && has_part_checksums { file_info.hydrate_data_movement_part_checksums()?; } if !self.part_numbers.is_empty() + && has_upload_integrity && rustfs_utils::http::contains_key_str(&file_info.metadata, crate::shard_integrity::SUFFIX_UPLOAD_INTEGRITY) { return Err(Error::FileCorrupt); } - if let Some(commitments) = crate::shard_integrity::descriptor_from_metadata(&file_info.metadata)? { + if has_shard_or_inline_integrity + && let Some(commitments) = crate::shard_integrity::descriptor_from_metadata(&file_info.metadata)? + { let layout = crate::shard_integrity::IntegrityLayout::new( self.erasure_m, self.erasure_n, @@ -2938,6 +2983,7 @@ impl From for MetaObject { pub fn get_internal_replication_state(metadata: &HashMap) -> Option { let mut rs = ReplicationState::default(); let mut has = false; + let mut has_delete_marker_version_keys = false; for (k, v) in metadata.iter() { if has_internal_suffix(k, SUFFIX_PURGESTATUS) { @@ -2961,6 +3007,11 @@ pub fn get_internal_replication_state(metadata: &HashMap) -> Opt has = true; rs.replication_status_internal = Some(v.clone()); rs.targets = replication_statuses_map(v.as_str()); + } else if sub_key + .get(..SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX.len()) + .is_some_and(|prefix| prefix.eq_ignore_ascii_case(SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX)) + { + has_delete_marker_version_keys = true; } else if let Some(arn) = rustfs_utils::http::internal_key_strip_suffix_prefix(k, SUFFIX_REPLICATION_RESET_ARN_PREFIX) { has = true; @@ -2981,9 +3032,12 @@ pub fn get_internal_replication_state(metadata: &HashMap) -> Opt // authority. `corrupt` means the dual internal prefixes disagreed: surface it // rather than guessing, so callers fail closed instead of purging the wrong // target version. - (rs.target_delete_marker_version_ids, rs.target_delete_marker_version_ids_corrupt) = target_delete_marker_versions(metadata); - has |= !rs.target_delete_marker_version_ids.is_empty(); - has |= rs.target_delete_marker_version_ids_corrupt; + if has_delete_marker_version_keys { + (rs.target_delete_marker_version_ids, rs.target_delete_marker_version_ids_corrupt) = + target_delete_marker_versions(metadata); + has |= !rs.target_delete_marker_version_ids.is_empty(); + has |= rs.target_delete_marker_version_ids_corrupt; + } if has { Some(rs) } else { None } } @@ -3101,6 +3155,10 @@ impl MetaDeleteMarker { } pub fn encode_to(&self, wr: &mut W) -> Result<()> { + self.encode_to_with_meta(wr, true) + } + + fn encode_to_with_meta(&self, wr: &mut W, include_meta_sys: bool) -> Result<()> { rmp::encode::write_map_len(wr, 3)?; // ID @@ -3114,27 +3172,35 @@ impl MetaDeleteMarker { // MetaSys rmp::encode::write_str(wr, "MetaSys")?; - rmp::encode::write_map_len(wr, self.meta_sys.len() as u32)?; - for (k, v) in &self.meta_sys { - rmp::encode::write_str(wr, k)?; - rmp::encode::write_bin(wr, v)?; + if !include_meta_sys || self.meta_sys.is_empty() { + rmp::encode::write_map_len(wr, 0)?; + } else { + rmp::encode::write_map_len(wr, self.meta_sys.len() as u32)?; + for (k, v) in &self.meta_sys { + rmp::encode::write_str(wr, k)?; + rmp::encode::write_bin(wr, v)?; + } } Ok(()) } pub fn decode_from(&mut self, rd: &mut R) -> Result<()> { - let mut fields = rmp::decode::read_map_len(rd)?; + let fields = rmp::decode::read_map_len(rd)? as usize; + self.decode_fields(rd, fields) + } + + fn decode_fields(&mut self, rd: &mut R, mut fields: usize) -> Result<()> { *self = MetaDeleteMarker::default(); + let mut inline_key = [0u8; MSGP_OBJECT_KEY_STACK_CAP]; + let mut heap_key = Vec::new(); while fields > 0 { fields -= 1; - let key_len = rmp::decode::read_str_len(rd)?; - let key_buf = read_exact_vec(rd, key_len as usize)?; - let key = String::from_utf8(key_buf)?; + let key = read_small_str(rd, &mut inline_key, &mut heap_key)?; - match key.as_str() { + match key { "ID" => { let _ = rmp::decode::read_bin_len(rd)?; let mut buf = [0u8; 16]; @@ -3193,14 +3259,11 @@ impl MetaDeleteMarker { /// Compute this delete marker's header signature, mirroring MinIO's /// `xlMetaV2DeleteMarker.Signature`. See [`FileMetaVersion::get_signature`]. pub fn get_signature(&self) -> [u8; 4] { - let mut c = self.clone(); - // MetaSys is order-unstable across disks; fold it in separately. - let mut crc = hash_deterministic_bytes(&c.meta_sys); - c.meta_sys.clear(); - - if let Ok(bytes) = c.marshal_msg() { - crc ^= xxhash_rust::xxh64::xxh64(&bytes, XXHASH_SEED); + let mut crc = hash_deterministic_bytes(&self.meta_sys); + let mut hasher = Xxh64Writer(xxhash_rust::xxh64::Xxh64::new(XXHASH_SEED)); + if self.encode_to_with_meta(&mut hasher, false).is_ok() { + crc ^= hasher.0.finish(); } fold_signature(crc) @@ -3578,33 +3641,18 @@ pub struct FileInfoOpts { } pub fn get_file_info(buf: &[u8], volume: &str, path: &str, version_id: &str, opts: FileInfoOpts) -> Result { - let vid = { - if version_id.is_empty() { - None - } else { - Some(Uuid::parse_str(version_id)?) - } - }; - - let meta = FileMeta::load(buf)?; - if meta.versions.is_empty() { - return Ok(FileInfo { - volume: volume.to_owned(), - name: path.to_owned(), - version_id: vid, - is_latest: true, - deleted: true, - mod_time: Some(OffsetDateTime::from_unix_timestamp(1)?), - ..Default::default() - }); - } - - let fi = if opts.include_part_checksums { - meta.into_fileinfo(volume, path, version_id, opts.data, opts.include_free_versions, true)? - } else { - meta.into_fileinfo_without_part_checksums(volume, path, version_id, opts.data, opts.include_free_versions)? - }; - Ok(fi) + FileMeta::decode_fileinfo_from_buf( + buf, + volume, + path, + version_id, + FileInfoDecodeOptions { + read_data: opts.data, + include_free_versions: opts.include_free_versions, + all_parts: true, + include_part_checksums: opts.include_part_checksums, + }, + ) } async fn read_more( @@ -5465,6 +5513,135 @@ mod tests { ); } + #[test] + fn into_fileinfo_preserves_upload_integrity_and_non_ascii_checksum_guards() { + for key in [ + format!("{RUSTFS_INTERNAL_PREFIX}{}", crate::shard_integrity::SUFFIX_UPLOAD_INTEGRITY), + format!( + "{}{}", + rustfs_utils::http::MINIO_INTERNAL_PREFIX, + crate::shard_integrity::SUFFIX_UPLOAD_INTEGRITY.to_ascii_uppercase() + ), + ] { + for all_parts in [false, true] { + let mut obj = MetaObject { + part_numbers: vec![1], + part_sizes: vec![1024], + part_actual_sizes: vec![1024], + ..Default::default() + }; + obj.meta_sys.insert(key.clone(), b"1".to_vec()); + assert!( + matches!(obj.into_fileinfo("bucket", "object", all_parts), Err(Error::FileCorrupt)), + "upload-integrity marker {key} with parts must fail closed (all_parts={all_parts})" + ); + } + } + + for prefix in [RUSTFS_INTERNAL_PREFIX, rustfs_utils::http::MINIO_INTERNAL_PREFIX] { + let mut obj = MetaObject { + part_numbers: vec![1], + part_sizes: vec![1024], + part_actual_sizes: vec![1024], + ..Default::default() + }; + let non_ascii_suffix = SUFFIX_PART_CHECKSUMS.replacen('k', "\u{212A}", 1); + obj.meta_sys + .insert(format!("{prefix}{non_ascii_suffix}"), br#"[[1,[["CRC32","AAAAAA=="]]]]"#.to_vec()); + assert!( + matches!(obj.into_fileinfo("bucket", "object", true), Err(Error::FileCorrupt)), + "non-ASCII part-checksums alias under {prefix} must fail closed" + ); + } + } + + #[test] + fn get_file_info_zero_copy_matches_loaded_filemeta_and_rejects_trailing_corruption() { + for raw in [ + crate::test_data::create_real_xlmeta().expect("real xl.meta"), + crate::test_data::create_complex_xlmeta().expect("complex xl.meta"), + crate::test_data::create_realistic_object_xlmeta().expect("realistic xl.meta"), + ] { + let loaded = FileMeta::load(&raw).expect("load fixture"); + for include_part_checksums in [false, true] { + let direct = get_file_info( + &raw, + "bucket", + "object", + "", + FileInfoOpts { + data: true, + include_free_versions: false, + include_part_checksums, + }, + ) + .expect("direct get_file_info"); + let via_loaded = if include_part_checksums { + loaded + .into_fileinfo("bucket", "object", "", true, false, true) + .expect("loaded into_fileinfo") + } else { + loaded + .into_fileinfo_without_part_checksums("bucket", "object", "", true, false) + .expect("loaded into_fileinfo_without_part_checksums") + }; + assert_eq!(direct, via_loaded); + } + } + + let mut fm = FileMeta::load(&crate::test_data::create_complex_xlmeta().expect("complex xl.meta")).expect("load"); + // Make version[0]'s body decode to FileCorrupt (mismatched part_sizes) while + // truncating the trailing version entry: the trailing framing error must still + // take precedence over version[0]'s body error, matching FileMeta::load. + let mut v0 = fm.versions[0].parse_version_meta().expect("parse v0"); + v0.object.as_mut().expect("v0 object").part_sizes.clear(); + fm.versions[0].meta = v0.marshal_msg().expect("marshal v0"); + fm.versions[1].header = FileMetaVersionHeader::default(); + let mut bad_buf = fm.marshal_msg().expect("marshal"); + let (_, remaining) = FileMeta::read_bytes_header(&bad_buf[8..]).expect("read bytes header"); + let meta_len = remaining.len() - 5; + let mut truncated_meta = remaining[..meta_len - 10].to_vec(); + let new_crc = xxhash_rust::xxh64::xxh64(&truncated_meta, XXHASH_SEED) as u32; + bad_buf.truncate(8); + bad_buf.push(0xc6); + bad_buf.extend_from_slice(&(truncated_meta.len() as u32).to_be_bytes()); + bad_buf.append(&mut truncated_meta); + bad_buf.push(0xce); + bad_buf.extend_from_slice(&new_crc.to_be_bytes()); + + let direct_err = get_file_info( + &bad_buf, + "bucket", + "object", + "", + FileInfoOpts { + data: false, + include_free_versions: false, + include_part_checksums: false, + }, + ) + .expect_err("truncated trailing version entry must fail even when version[0] matches"); + let load_err = FileMeta::load(&bad_buf).expect_err("FileMeta::load must fail on truncated trailing version"); + assert_eq!(direct_err, load_err, "get_file_info must match FileMeta::load error precedence"); + + // Submap non-nil error variants (`{"V2Obj": 0}` and truncated map16 `{"V2Obj": 0xde}`) + // must remain RmpDecodeValueRead. + for bad_ver in [&b"\x81\xa5V2Obj\x00"[..], &b"\x81\xa5V2Obj\xde"[..]] { + let mut ver = FileMetaVersion::default(); + assert!( + matches!(ver.unmarshal_msg(bad_ver), Err(Error::RmpDecodeValueRead(_))), + "invalid V2Obj marker must return RmpDecodeValueRead" + ); + } + + // InlineData validation must reject an empty key before reading a missing value. + let err = InlineData::validate_slice(&[0x01, 0x81, 0xa0]).expect_err("empty inline key must fail"); + assert!( + matches!(&err, Error::Io(io_err) if io_err.to_string().contains("InlineData key empty")), + "expected InlineData key empty error, got {err:?}" + ); + } + #[test] fn get_internal_replication_state_accepts_rfc3339_and_rustfs_display_timestamps() { let replica_timestamp = OffsetDateTime::UNIX_EPOCH + time::Duration::SECOND; diff --git a/crates/filemeta/src/filemeta_inline.rs b/crates/filemeta/src/filemeta_inline.rs index 658433e1e..e289ea6dd 100644 --- a/crates/filemeta/src/filemeta_inline.rs +++ b/crates/filemeta/src/filemeta_inline.rs @@ -13,8 +13,9 @@ // limitations under the License. use crate::error::{Error, Result}; -use crate::filemeta::msgp_decode::{prealloc_hint, read_exact_vec}; +use crate::filemeta::msgp_decode::{MAX_MSGP_ELEMENT_SIZE, prealloc_hint, read_exact_vec}; use serde::{Deserialize, Serialize}; +use std::collections::HashSet; use std::io::Cursor; use uuid::Uuid; @@ -63,110 +64,98 @@ impl InlineData { Ok(false) } + fn borrowed_entries(&self) -> Result> { + let buf = self.after_version(); + if buf.is_empty() { + return Ok(Vec::new()); + } + let mut cur = Cursor::new(buf); + let count = usize::try_from(rmp::decode::read_map_len(&mut cur)?).map_err(Error::other)?; + let mut entries = Vec::with_capacity(prealloc_hint(count)); + for _ in 0..count { + let key_len = usize::try_from(rmp::decode::read_str_len(&mut cur)?).map_err(Error::other)?; + if key_len > MAX_MSGP_ELEMENT_SIZE { + return Err(Error::other("InlineData key exceeds the msgpack element limit")); + } + let start = usize::try_from(cur.position()).map_err(Error::other)?; + let end = start + .checked_add(key_len) + .ok_or_else(|| Error::other("InlineData key out of range"))?; + let key = buf + .get(start..end) + .ok_or_else(|| Error::other("InlineData key out of range"))?; + let key = std::str::from_utf8(key).map_err(|err| Error::FromUtf8(err.to_string()))?; + cur.set_position(u64::try_from(end).map_err(Error::other)?); + + let value_len = usize::try_from(rmp::decode::read_bin_len(&mut cur)?).map_err(Error::other)?; + let start = usize::try_from(cur.position()).map_err(Error::other)?; + let end = start + .checked_add(value_len) + .ok_or_else(|| Error::other("InlineData value out of range"))?; + let value = buf + .get(start..end) + .ok_or_else(|| Error::other("InlineData value out of range"))?; + cur.set_position(u64::try_from(end).map_err(Error::other)?); + entries.push((key, value)); + } + Ok(entries) + } + + pub(crate) fn key_set(&self) -> Result>> { + if !self.version_ok() { + return Err(Error::FileCorrupt); + } + Ok(self + .borrowed_entries()? + .into_iter() + .map(|(key, _)| key.as_bytes().to_vec()) + .collect()) + } + + pub(crate) fn remove_keys(&mut self, keys: &HashSet>) -> Result { + if keys.is_empty() { + return Ok(false); + } + self.remove_keys_by(|key| keys.contains(key)) + } + fn remove_keys_by(&mut self, mut should_remove: F) -> Result where F: FnMut(&[u8]) -> bool, { - let buf = self.after_version(); - if buf.is_empty() { + let mut entries = self.borrowed_entries()?; + let original_count = entries.len(); + entries.retain(|(key, _)| !should_remove(key.as_bytes())); + if entries.len() == original_count { return Ok(false); } - - let mut cur = Cursor::new(buf); - let mut fields_len = rmp::decode::read_map_len(&mut cur)? as usize; - let mut keys = Vec::with_capacity(prealloc_hint(fields_len)); - let mut values = Vec::with_capacity(prealloc_hint(fields_len)); - let mut found = false; - - while fields_len > 0 { - fields_len -= 1; - - let str_len = rmp::decode::read_str_len(&mut cur)? as usize; - let field_buf = read_exact_vec(&mut cur, str_len)?; - - let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize; - let start = cur.position() as usize; - let end = start + bin_len; - let value = buf - .get(start..end) - .ok_or_else(|| Error::other("InlineData value out of range"))?; - cur.set_position(end as u64); - - if should_remove(field_buf.as_slice()) { - found = true; - continue; - } - - keys.push(String::from_utf8(field_buf)?); - values.push(value.to_vec()); - } - - if !found { - return Ok(false); - } - - if keys.is_empty() { - self.0 = Vec::new(); + if entries.is_empty() { + self.0.clear(); return Ok(true); } - self.serialize(keys, values)?; + // Borrow payloads until the final encoding so every retained body is + // copied only once, regardless of the number of removed versions. + let capacity = entries.iter().try_fold(6_usize, |size, (key, value)| { + // A string and binary value each need at most five header bytes. + size.checked_add(key.len()) + .and_then(|size| size.checked_add(value.len())) + .and_then(|size| size.checked_add(10)) + .ok_or_else(|| Error::other("InlineData encoded size overflow")) + })?; + let mut encoded = Vec::with_capacity(capacity); + encoded.push(INLINE_DATA_VER); + rmp::encode::write_map_len(&mut encoded, u32::try_from(entries.len()).map_err(Error::other)?)?; + for (key, value) in entries { + rmp::encode::write_str(&mut encoded, key)?; + rmp::encode::write_bin(&mut encoded, value)?; + } + self.0 = encoded; Ok(true) } fn remove_two_keys_by_bytes(&mut self, first_key: &[u8], second_key: &[u8]) -> Result { - let buf = self.after_version(); - if buf.is_empty() { - return Ok(false); - } - - let same = first_key == second_key; - let mut cur = Cursor::new(buf); - let mut fields_len = rmp::decode::read_map_len(&mut cur)? as usize; - let mut keys = Vec::with_capacity(prealloc_hint(fields_len) + 1); - let mut values = Vec::with_capacity(prealloc_hint(fields_len) + 1); - let mut found = false; - - while fields_len > 0 { - fields_len -= 1; - - let str_len = rmp::decode::read_str_len(&mut cur)? as usize; - let field_buf = read_exact_vec(&mut cur, str_len)?; - - let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize; - let start = cur.position() as usize; - let end = start + bin_len; - let value = buf - .get(start..end) - .ok_or_else(|| Error::other("InlineData value out of range"))?; - cur.set_position(end as u64); - - let should_remove = if same { - field_buf.as_slice() == first_key - } else { - field_buf.as_slice() == first_key || field_buf.as_slice() == second_key - }; - - if should_remove { - found = true; - continue; - } - - keys.push(String::from_utf8(field_buf)?); - values.push(value.to_vec()); - } - - if !found { - return Ok(false); - } - - if keys.is_empty() { - self.0 = Vec::new(); - return Ok(true); - } - - self.serialize(keys, values)?; - Ok(true) + self.remove_keys_by(|key| key == first_key || key == second_key) } pub fn new() -> Self { @@ -204,35 +193,56 @@ impl InlineData { Ok(fields_len as usize) } + #[inline] + fn read_entry_slice<'a>(buf: &'a [u8], cur: &mut Cursor<&'a [u8]>, reject_empty_key: bool) -> Result<(&'a str, &'a [u8])> { + let str_len = rmp::decode::read_str_len(cur)? as usize; + if str_len > MAX_MSGP_ELEMENT_SIZE { + return Err(Error::other(format!( + "corrupt msgpack element: length {str_len} exceeds the {MAX_MSGP_ELEMENT_SIZE} byte limit" + ))); + } + let key_start = cur.position() as usize; + let key_end = key_start + .checked_add(str_len) + .ok_or_else(|| Error::other("InlineData key out of range"))?; + let key_bytes = buf + .get(key_start..key_end) + .ok_or_else(|| Error::from(std::io::Error::new(std::io::ErrorKind::UnexpectedEof, "failed to fill whole buffer")))?; + let field = std::str::from_utf8(key_bytes).map_err(|e| Error::FromUtf8(e.to_string()))?; + if reject_empty_key && field.is_empty() { + return Err(Error::other("InlineData key empty")); + } + cur.set_position(key_end as u64); + + let bin_len = rmp::decode::read_bin_len(cur)? as usize; + let start = cur.position() as usize; + let end = start + .checked_add(bin_len) + .ok_or_else(|| Error::other("InlineData value out of range"))?; + let value = buf + .get(start..end) + .ok_or_else(|| Error::other("InlineData value out of range"))?; + cur.set_position(end as u64); + Ok((field, value)) + } + pub fn find(&self, key: &str) -> Result>> { - if self.0.is_empty() || !self.version_ok() { + Self::find_in_slice(&self.0, key) + } + + pub(crate) fn find_in_slice(raw: &[u8], key: &str) -> Result>> { + if raw.is_empty() || raw[0] == 0 || raw[0] > INLINE_DATA_VER { return Ok(None); } - let buf = self.after_version(); - + let buf = &raw[1..]; let mut cur = Cursor::new(buf); - let mut fields_len = rmp::decode::read_map_len(&mut cur)?; while fields_len > 0 { fields_len -= 1; - - let str_len = rmp::decode::read_str_len(&mut cur)?; - - let field_buff = read_exact_vec(&mut cur, str_len as usize)?; - - let field = String::from_utf8(field_buff)?; - - let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize; - let start = cur.position() as usize; - let end = start + bin_len; - let value = buf - .get(start..end) - .ok_or_else(|| Error::other("InlineData value out of range"))?; - cur.set_position(end as u64); - - if field.as_str() == key { + let (field, value) = Self::read_entry_slice(buf, &mut cur, false)?; + if field == key { return Ok(Some(value.to_vec())); } } @@ -241,34 +251,21 @@ impl InlineData { } pub fn validate(&self) -> Result<()> { - if self.0.is_empty() { + Self::validate_slice(&self.0) + } + + pub(crate) fn validate_slice(raw: &[u8]) -> Result<()> { + if raw.is_empty() { return Ok(()); } - let buf = self.after_version(); + let buf = &raw[1..]; let mut cur = Cursor::new(buf); - let mut fields_len = rmp::decode::read_map_len(&mut cur)?; while fields_len > 0 { fields_len -= 1; - - let str_len = rmp::decode::read_str_len(&mut cur)?; - - let field_buff = read_exact_vec(&mut cur, str_len as usize)?; - - let field = String::from_utf8(field_buff)?; - if field.is_empty() { - return Err(Error::other("InlineData key empty")); - } - - let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize; - let start = cur.position() as usize; - let end = start + bin_len; - if end > buf.len() { - return Err(Error::other("InlineData value out of range")); - } - cur.set_position(end as u64); + let _ = Self::read_entry_slice(buf, &mut cur, true)?; } Ok(()) @@ -420,4 +417,38 @@ mod tests { Some(b"keep".to_vec()) ); } + + #[test] + fn remove_keys_preserves_wire_bytes_and_rejects_truncated_payloads() { + let mut original = InlineData::new(); + for id in 0_u128..8 { + original + .replace(&Uuid::from_u128(id).to_string(), vec![u8::try_from(id).expect("small id"); 128]) + .expect("seed inline map"); + } + for mask in 0_u32..256 { + let keys: HashSet<_> = (0_u128..8) + .filter(|id| mask & (1 << id) != 0) + .map(|id| Uuid::from_u128(id).to_string().into_bytes()) + .collect(); + let mut actual = original.clone(); + actual.remove_keys(&keys).expect("batch removal"); + let retained: Vec<_> = (0_u128..8).filter(|id| mask & (1 << id) == 0).collect(); + let mut expected = InlineData::new(); + for id in retained { + expected + .replace(&Uuid::from_u128(id).to_string(), vec![u8::try_from(id).expect("small id"); 128]) + .expect("encode survivor"); + } + assert_eq!(actual, expected, "mask={mask}"); + } + let keys = HashSet::from([Uuid::nil().to_string().into_bytes()]); + for end in 2..original.as_slice().len() { + let mut truncated = InlineData::new(); + truncated.update(&original.as_slice()[..end]); + let before = truncated.clone(); + assert!(truncated.remove_keys(&keys).is_err(), "end={end}"); + assert_eq!(truncated, before); + } + } } diff --git a/crates/filemeta/src/metacache.rs b/crates/filemeta/src/metacache.rs index 61a2e4d8e..3109fd07c 100644 --- a/crates/filemeta/src/metacache.rs +++ b/crates/filemeta/src/metacache.rs @@ -12,7 +12,6 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::filemeta::msgp_decode::MAX_MSGP_ELEMENT_SIZE; use crate::{ Error, FileInfo, FileInfoOpts, FileInfoVersions, FileMeta, FileMetaShallowVersion, Result, VersionType, get_file_info, merge_file_meta_versions, merge_file_meta_versions_with_write_quorum, @@ -840,6 +839,7 @@ impl MetaCacheEntriesSorted { } const METACACHE_STREAM_VERSION: u8 = 2; +const METACACHE_READ_CHUNK_SIZE: usize = 64 << 10; #[derive(Debug)] pub struct MetacacheWriter { @@ -912,10 +912,10 @@ impl MetacacheWriter { pub struct MetacacheReader { rd: R, init: bool, + in_entry: bool, err: Option, buf: Vec, - offset: usize, - current: Option, + has_current: bool, } impl MetacacheReader { @@ -923,49 +923,66 @@ impl MetacacheReader { Self { rd, init: false, + in_entry: false, err: None, buf: Vec::new(), - offset: 0, - current: None, + has_current: false, } } pub async fn read_more(&mut self, read_size: usize) -> Result<&[u8]> { - // `read_size` is usually a length decoded from the stream itself, so - // it is untrusted: a corrupted length prefix must yield a decode - // error instead of a huge allocation that aborts the process - // (see rustfs/rustfs#2715). - if read_size > MAX_MSGP_ELEMENT_SIZE { - let err = Error::other(format!( - "metacache stream corrupt: element length {read_size} exceeds the {MAX_MSGP_ELEMENT_SIZE} byte limit" - )); - self.err = Some(err.clone()); - return Err(err); + if let Some(err) = &self.err { + return Err(err.clone()); } - let pref = self.offset; - let ext_size = pref + read_size; + let pref = self.buf.len(); + let Some(ext_size) = pref.checked_add(read_size) else { + let err = Error::other("metacache stream: element length overflow"); + self.err = Some(err.clone()); + return Err(err); + }; - if self.buf.len() < ext_size { - let extra = ext_size - self.buf.len(); + // A whole xl.meta can legitimately exceed the limit on its individual + // msgpack values. Grow only as payload arrives, so a corrupt length + // prefix cannot cause a speculative allocation of the advertised size. + while self.buf.len() < ext_size { + let start = self.buf.len(); + let extra = (ext_size - start).min(METACACHE_READ_CHUNK_SIZE); if let Err(e) = self.buf.try_reserve(extra) { let err = Error::other(format!("metacache stream: buffer allocation of {extra} bytes failed: {e}")); self.err = Some(err.clone()); return Err(err); } - self.buf.resize(ext_size, 0); + self.buf.resize(start + extra, 0); + if let Err(e) = self.rd.read_exact(&mut self.buf[start..]).await { + // Listings treat Unexpected/UnexpectedEof as clean exhaustion. + // An incomplete entry must not hide remaining keys. Preserve + // the existing empty/between-entry EOF handling. + let err = match e.kind() { + std::io::ErrorKind::UnexpectedEof if self.in_entry => Error::FileCorrupt, + _ => Error::from(e), + }; + self.err = Some(err.clone()); + return Err(err); + } } - self.rd.read_exact(&mut self.buf[pref..ext_size]).await?; - - self.offset += read_size; - Ok(&self.buf[pref..ext_size]) } + async fn discard(&mut self, mut size: usize) -> Result<()> { + self.reset(); + while size > 0 { + let chunk = size.min(METACACHE_READ_CHUNK_SIZE); + self.read_more(chunk).await?; + size -= chunk; + self.reset(); + } + Ok(()) + } + fn reset(&mut self) { self.buf.clear(); - self.offset = 0; } async fn check_init(&mut self) -> Result<()> { @@ -1048,11 +1065,15 @@ impl MetacacheReader { return Err(err.clone()); } + if size == 0 { + return Ok(()); + } + let mut n = size; - if self.current.is_some() { + if self.has_current { n -= 1; - self.current = None; + self.has_current = false; } while n > 0 { @@ -1069,10 +1090,12 @@ impl MetacacheReader { } }; + self.in_entry = true; let l = self.read_str_len().await?; - let _ = self.read_more(l as usize).await?; + self.discard(usize::try_from(l).map_err(Error::other)?).await?; let l = self.read_bin_len().await?; - let _ = self.read_more(l as usize).await?; + self.discard(usize::try_from(l).map_err(Error::other)?).await?; + self.in_entry = false; n -= 1; } @@ -1100,11 +1123,11 @@ impl MetacacheReader { } }; + self.in_entry = true; let l = self.read_str_len().await?; let buf = self.read_more(l as usize).await?; - let name_buf = buf.to_vec(); - let name = match from_utf8(&name_buf) { + let name = match from_utf8(buf) { Ok(decoded) => decoded.to_owned(), Err(err) => { self.err = Some(Error::other(err.to_string())); @@ -1118,17 +1141,16 @@ impl MetacacheReader { let metadata = buf.to_vec(); + self.in_entry = false; self.reset(); - let entry = Some(MetaCacheEntry { + self.has_current = true; + Ok(Some(MetaCacheEntry { name, metadata, cached: None, reusable: false, - }); - self.current = entry.clone(); - - Ok(entry) + })) } pub async fn read_all(&mut self) -> Result> { @@ -1294,6 +1316,41 @@ mod tests { assert_eq!(objs, nobjs); } + #[tokio::test] + async fn test_reader_consecutive_peeks_and_mixed_skips_preserve_entries() { + let entries: Vec<_> = [vec![], vec![1, 2, 3], vec![], vec![4, 5], vec![]] + .into_iter() + .enumerate() + .map(|(i, metadata)| MetaCacheEntry { + name: format!("object-{i}"), + metadata, + ..Default::default() + }) + .collect(); + let mut output = Cursor::new(Vec::new()); + let mut writer = MetacacheWriter::new(&mut output); + writer.write(&entries).await.expect("mixed metadata entries must encode"); + writer.close().await.expect("stream must close"); + let mut reader = MetacacheReader::new(Cursor::new(output.into_inner())); + assert_eq!(reader.peek().await.expect("first entry must decode"), Some(entries[0].clone())); + assert_eq!(reader.peek().await.expect("peek must advance without skip"), Some(entries[1].clone())); + reader.skip(0).await.expect("zero skip must preserve the current entry"); + reader + .skip(2) + .await + .expect("skip must consume the current and following entry"); + assert_eq!(reader.read_all().await.expect("remaining entries must decode"), entries[3..]); + } + + #[tokio::test] + async fn test_reader_invalid_utf8_name_remains_a_sticky_error() { + let stream = vec![0xcc, 2, 0xc3, 0xa1, 0xff, 0xc4, 0, 0xc2]; + let mut reader = MetacacheReader::new(Cursor::new(stream)); + let err = reader.peek().await.expect_err("invalid UTF-8 name must be rejected"); + assert!(err.to_string().contains("utf-8")); + assert_eq!(reader.peek().await.expect_err("invalid name must leave a sticky error"), err); + } + #[tokio::test] async fn empty_writer_emits_a_valid_stream() { let mut output = Cursor::new(Vec::new()); @@ -1304,6 +1361,45 @@ mod tests { assert!(reader.read_all().await.expect("empty stream should decode").is_empty()); } + #[tokio::test] + async fn test_reader_preserves_empty_and_between_entry_eof() { + let mut empty = MetacacheReader::new(Cursor::new(Vec::::new())); + assert_eq!(empty.peek().await.expect_err("legacy empty producer must remain EOF"), Error::Unexpected); + + // A complete entry without a final false marker is a supported legacy EOF. + let stream = vec![0xcc, 2, 0xc3, 0xa1, b'x', 0xc4, 1, 0x42]; + let mut reader = MetacacheReader::new(Cursor::new(stream.clone())); + assert!(reader.peek().await.expect("complete entry must decode").is_some()); + reader.skip(1).await.expect("current entry must advance"); + assert_eq!(reader.peek().await.expect_err("between-entry EOF must remain EOF"), Error::Unexpected); + + let mut reader = MetacacheReader::new(Cursor::new(stream)); + reader.skip(1).await.expect("complete entry must be skippable"); + assert_eq!(reader.peek().await.expect_err("skip must restore between-entry state"), Error::Unexpected); + } + + #[tokio::test] + async fn test_reader_truncated_entry_is_corruption_at_each_field() { + // Missing name marker, name bytes, bin marker, bin length, or bin payload. + for suffix in [vec![], vec![0xa1], vec![0xa0], vec![0xa0, 0xc4], vec![0xa0, 0xc4, 1]] { + let mut stream = vec![0xcc, 2, 0xc3]; + stream.extend(suffix); + let mut reader = MetacacheReader::new(Cursor::new(stream.clone())); + assert_eq!( + reader.peek().await.expect_err("incomplete entry must not become listing EOF"), + Error::FileCorrupt + ); + let mut reader = MetacacheReader::new(Cursor::new(stream)); + assert_eq!( + reader + .skip(1) + .await + .expect_err("incomplete skipped entry must not become listing EOF"), + Error::FileCorrupt + ); + } + } + fn corrupt_stream_with_metadata_len(len_marker: u8, len: u32) -> Vec { let mut data = Vec::new(); rmp::encode::write_u8(&mut data, METACACHE_STREAM_VERSION).unwrap(); @@ -1324,10 +1420,14 @@ mod tests { let data = corrupt_stream_with_metadata_len(0xc6, u32::MAX); let mut r = MetacacheReader::new(Cursor::new(data)); let err = r.peek().await.expect_err("corrupt bin length prefix must fail to decode"); - assert!(err.to_string().contains("exceeds"), "error should mention the exceeded limit, got: {err}"); + assert_eq!(err, Error::FileCorrupt); + assert!( + r.buf.capacity() <= 2 * METACACHE_READ_CHUNK_SIZE, + "a length prefix must not reserve its payload" + ); // The reader must stay in the error state instead of retrying. - assert!(r.peek().await.is_err()); + assert_eq!(r.peek().await.expect_err("decode errors must remain sticky"), err); } #[tokio::test] @@ -1341,35 +1441,150 @@ mod tests { let mut r = MetacacheReader::new(Cursor::new(data)); let err = r.peek().await.expect_err("corrupt str length prefix must fail to decode"); - assert!(err.to_string().contains("exceeds"), "error should mention the exceeded limit, got: {err}"); + assert_eq!(err, Error::FileCorrupt); + assert!( + r.buf.capacity() <= 2 * METACACHE_READ_CHUNK_SIZE, + "a length prefix must not reserve its payload" + ); + assert_eq!(r.peek().await.expect_err("decode errors must remain sticky"), err); } #[tokio::test] async fn test_reader_skip_rejects_corrupt_length_prefix() { let data = corrupt_stream_with_metadata_len(0xc6, u32::MAX); let mut r = MetacacheReader::new(Cursor::new(data)); - assert!(r.skip(1).await.is_err(), "skip over a corrupt length prefix must fail"); + let err = r.skip(1).await.expect_err("skip over a corrupt length prefix must fail"); + assert_eq!(err, Error::FileCorrupt); + assert!(r.buf.capacity() <= METACACHE_READ_CHUNK_SIZE); + assert_eq!(r.skip(1).await.expect_err("skip errors must remain sticky"), err); + assert_eq!(r.peek().await.expect_err("skip failure must prevent decoding another entry"), err); } - /// Large-but-legitimate records (well above typical sizes, below the - /// corruption guard) must still round-trip. + #[tokio::test] + async fn test_reader_truncated_payload_after_multiple_chunks() { + let mut data = corrupt_stream_with_metadata_len(0xc6, u32::MAX); + data.extend(std::iter::repeat_n(0xab, 2 * METACACHE_READ_CHUNK_SIZE + 7)); + let mut reader = MetacacheReader::new(Cursor::new(data)); + let err = reader.peek().await.expect_err("partial payload must fail"); + assert_eq!(err, Error::FileCorrupt); + assert!(reader.buf.capacity() <= 8 * METACACHE_READ_CHUNK_SIZE); + assert_eq!(reader.peek().await.expect_err("partial payload must leave a sticky error"), err); + } + + /// The stream carries a whole xl.meta, whose aggregate size can exceed + /// the limit for an individual value inside its metadata. #[tokio::test] async fn test_reader_accepts_large_legitimate_metadata() { + for size in [16 << 20, (16 << 20) + 1] { + let entry = MetaCacheEntry { + name: "big-object".to_string(), + metadata: vec![0xab; size], + ..Default::default() + }; + + let mut f = Cursor::new(Vec::new()); + let mut w = MetacacheWriter::new(&mut f); + w.write(std::slice::from_ref(&entry)).await.expect("large entry must encode"); + w.close().await.expect("large-entry stream must close"); + + let mut r = MetacacheReader::new(Cursor::new(f.into_inner())); + let decoded = r.read_all().await.expect("large entry must decode"); + assert_eq!(decoded, vec![entry]); + } + } + + #[tokio::test] + async fn test_reader_large_inline_version_history_remains_listable() { + const VERSIONS: u32 = 1700; + let payload = vec![0xab; 10 << 10]; + let mut meta = FileMeta::new(); + let mut inline = vec![1]; + rmp::encode::write_map_len(&mut inline, VERSIONS).expect("inline version map must encode"); + for i in 1..=VERSIONS { + let version_id = Uuid::from_u128(u128::from(i)); + let mut fi = FileInfo { + version_id: Some(version_id), + mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(i64::from(i))), + size: i64::try_from(payload.len()).expect("test payload size must fit"), + ..Default::default() + }; + fi.set_inline_data(); + meta.add_version(fi).expect("inline version metadata must be valid"); + rmp::encode::write_str(&mut inline, &version_id.to_string()).expect("inline version ID must encode"); + rmp::encode::write_bin(&mut inline, &payload).expect("inline payload must encode"); + } + // Encode the inline map once; adding payload with each version would + // repeatedly copy the growing map while constructing this fixture. + meta.data.update(&inline); + let metadata = meta.marshal_msg().expect("large xl.meta must encode"); + assert!(metadata.len() > 16 << 20); let entry = MetaCacheEntry { - name: "big-object".to_string(), - metadata: vec![0xab; 4 << 20], - cached: None, - reusable: false, + name: "versioned-object".to_string(), + metadata, + ..Default::default() }; + let mut output = Cursor::new(Vec::new()); + let mut writer = MetacacheWriter::new(&mut output); + writer + .write(std::slice::from_ref(&entry)) + .await + .expect("large xl.meta must enter the stream"); + writer.close().await.expect("stream must close"); + let mut reader = MetacacheReader::new(Cursor::new(output.into_inner())); + let decoded = reader + .peek() + .await + .expect("large xl.meta must decode") + .expect("object must be present"); + assert_eq!(decoded.metadata, entry.metadata); + let versions = decoded + .file_info_versions("bucket") + .expect("version listing must decode xl.meta"); + assert_eq!(versions.versions.len(), usize::try_from(VERSIONS).expect("version count must fit")); + let latest_id = Uuid::from_u128(u128::from(VERSIONS)); + assert_eq!(versions.versions[0].version_id, Some(latest_id)); + assert_eq!(versions.versions[0].size, i64::try_from(payload.len()).expect("payload size must fit")); + let loaded = FileMeta::load(&decoded.metadata).expect("stream must preserve valid xl.meta"); + assert_eq!( + loaded.data.find(&latest_id.to_string()).expect("inline payload must decode"), + Some(payload) + ); + assert!(reader.peek().await.expect("stream must terminate").is_none()); + } - let mut f = Cursor::new(Vec::new()); - let mut w = MetacacheWriter::new(&mut f); - w.write(std::slice::from_ref(&entry)).await.unwrap(); - w.close().await.unwrap(); - - let mut r = MetacacheReader::new(Cursor::new(f.into_inner())); - let decoded = r.read_all().await.unwrap(); - assert_eq!(decoded, vec![entry]); + #[tokio::test] + async fn test_reader_skip_large_entries_preserves_alignment_with_bounded_buffer() { + // Fixed v1/v2 framing: uint8 version, bool, fixstr name, bin32 payload. + // Build independently of MetacacheWriter to pin the existing wire shape. + for version in [1, 2] { + let mut stream = vec![0xcc, version]; + for size in [0, 2 * METACACHE_READ_CHUNK_SIZE, (16 << 20) + 1] { + stream.extend_from_slice(&[0xc3, 0xa1, b'x', 0xc6]); + stream.extend_from_slice(&u32::try_from(size).expect("payload length must fit bin32").to_be_bytes()); + stream.resize(stream.len() + size, 0xab); + } + stream.extend_from_slice(&[0xc3, 0xa1, b'y', 0xc4, 1, 0x42, 0xc2]); + let mut reader = MetacacheReader::new(Cursor::new(stream)); + reader.skip(3).await.expect("large entries must be skippable"); + assert!(reader.buf.is_empty()); + assert!( + reader.buf.capacity() <= METACACHE_READ_CHUNK_SIZE, + "skip must not retain discarded entries" + ); + let entry = reader + .peek() + .await + .expect("following entry must decode") + .expect("following entry must exist"); + assert_eq!(entry.name, "y"); + assert_eq!(entry.metadata, vec![0x42]); + reader + .skip(0) + .await + .expect("skipping zero with a current entry must be a no-op"); + reader.skip(1).await.expect("consume current entry without reading another"); + assert!(reader.peek().await.expect("stream must terminate").is_none()); + } } #[test] diff --git a/crates/filemeta/tests/xl_meta_codec_compat.rs b/crates/filemeta/tests/xl_meta_codec_compat.rs new file mode 100644 index 000000000..6a8cab5f9 --- /dev/null +++ b/crates/filemeta/tests/xl_meta_codec_compat.rs @@ -0,0 +1,258 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use bytes::Bytes; +use rustfs_filemeta::{ + Error, FileInfo, FileInfoOpts, FileMeta, FileMetaShallowVersion, FileMetaVersion, FileMetaVersionHeader, InlineData, + MetaDeleteMarker, VersionType, get_file_info, test_data::*, +}; +use rustfs_utils::http::{SUFFIX_FREE_VERSION, insert_bytes}; +use time::OffsetDateTime; +use uuid::Uuid; + +fn read_options(data: bool, include_free_versions: bool, include_part_checksums: bool) -> FileInfoOpts { + FileInfoOpts { + data, + include_free_versions, + include_part_checksums, + } +} + +#[test] +fn historical_codec_preserves_signatures_and_fileinfo() { + // Signatures were captured from the pre-optimization implementation at + // 6681dd12e39fda867c54cc4e576ecf7f4257aec4, not from a same-code round trip. + // MinIO fixtures pin RustFS recomputation, not MinIO's different hash algorithm. + let fixtures = [ + ( + "minio-small", + create_minio_small_object_xlmeta().expect("historical fixture"), + vec![[63, 254, 115, 180]], + ), + ( + "minio-versioned", + create_minio_versioned_object_xlmeta().expect("historical fixture"), + vec![[179, 89, 130, 162], [42, 145, 112, 214], [91, 28, 96, 74]], + ), + ( + "minio-large", + create_minio_large_object_xlmeta().expect("historical fixture"), + vec![[95, 172, 233, 51]], + ), + ( + "legacy-v1", + create_legacy_v1_object_xlmeta().expect("historical fixture"), + vec![[130, 178, 104, 208]], + ), + ( + "issue-2288", + create_issue_2288_legacy_xlmeta().expect("historical fixture"), + vec![[233, 25, 148, 246]], + ), + ( + "issue-2265-object", + create_issue_2265_legacy_meta_v2_object_xlmeta().expect("historical fixture"), + vec![[64, 154, 203, 216]], + ), + ( + "issue-2265-config", + create_issue_2265_legacy_meta_v2_config_xlmeta().expect("historical fixture"), + vec![[83, 146, 54, 27]], + ), + ( + "issue-2434-pool", + create_issue_2434_legacy_meta_v2_pool_xlmeta().expect("historical fixture"), + vec![[91, 227, 222, 255]], + ), + ( + "pre-metadata-keys", + create_pre_metadata_keys_xlmeta().expect("historical fixture"), + vec![[5, 84, 29, 43]], + ), + ]; + for (name, raw, expected_signatures) in fixtures { + let loaded = FileMeta::load(&raw).expect("load historical metadata"); + let signatures: Vec<_> = loaded + .versions + .iter() + .map(|version| version.parse_version_meta().expect("historical version").get_signature()) + .collect(); + assert_eq!(signatures, expected_signatures, "signature changed for {name}"); + + let mut selectors = vec![String::new(), Uuid::nil().to_string(), Uuid::from_u128(u128::MAX).to_string()]; + selectors.extend( + loaded + .versions + .iter() + .filter_map(|v| v.header.version_id.map(|id| id.to_string())), + ); + for selector in selectors { + for data in [false, true] { + for free in [false, true] { + for checksums in [false, true] { + let direct = get_file_info(&raw, "bucket", "object", &selector, read_options(data, free, checksums)); + let expected = if checksums { + loaded.into_fileinfo("bucket", "object", &selector, data, free, true) + } else { + loaded.into_fileinfo_without_part_checksums("bucket", "object", &selector, data, free) + }; + assert_eq!( + direct, expected, + "{name}: selector={selector}, data={data}, free={free}, checksums={checksums}" + ); + } + } + } + } + } +} + +#[test] +fn direct_fileinfo_reads_both_null_inline_keys() { + let mut meta = FileMeta::new(); + let payload = Bytes::from_static(b"legacy-inline"); + let mut fi = FileInfo::new("object", 2, 1); + fi.mod_time = Some(OffsetDateTime::from_unix_timestamp(1).expect("timestamp")); + fi.data = Some(payload.clone()); + fi.set_inline_data(); + meta.add_version(fi).expect("add inline null version"); + + for key in ["null".to_owned(), Uuid::nil().to_string()] { + meta.data = InlineData::default(); + meta.data.replace(&key, payload.to_vec()).expect("set inline key"); + let raw = meta.marshal_msg().expect("encode null version"); + for selector in ["", "00000000-0000-0000-0000-000000000000"] { + for data in [false, true] { + let fi = get_file_info(&raw, "bucket", "object", selector, read_options(data, false, true)) + .expect("read null version"); + assert_eq!(fi.version_id, None); + assert!(fi.is_latest); + assert_eq!(fi.num_versions, 1); + assert_eq!(fi.data, data.then(|| payload.clone()), "key={key}, selector={selector}"); + } + } + } +} + +#[test] +fn direct_fileinfo_preserves_free_version_selection() { + let mut marker = MetaDeleteMarker { + version_id: Some(Uuid::from_u128(1)), + mod_time: Some(OffsetDateTime::from_unix_timestamp(1).expect("timestamp")), + ..Default::default() + }; + insert_bytes(&mut marker.meta_sys, SUFFIX_FREE_VERSION, Vec::new()); + let mut meta = FileMeta::new(); + meta.versions.push( + FileMetaShallowVersion::try_from(FileMetaVersion { + version_type: VersionType::Delete, + delete_marker: Some(marker), + ..Default::default() + }) + .expect("free version"), + ); + let raw = meta.marshal_msg().expect("encode free version"); + assert_eq!( + get_file_info(&raw, "bucket", "object", "", read_options(false, false, true)), + Err(Error::FileNotFound) + ); + let free_info = get_file_info(&raw, "bucket", "object", "", read_options(false, true, true)).expect("include free version"); + assert!(free_info.tier_free_version()); + assert!(free_info.is_latest); + assert_eq!(free_info.num_versions, 0); + // Explicit addressing of the free record must retain the existing behavior. + let explicit = get_file_info( + &raw, + "bucket", + "object", + &Uuid::from_u128(1).to_string(), + read_options(false, false, true), + ) + .expect("explicit free version"); + assert_eq!(explicit, free_info); + + let object = FileMeta::load(&create_minio_small_object_xlmeta().expect("object fixture")).expect("load object"); + meta.versions.extend(object.versions); + let raw = meta.marshal_msg().expect("encode free and ordinary versions"); + for include_free in [false, true] { + let fi = get_file_info(&raw, "bucket", "object", "", read_options(false, include_free, true)) + .expect("ordinary object remains visible"); + assert!(!fi.deleted); + assert_eq!(fi.size, 19); + assert_eq!(fi.num_versions, 1); + } +} + +#[test] +fn direct_fileinfo_preserves_empty_history_and_invalid_selectors() { + let raw = create_empty_xlmeta().expect("empty history"); + for selector in [ + "", + "00000000-0000-0000-0000-000000000000", + "00000000-0000-0000-0000-000000000001", + ] { + let fi = + get_file_info(&raw, "bucket", "object", selector, read_options(true, true, true)).expect("empty history tombstone"); + assert_eq!( + fi.version_id, + if selector.is_empty() { + None + } else { + Some(Uuid::parse_str(selector).expect("UUID")) + } + ); + assert!(fi.deleted && fi.is_latest); + assert_eq!(fi.mod_time, Some(OffsetDateTime::from_unix_timestamp(1).expect("timestamp"))); + assert_eq!(fi.num_versions, 0); + } + for raw in [raw.as_slice(), &[]] { + assert!(matches!( + get_file_info(raw, "bucket", "object", "invalid", read_options(false, false, false)), + Err(Error::UuidParse(_)) + )); + } +} + +#[test] +fn large_version_histories_fit_initial_serialization_buffer() { + for count in [4, 128] { + for body_len in [255, 256, 65_535, 65_536] { + let mut meta = FileMeta::new(); + meta.versions = vec![ + FileMetaShallowVersion { + // Use the widest flags and erasure fields; version_type remains a fixint. + header: FileMetaVersionHeader { + version_id: Some(Uuid::nil()), + mod_time: Some(OffsetDateTime::from_unix_timestamp(1).expect("timestamp")), + version_type: VersionType::Object, + flags: u8::MAX, + ec_n: u8::MAX, + ec_m: u8::MAX, + ..Default::default() + }, + meta: vec![0; body_len], + }; + count + ]; + let encoded = meta.marshal_msg().expect("serialize large history"); + assert!( + encoded.capacity() - encoded.len() <= count * 4 + 32, + "unexpected buffer growth for {count} bodies of {body_len} bytes: length={}, capacity={}", + encoded.len(), + encoded.capacity() + ); + assert_eq!(FileMeta::load(&encoded).expect("load serialized history").versions, meta.versions); + } + } +} diff --git a/crates/heal/src/error.rs b/crates/heal/src/error.rs index d1db7292a..ac901e921 100644 --- a/crates/heal/src/error.rs +++ b/crates/heal/src/error.rs @@ -90,6 +90,9 @@ pub enum Error { source: Box, }, + #[error("Bucket discovery incomplete for the requested heal scope; not all buckets could be traversed")] + BucketDiscoveryIncomplete, + #[error("Invalid heal type: {heal_type}")] InvalidHealType { heal_type: String }, @@ -127,7 +130,7 @@ impl Error { match self { Error::TaskCancelled | Error::TaskTimeout | Error::StaleBucketIncarnation { .. } => false, Error::ReplacementTargetNotReady(_) | Error::DanglingDeleteDeferred { .. } => true, - Error::TransientSkip { .. } => true, + Error::TransientSkip { .. } | Error::BucketDiscoveryIncomplete => true, // Lock failures classify by LockError's own taxonomy: only the // fatal variants (ResourceNotFound / PermissionDenied / // Configuration) are terminal - retrying cannot fix them - while diff --git a/crates/heal/src/heal/channel.rs b/crates/heal/src/heal/channel.rs index 20b37eff2..5e78e43d8 100644 --- a/crates/heal/src/heal/channel.rs +++ b/crates/heal/src/heal/channel.rs @@ -853,6 +853,13 @@ mod tests { async fn get_bucket_info(&self, _bucket: &str) -> crate::Result> { Ok(None) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> crate::Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> crate::Result> { Ok(vec![]) } diff --git a/crates/heal/src/heal/erasure_healer.rs b/crates/heal/src/heal/erasure_healer.rs index 5a02978ba..e1fdf7884 100644 --- a/crates/heal/src/heal/erasure_healer.rs +++ b/crates/heal/src/heal/erasure_healer.rs @@ -112,6 +112,7 @@ mod admin; pub struct ErasureSetHealer { admin_task: Option, + unresolved_delete_recovery: bool, storage: Arc, progress: Arc>, cancel_token: tokio_util::sync::CancellationToken, @@ -386,6 +387,7 @@ impl ErasureSetHealer { ) -> Self { Self { admin_task: None, + unresolved_delete_recovery: false, storage, progress, cancel_token, @@ -406,6 +408,11 @@ impl ErasureSetHealer { self } + pub(crate) fn with_unresolved_delete_recovery(mut self, unresolved: bool) -> Self { + self.unresolved_delete_recovery = unresolved; + self + } + pub(crate) fn with_replacement_execution(mut self, execution: Option>) -> Self { self.replacement_execution = execution; self @@ -644,6 +651,22 @@ impl ErasureSetHealer { }; let state = resume_manager.get_state().await; + if state.delete_recovery_rescan_pending + && state.completed_buckets.is_empty() + && state.resume_cursor.is_none() + && state.processed_objects == 0 + && state.successful_objects == 0 + && state.failed_objects == 0 + && state.skipped_objects == 0 + && state.skipped_new_versions == 0 + && state.skipped_ilm_expired == 0 + && state.processed_bytes == 0 + { + // Resume is published before its companion checkpoint reset. + // Finish that rewind after a crash before clearing the obligation. + checkpoint_manager.reset_after_delete_recovery().await?; + resume_manager.set_delete_recovery_rescan_pending(false).await?; + } if self.admin_task.is_none() && (state.retry_count > 0 || state.dangling_delete_retry_not_before.is_some()) && state.completed_buckets.is_empty() @@ -706,6 +729,9 @@ impl ErasureSetHealer { resume_manager: &ResumeManager, checkpoint_manager: &CheckpointManager, ) -> Result<()> { + if self.unresolved_delete_recovery && !self.heal_opts.dry_run { + resume_manager.set_delete_recovery_rescan_pending(true).await?; + } // 1. get current state let state = resume_manager.get_state().await; if self.replacement_task_id.is_some() @@ -812,13 +838,27 @@ impl ErasureSetHealer { resume_manager.mark_counter_unknown().await?; } let mut failed_buckets = 0u64; + let mut unresolved_delete_recovery = self.unresolved_delete_recovery; // 4. process remaining buckets - for (bucket_idx, bucket) in buckets.iter().enumerate().skip(current_bucket_index) { - // check if completed - if state.completed_buckets.contains(bucket) { - checkpoint_manager.complete_bucket(bucket_idx.saturating_add(1)).await?; - current_object_index = 0; + let first_bucket_index = if self.admin_task.is_some() { 0 } else { current_bucket_index }; + for (bucket_idx, bucket) in buckets.iter().enumerate().skip(first_bucket_index) { + if bucket_idx < current_bucket_index || state.completed_buckets.contains(bucket) { + // Object progress does not certify hidden delete transactions. + // Retry recovery in the admitted incarnation without replaying + // payload work already acknowledged by the ordinary cursor. + if self.admin_task.is_some() { + let incarnation = self.admin_bucket_incarnation(checkpoint_manager, bucket).await?; + let unresolved = self.heal_admin_bucket_delete_recovery(bucket, incarnation).await?; + if unresolved && !self.heal_opts.dry_run { + resume_manager.set_delete_recovery_rescan_pending(true).await?; + } + unresolved_delete_recovery |= unresolved; + } + if bucket_idx >= current_bucket_index { + checkpoint_manager.complete_bucket(bucket_idx.saturating_add(1)).await?; + current_object_index = 0; + } continue; } @@ -843,7 +883,7 @@ impl ErasureSetHealer { .await; if matches!(bucket_result, Err(Error::TaskCancelled | Error::TaskTimeout)) { - return bucket_result; + return bucket_result.map(|_| ()); } // update progress @@ -899,7 +939,8 @@ impl ErasureSetHealer { // process bucket result match bucket_result { - Ok(_) => { + Ok(unresolved) => { + unresolved_delete_recovery |= unresolved; resume_manager.complete_bucket(bucket).await?; checkpoint_manager.complete_bucket(bucket_idx.saturating_add(1)).await?; debug!( @@ -970,8 +1011,24 @@ impl ErasureSetHealer { .await?; } + if !unresolved_delete_recovery && resume_manager.get_state().await.delete_recovery_rescan_pending { + // Recovery can publish formerly hidden objects, including when a + // background worker settled and collected its proof before this + // retry. Aggregate counters only support a whole-set rewind. + resume_manager.restart_scan_after_delete_recovery().await?; + checkpoint_manager.reset_after_delete_recovery().await?; + resume_manager.set_delete_recovery_rescan_pending(false).await?; + return Err(Error::transient_skip("Delete recovery settled; ordinary coverage must be rescanned")); + } + if self.admin_task.is_some() { - return Ok(()); + return if unresolved_delete_recovery { + Err(Error::transient_skip( + "Delete recovery remains unresolved after ordinary erasure-set heal", + )) + } else { + Ok(()) + }; } // 5. finalize. Only declare the set healed when nothing failed AND @@ -1081,6 +1138,12 @@ impl ErasureSetHealer { ))); } + if unresolved_delete_recovery { + return Err(Error::transient_skip( + "Delete recovery remains unresolved after ordinary erasure-set heal", + )); + } + // No failures — ordinary heals are complete now. Replacement heals // atomically transition to Verified after the terminal identity fence. if self.replacement_task_id.is_none() { @@ -1353,7 +1416,40 @@ impl ErasureSetHealer { Ok(()) } - /// heal single bucket with resume + async fn heal_admin_bucket_delete_recovery(&self, bucket: &str, incarnation: Option) -> Result { + let Some(task) = &self.admin_task else { + return Ok(false); + }; + let bucket_opts = HealOpts { + recursive: false, + remove: false, + ..self.heal_opts + }; + let result = match incarnation { + Some(expected) => { + self.storage.validate_bucket_incarnation(bucket, Some(expected)).await?; + self.storage + .heal_bucket_at_incarnation(bucket, expected, &bucket_opts) + .await? + } + None => self.storage.heal_bucket(bucket, &bucket_opts).await?, + }; + task.record_result_item(result).await; + // Recovery retains the checkpoint's admitted incarnation, and the + // publication bucket must exist before adopting survivor delete proof. + if let Some(result) = self + .storage + .recover_delete_transactions(bucket, None, incarnation, &self.heal_opts) + .await? + { + let unresolved = result.delete_recovery.is_some_and(|status| status.has_unresolved()); + task.record_result_item(result).await; + return Ok(unresolved); + } + Ok(false) + } + + /// Heal one bucket's ordinary objects and report unresolved delete recovery separately. #[allow(clippy::too_many_arguments)] #[tracing::instrument(skip(self, current_object_index, processed_objects, successful_objects, failed_objects, skipped_objects, resume_manager, checkpoint_manager), fields(bucket = %bucket, bucket_index = bucket_index))] async fn heal_bucket_with_resume( @@ -1369,7 +1465,7 @@ impl ErasureSetHealer { resume_manager: &ResumeManager, checkpoint_manager: &CheckpointManager, started_at_secs: u64, - ) -> Result<()> { + ) -> Result { debug!( target: "rustfs::heal::erasure_healer", event = EVENT_HEAL_ERASURE_BUCKET_STATE, @@ -1384,22 +1480,9 @@ impl ErasureSetHealer { ); let incarnation = self.admin_bucket_incarnation(checkpoint_manager, bucket).await?; - if let Some(task) = &self.admin_task { - let bucket_opts = HealOpts { - recursive: false, - remove: false, - ..self.heal_opts - }; - let result = match incarnation { - Some(expected) => { - self.storage.validate_bucket_incarnation(bucket, Some(expected)).await?; - self.storage - .heal_bucket_at_incarnation(bucket, expected, &bucket_opts) - .await? - } - None => self.storage.heal_bucket(bucket, &bucket_opts).await?, - }; - task.record_result_item(result).await; + let unresolved_delete_recovery = self.heal_admin_bucket_delete_recovery(bucket, incarnation).await?; + if unresolved_delete_recovery && !self.heal_opts.dry_run { + resume_manager.set_delete_recovery_rescan_pending(true).await?; } // 1. get bucket info @@ -1416,7 +1499,7 @@ impl ErasureSetHealer { state = "missing", "Erasure set bucket heal skipped because bucket is missing" ); - return Ok(()); + return Ok(unresolved_delete_recovery); } }; @@ -1937,7 +2020,7 @@ impl ErasureSetHealer { previous_page_last = page_last; } - Ok(()) + Ok(unresolved_delete_recovery) } /// initialize progress tracking @@ -2390,6 +2473,8 @@ mod resume_loop_tests { #[derive(Default)] struct FakeStorage { admin_disk: Mutex>, + bucket_discovery_incomplete: AtomicBool, + bucket_discovery_scopes: Mutex>>, bucket_pages: Mutex>, incarnations: Mutex>, receipts: Mutex>>, @@ -2469,6 +2554,17 @@ mod resume_loop_tests { ..Default::default() })) } + async fn list_buckets_for_admin_heal( + &self, + selector: Option, + ) -> Result> { + self.bucket_discovery_scopes.lock().expect("discovery scopes").push(selector); + if self.bucket_discovery_incomplete.load(Ordering::SeqCst) { + return Err(Error::BucketDiscoveryIncomplete); + } + self.list_buckets().await + } + async fn list_buckets(&self) -> Result> { Ok(self .bucket_pages @@ -2796,7 +2892,7 @@ mod resume_loop_tests { state.start_time, ) .await; - (processed, successful, failed, skipped, result) + (processed, successful, failed, skipped, result.map(|_| ())) } #[tokio::test] @@ -3866,6 +3962,112 @@ mod resume_loop_tests { ); } + #[tokio::test] + async fn unresolved_delete_recovery_rechecks_coverage_only_after_settlement() { + let mut env = make_env().await; + env.healer.unresolved_delete_recovery = true; + env.storage.set_page( + None, + Page { + items: vec![item("healthy", None, false)], + next: None, + truncated: false, + }, + ); + + let error = env + .healer + .execute_heal_with_resume(&["b".to_owned()], "pool_0_set_0", &env.resume, &env.checkpoint) + .await + .expect_err("unresolved recovery cannot publish set completion"); + assert!(matches!(error, Error::TransientSkip { .. })); + assert_eq!(env.storage.calls(), [("healthy".to_owned(), None)]); + assert!(!env.resume.get_state().await.completed); + assert!(CheckpointManager::has_checkpoint(&env.healer.disk, &env.task_id).await); + env.healer + .execute_heal_with_resume(&["b".to_owned()], "pool_0_set_0", &env.resume, &env.checkpoint) + .await + .expect_err("pending-only retries remain unresolved"); + assert_eq!( + env.storage.calls(), + [("healthy".to_owned(), None)], + "pending-only retries do not repeat payload scans" + ); + + env.healer.unresolved_delete_recovery = false; + env.storage.set_page( + None, + Page { + items: vec![item("healthy", None, false), item("restored", None, false)], + next: None, + truncated: false, + }, + ); + env.healer + .execute_heal_with_resume(&["b".to_owned()], "pool_0_set_0", &env.resume, &env.checkpoint) + .await + .expect_err("settlement invalidates coverage even if another worker already collected the transaction"); + assert_eq!(env.storage.calls(), [("healthy".to_owned(), None)]); + assert_eq!( + env.resume.get_state().await.retry_count, + 0, + "coverage invalidation does not spend failure retries" + ); + env.healer + .execute_heal_with_resume(&["b".to_owned()], "pool_0_set_0", &env.resume, &env.checkpoint) + .await + .expect("fresh coverage includes the restored object"); + assert!(env.resume.get_state().await.completed); + assert_eq!(env.storage.calls().iter().filter(|(object, _)| object == "restored").count(), 1); + assert_eq!(env.resume.get_state().await.processed_objects, 2); + } + + #[tokio::test] + async fn delete_recovery_coverage_rewind_survives_interrupted_checkpoint_reset() { + let mut env = make_env().await; + env.healer.unresolved_delete_recovery = true; + env.storage.set_page( + None, + Page { + items: vec![item("healthy", None, false)], + next: None, + truncated: false, + }, + ); + env.healer + .execute_heal_with_resume(&["b".to_owned()], "pool_0_set_0", &env.resume, &env.checkpoint) + .await + .expect_err("create overlapping coverage"); + assert_eq!(env.checkpoint.get_checkpoint().await.current_bucket_index, 1); + env.resume + .restart_scan_after_delete_recovery() + .await + .expect("publish rewind before its companion checkpoint"); + + let (resume, checkpoint) = env + .healer + .initialize_resume_state(&env.task_id, "pool_0_set_0", &["b".to_owned()]) + .await + .expect("restart must finish the interrupted checkpoint rewind"); + assert!(!resume.get_state().await.delete_recovery_rescan_pending); + assert_eq!(checkpoint.get_checkpoint().await.current_bucket_index, 0); + env.healer.unresolved_delete_recovery = false; + env.storage.set_page( + None, + Page { + items: vec![item("healthy", None, false), item("restored", None, false)], + next: None, + truncated: false, + }, + ); + env.healer + .execute_heal_with_resume(&["b".to_owned()], "pool_0_set_0", &resume, &checkpoint) + .await + .expect("restarted coverage must visit newly visible objects"); + assert!(resume.get_state().await.completed); + assert_eq!(env.storage.calls().iter().filter(|(object, _)| object == "restored").count(), 1); + } + #[tokio::test] async fn test_transient_skip_pass_not_marked_completed_state_survives() { // The finalize block must NOT declare a clean completion when a pass had diff --git a/crates/heal/src/heal/erasure_healer/admin.rs b/crates/heal/src/heal/erasure_healer/admin.rs index 5c9a686cd..bb88cdded 100644 --- a/crates/heal/src/heal/erasure_healer/admin.rs +++ b/crates/heal/src/heal/erasure_healer/admin.rs @@ -51,9 +51,10 @@ impl ErasureSetHealer { } let resolved; let buckets = if buckets.is_empty() { + let (pool_idx, set_idx) = crate::heal::utils::parse_set_disk_id(set_disk_id)?; resolved = self .storage - .list_buckets() + .list_buckets_for_admin_heal(Some(crate::heal::storage_api::storage::DiskSetSelector::new(pool_idx, set_idx))) .await? .into_iter() .map(|bucket| bucket.name) diff --git a/crates/heal/src/heal/erasure_healer/resume_loop_tests/admin_outcome.rs b/crates/heal/src/heal/erasure_healer/resume_loop_tests/admin_outcome.rs index e893f8eca..30a51fbe7 100644 --- a/crates/heal/src/heal/erasure_healer/resume_loop_tests/admin_outcome.rs +++ b/crates/heal/src/heal/erasure_healer/resume_loop_tests/admin_outcome.rs @@ -483,3 +483,50 @@ async fn admin_erasure_checkpoint_publication_cannot_undo_acknowledged_cancel() assert_eq!(outcome.coverage, HealTraversalCoverage::Partial); assert_eq!((outcome.counters.processed, outcome.counters.healed), (1, 1)); } + +#[tokio::test] +async fn admin_bucket_discovery_rejects_incomplete_checkpoint_and_freezes_complete_empty_scope() { + let env = make_env().await; + env.storage.bucket_discovery_incomplete.store(true, Ordering::SeqCst); + let result = env + .healer + .prepare_admin_checkpoint(&env.checkpoint, &[], "pool_0_set_0") + .await; + assert!(matches!(result, Err(Error::BucketDiscoveryIncomplete))); + assert!(env.checkpoint.get_checkpoint().await.admin.is_none()); + let reloaded = CheckpointManager::load_from_disk(env.healer.disk.clone(), &env.task_id) + .await + .expect("reload checkpoint"); + assert!(reloaded.get_checkpoint().await.admin.is_none(), "incomplete scope must not be persisted"); + assert_eq!( + env.storage.bucket_discovery_scopes.lock().expect("scopes").as_slice(), + &[Some(crate::heal::storage_api::storage::DiskSetSelector::new(0, 0))] + ); + + env.storage.bucket_discovery_incomplete.store(false, Ordering::SeqCst); + assert!( + env.healer + .prepare_admin_checkpoint(&env.checkpoint, &[], "pool_0_set_0") + .await + .expect("complete empty scope") + .is_empty() + ); + env.storage.bucket_discovery_incomplete.store(true, Ordering::SeqCst); + objects(&env.storage, "created-after-admission", &["object"]); + let reloaded = CheckpointManager::load_from_disk(env.healer.disk.clone(), &env.task_id) + .await + .expect("reload frozen scope"); + assert!( + env.healer + .prepare_admin_checkpoint(&reloaded, &[], "pool_0_set_0") + .await + .expect("resume frozen scope") + .is_empty() + ); + assert_eq!( + env.storage.bucket_discovery_scopes.lock().expect("scopes").len(), + 2, + "resume must not rediscover or expand scope" + ); + assert!(env.storage.calls().is_empty()); +} diff --git a/crates/heal/src/heal/manager/auto_scan.rs b/crates/heal/src/heal/manager/auto_scan.rs index b09f337db..751b276f4 100644 --- a/crates/heal/src/heal/manager/auto_scan.rs +++ b/crates/heal/src/heal/manager/auto_scan.rs @@ -13,6 +13,18 @@ // limitations under the License. /// The automatic disk scanner: replacement discovery and unformatted-disk enqueue. use super::*; +use crate::heal::DiskStore; +use std::collections::BTreeMap; + +pub(super) fn extend_replacement_recovery_disks( + inventory: &mut BTreeMap, + disks: impl IntoIterator, +) { + for disk in disks { + // Preserve the local snapshot when later inventory repeats its endpoint. + inventory.entry(disk.endpoint().to_string()).or_insert(disk); + } +} impl HealManager { /// Start background task to auto scan local disks and enqueue erasure set heal requests @@ -87,7 +99,20 @@ impl HealManager { let local_disk_map = local_disk_map_read().await; local_disk_map.values().flatten().cloned().collect::>() }; - let local_endpoints = local_disks.iter().map(|disk| disk.endpoint()).collect::>(); + let mut local_endpoints = HashMap::with_capacity(local_disks.len()); + for disk in &local_disks { + let endpoint = disk.endpoint(); + local_endpoints.entry(endpoint.to_string()).or_insert(endpoint); + } + // Formatting removes a target from blank-disk discovery, but its + // unfinished intent may still live on a remote survivor. + let mut recovery_sets = local_disks + .iter() + .filter_map(|disk| { + let endpoint = disk.endpoint(); + crate::heal::utils::format_set_disk_id_from_i32(endpoint.pool_idx, endpoint.set_idx) + }) + .collect::>(); let blocked_sets = replacement_recovery_blocked_sets .lock() .expect("replacement recovery blocked set lock poisoned") @@ -97,6 +122,7 @@ impl HealManager { let mut retry_succeeded = HashSet::new(); let mut retry_failed = HashSet::new(); for set_disk_id in blocked_sets { + recovery_sets.remove(&set_disk_id); let disks = match storage.replacement_intent_disks(&set_disk_id).await { Ok(disks) => disks, Err(error) => { @@ -235,11 +261,12 @@ impl HealManager { } } - let mut recovery_disks = local_disks.clone(); - recovery_disks.extend(retry_recovery_disks); - for set_disk_id in endpoints.keys() { + let mut recovery_disks = BTreeMap::new(); + extend_replacement_recovery_disks(&mut recovery_disks, local_disks.iter().cloned()); + extend_replacement_recovery_disks(&mut recovery_disks, retry_recovery_disks); + for set_disk_id in &recovery_sets { match storage.replacement_intent_disks(set_disk_id).await { - Ok(disks) => recovery_disks.extend(disks), + Ok(disks) => extend_replacement_recovery_disks(&mut recovery_disks, disks), Err(error) => { conflicted_recovery_sets.insert(set_disk_id.clone()); warn!( @@ -254,10 +281,6 @@ impl HealManager { } } } - recovery_disks.sort_by_key(|disk| disk.endpoint().to_string()); - recovery_disks - .dedup_by(|left, right| left.endpoint().to_string() == right.endpoint().to_string()); - // Once formatting succeeds a replacement is no longer // discoverable as UnformattedDisk. Re-admit exactly one // incomplete durable generation per set within its @@ -265,7 +288,7 @@ impl HealManager { // verified terminal cleanup. Multiple generations are a // durable conflict: leave every marker/state intact and // require reconciliation rather than choosing one. - for disk in &recovery_disks { + for (anchor, disk) in &recovery_disks { let endpoint = disk.endpoint(); let disk_set_disk_id = crate::heal::utils::format_set_disk_id_from_i32(endpoint.pool_idx, endpoint.set_idx); @@ -351,7 +374,7 @@ impl HealManager { // A remote survivor may own the durable intent, but only the // node mounting every replacement target may advance or replay it. if state.replacement_targets.iter().any(|target| { - !local_endpoints.iter().any(|endpoint| endpoint.to_string() == *target) + !local_endpoints.contains_key(target) }) { deferred_recovery_sets.insert(state.set_disk_id.clone()); continue; @@ -361,19 +384,20 @@ impl HealManager { } let mut selected = durable_recovery_candidates.remove(&task_id); let previous_set = selected.as_ref().map(|candidate| candidate.state.set_disk_id.clone()); + let candidate_set = state.set_disk_id.clone(); let candidate = match ReplacementRecoveryCandidate::new( - state.clone(), - disk.endpoint().to_string(), - disk.endpoint().is_local, + state, + anchor.clone(), + endpoint.is_local, ) { Ok(candidate) => candidate, Err(error) => { conflicted_recovery_tasks.insert(task_id.clone()); - conflicted_recovery_sets.insert(state.set_disk_id.clone()); + conflicted_recovery_sets.insert(candidate_set.clone()); if let Some(previous_set) = &previous_set { conflicted_recovery_sets.insert(previous_set.clone()); } - block_recovery_set(&state.set_disk_id); + block_recovery_set(&candidate_set); if let Some(previous_set) = &previous_set { block_recovery_set(previous_set); } @@ -432,12 +456,7 @@ impl HealManager { { continue; } - let Some(anchor_disk) = local_disks - .iter() - .chain(recovery_disks.iter()) - .find(|disk| disk.endpoint().to_string() == candidate.anchor) - .cloned() - else { + let Some(anchor_disk) = recovery_disks.get(&candidate.anchor).cloned() else { deferred_recovery_sets.insert(candidate.state.set_disk_id.clone()); continue; }; @@ -530,12 +549,7 @@ impl HealManager { let targets = state .replacement_targets .iter() - .filter_map(|target| { - local_endpoints - .iter() - .find(|endpoint| endpoint.to_string() == *target) - .cloned() - }) + .filter_map(|target| local_endpoints.get(target).cloned()) .collect::>(); if targets.len() != state.replacement_targets.len() { deferred_recovery_sets.insert(state.set_disk_id.clone()); @@ -607,20 +621,9 @@ impl HealManager { } for (set_disk_id, (_, targets, _, _)) in &durable_recoveries { - let expected = targets.iter().map(ToString::to_string).collect::>(); - let observed = endpoints - .get(set_disk_id) - .map(|endpoints| endpoints.iter().map(ToString::to_string).collect::>()) - .unwrap_or_default(); - if !observed.is_subset(&expected) { - replacement_recovery_blocked_sets - .lock() - .expect("replacement recovery blocked set lock poisoned") - .insert(set_disk_id.clone()); - conflicted_recovery_sets.insert(set_disk_id.clone()); - continue; - } - endpoints.entry(set_disk_id.clone()).or_default().extend(targets.clone()); + // The durable owner must finish its exact target scope before + // newly discovered replacements can start another generation. + endpoints.insert(set_disk_id.clone(), targets.clone()); } for set_disk_id in &conflicted_recovery_sets { durable_recoveries.remove(set_disk_id); diff --git a/crates/heal/src/heal/manager/root_recovery.rs b/crates/heal/src/heal/manager/root_recovery.rs index 5eb51df8e..aac67fbcc 100644 --- a/crates/heal/src/heal/manager/root_recovery.rs +++ b/crates/heal/src/heal/manager/root_recovery.rs @@ -19,6 +19,7 @@ //! or deletion. use super::*; +use crate::heal::outcome::HealExecutionOutcome; use crate::heal::resume::CheckpointManager; use crate::heal::storage_api::owner::{EcstoreConditionalFileUpdate, EcstoreDiskAPI, EcstoreDiskBytes}; use crate::heal::{DiskStore, RUSTFS_META_BUCKET}; @@ -36,12 +37,15 @@ const ROOT_RECOVERY_PREFIX: &str = "root-heal-"; const ROOT_TERMINAL_PREFIX: &str = "terminal-root-heal-"; const LEGACY_ROOT_RECOVERY_SCHEMA: u32 = 1; const SCOPED_ROOT_RECOVERY_SCHEMA: u32 = 2; -const ROOT_RECOVERY_SCHEMA: u32 = 3; +const BUCKET_ROOT_RECOVERY_SCHEMA: u32 = 3; +const ROOT_RECOVERY_SCHEMA: u32 = 4; const ROOT_TERMINAL_SCHEMA: u32 = 1; const ROOT_QUARANTINE_INTENT_PREFIX: &str = "quarantined-root-heal-intent-"; const ROOT_QUARANTINE_TERMINAL_PREFIX: &str = "quarantined-root-heal-terminal-"; const ROOT_QUARANTINE_SCHEMA: u32 = 1; pub(super) const MAX_ROOT_RECOVERY_BYTES: usize = 64 * 1024; +// Cluster checkpoints additionally retain a frozen inventory and the bounded outcome window. +pub(super) const MAX_CLUSTER_RECOVERY_BYTES: usize = 16 * 1024 * 1024; const ROOT_TERMINAL_GC_SCAN_BUDGET: usize = 1024; const ROOT_TERMINAL_GC_DELETE_BUDGET: usize = 64; @@ -237,7 +241,83 @@ fn default_recovery_heal_type() -> RecoveryHealType { RecoveryHealType::Cluster } -#[derive(Debug, PartialEq, Eq, Serialize, Deserialize)] +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct AdminClusterScope { + #[serde(deserialize_with = "decode_cluster_buckets")] + pub(crate) buckets: std::collections::BTreeMap, + pub(crate) next_bucket_index: usize, + pub(crate) outcome: HealTaskOutcome, + pub(crate) progress: HealProgress, +} + +fn decode_cluster_buckets<'de, D: serde::Deserializer<'de>>( + deserializer: D, +) -> std::result::Result, D::Error> { + struct Buckets; + impl<'de> serde::de::Visitor<'de> for Buckets { + type Value = std::collections::BTreeMap; + fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result { + formatter.write_str("unique bucket generation bindings") + } + fn visit_map>(self, mut map: M) -> std::result::Result { + let mut buckets = std::collections::BTreeMap::new(); + while let Some((bucket, incarnation)) = map.next_entry::()? { + if buckets.insert(bucket, incarnation).is_some() { + return Err(serde::de::Error::custom("duplicate administrator cluster bucket")); + } + } + Ok(buckets) + } + } + deserializer.deserialize_map(Buckets) +} + +impl AdminClusterScope { + pub(crate) fn checkpoint_interval(&self) -> usize { + // Cap full-inventory rewrites per traversal, including many empty buckets. + self.buckets.len().div_ceil(32).max(1) + } + + fn validate(&self) -> Result<()> { + if self.next_bucket_index > self.buckets.len() + || self.outcome.execution != HealExecutionOutcome::Running + || self.outcome.coverage != crate::heal::outcome::HealTraversalCoverage::Partial + || self.outcome.counters.failed != 0 + { + return Err(Error::InvalidCheckpoint("Invalid administrator cluster checkpoint progress".to_string())); + } + for (bucket, incarnation) in &self.buckets { + if bucket.contains('/') + || bucket.trim() != bucket + || crate::heal::storage_api::owner::ecstore_is_reserved_or_invalid_bucket(bucket, false) + { + return Err(Error::InvalidCheckpoint("Invalid administrator cluster bucket name".to_string())); + } + if incarnation.is_nil() { + return Err(Error::InvalidCheckpoint( + "Administrator cluster checkpoint lacks a bucket incarnation".to_string(), + )); + } + } + let completed_through = self + .next_bucket_index + .checked_sub(1) + .and_then(|index| self.buckets.keys().nth(index)); + if self.outcome.objects.iter().any(|item| { + completed_through.is_none_or(|last| item.identity.bucket > *last) + || self.buckets.get(&item.identity.bucket).copied() != item.identity.bucket_incarnation_id + }) || (self.next_bucket_index == 0 && self.outcome.counters.processed != 0) + { + return Err(Error::InvalidCheckpoint( + "Administrator cluster checkpoint outcome is outside its completed scope".to_string(), + )); + } + Ok(()) + } +} + +#[derive(Debug, PartialEq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] struct RootHealIntent { schema: u32, @@ -246,6 +326,8 @@ struct RootHealIntent { heal_type: RecoveryHealType, #[serde(default, skip_serializing_if = "Option::is_none")] bucket_incarnation_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + cluster_scope: Option, #[serde(deserialize_with = "decode_options")] options: HealOptions, priority: HealPriority, @@ -292,6 +374,26 @@ struct RootHealSource { source_oversized: bool, } +impl RootHealQuarantine { + fn matches_source(&self, source: &RootHealSource) -> bool { + if self.source_prefix_len == source.bytes.len() + && self.source_prefix_sha256 == source.source_prefix_sha256 + && self.source_oversized == source.source_oversized + { + return true; + } + // Before schema 4, oversized intents retained only the 64 KiB prefix. + // Keep that quarantine fence across upgrades without reinterpreting its source. + self.record_kind == RootHealRecordKind::Intent + && self.source_oversized + && self.source_prefix_len == MAX_ROOT_RECOVERY_BYTES + 1 + && source + .bytes + .get(..self.source_prefix_len) + .is_some_and(|prefix| Sha256::digest(prefix).as_slice() == self.source_prefix_sha256.as_slice()) + } +} + #[derive(Debug, Default)] pub(super) struct RootHealInventory { pub(super) requests: Vec, @@ -362,12 +464,38 @@ impl RootHealTerminal { } impl RootHealIntent { + fn encode(&self) -> Result { + let limit = match &self.cluster_scope { + // Reserve headroom for the bounded outcome window and progress so + // completing work cannot overflow an inventory admitted at the limit. + Some(scope) if scope.next_bucket_index == 0 => MAX_CLUSTER_RECOVERY_BYTES - 1024 * 1024, + Some(_) => MAX_CLUSTER_RECOVERY_BYTES, + None => MAX_ROOT_RECOVERY_BYTES, + }; + encode_root_record(self, limit) + } + + fn validate_task(&self, task: &HealTask) -> Result<()> { + let mut options = self.options.clone(); + options.timeout = task.options.timeout; + if HealType::from(self.heal_type.clone()) != task.heal_type + || self.bucket_incarnation_id != task.bucket_incarnation_id + || self.created_at != task.created_at + || self.priority != task.priority + || options != task.options + { + return Err(Error::InvalidCheckpoint(format!("Root heal recovery owner changed for {}", task.id))); + } + Ok(()) + } + fn from_request(request: &HealRequest) -> Result { Ok(Self { schema: ROOT_RECOVERY_SCHEMA, task_id: request.id.clone(), heal_type: RecoveryHealType::try_from(&request.heal_type)?, bucket_incarnation_id: request.bucket_incarnation_id, + cluster_scope: None, options: request.options.clone(), priority: request.priority, retry_attempts: request.retry_attempts, @@ -386,6 +514,32 @@ impl RootHealIntent { } } +fn encode_root_record(value: &impl Serialize, limit: usize) -> Result { + struct BoundedWriter { + bytes: Vec, + limit: usize, + } + impl std::io::Write for BoundedWriter { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + if bytes.len() > self.limit.saturating_sub(self.bytes.len()) { + return Err(std::io::Error::other("root heal record exceeds its size limit")); + } + self.bytes.extend_from_slice(bytes); + Ok(bytes.len()) + } + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + let mut writer = BoundedWriter { + bytes: Vec::new(), + limit, + }; + serde_json::to_writer(&mut writer, value) + .map_err(|error| Error::Serialization(format!("Serialize root heal record: {error}")))?; + Ok(writer.bytes.into()) +} + #[derive(Default)] pub(crate) struct RootHealRecovery { mutation: Mutex<()>, @@ -490,19 +644,27 @@ async fn read_bounded(disk: &DiskStore, path: &str) -> Result usize { + if path.starts_with(ROOT_RECOVERY_PREFIX) { + MAX_CLUSTER_RECOVERY_BYTES + } else { + MAX_ROOT_RECOVERY_BYTES + } +} + async fn read_source(disk: &DiskStore, path: &str) -> Result> { let Some(bytes) = read_bounded(disk, path).await? else { return Ok(None); }; Ok(Some(RootHealSource { source_prefix_sha256: Sha256::digest(&bytes).to_vec(), - source_oversized: bytes.len() > MAX_ROOT_RECOVERY_BYTES, + source_oversized: bytes.len() > root_record_read_limit(path), bytes, })) } @@ -544,7 +706,7 @@ fn encode_quarantine( fn decode_intent(task_id: &str, bytes: &[u8]) -> Result { let _ = intent_path(task_id)?; - if bytes.len() > MAX_ROOT_RECOVERY_BYTES { + if bytes.len() > MAX_CLUSTER_RECOVERY_BYTES { return Err(Error::Other(format!("Root heal recovery record exceeds its size limit {task_id}"))); } let mut intent: RootHealIntent = serde_json::from_slice(bytes) @@ -556,9 +718,25 @@ fn decode_intent(task_id: &str, bytes: &[u8]) -> Result { LEGACY_ROOT_RECOVERY_SCHEMA if intent.heal_type == RecoveryHealType::Cluster && intent.bucket_incarnation_id.is_none() => {} SCOPED_ROOT_RECOVERY_SCHEMA if intent.bucket_incarnation_id.is_none() => {} + BUCKET_ROOT_RECOVERY_SCHEMA if intent.cluster_scope.is_none() => {} ROOT_RECOVERY_SCHEMA => {} _ => return Err(Error::Other(format!("Unsupported or mismatched root heal recovery record {task_id}"))), } + if intent.heal_type == RecoveryHealType::Cluster && intent.schema != ROOT_RECOVERY_SCHEMA { + return Err(Error::InvalidCheckpoint( + "Legacy cluster heal has no frozen bucket generations; start a fresh task".to_string(), + )); + } + match &intent.cluster_scope { + Some(scope) if intent.schema == ROOT_RECOVERY_SCHEMA && intent.heal_type == RecoveryHealType::Cluster => { + scope.validate()?; + } + Some(_) => return Err(Error::InvalidCheckpoint("Unexpected administrator cluster scope".to_string())), + None if bytes.len() > MAX_ROOT_RECOVERY_BYTES => { + return Err(Error::InvalidCheckpoint("Root heal recovery record exceeds its size limit".to_string())); + } + None => {} + } if !matches!(intent.heal_type, RecoveryHealType::Bucket { .. }) && intent.bucket_incarnation_id.is_some() { return Err(Error::Other(format!( "Unexpected bucket incarnation in root heal recovery record {task_id}" @@ -723,10 +901,7 @@ impl RootHealRecovery { } }; let marker = decode_quarantine(task_id, record_kind, &marker_bytes)?; - if marker.source_prefix_len != source.bytes.len() - || marker.source_prefix_sha256 != source.source_prefix_sha256 - || marker.source_oversized != source.source_oversized - { + if !marker.matches_source(source) { return Err(Error::other(format!("Root heal quarantine source changed {task_id}"))); } Ok(marker) @@ -808,9 +983,7 @@ impl RootHealRecovery { if let Some((owner, source)) = terminal_raw.into_iter().next() { if let Some(marker) = markers.get(&(RootHealRecordKind::Terminal, task_id.clone())) { if EcstoreDiskAPI::endpoint(marker.0.as_ref()) != EcstoreDiskAPI::endpoint(owner.as_ref()) - || marker.1.source_prefix_len != source.bytes.len() - || marker.1.source_prefix_sha256 != source.source_prefix_sha256 - || marker.1.source_oversized != source.source_oversized + || !marker.1.matches_source(&source) { return Err(Error::other(format!("Root heal quarantine source changed {task_id}"))); } @@ -849,9 +1022,7 @@ impl RootHealRecovery { }; if let Some(marker) = markers.get(&(RootHealRecordKind::Intent, task_id.clone())) { if EcstoreDiskAPI::endpoint(marker.0.as_ref()) != EcstoreDiskAPI::endpoint(owner.as_ref()) - || marker.1.source_prefix_len != source.bytes.len() - || marker.1.source_prefix_sha256 != source.source_prefix_sha256 - || marker.1.source_oversized != source.source_oversized + || !marker.1.matches_source(&source) { return Err(Error::other(format!("Root heal quarantine source changed {task_id}"))); } @@ -924,18 +1095,29 @@ impl RootHealRecovery { if request.options.no_lock { return Err(Error::Other("Administrator root heal cannot skip namespace locking".to_string())); } - let intent = RootHealIntent::from_request(request)?; + let mut intent = RootHealIntent::from_request(request)?; intent.heal_type.validate()?; - let bytes = - serde_json::to_vec(&intent).map_err(|error| Error::Other(format!("Serialize root heal recovery record: {error}")))?; let path = intent_path(&request.id)?; if let Some((disk, expected)) = existing { + let previous = decode_intent(&request.id, &expected)?; + let mut previous_options = previous.options.clone(); + previous_options.timeout = intent.options.timeout; + if previous.heal_type != intent.heal_type + || previous.bucket_incarnation_id != intent.bucket_incarnation_id + || previous.created_at != intent.created_at + || previous.priority != intent.priority + || previous_options != intent.options + { + return Err(Error::InvalidCheckpoint("Root heal recovery owner changed during handoff".to_string())); + } + intent.cluster_scope = previous.cluster_scope; + let bytes = intent.encode()?; return match EcstoreDiskAPI::compare_and_update_file( disk.as_ref(), RUSTFS_META_BUCKET, &path, Some(expected), - Some(bytes.into()), + Some(bytes), ) .await? { @@ -944,19 +1126,15 @@ impl RootHealRecovery { }; } + let bytes = intent.encode()?; + if disks.is_empty() { return Err(Error::Other("No local disk available for root heal shutdown recovery".to_string())); } let mut last_not_committed = None; for disk in &disks { - match EcstoreDiskAPI::compare_and_update_file( - disk.as_ref(), - RUSTFS_META_BUCKET, - &path, - None, - Some(bytes.clone().into()), - ) - .await + match EcstoreDiskAPI::compare_and_update_file(disk.as_ref(), RUSTFS_META_BUCKET, &path, None, Some(bytes.clone())) + .await { Ok(EcstoreConditionalFileUpdate::Updated) => return Ok(()), Ok(_) => return Err(Error::Other(format!("Root heal recovery record changed for {}", request.id))), @@ -970,6 +1148,63 @@ impl RootHealRecovery { } } + pub(crate) async fn cluster_scope(&self, task: &HealTask) -> Result> { + #[cfg(any(test, feature = "test-util"))] + if self.disabled_for_tests { + return Ok(None); + } + let _guard = self.mutation.lock().await; + let (_, bytes) = Self::find(&self.disks().await?, &task.id) + .await? + .ok_or_else(|| Error::InvalidCheckpoint("Administrator cluster intent is missing".to_string()))?; + let intent = decode_intent(&task.id, &bytes)?; + intent.validate_task(task)?; + Ok(intent.cluster_scope) + } + + pub(crate) async fn checkpoint_cluster_scope(&self, task: &HealTask, scope: &AdminClusterScope) -> Result<()> { + scope.validate()?; + #[cfg(any(test, feature = "test-util"))] + if self.disabled_for_tests { + return Ok(()); + } + let _guard = self.mutation.lock().await; + let (disk, expected) = Self::find(&self.disks().await?, &task.id) + .await? + .ok_or_else(|| Error::InvalidCheckpoint("Administrator cluster intent is missing".to_string()))?; + let mut intent = decode_intent(&task.id, &expected)?; + intent.validate_task(task)?; + if intent.heal_type != RecoveryHealType::Cluster { + return Err(Error::InvalidCheckpoint("Cluster scope requires a cluster task".to_string())); + } + if let Some(previous) = &intent.cluster_scope { + if previous.buckets != scope.buckets + || scope.next_bucket_index < previous.next_bucket_index + || scope.next_bucket_index > previous.next_bucket_index.saturating_add(scope.checkpoint_interval()) + { + return Err(Error::InvalidCheckpoint( + "Administrator cluster scope changed during recovery".to_string(), + )); + } + } else if scope.next_bucket_index != 0 { + return Err(Error::InvalidCheckpoint("Administrator cluster scope was not initialized".to_string())); + } + intent.cluster_scope = Some(scope.clone()); + let bytes = intent.encode()?; + match EcstoreDiskAPI::compare_and_update_file( + disk.as_ref(), + RUSTFS_META_BUCKET, + &intent_path(&task.id)?, + Some(expected), + Some(bytes), + ) + .await? + { + EcstoreConditionalFileUpdate::Updated => Ok(()), + _ => Err(Error::InvalidCheckpoint("Administrator cluster checkpoint owner changed".to_string())), + } + } + pub(super) async fn remove(&self, task_id: &str, heal_type: &HealType, source: HealRequestSource) -> Result { if !is_admin_heal_recovery(heal_type, source) { return Ok(false); @@ -1015,16 +1250,7 @@ impl RootHealRecovery { return Ok(()); }; let mut intent = decode_intent(&task.id, &expected)?; - if HealType::from(intent.heal_type.clone()) != task.heal_type - || intent.bucket_incarnation_id != task.bucket_incarnation_id - { - return Err(Error::Other(format!("Root heal recovery owner changed for {}", task.id))); - } - let mut expected_options = intent.options.clone(); - expected_options.timeout = task.options.timeout; - if intent.created_at != task.created_at || intent.priority != task.priority || expected_options != task.options { - return Err(Error::Other(format!("Root heal recovery owner changed for {}", task.id))); - } + intent.validate_task(task)?; // A terminal timeout leaves no runtime owner for stop() to snapshot. // Checkpoint its consumed budget before publishing terminal status; // never refund time if an earlier checkpoint is already stricter. @@ -1033,14 +1259,13 @@ impl RootHealRecovery { (previous, remaining) => previous.or(remaining), }; intent.retry_attempts = intent.retry_attempts.max(task.retry_attempts); - let bytes = serde_json::to_vec(&intent) - .map_err(|error| Error::Other(format!("Serialize root heal recovery checkpoint: {error}")))?; + let bytes = intent.encode()?; match EcstoreDiskAPI::compare_and_update_file( disk.as_ref(), RUSTFS_META_BUCKET, &intent_path(&task.id)?, Some(expected), - Some(bytes.into()), + Some(bytes), ) .await? { diff --git a/crates/heal/src/heal/manager/root_recovery/report.rs b/crates/heal/src/heal/manager/root_recovery/report.rs index 923daf67b..fc646882e 100644 --- a/crates/heal/src/heal/manager/root_recovery/report.rs +++ b/crates/heal/src/heal/manager/root_recovery/report.rs @@ -14,7 +14,6 @@ use super::*; use crate::heal::outcome::{HealAbortReason, HealExecutionOutcome}; -use std::io::Write; use tokio::io::AsyncReadExt; // This prefix must not overlap either namespace understood by schema-1 readers. @@ -68,24 +67,7 @@ impl RootHealReport { } pub(super) fn encode(&self) -> Result { - struct BoundedWriter(Vec); - impl Write for BoundedWriter { - fn write(&mut self, bytes: &[u8]) -> std::io::Result { - if bytes.len() > MAX_ROOT_REPORT_BYTES.saturating_sub(self.0.len()) { - return Err(std::io::Error::other("root heal report exceeds its size limit")); - } - self.0.extend_from_slice(bytes); - Ok(bytes.len()) - } - - fn flush(&mut self) -> std::io::Result<()> { - Ok(()) - } - } - let mut writer = BoundedWriter(Vec::new()); - serde_json::to_writer(&mut writer, self) - .map_err(|error| Error::Serialization(format!("Serialize root heal report: {error}")))?; - Ok(writer.0.into()) + encode_root_record(self, MAX_ROOT_REPORT_BYTES) } } diff --git a/crates/heal/src/heal/manager/tests.rs b/crates/heal/src/heal/manager/tests.rs index 997c85e64..70121bac8 100644 --- a/crates/heal/src/heal/manager/tests.rs +++ b/crates/heal/src/heal/manager/tests.rs @@ -29,6 +29,7 @@ use tempfile::TempDir; pub(super) mod admin_overlap; mod root_recovery; use uuid::Uuid; +mod auto_scan; mod running_mainline; use super::super::{DiskOption, DiskStore, Endpoint, new_disk, storage_api::status::BucketInfo}; @@ -601,6 +602,13 @@ impl HealStorageAPI for MockStorage { Ok(None) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> Result> { if let Some(hook) = manager_recovery_test_hook() { *hook.listed.lock().expect("manager recovery listed lock should not poison") = true; @@ -5663,7 +5671,7 @@ async fn configured_task_timeout_applies_only_when_request_timeout_is_absent() { process_manager_queue_once(&manager).await; let explicit_status = tokio::time::timeout(Duration::from_secs(1), async { loop { - if let Ok(status @ HealTaskStatus::Failed { .. }) = manager.get_task_status(&explicit_id).await { + if let Ok(status @ HealTaskStatus::Completed) = manager.get_task_status(&explicit_id).await { break status; } tokio::task::yield_now().await; @@ -5671,7 +5679,8 @@ async fn configured_task_timeout_applies_only_when_request_timeout_is_absent() { }) .await .expect("explicit timeout request should finish without using the zero default"); - assert!(matches!(explicit_status, HealTaskStatus::Failed { .. })); + // Generation-bound admin tasks reach the successful mock heal without a namespace preflight. + assert_eq!(explicit_status, HealTaskStatus::Completed); } #[tokio::test] @@ -5902,3 +5911,25 @@ fn test_heal_config_respects_feature_flags() { }, ); } + +#[test] +fn admin_bucket_discovery_retries_keep_the_existing_budget() { + let storage: Arc = Arc::new(MockStorage); + let mut request = HealRequest::new(HealType::Cluster, HealOptions::default(), HealPriority::Normal); + request.source = HealRequestSource::Admin; + let id = request.id.clone(); + let error = Err(Error::BucketDiscoveryIncomplete); + for attempt in 1..=MAX_RECOVERABLE_HEAL_RETRIES { + let task = HealTask::from_request(request, storage.clone()); + let (retry, delay, _) = retry_request_for_result(&task, &error).expect("discovery can recover within budget"); + assert_eq!(retry.id, id); + assert_eq!(retry.retry_attempts, attempt); + assert!(delay > Duration::ZERO); + request = retry; + } + let task = HealTask::from_request(request, storage); + assert!( + retry_request_for_result(&task, &error).is_none(), + "persistent incompleteness must terminate" + ); +} diff --git a/crates/heal/src/heal/manager/tests/auto_scan.rs b/crates/heal/src/heal/manager/tests/auto_scan.rs new file mode 100644 index 000000000..091c8d498 --- /dev/null +++ b/crates/heal/src/heal/manager/tests/auto_scan.rs @@ -0,0 +1,377 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use crate::heal::storage::HealListItem; +use crate::heal::storage_api::{ + EcstoreEndpointServerPools as EndpointServerPools, EcstoreEndpoints as Endpoints, EcstorePoolEndpoints as PoolEndpoints, + ecstore_init_local_disks as init_local_disks, +}; +use std::{ + future::Future, + sync::atomic::{AtomicUsize, Ordering}, +}; + +const SET: &str = "pool_0_set_0"; +const SCAN_INTERVAL: Duration = Duration::from_secs(10); + +struct ScannerStorage { + survivor: DiskStore, + identity: ReplacementTargetIdentity, + inventory_calls: AtomicUsize, +} + +#[async_trait::async_trait] +impl HealStorageAPI for ScannerStorage { + async fn replacement_intent_disks(&self, set: &str) -> Result> { + assert_eq!(set, SET, "discovery must use the target's erasure set"); + self.inventory_calls.fetch_add(1, Ordering::SeqCst); + // This disk is deliberately outside the node's local registry. The + // mock supplies the same inventory boundary as a remote survivor, + // while real disk I/O exercises durable intent discovery and loading. + Ok(vec![self.survivor.clone()]) + } + + async fn replacement_target_identities(&self, targets: &[String]) -> Result> { + assert_eq!(targets, std::slice::from_ref(&self.identity.endpoint)); + Ok(vec![self.identity.clone()]) + } + + async fn get_object_meta(&self, _: &str, _: &str) -> Result> { + Err(Error::other("scanner admission must not inspect objects")) + } + + async fn ec_decode_rebuild(&self, _: &str, _: &str) -> Result> { + Err(Error::other("scanner admission must not rebuild objects")) + } + + async fn get_bucket_info(&self, _: &str) -> Result> { + Err(Error::other("scanner admission must not inspect buckets")) + } + + async fn list_buckets(&self) -> Result> { + Err(Error::other("same-incarnation discovery must reuse durable buckets")) + } + + async fn list_buckets_for_admin_heal( + &self, + _: Option, + ) -> Result> { + Err(Error::other("scanner admission must not start admin discovery")) + } + + async fn object_exists(&self, _: &str, _: &str) -> Result { + Err(Error::other("scanner admission must not inspect objects")) + } + + async fn heal_object(&self, _: &str, _: &str, _: Option<&str>, _: &HealOpts) -> Result<(HealResultItem, Option)> { + Err(Error::other("scanner admission must not execute object healing")) + } + + async fn heal_bucket(&self, _: &str, _: &HealOpts) -> Result { + Err(Error::other("scanner admission must not execute bucket healing")) + } + + async fn heal_format(&self, _: bool) -> Result<(HealResultItem, Option)> { + Err(Error::other("scanner admission must not execute format healing")) + } + + async fn list_objects_for_heal_page( + &self, + _: &str, + _: &str, + _: Option<&str>, + _: bool, + ) -> Result<(Vec, Option, bool)> { + Err(Error::other("scanner admission must not traverse objects")) + } + + async fn get_disk_for_resume(&self, _: &str) -> Result { + Err(Error::other("scanner admission must retain the durable survivor")) + } +} + +struct ScannerFixture { + _temp: TempDir, + manager: HealManager, + storage: Arc, + owner: ResumeManager, + task_id: String, + local_targets: Vec, +} + +impl Drop for ScannerFixture { + fn drop(&mut self) { + self.manager.cancel_token.cancel(); + } +} + +impl ScannerFixture { + async fn new(unformatted_siblings: bool) -> Self { + let temp = TempDir::new().expect("scanner replacement fixture directory"); + let disk_ids = (0..3).map(|_| Uuid::new_v4()).collect::>(); + let deployment_id = Uuid::new_v4(); + let mut endpoints = Vec::new(); + for slot in 0..3 { + let path = temp.path().join(format!("target-{slot}")); + let metadata_path = path.join(crate::heal::RUSTFS_META_BUCKET); + std::fs::create_dir_all(&metadata_path).expect("scanner local target metadata directory"); + if !unformatted_siblings || slot == 0 { + let format = serde_json::json!({ + "version": "1", + "format": "xl", + "id": deployment_id, + "xl": { + "version": "3", + "this": disk_ids[slot], + "sets": [disk_ids], + "distributionAlgo": "SIPMOD+PARITY" + } + }); + std::fs::write( + metadata_path.join("format.json"), + serde_json::to_vec(&format).expect("scanner format fixture serialization"), + ) + .expect("scanner formatted target fixture"); + } + let mut endpoint = + Endpoint::try_from(path.to_str().expect("scanner target path is UTF-8")).expect("scanner local endpoint"); + endpoint.set_pool_index(0); + endpoint.set_set_index(0); + endpoint.set_disk_index(slot); + endpoints.push(endpoint); + } + let local_targets = endpoints.iter().map(ToString::to_string).collect::>(); + init_local_disks(EndpointServerPools::from(vec![PoolEndpoints { + legacy: false, + set_count: 1, + drives_per_set: 3, + endpoints: Endpoints::from(endpoints), + cmd_line: "replacement-scanner-recovery-test".into(), + platform: "test".into(), + }])) + .await + .expect("scanner local registry initialization"); + + let survivor = make_manager_resume_disk(&temp, "survivor").await; + let local_disks = crate::heal::local_disk_map_read().await; + assert_eq!(local_disks.len(), 3); + assert!( + !local_disks.contains_key(&survivor.endpoint().to_string()), + "the durable anchor must be discoverable only through the set inventory" + ); + for disk in local_disks.values().flatten() { + let is_owner = disk.endpoint().to_string() == local_targets[0]; + match disk.get_disk_id().await { + Ok(Some(id)) => assert!(!id.is_nil()), + Err(DiskError::UnformattedDisk) => assert!(unformatted_siblings && !is_owner), + result => panic!("unexpected fixture disk identity: {result:?}"), + } + assert_eq!(disk.runtime_state().as_str(), "online"); + } + drop(local_disks); + + let identity = ReplacementTargetIdentity { + endpoint: local_targets[0].clone(), + canonical_path: local_targets[0].clone(), + physical_device_ids: vec!["replacement-target-0".into()], + filesystem_identity: "replacement-target-0-incarnation".into(), + }; + let task_id = ResumeUtils::generate_task_id(); + let owner = ResumeManager::new_replacement_intent( + survivor.clone(), + task_id.clone(), + SET.into(), + vec!["seeded-bucket".into()], + vec![local_targets[0].clone()], + vec![identity.clone()], + ) + .await + .expect("durable remote-survivor replacement intent"); + owner + .mark_replacement_rebuilding(vec![identity.clone()]) + .await + .expect("durable replacement rebuild phase"); + owner + .record_replacement_failure(&Error::Storage(EcstoreError::FileNotFound), 1) + .await + .expect("durable in-budget FileNotFound failure"); + let storage = Arc::new(ScannerStorage { + survivor, + identity, + inventory_calls: AtomicUsize::new(0), + }); + let manager = HealManager::new_without_root_recovery_for_test( + storage.clone(), + Some(HealConfig { + heal_interval: SCAN_INTERVAL, + ..HealConfig::default() + }), + ); + Self { + _temp: temp, + manager, + storage, + owner, + task_id, + local_targets, + } + } + + async fn queued_request(&self) -> HealRequest { + // Keep the paused clock from advancing while filesystem workers + // finish. A wall-clock deadline also bounds a broken admission path. + let deadline = std::time::Instant::now() + Duration::from_secs(5); + loop { + let queue = self.manager.heal_queue.lock().await; + if let Some(item) = queue.heap.peek() { + assert_eq!(queue.len(), 1, "one set must have exactly one admitted owner"); + return item.request.clone(); + } + drop(queue); + assert!(std::time::Instant::now() < deadline, "scanner tick did not admit replacement recovery"); + tokio::task::yield_now().await; + } + } + + async fn assert_original_owner_is_queued(&self) { + let request = self.queued_request().await; + assert_eq!(request.id, self.task_id, "scanner must replay the durable UUID"); + assert_eq!(request.heal_endpoints, vec![self.local_targets[0].clone()]); + assert_eq!(request.source, HealRequestSource::AutoHeal); + assert!(matches!( + request.heal_type, + HealType::ErasureSet { set_disk_id, buckets } + if set_disk_id == SET && buckets == ["seeded-bucket"] + )); + assert_eq!( + self.manager + .replacement_recovery_anchors + .lock() + .expect("scanner anchor registry") + .get(&self.task_id), + Some(&self.storage.survivor.endpoint().to_string()) + ); + assert!(!self.manager.replacement_recovery_set_is_blocked(SET)); + assert_eq!( + self.storage.inventory_calls.load(Ordering::SeqCst), + 1, + "three local slots in one set must share one inventory lookup per tick" + ); + let state = self.owner.get_state().await; + assert_eq!(state.retry_count, 1, "discovery must not reset the durable retry budget"); + assert_eq!(state.replacement_targets, request.heal_endpoints); + } +} + +async fn finish_without_advancing_clock(future: impl Future) -> T { + tokio::pin!(future); + loop { + tokio::select! { + biased; + result = &mut future => return result, + _ = tokio::task::yield_now() => {} + } + } +} + +#[tokio::test(start_paused = true)] +#[serial_test::serial(replacement_auto_scan)] +async fn scanner_tick_recovers_remote_survivor_after_all_local_targets_are_formatted() { + let fixture = ScannerFixture::new(false).await; + fixture + .manager + .start_auto_disk_scanner() + .await + .expect("start only the periodic replacement scanner"); + fixture.assert_original_owner_is_queued().await; +} + +#[tokio::test(start_paused = true)] +#[serial_test::serial(replacement_auto_scan)] +async fn scanner_tick_finishes_existing_owner_before_admitting_new_unformatted_targets() { + temp_env::async_with_vars([(rustfs_config::ENV_UNSAFE_BYPASS_DISK_CHECK, Some("true"))], async { + let fixture = ScannerFixture::new(true).await; + fixture + .manager + .start_auto_disk_scanner() + .await + .expect("start only the periodic replacement scanner"); + fixture.assert_original_owner_is_queued().await; + + // The real task's verified completion publishes the proof and + // removes its live intent before sibling admission is allowed. + let owner_removed = finish_without_advancing_clock(async { + fixture + .owner + .mark_replacement_completed_and_verified() + .await + .expect("durable owner completion proof"); + fixture + .owner + .mark_replacement_cleanup_pending() + .await + .expect("durable owner cleanup phase"); + fixture + .owner + .cleanup() + .await + .expect("completed owner releases its durable intent"); + !ResumeManager::has_replacement_intent(&fixture.storage.survivor, &fixture.task_id).await + }) + .await; + assert!(owner_removed, "completed ownership must be removed before the next generation"); + let finished = fixture + .manager + .heal_queue + .lock() + .await + .pop_next() + .expect("remove the completed queued owner"); + assert_eq!(finished.id, fixture.task_id); + tokio::time::advance(SCAN_INTERVAL).await; + + let next = fixture.queued_request().await; + assert_ne!(next.id, fixture.task_id, "the follow-up replacement gets a fresh generation"); + assert_eq!(next.heal_endpoints, fixture.local_targets[1..]); + assert_eq!(next.source, HealRequestSource::AutoHeal); + assert!(!fixture.manager.replacement_recovery_set_is_blocked(SET)); + assert_eq!(fixture.storage.inventory_calls.load(Ordering::SeqCst), 2); + }) + .await; +} + +#[tokio::test] +async fn recovery_disk_index_keeps_first_snapshot_and_sorted_unique_endpoints() { + let temp = TempDir::new().expect("recovery disk index fixture"); + let local = make_manager_resume_disk(&temp, "a-local").await; + let duplicate = make_manager_resume_disk(&temp, "a-local").await; + let survivor = make_manager_resume_disk(&temp, "z-survivor").await; + assert!(!Arc::ptr_eq(&local, &duplicate), "duplicate must be a different disk snapshot"); + let local_endpoint = local.endpoint().to_string(); + let survivor_endpoint = survivor.endpoint().to_string(); + let mut inventory = std::collections::BTreeMap::new(); + super::super::auto_scan::extend_replacement_recovery_disks(&mut inventory, [local.clone()]); + super::super::auto_scan::extend_replacement_recovery_disks(&mut inventory, [survivor.clone(), duplicate, survivor]); + assert_eq!(inventory.len(), 2, "each endpoint is discovered once"); + assert!( + Arc::ptr_eq(inventory.get(&local_endpoint).expect("local endpoint remains indexed"), &local), + "later set inventories must not replace the local snapshot" + ); + assert_eq!( + inventory.keys().collect::>(), + vec![&local_endpoint, &survivor_endpoint], + "discovery order remains lexical regardless of insertion order" + ); +} diff --git a/crates/heal/src/heal/manager/tests/root_recovery.rs b/crates/heal/src/heal/manager/tests/root_recovery.rs index 732c83383..669893b1a 100644 --- a/crates/heal/src/heal/manager/tests/root_recovery.rs +++ b/crates/heal/src/heal/manager/tests/root_recovery.rs @@ -1844,7 +1844,7 @@ async fn root_recovery_queued_non_root_admin_owner_is_not_priority_displaced() { } #[tokio::test] -async fn root_recovery_legacy_schema_replays_as_cluster() { +async fn root_recovery_legacy_cluster_requires_a_fresh_generation_bound_task() { #[derive(serde::Serialize)] struct LegacyRootHealIntent<'a> { schema: u32, @@ -1873,10 +1873,113 @@ async fn root_recovery_legacy_schema_replays_as_cluster() { let restarted = recovery_manager(vec![disk]); restarted.replay_root_heals().await.expect("replay legacy root record"); - let queued = restarted.heal_queue.lock().await.requests().cloned().collect::>(); - assert_eq!(queued.len(), 1); - assert_eq!(queued[0].id, request.id); - assert_eq!(queued[0].heal_type, HealType::Cluster); + assert!( + restarted.heal_queue.lock().await.requests().next().is_none(), + "unfenced legacy work must not bind to a later bucket generation" + ); + assert!( + restarted + .root_recovery + .inventory() + .await + .expect("quarantined legacy task") + .quarantined_task_ids + .contains(&request.id) + ); +} + +#[tokio::test] +async fn root_recovery_cluster_scope_preserves_generations_and_completed_prefix_on_handoff() { + use crate::heal::manager::root_recovery::AdminClusterScope; + use std::collections::BTreeMap; + let (_temp, disk) = recovery_disk().await; + let manager = recovery_manager(vec![disk.clone()]); + let request = root_request(); + manager.root_recovery.persist(&request).await.expect("admit cluster task"); + let task = HealTask::from_request(request.clone(), Arc::new(MockStorage)); + let mut outcome = HealTaskOutcome::default(); + outcome.start(); + let mut scope = AdminClusterScope { + buckets: BTreeMap::from([ + ("alpha-bucket".to_owned(), Uuid::from_u128(42)), + ("bravo-bucket".to_owned(), Uuid::from_u128(43)), + ]), + next_bucket_index: 0, + outcome, + progress: HealProgress::new(), + }; + manager + .root_recovery + .checkpoint_cluster_scope(&task, &scope) + .await + .expect("freeze bucket generations before mutation"); + scope.next_bucket_index = 1; + manager + .root_recovery + .checkpoint_cluster_scope(&task, &scope) + .await + .expect("checkpoint completed empty bucket"); + let retry = task.retry_request(); + manager + .root_recovery + .persist(&retry) + .await + .expect("shutdown/retry handoff retains scope"); + let mut changed_options = retry.clone(); + changed_options.options.recursive = !changed_options.options.recursive; + assert!( + manager.root_recovery.persist(&changed_options).await.is_err(), + "handoff cannot reinterpret completed work" + ); + drop(manager); + let restarted = recovery_manager(vec![disk]); + let replay = HealTask::from_request(retry, Arc::new(MockStorage)); + assert_eq!( + restarted + .root_recovery + .cluster_scope(&replay) + .await + .expect("read frozen scope"), + Some(scope.clone()) + ); + let mut changed = scope.clone(); + changed.buckets.insert("alpha-bucket".to_owned(), Uuid::from_u128(44)); + assert!( + restarted + .root_recovery + .checkpoint_cluster_scope(&replay, &changed) + .await + .is_err(), + "never rebind a saved generation" + ); + changed = scope.clone(); + changed.next_bucket_index = 0; + assert!( + restarted + .root_recovery + .checkpoint_cluster_scope(&replay, &changed) + .await + .is_err(), + "never rewind a committed prefix" + ); + changed = scope.clone(); + changed.buckets.insert("../invalid".to_owned(), Uuid::from_u128(45)); + assert!( + restarted + .root_recovery + .checkpoint_cluster_scope(&replay, &changed) + .await + .is_err(), + "reject decoded path aliases" + ); + assert_eq!( + restarted + .root_recovery + .cluster_scope(&replay) + .await + .expect("failed updates preserve scope"), + Some(scope) + ); } #[tokio::test] @@ -1899,6 +2002,155 @@ async fn root_recovery_rejected_admin_start_does_not_persist() { assert_eq!(restarted.get_queue_length().await, 0); } +#[tokio::test] +async fn root_recovery_cluster_replay_rejects_successor_and_preserves_completed_scope() { + use crate::heal::manager::root_recovery::AdminClusterScope; + use crate::heal::outcome::{HealObjectDisposition, HealObjectIdentity, HealObjectKind, HealObjectOutcome}; + let (_temp, disk) = recovery_disk().await; + let manager = recovery_manager(vec![disk]); + let request = root_request(); + manager.root_recovery.persist(&request).await.expect("admit cluster task"); + let task = HealTask::from_request(request.clone(), Arc::new(MockStorage)); + let bucket = format!("pinned-{}", Uuid::new_v4().simple()); + let expected = Uuid::from_u128(42); + let mut outcome = HealTaskOutcome::default(); + outcome.start(); + let mut scope = AdminClusterScope { + buckets: std::collections::BTreeMap::from([(bucket.clone(), expected)]), + next_bucket_index: 0, + outcome, + progress: HealProgress::new(), + }; + manager + .root_recovery + .checkpoint_cluster_scope(&task, &scope) + .await + .expect("freeze generation"); + bucket_incarnations() + .lock() + .expect("generation fixture") + .insert(bucket.clone(), Some(Uuid::from_u128(99))); + let replay = + HealTask::from_request(request.clone(), Arc::new(MockStorage)).with_admin_recovery(manager.root_recovery.clone()); + assert!( + matches!(replay.execute().await, Err(Error::StaleBucketIncarnation { .. })), + "replay must reject a successor before traversal" + ); + assert_eq!(replay.get_outcome().await.counters.processed, 0); + bucket_incarnations() + .lock() + .expect("generation fixture") + .insert(bucket.clone(), Some(expected)); + scope.next_bucket_index = 1; + scope.outcome.record(HealObjectOutcome { + identity: HealObjectIdentity { + kind: HealObjectKind::Object, + bucket: bucket.clone(), + object: "completed-object".to_owned(), + version_id: None, + bucket_incarnation_id: Some(expected), + pool_index: None, + set_index: None, + }, + disposition: HealObjectDisposition::VerifiedHealthy, + detail: None, + }); + scope.progress.objects_scanned = 1; + manager + .root_recovery + .checkpoint_cluster_scope(&task, &scope) + .await + .expect("completed bucket with retained outcome"); + let replay = HealTask::from_request(request, Arc::new(MockStorage)).with_admin_recovery(manager.root_recovery.clone()); + replay + .execute() + .await + .expect("completed frozen scope needs no rediscovery or repair"); + assert_eq!(replay.get_outcome().await.coverage, crate::heal::outcome::HealTraversalCoverage::Complete); + assert_eq!(replay.get_outcome().await.counters.processed, 1); + assert_eq!(replay.get_progress().await.objects_scanned, 1); + bucket_incarnations().lock().expect("generation fixture").remove(&bucket); +} + +#[tokio::test] +async fn root_recovery_cluster_inventory_exceeds_legacy_limit_without_losing_bindings() { + use crate::heal::manager::root_recovery::AdminClusterScope; + let (_temp, disk) = recovery_disk().await; + let manager = recovery_manager(vec![disk.clone()]); + let request = root_request(); + manager.root_recovery.persist(&request).await.expect("admit cluster task"); + let task = HealTask::from_request(request.clone(), Arc::new(MockStorage)); + let mut outcome = HealTaskOutcome::default(); + outcome.start(); + let scope = AdminClusterScope { + buckets: (0..1500) + .map(|index| (format!("bucket-{index:05}"), Uuid::from_u128(42))) + .collect(), + next_bucket_index: 0, + outcome, + progress: HealProgress::new(), + }; + assert!(scope.checkpoint_interval() > 1, "large inventories must batch durable rewrites"); + manager + .root_recovery + .checkpoint_cluster_scope(&task, &scope) + .await + .expect("persist large frozen inventory"); + let path = format!("root-heal-{}.json", request.id); + let bytes = disk.read_all(RUSTFS_META_BUCKET, &path).await.expect("scope bytes"); + assert!(bytes.len() > MAX_ROOT_RECOVERY_BYTES, "fixture must cross legacy size limit"); + assert_eq!( + manager + .root_recovery + .cluster_scope(&task) + .await + .expect("all bindings survive"), + Some(scope) + ); + let json = String::from_utf8(bytes.to_vec()).expect("UTF-8 scope"); + let binding = format!("\"bucket-00000\":\"{}\"", Uuid::from_u128(42)); + let duplicate = json.replacen(&binding, &format!("{binding},{binding}"), 1); + assert_ne!(duplicate, json, "inject a duplicate map key"); + disk.write_all(RUSTFS_META_BUCKET, &path, duplicate.into_bytes().into()) + .await + .expect("inject corrupt duplicate binding"); + assert!( + manager.root_recovery.cluster_scope(&task).await.is_err(), + "duplicate identity bindings fail closed" + ); +} + +#[tokio::test] +async fn root_recovery_cluster_rejects_inventory_without_checkpoint_headroom() { + use crate::heal::manager::root_recovery::AdminClusterScope; + let (_temp, disk) = recovery_disk().await; + let manager = recovery_manager(vec![disk]); + let request = root_request(); + manager.root_recovery.persist(&request).await.expect("admit cluster task"); + let task = HealTask::from_request(request, Arc::new(MockStorage)); + let mut outcome = HealTaskOutcome::default(); + outcome.start(); + let scope = AdminClusterScope { + // This inventory fits the 16 MiB record limit but leaves too little + // space for future object outcomes. Reject it before authorizing repair. + buckets: (0..152_000) + .map(|index| (format!("bucket-{index:056}"), Uuid::from_u128(42))) + .collect(), + next_bucket_index: 0, + outcome, + progress: HealProgress::new(), + }; + assert!(manager.root_recovery.checkpoint_cluster_scope(&task, &scope).await.is_err()); + assert!( + manager + .root_recovery + .cluster_scope(&task) + .await + .expect("original admission survives") + .is_none() + ); +} + #[tokio::test] async fn root_recovery_queued_owner_is_not_priority_displaced() { let (_temp, disk) = recovery_disk().await; @@ -2127,7 +2379,16 @@ async fn root_recovery_force_start_replaces_fresh_queued_and_retrying_admin_root #[tokio::test] async fn root_recovery_invalid_records_are_quarantined_without_blocking_replay() { - for kind in ["truncated", "schema", "identity", "option", "no_lock", "oversized"] { + use crate::heal::manager::root_recovery::MAX_CLUSTER_RECOVERY_BYTES; + for kind in [ + "truncated", + "schema", + "identity", + "option", + "no_lock", + "unscoped_oversized", + "oversized", + ] { let (_temp, disk) = recovery_disk().await; let manager = recovery_manager(vec![disk.clone()]); let valid = root_request(); @@ -2142,7 +2403,7 @@ async fn root_recovery_invalid_records_are_quarantined_without_blocking_replay() let original = disk.read_all(RUSTFS_META_BUCKET, &path).await.expect("read root record"); let mut value: serde_json::Value = serde_json::from_slice(&original).expect("record JSON"); match kind { - "schema" => value["schema"] = 4.into(), + "schema" => value["schema"] = u32::MAX.into(), "identity" => value["task_id"] = valid.id.clone().into(), "option" => value["options"]["future_delete_mode"] = true.into(), "no_lock" => value["options"]["no_lock"] = true.into(), @@ -2150,7 +2411,12 @@ async fn root_recovery_invalid_records_are_quarantined_without_blocking_replay() } let bytes = match kind { "truncated" => b"{".to_vec(), - "oversized" => vec![b' '; MAX_ROOT_RECOVERY_BYTES + 1], + "oversized" => vec![b' '; MAX_CLUSTER_RECOVERY_BYTES + 1], + "unscoped_oversized" => { + let mut bytes = serde_json::to_vec(&value).expect("valid unscoped record"); + bytes.resize(MAX_ROOT_RECOVERY_BYTES + 1, b' '); + bytes + } _ => serde_json::to_vec(&value).expect("modified record"), }; disk.write_all(RUSTFS_META_BUCKET, &path, bytes.clone().into()) @@ -2173,7 +2439,11 @@ async fn root_recovery_invalid_records_are_quarantined_without_blocking_replay() let marker: serde_json::Value = serde_json::from_slice(&marker).expect("quarantine marker JSON"); assert_eq!(marker["task_id"], invalid.id, "kind={kind}"); assert_eq!(marker["record_kind"], "intent", "kind={kind}"); - assert_eq!(marker["source_prefix_len"], bytes.len().min(MAX_ROOT_RECOVERY_BYTES + 1), "kind={kind}"); + assert_eq!( + marker["source_prefix_len"], + bytes.len().min(MAX_CLUSTER_RECOVERY_BYTES + 1), + "kind={kind}" + ); assert_eq!(marker["source_oversized"], kind == "oversized", "kind={kind}"); let mut reused = invalid.clone(); @@ -2195,6 +2465,51 @@ async fn root_recovery_invalid_records_are_quarantined_without_blocking_replay() } } +#[tokio::test] +async fn root_recovery_legacy_oversized_quarantine_survives_larger_intent_limit() { + use sha2::{Digest, Sha256}; + let (_temp, disk) = recovery_disk().await; + let manager = recovery_manager(vec![disk.clone()]); + let request = root_request(); + let path = format!("root-heal-{}.json", request.id); + let mut bytes = vec![b' '; MAX_ROOT_RECOVERY_BYTES + 100]; + disk.write_all(RUSTFS_META_BUCKET, &path, bytes.clone().into()) + .await + .expect("old oversized intent"); + let marker = serde_json::json!({ + "schema": 1, + "task_id": request.id, + "record_kind": "intent", + "source_prefix_len": MAX_ROOT_RECOVERY_BYTES + 1, + "source_prefix_sha256": Sha256::digest(&bytes[..MAX_ROOT_RECOVERY_BYTES + 1]).to_vec(), + "source_oversized": true, + "reason": "legacy size limit", + "quarantined_at": SystemTime::now(), + }); + disk.write_all( + RUSTFS_META_BUCKET, + &format!("quarantined-root-heal-intent-{}.json", request.id), + serde_json::to_vec(&marker).unwrap().into(), + ) + .await + .expect("old quarantine marker"); + let inventory = manager + .root_recovery + .inventory() + .await + .expect("upgrade preserves old quarantine"); + assert!(inventory.requests.is_empty()); + assert!(inventory.quarantined_task_ids.contains(&request.id)); + bytes[0] = b'{'; + disk.write_all(RUSTFS_META_BUCKET, &path, bytes.into()) + .await + .expect("alter retained prefix"); + assert!( + manager.root_recovery.inventory().await.is_err(), + "a changed legacy source still fails closed" + ); +} + #[tokio::test] async fn root_recovery_quarantine_source_change_fails_closed() { let (_temp, disk) = recovery_disk().await; diff --git a/crates/heal/src/heal/manager/tests/running_mainline.rs b/crates/heal/src/heal/manager/tests/running_mainline.rs index d7b827ce6..4cc153d28 100644 --- a/crates/heal/src/heal/manager/tests/running_mainline.rs +++ b/crates/heal/src/heal/manager/tests/running_mainline.rs @@ -80,6 +80,13 @@ impl HealStorageAPI for RunningStorage { ..Default::default() })) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> Result> { Ok(Vec::new()) } diff --git a/crates/heal/src/heal/resume.rs b/crates/heal/src/heal/resume.rs index b97617b2d..575922faf 100644 --- a/crates/heal/src/heal/resume.rs +++ b/crates/heal/src/heal/resume.rs @@ -321,6 +321,10 @@ pub struct ResumeState { /// the version listing. `None` means "start from the beginning". #[serde(default)] pub resume_cursor: Option, + /// Ordinary coverage overlapped unresolved hidden delete recovery. The + /// scan must restart after recovery settles, even if another worker settled it. + #[serde(default)] + pub delete_recovery_rescan_pending: bool, /// task id pub task_id: String, /// task type @@ -364,6 +368,11 @@ pub struct ResumeState { /// and may receive one bounded compatibility re-arm. #[serde(default)] pub replacement_legacy_retry_compatibility_done: bool, + /// New generations cannot consume the one-shot compatibility handoff for + /// a pre-format intent stranded by historical rebalance topology. Missing + /// fields in older schema-seven snapshots remain eligible for validation. + #[serde(default)] + pub replacement_preformat_handoff_retry_done: bool, /// The current pass was rewound because the replacement mount was not /// ready. Task-level error persistence must not turn this readiness wait /// back into a consumed retry attempt. @@ -434,6 +443,7 @@ impl ResumeState { Self { schema_version: CURRENT_RESUME_SCHEMA, resume_cursor: None, + delete_recovery_rescan_pending: false, task_id, task_type, set_disk_id, @@ -450,6 +460,7 @@ impl ResumeState { replacement_legacy_import: None, replacement_legacy_successor: None, replacement_legacy_retry_compatibility_done: false, + replacement_preformat_handoff_retry_done: false, replacement_retry_waiting_for_target: false, dangling_delete_retry_not_before: None, start_time: SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs(), @@ -495,6 +506,7 @@ impl ResumeState { // New generations already use readiness-aware persistence and must // never enter the one-shot compatibility path for legacy snapshots. state.replacement_legacy_retry_compatibility_done = true; + state.replacement_preformat_handoff_retry_done = true; state } @@ -1048,6 +1060,30 @@ impl ResumeManager { self.state.read().await.clone() } + pub(crate) async fn set_delete_recovery_rescan_pending(&self, pending: bool) -> Result<()> { + let mut state = self.state.write().await; + let previous = state.delete_recovery_rescan_pending; + if previous == pending { + return Ok(()); + } + state.delete_recovery_rescan_pending = pending; + drop(state); + if let Err(error) = self.save_state_strict().await { + self.state.write().await.delete_recovery_rescan_pending = previous; + return Err(error); + } + Ok(()) + } + + /// Keep the coverage obligation until the companion checkpoint rewind is durable. + pub(crate) async fn restart_scan_after_delete_recovery(&self) -> Result<()> { + let mut state = self.state.write().await; + state.delete_recovery_rescan_pending = true; + state.reset_for_retry(); + drop(state); + self.save_state_strict().await + } + /// update progress pub async fn update_progress(&self, processed: u64, successful: u64, failed: u64, skipped: u64) -> Result<()> { let mut state = self.state.write().await; @@ -1243,6 +1279,9 @@ impl ResumeManager { if state.max_retries == 0 { return Ok(false); } + if self.rearm_preformat_handoff_retry_if_needed(storage).await? { + return Ok(true); + } let legacy_rearm = replacement_retry_is_legacy_transient_skip(&state); let ready = storage .replacement_targets_ready_for_retry( diff --git a/crates/heal/src/heal/resume/checkpoint.rs b/crates/heal/src/heal/resume/checkpoint.rs index 50d27150c..e3049ee65 100644 --- a/crates/heal/src/heal/resume/checkpoint.rs +++ b/crates/heal/src/heal/resume/checkpoint.rs @@ -22,6 +22,9 @@ use std::time::{SystemTime, UNIX_EPOCH}; use tokio::sync::{Mutex as AsyncMutex, RwLock}; use tracing::{debug, warn}; +mod integrity; +use integrity::CheckpointJson; + use super::super::storage_api::owner::{EcstoreConditionalFileUpdate, EcstoreDiskAPI, EcstoreDiskBytes}; use super::super::{BUCKET_META_PREFIX, DiskStore, HealDiskExt, RUSTFS_META_BUCKET}; use super::{ @@ -406,8 +409,10 @@ impl CheckpointManager { async fn load_from_data(disk: DiskStore, task_id: &str, checkpoint_data: Vec) -> Result { validate_resume_task_id(task_id)?; - let mut checkpoint: ResumeCheckpoint = match serde_json::from_slice(&checkpoint_data) { - Ok(checkpoint) => checkpoint, + let (mut persisted, mut checkpoint) = match serde_json::from_slice::(&checkpoint_data) + .and_then(|checkpoint| serde_json::from_slice::(&checkpoint_data).map(|value| (value, checkpoint))) + { + Ok(decoded) => decoded, Err(error) => { Self::block_invalid_snapshot(&disk, task_id).await; return Err(Error::TaskExecutionFailed { @@ -451,7 +456,15 @@ impl CheckpointManager { } let integrity_verified = if let Some(expected) = checkpoint.integrity_digest.as_deref() { - let actual = Self::checkpoint_digest(&Self::serialize_without_digest(&checkpoint)?); + // Verify the persisted fields: typed defaults can add fields that + // were absent from an otherwise intact older signed checkpoint. + let mut actual = Self::checkpoint_digest(&persisted.unsigned_bytes()?); + if expected != actual { + // RUSTFS_COMPAT_TODO(#2871): retain until snapshots whose + // payload order differs from their digest order are retired. + persisted.sort_all_objects(); + actual = Self::checkpoint_digest(&persisted.unsigned_bytes()?); + } if expected != actual { Self::block_invalid_snapshot(&disk, task_id).await; return Err(Error::InvalidCheckpoint(format!( @@ -602,6 +615,18 @@ impl CheckpointManager { self.save_checkpoint_throttled().await } + pub(crate) async fn reset_after_delete_recovery(&self) -> Result<()> { + let mut checkpoint = self.checkpoint.write().await; + checkpoint.reset_for_retry(); + if let Some(admin) = checkpoint.admin.as_mut() { + admin.completed = false; + admin.outcome = Default::default(); + admin.outcome.start(); + } + drop(checkpoint); + self.save_checkpoint().await + } + /// Add a processed object. Called once per healed object, so persistence /// is batched (`PERSIST_EVERY_MUTATIONS` / `PERSIST_INTERVAL`); positions /// and page boundaries still persist unconditionally. @@ -792,10 +817,14 @@ impl CheckpointManager { let _save_guard = self.save_lock.lock().await; let checkpoint = self.checkpoint.read().await.clone(); validate_resume_task_id(&checkpoint.task_id)?; - let unsigned_checkpoint_data = Self::serialize_without_digest(&checkpoint)?; + let checkpoint_value = serde_json::to_value(&checkpoint).map_err(|e| Error::TaskExecutionFailed { + message: format!("Failed to serialize checkpoint: {e}"), + })?; + let mut persisted_checkpoint = CheckpointJson::from_value(checkpoint_value)?; + persisted_checkpoint.sort_all_objects(); + let unsigned_checkpoint_data = persisted_checkpoint.unsigned_bytes()?; let digest = Self::checkpoint_digest(&unsigned_checkpoint_data); - let mut persisted_checkpoint = checkpoint.clone(); - persisted_checkpoint.integrity_digest = Some(digest); + persisted_checkpoint.set_digest(Some(digest))?; let checkpoint_data = EcstoreDiskBytes::from(serde_json::to_vec(&persisted_checkpoint).map_err(|e| Error::TaskExecutionFailed { message: format!("Failed to serialize checkpoint: {e}"), @@ -943,25 +972,6 @@ impl CheckpointManager { }) } - fn serialize_without_digest(checkpoint: &ResumeCheckpoint) -> Result> { - let mut unsigned = checkpoint.clone(); - unsigned.integrity_digest = None; - let mut value = serde_json::to_value(&unsigned).map_err(|e| Error::TaskExecutionFailed { - message: format!("Failed to serialize checkpoint: {e}"), - })?; - for field in ["processed_objects", "failed_objects", "skipped_objects"] { - let Some(values) = value.get_mut(field).and_then(serde_json::Value::as_array_mut) else { - return Err(Error::TaskExecutionFailed { - message: format!("Failed to canonicalize checkpoint field: {field}"), - }); - }; - values.sort_by(|left, right| left.as_str().cmp(&right.as_str())); - } - serde_json::to_vec(&value).map_err(|e| Error::TaskExecutionFailed { - message: format!("Failed to serialize checkpoint: {e}"), - }) - } - fn checkpoint_digest(checkpoint_data: &[u8]) -> String { base64_simd::STANDARD.encode_to_string(Sha256::digest(checkpoint_data)) } diff --git a/crates/heal/src/heal/resume/checkpoint/integrity.rs b/crates/heal/src/heal/resume/checkpoint/integrity.rs new file mode 100644 index 000000000..4752fa7c9 --- /dev/null +++ b/crates/heal/src/heal/resume/checkpoint/integrity.rs @@ -0,0 +1,219 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use crate::{Error, Result}; +use serde::de::{MapAccess, SeqAccess, Visitor}; +use serde::ser::SerializeMap; +use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use serde_json::Value; +use std::fmt; + +// Checkpoint-local ordering preserves historical digests without affecting +// the server's serde_json feature set. +#[derive(Debug)] +pub(super) enum CheckpointJson { + Object(Vec<(String, Self)>), + Array(Vec), + Scalar(Value), +} + +impl CheckpointJson { + fn as_str(&self) -> Option<&str> { + match self { + Self::Scalar(Value::String(value)) => Some(value), + _ => None, + } + } + + pub(super) fn from_value(value: Value) -> Result { + fn convert(value: Value, depth: usize) -> Result { + if depth >= 128 { + return Err(Error::InvalidCheckpoint("Checkpoint JSON exceeds the nesting limit".to_string())); + } + match value { + Value::Object(fields) => fields + .into_iter() + .map(|(key, value)| convert(value, depth + 1).map(|value| (key, value))) + .collect::>() + .map(CheckpointJson::Object), + Value::Array(values) => values + .into_iter() + .map(|value| convert(value, depth + 1)) + .collect::>() + .map(CheckpointJson::Array), + scalar => Ok(CheckpointJson::Scalar(scalar)), + } + } + convert(value, 0) + } + + pub(super) fn sort_all_objects(&mut self) { + let mut pending = vec![self]; + while let Some(value) = pending.pop() { + match value { + Self::Object(fields) => { + fields.sort_unstable_by(|left, right| left.0.cmp(&right.0)); + pending.extend(fields.iter_mut().map(|(_, value)| value)); + } + Self::Array(values) => pending.extend(values), + Self::Scalar(_) => {} + } + } + } + + pub(super) fn set_digest(&mut self, digest: Option) -> Result<()> { + let Self::Object(fields) = self else { + return Err(Error::InvalidCheckpoint("Checkpoint must be a JSON object".to_string())); + }; + let (_, value) = fields + .iter_mut() + .find(|(key, _)| key == "integrity_digest") + .ok_or_else(|| Error::InvalidCheckpoint("Signed checkpoint is missing its integrity digest field".to_string()))?; + *value = Self::Scalar(digest.map_or(Value::Null, Value::String)); + Ok(()) + } + + pub(super) fn unsigned_bytes(&mut self) -> Result> { + self.set_digest(None)?; + let Self::Object(fields) = self else { + return Err(Error::InvalidCheckpoint("Checkpoint must be a JSON object".to_string())); + }; + for field in ["processed_objects", "failed_objects", "skipped_objects"] { + let Some((_, Self::Array(values))) = fields.iter_mut().find(|(key, _)| key == field) else { + return Err(Error::InvalidCheckpoint(format!("Checkpoint set is missing or invalid: {field}"))); + }; + values.sort_unstable_by(|left, right| left.as_str().cmp(&right.as_str())); + } + serde_json::to_vec(self).map_err(|error| Error::InvalidCheckpoint(format!("Failed to encode checkpoint digest: {error}"))) + } +} + +impl Serialize for CheckpointJson { + fn serialize(&self, serializer: S) -> std::result::Result { + match self { + Self::Object(fields) => { + let mut map = serializer.serialize_map(Some(fields.len()))?; + for (key, value) in fields { + map.serialize_entry(key, value)?; + } + map.end() + } + Self::Array(values) => values.serialize(serializer), + Self::Scalar(value) => value.serialize(serializer), + } + } +} + +impl<'de> Deserialize<'de> for CheckpointJson { + fn deserialize>(deserializer: D) -> std::result::Result { + struct JsonVisitor; + + impl<'de> Visitor<'de> for JsonVisitor { + type Value = CheckpointJson; + + fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("checkpoint JSON") + } + + fn visit_map>(self, mut map: A) -> std::result::Result { + let mut fields = Vec::new(); + while let Some(field) = map.next_entry()? { + fields.push(field); + } + Ok(CheckpointJson::Object(fields)) + } + + fn visit_seq>(self, mut seq: A) -> std::result::Result { + let mut values = Vec::new(); + while let Some(value) = seq.next_element()? { + values.push(value); + } + Ok(CheckpointJson::Array(values)) + } + + fn visit_bool(self, value: bool) -> std::result::Result { + Ok(CheckpointJson::Scalar(Value::Bool(value))) + } + + fn visit_i64(self, value: i64) -> std::result::Result { + Ok(CheckpointJson::Scalar(Value::Number(value.into()))) + } + + fn visit_u64(self, value: u64) -> std::result::Result { + Ok(CheckpointJson::Scalar(Value::Number(value.into()))) + } + + fn visit_f64(self, value: f64) -> std::result::Result { + serde_json::Number::from_f64(value) + .map(|value| CheckpointJson::Scalar(Value::Number(value))) + .ok_or_else(|| E::custom("non-finite checkpoint number")) + } + + fn visit_str(self, value: &str) -> std::result::Result { + self.visit_string(value.to_owned()) + } + + fn visit_string(self, value: String) -> std::result::Result { + Ok(CheckpointJson::Scalar(Value::String(value))) + } + + fn visit_unit(self) -> std::result::Result { + Ok(CheckpointJson::Scalar(Value::Null)) + } + } + + deserializer.deserialize_any(JsonVisitor) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn checkpoint_json_preserves_nested_order_and_scalar_values() { + let wire = r#"{"z":{"z":18446744073709551615,"a":-9223372036854775808,"f":1.5,"s":"hello\nworld","b":true,"n":null},"other":[2,1],"processed_objects":["b","a"],"failed_objects":[],"skipped_objects":[],"integrity_digest":"signed"}"#; + let mut value: CheckpointJson = serde_json::from_str(wire).expect("parse ordered checkpoint JSON"); + assert_eq!(serde_json::to_string(&value).expect("encode ordered checkpoint JSON"), wire); + let unsigned = value.unsigned_bytes().expect("normalize only checkpoint sets and digest"); + assert_eq!( + String::from_utf8(unsigned).expect("unsigned checkpoint must be UTF-8"), + wire.replace("[\"b\",\"a\"]", "[\"a\",\"b\"]").replace("\"signed\"", "null") + ); + } + + #[test] + fn checkpoint_json_sorts_nested_objects_without_reordering_other_arrays() { + let mut value: CheckpointJson = + serde_json::from_str(r#"{"z":[{"z":2,"a":1},3,1],"a":false}"#).expect("parse nested checkpoint JSON"); + value.sort_all_objects(); + assert_eq!( + serde_json::to_string(&value).expect("encode sorted nested checkpoint JSON"), + r#"{"a":false,"z":[{"a":1,"z":2},3,1]}"# + ); + } + + #[test] + fn checkpoint_json_bounds_both_input_and_value_conversion_depth() { + for depth in [127, 128] { + let wire = format!("{}null{}", "[".repeat(depth), "]".repeat(depth)); + assert_eq!(serde_json::from_str::(&wire).is_ok(), depth == 127); + let mut value = Value::Null; + for _ in 0..depth { + value = Value::Array(vec![value]); + } + assert_eq!(CheckpointJson::from_value(value).is_ok(), depth == 127); + } + } +} diff --git a/crates/heal/src/heal/resume/handoff.rs b/crates/heal/src/heal/resume/handoff.rs index 47f3a0bdf..001cc71aa 100644 --- a/crates/heal/src/heal/resume/handoff.rs +++ b/crates/heal/src/heal/resume/handoff.rs @@ -89,6 +89,65 @@ pub(super) fn validate_lineage(task_id: &str, set_disk_id: &str, lineage: &[Repl } impl ResumeManager { + // RUSTFS_COMPAT_TODO(8565): recover pre-format intents stranded by historical pool topology. Remove after supported upgrades no longer admit snapshots lacking replacement_preformat_handoff_retry_done and retained affected intents are resolved. + pub(super) async fn rearm_preformat_handoff_retry_if_needed( + &self, + storage: &dyn crate::heal::storage::HealStorageAPI, + ) -> Result { + let state = self.get_state().await; + if !preformat_handoff_retry_candidate(&state) { + return Ok(false); + } + let identities = storage.replacement_target_identities(&state.replacement_targets).await?; + if identities == state.replacement_target_identities { + return Ok(false); + } + // Lock order: target execution leases, the store's format fence during + // the probe, then the intent CAS after the probe releases its fence. + let execution = storage.replacement_execution(&state.replacement_targets).await?; + if execution.identities() != identities.as_slice() { + return Err(Error::ReplacementTargetNotReady( + "target changed before pre-format recovery admission".to_string(), + )); + } + // A cleared summary does not prove an unstarted pass. Any checkpoint, + // including an empty/corrupt file or an isolated checkpoint marker, + // prevents this compatibility admission. + for suffix in [RESUME_CHECKPOINT_FILE, RESUME_CHECKPOINT_BLOCKED_FILE] { + let path = Path::new(BUCKET_META_PREFIX).join(format!("{}_{suffix}", state.task_id)); + match self.disk.read_all(RUSTFS_META_BUCKET, path_to_str(&path)?).await { + Ok(_) => return Ok(false), + Err(DiskError::FileNotFound) => {} + Err(error) => return Err(error.into()), + } + } + if Self::replacement_completion_proof_if_present(self.disk.clone(), &state.task_id) + .await? + .is_some() + || !storage + .replacement_preformat_retry_is_safe(&state.set_disk_id, &state.replacement_targets) + .await? + { + return Ok(false); + } + let buckets = storage.list_buckets().await?.into_iter().map(|bucket| bucket.name).collect(); + if storage + .replacement_target_identities(&state.replacement_targets) + .await? + .as_slice() + != execution.identities() + { + return Err(Error::ReplacementTargetNotReady( + "target changed during pre-format recovery admission".to_string(), + )); + } + // Retain the execution leases through validation and the Prepared CAS. + // Budget admission, its one-shot marker, and its reserved successor are + // one durable decision, so a crash cannot grant an unbound retry. + self.prepare_replacement_handoff_inner(&execution, buckets, true).await?; + Ok(true) + } + /// Shared startup/scanner decision. A changed mount starts a full scan in /// one durable successor; an unfinished transfer always reuses that UUID. pub(crate) async fn resolve_replacement_recovery( @@ -184,11 +243,23 @@ impl ResumeManager { /// Select a successor only after every old executor is fenced by the target /// execution leases. The durable predecessor is the sole handoff authority. pub(crate) async fn prepare_replacement_handoff( + &self, + execution: &ReplacementExecution, + buckets: Vec, + ) -> Result { + self.prepare_replacement_handoff_inner(execution, buckets, false).await + } + + async fn prepare_replacement_handoff_inner( &self, execution: &ReplacementExecution, mut buckets: Vec, + preformat_retry: bool, ) -> Result { let state = self.get_state().await; + if preformat_retry && !preformat_handoff_retry_candidate(&state) { + return Err(replacement_recovery_conflict("pre-format recovery eligibility changed")); + } if let Some(handoff) = &state.replacement_handoff { Self::validate_handoff(handoff, &state)?; let raw = Self::read_state_file(&self.disk, &state.task_id, self.state_file).await?; @@ -219,7 +290,7 @@ impl ResumeManager { "replacement generation is not eligible for an online handoff", )); } - if state.retry_count.saturating_add(1) >= state.max_retries { + if !preformat_retry && state.retry_count.saturating_add(1) >= state.max_retries { return Err(Error::ReplacementRetryBudgetExhausted); } if state.replacement_phase == ReplacementPhase::OwnershipPending { @@ -280,6 +351,12 @@ impl ResumeManager { if current.replacement_revision != state.replacement_revision { return Err(replacement_recovery_conflict("replacement changed during handoff preparation")); } + if preformat_retry { + current.replacement_preformat_handoff_retry_done = true; + // One handoff consumes one attempt; its successor must still + // have exactly one actual rebuild attempt under the same cap. + current.retry_count = current.max_retries - 2; + } current.replacement_handoff = Some(ReplacementHandoff { phase: ReplacementHandoffPhase::Prepared, link, @@ -378,6 +455,8 @@ impl ResumeManager { link.targets.clone(), ); state.replacement_phase = ReplacementPhase::OwnershipPending; + // The constructor marks this compatibility grant consumed for every + // successor, including descendants of a recovered older generation. state.replacement_predecessor = Some(predecessor.task_id.clone()); state.retry_count = predecessor .retry_count @@ -493,6 +572,31 @@ impl ResumeManager { } } +fn preformat_handoff_retry_candidate(state: &ResumeState) -> bool { + replacement_retry_is_exhausted_active(state) + && state.replacement_generation.as_deref() == Some(state.task_id.as_str()) + && !state.replacement_preformat_handoff_retry_done + && state.replacement_phase == ReplacementPhase::Intent + && state.replacement_execution_protocol == 1 + && state.max_retries >= 2 + && state.replacement_predecessor.is_none() + && state.replacement_handoff.is_none() + && state.replacement_lineage.is_empty() + && state.replacement_legacy_import.is_none() + && state.replacement_legacy_successor.is_none() + && state.resume_cursor.is_none() + && state.current_bucket.is_none() + && state.current_object.is_none() + && state.completed_buckets.is_empty() + && state.processed_objects == 0 + && state.successful_objects == 0 + && state.failed_objects == 0 + && state.skipped_objects == 0 + && state.skipped_new_versions == 0 + && state.skipped_ilm_expired == 0 + && state.processed_bytes == 0 +} + fn successor_can_be_rebound(state: &ResumeState, predecessor: &str) -> bool { state.replacement_predecessor.as_deref() == Some(predecessor) && successor_has_no_progress(state) } diff --git a/crates/heal/src/heal/resume/handoff/tests.rs b/crates/heal/src/heal/resume/handoff/tests.rs index a6aaf9442..a358a0579 100644 --- a/crates/heal/src/heal/resume/handoff/tests.rs +++ b/crates/heal/src/heal/resume/handoff/tests.rs @@ -13,10 +13,451 @@ // limitations under the License. use super::*; -use crate::heal::{HEALING_MARKER_PATH, resume::tests::schema_test_disk}; -use std::sync::LazyLock; +use crate::heal::{ + HEALING_MARKER_PATH, HealObjectInfo, + resume::tests::schema_test_disk, + storage::{HealListItem, HealStorageAPI}, + storage_api::storage::{BucketInfo, DiskSetSelector}, +}; +use rustfs_heal_contracts::heal_channel::HealOpts; +use rustfs_madmin::heal_commands::HealResultItem; +use std::sync::{ + LazyLock, + atomic::{AtomicUsize, Ordering}, +}; static FAILURES: LazyLock>> = LazyLock::new(|| Mutex::new(HashMap::new())); + +struct PreformatRecoveryStorage { + execution: Arc, + historical_topology: bool, + probe_calls: AtomicUsize, + identity_calls: AtomicUsize, + change_identity_on_call: usize, +} + +#[async_trait::async_trait] +impl HealStorageAPI for PreformatRecoveryStorage { + async fn replacement_target_identities(&self, _: &[String]) -> Result> { + let mut identities = self.execution.identities().to_vec(); + if self.identity_calls.fetch_add(1, Ordering::SeqCst) == self.change_identity_on_call { + identities[0].filesystem_identity = "another-mount:fs:root".to_string(); + } + Ok(identities) + } + + async fn replacement_execution(&self, _: &[String]) -> Result> { + Ok(self.execution.clone()) + } + + async fn replacement_targets_ready_for_retry( + &self, + _: &str, + _: &[String], + expected: &[ReplacementTargetIdentity], + ) -> Result { + if expected != self.execution.identities() { + return Ok(false); + } + for disk in self.execution.test_disks() { + if !matches!(disk.get_disk_id().await, Ok(Some(id)) if !id.is_nil()) { + return Ok(false); + } + } + Ok(true) + } + + async fn replacement_preformat_retry_is_safe(&self, _: &str, _: &[String]) -> Result { + self.probe_calls.fetch_add(1, Ordering::SeqCst); + if !self.historical_topology { + return Ok(false); + } + for disk in self.execution.test_disks() { + match disk.read_all(RUSTFS_META_BUCKET, "format.json").await { + Err(DiskError::FileNotFound | DiskError::VolumeNotFound) => {} + Ok(_) => return Ok(false), + Err(error) => return Err(error.into()), + } + } + Ok(true) + } + + async fn list_buckets(&self) -> Result> { + Ok(Vec::new()) + } + + async fn get_object_meta(&self, _: &str, _: &str) -> Result> { + Err(Error::other("pre-format recovery must not inspect objects")) + } + + async fn ec_decode_rebuild(&self, _: &str, _: &str) -> Result> { + Err(Error::other("pre-format recovery must not rebuild objects")) + } + + async fn get_bucket_info(&self, _: &str) -> Result> { + Err(Error::other("pre-format recovery must not inspect buckets")) + } + + async fn list_buckets_for_admin_heal(&self, _: Option) -> Result> { + Err(Error::other("pre-format recovery must not start admin discovery")) + } + + async fn object_exists(&self, _: &str, _: &str) -> Result { + Err(Error::other("pre-format recovery must not inspect objects")) + } + + async fn heal_object(&self, _: &str, _: &str, _: Option<&str>, _: &HealOpts) -> Result<(HealResultItem, Option)> { + Err(Error::other("pre-format recovery must not heal objects")) + } + + async fn heal_bucket(&self, _: &str, _: &HealOpts) -> Result { + Err(Error::other("pre-format recovery must not heal buckets")) + } + + async fn heal_format(&self, _: bool) -> Result<(HealResultItem, Option)> { + Err(Error::other("pre-format recovery must not write formats")) + } + + async fn list_objects_for_heal_page( + &self, + _: &str, + _: &str, + _: Option<&str>, + _: bool, + ) -> Result<(Vec, Option, bool)> { + Err(Error::other("pre-format recovery must not traverse objects")) + } + + async fn get_disk_for_resume(&self, _: &str) -> Result { + Err(Error::other("pre-format recovery must retain its durable anchor")) + } +} + +async fn preformat_recovery_fixture() -> (Vec, ResumeManager, PreformatRecoveryStorage) { + let (dirs, manager, execution) = fixture(1).await; + execution.test_disks()[0] + .delete(RUSTFS_META_BUCKET, HEALING_MARKER_PATH, Default::default()) + .await + .expect("remove the fixture marker from the blank replacement"); + { + let mut state = manager.state.write().await; + state.retry_count = state.max_retries; + state.replacement_preformat_handoff_retry_done = false; + state.replacement_retry_waiting_for_target = true; + } + manager.save_state_strict().await.expect("persist exhausted older intent"); + ( + dirs, + manager, + PreformatRecoveryStorage { + execution, + historical_topology: true, + probe_calls: AtomicUsize::new(0), + identity_calls: AtomicUsize::new(0), + change_identity_on_call: usize::MAX, + }, + ) +} + +#[tokio::test] +async fn preformat_recovery_same_mount_reuses_existing_readiness_rearm() { + let (_dirs, manager, mut storage) = preformat_recovery_fixture().await; + let initial = manager.get_state().await; + storage.execution = + ReplacementExecution::for_test(storage.execution.test_disks().to_vec(), initial.replacement_target_identities.clone()); + assert!(matches!( + storage.execution.test_disks()[0].get_disk_id().await, + Err(DiskError::UnformattedDisk) + )); + assert!( + manager + .rearm_replacement_recovery_if_needed(&storage) + .await + .expect("rearm blank same mount") + ); + let resumed = manager + .resolve_replacement_recovery(&storage) + .await + .expect("resume the same generation"); + assert_eq!(resumed.task_id, initial.task_id); + assert_eq!(resumed.retry_count, resumed.max_retries - 1); + assert!(resumed.replacement_handoff.is_none()); + assert!(!resumed.replacement_preformat_handoff_retry_done); + assert_eq!(storage.probe_calls.load(Ordering::SeqCst), 0); +} + +#[tokio::test] +async fn preformat_recovery_reserves_one_handoff_and_one_rebuild() { + let (_dirs, manager, mut storage) = preformat_recovery_fixture().await; + assert!( + manager + .rearm_replacement_recovery_if_needed(&storage) + .await + .expect("admit historical pre-format recovery") + ); + let prepared = manager.get_state().await; + assert_eq!(prepared.replacement_phase, ReplacementPhase::HandoffPending); + assert!(prepared.replacement_preformat_handoff_retry_done); + let child_state = manager + .resolve_replacement_recovery(&storage) + .await + .expect("resolve the reserved successor"); + assert_eq!(child_state.retry_count, child_state.max_retries - 1); + assert!(child_state.replacement_preformat_handoff_retry_done); + assert_eq!(child_state.replacement_lineage.len(), 1); + let child = ResumeManager::load_replacement_intent(manager.disk.clone(), &child_state.task_id) + .await + .expect("reload the admitted successor"); + // Unlike a generic schema-test disk, an admitted replacement has an exact + // set/slot binding that get_disk_id validates against its persisted format. + let mut endpoint = storage.execution.test_disks()[0].endpoint(); + endpoint.set_pool_index(0); + endpoint.set_set_index(0); + endpoint.set_disk_index(0); + let disk = crate::heal::new_disk( + &endpoint, + &crate::heal::DiskOption { + cleanup: false, + health_check: false, + }, + ) + .await + .expect("open the replacement with its real erasure-set slot"); + storage.execution = ReplacementExecution::for_test(vec![disk.clone()], storage.execution.identities().to_vec()); + let disk_id = Uuid::new_v4(); + let format = serde_json::json!({ + "version": "1", "format": "xl", "id": Uuid::new_v4(), + "xl": {"version": "3", "this": disk_id, "sets": [[disk_id]], "distributionAlgo": "SIPMOD+PARITY"} + }); + storage.execution.test_disks()[0] + .write_all( + RUSTFS_META_BUCKET, + "format.json", + serde_json::to_vec(&format).expect("serialize target format").into(), + ) + .await + .expect("format the admitted target before rebuilding"); + crate::heal::storage_api::EcstoreDiskAPI::set_disk_id(disk.as_ref(), Some(disk_id)) + .await + .expect("publish the formatted disk identity as save_format_file does"); + assert_eq!(disk.get_disk_id().await.expect("formatted replacement must be ready"), Some(disk_id)); + child + .acquire_replacement_markers(&storage.execution) + .await + .expect("commit successor ownership"); + child + .mark_replacement_rebuilding(storage.execution.identities().to_vec()) + .await + .expect("admit actual rebuilding"); + child + .record_replacement_failure(&Error::other("actual rebuild failed"), child_state.max_retries) + .await + .expect("exhaust final attempt"); + assert!( + !child + .rearm_replacement_recovery_if_needed(&storage) + .await + .expect("never regrant the successor") + ); + let reloaded = ResumeManager::load_replacement_intent(manager.disk.clone(), &prepared.task_id) + .await + .expect("reload predecessor"); + assert!( + !reloaded + .rearm_preformat_handoff_retry_if_needed(&storage) + .await + .expect("never regrant the predecessor") + ); + assert_eq!(storage.probe_calls.load(Ordering::SeqCst), 1); +} + +#[tokio::test] +async fn preformat_recovery_replays_prepared_and_published_crashes_without_new_credit() { + for boundary in ["prepared", "successor_published"] { + let (_dirs, manager, storage) = preformat_recovery_fixture().await; + let source = manager.get_state().await.task_id; + FAILURES + .lock() + .expect("failure injection lock") + .insert(source.clone(), boundary); + assert!(manager.rearm_replacement_recovery_if_needed(&storage).await.is_err()); + let reloaded = ResumeManager::load_replacement_intent(manager.disk.clone(), &source) + .await + .expect("reload crashed grant"); + let prepared = reloaded.get_state().await; + assert!(prepared.replacement_preformat_handoff_retry_done); + let successor = prepared + .replacement_handoff + .expect("grant and successor are atomic") + .link + .successor; + assert!( + !reloaded + .rearm_preformat_handoff_retry_if_needed(&storage) + .await + .expect("grant already used") + ); + let recovered = reloaded + .resolve_replacement_recovery(&storage) + .await + .expect("replay the reserved handoff"); + assert_eq!(recovered.task_id, successor); + assert_eq!(recovered.retry_count, recovered.max_retries - 1); + assert_eq!(storage.probe_calls.load(Ordering::SeqCst), 1); + } +} + +#[tokio::test] +async fn preformat_recovery_rejects_progress_checkpoints_formats_and_unrelated_topology() { + for rejection in [ + "progress", + "checkpoint", + "blocked", + "completion_proof", + "format", + "topology", + "new_generation", + "single_retry", + ] { + let (_dirs, manager, mut storage) = preformat_recovery_fixture().await; + let source = manager.get_state().await.task_id; + match rejection { + "progress" => manager.state.write().await.processed_objects = 1, + "checkpoint" | "blocked" => { + let suffix = if rejection == "checkpoint" { + RESUME_CHECKPOINT_FILE + } else { + RESUME_CHECKPOINT_BLOCKED_FILE + }; + manager + .disk + .write_all(RUSTFS_META_BUCKET, &format!("{BUCKET_META_PREFIX}/{source}_{suffix}"), Vec::new().into()) + .await + .expect("persist checkpoint evidence, including an empty file"); + } + "format" => storage.execution.test_disks()[0] + .write_all(RUSTFS_META_BUCKET, "format.json", b"existing format".to_vec().into()) + .await + .expect("persist actual format evidence"), + "completion_proof" => { + let proof = super::super::replacement::ReplacementCompletionProof::from_state(&manager.get_state().await, 1) + .expect("construct a prior completion proof"); + manager + .disk + .write_all( + RUSTFS_META_BUCKET, + path_to_str(&replacement_completion_proof_path(&source)).expect("proof path"), + serde_json::to_vec(&proof).expect("serialize completion proof").into(), + ) + .await + .expect("persist existing completion proof"); + } + "topology" => storage.historical_topology = false, + "new_generation" => manager.state.write().await.replacement_preformat_handoff_retry_done = true, + "single_retry" => manager.state.write().await.max_retries = 1, + _ => unreachable!("known test rejection"), + } + manager.save_state_strict().await.expect("persist rejection fixture"); + let before = ResumeManager::read_state_file(&manager.disk, &source, manager.state_file) + .await + .expect("source bytes before admission"); + assert!( + !manager + .rearm_preformat_handoff_retry_if_needed(&storage) + .await + .expect("reject unsafe recovery"), + "{rejection}" + ); + assert_eq!( + ResumeManager::read_state_file(&manager.disk, &source, manager.state_file) + .await + .expect("source bytes after rejection"), + before, + "{rejection}" + ); + } +} + +#[tokio::test] +async fn preformat_recovery_rejects_mount_changes_before_or_during_probe() { + for change_identity_on_call in [0, 1] { + let (_dirs, manager, mut storage) = preformat_recovery_fixture().await; + storage.change_identity_on_call = change_identity_on_call; + let source = manager.get_state().await.task_id; + let before = ResumeManager::read_state_file(&manager.disk, &source, manager.state_file) + .await + .expect("source bytes before mount change"); + assert!(matches!( + manager.rearm_preformat_handoff_retry_if_needed(&storage).await, + Err(Error::ReplacementTargetNotReady(_)) + )); + assert_eq!( + ResumeManager::read_state_file(&manager.disk, &source, manager.state_file) + .await + .expect("source bytes after mount change"), + before + ); + } +} + +#[tokio::test] +async fn preformat_recovery_rejects_unknown_marker_and_stale_revision() { + let (_dirs, manager, storage) = preformat_recovery_fixture().await; + let source = manager.get_state().await.task_id; + storage.execution.test_disks()[0] + .write_all(RUSTFS_META_BUCKET, HEALING_MARKER_PATH, b"unknown".to_vec().into()) + .await + .expect("publish unknown marker owner"); + let before = ResumeManager::read_state_file(&manager.disk, &source, manager.state_file) + .await + .expect("source before owner conflict"); + assert!(matches!( + manager.rearm_preformat_handoff_retry_if_needed(&storage).await, + Err(Error::ReplacementOwnershipConflict(_)) + )); + assert_eq!( + ResumeManager::read_state_file(&manager.disk, &source, manager.state_file) + .await + .expect("source after owner conflict"), + before + ); + storage.execution.test_disks()[0] + .delete(RUSTFS_META_BUCKET, HEALING_MARKER_PATH, Default::default()) + .await + .expect("remove fixture owner conflict"); + let stale = ResumeManager::load_replacement_intent(manager.disk.clone(), &source) + .await + .expect("open stale source"); + manager.save_state_strict().await.expect("advance source revision"); + assert!(stale.rearm_preformat_handoff_retry_if_needed(&storage).await.is_err()); + let unchanged = ResumeManager::load_replacement_intent(manager.disk.clone(), &source) + .await + .expect("reload authoritative source") + .get_state() + .await; + assert!(!unchanged.replacement_preformat_handoff_retry_done); + assert!(unchanged.replacement_handoff.is_none()); +} + +#[test] +fn preformat_recovery_marker_defaults_only_old_snapshots_to_eligible() { + let state = ResumeState::replacement_intent( + Uuid::new_v4().to_string(), + "erasure_set".into(), + "pool_0_set_0".into(), + Vec::new(), + vec!["replacement-0".into()], + vec![identity("replacement-0", "mount:fs:root")], + ); + assert!(state.replacement_preformat_handoff_retry_done); + let mut wire = serde_json::to_value(state).expect("serialize new intent"); + wire.as_object_mut() + .expect("intent object") + .remove("replacement_preformat_handoff_retry_done"); + let old: ResumeState = serde_json::from_value(wire).expect("deserialize older schema-seven intent"); + assert!(!old.replacement_preformat_handoff_retry_done); +} + #[tokio::test] async fn handoff_rebinding_never_resets_an_exhausted_successor_budget() { let (_dirs, parent, execution) = fixture(1).await; diff --git a/crates/heal/src/heal/resume/tests.rs b/crates/heal/src/heal/resume/tests.rs index bc4cdc303..350ad5cb1 100644 --- a/crates/heal/src/heal/resume/tests.rs +++ b/crates/heal/src/heal/resume/tests.rs @@ -2339,6 +2339,162 @@ async fn test_checkpoint_schema_v5_discarded_on_load() { temp_dir.close().expect("remove schema test directory"); } +// Keep these signed wire fixtures independent of today's struct and its defaults. +const LEGACY_SIGNED_CHECKPOINTS: [&str; 3] = [ + r#"{"schema_version":5,"task_id":"00000000-0000-4000-8000-000000000002","checkpoint_time":1700000000,"current_bucket_index":2,"current_object_index":500,"processed_objects":["a","b"],"failed_objects":["c"],"skipped_objects":["d"],"integrity_digest":"/AZPOA1TZttBQ0ieKM0QBVMDnr5s2YWZqb+3rmg+vds="}"#, + r#"{"schema_version":6,"task_id":"00000000-0000-4000-8000-000000000002","checkpoint_time":1700000000,"current_bucket_index":2,"current_object_index":500,"processed_objects":["a","b"],"failed_objects":["c"],"skipped_objects":["d"],"integrity_digest":"4IBOlSw4ln91DXRnVZD0D7qDt+DKSp205Y6eFwNpm80="}"#, + r#"{"schema_version":6,"task_id":"00000000-0000-4000-8000-000000000002","checkpoint_time":1700000000,"current_bucket_index":2,"current_object_index":500,"processed_objects":["a","b"],"failed_objects":["c"],"skipped_objects":["d"],"integrity_digest":"mkGdi8fQRM6f1yS9NIyf+O8F9FRL4/pBXL79CD/il7k="}"#, +]; + +#[tokio::test] +async fn signed_legacy_checkpoint_verifies_before_schema_upgrade() { + for legacy in LEGACY_SIGNED_CHECKPOINTS { + let (temp_dir, disk) = schema_test_disk().await; + let task_id = "00000000-0000-4000-8000-000000000002"; + let path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}"); + disk.write_all(RUSTFS_META_BUCKET, &path, legacy.as_bytes().to_vec().into()) + .await + .expect("write signed legacy checkpoint"); + + assert!( + CheckpointManager::is_resumable(&disk, task_id) + .await + .expect("discovery must accept an intact signed legacy checkpoint") + ); + let manager = CheckpointManager::load_from_disk(disk.clone(), task_id) + .await + .expect("verify the original signed fields before filling defaults"); + let checkpoint = manager.get_checkpoint().await; + assert_eq!(checkpoint.schema_version, CURRENT_CHECKPOINT_SCHEMA); + assert_eq!(checkpoint.current_bucket_index, 0); + assert_eq!(checkpoint.current_object_index, 0); + assert!(checkpoint.processed_objects.is_empty()); + assert!(checkpoint.failed_objects.is_empty()); + assert!(checkpoint.skipped_objects.is_empty()); + assert_eq!(checkpoint.successful_objects, 0); + assert_eq!(checkpoint.failed_object_count, 0); + assert_eq!(checkpoint.skipped_object_count, 0); + assert!(!checkpoint.baseline_known); + assert!(!CheckpointManager::is_blocked(&disk, task_id).await); + + manager + .update_position(1, 3) + .await + .expect("save upgraded checkpoint with CAS"); + let restored = CheckpointManager::load_from_disk(disk, task_id) + .await + .expect("reload upgraded checkpoint") + .get_checkpoint() + .await; + assert_eq!(restored.current_bucket_index, 1); + assert_eq!(restored.current_object_index, 3); + temp_dir.close().expect("remove signed legacy checkpoint fixture"); + } +} + +#[tokio::test] +async fn signed_legacy_checkpoint_rejects_tampering() { + for legacy in LEGACY_SIGNED_CHECKPOINTS { + let (temp_dir, disk) = schema_test_disk().await; + let task_id = "00000000-0000-4000-8000-000000000002"; + let path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}"); + let tampered = legacy.replace("\"current_object_index\":500", "\"current_object_index\":501"); + disk.write_all(RUSTFS_META_BUCKET, &path, tampered.as_bytes().to_vec().into()) + .await + .expect("write modified signed legacy checkpoint"); + + assert!(matches!( + CheckpointManager::load_from_disk(disk.clone(), task_id).await, + Err(Error::InvalidCheckpoint(message)) if message.contains("digest does not match") + )); + assert!(CheckpointManager::is_blocked(&disk, task_id).await); + assert_eq!( + disk.read_all(RUSTFS_META_BUCKET, &path) + .await + .expect("invalid snapshot must remain available"), + tampered.as_bytes() + ); + temp_dir.close().expect("remove modified legacy checkpoint fixture"); + } +} + +#[tokio::test] +async fn signed_checkpoint_rejects_duplicate_fields() { + let (temp_dir, disk) = schema_test_disk().await; + let task_id = "00000000-0000-4000-8000-000000000002"; + let path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}"); + let duplicate = LEGACY_SIGNED_CHECKPOINTS[2].replace( + "\"current_object_index\":500", + "\"current_object_index\":501,\"current_object_index\":500", + ); + disk.write_all(RUSTFS_META_BUCKET, &path, duplicate.as_bytes().to_vec().into()) + .await + .expect("write checkpoint with conflicting duplicate fields"); + + assert!(matches!( + CheckpointManager::load_from_disk(disk.clone(), task_id).await, + Err(Error::TaskExecutionFailed { message }) if message.contains("duplicate field") + )); + assert!(CheckpointManager::is_blocked(&disk, task_id).await); + temp_dir.close().expect("remove duplicate field checkpoint fixture"); +} + +#[tokio::test] +async fn checkpoint_writer_preserves_base_reader_canonicalization() { + use sha2::{Digest, Sha256}; + + for admin in [false, true] { + let (temp_dir, disk) = schema_test_disk().await; + let task_id = ResumeUtils::generate_task_id(); + let manager = CheckpointManager::new(disk.clone(), task_id.clone()) + .await + .expect("create rollback-compatible checkpoint"); + if admin { + manager + .set_admin(AdminErasureCheckpoint { + set_disk_id: "pool_0_set_0".to_string(), + options: Default::default(), + buckets: [("bucket".to_string(), Some(uuid::Uuid::new_v4()))].into(), + outcome: Default::default(), + completed: false, + }) + .await + .expect("save nested administrator checkpoint"); + } + manager.add_processed_object("b".to_string()).await.expect("record object b"); + manager.add_processed_object("a".to_string()).await.expect("record object a"); + manager.update_position(2, 9).await.expect("flush rollback fixture"); + let path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}"); + let bytes = disk + .read_all(RUSTFS_META_BUCKET, &path) + .await + .expect("read real writer output"); + let mut wire: serde_json::Value = serde_json::from_slice(&bytes).expect("decode writer output"); + wire.sort_all_objects(); + assert_eq!( + bytes.as_ref(), + serde_json::to_vec(&wire).expect("encode recursively sorted payload"), + "persisted keys must have the production base writer's order, including nested admin fields" + ); + let signature = wire["integrity_digest"].take(); + let unsigned = serde_json::to_vec(&wire).expect("encode base reader signing representation"); + assert_eq!( + signature, + base64_simd::STANDARD.encode_to_string(Sha256::digest(&unsigned)), + "the production base reader must accept this writer's signature" + ); + let restored = CheckpointManager::load_from_disk(disk, &task_id) + .await + .expect("same-version control must also load") + .get_checkpoint() + .await; + assert_eq!(restored.current_bucket_index, 2); + assert_eq!(restored.current_object_index, 9); + assert_eq!(restored.admin.is_some(), admin); + temp_dir.close().expect("remove rollback canonicalization fixture"); + } +} + #[tokio::test] async fn downgraded_unsigned_checkpoint_resets_untrusted_progress() { let (temp_dir, disk) = schema_test_disk().await; diff --git a/crates/heal/src/heal/storage.rs b/crates/heal/src/heal/storage.rs index 051159c0e..fc3e8f3db 100644 --- a/crates/heal/src/heal/storage.rs +++ b/crates/heal/src/heal/storage.rs @@ -491,6 +491,10 @@ pub trait HealStorageAPI: Send + Sync { /// Get all buckets async fn list_buckets(&self) -> Result>; + /// Return only a complete bucket inventory for an administrator task. + /// Incomplete discovery must fail before traversal or scope checkpointing. + async fn list_buckets_for_admin_heal(&self, selector: Option) -> Result>; + /// Check object exists async fn object_exists(&self, bucket: &str, object: &str) -> Result; @@ -574,6 +578,29 @@ pub trait HealStorageAPI: Send + Sync { /// Heal bucket using ecstore async fn heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result; + /// Recover transactions omitted by ordinary listings. Alternate providers + /// may explicitly expose no transaction-recovery capability. + async fn recover_delete_transactions( + &self, + _bucket: &str, + _object: Option<&str>, + _expected: Option, + _opts: &HealOpts, + ) -> Result> { + Ok(None) + } + + /// Recover hidden transactions only within the requested object prefix. + async fn recover_delete_prefix_transactions( + &self, + _bucket: &str, + _prefix: &str, + _expected: Option, + _opts: &HealOpts, + ) -> Result> { + Ok(None) + } + /// Heal format using ecstore async fn heal_format(&self, dry_run: bool) -> Result<(HealResultItem, Option)>; @@ -709,6 +736,13 @@ pub trait HealStorageAPI: Send + Sync { Ok(true) } + /// Admit a one-shot recovery of an old pre-format intent only when the + /// current format fence permits its set, rebalance has a quiescent older + /// topology, and every target is a blank, unformatted replacement. + async fn replacement_preformat_retry_is_safe(&self, _set_disk_id: &str, _targets: &[String]) -> Result { + Ok(false) + } + async fn replacement_execution(&self, _targets: &[String]) -> Result> { Err(Error::other("replacement execution lease acquisition is unsupported")) } @@ -1312,6 +1346,14 @@ impl HealStorageAPI for ECStoreHealStorage { } } + async fn list_buckets_for_admin_heal(&self, selector: Option) -> Result> { + let listing = self.ecstore.list_buckets_for_admin_heal(selector).await?; + if !listing.topology_complete { + return Err(Error::BucketDiscoveryIncomplete); + } + Ok(listing.buckets) + } + async fn list_buckets(&self) -> Result> { debug!( target: "rustfs::heal::storage", @@ -1498,6 +1540,32 @@ impl HealStorageAPI for ECStoreHealStorage { .await) } + async fn recover_delete_transactions( + &self, + bucket: &str, + object: Option<&str>, + expected: Option, + opts: &HealOpts, + ) -> Result> { + self.ecstore + .recover_delete_transactions_at_incarnation(bucket, object, expected, opts) + .await + .map_err(Error::Storage) + } + + async fn recover_delete_prefix_transactions( + &self, + bucket: &str, + prefix: &str, + expected: Option, + opts: &HealOpts, + ) -> Result> { + self.ecstore + .recover_delete_prefix_at_incarnation(bucket, prefix, expected, opts) + .await + .map_err(Error::Storage) + } + async fn heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result { debug!( target: "rustfs::heal::storage", @@ -2048,6 +2116,14 @@ impl HealStorageAPI for ECStoreHealStorage { async fn replacement_execution(&self, targets: &[String]) -> Result> { ReplacementExecution::acquire(targets).await } + + async fn replacement_preformat_retry_is_safe(&self, set_disk_id: &str, targets: &[String]) -> Result { + let (pool_index, set_index) = crate::heal::utils::parse_set_disk_id(set_disk_id)?; + self.ecstore + .replacement_preformat_retry_is_safe(pool_index, set_index, targets) + .await + .map_err(Error::Storage) + } } #[cfg(test)] diff --git a/crates/heal/src/heal/storage_api.rs b/crates/heal/src/heal/storage_api.rs index 24a1e8cf0..c1640b757 100644 --- a/crates/heal/src/heal/storage_api.rs +++ b/crates/heal/src/heal/storage_api.rs @@ -12,6 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +pub(crate) use rustfs_ecstore::api::capacity::is_reserved_or_invalid_bucket as ecstore_is_reserved_or_invalid_bucket; pub(crate) use rustfs_ecstore::api::data_usage::{ DATA_USAGE_CACHE_NAME as ECSTORE_DATA_USAGE_CACHE_NAME, load_admin_data_usage_from_backend_cached as ecstore_load_admin_data_usage_from_backend_cached, @@ -26,7 +27,13 @@ pub(crate) use rustfs_ecstore::api::disk::{ }; pub(crate) use rustfs_ecstore::api::disk::{DiskOption as EcstoreDiskOption, new_disk as ecstore_new_disk}; pub(crate) use rustfs_ecstore::api::error::{Error as EcstoreErrorType, StorageError as EcstoreStorageError}; +#[cfg(test)] +pub(crate) use rustfs_ecstore::api::layout::{ + EndpointServerPools as EcstoreEndpointServerPools, Endpoints as EcstoreEndpoints, PoolEndpoints as EcstorePoolEndpoints, +}; pub(crate) use rustfs_ecstore::api::runtime::local_disk_map_read as ecstore_local_disk_map_read; +#[cfg(test)] +pub(crate) use rustfs_ecstore::api::storage::init_local_disks as ecstore_init_local_disks; pub(crate) use rustfs_ecstore::api::storage::{ ECStore as EcstoreStore, HealLifecycleExpiryContext as EcstoreHealLifecycleExpiryContext, HealObjectStorageResult as EcstoreHealObjectStorageResult, POOL_META_NAME as ECSTORE_POOL_META_NAME, @@ -40,8 +47,8 @@ pub(crate) mod owner { ECSTORE_BUCKET_META_PREFIX, ECSTORE_DATA_USAGE_CACHE_NAME, ECSTORE_HEALING_MARKER_PATH, ECSTORE_POOL_META_NAME, ECSTORE_RUSTFS_META_BUCKET, EcstoreConditionalFileUpdate, EcstoreDeleteOptions, EcstoreDiskAPI, EcstoreDiskBytes, EcstoreDiskError, EcstoreDiskResult, EcstoreDiskStore, EcstoreEndpoint, EcstoreErrorType, - EcstoreHealLifecycleExpiryContext, EcstoreStorageError, EcstoreStore, ecstore_load_admin_data_usage_from_backend_cached, - ecstore_local_disk_map_read, + EcstoreHealLifecycleExpiryContext, EcstoreStorageError, EcstoreStore, ecstore_is_reserved_or_invalid_bucket, + ecstore_load_admin_data_usage_from_backend_cached, ecstore_local_disk_map_read, }; pub(crate) use super::{EcstoreDiskOption, ecstore_new_disk}; diff --git a/crates/heal/src/heal/task/heal_bucket.rs b/crates/heal/src/heal/task/heal_bucket.rs index 73bc20291..c6edcea77 100644 --- a/crates/heal/src/heal/task/heal_bucket.rs +++ b/crates/heal/src/heal/task/heal_bucket.rs @@ -132,6 +132,21 @@ fn unavailable_recreate_error(result: &HealResultItem, opts: &HealOpts) -> Optio } impl HealTask { + async fn recover_bucket_delete_transactions(&self, bucket: &str, opts: &HealOpts) -> Result { + if let Some(result) = self + .await_with_control( + self.storage + .recover_delete_transactions(bucket, None, self.bucket_incarnation_id, opts), + ) + .await? + { + let unresolved = result.delete_recovery.is_some_and(|status| status.has_unresolved()); + self.record_result_item(result).await; + return Ok(unresolved); + } + Ok(false) + } + pub(super) async fn heal_bucket(&self, bucket: &str) -> Result<()> { self.pace_mainline().await?; if self.source == HealRequestSource::Admin && matches!(self.heal_type, HealType::Bucket { .. }) { @@ -169,8 +184,10 @@ impl HealTask { "Heal bucket stage entered" ); self.check_control_flags().await?; - let bucket_exists = self.await_with_control(self.storage.get_bucket_info(bucket)).await?.is_some(); - if !bucket_exists { + // Bound administrator repair validates existence and generation under the + // lifecycle fence. Ordinary read quorum can reject recoverable directory loss. + let bound_admin_repair = self.source == HealRequestSource::Admin && self.bucket_incarnation_id.is_some(); + if !bound_admin_repair && self.await_with_control(self.storage.get_bucket_info(bucket)).await?.is_none() { warn!( target: "rustfs::heal::task", event = EVENT_HEAL_BUCKET_RESULT, @@ -220,6 +237,15 @@ impl HealTask { set: self.options.set_index, }; + let recovery_opts = HealOpts { + remove: self.source == HealRequestSource::Admin && self.options.remove_corrupted, + ..heal_opts + }; + // Removal must settle survivor journals before deleting an empty + // replica's publication directory; repair must establish it first. + if heal_opts.remove && self.recover_bucket_delete_transactions(bucket, &recovery_opts).await? { + return Err(Error::transient_skip("Delete recovery must resolve before bucket removal")); + } let heal_result = match self.bucket_incarnation_id { Some(expected) => { self.await_with_control(self.storage.heal_bucket_at_incarnation(bucket, expected, &heal_opts)) @@ -245,9 +271,15 @@ impl HealTask { ); self.record_result_item(result).await; + let unresolved_delete_recovery = + !heal_opts.remove && self.recover_bucket_delete_transactions(bucket, &recovery_opts).await?; + if self.options.recursive { self.heal_bucket_objects(bucket, "").await?; } + if unresolved_delete_recovery { + return Err(Error::transient_skip("Delete recovery remains unresolved after ordinary bucket heal")); + } if !self.options.recursive { let mut progress = self.progress.write().await; @@ -292,17 +324,76 @@ impl HealTask { "Heal cluster started" ); - let bucket_infos = self.await_with_control(self.storage.list_buckets()).await?; + let mut cluster_scope = if self.source == HealRequestSource::Admin && !self.options.dry_run { + let saved = if let Some(recovery) = &self.admin_recovery { + recovery.cluster_scope(self).await? + } else { + None + }; + let scope = if let Some(scope) = saved { + scope + } else { + let mut buckets = std::collections::BTreeMap::new(); + for bucket in self + .await_with_control(self.storage.list_buckets_for_admin_heal(None)) + .await? + { + let incarnation = self + .await_with_control(self.storage.admit_bucket_incarnation(&bucket.name)) + .await?; + buckets.insert(bucket.name, incarnation); + } + let scope = super::super::manager::root_recovery::AdminClusterScope { + buckets, + next_bucket_index: 0, + outcome: self.get_outcome().await, + progress: self.progress.read().await.clone(), + }; + if let Some(recovery) = &self.admin_recovery { + recovery.checkpoint_cluster_scope(self, &scope).await?; + } + scope + }; + // Validate the frozen generations on every replay, including completed buckets. + for (bucket, incarnation) in &scope.buckets { + self.await_with_control(self.storage.validate_bucket_incarnation(bucket, Some(*incarnation))) + .await?; + } + self.restore_outcome(scope.outcome.clone()).await; + *self.progress.write().await = scope.progress.clone(); + Some(scope) + } else { + None + }; + let buckets = if let Some(scope) = &cluster_scope { + scope + .buckets + .iter() + .map(|(name, incarnation)| (name.clone(), Some(*incarnation))) + .collect::>() + } else { + let listed = if self.source == HealRequestSource::Admin { + self.await_with_control(self.storage.list_buckets_for_admin_heal(None)) + .await? + } else { + self.await_with_control(self.storage.list_buckets()).await? + }; + listed.into_iter().map(|bucket| (bucket.name, None)).collect() + }; let mut failed = 0_u64; let mut retryable = 0_u64; let mut permanent = 0_u64; let mut first_object = None; let mut first_error = None; - for bucket_info in bucket_infos { + let start = cluster_scope.as_ref().map_or(0, |scope| scope.next_bucket_index); + let mut checkpointed = start; + for (index, (bucket, incarnation)) in buckets.into_iter().enumerate().skip(start) { self.check_control_flags().await?; + let mut bucket_task = self.clone(); + bucket_task.bucket_incarnation_id = incarnation; let mut retry_attempt = 0_u32; loop { - match self.heal_bucket(&bucket_info.name).await { + match bucket_task.heal_bucket(&bucket).await { Ok(()) => break, Err(Error::TaskCancelled) => return Err(Error::TaskCancelled), Err(Error::TaskTimeout) => return Err(Error::TaskTimeout), @@ -331,15 +422,37 @@ impl HealTask { } else { permanent = permanent.saturating_add(1); } - first_object.get_or_insert(bucket_info.name.clone()); + first_object.get_or_insert(bucket.clone()); first_error.get_or_insert_with(|| err.to_string()); break; } } } + // Retain only the successful contiguous prefix. A later retry replays + // any work after a failure without inheriting its transient counters. + if failed == 0 + && let Some(scope) = &mut cluster_scope + { + scope.next_bucket_index = index.saturating_add(1); + scope.outcome = self.get_outcome().await; + scope.progress = self.progress.read().await.clone(); + if (scope.next_bucket_index.saturating_sub(checkpointed) >= scope.checkpoint_interval() + || scope.next_bucket_index == scope.buckets.len()) + && let Some(recovery) = &self.admin_recovery + { + recovery.checkpoint_cluster_scope(self, scope).await?; + checkpointed = scope.next_bucket_index; + } + } } if failed > 0 { + if let Some(scope) = &cluster_scope + && scope.next_bucket_index > checkpointed + && let Some(recovery) = &self.admin_recovery + { + recovery.checkpoint_cluster_scope(self, scope).await?; + } let failure = BatchHealFailure { scope: "cluster".to_string(), failed, @@ -446,7 +559,33 @@ impl HealTask { "Heal prefix started" ); - self.heal_bucket_objects(bucket, prefix).await + let opts = HealOpts { + dry_run: self.options.dry_run, + remove: self.source == HealRequestSource::Admin && self.options.remove_corrupted, + no_lock: self.options.no_lock, + pool: self.options.pool_index, + set: self.options.set_index, + ..Default::default() + }; + let mut unresolved_delete_recovery = false; + if let Some(result) = self + .await_with_control(self.storage.recover_delete_prefix_transactions( + bucket, + prefix, + self.bucket_incarnation_id, + &opts, + )) + .await? + { + unresolved_delete_recovery = result.delete_recovery.is_some_and(|status| status.has_unresolved()); + self.record_result_item(result).await; + } + + self.heal_bucket_objects(bucket, prefix).await?; + if unresolved_delete_recovery { + return Err(Error::transient_skip("Delete recovery remains unresolved after ordinary prefix heal")); + } + Ok(()) } #[hotpath::measure] diff --git a/crates/heal/src/heal/task/heal_erasure_set.rs b/crates/heal/src/heal/task/heal_erasure_set.rs index df9b852e9..ba32340e8 100644 --- a/crates/heal/src/heal/task/heal_erasure_set.rs +++ b/crates/heal/src/heal/task/heal_erasure_set.rs @@ -460,6 +460,7 @@ impl HealTask { set: self.options.set_index, }; + let mut unresolved_delete_recovery = false; for bucket in buckets.iter().filter(|_| self.source != HealRequestSource::Admin) { // Check control flags before starting each bucket heal self.check_control_flags().await?; @@ -490,6 +491,17 @@ impl HealTask { return Err(err); } } + // Adopt delete proofs only after their publication bucket exists. + if let Some(result) = self + .await_with_control( + self.storage + .recover_delete_transactions(bucket, None, None, &bucket_heal_opts), + ) + .await? + { + unresolved_delete_recovery |= result.delete_recovery.is_some_and(|status| status.has_unresolved()); + self.record_result_item(result).await; + } } // Create erasure set healer with resume support @@ -511,6 +523,7 @@ impl HealTask { heal_opts, self.source, ) + .with_unresolved_delete_recovery(unresolved_delete_recovery) .with_replacement_targets(replacement_targets, is_auto_replacement.then(|| self.id.clone())) .with_pool_metadata_targets(if self.options.recreate_missing && !self.options.dry_run { self.heal_endpoints.clone() diff --git a/crates/heal/src/heal/task/heal_object.rs b/crates/heal/src/heal/task/heal_object.rs index 0dc4ec298..e20d4f315 100644 --- a/crates/heal/src/heal/task/heal_object.rs +++ b/crates/heal/src/heal/task/heal_object.rs @@ -99,6 +99,40 @@ impl HealTask { // recreated bucket after the replay anchor was captured. return self.heal_mrf_partial_write_object(bucket, object, version_id).await; } + let heal_opts = HealOpts { + recursive: self.options.recursive, + dry_run: self.options.dry_run, + remove: self.options.remove_corrupted, + recreate: self.options.recreate_missing, + scan_mode: self.options.scan_mode, + update_parity: self.options.update_parity, + no_lock: self.options.no_lock, + read_repair: self.source == HealRequestSource::ReadRepair, + pool: self.options.pool_index, + set: self.options.set_index, + }; + let mut recovery_handled = false; + if matches!(self.source, HealRequestSource::Admin | HealRequestSource::Internal) { + let recovery_opts = HealOpts { + remove: heal_opts.remove && version_id.is_none_or(str::is_empty), + ..heal_opts + }; + if let Some(result) = self + .await_with_control(self.storage.recover_delete_transactions( + bucket, + Some(object), + self.bucket_incarnation_id, + &recovery_opts, + )) + .await? + { + recovery_handled = true; + self.record_result_item(result).await; + if self.options.dry_run { + return Ok(()); + } + } + } let mut object_exists = match self.await_with_control(self.storage.object_exists(bucket, object)).await { Ok(exists) => exists, Err(err @ Error::TransientSkip { .. }) => { @@ -129,7 +163,7 @@ impl HealTask { object }; - if !object_exists { + if !object_exists && !recovery_handled { // Background loops (scanner/MRF/autoheal/read-repair) routinely // race object deletion, so a missing target is per-object noise // for them; only foreground admin/internal requests keep the warn. @@ -197,18 +231,6 @@ impl HealTask { update_parity = self.options.update_parity, "Heal object stage entered" ); - let heal_opts = HealOpts { - recursive: self.options.recursive, - dry_run: self.options.dry_run, - remove: self.options.remove_corrupted, - recreate: self.options.recreate_missing, - scan_mode: self.options.scan_mode, - update_parity: self.options.update_parity, - no_lock: self.options.no_lock, - read_repair: self.source == HealRequestSource::ReadRepair, - pool: self.options.pool_index, - set: self.options.set_index, - }; let mut expected_identity = self.outcome_identity(bucket, object, version_id, self.options.pool_index, self.options.set_index); expected_identity.bucket_incarnation_id = self.outcome_bucket_incarnation_id(bucket, self.options.dry_run).await?; diff --git a/crates/heal/src/heal/task/tests.rs b/crates/heal/src/heal/task/tests.rs index 9919fb522..56e51e855 100644 --- a/crates/heal/src/heal/task/tests.rs +++ b/crates/heal/src/heal/task/tests.rs @@ -24,6 +24,59 @@ mod canonical_outcome { use super::*; use crate::heal::outcome::{HealExecutionOutcome, HealTraversalCoverage}; + #[tokio::test] + async fn admin_bucket_discovery_incomplete_cannot_finish_an_empty_cluster() { + for dry_run in [false, true] { + let storage = Arc::new(MockStorage { + listed_buckets: Mutex::new(Some(Vec::new())), + bucket_discovery_incomplete: true, + ..Default::default() + }); + let mut request = HealRequest::new( + HealType::Cluster, + HealOptions { + dry_run, + recursive: true, + ..Default::default() + }, + HealPriority::Normal, + ); + request.source = HealRequestSource::Admin; + let task = HealTask::from_request(request, storage.clone()); + assert!(matches!(task.execute().await, Err(Error::BucketDiscoveryIncomplete))); + let outcome = task.get_outcome().await; + assert_eq!(outcome.execution, HealExecutionOutcome::Aborted(HealAbortReason::Untraversable)); + assert_eq!(outcome.coverage, HealTraversalCoverage::Partial); + assert_eq!(outcome.counters.processed, 0); + assert_eq!(outcome.counters.failed, 0); + assert!(storage.bucket_heal_calls.lock().expect("bucket calls").is_empty()); + assert!(storage.heal_object_calls.lock().expect("object calls").is_empty()); + let wire = serde_json::to_value(&outcome).expect("outcome serializes"); + assert_eq!( + crate::heal::outcome::legacy_wire_status("finished", &wire, false) + .expect("legacy status") + .0, + "stopped" + ); + } + } + + #[tokio::test] + async fn admin_bucket_discovery_complete_empty_cluster_finishes() { + let storage = Arc::new(MockStorage { + listed_buckets: Mutex::new(Some(Vec::new())), + ..Default::default() + }); + let mut request = HealRequest::new(HealType::Cluster, HealOptions::default(), HealPriority::Normal); + request.source = HealRequestSource::Admin; + let task = HealTask::from_request(request, storage); + task.execute().await.expect("complete empty namespace is valid"); + let outcome = task.get_outcome().await; + assert_eq!(outcome.execution, HealExecutionOutcome::Completed); + assert_eq!(outcome.coverage, HealTraversalCoverage::Complete); + assert_eq!(outcome.counters.processed, 0); + } + fn bucket_task(storage: Arc) -> HealTask { HealTask::from_request( HealRequest::new( @@ -1481,8 +1534,11 @@ struct MockStorage { truncate_without_token: Mutex, include_object_dir_candidate: Mutex, listed_buckets: Mutex>>, + bucket_discovery_incomplete: bool, bucket_heal_errors: Mutex>>, bucket_heal_calls: Mutex>, + delete_recovery_results: Mutex>>>, + delete_recovery_calls: Mutex>, block_heal_object: Mutex, resume_disk: Mutex>, replacement_resume_disk: Mutex>, @@ -1490,6 +1546,7 @@ struct MockStorage { usage_baseline_error: Mutex, erasure_set_scopes: Mutex>, disk_walk_calls: Mutex>, + disk_walk_items: Mutex>>, } #[test] @@ -1980,6 +2037,16 @@ impl HealStorageAPI for MockStorage { Ok(*self.usage_baseline.lock().unwrap()) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> Result> { + if self.bucket_discovery_incomplete { + return Err(Error::BucketDiscoveryIncomplete); + } + self.list_buckets().await + } + async fn list_buckets(&self) -> Result> { let buckets = self .listed_buckets @@ -2259,6 +2326,34 @@ impl HealStorageAPI for MockStorage { Ok(HealResultItem::default()) } + async fn recover_delete_transactions( + &self, + bucket: &str, + _object: Option<&str>, + _expected: Option, + _opts: &HealOpts, + ) -> Result> { + self.delete_recovery_calls + .lock() + .expect("recovery calls") + .push(bucket.to_owned()); + self.delete_recovery_results + .lock() + .expect("recovery results") + .remove(bucket) + .unwrap_or(Ok(None)) + } + + async fn recover_delete_prefix_transactions( + &self, + bucket: &str, + _prefix: &str, + expected: Option, + opts: &HealOpts, + ) -> Result> { + self.recover_delete_transactions(bucket, None, expected, opts).await + } + async fn heal_format(&self, _dry_run: bool) -> Result<(HealResultItem, Option)> { *self.global_format_calls.lock().unwrap() += 1; if let Some(error) = self.format_error.lock().unwrap().take() { @@ -2424,7 +2519,15 @@ impl HealStorageAPI for MockStorage { if continuation_token.is_some() { return Ok((Vec::new(), None, false)); } - Ok((vec![heal_item(&format!("{set_disk_id}-object"))], None, false)) + Ok(( + self.disk_walk_items + .lock() + .expect("disk-walk items") + .clone() + .unwrap_or_else(|| vec![heal_item(&format!("{set_disk_id}-object"))]), + None, + false, + )) } async fn get_disk_for_resume(&self, _set_disk_id: &str) -> Result { @@ -4368,6 +4471,336 @@ async fn erasure_set_bucket_prepass_failure_stops_before_object_heal() { assert!(storage.object_heal_opts.lock().unwrap().is_empty()); } +#[tokio::test] +async fn erasure_set_unresolved_delete_recovery_preserves_other_buckets_and_ordinary_heal() { + use rustfs_madmin::heal_commands::DeleteRecoveryStatus; + + for status in [ + DeleteRecoveryStatus { + pending: 1, + ..Default::default() + }, + DeleteRecoveryStatus { + blocked: 1, + ..Default::default() + }, + DeleteRecoveryStatus { + failed: 1, + ..Default::default() + }, + ] { + let temp = TempDir::new().expect("temporary directory should be created"); + let disk = make_resume_disk(&temp).await; + let storage = Arc::new(MockStorage { + delete_recovery_results: Mutex::new(HashMap::from([( + "bucket-a".to_owned(), + Ok(Some(HealResultItem { + bucket: "bucket-a".to_owned(), + delete_recovery: Some(status), + ..Default::default() + })), + )])), + resume_disk: Mutex::new(Some(disk)), + list_each_bucket: true, + ..Default::default() + }); + let buckets = vec!["bucket-a".to_owned(), "bucket-b".to_owned()]; + let request = HealRequest::new( + HealType::ErasureSet { + buckets: buckets.clone(), + set_disk_id: "pool_0_set_0".to_owned(), + }, + HealOptions { + timeout: None, + ..Default::default() + }, + HealPriority::Normal, + ); + let task = HealTask::from_request(request, storage.clone()); + + let error = task + .heal_erasure_set(buckets.clone(), "pool_0_set_0".to_owned()) + .await + .expect_err("unresolved recovery must not be reported as complete"); + + assert!(matches!(error, Error::TransientSkip { .. }), "recovery must remain retryable: {error}"); + assert_eq!(*storage.delete_recovery_calls.lock().expect("recovery calls"), buckets); + assert_eq!(*storage.bucket_heal_calls.lock().expect("bucket calls"), buckets); + assert_eq!( + storage.object_heal_opts.lock().expect("object calls").len(), + 4, + "ordinary object heal must run for both buckets" + ); + assert!( + task.get_result_items() + .await + .iter() + .any(|item| item.bucket == "bucket-a" && item.delete_recovery == Some(status)), + "the result retains the exact unresolved recovery category" + ); + } +} + +#[tokio::test] +async fn erasure_set_delete_recovery_fatal_errors_stop_before_other_buckets() { + for error in [ + Error::TaskCancelled, + Error::TaskTimeout, + Error::Storage(EcstoreError::PreconditionFailed), + Error::Storage(EcstoreError::SlowDown), + Error::Storage(EcstoreError::InvalidPath), + ] { + let expected = error.to_string(); + let temp = TempDir::new().expect("temporary directory should be created"); + let disk = make_resume_disk(&temp).await; + let storage = Arc::new(MockStorage { + delete_recovery_results: Mutex::new(HashMap::from([("bucket-a".to_owned(), Err(error))])), + resume_disk: Mutex::new(Some(disk)), + ..Default::default() + }); + let buckets = vec!["bucket-a".to_owned(), "bucket-b".to_owned()]; + let request = HealRequest::new( + HealType::ErasureSet { + buckets: buckets.clone(), + set_disk_id: "pool_0_set_0".to_owned(), + }, + HealOptions { + timeout: None, + ..Default::default() + }, + HealPriority::Normal, + ); + let task = HealTask::from_request(request, storage.clone()); + + let actual = task + .heal_erasure_set(buckets, "pool_0_set_0".to_owned()) + .await + .expect_err("fence and control errors must terminate the prepass"); + + assert_eq!(actual.to_string(), expected); + assert_eq!(*storage.delete_recovery_calls.lock().expect("recovery calls"), ["bucket-a"]); + assert_eq!(*storage.bucket_heal_calls.lock().expect("bucket calls"), ["bucket-a"]); + assert!(storage.object_heal_opts.lock().expect("object calls").is_empty()); + } +} + +#[tokio::test] +async fn bucket_and_prefix_delete_recovery_stays_unresolved_after_ordinary_heal() { + for prefix in [None, Some("logs/")] { + let status = rustfs_madmin::heal_commands::DeleteRecoveryStatus { + pending: 1, + ..Default::default() + }; + let storage = Arc::new(MockStorage { + delete_recovery_results: Mutex::new(HashMap::from([( + "bucket-a".to_owned(), + Ok(Some(HealResultItem { + bucket: "bucket-a".to_owned(), + delete_recovery: Some(status), + ..Default::default() + })), + )])), + ..Default::default() + }); + let heal_type = prefix.map_or_else( + || HealType::Bucket { + bucket: "bucket-a".to_owned(), + }, + |prefix| HealType::Prefix { + bucket: "bucket-a".to_owned(), + prefix: prefix.to_owned(), + }, + ); + let task = HealTask::from_request( + HealRequest::new( + heal_type, + HealOptions { + recursive: true, + timeout: None, + ..Default::default() + }, + HealPriority::Normal, + ), + storage.clone(), + ); + + let result = match prefix { + Some(prefix) => task.heal_prefix("bucket-a", prefix).await, + None => task.heal_bucket("bucket-a").await, + }; + assert!( + matches!(result, Err(Error::TransientSkip { .. })), + "unfinished recovery must remain retryable: {result:?}" + ); + assert_eq!(storage.object_heal_opts.lock().expect("object calls").len(), 2); + assert!( + task.get_result_items() + .await + .iter() + .any(|item| item.delete_recovery == Some(status)) + ); + } +} + +#[tokio::test] +async fn unresolved_delete_recovery_prevents_bucket_removal() { + let storage = Arc::new(MockStorage { + delete_recovery_results: Mutex::new(HashMap::from([( + "bucket-a".to_owned(), + Ok(Some(HealResultItem { + delete_recovery: Some(rustfs_madmin::heal_commands::DeleteRecoveryStatus { + failed: 1, + ..Default::default() + }), + ..Default::default() + })), + )])), + ..Default::default() + }); + let task = HealTask::from_request( + HealRequest::new( + HealType::Bucket { + bucket: "bucket-a".to_owned(), + }, + HealOptions { + remove_corrupted: true, + timeout: None, + ..Default::default() + }, + HealPriority::Normal, + ), + storage.clone(), + ); + + assert!(matches!(task.heal_bucket("bucket-a").await, Err(Error::TransientSkip { .. }))); + assert!( + storage.bucket_heal_calls.lock().expect("bucket calls").is_empty(), + "publication directories must remain until recovery resolves" + ); +} + +#[tokio::test] +async fn admin_erasure_set_delete_recovery_rescans_only_after_settlement() { + let temp = TempDir::new().expect("temporary directory should be created"); + let disk = make_resume_disk(&temp).await; + let incarnation = Uuid::new_v4(); + let storage = Arc::new(MockStorage { + bucket_incarnation_id: Mutex::new(Some(incarnation)), + delete_recovery_results: Mutex::new(HashMap::from([( + "bucket-a".to_owned(), + Ok(Some(HealResultItem { + delete_recovery: Some(rustfs_madmin::heal_commands::DeleteRecoveryStatus { + blocked: 1, + ..Default::default() + }), + ..Default::default() + })), + )])), + resume_disk: Mutex::new(Some(disk.clone())), + disk_walk_items: Mutex::new(Some(vec![heal_item("object-a"), heal_item("object-b")])), + ..Default::default() + }); + let buckets = vec!["bucket-a".to_owned(), "bucket-b".to_owned()]; + let mut request = HealRequest::new( + HealType::ErasureSet { + buckets: buckets.clone(), + set_disk_id: "pool_0_set_0".to_owned(), + }, + HealOptions { + timeout: None, + ..Default::default() + }, + HealPriority::Normal, + ); + request.source = HealRequestSource::Admin; + let task = HealTask::from_request(request, storage.clone()); + + let error = task + .heal_erasure_set(buckets.clone(), "pool_0_set_0".to_owned()) + .await + .expect_err("administrator recovery cannot complete a set with unresolved recovery"); + assert!(matches!(error, Error::TransientSkip { .. }), "typed unresolved recovery: {error}"); + assert_eq!(storage.object_heal_opts.lock().expect("object calls").len(), 4); + let checkpoint = CheckpointManager::load_from_disk(disk.clone(), &task.id) + .await + .expect("retained administrator checkpoint") + .get_checkpoint() + .await; + assert_eq!( + checkpoint.current_bucket_index, 2, + "ordinary work advances independently from hidden recovery" + ); + assert!(!checkpoint.admin.expect("administrator outcome").completed); + assert_eq!(*storage.delete_recovery_calls.lock().expect("recovery calls"), buckets); + storage.delete_recovery_results.lock().expect("recovery results").insert( + "bucket-a".to_owned(), + Ok(Some(HealResultItem { + delete_recovery: Some(rustfs_madmin::heal_commands::DeleteRecoveryStatus { + failed: 1, + ..Default::default() + }), + ..Default::default() + })), + ); + let error = task + .heal_erasure_set(buckets.clone(), "pool_0_set_0".to_owned()) + .await + .expect_err("completed ordinary buckets must still retry their hidden recovery"); + assert!(matches!(error, Error::TransientSkip { .. })); + assert_eq!( + storage.object_heal_opts.lock().expect("object calls").len(), + 4, + "retry must not replay payload healing" + ); + + *storage.bucket_incarnation_id.lock().expect("bucket identity") = Some(Uuid::new_v4()); + let error = task + .heal_erasure_set(buckets.clone(), "pool_0_set_0".to_owned()) + .await + .expect_err("recovery-only retries must retain the admitted bucket generation"); + assert!(matches!(error, Error::StaleBucketIncarnation { .. })); + *storage.bucket_incarnation_id.lock().expect("bucket identity") = Some(incarnation); + + // A background worker may restore and collect the hidden transaction + // before this retry, so the recovery result can now be None. + *storage.disk_walk_items.lock().expect("disk-walk items") = + Some(vec![heal_item("object-a"), heal_item("object-b"), heal_item("restored")]); + let error = task + .heal_erasure_set(buckets.clone(), "pool_0_set_0".to_owned()) + .await + .expect_err("settlement must invalidate overlapping ordinary coverage"); + assert!(matches!(error, Error::TransientSkip { .. })); + assert_eq!(storage.object_heal_opts.lock().expect("object calls").len(), 4); + task.heal_erasure_set(buckets, "pool_0_set_0".to_owned()) + .await + .expect("a fresh coverage pass can complete"); + assert_eq!(storage.object_heal_opts.lock().expect("object calls").len(), 10); + assert_eq!( + storage + .healed_objects + .lock() + .expect("healed objects") + .iter() + .filter(|object| object.as_str() == "restored") + .count(), + 2 + ); + assert_eq!( + *storage.delete_recovery_calls.lock().expect("recovery calls"), + [ + "bucket-a", "bucket-b", "bucket-a", "bucket-b", "bucket-a", "bucket-b", "bucket-a", "bucket-b" + ] + ); + let checkpoint = CheckpointManager::load_from_disk(disk, &task.id) + .await + .expect("completed administrator receipt") + .get_checkpoint() + .await; + let admin = checkpoint.admin.expect("administrator outcome"); + assert!(admin.completed); + assert_eq!(admin.outcome.counters.processed, 6, "the fresh coverage pass owns its counters"); +} + #[tokio::test] async fn erasure_set_heal_applies_usage_baseline_to_progress() { let temp = TempDir::new().expect("temporary directory should be created"); diff --git a/crates/heal/src/lib.rs b/crates/heal/src/lib.rs index 8a4fdf8b5..d792aa877 100644 --- a/crates/heal/src/lib.rs +++ b/crates/heal/src/lib.rs @@ -398,6 +398,13 @@ mod tests { Ok(None) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> Result, Error> { + self.list_buckets().await + } + async fn list_buckets(&self) -> Result, Error> { Ok(Vec::new()) } diff --git a/crates/heal/tests/heal_bug_fixes_test.rs b/crates/heal/tests/heal_bug_fixes_test.rs index d2690b6be..832c71cf7 100644 --- a/crates/heal/tests/heal_bug_fixes_test.rs +++ b/crates/heal/tests/heal_bug_fixes_test.rs @@ -110,6 +110,13 @@ fn test_heal_task_status_atomic_update() { async fn get_bucket_info(&self, _bucket: &str) -> rustfs_heal::Result> { Ok(None) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> rustfs_heal::Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> rustfs_heal::Result> { Ok(vec![]) } @@ -205,6 +212,13 @@ async fn test_heal_task_transient_object_exists_skip_avoids_recreate() { Ok(None) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> rustfs_heal::Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> rustfs_heal::Result> { Ok(Vec::new()) } diff --git a/crates/heal/tests/heal_concurrent_delete_test.rs b/crates/heal/tests/heal_concurrent_delete_test.rs index 61657c4b7..c0808f7c8 100644 --- a/crates/heal/tests/heal_concurrent_delete_test.rs +++ b/crates/heal/tests/heal_concurrent_delete_test.rs @@ -79,6 +79,13 @@ impl HealStorageAPI for PausedTraversal { self.inner.get_bucket_info(bucket).await } + async fn list_buckets_for_admin_heal( + &self, + selector: Option, + ) -> Result> { + self.inner.list_buckets_for_admin_heal(selector).await + } + async fn list_buckets(&self) -> Result> { self.inner.list_buckets().await } @@ -95,6 +102,10 @@ impl HealStorageAPI for PausedTraversal { self.inner.admit_bucket_incarnation(bucket).await } + async fn heal_bucket_at_incarnation(&self, bucket: &str, expected: Uuid, opts: &HealOpts) -> Result { + self.inner.heal_bucket_at_incarnation(bucket, expected, opts).await + } + async fn heal_object_at_incarnation( &self, bucket: &str, @@ -563,9 +574,31 @@ fn admin_root_completes_after_unversioned_delete() { run_delete_case(DeleteCase::Unversioned); } -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[test] #[serial] -async fn admin_root_rejects_recreated_bucket_before_successor_repair() { +fn admin_root_rejects_recreated_bucket_before_successor_repair() { + // Incarnation-bound repairs use a spawned owner; retain the server's stack + // budget for the whole scenario. block_on alone would poll that future on + // libtest's smaller calling-thread stack. + let stack_size = 8 * 1024 * 1024; + std::thread::Builder::new() + .name("heal-concurrent-delete".to_owned()) + .stack_size(stack_size) + .spawn(move || { + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(4) + .thread_stack_size(stack_size) + .enable_all() + .build() + .expect("concurrent delete runtime"); + runtime.block_on(recreated_bucket_repair()); + }) + .expect("concurrent delete scenario thread starts") + .join() + .expect("concurrent delete scenario must not panic"); +} + +async fn recreated_bucket_repair() { let directory = tempfile::tempdir().expect("recreated bucket fixture"); let env = TestECStoreEnv::builder().base_dir(directory.path()).build().await; env.make_bucket(BUCKET, false).await; diff --git a/crates/heal/tests/heal_integration_test.rs b/crates/heal/tests/heal_integration_test.rs index 9559aa242..b9b93460c 100644 --- a/crates/heal/tests/heal_integration_test.rs +++ b/crates/heal/tests/heal_integration_test.rs @@ -17,10 +17,11 @@ use http::HeaderMap; use rustfs_heal::heal::{ manager::{HealConfig, HealManager}, + outcome::{HealExecutionOutcome, HealTraversalCoverage}, storage::{ECStoreHealStorage, HealObjectOptions as ObjectOptions, HealPutObjReader as PutObjReader, HealStorageAPI}, - task::{HealOptions, HealPriority, HealRequest, HealTaskStatus, HealType}, + task::{HealOptions, HealPriority, HealRequest, HealTask, HealTaskStatus, HealType}, }; -use rustfs_heal_contracts::heal_channel::{HealOpts, HealScanMode}; +use rustfs_heal_contracts::heal_channel::{HealOpts, HealRequestSource, HealScanMode}; use serial_test::serial; use std::{ path::{Path, PathBuf}, @@ -33,7 +34,7 @@ use walkdir::WalkDir; mod storage_api; -use storage_api::integration::{BucketOperations, ECStore, ObjectIO as _, ObjectOperations as _}; +use storage_api::integration::{BucketOperations, ECStore, ObjectIO as _, ObjectOperations as _, WriteCompletion}; const HEAL_FORMAT_WAIT_TIMEOUT: Duration = Duration::from_secs(25); const HEAL_FORMAT_WAIT_INTERVAL: Duration = Duration::from_millis(250); @@ -107,16 +108,183 @@ async fn create_test_bucket(ecstore: &Arc, bucket_name: &str) { async fn upload_test_object(ecstore: &Arc, bucket: &str, object: &str, data: &[u8]) { let mut reader = PutObjReader::from_vec(data.to_vec()); let object_info = (**ecstore) - .put_object(bucket, object, &mut reader, &ObjectOptions::default()) + .put_object( + bucket, + object, + &mut reader, + &ObjectOptions { + write_completion: WriteCompletion::TailDrained, + ..Default::default() + }, + ) .await .expect("Failed to upload test object"); info!("Uploaded test object: {}/{} ({} bytes)", bucket, object, object_info.size); } +async fn empty_replacement_with_committed_delete( + ecstore: &Arc, + disk_paths: &[PathBuf], + bucket: &str, + object: &str, +) -> (uuid::Uuid, uuid::Uuid) { + use rustfs_filemeta::FileInfo; + use storage_api::integration::{ + DeleteOptions, DeleteRecoveryAction, DeleteRecoveryState, DiskAPI as _, DiskSetSelector, StorageAdminApi as _, + }; + assert_eq!(disk_paths.len(), 4, "replacement fixture needs four paths"); + let incarnation = ecstore + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("admit the survivor bucket generation"); + let disks = ecstore + .disk_set_inventory(DiskSetSelector::new(0, 0)) + .await + .expect("read the live fixture set"); + assert_eq!(disks.len(), 4); + let transaction = uuid::Uuid::new_v4(); + // Retain a certified survivor decision without the normal delete's + // immediate recovery/cleanup consuming the fault input. + for disk in &disks[1..] { + let disk = disk.as_ref().expect("all three preparation survivors are online"); + disk.delete_version_recoverable( + bucket, + object, + FileInfo { + name: object.to_owned(), + ..Default::default() + }, + false, + DeleteOptions { + old_data_dir: Some(transaction), + delete_recovery_bucket_incarnation: Some(incarnation), + ..Default::default() + }, + ) + .await + .expect("prepare the same delete on every survivor"); + } + for disk in &disks[1..] { + let disk = disk.as_ref().expect("all three survivors are online"); + let prepared = disk + .inspect_delete_transaction(bucket, object, transaction) + .await + .expect("inspect prepared survivor decision"); + assert_eq!(prepared.state, DeleteRecoveryState::NewPrepared); + let committed = disk + .settle_delete_transaction(bucket, object, transaction, prepared.fingerprint, DeleteRecoveryAction::PublishCommit) + .await + .expect("publish the retained survivor decision"); + assert_eq!(committed.state, DeleteRecoveryState::QuorumCommitted); + } + // Linux disk I/O pins the root inode. Clearing its contents models the + // empty target on every platform; renaming it leaves Linux handles on + // the retired data and does not simulate a replacement for this test. + for entry in std::fs::read_dir(&disk_paths[0]).expect("read target root") { + let entry = entry.expect("read target entry"); + if entry.file_type().expect("target entry type").is_dir() { + std::fs::remove_dir_all(entry.path()).expect("clear target directory"); + } else { + std::fs::remove_file(entry.path()).expect("clear target file"); + } + } + (incarnation, transaction) +} + mod serial_tests { use super::*; + async fn bucket_directory_loss_task(cluster: bool, missing: usize, dry_run: bool, source: HealRequestSource) { + let temp = tempfile::tempdir().expect("isolated bucket recovery fixture"); + let env = rustfs_test_utils::TestECStoreEnv::builder() + .base_dir(temp.path()) + .build() + .await; + let bucket = "task-directory-recovery"; + env.make_bucket(bucket, false).await; + let storage = Arc::new(ECStoreHealStorage::new(env.ecstore.clone())); + let expected = storage + .admit_bucket_incarnation(bucket) + .await + .expect("committed bucket generation"); + for disk in env.disk_paths.iter().rev().take(missing) { + fs::remove_dir(disk.join(bucket)) + .await + .expect("remove only an empty bucket directory"); + } + assert_eq!(env.disk_paths.iter().filter(|disk| disk.join(bucket).exists()).count(), 4 - missing); + let mut request = HealRequest::new( + if cluster { + HealType::Cluster + } else { + HealType::Bucket { + bucket: bucket.to_owned(), + } + }, + HealOptions { + recursive: true, + dry_run, + recreate_missing: false, + timeout: Some(Duration::from_secs(30)), + ..Default::default() + }, + HealPriority::Normal, + ); + request.source = source; + if !cluster && source == HealRequestSource::Admin { + request.bucket_incarnation_id = Some(expected); + } + let task = HealTask::from_request(request, storage.clone()); + let result = task.execute().await; + if missing == 4 || source != HealRequestSource::Admin { + assert!( + result.is_err(), + "unproven or unbound recovery must fail: missing={missing}, source={source:?}" + ); + } else { + result.unwrap_or_else(|error| { + panic!( + "administrator task must reach generation-bound reconstruction: cluster={cluster}, missing={missing}: {error}" + ) + }); + let outcome = task.get_outcome().await; + assert_eq!(outcome.execution, HealExecutionOutcome::Completed); + assert_eq!(outcome.coverage, HealTraversalCoverage::Complete); + } + let remaining = if dry_run || missing == 4 || source != HealRequestSource::Admin { + 4 - missing + } else { + 4 + }; + assert_eq!(env.disk_paths.iter().filter(|disk| disk.join(bucket).is_dir()).count(), remaining); + assert_eq!(storage.admit_bucket_incarnation(bucket).await.expect("retained metadata"), expected); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + #[serial] + async fn test_admin_bucket_directory_recovery_from_task_entry() { + for cluster in [false, true] { + // Two surviving directories satisfy ordinary reads; one does not. + for missing in [2, 3] { + bucket_directory_loss_task(cluster, missing, false, HealRequestSource::Admin).await; + } + } + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + #[serial] + async fn test_admin_bucket_directory_recovery_dry_run_from_task_entry() { + bucket_directory_loss_task(false, 3, true, HealRequestSource::Admin).await; + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + #[serial] + async fn test_admin_bucket_directory_recovery_rejects_unproven_scope() { + bucket_directory_loss_task(false, 4, false, HealRequestSource::Admin).await; + bucket_directory_loss_task(false, 3, false, HealRequestSource::Internal).await; + } + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] #[serial] async fn test_heal_object_basic() { @@ -533,3 +701,383 @@ mod serial_tests { info!("Direct heal storage API test passed"); } } + +#[test] +#[serial] +fn historical_rebalance_topology_allows_format_and_object_rebuild() { + const STACK_SIZE: usize = 8 * 1024 * 1024; + std::thread::Builder::new() + .name("historical-rebalance-rebuild".to_owned()) + .stack_size(STACK_SIZE) + .spawn(|| { + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(4) + .thread_stack_size(STACK_SIZE) + .enable_all() + .build() + .expect("historical rebalance recovery runtime"); + runtime.block_on(historical_rebalance_topology_allows_format_and_object_rebuild_case()); + }) + .expect("historical rebalance recovery test thread") + .join() + .expect("historical rebalance recovery test result"); +} + +async fn historical_rebalance_topology_allows_format_and_object_rebuild_case() { + use storage_api::integration::{ + DiskAPI as _, DiskError, DiskOption, RUSTFS_META_BUCKET, RebalStatus, RebalanceMeta, RebalanceStats, new_disk, + }; + + let temp = tempfile::tempdir().expect("isolated historical rebalance fixture"); + let env = rustfs_test_utils::TestECStoreEnv::builder() + .base_dir(temp.path()) + .build() + .await; + let ecstore = env.ecstore; + let disks = env.disk_paths; + let storage = Arc::new(ECStoreHealStorage::new(ecstore.clone())); + assert_eq!(ecstore.pools.len(), 1); + let bucket = "historical-rebalance-rebuild"; + let object = "non-inline-object"; + let expected = non_inline_test_data(); + create_test_bucket(&ecstore, bucket).await; + upload_test_object(&ecstore, bucket, object, &expected).await; + wait_for_object_copies(&disks, bucket, object).await; + + // Model the persisted post-removal state. This test does not execute a + // rebalance worker, decommission lifecycle, or mount-identity handoff. + let mut historical = RebalanceMeta { + id: uuid::Uuid::new_v4().to_string(), + pool_stats: vec![RebalanceStats::default(), RebalanceStats::default()], + ..Default::default() + }; + for pool in &mut historical.pool_stats { + pool.participating = true; + pool.info.status = RebalStatus::Completed; + } + historical + .save_with_opts( + ecstore.pools[0].clone(), + ObjectOptions { + write_completion: WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("persist completed historical two-pool metadata on the current metadata pool"); + let historical_bytes = ecstore.pools[0] + .get_object_reader(RUSTFS_META_BUCKET, "rebalance.bin", None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("open persisted historical metadata") + .read_all() + .await + .expect("read persisted historical metadata"); + let old_format: serde_json::Value = serde_json::from_slice( + &fs::read(disks[0].join(RUSTFS_META_BUCKET).join("format.json")) + .await + .expect("read target format"), + ) + .expect("decode target format"); + let target_object = disks[0].join(bucket).join(object); + let old_shard = fs::read(find_part_file(&target_object).expect("original target shard")) + .await + .expect("read original target shard bytes"); + + // Preserve the root inode held by disk I/O, while replacing its contents + // with an empty target, as in the existing fresh-target recovery fixture. + for entry in std::fs::read_dir(&disks[0]).expect("read replacement target root") { + let entry = entry.expect("replacement target entry"); + if entry.file_type().expect("replacement entry type").is_dir() { + fs::remove_dir_all(entry.path()).await.expect("empty target directory"); + } else { + fs::remove_file(entry.path()).await.expect("empty target file"); + } + } + assert!(!disks[0].join(RUSTFS_META_BUCKET).join("format.json").exists()); + assert!(!target_object.exists()); + // A replacement reconnects with a fresh handle. Reusing the old handle + // would retain its cached format bytes despite the emptied directory. + let endpoint = ecstore.pools[0].endpoints.endpoints.as_ref()[0].clone(); + let replacement = new_disk( + &endpoint, + &DiskOption { + cleanup: false, + health_check: false, + }, + ) + .await + .expect("open a fresh replacement disk handle"); + assert!(matches!(replacement.get_disk_id().await, Err(DiskError::UnformattedDisk))); + ecstore.pools[0].get_disks(0).disks.write().await[0] = Some(replacement); + let target = endpoint.to_string(); + let (_, format_error) = storage + .heal_replacement_format(false, 0, 0, std::slice::from_ref(&target)) + .await + .expect("historical topology must admit target-scoped format repair"); + assert!(format_error.is_none(), "target format repair: {format_error:?}"); + let repaired_format: serde_json::Value = serde_json::from_slice( + &fs::read(disks[0].join(RUSTFS_META_BUCKET).join("format.json")) + .await + .expect("read repaired target format"), + ) + .expect("decode repaired target format"); + assert_eq!( + repaired_format["xl"]["this"], old_format["xl"]["this"], + "format repair must retain the original slot UUID" + ); + assert!( + !target_object.exists(), + "format repair alone must not be mistaken for object reconstruction" + ); + + let mut request = HealRequest::new( + HealType::ErasureSet { + buckets: vec![bucket.to_owned()], + set_disk_id: "pool_0_set_0".to_owned(), + }, + HealOptions { + recursive: true, + pool_index: Some(0), + set_index: Some(0), + scan_mode: HealScanMode::Deep, + ..Default::default() + }, + HealPriority::Normal, + ); + request.source = HealRequestSource::Admin; + let task = HealTask::from_request(request, storage); + task.execute() + .await + .expect("rebuild objects after historical-topology format repair"); + let outcome = task.get_outcome().await; + assert_eq!(outcome.execution, HealExecutionOutcome::Completed); + assert_eq!(outcome.coverage, HealTraversalCoverage::Complete); + // Verify target-side physical repair before a GET could perform read repair. + assert!(target_object.join("xl.meta").exists()); + let rebuilt_shard = fs::read(find_part_file(&target_object).expect("rebuilt target shard")) + .await + .expect("read rebuilt target shard bytes"); + assert_eq!(rebuilt_shard, old_shard, "the replacement shard must contain the original encoded bytes"); + let actual = ecstore + .get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("open rebuilt object") + .read_all() + .await + .expect("read rebuilt object bytes"); + assert_eq!(actual, expected, "reconstructed object bytes must be unchanged"); + let retained_history = ecstore.pools[0] + .get_object_reader(RUSTFS_META_BUCKET, "rebalance.bin", None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("open retained historical metadata") + .read_all() + .await + .expect("read retained historical metadata"); + assert_eq!(retained_history, historical_bytes, "healing must retain the historical rebalance payload"); +} + +#[test] +#[serial] +fn replacement_delete_recovery_on_fresh_target() { + // Match the debug server's worker stack for the composed Heal future. + // Libtest otherwise polls it on its smaller calling-thread stack. + const STACK_SIZE: usize = 8 * 1024 * 1024; + std::thread::Builder::new() + .name("replacement-delete-recovery".to_owned()) + .stack_size(STACK_SIZE) + .spawn(|| { + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(4) + .thread_stack_size(STACK_SIZE) + .enable_all() + .build() + .expect("replacement recovery runtime"); + runtime.block_on(replacement_delete_recovery_on_fresh_target_case()); + }) + .expect("replacement recovery test thread") + .join() + .expect("replacement recovery test result"); +} + +async fn replacement_delete_recovery_on_fresh_target_case() { + use rustfs_heal::Error; + use rustfs_heal::heal::task::HealTask; + use rustfs_heal_contracts::heal_channel::HealRequestSource; + use storage_api::integration::{ + DeleteRecoveryState, DiskAPI as _, DiskError, DiskSetSelector, EcstoreStorageError, StorageAdminApi as _, + }; + + for (source, erasure_set) in [ + (HealRequestSource::AutoHeal, true), + (HealRequestSource::Admin, true), + (HealRequestSource::Admin, false), + ] { + let (disk_paths, ecstore, storage) = heal_env().await; + let bucket = "replacement-delete-prepass"; + create_test_bucket(&ecstore, bucket).await; + let original = non_inline_test_data(); + for key in ["live", "deleted"] { + upload_test_object(&ecstore, bucket, key, &original).await; + wait_for_object_copies(&disk_paths, bucket, key).await; + } + let (incarnation, transaction) = empty_replacement_with_committed_delete(&ecstore, &disk_paths, bucket, "deleted").await; + let updated = vec![17; NON_INLINE_TEST_DATA_SIZE]; + upload_test_object(&ecstore, bucket, "live", &updated).await; + upload_test_object(&ecstore, bucket, "outage-created", &original).await; + let (_, error) = storage.heal_format(false).await.expect("format target"); + assert!(error.is_none(), "{error:?}"); + assert_eq!( + ecstore + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("survivor incarnation remains authoritative"), + incarnation + ); + let current_disks = ecstore + .disk_set_inventory(DiskSetSelector::new(0, 0)) + .await + .expect("read the formatted live set"); + assert!( + matches!( + current_disks[0] + .as_ref() + .expect("formatted target is attached") + .stat_volume(bucket) + .await, + Err(DiskError::VolumeNotFound) + ), + "the formatted target must still lack its publication bucket" + ); + for disk in ¤t_disks[1..] { + let retained = disk + .as_ref() + .expect("survivor remains attached") + .inspect_delete_transaction(bucket, "deleted", transaction) + .await + .expect("retained committed survivor proof"); + assert_eq!(retained.state, DeleteRecoveryState::QuorumCommitted); + } + // A committed survivor journal must be adopted, but publishing its + // witness cannot recreate a missing bucket outside lifecycle ownership. + let premature = storage + .recover_delete_transactions(bucket, None, None, &HealOpts::default()) + .await; + assert!( + matches!(&premature, Err(Error::Storage(EcstoreStorageError::FileNotFound))), + "{source:?} erasure_set={erasure_set}: {premature:?}" + ); + let heal_type = if erasure_set { + HealType::ErasureSet { + buckets: vec![bucket.to_owned()], + set_disk_id: "pool_0_set_0".to_owned(), + } + } else { + HealType::Bucket { + bucket: bucket.to_owned(), + } + }; + let mut request = HealRequest::new( + heal_type, + HealOptions { + recursive: true, + pool_index: Some(0), + set_index: Some(0), + scan_mode: HealScanMode::Deep, + ..Default::default() + }, + HealPriority::Normal, + ); + request.source = source; + if !erasure_set { + request.bucket_incarnation_id = Some(incarnation); + } + HealTask::from_request(request, storage.clone()) + .execute() + .await + .unwrap_or_else(|err| panic!("{source:?} erasure_set={erasure_set}: {err}")); + // Check physical repair before GET can hide missing shards via read repair. + for key in ["live", "outage-created"] { + assert!(disk_paths[0].join(bucket).join(key).join("xl.meta").exists()); + assert!(find_part_file(&disk_paths[0].join(bucket).join(key)).is_some()); + } + for (key, expected) in [("live", updated.as_slice()), ("outage-created", original.as_slice())] { + let mut reader = ecstore + .get_object_reader(bucket, key, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("open rebuilt object"); + assert_eq!(reader.read_all().await.expect("read rebuilt bytes"), expected); + } + assert!(matches!( + ecstore.get_object_info(bucket, "deleted", &ObjectOptions::default()).await, + Err(EcstoreStorageError::ObjectNotFound(_, _)) + )); + assert_eq!( + ecstore + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("retained incarnation"), + incarnation + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn bucket_removal_recovers_delete_proofs_before_removing_empty_volumes() { + use rustfs_heal::heal::task::HealTask; + use rustfs_heal_contracts::heal_channel::HealRequestSource; + use storage_api::integration::{DeleteRecoveryState, DiskAPI as _, DiskSetSelector, StorageAdminApi as _}; + let (disk_paths, ecstore, storage) = heal_env().await; + let bucket = "delete-proof-before-volume-removal"; + create_test_bucket(&ecstore, bucket).await; + upload_test_object(&ecstore, bucket, "deleted", &non_inline_test_data()).await; + wait_for_object_copies(&disk_paths, bucket, "deleted").await; + let (incarnation, transaction) = empty_replacement_with_committed_delete(&ecstore, &disk_paths, bucket, "deleted").await; + let (_, error) = storage.heal_format(false).await.expect("format target"); + assert!(error.is_none()); + storage + .heal_bucket( + bucket, + &HealOpts { + recreate: true, + ..Default::default() + }, + ) + .await + .expect("empty target volume"); + assert!(disk_paths[0].join(bucket).exists()); + let disks = ecstore + .disk_set_inventory(DiskSetSelector::new(0, 0)) + .await + .expect("live removal fixture set"); + for disk in &disks[1..] { + let retained = disk + .as_ref() + .expect("removal survivor remains attached") + .inspect_delete_transaction(bucket, "deleted", transaction) + .await + .expect("removal must still adopt the retained delete proof"); + assert_eq!(retained.state, DeleteRecoveryState::QuorumCommitted); + } + let mut request = HealRequest::new( + HealType::Bucket { + bucket: bucket.to_owned(), + }, + HealOptions { + recursive: false, + remove_corrupted: true, + ..Default::default() + }, + HealPriority::Normal, + ); + request.source = HealRequestSource::Admin; + request.bucket_incarnation_id = Some(incarnation); + HealTask::from_request(request, storage) + .execute() + .await + .expect("settle journals before deleting empty volumes"); + assert!( + disk_paths.iter().all(|path| !path.join(bucket).exists()), + "all empty replicas should be removed after proof settlement" + ); +} diff --git a/crates/heal/tests/storage_api.rs b/crates/heal/tests/storage_api.rs index b47737203..171ad216b 100644 --- a/crates/heal/tests/storage_api.rs +++ b/crates/heal/tests/storage_api.rs @@ -2,7 +2,7 @@ pub(crate) mod bug_fixes { pub(crate) use rustfs_ecstore::api::disk::DiskStore; pub(crate) use rustfs_ecstore::api::disk::endpoint::Endpoint; - pub(crate) use rustfs_storage_api::BucketInfo; + pub(crate) use rustfs_storage_api::{BucketInfo, DiskSetSelector}; } #[allow(unused_imports)] @@ -21,12 +21,14 @@ pub(crate) mod endpoint_index { #[allow(unused_imports)] pub(crate) mod integration { pub(crate) use rustfs_ecstore::api::disk::{ - DiskAPI, DiskError, DiskOption, DiskStore, Endpoint, RUSTFS_META_BUCKET, ReadOptions, new_disk, + DeleteOptions, DeleteRecoveryAction, DeleteRecoveryState, DiskAPI, DiskError, DiskOption, DiskStore, Endpoint, + RUSTFS_META_BUCKET, ReadOptions, new_disk, }; pub(crate) use rustfs_ecstore::api::error::StorageError as EcstoreStorageError; pub(crate) use rustfs_ecstore::api::object::{ ObjectOptions, PutObjReader, ShardIntegrityWriteMode, WriteCommitGuard, WriteCompletion, }; + pub(crate) use rustfs_ecstore::api::rebalance::{RebalStatus, RebalanceMeta, RebalanceStats}; pub(crate) use rustfs_ecstore::api::storage::ECStore; pub(crate) use rustfs_storage_api::BucketOperations; pub(crate) use rustfs_storage_api::MakeBucketOptions; diff --git a/crates/iam/Cargo.toml b/crates/iam/Cargo.toml index 7fb4f6b9f..dcfe70ce1 100644 --- a/crates/iam/Cargo.toml +++ b/crates/iam/Cargo.toml @@ -125,6 +125,7 @@ rustls-pki-types.workspace = true serial_test = { workspace = true } temp-env = { workspace = true, features = ["async_closure"] } tempfile = { workspace = true } +tracing-subscriber = { workspace = true, features = ["fmt"] } [lib] doctest = false diff --git a/crates/iam/src/federation/mapper.rs b/crates/iam/src/federation/mapper.rs index 1dca51e5c..6680a2471 100644 --- a/crates/iam/src/federation/mapper.rs +++ b/crates/iam/src/federation/mapper.rs @@ -12,12 +12,123 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::{FederatedAuthorization, VerifiedFederatedIdentity}; +use super::{FederatedClaims, VerifiedFederatedIdentity}; use crate::{EVENT_OIDC_DIAGNOSTICS, LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC}; use rustfs_policy::policy::{ClaimLookup, get_claim_case_insensitive}; +use rustfs_utils::HashAlgorithm; use std::{collections::HashMap, sync::Arc}; use tracing::debug; +#[cfg(any(test, feature = "test-util"))] +#[path = "test_support.rs"] +pub(super) mod test_support; + +/// Read-only authorization result produced by core federation mapping. +/// +/// The claims representation stays inside IAM: +/// +/// ```compile_fail +/// use rustfs_iam::federation::FederatedClaims; +/// ``` +/// +/// Callers cannot mutate mapped fields: +/// +/// ```compile_fail +/// use rustfs_iam::federation::FederatedAuthorization; +/// +/// fn mutate(authorization: &mut FederatedAuthorization) { +/// authorization.groups.clear(); +/// } +/// ``` +/// +/// Callers cannot construct authorization results: +/// +/// ```compile_fail +/// use rustfs_iam::federation::FederatedAuthorization; +/// +/// fn construct(authorization: FederatedAuthorization) -> FederatedAuthorization { +/// FederatedAuthorization { ..authorization } +/// } +/// ``` +#[derive(Debug, Clone)] +pub struct FederatedAuthorization { + provider_id: String, + claims: FederatedClaims, + policies: Vec, + group_claim_policies: Vec, + groups: Vec, + roles_claim_key: Option, + roles: Vec, +} + +impl FederatedAuthorization { + pub fn provider_id(&self) -> &str { + &self.provider_id + } + + pub fn subject(&self) -> &str { + &self.claims.sub + } + + pub fn email(&self) -> &str { + &self.claims.email + } + + pub fn username(&self) -> &str { + &self.claims.username + } + + pub fn session_identity(&self) -> String { + self.claims.session_identity() + } + + pub fn policies(&self) -> &[String] { + &self.policies + } + + /// Subset of `policies` derived only from the groups claim (including merged roles claim + /// values); these may be ignored when no policy of that name exists. Everything else in + /// `policies` must resolve. + pub fn group_claim_policies(&self) -> &[String] { + &self.group_claim_policies + } + + pub fn mapped_iam_groups(&self) -> &[String] { + &self.groups + } + + pub fn roles_claim_key(&self) -> Option<&str> { + self.roles_claim_key.as_deref() + } + + pub fn roles(&self) -> &[String] { + &self.roles + } + + pub fn has_authorization_context(&self) -> bool { + !self.policies.is_empty() || !self.groups.is_empty() + } + + pub fn oidc_virtual_parent(&self) -> Option { + let issuer = self.claims.raw.get("iss")?.as_str()?; + let subject = self.claims.sub.as_str(); + if issuer.is_empty() || subject.is_empty() { + return None; + } + + let subject_len = u64::try_from(subject.len()).ok()?; + let issuer_len = u64::try_from(issuer.len()).ok()?; + let mut source = Vec::with_capacity(23 + subject.len() + issuer.len()); + source.extend_from_slice(b"openid:"); + source.extend_from_slice(&subject_len.to_be_bytes()); + source.extend_from_slice(subject.as_bytes()); + source.extend_from_slice(&issuer_len.to_be_bytes()); + source.extend_from_slice(issuer.as_bytes()); + let digest = HashAlgorithm::SHA256.hash_encode(&source); + Some(base64_simd::URL_SAFE_NO_PAD.encode_to_string(digest.as_ref())) + } +} + #[derive(Clone)] pub(crate) struct FederatedAuthorizationRule { provider_id: String, @@ -308,8 +419,136 @@ fn claim_value_type(value: &serde_json::Value) -> &'static str { #[cfg(test)] mod tests { use super::*; - use crate::federation::{FederatedClaims, FederatedProviderRef, VerifiedFederatedIdentity}; - use serde_json::json; + use crate::federation::{FederatedAuthorizationFixture, FederatedClaims, FederatedProviderRef, VerifiedFederatedIdentity}; + use serde_json::{Value, json}; + + fn authorization(policies: Vec, groups: Vec) -> FederatedAuthorization { + FederatedAuthorizationFixture { + provider_id: "standard_oidc".to_string(), + sub: "subject".to_string(), + attributes: HashMap::from([("iss".to_string(), Value::String("https://idp.example.test".to_string()))]), + policies, + mapped_iam_groups: groups, + ..Default::default() + } + .into_authorization() + } + + #[test] + fn authorization_context_accepts_policy_or_group() { + assert!(!authorization(Vec::new(), Vec::new()).has_authorization_context()); + assert!(authorization(vec!["consoleAdmin".to_string()], Vec::new()).has_authorization_context()); + assert!(authorization(Vec::new(), vec!["RustFS.ConsoleAdmin".to_string()]).has_authorization_context()); + } + + #[test] + fn oidc_virtual_parent_is_issuer_scoped() { + let first = authorization(Vec::new(), Vec::new()); + let mut second = first.clone(); + second + .claims + .raw + .insert("iss".to_string(), Value::String("https://other-idp.example.test".to_string())); + + assert_eq!( + first.oidc_virtual_parent().as_deref(), + Some("HwDfWftzOy4jiuS3WjKytC_Sg_A2hKhrRAFtBDhoBr0") + ); + assert!(!rustfs_policy::auth::contains_reserved_chars( + first.oidc_virtual_parent().as_deref().expect("virtual parent") + )); + assert_ne!(first.oidc_virtual_parent(), second.oidc_virtual_parent()); + } + + #[test] + fn oidc_virtual_parent_length_delimits_identity_parts() { + let mut first = authorization(Vec::new(), Vec::new()); + first.claims.sub = "subject".to_string(); + first.claims.raw.insert( + "iss".to_string(), + Value::String("https://issuer.example/path:https://other.example".to_string()), + ); + let mut second = authorization(Vec::new(), Vec::new()); + second.claims.sub = "subject:https://issuer.example/path".to_string(); + second + .claims + .raw + .insert("iss".to_string(), Value::String("https://other.example".to_string())); + + assert_ne!(first.oidc_virtual_parent(), second.oidc_virtual_parent()); + } + + #[test] + fn oidc_virtual_parent_requires_verified_identity_parts() { + let mut missing_issuer = authorization(Vec::new(), Vec::new()); + missing_issuer.claims.raw.clear(); + let mut missing_subject = authorization(Vec::new(), Vec::new()); + missing_subject.claims.sub.clear(); + + assert!(missing_issuer.oidc_virtual_parent().is_none()); + assert!(missing_subject.oidc_virtual_parent().is_none()); + } + + #[test] + fn oidc_virtual_parent_preserves_exact_subject() { + let plain = authorization(Vec::new(), Vec::new()); + let mut padded = plain.clone(); + padded.claims.sub = format!(" {} ", plain.claims.sub); + + assert_ne!(plain.oidc_virtual_parent(), padded.oidc_virtual_parent()); + } + + #[test] + fn authorization_accessors_preserve_exact_values_and_distinct_groups() { + let attributes = HashMap::from([("custom".to_string(), json!(["value", null, 7]))]); + let authorization = FederatedAuthorizationFixture { + provider_id: " Corp Provider ".to_string(), + sub: " subject ".to_string(), + email: " email@example.test ".to_string(), + username: " username ".to_string(), + source_groups: vec!["different-source-group".to_string()], + attributes: attributes.clone(), + policies: vec![" policy-b ".to_string(), "policy-a".to_string(), " policy-b ".to_string()], + group_claim_policies: vec![" policy-b ".to_string(), " policy-b ".to_string()], + mapped_iam_groups: vec!["mapped-a".to_string(), "mapped-b".to_string()], + roles_claim_key: Some(" Roles ".to_string()), + roles: vec!["role-b".to_string(), " role-a ".to_string(), "role-b".to_string()], + } + .into_authorization(); + + assert_eq!(authorization.provider_id(), " Corp Provider "); + assert_eq!(authorization.subject(), " subject "); + assert_eq!(authorization.email(), " email@example.test "); + assert_eq!(authorization.username(), " username "); + assert_eq!(authorization.session_identity(), " username "); + assert_eq!(authorization.policies(), [" policy-b ", "policy-a", " policy-b "]); + assert_eq!(authorization.group_claim_policies(), [" policy-b ", " policy-b "]); + assert_eq!(authorization.claims.groups, ["different-source-group"]); + assert_eq!(authorization.claims.raw, attributes); + assert_eq!(authorization.mapped_iam_groups(), ["mapped-a", "mapped-b"]); + assert_eq!(authorization.roles_claim_key(), Some(" Roles ")); + assert_eq!(authorization.roles(), ["role-b", " role-a ", "role-b"]); + } + + #[test] + fn default_authorization_fixture_has_no_identity_or_mapping() { + let authorization = FederatedAuthorizationFixture::default().into_authorization(); + + assert_eq!(authorization.provider_id(), ""); + assert_eq!(authorization.subject(), ""); + assert_eq!(authorization.email(), ""); + assert_eq!(authorization.username(), ""); + assert_eq!(authorization.session_identity(), "oidc-user-unknown"); + assert!(authorization.claims.groups.is_empty()); + assert!(authorization.claims.raw.is_empty()); + assert!(authorization.policies().is_empty()); + assert!(authorization.group_claim_policies().is_empty()); + assert!(authorization.mapped_iam_groups().is_empty()); + assert!(authorization.roles_claim_key().is_none()); + assert!(authorization.roles().is_empty()); + assert!(!authorization.has_authorization_context()); + assert!(authorization.oidc_virtual_parent().is_none()); + } fn mapper(rule: FederatedAuthorizationRule) -> CoreFederatedAuthorizationMapper { CoreFederatedAuthorizationMapper::new(FederatedAuthorizationRules::new([rule])) diff --git a/crates/iam/src/federation/mod.rs b/crates/iam/src/federation/mod.rs index dc1d930f5..6b54f62e4 100644 --- a/crates/iam/src/federation/mod.rs +++ b/crates/iam/src/federation/mod.rs @@ -16,18 +16,21 @@ mod binding; mod error; mod mapper; mod model; -pub mod oidc; mod provider; mod transaction; pub use binding::FederatedSessionBinding; pub use error::{FederatedSessionBindingError, FederationError, Result}; -pub use mapper::{CoreFederatedAuthorizationMapper, FederatedAuthorizationRules}; -pub(crate) use mapper::{FederatedAuthorizationRule, FederatedAuthorizationRuleRef}; -pub(crate) use model::OpaqueLogoutContinuation; +pub(crate) use mapper::FederatedAuthorizationRule; +#[cfg(test)] +pub(crate) use mapper::FederatedAuthorizationRuleRef; +#[cfg(any(test, feature = "test-util"))] +pub use mapper::test_support::FederatedAuthorizationFixture; +pub use mapper::{CoreFederatedAuthorizationMapper, FederatedAuthorization, FederatedAuthorizationRules}; +pub(crate) use model::{FederatedClaims, OpaqueLogoutContinuation}; pub use model::{ - FederatedAuthorization, FederatedClaims, FederatedLoginSession, FederatedProviderRef, FederatedSession, - FederatedSessionTransaction, OIDC_VIRTUAL_PARENT_CLAIM, VerifiedFederatedIdentity, + FederatedLoginSession, FederatedProviderRef, FederatedSession, FederatedSessionTransaction, OIDC_VIRTUAL_PARENT_CLAIM, + VerifiedFederatedIdentity, }; pub(crate) use provider::{FederatedProviderQuery, StandardOidcAuthentication, VerifiedFederatedCodeExchange}; pub use provider::{FederatedProviderView, FederatedRedirectPolicy}; diff --git a/crates/iam/src/federation/model.rs b/crates/iam/src/federation/model.rs index 76b8a9816..54b6725b9 100644 --- a/crates/iam/src/federation/model.rs +++ b/crates/iam/src/federation/model.rs @@ -13,23 +13,24 @@ // limitations under the License. use rustfs_credentials::Credentials; -use rustfs_utils::HashAlgorithm; use serde_json::Value; use std::{collections::HashMap, fmt}; +use super::FederatedAuthorization; + pub const OIDC_VIRTUAL_PARENT_CLAIM: &str = "x-rustfs-internal-oidc-parent"; #[derive(Debug, Clone)] -pub struct FederatedClaims { - pub sub: String, - pub email: String, - pub username: String, - pub groups: Vec, - pub raw: HashMap, +pub(crate) struct FederatedClaims { + pub(crate) sub: String, + pub(crate) email: String, + pub(crate) username: String, + pub(crate) groups: Vec, + pub(crate) raw: HashMap, } impl FederatedClaims { - pub fn session_identity(&self) -> String { + pub(crate) fn session_identity(&self) -> String { if !self.username.is_empty() { self.username.clone() } else if !self.email.is_empty() { @@ -125,45 +126,6 @@ impl VerifiedFederatedIdentity { } } -#[derive(Debug, Clone)] -pub struct FederatedAuthorization { - pub provider_id: String, - pub claims: FederatedClaims, - pub policies: Vec, - /// Subset of `policies` derived only from the groups claim (including merged roles claim - /// values); these may be ignored when no policy of that name exists. Everything else in - /// `policies` must resolve. - pub group_claim_policies: Vec, - pub groups: Vec, - pub roles_claim_key: Option, - pub roles: Vec, -} - -impl FederatedAuthorization { - pub fn has_authorization_context(&self) -> bool { - !self.policies.is_empty() || !self.groups.is_empty() - } - - pub fn oidc_virtual_parent(&self) -> Option { - let issuer = self.claims.raw.get("iss")?.as_str()?; - let subject = self.claims.sub.as_str(); - if issuer.is_empty() || subject.is_empty() { - return None; - } - - let subject_len = u64::try_from(subject.len()).ok()?; - let issuer_len = u64::try_from(issuer.len()).ok()?; - let mut source = Vec::with_capacity(23 + subject.len() + issuer.len()); - source.extend_from_slice(b"openid:"); - source.extend_from_slice(&subject_len.to_be_bytes()); - source.extend_from_slice(subject.as_bytes()); - source.extend_from_slice(&issuer_len.to_be_bytes()); - source.extend_from_slice(issuer.as_bytes()); - let digest = HashAlgorithm::SHA256.hash_encode(&source); - Some(base64_simd::URL_SAFE_NO_PAD.encode_to_string(digest.as_ref())) - } -} - /// Carries the provider-bound ID token between code exchange and logout token creation. pub(crate) struct OpaqueLogoutContinuation { provider: FederatedProviderRef, @@ -224,21 +186,6 @@ mod tests { } } - fn authorization(policies: Vec, groups: Vec) -> FederatedAuthorization { - FederatedAuthorization { - provider_id: "standard_oidc".to_string(), - claims: FederatedClaims { - raw: HashMap::from([("iss".to_string(), Value::String("https://idp.example.test".to_string()))]), - ..claims("", "", "subject") - }, - policies, - group_claim_policies: Vec::new(), - groups, - roles_claim_key: None, - roles: Vec::new(), - } - } - #[test] fn session_identity_preserves_existing_fallback_order() { assert_eq!(claims("john", "john@example.com", "sub-1").session_identity(), "john"); @@ -247,70 +194,6 @@ mod tests { assert_eq!(claims("", "", "").session_identity(), "oidc-user-unknown"); } - #[test] - fn authorization_context_accepts_policy_or_group() { - assert!(!authorization(Vec::new(), Vec::new()).has_authorization_context()); - assert!(authorization(vec!["consoleAdmin".to_string()], Vec::new()).has_authorization_context()); - assert!(authorization(Vec::new(), vec!["RustFS.ConsoleAdmin".to_string()]).has_authorization_context()); - } - - #[test] - fn oidc_virtual_parent_is_issuer_scoped() { - let first = authorization(Vec::new(), Vec::new()); - let mut second = first.clone(); - second - .claims - .raw - .insert("iss".to_string(), Value::String("https://other-idp.example.test".to_string())); - - assert_eq!( - first.oidc_virtual_parent().as_deref(), - Some("HwDfWftzOy4jiuS3WjKytC_Sg_A2hKhrRAFtBDhoBr0") - ); - assert!(!rustfs_policy::auth::contains_reserved_chars( - first.oidc_virtual_parent().as_deref().expect("virtual parent") - )); - assert_ne!(first.oidc_virtual_parent(), second.oidc_virtual_parent()); - } - - #[test] - fn oidc_virtual_parent_length_delimits_identity_parts() { - let mut first = authorization(Vec::new(), Vec::new()); - first.claims.sub = "subject".to_string(); - first.claims.raw.insert( - "iss".to_string(), - Value::String("https://issuer.example/path:https://other.example".to_string()), - ); - let mut second = authorization(Vec::new(), Vec::new()); - second.claims.sub = "subject:https://issuer.example/path".to_string(); - second - .claims - .raw - .insert("iss".to_string(), Value::String("https://other.example".to_string())); - - assert_ne!(first.oidc_virtual_parent(), second.oidc_virtual_parent()); - } - - #[test] - fn oidc_virtual_parent_requires_verified_identity_parts() { - let mut missing_issuer = authorization(Vec::new(), Vec::new()); - missing_issuer.claims.raw.clear(); - let mut missing_subject = authorization(Vec::new(), Vec::new()); - missing_subject.claims.sub.clear(); - - assert!(missing_issuer.oidc_virtual_parent().is_none()); - assert!(missing_subject.oidc_virtual_parent().is_none()); - } - - #[test] - fn oidc_virtual_parent_preserves_exact_subject() { - let plain = authorization(Vec::new(), Vec::new()); - let mut padded = plain.clone(); - padded.claims.sub = format!(" {} ", plain.claims.sub); - - assert_ne!(plain.oidc_virtual_parent(), padded.oidc_virtual_parent()); - } - #[test] fn verified_identity_preserves_exact_values_and_distinct_groups() { let attributes = HashMap::from([ @@ -341,24 +224,6 @@ mod tests { assert!(!debug.contains(" username ")); assert!(!debug.contains(" source-b ")); assert!(!debug.contains("identity-attribute-poison")); - - let authorization = FederatedAuthorization { - provider_id: " Corp Provider ".to_string(), - claims: FederatedClaims { - sub: "subject".to_string(), - email: String::new(), - username: String::new(), - groups: vec!["different-source-group".to_string()], - raw: HashMap::new(), - }, - policies: Vec::new(), - group_claim_policies: Vec::new(), - groups: vec!["mapped-a".to_string(), "mapped-b".to_string()], - roles_claim_key: None, - roles: Vec::new(), - }; - assert_eq!(authorization.claims.groups, ["different-source-group"]); - assert_eq!(authorization.groups, ["mapped-a", "mapped-b"]); } #[test] diff --git a/crates/iam/src/federation/oidc/claims.rs b/crates/iam/src/federation/oidc/claims.rs deleted file mode 100644 index 210bef001..000000000 --- a/crates/iam/src/federation/oidc/claims.rs +++ /dev/null @@ -1,68 +0,0 @@ -// Copyright 2024 RustFS Team -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -use crate::{ - federation::{FederatedClaims, FederatedProviderRef, VerifiedFederatedIdentity}, - oidc::OidcClaims, -}; - -pub(super) fn verified_identity(provider: FederatedProviderRef, claims: OidcClaims) -> VerifiedFederatedIdentity { - VerifiedFederatedIdentity::from_claims( - provider, - FederatedClaims { - sub: claims.sub, - email: claims.email, - username: claims.username, - groups: claims.groups, - raw: claims.raw, - }, - ) -} - -#[cfg(test)] -mod tests { - use super::*; - use serde_json::json; - use std::collections::HashMap; - - #[test] - fn verified_identity_preserves_provider_and_claim_values() { - let raw = HashMap::from([ - ("iss".to_string(), json!("https://corp.example.test")), - ("department".to_string(), json!("engineering")), - ("roles".to_string(), json!(["reader", "admin"])), - ]); - let identity = verified_identity( - FederatedProviderRef::new("corp".to_string()), - OidcClaims { - sub: " subject-123 ".to_string(), - email: " user@example.test ".to_string(), - username: " user ".to_string(), - groups: vec![ - "source-ops".to_string(), - "source-developers".to_string(), - "source-ops".to_string(), - ], - raw: raw.clone(), - }, - ); - - assert_eq!(identity.provider().as_str(), "corp"); - assert_eq!(identity.subject(), " subject-123 "); - assert_eq!(identity.email(), " user@example.test "); - assert_eq!(identity.username(), " user "); - assert_eq!(identity.source_groups(), ["source-ops", "source-developers", "source-ops"]); - assert_eq!(identity.attributes(), &raw); - } -} diff --git a/crates/iam/src/federation/oidc/config.rs b/crates/iam/src/federation/oidc/config.rs deleted file mode 100644 index f2374c251..000000000 --- a/crates/iam/src/federation/oidc/config.rs +++ /dev/null @@ -1,68 +0,0 @@ -// Copyright 2024 RustFS Team -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -use crate::{ - federation::{FederatedAuthorizationRule, FederatedAuthorizationRules, FederatedRedirectPolicy}, - oidc::{OidcConfigSnapshot, OidcSys}, -}; - -pub(crate) fn authorization_rules(oidc: &OidcSys) -> FederatedAuthorizationRules { - FederatedAuthorizationRules::new(oidc.provider_configs().map(|config| { - FederatedAuthorizationRule::new( - config.id.clone(), - config.claim_name.clone(), - config.claim_prefix.clone(), - config.role_policy.clone(), - config.groups_claim.clone(), - config.roles_claim.clone(), - ) - })) -} - -/// Read-only access to the active standard OIDC configuration. -/// -/// This interface stays separate from authentication and generic provider -/// views because its snapshots include OIDC-specific fields and secrets. -pub trait OidcConfigQuery: Send + Sync { - fn config_snapshot(&self) -> OidcConfigSnapshot; -} - -pub(super) fn redirect_policy(oidc: &OidcSys, provider_id: &str) -> Option { - oidc.get_provider_config(provider_id).map(|config| FederatedRedirectPolicy { - redirect_uri: config.redirect_uri.clone(), - allow_request_origin: config.redirect_uri_dynamic, - }) -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn redirect_policy_keeps_configured_uri_and_dynamic_setting() { - let mut config = crate::oidc::test_config("default"); - config.redirect_uri = Some("https://console.example.test/callback".to_string()); - config.redirect_uri_dynamic = false; - let oidc = crate::oidc::make_test_sys(vec![config]); - - assert_eq!( - redirect_policy(&oidc, "default"), - Some(FederatedRedirectPolicy { - redirect_uri: Some("https://console.example.test/callback".to_string()), - allow_request_origin: false, - }) - ); - assert_eq!(redirect_policy(&oidc, "missing"), None); - } -} diff --git a/crates/iam/src/federation/oidc/discovery.rs b/crates/iam/src/federation/oidc/discovery.rs deleted file mode 100644 index dd50fe3ac..000000000 --- a/crates/iam/src/federation/oidc/discovery.rs +++ /dev/null @@ -1,63 +0,0 @@ -// Copyright 2024 RustFS Team -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -use crate::{federation::FederatedProviderView, oidc::OidcSys}; - -fn provider_view(provider: &crate::oidc::OidcProviderConfig) -> FederatedProviderView { - FederatedProviderView { - provider_id: provider.id.clone(), - display_name: provider.display_name.clone(), - } -} - -pub(super) fn list_providers(oidc: &OidcSys) -> Vec { - oidc.provider_configs().map(provider_view).collect() -} - -pub(super) fn list_visible_providers(oidc: &OidcSys) -> Vec { - oidc.provider_configs() - .filter(|provider| !provider.hide_from_ui) - .map(provider_view) - .collect() -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn provider_views_preserve_names_and_visibility() { - let mut visible = crate::oidc::test_config("default"); - visible.display_name = "Company SSO".to_string(); - let mut hidden = crate::oidc::test_config("workload"); - hidden.hide_from_ui = true; - let oidc = crate::oidc::make_test_sys(vec![visible, hidden]); - - let all = list_providers(&oidc); - assert_eq!(all.len(), 2); - assert!( - all.iter() - .any(|provider| { provider.provider_id == "default" && provider.display_name == "Company SSO" }) - ); - assert!(all.iter().any(|provider| provider.provider_id == "workload")); - - assert_eq!( - list_visible_providers(&oidc), - vec![FederatedProviderView { - provider_id: "default".to_string(), - display_name: "Company SSO".to_string(), - }] - ); - } -} diff --git a/crates/iam/src/federation/oidc/flow.rs b/crates/iam/src/federation/oidc/flow.rs deleted file mode 100644 index e338590e7..000000000 --- a/crates/iam/src/federation/oidc/flow.rs +++ /dev/null @@ -1,109 +0,0 @@ -// Copyright 2024 RustFS Team -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -use super::config::OidcConfigQuery; -use super::{claims, config, discovery, http}; -use crate::{ - federation::{ - CoreFederatedAuthorizationMapper, FederatedAuthorizationRules, FederatedIdentityService, FederatedProviderQuery, - FederatedProviderRef, FederatedProviderView, FederatedRedirectPolicy, FederationError, OpaqueLogoutContinuation, Result, - StandardOidcAuthentication, VerifiedFederatedCodeExchange, - }, - oidc::{OidcConfigSnapshot, OidcSys}, -}; -use std::sync::Arc; - -pub struct StandardOidcAdapter { - oidc: Arc, - authorization_rules: FederatedAuthorizationRules, -} - -impl StandardOidcAdapter { - pub fn new(oidc: Arc) -> Self { - let authorization_rules = config::authorization_rules(&oidc); - Self { - oidc, - authorization_rules, - } - } - - pub fn authorization_rules(&self) -> FederatedAuthorizationRules { - self.authorization_rules.clone() - } - - pub fn into_service(self: Arc, mapper: CoreFederatedAuthorizationMapper) -> FederatedIdentityService { - let authentication: Arc = self.clone(); - let provider_query: Arc = self; - FederatedIdentityService::from_standard_oidc_parts(provider_query, authentication, mapper) - } -} - -impl OidcConfigQuery for StandardOidcAdapter { - fn config_snapshot(&self) -> OidcConfigSnapshot { - self.oidc.config_snapshot() - } -} - -#[async_trait::async_trait] -impl StandardOidcAuthentication for StandardOidcAdapter { - async fn authorize_url(&self, provider_id: &str, redirect_uri: &str, redirect_after: Option) -> Result { - http::authorize_url(&self.oidc, provider_id, redirect_uri, redirect_after).await - } - - async fn exchange_identity(&self, state: &str, code: &str, redirect_uri: &str) -> Result { - let (oidc_claims, provider_id, session, id_token) = http::exchange_code(&self.oidc, state, code, redirect_uri).await?; - let provider = FederatedProviderRef::new(provider_id); - Ok(VerifiedFederatedCodeExchange::new( - claims::verified_identity(provider.clone(), oidc_claims), - session.redirect_after, - OpaqueLogoutContinuation::new(provider, id_token), - )) - } - - async fn verify_identity(&self, jwt: &str) -> Result { - let (oidc_claims, provider_id) = self - .oidc - .verify_web_identity_token(jwt) - .await - .map_err(FederationError::TokenVerification)?; - Ok(claims::verified_identity(FederatedProviderRef::new(provider_id), oidc_claims)) - } - - async fn create_logout_token(&self, continuation: OpaqueLogoutContinuation) -> Result { - let (provider, id_token) = continuation.into_parts(); - http::create_logout_token(&self.oidc, provider.as_str(), &id_token).await - } - - async fn build_logout_url(&self, logout_token: &str, post_logout_redirect_uri: &str) -> Result> { - http::build_logout_url(&self.oidc, logout_token, post_logout_redirect_uri).await - } -} - -impl FederatedProviderQuery for StandardOidcAdapter { - fn has_providers(&self) -> bool { - self.oidc.has_providers() - } - - fn list_providers(&self) -> Vec { - discovery::list_providers(&self.oidc) - } - - fn list_visible_providers(&self) -> Vec { - discovery::list_visible_providers(&self.oidc) - } - - fn redirect_policy(&self, provider_id: &str) -> Option { - config::redirect_policy(&self.oidc, provider_id) - } -} diff --git a/crates/iam/src/federation/oidc/http.rs b/crates/iam/src/federation/oidc/http.rs deleted file mode 100644 index 3b4f0dcd0..000000000 --- a/crates/iam/src/federation/oidc/http.rs +++ /dev/null @@ -1,57 +0,0 @@ -// Copyright 2024 RustFS Team -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -use crate::{ - federation::{FederationError, Result}, - oidc::{OidcClaims, OidcSys}, - oidc_state::OidcAuthSession, -}; - -pub(super) async fn authorize_url( - oidc: &OidcSys, - provider_id: &str, - redirect_uri: &str, - redirect_after: Option, -) -> Result { - oidc.authorize_url(provider_id, redirect_uri, redirect_after) - .await - .map_err(FederationError::Authorization) -} - -pub(super) async fn exchange_code( - oidc: &OidcSys, - state: &str, - code: &str, - redirect_uri: &str, -) -> Result<(OidcClaims, String, OidcAuthSession, String)> { - oidc.exchange_code(state, code, redirect_uri) - .await - .map_err(FederationError::CodeExchange) -} - -pub(super) async fn create_logout_token(oidc: &OidcSys, provider_id: &str, id_token: &str) -> Result { - oidc.create_logout_token(provider_id, id_token) - .await - .map_err(FederationError::Logout) -} - -pub(super) async fn build_logout_url( - oidc: &OidcSys, - logout_token: &str, - post_logout_redirect_uri: &str, -) -> Result> { - oidc.build_logout_url(logout_token, post_logout_redirect_uri) - .await - .map_err(FederationError::Logout) -} diff --git a/crates/iam/src/federation/oidc/mod.rs b/crates/iam/src/federation/oidc/mod.rs deleted file mode 100644 index 52e6b6432..000000000 --- a/crates/iam/src/federation/oidc/mod.rs +++ /dev/null @@ -1,22 +0,0 @@ -// Copyright 2024 RustFS Team -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -mod claims; -mod config; -mod discovery; -mod flow; -mod http; - -pub use config::OidcConfigQuery; -pub use flow::StandardOidcAdapter; diff --git a/crates/iam/src/federation/test_support.rs b/crates/iam/src/federation/test_support.rs new file mode 100644 index 000000000..3933ae910 --- /dev/null +++ b/crates/iam/src/federation/test_support.rs @@ -0,0 +1,54 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::{FederatedAuthorization, FederatedClaims}; +use serde_json::Value; +use std::collections::HashMap; + +/// Exact authorization inputs for tests that exercise the session binding boundary. +#[derive(Default)] +pub struct FederatedAuthorizationFixture { + pub provider_id: String, + pub sub: String, + pub email: String, + pub username: String, + pub source_groups: Vec, + pub attributes: HashMap, + pub policies: Vec, + pub group_claim_policies: Vec, + pub mapped_iam_groups: Vec, + pub roles_claim_key: Option, + pub roles: Vec, +} + +impl FederatedAuthorizationFixture { + /// Preserves supplied values without authentication, validation, or policy mapping. + pub fn into_authorization(self) -> FederatedAuthorization { + FederatedAuthorization { + provider_id: self.provider_id, + claims: FederatedClaims { + sub: self.sub, + email: self.email, + username: self.username, + groups: self.source_groups, + raw: self.attributes, + }, + policies: self.policies, + group_claim_policies: self.group_claim_policies, + groups: self.mapped_iam_groups, + roles_claim_key: self.roles_claim_key, + roles: self.roles, + } + } +} diff --git a/crates/iam/src/federation/transaction.rs b/crates/iam/src/federation/transaction.rs index b9bda054d..84f4c2969 100644 --- a/crates/iam/src/federation/transaction.rs +++ b/crates/iam/src/federation/transaction.rs @@ -17,6 +17,7 @@ use super::{ FederatedProviderView, FederatedRedirectPolicy, FederatedSession, FederatedSessionBinding, FederatedSessionTransaction, FederationError, OpaqueLogoutContinuation, Result, StandardOidcAuthentication, }; +use crate::{EVENT_OIDC_DIAGNOSTICS, LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC}; use std::sync::Arc; const DEFAULT_OIDC_PROVIDER_ID: &str = "default"; @@ -61,7 +62,7 @@ impl FederatedIdentityService { self.authentication.create_logout_token(continuation).await } - pub(super) fn from_standard_oidc_parts( + pub(crate) fn from_standard_oidc_parts( provider_query: Arc, authentication: Arc, mapper: CoreFederatedAuthorizationMapper, @@ -133,22 +134,28 @@ impl FederatedIdentityService { let authorization = self.verify_identity(jwt).await?; if !authorization.has_authorization_context() { tracing::warn!( - provider_id = %authorization.provider_id, - username = %authorization.claims.username, - sub = %authorization.claims.sub, - policy_count = authorization.policies.len(), - group_count = authorization.groups.len(), + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), + username = %authorization.username(), + sub = %authorization.subject(), + policy_count = authorization.policies().len(), + group_count = authorization.mapped_iam_groups().len(), "AssumeRoleWithWebIdentity has no mapped policies or groups" ); return Err(FederationError::NoAuthorizationContext); } tracing::debug!( - provider_id = %authorization.provider_id, - username = %authorization.claims.username, - policy_count = authorization.policies.len(), - group_count = authorization.groups.len(), - policies = ?authorization.policies, - groups = ?authorization.groups, + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), + username = %authorization.username(), + policy_count = authorization.policies().len(), + group_count = authorization.mapped_iam_groups().len(), + policies = ?authorization.policies(), + groups = ?authorization.mapped_iam_groups(), "AssumeRoleWithWebIdentity mapped OIDC policies and groups" ); @@ -379,19 +386,19 @@ mod tests { ) -> core::result::Result { self.events.lock().expect("event log should not be poisoned").push("bind"); self.transactions.lock().expect("transactions should not be poisoned").push(( - transaction.authorization.provider_id.clone(), + transaction.authorization.provider_id().to_string(), transaction.duration_seconds, transaction.session_policy.clone(), )); self.mapped_policies .lock() .expect("mapped policies should not be poisoned") - .push(transaction.authorization.policies.clone()); + .push(transaction.authorization.policies().to_vec()); if self.fail { return Err(FederatedSessionBindingError::Internal("binding failed".to_string())); } Ok(Credentials { - access_key: transaction.authorization.claims.session_identity(), + access_key: transaction.authorization.session_identity(), ..Default::default() }) } @@ -487,7 +494,7 @@ mod tests { .await .expect("callback flow should complete"); assert_eq!(login.session.credentials.access_key, "user"); - assert_eq!(login.session.authorization.provider_id, "corp"); + assert_eq!(login.session.authorization.provider_id(), "corp"); assert_eq!(login.redirect_after.as_deref(), Some("/browser")); assert_eq!(login.logout_token, "logout-token"); assert_eq!( @@ -501,7 +508,7 @@ mod tests { .await .expect("web identity flow should complete"); assert_eq!(web_identity.credentials.access_key, "user"); - assert_eq!(web_identity.authorization.provider_id, "partner"); + assert_eq!(web_identity.authorization.provider_id(), "partner"); assert_eq!(events.lock().expect("event log should not be poisoned").as_slice(), ["verify", "bind"]); assert_eq!( binding @@ -530,8 +537,8 @@ mod tests { .complete_authorization_code("state", "code", "https://console.example/callback", 3600, binding.as_ref()) .await .expect("standard callback flow should complete"); - assert_eq!(login.session.authorization.provider_id, "corp"); - assert_eq!(login.session.authorization.policies, ["readwrite"]); + assert_eq!(login.session.authorization.provider_id(), "corp"); + assert_eq!(login.session.authorization.policies(), ["readwrite"]); assert_eq!( events.lock().expect("event log should not be poisoned").as_slice(), ["exchange", "bind", "logout"] @@ -542,8 +549,8 @@ mod tests { .assume_role_with_web_identity("jwt", 7200, Some("session-policy".to_string()), binding.as_ref()) .await .expect("standard web identity flow should complete"); - assert_eq!(web_identity.authorization.provider_id, "partner"); - assert_eq!(web_identity.authorization.policies, ["readwrite"]); + assert_eq!(web_identity.authorization.provider_id(), "partner"); + assert_eq!(web_identity.authorization.policies(), ["readwrite"]); assert_eq!(events.lock().expect("event log should not be poisoned").as_slice(), ["verify", "bind"]); assert_eq!( binding @@ -586,8 +593,8 @@ mod tests { .await .expect("a mapped group is an authorization context"); - assert_eq!(session.authorization.policies, ["developers"]); - assert_eq!(session.authorization.groups, ["developers"]); + assert_eq!(session.authorization.policies(), ["developers"]); + assert_eq!(session.authorization.mapped_iam_groups(), ["developers"]); assert_eq!(events.lock().expect("event log should not be poisoned").as_slice(), ["verify", "bind"]); } diff --git a/crates/iam/src/lib.rs b/crates/iam/src/lib.rs index 6b10bad0f..6d7e2d610 100644 --- a/crates/iam/src/lib.rs +++ b/crates/iam/src/lib.rs @@ -35,7 +35,6 @@ pub mod keyring; pub mod manager; pub mod mfa; pub mod oidc; -pub mod oidc_state; mod root_credentials; mod runtime_sources; mod server_config; @@ -195,7 +194,6 @@ pub(crate) async fn notify_iam_load_policy_mapping( } static IAM_SYS: OnceLock>> = OnceLock::new(); -static OIDC_SYS: OnceLock> = OnceLock::new(); /// Build an IAM system bound to the given store without touching the process /// singleton (backlog#1052 S3): a per-server context can own the returned @@ -285,36 +283,11 @@ pub fn get_global_iam_sys() -> Option>> { IAM_SYS.get().cloned() } -/// Initialize the global OIDC system. Non-fatal if no OIDC providers are configured. -pub async fn init_oidc_sys() -> Result<()> { - init_oidc_sys_with_extra_root_ca(None).await -} - -/// Initialize the global OIDC system with an additional outbound root CA bundle. -pub async fn init_oidc_sys_with_extra_root_ca(root_ca_pem: Option<&[u8]>) -> Result<()> { - init_oidc_sys_with_extra_root_ca_provider_inner(None, root_ca_pem).await -} - -/// Initialize the global OIDC system with a reload-aware outbound root CA provider. -pub async fn init_oidc_sys_with_extra_root_ca_provider(extra_root_ca_provider: OidcExtraRootCaProvider) -> Result<()> { - init_oidc_sys_with_extra_root_ca_provider_inner(Some(extra_root_ca_provider), None).await -} - -async fn init_oidc_sys_with_extra_root_ca_provider_inner( - extra_root_ca_provider: Option, - root_ca_pem: Option<&[u8]>, -) -> Result<()> { - if OIDC_SYS.get().is_some() { - debug!( - event = EVENT_OIDC_STATE, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - state = "already_initialized", - "OIDC runtime already initialized" - ); - return Ok(()); - } - +/// Build the OIDC system with reload-aware outbound root CA material. +/// +/// Invalid OIDC setup remains non-fatal: the returned system has no providers, +/// matching the server startup behavior when OIDC cannot be initialized. +pub async fn build_oidc_sys_with_extra_root_ca_provider(extra_root_ca_provider: OidcExtraRootCaProvider) -> Result> { debug!( event = EVENT_OIDC_STATE, component = LOG_COMPONENT_IAM, @@ -323,10 +296,10 @@ async fn init_oidc_sys_with_extra_root_ca_provider_inner( "OIDC runtime starting" ); - let oidc_sys_result = match extra_root_ca_provider { - Some(provider) => OidcSys::new_with_extra_root_ca_provider(provider).await, - None => OidcSys::new_with_extra_root_ca(root_ca_pem).await, - }; + finish_oidc_sys_build(OidcSys::new_with_extra_root_ca_provider(extra_root_ca_provider).await) +} + +fn finish_oidc_sys_build(oidc_sys_result: std::result::Result) -> Result> { let oidc_sys = match oidc_sys_result { Ok(sys) => { if sys.has_providers() { @@ -362,20 +335,117 @@ async fn init_oidc_sys_with_extra_root_ca_provider_inner( } }; - if OIDC_SYS.set(Arc::new(oidc_sys)).is_err() { - warn!( - event = EVENT_OIDC_STATE, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - state = "singleton_set_race", - "OIDC runtime singleton set raced" - ); + Ok(Arc::new(oidc_sys)) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::{ + io::{self, Write}, + sync::{ + Mutex, + atomic::{AtomicUsize, Ordering}, + }, + }; + use tracing_subscriber::{Registry, fmt::MakeWriter, layer::SubscriberExt}; + + #[derive(Clone, Default)] + struct CapturedLogs(Arc>>); + + struct CapturedLogWriter(Arc>>); + + impl Write for CapturedLogWriter { + fn write(&mut self, buf: &[u8]) -> io::Result { + self.0 + .lock() + .expect("captured log lock should be available") + .extend_from_slice(buf); + Ok(buf.len()) + } + + fn flush(&mut self) -> io::Result<()> { + Ok(()) + } } - Ok(()) -} + impl<'writer> MakeWriter<'writer> for CapturedLogs { + type Writer = CapturedLogWriter; -/// Get the global OIDC system. -pub fn get_oidc() -> Option> { - OIDC_SYS.get().cloned() + fn make_writer(&'writer self) -> Self::Writer { + CapturedLogWriter(Arc::clone(&self.0)) + } + } + + fn capture_oidc_build(result: std::result::Result) -> (Arc, String) { + let logs = CapturedLogs::default(); + let captured = Arc::clone(&logs.0); + let subscriber = Registry::default().with( + tracing_subscriber::fmt::layer() + .without_time() + .with_target(false) + .with_level(true) + .with_ansi(false) + .with_writer(logs), + ); + + let oidc = tracing::subscriber::with_default(subscriber, || { + finish_oidc_sys_build(result).expect("OIDC build completion should succeed") + }); + let output = String::from_utf8(captured.lock().expect("captured log lock should be available").clone()) + .expect("captured logs should be UTF-8"); + (oidc, output) + } + + #[test] + fn oidc_build_reports_empty_configuration() { + let (oidc, output) = capture_oidc_build(OidcSys::empty()); + + assert!(!oidc.has_providers()); + assert!(output.contains(EVENT_OIDC_STATE), "{output}"); + assert!(output.contains("state=\"empty\""), "{output}"); + } + + #[test] + fn oidc_build_reports_ready_configuration() { + let oidc = crate::oidc::make_test_sys(vec![crate::oidc::test_config("default")]); + let (oidc, output) = capture_oidc_build(Ok(oidc)); + + assert!(oidc.has_providers()); + assert_eq!(oidc.config_snapshot().providers().len(), 1); + assert!(output.contains(EVENT_OIDC_STATE), "{output}"); + assert!(output.contains("state=\"ready\""), "{output}"); + assert!(output.contains("provider_count=1"), "{output}"); + } + + #[test] + fn oidc_build_reports_error_and_returns_empty_runtime() { + let (oidc, output) = capture_oidc_build(Err("configured OIDC root CA is invalid".to_string())); + + assert!(!oidc.has_providers()); + assert!(output.contains("WARN"), "{output}"); + assert!(output.contains(EVENT_OIDC_STATE), "{output}"); + assert!(output.contains("state=\"init_failed_non_fatal\""), "{output}"); + assert!(output.contains("configured OIDC root CA is invalid"), "{output}"); + } + + #[tokio::test] + async fn oidc_builder_loads_root_ca_provider_and_keeps_failure_non_fatal() { + let load_count = Arc::new(AtomicUsize::new(0)); + let observed_load_count = Arc::clone(&load_count); + let provider = OidcExtraRootCaProvider::new(move || { + let observed_load_count = Arc::clone(&observed_load_count); + async move { + observed_load_count.fetch_add(1, Ordering::SeqCst); + Err("test root CA load failure".to_string()) + } + }); + + let oidc = build_oidc_sys_with_extra_root_ca_provider(provider) + .await + .expect("root CA load failure should produce an empty OIDC runtime"); + + assert_eq!(load_count.load(Ordering::SeqCst), 1); + assert!(!oidc.has_providers()); + } } diff --git a/crates/iam/src/oidc.rs b/crates/iam/src/oidc.rs deleted file mode 100644 index 7a4adaa2f..000000000 --- a/crates/iam/src/oidc.rs +++ /dev/null @@ -1,4710 +0,0 @@ -// Copyright 2024 RustFS Team -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -//! OIDC Provider Manager -//! -//! Implements the OpenID Connect Authorization Code Flow with PKCE using the -//! `openidconnect` crate for standards-compliant discovery, token exchange, -//! and ID token verification. - -use crate::{ - EVENT_OIDC_DIAGNOSTICS, LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC, - federation::{CoreFederatedAuthorizationMapper, FederatedAuthorizationRuleRef}, - oidc_state::{OidcAuthSession, OidcLogoutSession, OidcStateStore}, -}; -use openidconnect::core::{ - CoreAuthenticationFlow, CoreClient, CoreIdToken, CoreIdTokenVerifier, CoreJsonWebKeySet, CoreJwsSigningAlgorithm, -}; -use openidconnect::{ - AsyncHttpClient, Audience, AuthType, AuthorizationCode, ClientId, ClientSecret, CsrfToken, DiscoveryError, IssuerUrl, - JsonWebKeySetUrl, LogoutRequest, Nonce, PkceCodeChallenge, PkceCodeVerifier, PostLogoutRedirectUrl, - ProviderMetadataWithLogout, RedirectUrl, RequestTokenError, Scope, TokenUrl, -}; -use reqwest::{Certificate, Client}; -use rustfs_config::oidc::*; -use rustfs_config::server_config::{Config as ServerConfig, KVS}; -use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, EnableState, MAX_OIDC_RESPONSE_SIZE}; -use rustfs_policy::policy::{ClaimLookup, get_claim_case_insensitive}; -use rustfs_utils::egress::{ENV_OUTBOUND_ALLOW_ORIGINS, OutboundPolicy, find_outbound_dns_policy_rejection}; -use serde::{Deserialize, Serialize}; -use std::borrow::Cow; -use std::collections::{HashMap, VecDeque}; -use std::fmt; -use std::future::Future; -use std::net::IpAddr; -use std::pin::Pin; -use std::sync::Arc; -use std::sync::{LazyLock, Mutex, MutexGuard, RwLock}; -use std::time::{Duration as StdDuration, Instant}; -use tokio::time::sleep; -use tracing::{debug, error, warn}; -use url::Url; - -const EVENT_OIDC_HTTP: &str = "oidc_http"; -const OIDC_JWKS_REFRESH_INTERVAL: StdDuration = StdDuration::from_secs(24 * 60 * 60); -const OIDC_DISCOVERY_TRANSPORT_RETRIES: usize = 3; -const OIDC_DISCOVERY_TRANSPORT_RETRY_DELAY: StdDuration = StdDuration::from_millis(50); -const OIDC_JWKS_BLOCKED_BY_OUTBOUND_POLICY: &str = "JWKS request blocked by outbound policy"; -const OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY: &str = "OIDC provider discovery blocked by outbound policy"; -const OIDC_HTTP_REQUEST_TIMEOUT: StdDuration = StdDuration::from_secs(10); -const OIDC_HTTP_CONNECT_TIMEOUT: StdDuration = StdDuration::from_secs(3); -const OIDC_PLUGIN_AUTHN_WINDOW: StdDuration = StdDuration::from_secs(60); - -#[derive(Debug, Clone, Copy, Default)] -pub struct OidcPluginAuthnMetricsSnapshot { - pub failed_requests_minute: u64, - pub last_fail_seconds: u64, - pub last_succ_seconds: u64, - pub succ_avg_rtt_ms_minute: u64, - pub succ_max_rtt_ms_minute: u64, - pub total_requests_minute: u64, -} - -#[derive(Debug, Clone)] -struct OidcPluginAuthnSample { - observed_at: Instant, - succeeded: bool, - rtt_ms: u64, -} - -#[derive(Debug, Default)] -struct OidcPluginAuthnMetrics { - samples: Mutex>, - last_fail_at: Mutex>, - last_succ_at: Mutex>, -} - -fn lock_oidc_plugin_authn_metrics<'a, T>(mutex: &'a Mutex, metric: &'static str) -> MutexGuard<'a, T> { - match mutex.lock() { - Ok(guard) => guard, - Err(err) => { - warn!(metric, "Recovering poisoned OIDC authn metrics lock"); - err.into_inner() - } - } -} - -fn seconds_since(now: Instant, observed_at: Option) -> u64 { - observed_at - .map(|instant| now.duration_since(instant).as_secs()) - .unwrap_or_default() -} - -impl OidcPluginAuthnMetrics { - fn record(&self, rtt_ms: u64, succeeded: bool) { - let now = Instant::now(); - let mut samples = lock_oidc_plugin_authn_metrics(&self.samples, "samples"); - samples.push_back(OidcPluginAuthnSample { - observed_at: now, - succeeded, - rtt_ms, - }); - while samples - .front() - .is_some_and(|sample| now.duration_since(sample.observed_at) > OIDC_PLUGIN_AUTHN_WINDOW) - { - samples.pop_front(); - } - drop(samples); - - if succeeded { - *lock_oidc_plugin_authn_metrics(&self.last_succ_at, "last_succ_at") = Some(now); - } else { - *lock_oidc_plugin_authn_metrics(&self.last_fail_at, "last_fail_at") = Some(now); - } - } - - fn snapshot(&self) -> OidcPluginAuthnMetricsSnapshot { - let now = Instant::now(); - let (total_requests_minute, failed_requests_minute, succ_avg_rtt_ms_minute, succ_max_rtt_ms_minute) = { - let mut samples = lock_oidc_plugin_authn_metrics(&self.samples, "samples"); - while samples - .front() - .is_some_and(|sample| now.duration_since(sample.observed_at) > OIDC_PLUGIN_AUTHN_WINDOW) - { - samples.pop_front(); - } - - let mut failed_requests_minute = 0u64; - let mut successful_requests = 0u64; - let mut successful_rtt_sum = 0u64; - let mut succ_max_rtt_ms_minute = 0u64; - - for sample in samples.iter() { - if sample.succeeded { - successful_requests += 1; - successful_rtt_sum += sample.rtt_ms; - succ_max_rtt_ms_minute = succ_max_rtt_ms_minute.max(sample.rtt_ms); - } else { - failed_requests_minute += 1; - } - } - - let succ_avg_rtt_ms_minute = successful_rtt_sum.checked_div(successful_requests).unwrap_or_default(); - - ( - samples.len() as u64, - failed_requests_minute, - succ_avg_rtt_ms_minute, - succ_max_rtt_ms_minute, - ) - }; - - let last_fail_seconds = seconds_since(now, *lock_oidc_plugin_authn_metrics(&self.last_fail_at, "last_fail_at")); - let last_succ_seconds = seconds_since(now, *lock_oidc_plugin_authn_metrics(&self.last_succ_at, "last_succ_at")); - - OidcPluginAuthnMetricsSnapshot { - failed_requests_minute, - last_fail_seconds, - last_succ_seconds, - succ_avg_rtt_ms_minute, - succ_max_rtt_ms_minute, - total_requests_minute, - } - } -} - -static OIDC_PLUGIN_AUTHN_METRICS: LazyLock = LazyLock::new(OidcPluginAuthnMetrics::default); - -pub fn oidc_plugin_authn_metrics_snapshot() -> OidcPluginAuthnMetricsSnapshot { - OIDC_PLUGIN_AUTHN_METRICS.snapshot() -} - -/// Header names whose values may carry OIDC secrets (client credentials, cookies, -/// bearer tokens). Their values are never emitted to logs, only their byte length. -const SENSITIVE_HEADER_NAMES: [&str; 4] = ["authorization", "proxy-authorization", "cookie", "set-cookie"]; - -fn is_sensitive_header(name: &str) -> bool { - SENSITIVE_HEADER_NAMES - .iter() - .any(|candidate| name.eq_ignore_ascii_case(candidate)) -} - -fn format_http_headers(headers: &http::HeaderMap) -> String { - headers - .iter() - .map(|(name, value)| { - if is_sensitive_header(name.as_str()) { - format!("{}=", name.as_str(), value.as_bytes().len()) - } else { - let value = value.to_str().unwrap_or(""); - format!("{}={}", name.as_str(), value) - } - }) - .collect::>() - .join("; ") -} - -#[derive(Debug, Default)] -struct TokenResponseBodyShape { - json_object: bool, - json_keys: String, - has_access_token: bool, - has_id_token: bool, - has_token_type: bool, - has_expires_in: bool, - has_error: bool, - has_error_description: bool, - looks_like_html: bool, -} - -fn inspect_token_response_body(body: &[u8]) -> TokenResponseBodyShape { - let mut shape = TokenResponseBodyShape { - looks_like_html: body - .iter() - .copied() - .find(|byte| !byte.is_ascii_whitespace()) - .is_some_and(|byte| byte == b'<'), - ..Default::default() - }; - - let Ok(value) = serde_json::from_slice::(body) else { - return shape; - }; - let Some(object) = value.as_object() else { - return shape; - }; - - shape.json_object = true; - shape.has_access_token = object.contains_key("access_token"); - shape.has_id_token = object.contains_key("id_token"); - shape.has_token_type = object.contains_key("token_type"); - shape.has_expires_in = object.contains_key("expires_in"); - shape.has_error = object.contains_key("error"); - shape.has_error_description = object.contains_key("error_description"); - - let mut keys: Vec<&str> = object.keys().map(String::as_str).collect(); - keys.sort_unstable(); - keys.truncate(16); - shape.json_keys = keys.join(","); - - shape -} - -fn oidc_http_error_diagnostics(error: &OidcHttpError) -> (&'static str, String) { - match error { - OidcHttpError::Reqwest(err) if err.is_timeout() => ("timeout", String::new()), - OidcHttpError::Reqwest(err) if err.is_connect() => ("connect", String::new()), - OidcHttpError::Reqwest(err) if err.status().is_some() => { - ("http_status", err.status().map(|status| status.as_u16().to_string()).unwrap_or_default()) - } - OidcHttpError::Reqwest(_) => ("request", String::new()), - OidcHttpError::Http(_) => ("http_build", String::new()), - OidcHttpError::ExtraRootCa(_) => ("extra_root_ca", String::new()), - OidcHttpError::ForbiddenOutbound(_) => ("forbidden_outbound", String::new()), - OidcHttpError::ResponseTooLarge(limit) => ("response_too_large", limit.to_string()), - } -} - -// ---- HTTP Client Adapter ---- - -/// Error type for the OIDC HTTP client adapter. -#[derive(Debug)] -pub enum OidcHttpError { - Reqwest(reqwest::Error), - Http(http::Error), - ExtraRootCa(String), - /// The outbound destination was rejected by the shared egress policy before any - /// connection was attempted (invalid URL, loopback/link-local/metadata/private IP, - /// or a malformed allow-origins configuration). - ForbiddenOutbound(String), - /// The provider response body exceeded [`MAX_OIDC_RESPONSE_SIZE`] and was abandoned - /// instead of being buffered in full. - ResponseTooLarge(usize), -} - -impl std::fmt::Display for OidcHttpError { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - Self::Reqwest(e) => write!(f, "{e}"), - Self::Http(e) => write!(f, "{e}"), - Self::ExtraRootCa(reason) => write!(f, "failed to load OIDC extra root CA bundle: {reason}"), - Self::ForbiddenOutbound(reason) => write!(f, "outbound request rejected: {reason}"), - Self::ResponseTooLarge(limit) => write!(f, "oidc response body exceeds {limit} bytes"), - } - } -} - -impl std::error::Error for OidcHttpError { - fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { - match self { - Self::Reqwest(e) => Some(e), - Self::Http(e) => Some(e), - Self::ExtraRootCa(_) | Self::ForbiddenOutbound(_) | Self::ResponseTooLarge(_) => None, - } - } -} - -#[derive(Clone, Debug, Default)] -pub struct OidcExtraRootCaMaterial { - pub generation: u64, - pub root_ca_pem: Option>, -} - -type OidcExtraRootCaFuture = Pin> + Send>>; -type OidcExtraRootCaLoader = dyn Fn() -> OidcExtraRootCaFuture + Send + Sync; - -#[derive(Clone)] -pub struct OidcExtraRootCaProvider { - loader: Arc, -} - -impl OidcExtraRootCaProvider { - pub fn new(loader: F) -> Self - where - F: Fn() -> Fut + Send + Sync + 'static, - Fut: Future> + Send + 'static, - { - Self { - loader: Arc::new(move || Box::pin(loader())), - } - } - - async fn load(&self) -> Result { - (self.loader)().await - } -} - -#[derive(Clone, Default)] -struct CachedOidcExtraRootCerts { - generation: u64, - initialized: bool, - certs: Vec, -} - -/// HTTP client adapter bridging reqwest 0.13 to the `openidconnect` `AsyncHttpClient` trait. -/// -/// A fresh client is built for every request so the destination is re-validated and the -/// resolved IP is re-classified at connection time. This closes the SSRF / DNS-rebinding -/// gap where a one-shot URL string check is bypassed by a hostname that resolves to an -/// internal address only at connection time, and it also covers endpoints discovered from -/// the provider metadata (JWKS, token) rather than only the operator-configured `config_url`. -pub(crate) struct ReqwestHttpClient { - /// `None` in production: the process-cached outbound policy from the environment is used. - /// `Some(..)` only in tests, to explicitly allow a loopback mock endpoint. - policy_override: Option, - extra_root_certs: Arc>, - extra_root_ca_provider: Option, - #[cfg(test)] - dns_resolver_override: Option>, -} - -fn parse_oidc_extra_root_certs(source: &str, pem: &[u8]) -> Result, String> { - if pem.iter().all(|byte| byte.is_ascii_whitespace()) { - return Ok(Vec::new()); - } - Certificate::from_pem_bundle(pem).map_err(|err| format!("failed to parse OIDC extra root CA bundle from {source}: {err}")) -} - -fn oidc_extra_root_certs(root_ca_pem: Option<&[u8]>) -> Result, String> { - match root_ca_pem { - Some(pem) => parse_oidc_extra_root_certs("RustFS outbound TLS material", pem), - None => Ok(Vec::new()), - } -} - -/// Build a reqwest client pinned to the shared outbound egress policy for a single request. -/// -/// [`OutboundPolicy::resolver_for`] validates the URL shape and rejects loopback, -/// link-local, metadata, multicast and unauthorized private addresses up front, and the -/// returned `OutboundDnsResolver` re-resolves and re-classifies the host on every new -/// connection so DNS rebinding fails closed. Redirects are not followed: a redirect target -/// would otherwise skip URL-shape re-validation. The timeouts bound how long a slow or -/// stalled provider can pin the calling task. -fn build_oidc_http_client( - uri: &str, - policy_override: Option<&OutboundPolicy>, - extra_root_certs: &[Certificate], - #[cfg(test)] dns_resolver_override: Option>, -) -> Result<(Client, Url), OidcHttpError> { - let url = Url::parse(uri).map_err(|_| OidcHttpError::ForbiddenOutbound("invalid outbound OIDC URL".to_string()))?; - let resolver = match policy_override { - Some(policy) => policy.resolver_for(&url), - None => OutboundPolicy::from_env_cached() - .map_err(|err| OidcHttpError::ForbiddenOutbound(err.to_string()))? - .resolver_for(&url), - } - .map_err(|err| { - let base = err.to_string(); - let origin = url.origin().ascii_serialization(); - let can_allow_origin = - OutboundPolicy::from_allowed_origins(&origin).is_ok_and(|allowlisted| allowlisted.validate_url(&url).is_ok()); - oidc_forbidden_outbound_error(&url, base, can_allow_origin) - })?; - let bypass_proxy = should_bypass_proxy_for_oidc_uri(uri); - #[cfg(test)] - let bypass_proxy = bypass_proxy || dns_resolver_override.is_some(); - #[cfg(test)] - let resolver: Arc = dns_resolver_override.unwrap_or_else(|| Arc::new(resolver)); - - let mut builder = reqwest::Client::builder() - .dns_resolver(resolver) - .redirect(reqwest::redirect::Policy::none()) - .timeout(OIDC_HTTP_REQUEST_TIMEOUT) - .connect_timeout(OIDC_HTTP_CONNECT_TIMEOUT); - if bypass_proxy { - builder = builder.no_proxy(); - } - if !extra_root_certs.is_empty() { - builder = builder.tls_certs_merge(extra_root_certs.iter().cloned()); - } - builder.build().map(|client| (client, url)).map_err(OidcHttpError::Reqwest) -} - -fn oidc_forbidden_outbound_error(url: &Url, base: String, can_allow_origin: bool) -> OidcHttpError { - let reason = if can_allow_origin { - let origin = url.origin().ascii_serialization(); - format!( - "{base}; add {origin} to {ENV_OUTBOUND_ALLOW_ORIGINS} (comma-separated) and restart RustFS to allow this operator-owned OIDC provider (origin only, no path)" - ) - } else { - base - }; - OidcHttpError::ForbiddenOutbound(reason) -} - -fn oidc_http_error_from_reqwest(url: &Url, error: reqwest::Error) -> OidcHttpError { - if let Some(rejection) = find_outbound_dns_policy_rejection(&error) { - let base = rejection.to_string(); - return oidc_forbidden_outbound_error(url, base, rejection.allow_origin_can_recover()); - } - OidcHttpError::Reqwest(error) -} - -/// Buffer a provider response body, failing closed once `limit` bytes have been seen. -/// -/// `Response::bytes` would buffer the whole body unconditionally, so a hostile or compromised -/// provider endpoint could stream an arbitrarily large (or endless) body into memory. -async fn read_bounded_response_body(response: reqwest::Response, limit: usize) -> Result, OidcHttpError> { - if response.content_length().is_some_and(|len| len > limit as u64) { - return Err(OidcHttpError::ResponseTooLarge(limit)); - } - - let mut response = response; - let mut body = Vec::new(); - while let Some(chunk) = response.chunk().await.map_err(OidcHttpError::Reqwest)? { - if body.len() + chunk.len() > limit { - return Err(OidcHttpError::ResponseTooLarge(limit)); - } - body.extend_from_slice(&chunk); - } - Ok(body) -} - -fn should_bypass_proxy_for_oidc_uri(uri: &str) -> bool { - let Some(host) = Url::parse(uri).ok().and_then(|url| url.host_str().map(str::to_owned)) else { - return false; - }; - let host = host.trim_matches(['[', ']']); - - host.eq_ignore_ascii_case("localhost") || host.parse::().is_ok_and(|addr| addr.is_loopback()) -} - -impl ReqwestHttpClient { - fn new() -> Result { - Self::new_with_extra_root_certs(Vec::new()) - } - - fn extra_root_cert_cache(certs: Vec) -> Arc> { - Arc::new(RwLock::new(CachedOidcExtraRootCerts { - generation: 0, - initialized: true, - certs, - })) - } - - fn new_with_extra_root_certs(extra_root_certs: Vec) -> Result { - Ok(Self { - policy_override: None, - extra_root_certs: Self::extra_root_cert_cache(extra_root_certs), - extra_root_ca_provider: None, - #[cfg(test)] - dns_resolver_override: None, - }) - } - - fn new_with_extra_root_ca_provider(extra_root_ca_provider: OidcExtraRootCaProvider) -> Result { - Ok(Self { - policy_override: None, - extra_root_certs: Arc::new(RwLock::new(CachedOidcExtraRootCerts::default())), - extra_root_ca_provider: Some(extra_root_ca_provider), - #[cfg(test)] - dns_resolver_override: None, - }) - } - - async fn current_extra_root_certs(&self) -> Result, OidcHttpError> { - let Some(provider) = self.extra_root_ca_provider.as_ref() else { - return self - .extra_root_certs - .read() - .map(|cache| cache.certs.clone()) - .map_err(|e| OidcHttpError::ExtraRootCa(format!("extra root certificate cache lock poisoned: {e}"))); - }; - - let material = provider.load().await.map_err(OidcHttpError::ExtraRootCa)?; - if let Ok(cache) = self.extra_root_certs.read() - && cache.initialized - && cache.generation == material.generation - { - return Ok(cache.certs.clone()); - } - - let certs = oidc_extra_root_certs(material.root_ca_pem.as_deref()).map_err(OidcHttpError::ExtraRootCa)?; - let mut cache = self - .extra_root_certs - .write() - .map_err(|e| OidcHttpError::ExtraRootCa(format!("extra root certificate cache lock poisoned: {e}")))?; - cache.generation = material.generation; - cache.initialized = true; - cache.certs = certs.clone(); - Ok(certs) - } - - /// Test-only constructor that pins outbound requests to an explicit policy, so a - /// loopback mock server can be reached without depending on process-wide environment. - #[cfg(test)] - fn with_policy(policy: OutboundPolicy) -> Self { - Self { - policy_override: Some(policy), - extra_root_certs: Self::extra_root_cert_cache(Vec::new()), - extra_root_ca_provider: None, - dns_resolver_override: None, - } - } - - #[cfg(test)] - fn with_policy_and_extra_root_certs(policy: OutboundPolicy, extra_root_certs: Vec) -> Self { - Self { - policy_override: Some(policy), - extra_root_certs: Self::extra_root_cert_cache(extra_root_certs), - extra_root_ca_provider: None, - dns_resolver_override: None, - } - } - - #[cfg(test)] - fn with_policy_and_extra_root_ca_provider(policy: OutboundPolicy, extra_root_ca_provider: OidcExtraRootCaProvider) -> Self { - Self { - policy_override: Some(policy), - extra_root_certs: Arc::new(RwLock::new(CachedOidcExtraRootCerts::default())), - extra_root_ca_provider: Some(extra_root_ca_provider), - dns_resolver_override: None, - } - } - - #[cfg(test)] - fn with_policy_and_dns_resolver(policy: OutboundPolicy, resolver: Arc) -> Self { - Self { - policy_override: Some(policy), - extra_root_certs: Self::extra_root_cert_cache(Vec::new()), - extra_root_ca_provider: None, - dns_resolver_override: Some(resolver), - } - } -} - -impl<'c> AsyncHttpClient<'c> for ReqwestHttpClient { - type Error = OidcHttpError; - type Future = Pin>, Self::Error>> + Send + 'c>>; - - fn call(&'c self, request: http::Request>) -> Self::Future { - Box::pin(async move { - let started_at = Instant::now(); - let (parts, body) = request.into_parts(); - let method = parts.method.clone(); - let uri = parts.uri.to_string(); - if tracing::enabled!(tracing::Level::DEBUG) { - let request_headers = format_http_headers(&parts.headers); - debug!( - event = EVENT_OIDC_HTTP, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "request", - method = %method, - uri = %uri, - request_headers = %request_headers, - request_body_len = body.len(), - "oidc outbound http" - ); - } - - let extra_root_certs = self.current_extra_root_certs().await?; - let (client, url) = build_oidc_http_client( - &uri, - self.policy_override.as_ref(), - &extra_root_certs, - #[cfg(test)] - self.dns_resolver_override.clone(), - )?; - let response = client - .request(parts.method, uri.clone()) - .headers(parts.headers) - .body(body) - .send() - .await; - - let elapsed_ms = started_at.elapsed().as_millis().min(u128::from(u64::MAX)) as u64; - let succeeded = response.as_ref().is_ok_and(|resp| resp.status().is_success()); - OIDC_PLUGIN_AUTHN_METRICS.record(elapsed_ms, succeeded); - - let response = response.map_err(|err| { - let error = oidc_http_error_from_reqwest(&url, err); - error!( - event = EVENT_OIDC_HTTP, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "request_failed", - method = %method, - uri = %uri, - elapsed_ms, - error = %error, - "oidc outbound http" - ); - error - })?; - - let status = response.status(); - let headers = response.headers().clone(); - let body_bytes = read_bounded_response_body(response, MAX_OIDC_RESPONSE_SIZE) - .await - .map_err(|err| { - error!( - event = EVENT_OIDC_HTTP, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "response_body_failed", - method = %method, - uri = %uri, - status = status.as_u16(), - elapsed_ms, - error = %err, - "oidc outbound http" - ); - err - })?; - if tracing::enabled!(tracing::Level::DEBUG) { - let response_headers = format_http_headers(&headers); - debug!( - event = EVENT_OIDC_HTTP, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "response", - method = %method, - uri = %uri, - status = status.as_u16(), - status_success = status.is_success(), - elapsed_ms, - response_headers = %response_headers, - response_body_len = body_bytes.len(), - "oidc outbound http" - ); - } - - let mut http_response = http::Response::builder() - .status(status) - .body(body_bytes) - .map_err(OidcHttpError::Http)?; - *http_response.headers_mut() = headers; - - Ok(http_response) - }) - } -} - -// ---- Public types ---- - -const REDACTED_SECRET: &str = "***redacted***"; - -fn redacted_optional_secret(value: Option<&str>) -> &'static str { - value.filter(|secret| !secret.is_empty()).map_or("", |_| REDACTED_SECRET) -} - -/// Parsed configuration for a single OIDC provider. -#[derive(Clone, PartialEq, Eq)] -pub struct OidcProviderConfig { - pub id: String, - pub enabled: bool, - pub config_url: String, - pub issuer: Option, - pub client_id: String, - pub client_secret: Option, - pub scopes: Vec, - pub other_audiences: Vec, - pub redirect_uri: Option, - pub redirect_uri_dynamic: bool, - pub claim_name: String, - pub claim_prefix: String, - pub role_policy: String, - pub display_name: String, - pub groups_claim: String, - pub roles_claim: String, - pub email_claim: String, - pub username_claim: String, - pub hide_from_ui: bool, -} - -impl fmt::Debug for OidcProviderConfig { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - f.debug_struct("OidcProviderConfig") - .field("id", &self.id) - .field("enabled", &self.enabled) - .field("config_url", &self.config_url) - .field("issuer", &self.issuer) - .field("client_id", &self.client_id) - .field("client_secret", &redacted_optional_secret(self.client_secret.as_deref())) - .field("scopes", &self.scopes) - .field("other_audiences", &self.other_audiences) - .field("redirect_uri", &self.redirect_uri) - .field("redirect_uri_dynamic", &self.redirect_uri_dynamic) - .field("claim_name", &self.claim_name) - .field("claim_prefix", &self.claim_prefix) - .field("role_policy", &self.role_policy) - .field("display_name", &self.display_name) - .field("groups_claim", &self.groups_claim) - .field("roles_claim", &self.roles_claim) - .field("email_claim", &self.email_claim) - .field("username_claim", &self.username_claim) - .field("hide_from_ui", &self.hide_from_ui) - .finish() - } -} - -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] -#[serde(rename_all = "snake_case")] -pub enum OidcProviderConfigSource { - Env, - Persisted, -} - -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct SourcedOidcProviderConfig { - pub config: OidcProviderConfig, - pub source: OidcProviderConfigSource, -} - -/// Immutable OIDC configuration entries returned by configuration queries. -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct OidcConfigSnapshot { - providers: Vec, -} - -impl OidcConfigSnapshot { - pub fn new(mut providers: Vec) -> Self { - providers.sort_by(|left, right| { - (left.config.id != "default") - .cmp(&(right.config.id != "default")) - .then_with(|| left.config.id.cmp(&right.config.id)) - }); - Self { providers } - } - - pub fn providers(&self) -> &[SourcedOidcProviderConfig] { - &self.providers - } - - pub fn into_providers(self) -> Vec { - self.providers - } -} - -#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] -pub struct OidcProviderValidationResult { - pub issuer: String, - pub authorization_endpoint: Option, - pub token_endpoint: Option, -} - -/// Compatibility name for the provider summary returned by OIDC list APIs. -pub type OidcProviderSummary = crate::federation::FederatedProviderView; - -/// Claims extracted from an OIDC ID token. -#[derive(Debug, Clone, Serialize, Deserialize, Default)] -pub struct OidcClaims { - pub sub: String, - pub email: String, - pub username: String, - pub groups: Vec, - pub raw: HashMap, -} - -// ---- Internal provider state ---- - -/// Discovered OIDC provider metadata. -/// We store metadata (which includes JWKS after discovery) separately rather than -/// a `CoreClient` because the crate uses type-state generics that make storing -/// the configured client in a HashMap impractical. The client is reconstructed -/// on-the-fly from metadata when needed. -#[derive(Clone)] -struct ProviderState { - metadata: DiscoveredProviderMetadata, - discovered_at: Instant, -} - -// Workload issuers do not implement the browser authorization flow. Keep their -// verification metadata separate rather than inventing an authorization URL. -#[serde_with::serde_as] -#[derive(Clone, Deserialize)] -struct WorkloadProviderMetadata { - issuer: IssuerUrl, - jwks_uri: JsonWebKeySetUrl, - token_endpoint: Option, - #[serde_as(as = "serde_with::VecSkipError<_>")] - id_token_signing_alg_values_supported: Vec, - #[serde(skip)] - jwks: CoreJsonWebKeySet, - // Discovery is extensible; report unsupported fields without logging values. - #[serde(flatten)] - additional_fields: HashMap, -} - -#[derive(Clone)] -enum DiscoveredProviderMetadata { - Console(Box), - Workload(Box), -} - -impl DiscoveredProviderMetadata { - fn parse(body: &[u8], hide_from_ui: bool) -> Result { - let document: serde_json::Value = serde_json::from_slice(body).map_err(|err| err.to_string())?; - if hide_from_ui - && document - .as_object() - .is_some_and(|fields| !fields.contains_key("authorization_endpoint")) - { - let mut metadata: WorkloadProviderMetadata = serde_json::from_slice(body).map_err(|err| err.to_string())?; - if !metadata.additional_fields.is_empty() { - warn!( - event = EVENT_OIDC_DIAGNOSTICS, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "workload_discovery_additional_fields", - field_count = metadata.additional_fields.len(), - "workload discovery contains additional fields" - ); - metadata.additional_fields.clear(); - } - Ok(Self::Workload(Box::new(metadata))) - } else { - serde_json::from_slice(body) - .map(|metadata| Self::Console(Box::new(metadata))) - .map_err(|err| err.to_string()) - } - } - - fn console(&self) -> Result<&ProviderMetadataWithLogout, String> { - match self { - Self::Console(metadata) => Ok(metadata), - Self::Workload(_) => Err("OIDC provider has no authorization endpoint; only web identity is supported".into()), - } - } - - fn issuer(&self) -> &IssuerUrl { - match self { - Self::Console(metadata) => metadata.issuer(), - Self::Workload(metadata) => &metadata.issuer, - } - } - - fn jwks_uri(&self) -> &JsonWebKeySetUrl { - match self { - Self::Console(metadata) => metadata.jwks_uri(), - Self::Workload(metadata) => &metadata.jwks_uri, - } - } - - fn set_jwks(self, jwks: CoreJsonWebKeySet) -> Self { - match self { - Self::Console(metadata) => Self::Console(Box::new(metadata.set_jwks(jwks))), - Self::Workload(mut metadata) => { - metadata.jwks = jwks; - Self::Workload(metadata) - } - } - } - - fn authorization_endpoint(&self) -> Option { - match self { - Self::Console(metadata) => Some(metadata.authorization_endpoint().to_string()), - Self::Workload(_) => None, - } - } - - fn token_endpoint(&self) -> Option<&TokenUrl> { - match self { - Self::Console(metadata) => metadata.token_endpoint(), - Self::Workload(metadata) => metadata.token_endpoint.as_ref(), - } - } - - fn verifier(&self, config: &OidcProviderConfig) -> CoreIdTokenVerifier<'static> { - let client_id = ClientId::new(config.client_id.clone()); - let secret = config.client_secret.as_ref().map(|secret| ClientSecret::new(secret.clone())); - let (issuer, jwks, algorithms) = match self { - Self::Console(metadata) => (metadata.issuer(), metadata.jwks(), metadata.id_token_signing_alg_values_supported()), - Self::Workload(metadata) => (&metadata.issuer, &metadata.jwks, &metadata.id_token_signing_alg_values_supported), - }; - let verifier = match secret { - Some(secret) => CoreIdTokenVerifier::new_confidential_client(client_id, secret, issuer.clone(), jwks.clone()), - None => CoreIdTokenVerifier::new_public_client(client_id, issuer.clone(), jwks.clone()), - }; - verifier.set_allowed_algs(algorithms.clone()) - } -} - -// This adapter is used only for discovery/JWKS fetches, never token exchange. -struct JwksAcceptClient<'a> { - inner: &'a ReqwestHttpClient, - discovery_url: Option, -} - -impl<'c> AsyncHttpClient<'c> for JwksAcceptClient<'_> { - type Error = OidcHttpError; - type Future = >::Future; - - fn call(&'c self, mut request: http::Request>) -> Self::Future { - if !self.discovery_url.as_ref().is_some_and(|url| request.uri() == url.as_str()) { - request.headers_mut().insert( - http::header::ACCEPT, - http::HeaderValue::from_static("application/json, application/jwk-set+json"), - ); - } - self.inner.call(request) - } -} - -impl ProviderState { - fn is_stale(&self) -> bool { - self.discovered_at.elapsed() >= OIDC_JWKS_REFRESH_INTERVAL - } -} - -// ---- Core OIDC system ---- - -/// Global OIDC manager for all configured providers. -pub struct OidcSys { - configs: HashMap, - provider_states: RwLock>, - state_store: OidcStateStore, - http_client: ReqwestHttpClient, -} - -fn trusted_aud(other_audiences: &[String], audience: &Audience) -> bool { - for aud in other_audiences { - if audience.as_str() == aud.as_str() { - return true; - } - } - false -} - -impl OidcSys { - /// Parse environment variables and discover all configured OIDC providers. - pub async fn new() -> Result { - Self::new_with_extra_root_ca(None).await - } - - /// Parse environment variables and discover providers with an additional outbound root CA bundle. - pub(crate) async fn new_with_extra_root_ca(root_ca_pem: Option<&[u8]>) -> Result { - let http_client = ReqwestHttpClient::new_with_extra_root_certs(oidc_extra_root_certs(root_ca_pem)?)?; - Self::new_with_http_client(http_client).await - } - - pub(crate) async fn new_with_extra_root_ca_provider(extra_root_ca_provider: OidcExtraRootCaProvider) -> Result { - let http_client = ReqwestHttpClient::new_with_extra_root_ca_provider(extra_root_ca_provider)?; - http_client.current_extra_root_certs().await.map_err(|err| err.to_string())?; - Self::new_with_http_client(http_client).await - } - - async fn new_with_http_client(http_client: ReqwestHttpClient) -> Result { - let server_config = crate::server_config::current_server_config(); - let parsed_configs = load_oidc_config_snapshot(server_config.as_ref()); - let mut configs = HashMap::new(); - let mut provider_states = HashMap::new(); - - for sourced_config in parsed_configs.into_providers() { - let config = &sourced_config.config; - if !config.enabled { - debug!(provider = %config.id, "OIDC provider disabled"); - continue; - } - - match Self::discover_provider(config, &http_client).await { - Ok(state) => { - debug!(provider = %config.id, "OIDC provider discovered"); - provider_states.insert(config.id.clone(), state); - configs.insert(config.id.clone(), sourced_config); - } - Err(e) => { - error!( - event = EVENT_OIDC_DIAGNOSTICS, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "provider_discovery_failed", - provider_id = %config.id, - config_url = %config.config_url, - client_id = %config.client_id, - scopes = ?config.scopes, - redirect_uri = %config.redirect_uri.as_deref().unwrap_or(""), - redirect_uri_dynamic = config.redirect_uri_dynamic, - error = %e, - "oidc provider discovery failed" - ); - } - } - } - - Ok(Self { - configs, - provider_states: RwLock::new(provider_states), - state_store: OidcStateStore::new(), - http_client, - }) - } - - /// Create an OidcSys with no providers (useful for when OIDC is not configured). - pub fn empty() -> Result { - Ok(Self { - configs: HashMap::new(), - provider_states: RwLock::new(HashMap::new()), - state_store: OidcStateStore::new(), - http_client: ReqwestHttpClient::new()?, - }) - } - - /// Return true if any OIDC providers are configured and enabled. - pub fn has_providers(&self) -> bool { - !self.configs.is_empty() - } - - pub(crate) fn provider_configs(&self) -> impl Iterator { - self.configs.values().map(|provider| &provider.config) - } - - /// List all providers, including providers hidden from the login UI. - pub fn list_providers(&self) -> Vec { - self.provider_configs() - .map(|config| OidcProviderSummary { - provider_id: config.id.clone(), - display_name: config.display_name.clone(), - }) - .collect() - } - - /// List providers visible in the login UI. - pub fn list_visible_providers(&self) -> Vec { - self.provider_configs() - .filter(|config| !config.hide_from_ui) - .map(|config| OidcProviderSummary { - provider_id: config.id.clone(), - display_name: config.display_name.clone(), - }) - .collect() - } - - pub fn config_snapshot(&self) -> OidcConfigSnapshot { - OidcConfigSnapshot::new(self.configs.values().cloned().collect()) - } - - /// Build the PKCE authorization URL for a provider, store state in the state store. - pub async fn authorize_url( - &self, - provider_id: &str, - redirect_uri: &str, - redirect_after: Option, - ) -> Result { - let config = self - .get_provider_config(provider_id) - .ok_or_else(|| format!("unknown OIDC provider: {provider_id}"))?; - let state = self.ensure_provider_state(provider_id, config).await?; - - let (pkce_challenge, pkce_verifier) = PkceCodeChallenge::new_random_sha256(); - - let redirect = RedirectUrl::new(redirect_uri.to_string()).map_err(|e| format!("invalid redirect URI: {e}"))?; - - let client = CoreClient::from_provider_metadata( - state.metadata.console()?.clone(), - ClientId::new(config.client_id.clone()), - config.client_secret.as_ref().map(|s| ClientSecret::new(s.clone())), - ) - .set_auth_type(AuthType::RequestBody); - - let mut auth_req = - client.authorize_url(CoreAuthenticationFlow::AuthorizationCode, CsrfToken::new_random, Nonce::new_random); - auth_req = auth_req.set_redirect_uri(Cow::Owned(redirect)); - - for scope in &config.scopes { - auth_req = auth_req.add_scope(Scope::new(scope.clone())); - } - - auth_req = auth_req.set_pkce_challenge(pkce_challenge); - - let (auth_url, csrf_token, nonce) = auth_req.url(); - - // Store the state for callback validation - self.state_store - .insert( - csrf_token.secret().clone(), - OidcAuthSession { - provider_id: provider_id.to_string(), - pkce_verifier: pkce_verifier.secret().clone(), - nonce: nonce.secret().clone(), - redirect_after, - }, - ) - .await; - - Ok(auth_url.to_string()) - } - - /// Exchange an authorization code for tokens and extract claims. - pub async fn exchange_code( - &self, - state: &str, - code: &str, - redirect_uri: &str, - ) -> Result<(OidcClaims, String, OidcAuthSession, String), String> { - // Retrieve and consume the state (single-use) - let session = self - .state_store - .take(state) - .await - .ok_or_else(|| "invalid or expired OIDC state".to_string())?; - - let config = self - .get_provider_config(&session.provider_id) - .ok_or_else(|| format!("unknown provider: {}", session.provider_id))?; - let provider_state = self.get_provider_state(&session.provider_id)?; - let issuer = provider_state.metadata.issuer().to_string(); - let token_endpoint = provider_state - .metadata - .token_endpoint() - .map(ToString::to_string) - .unwrap_or_default(); - - // Construct CoreClient on-the-fly with JWKS from discovery - let client = CoreClient::from_provider_metadata( - provider_state.metadata.console()?.clone(), - ClientId::new(config.client_id.clone()), - config.client_secret.as_ref().map(|s| ClientSecret::new(s.clone())), - ) - .set_auth_type(AuthType::RequestBody); - - let redirect = RedirectUrl::new(redirect_uri.to_string()).map_err(|e| format!("invalid redirect URI: {e}"))?; - - // Exchange code for tokens - let token_response = client - .exchange_code(AuthorizationCode::new(code.to_string())) - .map_err(|e| { - error!( - event = EVENT_OIDC_DIAGNOSTICS, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "token_endpoint_missing", - provider_id = %session.provider_id, - config_url = %config.config_url, - issuer = %issuer, - client_id = %config.client_id, - redirect_uri = %redirect_uri, - scopes = ?config.scopes, - error = %e, - "oidc token exchange failed" - ); - format!( - "token endpoint not configured: {e}: provider_id={}, config_url={}, issuer={}, redirect_uri={}, client_id={}", - session.provider_id, config.config_url, issuer, redirect_uri, config.client_id - ) - })? - .set_pkce_verifier(PkceCodeVerifier::new(session.pkce_verifier.clone())) - .set_redirect_uri(Cow::Owned(redirect)) - .request_async(&self.http_client) - .await - .map_err(|e| match &e { - RequestTokenError::ServerResponse(response) => { - error!( - event = EVENT_OIDC_DIAGNOSTICS, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "token_server_response", - provider_id = %session.provider_id, - config_url = %config.config_url, - issuer = %issuer, - token_endpoint = %token_endpoint, - client_id = %config.client_id, - client_secret_configured = config.client_secret.as_deref().is_some_and(|secret| !secret.is_empty()), - redirect_uri = %redirect_uri, - scopes = ?config.scopes, - oauth_error = %response.error(), - oauth_error_description = %response.error_description().map(String::as_str).unwrap_or(""), - oauth_error_uri = %response.error_uri().map(String::as_str).unwrap_or(""), - error = %e, - "oidc token exchange failed" - ); - format!( - "token exchange failed: {e}: stage=token_server_response, provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}, oauth_error={}, oauth_error_description={}", - session.provider_id, - config.config_url, - issuer, - token_endpoint, - redirect_uri, - config.client_id, - response.error(), - response.error_description().map(String::as_str).unwrap_or("") - ) - } - RequestTokenError::Request(err) => { - let (request_error_kind, request_error_status) = oidc_http_error_diagnostics(err); - error!( - event = EVENT_OIDC_DIAGNOSTICS, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "token_request_failed", - provider_id = %session.provider_id, - config_url = %config.config_url, - issuer = %issuer, - token_endpoint = %token_endpoint, - client_id = %config.client_id, - redirect_uri = %redirect_uri, - request_error_kind = %request_error_kind, - request_error_status = %request_error_status, - error = %err, - "oidc token exchange failed" - ); - format!( - "token exchange failed: stage=token_request_failed, provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}, request_error_kind={}, request_error_status={}, request_error={}", - session.provider_id, - config.config_url, - issuer, - token_endpoint, - redirect_uri, - config.client_id, - request_error_kind, - request_error_status, - err - ) - } - RequestTokenError::Parse(parse_err, body) => { - let shape = inspect_token_response_body(body); - error!( - event = EVENT_OIDC_DIAGNOSTICS, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "token_response_parse_failed", - provider_id = %session.provider_id, - config_url = %config.config_url, - issuer = %issuer, - token_endpoint = %token_endpoint, - client_id = %config.client_id, - redirect_uri = %redirect_uri, - parse_error_path = %parse_err.path(), - response_body_len = body.len(), - response_json_object = shape.json_object, - response_json_keys = %shape.json_keys, - response_has_access_token = shape.has_access_token, - response_has_id_token = shape.has_id_token, - response_has_token_type = shape.has_token_type, - response_has_expires_in = shape.has_expires_in, - response_has_error = shape.has_error, - response_has_error_description = shape.has_error_description, - response_looks_like_html = shape.looks_like_html, - error = %e, - "oidc token exchange failed" - ); - format!( - "token exchange failed: {e}: stage=token_response_parse_failed, provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}, parse_error_path={}, response_body_len={}, response_json_keys={}, response_has_id_token={}, response_has_error={}, response_looks_like_html={}", - session.provider_id, - config.config_url, - issuer, - token_endpoint, - redirect_uri, - config.client_id, - parse_err.path(), - body.len(), - shape.json_keys, - shape.has_id_token, - shape.has_error, - shape.looks_like_html - ) - } - RequestTokenError::Other(message) => { - error!( - event = EVENT_OIDC_DIAGNOSTICS, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "token_exchange_other_error", - provider_id = %session.provider_id, - config_url = %config.config_url, - issuer = %issuer, - token_endpoint = %token_endpoint, - client_id = %config.client_id, - redirect_uri = %redirect_uri, - error = %message, - "oidc token exchange failed" - ); - format!( - "token exchange failed: {e}: stage=token_exchange_other_error, provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}", - session.provider_id, config.config_url, issuer, token_endpoint, redirect_uri, config.client_id - ) - } - })?; - - // Verify the ID token (signature, issuer, audience, expiry, nonce) - let id_token = token_response - .extra_fields() - .id_token() - .ok_or_else(|| { - error!( - event = EVENT_OIDC_DIAGNOSTICS, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "token_response_missing_id_token", - provider_id = %session.provider_id, - config_url = %config.config_url, - issuer = %issuer, - token_endpoint = %token_endpoint, - client_id = %config.client_id, - redirect_uri = %redirect_uri, - scopes = ?config.scopes, - "oidc token exchange failed" - ); - format!( - "no id_token in token response: provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}, scopes={}", - session.provider_id, - config.config_url, - issuer, - token_endpoint, - redirect_uri, - config.client_id, - config.scopes.join(",") - ) - })?; - - let verifier = client - .id_token_verifier() - .set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud)); - let verified = id_token.claims(&verifier, &Nonce::new(session.nonce.clone())); - if let Err(e) = verified { - let verification_error = e.to_string(); - warn!( - event = EVENT_OIDC_DIAGNOSTICS, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "id_token_verification_retry", - provider_id = %session.provider_id, - config_url = %config.config_url, - issuer = %issuer, - token_endpoint = %token_endpoint, - client_id = %config.client_id, - other_audiences = ?config.other_audiences, - error = %verification_error, - "oidc id token verification failed" - ); - let refreshed_state = self - .refresh_provider_state(&session.provider_id, config) - .await - .map_err(|refresh_err| { - format!( - "ID token verification failed: {verification_error}; failed to refresh provider metadata: {refresh_err}" - ) - })?; - - warn!( - event = EVENT_OIDC_DIAGNOSTICS, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "jwks_metadata_refreshed", - provider_id = %session.provider_id, - config_url = %config.config_url, - issuer = %issuer, - "oidc provider metadata refreshed" - ); - - let client = CoreClient::from_provider_metadata( - refreshed_state.metadata.console()?.clone(), - ClientId::new(config.client_id.clone()), - config.client_secret.as_ref().map(|s| ClientSecret::new(s.clone())), - ) - .set_auth_type(AuthType::RequestBody); - - let verifier = client - .id_token_verifier() - .set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud)); - id_token - .claims(&verifier, &Nonce::new(session.nonce.clone())) - .map_err(|retry_err| { - error!( - event = EVENT_OIDC_DIAGNOSTICS, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "id_token_verification_failed", - provider_id = %session.provider_id, - config_url = %config.config_url, - issuer = %issuer, - token_endpoint = %token_endpoint, - client_id = %config.client_id, - other_audiences = ?config.other_audiences, - original_error = %verification_error, - retry_error = %retry_err, - "oidc id token verification failed" - ); - format!( - "ID token verification failed after JWKS refresh: {retry_err}; original_error={verification_error}; provider_id={}, config_url={}, issuer={}, token_endpoint={}, client_id={}, other_audiences={}", - session.provider_id, - config.config_url, - issuer, - token_endpoint, - config.client_id, - config.other_audiences.join(",") - ) - })?; - } - - // Extract raw claims from the verified JWT for custom claim support - // (the crate verifies signature/expiry/nonce; we decode payload for non-standard claims) - let raw_jwt = id_token.to_string(); - let raw = decode_jwt_payload(&raw_jwt); - - let claims = OidcClaims { - sub: extract_string_claim(&raw, "sub"), - email: extract_string_claim(&raw, &config.email_claim), - username: extract_string_claim(&raw, &config.username_claim), - groups: extract_canonical_group_values(&raw, &config.groups_claim, &config.roles_claim), - raw, - }; - - Ok((claims, session.provider_id.clone(), session, raw_jwt)) - } - - /// Store a one-time logout session keyed by an opaque token so the console can - /// trigger browser logout without persisting the raw ID token. - pub async fn create_logout_token(&self, provider_id: &str, id_token: &str) -> Result { - if !self.configs.contains_key(provider_id) { - return Err(format!("unknown OIDC provider: {provider_id}")); - } - - let token = CsrfToken::new_random().secret().clone(); - self.state_store - .insert_logout( - token.clone(), - OidcLogoutSession { - provider_id: provider_id.to_string(), - id_token: id_token.to_string(), - }, - ) - .await; - - Ok(token) - } - - /// Build the RP-initiated logout URL for a previously issued logout token. - /// Returns `Ok(None)` when the provider does not advertise an end-session endpoint. - pub async fn build_logout_url(&self, logout_token: &str, post_logout_redirect_uri: &str) -> Result, String> { - let session = self - .state_store - .take_logout(logout_token) - .await - .ok_or_else(|| "invalid or expired OIDC logout token".to_string())?; - - let config = self - .get_provider_config(&session.provider_id) - .ok_or_else(|| format!("unknown OIDC provider: {}", session.provider_id))?; - let state = self.ensure_provider_state(&session.provider_id, config).await?; - let Some(end_session_endpoint) = state.metadata.console()?.additional_metadata().end_session_endpoint.clone() else { - return Ok(None); - }; - - let id_token: CoreIdToken = session - .id_token - .parse() - .map_err(|e: serde_json::Error| format!("failed to parse ID token for logout: {e}"))?; - let post_logout_redirect_uri = PostLogoutRedirectUrl::new(post_logout_redirect_uri.to_string()) - .map_err(|e| format!("invalid post logout redirect URI: {e}"))?; - - let logout_url = LogoutRequest::from(end_session_endpoint) - .set_id_token_hint(&id_token) - .set_client_id(ClientId::new(config.client_id.clone())) - .set_post_logout_redirect_uri(post_logout_redirect_uri) - .http_get_url() - .to_string(); - - Ok(Some(logout_url)) - } - - /// Map OIDC claims to rustfs policy names. - pub fn map_claims_to_policies(&self, provider_id: &str, claims: &OidcClaims) -> (Vec, Vec) { - let Some(rule) = self.authorization_rule(provider_id) else { - return (Vec::new(), Vec::new()); - }; - CoreFederatedAuthorizationMapper::map_policies_and_groups(provider_id, rule, &claims.groups, &claims.raw, true) - } - - /// Policy names produced only by the canonical groups in claim-based mode: the groups claim - /// plus the roles claim values that `extract_canonical_group_values` merges into it. - /// - /// Identity providers emit built-in groups and roles that can never have a matching policy - /// (for example `DOMAIN\Domain Users` or Keycloak's `offline_access`), so the session binding - /// may ignore these names when no such policy exists. Names from a fixed role policy or a - /// dedicated policy claim are never included, so those configurations keep requiring every - /// policy to resolve. - pub fn group_claim_policy_names(&self, provider_id: &str, claims: &OidcClaims) -> Vec { - let Some(rule) = self.authorization_rule(provider_id) else { - return Vec::new(); - }; - CoreFederatedAuthorizationMapper::group_claim_policy_names(rule, &claims.groups, &claims.raw) - } - - fn authorization_rule(&self, provider_id: &str) -> Option> { - self.get_provider_config(provider_id).map(|config| { - FederatedAuthorizationRuleRef::new( - &config.claim_name, - &config.claim_prefix, - &config.role_policy, - &config.groups_claim, - &config.roles_claim, - ) - }) - } - - /// Verify a raw JWT (id_token) for the AssumeRoleWithWebIdentity flow. - /// - /// Unlike the authorization code flow, ARWWI receives a raw JWT directly - /// (not via code exchange). This method: - /// 1. Decodes the JWT payload to extract the `iss` claim - /// 2. Finds the OIDC provider whose issuer matches - /// 3. Verifies signature, issuer, audience, and expiry (nonce is skipped) - /// 4. Extracts claims using the provider's claim configuration - pub async fn verify_web_identity_token(&self, jwt: &str) -> Result<(OidcClaims, String /* provider_id */), String> { - // Decode JWT payload without verification to get the issuer claim - let raw_claims = decode_jwt_payload(jwt); - let issuer = raw_claims - .get("iss") - .and_then(|v| v.as_str()) - .ok_or_else(|| "JWT missing 'iss' claim".to_string())?; - - // Find matching provider by issuer - let (provider_id, config, mut state) = self - .find_provider_by_issuer(issuer) - .ok_or_else(|| format!("no OIDC provider configured for issuer: {issuer}"))?; - - state = self.ensure_provider_state_if_stale(&provider_id, &config, &state).await?; - - // Parse raw JWT string into CoreIdToken - let id_token: CoreIdToken = jwt - .parse() - .map_err(|e: serde_json::Error| format!("failed to parse JWT as ID token: {e}"))?; - - // Verify the token (signature, issuer, audience, expiry) — skip nonce - // (nonce is only required for the authorization code flow) - let verifier = state - .metadata - .verifier(&config) - .set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud)); - if let Err(e) = id_token.claims(&verifier, |_: Option<&Nonce>| Ok(())) { - state = self - .refresh_provider_state(&provider_id, &config) - .await - .map_err(|refresh_err| { - format!("ID token verification failed: {e}; failed to refresh provider metadata: {refresh_err}") - })?; - - let verifier = state - .metadata - .verifier(&config) - .set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud)); - id_token - .claims(&verifier, |_: Option<&Nonce>| Ok(())) - .map_err(|retry_err| format!("ID token verification failed after JWKS refresh: {retry_err}"))?; - } - - // Extract claims using the provider's claim configuration - let claims = OidcClaims { - sub: extract_string_claim(&raw_claims, "sub"), - email: extract_string_claim(&raw_claims, &config.email_claim), - username: extract_string_claim(&raw_claims, &config.username_claim), - groups: extract_canonical_group_values(&raw_claims, &config.groups_claim, &config.roles_claim), - raw: raw_claims, - }; - - Ok((claims, provider_id.to_string())) - } - - /// Find a provider whose discovered issuer matches the given JWT issuer string. - fn find_provider_by_issuer(&self, issuer: &str) -> Option<(String, OidcProviderConfig, ProviderState)> { - let (issuer_scheme, issuer_host, issuer_port, issuer_path) = normalize_issuer(issuer)?; - let map = self - .provider_states - .read() - .map_err(|e| format!("provider state lock poisoned: {e}")) - .ok()?; - for (id, state) in map.iter() { - let provider_issuer = state.metadata.issuer().as_str(); - let Some((provider_scheme, provider_host, provider_port, provider_path)) = normalize_issuer(provider_issuer) else { - continue; - }; - - if issuer_scheme == provider_scheme - && issuer_host == provider_host - && issuer_port == provider_port - && issuer_path == provider_path - && let Some(config) = self.get_provider_config(id) - { - return Some((id.clone(), config.clone(), state.clone())); - } - } - None - } - - fn get_provider_state(&self, provider_id: &str) -> Result { - self.provider_states - .read() - .map_err(|e| format!("provider state lock poisoned: {e}"))? - .get(provider_id) - .cloned() - .ok_or_else(|| format!("provider not discovered: {provider_id}")) - } - - async fn refresh_provider_state(&self, provider_id: &str, config: &OidcProviderConfig) -> Result { - let state = Self::discover_provider(config, &self.http_client).await?; - let mut map = self.provider_states.write().map_err(|e| { - let msg = e.to_string(); - format!("provider state lock poisoned: {msg}") - })?; - map.insert(provider_id.to_string(), state.clone()); - - Ok(state) - } - - async fn ensure_provider_state(&self, provider_id: &str, config: &OidcProviderConfig) -> Result { - let state = self.get_provider_state(provider_id)?; - if state.is_stale() { - self.refresh_provider_state(provider_id, config).await.or_else(|refresh_err| { - warn!( - "OIDC provider '{}' JWKS metadata refresh skipped due to transient network issue: {}", - provider_id, refresh_err - ); - Ok(state) - }) - } else { - Ok(state) - } - } - - async fn ensure_provider_state_if_stale( - &self, - provider_id: &str, - config: &OidcProviderConfig, - state: &ProviderState, - ) -> Result { - if state.is_stale() { - self.refresh_provider_state(provider_id, config).await.or_else(|refresh_err| { - warn!( - "OIDC provider '{}' JWKS metadata refresh skipped due to transient network issue: {}", - provider_id, refresh_err - ); - Ok(state.clone()) - }) - } else { - Ok(state.clone()) - } - } - - /// Get the state store (used by HTTP handlers). - pub fn state_store(&self) -> &OidcStateStore { - &self.state_store - } - - /// Get a provider config by ID. - pub fn get_provider_config(&self, id: &str) -> Option<&OidcProviderConfig> { - self.configs.get(id).map(|provider| &provider.config) - } - - /// Parse all OIDC provider configs from environment variables. - fn parse_env_configs() -> Vec { - let mut configs = Vec::new(); - - // Check for the default provider (no suffix) - if let Some(config) = Self::parse_single_provider("", "default") { - configs.push(config); - } - - // Scan for suffixed providers by checking all OIDC env var prefixes. - // This allows providers to be discovered without requiring a separate ENABLE_ key. - let mut provider_ids: Vec = Vec::new(); - let scan_prefixes: Vec = ENV_IDENTITY_OPENID_KEYS.iter().map(|k| format!("{k}_")).collect(); - for (key, _) in std::env::vars() { - for prefix in &scan_prefixes { - if let Some(suffix) = key.strip_prefix(prefix.as_str()) - && !suffix.is_empty() - && suffix != "default" - { - provider_ids.push(suffix.to_string()); - break; - } - } - } - - provider_ids.sort(); - provider_ids.dedup(); - - for id in provider_ids { - let suffix = format!("_{id}"); - if let Some(config) = Self::parse_single_provider(&suffix, &id) { - configs.push(config); - } - } - - configs - } - - fn parse_persisted_configs(cfg: &ServerConfig) -> Vec { - let Some(subsystem) = cfg.0.get(IDENTITY_OPENID_SUB_SYS) else { - return Vec::new(); - }; - - let mut configs = Vec::new(); - let mut provider_ids: Vec = subsystem.keys().cloned().collect(); - provider_ids.sort(); - - for raw_id in provider_ids { - let Some(kvs) = subsystem.get(&raw_id) else { - continue; - }; - - let id = if raw_id == DEFAULT_DELIMITER { - "default" - } else { - raw_id.as_str() - }; - if let Some(config) = Self::parse_single_persisted_provider(kvs, id) { - configs.push(config); - } - } - - configs - } - - /// Parse a string as an `EnableState` boolean. - /// Returns `default_if_empty` when the input is empty, and `default_on_error` - /// when parsing fails. - fn parse_enable_state(value: &str, default_if_empty: bool, default_on_error: bool) -> bool { - if value.is_empty() { - return default_if_empty; - } - value - .parse::() - .map(|s| s.is_enabled()) - .unwrap_or(default_on_error) - } - - /// Parse a single provider's config from env vars with the given suffix. - fn parse_single_provider(env_suffix: &str, id: &str) -> Option { - let get_env = |base: &str| -> String { std::env::var(format!("{base}{env_suffix}")).unwrap_or_default() }; - - let enable_val = get_env(ENV_IDENTITY_OPENID_ENABLE); - let config_url = get_env(ENV_IDENTITY_OPENID_CONFIG_URL); - let issuer = get_env(ENV_IDENTITY_OPENID_ISSUER); - - // Skip if no config URL - if config_url.is_empty() { - return None; - } - - let enabled = Self::parse_enable_state(&enable_val, true, false); - - let scopes_str = get_env(ENV_IDENTITY_OPENID_SCOPES); - let scopes = if scopes_str.is_empty() { - OIDC_DEFAULT_SCOPES.split(',').map(String::from).collect() - } else { - scopes_str.split(',').map(|s| s.trim().to_string()).collect() - }; - - let other_audiences_str = get_env(ENV_IDENTITY_OPENID_OTHER_AUDIENCES); - let other_audiences = other_audiences_str - .split(',') - .map(|s| s.trim()) - .filter(|s| !s.is_empty()) - .map(|s| s.to_string()) - .collect(); - - let redirect_uri_dynamic = Self::parse_enable_state(&get_env(ENV_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC), true, true); - - let claim_name = { - let v = get_env(ENV_IDENTITY_OPENID_CLAIM_NAME); - if v.is_empty() { - OIDC_DEFAULT_CLAIM_NAME.to_string() - } else { - v - } - }; - let groups_claim = { - let v = get_env(ENV_IDENTITY_OPENID_GROUPS_CLAIM); - if v.is_empty() { - OIDC_DEFAULT_GROUPS_CLAIM.to_string() - } else { - v - } - }; - let roles_claim = get_env(ENV_IDENTITY_OPENID_ROLES_CLAIM); - let email_claim = { - let v = get_env(ENV_IDENTITY_OPENID_EMAIL_CLAIM); - if v.is_empty() { - OIDC_DEFAULT_EMAIL_CLAIM.to_string() - } else { - v - } - }; - let username_claim = { - let v = get_env(ENV_IDENTITY_OPENID_USERNAME_CLAIM); - if v.is_empty() { - OIDC_DEFAULT_USERNAME_CLAIM.to_string() - } else { - v - } - }; - let display_name = { - let v = get_env(ENV_IDENTITY_OPENID_DISPLAY_NAME); - if v.is_empty() { id.to_string() } else { v } - }; - let redirect_uri = { - let v = get_env(ENV_IDENTITY_OPENID_REDIRECT_URI); - if v.is_empty() { None } else { Some(v) } - }; - let client_secret = { - let v = get_env(ENV_IDENTITY_OPENID_CLIENT_SECRET); - if v.is_empty() { None } else { Some(v) } - }; - let hide_from_ui = Self::parse_enable_state(&get_env(ENV_IDENTITY_OPENID_HIDE_FROM_UI), false, false); - - Some(OidcProviderConfig { - id: id.to_string(), - enabled, - config_url, - issuer: if issuer.is_empty() { None } else { Some(issuer) }, - client_id: get_env(ENV_IDENTITY_OPENID_CLIENT_ID), - client_secret, - scopes, - other_audiences, - redirect_uri, - redirect_uri_dynamic, - claim_name, - claim_prefix: get_env(ENV_IDENTITY_OPENID_CLAIM_PREFIX), - role_policy: get_env(ENV_IDENTITY_OPENID_ROLE_POLICY), - display_name, - groups_claim, - roles_claim, - email_claim, - username_claim, - hide_from_ui, - }) - } - - fn parse_single_persisted_provider(kvs: &KVS, id: &str) -> Option { - let config_url = kvs.get(OIDC_CONFIG_URL); - if config_url.is_empty() { - return None; - } - - let enabled = Self::parse_enable_state(&kvs.lookup(ENABLE_KEY).unwrap_or_default(), false, false); - - let scopes_str = kvs.get(OIDC_SCOPES); - let scopes = if scopes_str.is_empty() { - OIDC_DEFAULT_SCOPES.split(',').map(String::from).collect() - } else { - scopes_str.split(',').map(|s| s.trim().to_string()).collect() - }; - - let other_audiences_str = kvs.get(OIDC_OTHER_AUDIENCES); - let other_audiences = other_audiences_str - .split(',') - .map(|s| s.trim()) - .filter(|s| !s.is_empty()) - .map(|s| s.to_string()) - .collect(); - - let redirect_uri_dynamic = - Self::parse_enable_state(&kvs.lookup(OIDC_REDIRECT_URI_DYNAMIC).unwrap_or_default(), true, true); - - let claim_name = kvs - .lookup(OIDC_CLAIM_NAME) - .unwrap_or_else(|| OIDC_DEFAULT_CLAIM_NAME.to_string()); - let groups_claim = kvs - .lookup(OIDC_GROUPS_CLAIM) - .unwrap_or_else(|| OIDC_DEFAULT_GROUPS_CLAIM.to_string()); - let roles_claim = kvs - .lookup(OIDC_ROLES_CLAIM) - .unwrap_or_else(|| OIDC_DEFAULT_ROLES_CLAIM.to_string()); - let email_claim = kvs - .lookup(OIDC_EMAIL_CLAIM) - .unwrap_or_else(|| OIDC_DEFAULT_EMAIL_CLAIM.to_string()); - let username_claim = kvs - .lookup(OIDC_USERNAME_CLAIM) - .unwrap_or_else(|| OIDC_DEFAULT_USERNAME_CLAIM.to_string()); - let display_name = kvs.lookup(OIDC_DISPLAY_NAME).unwrap_or_else(|| id.to_string()); - let redirect_uri = kvs.lookup(OIDC_REDIRECT_URI).filter(|v| !v.is_empty()); - let client_secret = kvs.lookup(OIDC_CLIENT_SECRET).filter(|v| !v.is_empty()); - let hide_from_ui = Self::parse_enable_state(&kvs.lookup(OIDC_HIDE_FROM_UI).unwrap_or_default(), false, false); - - Some(OidcProviderConfig { - id: id.to_string(), - enabled, - config_url, - issuer: kvs.lookup(OIDC_ISSUER).filter(|v| !v.is_empty()), - client_id: kvs.get(OIDC_CLIENT_ID), - client_secret, - scopes, - other_audiences, - redirect_uri, - redirect_uri_dynamic, - claim_name, - claim_prefix: kvs.get(OIDC_CLAIM_PREFIX), - role_policy: kvs.get(OIDC_ROLE_POLICY), - display_name, - groups_claim, - roles_claim, - email_claim, - username_claim, - hide_from_ui, - }) - } - - /// Perform OIDC discovery for a provider. - /// `discover_async` fetches the discovery document and JWKS in one step. - async fn discover_provider(config: &OidcProviderConfig, http_client: &ReqwestHttpClient) -> Result { - if let Some(issuer) = config.issuer.as_deref().filter(|issuer| !issuer.trim().is_empty()) { - return Self::discover_provider_from_config_url(config, issuer, http_client).await; - } - - // The openidconnect crate expects the issuer URL (base), not the - // .well-known/openid-configuration URL. - let base_issuer = normalize_config_url(&config.config_url)?; - let candidates = issuer_candidates(&base_issuer); - let mut last_errors = Vec::new(); - - for candidate_issuer in candidates.iter() { - let issuer_url = IssuerUrl::new(candidate_issuer.clone()).map_err(|e| format!("invalid issuer URL: {e}"))?; - - for attempt in 0..OIDC_DISCOVERY_TRANSPORT_RETRIES { - let discovered = if config.hide_from_ui { - Self::discover_provider_from_config_url(config, candidate_issuer, http_client).await - } else { - let client = JwksAcceptClient { - inner: http_client, - discovery_url: Some( - issuer_url - .join(".well-known/openid-configuration") - .map_err(|err| err.to_string())?, - ), - }; - ProviderMetadataWithLogout::discover_async(issuer_url.clone(), &client) - .await - .map(|metadata| ProviderState { - metadata: DiscoveredProviderMetadata::Console(Box::new(metadata)), - discovered_at: Instant::now(), - }) - .map_err(|err| match err { - DiscoveryError::Request(OidcHttpError::ForbiddenOutbound(reason)) => { - format!("{OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY}: {reason}") - } - err => format!("discovery failed: {err}"), - }) - }; - match discovered { - Ok(state) => return Ok(state), - Err(error) - if error.starts_with(OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY) - || error.starts_with(OIDC_JWKS_BLOCKED_BY_OUTBOUND_POLICY) => - { - return Err(error); - } - Err(error) => { - let is_transient_transport = error.contains("Request failed"); - let should_retry = is_transient_transport && attempt + 1 < OIDC_DISCOVERY_TRANSPORT_RETRIES; - if should_retry { - warn!( - event = EVENT_OIDC_DIAGNOSTICS, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "provider_discovery_transport_retry", - provider_id = %config.id, - config_url = %config.config_url, - issuer_candidate = %candidate_issuer, - attempt = attempt + 1, - max_attempts = OIDC_DISCOVERY_TRANSPORT_RETRIES, - error = %error, - "oidc provider discovery failed" - ); - sleep(OIDC_DISCOVERY_TRANSPORT_RETRY_DELAY).await; - continue; - } - - last_errors.push(format!("issuer '{candidate_issuer}': {error}")); - warn!( - event = EVENT_OIDC_DIAGNOSTICS, - component = LOG_COMPONENT_IAM, - subsystem = LOG_SUBSYSTEM_OIDC, - result = "provider_discovery_candidate_failed", - provider_id = %config.id, - config_url = %config.config_url, - issuer_candidate = %candidate_issuer, - attempt = attempt + 1, - max_attempts = OIDC_DISCOVERY_TRANSPORT_RETRIES, - error = %error, - "oidc provider discovery failed" - ); - break; - } - } - } - } - - Err(format!( - "discovery failed for all issuer variants {:?}: {}", - candidates, - last_errors.join("; ") - )) - } - - async fn discover_provider_from_config_url( - config: &OidcProviderConfig, - issuer: &str, - http_client: &ReqwestHttpClient, - ) -> Result { - let issuer_url = IssuerUrl::new(issuer.trim().to_string()).map_err(|e| format!("invalid issuer URL: {e}"))?; - let explicit_issuer = config.issuer.as_deref().is_some_and(|issuer| !issuer.trim().is_empty()); - let discovery_url = if explicit_issuer { - discovery_url_from_config_url(&config.config_url)? - } else { - issuer_url - .join(".well-known/openid-configuration") - .map_err(|err| err.to_string())? - }; - let request = http::Request::builder() - .uri(discovery_url.to_string()) - .method(http::Method::GET) - .header(http::header::ACCEPT, "application/json") - .body(Vec::new()) - .map_err(|err| format!("failed to prepare discovery request: {err}"))?; - - let response = match http_client.call(request).await { - Ok(response) => response, - Err(OidcHttpError::ForbiddenOutbound(reason)) => { - return Err(format!("{OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY}: {reason}")); - } - Err(err) => return Err(format!("discovery request failed: Request failed: {err}")), - }; - if response.status() != http::StatusCode::OK { - return Err(format!("discovery failed: HTTP status code {} at {}", response.status(), discovery_url)); - } - - if !explicit_issuer - && let Some(content_type) = response.headers().get(http::header::CONTENT_TYPE) - && !content_type.to_str().ok().is_some_and(|value| { - value - .split(';') - .next() - .is_some_and(|essence| essence.eq_ignore_ascii_case("application/json")) - }) - { - return Err("Unexpected response Content-Type: expected application/json".into()); - } - - let provider_metadata = DiscoveredProviderMetadata::parse(response.body(), config.hide_from_ui) - .map_err(|err| format!("failed to parse discovery response: {err}"))?; - if provider_metadata.issuer() != &issuer_url { - return Err(format!( - "unexpected issuer URI `{}` (expected `{}`)", - provider_metadata.issuer().as_str(), - issuer_url.as_str() - )); - } - - let jwks_url = if explicit_issuer { - jwks_url_from_config_url(&config.config_url, &issuer_url, provider_metadata.jwks_uri())? - } else { - provider_metadata.jwks_uri().clone() - }; - let jwks = match CoreJsonWebKeySet::fetch_async( - &jwks_url, - &JwksAcceptClient { - inner: http_client, - discovery_url: None, - }, - ) - .await - { - Ok(jwks) => jwks, - Err(DiscoveryError::Request(OidcHttpError::ForbiddenOutbound(reason))) => { - return Err(format!("{OIDC_JWKS_BLOCKED_BY_OUTBOUND_POLICY}: {reason}")); - } - Err(err) => return Err(format!("failed to fetch JWKS: {err}")), - }; - - Ok(ProviderState { - metadata: provider_metadata.set_jwks(jwks), - discovered_at: Instant::now(), - }) - } -} - -pub fn load_oidc_provider_configs_from_env() -> Vec { - OidcSys::parse_env_configs() -} - -pub fn load_oidc_provider_configs_from_server_config(cfg: &ServerConfig) -> Vec { - OidcSys::parse_persisted_configs(cfg) -} - -pub fn merge_oidc_provider_configs( - env_configs: Vec, - persisted_configs: Vec, -) -> Vec { - let mut effective = HashMap::new(); - - for config in persisted_configs { - effective.insert( - config.id.clone(), - SourcedOidcProviderConfig { - config, - source: OidcProviderConfigSource::Persisted, - }, - ); - } - - for config in env_configs { - effective.insert( - config.id.clone(), - SourcedOidcProviderConfig { - config, - source: OidcProviderConfigSource::Env, - }, - ); - } - - let mut configs: Vec = effective.into_values().collect(); - configs.sort_by(|lhs, rhs| lhs.config.id.cmp(&rhs.config.id)); - configs -} - -pub fn load_effective_oidc_provider_configs(server_config: Option<&ServerConfig>) -> Vec { - let env_configs = load_oidc_provider_configs_from_env(); - let persisted_configs = server_config - .map(load_oidc_provider_configs_from_server_config) - .unwrap_or_default(); - merge_oidc_provider_configs(env_configs, persisted_configs) -} - -pub fn load_oidc_config_snapshot(server_config: Option<&ServerConfig>) -> OidcConfigSnapshot { - OidcConfigSnapshot::new(load_effective_oidc_provider_configs(server_config)) -} - -pub async fn validate_oidc_provider_config(config: &OidcProviderConfig) -> Result { - validate_oidc_provider_config_with_extra_root_ca(config, None).await -} - -pub async fn validate_oidc_provider_config_with_extra_root_ca( - config: &OidcProviderConfig, - root_ca_pem: Option<&[u8]>, -) -> Result { - let http_client = ReqwestHttpClient::new_with_extra_root_certs(oidc_extra_root_certs(root_ca_pem)?)?; - let state = OidcSys::discover_provider(config, &http_client).await?; - - Ok(OidcProviderValidationResult { - issuer: state.metadata.issuer().to_string(), - authorization_endpoint: state.metadata.authorization_endpoint(), - token_endpoint: state.metadata.token_endpoint().map(ToString::to_string), - }) -} - -// --- Helper functions --- - -fn normalize_issuer(raw: &str) -> Option<(String, String, u16, String)> { - let parsed = Url::parse(raw).ok()?; - if parsed.scheme() != "http" && parsed.scheme() != "https" { - return None; - } - - let host = parsed.host_str()?.to_ascii_lowercase(); - let port = parsed.port_or_known_default()?; - let normalized_path = { - let path = parsed.path().trim_end_matches('/').to_string(); - if path.is_empty() { "/".to_string() } else { path } - }; - - Some((parsed.scheme().to_string(), host, port, normalized_path)) -} - -fn normalize_config_url(config_url: &str) -> Result { - let config_url = config_url.trim(); - let url = Url::parse(config_url).map_err(|e| format!("invalid config_url: {e}"))?; - if url.scheme() != "http" && url.scheme() != "https" { - return Err(format!("invalid config_url scheme: {}", url.scheme())); - } - let host = url.host_str().ok_or_else(|| "config_url missing host".to_string())?; - let path = url.path(); - - // Strip `/.well-known/openid-configuration` (with optional trailing slash) if present. - // Everything else is preserved exactly so the issuer URL matches the provider's discovery - // document (e.g. Authentik includes a trailing slash, Keycloak does not). - let normalized_path = path - .strip_suffix('/') - .unwrap_or(path) - .strip_suffix("/.well-known/openid-configuration") - .unwrap_or(if path == "/" { "" } else { path }); - - if normalized_path.contains("/.well-known/") { - return Err("config_url uses an unsupported .well-known discovery URL".into()); - } - - let mut issuer = format!("{}://{host}", url.scheme()); - if let Some(port) = url.port() { - issuer.push(':'); - issuer.push_str(&port.to_string()); - } - - if !normalized_path.is_empty() { - issuer.push_str(normalized_path); - } - - Ok(issuer) -} - -fn discovery_url_from_config_url(config_url: &str) -> Result { - let mut url = Url::parse(config_url.trim()).map_err(|e| format!("invalid config_url: {e}"))?; - if url.scheme() != "http" && url.scheme() != "https" { - return Err(format!("invalid config_url scheme: {}", url.scheme())); - } - if url.host_str().is_none() { - return Err("config_url missing host".to_string()); - } - - let path = url.path().to_string(); - let without_trailing_slash = path.strip_suffix('/').unwrap_or(&path); - if without_trailing_slash.ends_with("/.well-known/openid-configuration") { - url.set_path(without_trailing_slash); - return Ok(url); - } - if without_trailing_slash.contains("/.well-known/") { - return Err("config_url uses an unsupported .well-known discovery URL".into()); - } - - let discovery_path = if without_trailing_slash.is_empty() || without_trailing_slash == "/" { - "/.well-known/openid-configuration".to_string() - } else { - format!("{without_trailing_slash}/.well-known/openid-configuration") - }; - url.set_path(&discovery_path); - Ok(url) -} - -fn jwks_url_from_config_url( - config_url: &str, - issuer_url: &IssuerUrl, - jwks_url: &JsonWebKeySetUrl, -) -> Result { - let issuer = issuer_url.url(); - let jwks = jwks_url.url(); - if issuer.origin() != jwks.origin() { - return Ok(jwks_url.clone()); - } - - let issuer_path = issuer.path().trim_end_matches('/'); - let Some(suffix) = jwks.path().strip_prefix(issuer_path) else { - return Ok(jwks_url.clone()); - }; - if !suffix.is_empty() && !suffix.starts_with('/') { - return Ok(jwks_url.clone()); - } - - let mut internal_url = - Url::parse(&normalize_config_url(config_url)?).map_err(|err| format!("invalid config_url issuer base: {err}"))?; - let internal_path = internal_url.path().trim_end_matches('/'); - internal_url.set_path(&format!("{internal_path}{suffix}")); - internal_url.set_query(jwks.query()); - Ok(JsonWebKeySetUrl::from_url(internal_url)) -} - -fn issuer_candidates(base: &str) -> Vec { - let original = base.trim(); - let mut variants = Vec::with_capacity(2); - variants.push(original.to_string()); - - let toggled = if original.ends_with('/') { - original.trim_end_matches('/').to_string() - } else { - format!("{original}/") - }; - variants.push(toggled); - - variants -} - -/// Decode the payload section of a JWT without validation (token must already be verified). -pub(crate) fn decode_jwt_payload(token: &str) -> HashMap { - let parts: Vec<&str> = token.split('.').collect(); - if parts.len() < 2 { - return HashMap::new(); - } - let payload_bytes = base64_simd::URL_SAFE_NO_PAD.decode_to_vec(parts[1]); - match payload_bytes { - Ok(bytes) => serde_json::from_slice(&bytes).unwrap_or_default(), - Err(_) => HashMap::new(), - } -} - -/// Extract a string claim from raw claims with case-insensitive fallback. -fn extract_string_claim(claims: &HashMap, key: &str) -> String { - match get_claim_case_insensitive(claims, key) { - ClaimLookup::Found(value) => value.as_str().unwrap_or_default().to_string(), - ClaimLookup::Missing | ClaimLookup::Ambiguous => String::new(), - } -} - -/// Extract a groups/array claim from raw claims with case-insensitive fallback. Handles both string arrays and single strings. -fn extract_groups_claim(claims: &HashMap, key: &str) -> Vec { - match get_claim_case_insensitive(claims, key) { - ClaimLookup::Found(serde_json::Value::Array(arr)) => arr.iter().filter_map(|v| v.as_str().map(String::from)).collect(), - ClaimLookup::Found(serde_json::Value::String(s)) => s.split(',').map(|s| s.trim().to_string()).collect(), - _ => vec![], - } -} - -fn extract_canonical_group_values( - claims: &HashMap, - groups_claim: &str, - roles_claim: &str, -) -> Vec { - let mut groups = extract_groups_claim(claims, groups_claim); - if !roles_claim.is_empty() && roles_claim != groups_claim { - groups.extend(extract_groups_claim(claims, roles_claim)); - } - groups.retain(|g| !g.is_empty()); - groups.sort(); - groups.dedup(); - groups -} - -#[cfg(any(test, feature = "test-util"))] -#[doc(hidden)] -pub fn make_test_sys(configs: Vec) -> OidcSys { - let configs = configs - .into_iter() - .map(test_sourced_config) - .map(|provider| (provider.config.id.clone(), provider)) - .collect(); - OidcSys { - configs, - provider_states: RwLock::new(HashMap::new()), - state_store: OidcStateStore::new(), - http_client: ReqwestHttpClient::new().expect("failed to initialize OIDC HTTP clients"), - } -} - -#[cfg(any(test, feature = "test-util"))] -fn test_sourced_config(config: OidcProviderConfig) -> SourcedOidcProviderConfig { - SourcedOidcProviderConfig { - config, - source: OidcProviderConfigSource::Persisted, - } -} - -#[cfg(any(test, feature = "test-util"))] -#[doc(hidden)] -pub fn test_config(id: &str) -> OidcProviderConfig { - OidcProviderConfig { - id: id.to_string(), - enabled: true, - config_url: format!("https://example.com/{id}/.well-known/openid-configuration"), - issuer: None, - client_id: "client-id".to_string(), - client_secret: None, - scopes: vec!["openid".to_string()], - other_audiences: vec![], - redirect_uri: None, - redirect_uri_dynamic: true, - claim_name: "groups".to_string(), - claim_prefix: String::new(), - role_policy: String::new(), - display_name: id.to_string(), - groups_claim: "groups".to_string(), - roles_claim: String::new(), - email_claim: "email".to_string(), - username_claim: "preferred_username".to_string(), - hide_from_ui: false, - } -} - -#[cfg(test)] -mod tests { - use super::*; - use jsonwebtoken::{Algorithm, EncodingKey, Header}; - use rustfs_utils::egress::OutboundDnsPolicyRejection; - use std::sync::atomic::{AtomicUsize, Ordering}; - - #[derive(Clone)] - struct RejectingDnsResolver { - allow_origin_can_recover: bool, - calls: Option>, - } - - impl reqwest::dns::Resolve for RejectingDnsResolver { - fn resolve(&self, name: reqwest::dns::Name) -> reqwest::dns::Resolving { - if let Some(calls) = &self.calls { - calls.fetch_add(1, Ordering::Relaxed); - } - let host = name.as_str().to_string(); - let rejection = OutboundDnsPolicyRejection::new(host, self.allow_origin_can_recover); - Box::pin(async move { Err(std::io::Error::new(std::io::ErrorKind::PermissionDenied, rejection).into()) }) - } - } - - #[test] - fn test_extract_string_claim() { - let mut claims = HashMap::new(); - claims.insert("email".to_string(), serde_json::json!("user@example.com")); - claims.insert("sub".to_string(), serde_json::json!("12345")); - - assert_eq!(extract_string_claim(&claims, "email"), "user@example.com"); - assert_eq!(extract_string_claim(&claims, "sub"), "12345"); - assert_eq!(extract_string_claim(&claims, "missing"), ""); - } - - #[test] - fn format_http_headers_redacts_sensitive_values() { - let mut headers = http::HeaderMap::new(); - headers.insert(http::header::AUTHORIZATION, "Basic Y2xpZW50OnNlY3JldA==".parse().unwrap()); - headers.insert(http::header::CONTENT_TYPE, "application/json".parse().unwrap()); - headers.insert(http::header::COOKIE, "session=super-secret".parse().unwrap()); - - let rendered = format_http_headers(&headers); - - // Sensitive header values never appear; only their length is emitted. - assert!(!rendered.contains("Y2xpZW50OnNlY3JldA=="), "authorization value leaked: {rendered}"); - assert!(!rendered.contains("super-secret"), "cookie value leaked: {rendered}"); - assert!( - rendered.contains("authorization= OidcProviderConfig { - OidcProviderConfig { - id: id.to_string(), - enabled: true, - config_url: config_url.to_string(), - issuer: None, - client_id: "rustfs-oidc-test".to_string(), - client_secret: None, - scopes: vec!["openid".to_string()], - other_audiences: vec![], - redirect_uri: None, - redirect_uri_dynamic: false, - claim_name: "sub".to_string(), - claim_prefix: "oidc".to_string(), - role_policy: String::new(), - display_name: "mock-oidc".to_string(), - groups_claim: "groups".to_string(), - roles_claim: String::new(), - email_claim: "email".to_string(), - username_claim: "username".to_string(), - hide_from_ui: false, - } - } - - fn read_mock_oidc_request(stream: &mut impl std::io::Read) -> String { - let mut request_bytes = Vec::new(); - let mut buffer = [0u8; 4096]; - loop { - match stream.read(&mut buffer) { - Ok(0) => break, - Ok(n) => request_bytes.extend_from_slice(&buffer[..n]), - Err(e) if matches!(e.kind(), std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut) => break, - Err(_) => break, - } - if request_bytes.windows(4).any(|w| w == b"\r\n\r\n") { - break; - } - if request_bytes.len() >= 8192 { - break; - } - } - String::from_utf8_lossy(&request_bytes).into_owned() - } - - fn read_mock_oidc_request_path(stream: &mut impl std::io::Read) -> String { - read_mock_oidc_request(stream) - .lines() - .next() - .unwrap_or("") - .split_whitespace() - .nth(1) - .unwrap_or("") - .to_string() - } - - fn mock_oidc_response(path: &str, discovery_body: &str, expected_jwks_path: &str, jwks_body: &str) -> String { - let (status, body) = if path.ends_with("/.well-known/openid-configuration") { - (200, discovery_body) - } else if path == expected_jwks_path { - (200, jwks_body) - } else { - (404, r#"{"error":"not found"}"#) - }; - - format!( - "HTTP/1.1 {status} {}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", - if status == 200 { "OK" } else { "Not Found" }, - body.len() - ) - } - - fn start_mock_oidc_discovery_server_with_jwks( - build_discovery_issuer: F, - max_requests: usize, - signing_alg: &'static str, - workload: bool, - jwks_response: J, - ) -> Option<(String, std::thread::JoinHandle<()>)> - where - F: Fn(&str) -> (String, String, String) + Send + 'static, - J: Fn(usize) -> String + Send + 'static, - { - use std::io::Write; - use std::net::{Shutdown, TcpListener}; - use std::sync::mpsc; - use std::time::{Duration, Instant}; - - // After the last completed response, exit if no new connection arrives within this window. - // Keep the mock server alive long enough for slower CI/macOS test environments to finish - // discovery + JWKS requests without racing the shutdown timer. - const IDLE_SHUTDOWN: Duration = Duration::from_secs(1); - const ABSOLUTE_CAP: Duration = Duration::from_secs(5); - - let listener = match TcpListener::bind("127.0.0.1:0") { - Ok(listener) => listener, - Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return None, - Err(err) => panic!("test listener should bind: {err}"), - }; - let base = format!("http://{}", listener.local_addr().expect("listener local address should be available")); - let (discovery_issuer, discovery_jwks_uri, expected_jwks_path) = build_discovery_issuer(&base); - let mut discovery_document = serde_json::json!({ - "issuer": discovery_issuer, - "authorization_endpoint": format!("{base}/authorize"), - "token_endpoint": format!("{base}/token"), - "jwks_uri": discovery_jwks_uri, - "response_types_supported": ["code"], - "response_modes_supported": ["query"], - "subject_types_supported": ["public"], - "id_token_signing_alg_values_supported": [signing_alg], - }); - if workload { - let fields = discovery_document.as_object_mut().expect("mock metadata is an object"); - fields.remove("authorization_endpoint"); - fields.remove("token_endpoint"); - fields.insert("response_types_supported".into(), serde_json::json!(["id_token"])); - } - let discovery_body = discovery_document.to_string(); - let (ready_tx, ready_rx) = mpsc::channel(); - - let handle = std::thread::spawn(move || { - listener - .set_nonblocking(true) - .expect("failed to set discovery mock listener non-blocking"); - let _ = ready_tx.send(()); - - let mut seen = 0usize; - let mut jwks_fetches = 0usize; - let start = Instant::now(); - let mut last_completed = Instant::now(); - - loop { - if seen > 0 && last_completed.elapsed() >= IDLE_SHUTDOWN { - break; - } - if start.elapsed() >= ABSOLUTE_CAP { - break; - } - - let mut stream = match listener.accept() { - Ok((stream, _)) => stream, - Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => { - std::thread::sleep(Duration::from_millis(5)); - continue; - } - Err(_) => break, - }; - stream - .set_nonblocking(false) - .expect("failed to set discovery mock stream blocking"); - - seen += 1; - stream - .set_nonblocking(false) - .expect("failed to set discovery mock stream blocking"); - stream - .set_read_timeout(Some(Duration::from_secs(1))) - .expect("failed to set discovery mock read timeout"); - - let request = read_mock_oidc_request(&mut stream); - let path = request - .lines() - .next() - .and_then(|line| line.split_whitespace().nth(1)) - .unwrap_or(""); - let jwks_body = jwks_response(jwks_fetches); - if path == expected_jwks_path { - jwks_fetches += 1; - } - let mut response = mock_oidc_response(path, &discovery_body, &expected_jwks_path, &jwks_body); - if path.contains("/.well-known/openid-configuration") { - assert!( - request - .lines() - .filter_map(|line| line.split_once(':')) - .any(|(name, value)| { name.eq_ignore_ascii_case("accept") && value.trim() == "application/json" }), - "discovery Accept must remain unchanged" - ); - } - if path == expected_jwks_path { - let expected_type = if workload { - "application/jwk-set+json" - } else { - "application/json" - }; - let accepts_type = request.lines().filter_map(|line| line.split_once(':')).any(|(name, value)| { - name.eq_ignore_ascii_case("accept") && value.split(',').any(|item| item.trim() == expected_type) - }); - if !accepts_type { - response = "HTTP/1.1 406 Not Acceptable\r\nContent-Length: 0\r\nConnection: close\r\n\r\n".into(); - } else if workload { - response = response.replace("Content-Type: application/json", "Content-Type: application/jwk-set+json"); - } - } - let _ = stream.write_all(response.as_bytes()); - let _ = stream.flush(); - let _ = stream.shutdown(Shutdown::Both); - last_completed = Instant::now(); - - if seen >= max_requests { - break; - } - } - }); - ready_rx - .recv_timeout(Duration::from_millis(100)) - .expect("mock OIDC discovery server should become ready"); - - Some((base, handle)) - } - - fn start_mock_oidc_discovery_server( - build_discovery_issuer: F, - max_requests: usize, - ) -> Option<(String, std::thread::JoinHandle<()>)> - where - F: Fn(&str) -> (String, String, String) + Send + 'static, - { - start_mock_oidc_discovery_server_with_jwks(build_discovery_issuer, max_requests, "RS256", false, |_| { - r#"{"keys":[]}"#.to_string() - }) - } - - fn oidc_es256_key_and_jwk(kid: &str) -> (EncodingKey, serde_json::Value) { - let certified = - rcgen::generate_simple_self_signed(vec![format!("{kid}.invalid")]).expect("OIDC signing key should generate"); - let encoding_key = - EncodingKey::from_ec_pem(certified.signing_key.serialize_pem().as_bytes()).expect("OIDC signing key should encode"); - let mut jwk = jsonwebtoken::jwk::Jwk::from_encoding_key(&encoding_key, Algorithm::ES256) - .expect("OIDC public JWK should derive from signing key"); - jwk.common.key_id = Some(kid.to_string()); - jwk.common.public_key_use = Some(jsonwebtoken::jwk::PublicKeyUse::Signature); - (encoding_key, serde_json::to_value(jwk).expect("OIDC JWK should serialize")) - } - - #[tokio::test] - async fn web_identity_verification_refreshes_rotated_jwks() { - let (_, initial_jwk) = oidc_es256_key_and_jwk("initial"); - let (rotated_key, rotated_jwk) = oidc_es256_key_and_jwk("rotated"); - let initial_jwks = serde_json::json!({ "keys": [initial_jwk] }).to_string(); - let rotated_jwks = serde_json::json!({ "keys": [rotated_jwk] }).to_string(); - let Some((base, handle)) = start_mock_oidc_discovery_server_with_jwks( - |base| (base.to_string(), format!("{base}/jwks"), "/jwks".to_string()), - 4, - "ES256", - false, - move |fetch| { - if fetch == 0 { - initial_jwks.clone() - } else { - rotated_jwks.clone() - } - }, - ) else { - return; - }; - - let config = build_mocked_oidc_provider_config("rotating", &base); - let policy = OutboundPolicy::from_allowed_origins(&base).expect("loopback origin should be allowed"); - let http_client = ReqwestHttpClient::with_policy(policy); - let state = OidcSys::discover_provider(&config, &http_client) - .await - .expect("initial OIDC discovery should succeed"); - let sys = OidcSys { - configs: HashMap::from([(config.id.clone(), test_sourced_config(config.clone()))]), - provider_states: RwLock::new(HashMap::from([(config.id.clone(), state)])), - state_store: OidcStateStore::new(), - http_client, - }; - - let now = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .expect("system clock should be after Unix epoch") - .as_secs(); - let mut header = Header::new(Algorithm::ES256); - header.kid = Some("rotated".to_string()); - let token = jsonwebtoken::encode( - &header, - &serde_json::json!({ - "iss": base, - "sub": "rotated-user", - "aud": config.client_id, - "iat": now, - "exp": now + 300, - "groups": ["readwrite"], - }), - &rotated_key, - ) - .expect("rotated OIDC token should sign"); - - let (claims, provider_id) = sys - .verify_web_identity_token(&token) - .await - .expect("verification should refresh JWKS and accept the rotated key"); - assert_eq!(provider_id, "rotating"); - assert_eq!(claims.sub, "rotated-user"); - assert_eq!(claims.groups, vec!["readwrite"]); - handle.join().expect("rotating JWKS mock server should exit cleanly"); - } - - #[tokio::test] - async fn complete_provider_console_login_preserves_hidden_and_issuer_modes() { - use tokio::io::{AsyncReadExt, AsyncWriteExt}; - - for hidden in [false, true] { - for explicit in [false, true] { - let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); - let base = format!("http://{}", listener.local_addr().unwrap()); - let issuer = if explicit { - "https://issuer.example.com".to_string() - } else { - base.clone() - }; - let mut config = - build_mocked_oidc_provider_config("console", &format!("{base}/.well-known/openid-configuration")); - config.hide_from_ui = hidden; - config.issuer = explicit.then(|| issuer.clone()); - config.client_secret = Some(Nonce::new_random().secret().clone()); - let server_config = config.clone(); - let server_base = base.clone(); - let redirect = "https://console.example.com/oauth_callback"; - let (key, jwk) = oidc_es256_key_and_jwk("console"); - let (auth_tx, auth_rx) = tokio::sync::oneshot::channel::>(); - let server = tokio::spawn(async move { - let mut auth_rx = Some(auth_rx); - for expected_path in ["/.well-known/openid-configuration", "/jwks", "/token"] { - let (mut stream, _) = tokio::time::timeout(StdDuration::from_secs(10), listener.accept()) - .await - .unwrap() - .unwrap(); - let mut bytes = Vec::new(); - let header_end = loop { - bytes.push(stream.read_u8().await.unwrap()); - assert!(bytes.len() < 8192); - if bytes.ends_with(b"\r\n\r\n") { - break bytes.len(); - } - }; - let headers = String::from_utf8(bytes).unwrap(); - let request_line = headers.lines().next().unwrap(); - assert_eq!(request_line.split_whitespace().nth(1), Some(expected_path)); - let headers_map: HashMap<_, _> = headers - .lines() - .skip(1) - .filter_map(|line| line.split_once(':')) - .map(|(name, value)| (name.to_ascii_lowercase(), value.trim().to_string())) - .collect(); - let body = match expected_path { - "/.well-known/openid-configuration" => { - assert!(request_line.starts_with("GET ")); - assert_eq!(headers_map["accept"], "application/json"); - serde_json::json!({ - "issuer": issuer, "authorization_endpoint": format!("{server_base}/authorize"), - "token_endpoint": format!("{server_base}/token"), "jwks_uri": format!("{server_base}/jwks"), - "response_types_supported": ["code"], "subject_types_supported": ["public"], - "id_token_signing_alg_values_supported": ["ES256"] - }) - } - "/jwks" => { - assert!(request_line.starts_with("GET ")); - assert!(headers_map["accept"].contains("application/json")); - serde_json::json!({"keys": [jwk]}) - } - "/token" => { - assert!(request_line.starts_with("POST ")); - assert_eq!(headers_map["accept"], "application/json"); - assert!(headers_map["content-type"].starts_with("application/x-www-form-urlencoded")); - let length: usize = headers_map["content-length"].parse().unwrap(); - assert!(header_end + length < 16384); - let mut body = vec![0; length]; - stream.read_exact(&mut body).await.unwrap(); - let form: HashMap = url::form_urlencoded::parse(&body).into_owned().collect(); - assert_eq!(form["grant_type"], "authorization_code"); - assert_eq!(form["code"], "test-authorization-code"); - assert_eq!(form["client_id"], server_config.client_id); - assert_eq!(Some(&form["client_secret"]), server_config.client_secret.as_ref()); - assert_eq!(form["redirect_uri"], redirect); - let auth = auth_rx.take().unwrap().await.unwrap(); - let challenge = PkceCodeChallenge::from_code_verifier_sha256(&PkceCodeVerifier::new(form["code_verifier"].clone())); - assert_eq!(challenge.as_str(), auth["code_challenge"]); - let now = time::OffsetDateTime::now_utc().unix_timestamp(); - let mut header = Header::new(Algorithm::ES256); - header.kid = Some("console".into()); - let token = jsonwebtoken::encode(&header, &serde_json::json!({ - "iss": issuer, "sub": "existing-user", "aud": server_config.client_id, - "iat": now, "exp": now + 300, "nonce": auth["nonce"], - "email": "user@example.com", "groups": ["readwrite"] - }), &key).unwrap(); - serde_json::json!({"access_token": "test-access-token", "token_type": "Bearer", "id_token": token}) - } - _ => unreachable!(), - }.to_string(); - stream.write_all(format!("HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", body.len()).as_bytes()).await.unwrap(); - } - }); - let http_client = ReqwestHttpClient::with_policy(OutboundPolicy::from_allowed_origins(&base).unwrap()); - let discovered = OidcSys::discover_provider(&config, &http_client).await.unwrap(); - let sys = OidcSys { - configs: HashMap::from([(config.id.clone(), test_sourced_config(config))]), - provider_states: RwLock::new(HashMap::from([("console".into(), discovered)])), - state_store: OidcStateStore::new(), - http_client, - }; - let auth_url = sys.authorize_url("console", redirect, Some("/buckets".into())).await.unwrap(); - let auth_url = Url::parse(&auth_url).unwrap(); - assert_eq!(auth_url.as_str().split('?').next(), Some(format!("{base}/authorize").as_str())); - let auth: HashMap = auth_url.query_pairs().into_owned().collect(); - assert_eq!(auth["response_type"], "code"); - assert_eq!(auth["client_id"], "rustfs-oidc-test"); - assert!(!auth["nonce"].is_empty()); - assert!(!auth["state"].is_empty()); - assert_eq!(auth["redirect_uri"], redirect); - assert_eq!(auth["code_challenge_method"], "S256"); - assert!(auth["scope"].split_whitespace().any(|scope| scope == "openid")); - let state = auth["state"].clone(); - auth_tx.send(auth).unwrap(); - let (claims, provider, session, _) = sys - .exchange_code(&state, "test-authorization-code", redirect) - .await - .unwrap_or_else(|err| panic!("hidden={hidden}, explicit={explicit}: {err}")); - assert_eq!(provider, "console"); - assert_eq!(claims.sub, "existing-user"); - assert_eq!(claims.email, "user@example.com"); - assert_eq!(claims.groups, vec!["readwrite"]); - assert_eq!(session.redirect_after.as_deref(), Some("/buckets")); - assert!(matches!(sys.exchange_code(&state, "test-authorization-code", redirect).await, - Err(error) if error == "invalid or expired OIDC state")); - server.await.unwrap(); - } - } - } - - #[test] - fn workload_metadata_requires_hidden_provider_and_valid_verification_fields() { - let document = serde_json::json!({ - "issuer": "https://issuer.example.com", - "jwks_uri": "https://issuer.example.com/jwks", - "response_types_supported": ["id_token"], - "subject_types_supported": ["public"], - "id_token_signing_alg_values_supported": ["ES256"], - }); - let parse = - |value: &serde_json::Value, hidden| DiscoveredProviderMetadata::parse(&serde_json::to_vec(value).unwrap(), hidden); - let metadata = parse(&document, true).expect("hidden workload metadata should parse"); - assert!(metadata.authorization_endpoint().is_none()); - assert!(metadata.console().err().unwrap().contains("only web identity")); - assert!(parse(&document, false).err().unwrap().contains("authorization_endpoint")); - for field in ["issuer", "jwks_uri", "id_token_signing_alg_values_supported"] { - let mut invalid = document.clone(); - invalid.as_object_mut().unwrap().remove(field); - assert!(parse(&invalid, true).err().unwrap().contains(field), "missing {field}"); - } - for endpoint in [serde_json::Value::Null, serde_json::json!(""), serde_json::json!("not a URL")] { - let mut invalid = document.clone(); - invalid["authorization_endpoint"] = endpoint; - assert!(parse(&invalid, true).is_err(), "invalid endpoint must not select workload metadata"); - } - let mut complete = document; - complete["authorization_endpoint"] = serde_json::json!("https://issuer.example.com/authorize"); - for hidden in [true, false] { - let metadata = parse(&complete, hidden).expect("full providers keep the existing parser"); - assert!(metadata.console().is_ok()); - assert!(metadata.token_endpoint().is_none(), "token endpoint remains optional"); - } - } - - #[test] - fn workload_metadata_rejects_duplicate_fields() { - for hidden in [false, true] { - let document = format!( - r#"{{"issuer":"https://wrong.example.com","issuer":"https://issuer.example.com",{}"jwks_uri":"https://issuer.example.com/jwks","response_types_supported":["id_token"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["ES256"]}}"#, - if hidden { - "" - } else { - r#""authorization_endpoint":"https://issuer.example.com/authorize","# - }, - ); - let error = DiscoveredProviderMetadata::parse(document.as_bytes(), hidden) - .err() - .expect("duplicate issuer must fail"); - assert!(error.contains("duplicate field"), "{error}"); - } - } - - #[test] - fn workload_verifier_preserves_algorithm_secret_and_audience_policy() { - let secret = Nonce::new_random().secret().to_string(); - let mut config = build_mocked_oidc_provider_config("workload", "https://issuer.example.com"); - config.client_secret = Some(secret.clone()); - config.other_audiences = vec!["additional-audience".into()]; - let now = time::OffsetDateTime::now_utc().unix_timestamp(); - let payload = serde_json::json!({ - "iss": config.config_url, "sub": "repo:example/project:ref:refs/heads/main", - "aud": [config.client_id, "additional-audience"], "iat": now, "exp": now + 300, - }); - let signed = - jsonwebtoken::encode(&Header::new(Algorithm::HS256), &payload, &EncodingKey::from_secret(secret.as_bytes())).unwrap(); - let token: CoreIdToken = signed.parse().unwrap(); - for workload in [false, true] { - for (algorithms, accepted) in [ - (serde_json::json!(["HS256", "unsupported-future-algorithm"]), true), - (serde_json::json!(["ES256"]), false), - (serde_json::json!(["unsupported-future-algorithm"]), false), - ] { - let mut document = serde_json::json!({ - "issuer": config.config_url, "jwks_uri": "https://issuer.example.com/jwks", - "response_types_supported": ["id_token"], "subject_types_supported": ["public"], - "id_token_signing_alg_values_supported": algorithms, - }); - if !workload { - document["authorization_endpoint"] = serde_json::json!("https://issuer.example.com/authorize"); - } - let metadata = DiscoveredProviderMetadata::parse(&serde_json::to_vec(&document).unwrap(), true).unwrap(); - let verifier = metadata - .verifier(&config) - .set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud)); - assert_eq!( - token.claims(&verifier, |_: Option<&Nonce>| Ok(())).is_ok(), - accepted, - "workload={workload}, algorithms={algorithms}" - ); - if accepted { - assert!( - token.claims(&metadata.verifier(&config), |_: Option<&Nonce>| Ok(())).is_err(), - "additional audiences require explicit trust" - ); - let mut wrong_secret = config.clone(); - wrong_secret.client_secret = Some(Nonce::new_random().secret().to_string()); - let verifier = metadata - .verifier(&wrong_secret) - .set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud)); - assert!( - token.claims(&verifier, |_: Option<&Nonce>| Ok(())).is_err(), - "incorrect client secret must fail" - ); - } - } - } - } - - #[tokio::test] - async fn workload_discovery_stops_after_forbidden_jwks() { - let requests = Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let seen = Arc::clone(&requests); - let (base, handle) = start_mock_oidc_discovery_server_with_jwks( - |base| (base.to_string(), "http://192.168.65.254:8080/jwks".into(), "/jwks".into()), - 2, - "ES256", - true, - move |_| { - seen.fetch_add(1, std::sync::atomic::Ordering::SeqCst); - serde_json::json!({"keys": []}).to_string() - }, - ) - .expect("workload discovery mock must bind"); - let mut config = build_mocked_oidc_provider_config("workload", &base); - config.hide_from_ui = true; - let client = ReqwestHttpClient::with_policy(OutboundPolicy::from_allowed_origins(&base).unwrap()); - let error = OidcSys::discover_provider(&config, &client) - .await - .err() - .expect("private JWKS must be blocked"); - assert!(error.starts_with(OIDC_JWKS_BLOCKED_BY_OUTBOUND_POLICY), "{error}"); - handle.join().unwrap(); - assert_eq!( - requests.load(std::sync::atomic::Ordering::SeqCst), - 1, - "a policy denial must not retry discovery" - ); - } - - #[tokio::test] - async fn workload_config_validation_reports_absent_console_endpoints() { - let (base, handle) = start_mock_oidc_discovery_server_with_jwks( - |base| (base.to_string(), format!("{base}/jwks"), "/jwks".into()), - 2, - "ES256", - true, - |_| serde_json::json!({"keys": []}).to_string(), - ) - .expect("workload discovery mock must bind"); - let mut config = build_mocked_oidc_provider_config("workload", &base); - config.hide_from_ui = true; - // Inferred issuers keep the library's discovery URL construction. - config.config_url = format!("{base}/.well-known/openid-configuration?ignored=1#ignored"); - let result = validate_mocked_oidc_provider_config(&config) - .await - .expect("hidden workload configuration should validate"); - assert_eq!(result.issuer, base); - assert!(result.authorization_endpoint.is_none()); - assert!(result.token_endpoint.is_none()); - handle.join().unwrap(); - } - - #[tokio::test] - async fn workload_discovery_verification_and_rotation() { - for explicit_issuer in [false, true] { - let (_, initial_jwk) = oidc_es256_key_and_jwk("initial"); - let (key, rotated_jwk) = oidc_es256_key_and_jwk("rotated"); - let initial_jwks = serde_json::json!({"keys": [initial_jwk]}).to_string(); - let rotated_jwks = serde_json::json!({"keys": [rotated_jwk]}).to_string(); - let (base, handle) = start_mock_oidc_discovery_server_with_jwks( - |base| (base.to_string(), format!("{base}/jwks"), "/jwks".into()), - 4, - "ES256", - true, - move |fetch| { - if fetch == 0 { - initial_jwks.clone() - } else { - rotated_jwks.clone() - } - }, - ) - .expect("workload discovery mock must bind"); - let mut config = build_mocked_oidc_provider_config("workload", &base); - config.hide_from_ui = true; - if explicit_issuer { - config.issuer = Some(base.clone()); - } - let http_client = ReqwestHttpClient::with_policy(OutboundPolicy::from_allowed_origins(&base).unwrap()); - let state = OidcSys::discover_provider(&config, &http_client) - .await - .expect("workload discovery must succeed"); - assert!(state.metadata.authorization_endpoint().is_none()); - let sys = OidcSys { - configs: HashMap::from([(config.id.clone(), test_sourced_config(config.clone()))]), - provider_states: RwLock::new(HashMap::from([(config.id.clone(), state)])), - state_store: OidcStateStore::new(), - http_client, - }; - assert!(sys.has_providers()); - assert!(sys.provider_configs().all(|provider| provider.hide_from_ui)); - let error = sys - .authorize_url(&config.id, "https://console.example.com/callback", None) - .await - .unwrap_err(); - assert!(error.contains("only web identity")); - let now = time::OffsetDateTime::now_utc().unix_timestamp(); - let payload = serde_json::json!({ - "iss": base, "sub": "system:serviceaccount:default:reader", "aud": [config.client_id], - "iat": now, "exp": now + 300, "groups": ["readonly"], - "kubernetes.io": {"namespace": "default", "serviceaccount": {"name": "reader"}}, - }); - let mut header = Header::new(Algorithm::ES256); - header.kid = Some("rotated".into()); - let token = jsonwebtoken::encode(&header, &payload, &key).unwrap(); - let (claims, provider) = sys - .verify_web_identity_token(&token) - .await - .expect("rotation must retain workload discovery support"); - assert_eq!(provider, config.id); - assert_eq!(claims.sub, "system:serviceaccount:default:reader"); - assert_eq!(claims.groups, ["readonly"]); - // Repeat after the mock exits: the verified snapshot must be cached. - handle.join().unwrap(); - assert!(sys.verify_web_identity_token(&token).await.is_ok()); - for (field, value, expected) in [ - ("iss", serde_json::json!("https://wrong.example.com"), "issuer"), - ("aud", serde_json::json!("wrong-audience"), "audience"), - ("exp", serde_json::json!(now - 60), "expired"), - ] { - let mut invalid = payload.clone(); - invalid[field] = value; - let token = jsonwebtoken::encode(&header, &invalid, &key).unwrap(); - let error = sys - .verify_web_identity_token(&token) - .await - .expect_err("invalid workload token must fail"); - assert!(error.to_lowercase().contains(expected), "{field}: {error}"); - } - let (wrong_key, _) = oidc_es256_key_and_jwk("wrong"); - let invalid = jsonwebtoken::encode(&header, &payload, &wrong_key).unwrap(); - let error = sys - .verify_web_identity_token(&invalid) - .await - .expect_err("wrong signature must fail"); - assert!(error.to_lowercase().contains("signature"), "{error}"); - assert!( - sys.verify_web_identity_token(&token).await.is_ok(), - "failed refresh must preserve the cached keys" - ); - } - } - - fn start_mock_oidc_tls_discovery_server( - build_discovery_issuer: F, - max_requests: usize, - ) -> Option<(String, String, std::thread::JoinHandle<()>)> - where - F: Fn(&str) -> (String, String, String) + Send + 'static, - { - use std::io::Write; - use std::net::{Shutdown, TcpListener}; - use std::sync::mpsc; - use std::time::{Duration, Instant}; - - const IDLE_SHUTDOWN: Duration = Duration::from_secs(1); - const ABSOLUTE_CAP: Duration = Duration::from_secs(5); - - let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); - let certified = - rcgen::generate_simple_self_signed(vec!["127.0.0.1".to_string()]).expect("generate OIDC TLS test certificate"); - let cert_pem = certified.cert.pem(); - let server_config = rustls::ServerConfig::builder() - .with_no_client_auth() - .with_single_cert( - vec![certified.cert.der().clone()], - rustls_pki_types::PrivateKeyDer::try_from(certified.signing_key.serialize_der()) - .expect("convert OIDC TLS test private key"), - ) - .expect("build OIDC TLS mock server config"); - - let listener = match TcpListener::bind("127.0.0.1:0") { - Ok(listener) => listener, - Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return None, - Err(err) => panic!("test TLS listener should bind: {err}"), - }; - let base = format!("https://{}", listener.local_addr().expect("listener local address should be available")); - let (discovery_issuer, discovery_jwks_uri, expected_jwks_path) = build_discovery_issuer(&base); - let discovery_body = serde_json::json!({ - "issuer": discovery_issuer, - "authorization_endpoint": format!("{base}/authorize"), - "token_endpoint": format!("{base}/token"), - "jwks_uri": discovery_jwks_uri, - "response_types_supported": ["code"], - "response_modes_supported": ["query"], - "subject_types_supported": ["public"], - "id_token_signing_alg_values_supported": ["RS256"], - }) - .to_string(); - let jwks_body = r#"{"keys":[]}"#; - let (ready_tx, ready_rx) = mpsc::channel(); - - let handle = std::thread::spawn(move || { - let server_config = Arc::new(server_config); - listener - .set_nonblocking(true) - .expect("failed to set TLS discovery mock listener non-blocking"); - let _ = ready_tx.send(()); - - let mut seen = 0usize; - let start = Instant::now(); - let mut last_completed = Instant::now(); - - loop { - if seen > 0 && last_completed.elapsed() >= IDLE_SHUTDOWN { - break; - } - if start.elapsed() >= ABSOLUTE_CAP { - break; - } - - let tcp_stream = match listener.accept() { - Ok((stream, _)) => stream, - Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => { - std::thread::sleep(Duration::from_millis(5)); - continue; - } - Err(_) => break, - }; - tcp_stream - .set_nonblocking(false) - .expect("failed to set TLS discovery mock stream blocking"); - tcp_stream - .set_read_timeout(Some(Duration::from_secs(1))) - .expect("failed to set TLS discovery mock read timeout"); - - seen += 1; - let connection = match rustls::ServerConnection::new(server_config.clone()) { - Ok(connection) => connection, - Err(_) => break, - }; - let mut stream = rustls::StreamOwned::new(connection, tcp_stream); - let path = read_mock_oidc_request_path(&mut stream); - let response = mock_oidc_response(&path, &discovery_body, &expected_jwks_path, jwks_body); - let _ = stream.write_all(response.as_bytes()); - let _ = stream.flush(); - let _ = stream.sock.shutdown(Shutdown::Both); - last_completed = Instant::now(); - - if seen >= max_requests { - break; - } - } - }); - ready_rx - .recv_timeout(Duration::from_millis(100)) - .expect("mock TLS OIDC discovery server should become ready"); - - Some((base, cert_pem, handle)) - } - - fn discovery_error_contains_all_variants(err: &str, base: &str) -> bool { - err.contains(base) && err.contains(&format!("{base}/")) && err.contains("discovery failed for all issuer variants") - } - - async fn validate_mocked_oidc_provider_config(config: &OidcProviderConfig) -> Result { - // The mock discovery/JWKS/token endpoints share the loopback origin of `config_url`. - // Explicitly allow that origin so the egress policy does not reject the loopback mock. - let origin = Url::parse(&config.config_url) - .map_err(|_| "invalid mock config_url".to_string())? - .origin() - .ascii_serialization(); - let policy = OutboundPolicy::from_allowed_origins(&origin).map_err(|err| err.to_string())?; - let http_client = ReqwestHttpClient::with_policy(policy); - let state = OidcSys::discover_provider(config, &http_client).await?; - - Ok(OidcProviderValidationResult { - issuer: state.metadata.issuer().to_string(), - authorization_endpoint: state.metadata.authorization_endpoint(), - token_endpoint: state.metadata.token_endpoint().map(ToString::to_string), - }) - } - - #[tokio::test] - async fn oidc_discovery_accepts_extra_root_ca_for_https_provider() { - let Some((base, ca_pem, handle)) = start_mock_oidc_tls_discovery_server( - |base| (format!("{base}/application/o/rustfs"), format!("{base}/jwks"), "/jwks".to_string()), - 4, - ) else { - return; - }; - let config_url = format!("{base}/application/o/rustfs"); - let config = build_mocked_oidc_provider_config("default", &config_url); - let origin = Url::parse(&config.config_url) - .expect("mock config_url should parse") - .origin() - .ascii_serialization(); - let policy = OutboundPolicy::from_allowed_origins(&origin).expect("loopback TLS origin should be allowed"); - let extra_root_certs = - parse_oidc_extra_root_certs("test OIDC TLS CA", ca_pem.as_bytes()).expect("test CA bundle should parse"); - let http_client = ReqwestHttpClient::with_policy_and_extra_root_certs(policy, extra_root_certs); - - let state = OidcSys::discover_provider(&config, &http_client) - .await - .expect("OIDC discovery should trust the extra root CA"); - - assert_eq!(state.metadata.issuer().to_string(), format!("{base}/application/o/rustfs")); - assert!(handle.join().is_ok()); - } - - #[tokio::test] - async fn oidc_discovery_refreshes_extra_root_ca_when_generation_changes() { - let Some((base_a, ca_pem_a, handle_a)) = start_mock_oidc_tls_discovery_server( - |base| (format!("{base}/application/o/rustfs-a"), format!("{base}/jwks"), "/jwks".to_string()), - 4, - ) else { - return; - }; - let Some((base_b, ca_pem_b, handle_b)) = start_mock_oidc_tls_discovery_server( - |base| (format!("{base}/application/o/rustfs-b"), format!("{base}/jwks"), "/jwks".to_string()), - 4, - ) else { - assert!(handle_a.join().is_ok()); - return; - }; - - let origin_a = Url::parse(&base_a) - .expect("mock base A should parse") - .origin() - .ascii_serialization(); - let origin_b = Url::parse(&base_b) - .expect("mock base B should parse") - .origin() - .ascii_serialization(); - let allowed_origins = format!("{origin_a},{origin_b}"); - let policy = OutboundPolicy::from_allowed_origins(&allowed_origins).expect("loopback TLS origins should be allowed"); - let material = Arc::new(Mutex::new(OidcExtraRootCaMaterial { - generation: 1, - root_ca_pem: Some(ca_pem_a.into_bytes()), - })); - let provider = OidcExtraRootCaProvider::new({ - let material = material.clone(); - move || { - let material = material.clone(); - async move { - material - .lock() - .map(|material| material.clone()) - .map_err(|e| format!("test OIDC extra CA material lock poisoned: {e}")) - } - } - }); - let http_client = ReqwestHttpClient::with_policy_and_extra_root_ca_provider(policy, provider); - - let config_a = build_mocked_oidc_provider_config("a", &format!("{base_a}/application/o/rustfs-a")); - let state_a = OidcSys::discover_provider(&config_a, &http_client) - .await - .expect("OIDC discovery should trust initial extra root CA"); - assert_eq!(state_a.metadata.issuer().to_string(), format!("{base_a}/application/o/rustfs-a")); - - { - let mut material = material - .lock() - .expect("test OIDC extra CA material lock should not be poisoned"); - material.generation = 2; - material.root_ca_pem = Some(ca_pem_b.into_bytes()); - } - let config_b = build_mocked_oidc_provider_config("b", &format!("{base_b}/application/o/rustfs-b")); - let state_b = OidcSys::discover_provider(&config_b, &http_client) - .await - .expect("OIDC discovery should refresh extra root CA after generation change"); - - assert_eq!(state_b.metadata.issuer().to_string(), format!("{base_b}/application/o/rustfs-b")); - assert!(handle_a.join().is_ok()); - assert!(handle_b.join().is_ok()); - } - - #[tokio::test] - async fn test_validate_oidc_provider_config_retries_with_issuer_candidates() { - // Discovery document must advertise the canonical issuer path. The first candidate has no - // trailing slash; openidconnect rejects issuer mismatch, then the second variant succeeds. - let Some((base, handle)) = start_mock_oidc_discovery_server( - |base| (format!("{base}/application/o/rustfs/"), format!("{base}/jwks"), "/jwks".to_string()), - 8, - ) else { - return; - }; - let config_url = format!("{base}/application/o/rustfs"); - let config = build_mocked_oidc_provider_config("default", &config_url); - - let result = validate_mocked_oidc_provider_config(&config).await; - - let validation_result = result.expect("OIDC provider validation should succeed"); - assert_eq!(validation_result.issuer, format!("{base}/application/o/rustfs/")); - assert!(handle.join().is_ok()); - } - - #[tokio::test] - async fn test_validate_oidc_provider_config_fetches_issuer_relative_jwks_from_config_url() { - let Some((base, handle)) = start_mock_oidc_discovery_server( - |_| { - ( - "http://127.0.0.1:1/public/realms/app".to_string(), - "http://127.0.0.1:1/public/realms/app/jwks?version=1".to_string(), - "/internal/realms/app/jwks?version=1".to_string(), - ) - }, - 2, - ) else { - return; - }; - let mut config = - build_mocked_oidc_provider_config("default", &format!("{base}/internal/realms/app/.well-known/openid-configuration")); - config.issuer = Some("http://127.0.0.1:1/public/realms/app".to_string()); - - let validation_result = validate_mocked_oidc_provider_config(&config) - .await - .expect("OIDC provider validation should succeed"); - - assert_eq!(validation_result.issuer, "http://127.0.0.1:1/public/realms/app"); - assert!(handle.join().is_ok()); - } - - #[test] - fn test_jwks_url_from_config_url_preserves_unrelated_urls() { - let issuer = IssuerUrl::new("https://public.example.com/realms/app".to_string()).expect("issuer URL should parse"); - for raw_jwks_url in [ - "https://keys.example.com/jwks", - "http://public.example.com/realms/app/jwks", - "https://public.example.com:8443/realms/app/jwks", - "https://public.example.com/keys/jwks", - "https://public.example.com/realms/application/jwks", - ] { - let jwks_url = JsonWebKeySetUrl::new(raw_jwks_url.to_string()).expect("JWKS URL should parse"); - - let resolved = jwks_url_from_config_url( - "http://keycloak.internal/realms/app/.well-known/openid-configuration", - &issuer, - &jwks_url, - ) - .expect("JWKS URL should resolve"); - - assert_eq!(resolved.as_str(), raw_jwks_url); - } - - let issuer_root_jwks = JsonWebKeySetUrl::new(issuer.as_str().to_string()).expect("JWKS URL should parse"); - let resolved = jwks_url_from_config_url( - "http://keycloak.internal/realms/app/.well-known/openid-configuration", - &issuer, - &issuer_root_jwks, - ) - .expect("issuer-root JWKS URL should resolve"); - assert_eq!(resolved.as_str(), "http://keycloak.internal/realms/app"); - - let issuer_with_slash = - IssuerUrl::new("https://public.example.com/realms/app/".to_string()).expect("issuer URL should parse"); - let jwks_url = - JsonWebKeySetUrl::new("https://public.example.com/realms/app/jwks".to_string()).expect("JWKS URL should parse"); - let resolved = jwks_url_from_config_url( - "http://keycloak.internal/realms/app/.well-known/openid-configuration", - &issuer_with_slash, - &jwks_url, - ) - .expect("issuer-relative JWKS URL should resolve"); - assert_eq!(resolved.as_str(), "http://keycloak.internal/realms/app/jwks"); - } - - #[tokio::test] - async fn test_validate_oidc_provider_config_rejects_separate_issuer_mismatch() { - let Some((base, handle)) = start_mock_oidc_discovery_server( - |base| { - ( - "https://public.example.com/realms/other".to_string(), - format!("{base}/jwks"), - "/jwks".to_string(), - ) - }, - 1, - ) else { - return; - }; - let mut config = - build_mocked_oidc_provider_config("default", &format!("{base}/internal/realms/app/.well-known/openid-configuration")); - config.issuer = Some("https://public.example.com/realms/app".to_string()); - - let err = validate_mocked_oidc_provider_config(&config) - .await - .expect_err("OIDC provider validation should fail"); - - assert!(err.contains("unexpected issuer URI")); - assert!(err.contains("https://public.example.com/realms/app")); - assert!(handle.join().is_ok()); - } - - #[tokio::test] - async fn test_validate_oidc_provider_config_returns_detailed_errors() { - let Some((base, handle)) = start_mock_oidc_discovery_server( - |base| (format!("{base}/application/o/other"), format!("{base}/jwks"), "/jwks".to_string()), - 8, - ) else { - return; - }; - let config_url = format!("{base}/application/o/rustfs"); - let config = build_mocked_oidc_provider_config("default", &config_url); - - let err = validate_mocked_oidc_provider_config(&config) - .await - .expect_err("OIDC provider validation should fail"); - assert!(discovery_error_contains_all_variants(&err, &base)); - assert!(err.contains("issuer '")); - assert!(err.contains(&format!("issuer '{base}/application/o/rustfs'"))); - assert!(err.contains(&format!("issuer '{base}/application/o/rustfs/'"))); - assert!(handle.join().is_ok()); - } - - #[test] - fn test_decode_jwt_payload_invalid() { - assert!(decode_jwt_payload("not-a-jwt").is_empty()); - assert!(decode_jwt_payload("").is_empty()); - } - - #[test] - fn test_core_token_response_accepts_rfc3339_updated_at() { - // Signature verification happens later; this test covers token response deserialization. - let id_token = "eyJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJodHRwczovL2F1dGguZXhhbXBsZS5jb20vb2lkYyIsInN1YiI6InVzZXItMSIsImF1ZCI6InJ1c3RmcyIsImV4cCI6MTc4NDQzMjc2OSwiaWF0IjoxNzgzMjIzMTY5LCJ1cGRhdGVkX2F0IjoiMjAyNi0wNy0wM1QwNDo1MDo1MC44MTFaIn0.c2ln"; - let body = serde_json::json!({ - "scope": "openid roles profile email", - "token_type": "Bearer", - "access_token": "access-token", - "expires_in": 1209600, - "id_token": id_token, - }) - .to_string(); - - let response: openidconnect::core::CoreTokenResponse = - serde_json::from_str(&body).expect("RFC3339 updated_at should parse in token response"); - - assert!(response.extra_fields().id_token().is_some()); - } - - #[test] - fn test_map_claims_to_policies_no_provider() { - let sys = OidcSys::empty().expect("failed to initialize empty OIDC system"); - - let claims = OidcClaims { - sub: "user123".to_string(), - email: "user@example.com".to_string(), - username: "user".to_string(), - groups: vec!["admin".to_string(), "devs".to_string()], - raw: HashMap::new(), - }; - - let (policies, groups) = sys.map_claims_to_policies("nonexistent", &claims); - assert!(policies.is_empty()); - assert!(groups.is_empty()); - } - - #[test] - fn test_oidc_claims_default() { - let claims = OidcClaims::default(); - assert!(claims.sub.is_empty()); - assert!(claims.email.is_empty()); - assert!(claims.username.is_empty()); - assert!(claims.groups.is_empty()); - assert!(claims.raw.is_empty()); - } - - #[test] - fn test_oidc_claims_serde_roundtrip() { - let claims = OidcClaims { - sub: "user123".to_string(), - email: "user@example.com".to_string(), - username: "testuser".to_string(), - groups: vec!["admin".to_string(), "devs".to_string()], - raw: { - let mut m = HashMap::new(); - m.insert("custom".to_string(), serde_json::json!("value")); - m - }, - }; - - let json = serde_json::to_string(&claims).unwrap(); - let deserialized: OidcClaims = serde_json::from_str(&json).unwrap(); - assert_eq!(deserialized.sub, "user123"); - assert_eq!(deserialized.email, "user@example.com"); - assert_eq!(deserialized.groups.len(), 2); - } - - #[test] - fn test_oidc_provider_summary_serde() { - let summary = OidcProviderSummary { - provider_id: "okta".to_string(), - display_name: "Okta SSO".to_string(), - }; - - let json = serde_json::to_string(&summary).unwrap(); - let deserialized: OidcProviderSummary = serde_json::from_str(&json).unwrap(); - assert_eq!(deserialized.provider_id, "okta"); - assert_eq!(deserialized.display_name, "Okta SSO"); - } - - #[test] - fn test_parse_single_provider_no_config_url() { - let config = OidcSys::parse_single_provider("_TEST_EMPTY", "test_empty"); - assert!(config.is_none()); - } - - #[test] - fn test_parse_single_provider_reads_issuer() { - temp_env::with_vars( - [ - ( - ENV_IDENTITY_OPENID_CONFIG_URL, - Some("http://keycloak.ns.svc.cluster.local:8080/realms/app/.well-known/openid-configuration"), - ), - (ENV_IDENTITY_OPENID_ISSUER, Some("https://app.local/realms/app")), - (ENV_IDENTITY_OPENID_CLIENT_ID, Some("console")), - ], - || { - let config = OidcSys::parse_single_provider("", "default").expect("provider config should parse"); - - assert_eq!(config.issuer.as_deref(), Some("https://app.local/realms/app")); - }, - ); - } - - #[test] - fn test_parse_persisted_provider_config() { - let mut cfg = ServerConfig::new(); - let mut kvs = KVS(vec![ - rustfs_config::server_config::KV { - key: ENABLE_KEY.to_string(), - value: EnableState::Off.to_string(), - hidden_if_empty: false, - }, - rustfs_config::server_config::KV { - key: OIDC_CONFIG_URL.to_string(), - value: String::new(), - hidden_if_empty: false, - }, - rustfs_config::server_config::KV { - key: OIDC_CLIENT_ID.to_string(), - value: String::new(), - hidden_if_empty: false, - }, - ]); - kvs.insert( - OIDC_CONFIG_URL.to_string(), - "https://example.com/.well-known/openid-configuration".to_string(), - ); - kvs.insert(OIDC_CLIENT_ID.to_string(), "console".to_string()); - kvs.insert(ENABLE_KEY.to_string(), EnableState::On.to_string()); - kvs.insert(OIDC_ISSUER.to_string(), "https://issuer.example".to_string()); - kvs.insert(OIDC_ROLES_CLAIM.to_string(), "app_roles".to_string()); - - cfg.0 - .entry(IDENTITY_OPENID_SUB_SYS.to_string()) - .or_default() - .insert(DEFAULT_DELIMITER.to_string(), kvs); - - let parsed = OidcSys::parse_persisted_configs(&cfg); - assert_eq!(parsed.len(), 1); - assert_eq!(parsed[0].id, "default"); - assert_eq!(parsed[0].client_id, "console"); - assert_eq!(parsed[0].issuer.as_deref(), Some("https://issuer.example")); - assert!(parsed[0].enabled); - assert_eq!(parsed[0].roles_claim, "app_roles"); - } - - #[test] - fn test_parse_persisted_provider_config_omitted_roles_claim_is_empty() { - let mut cfg = ServerConfig::new(); - let mut kvs = KVS(vec![ - rustfs_config::server_config::KV { - key: ENABLE_KEY.to_string(), - value: EnableState::Off.to_string(), - hidden_if_empty: false, - }, - rustfs_config::server_config::KV { - key: OIDC_CONFIG_URL.to_string(), - value: String::new(), - hidden_if_empty: false, - }, - rustfs_config::server_config::KV { - key: OIDC_CLIENT_ID.to_string(), - value: String::new(), - hidden_if_empty: false, - }, - ]); - kvs.insert( - OIDC_CONFIG_URL.to_string(), - "https://example.com/.well-known/openid-configuration".to_string(), - ); - kvs.insert(OIDC_CLIENT_ID.to_string(), "console".to_string()); - kvs.insert(ENABLE_KEY.to_string(), EnableState::On.to_string()); - - cfg.0 - .entry(IDENTITY_OPENID_SUB_SYS.to_string()) - .or_default() - .insert(DEFAULT_DELIMITER.to_string(), kvs); - - let parsed = OidcSys::parse_persisted_configs(&cfg); - assert_eq!(parsed.len(), 1); - assert_eq!(parsed[0].roles_claim, ""); - } - - #[test] - fn test_merge_oidc_provider_configs_prefers_env() { - let mut persisted = test_config("default"); - persisted.display_name = "Persisted".to_string(); - - let mut env = test_config("default"); - env.display_name = "Environment".to_string(); - - let merged = merge_oidc_provider_configs(vec![env], vec![persisted]); - assert_eq!(merged.len(), 1); - assert_eq!(merged[0].config.display_name, "Environment"); - assert_eq!(merged[0].source, OidcProviderConfigSource::Env); - } - - #[test] - fn test_oidc_sys_empty() { - let sys = OidcSys::empty().expect("failed to initialize empty OIDC system"); - assert!(!sys.has_providers()); - assert!(sys.config_snapshot().providers().is_empty()); - } - - #[test] - fn build_oidc_http_client_rejects_forbidden_targets_without_allowlist() { - // Cloud metadata endpoint is never allowed. - assert!( - matches!( - build_oidc_http_client("http://169.254.169.254/latest/meta-data/", None, &[], None), - Err(OidcHttpError::ForbiddenOutbound(_)) - ), - "metadata endpoint must be rejected" - ); - // Loopback is rejected by default (no allow-origins configured). - assert!( - matches!( - build_oidc_http_client("http://127.0.0.1:8080/.well-known/openid-configuration", None, &[], None), - Err(OidcHttpError::ForbiddenOutbound(_)) - ), - "loopback must be rejected by default" - ); - // A public hostname passes the up-front shape/host check; the resolved IP is still - // re-classified at connection time by the pinned resolver. - assert!( - build_oidc_http_client("https://accounts.example.com/.well-known/openid-configuration", None, &[], None).is_ok(), - "public https endpoint should build" - ); - } - - #[test] - fn test_should_bypass_proxy_for_oidc_uri_loopback_only() { - assert!(should_bypass_proxy_for_oidc_uri("http://127.0.0.1:9000/.well-known/openid-configuration")); - assert!(should_bypass_proxy_for_oidc_uri("http://localhost:9000/.well-known/openid-configuration")); - assert!(should_bypass_proxy_for_oidc_uri("http://[::1]:9000/.well-known/openid-configuration")); - assert!(!should_bypass_proxy_for_oidc_uri( - "https://idp.example.com/.well-known/openid-configuration" - )); - assert!(!should_bypass_proxy_for_oidc_uri("not-a-url")); - } - - #[test] - fn build_oidc_http_client_honors_explicit_allowlist_for_loopback() { - let policy = OutboundPolicy::from_allowed_origins("http://127.0.0.1:8080").expect("origin should parse"); - assert!( - build_oidc_http_client("http://127.0.0.1:8080/.well-known/openid-configuration", Some(&policy), &[], None).is_ok(), - "explicitly allow-listed loopback origin should build" - ); - // A metadata endpoint stays forbidden even when a loopback origin is allow-listed. - assert!( - matches!( - build_oidc_http_client("http://169.254.169.254/", Some(&policy), &[], None), - Err(OidcHttpError::ForbiddenOutbound(_)) - ), - "metadata endpoint stays forbidden despite an unrelated allow-list entry" - ); - } - - #[tokio::test] - async fn oidc_discovery_reports_forbidden_outbound_without_retrying() { - let config_url = "http://192.168.65.254:8080/realms/rustfs/.well-known/openid-configuration"; - let config = build_mocked_oidc_provider_config("default", config_url); - let http_client = ReqwestHttpClient::with_policy(OutboundPolicy::default()); - - let error = match OidcSys::discover_provider(&config, &http_client).await { - Ok(_) => panic!("private OIDC provider should require an explicit allowlist origin"), - Err(error) => error, - }; - - assert!(error.contains("OIDC provider discovery blocked by outbound policy")); - assert!(error.contains(&format!("add http://192.168.65.254:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); - assert!(!error.contains("discovery failed for all issuer variants")); - } - - #[tokio::test] - async fn oidc_explicit_issuer_reports_forbidden_discovery_endpoint() { - let config_url = "http://192.168.65.254:8080/realms/rustfs/.well-known/openid-configuration"; - let mut config = build_mocked_oidc_provider_config("default", config_url); - config.issuer = Some("https://idp.example.com/realms/rustfs".to_string()); - let http_client = ReqwestHttpClient::with_policy(OutboundPolicy::default()); - - let error = match OidcSys::discover_provider(&config, &http_client).await { - Ok(_) => panic!("private OIDC provider should require an explicit allowlist origin"), - Err(error) => error, - }; - - assert!(error.contains("OIDC provider discovery blocked by outbound policy")); - assert!(error.contains(&format!("add http://192.168.65.254:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); - } - - #[tokio::test] - async fn oidc_explicit_issuer_reports_forbidden_jwks_endpoint() { - let Some((base, handle)) = start_mock_oidc_discovery_server( - |base| { - ( - format!("{base}/realms/rustfs"), - "http://192.168.65.254:8080/realms/rustfs/protocol/openid-connect/certs".to_string(), - "/unused".to_string(), - ) - }, - 1, - ) else { - return; - }; - let mut config = - build_mocked_oidc_provider_config("default", &format!("{base}/realms/rustfs/.well-known/openid-configuration")); - config.issuer = Some(format!("{base}/realms/rustfs")); - let policy = OutboundPolicy::from_allowed_origins(&base).expect("loopback discovery origin should be allowed"); - let http_client = ReqwestHttpClient::with_policy(policy); - - let error = match OidcSys::discover_provider(&config, &http_client).await { - Ok(_) => panic!("private JWKS endpoint should require an explicit allowlist origin"), - Err(error) => error, - }; - - assert!(error.contains("JWKS request blocked by outbound policy")); - assert!(error.contains(&format!("add http://192.168.65.254:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); - assert!(handle.join().is_ok()); - } - - #[tokio::test] - async fn oidc_token_exchange_reports_forbidden_token_endpoint() { - let provider_id = "default"; - let token_endpoint = "http://192.168.65.254:8080/realms/rustfs/protocol/openid-connect/token"; - let metadata = serde_json::from_value::(serde_json::json!({ - "issuer": "https://idp.example.com/realms/rustfs", - "authorization_endpoint": "https://idp.example.com/realms/rustfs/protocol/openid-connect/auth", - "token_endpoint": token_endpoint, - "jwks_uri": "https://idp.example.com/realms/rustfs/protocol/openid-connect/certs", - "response_types_supported": ["code"], - "subject_types_supported": ["public"], - "id_token_signing_alg_values_supported": ["RS256"] - })) - .expect("provider metadata should parse"); - let config = build_mocked_oidc_provider_config( - provider_id, - "https://idp.example.com/realms/rustfs/.well-known/openid-configuration", - ); - let state_store = OidcStateStore::new(); - state_store - .insert( - "test-state".to_string(), - OidcAuthSession { - provider_id: provider_id.to_string(), - pkce_verifier: "test-pkce-verifier".to_string(), - nonce: "test-nonce".to_string(), - redirect_after: None, - }, - ) - .await; - let sys = OidcSys { - configs: HashMap::from([(provider_id.to_string(), test_sourced_config(config))]), - provider_states: RwLock::new(HashMap::from([( - provider_id.to_string(), - ProviderState { - metadata: DiscoveredProviderMetadata::Console(Box::new(metadata)), - discovered_at: Instant::now(), - }, - )])), - state_store, - http_client: ReqwestHttpClient::with_policy(OutboundPolicy::default()), - }; - - let error = match sys - .exchange_code("test-state", "test-code", "https://console.example.com/oauth_callback") - .await - { - Ok(_) => panic!("private token endpoint should require an explicit allowlist origin"), - Err(error) => error, - }; - - assert!(error.contains("request_error_kind=forbidden_outbound")); - assert!(error.contains(&format!("add http://192.168.65.254:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); - } - - #[tokio::test] - async fn oidc_reqwest_dns_policy_rejection_stays_typed() { - let calls = Arc::new(AtomicUsize::new(0)); - let config = build_mocked_oidc_provider_config( - "default", - "http://keycloak.internal:8080/realms/rustfs/.well-known/openid-configuration", - ); - let http_client = ReqwestHttpClient::with_policy_and_dns_resolver( - OutboundPolicy::default(), - Arc::new(RejectingDnsResolver { - allow_origin_can_recover: true, - calls: Some(calls.clone()), - }), - ); - - let error = match OidcSys::discover_provider(&config, &http_client).await { - Ok(_) => panic!("private DNS answer should fail discovery"), - Err(error) => error, - }; - - assert!(error.contains("OIDC provider discovery blocked by outbound policy")); - assert!(error.contains(&format!("add http://keycloak.internal:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); - assert_eq!(calls.load(Ordering::Relaxed), 1, "policy rejection must not be retried"); - } - - #[tokio::test] - async fn oidc_explicit_issuer_preserves_dns_policy_rejection() { - let calls = Arc::new(AtomicUsize::new(0)); - let mut config = build_mocked_oidc_provider_config( - "default", - "http://keycloak.internal:8080/realms/rustfs/.well-known/openid-configuration", - ); - config.issuer = Some("https://idp.example.com/realms/rustfs".to_string()); - let http_client = ReqwestHttpClient::with_policy_and_dns_resolver( - OutboundPolicy::default(), - Arc::new(RejectingDnsResolver { - allow_origin_can_recover: true, - calls: Some(calls.clone()), - }), - ); - - let error = match OidcSys::discover_provider(&config, &http_client).await { - Ok(_) => panic!("private DNS answer should fail discovery"), - Err(error) => error, - }; - - assert!(error.contains("OIDC provider discovery blocked by outbound policy")); - assert!(error.contains(&format!("add http://keycloak.internal:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); - assert_eq!(calls.load(Ordering::Relaxed), 1, "policy rejection must not be retried"); - } - - #[tokio::test] - async fn oidc_nonrecoverable_dns_policy_rejection_omits_allowlist_hint() { - let uri = "http://metadata.internal/latest/meta-data"; - let http_client = ReqwestHttpClient::with_policy_and_dns_resolver( - OutboundPolicy::default(), - Arc::new(RejectingDnsResolver { - allow_origin_can_recover: false, - calls: None, - }), - ); - let request = http::Request::builder() - .uri(uri) - .body(Vec::new()) - .expect("request should build"); - - let error = http_client - .call(request) - .await - .expect_err("metadata DNS answer should be rejected"); - let message = error.to_string(); - - assert!(matches!(error, OidcHttpError::ForbiddenOutbound(_))); - assert!(message.contains("metadata.internal")); - assert!(!message.contains(&format!("add http://metadata.internal to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); - } - - #[test] - fn oidc_metadata_endpoint_rejection_does_not_offer_allowlist_bypass() { - let error = - build_oidc_http_client("http://169.254.169.254/latest/meta-data/", Some(&OutboundPolicy::default()), &[], None) - .expect_err("metadata endpoint must remain forbidden"); - let message = error.to_string(); - - assert!(message.contains("metadata endpoint")); - assert!(!message.contains(&format!("add http://169.254.169.254 to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); - } - - /// Serve exactly `body_len` bytes with no `Content-Length`, so the body ends only at EOF - /// and the size guard cannot rely on an advertised length. - fn start_unbounded_body_server(body_len: usize) -> Option<(String, std::thread::JoinHandle<()>)> { - use std::io::{Read, Write}; - use std::net::TcpListener; - use std::sync::mpsc; - use std::time::Duration; - - let listener = match TcpListener::bind("127.0.0.1:0") { - Ok(listener) => listener, - Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return None, - Err(err) => panic!("test listener should bind: {err}"), - }; - let base = format!("http://{}", listener.local_addr().expect("listener local address should be available")); - let (ready_tx, ready_rx) = mpsc::channel(); - - let handle = std::thread::spawn(move || { - let _ = ready_tx.send(()); - let Ok((mut stream, _)) = listener.accept() else { - return; - }; - let _ = stream.set_read_timeout(Some(Duration::from_secs(1))); - let mut buffer = [0u8; 4096]; - let _ = stream.read(&mut buffer); - let _ = stream.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nConnection: close\r\n\r\n"); - - let chunk = vec![b'a'; 64 * 1024]; - let mut written = 0usize; - while written < body_len { - let take = chunk.len().min(body_len - written); - if stream.write_all(&chunk[..take]).is_err() { - break; - } - written += take; - } - let _ = stream.flush(); - }); - ready_rx - .recv_timeout(Duration::from_millis(100)) - .expect("mock body server should become ready"); - - Some((base, handle)) - } - - async fn fetch_oidc_mock_body(base: &str) -> Result, OidcHttpError> { - let policy = OutboundPolicy::from_allowed_origins(base).expect("origin should parse"); - let client = ReqwestHttpClient::with_policy(policy); - let request = http::Request::builder() - .method(http::Method::GET) - .uri(base) - .body(Vec::new()) - .expect("request should build"); - client.call(request).await.map(http::Response::into_body) - } - - #[tokio::test] - async fn oidc_response_body_at_the_limit_is_accepted() { - let Some((base, handle)) = start_unbounded_body_server(MAX_OIDC_RESPONSE_SIZE) else { - return; - }; - - let body = fetch_oidc_mock_body(&base) - .await - .expect("a body at the limit must be accepted"); - - assert_eq!(body.len(), MAX_OIDC_RESPONSE_SIZE); - handle.join().expect("mock body server thread should exit"); - } - - #[tokio::test] - async fn oidc_response_body_past_the_limit_is_rejected() { - let Some((base, handle)) = start_unbounded_body_server(MAX_OIDC_RESPONSE_SIZE + 1) else { - return; - }; - - let err = fetch_oidc_mock_body(&base) - .await - .map(|body| body.len()) - .expect_err("an oversized provider response must fail closed instead of being buffered"); - - assert!( - matches!(err, OidcHttpError::ResponseTooLarge(MAX_OIDC_RESPONSE_SIZE)), - "unexpected error: {err}" - ); - handle.join().expect("mock body server thread should exit"); - } - - #[test] - fn test_oidc_provider_config_debug_redacts_client_secret() { - let config = OidcProviderConfig { - client_secret: Some("oidc-client-secret".to_string()), - ..test_config("default") - }; - let sourced = SourcedOidcProviderConfig { - config, - source: OidcProviderConfigSource::Persisted, - }; - - let rendered = format!("{sourced:?}"); - - assert!(!rendered.contains("oidc-client-secret")); - assert!(rendered.contains(REDACTED_SECRET)); - assert!(rendered.contains("client-id")); - } - - #[test] - fn test_parse_enable_state_on() { - assert!(OidcSys::parse_enable_state("on", false, false)); - } - - #[test] - fn test_parse_enable_state_off() { - assert!(!OidcSys::parse_enable_state("off", true, true)); - } - - #[test] - fn test_parse_enable_state_empty_returns_default() { - assert!(OidcSys::parse_enable_state("", true, false)); - assert!(!OidcSys::parse_enable_state("", false, true)); - } - - #[test] - fn test_parse_enable_state_invalid_returns_error_default() { - assert!(!OidcSys::parse_enable_state("garbage", true, false)); - assert!(OidcSys::parse_enable_state("garbage", false, true)); - } - - #[test] - fn test_list_visible_providers_hides_hidden_provider() { - let visible = test_config("dex"); - let mut hidden = test_config("kubernetes"); - hidden.hide_from_ui = true; - - let sys = make_test_sys(vec![visible, hidden]); - let listed = sys.list_visible_providers(); - - assert_eq!(listed.len(), 1); - assert!(listed.iter().any(|provider| provider.provider_id == "dex")); - assert!(!listed.iter().any(|provider| provider.provider_id == "kubernetes")); - } - - #[test] - fn test_hidden_provider_still_resolvable_for_sts() { - let visible = test_config("dex"); - let mut hidden = test_config("kubernetes"); - hidden.hide_from_ui = true; - - let sys = make_test_sys(vec![visible, hidden]); - - assert!(sys.get_provider_config("kubernetes").is_some()); - assert!(sys.get_provider_config("dex").is_some()); - } - - #[test] - fn test_config_snapshot_includes_hidden_for_replication() { - let visible = test_config("dex"); - let mut hidden = test_config("kubernetes"); - hidden.hide_from_ui = true; - - let sys = make_test_sys(vec![visible, hidden]); - - assert_eq!(sys.config_snapshot().providers().len(), 2); - } - - #[test] - fn test_list_providers_includes_hidden_for_compatibility() { - let visible = test_config("dex"); - let mut hidden = test_config("kubernetes"); - hidden.hide_from_ui = true; - - let sys = make_test_sys(vec![visible, hidden]); - - assert_eq!(sys.list_providers().len(), 2); - assert_eq!(sys.list_visible_providers().len(), 1); - } - - #[test] - fn test_hide_from_ui_default_is_false() { - let config = test_config("default"); - assert!(!config.hide_from_ui); - } - - #[test] - fn test_parse_persisted_hide_from_ui_off_is_false() { - let mut cfg = ServerConfig::new(); - let mut kvs = KVS(vec![ - rustfs_config::server_config::KV { - key: ENABLE_KEY.to_string(), - value: EnableState::On.to_string(), - hidden_if_empty: false, - }, - rustfs_config::server_config::KV { - key: OIDC_CONFIG_URL.to_string(), - value: "https://example.com/.well-known/openid-configuration".to_string(), - hidden_if_empty: false, - }, - rustfs_config::server_config::KV { - key: OIDC_CLIENT_ID.to_string(), - value: "console".to_string(), - hidden_if_empty: false, - }, - ]); - kvs.insert(OIDC_HIDE_FROM_UI.to_string(), EnableState::Off.to_string()); - - cfg.0 - .entry(IDENTITY_OPENID_SUB_SYS.to_string()) - .or_default() - .insert(DEFAULT_DELIMITER.to_string(), kvs); - - let parsed = OidcSys::parse_persisted_configs(&cfg); - assert_eq!(parsed.len(), 1); - assert!(!parsed[0].hide_from_ui); - } - - #[test] - fn test_parse_persisted_hide_from_ui_missing_defaults_false() { - let mut cfg = ServerConfig::new(); - let kvs = KVS(vec![ - rustfs_config::server_config::KV { - key: ENABLE_KEY.to_string(), - value: EnableState::On.to_string(), - hidden_if_empty: false, - }, - rustfs_config::server_config::KV { - key: OIDC_CONFIG_URL.to_string(), - value: "https://example.com/.well-known/openid-configuration".to_string(), - hidden_if_empty: false, - }, - rustfs_config::server_config::KV { - key: OIDC_CLIENT_ID.to_string(), - value: "console".to_string(), - hidden_if_empty: false, - }, - ]); - - cfg.0 - .entry(IDENTITY_OPENID_SUB_SYS.to_string()) - .or_default() - .insert(DEFAULT_DELIMITER.to_string(), kvs); - - let parsed = OidcSys::parse_persisted_configs(&cfg); - assert_eq!(parsed.len(), 1); - assert!(!parsed[0].hide_from_ui); - } - - #[test] - fn test_parse_persisted_hide_from_ui() { - let mut cfg = ServerConfig::new(); - let mut kvs = KVS(vec![ - rustfs_config::server_config::KV { - key: ENABLE_KEY.to_string(), - value: EnableState::On.to_string(), - hidden_if_empty: false, - }, - rustfs_config::server_config::KV { - key: OIDC_CONFIG_URL.to_string(), - value: "https://example.com/.well-known/openid-configuration".to_string(), - hidden_if_empty: false, - }, - rustfs_config::server_config::KV { - key: OIDC_CLIENT_ID.to_string(), - value: "console".to_string(), - hidden_if_empty: false, - }, - ]); - kvs.insert(OIDC_HIDE_FROM_UI.to_string(), EnableState::On.to_string()); - - cfg.0 - .entry(IDENTITY_OPENID_SUB_SYS.to_string()) - .or_default() - .insert(DEFAULT_DELIMITER.to_string(), kvs); - - let parsed = OidcSys::parse_persisted_configs(&cfg); - assert_eq!(parsed.len(), 1); - assert!(parsed[0].hide_from_ui); - } - - #[test] - fn role_policy_does_not_map_groups_as_policies() { - let mut config = test_config("authentik"); - config.role_policy = "consoleAdmin".to_string(); - config.claim_name = "policy".to_string(); - - let sys = make_test_sys(vec![config]); - - let claims = OidcClaims { - groups: vec!["authentik Admins".to_string(), "users".to_string()], - raw: HashMap::from([("policy".to_string(), serde_json::json!(["readonly"]))]), - ..Default::default() - }; - - let (policies, groups) = sys.map_claims_to_policies("authentik", &claims); - assert_eq!(groups, vec!["authentik Admins", "users"]); - assert_eq!(policies, vec!["consoleAdmin"]); - } - - #[test] - fn group_claim_policy_names_cover_only_prefixed_group_values() { - let mut config = test_config("ad"); - config.claim_prefix = "oidc-".to_string(); - let sys = make_test_sys(vec![config]); - let claims = OidcClaims { - groups: vec![ - "EXAMPLE\\Domain Users".to_string(), - "admins".to_string(), - "admins".to_string(), - ], - ..Default::default() - }; - - assert_eq!( - sys.group_claim_policy_names("ad", &claims), - vec!["oidc-EXAMPLE\\Domain Users", "oidc-admins"] - ); - assert!(sys.group_claim_policy_names("missing-provider", &claims).is_empty()); - } - - #[test] - fn group_claim_policy_names_include_merged_roles_claim_values() { - let mut config = test_config("keycloak"); - config.roles_claim = "roles".to_string(); - let raw = HashMap::from([ - ("groups".to_string(), serde_json::json!(["readonly"])), - ("roles".to_string(), serde_json::json!(["offline_access", "default-roles-corp"])), - ]); - let claims = OidcClaims { - groups: extract_canonical_group_values(&raw, &config.groups_claim, &config.roles_claim), - raw, - ..Default::default() - }; - let sys = make_test_sys(vec![config]); - - assert_eq!( - sys.group_claim_policy_names("keycloak", &claims), - vec!["default-roles-corp", "offline_access", "readonly"] - ); - } - - #[test] - fn group_claim_policy_names_exclude_role_policy_and_dedicated_policy_claim() { - let mut role_config = test_config("role"); - role_config.role_policy = "consoleAdmin".to_string(); - let mut claim_config = test_config("claim"); - claim_config.claim_name = "policy".to_string(); - let sys = make_test_sys(vec![role_config, claim_config]); - let claims = OidcClaims { - groups: vec!["readonly".to_string(), "unmapped".to_string()], - raw: HashMap::from([("policy".to_string(), serde_json::json!(["readonly", "writeonly"]))]), - ..Default::default() - }; - - assert!(sys.group_claim_policy_names("role", &claims).is_empty()); - // `readonly` is also requested explicitly through the policy claim, so it must resolve. - assert_eq!(sys.group_claim_policy_names("claim", &claims), vec!["unmapped"]); - } - - #[test] - fn test_map_claims_to_policies_with_prefix() { - let mut config = test_config("azure"); - config.claim_prefix = "oidc-".to_string(); - config.display_name = "Azure AD".to_string(); - - let sys = make_test_sys(vec![config]); - - let claims = OidcClaims { - sub: "user456".to_string(), - email: "user@corp.com".to_string(), - username: "user".to_string(), - groups: vec!["engineers".to_string()], - raw: HashMap::new(), - }; - - let (policies, groups) = sys.map_claims_to_policies("azure", &claims); - assert_eq!(groups, vec!["engineers"]); - assert!(policies.contains(&"oidc-engineers".to_string())); - assert_eq!(policies.len(), 1); - } - - #[test] - fn blank_role_policy_uses_claim_mapping() { - let mut config = test_config("keycloak"); - config.role_policy = " ".to_string(); - - let sys = make_test_sys(vec![config]); - let claims = OidcClaims { - groups: vec!["readonly".to_string()], - ..Default::default() - }; - - let (policies, groups) = sys.map_claims_to_policies("keycloak", &claims); - assert_eq!(groups, vec!["readonly"]); - assert_eq!(policies, vec!["readonly"]); - } - - #[test] - fn claim_mapping_keeps_groups_with_distinct_primary_claim() { - let mut config = test_config("keycloak"); - config.claim_name = "policy".to_string(); - - let sys = make_test_sys(vec![config]); - let claims = OidcClaims { - groups: vec!["developers".to_string()], - raw: HashMap::from([("policy".to_string(), serde_json::json!(["readonly"]))]), - ..Default::default() - }; - - let (policies, groups) = sys.map_claims_to_policies("keycloak", &claims); - assert_eq!(groups, vec!["developers"]); - assert_eq!(policies, vec!["developers", "readonly"]); - } - - #[test] - fn test_config_snapshot_lists_providers() { - let mut config = test_config("keycloak"); - config.display_name = "Keycloak SSO".to_string(); - - let sys = make_test_sys(vec![config]); - - assert!(sys.has_providers()); - let snapshot = sys.config_snapshot(); - assert_eq!(snapshot.providers().len(), 1); - assert_eq!(snapshot.providers()[0].config.id, "keycloak"); - assert_eq!(snapshot.providers()[0].config.display_name, "Keycloak SSO"); - assert_eq!(snapshot.providers()[0].source, OidcProviderConfigSource::Persisted); - } - - #[test] - fn test_active_config_snapshot_puts_default_first() { - let sys = make_test_sys(vec![test_config("zeta"), test_config("default"), test_config("alpha")]); - - assert_eq!( - sys.config_snapshot() - .providers() - .iter() - .map(|provider| provider.config.id.as_str()) - .collect::>(), - ["default", "alpha", "zeta"] - ); - } - - #[test] - fn test_get_provider_config() { - let mut config = test_config("test"); - config.client_id = "my-client".to_string(); - config.client_secret = Some("secret".to_string()); - - let sys = make_test_sys(vec![config]); - - assert!(sys.get_provider_config("test").is_some()); - assert_eq!(sys.get_provider_config("test").unwrap().client_id, "my-client"); - assert!(sys.get_provider_config("nonexistent").is_none()); - } - - #[test] - fn test_oidc_provider_config_defaults() { - let config = OidcProviderConfig { - id: "test".to_string(), - enabled: true, - config_url: "https://example.com/.well-known/openid-configuration".to_string(), - issuer: None, - client_id: "my-client".to_string(), - client_secret: Some("secret".to_string()), - scopes: vec!["openid".to_string(), "profile".to_string(), "email".to_string()], - other_audiences: vec![], - redirect_uri: None, - redirect_uri_dynamic: true, - claim_name: "groups".to_string(), - claim_prefix: "".to_string(), - role_policy: "readwrite".to_string(), - display_name: "Test Provider".to_string(), - groups_claim: "groups".to_string(), - roles_claim: String::new(), - email_claim: "email".to_string(), - username_claim: "preferred_username".to_string(), - hide_from_ui: false, - }; - - assert_eq!(config.id, "test"); - assert!(config.enabled); - assert_eq!(config.scopes.len(), 3); - assert!(config.redirect_uri_dynamic); - } -} diff --git a/crates/iam/src/oidc/config.rs b/crates/iam/src/oidc/config.rs new file mode 100644 index 000000000..c8c48bebe --- /dev/null +++ b/crates/iam/src/oidc/config.rs @@ -0,0 +1,904 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +use crate::{EVENT_OIDC_DIAGNOSTICS, LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC}; +use base64_simd::URL_SAFE_NO_PAD; +use rustfs_config::oidc::*; +use rustfs_config::server_config::{Config as ServerConfig, KVS}; +use rustfs_config::{ + CONSOLE_OIDC_SESSION_DURATION_MAX_SECS, DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS, DEFAULT_DELIMITER, ENABLE_KEY, + ENV_CONSOLE_OIDC_SESSION_DURATION, EnableState, +}; +use rustfs_utils::egress::{OutboundUrlError, validate_outbound_url}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; +use std::collections::HashMap; +use std::fmt; +use tracing::warn; +use url::Url; + +pub(super) const REDACTED_SECRET: &str = "***redacted***"; + +fn redacted_optional_secret(value: Option<&str>) -> &'static str { + value.filter(|secret| !secret.is_empty()).map_or("", |_| REDACTED_SECRET) +} + +/// Parsed configuration for a single OIDC provider. +#[derive(Clone, PartialEq, Eq)] +pub struct OidcProviderConfig { + pub id: String, + pub enabled: bool, + pub config_url: String, + pub issuer: Option, + pub client_id: String, + pub client_secret: Option, + pub scopes: Vec, + pub other_audiences: Vec, + pub redirect_uri: Option, + pub redirect_uri_dynamic: bool, + pub claim_name: String, + pub claim_prefix: String, + pub role_policy: String, + pub display_name: String, + pub groups_claim: String, + pub roles_claim: String, + pub email_claim: String, + pub username_claim: String, + pub hide_from_ui: bool, +} + +#[derive(Deserialize)] +#[serde(default, deny_unknown_fields)] +pub struct OidcProviderConfigInput { + pub enabled: bool, + pub display_name: String, + pub config_url: String, + pub issuer: Option, + pub client_id: String, + pub client_secret: Option, + pub scopes: Vec, + pub other_audiences: Vec, + pub redirect_uri: Option, + pub redirect_uri_dynamic: bool, + pub claim_name: String, + pub claim_prefix: String, + pub role_policy: String, + pub groups_claim: String, + pub roles_claim: String, + pub email_claim: String, + pub username_claim: String, + pub hide_from_ui: bool, +} + +impl Default for OidcProviderConfigInput { + fn default() -> Self { + Self { + enabled: true, + display_name: String::new(), + config_url: String::new(), + issuer: None, + client_id: String::new(), + client_secret: None, + scopes: OIDC_DEFAULT_SCOPES.split(',').map(ToString::to_string).collect(), + other_audiences: Vec::new(), + redirect_uri: None, + redirect_uri_dynamic: true, + claim_name: OIDC_DEFAULT_CLAIM_NAME.to_string(), + claim_prefix: String::new(), + role_policy: String::new(), + groups_claim: OIDC_DEFAULT_GROUPS_CLAIM.to_string(), + roles_claim: OIDC_DEFAULT_ROLES_CLAIM.to_string(), + email_claim: OIDC_DEFAULT_EMAIL_CLAIM.to_string(), + username_claim: OIDC_DEFAULT_USERNAME_CLAIM.to_string(), + hide_from_ui: false, + } + } +} + +#[derive(Deserialize)] +#[serde(default, deny_unknown_fields)] +pub struct OidcProviderValidationInput { + pub provider_id: String, + pub enabled: bool, + pub display_name: String, + pub config_url: String, + pub issuer: Option, + pub client_id: String, + pub client_secret: Option, + pub scopes: Vec, + pub other_audiences: Vec, + pub redirect_uri: Option, + pub redirect_uri_dynamic: bool, + pub claim_name: String, + pub claim_prefix: String, + pub role_policy: String, + pub groups_claim: String, + pub roles_claim: String, + pub email_claim: String, + pub username_claim: String, + pub hide_from_ui: bool, +} + +impl Default for OidcProviderValidationInput { + fn default() -> Self { + let input = OidcProviderConfigInput::default(); + Self { + provider_id: "default".to_string(), + enabled: input.enabled, + display_name: input.display_name, + config_url: input.config_url, + issuer: input.issuer, + client_id: input.client_id, + client_secret: input.client_secret, + scopes: input.scopes, + other_audiences: input.other_audiences, + redirect_uri: input.redirect_uri, + redirect_uri_dynamic: input.redirect_uri_dynamic, + claim_name: input.claim_name, + claim_prefix: input.claim_prefix, + role_policy: input.role_policy, + groups_claim: input.groups_claim, + roles_claim: input.roles_claim, + email_claim: input.email_claim, + username_claim: input.username_claim, + hide_from_ui: input.hide_from_ui, + } + } +} + +impl From for (String, OidcProviderConfigInput) { + fn from(input: OidcProviderValidationInput) -> Self { + let provider_id = if input.provider_id.trim().is_empty() { + "default".to_string() + } else { + input.provider_id.trim().to_string() + }; + let config = OidcProviderConfigInput { + enabled: input.enabled, + display_name: input.display_name, + config_url: input.config_url, + issuer: input.issuer, + client_id: input.client_id, + client_secret: input.client_secret, + scopes: input.scopes, + other_audiences: input.other_audiences, + redirect_uri: input.redirect_uri, + redirect_uri_dynamic: input.redirect_uri_dynamic, + claim_name: input.claim_name, + claim_prefix: input.claim_prefix, + role_policy: input.role_policy, + groups_claim: input.groups_claim, + roles_claim: input.roles_claim, + email_claim: input.email_claim, + username_claim: input.username_claim, + hide_from_ui: input.hide_from_ui, + }; + (provider_id, config) + } +} + +#[derive(Debug, thiserror::Error)] +pub enum OidcConfigError { + #[error("invalid provider_id")] + InvalidProviderId, + #[error("provider is managed by environment variables")] + EnvironmentManaged, + #[error("provider not found")] + ProviderNotFound, + #[error("config_url is required")] + ConfigUrlRequired, + #[error("client_id is required")] + ClientIdRequired, + #[error("redirect_uri is required when redirect_uri_dynamic is off")] + RedirectUriRequired, + #[error("scopes must include openid")] + OpenidScopeRequired, + #[error("{0} must be an absolute http/https URL")] + InvalidAbsoluteUrl(&'static str), + #[error("{field} is not allowed: {source}")] + ForbiddenOutbound { + field: &'static str, + #[source] + source: OutboundUrlError, + }, +} + +impl fmt::Debug for OidcProviderConfig { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("OidcProviderConfig") + .field("id", &self.id) + .field("enabled", &self.enabled) + .field("config_url", &self.config_url) + .field("issuer", &self.issuer) + .field("client_id", &self.client_id) + .field("client_secret", &redacted_optional_secret(self.client_secret.as_deref())) + .field("scopes", &self.scopes) + .field("other_audiences", &self.other_audiences) + .field("redirect_uri", &self.redirect_uri) + .field("redirect_uri_dynamic", &self.redirect_uri_dynamic) + .field("claim_name", &self.claim_name) + .field("claim_prefix", &self.claim_prefix) + .field("role_policy", &self.role_policy) + .field("display_name", &self.display_name) + .field("groups_claim", &self.groups_claim) + .field("roles_claim", &self.roles_claim) + .field("email_claim", &self.email_claim) + .field("username_claim", &self.username_claim) + .field("hide_from_ui", &self.hide_from_ui) + .finish() + } +} + +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum OidcProviderConfigSource { + Env, + Persisted, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SourcedOidcProviderConfig { + pub config: OidcProviderConfig, + pub source: OidcProviderConfigSource, +} + +/// Immutable OIDC configuration entries returned by configuration queries. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct OidcConfigSnapshot { + providers: Vec, +} + +impl OidcConfigSnapshot { + pub fn new(mut providers: Vec) -> Self { + providers.sort_by(|left, right| { + (left.config.id != "default") + .cmp(&(right.config.id != "default")) + .then_with(|| left.config.id.cmp(&right.config.id)) + }); + Self { providers } + } + + pub fn providers(&self) -> &[SourcedOidcProviderConfig] { + &self.providers + } + + pub fn into_providers(self) -> Vec { + self.providers + } +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +pub struct OidcProviderValidationResult { + pub issuer: String, + pub authorization_endpoint: Option, + pub token_endpoint: Option, +} + +/// Compatibility name for the provider summary returned by OIDC list APIs. +pub(crate) type OidcProviderSummary = crate::federation::FederatedProviderView; + +/// OIDC fields required to compare identity providers across replicated sites. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct OidcSiteReplicationProvider { + pub provider_id: String, + pub claim_name: String, + pub role_policy: String, + pub client_id: String, + pub hashed_client_secret: String, +} + +/// Active OIDC settings exposed to site replication without raw credentials. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct OidcSiteReplicationSnapshot { + providers: Vec, +} + +impl OidcSiteReplicationSnapshot { + pub fn from_config_snapshot(snapshot: &OidcConfigSnapshot) -> Self { + let providers = snapshot + .providers() + .iter() + .map(|provider| { + let config = &provider.config; + OidcSiteReplicationProvider { + provider_id: config.id.clone(), + claim_name: config.claim_name.clone(), + role_policy: config.role_policy.clone(), + client_id: config.client_id.clone(), + hashed_client_secret: hash_client_secret(config.client_secret.as_deref()), + } + }) + .collect(); + Self { providers } + } + + pub fn providers(&self) -> &[OidcSiteReplicationProvider] { + &self.providers + } +} + +fn hash_client_secret(secret: Option<&str>) -> String { + let Some(secret) = secret.filter(|secret| !secret.is_empty()) else { + return String::new(); + }; + + let mut hasher = Sha256::new(); + hasher.update(secret.as_bytes()); + URL_SAFE_NO_PAD.encode_to_string(hasher.finalize()) +} + +/// Read-only access to the active OIDC settings needed by site replication. +pub trait OidcConfigQuery: Send + Sync { + fn site_replication_snapshot(&self) -> OidcSiteReplicationSnapshot; +} + +pub fn validate_mutable_provider_id(provider_id: &str) -> Result<(), OidcConfigError> { + if !is_valid_provider_id(provider_id) { + return Err(OidcConfigError::InvalidProviderId); + } + if load_oidc_provider_configs_from_env() + .iter() + .any(|config| config.id == provider_id) + { + return Err(OidcConfigError::EnvironmentManaged); + } + Ok(()) +} + +fn is_valid_provider_id(id: &str) -> bool { + !id.is_empty() && id.chars().all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-') +} + +fn provider_instance_key(provider_id: &str) -> String { + if provider_id == "default" { + DEFAULT_DELIMITER.to_string() + } else { + provider_id.to_string() + } +} + +fn normalize_optional(value: Option) -> Option { + value.map(|value| value.trim().to_string()).filter(|value| !value.is_empty()) +} + +fn or_default(value: &str, default: &str) -> String { + if value.trim().is_empty() { + default.to_string() + } else { + value.trim().to_string() + } +} + +fn validate_absolute_http_url(value: &str, field: &'static str, check_outbound: bool) -> Result<(), OidcConfigError> { + let parsed = Url::parse(value).map_err(|_| OidcConfigError::InvalidAbsoluteUrl(field))?; + if !matches!(parsed.scheme(), "http" | "https") || parsed.host_str().is_none() { + return Err(OidcConfigError::InvalidAbsoluteUrl(field)); + } + if check_outbound { + validate_outbound_url(&parsed).map_err(|source| OidcConfigError::ForbiddenOutbound { field, source })?; + } + Ok(()) +} + +fn normalize_provider_config(mut config: OidcProviderConfig) -> OidcProviderConfig { + config.config_url = config.config_url.trim().to_string(); + config.issuer = normalize_optional(config.issuer); + config.client_id = config.client_id.trim().to_string(); + config.scopes = config + .scopes + .iter() + .map(|scope| scope.trim().to_string()) + .filter(|scope| !scope.is_empty()) + .collect(); + config.redirect_uri = normalize_optional(config.redirect_uri); + config.claim_name = or_default(&config.claim_name, OIDC_DEFAULT_CLAIM_NAME); + config.claim_prefix = config.claim_prefix.trim().to_string(); + config.role_policy = config.role_policy.trim().to_string(); + config.display_name = or_default(&config.display_name, &config.id); + config.groups_claim = or_default(&config.groups_claim, OIDC_DEFAULT_GROUPS_CLAIM); + config.roles_claim = or_default(&config.roles_claim, OIDC_DEFAULT_ROLES_CLAIM); + config.email_claim = or_default(&config.email_claim, OIDC_DEFAULT_EMAIL_CLAIM); + config.username_claim = or_default(&config.username_claim, OIDC_DEFAULT_USERNAME_CLAIM); + config +} + +fn validate_provider_config_fields(config: &OidcProviderConfig) -> Result<(), OidcConfigError> { + if !is_valid_provider_id(&config.id) { + return Err(OidcConfigError::InvalidProviderId); + } + if config.config_url.trim().is_empty() { + return Err(OidcConfigError::ConfigUrlRequired); + } + validate_absolute_http_url(&config.config_url, "config_url", true)?; + if let Some(issuer) = config.issuer.as_deref() { + validate_absolute_http_url(issuer, "issuer", false)?; + } + if config.client_id.trim().is_empty() { + return Err(OidcConfigError::ClientIdRequired); + } + if !config.redirect_uri_dynamic { + let redirect_uri = config.redirect_uri.as_deref().ok_or(OidcConfigError::RedirectUriRequired)?; + validate_absolute_http_url(redirect_uri, "redirect_uri", true)?; + } else if let Some(redirect_uri) = config.redirect_uri.as_deref() { + validate_absolute_http_url(redirect_uri, "redirect_uri", true)?; + } + if !config.scopes.iter().any(|scope| scope == "openid") { + return Err(OidcConfigError::OpenidScopeRequired); + } + Ok(()) +} + +fn build_provider_config( + provider_id: &str, + input: OidcProviderConfigInput, + existing_secret: Option, +) -> Result { + let client_secret = match input.client_secret { + Some(value) if !value.trim().is_empty() => Some(value), + _ => existing_secret.filter(|value| !value.trim().is_empty()), + }; + let config = normalize_provider_config(OidcProviderConfig { + id: provider_id.to_string(), + enabled: input.enabled, + config_url: input.config_url, + issuer: input.issuer, + client_id: input.client_id, + client_secret, + scopes: input.scopes, + other_audiences: input.other_audiences, + redirect_uri: input.redirect_uri, + redirect_uri_dynamic: input.redirect_uri_dynamic, + claim_name: input.claim_name, + claim_prefix: input.claim_prefix, + role_policy: input.role_policy, + display_name: input.display_name, + groups_claim: input.groups_claim, + roles_claim: input.roles_claim, + email_claim: input.email_claim, + username_claim: input.username_claim, + hide_from_ui: input.hide_from_ui, + }); + validate_provider_config_fields(&config)?; + Ok(config) +} + +pub fn build_upsert_provider_config( + provider_id: &str, + input: OidcProviderConfigInput, + existing_secret: Option, +) -> Result { + build_provider_config(provider_id, input, existing_secret) +} + +pub fn build_validation_provider_config(input: OidcProviderValidationInput) -> Result { + let (provider_id, input) = input.into(); + build_provider_config(&provider_id, input, None) +} + +pub fn persisted_provider_secret(config: &ServerConfig, provider_id: &str) -> Option { + config + .0 + .get(IDENTITY_OPENID_SUB_SYS) + .and_then(|subsystem| subsystem.get(&provider_instance_key(provider_id))) + .and_then(|kvs| kvs.lookup(OIDC_CLIENT_SECRET)) + .filter(|value| !value.trim().is_empty()) +} + +fn set_kvs_value(kvs: &mut KVS, key: &str, value: String) { + if let Some(existing) = kvs.0.iter_mut().find(|kv| kv.key == key) { + existing.value = value; + return; + } + kvs.insert(key.to_string(), value); +} + +pub fn upsert_persisted_provider_config(config: &mut ServerConfig, provider: &OidcProviderConfig) { + let mut kvs = ServerConfig::new() + .get_value(IDENTITY_OPENID_SUB_SYS, DEFAULT_DELIMITER) + .unwrap_or_default(); + set_kvs_value( + &mut kvs, + ENABLE_KEY, + if provider.enabled { + EnableState::On.to_string() + } else { + EnableState::Off.to_string() + }, + ); + set_kvs_value(&mut kvs, OIDC_CONFIG_URL, provider.config_url.clone()); + set_kvs_value(&mut kvs, OIDC_ISSUER, provider.issuer.clone().unwrap_or_default()); + set_kvs_value(&mut kvs, OIDC_CLIENT_ID, provider.client_id.clone()); + set_kvs_value(&mut kvs, OIDC_CLIENT_SECRET, provider.client_secret.clone().unwrap_or_default()); + set_kvs_value(&mut kvs, OIDC_SCOPES, provider.scopes.join(",")); + set_kvs_value(&mut kvs, OIDC_OTHER_AUDIENCES, provider.other_audiences.join(",")); + set_kvs_value(&mut kvs, OIDC_REDIRECT_URI, provider.redirect_uri.clone().unwrap_or_default()); + set_kvs_value( + &mut kvs, + OIDC_REDIRECT_URI_DYNAMIC, + if provider.redirect_uri_dynamic { + EnableState::On.to_string() + } else { + EnableState::Off.to_string() + }, + ); + set_kvs_value(&mut kvs, OIDC_CLAIM_NAME, provider.claim_name.clone()); + set_kvs_value(&mut kvs, OIDC_CLAIM_PREFIX, provider.claim_prefix.clone()); + set_kvs_value(&mut kvs, OIDC_ROLE_POLICY, provider.role_policy.clone()); + set_kvs_value(&mut kvs, OIDC_DISPLAY_NAME, provider.display_name.clone()); + set_kvs_value(&mut kvs, OIDC_GROUPS_CLAIM, provider.groups_claim.clone()); + set_kvs_value(&mut kvs, OIDC_ROLES_CLAIM, provider.roles_claim.clone()); + set_kvs_value(&mut kvs, OIDC_EMAIL_CLAIM, provider.email_claim.clone()); + set_kvs_value(&mut kvs, OIDC_USERNAME_CLAIM, provider.username_claim.clone()); + set_kvs_value( + &mut kvs, + OIDC_HIDE_FROM_UI, + if provider.hide_from_ui { + EnableState::On.to_string() + } else { + EnableState::Off.to_string() + }, + ); + config + .0 + .entry(IDENTITY_OPENID_SUB_SYS.to_string()) + .or_default() + .insert(provider_instance_key(&provider.id), kvs); +} + +pub fn delete_persisted_provider_config(config: &mut ServerConfig, provider_id: &str) -> Result<(), OidcConfigError> { + let Some(subsystem) = config.0.get_mut(IDENTITY_OPENID_SUB_SYS) else { + return Err(OidcConfigError::ProviderNotFound); + }; + if subsystem.remove(&provider_instance_key(provider_id)).is_none() { + return Err(OidcConfigError::ProviderNotFound); + } + if subsystem.is_empty() { + config.0.remove(IDENTITY_OPENID_SUB_SYS); + } + Ok(()) +} + +/// Parse all OIDC provider configs from environment variables. +pub(super) fn parse_env_configs() -> Vec { + let mut configs = Vec::new(); + + // Check for the default provider (no suffix) + if let Some(config) = parse_single_provider("", "default") { + configs.push(config); + } + + // Scan for suffixed providers by checking all OIDC env var prefixes. + // This allows providers to be discovered without requiring a separate ENABLE_ key. + let mut provider_ids: Vec = Vec::new(); + let scan_prefixes: Vec = ENV_IDENTITY_OPENID_KEYS.iter().map(|k| format!("{k}_")).collect(); + for (key, _) in std::env::vars() { + for prefix in &scan_prefixes { + if let Some(suffix) = key.strip_prefix(prefix.as_str()) + && !suffix.is_empty() + && suffix != "default" + { + provider_ids.push(suffix.to_string()); + break; + } + } + } + + provider_ids.sort(); + provider_ids.dedup(); + + for id in provider_ids { + let suffix = format!("_{id}"); + if let Some(config) = parse_single_provider(&suffix, &id) { + configs.push(config); + } + } + + configs +} + +pub(super) fn parse_persisted_configs(cfg: &ServerConfig) -> Vec { + let Some(subsystem) = cfg.0.get(IDENTITY_OPENID_SUB_SYS) else { + return Vec::new(); + }; + + let mut configs = Vec::new(); + let mut provider_ids: Vec = subsystem.keys().cloned().collect(); + provider_ids.sort(); + + for raw_id in provider_ids { + let Some(kvs) = subsystem.get(&raw_id) else { + continue; + }; + + let id = if raw_id == DEFAULT_DELIMITER { + "default" + } else { + raw_id.as_str() + }; + if let Some(config) = parse_single_persisted_provider(kvs, id) { + configs.push(config); + } + } + + configs +} + +/// Parse a string as an `EnableState` boolean. +/// Returns `default_if_empty` when the input is empty, and `default_on_error` +/// when parsing fails. +pub(super) fn parse_enable_state(value: &str, default_if_empty: bool, default_on_error: bool) -> bool { + if value.is_empty() { + return default_if_empty; + } + value + .parse::() + .map(|s| s.is_enabled()) + .unwrap_or(default_on_error) +} + +/// Parse a single provider's config from env vars with the given suffix. +pub(super) fn parse_single_provider(env_suffix: &str, id: &str) -> Option { + let get_env = |base: &str| -> String { std::env::var(format!("{base}{env_suffix}")).unwrap_or_default() }; + + let enable_val = get_env(ENV_IDENTITY_OPENID_ENABLE); + let config_url = get_env(ENV_IDENTITY_OPENID_CONFIG_URL); + let issuer = get_env(ENV_IDENTITY_OPENID_ISSUER); + + // Skip if no config URL + if config_url.is_empty() { + return None; + } + + let enabled = parse_enable_state(&enable_val, true, false); + + let scopes_str = get_env(ENV_IDENTITY_OPENID_SCOPES); + let scopes = if scopes_str.is_empty() { + OIDC_DEFAULT_SCOPES.split(',').map(String::from).collect() + } else { + scopes_str.split(',').map(|s| s.trim().to_string()).collect() + }; + + let other_audiences_str = get_env(ENV_IDENTITY_OPENID_OTHER_AUDIENCES); + let other_audiences = other_audiences_str + .split(',') + .map(|s| s.trim()) + .filter(|s| !s.is_empty()) + .map(|s| s.to_string()) + .collect(); + + let redirect_uri_dynamic = parse_enable_state(&get_env(ENV_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC), true, true); + + let claim_name = { + let v = get_env(ENV_IDENTITY_OPENID_CLAIM_NAME); + if v.is_empty() { + OIDC_DEFAULT_CLAIM_NAME.to_string() + } else { + v + } + }; + let groups_claim = { + let v = get_env(ENV_IDENTITY_OPENID_GROUPS_CLAIM); + if v.is_empty() { + OIDC_DEFAULT_GROUPS_CLAIM.to_string() + } else { + v + } + }; + let roles_claim = get_env(ENV_IDENTITY_OPENID_ROLES_CLAIM); + let email_claim = { + let v = get_env(ENV_IDENTITY_OPENID_EMAIL_CLAIM); + if v.is_empty() { + OIDC_DEFAULT_EMAIL_CLAIM.to_string() + } else { + v + } + }; + let username_claim = { + let v = get_env(ENV_IDENTITY_OPENID_USERNAME_CLAIM); + if v.is_empty() { + OIDC_DEFAULT_USERNAME_CLAIM.to_string() + } else { + v + } + }; + let display_name = { + let v = get_env(ENV_IDENTITY_OPENID_DISPLAY_NAME); + if v.is_empty() { id.to_string() } else { v } + }; + let redirect_uri = { + let v = get_env(ENV_IDENTITY_OPENID_REDIRECT_URI); + if v.is_empty() { None } else { Some(v) } + }; + let client_secret = { + let v = get_env(ENV_IDENTITY_OPENID_CLIENT_SECRET); + if v.is_empty() { None } else { Some(v) } + }; + let hide_from_ui = parse_enable_state(&get_env(ENV_IDENTITY_OPENID_HIDE_FROM_UI), false, false); + + Some(OidcProviderConfig { + id: id.to_string(), + enabled, + config_url, + issuer: if issuer.is_empty() { None } else { Some(issuer) }, + client_id: get_env(ENV_IDENTITY_OPENID_CLIENT_ID), + client_secret, + scopes, + other_audiences, + redirect_uri, + redirect_uri_dynamic, + claim_name, + claim_prefix: get_env(ENV_IDENTITY_OPENID_CLAIM_PREFIX), + role_policy: get_env(ENV_IDENTITY_OPENID_ROLE_POLICY), + display_name, + groups_claim, + roles_claim, + email_claim, + username_claim, + hide_from_ui, + }) +} + +pub(super) fn parse_single_persisted_provider(kvs: &KVS, id: &str) -> Option { + let config_url = kvs.get(OIDC_CONFIG_URL); + if config_url.is_empty() { + return None; + } + + let enabled = parse_enable_state(&kvs.lookup(ENABLE_KEY).unwrap_or_default(), false, false); + + let scopes_str = kvs.get(OIDC_SCOPES); + let scopes = if scopes_str.is_empty() { + OIDC_DEFAULT_SCOPES.split(',').map(String::from).collect() + } else { + scopes_str.split(',').map(|s| s.trim().to_string()).collect() + }; + + let other_audiences_str = kvs.get(OIDC_OTHER_AUDIENCES); + let other_audiences = other_audiences_str + .split(',') + .map(|s| s.trim()) + .filter(|s| !s.is_empty()) + .map(|s| s.to_string()) + .collect(); + + let redirect_uri_dynamic = parse_enable_state(&kvs.lookup(OIDC_REDIRECT_URI_DYNAMIC).unwrap_or_default(), true, true); + + let claim_name = kvs + .lookup(OIDC_CLAIM_NAME) + .unwrap_or_else(|| OIDC_DEFAULT_CLAIM_NAME.to_string()); + let groups_claim = kvs + .lookup(OIDC_GROUPS_CLAIM) + .unwrap_or_else(|| OIDC_DEFAULT_GROUPS_CLAIM.to_string()); + let roles_claim = kvs + .lookup(OIDC_ROLES_CLAIM) + .unwrap_or_else(|| OIDC_DEFAULT_ROLES_CLAIM.to_string()); + let email_claim = kvs + .lookup(OIDC_EMAIL_CLAIM) + .unwrap_or_else(|| OIDC_DEFAULT_EMAIL_CLAIM.to_string()); + let username_claim = kvs + .lookup(OIDC_USERNAME_CLAIM) + .unwrap_or_else(|| OIDC_DEFAULT_USERNAME_CLAIM.to_string()); + let display_name = kvs.lookup(OIDC_DISPLAY_NAME).unwrap_or_else(|| id.to_string()); + let redirect_uri = kvs.lookup(OIDC_REDIRECT_URI).filter(|v| !v.is_empty()); + let client_secret = kvs.lookup(OIDC_CLIENT_SECRET).filter(|v| !v.is_empty()); + let hide_from_ui = parse_enable_state(&kvs.lookup(OIDC_HIDE_FROM_UI).unwrap_or_default(), false, false); + + Some(OidcProviderConfig { + id: id.to_string(), + enabled, + config_url, + issuer: kvs.lookup(OIDC_ISSUER).filter(|v| !v.is_empty()), + client_id: kvs.get(OIDC_CLIENT_ID), + client_secret, + scopes, + other_audiences, + redirect_uri, + redirect_uri_dynamic, + claim_name, + claim_prefix: kvs.get(OIDC_CLAIM_PREFIX), + role_policy: kvs.get(OIDC_ROLE_POLICY), + display_name, + groups_claim, + roles_claim, + email_claim, + username_claim, + hide_from_ui, + }) +} + +pub fn load_oidc_provider_configs_from_env() -> Vec { + parse_env_configs() +} + +/// Credential lifetime, in seconds, for Console credentials issued by the OIDC callback. +/// +/// `RUSTFS_CONSOLE_OIDC_SESSION_DURATION` overrides the default for every provider. An unset, +/// unparseable, or zero value keeps [`DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS`], so a +/// misconfigured variable can never issue credentials that are already expired. Values above +/// [`CONSOLE_OIDC_SESSION_DURATION_MAX_SECS`] are clamped to that maximum with a warning. +/// +/// Both the credential lifetime and the Console logout-token retention derive from this single +/// value, so they stay in step whichever way the variable is set. +pub fn console_session_duration_secs() -> usize { + let configured = rustfs_utils::get_env_opt_u64(ENV_CONSOLE_OIDC_SESSION_DURATION) + .and_then(|seconds| usize::try_from(seconds).ok()) + .filter(|seconds| *seconds > 0) + .unwrap_or(DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS); + + if configured > CONSOLE_OIDC_SESSION_DURATION_MAX_SECS { + warn!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + state = "console_session_duration_clamped", + config_key = ENV_CONSOLE_OIDC_SESSION_DURATION, + configured, + max = CONSOLE_OIDC_SESSION_DURATION_MAX_SECS, + "Console OIDC session duration exceeds the maximum" + ); + return CONSOLE_OIDC_SESSION_DURATION_MAX_SECS; + } + + configured +} + +pub fn load_oidc_provider_configs_from_server_config(cfg: &ServerConfig) -> Vec { + parse_persisted_configs(cfg) +} + +pub(super) fn merge_oidc_provider_configs( + env_configs: Vec, + persisted_configs: Vec, +) -> Vec { + let mut effective = HashMap::new(); + + for config in persisted_configs { + effective.insert( + config.id.clone(), + SourcedOidcProviderConfig { + config, + source: OidcProviderConfigSource::Persisted, + }, + ); + } + + for config in env_configs { + effective.insert( + config.id.clone(), + SourcedOidcProviderConfig { + config, + source: OidcProviderConfigSource::Env, + }, + ); + } + + let mut configs: Vec = effective.into_values().collect(); + configs.sort_by(|lhs, rhs| lhs.config.id.cmp(&rhs.config.id)); + configs +} + +pub(super) fn load_effective_oidc_provider_configs(server_config: Option<&ServerConfig>) -> Vec { + let env_configs = load_oidc_provider_configs_from_env(); + let persisted_configs = server_config + .map(load_oidc_provider_configs_from_server_config) + .unwrap_or_default(); + merge_oidc_provider_configs(env_configs, persisted_configs) +} + +pub fn load_oidc_config_snapshot(server_config: Option<&ServerConfig>) -> OidcConfigSnapshot { + OidcConfigSnapshot::new(load_effective_oidc_provider_configs(server_config)) +} + +#[cfg(test)] +#[path = "config_tests.rs"] +mod tests; diff --git a/crates/iam/src/oidc/config_tests.rs b/crates/iam/src/oidc/config_tests.rs new file mode 100644 index 000000000..37370a434 --- /dev/null +++ b/crates/iam/src/oidc/config_tests.rs @@ -0,0 +1,475 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use crate::oidc::test_support::*; +use rustfs_config::server_config::{Config as ServerConfig, KVS}; +use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY}; + +#[test] +#[serial_test::serial] +fn console_session_duration_follows_the_environment() { + for (configured, expected) in [ + (None, DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS), + (Some("1"), 1), + (Some("7200"), 7200), + (Some("604800"), CONSOLE_OIDC_SESSION_DURATION_MAX_SECS), + (Some("604801"), CONSOLE_OIDC_SESSION_DURATION_MAX_SECS), + (Some("999999999"), CONSOLE_OIDC_SESSION_DURATION_MAX_SECS), + (Some("18446744073709551615"), CONSOLE_OIDC_SESSION_DURATION_MAX_SECS), + (Some("18446744073709551616"), DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS), + (Some("0"), DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS), + (Some("-1"), DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS), + (Some("60s"), DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS), + (Some(""), DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS), + ] { + let observed = temp_env::with_var(ENV_CONSOLE_OIDC_SESSION_DURATION, configured, console_session_duration_secs); + assert_eq!(observed, expected, "Console OIDC session duration for {configured:?}"); + } +} + +#[test] +fn test_parse_single_provider_no_config_url() { + let config = parse_single_provider("_TEST_EMPTY", "test_empty"); + assert!(config.is_none()); +} +#[test] +fn test_parse_single_provider_reads_issuer() { + temp_env::with_vars( + [ + ( + ENV_IDENTITY_OPENID_CONFIG_URL, + Some("http://keycloak.ns.svc.cluster.local:8080/realms/app/.well-known/openid-configuration"), + ), + (ENV_IDENTITY_OPENID_ISSUER, Some("https://app.local/realms/app")), + (ENV_IDENTITY_OPENID_CLIENT_ID, Some("console")), + ], + || { + let config = parse_single_provider("", "default").expect("provider config should parse"); + + assert_eq!(config.issuer.as_deref(), Some("https://app.local/realms/app")); + }, + ); +} +#[test] +fn test_parse_persisted_provider_config() { + let mut cfg = ServerConfig::new(); + let mut kvs = KVS(vec![ + rustfs_config::server_config::KV { + key: ENABLE_KEY.to_string(), + value: EnableState::Off.to_string(), + hidden_if_empty: false, + }, + rustfs_config::server_config::KV { + key: OIDC_CONFIG_URL.to_string(), + value: String::new(), + hidden_if_empty: false, + }, + rustfs_config::server_config::KV { + key: OIDC_CLIENT_ID.to_string(), + value: String::new(), + hidden_if_empty: false, + }, + ]); + kvs.insert( + OIDC_CONFIG_URL.to_string(), + "https://example.com/.well-known/openid-configuration".to_string(), + ); + kvs.insert(OIDC_CLIENT_ID.to_string(), "console".to_string()); + kvs.insert(ENABLE_KEY.to_string(), EnableState::On.to_string()); + kvs.insert(OIDC_ISSUER.to_string(), "https://issuer.example".to_string()); + kvs.insert(OIDC_ROLES_CLAIM.to_string(), "app_roles".to_string()); + + cfg.0 + .entry(IDENTITY_OPENID_SUB_SYS.to_string()) + .or_default() + .insert(DEFAULT_DELIMITER.to_string(), kvs); + + let parsed = parse_persisted_configs(&cfg); + assert_eq!(parsed.len(), 1); + assert_eq!(parsed[0].id, "default"); + assert_eq!(parsed[0].client_id, "console"); + assert_eq!(parsed[0].issuer.as_deref(), Some("https://issuer.example")); + assert!(parsed[0].enabled); + assert_eq!(parsed[0].roles_claim, "app_roles"); +} +#[test] +fn test_parse_persisted_provider_config_omitted_roles_claim_is_empty() { + let mut cfg = ServerConfig::new(); + let mut kvs = KVS(vec![ + rustfs_config::server_config::KV { + key: ENABLE_KEY.to_string(), + value: EnableState::Off.to_string(), + hidden_if_empty: false, + }, + rustfs_config::server_config::KV { + key: OIDC_CONFIG_URL.to_string(), + value: String::new(), + hidden_if_empty: false, + }, + rustfs_config::server_config::KV { + key: OIDC_CLIENT_ID.to_string(), + value: String::new(), + hidden_if_empty: false, + }, + ]); + kvs.insert( + OIDC_CONFIG_URL.to_string(), + "https://example.com/.well-known/openid-configuration".to_string(), + ); + kvs.insert(OIDC_CLIENT_ID.to_string(), "console".to_string()); + kvs.insert(ENABLE_KEY.to_string(), EnableState::On.to_string()); + + cfg.0 + .entry(IDENTITY_OPENID_SUB_SYS.to_string()) + .or_default() + .insert(DEFAULT_DELIMITER.to_string(), kvs); + + let parsed = parse_persisted_configs(&cfg); + assert_eq!(parsed.len(), 1); + assert_eq!(parsed[0].roles_claim, ""); +} +#[test] +fn test_merge_oidc_provider_configs_prefers_env() { + let mut persisted = test_config("default"); + persisted.display_name = "Persisted".to_string(); + + let mut env = test_config("default"); + env.display_name = "Environment".to_string(); + + let merged = merge_oidc_provider_configs(vec![env], vec![persisted]); + assert_eq!(merged.len(), 1); + assert_eq!(merged[0].config.display_name, "Environment"); + assert_eq!(merged[0].source, OidcProviderConfigSource::Env); +} +#[test] +fn test_oidc_provider_config_debug_redacts_client_secret() { + let config = OidcProviderConfig { + client_secret: Some("oidc-client-secret".to_string()), + ..test_config("default") + }; + let sourced = SourcedOidcProviderConfig { + config, + source: OidcProviderConfigSource::Persisted, + }; + + let rendered = format!("{sourced:?}"); + + assert!(!rendered.contains("oidc-client-secret")); + assert!(rendered.contains(REDACTED_SECRET)); + assert!(rendered.contains("client-id")); +} +#[test] +fn test_parse_enable_state_on() { + assert!(parse_enable_state("on", false, false)); +} +#[test] +fn test_parse_enable_state_off() { + assert!(!parse_enable_state("off", true, true)); +} +#[test] +fn test_parse_enable_state_empty_returns_default() { + assert!(parse_enable_state("", true, false)); + assert!(!parse_enable_state("", false, true)); +} +#[test] +fn test_parse_enable_state_invalid_returns_error_default() { + assert!(!parse_enable_state("garbage", true, false)); + assert!(parse_enable_state("garbage", false, true)); +} +#[test] +fn test_hide_from_ui_default_is_false() { + let config = test_config("default"); + assert!(!config.hide_from_ui); +} +#[test] +fn test_parse_persisted_hide_from_ui_off_is_false() { + let mut cfg = ServerConfig::new(); + let mut kvs = KVS(vec![ + rustfs_config::server_config::KV { + key: ENABLE_KEY.to_string(), + value: EnableState::On.to_string(), + hidden_if_empty: false, + }, + rustfs_config::server_config::KV { + key: OIDC_CONFIG_URL.to_string(), + value: "https://example.com/.well-known/openid-configuration".to_string(), + hidden_if_empty: false, + }, + rustfs_config::server_config::KV { + key: OIDC_CLIENT_ID.to_string(), + value: "console".to_string(), + hidden_if_empty: false, + }, + ]); + kvs.insert(OIDC_HIDE_FROM_UI.to_string(), EnableState::Off.to_string()); + + cfg.0 + .entry(IDENTITY_OPENID_SUB_SYS.to_string()) + .or_default() + .insert(DEFAULT_DELIMITER.to_string(), kvs); + + let parsed = parse_persisted_configs(&cfg); + assert_eq!(parsed.len(), 1); + assert!(!parsed[0].hide_from_ui); +} +#[test] +fn test_parse_persisted_hide_from_ui_missing_defaults_false() { + let mut cfg = ServerConfig::new(); + let kvs = KVS(vec![ + rustfs_config::server_config::KV { + key: ENABLE_KEY.to_string(), + value: EnableState::On.to_string(), + hidden_if_empty: false, + }, + rustfs_config::server_config::KV { + key: OIDC_CONFIG_URL.to_string(), + value: "https://example.com/.well-known/openid-configuration".to_string(), + hidden_if_empty: false, + }, + rustfs_config::server_config::KV { + key: OIDC_CLIENT_ID.to_string(), + value: "console".to_string(), + hidden_if_empty: false, + }, + ]); + + cfg.0 + .entry(IDENTITY_OPENID_SUB_SYS.to_string()) + .or_default() + .insert(DEFAULT_DELIMITER.to_string(), kvs); + + let parsed = parse_persisted_configs(&cfg); + assert_eq!(parsed.len(), 1); + assert!(!parsed[0].hide_from_ui); +} +#[test] +fn test_parse_persisted_hide_from_ui() { + let mut cfg = ServerConfig::new(); + let mut kvs = KVS(vec![ + rustfs_config::server_config::KV { + key: ENABLE_KEY.to_string(), + value: EnableState::On.to_string(), + hidden_if_empty: false, + }, + rustfs_config::server_config::KV { + key: OIDC_CONFIG_URL.to_string(), + value: "https://example.com/.well-known/openid-configuration".to_string(), + hidden_if_empty: false, + }, + rustfs_config::server_config::KV { + key: OIDC_CLIENT_ID.to_string(), + value: "console".to_string(), + hidden_if_empty: false, + }, + ]); + kvs.insert(OIDC_HIDE_FROM_UI.to_string(), EnableState::On.to_string()); + + cfg.0 + .entry(IDENTITY_OPENID_SUB_SYS.to_string()) + .or_default() + .insert(DEFAULT_DELIMITER.to_string(), kvs); + + let parsed = parse_persisted_configs(&cfg); + assert_eq!(parsed.len(), 1); + assert!(parsed[0].hide_from_ui); +} +#[test] +fn test_active_config_snapshot_puts_default_first() { + let sys = make_test_sys(vec![test_config("zeta"), test_config("default"), test_config("alpha")]); + + assert_eq!( + sys.config_snapshot() + .providers() + .iter() + .map(|provider| provider.config.id.as_str()) + .collect::>(), + ["default", "alpha", "zeta"] + ); +} +#[test] +fn test_oidc_provider_config_defaults() { + let config = OidcProviderConfig { + id: "test".to_string(), + enabled: true, + config_url: "https://example.com/.well-known/openid-configuration".to_string(), + issuer: None, + client_id: "my-client".to_string(), + client_secret: Some("secret".to_string()), + scopes: vec!["openid".to_string(), "profile".to_string(), "email".to_string()], + other_audiences: vec![], + redirect_uri: None, + redirect_uri_dynamic: true, + claim_name: "groups".to_string(), + claim_prefix: "".to_string(), + role_policy: "readwrite".to_string(), + display_name: "Test Provider".to_string(), + groups_claim: "groups".to_string(), + roles_claim: String::new(), + email_claim: "email".to_string(), + username_claim: "preferred_username".to_string(), + hide_from_ui: false, + }; + + assert_eq!(config.id, "test"); + assert!(config.enabled); + assert_eq!(config.scopes.len(), 3); + assert!(config.redirect_uri_dynamic); +} + +#[test] +fn config_input_preserves_secret_and_defaults() { + let input = OidcProviderConfigInput { + config_url: " https://example.com/.well-known/openid-configuration ".to_string(), + client_id: " client ".to_string(), + client_secret: Some(String::new()), + scopes: vec![" openid ".to_string(), " profile ".to_string()], + roles_claim: "app_roles".to_string(), + ..Default::default() + }; + let config = build_upsert_provider_config("default", input, Some("existing-secret".to_string())) + .expect("valid config should preserve existing secret"); + + assert_eq!(config.client_secret.as_deref(), Some("existing-secret")); + assert_eq!(config.config_url, "https://example.com/.well-known/openid-configuration"); + assert_eq!(config.client_id, "client"); + assert_eq!(config.scopes, ["openid", "profile"]); + assert_eq!(config.roles_claim, "app_roles"); + assert_eq!(config.claim_name, OIDC_DEFAULT_CLAIM_NAME); + + let raw_secret = build_upsert_provider_config( + "default", + OidcProviderConfigInput { + config_url: config.config_url, + client_id: config.client_id, + client_secret: Some(" raw secret ".to_string()), + ..Default::default() + }, + None, + ) + .expect("a nonempty client secret should be accepted"); + assert_eq!(raw_secret.client_secret.as_deref(), Some(" raw secret ")); +} + +#[test] +fn config_input_rejects_invalid_fields() { + let input = OidcProviderConfigInput { + config_url: "https://example.com/.well-known/openid-configuration".to_string(), + client_id: "client".to_string(), + scopes: vec!["profile".to_string()], + ..Default::default() + }; + assert!(matches!( + build_upsert_provider_config("default", input, None), + Err(OidcConfigError::OpenidScopeRequired) + )); + + let input = OidcProviderConfigInput { + config_url: "https://127.0.0.1/.well-known/openid-configuration".to_string(), + client_id: "client".to_string(), + ..Default::default() + }; + assert!(matches!( + build_upsert_provider_config("default", input, None), + Err(OidcConfigError::ForbiddenOutbound { field: "config_url", .. }) + )); + + assert!(serde_json::from_str::(r#"{"unexpected_field":true}"#).is_err()); + assert!(serde_json::from_str::(r#"{"unexpected_field":true}"#).is_err()); + assert!(matches!( + validate_mutable_provider_id("../escape"), + Err(OidcConfigError::InvalidProviderId) + )); +} + +#[test] +fn validation_input_defaults_to_default_provider() { + let input = OidcProviderValidationInput { + provider_id: " ".to_string(), + config_url: "https://example.com/.well-known/openid-configuration".to_string(), + client_id: "client".to_string(), + ..Default::default() + }; + let config = build_validation_provider_config(input).expect("validation input should use default provider"); + assert_eq!(config.id, "default"); +} + +#[test] +fn persisted_config_codec_keeps_instance_keys_and_fields() { + let mut config = ServerConfig::new(); + let mut provider = build_upsert_provider_config( + "default", + OidcProviderConfigInput { + config_url: "https://example.com/.well-known/openid-configuration".to_string(), + issuer: Some("https://issuer.example.com".to_string()), + client_id: "client".to_string(), + client_secret: Some("secret".to_string()), + hide_from_ui: true, + ..Default::default() + }, + None, + ) + .expect("provider should be valid"); + upsert_persisted_provider_config(&mut config, &provider); + + let default = config + .0 + .get(IDENTITY_OPENID_SUB_SYS) + .and_then(|subsystem| subsystem.get(DEFAULT_DELIMITER)) + .expect("default provider KVS should exist"); + assert_eq!(default.get(OIDC_ISSUER), "https://issuer.example.com"); + assert_eq!(default.get(OIDC_HIDE_FROM_UI), EnableState::On.to_string()); + assert_eq!(persisted_provider_secret(&config, "default").as_deref(), Some("secret")); + + provider.hide_from_ui = false; + upsert_persisted_provider_config(&mut config, &provider); + let default = config + .0 + .get(IDENTITY_OPENID_SUB_SYS) + .and_then(|subsystem| subsystem.get(DEFAULT_DELIMITER)) + .expect("updated default provider KVS should exist"); + assert_eq!(default.get(OIDC_HIDE_FROM_UI), EnableState::Off.to_string()); + + delete_persisted_provider_config(&mut config, "default").expect("default provider should be deleted"); + assert!(matches!( + delete_persisted_provider_config(&mut config, "default"), + Err(OidcConfigError::ProviderNotFound) + )); +} + +#[test] +fn site_replication_snapshot_hashes_secret_without_exposing_config() { + let provider = build_upsert_provider_config( + "default", + OidcProviderConfigInput { + config_url: "https://example.com/.well-known/openid-configuration".to_string(), + client_id: "client".to_string(), + client_secret: Some("secret".to_string()), + ..Default::default() + }, + None, + ) + .expect("provider should be valid"); + let snapshot = OidcSiteReplicationSnapshot::from_config_snapshot(&OidcConfigSnapshot::new(vec![SourcedOidcProviderConfig { + config: provider, + source: OidcProviderConfigSource::Persisted, + }])); + assert_eq!( + snapshot.providers()[0].hashed_client_secret, + "K7gNU3sdo-OL0wNhqoVWhr3g6s1xYv72ol_pe_Unols" + ); + assert!(!format!("{snapshot:?}").contains("\"secret\"")); + assert_eq!(hash_client_secret(None), ""); + assert_eq!(hash_client_secret(Some("")), ""); +} diff --git a/crates/iam/src/oidc/mod.rs b/crates/iam/src/oidc/mod.rs new file mode 100644 index 000000000..a171b776e --- /dev/null +++ b/crates/iam/src/oidc/mod.rs @@ -0,0 +1,40 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +//! OpenID Connect configuration, provider discovery, runtime flows, and state. + +mod config; +mod provider; +mod runtime; +mod state; +mod transport; + +#[cfg(any(test, feature = "test-util"))] +mod test_support; + +pub use config::{ + OidcConfigError, OidcConfigQuery, OidcConfigSnapshot, OidcProviderConfig, OidcProviderConfigInput, OidcProviderConfigSource, + OidcProviderValidationInput, OidcProviderValidationResult, OidcSiteReplicationProvider, OidcSiteReplicationSnapshot, + SourcedOidcProviderConfig, build_upsert_provider_config, build_validation_provider_config, console_session_duration_secs, + delete_persisted_provider_config, load_oidc_config_snapshot, load_oidc_provider_configs_from_env, + load_oidc_provider_configs_from_server_config, persisted_provider_secret, upsert_persisted_provider_config, + validate_mutable_provider_id, +}; +pub use provider::validate_oidc_provider_config_with_extra_root_ca; +pub use runtime::{OidcSys, StandardOidcAdapter}; +pub use transport::{ + OidcExtraRootCaMaterial, OidcExtraRootCaProvider, OidcPluginAuthnMetricsSnapshot, oidc_plugin_authn_metrics_snapshot, +}; + +#[cfg(any(test, feature = "test-util"))] +pub use test_support::{make_test_sys, test_config}; diff --git a/crates/iam/src/oidc/provider.rs b/crates/iam/src/oidc/provider.rs new file mode 100644 index 000000000..6b4f083f8 --- /dev/null +++ b/crates/iam/src/oidc/provider.rs @@ -0,0 +1,713 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +use super::{ + config::{OidcProviderConfig, OidcProviderValidationResult, SourcedOidcProviderConfig}, + transport::{DiscoveryRequest, OidcHttpError, ReqwestHttpClient, oidc_extra_root_certs, oidc_http_error_diagnostics}, +}; +use crate::{EVENT_OIDC_DIAGNOSTICS, LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC}; +use openidconnect::core::{CoreIdTokenVerifier, CoreJsonWebKeySet, CoreJwsSigningAlgorithm}; +use openidconnect::{ + AsyncHttpClient, ClientId, ClientSecret, DiscoveryError, IssuerUrl, JsonWebKeySetUrl, ProviderMetadataWithLogout, TokenUrl, +}; +use serde::Deserialize; +use std::collections::HashMap; +use std::sync::RwLock; +use std::time::{Duration as StdDuration, Instant}; +use tokio::time::sleep; +use tracing::{debug, warn}; +use url::Url; + +pub(super) const OIDC_JWKS_REFRESH_INTERVAL: StdDuration = StdDuration::from_secs(24 * 60 * 60); +pub(super) const OIDC_DISCOVERY_TRANSPORT_RETRIES: usize = 3; +pub(super) const OIDC_DISCOVERY_TRANSPORT_RETRY_DELAY: StdDuration = StdDuration::from_millis(50); +pub(super) const OIDC_JWKS_BLOCKED_BY_OUTBOUND_POLICY: &str = "JWKS request blocked by outbound policy"; +pub(super) const OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY: &str = "OIDC provider discovery blocked by outbound policy"; + +/// Preserve recovery eligibility before discovery errors become public diagnostic strings. +#[derive(Debug, thiserror::Error)] +#[error("{message}")] +pub(super) struct DiscoveryFailure { + message: String, + pub(super) retryable: bool, + pub(super) diagnostics: (&'static str, String), +} + +impl From for DiscoveryFailure { + fn from(message: String) -> Self { + Self { + message, + retryable: false, + diagnostics: ("validation", String::new()), + } + } +} + +fn retryable_discovery_status(status: http::StatusCode) -> bool { + status.is_server_error() || matches!(status, http::StatusCode::REQUEST_TIMEOUT | http::StatusCode::TOO_MANY_REQUESTS) +} + +fn retryable_discovery_transport(error: &OidcHttpError) -> bool { + matches!(error, OidcHttpError::Reqwest(error) if !error.is_builder()) +} + +fn discovery_failure(error: DiscoveryError, context: &str, blocked_context: &str) -> DiscoveryFailure { + let retryable = match &error { + DiscoveryError::Request(error) => retryable_discovery_transport(error), + DiscoveryError::Response(status, _, _) => retryable_discovery_status(*status), + _ => false, + }; + let diagnostics = match &error { + DiscoveryError::Request(error) => oidc_http_error_diagnostics(error), + DiscoveryError::Response(status, _, _) => ("http_status", status.as_u16().to_string()), + _ => ("validation", String::new()), + }; + let message = match error { + DiscoveryError::Request(OidcHttpError::ForbiddenOutbound(reason)) => format!("{blocked_context}: {reason}"), + error => format!("{context}: {error}"), + }; + DiscoveryFailure { + message, + retryable, + diagnostics, + } +} + +// ---- Internal provider state ---- + +/// Discovered OIDC provider metadata. +/// We store metadata (which includes JWKS after discovery) separately rather than +/// a `CoreClient` because the crate uses type-state generics that make storing +/// the configured client in a HashMap impractical. The client is reconstructed +/// on-the-fly from metadata when needed. +#[derive(Clone)] +pub(super) struct ProviderState { + pub(super) metadata: DiscoveredProviderMetadata, + pub(super) discovered_at: Instant, +} + +pub(super) struct ProviderRuntime { + states: RwLock>, + http_client: ReqwestHttpClient, +} + +// Workload issuers do not implement the browser authorization flow. Keep their +// verification metadata separate rather than inventing an authorization URL. +#[serde_with::serde_as] +#[derive(Clone, Deserialize)] +pub(super) struct WorkloadProviderMetadata { + issuer: IssuerUrl, + jwks_uri: JsonWebKeySetUrl, + token_endpoint: Option, + #[serde_as(as = "serde_with::VecSkipError<_>")] + id_token_signing_alg_values_supported: Vec, + #[serde(skip)] + jwks: CoreJsonWebKeySet, + // Discovery is extensible; report unsupported fields without logging values. + #[serde(flatten)] + additional_fields: HashMap, +} + +#[derive(Clone)] +pub(super) enum DiscoveredProviderMetadata { + Console(Box), + Workload(Box), +} + +impl DiscoveredProviderMetadata { + pub(super) fn parse(body: &[u8], hide_from_ui: bool) -> Result { + let document: serde_json::Value = serde_json::from_slice(body).map_err(|err| err.to_string())?; + if hide_from_ui + && document + .as_object() + .is_some_and(|fields| !fields.contains_key("authorization_endpoint")) + { + let mut metadata: WorkloadProviderMetadata = serde_json::from_slice(body).map_err(|err| err.to_string())?; + if !metadata.additional_fields.is_empty() { + warn!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "workload_discovery_additional_fields", + field_count = metadata.additional_fields.len(), + "workload discovery contains additional fields" + ); + metadata.additional_fields.clear(); + } + Ok(Self::Workload(Box::new(metadata))) + } else { + serde_json::from_slice(body) + .map(|metadata| Self::Console(Box::new(metadata))) + .map_err(|err| err.to_string()) + } + } + + pub(super) fn console(&self) -> Result<&ProviderMetadataWithLogout, String> { + match self { + Self::Console(metadata) => Ok(metadata), + Self::Workload(_) => Err("OIDC provider has no authorization endpoint; only web identity is supported".into()), + } + } + + pub(super) fn issuer(&self) -> &IssuerUrl { + match self { + Self::Console(metadata) => metadata.issuer(), + Self::Workload(metadata) => &metadata.issuer, + } + } + + pub(super) fn jwks_uri(&self) -> &JsonWebKeySetUrl { + match self { + Self::Console(metadata) => metadata.jwks_uri(), + Self::Workload(metadata) => &metadata.jwks_uri, + } + } + + pub(super) fn set_jwks(self, jwks: CoreJsonWebKeySet) -> Self { + match self { + Self::Console(metadata) => Self::Console(Box::new(metadata.set_jwks(jwks))), + Self::Workload(mut metadata) => { + metadata.jwks = jwks; + Self::Workload(metadata) + } + } + } + + pub(super) fn authorization_endpoint(&self) -> Option { + match self { + Self::Console(metadata) => Some(metadata.authorization_endpoint().to_string()), + Self::Workload(_) => None, + } + } + + pub(super) fn token_endpoint(&self) -> Option<&TokenUrl> { + match self { + Self::Console(metadata) => metadata.token_endpoint(), + Self::Workload(metadata) => metadata.token_endpoint.as_ref(), + } + } + + pub(super) fn verifier(&self, config: &OidcProviderConfig) -> CoreIdTokenVerifier<'static> { + let client_id = ClientId::new(config.client_id.clone()); + let secret = config.client_secret.as_ref().map(|secret| ClientSecret::new(secret.clone())); + let (issuer, jwks, algorithms) = match self { + Self::Console(metadata) => (metadata.issuer(), metadata.jwks(), metadata.id_token_signing_alg_values_supported()), + Self::Workload(metadata) => (&metadata.issuer, &metadata.jwks, &metadata.id_token_signing_alg_values_supported), + }; + let verifier = match secret { + Some(secret) => CoreIdTokenVerifier::new_confidential_client(client_id, secret, issuer.clone(), jwks.clone()), + None => CoreIdTokenVerifier::new_public_client(client_id, issuer.clone(), jwks.clone()), + }; + verifier.set_allowed_algs(algorithms.clone()) + } +} + +// This adapter is used only for discovery/JWKS fetches, never token exchange. +struct JwksAcceptClient<'a> { + inner: &'a ReqwestHttpClient, + discovery_url: Option, +} + +impl<'c> AsyncHttpClient<'c> for JwksAcceptClient<'_> { + type Error = OidcHttpError; + type Future = >::Future; + + fn call(&'c self, mut request: http::Request>) -> Self::Future { + if !self.discovery_url.as_ref().is_some_and(|url| request.uri() == url.as_str()) { + request.headers_mut().insert( + http::header::ACCEPT, + http::HeaderValue::from_static("application/json, application/jwk-set+json"), + ); + } + request.extensions_mut().insert(DiscoveryRequest); + self.inner.call(request) + } +} + +impl ProviderState { + fn is_stale(&self) -> bool { + self.discovered_at.elapsed() >= OIDC_JWKS_REFRESH_INTERVAL + } +} + +impl ProviderRuntime { + pub(super) fn new(http_client: ReqwestHttpClient, states: HashMap) -> Self { + Self { + states: RwLock::new(states), + http_client, + } + } + + pub(super) fn http_client(&self) -> &ReqwestHttpClient { + &self.http_client + } + + /// Find a provider whose discovered issuer matches the given JWT issuer string. + pub(super) fn find_provider_by_issuer( + &self, + issuer: &str, + configs: &HashMap, + ) -> Option<(String, OidcProviderConfig, ProviderState)> { + let (issuer_scheme, issuer_host, issuer_port, issuer_path) = normalize_issuer(issuer)?; + let map = self + .states + .read() + .map_err(|e| format!("provider state lock poisoned: {e}")) + .ok()?; + for (id, state) in map.iter() { + let provider_issuer = state.metadata.issuer().as_str(); + let Some((provider_scheme, provider_host, provider_port, provider_path)) = normalize_issuer(provider_issuer) else { + continue; + }; + + if issuer_scheme == provider_scheme + && issuer_host == provider_host + && issuer_port == provider_port + && issuer_path == provider_path + && let Some(config) = configs.get(id) + { + return Some((id.clone(), config.config.clone(), state.clone())); + } + } + None + } + + pub(super) fn get_provider_state(&self, provider_id: &str) -> Result { + self.states + .read() + .map_err(|e| format!("provider state lock poisoned: {e}"))? + .get(provider_id) + .cloned() + .ok_or_else(|| format!("provider not discovered: {provider_id}")) + } + + pub(super) async fn refresh_provider_state( + &self, + provider_id: &str, + config: &OidcProviderConfig, + ) -> Result { + self.recover_provider_state(provider_id, config) + .await + .map_err(|err| err.to_string()) + } + + pub(super) async fn recover_provider_state( + &self, + provider_id: &str, + config: &OidcProviderConfig, + ) -> Result { + let state = discover_provider_for_recovery(config, &self.http_client).await?; + let mut map = self.states.write().map_err(|e| { + let msg = e.to_string(); + format!("provider state lock poisoned: {msg}") + })?; + map.insert(provider_id.to_string(), state.clone()); + + Ok(state) + } + + pub(super) async fn ensure_provider_state( + &self, + provider_id: &str, + config: &OidcProviderConfig, + ) -> Result { + let state = self.get_provider_state(provider_id)?; + if state.is_stale() { + self.refresh_provider_state(provider_id, config).await.or_else(|refresh_err| { + warn!( + "OIDC provider '{}' JWKS metadata refresh skipped due to transient network issue: {}", + provider_id, refresh_err + ); + Ok(state) + }) + } else { + Ok(state) + } + } + + pub(super) async fn ensure_provider_state_if_stale( + &self, + provider_id: &str, + config: &OidcProviderConfig, + state: &ProviderState, + ) -> Result { + if state.is_stale() { + self.refresh_provider_state(provider_id, config).await.or_else(|refresh_err| { + warn!( + "OIDC provider '{}' JWKS metadata refresh skipped due to transient network issue: {}", + provider_id, refresh_err + ); + Ok(state.clone()) + }) + } else { + Ok(state.clone()) + } + } +} + +/// Perform OIDC discovery for a provider. +/// `discover_async` fetches the discovery document and JWKS in one step. +pub(super) async fn discover_provider( + config: &OidcProviderConfig, + http_client: &ReqwestHttpClient, +) -> Result { + discover_provider_for_recovery(config, http_client) + .await + .map_err(|err| err.to_string()) +} + +pub(super) async fn discover_provider_for_recovery( + config: &OidcProviderConfig, + http_client: &ReqwestHttpClient, +) -> Result { + if let Some(issuer) = config.issuer.as_deref().filter(|issuer| !issuer.trim().is_empty()) { + return discover_provider_from_config_url(config, issuer, http_client).await; + } + + // The openidconnect crate expects the issuer URL (base), not the + // .well-known/openid-configuration URL. + let base_issuer = normalize_config_url(&config.config_url)?; + let candidates = issuer_candidates(&base_issuer); + let mut last_errors = Vec::new(); + let mut retryable = false; + let mut diagnostics = ("validation", String::new()); + + for candidate_issuer in candidates.iter() { + let issuer_url = IssuerUrl::new(candidate_issuer.clone()).map_err(|e| format!("invalid issuer URL: {e}"))?; + + for attempt in 0..OIDC_DISCOVERY_TRANSPORT_RETRIES { + let discovered = if config.hide_from_ui { + discover_provider_from_config_url(config, candidate_issuer, http_client).await + } else { + let client = JwksAcceptClient { + inner: http_client, + discovery_url: Some( + issuer_url + .join(".well-known/openid-configuration") + .map_err(|err| err.to_string())?, + ), + }; + ProviderMetadataWithLogout::discover_async(issuer_url.clone(), &client) + .await + .map(|metadata| ProviderState { + metadata: DiscoveredProviderMetadata::Console(Box::new(metadata)), + discovered_at: Instant::now(), + }) + .map_err(|err| discovery_failure(err, "discovery failed", OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY)) + }; + match discovered { + Ok(state) => return Ok(state), + Err(error) + if error.message.starts_with(OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY) + || error.message.starts_with(OIDC_JWKS_BLOCKED_BY_OUTBOUND_POLICY) => + { + return Err(error); + } + Err(error) => { + let is_transient_transport = error.message.contains("Request failed"); + let should_retry = is_transient_transport && attempt + 1 < OIDC_DISCOVERY_TRANSPORT_RETRIES; + if should_retry { + debug!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "provider_discovery_transport_retry", + provider_id = %config.id, + config_url = %config.config_url, + issuer_candidate = %candidate_issuer, + attempt = attempt + 1, + max_attempts = OIDC_DISCOVERY_TRANSPORT_RETRIES, + error = %error, + "oidc provider discovery failed" + ); + sleep(OIDC_DISCOVERY_TRANSPORT_RETRY_DELAY).await; + continue; + } + + if error.retryable || !retryable { + diagnostics = error.diagnostics.clone(); + } + retryable |= error.retryable; + last_errors.push(format!("issuer '{candidate_issuer}': {error}")); + debug!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "provider_discovery_candidate_failed", + provider_id = %config.id, + config_url = %config.config_url, + issuer_candidate = %candidate_issuer, + attempt = attempt + 1, + max_attempts = OIDC_DISCOVERY_TRANSPORT_RETRIES, + error = %error, + "oidc provider discovery failed" + ); + break; + } + } + } + } + + Err(DiscoveryFailure { + message: format!("discovery failed for all issuer variants {:?}: {}", candidates, last_errors.join("; ")), + retryable, + diagnostics, + }) +} + +pub(super) async fn validate_oidc_provider_config(config: &OidcProviderConfig) -> Result { + let http_client = ReqwestHttpClient::new()?; + validate_oidc_provider_config_with_http_client(config, &http_client).await +} + +pub async fn validate_oidc_provider_config_with_extra_root_ca( + config: &OidcProviderConfig, + root_ca_pem: Option<&[u8]>, +) -> Result { + if root_ca_pem.is_none() { + return validate_oidc_provider_config(config).await; + } + + let http_client = ReqwestHttpClient::new_with_extra_root_certs(oidc_extra_root_certs(root_ca_pem)?)?; + validate_oidc_provider_config_with_http_client(config, &http_client).await +} + +async fn validate_oidc_provider_config_with_http_client( + config: &OidcProviderConfig, + http_client: &ReqwestHttpClient, +) -> Result { + let state = discover_provider(config, http_client).await?; + + Ok(OidcProviderValidationResult { + issuer: state.metadata.issuer().to_string(), + authorization_endpoint: state.metadata.authorization_endpoint(), + token_endpoint: state.metadata.token_endpoint().map(ToString::to_string), + }) +} + +async fn discover_provider_from_config_url( + config: &OidcProviderConfig, + issuer: &str, + http_client: &ReqwestHttpClient, +) -> Result { + let issuer_url = IssuerUrl::new(issuer.trim().to_string()).map_err(|e| format!("invalid issuer URL: {e}"))?; + let explicit_issuer = config.issuer.as_deref().is_some_and(|issuer| !issuer.trim().is_empty()); + let discovery_url = if explicit_issuer { + discovery_url_from_config_url(&config.config_url)? + } else { + issuer_url + .join(".well-known/openid-configuration") + .map_err(|err| err.to_string())? + }; + let request = http::Request::builder() + .extension(DiscoveryRequest) + .uri(discovery_url.to_string()) + .method(http::Method::GET) + .header(http::header::ACCEPT, "application/json") + .body(Vec::new()) + .map_err(|err| format!("failed to prepare discovery request: {err}"))?; + + let response = match http_client.call(request).await { + Ok(response) => response, + Err(OidcHttpError::ForbiddenOutbound(reason)) => { + return Err(format!("{OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY}: {reason}").into()); + } + Err(err) => { + return Err(DiscoveryFailure { + retryable: retryable_discovery_transport(&err), + diagnostics: oidc_http_error_diagnostics(&err), + message: format!("discovery request failed: Request failed: {err}"), + }); + } + }; + if response.status() != http::StatusCode::OK { + return Err(DiscoveryFailure { + retryable: retryable_discovery_status(response.status()), + diagnostics: ("http_status", response.status().as_u16().to_string()), + message: format!("discovery failed: HTTP status code {} at {}", response.status(), discovery_url), + }); + } + + if !explicit_issuer + && let Some(content_type) = response.headers().get(http::header::CONTENT_TYPE) + && !content_type.to_str().ok().is_some_and(|value| { + value + .split(';') + .next() + .is_some_and(|essence| essence.eq_ignore_ascii_case("application/json")) + }) + { + return Err("Unexpected response Content-Type: expected application/json" + .to_string() + .into()); + } + + let provider_metadata = DiscoveredProviderMetadata::parse(response.body(), config.hide_from_ui) + .map_err(|err| format!("failed to parse discovery response: {err}"))?; + if provider_metadata.issuer() != &issuer_url { + return Err(format!( + "unexpected issuer URI `{}` (expected `{}`)", + provider_metadata.issuer().as_str(), + issuer_url.as_str() + ) + .into()); + } + + let jwks_url = if explicit_issuer { + jwks_url_from_config_url(&config.config_url, &issuer_url, provider_metadata.jwks_uri())? + } else { + provider_metadata.jwks_uri().clone() + }; + let jwks = match CoreJsonWebKeySet::fetch_async( + &jwks_url, + &JwksAcceptClient { + inner: http_client, + discovery_url: None, + }, + ) + .await + { + Ok(jwks) => jwks, + Err(err) => return Err(discovery_failure(err, "failed to fetch JWKS", OIDC_JWKS_BLOCKED_BY_OUTBOUND_POLICY)), + }; + + Ok(ProviderState { + metadata: provider_metadata.set_jwks(jwks), + discovered_at: Instant::now(), + }) +} + +pub(super) fn normalize_issuer(raw: &str) -> Option<(String, String, u16, String)> { + let parsed = Url::parse(raw).ok()?; + if parsed.scheme() != "http" && parsed.scheme() != "https" { + return None; + } + + let host = parsed.host_str()?.to_ascii_lowercase(); + let port = parsed.port_or_known_default()?; + let normalized_path = { + let path = parsed.path().trim_end_matches('/').to_string(); + if path.is_empty() { "/".to_string() } else { path } + }; + + Some((parsed.scheme().to_string(), host, port, normalized_path)) +} + +pub(super) fn normalize_config_url(config_url: &str) -> Result { + let config_url = config_url.trim(); + let url = Url::parse(config_url).map_err(|e| format!("invalid config_url: {e}"))?; + if url.scheme() != "http" && url.scheme() != "https" { + return Err(format!("invalid config_url scheme: {}", url.scheme())); + } + let host = url.host_str().ok_or_else(|| "config_url missing host".to_string())?; + let path = url.path(); + + // Strip `/.well-known/openid-configuration` (with optional trailing slash) if present. + // Everything else is preserved exactly so the issuer URL matches the provider's discovery + // document (e.g. Authentik includes a trailing slash, Keycloak does not). + let normalized_path = path + .strip_suffix('/') + .unwrap_or(path) + .strip_suffix("/.well-known/openid-configuration") + .unwrap_or(if path == "/" { "" } else { path }); + + if normalized_path.contains("/.well-known/") { + return Err("config_url uses an unsupported .well-known discovery URL".into()); + } + + let mut issuer = format!("{}://{host}", url.scheme()); + if let Some(port) = url.port() { + issuer.push(':'); + issuer.push_str(&port.to_string()); + } + + if !normalized_path.is_empty() { + issuer.push_str(normalized_path); + } + + Ok(issuer) +} + +pub(super) fn discovery_url_from_config_url(config_url: &str) -> Result { + let mut url = Url::parse(config_url.trim()).map_err(|e| format!("invalid config_url: {e}"))?; + if url.scheme() != "http" && url.scheme() != "https" { + return Err(format!("invalid config_url scheme: {}", url.scheme())); + } + if url.host_str().is_none() { + return Err("config_url missing host".to_string()); + } + + let path = url.path().to_string(); + let without_trailing_slash = path.strip_suffix('/').unwrap_or(&path); + if without_trailing_slash.ends_with("/.well-known/openid-configuration") { + url.set_path(without_trailing_slash); + return Ok(url); + } + if without_trailing_slash.contains("/.well-known/") { + return Err("config_url uses an unsupported .well-known discovery URL".into()); + } + + let discovery_path = if without_trailing_slash.is_empty() || without_trailing_slash == "/" { + "/.well-known/openid-configuration".to_string() + } else { + format!("{without_trailing_slash}/.well-known/openid-configuration") + }; + url.set_path(&discovery_path); + Ok(url) +} + +pub(super) fn jwks_url_from_config_url( + config_url: &str, + issuer_url: &IssuerUrl, + jwks_url: &JsonWebKeySetUrl, +) -> Result { + let issuer = issuer_url.url(); + let jwks = jwks_url.url(); + if issuer.origin() != jwks.origin() { + return Ok(jwks_url.clone()); + } + + let issuer_path = issuer.path().trim_end_matches('/'); + let Some(suffix) = jwks.path().strip_prefix(issuer_path) else { + return Ok(jwks_url.clone()); + }; + if !suffix.is_empty() && !suffix.starts_with('/') { + return Ok(jwks_url.clone()); + } + + let mut internal_url = + Url::parse(&normalize_config_url(config_url)?).map_err(|err| format!("invalid config_url issuer base: {err}"))?; + let internal_path = internal_url.path().trim_end_matches('/'); + internal_url.set_path(&format!("{internal_path}{suffix}")); + internal_url.set_query(jwks.query()); + Ok(JsonWebKeySetUrl::from_url(internal_url)) +} + +pub(super) fn issuer_candidates(base: &str) -> Vec { + let original = base.trim(); + let mut variants = Vec::with_capacity(2); + variants.push(original.to_string()); + + let toggled = if original.ends_with('/') { + original.trim_end_matches('/').to_string() + } else { + format!("{original}/") + }; + variants.push(toggled); + + variants +} + +#[cfg(test)] +#[path = "provider_tests.rs"] +mod tests; diff --git a/crates/iam/src/oidc/provider_tests.rs b/crates/iam/src/oidc/provider_tests.rs new file mode 100644 index 000000000..849b0d4d5 --- /dev/null +++ b/crates/iam/src/oidc/provider_tests.rs @@ -0,0 +1,713 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use crate::oidc::{runtime::*, state::*, test_support::*, transport::*}; +use jsonwebtoken::{Algorithm, EncodingKey, Header}; +use openidconnect::core::CoreIdToken; +use openidconnect::{IssuerUrl, JsonWebKeySetUrl, Nonce}; +use rustfs_utils::egress::{ENV_OUTBOUND_ALLOW_ORIGINS, OutboundPolicy}; +use std::collections::HashMap; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; +use url::Url; + +#[test] +fn test_normalize_issuer_matches() { + let lhs = normalize_issuer("https://idp.example.com/.well-known/openid-configuration/").unwrap(); + let rhs = normalize_issuer("https://idp.example.com/.well-known/openid-configuration").unwrap(); + assert_eq!(lhs, rhs); + assert_eq!( + lhs, + ( + "https".to_string(), + "idp.example.com".to_string(), + 443, + "/.well-known/openid-configuration".to_string() + ) + ); +} +#[test] +fn test_normalize_config_url() { + // --- Well-known suffix stripping --- + // Bare well-known URL → stripped to just the host + assert_eq!( + normalize_config_url("https://idp.example.com/.well-known/openid-configuration").unwrap(), + "https://idp.example.com" + ); + // Trailing slash after well-known suffix is also stripped + assert_eq!( + normalize_config_url("https://idp.example.com/.well-known/openid-configuration/").unwrap(), + "https://idp.example.com" + ); + // Well-known under a sub-path (Keycloak realms) + assert_eq!( + normalize_config_url("https://keycloak.example.com/realms/myrealm/.well-known/openid-configuration").unwrap(), + "https://keycloak.example.com/realms/myrealm" + ); + + // --- Providers WITHOUT trailing slash (Keycloak, Auth0, Okta, Google) --- + assert_eq!( + normalize_config_url("https://keycloak.example.com/realms/myrealm").unwrap(), + "https://keycloak.example.com/realms/myrealm" + ); + assert_eq!( + normalize_config_url("https://idp.example.com/custom/realm").unwrap(), + "https://idp.example.com/custom/realm" + ); + + // --- Providers WITH trailing slash (Authentik) --- + assert_eq!( + normalize_config_url("https://auth.example.com/application/o/myapp/").unwrap(), + "https://auth.example.com/application/o/myapp/" + ); + + // --- Root-level issuer (bare host) --- + assert_eq!(normalize_config_url("https://idp.example.com").unwrap(), "https://idp.example.com"); + assert_eq!(normalize_config_url("https://idp.example.com/").unwrap(), "https://idp.example.com"); + + // --- Custom port --- + assert_eq!( + normalize_config_url("https://idp.example.com:8443/auth/realms/test").unwrap(), + "https://idp.example.com:8443/auth/realms/test" + ); + assert_eq!( + normalize_config_url("http://localhost:8080/application/o/app/").unwrap(), + "http://localhost:8080/application/o/app/" + ); + + // --- Error cases --- + assert!(normalize_config_url("https://idp.example.com/.well-known/invalid").is_err()); + assert!(normalize_config_url("gopher://idp.example.com").is_err()); + assert!(normalize_config_url("not-a-url").is_err()); +} +#[test] +fn test_discovery_url_from_config_url() { + assert_eq!( + discovery_url_from_config_url("https://idp.example.com/.well-known/openid-configuration") + .expect("config URL should parse") + .as_str(), + "https://idp.example.com/.well-known/openid-configuration" + ); + assert_eq!( + discovery_url_from_config_url("https://idp.example.com/realms/app") + .expect("issuer URL should derive discovery URL") + .as_str(), + "https://idp.example.com/realms/app/.well-known/openid-configuration" + ); + assert!(discovery_url_from_config_url("https://idp.example.com/.well-known/not-openid").is_err()); +} +#[test] +fn test_issuer_candidates() { + assert_eq!( + issuer_candidates("https://idp.example.com/realm"), + vec![ + "https://idp.example.com/realm".to_string(), + "https://idp.example.com/realm/".to_string() + ] + ); + assert_eq!( + issuer_candidates("https://idp.example.com/realm/"), + vec![ + "https://idp.example.com/realm/".to_string(), + "https://idp.example.com/realm".to_string() + ] + ); + assert_eq!( + issuer_candidates("https://idp.example.com"), + vec!["https://idp.example.com".to_string(), "https://idp.example.com/".to_string()] + ); +} +#[test] +fn workload_metadata_requires_hidden_provider_and_valid_verification_fields() { + let document = serde_json::json!({ + "issuer": "https://issuer.example.com", + "jwks_uri": "https://issuer.example.com/jwks", + "response_types_supported": ["id_token"], + "subject_types_supported": ["public"], + "id_token_signing_alg_values_supported": ["ES256"], + }); + let parse = + |value: &serde_json::Value, hidden| DiscoveredProviderMetadata::parse(&serde_json::to_vec(value).unwrap(), hidden); + let metadata = parse(&document, true).expect("hidden workload metadata should parse"); + assert!(metadata.authorization_endpoint().is_none()); + assert!(metadata.console().err().unwrap().contains("only web identity")); + assert!(parse(&document, false).err().unwrap().contains("authorization_endpoint")); + for field in ["issuer", "jwks_uri", "id_token_signing_alg_values_supported"] { + let mut invalid = document.clone(); + invalid.as_object_mut().unwrap().remove(field); + assert!(parse(&invalid, true).err().unwrap().contains(field), "missing {field}"); + } + for endpoint in [serde_json::Value::Null, serde_json::json!(""), serde_json::json!("not a URL")] { + let mut invalid = document.clone(); + invalid["authorization_endpoint"] = endpoint; + assert!(parse(&invalid, true).is_err(), "invalid endpoint must not select workload metadata"); + } + let mut complete = document; + complete["authorization_endpoint"] = serde_json::json!("https://issuer.example.com/authorize"); + for hidden in [true, false] { + let metadata = parse(&complete, hidden).expect("full providers keep the existing parser"); + assert!(metadata.console().is_ok()); + assert!(metadata.token_endpoint().is_none(), "token endpoint remains optional"); + } +} +#[test] +fn workload_metadata_rejects_duplicate_fields() { + for hidden in [false, true] { + let document = format!( + r#"{{"issuer":"https://wrong.example.com","issuer":"https://issuer.example.com",{}"jwks_uri":"https://issuer.example.com/jwks","response_types_supported":["id_token"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["ES256"]}}"#, + if hidden { + "" + } else { + r#""authorization_endpoint":"https://issuer.example.com/authorize","# + }, + ); + let error = DiscoveredProviderMetadata::parse(document.as_bytes(), hidden) + .err() + .expect("duplicate issuer must fail"); + assert!(error.contains("duplicate field"), "{error}"); + } +} +#[test] +fn workload_verifier_preserves_algorithm_secret_and_audience_policy() { + let secret = Nonce::new_random().secret().to_string(); + let mut config = build_mocked_oidc_provider_config("workload", "https://issuer.example.com"); + config.client_secret = Some(secret.clone()); + config.other_audiences = vec!["additional-audience".into()]; + let now = time::OffsetDateTime::now_utc().unix_timestamp(); + let payload = serde_json::json!({ + "iss": config.config_url, "sub": "repo:example/project:ref:refs/heads/main", + "aud": [config.client_id, "additional-audience"], "iat": now, "exp": now + 300, + }); + let signed = + jsonwebtoken::encode(&Header::new(Algorithm::HS256), &payload, &EncodingKey::from_secret(secret.as_bytes())).unwrap(); + let token: CoreIdToken = signed.parse().unwrap(); + for workload in [false, true] { + for (algorithms, accepted) in [ + (serde_json::json!(["HS256", "unsupported-future-algorithm"]), true), + (serde_json::json!(["ES256"]), false), + (serde_json::json!(["unsupported-future-algorithm"]), false), + ] { + let mut document = serde_json::json!({ + "issuer": config.config_url, "jwks_uri": "https://issuer.example.com/jwks", + "response_types_supported": ["id_token"], "subject_types_supported": ["public"], + "id_token_signing_alg_values_supported": algorithms, + }); + if !workload { + document["authorization_endpoint"] = serde_json::json!("https://issuer.example.com/authorize"); + } + let metadata = DiscoveredProviderMetadata::parse(&serde_json::to_vec(&document).unwrap(), true).unwrap(); + let verifier = metadata + .verifier(&config) + .set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud)); + assert_eq!( + token.claims(&verifier, |_: Option<&Nonce>| Ok(())).is_ok(), + accepted, + "workload={workload}, algorithms={algorithms}" + ); + if accepted { + assert!( + token.claims(&metadata.verifier(&config), |_: Option<&Nonce>| Ok(())).is_err(), + "additional audiences require explicit trust" + ); + let mut wrong_secret = config.clone(); + wrong_secret.client_secret = Some(Nonce::new_random().secret().to_string()); + let verifier = metadata + .verifier(&wrong_secret) + .set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud)); + assert!( + token.claims(&verifier, |_: Option<&Nonce>| Ok(())).is_err(), + "incorrect client secret must fail" + ); + } + } + } +} +#[tokio::test] +async fn workload_discovery_stops_after_forbidden_jwks() { + let requests = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let seen = Arc::clone(&requests); + let (base, handle) = start_mock_oidc_discovery_server_with_jwks( + |base| (base.to_string(), "http://192.168.65.254:8080/jwks".into(), "/jwks".into()), + 2, + "ES256", + true, + move |_| { + seen.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + serde_json::json!({"keys": []}).to_string() + }, + ) + .expect("workload discovery mock must bind"); + let mut config = build_mocked_oidc_provider_config("workload", &base); + config.hide_from_ui = true; + let client = ReqwestHttpClient::with_policy(OutboundPolicy::from_allowed_origins(&base).unwrap()); + let error = discover_provider(&config, &client) + .await + .err() + .expect("private JWKS must be blocked"); + assert!(error.starts_with(OIDC_JWKS_BLOCKED_BY_OUTBOUND_POLICY), "{error}"); + handle.join().unwrap(); + assert_eq!( + requests.load(std::sync::atomic::Ordering::SeqCst), + 1, + "a policy denial must not retry discovery" + ); +} +#[tokio::test] +async fn workload_discovery_verification_and_rotation() { + for explicit_issuer in [false, true] { + let (_, initial_jwk) = oidc_es256_key_and_jwk("initial"); + let (key, rotated_jwk) = oidc_es256_key_and_jwk("rotated"); + let initial_jwks = serde_json::json!({"keys": [initial_jwk]}).to_string(); + let rotated_jwks = serde_json::json!({"keys": [rotated_jwk]}).to_string(); + let (base, handle) = start_mock_oidc_discovery_server_with_jwks( + |base| (base.to_string(), format!("{base}/jwks"), "/jwks".into()), + 4, + "ES256", + true, + move |fetch| { + if fetch == 0 { + initial_jwks.clone() + } else { + rotated_jwks.clone() + } + }, + ) + .expect("workload discovery mock must bind"); + let mut config = build_mocked_oidc_provider_config("workload", &base); + config.hide_from_ui = true; + if explicit_issuer { + config.issuer = Some(base.clone()); + } + let http_client = ReqwestHttpClient::with_policy(OutboundPolicy::from_allowed_origins(&base).unwrap()); + let state = discover_provider(&config, &http_client) + .await + .expect("workload discovery must succeed"); + assert!(state.metadata.authorization_endpoint().is_none()); + let sys = OidcSys { + configs: HashMap::from([(config.id.clone(), test_sourced_config(config.clone()))]), + provider_runtime: ProviderRuntime::new(http_client, HashMap::from([(config.id.clone(), state)])), + state_store: OidcStateStore::new(), + pending_discovery: Default::default(), + }; + assert!(sys.has_providers()); + assert!(sys.provider_configs().all(|provider| provider.hide_from_ui)); + let error = sys + .authorize_url(&config.id, "https://console.example.com/callback", None) + .await + .unwrap_err(); + assert!(error.contains("only web identity")); + let now = time::OffsetDateTime::now_utc().unix_timestamp(); + let payload = serde_json::json!({ + "iss": base, "sub": "system:serviceaccount:default:reader", "aud": [config.client_id], + "iat": now, "exp": now + 300, "groups": ["readonly"], + "kubernetes.io": {"namespace": "default", "serviceaccount": {"name": "reader"}}, + }); + let mut header = Header::new(Algorithm::ES256); + header.kid = Some("rotated".into()); + let token = jsonwebtoken::encode(&header, &payload, &key).unwrap(); + let (claims, provider) = sys + .verify_web_identity_token(&token) + .await + .expect("rotation must retain workload discovery support"); + assert_eq!(provider, config.id); + assert_eq!(claims.sub, "system:serviceaccount:default:reader"); + assert_eq!(claims.groups, ["readonly"]); + // Repeat after the mock exits: the verified snapshot must be cached. + handle.join().unwrap(); + assert!(sys.verify_web_identity_token(&token).await.is_ok()); + for (field, value, expected) in [ + ("iss", serde_json::json!("https://wrong.example.com"), "issuer"), + ("aud", serde_json::json!("wrong-audience"), "audience"), + ("exp", serde_json::json!(now - 60), "expired"), + ] { + let mut invalid = payload.clone(); + invalid[field] = value; + let token = jsonwebtoken::encode(&header, &invalid, &key).unwrap(); + let error = sys + .verify_web_identity_token(&token) + .await + .expect_err("invalid workload token must fail"); + assert!(error.to_lowercase().contains(expected), "{field}: {error}"); + } + let (wrong_key, _) = oidc_es256_key_and_jwk("wrong"); + let invalid = jsonwebtoken::encode(&header, &payload, &wrong_key).unwrap(); + let error = sys + .verify_web_identity_token(&invalid) + .await + .expect_err("wrong signature must fail"); + assert!(error.to_lowercase().contains("signature"), "{error}"); + assert!( + sys.verify_web_identity_token(&token).await.is_ok(), + "failed refresh must preserve the cached keys" + ); + } +} +#[tokio::test] +async fn oidc_discovery_accepts_extra_root_ca_for_https_provider() { + let Some((base, ca_pem, handle)) = start_mock_oidc_tls_discovery_server( + |base| (format!("{base}/application/o/rustfs"), format!("{base}/jwks"), "/jwks".to_string()), + 4, + ) else { + return; + }; + let config_url = format!("{base}/application/o/rustfs"); + let config = build_mocked_oidc_provider_config("default", &config_url); + let origin = Url::parse(&config.config_url) + .expect("mock config_url should parse") + .origin() + .ascii_serialization(); + let policy = OutboundPolicy::from_allowed_origins(&origin).expect("loopback TLS origin should be allowed"); + let extra_root_certs = + parse_oidc_extra_root_certs("test OIDC TLS CA", ca_pem.as_bytes()).expect("test CA bundle should parse"); + let http_client = ReqwestHttpClient::with_policy_and_extra_root_certs(policy, extra_root_certs); + + let state = discover_provider(&config, &http_client) + .await + .expect("OIDC discovery should trust the extra root CA"); + + assert_eq!(state.metadata.issuer().to_string(), format!("{base}/application/o/rustfs")); + assert!(handle.join().is_ok()); +} +#[tokio::test] +async fn oidc_discovery_refreshes_extra_root_ca_when_generation_changes() { + let Some((base_a, ca_pem_a, handle_a)) = start_mock_oidc_tls_discovery_server( + |base| (format!("{base}/application/o/rustfs-a"), format!("{base}/jwks"), "/jwks".to_string()), + 4, + ) else { + return; + }; + let Some((base_b, ca_pem_b, handle_b)) = start_mock_oidc_tls_discovery_server( + |base| (format!("{base}/application/o/rustfs-b"), format!("{base}/jwks"), "/jwks".to_string()), + 4, + ) else { + assert!(handle_a.join().is_ok()); + return; + }; + + let origin_a = Url::parse(&base_a) + .expect("mock base A should parse") + .origin() + .ascii_serialization(); + let origin_b = Url::parse(&base_b) + .expect("mock base B should parse") + .origin() + .ascii_serialization(); + let allowed_origins = format!("{origin_a},{origin_b}"); + let policy = OutboundPolicy::from_allowed_origins(&allowed_origins).expect("loopback TLS origins should be allowed"); + let material = Arc::new(Mutex::new(OidcExtraRootCaMaterial { + generation: 1, + root_ca_pem: Some(ca_pem_a.into_bytes()), + })); + let provider = OidcExtraRootCaProvider::new({ + let material = material.clone(); + move || { + let material = material.clone(); + async move { + material + .lock() + .map(|material| material.clone()) + .map_err(|e| format!("test OIDC extra CA material lock poisoned: {e}")) + } + } + }); + let http_client = ReqwestHttpClient::with_policy_and_extra_root_ca_provider(policy, provider); + + let config_a = build_mocked_oidc_provider_config("a", &format!("{base_a}/application/o/rustfs-a")); + let state_a = discover_provider(&config_a, &http_client) + .await + .expect("OIDC discovery should trust initial extra root CA"); + assert_eq!(state_a.metadata.issuer().to_string(), format!("{base_a}/application/o/rustfs-a")); + + { + let mut material = material + .lock() + .expect("test OIDC extra CA material lock should not be poisoned"); + material.generation = 2; + material.root_ca_pem = Some(ca_pem_b.into_bytes()); + } + let config_b = build_mocked_oidc_provider_config("b", &format!("{base_b}/application/o/rustfs-b")); + let state_b = discover_provider(&config_b, &http_client) + .await + .expect("OIDC discovery should refresh extra root CA after generation change"); + + assert_eq!(state_b.metadata.issuer().to_string(), format!("{base_b}/application/o/rustfs-b")); + assert!(handle_a.join().is_ok()); + assert!(handle_b.join().is_ok()); +} +#[test] +fn test_jwks_url_from_config_url_preserves_unrelated_urls() { + let issuer = IssuerUrl::new("https://public.example.com/realms/app".to_string()).expect("issuer URL should parse"); + for raw_jwks_url in [ + "https://keys.example.com/jwks", + "http://public.example.com/realms/app/jwks", + "https://public.example.com:8443/realms/app/jwks", + "https://public.example.com/keys/jwks", + "https://public.example.com/realms/application/jwks", + ] { + let jwks_url = JsonWebKeySetUrl::new(raw_jwks_url.to_string()).expect("JWKS URL should parse"); + + let resolved = + jwks_url_from_config_url("http://keycloak.internal/realms/app/.well-known/openid-configuration", &issuer, &jwks_url) + .expect("JWKS URL should resolve"); + + assert_eq!(resolved.as_str(), raw_jwks_url); + } + + let issuer_root_jwks = JsonWebKeySetUrl::new(issuer.as_str().to_string()).expect("JWKS URL should parse"); + let resolved = jwks_url_from_config_url( + "http://keycloak.internal/realms/app/.well-known/openid-configuration", + &issuer, + &issuer_root_jwks, + ) + .expect("issuer-root JWKS URL should resolve"); + assert_eq!(resolved.as_str(), "http://keycloak.internal/realms/app"); + + let issuer_with_slash = + IssuerUrl::new("https://public.example.com/realms/app/".to_string()).expect("issuer URL should parse"); + let jwks_url = + JsonWebKeySetUrl::new("https://public.example.com/realms/app/jwks".to_string()).expect("JWKS URL should parse"); + let resolved = jwks_url_from_config_url( + "http://keycloak.internal/realms/app/.well-known/openid-configuration", + &issuer_with_slash, + &jwks_url, + ) + .expect("issuer-relative JWKS URL should resolve"); + assert_eq!(resolved.as_str(), "http://keycloak.internal/realms/app/jwks"); +} +#[tokio::test] +async fn oidc_discovery_reports_forbidden_outbound_without_retrying() { + let config_url = "http://192.168.65.254:8080/realms/rustfs/.well-known/openid-configuration"; + let config = build_mocked_oidc_provider_config("default", config_url); + let http_client = ReqwestHttpClient::with_policy(OutboundPolicy::default()); + + let error = match discover_provider(&config, &http_client).await { + Ok(_) => panic!("private OIDC provider should require an explicit allowlist origin"), + Err(error) => error, + }; + + assert!(error.contains("OIDC provider discovery blocked by outbound policy")); + assert!(error.contains(&format!("add http://192.168.65.254:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); + assert!(!error.contains("discovery failed for all issuer variants")); +} +#[tokio::test] +async fn oidc_explicit_issuer_reports_forbidden_discovery_endpoint() { + let config_url = "http://192.168.65.254:8080/realms/rustfs/.well-known/openid-configuration"; + let mut config = build_mocked_oidc_provider_config("default", config_url); + config.issuer = Some("https://idp.example.com/realms/rustfs".to_string()); + let http_client = ReqwestHttpClient::with_policy(OutboundPolicy::default()); + + let error = match discover_provider(&config, &http_client).await { + Ok(_) => panic!("private OIDC provider should require an explicit allowlist origin"), + Err(error) => error, + }; + + assert!(error.contains("OIDC provider discovery blocked by outbound policy")); + assert!(error.contains(&format!("add http://192.168.65.254:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); +} +#[tokio::test] +async fn oidc_explicit_issuer_reports_forbidden_jwks_endpoint() { + let Some((base, handle)) = start_mock_oidc_discovery_server( + |base| { + ( + format!("{base}/realms/rustfs"), + "http://192.168.65.254:8080/realms/rustfs/protocol/openid-connect/certs".to_string(), + "/unused".to_string(), + ) + }, + 1, + ) else { + return; + }; + let mut config = + build_mocked_oidc_provider_config("default", &format!("{base}/realms/rustfs/.well-known/openid-configuration")); + config.issuer = Some(format!("{base}/realms/rustfs")); + let policy = OutboundPolicy::from_allowed_origins(&base).expect("loopback discovery origin should be allowed"); + let http_client = ReqwestHttpClient::with_policy(policy); + + let error = match discover_provider(&config, &http_client).await { + Ok(_) => panic!("private JWKS endpoint should require an explicit allowlist origin"), + Err(error) => error, + }; + + assert!(error.contains("JWKS request blocked by outbound policy")); + assert!(error.contains(&format!("add http://192.168.65.254:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); + assert!(handle.join().is_ok()); +} +#[tokio::test] +async fn oidc_reqwest_dns_policy_rejection_stays_typed() { + let calls = Arc::new(AtomicUsize::new(0)); + let config = build_mocked_oidc_provider_config( + "default", + "http://keycloak.internal:8080/realms/rustfs/.well-known/openid-configuration", + ); + let http_client = ReqwestHttpClient::with_policy_and_dns_resolver( + OutboundPolicy::default(), + Arc::new(RejectingDnsResolver { + allow_origin_can_recover: true, + calls: Some(calls.clone()), + }), + ); + + let error = match discover_provider(&config, &http_client).await { + Ok(_) => panic!("private DNS answer should fail discovery"), + Err(error) => error, + }; + + assert!(error.contains("OIDC provider discovery blocked by outbound policy")); + assert!(error.contains(&format!("add http://keycloak.internal:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); + assert_eq!(calls.load(Ordering::Relaxed), 1, "policy rejection must not be retried"); +} +#[tokio::test] +async fn oidc_explicit_issuer_preserves_dns_policy_rejection() { + let calls = Arc::new(AtomicUsize::new(0)); + let mut config = build_mocked_oidc_provider_config( + "default", + "http://keycloak.internal:8080/realms/rustfs/.well-known/openid-configuration", + ); + config.issuer = Some("https://idp.example.com/realms/rustfs".to_string()); + let http_client = ReqwestHttpClient::with_policy_and_dns_resolver( + OutboundPolicy::default(), + Arc::new(RejectingDnsResolver { + allow_origin_can_recover: true, + calls: Some(calls.clone()), + }), + ); + + let error = match discover_provider(&config, &http_client).await { + Ok(_) => panic!("private DNS answer should fail discovery"), + Err(error) => error, + }; + + assert!(error.contains("OIDC provider discovery blocked by outbound policy")); + assert!(error.contains(&format!("add http://keycloak.internal:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); + assert_eq!(calls.load(Ordering::Relaxed), 1, "policy rejection must not be retried"); +} + +#[tokio::test] +async fn workload_config_validation_reports_absent_console_endpoints() { + let (base, handle) = start_mock_oidc_discovery_server_with_jwks( + |base| (base.to_string(), format!("{base}/jwks"), "/jwks".into()), + 2, + "ES256", + true, + |_| serde_json::json!({"keys": []}).to_string(), + ) + .expect("workload discovery mock must bind"); + let mut config = build_mocked_oidc_provider_config("workload", &base); + config.hide_from_ui = true; + // Inferred issuers keep the library's discovery URL construction. + config.config_url = format!("{base}/.well-known/openid-configuration?ignored=1#ignored"); + let result = validate_mocked_oidc_provider_config(&config) + .await + .expect("hidden workload configuration should validate"); + assert_eq!(result.issuer, base); + assert!(result.authorization_endpoint.is_none()); + assert!(result.token_endpoint.is_none()); + handle.join().unwrap(); +} + +#[tokio::test] +async fn test_validate_oidc_provider_config_retries_with_issuer_candidates() { + // Discovery document must advertise the canonical issuer path. The first candidate has no + // trailing slash; openidconnect rejects issuer mismatch, then the second variant succeeds. + let Some((base, handle)) = start_mock_oidc_discovery_server( + |base| (format!("{base}/application/o/rustfs/"), format!("{base}/jwks"), "/jwks".to_string()), + 8, + ) else { + return; + }; + let config_url = format!("{base}/application/o/rustfs"); + let config = build_mocked_oidc_provider_config("default", &config_url); + + let result = validate_mocked_oidc_provider_config(&config).await; + + let validation_result = result.expect("OIDC provider validation should succeed"); + assert_eq!(validation_result.issuer, format!("{base}/application/o/rustfs/")); + assert!(handle.join().is_ok()); +} + +#[tokio::test] +async fn test_validate_oidc_provider_config_fetches_issuer_relative_jwks_from_config_url() { + let Some((base, handle)) = start_mock_oidc_discovery_server( + |_| { + ( + "http://127.0.0.1:1/public/realms/app".to_string(), + "http://127.0.0.1:1/public/realms/app/jwks?version=1".to_string(), + "/internal/realms/app/jwks?version=1".to_string(), + ) + }, + 2, + ) else { + return; + }; + let mut config = + build_mocked_oidc_provider_config("default", &format!("{base}/internal/realms/app/.well-known/openid-configuration")); + config.issuer = Some("http://127.0.0.1:1/public/realms/app".to_string()); + + let validation_result = validate_mocked_oidc_provider_config(&config) + .await + .expect("OIDC provider validation should succeed"); + + assert_eq!(validation_result.issuer, "http://127.0.0.1:1/public/realms/app"); + assert!(handle.join().is_ok()); +} + +#[tokio::test] +async fn test_validate_oidc_provider_config_rejects_separate_issuer_mismatch() { + let Some((base, handle)) = start_mock_oidc_discovery_server( + |base| { + ( + "https://public.example.com/realms/other".to_string(), + format!("{base}/jwks"), + "/jwks".to_string(), + ) + }, + 1, + ) else { + return; + }; + let mut config = + build_mocked_oidc_provider_config("default", &format!("{base}/internal/realms/app/.well-known/openid-configuration")); + config.issuer = Some("https://public.example.com/realms/app".to_string()); + + let err = validate_mocked_oidc_provider_config(&config) + .await + .expect_err("OIDC provider validation should fail"); + + assert!(err.contains("unexpected issuer URI")); + assert!(err.contains("https://public.example.com/realms/app")); + assert!(handle.join().is_ok()); +} + +#[tokio::test] +async fn test_validate_oidc_provider_config_returns_detailed_errors() { + let Some((base, handle)) = start_mock_oidc_discovery_server( + |base| (format!("{base}/application/o/other"), format!("{base}/jwks"), "/jwks".to_string()), + 8, + ) else { + return; + }; + let config_url = format!("{base}/application/o/rustfs"); + let config = build_mocked_oidc_provider_config("default", &config_url); + + let err = validate_mocked_oidc_provider_config(&config) + .await + .expect_err("OIDC provider validation should fail"); + assert!(discovery_error_contains_all_variants(&err, &base)); + assert!(err.contains("issuer '")); + assert!(err.contains(&format!("issuer '{base}/application/o/rustfs'"))); + assert!(err.contains(&format!("issuer '{base}/application/o/rustfs/'"))); + assert!(handle.join().is_ok()); +} diff --git a/crates/iam/src/oidc/recovery_tests.rs b/crates/iam/src/oidc/recovery_tests.rs new file mode 100644 index 000000000..96fece5bc --- /dev/null +++ b/crates/iam/src/oidc/recovery_tests.rs @@ -0,0 +1,489 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use crate::oidc::test_support::{build_mocked_oidc_provider_config, oidc_es256_key_and_jwk, test_sourced_config}; +use jsonwebtoken::{Algorithm, EncodingKey, Header}; +use rustfs_utils::egress::OutboundPolicy; +use std::sync::atomic::{AtomicU16, AtomicUsize, Ordering}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio_util::task::AbortOnDropHandle; + +struct RecoveryIdp { + base: String, + config: OidcProviderConfig, + key: EncodingKey, + discovery_status: Arc, + jwks_status: Arc, + requests: Arc, + cancelled_requests: Arc, + _server: AbortOnDropHandle<()>, +} + +impl RecoveryIdp { + async fn start(id: &str, workload: bool, explicit_issuer: bool) -> Self { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("bind recovery IdP"); + let base = format!("http://{}", listener.local_addr().expect("recovery IdP address")); + let mut config = build_mocked_oidc_provider_config(id, &base); + config.hide_from_ui = workload; + config.claim_name = "groups".into(); + config.claim_prefix = "team/".into(); + config.issuer = explicit_issuer.then(|| format!("https://issuer.example.com/{id}")); + let (key, jwk) = oidc_es256_key_and_jwk(id); + let mut document = serde_json::json!({ + "issuer": config.issuer.as_deref().unwrap_or(&base), + "authorization_endpoint": format!("{base}/authorize"), + "token_endpoint": format!("{base}/token"), + "jwks_uri": format!("{base}/jwks"), + "response_types_supported": ["code"], + "subject_types_supported": ["public"], + "id_token_signing_alg_values_supported": ["ES256"], + }); + if workload { + let fields = document.as_object_mut().expect("discovery document"); + fields.remove("authorization_endpoint"); + fields.remove("token_endpoint"); + } + let document = document.to_string(); + let jwks = serde_json::json!({"keys": [jwk]}).to_string(); + let discovery_status = Arc::new(AtomicU16::new(503)); + let jwks_status = Arc::new(AtomicU16::new(200)); + let requests = Arc::new(AtomicUsize::new(0)); + let cancelled_requests = Arc::new(AtomicUsize::new(0)); + let server_discovery_status = discovery_status.clone(); + let server_jwks_status = jwks_status.clone(); + let server_requests = requests.clone(); + let server_cancelled = cancelled_requests.clone(); + let server = tokio::spawn(async move { + loop { + let (mut stream, _) = listener.accept().await.expect("accept discovery request"); + let mut request = Vec::new(); + while !request.ends_with(b"\r\n\r\n") { + request.push(stream.read_u8().await.expect("read discovery request")); + assert!(request.len() < 8192); + } + let headers = String::from_utf8(request).expect("HTTP request headers"); + let path = headers + .lines() + .next() + .expect("request line") + .split_whitespace() + .nth(1) + .expect("path"); + let (status, body) = if path.ends_with("/.well-known/openid-configuration") { + (server_discovery_status.load(Ordering::SeqCst), document.as_str()) + } else { + assert_eq!(path, "/jwks"); + (server_jwks_status.load(Ordering::SeqCst), jwks.as_str()) + }; + server_requests.fetch_add(1, Ordering::SeqCst); + if status == 0 { + continue; // Simulate a transport failure before a response is available. + } + if status == 1 { + let mut byte = [0]; + assert_eq!(stream.read(&mut byte).await.expect("client closes cancelled discovery"), 0); + server_cancelled.fetch_add(1, Ordering::SeqCst); + continue; + } + let body = if status == 200 { body } else { "" }; + let response = format!( + "HTTP/1.1 {status} Test\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + stream.write_all(response.as_bytes()).await.expect("write discovery response"); + } + }); + Self { + base, + config, + key, + discovery_status, + jwks_status, + requests, + cancelled_requests, + _server: AbortOnDropHandle::new(server), + } + } + + fn token(&self, key: &EncodingKey, audience: &str) -> String { + let mut header = Header::new(Algorithm::ES256); + header.kid = Some(self.config.id.clone()); + let now = time::OffsetDateTime::now_utc().unix_timestamp(); + jsonwebtoken::encode( + &header, + &serde_json::json!({ + "iss": self.config.issuer.as_deref().unwrap_or(&self.base), + "sub": "recovered-user", "aud": audience, "iat": now, "exp": now + 300, + "groups": ["readonly"], + }), + key, + ) + .expect("sign recovery test token") + } +} + +async fn initialize(idps: &[&RecoveryIdp]) -> Arc { + let origins = idps.iter().map(|idp| idp.base.as_str()).collect::>().join(","); + let client = ReqwestHttpClient::with_policy(OutboundPolicy::from_allowed_origins(&origins).expect("allow test IdPs")); + let snapshot = OidcConfigSnapshot::new(idps.iter().map(|idp| test_sourced_config(idp.config.clone())).collect()); + Arc::new( + OidcSys::new_with_config_snapshot(client, snapshot) + .await + .expect("initialize OIDC with unavailable IdPs"), + ) +} + +async fn wait_until(mut condition: impl FnMut() -> bool) { + tokio::time::timeout(Duration::from_secs(5), async { + while !condition() { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("recovery condition should be reached"); +} + +#[tokio::test] +async fn web_identity_recovers_transport_failure_and_preserves_verification() { + for explicit in [false, true] { + let idp = RecoveryIdp::start("workload", true, explicit).await; + idp.discovery_status.store(0, Ordering::SeqCst); + let sys = initialize(&[&idp]).await; + let adapter = StandardOidcAdapter::new(sys.clone()); + let mapper = CoreFederatedAuthorizationMapper::new(adapter.authorization_rules()); + assert!(sys.has_providers()); + assert!(sys.list_visible_providers().is_empty()); + let token = idp.token(&idp.key, &idp.config.client_id); + assert!(adapter.verify_identity(&token).await.is_err()); + + idp.discovery_status.store(200, Ordering::SeqCst); + tokio::time::timeout(Duration::from_secs(5), sys.recover_failed_providers()) + .await + .expect("recover workload discovery"); + let identity = adapter + .verify_identity(&token) + .await + .expect("verify recovered workload token"); + let authorization = mapper.map(identity); + assert_eq!(authorization.provider_id(), "workload"); + assert_eq!(authorization.subject(), "recovered-user"); + assert_eq!(authorization.policies(), ["team/readonly"]); + assert!( + sys.authorize_url("workload", "https://console.example.com/callback", None) + .await + .is_err() + ); + + let (wrong_key, _) = oidc_es256_key_and_jwk("wrong"); + assert!( + adapter + .verify_identity(&idp.token(&wrong_key, &idp.config.client_id)) + .await + .is_err() + ); + assert!(adapter.verify_identity(&idp.token(&idp.key, "wrong-audience")).await.is_err()); + } +} + +#[tokio::test] +async fn recovery_waits_for_jwks_and_claims_pending_providers_once() { + let idp = RecoveryIdp::start("console", false, true).await; + idp.discovery_status.store(200, Ordering::SeqCst); + idp.jwks_status.store(503, Ordering::SeqCst); + let sys = initialize(&[&idp]).await; + assert!(sys.provider_runtime.get_provider_state("console").is_err()); + let requests = idp.requests.load(Ordering::SeqCst); + idp.jwks_status.store(200, Ordering::SeqCst); + tokio::time::timeout( + Duration::from_secs(5), + futures::future::join_all((0..16).map(|_| sys.recover_failed_providers())), + ) + .await + .expect("concurrent recovery calls complete"); + assert!(sys.provider_runtime.get_provider_state("console").is_ok()); + assert_eq!( + idp.requests.load(Ordering::SeqCst), + requests + 2, + "only one discovery/JWKS round should run" + ); +} + +#[tokio::test] +async fn recovery_backoff_is_independent_of_login_traffic() { + let idp = RecoveryIdp::start("console", false, true).await; + let sys = initialize(&[&idp]).await; + let requests = idp.requests.load(Ordering::SeqCst); + let recovery_sys = sys.clone(); + let recovery = AbortOnDropHandle::new(tokio::spawn(async move { recovery_sys.recover_failed_providers().await })); + wait_until(|| idp.requests.load(Ordering::SeqCst) > requests).await; + assert_eq!(idp.requests.load(Ordering::SeqCst), requests + 1); + for _ in 0..32 { + assert!( + sys.authorize_url("console", "https://console.example.com/callback", None) + .await + .is_err() + ); + } + tokio::time::sleep(Duration::from_millis(300)).await; + assert_eq!( + idp.requests.load(Ordering::SeqCst), + requests + 1, + "login attempts must not bypass discovery backoff" + ); + idp.discovery_status.store(200, Ordering::SeqCst); + tokio::time::timeout(Duration::from_secs(5), recovery) + .await + .expect("recovery deadline") + .expect("recovery task"); + assert!( + sys.authorize_url("console", "https://console.example.com/callback", None) + .await + .is_ok() + ); +} + +#[tokio::test] +async fn unavailable_provider_does_not_block_another_provider() { + let unavailable = RecoveryIdp::start("unavailable", false, true).await; + let available = RecoveryIdp::start("available", false, true).await; + let sys = initialize(&[&unavailable, &available]).await; + available.discovery_status.store(200, Ordering::SeqCst); + let recovery_sys = sys.clone(); + let recovery = AbortOnDropHandle::new(tokio::spawn(async move { recovery_sys.recover_failed_providers().await })); + wait_until(|| sys.provider_runtime.get_provider_state("available").is_ok()).await; + assert!(sys.provider_runtime.get_provider_state("unavailable").is_err()); + assert!( + sys.authorize_url("available", "https://console.example.com/callback", None) + .await + .is_ok() + ); + drop(recovery); +} + +#[tokio::test] +async fn recovery_preserves_ready_provider_and_inflight_sessions() { + let ready = RecoveryIdp::start("ready", false, true).await; + let recovering = RecoveryIdp::start("recovering", false, true).await; + ready.discovery_status.store(200, Ordering::SeqCst); + let sys = initialize(&[&ready, &recovering]).await; + let auth_url = sys + .authorize_url("ready", "https://console.example.com/callback", Some("/buckets".into())) + .await + .expect("begin login with ready provider"); + let auth: HashMap = url::Url::parse(&auth_url) + .expect("authorization URL") + .query_pairs() + .into_owned() + .collect(); + let logout = sys + .create_logout_token("ready", "verified-id-token") + .await + .expect("retain logout continuation"); + let discovered_at = sys + .provider_runtime + .get_provider_state("ready") + .expect("ready state") + .discovered_at; + let ready_requests = ready.requests.load(Ordering::SeqCst); + + recovering.discovery_status.store(200, Ordering::SeqCst); + tokio::time::timeout(Duration::from_secs(5), sys.recover_failed_providers()) + .await + .expect("recover second provider"); + assert_eq!(ready.requests.load(Ordering::SeqCst), ready_requests); + assert_eq!( + sys.provider_runtime + .get_provider_state("ready") + .expect("ready state remains") + .discovered_at, + discovered_at + ); + let session = sys + .state_store + .take(&auth["state"]) + .await + .expect("in-flight login remains available"); + assert_eq!(session.provider_id, "ready"); + assert_eq!(session.nonce, auth["nonce"]); + assert_eq!(session.redirect_after.as_deref(), Some("/buckets")); + assert_eq!( + PkceCodeChallenge::from_code_verifier_sha256(&PkceCodeVerifier::new(session.pkce_verifier)).as_str(), + auth["code_challenge"] + ); + assert_eq!( + sys.state_store + .take_logout(&logout) + .await + .expect("logout continuation remains available") + .provider_id, + "ready" + ); +} + +#[tokio::test] +async fn startup_schedules_only_recoverable_http_errors() { + for explicit in [false, true] { + let idp = RecoveryIdp::start("status", false, explicit).await; + for (status, retryable) in [ + (408, true), + (429, true), + (500, true), + (503, true), + (400, false), + (401, false), + (403, false), + (404, false), + ] { + idp.discovery_status.store(status, Ordering::SeqCst); + let sys = initialize(&[&idp]).await; + assert_eq!( + sys.pending_discovery.lock().await.len(), + usize::from(retryable), + "explicit={explicit}, status={status}" + ); + assert!(sys.provider_runtime.get_provider_state("status").is_err()); + } + } +} + +#[tokio::test] +async fn recovery_bounds_concurrency_and_cancels_inflight_requests() { + let mut idps = Vec::new(); + for index in 0..6 { + idps.push(RecoveryIdp::start(&format!("provider-{index}"), false, true).await); + } + let refs: Vec<_> = idps.iter().collect(); + let sys = initialize(&refs).await; + let initial_requests: usize = idps.iter().map(|idp| idp.requests.load(Ordering::SeqCst)).sum(); + for idp in &idps { + idp.discovery_status.store(1, Ordering::SeqCst); + } + let recovery_sys = sys.clone(); + let recovery = AbortOnDropHandle::new(tokio::spawn(async move { recovery_sys.recover_failed_providers().await })); + let requests = || idps.iter().map(|idp| idp.requests.load(Ordering::SeqCst)).sum::(); + wait_until(|| requests() >= initial_requests + 4).await; + tokio::time::sleep(Duration::from_millis(300)).await; + assert_eq!(requests(), initial_requests + 4, "at most four discovery rounds may run concurrently"); + drop(recovery); + wait_until(|| { + idps.iter() + .map(|idp| idp.cancelled_requests.load(Ordering::SeqCst)) + .sum::() + == 4 + }) + .await; + tokio::time::sleep(Duration::from_millis(100)).await; + assert_eq!(requests(), initial_requests + 4, "cancelled recovery must not start queued work"); + assert!( + idps.iter() + .all(|idp| sys.provider_runtime.get_provider_state(&idp.config.id).is_err()) + ); +} + +#[tokio::test] +async fn disabled_and_rejected_providers_do_not_schedule_recovery() { + let mut disabled = RecoveryIdp::start("disabled", false, true).await; + disabled.config.enabled = false; + let sys = initialize(&[&disabled]).await; + sys.recover_failed_providers().await; + assert!(!sys.has_providers()); + assert_eq!(disabled.requests.load(Ordering::SeqCst), 0); + + let blocked = RecoveryIdp::start("blocked", false, true).await; + let sys = OidcSys::new_with_config_snapshot( + ReqwestHttpClient::with_policy(OutboundPolicy::default()), + OidcConfigSnapshot::new(vec![test_sourced_config(blocked.config.clone())]), + ) + .await + .expect("policy rejection remains non-fatal for startup"); + assert!(sys.pending_discovery.lock().await.is_empty()); + sys.recover_failed_providers().await; + assert_eq!(blocked.requests.load(Ordering::SeqCst), 0); + assert!(sys.provider_runtime.get_provider_state("blocked").is_err()); +} + +#[tokio::test] +async fn recovery_stops_on_issuer_validation_failure() { + let mut idp = RecoveryIdp::start("console", false, true).await; + idp.config.issuer = Some("https://unexpected.example.com".into()); + let sys = initialize(&[&idp]).await; + idp.discovery_status.store(200, Ordering::SeqCst); + tokio::time::timeout(Duration::from_secs(5), sys.recover_failed_providers()) + .await + .expect("validation failure stops recovery"); + assert_eq!( + idp.requests.load(Ordering::SeqCst), + 2, + "issuer rejection must not fetch JWKS or schedule another attempt" + ); + assert!(sys.provider_runtime.get_provider_state("console").is_err()); + assert!( + sys.authorize_url("console", "https://console.example.com/callback", None) + .await + .is_err() + ); +} + +#[tokio::test] +async fn recovery_logs_failure_reasons_without_repeating_warnings() { + use std::io::{Read, Seek}; + + for (explicit, status, expected_kind) in [ + (false, 503, "http_status"), + (true, 503, "http_status"), + (true, 0, "request"), + (true, 1, "timeout"), + ] { + let idp = RecoveryIdp::start("logging", false, explicit).await; + let sys = initialize(&[&idp]).await; + let requests = idp.requests.load(Ordering::SeqCst); + idp.discovery_status.store(status, Ordering::SeqCst); + let mut output = tempfile::tempfile().expect("create captured log file"); + let subscriber = tracing_subscriber::fmt() + .without_time() + .with_ansi(false) + .with_max_level(tracing::Level::WARN) + .with_writer(output.try_clone().expect("clone captured log file")) + .finish(); + let _subscriber = tracing::subscriber::set_default(subscriber); + tracing::callsite::rebuild_interest_cache(); + tokio::time::timeout(Duration::from_secs(25), async { + tokio::join!(sys.recover_failed_providers(), async { + // Two failing rounds exercise suppression; a timeout is already bounded by the HTTP client. + while if status == 1 { + idp.cancelled_requests.load(Ordering::SeqCst) == 0 + } else { + idp.requests.load(Ordering::SeqCst) < requests + if explicit { 2 } else { 4 } + } { + tokio::time::sleep(Duration::from_millis(10)).await; + } + idp.discovery_status.store(200, Ordering::SeqCst); + }); + }) + .await + .expect("recover after recording failure diagnostics"); + output.rewind().expect("rewind captured log file"); + let mut logs = String::new(); + output.read_to_string(&mut logs).expect("read captured log file"); + assert_eq!(logs.lines().count(), 1, "explicit={explicit}, status={status}: {logs}"); + assert!(logs.contains("provider_discovery_recovery_pending"), "{logs}"); + assert!(logs.contains(&format!("error_kind=\"{expected_kind}\"")), "{logs}"); + if status == 503 { + assert!(logs.contains("error_detail=503"), "{logs}"); + } + assert!(logs.contains("provider_id=logging"), "{logs}"); + assert!(logs.contains("suppressed=0"), "{logs}"); + } +} diff --git a/crates/iam/src/oidc/runtime.rs b/crates/iam/src/oidc/runtime.rs new file mode 100644 index 000000000..7088d31d9 --- /dev/null +++ b/crates/iam/src/oidc/runtime.rs @@ -0,0 +1,994 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +use super::{ + config::{ + OidcConfigQuery, OidcConfigSnapshot, OidcProviderConfig, OidcProviderSummary, OidcSiteReplicationSnapshot, + SourcedOidcProviderConfig, load_oidc_config_snapshot, + }, + provider::{ProviderRuntime, discover_provider_for_recovery}, + state::{OidcAuthSession, OidcLogoutSession, OidcStateStore}, + transport::{OidcExtraRootCaProvider, ReqwestHttpClient, inspect_token_response_body, oidc_http_error_diagnostics}, +}; +#[cfg(test)] +use crate::federation::FederatedAuthorizationRuleRef; +use crate::{ + EVENT_OIDC_DIAGNOSTICS, LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC, + federation::{ + CoreFederatedAuthorizationMapper, FederatedAuthorizationRule, FederatedAuthorizationRules, FederatedClaims, + FederatedIdentityService, FederatedProviderQuery, FederatedProviderRef, FederatedProviderView, FederatedRedirectPolicy, + FederationError, OpaqueLogoutContinuation, Result as FederationResult, StandardOidcAuthentication, + VerifiedFederatedCodeExchange, VerifiedFederatedIdentity, + }, +}; +use openidconnect::core::{CoreAuthenticationFlow, CoreClient, CoreIdToken}; +use openidconnect::{ + Audience, AuthType, AuthorizationCode, ClientId, ClientSecret, CsrfToken, LogoutRequest, Nonce, PkceCodeChallenge, + PkceCodeVerifier, PostLogoutRedirectUrl, RedirectUrl, RequestTokenError, Scope, +}; +use rustfs_policy::policy::{ClaimLookup, get_claim_case_insensitive}; +use rustfs_utils::logging::LogThrottle; +use serde::{Deserialize, Serialize}; +use std::borrow::Cow; +use std::collections::HashMap; +use std::sync::Arc; +use std::time::Duration; +use tokio::sync::{Mutex, Semaphore}; +use tracing::{debug, error, info, warn}; + +const OIDC_DISCOVERY_RECOVERY_CONCURRENCY: usize = 4; +const OIDC_DISCOVERY_RECOVERY_MAX_DELAY_MS: u64 = 30_000; +// A round can try two issuer variants three times, each fetching discovery and JWKS. +const OIDC_DISCOVERY_RECOVERY_TIMEOUT: Duration = Duration::from_secs(125); + +/// Claims extracted from an OIDC ID token. +#[derive(Debug, Clone, Serialize, Deserialize, Default)] +pub(crate) struct OidcClaims { + pub sub: String, + pub email: String, + pub username: String, + pub groups: Vec, + pub raw: HashMap, +} + +// ---- Core OIDC system ---- + +/// Global OIDC manager for all configured providers. +pub struct OidcSys { + pub(super) configs: HashMap, + pub(super) provider_runtime: ProviderRuntime, + pub(super) state_store: OidcStateStore, + pub(super) pending_discovery: Mutex>, +} + +pub(super) fn trusted_aud(other_audiences: &[String], audience: &Audience) -> bool { + for aud in other_audiences { + if audience.as_str() == aud.as_str() { + return true; + } + } + false +} + +impl OidcSys { + pub(crate) async fn new_with_extra_root_ca_provider(extra_root_ca_provider: OidcExtraRootCaProvider) -> Result { + let http_client = ReqwestHttpClient::new_with_extra_root_ca_provider(extra_root_ca_provider)?; + http_client.current_extra_root_certs().await.map_err(|err| err.to_string())?; + Self::new_with_http_client(http_client).await + } + + async fn new_with_http_client(http_client: ReqwestHttpClient) -> Result { + let server_config = crate::server_config::current_server_config(); + let parsed_configs = load_oidc_config_snapshot(server_config.as_ref()); + Self::new_with_config_snapshot(http_client, parsed_configs).await + } + + async fn new_with_config_snapshot( + http_client: ReqwestHttpClient, + parsed_configs: OidcConfigSnapshot, + ) -> Result { + let mut configs = HashMap::new(); + let mut provider_states = HashMap::new(); + let mut pending_discovery = Vec::new(); + + for sourced_config in parsed_configs.into_providers() { + let config = &sourced_config.config; + if !config.enabled { + debug!(provider = %config.id, "OIDC provider disabled"); + continue; + } + + match discover_provider_for_recovery(config, &http_client).await { + Ok(state) => { + debug!(provider = %config.id, "OIDC provider discovered"); + provider_states.insert(config.id.clone(), state); + } + Err(e) => { + if e.retryable { + pending_discovery.push(config.id.clone()); + } + error!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "provider_discovery_failed", + provider_id = %config.id, + config_url = %config.config_url, + client_id = %config.client_id, + scopes = ?config.scopes, + redirect_uri = %config.redirect_uri.as_deref().unwrap_or(""), + redirect_uri_dynamic = config.redirect_uri_dynamic, + error = %e, + "oidc provider discovery failed" + ); + } + } + // Configuration also supplies redirect and authorization rules before recovery. + configs.insert(config.id.clone(), sourced_config); + } + + Ok(Self { + configs, + provider_runtime: ProviderRuntime::new(http_client, provider_states), + state_store: OidcStateStore::new(), + pending_discovery: Mutex::new(pending_discovery), + }) + } + + /// Recover providers that failed startup discovery due to a transient error. + /// The server owns and cancels this future; dropping it cancels all outstanding requests. + /// Pending providers are claimed once so concurrent callers cannot duplicate recovery. + pub async fn recover_failed_providers(&self) { + let pending = std::mem::take(&mut *self.pending_discovery.lock().await); + let permits = Semaphore::new(OIDC_DISCOVERY_RECOVERY_CONCURRENCY); + futures::future::join_all(pending.iter().filter_map(|id| self.get_provider_config(id)).map(|config| { + let permits = &permits; + async move { + let mut delay_ms = 1_000_u64; + let failure_log = LogThrottle::new(60_000); + loop { + let sleep_ms = rand::random_range(delay_ms * 3 / 4..=delay_ms); + tokio::time::sleep(Duration::from_millis(sleep_ms)).await; + let Ok(permit) = permits.acquire().await else { + return; + }; + let result = tokio::time::timeout( + OIDC_DISCOVERY_RECOVERY_TIMEOUT, + self.provider_runtime.recover_provider_state(&config.id, config), + ) + .await; + drop(permit); + let (error_kind, error_detail) = match result { + Ok(Ok(_)) => { + info!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "provider_discovery_recovered", + provider_id = %config.id, + "oidc provider discovery recovered" + ); + return; + } + Ok(Err(error)) if !error.retryable => { + error!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "provider_discovery_recovery_stopped", + provider_id = %config.id, + error = %error, + "oidc provider discovery requires configuration correction" + ); + return; + } + Ok(Err(error)) => error.diagnostics, + Err(_) => ("attempt_timeout", OIDC_DISCOVERY_RECOVERY_TIMEOUT.as_secs().to_string()), + }; + if let Some(suppressed) = failure_log.claim() { + warn!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "provider_discovery_recovery_pending", + provider_id = %config.id, + error_kind, + error_detail = %error_detail, + suppressed, + "oidc provider discovery remains unavailable" + ); + } + delay_ms = delay_ms.saturating_mul(2).min(OIDC_DISCOVERY_RECOVERY_MAX_DELAY_MS); + } + } + })) + .await; + } + + /// Create an OidcSys with no providers (useful for when OIDC is not configured). + pub fn empty() -> Result { + Ok(Self { + configs: HashMap::new(), + provider_runtime: ProviderRuntime::new(ReqwestHttpClient::new()?, HashMap::new()), + state_store: OidcStateStore::new(), + pending_discovery: Mutex::default(), + }) + } + + /// Return true if any OIDC providers are configured and enabled. + pub(crate) fn has_providers(&self) -> bool { + !self.configs.is_empty() + } + + pub(crate) fn provider_configs(&self) -> impl Iterator { + self.configs.values().map(|provider| &provider.config) + } + + /// List all providers, including providers hidden from the login UI. + pub(crate) fn list_providers(&self) -> Vec { + self.provider_configs() + .map(|config| OidcProviderSummary { + provider_id: config.id.clone(), + display_name: config.display_name.clone(), + }) + .collect() + } + + /// List providers visible in the login UI. + pub(crate) fn list_visible_providers(&self) -> Vec { + self.provider_configs() + .filter(|config| !config.hide_from_ui) + .map(|config| OidcProviderSummary { + provider_id: config.id.clone(), + display_name: config.display_name.clone(), + }) + .collect() + } + + pub(crate) fn config_snapshot(&self) -> OidcConfigSnapshot { + OidcConfigSnapshot::new(self.configs.values().cloned().collect()) + } + + /// Build the PKCE authorization URL for a provider, store state in the state store. + pub(crate) async fn authorize_url( + &self, + provider_id: &str, + redirect_uri: &str, + redirect_after: Option, + ) -> Result { + let config = self + .get_provider_config(provider_id) + .ok_or_else(|| format!("unknown OIDC provider: {provider_id}"))?; + let state = self.provider_runtime.ensure_provider_state(provider_id, config).await?; + + let (pkce_challenge, pkce_verifier) = PkceCodeChallenge::new_random_sha256(); + + let redirect = RedirectUrl::new(redirect_uri.to_string()).map_err(|e| format!("invalid redirect URI: {e}"))?; + + let client = CoreClient::from_provider_metadata( + state.metadata.console()?.clone(), + ClientId::new(config.client_id.clone()), + config.client_secret.as_ref().map(|s| ClientSecret::new(s.clone())), + ) + .set_auth_type(AuthType::RequestBody); + + let mut auth_req = + client.authorize_url(CoreAuthenticationFlow::AuthorizationCode, CsrfToken::new_random, Nonce::new_random); + auth_req = auth_req.set_redirect_uri(Cow::Owned(redirect)); + + for scope in &config.scopes { + auth_req = auth_req.add_scope(Scope::new(scope.clone())); + } + + auth_req = auth_req.set_pkce_challenge(pkce_challenge); + + let (auth_url, csrf_token, nonce) = auth_req.url(); + + // Store the state for callback validation + self.state_store + .insert( + csrf_token.secret().clone(), + OidcAuthSession { + provider_id: provider_id.to_string(), + pkce_verifier: pkce_verifier.secret().clone(), + nonce: nonce.secret().clone(), + redirect_after, + }, + ) + .await; + + Ok(auth_url.to_string()) + } + + /// Exchange an authorization code for tokens and extract claims. + pub(super) async fn exchange_code( + &self, + state: &str, + code: &str, + redirect_uri: &str, + ) -> Result<(OidcClaims, String, OidcAuthSession, String), String> { + // Retrieve and consume the state (single-use) + let session = self + .state_store + .take(state) + .await + .ok_or_else(|| "invalid or expired OIDC state".to_string())?; + + let config = self + .get_provider_config(&session.provider_id) + .ok_or_else(|| format!("unknown provider: {}", session.provider_id))?; + let provider_state = self.provider_runtime.get_provider_state(&session.provider_id)?; + let issuer = provider_state.metadata.issuer().to_string(); + let token_endpoint = provider_state + .metadata + .token_endpoint() + .map(ToString::to_string) + .unwrap_or_default(); + + // Construct CoreClient on-the-fly with JWKS from discovery + let client = CoreClient::from_provider_metadata( + provider_state.metadata.console()?.clone(), + ClientId::new(config.client_id.clone()), + config.client_secret.as_ref().map(|s| ClientSecret::new(s.clone())), + ) + .set_auth_type(AuthType::RequestBody); + + let redirect = RedirectUrl::new(redirect_uri.to_string()).map_err(|e| format!("invalid redirect URI: {e}"))?; + + // Exchange code for tokens + let token_response = client + .exchange_code(AuthorizationCode::new(code.to_string())) + .map_err(|e| { + error!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "token_endpoint_missing", + provider_id = %session.provider_id, + config_url = %config.config_url, + issuer = %issuer, + client_id = %config.client_id, + redirect_uri = %redirect_uri, + scopes = ?config.scopes, + error = %e, + "oidc token exchange failed" + ); + format!( + "token endpoint not configured: {e}: provider_id={}, config_url={}, issuer={}, redirect_uri={}, client_id={}", + session.provider_id, config.config_url, issuer, redirect_uri, config.client_id + ) + })? + .set_pkce_verifier(PkceCodeVerifier::new(session.pkce_verifier.clone())) + .set_redirect_uri(Cow::Owned(redirect)) + .request_async(self.provider_runtime.http_client()) + .await + .map_err(|e| match &e { + RequestTokenError::ServerResponse(response) => { + error!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "token_server_response", + provider_id = %session.provider_id, + config_url = %config.config_url, + issuer = %issuer, + token_endpoint = %token_endpoint, + client_id = %config.client_id, + client_secret_configured = config.client_secret.as_deref().is_some_and(|secret| !secret.is_empty()), + redirect_uri = %redirect_uri, + scopes = ?config.scopes, + oauth_error = %response.error(), + oauth_error_description = %response.error_description().map(String::as_str).unwrap_or(""), + oauth_error_uri = %response.error_uri().map(String::as_str).unwrap_or(""), + error = %e, + "oidc token exchange failed" + ); + format!( + "token exchange failed: {e}: stage=token_server_response, provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}, oauth_error={}, oauth_error_description={}", + session.provider_id, + config.config_url, + issuer, + token_endpoint, + redirect_uri, + config.client_id, + response.error(), + response.error_description().map(String::as_str).unwrap_or("") + ) + } + RequestTokenError::Request(err) => { + let (request_error_kind, request_error_status) = oidc_http_error_diagnostics(err); + error!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "token_request_failed", + provider_id = %session.provider_id, + config_url = %config.config_url, + issuer = %issuer, + token_endpoint = %token_endpoint, + client_id = %config.client_id, + redirect_uri = %redirect_uri, + request_error_kind = %request_error_kind, + request_error_status = %request_error_status, + error = %err, + "oidc token exchange failed" + ); + format!( + "token exchange failed: stage=token_request_failed, provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}, request_error_kind={}, request_error_status={}, request_error={}", + session.provider_id, + config.config_url, + issuer, + token_endpoint, + redirect_uri, + config.client_id, + request_error_kind, + request_error_status, + err + ) + } + RequestTokenError::Parse(parse_err, body) => { + let shape = inspect_token_response_body(body); + error!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "token_response_parse_failed", + provider_id = %session.provider_id, + config_url = %config.config_url, + issuer = %issuer, + token_endpoint = %token_endpoint, + client_id = %config.client_id, + redirect_uri = %redirect_uri, + parse_error_path = %parse_err.path(), + response_body_len = body.len(), + response_json_object = shape.json_object, + response_json_keys = %shape.json_keys, + response_has_access_token = shape.has_access_token, + response_has_id_token = shape.has_id_token, + response_has_token_type = shape.has_token_type, + response_has_expires_in = shape.has_expires_in, + response_has_error = shape.has_error, + response_has_error_description = shape.has_error_description, + response_looks_like_html = shape.looks_like_html, + error = %e, + "oidc token exchange failed" + ); + format!( + "token exchange failed: {e}: stage=token_response_parse_failed, provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}, parse_error_path={}, response_body_len={}, response_json_keys={}, response_has_id_token={}, response_has_error={}, response_looks_like_html={}", + session.provider_id, + config.config_url, + issuer, + token_endpoint, + redirect_uri, + config.client_id, + parse_err.path(), + body.len(), + shape.json_keys, + shape.has_id_token, + shape.has_error, + shape.looks_like_html + ) + } + RequestTokenError::Other(message) => { + error!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "token_exchange_other_error", + provider_id = %session.provider_id, + config_url = %config.config_url, + issuer = %issuer, + token_endpoint = %token_endpoint, + client_id = %config.client_id, + redirect_uri = %redirect_uri, + error = %message, + "oidc token exchange failed" + ); + format!( + "token exchange failed: {e}: stage=token_exchange_other_error, provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}", + session.provider_id, config.config_url, issuer, token_endpoint, redirect_uri, config.client_id + ) + } + })?; + + // Verify the ID token (signature, issuer, audience, expiry, nonce) + let id_token = token_response + .extra_fields() + .id_token() + .ok_or_else(|| { + error!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "token_response_missing_id_token", + provider_id = %session.provider_id, + config_url = %config.config_url, + issuer = %issuer, + token_endpoint = %token_endpoint, + client_id = %config.client_id, + redirect_uri = %redirect_uri, + scopes = ?config.scopes, + "oidc token exchange failed" + ); + format!( + "no id_token in token response: provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}, scopes={}", + session.provider_id, + config.config_url, + issuer, + token_endpoint, + redirect_uri, + config.client_id, + config.scopes.join(",") + ) + })?; + + let verifier = client + .id_token_verifier() + .set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud)); + let verified = id_token.claims(&verifier, &Nonce::new(session.nonce.clone())); + if let Err(e) = verified { + let verification_error = e.to_string(); + warn!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "id_token_verification_retry", + provider_id = %session.provider_id, + config_url = %config.config_url, + issuer = %issuer, + token_endpoint = %token_endpoint, + client_id = %config.client_id, + other_audiences = ?config.other_audiences, + error = %verification_error, + "oidc id token verification failed" + ); + let refreshed_state = self + .provider_runtime + .refresh_provider_state(&session.provider_id, config) + .await + .map_err(|refresh_err| { + format!( + "ID token verification failed: {verification_error}; failed to refresh provider metadata: {refresh_err}" + ) + })?; + + warn!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "jwks_metadata_refreshed", + provider_id = %session.provider_id, + config_url = %config.config_url, + issuer = %issuer, + "oidc provider metadata refreshed" + ); + + let client = CoreClient::from_provider_metadata( + refreshed_state.metadata.console()?.clone(), + ClientId::new(config.client_id.clone()), + config.client_secret.as_ref().map(|s| ClientSecret::new(s.clone())), + ) + .set_auth_type(AuthType::RequestBody); + + let verifier = client + .id_token_verifier() + .set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud)); + id_token + .claims(&verifier, &Nonce::new(session.nonce.clone())) + .map_err(|retry_err| { + error!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "id_token_verification_failed", + provider_id = %session.provider_id, + config_url = %config.config_url, + issuer = %issuer, + token_endpoint = %token_endpoint, + client_id = %config.client_id, + other_audiences = ?config.other_audiences, + original_error = %verification_error, + retry_error = %retry_err, + "oidc id token verification failed" + ); + format!( + "ID token verification failed after JWKS refresh: {retry_err}; original_error={verification_error}; provider_id={}, config_url={}, issuer={}, token_endpoint={}, client_id={}, other_audiences={}", + session.provider_id, + config.config_url, + issuer, + token_endpoint, + config.client_id, + config.other_audiences.join(",") + ) + })?; + } + + // Extract raw claims from the verified JWT for custom claim support + // (the crate verifies signature/expiry/nonce; we decode payload for non-standard claims) + let raw_jwt = id_token.to_string(); + let raw = decode_jwt_payload(&raw_jwt); + + let claims = OidcClaims { + sub: extract_string_claim(&raw, "sub"), + email: extract_string_claim(&raw, &config.email_claim), + username: extract_string_claim(&raw, &config.username_claim), + groups: extract_canonical_group_values(&raw, &config.groups_claim, &config.roles_claim), + raw, + }; + + Ok((claims, session.provider_id.clone(), session, raw_jwt)) + } + + /// Store a one-time logout session keyed by an opaque token so the console can + /// trigger browser logout without persisting the raw ID token. + pub(crate) async fn create_logout_token(&self, provider_id: &str, id_token: &str) -> Result { + if !self.configs.contains_key(provider_id) { + return Err(format!("unknown OIDC provider: {provider_id}")); + } + + let token = CsrfToken::new_random().secret().clone(); + self.state_store + .insert_logout( + token.clone(), + OidcLogoutSession { + provider_id: provider_id.to_string(), + id_token: id_token.to_string(), + }, + ) + .await; + + Ok(token) + } + + /// Build the RP-initiated logout URL for a previously issued logout token. + /// Returns `Ok(None)` when the provider does not advertise an end-session endpoint. + pub(crate) async fn build_logout_url( + &self, + logout_token: &str, + post_logout_redirect_uri: &str, + ) -> Result, String> { + let session = self + .state_store + .take_logout(logout_token) + .await + .ok_or_else(|| "invalid or expired OIDC logout token".to_string())?; + + let config = self + .get_provider_config(&session.provider_id) + .ok_or_else(|| format!("unknown OIDC provider: {}", session.provider_id))?; + let state = self + .provider_runtime + .ensure_provider_state(&session.provider_id, config) + .await?; + let Some(end_session_endpoint) = state.metadata.console()?.additional_metadata().end_session_endpoint.clone() else { + return Ok(None); + }; + + let id_token: CoreIdToken = session + .id_token + .parse() + .map_err(|e: serde_json::Error| format!("failed to parse ID token for logout: {e}"))?; + let post_logout_redirect_uri = PostLogoutRedirectUrl::new(post_logout_redirect_uri.to_string()) + .map_err(|e| format!("invalid post logout redirect URI: {e}"))?; + + let logout_url = LogoutRequest::from(end_session_endpoint) + .set_id_token_hint(&id_token) + .set_client_id(ClientId::new(config.client_id.clone())) + .set_post_logout_redirect_uri(post_logout_redirect_uri) + .http_get_url() + .to_string(); + + Ok(Some(logout_url)) + } + + /// Map OIDC claims to rustfs policy names. + #[cfg(test)] + pub(crate) fn map_claims_to_policies(&self, provider_id: &str, claims: &OidcClaims) -> (Vec, Vec) { + let Some(rule) = self.authorization_rule(provider_id) else { + return (Vec::new(), Vec::new()); + }; + CoreFederatedAuthorizationMapper::map_policies_and_groups(provider_id, rule, &claims.groups, &claims.raw, true) + } + + /// Policy names produced only by the canonical groups in claim-based mode: the groups claim + /// plus the roles claim values that `extract_canonical_group_values` merges into it. + /// + /// Identity providers emit built-in groups and roles that can never have a matching policy + /// (for example `DOMAIN\Domain Users` or Keycloak's `offline_access`), so the session binding + /// may ignore these names when no such policy exists. Names from a fixed role policy or a + /// dedicated policy claim are never included, so those configurations keep requiring every + /// policy to resolve. + #[cfg(test)] + pub(crate) fn group_claim_policy_names(&self, provider_id: &str, claims: &OidcClaims) -> Vec { + let Some(rule) = self.authorization_rule(provider_id) else { + return Vec::new(); + }; + CoreFederatedAuthorizationMapper::group_claim_policy_names(rule, &claims.groups, &claims.raw) + } + + #[cfg(test)] + fn authorization_rule(&self, provider_id: &str) -> Option> { + self.get_provider_config(provider_id).map(|config| { + FederatedAuthorizationRuleRef::new( + &config.claim_name, + &config.claim_prefix, + &config.role_policy, + &config.groups_claim, + &config.roles_claim, + ) + }) + } + + /// Verify a raw JWT (id_token) for the AssumeRoleWithWebIdentity flow. + /// + /// Unlike the authorization code flow, ARWWI receives a raw JWT directly + /// (not via code exchange). This method: + /// 1. Decodes the JWT payload to extract the `iss` claim + /// 2. Finds the OIDC provider whose issuer matches + /// 3. Verifies signature, issuer, audience, and expiry (nonce is skipped) + /// 4. Extracts claims using the provider's claim configuration + pub(crate) async fn verify_web_identity_token(&self, jwt: &str) -> Result<(OidcClaims, String /* provider_id */), String> { + // Decode JWT payload without verification to get the issuer claim + let raw_claims = decode_jwt_payload(jwt); + let issuer = raw_claims + .get("iss") + .and_then(|v| v.as_str()) + .ok_or_else(|| "JWT missing 'iss' claim".to_string())?; + + // Find matching provider by issuer + let (provider_id, config, mut state) = self + .provider_runtime + .find_provider_by_issuer(issuer, &self.configs) + .ok_or_else(|| format!("no OIDC provider configured for issuer: {issuer}"))?; + + state = self + .provider_runtime + .ensure_provider_state_if_stale(&provider_id, &config, &state) + .await?; + + // Parse raw JWT string into CoreIdToken + let id_token: CoreIdToken = jwt + .parse() + .map_err(|e: serde_json::Error| format!("failed to parse JWT as ID token: {e}"))?; + + // Verify the token (signature, issuer, audience, expiry) — skip nonce + // (nonce is only required for the authorization code flow) + let verifier = state + .metadata + .verifier(&config) + .set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud)); + if let Err(e) = id_token.claims(&verifier, |_: Option<&Nonce>| Ok(())) { + state = self + .provider_runtime + .refresh_provider_state(&provider_id, &config) + .await + .map_err(|refresh_err| { + format!("ID token verification failed: {e}; failed to refresh provider metadata: {refresh_err}") + })?; + + let verifier = state + .metadata + .verifier(&config) + .set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud)); + id_token + .claims(&verifier, |_: Option<&Nonce>| Ok(())) + .map_err(|retry_err| format!("ID token verification failed after JWKS refresh: {retry_err}"))?; + } + + // Extract claims using the provider's claim configuration + let claims = OidcClaims { + sub: extract_string_claim(&raw_claims, "sub"), + email: extract_string_claim(&raw_claims, &config.email_claim), + username: extract_string_claim(&raw_claims, &config.username_claim), + groups: extract_canonical_group_values(&raw_claims, &config.groups_claim, &config.roles_claim), + raw: raw_claims, + }; + + Ok((claims, provider_id.to_string())) + } + /// Get a provider config by ID. + pub(crate) fn get_provider_config(&self, id: &str) -> Option<&OidcProviderConfig> { + self.configs.get(id).map(|provider| &provider.config) + } +} + +/// Decode the payload section of a JWT without validation (token must already be verified). +pub(crate) fn decode_jwt_payload(token: &str) -> HashMap { + let parts: Vec<&str> = token.split('.').collect(); + if parts.len() < 2 { + return HashMap::new(); + } + let payload_bytes = base64_simd::URL_SAFE_NO_PAD.decode_to_vec(parts[1]); + match payload_bytes { + Ok(bytes) => serde_json::from_slice(&bytes).unwrap_or_default(), + Err(_) => HashMap::new(), + } +} + +/// Extract a string claim from raw claims with case-insensitive fallback. +fn extract_string_claim(claims: &HashMap, key: &str) -> String { + match get_claim_case_insensitive(claims, key) { + ClaimLookup::Found(value) => value.as_str().unwrap_or_default().to_string(), + ClaimLookup::Missing | ClaimLookup::Ambiguous => String::new(), + } +} + +/// Extract a groups/array claim from raw claims with case-insensitive fallback. Handles both string arrays and single strings. +fn extract_groups_claim(claims: &HashMap, key: &str) -> Vec { + match get_claim_case_insensitive(claims, key) { + ClaimLookup::Found(serde_json::Value::Array(arr)) => arr.iter().filter_map(|v| v.as_str().map(String::from)).collect(), + ClaimLookup::Found(serde_json::Value::String(s)) => s.split(',').map(|s| s.trim().to_string()).collect(), + _ => vec![], + } +} + +fn extract_canonical_group_values( + claims: &HashMap, + groups_claim: &str, + roles_claim: &str, +) -> Vec { + let mut groups = extract_groups_claim(claims, groups_claim); + if !roles_claim.is_empty() && roles_claim != groups_claim { + groups.extend(extract_groups_claim(claims, roles_claim)); + } + groups.retain(|g| !g.is_empty()); + groups.sort(); + groups.dedup(); + groups +} + +fn verified_identity(provider: FederatedProviderRef, claims: OidcClaims) -> VerifiedFederatedIdentity { + VerifiedFederatedIdentity::from_claims( + provider, + FederatedClaims { + sub: claims.sub, + email: claims.email, + username: claims.username, + groups: claims.groups, + raw: claims.raw, + }, + ) +} + +fn authorization_rules(oidc: &OidcSys) -> FederatedAuthorizationRules { + FederatedAuthorizationRules::new(oidc.provider_configs().map(|config| { + FederatedAuthorizationRule::new( + config.id.clone(), + config.claim_name.clone(), + config.claim_prefix.clone(), + config.role_policy.clone(), + config.groups_claim.clone(), + config.roles_claim.clone(), + ) + })) +} + +pub struct StandardOidcAdapter { + oidc: Arc, + authorization_rules: FederatedAuthorizationRules, +} + +impl StandardOidcAdapter { + pub fn new(oidc: Arc) -> Self { + let authorization_rules = authorization_rules(&oidc); + Self { + oidc, + authorization_rules, + } + } + + pub fn authorization_rules(&self) -> FederatedAuthorizationRules { + self.authorization_rules.clone() + } + + pub fn into_service(self: Arc, mapper: CoreFederatedAuthorizationMapper) -> FederatedIdentityService { + let authentication: Arc = self.clone(); + let provider_query: Arc = self; + FederatedIdentityService::from_standard_oidc_parts(provider_query, authentication, mapper) + } +} + +impl OidcConfigQuery for StandardOidcAdapter { + fn site_replication_snapshot(&self) -> OidcSiteReplicationSnapshot { + OidcSiteReplicationSnapshot::from_config_snapshot(&self.oidc.config_snapshot()) + } +} + +#[async_trait::async_trait] +impl StandardOidcAuthentication for StandardOidcAdapter { + async fn authorize_url( + &self, + provider_id: &str, + redirect_uri: &str, + redirect_after: Option, + ) -> FederationResult { + self.oidc + .authorize_url(provider_id, redirect_uri, redirect_after) + .await + .map_err(FederationError::Authorization) + } + + async fn exchange_identity( + &self, + state: &str, + code: &str, + redirect_uri: &str, + ) -> FederationResult { + let (claims, provider_id, session, id_token) = self + .oidc + .exchange_code(state, code, redirect_uri) + .await + .map_err(FederationError::CodeExchange)?; + let provider = FederatedProviderRef::new(provider_id); + Ok(VerifiedFederatedCodeExchange::new( + verified_identity(provider.clone(), claims), + session.redirect_after, + OpaqueLogoutContinuation::new(provider, id_token), + )) + } + + async fn verify_identity(&self, jwt: &str) -> FederationResult { + let (claims, provider_id) = self + .oidc + .verify_web_identity_token(jwt) + .await + .map_err(FederationError::TokenVerification)?; + Ok(verified_identity(FederatedProviderRef::new(provider_id), claims)) + } + + async fn create_logout_token(&self, continuation: OpaqueLogoutContinuation) -> FederationResult { + let (provider, id_token) = continuation.into_parts(); + self.oidc + .create_logout_token(provider.as_str(), &id_token) + .await + .map_err(FederationError::Logout) + } + + async fn build_logout_url(&self, logout_token: &str, post_logout_redirect_uri: &str) -> FederationResult> { + self.oidc + .build_logout_url(logout_token, post_logout_redirect_uri) + .await + .map_err(FederationError::Logout) + } +} + +impl FederatedProviderQuery for StandardOidcAdapter { + fn has_providers(&self) -> bool { + self.oidc.has_providers() + } + + fn list_providers(&self) -> Vec { + self.oidc.list_providers() + } + + fn list_visible_providers(&self) -> Vec { + self.oidc.list_visible_providers() + } + + fn redirect_policy(&self, provider_id: &str) -> Option { + self.oidc + .get_provider_config(provider_id) + .map(|config| FederatedRedirectPolicy { + redirect_uri: config.redirect_uri.clone(), + allow_request_origin: config.redirect_uri_dynamic, + }) + } +} + +#[cfg(test)] +#[path = "runtime_tests.rs"] +mod tests; + +#[cfg(test)] +#[path = "recovery_tests.rs"] +mod recovery_tests; diff --git a/crates/iam/src/oidc/runtime_tests.rs b/crates/iam/src/oidc/runtime_tests.rs new file mode 100644 index 000000000..52181e530 --- /dev/null +++ b/crates/iam/src/oidc/runtime_tests.rs @@ -0,0 +1,796 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use crate::oidc::{config::*, provider::*, test_support::*}; +use jsonwebtoken::{Algorithm, Header}; +use openidconnect::ProviderMetadataWithLogout; +use rustfs_utils::egress::{ENV_OUTBOUND_ALLOW_ORIGINS, OutboundPolicy}; +use std::time::{Duration as StdDuration, Instant}; +use url::Url; + +#[test] +fn test_extract_string_claim() { + let mut claims = HashMap::new(); + claims.insert("email".to_string(), serde_json::json!("user@example.com")); + claims.insert("sub".to_string(), serde_json::json!("12345")); + + assert_eq!(extract_string_claim(&claims, "email"), "user@example.com"); + assert_eq!(extract_string_claim(&claims, "sub"), "12345"); + assert_eq!(extract_string_claim(&claims, "missing"), ""); +} +#[test] +fn test_extract_groups_claim_array() { + let mut claims = HashMap::new(); + claims.insert("groups".to_string(), serde_json::json!(["admin", "developers", "readonly"])); + + let groups = extract_groups_claim(&claims, "groups"); + assert_eq!(groups, vec!["admin", "developers", "readonly"]); +} +#[test] +fn test_extract_groups_claim_string() { + let mut claims = HashMap::new(); + claims.insert("groups".to_string(), serde_json::json!("admin,developers")); + + let groups = extract_groups_claim(&claims, "groups"); + assert_eq!(groups, vec!["admin", "developers"]); +} +#[test] +fn test_extract_groups_claim_missing() { + let claims = HashMap::new(); + let groups = extract_groups_claim(&claims, "groups"); + assert!(groups.is_empty()); +} +#[test] +fn test_extract_groups_claim_number() { + let mut claims = HashMap::new(); + claims.insert("groups".to_string(), serde_json::json!(42)); + let groups = extract_groups_claim(&claims, "groups"); + assert!(groups.is_empty()); +} +#[test] +fn test_extract_canonical_group_values_merges_groups_and_roles() { + let mut claims = HashMap::new(); + claims.insert("groups".to_string(), serde_json::json!(["devs", "admins"])); + claims.insert("roles".to_string(), serde_json::json!(["admins", "consoleAdmin"])); + + let merged = extract_canonical_group_values(&claims, "groups", "roles"); + assert_eq!(merged, vec!["admins", "consoleAdmin", "devs"]); +} +#[test] +fn test_extract_canonical_group_values_skips_duplicate_claim_name() { + let mut claims = HashMap::new(); + claims.insert("roles".to_string(), serde_json::json!(["consoleAdmin"])); + + let merged = extract_canonical_group_values(&claims, "roles", "roles"); + assert_eq!(merged, vec!["consoleAdmin"]); +} +#[test] +fn test_extract_canonical_group_values_roles_only() { + let mut claims = HashMap::new(); + claims.insert("roles".to_string(), serde_json::json!(["consoleAdmin", "bucket-reader"])); + + let merged = extract_canonical_group_values(&claims, "groups", "roles"); + assert_eq!(merged, vec!["bucket-reader", "consoleAdmin"]); +} +#[test] +fn test_extract_string_claim_case_insensitive() { + let mut claims = HashMap::new(); + claims.insert("policyminio".to_string(), serde_json::json!("consoleAdmin")); + + assert_eq!(extract_string_claim(&claims, "policyMinio"), "consoleAdmin"); + assert_eq!(extract_string_claim(&claims, "POLICYMINIO"), "consoleAdmin"); + assert_eq!(extract_string_claim(&claims, "policyminio"), "consoleAdmin"); +} +#[test] +fn test_extract_groups_claim_case_insensitive() { + let mut claims = HashMap::new(); + claims.insert("policyminio".to_string(), serde_json::json!(["consoleAdmin", "readwrite"])); + + let groups = extract_groups_claim(&claims, "policyMinio"); + assert_eq!(groups, vec!["consoleAdmin", "readwrite"]); + + let groups = extract_groups_claim(&claims, "POLICYMINIO"); + assert_eq!(groups, vec!["consoleAdmin", "readwrite"]); + + let groups = extract_groups_claim(&claims, "policyminio"); + assert_eq!(groups, vec!["consoleAdmin", "readwrite"]); +} +#[test] +fn test_extract_groups_claim_exact_match_preferred() { + let mut claims = HashMap::new(); + claims.insert("Policy".to_string(), serde_json::json!(["exact_match"])); + claims.insert("policy".to_string(), serde_json::json!(["lowercase"])); + + let groups = extract_groups_claim(&claims, "Policy"); + assert_eq!(groups, vec!["exact_match"]); +} +#[test] +fn test_extract_string_claim_ambiguous_case_insensitive_match_returns_empty() { + let mut claims = HashMap::new(); + claims.insert("Policy".to_string(), serde_json::json!("exact_match")); + claims.insert("policy".to_string(), serde_json::json!("lowercase")); + + assert_eq!(extract_string_claim(&claims, "POLICY"), ""); +} +#[test] +fn test_extract_groups_claim_ambiguous_case_insensitive_match_returns_empty() { + let mut claims = HashMap::new(); + claims.insert("Policy".to_string(), serde_json::json!(["exact_match"])); + claims.insert("policy".to_string(), serde_json::json!(["lowercase"])); + + let groups = extract_groups_claim(&claims, "POLICY"); + assert!(groups.is_empty()); +} +#[test] +fn test_decode_jwt_payload() { + let payload = r#"{"sub":"user123","email":"user@example.com"}"#; + let payload_b64 = base64_simd::URL_SAFE_NO_PAD.encode_to_string(payload.as_bytes()); + let token = format!("eyJhbGciOiJSUzI1NiJ9.{payload_b64}.signature"); + + let claims = decode_jwt_payload(&token); + assert_eq!(claims.get("sub").and_then(|v| v.as_str()), Some("user123")); + assert_eq!(claims.get("email").and_then(|v| v.as_str()), Some("user@example.com")); +} +#[tokio::test] +async fn web_identity_verification_refreshes_rotated_jwks() { + let (_, initial_jwk) = oidc_es256_key_and_jwk("initial"); + let (rotated_key, rotated_jwk) = oidc_es256_key_and_jwk("rotated"); + let initial_jwks = serde_json::json!({ "keys": [initial_jwk] }).to_string(); + let rotated_jwks = serde_json::json!({ "keys": [rotated_jwk] }).to_string(); + let Some((base, handle)) = start_mock_oidc_discovery_server_with_jwks( + |base| (base.to_string(), format!("{base}/jwks"), "/jwks".to_string()), + 4, + "ES256", + false, + move |fetch| { + if fetch == 0 { + initial_jwks.clone() + } else { + rotated_jwks.clone() + } + }, + ) else { + return; + }; + + let config = build_mocked_oidc_provider_config("rotating", &base); + let policy = OutboundPolicy::from_allowed_origins(&base).expect("loopback origin should be allowed"); + let http_client = ReqwestHttpClient::with_policy(policy); + let state = discover_provider(&config, &http_client) + .await + .expect("initial OIDC discovery should succeed"); + let sys = OidcSys { + configs: HashMap::from([(config.id.clone(), test_sourced_config(config.clone()))]), + provider_runtime: ProviderRuntime::new(http_client, HashMap::from([(config.id.clone(), state)])), + state_store: OidcStateStore::new(), + pending_discovery: Default::default(), + }; + + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("system clock should be after Unix epoch") + .as_secs(); + let mut header = Header::new(Algorithm::ES256); + header.kid = Some("rotated".to_string()); + let token = jsonwebtoken::encode( + &header, + &serde_json::json!({ + "iss": base, + "sub": "rotated-user", + "aud": config.client_id, + "iat": now, + "exp": now + 300, + "groups": ["readwrite"], + }), + &rotated_key, + ) + .expect("rotated OIDC token should sign"); + + let (claims, provider_id) = sys + .verify_web_identity_token(&token) + .await + .expect("verification should refresh JWKS and accept the rotated key"); + assert_eq!(provider_id, "rotating"); + assert_eq!(claims.sub, "rotated-user"); + assert_eq!(claims.groups, vec!["readwrite"]); + handle.join().expect("rotating JWKS mock server should exit cleanly"); +} +#[tokio::test] +async fn complete_provider_console_login_preserves_hidden_and_issuer_modes() { + assert_complete_console_login(false).await; +} + +#[tokio::test] +async fn console_login_recovers_startup_discovery_with_authorization_rules() { + assert_complete_console_login(true).await; +} + +async fn assert_complete_console_login(initially_unavailable: bool) { + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + for hidden in [false, true] { + for explicit in [false, true] { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let base = format!("http://{}", listener.local_addr().unwrap()); + let issuer = if explicit { + "https://issuer.example.com".to_string() + } else { + base.clone() + }; + let mut config = build_mocked_oidc_provider_config("console", &format!("{base}/.well-known/openid-configuration")); + config.hide_from_ui = hidden; + config.claim_name = "groups".into(); + config.claim_prefix = "team/".into(); + config.issuer = explicit.then(|| issuer.clone()); + config.client_secret = Some(Nonce::new_random().secret().clone()); + let server_config = config.clone(); + let server_base = base.clone(); + let redirect = "https://console.example.com/oauth_callback"; + let (key, jwk) = oidc_es256_key_and_jwk("console"); + let (auth_tx, auth_rx) = tokio::sync::oneshot::channel::>(); + let server = tokio::spawn(async move { + let mut auth_rx = Some(auth_rx); + let failures = if initially_unavailable { + if explicit { 1 } else { 2 } + } else { + 0 + }; + for _ in 0..failures { + let (mut stream, _) = listener.accept().await.expect("startup discovery request"); + let mut request = Vec::new(); + while !request.ends_with(b"\r\n\r\n") { + request.push(stream.read_u8().await.expect("discovery request headers")); + assert!(request.len() < 8192); + } + stream + .write_all(b"HTTP/1.1 503 Service Unavailable\r\nContent-Length: 0\r\nConnection: close\r\n\r\n") + .await + .expect("startup discovery failure response"); + } + for expected_path in ["/.well-known/openid-configuration", "/jwks", "/token"] { + let (mut stream, _) = tokio::time::timeout(StdDuration::from_secs(10), listener.accept()) + .await + .unwrap() + .unwrap(); + let mut bytes = Vec::new(); + let header_end = loop { + bytes.push(stream.read_u8().await.unwrap()); + assert!(bytes.len() < 8192); + if bytes.ends_with(b"\r\n\r\n") { + break bytes.len(); + } + }; + let headers = String::from_utf8(bytes).unwrap(); + let request_line = headers.lines().next().unwrap(); + assert_eq!(request_line.split_whitespace().nth(1), Some(expected_path)); + let headers_map: HashMap<_, _> = headers + .lines() + .skip(1) + .filter_map(|line| line.split_once(':')) + .map(|(name, value)| (name.to_ascii_lowercase(), value.trim().to_string())) + .collect(); + let body = match expected_path { + "/.well-known/openid-configuration" => { + assert!(request_line.starts_with("GET ")); + assert_eq!(headers_map["accept"], "application/json"); + serde_json::json!({ + "issuer": issuer, "authorization_endpoint": format!("{server_base}/authorize"), + "token_endpoint": format!("{server_base}/token"), "jwks_uri": format!("{server_base}/jwks"), + "response_types_supported": ["code"], "subject_types_supported": ["public"], + "id_token_signing_alg_values_supported": ["ES256"] + }) + } + "/jwks" => { + assert!(request_line.starts_with("GET ")); + assert!(headers_map["accept"].contains("application/json")); + serde_json::json!({"keys": [jwk]}) + } + "/token" => { + assert!(request_line.starts_with("POST ")); + assert_eq!(headers_map["accept"], "application/json"); + assert!(headers_map["content-type"].starts_with("application/x-www-form-urlencoded")); + let length: usize = headers_map["content-length"].parse().unwrap(); + assert!(header_end + length < 16384); + let mut body = vec![0; length]; + stream.read_exact(&mut body).await.unwrap(); + let form: HashMap = url::form_urlencoded::parse(&body).into_owned().collect(); + assert_eq!(form["grant_type"], "authorization_code"); + assert_eq!(form["code"], "test-authorization-code"); + assert_eq!(form["client_id"], server_config.client_id); + assert_eq!(Some(&form["client_secret"]), server_config.client_secret.as_ref()); + assert_eq!(form["redirect_uri"], redirect); + let auth = auth_rx.take().unwrap().await.unwrap(); + let challenge = PkceCodeChallenge::from_code_verifier_sha256(&PkceCodeVerifier::new( + form["code_verifier"].clone(), + )); + assert_eq!(challenge.as_str(), auth["code_challenge"]); + let now = time::OffsetDateTime::now_utc().unix_timestamp(); + let mut header = Header::new(Algorithm::ES256); + header.kid = Some("console".into()); + let token = jsonwebtoken::encode( + &header, + &serde_json::json!({ + "iss": issuer, "sub": "existing-user", "aud": server_config.client_id, + "iat": now, "exp": now + 300, "nonce": auth["nonce"], + "email": "user@example.com", "groups": ["readwrite"] + }), + &key, + ) + .unwrap(); + serde_json::json!({"access_token": "test-access-token", "token_type": "Bearer", "id_token": token}) + } + _ => unreachable!(), + } + .to_string(); + stream.write_all(format!("HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", body.len()).as_bytes()).await.unwrap(); + } + }); + let http_client = ReqwestHttpClient::with_policy(OutboundPolicy::from_allowed_origins(&base).unwrap()); + let sys = Arc::new( + OidcSys::new_with_config_snapshot(http_client, OidcConfigSnapshot::new(vec![test_sourced_config(config)])) + .await + .expect("OIDC initialization remains available during an IdP outage"), + ); + let adapter = StandardOidcAdapter::new(sys.clone()); + let mapper = CoreFederatedAuthorizationMapper::new(adapter.authorization_rules()); + assert_eq!(sys.list_visible_providers().len(), usize::from(!hidden)); + if initially_unavailable { + assert!(sys.provider_runtime.get_provider_state("console").is_err()); + assert!(sys.authorize_url("console", redirect, None).await.is_err()); + tokio::time::timeout(StdDuration::from_secs(10), sys.recover_failed_providers()) + .await + .expect("discovery should recover after the IdP becomes available"); + } + let auth_url = sys.authorize_url("console", redirect, Some("/buckets".into())).await.unwrap(); + let auth_url = Url::parse(&auth_url).unwrap(); + assert_eq!(auth_url.as_str().split('?').next(), Some(format!("{base}/authorize").as_str())); + let auth: HashMap = auth_url.query_pairs().into_owned().collect(); + assert_eq!(auth["response_type"], "code"); + assert_eq!(auth["client_id"], "rustfs-oidc-test"); + assert!(!auth["nonce"].is_empty()); + assert!(!auth["state"].is_empty()); + assert_eq!(auth["redirect_uri"], redirect); + assert_eq!(auth["code_challenge_method"], "S256"); + assert!(auth["scope"].split_whitespace().any(|scope| scope == "openid")); + let state = auth["state"].clone(); + auth_tx.send(auth).unwrap(); + let exchange = adapter + .exchange_identity(&state, "test-authorization-code", redirect) + .await + .unwrap_or_else(|err| panic!("hidden={hidden}, explicit={explicit}: {err}")); + let (identity, redirect_after, _) = exchange.into_parts(); + assert_eq!(identity.source_groups(), ["readwrite"]); + let authorization = mapper.map(identity); + assert_eq!(authorization.provider_id(), "console"); + assert_eq!(authorization.subject(), "existing-user"); + assert_eq!(authorization.email(), "user@example.com"); + assert_eq!(authorization.policies(), ["team/readwrite"]); + assert_eq!(redirect_after.as_deref(), Some("/buckets")); + assert!(matches!(sys.exchange_code(&state, "test-authorization-code", redirect).await, + Err(error) if error == "invalid or expired OIDC state")); + server.await.unwrap(); + } + } +} +#[test] +fn test_decode_jwt_payload_invalid() { + assert!(decode_jwt_payload("not-a-jwt").is_empty()); + assert!(decode_jwt_payload("").is_empty()); +} +#[test] +fn test_core_token_response_accepts_rfc3339_updated_at() { + // Signature verification happens later; this test covers token response deserialization. + let id_token = "eyJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJodHRwczovL2F1dGguZXhhbXBsZS5jb20vb2lkYyIsInN1YiI6InVzZXItMSIsImF1ZCI6InJ1c3RmcyIsImV4cCI6MTc4NDQzMjc2OSwiaWF0IjoxNzgzMjIzMTY5LCJ1cGRhdGVkX2F0IjoiMjAyNi0wNy0wM1QwNDo1MDo1MC44MTFaIn0.c2ln"; + let body = serde_json::json!({ + "scope": "openid roles profile email", + "token_type": "Bearer", + "access_token": "access-token", + "expires_in": 1209600, + "id_token": id_token, + }) + .to_string(); + + let response: openidconnect::core::CoreTokenResponse = + serde_json::from_str(&body).expect("RFC3339 updated_at should parse in token response"); + + assert!(response.extra_fields().id_token().is_some()); +} +#[test] +fn test_map_claims_to_policies_no_provider() { + let sys = OidcSys::empty().expect("failed to initialize empty OIDC system"); + + let claims = OidcClaims { + sub: "user123".to_string(), + email: "user@example.com".to_string(), + username: "user".to_string(), + groups: vec!["admin".to_string(), "devs".to_string()], + raw: HashMap::new(), + }; + + let (policies, groups) = sys.map_claims_to_policies("nonexistent", &claims); + assert!(policies.is_empty()); + assert!(groups.is_empty()); +} +#[test] +fn test_oidc_claims_default() { + let claims = OidcClaims::default(); + assert!(claims.sub.is_empty()); + assert!(claims.email.is_empty()); + assert!(claims.username.is_empty()); + assert!(claims.groups.is_empty()); + assert!(claims.raw.is_empty()); +} +#[test] +fn test_oidc_claims_serde_roundtrip() { + let claims = OidcClaims { + sub: "user123".to_string(), + email: "user@example.com".to_string(), + username: "testuser".to_string(), + groups: vec!["admin".to_string(), "devs".to_string()], + raw: { + let mut m = HashMap::new(); + m.insert("custom".to_string(), serde_json::json!("value")); + m + }, + }; + + let json = serde_json::to_string(&claims).unwrap(); + let deserialized: OidcClaims = serde_json::from_str(&json).unwrap(); + assert_eq!(deserialized.sub, "user123"); + assert_eq!(deserialized.email, "user@example.com"); + assert_eq!(deserialized.groups.len(), 2); +} +#[test] +fn test_oidc_provider_summary_serde() { + let summary = OidcProviderSummary { + provider_id: "okta".to_string(), + display_name: "Okta SSO".to_string(), + }; + + let json = serde_json::to_string(&summary).unwrap(); + let deserialized: OidcProviderSummary = serde_json::from_str(&json).unwrap(); + assert_eq!(deserialized.provider_id, "okta"); + assert_eq!(deserialized.display_name, "Okta SSO"); +} +#[test] +fn test_oidc_sys_empty() { + let sys = OidcSys::empty().expect("failed to initialize empty OIDC system"); + assert!(!sys.has_providers()); + assert!(sys.config_snapshot().providers().is_empty()); +} +#[tokio::test] +async fn oidc_token_exchange_reports_forbidden_token_endpoint() { + let provider_id = "default"; + let token_endpoint = "http://192.168.65.254:8080/realms/rustfs/protocol/openid-connect/token"; + let metadata = serde_json::from_value::(serde_json::json!({ + "issuer": "https://idp.example.com/realms/rustfs", + "authorization_endpoint": "https://idp.example.com/realms/rustfs/protocol/openid-connect/auth", + "token_endpoint": token_endpoint, + "jwks_uri": "https://idp.example.com/realms/rustfs/protocol/openid-connect/certs", + "response_types_supported": ["code"], + "subject_types_supported": ["public"], + "id_token_signing_alg_values_supported": ["RS256"] + })) + .expect("provider metadata should parse"); + let config = + build_mocked_oidc_provider_config(provider_id, "https://idp.example.com/realms/rustfs/.well-known/openid-configuration"); + let state_store = OidcStateStore::new(); + state_store + .insert( + "test-state".to_string(), + OidcAuthSession { + provider_id: provider_id.to_string(), + pkce_verifier: "test-pkce-verifier".to_string(), + nonce: "test-nonce".to_string(), + redirect_after: None, + }, + ) + .await; + let sys = OidcSys { + configs: HashMap::from([(provider_id.to_string(), test_sourced_config(config))]), + provider_runtime: ProviderRuntime::new( + ReqwestHttpClient::with_policy(OutboundPolicy::default()), + HashMap::from([( + provider_id.to_string(), + ProviderState { + metadata: DiscoveredProviderMetadata::Console(Box::new(metadata)), + discovered_at: Instant::now(), + }, + )]), + ), + state_store, + pending_discovery: Default::default(), + }; + + let error = match sys + .exchange_code("test-state", "test-code", "https://console.example.com/oauth_callback") + .await + { + Ok(_) => panic!("private token endpoint should require an explicit allowlist origin"), + Err(error) => error, + }; + + assert!(error.contains("request_error_kind=forbidden_outbound")); + assert!(error.contains(&format!("add http://192.168.65.254:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}"))); +} +#[test] +fn test_list_visible_providers_hides_hidden_provider() { + let visible = test_config("dex"); + let mut hidden = test_config("kubernetes"); + hidden.hide_from_ui = true; + + let sys = make_test_sys(vec![visible, hidden]); + let listed = sys.list_visible_providers(); + + assert_eq!(listed.len(), 1); + assert!(listed.iter().any(|provider| provider.provider_id == "dex")); + assert!(!listed.iter().any(|provider| provider.provider_id == "kubernetes")); +} +#[test] +fn test_hidden_provider_still_resolvable_for_sts() { + let visible = test_config("dex"); + let mut hidden = test_config("kubernetes"); + hidden.hide_from_ui = true; + + let sys = make_test_sys(vec![visible, hidden]); + + assert!(sys.get_provider_config("kubernetes").is_some()); + assert!(sys.get_provider_config("dex").is_some()); +} +#[test] +fn test_config_snapshot_includes_hidden_for_replication() { + let visible = test_config("dex"); + let mut hidden = test_config("kubernetes"); + hidden.hide_from_ui = true; + + let sys = make_test_sys(vec![visible, hidden]); + + assert_eq!(sys.config_snapshot().providers().len(), 2); +} +#[test] +fn test_list_providers_includes_hidden_for_compatibility() { + let visible = test_config("dex"); + let mut hidden = test_config("kubernetes"); + hidden.hide_from_ui = true; + + let sys = make_test_sys(vec![visible, hidden]); + + assert_eq!(sys.list_providers().len(), 2); + assert_eq!(sys.list_visible_providers().len(), 1); +} +#[test] +fn role_policy_does_not_map_groups_as_policies() { + let mut config = test_config("authentik"); + config.role_policy = "consoleAdmin".to_string(); + config.claim_name = "policy".to_string(); + + let sys = make_test_sys(vec![config]); + + let claims = OidcClaims { + groups: vec!["authentik Admins".to_string(), "users".to_string()], + raw: HashMap::from([("policy".to_string(), serde_json::json!(["readonly"]))]), + ..Default::default() + }; + + let (policies, groups) = sys.map_claims_to_policies("authentik", &claims); + assert_eq!(groups, vec!["authentik Admins", "users"]); + assert_eq!(policies, vec!["consoleAdmin"]); +} +#[test] +fn group_claim_policy_names_cover_only_prefixed_group_values() { + let mut config = test_config("ad"); + config.claim_prefix = "oidc-".to_string(); + let sys = make_test_sys(vec![config]); + let claims = OidcClaims { + groups: vec![ + "EXAMPLE\\Domain Users".to_string(), + "admins".to_string(), + "admins".to_string(), + ], + ..Default::default() + }; + + assert_eq!( + sys.group_claim_policy_names("ad", &claims), + vec!["oidc-EXAMPLE\\Domain Users", "oidc-admins"] + ); + assert!(sys.group_claim_policy_names("missing-provider", &claims).is_empty()); +} +#[test] +fn group_claim_policy_names_include_merged_roles_claim_values() { + let mut config = test_config("keycloak"); + config.roles_claim = "roles".to_string(); + let raw = HashMap::from([ + ("groups".to_string(), serde_json::json!(["readonly"])), + ("roles".to_string(), serde_json::json!(["offline_access", "default-roles-corp"])), + ]); + let claims = OidcClaims { + groups: extract_canonical_group_values(&raw, &config.groups_claim, &config.roles_claim), + raw, + ..Default::default() + }; + let sys = make_test_sys(vec![config]); + + assert_eq!( + sys.group_claim_policy_names("keycloak", &claims), + vec!["default-roles-corp", "offline_access", "readonly"] + ); +} +#[test] +fn group_claim_policy_names_exclude_role_policy_and_dedicated_policy_claim() { + let mut role_config = test_config("role"); + role_config.role_policy = "consoleAdmin".to_string(); + let mut claim_config = test_config("claim"); + claim_config.claim_name = "policy".to_string(); + let sys = make_test_sys(vec![role_config, claim_config]); + let claims = OidcClaims { + groups: vec!["readonly".to_string(), "unmapped".to_string()], + raw: HashMap::from([("policy".to_string(), serde_json::json!(["readonly", "writeonly"]))]), + ..Default::default() + }; + + assert!(sys.group_claim_policy_names("role", &claims).is_empty()); + // `readonly` is also requested explicitly through the policy claim, so it must resolve. + assert_eq!(sys.group_claim_policy_names("claim", &claims), vec!["unmapped"]); +} +#[test] +fn test_map_claims_to_policies_with_prefix() { + let mut config = test_config("azure"); + config.claim_prefix = "oidc-".to_string(); + config.display_name = "Azure AD".to_string(); + + let sys = make_test_sys(vec![config]); + + let claims = OidcClaims { + sub: "user456".to_string(), + email: "user@corp.com".to_string(), + username: "user".to_string(), + groups: vec!["engineers".to_string()], + raw: HashMap::new(), + }; + + let (policies, groups) = sys.map_claims_to_policies("azure", &claims); + assert_eq!(groups, vec!["engineers"]); + assert!(policies.contains(&"oidc-engineers".to_string())); + assert_eq!(policies.len(), 1); +} +#[test] +fn blank_role_policy_uses_claim_mapping() { + let mut config = test_config("keycloak"); + config.role_policy = " ".to_string(); + + let sys = make_test_sys(vec![config]); + let claims = OidcClaims { + groups: vec!["readonly".to_string()], + ..Default::default() + }; + + let (policies, groups) = sys.map_claims_to_policies("keycloak", &claims); + assert_eq!(groups, vec!["readonly"]); + assert_eq!(policies, vec!["readonly"]); +} +#[test] +fn claim_mapping_keeps_groups_with_distinct_primary_claim() { + let mut config = test_config("keycloak"); + config.claim_name = "policy".to_string(); + + let sys = make_test_sys(vec![config]); + let claims = OidcClaims { + groups: vec!["developers".to_string()], + raw: HashMap::from([("policy".to_string(), serde_json::json!(["readonly"]))]), + ..Default::default() + }; + + let (policies, groups) = sys.map_claims_to_policies("keycloak", &claims); + assert_eq!(groups, vec!["developers"]); + assert_eq!(policies, vec!["developers", "readonly"]); +} +#[test] +fn test_config_snapshot_lists_providers() { + let mut config = test_config("keycloak"); + config.display_name = "Keycloak SSO".to_string(); + + let sys = make_test_sys(vec![config]); + + assert!(sys.has_providers()); + let snapshot = sys.config_snapshot(); + assert_eq!(snapshot.providers().len(), 1); + assert_eq!(snapshot.providers()[0].config.id, "keycloak"); + assert_eq!(snapshot.providers()[0].config.display_name, "Keycloak SSO"); + assert_eq!(snapshot.providers()[0].source, OidcProviderConfigSource::Persisted); +} +#[test] +fn test_get_provider_config() { + let mut config = test_config("test"); + config.client_id = "my-client".to_string(); + config.client_secret = Some("secret".to_string()); + + let sys = make_test_sys(vec![config]); + + assert!(sys.get_provider_config("test").is_some()); + assert_eq!(sys.get_provider_config("test").unwrap().client_id, "my-client"); + assert!(sys.get_provider_config("nonexistent").is_none()); +} +#[test] +fn verified_identity_preserves_provider_and_claim_values() { + let raw = HashMap::from([ + ("iss".to_string(), serde_json::json!("https://corp.example.test")), + ("department".to_string(), serde_json::json!("engineering")), + ("roles".to_string(), serde_json::json!(["reader", "admin"])), + ]); + let identity = verified_identity( + FederatedProviderRef::new("corp".to_string()), + OidcClaims { + sub: " subject-123 ".to_string(), + email: " user@example.test ".to_string(), + username: " user ".to_string(), + groups: vec![ + "source-ops".to_string(), + "source-developers".to_string(), + "source-ops".to_string(), + ], + raw: raw.clone(), + }, + ); + + assert_eq!(identity.provider().as_str(), "corp"); + assert_eq!(identity.subject(), " subject-123 "); + assert_eq!(identity.email(), " user@example.test "); + assert_eq!(identity.username(), " user "); + assert_eq!(identity.source_groups(), ["source-ops", "source-developers", "source-ops"]); + assert_eq!(identity.attributes(), &raw); +} +#[test] +fn adapter_preserves_provider_views_and_redirect_policy() { + let mut visible = test_config("default"); + visible.display_name = "Company SSO".to_string(); + visible.redirect_uri = Some("https://console.example.test/callback".to_string()); + visible.redirect_uri_dynamic = false; + let mut hidden = test_config("workload"); + hidden.display_name = "Workload Identity".to_string(); + hidden.hide_from_ui = true; + let adapter = StandardOidcAdapter::new(Arc::new(make_test_sys(vec![visible, hidden]))); + + let mut providers = adapter.list_providers(); + providers.sort_by(|left, right| left.provider_id.cmp(&right.provider_id)); + assert_eq!( + providers, + vec![ + FederatedProviderView { + provider_id: "default".to_string(), + display_name: "Company SSO".to_string(), + }, + FederatedProviderView { + provider_id: "workload".to_string(), + display_name: "Workload Identity".to_string(), + }, + ] + ); + assert_eq!( + adapter.list_visible_providers(), + vec![FederatedProviderView { + provider_id: "default".to_string(), + display_name: "Company SSO".to_string(), + }] + ); + assert_eq!( + adapter.redirect_policy("default"), + Some(FederatedRedirectPolicy { + redirect_uri: Some("https://console.example.test/callback".to_string()), + allow_request_origin: false, + }) + ); + assert_eq!(adapter.redirect_policy("missing"), None); +} diff --git a/crates/iam/src/oidc_state.rs b/crates/iam/src/oidc/state.rs similarity index 82% rename from crates/iam/src/oidc_state.rs rename to crates/iam/src/oidc/state.rs index 5cdd962ee..48af714b8 100644 --- a/crates/iam/src/oidc_state.rs +++ b/crates/iam/src/oidc/state.rs @@ -18,6 +18,8 @@ use std::sync::atomic::{AtomicU64, Ordering}; use std::time::{Duration, SystemTime, UNIX_EPOCH}; use tracing::warn; +use super::config::console_session_duration_secs; + const OIDC_STATE_CAPACITY: u64 = 10_000; const OIDC_STATE_CAPACITY_WARNING: u64 = 9_000; const OIDC_STATE_CAPACITY_CRITICAL: u64 = 10_000; @@ -25,39 +27,42 @@ const OIDC_STATE_CAPACITY_LOG_INTERVAL_SECS: u64 = 60; /// Stores the PKCE verifier and nonce for an in-flight OIDC authorization flow. #[derive(Debug, Clone)] -pub struct OidcAuthSession { - pub provider_id: String, - pub pkce_verifier: String, - pub nonce: String, - pub redirect_after: Option, +pub(super) struct OidcAuthSession { + pub(super) provider_id: String, + pub(super) pkce_verifier: String, + pub(super) nonce: String, + pub(super) redirect_after: Option, } /// Stores an ID token behind a one-time opaque handle so the console can trigger /// RP-initiated logout without persisting the raw token in browser storage. #[derive(Debug, Clone)] -pub struct OidcLogoutSession { - pub provider_id: String, - pub id_token: String, +pub(super) struct OidcLogoutSession { + pub(super) provider_id: String, + pub(super) id_token: String, } /// TTL cache for OIDC auth state (PKCE verifiers + nonces) during the authorization flow. /// Entries expire after 5 minutes and are single-use (removed on retrieval). #[derive(Clone)] -pub struct OidcStateStore { +pub(super) struct OidcStateStore { cache: Cache, logout_cache: Cache, last_capacity_log_at: Arc, } impl OidcStateStore { - pub fn new() -> Self { + pub(super) fn new() -> Self { let cache = Cache::builder() .max_capacity(OIDC_STATE_CAPACITY) .time_to_live(Duration::from_secs(300)) // 5 minute TTL .build(); let logout_cache = Cache::builder() .max_capacity(OIDC_STATE_CAPACITY) - .time_to_live(Duration::from_secs(3600)) // 1 hour TTL to match console OIDC sessions + // Keep the logout token alive for as long as the credentials it belongs to: if it + // expired sooner, a session longer than the old hard-coded hour could no longer + // trigger RP-initiated logout. + .time_to_live(Duration::from_secs(console_session_duration_secs() as u64)) .build(); Self { cache, @@ -67,7 +72,7 @@ impl OidcStateStore { } /// Store a new auth session keyed by the OAuth2 `state` parameter. - pub async fn insert(&self, state: String, session: OidcAuthSession) { + pub(super) async fn insert(&self, state: String, session: OidcAuthSession) { self.cache.insert(state, session).await; let size = self.cache.entry_count(); @@ -104,27 +109,29 @@ impl OidcStateStore { /// Retrieve and remove an auth session (single-use). Returns None if expired or not found. /// Uses `remove` which returns the value and removes it in a single operation. - pub async fn take(&self, state: &str) -> Option { + pub(super) async fn take(&self, state: &str) -> Option { self.cache.remove(state).await } /// Check if a state key exists (without consuming it). - pub async fn contains(&self, state: &str) -> bool { + #[cfg(test)] + pub(super) async fn contains(&self, state: &str) -> bool { self.cache.get(state).await.is_some() } /// Store a new one-time logout session keyed by an opaque logout token. - pub async fn insert_logout(&self, token: String, session: OidcLogoutSession) { + pub(super) async fn insert_logout(&self, token: String, session: OidcLogoutSession) { self.logout_cache.insert(token, session).await; } /// Retrieve and remove a logout session (single-use). Returns None if expired or not found. - pub async fn take_logout(&self, token: &str) -> Option { + pub(super) async fn take_logout(&self, token: &str) -> Option { self.logout_cache.remove(token).await } /// Check if a logout token exists (without consuming it). - pub async fn contains_logout(&self, token: &str) -> bool { + #[cfg(test)] + pub(super) async fn contains_logout(&self, token: &str) -> bool { self.logout_cache.get(token).await.is_some() } } @@ -138,6 +145,8 @@ impl Default for OidcStateStore { #[cfg(test)] mod tests { use super::*; + use rustfs_config::ENV_CONSOLE_OIDC_SESSION_DURATION; + use serial_test::serial; use std::sync::Arc; use tokio::sync::Barrier; @@ -300,4 +309,26 @@ mod tests { assert!(store.take_logout("logout_abc").await.is_none()); } + + #[test] + #[serial] + fn logout_retention_follows_the_bounded_session_duration() { + for (configured, expected_seconds) in [ + (None, 3600), + (Some("1"), 1), + (Some("7200"), 7200), + (Some("604801"), 604_800), + (Some("18446744073709551615"), 604_800), + ] { + temp_env::with_var(ENV_CONSOLE_OIDC_SESSION_DURATION, configured, || { + let store = OidcStateStore::new(); + assert_eq!( + store.logout_cache.policy().time_to_live(), + Some(Duration::from_secs(expected_seconds)), + "logout retention for {configured:?} must match the issued session" + ); + assert_eq!(store.cache.policy().time_to_live(), Some(Duration::from_secs(300))); + }); + } + } } diff --git a/crates/iam/src/oidc/test_support.rs b/crates/iam/src/oidc/test_support.rs new file mode 100644 index 000000000..51262f675 --- /dev/null +++ b/crates/iam/src/oidc/test_support.rs @@ -0,0 +1,527 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +use super::{ + config::{OidcProviderConfig, OidcProviderConfigSource, SourcedOidcProviderConfig}, + provider::ProviderRuntime, + runtime::OidcSys, + state::OidcStateStore, + transport::ReqwestHttpClient, +}; +use std::collections::HashMap; + +#[cfg(any(test, feature = "test-util"))] +#[doc(hidden)] +pub fn make_test_sys(configs: Vec) -> OidcSys { + let configs = configs + .into_iter() + .map(test_sourced_config) + .map(|provider| (provider.config.id.clone(), provider)) + .collect(); + OidcSys { + configs, + provider_runtime: ProviderRuntime::new( + ReqwestHttpClient::new().expect("failed to initialize OIDC HTTP clients"), + HashMap::new(), + ), + state_store: OidcStateStore::new(), + pending_discovery: Default::default(), + } +} + +#[cfg(any(test, feature = "test-util"))] +pub(super) fn test_sourced_config(config: OidcProviderConfig) -> SourcedOidcProviderConfig { + SourcedOidcProviderConfig { + config, + source: OidcProviderConfigSource::Persisted, + } +} + +#[cfg(any(test, feature = "test-util"))] +#[doc(hidden)] +pub fn test_config(id: &str) -> OidcProviderConfig { + OidcProviderConfig { + id: id.to_string(), + enabled: true, + config_url: format!("https://example.com/{id}/.well-known/openid-configuration"), + issuer: None, + client_id: "client-id".to_string(), + client_secret: None, + scopes: vec!["openid".to_string()], + other_audiences: vec![], + redirect_uri: None, + redirect_uri_dynamic: true, + claim_name: "groups".to_string(), + claim_prefix: String::new(), + role_policy: String::new(), + display_name: id.to_string(), + groups_claim: "groups".to_string(), + roles_claim: String::new(), + email_claim: "email".to_string(), + username_claim: "preferred_username".to_string(), + hide_from_ui: false, + } +} + +#[cfg(test)] +use super::{config::OidcProviderValidationResult, provider::discover_provider, transport::OidcHttpError}; +#[cfg(test)] +use jsonwebtoken::{Algorithm, EncodingKey}; +#[cfg(test)] +use openidconnect::AsyncHttpClient; +#[cfg(test)] +use rustfs_utils::egress::{OutboundDnsPolicyRejection, OutboundPolicy}; +#[cfg(test)] +use std::sync::Arc; +#[cfg(test)] +use std::sync::atomic::{AtomicUsize, Ordering}; +#[cfg(test)] +use url::Url; + +#[cfg(test)] +#[derive(Clone)] +pub(super) struct RejectingDnsResolver { + pub(super) allow_origin_can_recover: bool, + pub(super) calls: Option>, +} + +#[cfg(test)] +impl reqwest::dns::Resolve for RejectingDnsResolver { + fn resolve(&self, name: reqwest::dns::Name) -> reqwest::dns::Resolving { + if let Some(calls) = &self.calls { + calls.fetch_add(1, Ordering::Relaxed); + } + let host = name.as_str().to_string(); + let rejection = OutboundDnsPolicyRejection::new(host, self.allow_origin_can_recover); + Box::pin(async move { Err(std::io::Error::new(std::io::ErrorKind::PermissionDenied, rejection).into()) }) + } +} + +#[cfg(test)] +pub(super) fn build_mocked_oidc_provider_config(id: &str, config_url: &str) -> OidcProviderConfig { + OidcProviderConfig { + id: id.to_string(), + enabled: true, + config_url: config_url.to_string(), + issuer: None, + client_id: "rustfs-oidc-test".to_string(), + client_secret: None, + scopes: vec!["openid".to_string()], + other_audiences: vec![], + redirect_uri: None, + redirect_uri_dynamic: false, + claim_name: "sub".to_string(), + claim_prefix: "oidc".to_string(), + role_policy: String::new(), + display_name: "mock-oidc".to_string(), + groups_claim: "groups".to_string(), + roles_claim: String::new(), + email_claim: "email".to_string(), + username_claim: "username".to_string(), + hide_from_ui: false, + } +} +#[cfg(test)] +pub(super) fn read_mock_oidc_request(stream: &mut impl std::io::Read) -> String { + let mut request_bytes = Vec::new(); + let mut buffer = [0u8; 4096]; + loop { + match stream.read(&mut buffer) { + Ok(0) => break, + Ok(n) => request_bytes.extend_from_slice(&buffer[..n]), + Err(e) if matches!(e.kind(), std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut) => break, + Err(_) => break, + } + if request_bytes.windows(4).any(|w| w == b"\r\n\r\n") { + break; + } + if request_bytes.len() >= 8192 { + break; + } + } + String::from_utf8_lossy(&request_bytes).into_owned() +} +#[cfg(test)] +pub(super) fn read_mock_oidc_request_path(stream: &mut impl std::io::Read) -> String { + read_mock_oidc_request(stream) + .lines() + .next() + .unwrap_or("") + .split_whitespace() + .nth(1) + .unwrap_or("") + .to_string() +} +#[cfg(test)] +pub(super) fn mock_oidc_response(path: &str, discovery_body: &str, expected_jwks_path: &str, jwks_body: &str) -> String { + let (status, body) = if path.ends_with("/.well-known/openid-configuration") { + (200, discovery_body) + } else if path == expected_jwks_path { + (200, jwks_body) + } else { + (404, r#"{"error":"not found"}"#) + }; + + format!( + "HTTP/1.1 {status} {}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + if status == 200 { "OK" } else { "Not Found" }, + body.len() + ) +} +#[cfg(test)] +pub(super) fn start_mock_oidc_discovery_server_with_jwks( + build_discovery_issuer: F, + max_requests: usize, + signing_alg: &'static str, + workload: bool, + jwks_response: J, +) -> Option<(String, std::thread::JoinHandle<()>)> +where + F: Fn(&str) -> (String, String, String) + Send + 'static, + J: Fn(usize) -> String + Send + 'static, +{ + use std::io::Write; + use std::net::{Shutdown, TcpListener}; + use std::sync::mpsc; + use std::time::{Duration, Instant}; + + // After the last completed response, exit if no new connection arrives within this window. + // Keep the mock server alive long enough for slower CI/macOS test environments to finish + // discovery + JWKS requests without racing the shutdown timer. + const IDLE_SHUTDOWN: Duration = Duration::from_secs(1); + const ABSOLUTE_CAP: Duration = Duration::from_secs(5); + + let listener = match TcpListener::bind("127.0.0.1:0") { + Ok(listener) => listener, + Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return None, + Err(err) => panic!("test listener should bind: {err}"), + }; + let base = format!("http://{}", listener.local_addr().expect("listener local address should be available")); + let (discovery_issuer, discovery_jwks_uri, expected_jwks_path) = build_discovery_issuer(&base); + let mut discovery_document = serde_json::json!({ + "issuer": discovery_issuer, + "authorization_endpoint": format!("{base}/authorize"), + "token_endpoint": format!("{base}/token"), + "jwks_uri": discovery_jwks_uri, + "response_types_supported": ["code"], + "response_modes_supported": ["query"], + "subject_types_supported": ["public"], + "id_token_signing_alg_values_supported": [signing_alg], + }); + if workload { + let fields = discovery_document.as_object_mut().expect("mock metadata is an object"); + fields.remove("authorization_endpoint"); + fields.remove("token_endpoint"); + fields.insert("response_types_supported".into(), serde_json::json!(["id_token"])); + } + let discovery_body = discovery_document.to_string(); + let (ready_tx, ready_rx) = mpsc::channel(); + + let handle = std::thread::spawn(move || { + listener + .set_nonblocking(true) + .expect("failed to set discovery mock listener non-blocking"); + let _ = ready_tx.send(()); + + let mut seen = 0usize; + let mut jwks_fetches = 0usize; + let start = Instant::now(); + let mut last_completed = Instant::now(); + + loop { + if seen > 0 && last_completed.elapsed() >= IDLE_SHUTDOWN { + break; + } + if start.elapsed() >= ABSOLUTE_CAP { + break; + } + + let mut stream = match listener.accept() { + Ok((stream, _)) => stream, + Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => { + std::thread::sleep(Duration::from_millis(5)); + continue; + } + Err(_) => break, + }; + stream + .set_nonblocking(false) + .expect("failed to set discovery mock stream blocking"); + + seen += 1; + stream + .set_nonblocking(false) + .expect("failed to set discovery mock stream blocking"); + stream + .set_read_timeout(Some(Duration::from_secs(1))) + .expect("failed to set discovery mock read timeout"); + + let request = read_mock_oidc_request(&mut stream); + let path = request + .lines() + .next() + .and_then(|line| line.split_whitespace().nth(1)) + .unwrap_or(""); + let jwks_body = jwks_response(jwks_fetches); + if path == expected_jwks_path { + jwks_fetches += 1; + } + let mut response = mock_oidc_response(path, &discovery_body, &expected_jwks_path, &jwks_body); + if path.contains("/.well-known/openid-configuration") { + assert!( + request + .lines() + .filter_map(|line| line.split_once(':')) + .any(|(name, value)| { name.eq_ignore_ascii_case("accept") && value.trim() == "application/json" }), + "discovery Accept must remain unchanged" + ); + } + if path == expected_jwks_path { + let expected_type = if workload { + "application/jwk-set+json" + } else { + "application/json" + }; + let accepts_type = request.lines().filter_map(|line| line.split_once(':')).any(|(name, value)| { + name.eq_ignore_ascii_case("accept") && value.split(',').any(|item| item.trim() == expected_type) + }); + if !accepts_type { + response = "HTTP/1.1 406 Not Acceptable\r\nContent-Length: 0\r\nConnection: close\r\n\r\n".into(); + } else if workload { + response = response.replace("Content-Type: application/json", "Content-Type: application/jwk-set+json"); + } + } + let _ = stream.write_all(response.as_bytes()); + let _ = stream.flush(); + let _ = stream.shutdown(Shutdown::Both); + last_completed = Instant::now(); + + if seen >= max_requests { + break; + } + } + }); + ready_rx + .recv_timeout(Duration::from_millis(100)) + .expect("mock OIDC discovery server should become ready"); + + Some((base, handle)) +} +#[cfg(test)] +pub(super) fn start_mock_oidc_discovery_server( + build_discovery_issuer: F, + max_requests: usize, +) -> Option<(String, std::thread::JoinHandle<()>)> +where + F: Fn(&str) -> (String, String, String) + Send + 'static, +{ + start_mock_oidc_discovery_server_with_jwks(build_discovery_issuer, max_requests, "RS256", false, |_| { + r#"{"keys":[]}"#.to_string() + }) +} +#[cfg(test)] +pub(super) fn oidc_es256_key_and_jwk(kid: &str) -> (EncodingKey, serde_json::Value) { + let certified = rcgen::generate_simple_self_signed(vec![format!("{kid}.invalid")]).expect("OIDC signing key should generate"); + let encoding_key = + EncodingKey::from_ec_pem(certified.signing_key.serialize_pem().as_bytes()).expect("OIDC signing key should encode"); + let mut jwk = jsonwebtoken::jwk::Jwk::from_encoding_key(&encoding_key, Algorithm::ES256) + .expect("OIDC public JWK should derive from signing key"); + jwk.common.key_id = Some(kid.to_string()); + jwk.common.public_key_use = Some(jsonwebtoken::jwk::PublicKeyUse::Signature); + (encoding_key, serde_json::to_value(jwk).expect("OIDC JWK should serialize")) +} +#[cfg(test)] +pub(super) fn start_mock_oidc_tls_discovery_server( + build_discovery_issuer: F, + max_requests: usize, +) -> Option<(String, String, std::thread::JoinHandle<()>)> +where + F: Fn(&str) -> (String, String, String) + Send + 'static, +{ + use std::io::Write; + use std::net::{Shutdown, TcpListener}; + use std::sync::mpsc; + use std::time::{Duration, Instant}; + + const IDLE_SHUTDOWN: Duration = Duration::from_secs(1); + const ABSOLUTE_CAP: Duration = Duration::from_secs(5); + + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); + let certified = + rcgen::generate_simple_self_signed(vec!["127.0.0.1".to_string()]).expect("generate OIDC TLS test certificate"); + let cert_pem = certified.cert.pem(); + let server_config = rustls::ServerConfig::builder() + .with_no_client_auth() + .with_single_cert( + vec![certified.cert.der().clone()], + rustls_pki_types::PrivateKeyDer::try_from(certified.signing_key.serialize_der()) + .expect("convert OIDC TLS test private key"), + ) + .expect("build OIDC TLS mock server config"); + + let listener = match TcpListener::bind("127.0.0.1:0") { + Ok(listener) => listener, + Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return None, + Err(err) => panic!("test TLS listener should bind: {err}"), + }; + let base = format!("https://{}", listener.local_addr().expect("listener local address should be available")); + let (discovery_issuer, discovery_jwks_uri, expected_jwks_path) = build_discovery_issuer(&base); + let discovery_body = serde_json::json!({ + "issuer": discovery_issuer, + "authorization_endpoint": format!("{base}/authorize"), + "token_endpoint": format!("{base}/token"), + "jwks_uri": discovery_jwks_uri, + "response_types_supported": ["code"], + "response_modes_supported": ["query"], + "subject_types_supported": ["public"], + "id_token_signing_alg_values_supported": ["RS256"], + }) + .to_string(); + let jwks_body = r#"{"keys":[]}"#; + let (ready_tx, ready_rx) = mpsc::channel(); + + let handle = std::thread::spawn(move || { + let server_config = Arc::new(server_config); + listener + .set_nonblocking(true) + .expect("failed to set TLS discovery mock listener non-blocking"); + let _ = ready_tx.send(()); + + let mut seen = 0usize; + let start = Instant::now(); + let mut last_completed = Instant::now(); + + loop { + if seen > 0 && last_completed.elapsed() >= IDLE_SHUTDOWN { + break; + } + if start.elapsed() >= ABSOLUTE_CAP { + break; + } + + let tcp_stream = match listener.accept() { + Ok((stream, _)) => stream, + Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => { + std::thread::sleep(Duration::from_millis(5)); + continue; + } + Err(_) => break, + }; + tcp_stream + .set_nonblocking(false) + .expect("failed to set TLS discovery mock stream blocking"); + tcp_stream + .set_read_timeout(Some(Duration::from_secs(1))) + .expect("failed to set TLS discovery mock read timeout"); + + seen += 1; + let connection = match rustls::ServerConnection::new(server_config.clone()) { + Ok(connection) => connection, + Err(_) => break, + }; + let mut stream = rustls::StreamOwned::new(connection, tcp_stream); + let path = read_mock_oidc_request_path(&mut stream); + let response = mock_oidc_response(&path, &discovery_body, &expected_jwks_path, jwks_body); + let _ = stream.write_all(response.as_bytes()); + let _ = stream.flush(); + let _ = stream.sock.shutdown(Shutdown::Both); + last_completed = Instant::now(); + + if seen >= max_requests { + break; + } + } + }); + ready_rx + .recv_timeout(Duration::from_millis(100)) + .expect("mock TLS OIDC discovery server should become ready"); + + Some((base, cert_pem, handle)) +} +#[cfg(test)] +pub(super) fn discovery_error_contains_all_variants(err: &str, base: &str) -> bool { + err.contains(base) && err.contains(&format!("{base}/")) && err.contains("discovery failed for all issuer variants") +} +#[cfg(test)] +pub(super) async fn validate_mocked_oidc_provider_config( + config: &OidcProviderConfig, +) -> Result { + // The mock discovery/JWKS/token endpoints share the loopback origin of `config_url`. + // Explicitly allow that origin so the egress policy does not reject the loopback mock. + let origin = Url::parse(&config.config_url) + .map_err(|_| "invalid mock config_url".to_string())? + .origin() + .ascii_serialization(); + let policy = OutboundPolicy::from_allowed_origins(&origin).map_err(|err| err.to_string())?; + let http_client = ReqwestHttpClient::with_policy(policy); + let state = discover_provider(config, &http_client).await?; + + Ok(OidcProviderValidationResult { + issuer: state.metadata.issuer().to_string(), + authorization_endpoint: state.metadata.authorization_endpoint(), + token_endpoint: state.metadata.token_endpoint().map(ToString::to_string), + }) +} +#[cfg(test)] +pub(super) fn start_unbounded_body_server(body_len: usize) -> Option<(String, std::thread::JoinHandle<()>)> { + use std::io::{Read, Write}; + use std::net::TcpListener; + use std::sync::mpsc; + use std::time::Duration; + + let listener = match TcpListener::bind("127.0.0.1:0") { + Ok(listener) => listener, + Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return None, + Err(err) => panic!("test listener should bind: {err}"), + }; + let base = format!("http://{}", listener.local_addr().expect("listener local address should be available")); + let (ready_tx, ready_rx) = mpsc::channel(); + + let handle = std::thread::spawn(move || { + let _ = ready_tx.send(()); + let Ok((mut stream, _)) = listener.accept() else { + return; + }; + let _ = stream.set_read_timeout(Some(Duration::from_secs(1))); + let mut buffer = [0u8; 4096]; + let _ = stream.read(&mut buffer); + let _ = stream.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nConnection: close\r\n\r\n"); + + let chunk = vec![b'a'; 64 * 1024]; + let mut written = 0usize; + while written < body_len { + let take = chunk.len().min(body_len - written); + if stream.write_all(&chunk[..take]).is_err() { + break; + } + written += take; + } + let _ = stream.flush(); + }); + ready_rx + .recv_timeout(Duration::from_millis(100)) + .expect("mock body server should become ready"); + + Some((base, handle)) +} +#[cfg(test)] +pub(super) async fn fetch_oidc_mock_body(base: &str) -> Result, OidcHttpError> { + let policy = OutboundPolicy::from_allowed_origins(base).expect("origin should parse"); + let client = ReqwestHttpClient::with_policy(policy); + let request = http::Request::builder() + .method(http::Method::GET) + .uri(base) + .body(Vec::new()) + .expect("request should build"); + client.call(request).await.map(http::Response::into_body) +} diff --git a/crates/iam/src/oidc/transport.rs b/crates/iam/src/oidc/transport.rs new file mode 100644 index 000000000..18a708a4d --- /dev/null +++ b/crates/iam/src/oidc/transport.rs @@ -0,0 +1,669 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +use crate::{LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC}; +use openidconnect::AsyncHttpClient; +use reqwest::{Certificate, Client}; +use rustfs_config::MAX_OIDC_RESPONSE_SIZE; +use rustfs_utils::egress::{ENV_OUTBOUND_ALLOW_ORIGINS, OutboundPolicy, find_outbound_dns_policy_rejection}; +use std::collections::VecDeque; +use std::future::Future; +use std::net::IpAddr; +use std::pin::Pin; +use std::sync::{Arc, LazyLock, Mutex, MutexGuard, RwLock}; +use std::time::{Duration as StdDuration, Instant}; +use tracing::{debug, error, warn}; +use url::Url; + +pub(super) const EVENT_OIDC_HTTP: &str = "oidc_http"; +const OIDC_HTTP_REQUEST_TIMEOUT: StdDuration = StdDuration::from_secs(10); +const OIDC_HTTP_CONNECT_TIMEOUT: StdDuration = StdDuration::from_secs(3); +const OIDC_PLUGIN_AUTHN_WINDOW: StdDuration = StdDuration::from_secs(60); + +#[derive(Debug, Clone, Copy, Default)] +pub struct OidcPluginAuthnMetricsSnapshot { + pub failed_requests_minute: u64, + pub last_fail_seconds: u64, + pub last_succ_seconds: u64, + pub succ_avg_rtt_ms_minute: u64, + pub succ_max_rtt_ms_minute: u64, + pub total_requests_minute: u64, +} + +#[derive(Debug, Clone)] +struct OidcPluginAuthnSample { + observed_at: Instant, + succeeded: bool, + rtt_ms: u64, +} + +#[derive(Debug, Default)] +struct OidcPluginAuthnMetrics { + samples: Mutex>, + last_fail_at: Mutex>, + last_succ_at: Mutex>, +} + +fn lock_oidc_plugin_authn_metrics<'a, T>(mutex: &'a Mutex, metric: &'static str) -> MutexGuard<'a, T> { + match mutex.lock() { + Ok(guard) => guard, + Err(err) => { + warn!(metric, "Recovering poisoned OIDC authn metrics lock"); + err.into_inner() + } + } +} + +fn seconds_since(now: Instant, observed_at: Option) -> u64 { + observed_at + .map(|instant| now.duration_since(instant).as_secs()) + .unwrap_or_default() +} + +impl OidcPluginAuthnMetrics { + fn record(&self, rtt_ms: u64, succeeded: bool) { + let now = Instant::now(); + let mut samples = lock_oidc_plugin_authn_metrics(&self.samples, "samples"); + samples.push_back(OidcPluginAuthnSample { + observed_at: now, + succeeded, + rtt_ms, + }); + while samples + .front() + .is_some_and(|sample| now.duration_since(sample.observed_at) > OIDC_PLUGIN_AUTHN_WINDOW) + { + samples.pop_front(); + } + drop(samples); + + if succeeded { + *lock_oidc_plugin_authn_metrics(&self.last_succ_at, "last_succ_at") = Some(now); + } else { + *lock_oidc_plugin_authn_metrics(&self.last_fail_at, "last_fail_at") = Some(now); + } + } + + fn snapshot(&self) -> OidcPluginAuthnMetricsSnapshot { + let now = Instant::now(); + let (total_requests_minute, failed_requests_minute, succ_avg_rtt_ms_minute, succ_max_rtt_ms_minute) = { + let mut samples = lock_oidc_plugin_authn_metrics(&self.samples, "samples"); + while samples + .front() + .is_some_and(|sample| now.duration_since(sample.observed_at) > OIDC_PLUGIN_AUTHN_WINDOW) + { + samples.pop_front(); + } + + let mut failed_requests_minute = 0u64; + let mut successful_requests = 0u64; + let mut successful_rtt_sum = 0u64; + let mut succ_max_rtt_ms_minute = 0u64; + + for sample in samples.iter() { + if sample.succeeded { + successful_requests += 1; + successful_rtt_sum += sample.rtt_ms; + succ_max_rtt_ms_minute = succ_max_rtt_ms_minute.max(sample.rtt_ms); + } else { + failed_requests_minute += 1; + } + } + + let succ_avg_rtt_ms_minute = successful_rtt_sum.checked_div(successful_requests).unwrap_or_default(); + + ( + samples.len() as u64, + failed_requests_minute, + succ_avg_rtt_ms_minute, + succ_max_rtt_ms_minute, + ) + }; + + let last_fail_seconds = seconds_since(now, *lock_oidc_plugin_authn_metrics(&self.last_fail_at, "last_fail_at")); + let last_succ_seconds = seconds_since(now, *lock_oidc_plugin_authn_metrics(&self.last_succ_at, "last_succ_at")); + + OidcPluginAuthnMetricsSnapshot { + failed_requests_minute, + last_fail_seconds, + last_succ_seconds, + succ_avg_rtt_ms_minute, + succ_max_rtt_ms_minute, + total_requests_minute, + } + } +} + +static OIDC_PLUGIN_AUTHN_METRICS: LazyLock = LazyLock::new(OidcPluginAuthnMetrics::default); + +pub fn oidc_plugin_authn_metrics_snapshot() -> OidcPluginAuthnMetricsSnapshot { + OIDC_PLUGIN_AUTHN_METRICS.snapshot() +} + +/// Header names whose values may carry OIDC secrets (client credentials, cookies, +/// bearer tokens). Their values are never emitted to logs, only their byte length. +const SENSITIVE_HEADER_NAMES: [&str; 4] = ["authorization", "proxy-authorization", "cookie", "set-cookie"]; + +pub(super) fn is_sensitive_header(name: &str) -> bool { + SENSITIVE_HEADER_NAMES + .iter() + .any(|candidate| name.eq_ignore_ascii_case(candidate)) +} + +pub(super) fn format_http_headers(headers: &http::HeaderMap) -> String { + headers + .iter() + .map(|(name, value)| { + if is_sensitive_header(name.as_str()) { + format!("{}=", name.as_str(), value.as_bytes().len()) + } else { + let value = value.to_str().unwrap_or(""); + format!("{}={}", name.as_str(), value) + } + }) + .collect::>() + .join("; ") +} + +#[derive(Debug, Default)] +pub(super) struct TokenResponseBodyShape { + pub(super) json_object: bool, + pub(super) json_keys: String, + pub(super) has_access_token: bool, + pub(super) has_id_token: bool, + pub(super) has_token_type: bool, + pub(super) has_expires_in: bool, + pub(super) has_error: bool, + pub(super) has_error_description: bool, + pub(super) looks_like_html: bool, +} + +pub(super) fn inspect_token_response_body(body: &[u8]) -> TokenResponseBodyShape { + let mut shape = TokenResponseBodyShape { + looks_like_html: body + .iter() + .copied() + .find(|byte| !byte.is_ascii_whitespace()) + .is_some_and(|byte| byte == b'<'), + ..Default::default() + }; + + let Ok(value) = serde_json::from_slice::(body) else { + return shape; + }; + let Some(object) = value.as_object() else { + return shape; + }; + + shape.json_object = true; + shape.has_access_token = object.contains_key("access_token"); + shape.has_id_token = object.contains_key("id_token"); + shape.has_token_type = object.contains_key("token_type"); + shape.has_expires_in = object.contains_key("expires_in"); + shape.has_error = object.contains_key("error"); + shape.has_error_description = object.contains_key("error_description"); + + let mut keys: Vec<&str> = object.keys().map(String::as_str).collect(); + keys.sort_unstable(); + keys.truncate(16); + shape.json_keys = keys.join(","); + + shape +} + +pub(super) fn oidc_http_error_diagnostics(error: &OidcHttpError) -> (&'static str, String) { + match error { + OidcHttpError::Reqwest(err) if err.is_timeout() => ("timeout", String::new()), + OidcHttpError::Reqwest(err) if err.is_connect() => ("connect", String::new()), + OidcHttpError::Reqwest(err) if err.status().is_some() => { + ("http_status", err.status().map(|status| status.as_u16().to_string()).unwrap_or_default()) + } + OidcHttpError::Reqwest(_) => ("request", String::new()), + OidcHttpError::Http(_) => ("http_build", String::new()), + OidcHttpError::ExtraRootCa(_) => ("extra_root_ca", String::new()), + OidcHttpError::ForbiddenOutbound(_) => ("forbidden_outbound", String::new()), + OidcHttpError::ResponseTooLarge(limit) => ("response_too_large", limit.to_string()), + } +} + +// ---- HTTP Client Adapter ---- + +/// Error type for the OIDC HTTP client adapter. +#[derive(Debug)] +pub(super) enum OidcHttpError { + Reqwest(reqwest::Error), + Http(http::Error), + ExtraRootCa(String), + /// The outbound destination was rejected by the shared egress policy before any + /// connection was attempted (invalid URL, loopback/link-local/metadata/private IP, + /// or a malformed allow-origins configuration). + ForbiddenOutbound(String), + /// The provider response body exceeded [`MAX_OIDC_RESPONSE_SIZE`] and was abandoned + /// instead of being buffered in full. + ResponseTooLarge(usize), +} + +impl std::fmt::Display for OidcHttpError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Reqwest(e) => write!(f, "{e}"), + Self::Http(e) => write!(f, "{e}"), + Self::ExtraRootCa(reason) => write!(f, "failed to load OIDC extra root CA bundle: {reason}"), + Self::ForbiddenOutbound(reason) => write!(f, "outbound request rejected: {reason}"), + Self::ResponseTooLarge(limit) => write!(f, "oidc response body exceeds {limit} bytes"), + } + } +} + +impl std::error::Error for OidcHttpError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Reqwest(e) => Some(e), + Self::Http(e) => Some(e), + Self::ExtraRootCa(_) | Self::ForbiddenOutbound(_) | Self::ResponseTooLarge(_) => None, + } + } +} + +#[derive(Clone, Debug, Default)] +pub struct OidcExtraRootCaMaterial { + pub generation: u64, + pub root_ca_pem: Option>, +} + +type OidcExtraRootCaFuture = Pin> + Send>>; +type OidcExtraRootCaLoader = dyn Fn() -> OidcExtraRootCaFuture + Send + Sync; + +#[derive(Clone)] +pub struct OidcExtraRootCaProvider { + loader: Arc, +} + +impl OidcExtraRootCaProvider { + pub fn new(loader: F) -> Self + where + F: Fn() -> Fut + Send + Sync + 'static, + Fut: Future> + Send + 'static, + { + Self { + loader: Arc::new(move || Box::pin(loader())), + } + } + + async fn load(&self) -> Result { + (self.loader)().await + } +} + +#[derive(Clone, Default)] +struct CachedOidcExtraRootCerts { + generation: u64, + initialized: bool, + certs: Vec, +} + +/// Discovery callers report failures with provider context and recovery throttling. +#[derive(Clone)] +pub(super) struct DiscoveryRequest; + +/// HTTP client adapter bridging reqwest 0.13 to the `openidconnect` `AsyncHttpClient` trait. +/// +/// A fresh client is built for every request so the destination is re-validated and the +/// resolved IP is re-classified at connection time. This closes the SSRF / DNS-rebinding +/// gap where a one-shot URL string check is bypassed by a hostname that resolves to an +/// internal address only at connection time, and it also covers endpoints discovered from +/// the provider metadata (JWKS, token) rather than only the operator-configured `config_url`. +pub(super) struct ReqwestHttpClient { + /// `None` in production: the process-cached outbound policy from the environment is used. + /// `Some(..)` only in tests, to explicitly allow a loopback mock endpoint. + policy_override: Option, + extra_root_certs: Arc>, + extra_root_ca_provider: Option, + #[cfg(test)] + dns_resolver_override: Option>, +} + +pub(super) fn parse_oidc_extra_root_certs(source: &str, pem: &[u8]) -> Result, String> { + if pem.iter().all(|byte| byte.is_ascii_whitespace()) { + return Ok(Vec::new()); + } + Certificate::from_pem_bundle(pem).map_err(|err| format!("failed to parse OIDC extra root CA bundle from {source}: {err}")) +} + +pub(super) fn oidc_extra_root_certs(root_ca_pem: Option<&[u8]>) -> Result, String> { + match root_ca_pem { + Some(pem) => parse_oidc_extra_root_certs("RustFS outbound TLS material", pem), + None => Ok(Vec::new()), + } +} + +/// Build a reqwest client pinned to the shared outbound egress policy for a single request. +/// +/// [`OutboundPolicy::resolver_for`] validates the URL shape and rejects loopback, +/// link-local, metadata, multicast and unauthorized private addresses up front, and the +/// returned `OutboundDnsResolver` re-resolves and re-classifies the host on every new +/// connection so DNS rebinding fails closed. Redirects are not followed: a redirect target +/// would otherwise skip URL-shape re-validation. The timeouts bound how long a slow or +/// stalled provider can pin the calling task. +pub(super) fn build_oidc_http_client( + uri: &str, + policy_override: Option<&OutboundPolicy>, + extra_root_certs: &[Certificate], + #[cfg(test)] dns_resolver_override: Option>, +) -> Result<(Client, Url), OidcHttpError> { + let url = Url::parse(uri).map_err(|_| OidcHttpError::ForbiddenOutbound("invalid outbound OIDC URL".to_string()))?; + let resolver = match policy_override { + Some(policy) => policy.resolver_for(&url), + None => OutboundPolicy::from_env_cached() + .map_err(|err| OidcHttpError::ForbiddenOutbound(err.to_string()))? + .resolver_for(&url), + } + .map_err(|err| { + let base = err.to_string(); + let origin = url.origin().ascii_serialization(); + let can_allow_origin = + OutboundPolicy::from_allowed_origins(&origin).is_ok_and(|allowlisted| allowlisted.validate_url(&url).is_ok()); + oidc_forbidden_outbound_error(&url, base, can_allow_origin) + })?; + let bypass_proxy = should_bypass_proxy_for_oidc_uri(uri); + #[cfg(test)] + let bypass_proxy = bypass_proxy || dns_resolver_override.is_some(); + #[cfg(test)] + let resolver: Arc = dns_resolver_override.unwrap_or_else(|| Arc::new(resolver)); + + let mut builder = reqwest::Client::builder() + .dns_resolver(resolver) + .redirect(reqwest::redirect::Policy::none()) + .timeout(OIDC_HTTP_REQUEST_TIMEOUT) + .connect_timeout(OIDC_HTTP_CONNECT_TIMEOUT); + if bypass_proxy { + builder = builder.no_proxy(); + } + if !extra_root_certs.is_empty() { + builder = builder.tls_certs_merge(extra_root_certs.iter().cloned()); + } + builder.build().map(|client| (client, url)).map_err(OidcHttpError::Reqwest) +} + +fn oidc_forbidden_outbound_error(url: &Url, base: String, can_allow_origin: bool) -> OidcHttpError { + let reason = if can_allow_origin { + let origin = url.origin().ascii_serialization(); + format!( + "{base}; add {origin} to {ENV_OUTBOUND_ALLOW_ORIGINS} (comma-separated) and restart RustFS to allow this operator-owned OIDC provider (origin only, no path)" + ) + } else { + base + }; + OidcHttpError::ForbiddenOutbound(reason) +} + +fn oidc_http_error_from_reqwest(url: &Url, error: reqwest::Error) -> OidcHttpError { + if let Some(rejection) = find_outbound_dns_policy_rejection(&error) { + let base = rejection.to_string(); + return oidc_forbidden_outbound_error(url, base, rejection.allow_origin_can_recover()); + } + OidcHttpError::Reqwest(error) +} + +/// Buffer a provider response body, failing closed once `limit` bytes have been seen. +/// +/// `Response::bytes` would buffer the whole body unconditionally, so a hostile or compromised +/// provider endpoint could stream an arbitrarily large (or endless) body into memory. +async fn read_bounded_response_body(response: reqwest::Response, limit: usize) -> Result, OidcHttpError> { + if response.content_length().is_some_and(|len| len > limit as u64) { + return Err(OidcHttpError::ResponseTooLarge(limit)); + } + + let mut response = response; + let mut body = Vec::new(); + while let Some(chunk) = response.chunk().await.map_err(OidcHttpError::Reqwest)? { + if body.len() + chunk.len() > limit { + return Err(OidcHttpError::ResponseTooLarge(limit)); + } + body.extend_from_slice(&chunk); + } + Ok(body) +} + +pub(super) fn should_bypass_proxy_for_oidc_uri(uri: &str) -> bool { + let Some(host) = Url::parse(uri).ok().and_then(|url| url.host_str().map(str::to_owned)) else { + return false; + }; + let host = host.trim_matches(['[', ']']); + + host.eq_ignore_ascii_case("localhost") || host.parse::().is_ok_and(|addr| addr.is_loopback()) +} + +impl ReqwestHttpClient { + pub(super) fn new() -> Result { + Self::new_with_extra_root_certs(Vec::new()) + } + + fn extra_root_cert_cache(certs: Vec) -> Arc> { + Arc::new(RwLock::new(CachedOidcExtraRootCerts { + generation: 0, + initialized: true, + certs, + })) + } + + pub(super) fn new_with_extra_root_certs(extra_root_certs: Vec) -> Result { + Ok(Self { + policy_override: None, + extra_root_certs: Self::extra_root_cert_cache(extra_root_certs), + extra_root_ca_provider: None, + #[cfg(test)] + dns_resolver_override: None, + }) + } + + pub(super) fn new_with_extra_root_ca_provider(extra_root_ca_provider: OidcExtraRootCaProvider) -> Result { + Ok(Self { + policy_override: None, + extra_root_certs: Arc::new(RwLock::new(CachedOidcExtraRootCerts::default())), + extra_root_ca_provider: Some(extra_root_ca_provider), + #[cfg(test)] + dns_resolver_override: None, + }) + } + + pub(super) async fn current_extra_root_certs(&self) -> Result, OidcHttpError> { + let Some(provider) = self.extra_root_ca_provider.as_ref() else { + return self + .extra_root_certs + .read() + .map(|cache| cache.certs.clone()) + .map_err(|e| OidcHttpError::ExtraRootCa(format!("extra root certificate cache lock poisoned: {e}"))); + }; + + let material = provider.load().await.map_err(OidcHttpError::ExtraRootCa)?; + if let Ok(cache) = self.extra_root_certs.read() + && cache.initialized + && cache.generation == material.generation + { + return Ok(cache.certs.clone()); + } + + let certs = oidc_extra_root_certs(material.root_ca_pem.as_deref()).map_err(OidcHttpError::ExtraRootCa)?; + let mut cache = self + .extra_root_certs + .write() + .map_err(|e| OidcHttpError::ExtraRootCa(format!("extra root certificate cache lock poisoned: {e}")))?; + cache.generation = material.generation; + cache.initialized = true; + cache.certs = certs.clone(); + Ok(certs) + } + + /// Test-only constructor that pins outbound requests to an explicit policy, so a + /// loopback mock server can be reached without depending on process-wide environment. + #[cfg(test)] + pub(super) fn with_policy(policy: OutboundPolicy) -> Self { + Self { + policy_override: Some(policy), + extra_root_certs: Self::extra_root_cert_cache(Vec::new()), + extra_root_ca_provider: None, + dns_resolver_override: None, + } + } + + #[cfg(test)] + pub(super) fn with_policy_and_extra_root_certs(policy: OutboundPolicy, extra_root_certs: Vec) -> Self { + Self { + policy_override: Some(policy), + extra_root_certs: Self::extra_root_cert_cache(extra_root_certs), + extra_root_ca_provider: None, + dns_resolver_override: None, + } + } + + #[cfg(test)] + pub(super) fn with_policy_and_extra_root_ca_provider( + policy: OutboundPolicy, + extra_root_ca_provider: OidcExtraRootCaProvider, + ) -> Self { + Self { + policy_override: Some(policy), + extra_root_certs: Arc::new(RwLock::new(CachedOidcExtraRootCerts::default())), + extra_root_ca_provider: Some(extra_root_ca_provider), + dns_resolver_override: None, + } + } + + #[cfg(test)] + pub(super) fn with_policy_and_dns_resolver(policy: OutboundPolicy, resolver: Arc) -> Self { + Self { + policy_override: Some(policy), + extra_root_certs: Self::extra_root_cert_cache(Vec::new()), + extra_root_ca_provider: None, + dns_resolver_override: Some(resolver), + } + } +} + +impl<'c> AsyncHttpClient<'c> for ReqwestHttpClient { + type Error = OidcHttpError; + type Future = Pin>, Self::Error>> + Send + 'c>>; + + fn call(&'c self, request: http::Request>) -> Self::Future { + Box::pin(async move { + let started_at = Instant::now(); + let (parts, body) = request.into_parts(); + let discovery_request = parts.extensions.get::().is_some(); + let method = parts.method.clone(); + let uri = parts.uri.to_string(); + if tracing::enabled!(tracing::Level::DEBUG) { + let request_headers = format_http_headers(&parts.headers); + debug!( + event = EVENT_OIDC_HTTP, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "request", + method = %method, + uri = %uri, + request_headers = %request_headers, + request_body_len = body.len(), + "oidc outbound http" + ); + } + + let extra_root_certs = self.current_extra_root_certs().await?; + let (client, url) = build_oidc_http_client( + &uri, + self.policy_override.as_ref(), + &extra_root_certs, + #[cfg(test)] + self.dns_resolver_override.clone(), + )?; + let response = client + .request(parts.method, uri.clone()) + .headers(parts.headers) + .body(body) + .send() + .await; + + let elapsed_ms = started_at.elapsed().as_millis().min(u128::from(u64::MAX)) as u64; + let succeeded = response.as_ref().is_ok_and(|resp| resp.status().is_success()); + OIDC_PLUGIN_AUTHN_METRICS.record(elapsed_ms, succeeded); + + let response = response.map_err(|err| { + let error = oidc_http_error_from_reqwest(&url, err); + if !discovery_request { + error!( + event = EVENT_OIDC_HTTP, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "request_failed", + method = %method, + uri = %uri, + elapsed_ms, + error = %error, + "oidc outbound http" + ); + } + error + })?; + + let status = response.status(); + let headers = response.headers().clone(); + let body_bytes = read_bounded_response_body(response, MAX_OIDC_RESPONSE_SIZE) + .await + .map_err(|err| { + if !discovery_request { + error!( + event = EVENT_OIDC_HTTP, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "response_body_failed", + method = %method, + uri = %uri, + status = status.as_u16(), + elapsed_ms, + error = %err, + "oidc outbound http" + ); + } + err + })?; + if tracing::enabled!(tracing::Level::DEBUG) { + let response_headers = format_http_headers(&headers); + debug!( + event = EVENT_OIDC_HTTP, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + result = "response", + method = %method, + uri = %uri, + status = status.as_u16(), + status_success = status.is_success(), + elapsed_ms, + response_headers = %response_headers, + response_body_len = body_bytes.len(), + "oidc outbound http" + ); + } + + let mut http_response = http::Response::builder() + .status(status) + .body(body_bytes) + .map_err(OidcHttpError::Http)?; + *http_response.headers_mut() = headers; + + Ok(http_response) + }) + } +} + +#[cfg(test)] +#[path = "transport_tests.rs"] +mod tests; diff --git a/crates/iam/src/oidc/transport_tests.rs b/crates/iam/src/oidc/transport_tests.rs new file mode 100644 index 000000000..e77fe561a --- /dev/null +++ b/crates/iam/src/oidc/transport_tests.rs @@ -0,0 +1,167 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use crate::oidc::test_support::*; +use openidconnect::AsyncHttpClient; +use rustfs_config::MAX_OIDC_RESPONSE_SIZE; +use rustfs_utils::egress::{ENV_OUTBOUND_ALLOW_ORIGINS, OutboundPolicy}; +use std::sync::Arc; + +#[test] +fn format_http_headers_redacts_sensitive_values() { + let mut headers = http::HeaderMap::new(); + headers.insert(http::header::AUTHORIZATION, "Basic Y2xpZW50OnNlY3JldA==".parse().unwrap()); + headers.insert(http::header::CONTENT_TYPE, "application/json".parse().unwrap()); + headers.insert(http::header::COOKIE, "session=super-secret".parse().unwrap()); + + let rendered = format_http_headers(&headers); + + // Sensitive header values never appear; only their length is emitted. + assert!(!rendered.contains("Y2xpZW50OnNlY3JldA=="), "authorization value leaked: {rendered}"); + assert!(!rendered.contains("super-secret"), "cookie value leaked: {rendered}"); + assert!( + rendered.contains("authorization=> = None; + let with_header = HashMap::from([("if-none-match".to_string(), vec!["*".to_string()])]); + let without_header = HashMap::new(); + + let mut outcomes = Vec::new(); + for conditions in [&with_header, &without_header] { + let args = Args { + account: "sts-conditional-write-user", + groups: &groups, + action: Action::S3Action(S3Action::PutObjectAction), + bucket: "bucket", + conditions, + is_owner: false, + object: "obj", + claims: &claims, + deny_only: false, + }; + let prepared = prepare_session_policy(&args, false); + assert!(matches!(prepared, PreparedSessionPolicy::Policy(_))); + outcomes.push(evaluate_prepared_session_policy(&prepared, &args).await); + } + + assert_eq!(outcomes, [Some(true), Some(false)]); + } + #[tokio::test] async fn test_prepare_auth_service_account_inherited_ignores_session_policy_tag_hint() { let store = StsTestMockStore::new(false); diff --git a/crates/io-metrics/Cargo.toml b/crates/io-metrics/Cargo.toml index 0591aa4e7..5cfd022a5 100644 --- a/crates/io-metrics/Cargo.toml +++ b/crates/io-metrics/Cargo.toml @@ -30,6 +30,7 @@ harness = false [features] default = [] +diagnostic-read-version-caller = [] hotpath = [ "hotpath/hotpath", "hotpath/tokio", diff --git a/crates/io-metrics/benches/metrics_pipeline.rs b/crates/io-metrics/benches/metrics_pipeline.rs index fb1aa6857..3f213e586 100644 --- a/crates/io-metrics/benches/metrics_pipeline.rs +++ b/crates/io-metrics/benches/metrics_pipeline.rs @@ -1,13 +1,15 @@ use criterion::{Criterion, criterion_group, criterion_main}; -use rustfs_io_metrics::{MetricsCollector, PerformanceMetrics, record_get_object_request_started}; +use rustfs_io_metrics::{MetricsCollector, PerformanceMetrics, record_get_object_timeout_operation_start}; use rustfs_io_metrics::{record_s3_op, s3_http_metrics::S3HttpRequestGuard}; use rustfs_s3_ops::S3Operation; use std::hint::black_box; use std::sync::Arc; use std::time::Duration; -fn bench_record_get_object_request_started(c: &mut Criterion) { - c.bench_function("record_get_object_request_started", |b| b.iter(record_get_object_request_started)); +fn bench_record_get_object_timeout_operation_start(c: &mut Criterion) { + c.bench_function("record_get_object_timeout_operation_start", |b| { + b.iter(record_get_object_timeout_operation_start) + }); } fn bench_s3_http_outcomes(c: &mut Criterion) { @@ -49,7 +51,7 @@ fn bench_metrics_collector_record_io_operation(c: &mut Criterion) { criterion_group!( benches, - bench_record_get_object_request_started, + bench_record_get_object_timeout_operation_start, bench_s3_http_outcomes, bench_update_concurrent_requests, bench_metrics_collector_record_io_operation diff --git a/crates/io-metrics/src/internode_metrics.rs b/crates/io-metrics/src/internode_metrics.rs index 2bc9b1ac1..d677cba5a 100644 --- a/crates/io-metrics/src/internode_metrics.rs +++ b/crates/io-metrics/src/internode_metrics.rs @@ -12,7 +12,10 @@ // See the License for the specific language governing permissions and // limitations under the License. +use crate::s3_http_metrics::S3HttpOperationContext; use metrics::{counter, gauge}; +#[cfg(not(test))] +use rustfs_s3_ops::S3Operation; use std::collections::HashMap; use std::sync::{ Arc, LazyLock, OnceLock, RwLock, @@ -65,6 +68,340 @@ pub const INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_DECODE: &str = "batch_read pub const INTERNODE_STAGE_BATCH_READ_VERSION_COALESCER_WAIT: &str = "batch_read_version_coalescer_wait"; pub const INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_MAP: &str = "batch_read_version_response_map"; +const INTERNODE_READ_VERSION_REQUESTS_BY_CALLER_TOTAL: &str = + "rustfs_system_network_internode_read_version_requests_by_caller_total"; + +/// An attribution dimension is absent, common to all items, or heterogeneous. +#[cfg(feature = "diagnostic-read-version-caller")] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum ReadVersionAttribution { + Absent, + Common(T), + Mixed, +} + +/// Storage entry names describe call paths, not proof of an external request. +#[cfg(feature = "diagnostic-read-version-caller")] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum ReadVersionEntrySource { + Unclassified, + ConfigRead, + ConfigWrite, + ScannerConfigWrite, + WritePreconditionRead, + ObjectInfoRead, + AppPutOwner, + PutCommitOwner, + LifecycleRecovery, + RecoveryDispositionMaintenance, + StoreObjectReader, + DataUsageCacheRead, + ServerInfoPools, + ScannerBacklogRead, + ScannerRevisionRead, + ScannerCycleStateRead, + TierConfigReload, + ObjectRead, + ListObjects, + MetadataRead, +} + +#[cfg(feature = "diagnostic-read-version-caller")] +impl ReadVersionEntrySource { + pub const fn as_str(self) -> &'static str { + match self { + Self::Unclassified => "unclassified", + Self::ConfigRead => "config_read", + Self::ConfigWrite => "config_write", + Self::ScannerConfigWrite => "scanner_config_write", + Self::WritePreconditionRead => "write_precondition_read", + Self::ObjectInfoRead => "object_info_read", + Self::AppPutOwner => "app_put_owner", + Self::PutCommitOwner => "put_commit_owner", + Self::LifecycleRecovery => "lifecycle_recovery", + Self::RecoveryDispositionMaintenance => "recovery_disposition_maintenance", + Self::StoreObjectReader => "store_object_reader", + Self::DataUsageCacheRead => "data_usage_cache_read", + Self::ServerInfoPools => "server_info_pools", + Self::ScannerBacklogRead => "scanner_backlog_read", + Self::ScannerRevisionRead => "scanner_revision_read", + Self::ScannerCycleStateRead => "scanner_cycle_state_read", + Self::TierConfigReload => "tier_config_reload", + Self::ObjectRead => "object_read", + Self::ListObjects => "list_objects", + Self::MetadataRead => "metadata_read", + } + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +#[derive(Clone, Copy, Debug)] +pub enum ReadVersionContextBoundary { + Entry, + FullWait, + EarlyStop, + ListProducer, + ListGather, + CoalescerEnqueue, + UnaryFallback, + EntryToRpc, + RemoteRpc, + ScannerCacheLoad, + ScannerCycleStateLoad, + ConfigWrite, + ScannerConfigWrite, + WritePreconditionLookup, + ObjectInfoRead, + AppPutOwner, + PutCommitOwner, + TierFreeVersionSourceLookup, +} + +#[cfg(feature = "diagnostic-read-version-caller")] +impl ReadVersionContextBoundary { + const fn as_str(self) -> &'static str { + match self { + Self::Entry => "entry", + Self::FullWait => "full_wait", + Self::EarlyStop => "early_stop", + Self::ListProducer => "list_producer", + Self::ListGather => "list_gather", + Self::CoalescerEnqueue => "coalescer_enqueue", + Self::UnaryFallback => "unary_fallback", + Self::EntryToRpc => "entry_to_rpc", + Self::RemoteRpc => "remote_rpc", + Self::ScannerCacheLoad => "scanner_cache_load", + Self::ScannerCycleStateLoad => "scanner_cycle_state_load", + Self::ConfigWrite => "config_write", + Self::ScannerConfigWrite => "scanner_config_write", + Self::WritePreconditionLookup => "write_precondition_lookup", + Self::ObjectInfoRead => "object_info_read", + Self::AppPutOwner => "app_put_owner", + Self::PutCommitOwner => "put_commit_owner", + Self::TierFreeVersionSourceLookup => "tier_free_version_source_lookup", + } + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn context_transition(parent: Option, child: Option) -> &'static str { + match (parent, child) { + (None, None) => "absent", + (Some(_), None) => "lost", + (None, Some(_)) => "gained", + (Some(parent), Some(child)) if parent == child => "preserved", + (Some(_), Some(_)) => "changed", + } +} + +/// Count observations separately from RPCs, using only bounded context labels. +#[cfg(feature = "diagnostic-read-version-caller")] +pub fn record_read_version_context_transition( + source: ReadVersionEntrySource, + boundary: ReadVersionContextBoundary, + parent_api: Option, + child_api: Option, + parent_caller: Option, + child_caller: Option, + mode: Option, +) { + if !crate::get_stage_metrics_enabled() { + return; + } + counter!( + "rustfs_storage_diagnostic_context_transitions_total", + "entry_source" => source.as_str(), + "boundary" => boundary.as_str(), + "parent_api_operation" => parent_api.map(S3HttpOperationContext::operation_name).unwrap_or("unattributed"), + "child_api_operation" => child_api.map(S3HttpOperationContext::operation_name).unwrap_or("unattributed"), + "parent_caller" => parent_caller.map(ReadVersionRequestCaller::as_str).unwrap_or("unattributed"), + "child_caller" => child_caller.map(ReadVersionRequestCaller::as_str).unwrap_or("unattributed"), + "mode" => mode.map(ReadVersionRequestMode::as_str).unwrap_or("not_applicable"), + "api_context" => if source == ReadVersionEntrySource::Unclassified && matches!(boundary, ReadVersionContextBoundary::RemoteRpc) { + "unobserved" + } else { context_transition(parent_api, child_api) }, + "caller_context" => if source == ReadVersionEntrySource::Unclassified && matches!(boundary, ReadVersionContextBoundary::RemoteRpc) { + "unobserved" + } else { context_transition(parent_caller, child_caller) }, + ) + .increment(1); +} + +/// Count a native batch RPC once; item observations and unary fallback are separate. +#[cfg(feature = "diagnostic-read-version-caller")] +pub fn record_batch_read_version_request_by_caller( + operation: ReadVersionAttribution, + caller: ReadVersionAttribution, + source: ReadVersionAttribution, + mode: ReadVersionRequestMode, +) { + let api_label = match operation { + ReadVersionAttribution::Absent => "unattributed", + ReadVersionAttribution::Common(operation) => operation.operation_name(), + ReadVersionAttribution::Mixed => "mixed", + }; + let caller_label = match caller { + ReadVersionAttribution::Absent => "unattributed", + ReadVersionAttribution::Common(caller) => caller.as_str(), + ReadVersionAttribution::Mixed => "mixed", + }; + let source_label = match source { + ReadVersionAttribution::Absent => "unclassified", + ReadVersionAttribution::Common(source) => source.as_str(), + ReadVersionAttribution::Mixed => "mixed", + }; + counter!( + "rustfs_system_network_internode_batch_read_version_requests_by_caller_total", + "api_operation" => api_label, + "caller" => caller_label, + "entry_source" => source_label, + "mode" => mode.as_str(), + ) + .increment(1); +} +#[cfg(not(test))] +const READ_VERSION_REQUEST_MODE_COUNT: usize = 5; +const READ_VERSION_REQUEST_CALLER_COUNT: usize = 18; + +#[cfg(not(test))] +static READ_VERSION_CALLER_COUNTERS: LazyLock]>> = LazyLock::new(|| { + std::iter::repeat_with(OnceLock::new) + .take((S3Operation::ALL.len() + 1) * READ_VERSION_REQUEST_MODE_COUNT * READ_VERSION_REQUEST_CALLER_COUNT) + .collect::>() + .into_boxed_slice() +}); + +/// Coarse read mode for diagnostic attribution of remote metadata reads. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum ReadVersionRequestMode { + MetadataOnly, + Data, + Healing, + IncludeFreeVersions, + Combined, +} + +/// Bounded internal call-site class for an outgoing ReadVersion RPC. +#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] +#[repr(usize)] +pub enum ReadVersionRequestCaller { + #[default] + Unattributed, + Peer, + Startup, + Heal, + ListObjects, + Object, + MetadataRead, + Multipart, + DiskStore, + DiskDispatch, + SetDiskRead, + SetDiskIo, + Rebalance, + Decommission, + Lifecycle, + TierRecovery, + CacheValue, + Other, +} + +impl ReadVersionRequestCaller { + pub const ALL: [Self; READ_VERSION_REQUEST_CALLER_COUNT] = [ + Self::Unattributed, + Self::Peer, + Self::Startup, + Self::Heal, + Self::ListObjects, + Self::Object, + Self::MetadataRead, + Self::Multipart, + Self::DiskStore, + Self::DiskDispatch, + Self::SetDiskRead, + Self::SetDiskIo, + Self::Rebalance, + Self::Decommission, + Self::Lifecycle, + Self::TierRecovery, + Self::CacheValue, + Self::Other, + ]; + + pub const fn as_str(self) -> &'static str { + match self { + Self::Unattributed => "unattributed", + Self::Peer => "peer", + Self::Startup => "startup", + Self::Heal => "heal", + Self::ListObjects => "list_objects", + Self::Object => "object", + Self::MetadataRead => "metadata_read", + Self::Multipart => "multipart", + Self::DiskStore => "disk_store", + Self::DiskDispatch => "disk_dispatch", + Self::SetDiskRead => "set_disk_read", + Self::SetDiskIo => "set_disk_io", + Self::Rebalance => "rebalance", + Self::Decommission => "decommission", + Self::Lifecycle => "lifecycle", + Self::TierRecovery => "tier_recovery", + Self::CacheValue => "cache_value", + Self::Other => "other", + } + } +} + +impl ReadVersionRequestMode { + const fn as_str(self) -> &'static str { + match self { + Self::MetadataOnly => "metadata_only", + Self::Data => "data", + Self::Healing => "healing", + Self::IncludeFreeVersions => "include_free_versions", + Self::Combined => "combined", + } + } +} + +/// Record a bounded diagnostic caller class for one outgoing ReadVersion RPC. +/// The caller label is either a fixed S3 operation name or `unattributed`. +pub fn record_read_version_request_by_caller( + operation: Option, + caller: ReadVersionRequestCaller, + mode: ReadVersionRequestMode, +) { + let api_operation = operation + .map(S3HttpOperationContext::operation_name) + .unwrap_or("unattributed"); + + #[cfg(test)] + counter!( + INTERNODE_READ_VERSION_REQUESTS_BY_CALLER_TOTAL, + "api_operation" => api_operation, + "caller" => caller.as_str(), + "mode" => mode.as_str(), + ) + .increment(1); + + #[cfg(not(test))] + { + let caller_index = operation.map_or(S3Operation::ALL.len(), |context| context.operation_index()); + let counter_index = (caller_index * READ_VERSION_REQUEST_MODE_COUNT + mode as usize) * READ_VERSION_REQUEST_CALLER_COUNT + + caller as usize; + READ_VERSION_CALLER_COUNTERS[counter_index] + .get_or_init(|| { + counter!( + INTERNODE_READ_VERSION_REQUESTS_BY_CALLER_TOTAL, + "api_operation" => api_operation, + "caller" => caller.as_str(), + "mode" => mode.as_str(), + ) + }) + .increment(1); + } +} + const OPERATION_LABEL: &str = "operation"; const BACKEND_LABEL: &str = "backend"; const SERVER_LABEL: &str = "server"; @@ -1231,6 +1568,61 @@ mod tests { assert_eq!(snapshot.replay_cache_evictions_total, 3); } + #[test] + fn read_version_caller_metric_uses_only_bounded_labels() { + let recorder = DebuggingRecorder::new(); + let snapshotter = recorder.snapshotter(); + with_local_recorder(&recorder, || { + let mut request = crate::s3_http_metrics::S3HttpRequestGuard::new("GET"); + let operation = request + .in_scope(|| { + crate::s3_http_metrics::observe_s3_http_operation(rustfs_s3_ops::S3Operation::GetObject); + crate::s3_http_metrics::current_s3_http_operation_context() + }) + .expect("S3 request should expose its dispatched operation"); + record_read_version_request_by_caller( + Some(operation), + ReadVersionRequestCaller::SetDiskRead, + ReadVersionRequestMode::Data, + ); + record_read_version_request_by_caller(None, ReadVersionRequestCaller::Peer, ReadVersionRequestMode::MetadataOnly); + }); + + let rows = snapshotter + .snapshot() + .into_vec() + .into_iter() + .filter(|(composite, _, _, _)| composite.key().name() == INTERNODE_READ_VERSION_REQUESTS_BY_CALLER_TOTAL) + .collect::>(); + assert_eq!(rows.len(), 2); + let labels = rows + .iter() + .map(|(composite, _, _, _)| { + composite + .key() + .labels() + .map(|label| (label.key().to_string(), label.value().to_string())) + .collect::>() + }) + .collect::>(); + assert!(labels.iter().any(|labels| { + labels.get("api_operation").map(String::as_str) == Some("s3:GetObject") + && labels.get("caller").map(String::as_str) == Some("set_disk_read") + && labels.get("mode").map(String::as_str) == Some("data") + })); + assert!(labels.iter().any(|labels| { + labels.get("api_operation").map(String::as_str) == Some("unattributed") + && labels.get("mode").map(String::as_str) == Some("metadata_only") + && labels.get("caller").map(String::as_str) == Some("peer") + })); + assert!(labels.iter().all(|labels| { + labels.len() == 3 + && labels + .keys() + .all(|key| key == "api_operation" || key == "caller" || key == "mode") + })); + } + #[test] fn operation_stage_duration_records_low_cardinality_stage_labels() { let recorder = DebuggingRecorder::new(); diff --git a/crates/io-metrics/src/lib.rs b/crates/io-metrics/src/lib.rs index f499c0f02..7fba337bb 100644 --- a/crates/io-metrics/src/lib.rs +++ b/crates/io-metrics/src/lib.rs @@ -49,8 +49,9 @@ #[macro_use] extern crate metrics; +use std::collections::HashMap; use std::sync::{ - Mutex, + LazyLock, Mutex, atomic::{AtomicBool, AtomicU64, Ordering}, }; @@ -61,6 +62,25 @@ use std::sync::{ /// Enabled only through an explicit runtime opt-in. static PUT_STAGE_METRICS_ENABLED: AtomicBool = AtomicBool::new(false); static GET_STAGE_METRICS_ENABLED: AtomicBool = AtomicBool::new(false); +static GET_STAGE_LOCAL_SUMMARY_ENABLED: AtomicBool = AtomicBool::new(false); +static GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE: AtomicU64 = AtomicU64::new(64); +static GET_STAGE_LOCAL_SUMMARY_SEQUENCE: AtomicU64 = AtomicU64::new(0); + +type GetStageLocalSummaryKey = (&'static str, &'static str, &'static str, &'static str); + +/// Sampled local aggregate for one bounded GetObject stage label set. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct GetStageLocalSummary { + pub path: &'static str, + pub stage: &'static str, + pub object_class: &'static str, + pub size_bucket: &'static str, + pub sampled_count: u64, + pub duration_nanoseconds: u64, +} + +static GET_STAGE_LOCAL_SUMMARY: LazyLock>> = + LazyLock::new(|| Mutex::new(HashMap::new())); /// Global switch for all remaining (non GET/PUT-stage) metric emission in this /// crate's free `record_*` functions — I/O scheduler, bytes-pool, zero-copy, @@ -87,6 +107,70 @@ pub fn set_get_stage_metrics_enabled(enabled: bool) { GET_STAGE_METRICS_ENABLED.store(enabled, Ordering::Relaxed); } +pub fn set_get_stage_local_summary_enabled(enabled: bool) { + GET_STAGE_LOCAL_SUMMARY_ENABLED.store(enabled, Ordering::Relaxed); +} + +pub fn set_get_stage_local_summary_sample_rate(sample_rate: u64) { + GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE.store(sample_rate.max(1), Ordering::Relaxed); +} + +pub fn get_stage_local_summary_enabled() -> bool { + GET_STAGE_LOCAL_SUMMARY_ENABLED.load(Ordering::Relaxed) +} + +pub fn get_stage_local_summary_sample_rate() -> u64 { + GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE.load(Ordering::Relaxed) +} + +pub fn take_get_stage_local_summary() -> Vec { + let Ok(mut summary) = GET_STAGE_LOCAL_SUMMARY.lock() else { + return Vec::new(); + }; + let mut rows = summary.drain().map(|(_, row)| row).collect::>(); + rows.sort_unstable_by_key(|row| (row.path, row.stage, row.object_class, row.size_bucket)); + rows +} + +fn record_get_stage_local_summary( + path: &'static str, + stage: &'static str, + object_class: &'static str, + size_bucket: &'static str, + duration_secs: f64, +) { + let sample_rate = get_stage_local_summary_sample_rate(); + if sample_rate > 1 + && !GET_STAGE_LOCAL_SUMMARY_SEQUENCE + .fetch_add(1, Ordering::Relaxed) + .is_multiple_of(sample_rate) + { + return; + } + + let Ok(duration) = std::time::Duration::try_from_secs_f64(duration_secs) else { + return; + }; + let Ok(duration_nanoseconds) = u64::try_from(duration.as_nanos()) else { + return; + }; + + let key = (path, stage, object_class, size_bucket); + let Ok(mut summary) = GET_STAGE_LOCAL_SUMMARY.lock() else { + return; + }; + let row = summary.entry(key).or_insert_with(|| GetStageLocalSummary { + path, + stage, + object_class, + size_bucket, + sampled_count: 0, + duration_nanoseconds: 0, + }); + row.sampled_count = row.sampled_count.saturating_add(1); + row.duration_nanoseconds = row.duration_nanoseconds.saturating_add(duration_nanoseconds); +} + /// Enable or disable general (non GET/PUT-stage) metric emission. /// /// Called once during startup, typically gated by `rustfs_obs::observability_metric_enabled()`. @@ -110,8 +194,210 @@ pub fn put_stage_timer() -> Option { } pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_WAIT: &str = "put_object_commit_namespace_lock_wait"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD: &str = "put_object_commit_namespace_lock_held"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_RENAME: &str = "put_object_commit_namespace_lock_held_rename"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_PRE_RENAME_PREPARE: &str = + "put_object_commit_namespace_lock_held_pre_rename_prepare"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_RENAME_RESULT_TO_TAIL_HANDOFF: &str = + "put_object_commit_namespace_lock_held_rename_result_to_tail_handoff"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_TAIL_HANDOFF: &str = + "put_object_commit_namespace_lock_held_tail_handoff"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_POST_RENAME: &str = "put_object_commit_namespace_lock_held_post_rename"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_GUARD_RELEASE: &str = + "put_object_commit_namespace_lock_held_guard_release"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE: &str = + "put_object_commit_namespace_lock_held_quota_fence_release"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_COMMIT: &str = + "put_object_commit_namespace_lock_held_quota_commit"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT: &str = "put_object_commit_namespace_lock_held_heal_submit"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT: &str = + "put_object_commit_namespace_lock_held_cleanup_receipt"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_METADATA_INVALIDATE: &str = + "put_object_commit_namespace_lock_held_metadata_invalidate"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_OBJECT_INFO_BUILD: &str = + "put_object_commit_namespace_lock_held_object_info_build"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_POST_OBJECT_INFO_TO_RETURN: &str = + "put_object_commit_namespace_lock_held_post_object_info_to_return"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_TASK_WAIT: &str = "put_object_commit_task_wait"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_FUTURE_TOTAL: &str = "put_object_commit_future_total"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_OPERATION_SELECT_WAIT: &str = "put_object_commit_operation_select_wait"; +pub const PUT_STAGE_PUT_OBJECT_OPERATION_AWAIT_TOTAL: &str = "put_object_operation_await_total"; +pub const PUT_STAGE_PUT_OBJECT_OPERATION_BODY_TO_COMMIT_SUBMIT: &str = "put_object_operation_body_to_commit_submit"; +pub const PUT_STAGE_PUT_OBJECT_OPERATION_RESULT_RETURN: &str = "put_object_operation_result_return"; +pub const PUT_STAGE_PUT_OBJECT_OPERATION_FIRST_POLL_DELAY: &str = "put_object_operation_first_poll_delay"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_ACQUIRED_TO_COMMIT_READY: &str = + "put_object_commit_namespace_lock_acquired_to_commit_ready"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_PRECONDITION_AND_TIMESTAMP: &str = + "put_object_commit_ready_precondition_and_timestamp"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_EXPECTED_VERSION_LOOKUP: &str = "put_object_commit_ready_expected_version_lookup"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_EXPLICIT_VERSION_LOOKUP: &str = "put_object_commit_ready_explicit_version_lookup"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_RESTORE_VERIFY: &str = "put_object_commit_ready_restore_verify"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_FENCE_VERIFY: &str = "put_object_commit_ready_fence_verify"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_DECOMMISSION_CAPACITY_FENCE: &str = + "put_object_commit_ready_decommission_capacity_fence"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_TRANSACTION_EPOCH_FENCE_READ: &str = + "put_object_commit_ready_transaction_epoch_fence_read"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_BEGIN: &str = "put_object_commit_ready_quota_begin"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_METADATA_ADJUST: &str = "put_object_commit_ready_metadata_adjust"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_RESERVE: &str = "put_object_commit_ready_quota_reserve"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_MUTATION_FENCE_PREPARE: &str = + "put_object_commit_ready_quota_mutation_fence_prepare"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_CONTEXT_PREPARE: &str = "put_object_commit_ready_context_prepare"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_META_BUCKET_FAST_PATH: &str = "put_object_quota_begin_meta_bucket_fast_path"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_METADATA_LOCK: &str = "put_object_quota_begin_metadata_lock"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CACHE_LOOKUP: &str = "put_object_quota_begin_cache_lookup"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONFIG_READ: &str = "put_object_quota_begin_config_read"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_LOCK_LOST_CHECK: &str = "put_object_quota_begin_lock_lost_check"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_POLICY_EVALUATE: &str = "put_object_quota_begin_policy_evaluate"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CAPABILITY_PROOF: &str = "put_object_quota_begin_capability_proof"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_SNAPSHOT_ADMISSION: &str = "put_object_quota_begin_snapshot_admission"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_OBJECT_STORE_LOOKUP: &str = "put_object_quota_begin_object_store_lookup"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONTEXT_BUILD: &str = "put_object_quota_begin_context_build"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_SUBMIT_TO_CLOSURE_ENTER: &str = "put_object_commit_submit_to_closure_enter"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_ENTER_TO_FIRST_AWAIT: &str = "put_object_commit_closure_enter_to_first_await"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_TOTAL: &str = "put_object_commit_closure_total"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_BEFORE_RENAME: &str = "put_object_commit_before_rename"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_AWAIT_TOTAL: &str = "put_object_commit_pre_rename_await_total"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_QUOTA_MARK_COMMIT_STARTED: &str = + "put_object_commit_pre_rename_quota_mark_commit_started"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_FIRST_FENCE_CHECK: &str = "put_object_commit_pre_rename_first_fence_check"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_RESTORE_VERIFY: &str = "put_object_commit_pre_rename_restore_verify"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_TRANSACTION_FENCE_VERIFY: &str = + "put_object_commit_pre_rename_transaction_fence_verify"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_SECOND_FENCE_CHECK: &str = "put_object_commit_pre_rename_second_fence_check"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_SCANNER_SCOPE_CHECK: &str = "put_object_commit_pre_rename_scanner_scope_check"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_TIER_FREE_VERSION_SOURCE_LOOKUP: &str = + "put_object_commit_pre_rename_tier_free_version_source_lookup"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_ASSIGN_RENAME_INDEXES: &str = + "put_object_commit_pre_rename_assign_rename_indexes"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_POST_AWAIT_TO_RENAME: &str = "put_object_commit_pre_rename_post_await_to_rename"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_AFTER_RENAME_TO_RETURN: &str = "put_object_commit_after_rename_to_return"; pub const PUT_STAGE_SET_DISK_RENAME_QUORUM_WAIT: &str = "set_disk_rename_quorum_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_SYNC_FULL_FANOUT_WAIT: &str = "set_disk_rename_sync_full_fanout_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_EARLY_ACK_SPAWN_TO_QUORUM_WAIT: &str = "set_disk_rename_early_ack_spawn_to_quorum_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_EARLY_ACK_ATTACH_TAIL_OWNER: &str = "set_disk_rename_early_ack_attach_tail_owner"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_AFTER_QUORUM_WAIT: &str = "set_disk_rename_tail_after_quorum_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_ASYNC_WAIT: &str = "set_disk_rename_tail_async_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_GUARD_RELEASE_WAIT: &str = "set_disk_rename_tail_guard_release_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_FINALIZE: &str = "set_disk_rename_tail_finalize"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP: &str = "set_disk_rename_tail_cleanup"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_SUBMIT: &str = "set_disk_rename_tail_cleanup_submit"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP: &str = "set_disk_rename_tail_cleanup_counterfactual_skip"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_ENQUEUED: &str = "set_disk_rename_tail_cleanup_defer_enqueued"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_FALLBACK_SYNC: &str = "set_disk_rename_tail_cleanup_defer_fallback_sync"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_PUSH: &str = "set_disk_rename_tail_cleanup_defer_queue_push"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH: &str = "set_disk_rename_tail_cleanup_defer_queue_depth"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH_MAX_SAMPLE: &str = + "set_disk_rename_tail_cleanup_defer_queue_depth_max_sample"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE: &str = "set_disk_rename_tail_cleanup_defer_oldest_age"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE_MAX_SAMPLE: &str = + "set_disk_rename_tail_cleanup_defer_oldest_age_max_sample"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_TARGETS: &str = "set_disk_rename_tail_cleanup_defer_targets"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_START_DELAY: &str = "set_disk_rename_tail_cleanup_defer_start_delay"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_COMPLETED_LAG: &str = "set_disk_rename_tail_cleanup_defer_completed_lag"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER: &str = "set_disk_rename_tail_cleanup_defer_hold_worker"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_DELAY: &str = "set_disk_rename_tail_cleanup_defer_worker_delay"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_PER_DISK_PERMIT_WAIT: &str = "set_disk_rename_tail_cleanup_per_disk_permit_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_PER_DISK_INFLIGHT: &str = "set_disk_rename_tail_cleanup_per_disk_inflight"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_FOREGROUND_YIELD: &str = "set_disk_rename_tail_cleanup_foreground_yield"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFERRED_FOREGROUND_ACTIVE: &str = + "set_disk_rename_tail_cleanup_deferred_foreground_active"; +pub const PUT_STAGE_SET_DISK_RENAME_FOREGROUND_ACTIVE: &str = "set_disk_rename_foreground_active"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BATCH_WINDOW: &str = "set_disk_rename_tail_cleanup_defer_batch_window"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BATCH_SIZE: &str = "set_disk_rename_tail_cleanup_defer_batch_size"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BURST_BATCH_SIZE: &str = + "set_disk_rename_tail_cleanup_defer_burst_batch_size"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_ACTIVE: &str = "set_disk_rename_tail_cleanup_defer_worker_active"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_POLL_WAIT: &str = + "set_disk_rename_tail_cleanup_defer_worker_poll_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_IDLE_WAIT: &str = + "set_disk_rename_tail_cleanup_defer_worker_idle_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_JOB_WAIT: &str = + "set_disk_rename_tail_cleanup_defer_worker_job_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_COMPLETED: &str = "set_disk_rename_tail_cleanup_defer_completed"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_DURATION: &str = + "set_disk_rename_tail_cleanup_defer_worker_duration"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_RECEIPT: &str = "set_disk_rename_tail_cleanup_target_receipt"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_COMMIT_CAPACITY: &str = + "set_disk_rename_tail_cleanup_target_commit_capacity"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_REPORT: &str = "set_disk_rename_tail_cleanup_target_report"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_COUNT: &str = "set_disk_rename_tail_cleanup_target_count"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_ZERO_TARGET: &str = "set_disk_rename_tail_cleanup_zero_target"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TMP_DELETE: &str = "set_disk_rename_tail_cleanup_tmp_delete"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TMP_DELETE_COUNTERFACTUAL_SKIP: &str = + "set_disk_rename_tail_cleanup_tmp_delete_counterfactual_skip"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DECOMMISSION_GUARD_DROP: &str = + "set_disk_rename_tail_cleanup_decommission_guard_drop"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH: &str = "set_disk_rename_tail_cleanup_lane_queue_depth"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_PUSH: &str = "set_disk_rename_tail_cleanup_lane_queue_push"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH_MAX_SAMPLE: &str = + "set_disk_rename_tail_cleanup_lane_queue_depth_max_sample"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE: &str = "set_disk_rename_tail_cleanup_lane_oldest_age"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE_MAX_SAMPLE: &str = + "set_disk_rename_tail_cleanup_lane_oldest_age_max_sample"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_BATCH_SIZE: &str = "set_disk_rename_tail_cleanup_lane_batch_size"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_BURST_BATCH_SIZE: &str = + "set_disk_rename_tail_cleanup_lane_burst_batch_size"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_ACTIVE: &str = "set_disk_rename_tail_cleanup_lane_worker_active"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_POLL_WAIT: &str = + "set_disk_rename_tail_cleanup_lane_worker_poll_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_IDLE_WAIT: &str = + "set_disk_rename_tail_cleanup_lane_worker_idle_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_JOB_WAIT: &str = "set_disk_rename_tail_cleanup_lane_worker_job_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_SERVICE_TIME: &str = "set_disk_rename_tail_cleanup_lane_service_time"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_REQUEUE: &str = "set_disk_rename_tail_cleanup_lane_requeue"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_FOREGROUND_ACTIVE_HIT: &str = + "set_disk_rename_tail_cleanup_lane_foreground_active_hit"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_HEAL_SUBMIT: &str = "set_disk_rename_tail_heal_submit"; pub const PUT_STAGE_SET_DISK_RENAME_DISK_WAIT: &str = "set_disk_rename_disk_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_LOCAL: &str = "set_disk_rename_disk_wait_local"; +pub const PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_REMOTE: &str = "set_disk_rename_disk_wait_remote"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE: &str = "set_disk_rename_remote_client_prepare"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC: &str = "set_disk_rename_remote_client_rpc"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT: &str = "set_disk_rename_remote_client_rpc_await"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT: &str = + "set_disk_rename_remote_client_rpc_await_poll_pending_count"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY: &str = + "set_disk_rename_remote_client_rpc_await_first_pending_to_ready"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REQUEST_SCOPE: &str = "set_disk_rename_remote_client_request_scope"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL: &str = "set_disk_rename_remote_client_transport_call"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_POLL_PENDING_COUNT: &str = + "set_disk_rename_remote_client_transport_call_poll_pending_count"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_FIRST_PENDING_TO_READY: &str = + "set_disk_rename_remote_client_transport_call_first_pending_to_ready"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REPLAY_RESPONSE: &str = "set_disk_rename_remote_client_replay_response"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_INTO_INNER: &str = "set_disk_rename_remote_client_into_inner"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_SUCCESS_CHECK: &str = "set_disk_rename_remote_client_success_check"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_DECODE: &str = "set_disk_rename_remote_client_decode"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_PROCESS: &str = "set_disk_rename_remote_client_response_process"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_HEADERS: &str = "set_disk_rename_remote_client_response_headers"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_ENCODE: &str = "set_disk_rename_remote_service_response_encode"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE_TRIGGERED: &str = + "put_object_commit_namespace_lock_held_quota_fence_release_triggered"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE_SKIPPED: &str = + "put_object_commit_namespace_lock_held_quota_fence_release_skipped"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT_TRIGGERED: &str = + "put_object_commit_namespace_lock_held_heal_submit_triggered"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT_SKIPPED: &str = + "put_object_commit_namespace_lock_held_heal_submit_skipped"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT_TRIGGERED: &str = + "put_object_commit_namespace_lock_held_cleanup_receipt_triggered"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT_SKIPPED: &str = + "put_object_commit_namespace_lock_held_cleanup_receipt_skipped"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_TOTAL: &str = "set_disk_rename_remote_handler_total"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_OVERHEAD: &str = "set_disk_rename_remote_handler_overhead"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOOKUP: &str = "set_disk_rename_remote_handler_lookup"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_ADMIT: &str = "set_disk_rename_remote_handler_admit"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOCAL_RENAME: &str = "set_disk_rename_remote_handler_local_rename"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_ENTRY: &str = "set_disk_rename_remote_service_entry"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_BEFORE_HANDLER: &str = "set_disk_rename_remote_service_before_handler"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_HANDLER_AWAIT: &str = "set_disk_rename_remote_service_handler_await"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_AFTER_HANDLER: &str = "set_disk_rename_remote_service_after_handler"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_BUILD: &str = "set_disk_rename_remote_service_response_build"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN: &str = "set_disk_rename_remote_service_response_return"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_SERVICE_FUTURE: &str = "set_disk_rename_remote_transport_service_future"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_FIRST_FRAME: &str = "set_disk_rename_remote_transport_first_frame"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_BODY_COMPLETE: &str = "set_disk_rename_remote_transport_body_complete"; pub const PUT_STAGE_SET_DISK_RENAME_FILE_SYNC_PERMIT_WAIT: &str = "set_disk_rename_file_sync_permit_wait"; pub const PUT_STAGE_SET_DISK_RENAME_GLOBAL_FILE_SYNC_PERMIT_WAIT: &str = "set_disk_rename_global_file_sync_permit_wait"; pub const PUT_STAGE_SET_DISK_RENAME_FILE_FDATASYNC: &str = "set_disk_rename_file_fdatasync"; @@ -260,8 +546,8 @@ pub use io_metrics::{ pub use list_objects_metrics::{ LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED, LIST_OBJECTS_GATHER_OUTCOME_LIMIT_REACHED, LIST_OBJECTS_LOCAL_READ_DIR_OUTCOME_ERROR, LIST_OBJECTS_LOCAL_READ_DIR_OUTCOME_OK, LIST_OBJECTS_SOURCE_WALKER, - ListObjectsGatherObservation, ListObjectsIndexPageObservation, ListObjectsLocalReadDirObservation, init_list_objects_metrics, - record_list_objects_gather, record_list_objects_index_attempt, record_list_objects_index_fallback, + ListObjectsGatherCaller, ListObjectsGatherObservation, ListObjectsIndexPageObservation, ListObjectsLocalReadDirObservation, + init_list_objects_metrics, record_list_objects_gather, record_list_objects_index_attempt, record_list_objects_index_fallback, record_list_objects_index_live_verify_failure, record_list_objects_index_served, record_list_objects_local_read_dir, record_list_objects_merge, }; @@ -460,7 +746,16 @@ pub fn record_get_object_request_start(concurrent_requests: usize) { gauge!("rustfs_io_get_object_concurrent_requests").set(concurrent_requests as f64); } -/// Record GetObject request start without concurrency context. +/// Record one timeout-wrapped GetObject storage operation start. +#[inline(always)] +pub fn record_get_object_timeout_operation_start() { + if !get_stage_metrics_enabled() { + return; + } + counter!("rustfs_io_get_object_timeout_operations_total").increment(1); +} + +/// Record a GetObject request start without concurrency context. #[inline(always)] pub fn record_get_object_request_started() { if !get_stage_metrics_enabled() { @@ -479,6 +774,16 @@ pub fn record_get_object_request_result(status: &str, duration_secs: f64) { histogram!("rustfs_io_get_object_request_duration_seconds", "status" => status.to_string()).record(duration_secs); } +/// Record the result of one timeout-wrapped GetObject storage operation. +#[inline(always)] +pub fn record_get_object_timeout_operation_result(status: &str, duration_secs: f64) { + if !get_stage_metrics_enabled() { + return; + } + counter!("rustfs_io_get_object_timeout_operation_results_total", "status" => status.to_string()).increment(1); + histogram!("rustfs_io_get_object_timeout_operation_duration_seconds", "status" => status.to_string()).record(duration_secs); +} + /// Record PutObject request start. #[inline(always)] pub fn record_put_object_request_start(concurrent_requests: usize) { @@ -760,6 +1065,10 @@ pub fn record_get_object_io_state( /// Record GetObject phase duration for the current read path. #[inline(always)] pub fn record_get_object_stage_duration(path: &'static str, stage: &'static str, duration_secs: f64) { + if get_stage_local_summary_enabled() { + record_get_stage_local_summary(path, stage, "", "", duration_secs); + return; + } if !get_stage_metrics_enabled() { return; } @@ -775,6 +1084,10 @@ pub fn record_get_object_stage_duration_by_size( size_bucket: &'static str, duration_secs: f64, ) { + if get_stage_local_summary_enabled() { + record_get_stage_local_summary(path, stage, object_class, size_bucket, duration_secs); + return; + } if !get_stage_metrics_enabled() { return; } @@ -827,6 +1140,15 @@ pub fn record_get_object_metadata_response(path: &'static str, outcome: &'static counter!("rustfs_io_get_object_metadata_response_total", "path" => path, "outcome" => outcome).increment(1); } +/// Record whether metadata resolution selected an object identity at read quorum. +#[inline(always)] +pub fn record_get_object_metadata_quorum_result(path: &'static str, outcome: &'static str) { + if !get_stage_metrics_enabled() { + return; + } + counter!("rustfs_io_get_object_metadata_quorum_results_total", "path" => path, "outcome" => outcome).increment(1); +} + /// Record one bounded metadata cache decision. #[inline(always)] pub fn record_get_object_metadata_cache_decision(path: &'static str, decision: &'static str, reason: &'static str) { @@ -1668,6 +1990,10 @@ pub fn record_get_object_metadata_phase_duration(duration_secs: f64) { /// Record metadata phase duration with early-stop state label. #[inline(always)] pub fn record_get_object_metadata_phase_duration_with_early_stop(duration_secs: f64, early_stop_active: &'static str) { + if get_stage_local_summary_enabled() { + record_get_stage_local_summary("legacy_duplex", "metadata", early_stop_active, "", duration_secs); + return; + } if !get_stage_metrics_enabled() { return; } @@ -2073,6 +2399,57 @@ pub fn record_put_object_stage_duration_from(stage: &'static str, started_at: Op } } +#[inline(always)] +pub fn record_put_object_stage_count(stage: &'static str, count: u64) { + if !put_stage_metrics_enabled() { + return; + } + counter!("rustfs_s3_put_object_stage_count", "stage" => stage).increment(count); +} + +/// Observe a future's latency and wakeup behavior only while detailed PUT +/// attribution is enabled. The disabled path awaits the original future +/// directly, so normal remote mutation RPCs do not gain a polling wrapper. +pub async fn observe_put_stage_future( + future: F, + duration_stage: &'static str, + pending_count_stage: &'static str, + first_pending_to_ready_stage: &'static str, +) -> F::Output +where + F: std::future::Future, +{ + if !put_stage_metrics_enabled() { + return future.await; + } + + let duration_started = put_stage_timer(); + let mut first_pending_at = None; + let mut pending_count = 0_u64; + let mut future = std::pin::pin!(future); + let output = std::future::poll_fn(|cx| match future.as_mut().poll(cx) { + std::task::Poll::Ready(output) => std::task::Poll::Ready(output), + std::task::Poll::Pending => { + pending_count = pending_count.saturating_add(1); + first_pending_at.get_or_insert_with(std::time::Instant::now); + std::task::Poll::Pending + } + }) + .await; + record_put_object_stage_duration_from(duration_stage, duration_started); + record_put_object_stage_count(pending_count_stage, pending_count); + record_put_object_stage_duration_from(first_pending_to_ready_stage, first_pending_at); + output +} + +#[inline(always)] +pub fn record_put_object_quota_cache_decision(decision: &'static str, reason: &'static str) { + if !put_stage_metrics_enabled() { + return; + } + counter!("rustfs_s3_put_object_quota_cache_total", "decision" => decision, "reason" => reason).increment(1); +} + #[inline(always)] pub fn record_put_object_commit_lock_admission(budget: &'static str, outcome: &'static str) { if !put_stage_metrics_enabled() { @@ -3219,8 +3596,162 @@ mod tests { assert_eq!(PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_WAIT, "put_object_commit_namespace_lock_wait"); let stages = [ PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_WAIT, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_RENAME, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_PRE_RENAME_PREPARE, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_RENAME_RESULT_TO_TAIL_HANDOFF, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_TAIL_HANDOFF, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_POST_RENAME, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_GUARD_RELEASE, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_COMMIT, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_METADATA_INVALIDATE, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_OBJECT_INFO_BUILD, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_POST_OBJECT_INFO_TO_RETURN, + PUT_STAGE_PUT_OBJECT_COMMIT_TASK_WAIT, + PUT_STAGE_PUT_OBJECT_COMMIT_FUTURE_TOTAL, + PUT_STAGE_PUT_OBJECT_COMMIT_OPERATION_SELECT_WAIT, + PUT_STAGE_PUT_OBJECT_OPERATION_AWAIT_TOTAL, + PUT_STAGE_PUT_OBJECT_OPERATION_BODY_TO_COMMIT_SUBMIT, + PUT_STAGE_PUT_OBJECT_OPERATION_RESULT_RETURN, + PUT_STAGE_PUT_OBJECT_OPERATION_FIRST_POLL_DELAY, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_ACQUIRED_TO_COMMIT_READY, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_PRECONDITION_AND_TIMESTAMP, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_EXPECTED_VERSION_LOOKUP, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_EXPLICIT_VERSION_LOOKUP, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_RESTORE_VERIFY, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_FENCE_VERIFY, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_DECOMMISSION_CAPACITY_FENCE, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_TRANSACTION_EPOCH_FENCE_READ, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_BEGIN, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_METADATA_ADJUST, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_RESERVE, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_MUTATION_FENCE_PREPARE, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_CONTEXT_PREPARE, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_META_BUCKET_FAST_PATH, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_METADATA_LOCK, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CACHE_LOOKUP, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONFIG_READ, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_LOCK_LOST_CHECK, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_POLICY_EVALUATE, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CAPABILITY_PROOF, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_SNAPSHOT_ADMISSION, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_OBJECT_STORE_LOOKUP, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONTEXT_BUILD, + PUT_STAGE_PUT_OBJECT_COMMIT_SUBMIT_TO_CLOSURE_ENTER, + PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_ENTER_TO_FIRST_AWAIT, + PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_TOTAL, + PUT_STAGE_PUT_OBJECT_COMMIT_BEFORE_RENAME, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_AWAIT_TOTAL, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_QUOTA_MARK_COMMIT_STARTED, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_FIRST_FENCE_CHECK, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_RESTORE_VERIFY, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_TRANSACTION_FENCE_VERIFY, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_SECOND_FENCE_CHECK, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_SCANNER_SCOPE_CHECK, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_TIER_FREE_VERSION_SOURCE_LOOKUP, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_ASSIGN_RENAME_INDEXES, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_POST_AWAIT_TO_RENAME, + PUT_STAGE_PUT_OBJECT_COMMIT_AFTER_RENAME_TO_RETURN, PUT_STAGE_SET_DISK_RENAME_QUORUM_WAIT, + PUT_STAGE_SET_DISK_RENAME_SYNC_FULL_FANOUT_WAIT, + PUT_STAGE_SET_DISK_RENAME_EARLY_ACK_SPAWN_TO_QUORUM_WAIT, + PUT_STAGE_SET_DISK_RENAME_EARLY_ACK_ATTACH_TAIL_OWNER, + PUT_STAGE_SET_DISK_RENAME_TAIL_AFTER_QUORUM_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_ASYNC_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_GUARD_RELEASE_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_FINALIZE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_SUBMIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_ENQUEUED, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_FALLBACK_SYNC, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_PUSH, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH_MAX_SAMPLE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE_MAX_SAMPLE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_TARGETS, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_START_DELAY, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_COMPLETED_LAG, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_DELAY, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_PER_DISK_PERMIT_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_PER_DISK_INFLIGHT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_FOREGROUND_YIELD, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFERRED_FOREGROUND_ACTIVE, + PUT_STAGE_SET_DISK_RENAME_FOREGROUND_ACTIVE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BATCH_WINDOW, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BATCH_SIZE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BURST_BATCH_SIZE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_ACTIVE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_POLL_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_IDLE_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_JOB_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_COMPLETED, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_DURATION, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_RECEIPT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_COMMIT_CAPACITY, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_REPORT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_COUNT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_ZERO_TARGET, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TMP_DELETE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TMP_DELETE_COUNTERFACTUAL_SKIP, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DECOMMISSION_GUARD_DROP, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_PUSH, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH_MAX_SAMPLE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE_MAX_SAMPLE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_BATCH_SIZE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_BURST_BATCH_SIZE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_ACTIVE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_POLL_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_IDLE_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_JOB_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_SERVICE_TIME, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_REQUEUE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_FOREGROUND_ACTIVE_HIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_HEAL_SUBMIT, PUT_STAGE_SET_DISK_RENAME_DISK_WAIT, + PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_LOCAL, + PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_REMOTE, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REQUEST_SCOPE, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_FIRST_PENDING_TO_READY, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REPLAY_RESPONSE, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_INTO_INNER, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_SUCCESS_CHECK, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_DECODE, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_PROCESS, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_HEADERS, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_ENCODE, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE_TRIGGERED, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE_SKIPPED, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT_TRIGGERED, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT_SKIPPED, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT_TRIGGERED, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT_SKIPPED, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_TOTAL, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_OVERHEAD, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOOKUP, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_ADMIT, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOCAL_RENAME, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_ENTRY, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_BEFORE_HANDLER, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_HANDLER_AWAIT, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_AFTER_HANDLER, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_BUILD, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_SERVICE_FUTURE, + PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_FIRST_FRAME, + PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_BODY_COMPLETE, PUT_STAGE_SET_DISK_RENAME_FILE_SYNC_PERMIT_WAIT, PUT_STAGE_SET_DISK_RENAME_GLOBAL_FILE_SYNC_PERMIT_WAIT, PUT_STAGE_SET_DISK_RENAME_FILE_FDATASYNC, @@ -3233,11 +3764,36 @@ mod tests { let unique = stages.iter().copied().collect::>(); assert_eq!(unique.len(), stages.len()); assert!(stages.iter().all(|stage| { - (stage.starts_with("set_disk_rename_") || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_WAIT) + (stage.starts_with("set_disk_rename_") + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_WAIT + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_TASK_WAIT + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_FUTURE_TOTAL + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_OPERATION_SELECT_WAIT + || *stage == PUT_STAGE_PUT_OBJECT_OPERATION_AWAIT_TOTAL + || *stage == PUT_STAGE_PUT_OBJECT_OPERATION_BODY_TO_COMMIT_SUBMIT + || *stage == PUT_STAGE_PUT_OBJECT_OPERATION_RESULT_RETURN + || *stage == PUT_STAGE_PUT_OBJECT_OPERATION_FIRST_POLL_DELAY + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_ACQUIRED_TO_COMMIT_READY + || stage.starts_with("put_object_commit_ready_") + || stage.starts_with("put_object_quota_begin_") + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_SUBMIT_TO_CLOSURE_ENTER + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_ENTER_TO_FIRST_AWAIT + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_TOTAL + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_BEFORE_RENAME + || stage.starts_with("put_object_commit_pre_rename_") + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_AFTER_RENAME_TO_RETURN + || stage.starts_with("put_object_commit_namespace_lock_held_")) && !stage.contains('/') && !stage.contains('{') })); + let count_stages = [ + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_POLL_PENDING_COUNT, + ]; + assert_eq!(count_stages.iter().copied().collect::>().len(), count_stages.len()); + let recorder = DebuggingRecorder::new(); let snapshotter = recorder.snapshotter(); metrics::with_local_recorder(&recorder, || { @@ -3245,16 +3801,25 @@ mod tests { for stage in stages { record_put_object_stage_duration(stage, 1.0); } + for stage in count_stages { + record_put_object_stage_count(stage, 1); + } set_put_stage_metrics_enabled(true); for stage in stages { record_put_object_stage_duration(stage, 1.0); } + for stage in count_stages { + record_put_object_stage_count(stage, 1); + } set_put_stage_metrics_enabled(false); }); - let recorded = snapshotter - .snapshot() - .into_vec() + let rows = snapshotter.snapshot().into_vec(); + assert_eq!( + counter_total(&rows, "rustfs_s3_put_object_stage_count"), + Some(u64::try_from(count_stages.len()).expect("count stage length fits u64")), + ); + let recorded = rows .into_iter() .filter(|(composite, _, _, _)| { composite.kind() == MetricKind::Histogram && composite.key().name() == "rustfs_s3_put_object_stage_duration_ms" @@ -3349,6 +3914,63 @@ mod tests { ]))); } + #[test] + fn put_stage_future_observer_preserves_outputs_and_counts_pending() { + let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + let recorder = DebuggingRecorder::new(); + let snapshotter = recorder.snapshotter(); + let runtime = tokio::runtime::Builder::new_current_thread() + .build() + .expect("build current-thread runtime for observer test"); + let duration_stage = PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT; + let count_stage = PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT; + let pending_stage = PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY; + + metrics::with_local_recorder(&recorder, || { + set_put_stage_metrics_enabled(false); + let output = runtime.block_on(observe_put_stage_future( + std::future::ready(42), + duration_stage, + count_stage, + pending_stage, + )); + assert_eq!(output, 42, "disabled observer must preserve the future output"); + }); + assert!(snapshotter.snapshot().into_vec().is_empty(), "disabled observer must emit no metrics"); + + metrics::with_local_recorder(&recorder, || { + set_put_stage_metrics_enabled(true); + let mut pending = 2; + let future = std::future::poll_fn(|cx| { + if pending > 0 { + pending -= 1; + cx.waker().wake_by_ref(); + std::task::Poll::Pending + } else { + std::task::Poll::Ready(Err::<(), _>("original error")) + } + }); + let output = runtime.block_on(observe_put_stage_future(future, duration_stage, count_stage, pending_stage)); + set_put_stage_metrics_enabled(false); + assert_eq!(output, Err("original error"), "enabled observer must preserve errors"); + }); + + let rows = snapshotter.snapshot().into_vec(); + assert_eq!(counter_total(&rows, "rustfs_s3_put_object_stage_count"), Some(2)); + let count_labels = rows + .iter() + .filter(|(composite, _, _, _)| composite.key().name() == "rustfs_s3_put_object_stage_count") + .flat_map(|(composite, _, _, _)| composite.key().labels().map(|label| label.value().to_string())) + .collect::>(); + assert_eq!(count_labels, vec![count_stage.to_string()]); + let durations = rows + .iter() + .filter(|(composite, _, _, _)| composite.key().name() == "rustfs_s3_put_object_stage_duration_ms") + .flat_map(|(composite, _, _, _)| composite.key().labels().map(|label| label.value().to_string())) + .collect::>(); + assert_eq!(durations, HashSet::from([duration_stage.to_string(), pending_stage.to_string()])); + } + #[test] fn put_rename_code_level_metrics_are_static_and_gated() { let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner()); @@ -3578,6 +4200,35 @@ mod tests { set_put_stage_metrics_enabled(false); } + #[test] + fn get_timeout_operation_and_metadata_quorum_metrics_have_separate_scopes() { + let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + let recorder = DebuggingRecorder::new(); + let snapshotter = recorder.snapshotter(); + + metrics::with_local_recorder(&recorder, || { + set_get_stage_metrics_enabled(true); + record_get_object_request_start(1); + record_get_object_request_result("ok", 0.01); + record_get_object_timeout_operation_start(); + record_get_object_timeout_operation_result("success", 0.005); + record_get_object_metadata_quorum_result("legacy_duplex", "reached"); + record_get_object_metadata_quorum_result("legacy_duplex", "not_reached"); + set_get_stage_metrics_enabled(false); + }); + + let rows = snapshotter.snapshot().into_vec(); + assert_eq!(counter_total(&rows, "rustfs_io_get_object_requests_total"), Some(1)); + assert_eq!(counter_total(&rows, "rustfs_io_get_object_request_results_total"), Some(1)); + assert_eq!(counter_total(&rows, "rustfs_io_get_object_timeout_operations_total"), Some(1)); + assert_eq!(counter_total(&rows, "rustfs_io_get_object_timeout_operation_results_total"), Some(1)); + assert_eq!(counter_total(&rows, "rustfs_io_get_object_metadata_quorum_results_total"), Some(2)); + assert_eq!( + histogram_samples(&rows, "rustfs_io_get_object_timeout_operation_duration_seconds"), + vec![0.005] + ); + } + #[test] fn test_record_get_object_path_and_stage() { let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner()); diff --git a/crates/io-metrics/src/list_objects_metrics.rs b/crates/io-metrics/src/list_objects_metrics.rs index b5f233557..3c8dd863d 100644 --- a/crates/io-metrics/src/list_objects_metrics.rs +++ b/crates/io-metrics/src/list_objects_metrics.rs @@ -22,7 +22,31 @@ pub const LIST_OBJECTS_GATHER_OUTCOME_LIMIT_REACHED: &str = "limit_reached"; pub const LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED: &str = "input_closed"; pub const LIST_OBJECTS_MERGE_OUTCOME_STARTED: &str = "started"; +/// Bounded caller class for a ListObjects walker gather. +#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] +pub enum ListObjectsGatherCaller { + #[default] + Unattributed, + S3List, + S3ListVersions, + Lifecycle, + Other, +} + +impl ListObjectsGatherCaller { + pub const fn as_str(self) -> &'static str { + match self { + Self::Unattributed => "unattributed", + Self::S3List => "s3_list", + Self::S3ListVersions => "s3_list_versions", + Self::Lifecycle => "lifecycle", + Self::Other => "other", + } + } +} + const LIST_OBJECTS_GATHER_TOTAL: &str = "rustfs_s3_list_objects_gather_total"; +const LIST_OBJECTS_GATHER_CALLER_TOTAL: &str = "rustfs_s3_list_objects_gather_caller_total"; const LIST_OBJECTS_GATHER_DURATION_MS: &str = "rustfs_s3_list_objects_gather_duration_ms"; const LIST_OBJECTS_GATHER_SCANNED_ENTRIES: &str = "rustfs_s3_list_objects_gather_scanned_entries"; const LIST_OBJECTS_GATHER_RETURNED_ENTRIES: &str = "rustfs_s3_list_objects_gather_returned_entries"; @@ -53,6 +77,7 @@ pub const LIST_OBJECTS_LOCAL_READ_DIR_OUTCOME_ERROR: &str = "error"; #[derive(Clone, Copy, Debug, PartialEq)] pub struct ListObjectsGatherObservation { pub source: &'static str, + pub caller: ListObjectsGatherCaller, pub outcome: &'static str, pub limit: i32, pub scanned_entries: usize, @@ -213,6 +238,7 @@ pub fn record_list_objects_gather(observation: ListObjectsGatherObservation) { } else { count_as_f64(observation.scanned_entries) / count_as_f64(observation.returned_entries) }; + let api_operation = crate::s3_http_metrics::current_s3_http_operation_name().unwrap_or("unattributed"); counter!( LIST_OBJECTS_GATHER_TOTAL, @@ -223,6 +249,14 @@ pub fn record_list_objects_gather(observation: ListObjectsGatherObservation) { "has_marker" => bool_label(observation.has_marker), ) .increment(1); + counter!( + LIST_OBJECTS_GATHER_CALLER_TOTAL, + "api_operation" => api_operation, + "caller" => observation.caller.as_str(), + "source" => observation.source, + "outcome" => observation.outcome, + ) + .increment(1); histogram!( LIST_OBJECTS_GATHER_DURATION_MS, "source" => observation.source, @@ -432,6 +466,7 @@ mod tests { let rows = recorded(|| { record_list_objects_gather(ListObjectsGatherObservation { source: LIST_OBJECTS_SOURCE_WALKER, + caller: ListObjectsGatherCaller::Unattributed, outcome: LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED, limit: 1001, scanned_entries: 42, @@ -444,6 +479,17 @@ mod tests { }); assert_eq!(counter_total(&rows, LIST_OBJECTS_GATHER_TOTAL), Some(1)); + assert_eq!(counter_total(&rows, LIST_OBJECTS_GATHER_CALLER_TOTAL), Some(1)); + let caller_labels = rows + .iter() + .find(|(composite, _, _, _)| composite.key().name() == LIST_OBJECTS_GATHER_CALLER_TOTAL) + .expect("internal caller counter should be emitted") + .0 + .key() + .labels() + .map(|label| (label.key().to_string(), label.value().to_string())) + .collect::>(); + assert_eq!(caller_labels.get("api_operation").map(String::as_str), Some("unattributed")); // Zero returned entries must not divide: the amplification reports the // scan count itself rather than an infinity or a NaN. assert_eq!(histogram_samples(&rows, LIST_OBJECTS_GATHER_SCAN_AMPLIFICATION), vec![42.0]); @@ -452,6 +498,44 @@ mod tests { assert_eq!(histogram_samples(&rows, LIST_OBJECTS_GATHER_DURATION_MS), vec![3.5]); } + #[test] + fn gather_caller_metric_attributes_the_s3_api_operation() { + use rustfs_s3_ops::S3Operation; + + let rows = recorded(|| { + let mut request = crate::s3_http_metrics::S3HttpRequestGuard::new("GET"); + request.in_scope(|| { + crate::s3_http_metrics::observe_s3_http_operation(S3Operation::ListObjectsV2); + record_list_objects_gather(ListObjectsGatherObservation { + source: LIST_OBJECTS_SOURCE_WALKER, + caller: ListObjectsGatherCaller::S3List, + outcome: LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED, + limit: 1000, + scanned_entries: 1, + returned_entries: 1, + duration_ms: 1.0, + has_prefix: false, + has_delimiter: false, + has_marker: false, + }); + }); + }); + + assert_eq!(counter_total(&rows, LIST_OBJECTS_GATHER_CALLER_TOTAL), Some(1)); + let labels = rows + .iter() + .find(|(composite, _, _, _)| composite.key().name() == LIST_OBJECTS_GATHER_CALLER_TOTAL) + .expect("caller counter should be emitted") + .0 + .key() + .labels() + .map(|label| (label.key().to_string(), label.value().to_string())) + .collect::>(); + assert_eq!(labels.get("api_operation").map(String::as_str), Some("s3:ListObjectsV2")); + assert_eq!(labels.get("caller").map(String::as_str), Some("s3_list")); + assert_eq!(labels.get("source").map(String::as_str), Some(LIST_OBJECTS_SOURCE_WALKER)); + } + #[test] fn merge_records_a_zero_read_quorum_rather_than_skipping_it() { let rows = recorded(|| record_list_objects_merge(LIST_OBJECTS_SOURCE_WALKER, 4, 0)); diff --git a/crates/io-metrics/src/s3_http_metrics.rs b/crates/io-metrics/src/s3_http_metrics.rs index 7e0cd5b3d..07bc7df3e 100644 --- a/crates/io-metrics/src/s3_http_metrics.rs +++ b/crates/io-metrics/src/s3_http_metrics.rs @@ -18,6 +18,7 @@ use rustfs_s3_ops::S3Operation; use std::cell::Cell; +use std::future::Future; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{LazyLock, OnceLock}; use std::time::{Duration, Instant}; @@ -38,6 +39,81 @@ const UNKNOWN_OPERATION: usize = S3Operation::ALL.len(); static COUNTERS: LazyLock = LazyLock::new(HttpOutcomeCounters::new); static COMPLETION_OBSERVER: OnceLock<(S3HttpCompletionObserverEnabled, S3HttpCompletionObserver)> = OnceLock::new(); +/// A bounded S3 operation captured from the current external-request context. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct S3HttpOperationContext(S3Operation); + +impl S3HttpOperationContext { + /// Return the stable operation index used for bounded metric-handle lookup. + pub fn operation_index(self) -> usize { + self.0.metric_index() + } + + /// Return the stable S3 operation label associated with this context. + pub fn operation_name(self) -> &'static str { + self.0.as_str() + } +} + +/// Return the current S3 operation, when the task is inside a dispatched S3 request. +pub fn current_s3_http_operation_context() -> Option { + #[cfg(feature = "diagnostic-read-version-caller")] + if let Ok(index) = ISOLATED_OPERATION.try_with(Cell::get) { + return S3Operation::ALL.get(index).copied().map(S3HttpOperationContext); + } + CURRENT_OPERATION + .try_with(|current| { + let index = current.get(); + S3Operation::ALL.get(index).copied().map(S3HttpOperationContext) + }) + .ok() + .flatten() +} + +/// Return the current S3 operation label, or `None` for internal/non-S3 work. +pub fn current_s3_http_operation_name() -> Option<&'static str> { + current_s3_http_operation_context().map(S3HttpOperationContext::operation_name) +} + +/// Reapply a captured S3 operation while polling work on a child Tokio task. +pub async fn with_s3_http_operation_context(context: Option, future: F) -> F::Output +where + F: Future, +{ + match context { + Some(context) => { + #[cfg(feature = "diagnostic-read-version-caller")] + return with_isolated_s3_http_operation_context(Some(context), future).await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] + CURRENT_OPERATION.scope(Cell::new(context.operation_index()), future).await + } + None => future.await, + } +} + +/// Restore an item's operation, including explicit absence, without changing +/// the ambient HTTP request guard or borrowing the request that flushed a batch. +#[cfg(feature = "diagnostic-read-version-caller")] +pub async fn with_isolated_s3_http_operation_context(context: Option, future: F) -> F::Output +where + F: Future, +{ + let index = context.map_or(UNKNOWN_OPERATION, S3HttpOperationContext::operation_index); + ISOLATED_OPERATION + .scope(Cell::new(index), async { + match context { + Some(context) => CURRENT_OPERATION.scope(Cell::new(context.operation_index()), future).await, + None => future.await, + } + }) + .await +} + +#[cfg(feature = "diagnostic-read-version-caller")] +tokio::task_local! { + static ISOLATED_OPERATION: Cell; +} + tokio::task_local! { static CURRENT_OPERATION: Cell; } @@ -240,6 +316,25 @@ mod tests { ); } + #[tokio::test] + async fn captured_operation_context_can_be_reapplied_to_child_task() { + let mut request = S3HttpRequestGuard::new("GET"); + let operation = request + .in_scope(|| { + observe_s3_http_operation(S3Operation::GetObject); + current_s3_http_operation_context() + }) + .expect("dispatched S3 operation should be captured"); + + let child_operation = tokio::spawn(with_s3_http_operation_context(Some(operation), async { + current_s3_http_operation_name() + })) + .await + .expect("child task should finish"); + + assert_eq!(child_operation, Some("s3:GetObject")); + } + #[test] fn outcome_counters_distinguish_partial_and_complete_write_failure() { let counters = HttpOutcomeCounters::new(); @@ -279,6 +374,39 @@ mod tests { } } + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn isolated_absent_operation_does_not_inherit_the_flush_request() { + let mut request = S3HttpRequestGuard::new("GET"); + let operation = request.in_scope(|| { + observe_s3_http_operation(S3Operation::GetObject); + current_s3_http_operation_context().expect("GetObject operation should be captured") + }); + with_s3_http_operation_context(Some(operation), async { + assert!(S3HttpRequestGuard::is_active()); + with_isolated_s3_http_operation_context(None, async { + assert_eq!(current_s3_http_operation_context(), None); + assert!( + S3HttpRequestGuard::is_active(), + "isolating an item must leave ambient HTTP accounting intact" + ); + tokio::task::yield_now().await; + assert_eq!(current_s3_http_operation_context(), None); + }) + .await; + assert_eq!(current_s3_http_operation_context(), Some(operation)); + }) + .await; + assert_eq!(current_s3_http_operation_context(), None); + with_isolated_s3_http_operation_context(None, async { + assert!( + !S3HttpRequestGuard::is_active(), + "absent operation must not fabricate an HTTP request guard" + ); + }) + .await; + } + #[test] fn request_guard_preserves_operation_across_polls_and_finishes_once() { let totals = || { diff --git a/crates/lifecycle/src/core.rs b/crates/lifecycle/src/core.rs index fbbf35090..138e7cc84 100644 --- a/crates/lifecycle/src/core.rs +++ b/crates/lifecycle/src/core.rs @@ -648,7 +648,11 @@ impl Lifecycle for BucketLifecycleConfiguration { async fn eval(&self, obj: &ObjectOpts) -> Event { // A single-object lookup cannot prove how many newer historical versions // survive. Count-dependent actions wait for the complete-group evaluator. - self.eval_inner(obj, OffsetDateTime::now_utc(), 0).await + let mut event = self.eval_inner(obj, OffsetDateTime::now_utc(), 0).await; + if event.requires_expiry_policy() { + event.expiry_policy = Some(Arc::new(self.clone()).into()); + } + event } async fn predict_expiration(&self, obj: &ObjectOpts) -> Event { @@ -687,6 +691,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }; let predicted_due = predicted.due.unwrap_or(OffsetDateTime::UNIX_EPOCH).unix_timestamp(); let should_replace = event @@ -749,6 +754,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }); // Stop after scheduling an expired delete-marker event. break; @@ -764,6 +770,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }); } } @@ -783,6 +790,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }); } continue; @@ -812,6 +820,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }); } } else { @@ -877,6 +886,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }); } } else if let Some(days) = expiration.days { @@ -898,6 +908,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }; // MinIO extension: ExpiredObjectAllVersions deletes all versions if rule @@ -935,6 +946,7 @@ impl Lifecycle for BucketLifecycleConfiguration { rule_id: rule.id.clone().unwrap_or_default(), due, storage_class: storage_class.as_str().to_string(), + expiry_policy: None, noncurrent_days: 0, newer_noncurrent_versions: 0, }); @@ -983,6 +995,7 @@ impl Lifecycle for BucketLifecycleConfiguration { newer_noncurrent_versions: retained, due: Some(OffsetDateTime::UNIX_EPOCH), storage_class: "".into(), + expiry_policy: Some(Arc::clone(&self).into()), } } else { Event { @@ -993,6 +1006,7 @@ impl Lifecycle for BucketLifecycleConfiguration { newer_noncurrent_versions: 0, due: Some(OffsetDateTime::UNIX_EPOCH), storage_class: "".into(), + expiry_policy: Some(Arc::clone(&self).into()), } }; } @@ -1284,7 +1298,27 @@ fn ilm_action_priority_rank(action: &IlmAction) -> u8 { } } -#[derive(Debug, Clone)] +/// Immutable policy identity carried from lifecycle evaluation to deletion. +/// Consumers can compare authoritative metadata without exposing the policy's +/// wire representation in storage operation options. Clones share the policy. +#[derive(Clone)] +pub struct ExpiryPolicySnapshot { + configuration: Arc, +} + +impl From> for ExpiryPolicySnapshot { + fn from(configuration: Arc) -> Self { + Self { configuration } + } +} + +impl ExpiryPolicySnapshot { + pub fn matches_configuration(&self, current: Option<&BucketLifecycleConfiguration>) -> bool { + current == Some(self.configuration.as_ref()) + } +} + +#[derive(Clone)] pub struct Event { pub action: IlmAction, pub rule_id: String, @@ -1292,6 +1326,35 @@ pub struct Event { pub noncurrent_days: u32, pub newer_noncurrent_versions: usize, pub storage_class: String, + /// In-memory policy that authorized logical expiration. Execution must + /// compare it with authoritative bucket metadata under the transaction lock. + pub expiry_policy: Option, +} + +impl Event { + pub fn requires_expiry_policy(&self) -> bool { + matches!( + self.action, + IlmAction::DeleteAction + | IlmAction::DeleteVersionAction + | IlmAction::DeleteAllVersionsAction + | IlmAction::DelMarkerDeleteAllVersionsAction + ) + } +} + +impl std::fmt::Debug for Event { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Event") + .field("action", &self.action) + .field("rule_id", &self.rule_id) + .field("due", &self.due) + .field("noncurrent_days", &self.noncurrent_days) + .field("newer_noncurrent_versions", &self.newer_noncurrent_versions) + .field("storage_class", &self.storage_class) + .field("has_expiry_policy", &self.expiry_policy.is_some()) + .finish() + } } impl Default for Event { @@ -1303,6 +1366,7 @@ impl Default for Event { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, } } } diff --git a/crates/lifecycle/src/evaluator.rs b/crates/lifecycle/src/evaluator.rs index 7f9089610..e4e155984 100644 --- a/crates/lifecycle/src/evaluator.rs +++ b/crates/lifecycle/src/evaluator.rs @@ -141,6 +141,11 @@ impl Evaluator { events[i] = event; } } + for event in &mut events { + if event.requires_expiry_policy() { + event.expiry_policy = Some(Arc::clone(&self.policy).into()); + } + } events } @@ -218,6 +223,30 @@ mod tests { use super::*; use rustfs_replication::{ReplicationStatusType, VersionPurgeStatusType}; + #[tokio::test] + async fn expiration_events_retain_the_evaluated_policy() { + let policy = latest_expiration_lifecycle(); + let object = current_object_opts(ReplicationStatusType::Completed); + let event = Evaluator::new(Arc::clone(&policy)) + .eval(std::slice::from_ref(&object)) + .await + .unwrap() + .remove(0); + assert_eq!(event.action, IlmAction::DeleteAction); + assert_eq!(Arc::strong_count(&policy), 2, "the event should share the evaluated policy"); + assert!(event.expiry_policy.as_ref().unwrap().matches_configuration(Some(&policy))); + assert!( + policy + .eval(&object) + .await + .expiry_policy + .as_ref() + .unwrap() + .matches_configuration(Some(&policy)) + ); + assert!(!format!("{event:?}").contains("BucketLifecycleConfiguration")); + } + #[tokio::test] async fn adversarial_restore_expiry_survives_legal_hold() { let mut policy = (*latest_expiration_lifecycle()).clone(); @@ -247,6 +276,7 @@ mod tests { .await .expect("unlocked restored versions should evaluate"); assert_eq!(unlocked.iter().map(|event| event.action).collect::>(), expected); + assert!(unlocked.iter().all(|event| event.expiry_policy.is_none())); for object in &mut objects { object diff --git a/crates/log-analyzer/src/ingest/mod.rs b/crates/log-analyzer/src/ingest/mod.rs index 2ad960479..ba8b97b27 100644 --- a/crates/log-analyzer/src/ingest/mod.rs +++ b/crates/log-analyzer/src/ingest/mod.rs @@ -307,7 +307,13 @@ fn walk_zip_entries( if entry.is_dir() { continue; } - let entry_path = entry.name().to_string(); + let entry_path = match entry.name() { + Ok(name) => name.into_owned(), + Err(_) => { + ctx.skip(format!("{provenance}!"), SkipReason::Unreadable); + continue; + } + }; let child_provenance = format!("{provenance}!{entry_path}"); let child_node = node_from_entry_path(&entry_path).or_else(|| node.clone()); let child_name = leaf_name(&entry_path).to_string(); diff --git a/crates/log-analyzer/src/rules/seed/heal.rs b/crates/log-analyzer/src/rules/seed/heal.rs index b5564f49e..8957b97d6 100644 --- a/crates/log-analyzer/src/rules/seed/heal.rs +++ b/crates/log-analyzer/src/rules/seed/heal.rs @@ -15,7 +15,7 @@ //! Self-heal rules. use super::super::model::{Rule, Severity::*}; -use super::{any, base, contains, prefix, strings}; +use super::{all, any, base, contains, field, prefix, strings}; pub(super) fn rules() -> Vec { vec![ @@ -33,14 +33,29 @@ pub(super) fn rules() -> Vec { ) }, Rule { - anchors: strings(["all drives had write errors, unable to heal"]), + evidence_fields: strings([ + "bucket", + "object", + "version_id", + "target_count", + "writer_failure_count", + "error", + ]), + anchors: strings(["Set disk object heal writer failures"]), ..base( "heal-all-writes-failed", P0DataRisk, "heal", - "heal 结果无法写入任何盘", - contains("all drives had write errors, unable to heal"), - "heal 结果无法写入任何盘(全盘故障/满盘)。", + "所有自愈目标不可写", + any([ + contains("all drives had write errors, unable to heal"), + all([ + contains("Set disk object heal writer failures"), + field("state", "writer_unavailable"), + field("result", "all_targets_unavailable"), + ]), + ]), + "所有自愈目标不可写,修复无法继续;健康源盘可能仍可读。", "先解决容量/盘故障 finding,再重跑 heal。", ) }, diff --git a/crates/log-analyzer/src/rules/seed/tests.rs b/crates/log-analyzer/src/rules/seed/tests.rs index 88bf61a6a..d6a87f094 100644 --- a/crates/log-analyzer/src/rules/seed/tests.rs +++ b/crates/log-analyzer/src/rules/seed/tests.rs @@ -176,7 +176,15 @@ fn every_rule_has_a_positive_sample() { "heal-no-datadir", msg("heal: latest metadata for b/o has no data_dir, cannot heal object data"), ), - ("heal-all-writes-failed", msg("all drives had write errors, unable to heal b/o")), + ( + "heal-all-writes-failed", + Sample { + message: "Set disk object heal writer failures", + level: Some(LogLevel::Warn), + fields: &[("state", "writer_unavailable"), ("result", "all_targets_unavailable")], + ..Default::default() + }, + ), ("heal-rename-failed", msg("heal rename incomplete: 1 of 2 targets committed for b/o")), ( "heal-xlmeta-regen-failed", @@ -297,6 +305,46 @@ fn every_rule_has_a_positive_sample() { } } +#[test] +fn heal_writer_rule_requires_all_targets_and_retains_legacy_messages() { + let engine = engine(); + for (sample, expected) in [ + ( + Sample { + message: "Set disk object heal writer failures", + level: Some(LogLevel::Warn), + fields: &[("state", "writer_unavailable"), ("result", "all_targets_unavailable")], + ..Default::default() + }, + true, + ), + ( + Sample { + message: "Set disk object heal writer failures", + level: Some(LogLevel::Warn), + fields: &[("state", "writer_unavailable"), ("result", "partial_targets_unavailable")], + ..Default::default() + }, + false, + ), + ( + Sample { + message: "Set disk object heal writer failures", + fields: &[("result", "all_targets_unavailable")], + ..Default::default() + }, + false, + ), + (msg("all drives had write errors, unable to heal b/o"), true), + ] { + let matched = engine + .matches(&event(&sample)) + .iter() + .any(|&index| engine.rules()[index].id == "heal-all-writes-failed"); + assert_eq!(matched, expected); + } +} + #[test] fn smoke_samples_hit_exact_rule_sets() { let engine = engine(); @@ -372,3 +420,44 @@ fn smoke_samples_hit_exact_rule_sets() { &[], ); } + +#[test] +fn heal_all_writes_failed_matches_current_and_legacy_logs() { + let engine = engine(); + let rule_ids = |sample: &Sample| { + engine + .matches(&event(sample)) + .into_iter() + .map(|index| engine.rules()[index].id.as_str()) + .collect::>() + }; + + assert_eq!( + rule_ids(&Sample { + message: "Set disk object heal writer failures", + level: Some(LogLevel::Warn), + fields: &[("state", "writer_unavailable"), ("result", "all_targets_unavailable")], + ..Default::default() + }), + vec!["heal-all-writes-failed"] + ); + assert_eq!( + rule_ids(&msg("all drives had write errors, unable to heal b/o")), + vec!["heal-all-writes-failed"] + ); + for sample in [ + Sample { + message: "Set disk object heal writer failures", + fields: &[("state", "writer_unavailable"), ("result", "partial_targets_unavailable")], + ..Default::default() + }, + msg("Set disk object heal writer failures"), + Sample { + message: "unrelated operation failed", + fields: &[("result", "all_targets_unavailable")], + ..Default::default() + }, + ] { + assert!(rule_ids(&sample).is_empty(), "unexpected rule match: {}", sample.message); + } +} diff --git a/crates/madmin/src/heal_commands.rs b/crates/madmin/src/heal_commands.rs index b1a22efcd..807d3699b 100644 --- a/crates/madmin/src/heal_commands.rs +++ b/crates/madmin/src/heal_commands.rs @@ -34,6 +34,20 @@ pub struct Infos { /// enum's crate). const DRIVE_STATE_OK: &str = "ok"; +/// Unresolved delete recovery work, independent of ordinary object healing. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct DeleteRecoveryStatus { + pub pending: usize, + pub blocked: usize, + pub failed: usize, +} + +impl DeleteRecoveryStatus { + pub fn has_unresolved(&self) -> bool { + self.pending != 0 || self.blocked != 0 || self.failed != 0 + } +} + #[derive(Clone, Debug, Default, Serialize, Deserialize)] pub struct HealResultItem { /// Only the coordinator's authenticated full scan may set this. Transported @@ -70,6 +84,8 @@ pub struct HealResultItem { pub resolved_version_id: Option<[u8; 16]>, #[serde(rename = "detail")] pub detail: String, + #[serde(default, rename = "deleteRecovery", skip_serializing_if = "Option::is_none")] + pub delete_recovery: Option, #[serde(rename = "parityBlocks")] pub parity_blocks: usize, #[serde(rename = "dataBlocks")] @@ -118,6 +134,38 @@ impl HealResultItem { mod tests { use super::*; + #[test] + fn delete_recovery_status_preserves_partial_results_and_legacy_shape() { + let legacy = serde_json::to_value(HealResultItem::default()).expect("serialize legacy result"); + assert!(legacy.get("deleteRecovery").is_none()); + let decoded: HealResultItem = serde_json::from_value(legacy).expect("decode result without recovery status"); + assert!(decoded.delete_recovery.is_none()); + for status in [ + DeleteRecoveryStatus { + pending: 1, + ..Default::default() + }, + DeleteRecoveryStatus { + blocked: 1, + ..Default::default() + }, + DeleteRecoveryStatus { + failed: 1, + ..Default::default() + }, + ] { + let item = HealResultItem { + delete_recovery: Some(status), + ..Default::default() + }; + let wire = serde_json::to_value(item).expect("serialize recovery status"); + let decoded: HealResultItem = serde_json::from_value(wire).expect("decode recovery status"); + assert_eq!(decoded.delete_recovery, Some(status)); + assert!(status.has_unresolved()); + } + assert!(!DeleteRecoveryStatus::default().has_unresolved()); + } + #[test] fn resolved_heal_version_is_not_wire_proof() { let item = HealResultItem { diff --git a/crates/madmin/src/metrics.rs b/crates/madmin/src/metrics.rs index 1753bde2b..3ca3daca9 100644 --- a/crates/madmin/src/metrics.rs +++ b/crates/madmin/src/metrics.rs @@ -318,6 +318,13 @@ pub struct ScannerUsageFreshnessSnapshot { pub last_deferred_unix_secs: u64, #[serde(rename = "last_deferred_reason", default)] pub last_deferred_reason: String, + /// Cluster-activity guardrail vetoes, attributed as `@`. + #[serde(rename = "guardrail_veto_total", default)] + pub guardrail_veto_total: u64, + #[serde(rename = "last_guardrail_veto_unix_secs", default)] + pub last_guardrail_veto_unix_secs: u64, + #[serde(rename = "last_guardrail_veto_reason", default)] + pub last_guardrail_veto_reason: String, } impl ScannerUsageFreshnessSnapshot { @@ -343,6 +350,11 @@ impl ScannerUsageFreshnessSnapshot { self.last_publication_reason = other.last_publication_reason.clone(); } self.deferred_total = self.deferred_total.saturating_add(other.deferred_total); + self.guardrail_veto_total = self.guardrail_veto_total.saturating_add(other.guardrail_veto_total); + if other.last_guardrail_veto_unix_secs > self.last_guardrail_veto_unix_secs { + self.last_guardrail_veto_unix_secs = other.last_guardrail_veto_unix_secs; + self.last_guardrail_veto_reason = other.last_guardrail_veto_reason.clone(); + } if other_deferred_state_at > self_deferred_state_at { self.deferred_pending = other.deferred_pending; } else if other_deferred_state_at == self_deferred_state_at { @@ -1719,6 +1731,26 @@ mod tests { .expect("legacy usage freshness should decode"); assert!(!legacy.deferred_pending); assert_eq!(legacy.deferred_total, 0); + assert_eq!(legacy.guardrail_veto_total, 0); + assert_eq!(legacy.last_guardrail_veto_unix_secs, 0); + assert!(legacy.last_guardrail_veto_reason.is_empty()); + + let mut vetoes = ScannerUsageFreshnessSnapshot::default(); + vetoes.merge(&ScannerUsageFreshnessSnapshot { + guardrail_veto_total: 2, + last_guardrail_veto_unix_secs: 30, + last_guardrail_veto_reason: "publication_blocked@".to_string(), + ..Default::default() + }); + vetoes.merge(&ScannerUsageFreshnessSnapshot { + guardrail_veto_total: 1, + last_guardrail_veto_unix_secs: 20, + last_guardrail_veto_reason: "movement_generation@node-2".to_string(), + ..Default::default() + }); + assert_eq!(vetoes.guardrail_veto_total, 3); + assert_eq!(vetoes.last_guardrail_veto_unix_secs, 30); + assert_eq!(vetoes.last_guardrail_veto_reason, "publication_blocked@"); let mut merged = ScannerUsageFreshnessSnapshot::default(); merged.merge(&ScannerUsageFreshnessSnapshot { diff --git a/crates/obs/README.md b/crates/obs/README.md index a66fc5dea..0a1a9caaf 100644 --- a/crates/obs/README.md +++ b/crates/obs/README.md @@ -127,7 +127,9 @@ All configuration is read from environment variables at startup. `RUSTFS_OBS_TLS_CA_FILE` is read only while OTLP exporters are initialized. Its PEM certificates are added to, not substituted for, the platform trust roots, and TLS chain, expiry, and hostname checks remain enabled. Restart RustFS after rotating the bundle. A missing, unreadable, non-absolute, empty, or invalid bundle fails OTLP initialization. -The export interval is separate from application metric collection. Without interval overrides, node/disk metrics refresh every 60 seconds (`RUSTFS_METRICS_NODE_INTERVAL_SEC`), cluster metrics every 60 seconds (`RUSTFS_METRICS_CLUSTER_INTERVAL_SEC`), and per-bucket metrics every 300 seconds (`RUSTFS_METRICS_BUCKET_INTERVAL_SEC`). See the [collection defaults](src/metrics/config.rs) and [interval configuration](src/metrics/scheduler.rs) for collector-specific and global overrides. +The export interval is separate from application metric collection. Without interval overrides, node/disk metrics refresh every 60 seconds (`RUSTFS_METRICS_NODE_INTERVAL_SEC`), cluster metrics including scanner bucket usage every 60 seconds (`RUSTFS_METRICS_CLUSTER_INTERVAL_SEC`), and the separate bucket collector every 300 seconds (`RUSTFS_METRICS_BUCKET_INTERVAL_SEC`). See the [collection defaults](src/metrics/config.rs) and [interval configuration](src/metrics/scheduler.rs) for collector-specific and global overrides. + +For bucket size and object-count metric names, all-bucket PromQL queries, accounting semantics, and scanner freshness, see [Bucket usage metrics](../../docs/operations/bucket-usage-metrics.md). With OTLP → Collector → Prometheus, the time from a failure to an alert also includes underlying health detection, collection and export work, transport and Collector batching, Prometheus `scrape_interval` and `evaluation_interval`, and any alert rule `for` duration. A shorter scrape interval cannot refresh a value that RustFS has not yet collected and exported. These periodic intervals do not guarantee a fixed detection or alert latency; see the [Prometheus alert rule documentation](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) for the role of `for`. diff --git a/crates/obs/src/config.rs b/crates/obs/src/config.rs index c637719ff..4bc7c7705 100644 --- a/crates/obs/src/config.rs +++ b/crates/obs/src/config.rs @@ -238,8 +238,10 @@ impl OtelConfig { /// that know their display/build version should use /// [`OtelConfig::extract_otel_config_from_env_with_version`]. /// - /// When no endpoint is configured at all, `use_stdout` is forced to `true` - /// so that logs are still visible during development. + /// When no endpoint is configured at all (neither the root endpoint nor any + /// per-signal one), `use_stdout` is forced to `true` so that logs are still + /// visible during development. With any endpoint set, `RUSTFS_OBS_USE_STDOUT` + /// is honoured. /// /// # Example /// ```no_run @@ -268,9 +270,15 @@ impl OtelConfig { _ => env::var(ENV_OBS_ENDPOINT).unwrap_or_default(), }; - // Force stdout when there is no remote endpoint so that operators - // always have *some* log output in the default configuration. - let use_stdout = if endpoint.is_empty() { + // Force stdout only when no OTLP endpoint is configured at all, so that operators + // always have *some* log output in the default configuration. A per-signal endpoint + // (e.g. `RUSTFS_OBS_TRACE_ENDPOINT`) counts as configured: otherwise setting only that + // one forces the stdout exporters on and silently ignores `RUSTFS_OBS_USE_STDOUT=false`, + // printing every exported span to stdout. + let has_signal_endpoint = [ENV_OBS_TRACE_ENDPOINT, ENV_OBS_METRIC_ENDPOINT, ENV_OBS_LOG_ENDPOINT] + .into_iter() + .any(|key| get_env_opt_str(key).is_some_and(|value| !value.is_empty())); + let use_stdout = if endpoint.is_empty() && !has_signal_endpoint { true } else { get_env_bool(ENV_OBS_USE_STDOUT, USE_STDOUT) @@ -538,6 +546,45 @@ mod tests { }); } + #[test] + fn use_stdout_is_forced_only_when_no_endpoint_is_configured() { + with_profiling_env_lock(|| { + temp_env::with_vars_unset( + [ + ENV_OBS_ENDPOINT, + ENV_OBS_TRACE_ENDPOINT, + ENV_OBS_METRIC_ENDPOINT, + ENV_OBS_LOG_ENDPOINT, + ], + || { + temp_env::with_var(ENV_OBS_USE_STDOUT, Some("false"), || { + // No endpoint at all: forced on, the explicit false is ignored. + assert_eq!(OtelConfig::extract_otel_config_from_env(None).use_stdout, Some(true)); + }); + for key in [ENV_OBS_TRACE_ENDPOINT, ENV_OBS_METRIC_ENDPOINT, ENV_OBS_LOG_ENDPOINT] { + temp_env::with_var(key, Some("http://localhost:4318/v1/signal"), || { + temp_env::with_var(ENV_OBS_USE_STDOUT, Some("false"), || { + assert_eq!(OtelConfig::extract_otel_config_from_env(None).use_stdout, Some(false), "{key}"); + }); + temp_env::with_var(ENV_OBS_USE_STDOUT, Some("true"), || { + assert_eq!(OtelConfig::extract_otel_config_from_env(None).use_stdout, Some(true), "{key}"); + }); + temp_env::with_var_unset(ENV_OBS_USE_STDOUT, || { + assert_eq!(OtelConfig::extract_otel_config_from_env(None).use_stdout, Some(USE_STDOUT), "{key}"); + }); + }); + } + temp_env::with_var(ENV_OBS_TRACE_ENDPOINT, Some(""), || { + temp_env::with_var(ENV_OBS_USE_STDOUT, Some("false"), || { + // An empty per-signal endpoint is not a configured endpoint. + assert_eq!(OtelConfig::extract_otel_config_from_env(None).use_stdout, Some(true)); + }); + }); + }, + ); + }); + } + #[test] fn stdout_mirror_environment_preserves_unset_true_and_false() { with_profiling_env_lock(|| { diff --git a/crates/obs/src/global.rs b/crates/obs/src/global.rs index 08f1e83b0..a413a5d01 100644 --- a/crates/obs/src/global.rs +++ b/crates/obs/src/global.rs @@ -260,9 +260,26 @@ mod tests { } #[test] - fn test_grafana_dashboard_deduplicates_cluster_bucket_usage() { - assert!(GRAFANA_DASHBOARD.contains("max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count")); - assert!(GRAFANA_DASHBOARD.contains("max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes")); + fn test_grafana_dashboard_scopes_and_deduplicates_cluster_bucket_usage() { + let dashboard: serde_json::Value = serde_json::from_str(GRAFANA_DASHBOARD).expect("dashboard must be valid JSON"); + for (title, metric) in [ + ("Bucket Objects", "rustfs_cluster_usage_buckets_objects_count"), + ("Bucket Size", "rustfs_cluster_usage_buckets_total_bytes"), + ] { + let panel = dashboard["panels"] + .as_array() + .expect("dashboard panels") + .iter() + .find(|panel| panel["title"] == title) + .expect("bucket usage panel"); + assert_eq!( + panel["targets"][0]["expr"], + format!( + "max by (rustfs_cluster_id, job, bucket) ({metric}{{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"}})" + ), + "{title} must select one cluster and deduplicate its reporting nodes" + ); + } assert!(!GRAFANA_DASHBOARD.contains("sum by (bucket) (rustfs_bucket_api_objects_total")); assert!(!GRAFANA_DASHBOARD.contains("sum by (bucket) (rustfs_bucket_api_usage_bytes")); } diff --git a/crates/obs/src/metrics/storage_snapshot.rs b/crates/obs/src/metrics/storage_snapshot.rs index 8977403e9..d3e1a74ee 100644 --- a/crates/obs/src/metrics/storage_snapshot.rs +++ b/crates/obs/src/metrics/storage_snapshot.rs @@ -408,6 +408,69 @@ mod tests { provider.shutdown().unwrap(); } + #[test] + fn actual_sdk_exports_bucket_size_and_object_count_as_current_gauges() { + use crate::metrics::collectors::{BucketUsageStats, collect_bucket_usage_metrics}; + + let (provider, exporter, mut snapshot) = setup_unscoped(); + let mut buckets = vec![ + BucketUsageStats { + bucket: "client-a".into(), + total_bytes: 4096, + objects_count: 3, + ..Default::default() + }, + BucketUsageStats { + bucket: "client-b".into(), + total_bytes: 8192, + objects_count: 7, + ..Default::default() + }, + BucketUsageStats { + bucket: "empty".into(), + ..Default::default() + }, + ]; + + for expected in [ + vec![("client-a", 4096.0, 3.0), ("client-b", 8192.0, 7.0), ("empty", 0.0, 0.0)], + vec![("client-a", 1024.0, 1.0), ("empty", 0.0, 0.0)], + ] { + let metrics = collect_bucket_usage_metrics(&buckets); + assert!( + metrics.iter().all(|metric| metric.metric_type == MetricType::Gauge), + "bucket inventory values must decrease without counter semantics" + ); + snapshot.replace(metrics).expect("publish current bucket usage"); + let points = flush(&provider, &exporter); + for (name, index) in [ + ("rustfs_cluster_usage_buckets_total_bytes", 0), + ("rustfs_cluster_usage_buckets_objects_count", 1), + ] { + let mut actual = points + .iter() + .filter(|point| point.name == name) + .map(|point| { + assert_eq!(point.scope, "service-test"); + assert_eq!(point.attributes.len(), 1, "usage is cluster-wide, not a per-node count"); + assert_eq!(point.attributes[0].key.as_str(), "bucket"); + (point.attributes[0].value.to_string(), point.value) + }) + .collect::>(); + actual.sort_by(|a, b| a.0.cmp(&b.0)); + let values = expected + .iter() + .map(|(bucket, bytes, objects)| (bucket.to_string(), if index == 0 { *bytes } else { *objects })) + .collect::>(); + assert_eq!(actual, values, "{name} must preserve each bucket's current value"); + } + buckets.remove(1); + buckets[0].total_bytes = 1024; + buckets[0].objects_count = 1; + } + provider.shutdown().expect("shut down bucket usage test provider"); + } + #[test] fn failed_or_stalled_collection_cannot_refresh_last_success_or_export_old_points() { let (provider, exporter, mut snapshot) = setup(); diff --git a/crates/obs/src/telemetry/resource.rs b/crates/obs/src/telemetry/resource.rs index 8a7d003ab..5f9c4aa33 100644 --- a/crates/obs/src/telemetry/resource.rs +++ b/crates/obs/src/telemetry/resource.rs @@ -16,9 +16,9 @@ //! //! A `Resource` describes the entity producing telemetry data. The resource //! built here includes the service name, service version, deployment -//! environment, the stable RustFS node identity, and the local machine IP -//! address so that data can be correlated across services in a distributed -//! system. +//! environment, the stable RustFS node identity, the process ID, and the local +//! machine IP address so telemetry series can be correlated across service +//! restarts and nodes. use crate::config::OtelConfig; use crate::node_identity::{RUSTFS_NODE_ATTRIBUTE, local_node_identity}; @@ -27,7 +27,7 @@ use opentelemetry_sdk::Resource; use opentelemetry_semantic_conventions::{ SCHEMA_URL, attribute::{ - DEPLOYMENT_ENVIRONMENT_NAME, NETWORK_LOCAL_ADDRESS, SERVICE_INSTANCE_ID as OTEL_SERVICE_INSTANCE_ID, + DEPLOYMENT_ENVIRONMENT_NAME, NETWORK_LOCAL_ADDRESS, PROCESS_PID, SERVICE_INSTANCE_ID as OTEL_SERVICE_INSTANCE_ID, SERVICE_VERSION as OTEL_SERVICE_VERSION, }, }; @@ -48,6 +48,8 @@ use std::borrow::Cow; /// - `network.local.address` — the primary local IP of the current host, /// useful as an operational fallback when the stable node name is not yet /// initialized. +/// - `process.pid` — distinguishes telemetry emitted by different processes +/// running on the same node. /// /// All attributes are attached to the resource using the semantic conventions /// schema URL to ensure compatibility with standard OTLP backends. @@ -70,6 +72,7 @@ pub(super) fn build_resource(config: &OtelConfig) -> Resource { KeyValue::new(RUSTFS_NODE_ATTRIBUTE, node_identity.clone()), KeyValue::new(OTEL_SERVICE_INSTANCE_ID, node_identity), KeyValue::new(NETWORK_LOCAL_ADDRESS, local_ip), + KeyValue::new(PROCESS_PID, i64::from(std::process::id())), ], SCHEMA_URL, ) @@ -117,6 +120,12 @@ mod tests { .map(|value| value.to_string()), Some("node1:9000".to_string()) ); + assert_eq!( + resource + .get(&Key::from_static_str(PROCESS_PID)) + .map(|value| value.to_string()), + Some(std::process::id().to_string()) + ); rustfs_common::set_global_local_node_name(&previous).await; } diff --git a/crates/policy/src/policy/action.rs b/crates/policy/src/policy/action.rs index d7b0452cd..2388272ce 100644 --- a/crates/policy/src/policy/action.rs +++ b/crates/policy/src/policy/action.rs @@ -447,6 +447,10 @@ pub enum AdminAction { ServiceFreezeAdminAction, #[strum(serialize = "admin:ConfigUpdate")] ConfigUpdateAdminAction, + #[strum(serialize = "admin:TLSCertificatePublish")] + TLSCertificatePublishAdminAction, + #[strum(serialize = "admin:TLSCertificateStatus")] + TLSCertificateStatusAdminAction, #[strum(serialize = "admin:CreateUser")] CreateUserAdminAction, #[strum(serialize = "admin:DeleteUser")] @@ -688,6 +692,8 @@ impl AdminAction { | AdminAction::ServiceStopAdminAction | AdminAction::ServiceFreezeAdminAction | AdminAction::ConfigUpdateAdminAction + | AdminAction::TLSCertificatePublishAdminAction + | AdminAction::TLSCertificateStatusAdminAction | AdminAction::CreateUserAdminAction | AdminAction::DeleteUserAdminAction | AdminAction::ListUsersAdminAction @@ -1027,6 +1033,34 @@ mod tests { assert!(!Action::AdminAction(get_action).is_match(&Action::AdminAction(set_action))); } + #[test] + fn tls_certificate_actions_preserve_least_privilege() { + let publish = Action::try_from("admin:TLSCertificatePublish").expect("parse certificate publish action"); + let status = Action::try_from("admin:TLSCertificateStatus").expect("parse certificate status action"); + for (action, name) in [ + (&publish, "admin:TLSCertificatePublish"), + (&status, "admin:TLSCertificateStatus"), + ] { + let Action::AdminAction(admin_action) = action else { + panic!("certificate policy must use an admin action"); + }; + assert!(admin_action.is_valid()); + assert_eq!(serde_json::to_string(action).expect("serialize action"), format!("\"{name}\"")); + } + assert!(!status.is_match(&publish)); + assert!(!publish.is_match(&status)); + let config = Action::AdminAction(AdminAction::ConfigUpdateAdminAction); + let server_info = Action::AdminAction(AdminAction::ServerInfoAdminAction); + assert!(!config.is_match(&publish)); + assert!(!server_info.is_match(&publish)); + let admin = Action::AdminAction(AdminAction::AllAdminActions); + assert!(admin.is_match(&publish)); + assert!(admin.is_match(&status)); + let s3 = Action::try_from("s3:*").expect("parse S3 wildcard"); + assert!(!s3.is_match(&publish)); + assert!(!s3.is_match(&status)); + } + #[test] fn test_table_catalog_admin_action_is_valid() { let get_action = AdminAction::try_from("admin:GetTableCatalog").expect("Should parse GetTableCatalog action"); diff --git a/crates/policy/src/policy/function/key_name.rs b/crates/policy/src/policy/function/key_name.rs index 9ead5ca31..807653e02 100644 --- a/crates/policy/src/policy/function/key_name.rs +++ b/crates/policy/src/policy/function/key_name.rs @@ -57,6 +57,7 @@ impl KeyName { KeyName::S3(S3KeyName::S3XAmzContentSha256), KeyName::S3(S3KeyName::S3LocationConstraint), KeyName::S3(S3KeyName::S3VersionId), + KeyName::S3(S3KeyName::S3ObjectCreationOperation), //aws KeyName::Aws(AwsKeyName::AWSReferer), KeyName::Aws(AwsKeyName::AWSSourceIP), @@ -121,12 +122,13 @@ impl KeyName { /// values for these keys. Without that check a caller could satisfy conditions such /// as `aws:userid` or `jwt:groups` simply by sending a header of the same name. /// - /// The `s3:` namespace is mixed: `s3:x-amz-*` keys mirror request headers by - /// design, while the rest (`s3:authType`, `s3:LocationConstraint`, ...) are - /// computed by the server. + /// The `s3:` namespace is mixed: `s3:x-amz-*`, `s3:if-match` and `s3:if-none-match` + /// mirror request headers by design, while the rest (`s3:authType`, + /// `s3:LocationConstraint`, ...) are computed by the server. pub fn is_server_derived(&self) -> bool { match self { KeyName::Aws(_) | KeyName::Jwt(_) | KeyName::Ldap(_) | KeyName::Sts(_) | KeyName::Svc(_) => true, + KeyName::S3(S3KeyName::S3IfMatch | S3KeyName::S3IfNoneMatch) => false, KeyName::S3(_) => !self.name().starts_with("x-amz-"), } } @@ -165,6 +167,9 @@ impl From<&KeyName> for &'static str { #[derive(Clone, EnumString, Debug, IntoStaticStr, Eq, PartialEq, Serialize, Deserialize)] #[serde(try_from = "&str", into = "&str")] pub enum S3KeyName { + #[strum(serialize = "s3:ObjectCreationOperation")] + S3ObjectCreationOperation, + #[strum(serialize = "s3:x-amz-copy-source")] S3XAmzCopySource, @@ -237,6 +242,12 @@ pub enum S3KeyName { #[strum(serialize = "s3:x-amz-grant-write-acp")] S3XAmzGrantWriteAcp, + #[strum(serialize = "s3:if-match")] + S3IfMatch, + + #[strum(serialize = "s3:if-none-match")] + S3IfNoneMatch, + #[strum(serialize = "s3:ExistingObjectTag")] S3ExistingObjectTag, #[strum(serialize = "s3:RequestObjectTagKeys")] @@ -391,6 +402,18 @@ mod tests { use serde::Deserialize; use test_case::test_case; + #[test] + fn object_creation_operation_is_registered_as_server_derived() { + let key = KeyName::try_from("s3:ObjectCreationOperation").expect("AWS object creation condition key should parse"); + assert_eq!(key.name(), "ObjectCreationOperation"); + assert!(key.is_server_derived()); + assert!(KeyName::server_derived_key_names().any(|name| name == "ObjectCreationOperation")); + assert_eq!( + serde_json::to_string(&key).expect("condition key should serialize"), + "\"s3:ObjectCreationOperation\"" + ); + } + #[test_case("s3:x-amz-copy-source", KeyName::S3(S3KeyName::S3XAmzCopySource))] #[test_case("s3:VersionId", KeyName::S3(S3KeyName::S3VersionId) ; "aws_version_id")] #[test_case("s3:versionid", KeyName::S3(S3KeyName::S3VersionId) ; "minio_version_id")] @@ -400,6 +423,8 @@ mod tests { "s3:object-lock-remaining-retention-days", KeyName::S3(S3KeyName::S3ObjectLockRemainingRetentionDays) )] + #[test_case("s3:if-match", KeyName::S3(S3KeyName::S3IfMatch))] + #[test_case("s3:if-none-match", KeyName::S3(S3KeyName::S3IfNoneMatch))] #[test_case("aws:SecureTransport", KeyName::Aws(AwsKeyName::AWSSecureTransport))] #[test_case("jwt:sub", KeyName::Jwt(JwtKeyName::JWTSub))] #[test_case("ldap:user", KeyName::Ldap(LdapKeyName::User))] @@ -428,6 +453,8 @@ mod tests { "s3:object-lock-remaining-retention-days", KeyName::S3(S3KeyName::S3ObjectLockRemainingRetentionDays) )] + #[test_case("s3:if-match", KeyName::S3(S3KeyName::S3IfMatch))] + #[test_case("s3:if-none-match", KeyName::S3(S3KeyName::S3IfNoneMatch))] #[test_case("aws:SecureTransport", KeyName::Aws(AwsKeyName::AWSSecureTransport))] #[test_case("jwt:sub", KeyName::Jwt(JwtKeyName::JWTSub))] #[test_case("ldap:user", KeyName::Ldap(LdapKeyName::User))] @@ -452,6 +479,8 @@ mod tests { "s3:object-lock-remaining-retention-days", KeyName::S3(S3KeyName::S3ObjectLockRemainingRetentionDays) )] + #[test_case("s3:if-match", KeyName::S3(S3KeyName::S3IfMatch))] + #[test_case("s3:if-none-match", KeyName::S3(S3KeyName::S3IfNoneMatch))] #[test_case("aws:SecureTransport", KeyName::Aws(AwsKeyName::AWSSecureTransport))] #[test_case("jwt:sub", KeyName::Jwt(JwtKeyName::JWTSub))] #[test_case("ldap:user", KeyName::Ldap(LdapKeyName::User))] @@ -468,6 +497,14 @@ mod tests { assert_eq!(data, except); } + #[test_case("s3:if-match", "if-match")] + #[test_case("s3:if-none-match", "if-none-match")] + fn conditional_write_key_names_match_request_header_names(key: &str, header: &str) { + let key_name = KeyName::try_from(key).expect("conditional write key must parse"); + assert_eq!(key_name.name(), header); + assert!(!key_name.is_server_derived(), "{key} is filled from a request header"); + } + #[test] fn key_name_from_str_supports_jwt_roles() { assert!(KeyName::try_from("jwt:roles").is_ok()); diff --git a/crates/policy/src/policy/resource.rs b/crates/policy/src/policy/resource.rs index 262b13f4e..6d203accc 100644 --- a/crates/policy/src/policy/resource.rs +++ b/crates/policy/src/policy/resource.rs @@ -38,11 +38,7 @@ impl Serialize for ResourceSet { let mut seq = serializer.serialize_seq(Some(self.0.len()))?; for resource in &self.0 { - let resource_str = match resource { - Resource::S3(value) => format!("{}{}", Resource::S3_PREFIX, value), - Resource::Kms(value) => format!("{}{}", Resource::KMS_PREFIX, value), - }; - seq.serialize_element(&resource_str)?; + seq.serialize_element(resource)?; } seq.end() } @@ -165,6 +161,7 @@ impl<'de> Deserialize<'de> for ResourceSet { #[derive(Hash, Eq, PartialEq, Clone, Debug)] pub enum Resource { + All, S3(String), Kms(String), } @@ -197,6 +194,7 @@ impl Resource { resolver: Option<&dyn PolicyVariableResolver>, ) -> bool { let pattern = match self { + Resource::All => return true, Resource::S3(s) => s.to_owned(), Resource::Kms(s) => s.to_owned(), }; @@ -242,7 +240,9 @@ impl Resource { impl TryFrom<&str> for Resource { type Error = Error; fn try_from(value: &str) -> std::result::Result { - let resource = if let Some(suffix) = value.strip_prefix(Self::S3_PREFIX) { + let resource = if value == "*" { + Resource::All + } else if let Some(suffix) = value.strip_prefix(Self::S3_PREFIX) { Resource::S3(suffix.into()) } else if let Some(suffix) = value.strip_prefix(Self::KMS_PREFIX) { Resource::Kms(suffix.into()) @@ -259,6 +259,7 @@ impl Validator for Resource { type Error = Error; fn is_valid(&self) -> std::result::Result<(), Error> { match self { + Self::All => {} Self::S3(pattern) => { if pattern.is_empty() || pattern.starts_with('/') { return Err(IamError::InvalidResource("s3".into(), pattern.into()).into()); @@ -290,6 +291,7 @@ impl Serialize for Resource { S: serde::Serializer, { match self { + Resource::All => serializer.serialize_str("*"), Resource::S3(s) => serializer.serialize_str(&format!("{}{}", Self::S3_PREFIX, s)), Resource::Kms(s) => serializer.serialize_str(&format!("{}{}", Self::KMS_PREFIX, s)), } @@ -312,6 +314,10 @@ mod tests { use std::collections::HashMap; use test_case::test_case; + #[test_case("*", "mybucket" => true; "all buckets")] + #[test_case("*", "mybucket/myobject" => true; "all objects")] + #[test_case("*", "key/mykey" => true; "all keys")] + #[test_case("*", "" => true; "unscoped action")] #[test_case("arn:aws:s3:::*","mybucket" => true; "1")] #[test_case("arn:aws:s3:::*","mybucket/myobject" => true; "2")] #[test_case("arn:aws:s3:::mybucket*","mybucket" => true; "3")] @@ -346,6 +352,32 @@ mod tests { pollster::block_on(resource.is_match(object, &HashMap::new())) } + #[test] + fn test_wildcard_resource_serialization_round_trip() { + use crate::policy::resource::ResourceSet; + + let resource: Resource = serde_json::from_str(r#""*""#).expect("wildcard resource should parse"); + assert_eq!(serde_json::to_string(&resource).expect("resource should serialize"), r#""*""#); + + for json in [r#""*""#, r#"["*"]"#, r#"["*","*"]"#] { + let set: ResourceSet = serde_json::from_str(json).expect("wildcard resource set should parse"); + assert_eq!(set.len(), 1, "duplicate wildcard resources should be removed"); + let serialized = serde_json::to_string(&set).expect("resource set should serialize"); + assert_eq!(serialized, r#"["*"]"#); + let round_trip: ResourceSet = serde_json::from_str(&serialized).expect("resource set should parse again"); + assert_eq!(round_trip, set); + } + } + + #[test_case(""; "empty resource")] + #[test_case("**"; "multiple wildcards")] + #[test_case("*/*"; "unqualified path")] + #[test_case(" *"; "leading whitespace")] + #[test_case("arn:aws:unknown:::*"; "unknown service")] + fn test_invalid_resource_is_rejected(resource: &str) { + assert!(Resource::try_from(resource).is_err(), "invalid resource must not become a wildcard"); + } + #[test_case("arn:aws:kms:::key/mykey" => true; "key id parses")] #[test_case("arn:aws:kms:::key/*" => true; "key wildcard parses")] #[test_case("arn:aws:kms:::key/app-key.v2" => true; "key id with dot parses")] diff --git a/crates/policy/src/policy/statement.rs b/crates/policy/src/policy/statement.rs index 961d6e2eb..26e51b57c 100644 --- a/crates/policy/src/policy/statement.rs +++ b/crates/policy/src/policy/statement.rs @@ -182,15 +182,24 @@ impl Statement { /// left empty. An empty `args.object` means the caller did not scope the request to a /// key, which preserves the legacy match-every-key behaviour. async fn kms_key_scope_matches(&self, args: &Args<'_>, resolver: &VariableResolver) -> bool { + if self.not_resources.iter().any(|resource| matches!(resource, Resource::All)) { + return false; + } + if matches!(args.action, Action::KmsAction(KmsAction::BackupAction | KmsAction::RestoreAction)) - && (!self.resources.is_empty() || !self.not_resources.is_empty()) + && ((!self.resources.is_empty() && !self.resources.iter().any(|resource| matches!(resource, Resource::All))) + || !self.not_resources.is_empty()) { // Global bundle operations require an unscoped Allow, while a Deny // covering any key must still block an operation covering every key. return matches!(self.effect, Effect::Deny); } - let kms_resources: Vec<&Resource> = self.resources.iter().filter(|resource| resource.is_kms()).collect(); + let kms_resources: Vec<&Resource> = self + .resources + .iter() + .filter(|resource| matches!(resource, Resource::All | Resource::Kms(_))) + .collect(); let kms_not_resources: Vec<&Resource> = self.not_resources.iter().filter(|resource| resource.is_kms()).collect(); if kms_resources.len() != self.resources.len() || kms_not_resources.len() != self.not_resources.len() { diff --git a/crates/policy/tests/policy_is_allowed.rs b/crates/policy/tests/policy_is_allowed.rs index 868f25072..d29f36438 100644 --- a/crates/policy/tests/policy_is_allowed.rs +++ b/crates/policy/tests/policy_is_allowed.rs @@ -808,3 +808,215 @@ fn merge_keeps_allow_statements_differing_only_in_not_resource() { "both Allow statements must survive the merge; neither may be dropped as a duplicate" ); } + +#[test] +fn wildcard_deny_only_session_grants_no_storage_access() { + let policy = Policy::parse_config( + br#"{"Statement":[{"Action":["s3:GetObject"],"Effect":"Deny","Resource":"*","Sid":"1"}],"Version":"2012-10-17"}"#, + ) + .expect("deny-only session policy should parse"); + let encoded = serde_json::to_vec(&policy).expect("session policy should serialize into claims"); + let decoded = Policy::parse_config(&encoded).expect("session policy should parse from claims"); + for action in ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"] { + for bucket in ["bucket", "other-bucket"] { + assert!( + !policy_is_allowed( + decoded.clone(), + ArgsBuilder { + action: action.into(), + bucket: bucket.into(), + object: "data/object".into(), + ..Default::default() + } + ), + "deny-only session must not grant {action} on {bucket}" + ); + } + } +} + +#[test_case("s3:GetObject", "bucket", "object" => true; "matching object action")] +#[test_case("s3:PutObject", "bucket", "object" => false; "ungranted action")] +#[test_case("s3:GetObject", "other-bucket", "object" => true; "other bucket")] +fn wildcard_resource_allow_preserves_action_scope(action: &str, bucket: &str, object: &str) -> bool { + let policy = Policy::parse_config( + br#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"s3:GetObject","Resource":["*"]}]}"#, + ) + .expect("wildcard allow policy should parse"); + policy_is_allowed( + policy, + ArgsBuilder { + action: action.into(), + bucket: bucket.into(), + object: object.into(), + ..Default::default() + }, + ) +} + +#[test_case("s3:GetObject", "bucket", "object"; "object")] +#[test_case("s3:ListBucket", "bucket", ""; "bucket")] +#[test_case("s3:ListAllMyBuckets", "", ""; "all buckets")] +#[test_case("kms:Decrypt", "", "mykey"; "kms key")] +#[test_case("kms:Backup", "", ""; "kms backup")] +#[test_case("kms:Restore", "", ""; "kms restore")] +fn wildcard_resource_allow_deny_and_exclusion(action: &str, bucket: &str, object: &str) { + let other_resource = if action.starts_with("kms:") { + "arn:aws:kms:::key/other-key" + } else { + "arn:aws:s3:::other-bucket/*" + }; + for (field, resource) in [ + ("Resource", serde_json::json!("*")), + ("Resource", serde_json::json!(["*"])), + ("Resource", serde_json::json!([other_resource, "*"])), + ("NotResource", serde_json::json!("*")), + ("NotResource", serde_json::json!([other_resource, "*"])), + ] { + let mut json = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [{"Effect": "Allow", "Action": action, field: resource}] + }); + let allow = Policy::parse_config(&serde_json::to_vec(&json).expect("test policy should serialize")) + .expect("wildcard policy should parse"); + let args = ArgsBuilder { + action: action.into(), + bucket: bucket.into(), + object: object.into(), + ..Default::default() + }; + assert_eq!(policy_is_allowed(allow, args), field == "Resource", "{action} with {field}"); + + json["Statement"][0]["Effect"] = serde_json::json!("Deny"); + json["Statement"] + .as_array_mut() + .expect("statements are an array") + .push(serde_json::json!({ + "Effect": "Allow", "Action": action, "Resource": "*" + })); + let deny = Policy::parse_config(&serde_json::to_vec(&json).expect("test policy should serialize")) + .expect("wildcard deny policy should parse"); + assert_eq!( + policy_is_allowed( + deny, + ArgsBuilder { + action: action.into(), + bucket: bucket.into(), + object: object.into(), + ..Default::default() + } + ), + field == "NotResource", + "explicit deny must win unless {field} excludes the resource for {action}" + ); + } +} + +#[test_case(Some("allowed/data/") => true; "matching prefix")] +#[test_case(Some("other/data/") => false; "outside allowed prefix")] +#[test_case(None => false; "missing prefix")] +fn wildcard_resource_preserves_conditions(prefix: Option<&str>) -> bool { + let policy = Policy::parse_config( + br#"{"Version":"2012-10-17","Statement":[{ + "Effect":"Allow","Action":"s3:ListBucket","Resource":"*", + "Condition":{"StringLike":{"s3:prefix":"allowed/*"}} + }]}"#, + ) + .expect("conditional wildcard policy should parse"); + let mut args = ArgsBuilder { + action: "s3:ListBucket".into(), + bucket: "bucket".into(), + ..Default::default() + }; + if let Some(prefix) = prefix { + args.conditions.insert("prefix".into(), vec![prefix.into()]); + } + policy_is_allowed(policy, args) +} + +// Conditional write enforcement (`s3:if-none-match` / `s3:if-match`). Condition values +// carry request headers under their lowercase header name; a missing header leaves +// the key absent, which `Null: true` matches and `StringEquals` never satisfies. + +fn header_conditions(headers: &[(&str, &str)]) -> HashMap> { + headers + .iter() + .map(|(name, value)| ((*name).to_string(), vec![(*value).to_string()])) + .collect() +} + +fn is_allowed_put_object(policy: &Policy, conditions: &HashMap>) -> bool { + let claims: HashMap = HashMap::new(); + let groups: Option> = None; + pollster::block_on(policy.is_allowed(&Args { + account: "Q3AM3UQ867SPQQA43P2F", + groups: &groups, + action: rustfs_policy::policy::action::Action::S3Action(PutObjectAction), + bucket: "mybucket", + conditions, + is_owner: false, + object: "uploads/report.csv", + claims: &claims, + deny_only: false, + })) +} + +fn put_object_policy_with_condition(condition: &str) -> Policy { + let json = format!( + r#"{{"Version":"2012-10-17","Statement":[{{"Effect":"Allow","Action":["s3:PutObject"],"Resource":["arn:aws:s3:::mybucket/*"],"Condition":{condition}}}]}}"# + ); + Policy::parse_config(json.as_bytes()).expect("policy with conditional write key should parse") +} + +#[test_case(r#"{"StringEquals":{"s3:if-none-match":"*"}}"#, &[("if-none-match", "*")] => true ; "if_none_match_string_equals_header_present")] +#[test_case(r#"{"StringEquals":{"s3:if-none-match":"*"}}"#, &[] => false ; "if_none_match_string_equals_header_absent")] +#[test_case(r#"{"StringEquals":{"s3:if-none-match":"*"}}"#, &[("if-match", "\"etag\"")] => false ; "if_none_match_string_equals_other_header")] +#[test_case(r#"{"Null":{"s3:if-none-match":"false"}}"#, &[("if-none-match", "*")] => true ; "if_none_match_not_null_header_present")] +#[test_case(r#"{"Null":{"s3:if-none-match":"false"}}"#, &[] => false ; "if_none_match_not_null_header_absent")] +#[test_case(r#"{"Null":{"s3:if-match":"false"}}"#, &[("if-match", "\"etag\"")] => true ; "if_match_not_null_header_present")] +#[test_case(r#"{"Null":{"s3:if-match":"false"}}"#, &[] => false ; "if_match_not_null_header_absent")] +#[test_case(r#"{"StringEquals":{"s3:if-match":"\"etag\""}}"#, &[("if-match", "\"etag\"")] => true ; "if_match_string_equals_matching_value")] +#[test_case(r#"{"StringEquals":{"s3:if-match":"\"etag\""}}"#, &[("if-match", "\"other\"")] => false ; "if_match_string_equals_different_value")] +fn conditional_write_condition_keys(condition: &str, headers: &[(&str, &str)]) -> bool { + let policy = put_object_policy_with_condition(condition); + is_allowed_put_object(&policy, &header_conditions(headers)) +} + +/// Mirrors the AWS deny-based enforcement example: allow PutObject, but deny it +/// when the `If-Match` header is missing. +#[test_case(&[("if-match", "\"etag\"")] => true ; "header_present")] +#[test_case(&[] => false ; "header_absent")] +fn bucket_policy_denies_put_object_without_if_match(headers: &[(&str, &str)]) -> bool { + let policy: BucketPolicy = serde_json::from_str( + r#"{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": {"AWS": ["*"]}, + "Action": ["s3:PutObject"], + "Resource": ["arn:aws:s3:::mybucket/*"] + }, + { + "Effect": "Deny", + "Principal": {"AWS": ["*"]}, + "Action": ["s3:PutObject"], + "Resource": ["arn:aws:s3:::mybucket/*"], + "Condition": {"Null": {"s3:if-match": "true"}} + } + ] + }"#, + ) + .expect("bucket policy with conditional write key should parse"); + let conditions = header_conditions(headers); + let groups: Option> = None; + pollster::block_on(policy.is_allowed(&BucketPolicyArgs { + account: "Q3AM3UQ867SPQQA43P2F", + groups: &groups, + action: rustfs_policy::policy::action::Action::S3Action(PutObjectAction), + bucket: "mybucket", + conditions: &conditions, + is_owner: false, + object: "uploads/report.csv", + })) +} diff --git a/crates/protos/src/generated/proto_gen/node_service.rs b/crates/protos/src/generated/proto_gen/node_service.rs index 75b1ff898..1ea00d610 100644 --- a/crates/protos/src/generated/proto_gen/node_service.rs +++ b/crates/protos/src/generated/proto_gen/node_service.rs @@ -571,6 +571,29 @@ pub struct DeletePathsResponse { #[prost(message, optional, tag = "2")] pub error: ::core::option::Option, } +/// Protocol 1 is additive and has no fallback to generic file mutation RPCs. +#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] +pub struct RecoverDeleteTransactionRequest { + #[prost(string, tag = "1")] + pub disk: ::prost::alloc::string::String, + #[prost(string, tag = "2")] + pub volume: ::prost::alloc::string::String, + #[prost(uint32, tag = "3")] + pub protocol_version: u32, + #[prost(bytes = "bytes", tag = "4")] + pub operation: ::prost::bytes::Bytes, +} +#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] +pub struct RecoverDeleteTransactionResponse { + #[prost(bool, tag = "1")] + pub success: bool, + #[prost(uint32, tag = "2")] + pub protocol_version: u32, + #[prost(bytes = "bytes", tag = "3")] + pub result: ::prost::bytes::Bytes, + #[prost(message, optional, tag = "4")] + pub error: ::core::option::Option, +} #[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] pub struct SnapshotLeaseRequest { #[prost(string, tag = "1")] @@ -806,6 +829,18 @@ pub struct DeleteVersionsRequest { #[prost(bytes = "bytes", tag = "6")] pub opts_bin: ::prost::bytes::Bytes, } +/// Additive, fail-closed conditional batch protocol. Probe requests never mutate disk state. +#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] +pub struct GuardedDeleteVersionsRequest { + #[prost(string, tag = "1")] + pub disk: ::prost::alloc::string::String, + #[prost(string, tag = "2")] + pub volume: ::prost::alloc::string::String, + #[prost(bytes = "bytes", tag = "3")] + pub payload: ::prost::bytes::Bytes, + #[prost(bool, tag = "4")] + pub probe_only: bool, +} #[derive(Clone, PartialEq, ::prost::Message)] pub struct DeleteVersionsResponse { #[prost(bool, tag = "1")] @@ -2126,6 +2161,21 @@ pub mod node_service_client { .insert(GrpcMethod::new("node_service.NodeService", "DeleteAtIncarnation")); self.inner.unary(req, path, codec).await } + pub async fn recover_delete_transaction( + &mut self, + request: impl tonic::IntoRequest, + ) -> std::result::Result, tonic::Status> { + self.inner + .ready() + .await + .map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?; + let codec = tonic_prost::ProstCodec::default(); + let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/RecoverDeleteTransaction"); + let mut req = request.into_request(); + req.extensions_mut() + .insert(GrpcMethod::new("node_service.NodeService", "RecoverDeleteTransaction")); + self.inner.unary(req, path, codec).await + } pub async fn acquire_snapshot_lease( &mut self, request: impl tonic::IntoRequest, @@ -2141,6 +2191,21 @@ pub mod node_service_client { .insert(GrpcMethod::new("node_service.NodeService", "AcquireSnapshotLease")); self.inner.unary(req, path, codec).await } + pub async fn acquire_quota_mutation_fence( + &mut self, + request: impl tonic::IntoRequest, + ) -> std::result::Result, tonic::Status> { + self.inner + .ready() + .await + .map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?; + let codec = tonic_prost::ProstCodec::default(); + let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/AcquireQuotaMutationFence"); + let mut req = request.into_request(); + req.extensions_mut() + .insert(GrpcMethod::new("node_service.NodeService", "AcquireQuotaMutationFence")); + self.inner.unary(req, path, codec).await + } pub async fn renew_snapshot_lease( &mut self, request: impl tonic::IntoRequest, @@ -2171,6 +2236,21 @@ pub mod node_service_client { .insert(GrpcMethod::new("node_service.NodeService", "ReleaseSnapshotLease")); self.inner.unary(req, path, codec).await } + pub async fn release_quota_mutation_fence( + &mut self, + request: impl tonic::IntoRequest, + ) -> std::result::Result, tonic::Status> { + self.inner + .ready() + .await + .map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?; + let codec = tonic_prost::ProstCodec::default(); + let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/ReleaseQuotaMutationFence"); + let mut req = request.into_request(); + req.extensions_mut() + .insert(GrpcMethod::new("node_service.NodeService", "ReleaseQuotaMutationFence")); + self.inner.unary(req, path, codec).await + } pub async fn verify_file( &mut self, request: impl tonic::IntoRequest, @@ -2457,6 +2537,21 @@ pub mod node_service_client { .insert(GrpcMethod::new("node_service.NodeService", "SealArtifactAttemptPuts")); self.inner.unary(req, path, codec).await } + pub async fn rename_data_with_quota_fence( + &mut self, + request: impl tonic::IntoRequest, + ) -> std::result::Result, tonic::Status> { + self.inner + .ready() + .await + .map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?; + let codec = tonic_prost::ProstCodec::default(); + let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/RenameDataWithQuotaFence"); + let mut req = request.into_request(); + req.extensions_mut() + .insert(GrpcMethod::new("node_service.NodeService", "RenameDataWithQuotaFence")); + self.inner.unary(req, path, codec).await + } pub async fn make_volumes( &mut self, request: impl tonic::IntoRequest, @@ -2697,6 +2792,21 @@ pub mod node_service_client { .insert(GrpcMethod::new("node_service.NodeService", "DeleteVersions")); self.inner.unary(req, path, codec).await } + pub async fn guarded_delete_versions( + &mut self, + request: impl tonic::IntoRequest, + ) -> std::result::Result, tonic::Status> { + self.inner + .ready() + .await + .map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?; + let codec = tonic_prost::ProstCodec::default(); + let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/GuardedDeleteVersions"); + let mut req = request.into_request(); + req.extensions_mut() + .insert(GrpcMethod::new("node_service.NodeService", "GuardedDeleteVersions")); + self.inner.unary(req, path, codec).await + } pub async fn read_multiple( &mut self, request: impl tonic::IntoRequest, @@ -3582,10 +3692,18 @@ pub mod node_service_server { &self, request: tonic::Request, ) -> std::result::Result, tonic::Status>; + async fn recover_delete_transaction( + &self, + request: tonic::Request, + ) -> std::result::Result, tonic::Status>; async fn acquire_snapshot_lease( &self, request: tonic::Request, ) -> std::result::Result, tonic::Status>; + async fn acquire_quota_mutation_fence( + &self, + request: tonic::Request, + ) -> std::result::Result, tonic::Status>; async fn renew_snapshot_lease( &self, request: tonic::Request, @@ -3594,6 +3712,10 @@ pub mod node_service_server { &self, request: tonic::Request, ) -> std::result::Result, tonic::Status>; + async fn release_quota_mutation_fence( + &self, + request: tonic::Request, + ) -> std::result::Result, tonic::Status>; async fn verify_file( &self, request: tonic::Request, @@ -3683,6 +3805,10 @@ pub mod node_service_server { &self, request: tonic::Request, ) -> std::result::Result, tonic::Status>; + async fn rename_data_with_quota_fence( + &self, + request: tonic::Request, + ) -> std::result::Result, tonic::Status>; async fn make_volumes( &self, request: tonic::Request, @@ -3747,6 +3873,10 @@ pub mod node_service_server { &self, request: tonic::Request, ) -> std::result::Result, tonic::Status>; + async fn guarded_delete_versions( + &self, + request: tonic::Request, + ) -> std::result::Result, tonic::Status>; async fn read_multiple( &self, request: tonic::Request, @@ -4376,6 +4506,34 @@ pub mod node_service_server { }; Box::pin(fut) } + "/node_service.NodeService/RecoverDeleteTransaction" => { + #[allow(non_camel_case_types)] + struct RecoverDeleteTransactionSvc(pub Arc); + impl tonic::server::UnaryService for RecoverDeleteTransactionSvc { + type Response = super::RecoverDeleteTransactionResponse; + type Future = BoxFuture, tonic::Status>; + fn call(&mut self, request: tonic::Request) -> Self::Future { + let inner = Arc::clone(&self.0); + let fut = async move { ::recover_delete_transaction(&inner, request).await }; + Box::pin(fut) + } + } + let accept_compression_encodings = self.accept_compression_encodings; + let send_compression_encodings = self.send_compression_encodings; + let max_decoding_message_size = self.max_decoding_message_size; + let max_encoding_message_size = self.max_encoding_message_size; + let inner = self.inner.clone(); + let fut = async move { + let method = RecoverDeleteTransactionSvc(inner); + let codec = tonic_prost::ProstCodec::default(); + let mut grpc = tonic::server::Grpc::new(codec) + .apply_compression_config(accept_compression_encodings, send_compression_encodings) + .apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size); + let res = grpc.unary(method, req).await; + Ok(res) + }; + Box::pin(fut) + } "/node_service.NodeService/AcquireSnapshotLease" => { #[allow(non_camel_case_types)] struct AcquireSnapshotLeaseSvc(pub Arc); @@ -4404,6 +4562,34 @@ pub mod node_service_server { }; Box::pin(fut) } + "/node_service.NodeService/AcquireQuotaMutationFence" => { + #[allow(non_camel_case_types)] + struct AcquireQuotaMutationFenceSvc(pub Arc); + impl tonic::server::UnaryService for AcquireQuotaMutationFenceSvc { + type Response = super::SnapshotLeaseResponse; + type Future = BoxFuture, tonic::Status>; + fn call(&mut self, request: tonic::Request) -> Self::Future { + let inner = Arc::clone(&self.0); + let fut = async move { ::acquire_quota_mutation_fence(&inner, request).await }; + Box::pin(fut) + } + } + let accept_compression_encodings = self.accept_compression_encodings; + let send_compression_encodings = self.send_compression_encodings; + let max_decoding_message_size = self.max_decoding_message_size; + let max_encoding_message_size = self.max_encoding_message_size; + let inner = self.inner.clone(); + let fut = async move { + let method = AcquireQuotaMutationFenceSvc(inner); + let codec = tonic_prost::ProstCodec::default(); + let mut grpc = tonic::server::Grpc::new(codec) + .apply_compression_config(accept_compression_encodings, send_compression_encodings) + .apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size); + let res = grpc.unary(method, req).await; + Ok(res) + }; + Box::pin(fut) + } "/node_service.NodeService/RenewSnapshotLease" => { #[allow(non_camel_case_types)] struct RenewSnapshotLeaseSvc(pub Arc); @@ -4460,6 +4646,34 @@ pub mod node_service_server { }; Box::pin(fut) } + "/node_service.NodeService/ReleaseQuotaMutationFence" => { + #[allow(non_camel_case_types)] + struct ReleaseQuotaMutationFenceSvc(pub Arc); + impl tonic::server::UnaryService for ReleaseQuotaMutationFenceSvc { + type Response = super::SnapshotLeaseMutationResponse; + type Future = BoxFuture, tonic::Status>; + fn call(&mut self, request: tonic::Request) -> Self::Future { + let inner = Arc::clone(&self.0); + let fut = async move { ::release_quota_mutation_fence(&inner, request).await }; + Box::pin(fut) + } + } + let accept_compression_encodings = self.accept_compression_encodings; + let send_compression_encodings = self.send_compression_encodings; + let max_decoding_message_size = self.max_decoding_message_size; + let max_encoding_message_size = self.max_encoding_message_size; + let inner = self.inner.clone(); + let fut = async move { + let method = ReleaseQuotaMutationFenceSvc(inner); + let codec = tonic_prost::ProstCodec::default(); + let mut grpc = tonic::server::Grpc::new(codec) + .apply_compression_config(accept_compression_encodings, send_compression_encodings) + .apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size); + let res = grpc.unary(method, req).await; + Ok(res) + }; + Box::pin(fut) + } "/node_service.NodeService/VerifyFile" => { #[allow(non_camel_case_types)] struct VerifyFileSvc(pub Arc); @@ -4997,6 +5211,34 @@ pub mod node_service_server { }; Box::pin(fut) } + "/node_service.NodeService/RenameDataWithQuotaFence" => { + #[allow(non_camel_case_types)] + struct RenameDataWithQuotaFenceSvc(pub Arc); + impl tonic::server::UnaryService for RenameDataWithQuotaFenceSvc { + type Response = super::RenameDataResponse; + type Future = BoxFuture, tonic::Status>; + fn call(&mut self, request: tonic::Request) -> Self::Future { + let inner = Arc::clone(&self.0); + let fut = async move { ::rename_data_with_quota_fence(&inner, request).await }; + Box::pin(fut) + } + } + let accept_compression_encodings = self.accept_compression_encodings; + let send_compression_encodings = self.send_compression_encodings; + let max_decoding_message_size = self.max_decoding_message_size; + let max_encoding_message_size = self.max_encoding_message_size; + let inner = self.inner.clone(); + let fut = async move { + let method = RenameDataWithQuotaFenceSvc(inner); + let codec = tonic_prost::ProstCodec::default(); + let mut grpc = tonic::server::Grpc::new(codec) + .apply_compression_config(accept_compression_encodings, send_compression_encodings) + .apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size); + let res = grpc.unary(method, req).await; + Ok(res) + }; + Box::pin(fut) + } "/node_service.NodeService/MakeVolumes" => { #[allow(non_camel_case_types)] struct MakeVolumesSvc(pub Arc); @@ -5445,6 +5687,34 @@ pub mod node_service_server { }; Box::pin(fut) } + "/node_service.NodeService/GuardedDeleteVersions" => { + #[allow(non_camel_case_types)] + struct GuardedDeleteVersionsSvc(pub Arc); + impl tonic::server::UnaryService for GuardedDeleteVersionsSvc { + type Response = super::DeleteVersionsResponse; + type Future = BoxFuture, tonic::Status>; + fn call(&mut self, request: tonic::Request) -> Self::Future { + let inner = Arc::clone(&self.0); + let fut = async move { ::guarded_delete_versions(&inner, request).await }; + Box::pin(fut) + } + } + let accept_compression_encodings = self.accept_compression_encodings; + let send_compression_encodings = self.send_compression_encodings; + let max_decoding_message_size = self.max_decoding_message_size; + let max_encoding_message_size = self.max_encoding_message_size; + let inner = self.inner.clone(); + let fut = async move { + let method = GuardedDeleteVersionsSvc(inner); + let codec = tonic_prost::ProstCodec::default(); + let mut grpc = tonic::server::Grpc::new(codec) + .apply_compression_config(accept_compression_encodings, send_compression_encodings) + .apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size); + let res = grpc.unary(method, req).await; + Ok(res) + }; + Box::pin(fut) + } "/node_service.NodeService/ReadMultiple" => { #[allow(non_camel_case_types)] struct ReadMultipleSvc(pub Arc); diff --git a/crates/protos/src/lib.rs b/crates/protos/src/lib.rs index 4be4d1059..d1f478fff 100644 --- a/crates/protos/src/lib.rs +++ b/crates/protos/src/lib.rs @@ -25,8 +25,8 @@ use rustfs_common::{cache_connection, cached_connection, evict_connection_with_l use std::{ collections::HashMap, error::Error, - sync::LazyLock, sync::atomic::{AtomicU8, AtomicUsize, Ordering}, + sync::{Arc, LazyLock, Mutex as StdMutex, Weak}, time::{Duration, Instant}, }; use tokio::sync::Mutex; @@ -59,6 +59,44 @@ const INTERNODE_RPC_MSGPACK_ONLY_CACHE_UNSET: u8 = 0; const INTERNODE_RPC_MSGPACK_ONLY_CACHE_FALSE: u8 = 1; const INTERNODE_RPC_MSGPACK_ONLY_CACHE_TRUE: u8 = 2; static TLS_GENERATION_CACHE: LazyLock>> = LazyLock::new(|| Mutex::new(HashMap::new())); +// Lock order: generation metadata, then the common connection cache. The dial +// registry's standard mutex is released before waiting on any async lock. +static DIAL_LOCK_REGISTRY: LazyLock>>>> = LazyLock::new(|| StdMutex::new(HashMap::new())); + +struct DialLock { + key: String, + mutex: Arc>, +} + +impl DialLock { + fn for_key(key: &str) -> Self { + let mut registry = DIAL_LOCK_REGISTRY.lock().unwrap_or_else(|poisoned| poisoned.into_inner()); + registry.retain(|_, lock| lock.strong_count() != 0); + let mutex = registry.get(key).and_then(Weak::upgrade).unwrap_or_else(|| { + let mutex = Arc::new(Mutex::new(())); + registry.insert(key.to_string(), Arc::downgrade(&mutex)); + mutex + }); + Self { + key: key.to_string(), + mutex, + } + } +} + +impl Drop for DialLock { + fn drop(&mut self) { + let mut registry = DIAL_LOCK_REGISTRY.lock().unwrap_or_else(|poisoned| poisoned.into_inner()); + if Arc::strong_count(&self.mutex) == 1 + && registry + .get(&self.key) + .is_some_and(|slot| Weak::ptr_eq(slot, &Arc::downgrade(&self.mutex))) + { + registry.remove(&self.key); + } + } +} + static INTERNODE_RPC_MSGPACK_ONLY_CACHE: AtomicU8 = AtomicU8::new(INTERNODE_RPC_MSGPACK_ONLY_CACHE_UNSET); fn enforce_tls_generation_cache_bound(generation_cache: &mut HashMap, generation: u64, addr: &str) { @@ -1134,6 +1172,20 @@ pub fn canonical_seal_artifact_attempt_repairs_response_body( Ok(body.finish()) } +pub const QUOTA_MUTATION_TOPOLOGY_HEADER: &str = "x-rustfs-quota-topology"; + +/// Binds a quota mutation to the coordinator's complete, ordered storage layout. +/// The inner canonical body retains its operation-specific domain and wire fields. +pub fn canonical_quota_mutation_request_body( + topology_fingerprint: &str, + mutation_body: &[u8], +) -> Result, std::num::TryFromIntError> { + let mut body = CanonicalBodyBuilder::new(b"rustfs-quota-mutation-topology-request-v1\0"); + body.push_str(topology_fingerprint)?; + body.push_bytes(mutation_body)?; + Ok(body.finish()) +} + pub fn canonical_delete_version_request_body( request: &proto_gen::node_service::DeleteVersionRequest, ) -> Result, std::num::TryFromIntError> { @@ -1171,6 +1223,17 @@ pub fn canonical_delete_versions_request_body( Ok(body.finish()) } +pub fn canonical_guarded_delete_versions_request_body( + request: &proto_gen::node_service::GuardedDeleteVersionsRequest, +) -> Result, std::num::TryFromIntError> { + let mut body = CanonicalBodyBuilder::new(b"rustfs-guarded-delete-versions-request-v1\0"); + body.push_str(&request.disk)?; + body.push_str(&request.volume)?; + body.push_bytes(&request.payload)?; + body.push_bool(request.probe_only); + Ok(body.finish()) +} + pub fn canonical_write_metadata_request_body( request: &proto_gen::node_service::WriteMetadataRequest, ) -> Result, std::num::TryFromIntError> { @@ -1334,6 +1397,37 @@ pub fn canonical_prepare_part_transaction_request_body( Ok(body.finish()) } +/// Recovery messages are bounded and must contain exactly one MessagePack value. +pub const DELETE_RECOVERY_MAX_PAYLOAD_BYTES: usize = DEFAULT_GRPC_SERVER_MESSAGE_LEN; +pub const DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES: usize = 1024 * 1024; + +pub fn decode_delete_recovery_payload(payload: &[u8]) -> std::io::Result { + if payload.is_empty() || payload.len() > DELETE_RECOVERY_MAX_PAYLOAD_BYTES { + return Err(std::io::Error::new( + std::io::ErrorKind::InvalidData, + "invalid delete recovery payload length", + )); + } + let mut cursor = std::io::Cursor::new(payload); + let value = rmp_serde::from_read(&mut cursor) + .map_err(|_| std::io::Error::new(std::io::ErrorKind::InvalidData, "invalid delete recovery payload"))?; + if cursor.position() != u64::try_from(payload.len()).map_err(std::io::Error::other)? { + return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, "trailing delete recovery payload")); + } + Ok(value) +} + +pub fn canonical_recover_delete_transaction_request_body( + request: &proto_gen::node_service::RecoverDeleteTransactionRequest, +) -> Result, std::num::TryFromIntError> { + let mut body = CanonicalBodyBuilder::new(b"rustfs-recover-delete-transaction-request-v1\0"); + body.push_str(&request.disk)?; + body.push_str(&request.volume)?; + body.push_u32(request.protocol_version); + body.push_bytes(&request.operation)?; + Ok(body.finish()) +} + pub fn canonical_settle_part_transaction_request_body( request: &proto_gen::node_service::SettlePartTransactionRequest, ) -> Result, std::num::TryFromIntError> { @@ -1388,6 +1482,92 @@ mod disk_mutation_canonical_tests { }; use super::*; + #[test] + fn quota_mutation_envelope_binds_topology_operation_and_field_boundaries() { + let acquire = canonical_snapshot_lease_request_body(&SnapshotLeaseRequest::default()).expect("acquire body"); + let release = + canonical_snapshot_lease_release_request_body(&SnapshotLeaseReleaseRequest::default()).expect("release body"); + let rename = canonical_rename_data_request_body(&RenameDataRequest::default()).expect("rename body"); + let topology = "a".repeat(64); + let envelope = canonical_quota_mutation_request_body(&topology, &acquire).expect("quota envelope"); + assert_ne!(envelope, acquire, "quota and ordinary mutation domains must differ"); + assert_all_distinct(&[ + envelope, + canonical_quota_mutation_request_body(&"b".repeat(64), &acquire).expect("changed topology"), + canonical_quota_mutation_request_body(&topology, &release).expect("release envelope"), + canonical_quota_mutation_request_body(&topology, &rename).expect("rename envelope"), + ]); + assert_ne!( + canonical_quota_mutation_request_body("ab", b"c").expect("first boundary"), + canonical_quota_mutation_request_body("a", b"bc").expect("second boundary"), + "topology and operation bytes must be independently length delimited" + ); + } + + #[test] + fn guarded_delete_digest_binds_probe_and_payload() { + let request = proto_gen::node_service::GuardedDeleteVersionsRequest { + disk: "disk-a".into(), + volume: "bucket-a".into(), + payload: vec![0x80].into(), + probe_only: false, + }; + let original = canonical_guarded_delete_versions_request_body(&request).expect("encode guarded body"); + let mut candidates = Vec::new(); + let mut changed = request.clone(); + changed.disk = "disk-b".into(); + candidates.push(changed); + let mut changed = request.clone(); + changed.volume = "bucket-b".into(); + candidates.push(changed); + let mut changed = request.clone(); + changed.payload = vec![0x81].into(); + candidates.push(changed); + let mut changed = request; + changed.probe_only = true; + candidates.push(changed); + for changed in candidates { + assert_ne!( + canonical_guarded_delete_versions_request_body(&changed).expect("encode changed body"), + original + ); + } + } + + #[test] + fn delete_recovery_canonical_body_binds_all_wire_fields() { + let request = proto_gen::node_service::RecoverDeleteTransactionRequest { + disk: "disk".into(), + volume: "volume".into(), + protocol_version: 1, + operation: vec![0x80].into(), + }; + let mut bodies = vec![canonical_recover_delete_transaction_request_body(&request).expect("canonical body")]; + for mutate in [ + |r: &mut proto_gen::node_service::RecoverDeleteTransactionRequest| r.disk = "other".into(), + |r: &mut proto_gen::node_service::RecoverDeleteTransactionRequest| r.volume = "other".into(), + |r: &mut proto_gen::node_service::RecoverDeleteTransactionRequest| r.protocol_version = 2, + |r: &mut proto_gen::node_service::RecoverDeleteTransactionRequest| r.operation = vec![0x81].into(), + ] { + let mut changed = request.clone(); + mutate(&mut changed); + bodies.push(canonical_recover_delete_transaction_request_body(&changed).expect("changed canonical body")); + } + assert_all_distinct(&bodies); + } + + #[test] + fn delete_recovery_payload_rejects_trailing_malformed_and_oversized_values() { + let payload = rmp_serde::to_vec_named(&vec![1_u32, 2]).expect("encode test payload"); + assert_eq!(decode_delete_recovery_payload::>(&payload).expect("one payload"), vec![1, 2]); + let mut trailing = payload; + trailing.push(0xc0); + assert!(decode_delete_recovery_payload::>(&trailing).is_err()); + assert!(decode_delete_recovery_payload::>(&[]).is_err()); + assert!(decode_delete_recovery_payload::>(&[0xc1]).is_err()); + assert!(decode_delete_recovery_payload::>(&vec![0xc0; DELETE_RECOVERY_MAX_PAYLOAD_BYTES + 1]).is_err()); + } + fn assert_all_distinct(bodies: &[Vec]) { for (i, a) in bodies.iter().enumerate() { for (j, b) in bodies.iter().enumerate() { @@ -3359,6 +3539,53 @@ fn bulk_cache_key(addr: &str, idx: usize) -> String { format!("{addr}\u{0}bulk\u{0}{idx}") } +async fn cached_channel_for_tls_generation(dial_addr: &str, cache_key: &str) -> Option { + if !dial_addr.starts_with(RUSTFS_HTTPS_PREFIX) { + return cached_connection(cache_key).await; + } + let mut generations = TLS_GENERATION_CACHE.lock().await; + let Some(channel) = cached_connection(cache_key).await else { + generations.remove(cache_key); + return None; + }; + if generations.get(cache_key).copied() == Some(runtime_sources::outbound_tls_generation()) { + return Some(channel); + } + evict_connection_with_log_level(cache_key, ConnectionEvictionLogLevel::Debug).await; + generations.remove(cache_key); + runtime_sources::record_stale_grpc_channel_tls_generation(); + None +} + +async fn cache_dialed_channel(dial_addr: &str, cache_key: &str, channel: &Channel, generation: Option) -> bool { + if !dial_addr.starts_with(RUSTFS_HTTPS_PREFIX) { + cache_connection(cache_key.to_string(), channel.clone()).await; + return true; + } + let mut generations = TLS_GENERATION_CACHE.lock().await; + if generation != Some(runtime_sources::outbound_tls_generation()) { + runtime_sources::record_stale_grpc_channel_tls_generation(); + return false; + } + cache_connection(cache_key.to_string(), channel.clone()).await; + // Publication can advance while the common cache's async lock is awaited. + if generation != Some(runtime_sources::outbound_tls_generation()) { + evict_connection_with_log_level(cache_key, ConnectionEvictionLogLevel::Debug).await; + generations.remove(cache_key); + runtime_sources::record_stale_grpc_channel_tls_generation(); + return false; + } + if let Some(generation) = generation { + enforce_tls_generation_cache_bound(&mut generations, generation, cache_key); + generations.insert(cache_key.to_string(), generation); + } + true +} + +fn channel_acquisition_timeout(_: tokio::time::error::Elapsed) -> std::io::Error { + std::io::Error::new(std::io::ErrorKind::TimedOut, "internode channel acquisition deadline elapsed") +} + /// Acquire a cached-or-newly-dialed channel for the given peer and channel class. /// /// For [`ChannelClass::Control`] (or whenever isolation is disabled) this is exactly the legacy @@ -3367,22 +3594,27 @@ fn bulk_cache_key(addr: &str, idx: usize) -> String { /// per-peer bulk pool and dialed on demand, physically isolating large transfers from /// control-plane RPCs. pub async fn get_channel_for_class(addr: &str, class: ChannelClass) -> Result> { - if class == ChannelClass::Control || !channel_isolation_enabled() { - if let Some(channel) = cached_connection(addr).await { - debug!("Using cached control gRPC channel for: {}", addr); + let deadline = tokio::time::Instant::now() + internode_connect_timeout(); + tokio::time::timeout_at(deadline, async { + if class == ChannelClass::Control || !channel_isolation_enabled() { + if let Some(channel) = cached_channel_for_tls_generation(addr, addr).await { + debug!("Using cached control gRPC channel for: {}", addr); + return Ok(channel); + } + return build_channel_until(addr, addr, deadline, internode_rpc_timeout()).await; + } + + let pool_size = bulk_channel_pool_size(); + let idx = BULK_CHANNEL_CURSOR.fetch_add(1, Ordering::Relaxed) % pool_size; + let cache_key = bulk_cache_key(addr, idx); + if let Some(channel) = cached_channel_for_tls_generation(addr, &cache_key).await { + debug!("Using cached bulk gRPC channel {} for: {}", idx, addr); return Ok(channel); } - return create_new_channel(addr).await; - } - - let pool_size = bulk_channel_pool_size(); - let idx = BULK_CHANNEL_CURSOR.fetch_add(1, Ordering::Relaxed) % pool_size; - let cache_key = bulk_cache_key(addr, idx); - if let Some(channel) = cached_connection(&cache_key).await { - debug!("Using cached bulk gRPC channel {} for: {}", idx, addr); - return Ok(channel); - } - build_channel(addr, &cache_key).await + build_channel_until(addr, &cache_key, deadline, internode_rpc_timeout()).await + }) + .await + .map_err(channel_acquisition_timeout)? } /// Creates a new gRPC channel with optimized keepalive settings for cluster resilience. @@ -3406,14 +3638,83 @@ pub async fn create_new_channel(addr: &str) -> Result> { /// for isolated bulk channels (see [`bulk_cache_key`]), letting several physically distinct /// channels to the same peer be cached independently. async fn build_channel(dial_addr: &str, cache_key: &str) -> Result> { + let deadline = tokio::time::Instant::now() + internode_connect_timeout(); + build_channel_until(dial_addr, cache_key, deadline, internode_rpc_timeout()).await +} + +/// A separate channel deadline for slow authenticated recovery work. The cache +/// key keeps this deadline from changing foreground lock, read, and write RPCs. +#[derive(Debug)] +pub enum DeleteRecoveryChannelError { + Connection(Box), +} + +impl std::fmt::Display for DeleteRecoveryChannelError { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Connection(error) => write!(formatter, "delete recovery channel connection failed: {error}"), + } + } +} + +impl Error for DeleteRecoveryChannelError { + fn source(&self) -> Option<&(dyn Error + 'static)> { + match self { + Self::Connection(error) => Some(error.as_ref()), + } + } +} + +pub async fn create_channel_with_timeout(addr: &str, rpc_timeout: Duration) -> Result { + let cache_key = format!("delete-recovery:{}:{addr}", rpc_timeout.as_millis()); + let deadline = tokio::time::Instant::now() + internode_connect_timeout(); + build_channel_until(addr, &cache_key, deadline, rpc_timeout) + .await + .map_err(DeleteRecoveryChannelError::Connection) +} + +async fn build_channel_until( + dial_addr: &str, + cache_key: &str, + deadline: tokio::time::Instant, + rpc_timeout: Duration, +) -> Result> { + tokio::time::timeout_at(deadline, async { + if let Some(channel) = cached_channel_for_tls_generation(dial_addr, cache_key).await { + return Ok(channel); + } + let slot = DialLock::for_key(cache_key); + let _guard = slot.mutex.lock().await; + loop { + // A successful concurrent miss has already populated this exact key. + if let Some(channel) = cached_channel_for_tls_generation(dial_addr, cache_key).await { + return Ok(channel); + } + let (channel, generation) = dial_channel(dial_addr, cache_key, rpc_timeout).await?; + if cache_dialed_channel(dial_addr, cache_key, &channel, generation).await + && let Some(channel) = cached_channel_for_tls_generation(dial_addr, cache_key).await + { + debug!( + "Successfully created and cached gRPC channel to: {} (cache key: {})", + dial_addr, cache_key + ); + return Ok(channel); + } + // A generation published during dial cannot be returned; rebuilding + // shares the original deadline with lock wait and all earlier dials. + } + }) + .await + .map_err(channel_acquisition_timeout)? +} + +async fn dial_channel(dial_addr: &str, cache_key: &str, rpc_timeout: Duration) -> Result<(Channel, Option), Box> { debug!("Creating new gRPC channel to: {} (cache key: {})", dial_addr, cache_key); let dial_started_at = Instant::now(); let connect_timeout = internode_connect_timeout(); let tcp_keepalive = internode_tcp_keepalive(); let http2_keepalive_interval = internode_http2_keep_alive_interval(); let http2_keepalive_timeout = internode_http2_keep_alive_timeout(); - let rpc_timeout = internode_rpc_timeout(); - let mut connector = Endpoint::from_shared(dial_addr.to_string())? // Fast connection timeout for dead peer detection .connect_timeout(connect_timeout) @@ -3441,18 +3742,10 @@ async fn build_channel(dial_addr: &str, cache_key: &str) -> Result Result = tokio::sync::Mutex::const_new(()); + static INTERNODE_RPC_MSGPACK_ONLY_ENV_LOCK: Mutex<()> = Mutex::new(()); #[derive(Clone, PartialEq, prost::Message)] @@ -3698,6 +3979,268 @@ mod tests { assert_eq!(manifest_fields, proto_bin_json_fields("Response")); } + #[tokio::test] + async fn tls02_stale_https_control_and_bulk_cache_entries_are_evicted() { + let _test_guard = TLS02_TEST_LOCK.lock().await; + let addr = "https://tls02-stale-cache.invalid:9000"; + let generation = rustfs_tls_runtime::load_global_outbound_tls_generation().0; + let stale_generation = generation.checked_add(1).unwrap_or(0); + for key in [addr.to_string(), bulk_cache_key(addr, 0)] { + cache_connection(key.clone(), Endpoint::from_static("http://127.0.0.1:1").connect_lazy()).await; + TLS_GENERATION_CACHE.lock().await.insert(key.clone(), stale_generation); + assert!( + cached_channel_for_tls_generation(addr, &key).await.is_none(), + "TLS02_INVARIANT stale HTTPS cache entry cannot be returned" + ); + assert!( + !rustfs_common::has_cached_connection(&key).await, + "TLS02_INVARIANT stale entry is removed from the real connection cache" + ); + } + } + + #[tokio::test] + async fn tls02_missing_https_generation_control_and_bulk_cache_entries_are_evicted() { + let _test_guard = TLS02_TEST_LOCK.lock().await; + let addr = "https://tls02-missing-generation.invalid:9000"; + for key in [addr.to_string(), bulk_cache_key(addr, 0)] { + cache_connection(key.clone(), Endpoint::from_static("http://127.0.0.1:1").connect_lazy()).await; + TLS_GENERATION_CACHE.lock().await.remove(&key); + assert!( + cached_channel_for_tls_generation(addr, &key).await.is_none(), + "TLS02_INVARIANT HTTPS channel without a generation cannot be returned" + ); + assert!( + !rustfs_common::has_cached_connection(&key).await, + "TLS02_INVARIANT unbound entry is removed from the real connection cache" + ); + } + } + + #[tokio::test] + async fn tls02_delete_recovery_channels_reject_unbound_and_stale_https_cache() { + let _test_guard = TLS02_TEST_LOCK.lock().await; + let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("reserve recovery peer address"); + let addr = format!("https://{}", listener.local_addr().expect("recovery peer address")); + drop(listener); + let rpc_timeout = Duration::from_secs(20); + let cache_key = format!("delete-recovery:{}:{addr}", rpc_timeout.as_millis()); + let generation = runtime_sources::outbound_tls_generation(); + for cached_generation in [None, Some(generation.checked_add(1).unwrap_or(0))] { + cache_connection(cache_key.clone(), Endpoint::from_static("http://127.0.0.1:1").connect_lazy()).await; + if let Some(generation) = cached_generation { + TLS_GENERATION_CACHE.lock().await.insert(cache_key.clone(), generation); + } else { + TLS_GENERATION_CACHE.lock().await.remove(&cache_key); + } + assert!( + create_channel_with_timeout(&addr, rpc_timeout).await.is_err(), + "recovery must dial the unavailable HTTPS peer instead of returning an unbound or stale cached channel" + ); + assert!(!rustfs_common::has_cached_connection(&cache_key).await); + assert!(!TLS_GENERATION_CACHE.lock().await.contains_key(&cache_key)); + } + } + + #[tokio::test] + async fn tls02_current_https_control_and_bulk_cache_entries_are_reused() { + let _test_guard = TLS02_TEST_LOCK.lock().await; + let addr = "https://tls02-current-generation.invalid:9000"; + let generation = rustfs_tls_runtime::load_global_outbound_tls_generation().0; + for key in [addr.to_string(), bulk_cache_key(addr, 0)] { + cache_connection(key.clone(), Endpoint::from_static("http://127.0.0.1:1").connect_lazy()).await; + TLS_GENERATION_CACHE.lock().await.insert(key.clone(), generation); + assert!( + cached_channel_for_tls_generation(addr, &key).await.is_some(), + "TLS02_INVARIANT HTTPS channel at the canonical generation is reused" + ); + assert!(rustfs_common::has_cached_connection(&key).await); + evict_connection_with_log_level(&key, ConnectionEvictionLogLevel::Debug).await; + TLS_GENERATION_CACHE.lock().await.remove(&key); + } + } + + #[tokio::test] + async fn tls02_http_control_and_bulk_cache_ignore_tls_generation() { + let _test_guard = TLS02_TEST_LOCK.lock().await; + let addr = "http://tls02-http-cache.invalid:9000"; + let generation = rustfs_tls_runtime::load_global_outbound_tls_generation().0; + for key in [addr.to_string(), bulk_cache_key(addr, 0)] { + cache_connection(key.clone(), Endpoint::from_static("http://127.0.0.1:1").connect_lazy()).await; + TLS_GENERATION_CACHE + .lock() + .await + .insert(key.clone(), generation.checked_add(1).unwrap_or(0)); + assert!( + cached_channel_for_tls_generation(addr, &key).await.is_some(), + "TLS02_INVARIANT HTTP cache is independent of TLS generations" + ); + TLS_GENERATION_CACHE.lock().await.remove(&key); + assert!( + cached_channel_for_tls_generation(addr, &key).await.is_some(), + "TLS02_INVARIANT HTTP cache needs no TLS generation metadata" + ); + evict_connection_with_log_level(&key, ConnectionEvictionLogLevel::Debug).await; + } + } + + #[tokio::test] + async fn tls02_captured_identity_is_immutable_and_late_dial_keeps_its_actual_generation() { + use rustfs_common::MtlsIdentityPem; + use rustfs_tls_runtime::{OutboundTlsMaterial, TlsGeneration, publish_global_outbound_tls_state}; + let _test_guard = TLS02_TEST_LOCK.lock().await; + let previous = runtime_sources::outbound_tls_state().await; + let first_generation = previous.generation.0.checked_add(1).expect("test generation capacity"); + let first = OutboundTlsMaterial { + root_ca_pem: b"root-a".to_vec(), + mtls_identity: Some(MtlsIdentityPem { + cert_pem: b"certificate-a".to_vec(), + key_pem: b"test-key-a".to_vec(), + }), + }; + publish_global_outbound_tls_state(TlsGeneration(first_generation), &first).await; + let captured = runtime_sources::outbound_tls_state().await; + let second = OutboundTlsMaterial { + root_ca_pem: b"root-b".to_vec(), + mtls_identity: Some(MtlsIdentityPem { + cert_pem: b"certificate-b".to_vec(), + key_pem: b"test-key-b".to_vec(), + }), + }; + publish_global_outbound_tls_state(TlsGeneration(first_generation + 1), &second).await; + let current = runtime_sources::outbound_tls_state().await; + let addr = "https://tls02-late-dial.invalid:9000"; + let channel = Endpoint::from_static("http://127.0.0.1:1").connect_lazy(); + let cached = cache_dialed_channel(addr, addr, &channel, Some(captured.generation.0)).await; + let original_key = captured.mtls_identity.as_ref().expect("captured identity").key_pem.clone(); + let current_key = current.mtls_identity.as_ref().expect("current identity").key_pem.clone(); + let restore = OutboundTlsMaterial { + root_ca_pem: previous.root_ca_pem.unwrap_or_default(), + mtls_identity: previous.mtls_identity, + }; + publish_global_outbound_tls_state(TlsGeneration(first_generation + 2), &restore).await; + assert_eq!( + original_key, b"test-key-a", + "TLS02_INVARIANT captured identity cannot change with publication" + ); + assert_eq!(captured.root_ca_pem.as_deref(), Some(b"root-a".as_slice())); + assert_eq!(current_key, b"test-key-b"); + assert!(!cached, "TLS02_INVARIANT late dial cannot be relabeled with the current generation"); + assert!( + !rustfs_common::has_cached_connection(addr).await, + "TLS02_INVARIANT late stale dial is never cached" + ); + assert!(!TLS_GENERATION_CACHE.lock().await.contains_key(addr)); + } + + #[tokio::test] + async fn tls02_concurrent_control_and_bulk_misses_merge_successful_real_dials() { + use tonic::codegen::tokio_stream::{StreamExt, wrappers::TcpListenerStream}; + let _test_guard = TLS02_TEST_LOCK.lock().await; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind HTTP/2 test server"); + let addr = format!("http://{}", listener.local_addr().expect("test server address")); + let accepted = Arc::new(AtomicUsize::new(0)); + let counter = accepted.clone(); + let incoming = TcpListenerStream::new(listener).map(move |stream| { + if stream.is_ok() { + counter.fetch_add(1, Ordering::SeqCst); + } + stream + }); + let server = tokio::spawn(async move { + tonic::transport::Server::builder() + .add_routes(tonic::service::Routes::default()) + .serve_with_incoming(incoming) + .await + }); + for key in [addr.clone(), bulk_cache_key(&addr, 0)] { + let mut tasks = Vec::new(); + for _ in 0..32 { + let dial_addr = addr.clone(); + let cache_key = key.clone(); + tasks.push(tokio::spawn(async move { + build_channel(&dial_addr, &cache_key).await.map_err(|error| error.to_string()) + })); + } + for task in tasks { + let channel = task.await.expect("channel acquisition task").expect("real HTTP/2 dial"); + drop(channel); + } + assert!(rustfs_common::has_cached_connection(&key).await); + assert!( + !DIAL_LOCK_REGISTRY.lock().expect("dial registry").contains_key(&key), + "TLS02_INVARIANT completed miss groups leave no lock entries" + ); + } + // Tonic returns its channel before the server task necessarily polls the + // already-established sockets out of the listener's accept queue. + tokio::time::sleep(Duration::from_millis(20)).await; + let connections = accepted.load(Ordering::SeqCst); + evict_connection_with_log_level(&addr, ConnectionEvictionLogLevel::Debug).await; + evict_connection_with_log_level(&bulk_cache_key(&addr, 0), ConnectionEvictionLogLevel::Debug).await; + server.abort(); + let _ = server.await; + assert_eq!(connections, 2, "TLS02_INVARIANT 32 misses per control/bulk key dial once each"); + } + + #[tokio::test] + async fn tls02_concurrent_waiters_share_bounded_total_deadlines_and_release_registry() { + let _test_guard = TLS02_TEST_LOCK.lock().await; + let addr = "http://tls02-contended-dial.invalid:9000"; + let owner = DialLock::for_key(addr); + let guard = owner.mutex.lock().await; + let mut tasks = Vec::new(); + for _ in 0..8 { + tasks.push(tokio::spawn(async move { + let deadline = tokio::time::Instant::now() + Duration::from_millis(40); + let result = build_channel_until(addr, addr, deadline, internode_rpc_timeout()).await; + result.err().is_some_and(|error| { + error + .downcast_ref::() + .is_some_and(|error| error.kind() == std::io::ErrorKind::TimedOut) + }) + })); + } + let completed = tokio::time::timeout(Duration::from_millis(250), async { + for task in tasks { + assert!( + task.await.expect("deadline waiter"), + "TLS02_INVARIANT wait and rebuild share a typed timeout" + ); + } + }) + .await; + drop(guard); + drop(owner); + assert!( + completed.is_ok(), + "TLS02_INVARIANT eight waiters cannot accumulate eight connect timeouts" + ); + assert!( + !DIAL_LOCK_REGISTRY.lock().expect("dial registry").contains_key(addr), + "TLS02_INVARIANT cancelled waiters release the lock registry" + ); + } + + #[test] + fn tls02_dial_registry_reclaims_unique_keys_and_preserves_live_lock_identity() { + for index in 0..1000 { + let key = format!("http://tls02-registry-{index}.invalid"); + let first = DialLock::for_key(&key); + let second = DialLock::for_key(&key); + assert!(Arc::ptr_eq(&first.mutex, &second.mutex), "TLS02_INVARIANT one live lock per cache key"); + drop(first); + assert!(DIAL_LOCK_REGISTRY.lock().expect("dial registry").contains_key(&key)); + drop(second); + assert!( + !DIAL_LOCK_REGISTRY.lock().expect("dial registry").contains_key(&key), + "TLS02_INVARIANT last owner removes its own weak entry" + ); + } + } + #[test] fn enforce_tls_generation_cache_bound_evicts_when_retained_entries_still_full() { let mut cache = HashMap::new(); diff --git a/crates/protos/src/node.proto b/crates/protos/src/node.proto index 1c299cc26..ea0b47a20 100644 --- a/crates/protos/src/node.proto +++ b/crates/protos/src/node.proto @@ -422,6 +422,21 @@ message DeletePathsResponse { optional Error error = 2; } +// Protocol 1 is additive and has no fallback to generic file mutation RPCs. +message RecoverDeleteTransactionRequest { + string disk = 1; + string volume = 2; + uint32 protocol_version = 3; + bytes operation = 4; +} + +message RecoverDeleteTransactionResponse { + bool success = 1; + uint32 protocol_version = 2; + bytes result = 3; + optional Error error = 4; +} + message SnapshotLeaseRequest { string disk = 1; string volume = 2; @@ -571,6 +586,14 @@ message DeleteVersionsRequest { bytes opts_bin = 6; } +// Additive, fail-closed conditional batch protocol. Probe requests never mutate disk state. +message GuardedDeleteVersionsRequest { + string disk = 1; + string volume = 2; + bytes payload = 3; + bool probe_only = 4; +} + message DeleteVersionsResponse { bool success = 1; repeated string errors = 2; @@ -1249,9 +1272,12 @@ service NodeService { rpc CompareAndUpdateFile(CompareAndUpdateFileRequest) returns (CompareAndUpdateFileResponse) {}; // auth-policy: body-bound rpc Delete(DeleteRequest) returns (DeleteResponse) {}; // auth-policy: body-bound rpc DeleteAtIncarnation(DeleteRequest) returns (DeleteResponse) {}; // auth-policy: body-bound + rpc RecoverDeleteTransaction(RecoverDeleteTransactionRequest) returns (RecoverDeleteTransactionResponse) {}; // auth-policy: body-bound rpc AcquireSnapshotLease(SnapshotLeaseRequest) returns (SnapshotLeaseResponse) {}; // auth-policy: body-bound + rpc AcquireQuotaMutationFence(SnapshotLeaseRequest) returns (SnapshotLeaseResponse) {}; // auth-policy: body-bound rpc RenewSnapshotLease(SnapshotLeaseRenewRequest) returns (SnapshotLeaseResponse) {}; // auth-policy: body-bound rpc ReleaseSnapshotLease(SnapshotLeaseReleaseRequest) returns (SnapshotLeaseMutationResponse) {}; // auth-policy: body-bound + rpc ReleaseQuotaMutationFence(SnapshotLeaseReleaseRequest) returns (SnapshotLeaseMutationResponse) {}; // auth-policy: body-bound rpc VerifyFile(VerifyFileRequest) returns (VerifyFileResponse) {}; // auth-policy: read-only rpc ReadParts(ReadPartsRequest) returns (ReadPartsResponse) {}; // auth-policy: read-only rpc CheckParts(CheckPartsRequest) returns (CheckPartsResponse) {}; // auth-policy: read-only @@ -1272,6 +1298,7 @@ service NodeService { rpc ArtifactRepairWriteMetadata(ArtifactRepairWriteMetadataRequest) returns (WriteMetadataResponse) {}; // auth-policy: body-bound rpc SealArtifactAttemptRepairs(SealArtifactAttemptPutsRequest) returns (SealArtifactAttemptPutsResponse) {}; // auth-policy: body-bound rpc SealArtifactAttemptPuts(SealArtifactAttemptPutsRequest) returns (SealArtifactAttemptPutsResponse) {}; // auth-policy: body-bound + rpc RenameDataWithQuotaFence(RenameDataRequest) returns (RenameDataResponse) {}; // auth-policy: body-bound rpc MakeVolumes(MakeVolumesRequest) returns (MakeVolumesResponse) {}; // auth-policy: body-bound rpc MakeVolume(MakeVolumeRequest) returns (MakeVolumeResponse) {}; // auth-policy: body-bound rpc ListVolumes(ListVolumesRequest) returns (ListVolumesResponse) {}; // auth-policy: read-only @@ -1288,6 +1315,7 @@ service NodeService { rpc DeleteVersionAtIncarnation(DeleteVersionRequest) returns (DeleteVersionResponse) {}; // auth-policy: body-bound rpc DeleteRetiredMarker(DeleteVersionRequest) returns (DeleteVersionResponse) {}; // auth-policy: body-bound rpc DeleteVersions(DeleteVersionsRequest) returns (DeleteVersionsResponse) {}; // auth-policy: body-bound + rpc GuardedDeleteVersions(GuardedDeleteVersionsRequest) returns (DeleteVersionsResponse) {}; // auth-policy: body-bound rpc ReadMultiple(ReadMultipleRequest) returns (ReadMultipleResponse) {}; // auth-policy: read-only rpc DeleteVolume(DeleteVolumeRequest) returns (DeleteVolumeResponse) {}; // auth-policy: body-bound rpc DiskInfo(DiskInfoRequest) returns (DiskInfoResponse) {}; // auth-policy: read-only diff --git a/crates/protos/src/runtime_sources.rs b/crates/protos/src/runtime_sources.rs index 135e3a9f5..ac685d890 100644 --- a/crates/protos/src/runtime_sources.rs +++ b/crates/protos/src/runtime_sources.rs @@ -22,6 +22,10 @@ pub(crate) async fn outbound_tls_state() -> GlobalPublishedOutboundTlsState { load_global_outbound_tls_state().await } +pub(crate) fn outbound_tls_generation() -> u64 { + rustfs_tls_runtime::load_global_outbound_tls_generation().0 +} + pub(crate) fn record_stale_grpc_channel_tls_generation() { record_tls_consumer_stale_generation(PROTOS_GRPC_CHANNEL_TLS_CONSUMER); } diff --git a/crates/replication/src/object.rs b/crates/replication/src/object.rs index 5d12f8c5a..c33348694 100644 --- a/crates/replication/src/object.rs +++ b/crates/replication/src/object.rs @@ -120,8 +120,11 @@ pub fn object_lock_put_integrity( /// stored bytes differ from what the source sent — e.g. a target that does not /// decode `aws-chunked` framing stores the frames verbatim and returns their /// ETag. Only a plain single-part MD5 ETag on both sides is decidable; a -/// multipart or opaque (encrypted) ETag, or a withheld replica ETag, returns -/// `false` because no corruption can be concluded from it. +/// multipart ETag, or a withheld replica ETag, returns `false` because no +/// corruption can be concluded from it. An encrypted source's keyed +/// fingerprint is also 32 hex characters and cannot be told apart from an MD5 +/// here, so callers must exempt encrypted sources before asking (the ecstore +/// resyncer does so through its source-encryption classifier). pub fn single_part_replica_etag_mismatch(source_etag: Option<&str>, replica_etag: Option<&str>) -> bool { let Some(source) = source_etag.map(trim_etag) else { return false; diff --git a/crates/s3select-query/Cargo.toml b/crates/s3select-query/Cargo.toml index b786f2197..3fbf7778c 100644 --- a/crates/s3select-query/Cargo.toml +++ b/crates/s3select-query/Cargo.toml @@ -57,6 +57,8 @@ tracing = { workspace = true } [dev-dependencies] rustfs-test-utils = { workspace = true, features = ["put-object-commit-barrier"] } +# Match the streaming inference's field order in Arrow's JSON reference. +serde_json = { workspace = true, features = ["preserve_order"] } [lib] doctest = false diff --git a/crates/s3select-query/src/dispatcher/json_document_schema.rs b/crates/s3select-query/src/dispatcher/json_document_schema.rs index d61d9ff0b..1f8c33c46 100644 --- a/crates/s3select-query/src/dispatcher/json_document_schema.rs +++ b/crates/s3select-query/src/dispatcher/json_document_schema.rs @@ -430,7 +430,7 @@ mod tests { const TEST_SCHEMA_LIMIT: usize = 4 * 1024 * 1024; - fn assert_matches_arrow(input: &[u8]) { + fn assert_matches_arrow(input: &[u8]) -> Schema { let records = input.iter().filter(|byte| **byte == b'\n').count(); let (expected, expected_records) = arrow_infer_json_schema(io::BufReader::new(input), Some(records)) .expect("Arrow should infer the compatibility fixture"); @@ -438,6 +438,33 @@ mod tests { .expect("streaming inference should accept the compatibility fixture"); assert_eq!(actual_records, expected_records); assert_eq!(actual, expected); + actual + } + + #[test] + fn inference_preserves_first_seen_field_order_for_nested_objects() { + let schema = assert_matches_arrow( + br#"{"z":1,"nested":{"z":true,"a":"first"}} +{"a":2,"nested":{"m":3}} +"#, + ); + let expected = Schema::new(vec![ + Field::new("z", DataType::Int64, true), + Field::new( + "nested", + DataType::Struct( + vec![ + Field::new("z", DataType::Boolean, true), + Field::new("a", DataType::Utf8, true), + Field::new("m", DataType::Int64, true), + ] + .into(), + ), + true, + ), + Field::new("a", DataType::Int64, true), + ]); + assert_eq!(schema, expected); } #[test] diff --git a/crates/scanner-metrics/Cargo.toml b/crates/scanner-metrics/Cargo.toml index cfc0d3e19..eeedf0791 100644 --- a/crates/scanner-metrics/Cargo.toml +++ b/crates/scanner-metrics/Cargo.toml @@ -38,7 +38,7 @@ tokio = { workspace = true, features = ["sync","rt"] } [dev-dependencies] serde_json = { workspace = true } -tokio = { workspace = true, features = ["macros", "rt-multi-thread"] } +tokio = { workspace = true, features = ["macros", "rt-multi-thread", "time"] } uuid = { workspace = true, features = ["v4"] } [lib] diff --git a/crates/scanner-metrics/src/metrics.rs b/crates/scanner-metrics/src/metrics.rs index d91d08497..5044b4ac9 100644 --- a/crates/scanner-metrics/src/metrics.rs +++ b/crates/scanner-metrics/src/metrics.rs @@ -18,7 +18,7 @@ use jiff::Timestamp; use rustfs_heal_contracts::heal_channel::HealScanMode; use serde::{Deserialize, Serialize}; use std::{ - collections::{BTreeSet, HashMap}, + collections::{BTreeMap, BTreeSet, HashMap}, fmt::Display, future::Future, pin::Pin, @@ -880,6 +880,24 @@ pub struct TierRequestCount { const TIER_REQUEST_COUNTER_SLOTS: usize = TierRequestOperation::ALL.len() * TierRequestOutcome::ALL.len(); +/// Remote work belongs to one cycle; retained worker handles cannot update a later cycle. +#[derive(Default)] +pub struct RemoteScanProgress { + objects: AtomicU64, + directories: AtomicU64, +} + +impl RemoteScanProgress { + /// Record authenticated per-walk deltas without changing process-local operation counters. + pub fn record(&self, objects: u64, directories: u64) { + for (counter, delta) in [(&self.objects, objects), (&self.directories, directories)] { + if delta != 0 { + let _ = counter.try_update(Ordering::Relaxed, Ordering::Relaxed, |value| Some(value.saturating_add(delta))); + } + } + } +} + pub struct Metrics { operations: Vec, latency: Vec, @@ -889,6 +907,10 @@ pub struct Metrics { cycle_info: Arc>>, current_scan_mode: AtomicU8, current_scan_cycle_work_active: AtomicBool, + // Cycle transitions and reports lock cycle_info before remote_scan_progress. + // Cycle transitions take cycle_info before this lock; never hold it across awaits + // or while acquiring another metrics lock. Workers update their own handle directly. + remote_scan_progress: Mutex>, current_scan_cycle_objects_start: AtomicU64, current_scan_cycle_directories_start: AtomicU64, current_scan_cycle_bucket_drive_scans_start: AtomicU64, @@ -994,6 +1016,8 @@ pub struct Metrics { scanner_bitrot_cycle_enabled: AtomicBool, scanner_bitrot_cycle_millis: AtomicU64, scanner_checkpoint: Mutex>, + scanner_checkpoints_persisted: Mutex>, + scanner_checkpoint_save_failures: AtomicU64, scanner_checkpoint_used: AtomicU64, scanner_checkpoint_cleared: AtomicU64, scanner_checkpoint_ignored: AtomicU64, @@ -1008,6 +1032,9 @@ pub struct Metrics { scanner_usage_last_publication_unix_secs: AtomicU64, scanner_usage_last_publication_state: Mutex, scanner_usage_last_publication_reason: Mutex, + scanner_guardrail_veto_total: AtomicU64, + scanner_guardrail_last_veto_unix_secs: AtomicU64, + scanner_guardrail_last_veto_reason: Mutex, scanner_usage_last_save_result: AtomicU8, scanner_usage_deferred_pending: AtomicBool, scanner_usage_deferred_total: AtomicU64, @@ -1137,6 +1164,15 @@ pub struct ScannerCheckpointReport { pub last_event: String, } +#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)] +pub struct ScannerPersistedCheckpointReport { + pub pool_index: usize, + pub set_index: usize, + pub bucket: String, + pub saved_unix_secs: u64, + pub checkpoint: Option, +} + #[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)] pub struct ScannerSourceWorkSnapshot { pub source: String, @@ -1326,6 +1362,14 @@ pub struct ScannerUsageFreshnessSnapshot { pub last_deferred_unix_secs: u64, #[serde(default)] pub last_deferred_reason: String, + /// Guardrail vetoes attributed as `@`; a phase-2 peer + /// dirty-usage fallback is reported through the same fields. + #[serde(default)] + pub guardrail_veto_total: u64, + #[serde(default)] + pub last_guardrail_veto_unix_secs: u64, + #[serde(default)] + pub last_guardrail_veto_reason: String, } #[derive(Clone, Debug, Default, Serialize, Deserialize)] @@ -1499,6 +1543,10 @@ pub struct ScannerMetricsReport { #[serde(skip_serializing_if = "Option::is_none")] pub scan_checkpoint: Option, #[serde(default)] + pub scan_checkpoints_persisted: Vec, + #[serde(default)] + pub scan_checkpoint_save_failures: u64, + #[serde(default)] pub scan_checkpoint_used: u64, #[serde(default)] pub scan_checkpoint_cleared: u64, @@ -1948,6 +1996,7 @@ impl Metrics { cycle_info: Arc::new(RwLock::new(None)), current_scan_mode: AtomicU8::new(HealScanMode::Unknown as u8), current_scan_cycle_work_active: AtomicBool::new(false), + remote_scan_progress: Mutex::new(Arc::new(RemoteScanProgress::default())), current_scan_cycle_objects_start: AtomicU64::new(0), current_scan_cycle_directories_start: AtomicU64::new(0), current_scan_cycle_bucket_drive_scans_start: AtomicU64::new(0), @@ -2046,6 +2095,8 @@ impl Metrics { scanner_bitrot_cycle_enabled: AtomicBool::new(false), scanner_bitrot_cycle_millis: AtomicU64::new(0), scanner_checkpoint: Mutex::new(None), + scanner_checkpoints_persisted: Mutex::new(BTreeMap::new()), + scanner_checkpoint_save_failures: AtomicU64::new(0), scanner_checkpoint_used: AtomicU64::new(0), scanner_checkpoint_cleared: AtomicU64::new(0), scanner_checkpoint_ignored: AtomicU64::new(0), @@ -2060,6 +2111,9 @@ impl Metrics { scanner_usage_last_publication_unix_secs: AtomicU64::new(0), scanner_usage_last_publication_state: Mutex::new(String::new()), scanner_usage_last_publication_reason: Mutex::new(String::new()), + scanner_guardrail_veto_total: AtomicU64::new(0), + scanner_guardrail_last_veto_unix_secs: AtomicU64::new(0), + scanner_guardrail_last_veto_reason: Mutex::new(String::new()), scanner_usage_last_save_result: AtomicU8::new(ScannerUsageSaveResult::Unknown as u8), scanner_usage_deferred_pending: AtomicBool::new(false), scanner_usage_deferred_total: AtomicU64::new(0), @@ -2353,6 +2407,34 @@ impl Metrics { } } + /// Keep only the latest confirmed periodic save per set on this process. + pub fn record_scanner_checkpoint_saved( + &self, + pool_index: usize, + set_index: usize, + bucket: &str, + checkpoint: Option, + ) { + let mut saved = self + .scanner_checkpoints_persisted + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + saved.insert( + (pool_index, set_index), + ScannerPersistedCheckpointReport { + pool_index, + set_index, + bucket: bucket.to_string(), + saved_unix_secs: unix_now_secs(), + checkpoint, + }, + ); + } + + pub fn record_scanner_checkpoint_save_failed(&self) { + self.scanner_checkpoint_save_failures.fetch_add(1, Ordering::Relaxed); + } + pub fn record_scanner_checkpoint_used(&self) { self.scanner_checkpoint_used.fetch_add(1, Ordering::Relaxed); self.update_scanner_checkpoint_event(SCANNER_CHECKPOINT_EVENT_USED); @@ -2432,6 +2514,19 @@ impl Metrics { *last_reason = reason; } + /// Record an attributed cluster-activity guardrail veto (`@`) + /// so an operator can tell a rebalance or a PUT tail from a broken probe. + pub fn record_scanner_guardrail_veto(&self, reason: impl Into) { + self.scanner_guardrail_veto_total.fetch_add(1, Ordering::Relaxed); + self.scanner_guardrail_last_veto_unix_secs + .store(unix_now_secs(), Ordering::Relaxed); + let mut last_reason = match self.scanner_guardrail_last_veto_reason.lock() { + Ok(guard) => guard, + Err(poisoned) => poisoned.into_inner(), + }; + *last_reason = reason.into(); + } + pub fn record_scanner_usage_durable_success(&self) { self.record_scanner_usage_publication("success", ""); self.scanner_usage_last_durable_success_unix_secs @@ -2749,8 +2844,18 @@ impl Metrics { /// Publish a scanner cycle and its work-accounting baseline as one state transition. pub async fn start_scan_cycle_work_with_cycle(&self, cycle: CurrentCycle) -> ScanCycleWorkSnapshot { + self.start_scan_cycle_work_with_cycle_and_remote_progress(cycle, Arc::new(RemoteScanProgress::default())) + .await + } + + /// Install the cycle's remote work handle together with its cycle identity. + pub async fn start_scan_cycle_work_with_cycle_and_remote_progress( + &self, + cycle: CurrentCycle, + remote_progress: Arc, + ) -> ScanCycleWorkSnapshot { let mut current_cycle = self.cycle_info.write().await; - let snapshot = self.start_scan_cycle_work(); + let snapshot = self.begin_scan_cycle_work(remote_progress); *current_cycle = Some(cycle); snapshot } @@ -2865,6 +2970,15 @@ impl Metrics { } pub fn start_scan_cycle_work(&self) -> ScanCycleWorkSnapshot { + self.begin_scan_cycle_work(Arc::new(RemoteScanProgress::default())) + } + + fn begin_scan_cycle_work(&self, remote_progress: Arc) -> ScanCycleWorkSnapshot { + // The cycle transition serializes replacement; workers keep their own Arc. + *self + .remote_scan_progress + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) = remote_progress; let snapshot = self.scan_cycle_work_snapshot(); let source_snapshot = self.scanner_source_work_values(); let replication_repair_snapshot = self.scanner_replication_repair_work_values(); @@ -2982,9 +3096,22 @@ impl Metrics { fn scan_cycle_work_since(&self, start: ScanCycleWorkSnapshot) -> ScanCycleWorkSnapshot { let current = self.scan_cycle_work_snapshot(); + let (remote_objects, remote_directories) = { + let remote = self + .remote_scan_progress + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + (remote.objects.load(Ordering::Relaxed), remote.directories.load(Ordering::Relaxed)) + }; ScanCycleWorkSnapshot { - objects_scanned: current.objects_scanned.saturating_sub(start.objects_scanned), - directories_scanned: current.directories_scanned.saturating_sub(start.directories_scanned), + objects_scanned: current + .objects_scanned + .saturating_sub(start.objects_scanned) + .saturating_add(remote_objects), + directories_scanned: current + .directories_scanned + .saturating_sub(start.directories_scanned) + .saturating_add(remote_directories), bucket_drive_scans: current.bucket_drive_scans.saturating_sub(start.bucket_drive_scans), bucket_drive_failures: current.bucket_drive_failures.saturating_sub(start.bucket_drive_failures), yield_events: current.yield_events.saturating_sub(start.yield_events), @@ -3493,6 +3620,12 @@ impl Metrics { Ok(reason) => reason.clone(), Err(poisoned) => poisoned.into_inner().clone(), }, + guardrail_veto_total: self.scanner_guardrail_veto_total.load(Ordering::Relaxed), + last_guardrail_veto_unix_secs: self.scanner_guardrail_last_veto_unix_secs.load(Ordering::Relaxed), + last_guardrail_veto_reason: match self.scanner_guardrail_last_veto_reason.lock() { + Ok(reason) => reason.clone(), + Err(poisoned) => poisoned.into_inner().clone(), + }, }; m.throttle_idle_mode_enabled = self.scanner_throttle_idle_mode_enabled.load(Ordering::Relaxed); m.throttle_sleep_factor = self.scanner_throttle_sleep_factor_micros.load(Ordering::Relaxed) as f64 / 1_000_000.0; @@ -3512,6 +3645,14 @@ impl Metrics { Ok(checkpoint) => checkpoint.clone(), Err(poisoned) => poisoned.into_inner().clone(), }; + m.scan_checkpoints_persisted = self + .scanner_checkpoints_persisted + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .values() + .cloned() + .collect(); + m.scan_checkpoint_save_failures = self.scanner_checkpoint_save_failures.load(Ordering::Relaxed); m.scan_checkpoint_used = self.scanner_checkpoint_used.load(Ordering::Relaxed); m.scan_checkpoint_cleared = self.scanner_checkpoint_cleared.load(Ordering::Relaxed); m.scan_checkpoint_ignored = self.scanner_checkpoint_ignored.load(Ordering::Relaxed); @@ -3597,14 +3738,18 @@ pub type CloseDiskFn = Arc Pin + Send>> /// Register a new disk in the global path tracker and return two callbacks: /// one to update the current path and one to deregister the disk when done. pub async fn current_path_updater(disk: &str, initial: &str) -> (UpdateCurrentPathFn, CloseDiskFn) { + register_current_path(&global_metrics().current_paths, disk, initial).await +} + +async fn register_current_path( + current_paths: &Arc>>>, + disk: &str, + initial: &str, +) -> (UpdateCurrentPathFn, CloseDiskFn) { let tracker = Arc::new(CurrentPathTracker::new(initial.to_string())); let disk_name = disk.to_string(); - global_metrics() - .current_paths - .write() - .await - .insert(disk_name.clone(), Arc::clone(&tracker)); + current_paths.write().await.insert(disk_name.clone(), Arc::clone(&tracker)); let update_fn: UpdateCurrentPathFn = { let tracker = Arc::clone(&tracker); @@ -3617,10 +3762,17 @@ pub async fn current_path_updater(disk: &str, initial: &str) -> (UpdateCurrentPa let done_fn: CloseDiskFn = { let disk = disk_name; + let current_paths = Arc::clone(current_paths); + let tracker = Arc::clone(&tracker); Arc::new(move || { let disk = disk.clone(); + let current_paths = Arc::clone(¤t_paths); + let tracker = Arc::clone(&tracker); Box::pin(async move { - global_metrics().current_paths.write().await.remove(&disk); + let mut paths = current_paths.write().await; + if paths.get(&disk).is_some_and(|current| Arc::ptr_eq(current, &tracker)) { + paths.remove(&disk); + } }) }) }; @@ -3804,6 +3956,33 @@ mod tests { assert!(!global_metrics().current_paths.read().await.contains_key(&disk)); } + #[tokio::test] + async fn stale_path_cleanup_preserves_a_replacement_scan() { + let metrics = Metrics::new(); + let (_, close_first) = register_current_path(&metrics.current_paths, "remote-node/disk", "first-bucket").await; + let (_, close_second) = register_current_path(&metrics.current_paths, "remote-node/disk", "second-bucket").await; + close_first().await; + assert_eq!(metrics.get_current_paths().await, vec!["remote-node/disk/second-bucket"]); + close_second().await; + assert!(metrics.get_current_paths().await.is_empty()); + } + + #[tokio::test] + async fn cancelled_path_guard_cleans_up_its_remote_scan() { + let metrics = Metrics::new(); + let (_, close_path) = register_current_path(&metrics.current_paths, "remote-node/disk", "bucket").await; + let guard = CloseDiskGuard::new(close_path); + assert_eq!(metrics.report().await.active_paths, vec!["remote-node/disk/bucket"]); + drop(guard); + tokio::time::timeout(Duration::from_secs(1), async { + while !metrics.get_current_paths().await.is_empty() { + tokio::task::yield_now().await; + } + }) + .await + .expect("cancelled remote path guard must remove its registration"); + } + #[tokio::test] async fn report_counts_active_scan_paths() { let metrics = Metrics::new(); @@ -3975,6 +4154,37 @@ mod tests { assert_eq!(report.scan_checkpoint_cleared, 1); } + #[tokio::test] + async fn report_distinguishes_persisted_checkpoints_from_memory_and_failures() { + let metrics = Metrics::new(); + let saved = ScannerCheckpointReport { + version: 1, + resume_after: "bucket/first".to_string(), + reason: "objects".to_string(), + last_event: "saved".to_string(), + }; + metrics.record_scanner_checkpoint_saved(0, 1, "bucket", Some(saved.clone())); + metrics.record_scanner_checkpoint_set(1, "bucket/second", "objects"); + metrics.record_scanner_checkpoint_save_failed(); + let report = metrics.report().await; + assert_eq!(report.scan_checkpoint_save_failures, 1); + assert_eq!(report.scan_checkpoint.expect("in-memory cursor").resume_after, "bucket/second"); + assert_eq!(report.scan_checkpoints_persisted.len(), 1); + assert_eq!(report.scan_checkpoints_persisted[0].checkpoint, Some(saved)); + assert_eq!(report.scan_checkpoints_persisted[0].set_index, 1); + assert!(report.scan_checkpoints_persisted[0].saved_unix_secs > 0); + metrics.record_scanner_checkpoint_saved(0, 1, "other-bucket", None); + metrics.record_scanner_checkpoint_saved(0, 2, "bucket", None); + let report = metrics.report().await; + assert_eq!( + report.scan_checkpoints_persisted.len(), + 2, + "retain at most one save per set, not per bucket" + ); + assert_eq!(report.scan_checkpoints_persisted[0].bucket, "other-bucket"); + assert!(report.scan_checkpoints_persisted[0].checkpoint.is_none()); + } + #[tokio::test] async fn report_includes_scanner_source_work() { let metrics = Metrics::new(); @@ -4977,6 +5187,86 @@ mod tests { assert_eq!(report.usage_freshness.last_cycle_cleared_dirty_buckets, 0); } + #[test] + fn remote_scan_progress_saturates_instead_of_wrapping() { + let remote = RemoteScanProgress::default(); + remote.record(u64::MAX - 1, u64::MAX - 2); + remote.record(3, 4); + remote.record(0, 0); + assert_eq!(remote.objects.load(Ordering::Relaxed), u64::MAX); + assert_eq!(remote.directories.load(Ordering::Relaxed), u64::MAX); + } + + #[tokio::test] + async fn saturated_remote_cycle_does_not_hide_later_remote_and_local_work() { + let metrics = Metrics::new(); + let old_remote = Arc::new(RemoteScanProgress::default()); + let previous = metrics + .start_scan_cycle_work_with_cycle_and_remote_progress(CurrentCycle::default(), old_remote.clone()) + .await; + old_remote.record(u64::MAX, u64::MAX); + metrics.finish_scan_cycle_work(previous); + + let remote = Arc::new(RemoteScanProgress::default()); + let next = metrics + .start_scan_cycle_work_with_cycle_and_remote_progress(CurrentCycle::default(), remote.clone()) + .await; + old_remote.record(9, 7); + let empty = metrics.report().await; + assert_eq!(empty.current_cycle_objects_scanned, 0); + assert_eq!(empty.current_cycle_directories_scanned, 0); + remote.record(2, 1); + metrics.operations[Metric::ScanObject as usize].fetch_add(3, Ordering::Relaxed); + metrics.operations[Metric::ScanFolder as usize].fetch_add(2, Ordering::Relaxed); + let active = metrics.report().await; + assert_eq!(active.current_cycle_objects_scanned, 5); + assert_eq!(active.current_cycle_directories_scanned, 3); + metrics.finish_scan_cycle_work(next); + let finished = metrics.report().await; + assert_eq!(finished.last_cycle_objects_scanned, 5); + assert_eq!(finished.last_cycle_directories_scanned, 3); + } + + #[tokio::test] + async fn cycle_work_combines_local_and_remote_progress_without_changing_local_operations() { + let metrics = Metrics::new(); + metrics.operations[Metric::ScanObject as usize].store(10, Ordering::Relaxed); + metrics.operations[Metric::ScanFolder as usize].store(5, Ordering::Relaxed); + let remote = Arc::new(RemoteScanProgress::default()); + let start = metrics + .start_scan_cycle_work_with_cycle_and_remote_progress(CurrentCycle::default(), remote.clone()) + .await; + metrics.operations[Metric::ScanObject as usize].fetch_add(3, Ordering::Relaxed); + metrics.operations[Metric::ScanFolder as usize].fetch_add(2, Ordering::Relaxed); + remote.record(7, 4); + let active = metrics.report().await; + assert_eq!(active.current_cycle_objects_scanned, 10); + assert_eq!(active.current_cycle_directories_scanned, 6); + assert_eq!(metrics.lifetime(Metric::ScanObject), 13); + assert_eq!(metrics.lifetime(Metric::ScanFolder), 7); + metrics.finish_scan_cycle_work(start); + let finished = metrics.report().await; + assert_eq!(finished.current_cycle_objects_scanned, 0); + assert_eq!(finished.current_cycle_directories_scanned, 0); + assert_eq!(finished.last_cycle_objects_scanned, 10); + assert_eq!(finished.last_cycle_directories_scanned, 6); + + remote.record(9, 3); + let next_remote = Arc::new(RemoteScanProgress::default()); + let next = metrics + .start_scan_cycle_work_with_cycle_and_remote_progress(CurrentCycle::default(), next_remote.clone()) + .await; + let empty = metrics.report().await; + assert_eq!(empty.current_cycle_objects_scanned, 0); + assert_eq!(empty.current_cycle_directories_scanned, 0); + remote.record(11, 5); + next_remote.record(2, 1); + metrics.finish_scan_cycle_work(next); + let finished = metrics.report().await; + assert_eq!(finished.last_cycle_objects_scanned, 2); + assert_eq!(finished.last_cycle_directories_scanned, 1); + } + #[tokio::test] async fn report_includes_active_scan_cycle_work() { let metrics = Metrics::new(); diff --git a/crates/scanner/Cargo.toml b/crates/scanner/Cargo.toml index 6e92392e4..0205328c6 100644 --- a/crates/scanner/Cargo.toml +++ b/crates/scanner/Cargo.toml @@ -31,6 +31,7 @@ workspace = true [features] default = [] +diagnostic-read-version-caller = ["rustfs-ecstore/diagnostic-read-version-caller"] hotpath = [ "hotpath/hotpath", "hotpath/tokio", diff --git a/crates/scanner/src/data_usage_define.rs b/crates/scanner/src/data_usage_define.rs index f711f11fc..e4c4530e3 100644 --- a/crates/scanner/src/data_usage_define.rs +++ b/crates/scanner/src/data_usage_define.rs @@ -56,9 +56,10 @@ pub const DATA_USAGE_CACHE_NAME: &str = ".usage-cache.bin"; pub(crate) const DATA_USAGE_CACHE_KEY_FORMAT: u16 = 1; const DATA_USAGE_CACHE_SAVE_RETRIES: u32 = 2; -const DATA_USAGE_CACHE_BACKUP_SAVE_TIMEOUT_SECS_MAX: u64 = 5; +const DATA_USAGE_CACHE_BACKUP_SAVE_TIMEOUT_SECS_BASE: u64 = 5; const DATA_USAGE_CACHE_BACKUP_SAVE_RETRIES: u32 = 0; const DATA_USAGE_CACHE_SAVE_RETRY_BACKOFF_MAX: Duration = Duration::from_millis(350); +const DATA_USAGE_CACHE_RECONCILE_BACKOFF_MAX: Duration = Duration::from_millis(150); const DATA_USAGE_CACHE_PERSISTENCE_MARGIN: Duration = Duration::from_secs(5); const METRIC_CACHE_SAVE_ATTEMPT_TOTAL: &str = "rustfs_scanner_cache_save_attempt_total"; const METRIC_CACHE_SAVE_TIMEOUT_TOTAL: &str = "rustfs_scanner_cache_save_timeout_total"; @@ -100,19 +101,17 @@ pub(crate) async fn read_config_with_revision( store: Arc, path: &str, ) -> StorageResult<(Option>, DataUsageCacheRevision)> { - match store - .get_object_reader( - RUSTFS_META_BUCKET, - path, - None, - HeaderMap::new(), - &ObjectOptions { - no_lock: true, - ..Default::default() - }, - ) - .await - { + let opts = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let read = store.get_object_reader(RUSTFS_META_BUCKET, path, None, HeaderMap::new(), &opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::storage_api::ecstore_diagnostic::with_read_version_entry( + crate::storage_api::ecstore_diagnostic::ReadVersionEntrySource::ScannerRevisionRead, + read, + ); + match read.await { Ok(mut reader) => { let revision = reader .object_info @@ -170,19 +169,17 @@ fn error_chain_has_usage_floor_corruption_signature(error: &(dyn std::error::Err /// Read only the object revision without materializing its body. pub(crate) async fn read_config_revision(store: Arc, path: &str) -> StorageResult { - match store - .get_object_reader( - RUSTFS_META_BUCKET, - path, - None, - HeaderMap::new(), - &ObjectOptions { - no_lock: true, - ..Default::default() - }, - ) - .await - { + let opts = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let read = store.get_object_reader(RUSTFS_META_BUCKET, path, None, HeaderMap::new(), &opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::storage_api::ecstore_diagnostic::with_read_version_entry( + crate::storage_api::ecstore_diagnostic::ReadVersionEntrySource::ScannerRevisionRead, + read, + ); + match read.await { Ok(reader) => reader .object_info .etag @@ -234,7 +231,7 @@ pub static DATA_USAGE_BLOOM_RECOVERY_PATH: LazyLock = pub static BACKGROUND_HEAL_INFO_PATH: LazyLock = LazyLock::new(|| format!("{BUCKET_META_PREFIX}{SLASH_SEPARATOR}.background-heal.json")); -const MAX_DATA_USAGE_CACHE_DEPTH: usize = 1024; +pub(crate) const MAX_DATA_USAGE_CACHE_DEPTH: usize = 1024; /// Scanner-side accounting on the shared [`SizeSummary`]. /// @@ -518,6 +515,14 @@ impl Serialize for DataUsageScanCoverageReceipt { } } +/// Coverage follows depth-first directory traversal, rather than raw path order. +/// A completed subtree includes its descendants but never an unfinished ancestor. +pub(crate) fn scan_frontier_covers_path(path: &str, frontier: &str) -> bool { + let ancestor = frontier.strip_prefix(path).is_some_and(|suffix| suffix.starts_with('/')); + let descendant = path.strip_prefix(frontier).is_some_and(|suffix| suffix.starts_with('/')); + (path.split('/').cmp(frontier.split('/')).is_le() && !ancestor) || descendant +} + struct CheckpointDigestWriter(Sha256); impl std::io::Write for CheckpointDigestWriter { @@ -895,6 +900,14 @@ impl DataUsageCache { } } + pub(crate) fn has_scan_position(&self) -> bool { + self.info.scan_resume_after.is_some() + || self.info.scan_checkpoint.is_some() + || self.info.scan_raw_enumeration_cursor.is_some() + || self.info.scan_raw_enumeration_page_index.is_some() + || self.info.scan_coverage_receipt.is_some() + } + pub(crate) fn prepare_bucket_checkpoint( &mut self, name: &str, @@ -910,6 +923,8 @@ impl DataUsageCache { if self.info.leader_epoch > leader_epoch { return DataUsageCachePrepareOutcome::RejectedNewerLeader; } + let previous_cycle = self.info.next_cycle; + let previous_leader_epoch = self.info.leader_epoch; // A checkpoint is a durable traversal frontier, not a publication // result. A newer leader may adopt it only when the frontier is // independently fenced by the bucket identity and has a validated @@ -938,16 +953,43 @@ impl DataUsageCache { if reusable && self.info.snapshot_complete && self.info.scan_progress.is_none() - && self.info.scan_checkpoint.is_none() - && self.info.scan_raw_enumeration_cursor.is_none() - && self.info.scan_raw_enumeration_page_index.is_none() - && self.info.scan_resume_after.is_none() - && self.info.scan_coverage_receipt.is_none() + && !self.has_scan_position() && self.info.scan_plan_digest == Some(scan_plan_digest) { return self.prepare_for_scan(name, next_cycle, leader_epoch, source, scan_plan_digest, true); } if !reusable { + if self.has_scan_position() { + let reason = if name == DATA_USAGE_ROOT { + "invalid_bucket_scope" + } else if !identity.is_valid() { + "invalid_identity" + } else if self.info.name != name { + "bucket_mismatch" + } else if self.info.source != Some(source) { + "source_mismatch" + } else if self.info.cache_key_format != DATA_USAGE_CACHE_KEY_FORMAT { + "key_format_mismatch" + } else if self.info.scan_identity != Some(identity) + || self.info.tier_registry_generation != Some(identity.tier_registry_generation) + { + "identity_mismatch" + } else if self.info.leader_epoch != leader_epoch && !cross_epoch_checkpoint { + "unvalidated_handoff" + } else { + "incomplete_cache_scope" + }; + warn!( + target: "rustfs::scanner::data_usage", + event = EVENT_SCANNER_CACHE_LOAD_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_CACHE, + state = "checkpoint_reset", + bucket = %name, + reason, + "Scanner bucket checkpoint cannot be adopted" + ); + } let keep_debts = self.info.name == name && self .info @@ -971,11 +1013,7 @@ impl DataUsageCache { if self.validated_raw_enumeration_page_index().is_none() { self.info.scan_raw_enumeration_page_index = None; } - let cursor_is_valid = (self.info.scan_checkpoint.is_none() - && self.info.scan_raw_enumeration_cursor.is_none() - && self.info.scan_raw_enumeration_page_index.is_none() - && self.info.scan_resume_after.is_none() - && self.info.scan_coverage_receipt.is_none()) + let cursor_is_valid = !self.has_scan_position() || self.validated_scan_frontier().is_some() || self.info.scan_raw_enumeration_cursor.is_some() || self.info.scan_raw_enumeration_page_index.is_some(); @@ -1003,11 +1041,7 @@ impl DataUsageCache { // so it observes nothing under the plan that finished the previous // sweep. Keeping that plan would mark every sweep of a bucket written // between cycles as mixed, and the bucket could never certify. - let has_position = self.info.scan_resume_after.is_some() - || self.info.scan_checkpoint.is_some() - || self.info.scan_raw_enumeration_cursor.is_some() - || self.info.scan_raw_enumeration_page_index.is_some() - || self.info.scan_coverage_receipt.is_some(); + let has_position = self.has_scan_position(); if let Some(progress) = &mut self.info.scan_progress && has_position { @@ -1026,6 +1060,23 @@ impl DataUsageCache { // Old readers do not understand coverage sweeps. An absent plan makes // their existing prepare path rebuild instead of promoting mixed data. self.info.scan_plan_digest = None; + debug!( + target: "rustfs::scanner::data_usage", + event = EVENT_SCANNER_CACHE_LOAD_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_CACHE, + state = "bucket_checkpoint_prepared", + bucket = %name, + pool = source.pool_index, + set = source.set_index, + cycle = next_cycle, + leader_epoch, + previous_cycle, + previous_leader_epoch, + resume_position = self.has_scan_position(), + mixed_coverage = self.info.scan_progress.is_some_and(|progress| progress.started_plan != progress.requested_plan), + "Scanner bucket checkpoint prepared for a coverage sweep" + ); if reusable { DataUsageCachePrepareOutcome::Reused } else { @@ -1034,17 +1085,22 @@ impl DataUsageCache { } fn coverage_prefix_digest(&self, through: &str) -> Result<[u8; 32], serde_json::Error> { - self.coverage_prefix_digest_with_epoch(through, None) + self.coverage_prefix_digest_with_epoch(through, None, 3) + } + + fn v2_coverage_prefix_digest(&self, through: &str) -> Result<[u8; 32], serde_json::Error> { + self.coverage_prefix_digest_with_epoch(through, None, 2) } fn legacy_coverage_prefix_digest(&self, through: &str) -> Result<[u8; 32], serde_json::Error> { - self.coverage_prefix_digest_with_epoch(through, Some(self.info.leader_epoch)) + self.coverage_prefix_digest_with_epoch(through, Some(self.info.leader_epoch), 0) } fn coverage_prefix_digest_with_epoch( &self, through: &str, legacy_leader_epoch: Option, + domain_version: u8, ) -> Result<[u8; 32], serde_json::Error> { let mut writer = CheckpointDigestWriter(Sha256::new()); if let Some(leader_epoch) = legacy_leader_epoch { @@ -1068,10 +1124,12 @@ impl DataUsageCache { // process-local leader epoch is intentionally excluded because it // changes during a safe handoff. The domain version also keeps // old epoch-bound receipts from colliding with this digest. + // Domain 3 permits object-health debt behind the frontier. Older + // readers must reject it because they skip covered failures. serde_json::to_writer( &mut writer, &( - 2u8, + domain_version, &self.info.name, self.info.scan_identity, self.info.source, @@ -1085,6 +1143,11 @@ impl DataUsageCache { .cache .iter() .filter(|(key, _)| { + // Sign exactly the subtrees the reader can reuse, including + // descendants whose raw path order differs from traversal order. + if legacy_leader_epoch.is_none() && domain_version >= 3 { + return scan_frontier_covers_path(key, through); + } let ancestor = through .strip_prefix(key.as_str()) .is_some_and(|suffix| suffix.starts_with('/')); @@ -1120,6 +1183,7 @@ impl DataUsageCache { .is_some_and(|suffix| suffix.starts_with('/')) && self.find(&receipt.through).is_some() && (self.coverage_prefix_digest(&receipt.through).ok() == Some(receipt.digest) + || self.v2_coverage_prefix_digest(&receipt.through).ok() == Some(receipt.digest) || self.legacy_coverage_prefix_digest(&receipt.through).ok() == Some(receipt.digest))) .then_some(receipt.through.as_str()) } @@ -1221,8 +1285,8 @@ impl DataUsageCache { self.cache.get(&hash_path(path).key()) } - pub fn find_children_copy(&mut self, h: DataUsageHash) -> DataUsageHashMap { - self.cache.entry(h.string()).or_default().children.clone() + pub fn find_children_copy(&self, h: &DataUsageHash) -> DataUsageHashMap { + self.cache.get(&h.0).map(|entry| entry.children.clone()).unwrap_or_default() } pub fn flatten(&self, root: &DataUsageEntry) -> DataUsageEntry { diff --git a/crates/scanner/src/data_usage_define/persistence.rs b/crates/scanner/src/data_usage_define/persistence.rs index 00851af64..29d15e4de 100644 --- a/crates/scanner/src/data_usage_define/persistence.rs +++ b/crates/scanner/src/data_usage_define/persistence.rs @@ -370,6 +370,17 @@ impl DataUsageCache { } }; + #[cfg(feature = "diagnostic-read-version-caller")] + let load_fut = crate::storage_api::ecstore_diagnostic::with_read_version_entry( + crate::storage_api::ecstore_diagnostic::ReadVersionEntrySource::DataUsageCacheRead, + async { + crate::storage_api::ecstore_diagnostic::observe_read_version_context_boundary( + crate::storage_api::ecstore_diagnostic::ReadVersionContextBoundary::ScannerCacheLoad, + ); + load_fut.await + }, + ); + match timeout(timeout_duration, load_fut).await { Ok(Ok((Some(cache), revision, _))) => Ok(DataUsageCacheLoadAttempt::Loaded { cache: Box::new(cache), @@ -386,16 +397,36 @@ impl DataUsageCache { crate::runtime_config::scanner_cache_save_timeout() } - pub(crate) fn persistence_timeout() -> Duration { - Self::cache_save_timeout() + pub(crate) fn publication_timeout() -> Duration { + // Aggregate statistics publication must fit its lease window rather + // than inherit the longer checkpoint save/reconciliation allowance. + let timeout_duration = Self::cache_save_timeout(); + timeout_duration .saturating_mul(DATA_USAGE_CACHE_SAVE_RETRIES + 1) .saturating_add(DATA_USAGE_CACHE_SAVE_RETRY_BACKOFF_MAX) - .saturating_add(Self::backup_cache_save_timeout(Self::cache_save_timeout())) + .saturating_add(Self::backup_cache_save_timeout(timeout_duration, Duration::ZERO)) .saturating_add(DATA_USAGE_CACHE_PERSISTENCE_MARGIN) } - fn backup_cache_save_timeout(timeout_duration: Duration) -> Duration { - timeout_duration.min(Duration::from_secs(DATA_USAGE_CACHE_BACKUP_SAVE_TIMEOUT_SECS_MAX)) + pub(crate) fn persistence_timeout() -> Duration { + // Each save attempt can be followed by a full read-after-error + // reconciliation attempt, including the best-effort backup. + Self::cache_save_timeout() + .saturating_mul(DATA_USAGE_CACHE_SAVE_RETRIES + DATA_USAGE_CACHE_BACKUP_SAVE_RETRIES + 2) + .saturating_mul(2) + .saturating_add(DATA_USAGE_CACHE_SAVE_RETRY_BACKOFF_MAX) + .saturating_add(DATA_USAGE_CACHE_RECONCILE_BACKOFF_MAX) + .saturating_add(DATA_USAGE_CACHE_PERSISTENCE_MARGIN) + } + + fn backup_cache_save_timeout(timeout_duration: Duration, primary_elapsed: Duration) -> Duration { + // Both copies write and verify the same body. Allow headroom over the + // observed primary cost without exceeding the configured attempt cap; + // fast checkpoints keep the short best-effort budget and zero retries. + primary_elapsed + .saturating_mul(2) + .max(Duration::from_secs(DATA_USAGE_CACHE_BACKUP_SAVE_TIMEOUT_SECS_BASE)) + .min(timeout_duration) } fn record_save_attempt(path_type: &'static str, result: &'static str, duration: Duration) { @@ -617,6 +648,7 @@ impl DataUsageCache { let timeout_duration = Self::cache_save_timeout(); let path = path_join_buf(&[BUCKET_META_PREFIX, name]); + let primary_save_started = Instant::now(); Self::save_path_with_retry( store.clone(), &path, @@ -630,7 +662,7 @@ impl DataUsageCache { let backup_name = format!("{name}.bkp"); let backup_path = path_join_buf(&[BUCKET_META_PREFIX, &backup_name]); - let backup_timeout_duration = Self::backup_cache_save_timeout(timeout_duration); + let backup_timeout_duration = Self::backup_cache_save_timeout(timeout_duration, primary_save_started.elapsed()); let backup_revision = revisions.and_then(|revisions| revisions.backup.clone()); if revisions.is_some() && backup_revision.is_none() { return Ok(()); @@ -669,6 +701,28 @@ impl DataUsageCache { mod tests { use super::*; + #[test] + fn backup_budget_tracks_cost_within_configured_cap() { + for (cap, cost, expected) in [ + (Duration::ZERO, Duration::ZERO, Duration::ZERO), + (Duration::from_secs(14), Duration::ZERO, Duration::from_secs(5)), + (Duration::from_secs(14), Duration::from_millis(2499), Duration::from_secs(5)), + (Duration::from_secs(14), Duration::from_millis(2500), Duration::from_secs(5)), + (Duration::from_secs(14), Duration::from_millis(2501), Duration::from_millis(5002)), + (Duration::from_secs(14), Duration::from_secs(6), Duration::from_secs(12)), + (Duration::from_secs(14), Duration::from_secs(7), Duration::from_secs(14)), + (Duration::from_secs(14), Duration::from_secs(60), Duration::from_secs(14)), + (Duration::from_secs(2), Duration::from_secs(6), Duration::from_secs(2)), + (Duration::MAX, Duration::MAX, Duration::MAX), + ] { + assert_eq!( + DataUsageCache::backup_cache_save_timeout(cap, cost), + expected, + "cap={cap:?}, cost={cost:?}" + ); + } + } + #[test] fn interrupted_checkpoint_admission_is_not_retried() { let error = StorageError::Io(std::io::Error::new(std::io::ErrorKind::Interrupted, "cancelled while queued")); diff --git a/crates/scanner/src/data_usage_define/tests.rs b/crates/scanner/src/data_usage_define/tests.rs index 047d92105..fb6176227 100644 --- a/crates/scanner/src/data_usage_define/tests.rs +++ b/crates/scanner/src/data_usage_define/tests.rs @@ -71,6 +71,10 @@ struct BackupFallbackStore { recover_main_revision: AtomicBool, backup_reads: Mutex, puts: Mutex>, + put_delays: [Duration; 2], + read_delays: [Duration; 2], + put_attempts: [AtomicUsize; 2], + main_put_failures: usize, } impl BackupFallbackStore { @@ -82,6 +86,10 @@ impl BackupFallbackStore { recover_main_revision: AtomicBool::new(recover_main_revision), backup_reads: Mutex::new(0), puts: Mutex::new(Vec::new()), + put_delays: [Duration::ZERO; 2], + read_delays: [Duration::ZERO; 2], + put_attempts: std::array::from_fn(|_| AtomicUsize::new(0)), + main_put_failures: 0, } } @@ -93,6 +101,10 @@ impl BackupFallbackStore { recover_main_revision: AtomicBool::new(true), backup_reads: Mutex::new(0), puts: Mutex::new(Vec::new()), + put_delays: [Duration::ZERO; 2], + read_delays: [Duration::ZERO; 2], + put_attempts: std::array::from_fn(|_| AtomicUsize::new(0)), + main_put_failures: 0, } } @@ -113,6 +125,18 @@ impl BackupFallbackStore { enum CacheReadBody { Bytes(Vec), PrefixThenError(Vec), + #[cfg(feature = "diagnostic-read-version-caller")] + Pending, +} + +#[cfg(feature = "diagnostic-read-version-caller")] +struct PendingCacheReader; + +#[cfg(feature = "diagnostic-read-version-caller")] +impl AsyncRead for PendingCacheReader { + fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &mut ReadBuf<'_>) -> Poll> { + Poll::Pending + } } #[derive(Debug)] @@ -138,7 +162,10 @@ impl AsyncRead for PrefixThenErrorReader { struct CacheReadStore { main: CacheReadBody, backup: Option>, + legacy: Option>, puts: AtomicUsize, + #[cfg(feature = "diagnostic-read-version-caller")] + entry_sources: Mutex>>, } #[derive(Debug, Default)] @@ -152,7 +179,10 @@ impl CacheReadStore { Self { main, backup, + legacy: None, puts: AtomicUsize::new(0), + #[cfg(feature = "diagnostic-read-version-caller")] + entry_sources: Mutex::new(Vec::new()), } } @@ -163,6 +193,8 @@ impl CacheReadStore { prefix: Cursor::new(prefix), failed: false, }), + #[cfg(feature = "diagnostic-read-version-caller")] + CacheReadBody::Pending => Box::new(PendingCacheReader), }; ScannerGetObjectReader { stream, @@ -194,6 +226,18 @@ impl ObjectIO for CacheReadStore { _h: Self::HeaderMap, _opts: &Self::ObjectOptions, ) -> StorageResult { + #[cfg(feature = "diagnostic-read-version-caller")] + self.entry_sources + .lock() + .await + .push(crate::storage_api::ecstore_diagnostic::current_read_version_entry_source()); + if bucket == DATA_USAGE_BUCKET.as_str() { + return self + .legacy + .clone() + .map(|data| Self::reader(CacheReadBody::Bytes(data), "legacy-etag")) + .ok_or(Error::FileNotFound); + } if bucket != RUSTFS_META_BUCKET { return Err(Error::FileNotFound); } @@ -326,6 +370,10 @@ impl ObjectIO for BackupFallbackStore { let main_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]); let backup_path = format!("{main_path}.bkp"); + let delay = self.read_delays[usize::from(object.ends_with(".bkp"))]; + if !delay.is_zero() { + sleep(delay).await; + } if object == main_path { let read = self.main_reads.fetch_add(1, Ordering::SeqCst); if read == 0 || !self.recover_main_revision.load(Ordering::SeqCst) { @@ -361,6 +409,15 @@ impl ObjectIO for BackupFallbackStore { .as_ref() .and_then(HTTPPreconditions::if_none_match_value) .map(str::to_owned); + let path_index = usize::from(object.ends_with(".bkp")); + let attempt = self.put_attempts[path_index].fetch_add(1, Ordering::SeqCst); + let delay = self.put_delays[path_index]; + if !delay.is_zero() { + sleep(delay).await; + } + if path_index == 0 && attempt < self.main_put_failures { + return Err(Error::other("ambiguous cache write failure")); + } self.puts.lock().await.push(CachePutRecord { object: object.to_string(), if_match, @@ -400,6 +457,53 @@ fn cache_revisions_map_to_compare_and_swap_preconditions() { assert!(existing.if_none_match_value().is_none()); } +#[cfg(feature = "diagnostic-read-version-caller")] +#[tokio::test] +async fn diagnostic_revision_readers_keep_scanner_or_earlier_entry() { + use crate::storage_api::ecstore_diagnostic::{ + ReadVersionEntrySource, current_read_version_entry_source, with_read_version_entry, + }; + let path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]); + let payload = b"scanner revision diagnostic".to_vec(); + for earlier in [None, Some(ReadVersionEntrySource::TierConfigReload)] { + let store = Arc::new(CacheReadStore::new(CacheReadBody::Bytes(payload.clone()), None)); + let read = async { + let (body, revision) = read_config_with_revision(store.clone(), &path) + .await + .expect("revision and body read"); + assert_eq!(body, Some(payload.clone())); + assert_eq!(revision, DataUsageCacheRevision::Etag("main-etag".to_string())); + assert_eq!( + read_config_revision(store.clone(), &path).await.expect("revision-only read"), + DataUsageCacheRevision::Etag("main-etag".to_string()) + ); + let missing = format!("{path}.missing"); + assert_eq!( + read_config_with_revision(store.clone(), &missing) + .await + .expect("missing body is harmless"), + (None, DataUsageCacheRevision::Missing) + ); + assert_eq!( + read_config_revision(store.clone(), &missing) + .await + .expect("missing revision is harmless"), + DataUsageCacheRevision::Missing + ); + }; + match earlier { + Some(source) => with_read_version_entry(source, read).await, + None => read.await, + } + assert_eq!( + *store.entry_sources.lock().await, + vec![Some(earlier.unwrap_or(ReadVersionEntrySource::ScannerRevisionRead)); 4], + "both readers and missing paths preserve the earliest diagnostic source" + ); + assert_eq!(current_read_version_entry_source(), None, "reader scopes must not leak to the caller"); + } +} + #[tokio::test] async fn backup_cache_load_recovers_main_revision_before_cas_save() { let mut expected = DataUsageCache::default(); @@ -497,6 +601,71 @@ async fn cache_save_reconciles_an_ambiguous_committed_write() { ); } +#[cfg(feature = "diagnostic-read-version-caller")] +#[tokio::test] +async fn diagnostic_cache_load_attempts_preserve_source_through_fallback_and_timeout() { + use crate::storage_api::ecstore_diagnostic::{ + ReadVersionEntrySource, current_read_version_entry_source, with_read_version_entry, + }; + + let mut cache = DataUsageCache::default(); + cache.info.name = "diagnostic-cache".to_string(); + let bytes = cache.marshal_msg().expect("cache fixture should serialize"); + for earlier in [None, Some(ReadVersionEntrySource::TierConfigReload)] { + for case in ["primary", "legacy", "missing", "body_error", "timeout"] { + let body = match case { + "body_error" => CacheReadBody::PrefixThenError(vec![0x81, 0xa4, b'n', b'a']), + "timeout" => CacheReadBody::Pending, + _ => CacheReadBody::Bytes(bytes.clone()), + }; + let mut fixture = CacheReadStore::new(body, None); + if case == "legacy" { + fixture.legacy = Some(bytes.clone()); + } + let store = Arc::new(fixture); + let name = if matches!(case, "legacy" | "missing") { + "diagnostic-legacy-cache" + } else { + DATA_USAGE_CACHE_NAME + }; + let timeout = if case == "timeout" { + Duration::from_millis(1) + } else { + Duration::from_secs(1) + }; + let load = DataUsageCache::try_load_inner(store.clone(), name, timeout); + let attempt = match earlier { + Some(source) => with_read_version_entry(source, load).await, + None => load.await, + } + .expect("diagnostic wrapper must preserve load-attempt results"); + match (case, attempt) { + ("primary", DataUsageCacheLoadAttempt::Loaded { cache, revision }) => { + assert_eq!(cache.info.name, "diagnostic-cache"); + assert_eq!(revision, Some(DataUsageCacheRevision::Etag("main-etag".to_string()))); + } + ("legacy", DataUsageCacheLoadAttempt::Loaded { cache, revision }) => { + assert_eq!(cache.info.name, "diagnostic-cache"); + assert_eq!(revision, Some(DataUsageCacheRevision::Missing)); + } + ("missing", DataUsageCacheLoadAttempt::Missing { revision }) => { + assert_eq!(revision, Some(DataUsageCacheRevision::Missing)); + } + ("body_error" | "timeout", DataUsageCacheLoadAttempt::Retryable(_)) => {} + _ => panic!("unexpected load-attempt outcome for {case}"), + } + let calls = if matches!(case, "legacy" | "missing") { 2 } else { 1 }; + assert_eq!( + *store.entry_sources.lock().await, + vec![Some(earlier.unwrap_or(ReadVersionEntrySource::DataUsageCacheRead)); calls], + "primary and legacy reads must retain this attempt's entry source: {case}" + ); + assert_eq!(store.puts.load(Ordering::SeqCst), 0, "load attempts must not save: {case}"); + assert_eq!(current_read_version_entry_source(), None, "scope must restore even after timeout: {case}"); + } + } +} + #[tokio::test] async fn backup_cache_load_fails_closed_without_main_revision_quorum() { let backup = DataUsageCache::default().marshal_msg().expect("serialize backup cache"); @@ -2076,22 +2245,36 @@ fn prepare_bucket_checkpoint_migrates_legacy_epoch_bound_receipt() { ..Default::default() }, ); - let legacy_digest = cache - .legacy_coverage_prefix_digest("bucket/a") - .expect("legacy receipt digest"); - cache.info.scan_coverage_receipt = Some(DataUsageScanCoverageReceipt { - through: "bucket/a".to_string(), - digest: legacy_digest, - }); - assert_eq!(cache.validated_scan_frontier(), Some("bucket/a")); - - assert_eq!( - cache.prepare_bucket_checkpoint("bucket", 9, 2, source, TEST_PLAN_DIGEST, identity), - DataUsageCachePrepareOutcome::Reused - ); - assert_eq!(cache.info.leader_epoch, 2); - assert_eq!(cache.validated_scan_frontier(), Some("bucket/a")); - assert_ne!(cache.info.scan_coverage_receipt.expect("migrated receipt").digest, legacy_digest); + for legacy_digest in [ + cache + .legacy_coverage_prefix_digest("bucket/a") + .expect("epoch-bound receipt digest"), + cache.v2_coverage_prefix_digest("bucket/a").expect("v2 receipt digest"), + ] { + let mut cache = cache.clone(); + cache.info.scan_coverage_receipt = Some(DataUsageScanCoverageReceipt { + through: "bucket/a".to_string(), + digest: legacy_digest, + }); + assert_eq!(cache.validated_scan_frontier(), Some("bucket/a")); + assert_eq!( + cache.prepare_bucket_checkpoint("bucket", 9, 2, source, TEST_PLAN_DIGEST, identity), + DataUsageCachePrepareOutcome::Reused + ); + assert_eq!(cache.info.leader_epoch, 2); + assert_eq!(cache.validated_scan_frontier(), Some("bucket/a")); + let digest = cache.info.scan_coverage_receipt.as_ref().expect("migrated receipt").digest; + assert_ne!(digest, legacy_digest); + assert_ne!(digest, cache.v2_coverage_prefix_digest("bucket/a").expect("old reader v2 digest")); + assert_ne!( + digest, + cache + .legacy_coverage_prefix_digest("bucket/a") + .expect("old reader epoch digest") + ); + cache.cache.get_mut("bucket/a").expect("covered entry").failed_objects = 1; + assert!(cache.validated_scan_frontier().is_none(), "tampered health debt invalidates coverage"); + } } #[test] @@ -2491,12 +2674,32 @@ fn checked_flatten_accepts_depth_limit_and_rejects_deeper_tree() { } #[test] -fn test_find_children_copy_preserves_missing_entry_behavior() { - let mut cache = DataUsageCache::default(); +fn test_find_children_copy_does_not_insert_missing_entries() { + let cache = DataUsageCache::default(); let missing_hash = hash_path("missing"); - assert!(cache.find_children_copy(missing_hash.clone()).is_empty()); - assert!(cache.cache.contains_key(&missing_hash.key())); + assert!(cache.find_children_copy(&missing_hash).is_empty()); + assert!(cache.cache.is_empty()); + assert_eq!(cache.cache.capacity(), 0); +} + +#[test] +fn test_find_children_copy_preserves_existing_children() { + let hash = hash_path("bucket"); + let children = HashSet::from(["bucket/child".to_string()]); + let mut cache = DataUsageCache::default(); + cache.cache.insert( + hash.key(), + DataUsageEntry { + children: children.clone(), + ..Default::default() + }, + ); + let mut copied = cache.find_children_copy(&hash); + assert_eq!(copied, children); + copied.clear(); + assert_eq!(cache.cache[&hash.0].children, children); + assert_eq!(cache.cache.len(), 1); } #[test] @@ -2533,6 +2736,88 @@ fn test_cache_path_type_distinguishes_main_and_backup() { assert_eq!(DataUsageCache::cache_path_type("buckets/.usage-cache.bin.bkp"), "backup"); } +#[tokio::test(start_paused = true)] +#[serial_test::serial] +async fn test_save_backup_budget_tracks_primary_cost() { + temp_env::async_with_vars([(ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, Some("14"))], async { + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + let mut store = BackupFallbackStore::new(b"old backup".to_vec(), true); + store.put_delays = [Duration::from_secs(6); 2]; + let store = Arc::new(store); + let revisions = DataUsageCacheRevisions { + main: DataUsageCacheRevision::Missing, + backup: Some(DataUsageCacheRevision::Etag("backup-etag".to_string())), + }; + DataUsageCache::default() + .save_with_revisions(store.clone(), DATA_USAGE_CACHE_NAME, &revisions) + .await + .expect("a backup with the observed primary cost should complete"); + let puts = store.puts.lock().await; + assert_eq!(puts.len(), 2, "best-effort save success must not hide a stale backup"); + assert!(puts[1].object.ends_with(".bkp")); + assert_eq!(puts[0].if_none_match.as_deref(), Some("*")); + assert_eq!(puts[1].if_match.as_deref(), Some("backup-etag")); + }) + .await; + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); +} + +#[tokio::test(start_paused = true)] +#[serial_test::serial] +async fn test_save_backup_budget_still_bounds_a_stalled_backup() { + temp_env::async_with_vars([(ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, Some("14"))], async { + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + let mut store = BackupFallbackStore::new(b"old backup".to_vec(), true); + store.put_delays = [Duration::from_secs(6), Duration::from_secs(60)]; + let store = Arc::new(store); + let started = Instant::now(); + DataUsageCache::default() + .save(store.clone(), DATA_USAGE_CACHE_NAME) + .await + .expect("backup failure remains best effort after a successful main save"); + assert_eq!(store.puts.lock().await.len(), 1, "stalled backup must not commit"); + assert_eq!(*store.backup_reads.lock().await, 1, "one bounded reconciliation, no backup retry"); + assert!(started.elapsed() <= Duration::from_secs(6) + DataUsageCache::cache_save_timeout() + Duration::from_millis(10)); + }) + .await; + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); +} + +#[tokio::test(start_paused = true)] +#[serial_test::serial] +async fn test_checkpoint_budget_covers_retries_and_reconciliation() { + temp_env::async_with_vars([(ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, Some("14"))], async { + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + let cache = DataUsageCache::default(); + let encoded = rmp_serde::to_vec(&cache).expect("checkpoint should encode"); + for main_put_failures in [2, 3] { + let mut store = BackupFallbackStore::with_recovered_main(b"old backup".to_vec(), encoded.clone()); + store.put_delays = [Duration::from_secs(13), Duration::from_secs(60)]; + store.read_delays = [Duration::from_secs(13); 2]; + store.main_put_failures = main_put_failures; + let store = Arc::new(store); + let revisions = DataUsageCacheRevisions { + main: DataUsageCacheRevision::Missing, + backup: Some(DataUsageCacheRevision::Etag("backup-etag".to_string())), + }; + let started = Instant::now(); + timeout( + DataUsageCache::persistence_timeout(), + cache.save_with_revisions(store.clone(), DATA_USAGE_CACHE_NAME, &revisions), + ) + .await + .expect("the outer checkpoint budget must cover save and reconciliation") + .expect("a successful or reconciled primary must survive a best-effort backup failure"); + assert_eq!(store.put_attempts[0].load(Ordering::SeqCst), 3); + assert_eq!(store.put_attempts[1].load(Ordering::SeqCst), 1, "backup writes must not retry"); + assert_eq!(*store.backup_reads.lock().await, 1, "backup must finish its bounded reconciliation"); + assert!(started.elapsed() < DataUsageCache::persistence_timeout()); + } + }) + .await; + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); +} + #[test] fn test_cache_save_timeout_uses_default_when_env_missing() { with_var_unset(ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, || { @@ -2541,7 +2826,7 @@ fn test_cache_save_timeout_uses_default_when_env_missing() { DataUsageCache::cache_save_timeout(), Duration::from_secs(rustfs_config::DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS) ); - assert_eq!(DataUsageCache::persistence_timeout(), Duration::from_millis(52_350)); + assert_eq!(DataUsageCache::persistence_timeout(), Duration::from_millis(117_500)); }); crate::runtime_config::refresh_scanner_runtime_config_for_tests(); } @@ -2561,10 +2846,10 @@ fn test_cache_save_timeout_respects_env_and_minimum_bound() { } #[test] -fn test_cache_persistence_timeout_covers_all_save_attempts() { +fn test_cache_persistence_timeout_covers_all_save_and_reconciliation_attempts() { with_var(ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, Some("7"), || { crate::runtime_config::refresh_scanner_runtime_config_for_tests(); - assert_eq!(DataUsageCache::persistence_timeout(), Duration::from_millis(31_350)); + assert_eq!(DataUsageCache::persistence_timeout(), Duration::from_millis(61_500)); }); crate::runtime_config::refresh_scanner_runtime_config_for_tests(); } @@ -2821,3 +3106,39 @@ async fn test_retry_save_op_times_out_and_returns_error_after_retries() { assert!(result.is_err()); assert_eq!(attempts.load(Ordering::SeqCst), (DATA_USAGE_CACHE_SAVE_RETRIES + 1) as usize); } + +#[test] +fn scan_frontier_receipt_binds_nested_debt_outside_raw_lexical_prefix() { + let mut cache = DataUsageCache::default(); + cache.prepare_bucket_checkpoint("bucket", 8, 1, DataUsageCacheSource::new(0, 0), TEST_PLAN_DIGEST, valid_scan_identity()); + cache.replace("bucket", "", DataUsageEntry::default()); + cache.replace("bucket/a", "bucket", DataUsageEntry::default()); + cache.replace( + "bucket/a/z", + "bucket/a", + DataUsageEntry { + failed_objects: 1, + ..Default::default() + }, + ); + cache.replace( + "bucket/a-", + "bucket", + DataUsageEntry { + objects: 1, + size: 1, + ..Default::default() + }, + ); + cache + .seal_scan_frontier(Some("bucket/a-")) + .expect("seal traversal after failed subtree"); + assert_eq!(cache.validated_scan_frontier(), Some("bucket/a-")); + let digest = cache.info.scan_coverage_receipt.as_ref().expect("new receipt").digest; + assert_ne!(digest, cache.v2_coverage_prefix_digest("bucket/a-").expect("old reader digest")); + cache.cache.get_mut("bucket/a/z").expect("nested failure").failed_objects = 0; + assert!( + cache.validated_scan_frontier().is_none(), + "nested debt must be signed even when '/' sorts after '-'" + ); +} diff --git a/crates/scanner/src/lib.rs b/crates/scanner/src/lib.rs index 5a4b57d1e..33b7a0e72 100644 --- a/crates/scanner/src/lib.rs +++ b/crates/scanner/src/lib.rs @@ -103,9 +103,9 @@ pub use scanner_io::{ acknowledge_scoped_dirty_usage, clear_dirty_usage_bucket, encode_durable_dirty_usage_producer_replay_record, record_dirty_usage_bucket, record_dirty_usage_bucket_from_producer, record_dirty_usage_bucket_from_producers, record_dirty_usage_object, record_dirty_usage_object_from_producer, record_scanner_maintenance_change, - replay_durable_dirty_usage_producer_record, scanner_activity_epoch, scanner_dirty_usage_bucket_generation, - scanner_dirty_usage_snapshot, scanner_dirty_usage_state, scanner_maintenance_generation, - set_scanner_dirty_usage_clear_observer, set_scanner_dirty_usage_mutation_observer, + record_status_class_mutation, replay_durable_dirty_usage_producer_record, scanner_activity_epoch, + scanner_dirty_usage_bucket_generation, scanner_dirty_usage_snapshot, scanner_dirty_usage_state, + scanner_maintenance_generation, set_scanner_dirty_usage_clear_observer, set_scanner_dirty_usage_mutation_observer, }; pub use segment_invalidation::SegmentInvalidationProducerIdentity; pub use sleeper::{DynamicSleeper, SCANNER_IDLE_MODE, SCANNER_SLEEPER}; @@ -734,17 +734,23 @@ where S: ScannerObjectIO, { let mut reader = ScannerPutObjReader::from_vec(data); - api.put_object( - RUSTFS_META_BUCKET, - file, - &mut reader, - &ScannerObjectOptions { - max_parity: true, - http_preconditions: Some(preconditions), - ..Default::default() + let opts = ScannerObjectOptions { + max_parity: true, + http_preconditions: Some(preconditions), + ..Default::default() + }; + let write = api.put_object(RUSTFS_META_BUCKET, file, &mut reader, &opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let write = crate::storage_api::ecstore_diagnostic::with_read_version_entry( + crate::storage_api::ecstore_diagnostic::ReadVersionEntrySource::ScannerConfigWrite, + async { + crate::storage_api::ecstore_diagnostic::observe_read_version_context_boundary( + crate::storage_api::ecstore_diagnostic::ReadVersionContextBoundary::ScannerConfigWrite, + ); + write.await }, - ) - .await + ); + write.await } pub(crate) async fn save_config_with_publication_admission_for_epoch( @@ -859,19 +865,25 @@ where fence.to_string(), ); } - api.put_object( - RUSTFS_META_BUCKET, - file, - &mut reader, - &ScannerObjectOptions { - max_parity: true, - http_preconditions: Some(preconditions), - scanner_publication_commit_scope, - user_defined, - ..Default::default() + let opts = ScannerObjectOptions { + max_parity: true, + http_preconditions: Some(preconditions), + scanner_publication_commit_scope, + user_defined, + ..Default::default() + }; + let write = api.put_object(RUSTFS_META_BUCKET, file, &mut reader, &opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let write = crate::storage_api::ecstore_diagnostic::with_read_version_entry( + crate::storage_api::ecstore_diagnostic::ReadVersionEntrySource::ScannerConfigWrite, + async { + crate::storage_api::ecstore_diagnostic::observe_read_version_context_boundary( + crate::storage_api::ecstore_diagnostic::ReadVersionContextBoundary::ScannerConfigWrite, + ); + write.await }, - ) - .await + ); + write.await } pub(crate) async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> std::result::Result<(), DiskError> { diff --git a/crates/scanner/src/raw_page_index.rs b/crates/scanner/src/raw_page_index.rs index 023f9f585..dc8d75123 100644 --- a/crates/scanner/src/raw_page_index.rs +++ b/crates/scanner/src/raw_page_index.rs @@ -310,7 +310,8 @@ impl RawEnumerationPageIndex { return Err(RawEnumerationPageIndexError::StaleGeneration); } let entries_start = inner.validated_committed_entries()?.len(); - inner.commit_building_page(entries_start) + inner.commit_building_page(entries_start)?; + inner.pages.last().cloned().ok_or(RawEnumerationPageIndexError::EmptyCommit) } pub fn page(&self, page_index: u64, expected_digest: [u8; 32]) -> Result<&RawEnumerationPage, RawEnumerationPageIndexError> { @@ -347,8 +348,9 @@ impl RawEnumerationPageIndex { impl RawEnumerationPageIndexInner { // Callers validate the committed prefix at the restore/API boundary. The - // live writer maintains its entry count without revisiting that prefix. - fn commit_building_page(&mut self, entries_start: usize) -> Result { + // live writer maintains its entry count without revisiting that prefix, + // and the committed page moves into `pages` without a copy. + fn commit_building_page(&mut self, entries_start: usize) -> Result<(), RawEnumerationPageIndexError> { let Some(building) = self.building.as_ref() else { return Err(RawEnumerationPageIndexError::EmptyCommit); }; @@ -365,9 +367,9 @@ impl RawEnumerationPageIndexInner { }; let page = RawEnumerationPage::new(&self.parent, building); self.complete = page.terminal; - self.pages.push(page.clone()); + self.pages.push(page); self.generation = self.generation.saturating_add(1); - Ok(page) + Ok(()) } fn status(&self) -> RawEnumerationPageOwnerStatus { diff --git a/crates/scanner/src/raw_page_index/writer.rs b/crates/scanner/src/raw_page_index/writer.rs index 9cd7f04da..a6f93738d 100644 --- a/crates/scanner/src/raw_page_index/writer.rs +++ b/crates/scanner/src/raw_page_index/writer.rs @@ -112,8 +112,10 @@ impl RawEnumerationPageWriter { terminal: false, }); self.indexed.insert(entry.clone()); - building.entries.push(entry); - building.entries.sort(); + // The building page is normalized (strictly increasing) on every + // path that reaches here, so a positional insert keeps it so. + let position = building.entries.binary_search(&entry).unwrap_or_else(|position| position); + building.entries.insert(position, entry); building.terminal = false; self.inner.generation = self.inner.generation.saturating_add(1); } diff --git a/crates/scanner/src/remote_scanner.rs b/crates/scanner/src/remote_scanner.rs index 61036e1c8..932e73045 100644 --- a/crates/scanner/src/remote_scanner.rs +++ b/crates/scanner/src/remote_scanner.rs @@ -318,3 +318,60 @@ fn try_admit_remote_scanner_key(disk_key: String) -> Result, +} + +#[derive(Default)] +struct LocalDiskAdmissionState { + admission: Option, + holders: usize, +} + +/// One walker's stake in a [`LocalDiskAdmissionShare`]. +pub(crate) struct LocalDiskAdmissionHold { + share: Arc, +} + +impl LocalDiskAdmissionShare { + pub(crate) fn new() -> Arc { + Arc::new(Self { + state: Mutex::new(LocalDiskAdmissionState::default()), + }) + } + + pub(crate) fn acquire(self: &Arc, disk: &Disk) -> Result { + self.acquire_with(|| try_admit_remote_scanner(disk)) + } + + fn acquire_with( + self: &Arc, + admit: impl FnOnce() -> Result, + ) -> Result { + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if state.admission.is_none() { + state.admission = Some(admit()?); + } + state.holders = state.holders.saturating_add(1); + Ok(LocalDiskAdmissionHold { share: Arc::clone(self) }) + } +} + +impl Drop for LocalDiskAdmissionHold { + fn drop(&mut self) { + let mut state = self.share.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + state.holders = state.holders.saturating_sub(1); + if state.holders == 0 { + state.admission = None; + } + } +} diff --git a/crates/scanner/src/remote_scanner/stream.rs b/crates/scanner/src/remote_scanner/stream.rs index 9ffc3ea2b..3ce3740fd 100644 --- a/crates/scanner/src/remote_scanner/stream.rs +++ b/crates/scanner/src/remote_scanner/stream.rs @@ -18,7 +18,8 @@ use crate::scanner_budget::{ScannerCycleBudget, ScannerCycleBudgetConfig}; use crate::scanner_io::{ DataUsageCacheReuseOptions, DataUsageCacheScanState, ScannerCheckpointPersistContext, ScannerCheckpointPersistResult, ScannerDiskScanOptions, ScannerDiskScanOutcome, ScannerIODisk, acquire_scanner_cache_locks, cache_root_entry_info, - current_cache_root_or_prepare_with_generation, persist_scanner_checkpoint, scanner_set_disk_inventory, + current_cache_root_or_prepare_with_generation, log_saved_partial_bucket_cache, persist_scanner_checkpoint, + scanner_set_disk_inventory, }; use crate::storage_api::owner::NS_SCANNER_PROTOCOL_VERSION; use crate::{ @@ -29,7 +30,7 @@ use crate::{ use hmac::{Hmac, KeyInit, Mac}; use rustfs_credentials::try_get_rpc_token; use rustfs_heal_contracts::heal_channel::HealScanMode; -use rustfs_scanner_metrics::metrics::{Metric, Metrics}; +use rustfs_scanner_metrics::metrics::{CloseDiskGuard, Metric, Metrics, current_path_updater}; use rustfs_utils::path::path_join_buf; use serde::{Deserialize, Serialize}; use sha2::Sha256; @@ -681,7 +682,9 @@ async fn scan_and_persist_local_bucket( .ok_or_else(|| RemoteScannerServerError::worker("remote namespace scanner object layer is unavailable"))?; validate_remote_scanner_request_fence_with_store(next_cycle, leader_epoch, store.clone()) .await - .map_err(|err| RemoteScannerServerError::worker(format!("remote namespace scanner leader fence is stale: {err}")))?; + .map_err(|err| { + RemoteScannerServerError::worker(format!("remote namespace scanner leader fence verification failed: {err}")) + })?; let source = cache_source_for_disk(disk.as_ref()) .map_err(|err| RemoteScannerServerError::worker(format!("remote namespace scanner source is unavailable: {err}")))?; let set = store @@ -853,7 +856,7 @@ async fn scan_and_persist_local_bucket( ) .await { - ScannerCheckpointPersistResult::Saved => { + ScannerCheckpointPersistResult::Saved | ScannerCheckpointPersistResult::Skipped => { if guard.is_lock_lost() { scan_ctx.cancel(); let _ = tokio::time::timeout(NS_SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, scan.as_mut()).await; @@ -869,8 +872,12 @@ async fn scan_and_persist_local_bucket( "remote namespace scanner cache fence changed during checkpoint save", )); } - ScannerCheckpointPersistResult::Failed(_error) => { - checkpoint_channel_closed = true; + ScannerCheckpointPersistResult::RetryBucket(error) => { + scan_ctx.cancel(); + let _ = tokio::time::timeout(NS_SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, scan.as_mut()).await; + return Err(RemoteScannerServerError::retry_bucket(format!( + "remote namespace scanner checkpoint needs reload: {error}", + ))); } } } @@ -882,7 +889,7 @@ async fn scan_and_persist_local_bucket( Err(err) => err, }; return Err(RemoteScannerServerError::worker(format!( - "remote namespace scanner leader fence changed during bucket scan: {}", + "remote namespace scanner leader fence verification failed during bucket scan: {}", err ))); } @@ -919,7 +926,9 @@ async fn scan_and_persist_local_bucket( phase_changed.notify_one(); validate_remote_scanner_request_fence_with_store(next_cycle, leader_epoch, store.clone()) .await - .map_err(|err| RemoteScannerServerError::worker(format!("remote namespace scanner leader fence changed: {err}")))?; + .map_err(|err| { + RemoteScannerServerError::worker(format!("remote namespace scanner leader fence verification failed: {err}")) + })?; let done_save = Metrics::time(Metric::SaveUsage); // Each physical main/backup PUT must still prove the epoch captured before // the scan. A movement transition that starts and ends during the scan @@ -939,7 +948,9 @@ async fn scan_and_persist_local_bucket( } validate_remote_scanner_request_fence_with_store(next_cycle, leader_epoch, store) .await - .map_err(|err| RemoteScannerServerError::worker(format!("remote namespace scanner leader fence changed: {err}")))?; + .map_err(|err| { + RemoteScannerServerError::worker(format!("remote namespace scanner leader fence verification failed: {err}")) + })?; if guard.is_lock_lost() { return Err(RemoteScannerServerError::worker( "remote namespace scanner cache lock was lost during persistence", @@ -947,6 +958,9 @@ async fn scan_and_persist_local_bucket( } if let Some(partial_result) = partial_result { + if matches!(partial_result, RemoteScannerFrameResult::Partial) { + log_saved_partial_bucket_cache(&cache, &cache_name); + } return Ok(partial_result); } if cache.info.source != Some(source) || !cache.info.snapshot_complete { @@ -1013,71 +1027,79 @@ pub(crate) async fn scan_remote_bucket( ))); } - let rpc_deadline = Instant::now() + NS_SCANNER_MAX_RPC_LIFETIME; - let open_deadline = (Instant::now() + NS_SCANNER_STALL_TIMEOUT).min(rpc_deadline); - let open_stream = disk.open_ns_scanner_stream(crate::NsScannerOpenRequest { - request_id, - server_epoch, - session_id, - session_sequence, - next_cycle, - leader_epoch, - body, - stall_timeout: Some(NS_SCANNER_STALL_TIMEOUT), - }); - tokio::pin!(open_stream); - let reader = tokio::select! { - _ = ctx.cancelled() => { - budget.cancel_after_unreported_remote_progress(); - return Err(RemoteScannerFailure::transport(StorageError::other( - "remote namespace scanner cancelled while opening stream", - ))); - } - result = tokio::time::timeout_at(open_deadline, &mut open_stream) => { - match result { - Err(_) => { - budget.cancel_after_unreported_remote_progress(); - return Err(RemoteScannerFailure::transport(StorageError::other( - "remote namespace scanner response headers timed out", - ))); - } - Ok(Err(err)) if err.is_internode_http_status(429) => { - return Err(RemoteScannerFailure::retry_bucket(StorageError::other(format!( - "remote namespace scanner worker rejected zero-progress work: {err}" - )))); - } - Ok(Err(err)) => { - budget.cancel_after_unreported_remote_progress(); - return Err(RemoteScannerFailure::transport(StorageError::other(format!( - "failed to open remote namespace scanner stream: {err}" - )))); - } - Ok(Ok(reader)) => reader, - } - } - }; - let authenticator = FrameAuthenticator::from_rpc_secret(request_id).map_err(|err| { - RemoteScannerFailure::transport(StorageError::other(format!( - "failed to authenticate remote namespace scanner stream: {err}" - ))) - })?; - - let stream_result = consume_remote_scanner_stream_until( - reader, - ctx, - budget.clone(), - RemoteScannerResponseExpectation { - bucket, - source: expected_source, + let disk_name = format!("{}{}", disk.host_name(), disk.path().display()); + let (_, close_path) = current_path_updater(&disk_name, bucket).await; + let mut path_guard = CloseDiskGuard::new(close_path); + let result = async { + let rpc_deadline = Instant::now() + NS_SCANNER_MAX_RPC_LIFETIME; + let open_deadline = (Instant::now() + NS_SCANNER_STALL_TIMEOUT).min(rpc_deadline); + let open_stream = disk.open_ns_scanner_stream(crate::NsScannerOpenRequest { + request_id, + server_epoch, + session_id, + session_sequence, next_cycle, - scan_plan_digest, - tier_registry_generation, - }, - authenticator, - rpc_deadline, - ) + leader_epoch, + body, + stall_timeout: Some(NS_SCANNER_STALL_TIMEOUT), + }); + tokio::pin!(open_stream); + let reader = tokio::select! { + _ = ctx.cancelled() => { + budget.cancel_after_unreported_remote_progress(); + return Err(RemoteScannerFailure::transport(StorageError::other( + "remote namespace scanner cancelled while opening stream", + ))); + } + result = tokio::time::timeout_at(open_deadline, &mut open_stream) => { + match result { + Err(_) => { + budget.cancel_after_unreported_remote_progress(); + return Err(RemoteScannerFailure::transport(StorageError::other( + "remote namespace scanner response headers timed out", + ))); + } + Ok(Err(err)) if err.is_internode_http_status(429) => { + return Err(RemoteScannerFailure::retry_bucket(StorageError::other(format!( + "remote namespace scanner worker rejected zero-progress work: {err}" + )))); + } + Ok(Err(err)) => { + budget.cancel_after_unreported_remote_progress(); + return Err(RemoteScannerFailure::transport(StorageError::other(format!( + "failed to open remote namespace scanner stream: {err}" + )))); + } + Ok(Ok(reader)) => reader, + } + } + }; + let authenticator = FrameAuthenticator::from_rpc_secret(request_id).map_err(|err| { + RemoteScannerFailure::transport(StorageError::other(format!( + "failed to authenticate remote namespace scanner stream: {err}" + ))) + })?; + + let stream_result = consume_remote_scanner_stream_until( + reader, + ctx, + budget.clone(), + RemoteScannerResponseExpectation { + bucket, + source: expected_source, + next_cycle, + scan_plan_digest, + tier_registry_generation, + }, + authenticator, + rpc_deadline, + ) + .await; + finish_remote_scanner_stream(stream_result, budget.as_ref()) + } .await; - finish_remote_scanner_stream(stream_result, budget.as_ref()) + path_guard.close().await; + result } fn finish_remote_scanner_stream( diff --git a/crates/scanner/src/remote_scanner/stream/tests.rs b/crates/scanner/src/remote_scanner/stream/tests.rs index b73b1077d..06f2e6328 100644 --- a/crates/scanner/src/remote_scanner/stream/tests.rs +++ b/crates/scanner/src/remote_scanner/stream/tests.rs @@ -785,6 +785,50 @@ fn admission_rejects_busy_disk_before_replay_state_changes() { assert!(try_admit_remote_scanner_key(key).is_ok()); } +#[test] +fn local_disk_admission_share_holds_one_permit_until_the_last_walker_leaves() { + let key = format!("test-disk-{}", Uuid::new_v4()); + let share = LocalDiskAdmissionShare::new(); + let first = share + .acquire_with(|| try_admit_remote_scanner_key(key.clone())) + .expect("first walker acquires the disk permit"); + let second = share + .acquire_with(|| try_admit_remote_scanner_key(key.clone())) + .expect("second walker shares the held permit"); + assert!( + matches!(try_admit_remote_scanner_key(key.clone()), Err(ScannerError::RemoteDiskBusy)), + "a remote request sees one busy disk while any walker is active" + ); + drop(first); + assert!( + matches!(try_admit_remote_scanner_key(key.clone()), Err(ScannerError::RemoteDiskBusy)), + "the permit is released by the last holder, not the first" + ); + drop(second); + assert!(try_admit_remote_scanner_key(key).is_ok(), "no walker left: the disk is free again"); +} + +#[test] +fn local_disk_admission_share_retries_after_a_rejected_admission() { + let key = format!("test-disk-{}", Uuid::new_v4()); + let share = LocalDiskAdmissionShare::new(); + let remote = try_admit_remote_scanner_key(key.clone()).expect("remote request occupies the disk"); + assert!( + matches!( + share.acquire_with(|| try_admit_remote_scanner_key(key.clone())), + Err(ScannerError::RemoteDiskBusy) + ), + "walkers do not pre-empt an active remote session" + ); + drop(remote); + let hold = share + .acquire_with(|| try_admit_remote_scanner_key(key.clone())) + .expect("a rejected attempt leaves the share empty for a later retry"); + assert!(matches!(try_admit_remote_scanner_key(key.clone()), Err(ScannerError::RemoteDiskBusy))); + drop(hold); + assert!(try_admit_remote_scanner_key(key).is_ok()); +} + #[test] fn replay_cache_requires_contiguous_sequence_per_disk_session() { let session_id = Uuid::new_v4(); @@ -917,18 +961,20 @@ async fn complete_terminal_frame_reconciles_progress_and_usage() { let (mut writer, reader) = tokio::io::duplex(4096); tokio::spawn(async move { let mut sequence = 0; - write_frame( - &mut writer, - &writer_auth, - &mut sequence, - &RemoteScannerFrame::progress(RemoteScannerProgress { - objects_scanned: 2, - directories_started: 1, - ..Default::default() - }), - ) - .await - .expect("progress should write"); + for _ in 0..2 { + write_frame( + &mut writer, + &writer_auth, + &mut sequence, + &RemoteScannerFrame::progress(RemoteScannerProgress { + objects_scanned: 2, + directories_started: 1, + ..Default::default() + }), + ) + .await + .expect("progress should write"); + } write_frame( &mut writer, &writer_auth, @@ -961,10 +1007,26 @@ async fn complete_terminal_frame_reconciles_progress_and_usage() { ..Default::default() }, ); - let outcome = - consume_remote_scanner_stream(reader, parent, budget.clone(), "bucket", TEST_SOURCE, TEST_PLAN_DIGEST, reader_auth) - .await - .expect("stream should complete"); + let metrics = Metrics::new(); + let start = metrics + .start_scan_cycle_work_with_cycle_and_remote_progress(Default::default(), budget.remote_scan_progress()) + .await; + let outcome = consume_remote_scanner_stream_until( + reader, + parent, + budget.clone(), + RemoteScannerResponseExpectation { + bucket: "bucket", + source: TEST_SOURCE, + next_cycle: TEST_NEXT_CYCLE, + scan_plan_digest: TEST_PLAN_DIGEST, + tier_registry_generation: 0, + }, + reader_auth, + Instant::now() + NS_SCANNER_MAX_RPC_LIFETIME, + ) + .await + .expect("stream should complete"); let RemoteScannerOutcome::Complete { usage, @@ -976,6 +1038,15 @@ async fn complete_terminal_frame_reconciles_progress_and_usage() { assert_eq!(usage.entry.objects, 3); assert!(pending_maintenance_work); assert_eq!(budget.progress(), (3, 2)); + let active = metrics.report().await; + assert_eq!(active.current_cycle_objects_scanned, 3); + assert_eq!(active.current_cycle_directories_scanned, 2); + assert_eq!(metrics.lifetime(Metric::ScanObject), 0); + assert_eq!(metrics.lifetime(Metric::ScanFolder), 0); + metrics.finish_scan_cycle_work(start); + let finished = metrics.report().await; + assert_eq!(finished.last_cycle_objects_scanned, 3); + assert_eq!(finished.last_cycle_directories_scanned, 2); } #[tokio::test] @@ -1271,6 +1342,39 @@ fn uncertain_stream_failure_cancels_count_budget() { assert_eq!(budget.reason(), Some(crate::scanner_budget::ScannerCycleBudgetReason::Objects)); } +#[tokio::test] +async fn remote_progress_rejects_regression_and_counts_new_attempts_once() { + let parent = CancellationToken::new(); + let budget = ScannerCycleBudget::new_with_progress_tracking(&parent, ScannerCycleBudgetConfig::default()); + let metrics = Metrics::new(); + let _start = metrics + .start_scan_cycle_work_with_cycle_and_remote_progress(Default::default(), budget.remote_scan_progress()) + .await; + let mut last = RemoteScannerProgress::default(); + let progress = RemoteScannerProgress { + objects_scanned: 4, + directories_started: 2, + entries_visited: 6, + }; + assert!(apply_remote_progress(&budget, &mut last, progress).expect("first progress must advance")); + assert!(!apply_remote_progress(&budget, &mut last, progress).expect("duplicate progress must be accepted")); + let regression = RemoteScannerProgress { + objects_scanned: 3, + ..progress + }; + assert!(apply_remote_progress(&budget, &mut last, regression).is_err()); + assert_eq!(last, progress); + assert_eq!(metrics.report().await.current_cycle_objects_scanned, 4); + assert_eq!(metrics.report().await.current_cycle_directories_scanned, 2); + + let mut retry = RemoteScannerProgress::default(); + apply_remote_progress(&budget, &mut retry, progress).expect("new attempt counts repeated work"); + assert_eq!(budget.progress(), (8, 4)); + let report = metrics.report().await; + assert_eq!(report.current_cycle_objects_scanned, 8); + assert_eq!(report.current_cycle_directories_scanned, 4); +} + #[test] fn directory_entry_activity_counts_as_semantic_progress() { let parent = CancellationToken::new(); diff --git a/crates/scanner/src/runtime_config.rs b/crates/scanner/src/runtime_config.rs index cdace6a8f..6cf216092 100644 --- a/crates/scanner/src/runtime_config.rs +++ b/crates/scanner/src/runtime_config.rs @@ -20,17 +20,17 @@ use rustfs_config::{ DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE, DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS, DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS, DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES, DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS, DEFAULT_SCANNER_CYCLE_MAX_OBJECTS, DEFAULT_SCANNER_IDLE_MODE, DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS, - DEFAULT_SCANNER_SPEED, DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, ENV_SCANNER_ALERT_EXCESS_FOLDERS, - ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE, ENV_SCANNER_ALERT_EXCESS_VERSIONS, ENV_SCANNER_BITROT_CYCLE_SECS, - ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, ENV_SCANNER_CYCLE, ENV_SCANNER_CYCLE_MAX_DIRECTORIES, + DEFAULT_SCANNER_SPEED, DEFAULT_SCANNER_WALKERS_PER_DISK, DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, + ENV_SCANNER_ALERT_EXCESS_FOLDERS, ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE, ENV_SCANNER_ALERT_EXCESS_VERSIONS, + ENV_SCANNER_BITROT_CYCLE_SECS, ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, ENV_SCANNER_CYCLE, ENV_SCANNER_CYCLE_MAX_DIRECTORIES, ENV_SCANNER_CYCLE_MAX_DURATION_SECS, ENV_SCANNER_CYCLE_MAX_OBJECTS, ENV_SCANNER_DELAY, ENV_SCANNER_IDLE_MODE, ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS, ENV_SCANNER_MAX_CONCURRENT_SET_SCANS, ENV_SCANNER_MAX_WAIT_SECS, ENV_SCANNER_SPEED, - ENV_SCANNER_START_DELAY_SECS, ENV_SCANNER_YIELD_EVERY_N_OBJECTS, HEAL_BITROT_CYCLE, HEAL_SUB_SYS, - SCANNER_ALERT_EXCESS_FOLDERS, SCANNER_ALERT_EXCESS_VERSION_SIZE, SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, - SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, - SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE, SCANNER_MAX_CONCURRENT_DISK_SCANS, - SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_SPEED, SCANNER_START_DELAY, SCANNER_SUB_SYS, - SCANNER_YIELD_EVERY_N_OBJECTS, ScannerSpeed, + ENV_SCANNER_START_DELAY_SECS, ENV_SCANNER_WALKERS_PER_DISK, ENV_SCANNER_YIELD_EVERY_N_OBJECTS, HEAL_BITROT_CYCLE, + HEAL_SUB_SYS, MAX_SCANNER_WALKERS_PER_DISK, SCANNER_ALERT_EXCESS_FOLDERS, SCANNER_ALERT_EXCESS_VERSION_SIZE, + SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, + SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE, + SCANNER_MAX_CONCURRENT_DISK_SCANS, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_SPEED, SCANNER_START_DELAY, + SCANNER_SUB_SYS, SCANNER_WALKERS_PER_DISK, SCANNER_YIELD_EVERY_N_OBJECTS, ScannerSpeed, server_config::{Config as ServerConfig, KVS}, }; use serde::Serialize; @@ -53,6 +53,7 @@ const ENV_SCANNER_START_DELAY_SECS_DEPRECATED: &str = "RUSTFS_DATA_SCANNER_START const NO_DEFAULT_CYCLE_OVERRIDE: u64 = 0; const MAX_SCANNER_DELAY_FACTOR: f64 = 10_000.0; const SCANNER_DELAY_RANGE_REASON: &str = "expected scanner delay between 0 and 10000"; +const SCANNER_WALKERS_PER_DISK_RANGE_REASON: &str = "expected walkers per disk between 1 and 16"; static SCANNER_DEFAULT_CYCLE_SECS: AtomicU64 = AtomicU64::new(NO_DEFAULT_CYCLE_OVERRIDE); static SCANNER_RUNTIME_CONFIG_GENERATION: AtomicU64 = AtomicU64::new(0); @@ -96,6 +97,8 @@ pub(crate) struct ScannerRuntimeConfig { pub(crate) max_concurrent_set_scans_source: ScannerRuntimeConfigSource, pub(crate) max_concurrent_disk_scans: usize, pub(crate) max_concurrent_disk_scans_source: ScannerRuntimeConfigSource, + pub(crate) walkers_per_disk: usize, + pub(crate) walkers_per_disk_source: ScannerRuntimeConfigSource, pub(crate) yield_every_n_objects: u64, pub(crate) yield_every_n_objects_source: ScannerRuntimeConfigSource, pub(crate) alert_excess_versions: u64, @@ -138,6 +141,8 @@ impl Default for ScannerRuntimeConfig { max_concurrent_set_scans_source: ScannerRuntimeConfigSource::Default, max_concurrent_disk_scans: DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, max_concurrent_disk_scans_source: ScannerRuntimeConfigSource::Default, + walkers_per_disk: DEFAULT_SCANNER_WALKERS_PER_DISK, + walkers_per_disk_source: ScannerRuntimeConfigSource::Default, yield_every_n_objects: DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, yield_every_n_objects_source: ScannerRuntimeConfigSource::Default, alert_excess_versions: DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS, @@ -181,6 +186,7 @@ pub struct ScannerRuntimeConfigStatus { pub cache_save_timeout_seconds: ScannerRuntimeConfigValue, pub max_concurrent_set_scans: ScannerRuntimeConfigValue, pub max_concurrent_disk_scans: ScannerRuntimeConfigValue, + pub walkers_per_disk: ScannerRuntimeConfigValue, pub yield_every_n_objects: ScannerRuntimeConfigValue, pub alert_excess_versions: ScannerRuntimeConfigValue, pub alert_excess_version_size: ScannerRuntimeConfigValue, @@ -392,6 +398,9 @@ fn validate_persisted_scanner_runtime_config(config: &ServerConfig) -> Result<() validate_optional_config_u64(scanner_kvs, SCANNER_CACHE_SAVE_TIMEOUT, DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS)?; validate_optional_config_usize(scanner_kvs, SCANNER_MAX_CONCURRENT_SET_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS)?; validate_optional_config_usize(scanner_kvs, SCANNER_MAX_CONCURRENT_DISK_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS)?; + if let Some(value) = config_value(scanner_kvs, SCANNER_WALKERS_PER_DISK, DEFAULT_SCANNER_WALKERS_PER_DISK) { + walkers_per_disk_from_count(SCANNER_WALKERS_PER_DISK, parse_config_usize(SCANNER_WALKERS_PER_DISK, value)?)?; + } validate_optional_config_u64(scanner_kvs, SCANNER_YIELD_EVERY_N_OBJECTS, DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS)?; validate_optional_config_u64(scanner_kvs, SCANNER_ALERT_EXCESS_VERSIONS, DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS)?; validate_optional_config_u64(scanner_kvs, SCANNER_ALERT_EXCESS_VERSION_SIZE, DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE)?; @@ -484,6 +493,39 @@ fn cycle_duration_from_secs(key: &'static str, secs: u64) -> Result Result { + if (1..=MAX_SCANNER_WALKERS_PER_DISK).contains(&count) { + Ok(count) + } else { + Err(invalid_value(key, count.to_string(), SCANNER_WALKERS_PER_DISK_RANGE_REASON)) + } +} + +fn lookup_walkers_per_disk(kvs: Option<&KVS>) -> Result<(usize, ScannerRuntimeConfigSource), ScannerRuntimeConfigError> { + match rustfs_utils::get_env_parse_outcome::(ENV_SCANNER_WALKERS_PER_DISK) { + rustfs_utils::EnvParseOutcome::Parsed(count) => { + return walkers_per_disk_from_count(ENV_SCANNER_WALKERS_PER_DISK, count) + .map(|count| (count, ScannerRuntimeConfigSource::Env)); + } + rustfs_utils::EnvParseOutcome::Invalid => { + return Err(invalid_value( + ENV_SCANNER_WALKERS_PER_DISK, + "", + "expected unsigned integer count", + )); + } + rustfs_utils::EnvParseOutcome::Absent => {} + } + + if let Some(value) = config_value(kvs, SCANNER_WALKERS_PER_DISK, DEFAULT_SCANNER_WALKERS_PER_DISK) { + let count = parse_config_usize(SCANNER_WALKERS_PER_DISK, value)?; + return walkers_per_disk_from_count(SCANNER_WALKERS_PER_DISK, count) + .map(|count| (count, ScannerRuntimeConfigSource::Config)); + } + + Ok((DEFAULT_SCANNER_WALKERS_PER_DISK, ScannerRuntimeConfigSource::Default)) +} + fn lookup_start_delay(kvs: Option<&KVS>) -> Result<(Option, ScannerRuntimeConfigSource), ScannerRuntimeConfigError> { let aliases = [ENV_SCANNER_START_DELAY_SECS_DEPRECATED]; if let Some(secs) = rustfs_utils::get_env_opt_u64_with_aliases(ENV_SCANNER_START_DELAY_SECS, &aliases) { @@ -639,6 +681,7 @@ pub(crate) fn lookup_scanner_runtime_config( ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, )?; + let (walkers_per_disk, walkers_per_disk_source) = lookup_walkers_per_disk(scanner_kvs)?; let (yield_every_n_objects, yield_every_n_objects_source) = lookup_u64( scanner_kvs, SCANNER_YIELD_EVERY_N_OBJECTS, @@ -689,6 +732,8 @@ pub(crate) fn lookup_scanner_runtime_config( max_concurrent_set_scans_source, max_concurrent_disk_scans, max_concurrent_disk_scans_source, + walkers_per_disk, + walkers_per_disk_source, yield_every_n_objects, yield_every_n_objects_source, alert_excess_versions, @@ -825,6 +870,10 @@ pub fn scanner_runtime_config_status() -> ScannerRuntimeConfigStatus { value: config.max_concurrent_disk_scans, source: config.max_concurrent_disk_scans_source, }, + walkers_per_disk: ScannerRuntimeConfigValue { + value: config.walkers_per_disk, + source: config.walkers_per_disk_source, + }, yield_every_n_objects: ScannerRuntimeConfigValue { value: config.yield_every_n_objects, source: config.yield_every_n_objects_source, @@ -872,6 +921,10 @@ pub(crate) fn scanner_max_concurrent_disk_scans_configured() -> usize { current_scanner_runtime_config().max_concurrent_disk_scans } +pub(crate) fn scanner_walkers_per_disk_configured() -> usize { + current_scanner_runtime_config().walkers_per_disk +} + pub(crate) fn scanner_alert_excess_versions() -> u64 { current_scanner_runtime_config().alert_excess_versions } @@ -892,9 +945,10 @@ mod tests { use rustfs_config::{ DEFAULT_DELIMITER, DEFAULT_HEAL_BITROT_CYCLE_SECS, ENV_SCANNER_BITROT_CYCLE_SECS, ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, ENV_SCANNER_CYCLE, ENV_SCANNER_CYCLE_MAX_DURATION_SECS, ENV_SCANNER_CYCLE_MAX_OBJECTS, ENV_SCANNER_DELAY, - ENV_SCANNER_MAX_WAIT_SECS, ENV_SCANNER_SPEED, HEAL_BITROT_CYCLE, HEAL_SUB_SYS, SCANNER_BITROT_CYCLE, - SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, - SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE, SCANNER_SPEED, SCANNER_SUB_SYS, ScannerSpeed, + ENV_SCANNER_MAX_WAIT_SECS, ENV_SCANNER_SPEED, ENV_SCANNER_WALKERS_PER_DISK, HEAL_BITROT_CYCLE, HEAL_SUB_SYS, + SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, + SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE, SCANNER_SPEED, SCANNER_SUB_SYS, + SCANNER_WALKERS_PER_DISK, ScannerSpeed, }; use serial_test::serial; use std::collections::HashMap; @@ -1150,6 +1204,65 @@ mod tests { }); } + #[test] + #[serial] + fn scanner_runtime_config_walkers_per_disk_defaults_to_one_walker() { + let config = server_config_with_scanner(&[]); + + with_var_unset(ENV_SCANNER_WALKERS_PER_DISK, || { + let resolved = lookup_scanner_runtime_config(Some(&config)).expect("scanner runtime config"); + + assert_eq!(resolved.walkers_per_disk, 1); + assert_eq!(resolved.walkers_per_disk_source, ScannerRuntimeConfigSource::Default); + }); + } + + #[test] + #[serial] + fn scanner_runtime_config_walkers_per_disk_accepts_persisted_range_and_rejects_zero_or_excess() { + with_var_unset(ENV_SCANNER_WALKERS_PER_DISK, || { + for (value, expected) in [("1", 1), ("4", 4), ("16", 16)] { + let config = server_config_with_scanner(&[(SCANNER_WALKERS_PER_DISK, value)]); + validate_scanner_runtime_config(&config).expect("walkers per disk inside the range should validate"); + let resolved = lookup_scanner_runtime_config(Some(&config)).expect("scanner runtime config"); + assert_eq!(resolved.walkers_per_disk, expected, "persisted {value}"); + let expected_source = if expected == 1 { + ScannerRuntimeConfigSource::Default + } else { + ScannerRuntimeConfigSource::Config + }; + assert_eq!(resolved.walkers_per_disk_source, expected_source, "persisted {value}"); + } + for value in ["0", "17", "-1", "many"] { + let config = server_config_with_scanner(&[(SCANNER_WALKERS_PER_DISK, value)]); + let err = validate_scanner_runtime_config(&config).expect_err("walkers per disk outside the range must fail"); + assert!(err.to_string().contains("walkers_per_disk"), "persisted {value}: {err}"); + assert!( + lookup_scanner_runtime_config(Some(&config)).is_err(), + "persisted {value} must not resolve into a runtime config" + ); + } + }); + } + + #[test] + #[serial] + fn scanner_runtime_config_walkers_per_disk_env_overrides_and_is_range_checked() { + let config = server_config_with_scanner(&[(SCANNER_WALKERS_PER_DISK, "2")]); + + with_var(ENV_SCANNER_WALKERS_PER_DISK, Some("3"), || { + let resolved = lookup_scanner_runtime_config(Some(&config)).expect("scanner runtime config"); + assert_eq!(resolved.walkers_per_disk, 3); + assert_eq!(resolved.walkers_per_disk_source, ScannerRuntimeConfigSource::Env); + }); + for value in ["0", "17", "x"] { + with_var(ENV_SCANNER_WALKERS_PER_DISK, Some(value), || { + let err = lookup_scanner_runtime_config(Some(&config)).expect_err("env walkers per disk is range checked"); + assert!(err.to_string().contains(ENV_SCANNER_WALKERS_PER_DISK), "env {value}: {err}"); + }); + } + } + #[test] fn scanner_runtime_config_validation_rejects_non_default_target() { let config = server_config_with_scanner_target("analytics", &[(SCANNER_SPEED, "slow")]); diff --git a/crates/scanner/src/scanner.rs b/crates/scanner/src/scanner.rs index c41b01176..1638e133e 100644 --- a/crates/scanner/src/scanner.rs +++ b/crates/scanner/src/scanner.rs @@ -51,9 +51,9 @@ use rustfs_data_usage::observed_data_usage_is_newer; use rustfs_heal_contracts::heal_channel::HealScanMode; use rustfs_lock::{NamespaceLockGuard, error::LockError}; use rustfs_scanner_metrics::metrics::{ - CurrentCycle, Metric, Metrics, ScanCyclePartialReason, ScanCycleWorkSnapshot, ScannerUsageSaveResult, ScannerWorkSource, - emit_scan_cycle_complete, emit_scan_cycle_deferred, emit_scan_cycle_partial_with_source, emit_scan_cycle_superseded, - global_metrics, + CurrentCycle, Metric, Metrics, RemoteScanProgress, ScanCyclePartialReason, ScanCycleWorkSnapshot, ScannerUsageSaveResult, + ScannerWorkSource, emit_scan_cycle_complete, emit_scan_cycle_deferred, emit_scan_cycle_partial_with_source, + emit_scan_cycle_superseded, global_metrics, }; use serde::{Deserialize, Serialize}; use sha2::{Digest as _, Sha256}; @@ -1157,10 +1157,17 @@ impl ScannerMaintenanceFeatures { self.lifecycle || self.replication || self.inspection_failed } - fn requires_full_scan(self, observed_generation: Option, current_generation: u64, wake: ScannerCycleWakeReason) -> bool { + fn requires_full_scan( + self, + observed_generation: Option, + current_generation: u64, + wake: ScannerCycleWakeReason, + superseded_retry: bool, + ) -> bool { self.needs_regular_cycle() || observed_generation != Some(current_generation) - || !matches!(wake, ScannerCycleWakeReason::DirtyUsage | ScannerCycleWakeReason::ClusterActivity) + || !(matches!(wake, ScannerCycleWakeReason::DirtyUsage | ScannerCycleWakeReason::ClusterActivity) + || (wake == ScannerCycleWakeReason::Timer && superseded_retry)) } } @@ -1191,16 +1198,6 @@ impl ScannerUsageBootstrapRebuild { enabled && !self.pending } - fn requires_full_scan( - self, - maintenance_features: ScannerMaintenanceFeatures, - observed_generation: Option, - current_generation: u64, - wake: ScannerCycleWakeReason, - ) -> bool { - self.pending || maintenance_features.requires_full_scan(observed_generation, current_generation, wake) - } - fn record_cycle(&mut self, outcome: ScannerCycleOutcome) -> bool { if matches!( outcome, @@ -1597,7 +1594,7 @@ fn get_lock_acquire_timeout() -> Duration { } fn data_usage_persist_timeout() -> Duration { - DataUsageCache::persistence_timeout() + DataUsageCache::publication_timeout() } #[cfg(not(test))] @@ -1777,7 +1774,7 @@ where let now = Instant::now(); cycle_info.started = Utc::now(); - let mut cycle_metrics_guard = ScannerCycleMetricsGuard::new(cycle_info.clone()).await; + let mut cycle_metrics_guard = ScannerCycleMetricsGuard::new(cycle_info.clone(), cycle_budget.remote_scan_progress()).await; // Refresh the storage-owned movement snapshot before reading background // heal state. A missing heal object yields an in-memory default; do not @@ -1907,9 +1904,15 @@ where service_cohort: scheduling.service_cohort, #[cfg(test)] resolved_scope_observer: None, + #[cfg(test)] + post_walk_gate: None, }, ) .await; + let publication_plan = scan_result + .as_ref() + .map(ScannerCyclePublicationPlan::for_result) + .unwrap_or(ScannerCyclePublicationPlan::ProgressOnly); let publication_defer_reason = match &scan_result { Ok(result) if result @@ -1924,7 +1927,7 @@ where // fence remains enforced by the usage store and the cycle is advanced // as partial without acknowledging dirty usage. Ok(result) - if scanner_cycle_uses_observational_publication(result.status, result.has_observational_snapshot()) + if publication_plan.is_observational() && matches!( result.status, ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable) @@ -1933,18 +1936,11 @@ where None } Ok(result) => { - final_data_usage_publication_defer_reason( - storeapi.as_ref(), - result.status, - scanner_cycle_uses_observational_publication(result.status, result.has_observational_snapshot()), - ) - .await + final_data_usage_publication_defer_reason(storeapi.as_ref(), result.status, publication_plan.is_observational()).await } Err(_) => Some(ScannerCycleDeferReason::ActivityBaselineUnavailable), }; - let observational_only = scan_result - .as_ref() - .is_ok_and(|result| scanner_cycle_uses_observational_publication(result.status, result.has_observational_snapshot())); + let observational_only = publication_plan.is_observational(); let publication_epoch = scan_result.as_ref().ok().and_then(ScannerCycleResult::publication_epoch); let remote_publication_lease_targets = if publication_defer_reason.is_none() { scan_result @@ -1959,16 +1955,12 @@ where let remote_lease_defer_reason = if remote_publication_lease_targets.is_empty() { None } else if let Some(notification_system) = storeapi.scanner_notification_system() { - let publication_proof_ctx = cycle_budget.token(); - let lease_result = await_scanner_publication_proof( - &publication_proof_ctx, - cycle_info.current, - "lease_acquire", - || { + let lease_result = publication_plan + .await_leases(ctx, &cycle_budget, cycle_info.current, |observational_lease| { let targets = remote_publication_lease_targets.clone(); let notification_system = Arc::clone(¬ification_system); async move { - if observational_only { + if observational_lease { notification_system .acquire_observational_scanner_publication_leases(targets) .await @@ -1976,10 +1968,8 @@ where notification_system.acquire_scanner_publication_leases(targets).await } } - }, - |err| scanner_publication_lease_error_is_retryable(&err.to_string()), - ) - .await; + }) + .await; match lease_result { ScannerPublicationProofWait::Ready(grants) => { remote_publication_leases = Some((notification_system, grants)); @@ -1999,34 +1989,40 @@ where // it is never inserted into FileInfo metadata. Keep the representation // bounded and require one authenticated token per remote target so a // partial grant can never silently fall back to an unfenced rename. - let remote_lease_fence = remote_publication_leases.as_ref().and_then(|(_, grants)| { - if grants.len() != remote_publication_lease_targets.len() || grants.len() > SCANNER_PUBLICATION_LEASE_FENCE_MAX_ENTRIES { - return None; - } - let mut fence = BTreeMap::new(); - for grant in grants { - if grant.host.is_empty() || grant.host.len() > 1024 { + let remote_lease_fence = remote_publication_leases + .as_ref() + .filter(|_| publication_plan.has_usage_update()) + .and_then(|(_, grants)| { + if grants.len() != remote_publication_lease_targets.len() + || grants.len() > SCANNER_PUBLICATION_LEASE_FENCE_MAX_ENTRIES + { return None; } - if fence.insert(grant.host.clone(), grant.lease.token.to_string()).is_some() { + let mut fence = BTreeMap::new(); + for grant in grants { + if grant.host.is_empty() || grant.host.len() > 1024 { + return None; + } + if fence.insert(grant.host.clone(), grant.lease.token.to_string()).is_some() { + return None; + } + } + if remote_publication_lease_targets + .iter() + .any(|(host, _, _)| !fence.contains_key(host)) + { return None; } - } - if remote_publication_lease_targets - .iter() - .any(|(host, _, _)| !fence.contains_key(host)) - { - return None; - } - serde_json::to_string(&fence) - .ok() - .filter(|encoded| encoded.len() <= SCANNER_PUBLICATION_LEASE_FENCE_MAX_BYTES) - }); - let remote_lease_fence_defer_reason = (remote_publication_lease_fence_targets_are_required( - remote_publication_lease_targets.len(), - remote_publication_leases.is_some(), - remote_lease_fence.is_some(), - )) + serde_json::to_string(&fence) + .ok() + .filter(|encoded| encoded.len() <= SCANNER_PUBLICATION_LEASE_FENCE_MAX_BYTES) + }); + let remote_lease_fence_defer_reason = (publication_plan.has_usage_update() + && remote_publication_lease_fence_targets_are_required( + remote_publication_lease_targets.len(), + remote_publication_leases.is_some(), + remote_lease_fence.is_some(), + )) .then_some(ScannerCycleDeferReason::ActivityBaselineUnavailable); let publication_defer_reason = publication_defer_reason .or(remote_lease_defer_reason) @@ -2039,7 +2035,10 @@ where && let Ok(result) = &scan_result && result.status == ScannerCycleStatus::Complete { - scanner_post_lease_activity_defer_reason(result.activity_digest(), probe_scanner_activity(storeapi.as_ref(), true).await) + scanner_post_lease_activity_defer_reason( + result.activity_baseline(), + probe_scanner_activity(storeapi.as_ref(), true).await, + ) } else { None }; @@ -2049,6 +2048,7 @@ where let budget_elapsed = cycle_budget.budget_elapsed() && !ctx.is_cancelled(); let remote_lease_probe = remote_publication_leases .as_ref() + .filter(|_| publication_plan.has_usage_update()) .map(|(notification_system, grants)| (Arc::clone(notification_system), grants.clone())); let remote_lease_release_safe = Arc::new(AtomicBool::new(true)); let mut usage_publication_result = match publication_defer_reason { @@ -2056,6 +2056,10 @@ where drop(receiver); DataUsagePublicationResult::from(DataUsagePersistOutcome::Deferred(reason)) } + None if !publication_plan.has_usage_update() => { + drop(receiver); + DataUsagePublicationResult::from(DataUsagePersistOutcome::NoUpdate) + } None => { // ScannerIO emits its complete or observational update only after // all set workers finish. Persist after the final activity fence; @@ -2064,7 +2068,6 @@ where let ctx_clone = ctx.clone(); let route_probe_store = storeapi.clone(); let route_probe_observational_only = observational_only; - let remote_lease_fence = remote_lease_fence.clone(); let remote_lease_release_safe_for_task = Arc::clone(&remote_lease_release_safe); let remote_lease_tokens = remote_publication_leases .as_ref() @@ -2366,15 +2369,19 @@ where } usage_publication_result.restrict_outcome(usage_persist_outcome); - let partial_cause = if scan_cycle_result.has_observational_snapshot() + let partial_cause = if scan_cycle_result.has_partial_bucket_progress() && !scan_cycle_result.has_usage_update() { + "saved_bucket_progress" + } else if scan_cycle_result.has_usage_update() && matches!( scan_cycle_result.status, ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable) - ) { + ) + { "activity_unverified_observation" } else { "incomplete_coverage" }; + let pending_dirty_usage_buckets = scan_cycle_result.pending_dirty_usage_buckets(); let (completion_outcome, scanner_pending_maintenance_work, remote_dirty_usage_acknowledgements) = finalize_scanner_cycle_result(scan_cycle_result, usage_publication_result); let remote_dirty_usage_pending = if remote_dirty_usage_acknowledgements.is_empty() { @@ -2586,6 +2593,7 @@ where state = "completed", duration = ?now.elapsed(), cycles_total = cycle_info.cycle_completed.len(), + pending_dirty_buckets = pending_dirty_usage_buckets, "Scanner cycle completed" ); @@ -2598,9 +2606,13 @@ struct ScannerCycleMetricsGuard { } impl ScannerCycleMetricsGuard { - async fn new(cycle: CurrentCycle) -> Self { + async fn new(cycle: CurrentCycle, remote_progress: Arc) -> Self { Self { - start: Some(global_metrics().start_scan_cycle_work_with_cycle(cycle).await), + start: Some( + global_metrics() + .start_scan_cycle_work_with_cycle_and_remote_progress(cycle, remote_progress) + .await, + ), } } @@ -2739,6 +2751,7 @@ where Err(e) => { match classify_scanner_leader_lock_failure(&e) { ScannerLeaderLockFailure::Contended => { + clear_scanner_transient_cycle_recovery_status(); record_scanner_cycle_schedule_role("follower"); record_scanner_leader_lock_state("contended"); global_metrics().record_scanner_leader_liveness("contended", false, "").await; @@ -3180,7 +3193,8 @@ where )); let mut wait_plan = scanner_cycle_wait_plan(&runtime_config, clean_idle_backoff, backoff_enabled, randomized_cycle_delay_for); - let superseded_retry_interval = scanner_superseded_retry_interval(superseded_backoff, &runtime_config); + let superseded_retry_interval = + scanner_superseded_retry_interval(superseded_backoff, &runtime_config, maintenance_features); let deferred_retry_interval = deferred_backoff.retry_interval(runtime_config.cycle_interval); let mut convergence_retry_interval = superseded_retry_interval.or(deferred_retry_interval); if let Some(retry_interval) = convergence_retry_interval { @@ -3358,11 +3372,13 @@ where leader_epoch, cycle_budget.clone(), ScannerCycleScheduling { - requires_full_scan: usage_bootstrap_rebuild.requires_full_scan( - maintenance_features, + // Cache reuse is admitted by the persisted coverage proof in ScannerIO. + // Bootstrap remains pending until authoritative publication succeeds. + requires_full_scan: maintenance_features.requires_full_scan( maintenance_generation_seen, scanner_maintenance_generation(), wake_reason, + superseded_retry_interval.is_some(), ), service_cohort: Some(service_cohort.clone()), }, @@ -3594,22 +3610,109 @@ where } } -fn scanner_cycle_uses_observational_publication(status: ScannerCycleStatus, has_observational_candidate: bool) -> bool { - has_observational_candidate - && matches!( - status, - ScannerCycleStatus::Superseded | ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable) - ) +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum ScannerCyclePublicationPlan { + ProgressOnly, + Observational, + Authoritative, } +impl ScannerCyclePublicationPlan { + fn for_result(result: &ScannerCycleResult) -> Self { + match (result.has_usage_update(), result.status) { + (false, _) => Self::ProgressOnly, + ( + true, + ScannerCycleStatus::Superseded + | ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable) + | ScannerCycleStatus::Incomplete, + ) => Self::Observational, + (true, _) => Self::Authoritative, + } + } + + fn has_usage_update(self) -> bool { + self != Self::ProgressOnly + } + + fn is_observational(self) -> bool { + self == Self::Observational + } + + fn uses_observational_lease(self) -> bool { + self != Self::Authoritative + } + + async fn await_leases( + self, + parent: &CancellationToken, + budget: &ScannerCycleBudget, + cycle: u64, + mut acquire: F, + ) -> ScannerPublicationProofWait + where + E: std::fmt::Display, + F: FnMut(bool) -> Fut, + Fut: Future>, + { + // Exhausting the walk budget must not cancel the movement proof needed + // to save partial progress. No usage update means namespace stability + // is unnecessary, but peer session/generation and movement checks stay. + let proof_ctx = if self == Self::ProgressOnly { + parent.clone() + } else { + budget.token() + }; + debug!( + target: "rustfs::scanner", + event = EVENT_SCANNER_CYCLE_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + state = "publication_proof_started", + cycle, + stage = "lease_acquire", + publication_kind = match self { + Self::ProgressOnly => "progress_only", + Self::Observational => "observational", + Self::Authoritative => "authoritative", + }, + "Scanner is acquiring peer publication proof" + ); + let proof = await_scanner_publication_proof( + &proof_ctx, + cycle, + "lease_acquire", + || acquire(self.uses_observational_lease()), + |err| scanner_publication_lease_error_is_retryable(&err.to_string()), + ); + if self == Self::ProgressOnly { + // Object/directory budgets can expire with no runtime deadline. + // Bound the control-plane drain independently of the walk budget. + tokio::time::timeout(SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, proof) + .await + .unwrap_or(ScannerPublicationProofWait::Cancelled) + } else { + proof.await + } + } +} + +/// Re-prove the guardrail after every peer granted a publication lease. The +/// comparison uses the guardrail fields (`scanner_activity_guardrail_veto`), +/// not the full digest: local generation movement in this window is the next +/// cycle's work and its buckets stay in the dirty map. A peer whose dirty +/// usage moved keeps deferring in this phase, as it does during the walk. fn scanner_post_lease_activity_defer_reason( - expected_digest: Option<[u8; 32]>, + baseline: Option<&ScannerActivitySnapshot>, activity: Result, ) -> Option { match activity { Ok(snapshot) if scanner_activity_allows_usage_publication(&snapshot) - && expected_digest == Some(scanner_activity_snapshot_digest(&snapshot)) => + && baseline.is_some_and(|baseline| { + scanner_activity_guardrail_veto(baseline, &snapshot).is_none() + && scanner_peer_dirty_usage_moved(baseline, &snapshot).is_none() + }) => { None } @@ -3617,8 +3720,13 @@ fn scanner_post_lease_activity_defer_reason( let blocker = match &observed { Err(_) => "probe_failed", Ok(snapshot) if !scanner_activity_allows_usage_publication(snapshot) => "publication_blocked", - Ok(_) if expected_digest.is_none() => "baseline_missing", - Ok(_) => "activity_changed", + Ok(_) if baseline.is_none() => "baseline_missing", + Ok(snapshot) => baseline + .and_then(|baseline| { + scanner_activity_guardrail_veto(baseline, snapshot) + .or_else(|| scanner_peer_dirty_usage_moved(baseline, snapshot)) + }) + .map_or("activity_changed", |veto| veto.field), }; debug!( target: "rustfs::scanner", @@ -3798,7 +3906,17 @@ fn scanner_cycle_completion_outcome_for_result( ) -> ScannerCycleOutcome { let has_dirty_usage = scan_cycle_result.has_dirty_usage_to_acknowledge(); let has_failed_dirty_usage = scan_cycle_result.has_failed_dirty_usage(); - if scan_cycle_result.has_observational_snapshot() + if scan_cycle_result.status == ScannerCycleStatus::Incomplete + && scan_cycle_result.has_partial_bucket_progress() + && !scan_cycle_result.has_usage_update() + && !has_failed_dirty_usage + && usage_persist_outcome == DataUsagePersistOutcome::NoUpdate + { + // No root was submitted: retain dirty usage and advance only the + // fenced partial-cycle state. A fresh sweep still has to certify usage. + return ScannerCycleOutcome::Partial; + } + if scan_cycle_result.has_usage_update() && matches!( scan_cycle_result.status, ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable) @@ -3897,10 +4015,11 @@ use usage_store::*; pub(crate) use activity::scanner_node_activity_for_tests; pub use activity::scanner_topology_digest; pub(crate) use activity::{ - ScannerActivitySnapshot, ScannerDirtyUsageAcknowledgement, ScannerDirtyUsageAcknowledgementKind, probe_scanner_activity, - scanner_activity_allows_usage_publication, scanner_activity_dirty_usage_state_for_host, - scanner_activity_publication_lease_targets, scanner_activity_snapshot_digest, scanner_activity_structural_digest, - scanner_dirty_usage_acknowledgements, + LOCAL_SCANNER_ACTIVITY_NODE, ScannerActivityGuardrailVeto, ScannerActivitySnapshot, ScannerDirtyUsageAcknowledgement, + ScannerDirtyUsageAcknowledgementKind, probe_scanner_activity, scanner_activity_allows_usage_publication, + scanner_activity_dirty_usage_state_for_host, scanner_activity_guardrail_veto, scanner_activity_publication_lease_targets, + scanner_activity_snapshot_digest, scanner_activity_structural_digest, scanner_dirty_usage_acknowledgements, + scanner_peer_dirty_usage_moved, }; pub(crate) use activity::{ScannerCycleOutcome, scanner_cycle_outcome_with_pending_maintenance}; pub use backlog::{ diff --git a/crates/scanner/src/scanner/activity.rs b/crates/scanner/src/scanner/activity.rs index 2b63d115a..e018dabc9 100644 --- a/crates/scanner/src/scanner/activity.rs +++ b/crates/scanner/src/scanner/activity.rs @@ -152,17 +152,19 @@ impl ScannerRetryBackoff { pub(super) fn scanner_superseded_retry_interval( backoff: ScannerRetryBackoff, runtime_config: &ScannerRuntimeConfig, + maintenance_features: ScannerMaintenanceFeatures, ) -> Option { - let retry_interval = backoff.retry_interval(runtime_config.cycle_interval)?; - if runtime_config.cycle_interval_source == ScannerRuntimeConfigSource::Default { - return Some(retry_interval); + let mut retry_interval = backoff.retry_interval(runtime_config.cycle_interval)?; + // Preserve an operator-selected duty-cycle floor during convergence. + if runtime_config.cycle_interval_source != ScannerRuntimeConfigSource::Default { + retry_interval = retry_interval.max(runtime_config.cycle_interval); } - - // An explicit cycle is an operator-selected duty-cycle floor. A - // superseded snapshot keeps its dirty work pending, but retrying that work - // sooner than the configured cadence would turn continuous writes into a - // repeated full-walk loop despite the override. - Some(retry_interval.max(runtime_config.cycle_interval)) + // A complete but non-authoritative walk must not postpone lifecycle or + // replication evaluation beyond its ordinary maintenance cadence. + if maintenance_features.needs_regular_cycle() { + retry_interval = retry_interval.min(runtime_config.cycle_interval.max(Duration::from_secs(1))); + } + Some(retry_interval) } const SCANNER_PUBLICATION_PROOF_RETRY_MAX_INTERVAL: Duration = Duration::from_secs(30); @@ -260,8 +262,9 @@ where stage, attempt = consecutive_failures, retry_delay = ?retry_delay, + pending_duration = ?started_at.elapsed(), error = %err, - "Scanner retained a completed scan while publication proof is unavailable" + "Scanner is waiting for publication proof" ); } else { debug!( @@ -466,7 +469,7 @@ pub(super) struct ScannerMovementWaitContext<'a, G, F> { pub(super) recovery_wake: Option<&'a Notify>, } -pub(super) const LOCAL_SCANNER_ACTIVITY_NODE: &str = ""; +pub(crate) const LOCAL_SCANNER_ACTIVITY_NODE: &str = ""; #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) struct ScannerNodeActivity { @@ -1059,6 +1062,109 @@ pub(crate) fn scanner_activity_structural_digest(snapshot: &ScannerActivitySnaps hasher.finalize().into() } +/// An attributed veto of the cluster activity guardrail: the named field on +/// the named host moved between the pre-walk and post-walk probes. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct ScannerActivityGuardrailVeto { + pub(crate) host: String, + pub(crate) field: &'static str, +} + +impl ScannerActivityGuardrailVeto { + pub(crate) fn new(host: &str, field: &'static str) -> Self { + Self { + host: host.to_string(), + field, + } + } + + /// Operator-facing label, `@`. + pub(crate) fn reason(&self) -> String { + format!("{}@{}", self.field, self.host) + } +} + +/// `ScannerNodeActivity` fields whose movement means the heap the probe +/// describes is not the heap that was walked. Each is a guardrail veto on its +/// own; `peer_set` covers a host that appeared or disappeared. +#[cfg(test)] +pub(crate) const SCANNER_ACTIVITY_GUARDRAIL_VETO_FIELDS: [&str; 8] = [ + "peer_set", + "instance_id", + "protocol_version", + "topology_digest", + "maintenance_generation", + "data_movement_active", + "movement_generation", + "publication_blocked", +]; + +/// `ScannerNodeActivity` fields that only shade buckets: they are consumed +/// per bucket by dirty-usage certification and never veto on their own. +#[cfg(test)] +pub(crate) const SCANNER_ACTIVITY_SHADE_FIELDS: [&str; 3] = + ["namespace_generation", "dirty_usage_generation", "dirty_usage_pending"]; + +/// Compare the guardrail fields of two activity probes and name the first +/// field that moved. `maintenance_generation` is a veto because no cycle today +/// distinguishes whether it published maintenance results; it is also part of +/// `scanner_activity_structural_digest`. +pub(crate) fn scanner_activity_guardrail_veto( + before: &ScannerActivitySnapshot, + after: &ScannerActivitySnapshot, +) -> Option { + for (host, before_activity) in before { + let Some(after_activity) = after.get(host) else { + return Some(ScannerActivityGuardrailVeto::new(host, "peer_set")); + }; + let moved = if before_activity.instance_id != after_activity.instance_id { + "instance_id" + } else if before_activity.protocol_version != after_activity.protocol_version { + "protocol_version" + } else if before_activity.topology_digest != after_activity.topology_digest { + "topology_digest" + } else if before_activity.maintenance_generation != after_activity.maintenance_generation { + "maintenance_generation" + } else if before_activity.data_movement_active != after_activity.data_movement_active { + "data_movement_active" + } else if before_activity.movement_generation != after_activity.movement_generation { + "movement_generation" + } else if before_activity.publication_blocked != after_activity.publication_blocked { + "publication_blocked" + } else { + continue; + }; + return Some(ScannerActivityGuardrailVeto::new(host, moved)); + } + after + .keys() + .find(|host| !before.contains_key(*host)) + .map(|host| ScannerActivityGuardrailVeto::new(host, "peer_set")) +} + +/// Name the first remote peer whose dirty-usage state moved between the two +/// probes. The local node's dirty state is certified per bucket, but a peer's +/// moved buckets cannot be named without a post-walk scoped snapshot, so this +/// phase keeps the whole-cycle fallback for peers; phase 5 lifts it. +pub(crate) fn scanner_peer_dirty_usage_moved( + before: &ScannerActivitySnapshot, + after: &ScannerActivitySnapshot, +) -> Option { + before + .iter() + .filter(|(host, _)| host.as_str() != LOCAL_SCANNER_ACTIVITY_NODE) + .find_map(|(host, before_activity)| { + let after_activity = after.get(host)?; + if before_activity.dirty_usage_generation != after_activity.dirty_usage_generation { + Some(ScannerActivityGuardrailVeto::new(host, "dirty_usage_generation")) + } else if before_activity.dirty_usage_pending != after_activity.dirty_usage_pending { + Some(ScannerActivityGuardrailVeto::new(host, "dirty_usage_pending")) + } else { + None + } + }) +} + pub(crate) fn scanner_activity_allows_usage_publication(snapshot: &ScannerActivitySnapshot) -> bool { !snapshot.is_empty() && snapshot.values().all(|activity| { diff --git a/crates/scanner/src/scanner/backlog.rs b/crates/scanner/src/scanner/backlog.rs index ba5df0feb..8bc63669a 100644 --- a/crates/scanner/src/scanner/backlog.rs +++ b/crates/scanner/src/scanner/backlog.rs @@ -2192,7 +2192,7 @@ mod tests { }) .expect("native retirement test thread") .join() - .expect("native retirement test completed"); + .unwrap_or_else(|panic| std::panic::resume_unwind(panic)); } async fn native_retirement_store() -> (tempfile::TempDir, Arc) { @@ -2631,13 +2631,13 @@ mod tests { ) .await; eprintln!("native expansion fixture old_pools={old_pool_count} stage=seed_old_authority"); - let fault = unstable_source - .then(|| NativeScannerPauseBacklogWriteFault::fail_before_write(Arc::clone(&old.pools[0].disk_set[0]), "publish", 2)); let now = unix_now(); - let mut controller = ScannerPauseBacklogController::claim(Arc::clone(&old), now) - .await - .expect("the native controller commits the actual old cohort"); - if unstable_source { + let original = if unstable_source { + let fault = + NativeScannerPauseBacklogWriteFault::fail_before_write(Arc::clone(&old.pools[0].disk_set[0]), "publish", 2); + let controller = ScannerPauseBacklogController::claim(Arc::clone(&old), now) + .await + .expect("the native controller commits the actual old cohort"); assert!(controller.loaded.requires_reload); let (bytes, _) = native_replica_bytes(&old.pools[0].disk_set[0]).await; let ScannerPauseBacklogReplicaState::Valid(record) = decode_scanner_pause_backlog_ledger(&bytes) else { @@ -2645,23 +2645,14 @@ mod tests { }; assert!(record.stable.is_none()); assert_eq!(record.committed.unwrap().ledger, controller.loaded.ledger); + let original = controller.loaded.ledger.clone(); + assert!(controller.loaded.durable); + drop(controller); + drop(fault); + original } else { - controller.observe(observation(now + 1, true, 4)).await; - controller.observe(observation(now + 2, false, 4)).await; - let decision = controller.begin_attempt(now + 2).await; - assert!( - matches!(decision, ScannerPauseBacklogAttemptDecision::Tracked(_)), - "expected tracked native expansion setup attempt, got {decision:?}; ledger={:?}; persistence_disabled={}; runtime_error={:?}", - controller.loaded.ledger, - controller.persistence_disabled, - runtime_error() - ); - assert!(controller.loaded.ledger.has_unfinished_attempt()); - } - let original = controller.loaded.ledger.clone(); - assert!(controller.loaded.durable); - drop(controller); - drop(fault); + seed_native_expansion_ledger(&old, now).await + }; old.pool_meta_write_status() .await .expect("healthy pool metadata keeps the background recovery loop read-only"); @@ -2686,6 +2677,81 @@ mod tests { (root, expanded, original) } + async fn seed_native_expansion_ledger(store: &Arc, now: u64) -> ScannerPauseBacklogLedger { + let loaded = load_scanner_pause_backlog(Arc::clone(store)) + .await + .expect("empty native expansion cohort"); + assert!( + loaded + .replicas + .iter() + .all(|replica| matches!(replica.state, ScannerPauseBacklogReplicaState::Missing)) + ); + let mut ledger = claim_scanner_pause_backlog_writer(&loaded.ledger, now).expect("fixture writer generation"); + // The fixture needs the final unfinished attempt, not four separate + // durable controller transitions before the fault under test. Retain + // their generations and timestamps, then publish the final state once + // through the real native commit and stabilization protocol. + ledger.apply_observation(observation(now + 1, true, 4)); + prepare_scanner_pause_backlog_persist(&mut ledger, now + 1).expect("fixture pause generation"); + ledger.apply_observation(observation(now + 2, false, 4)); + prepare_scanner_pause_backlog_persist(&mut ledger, now + 2).expect("fixture resume generation"); + assert!(matches!(ledger.begin_attempt(now + 2), ScannerPauseBacklogAttemptDecision::Tracked(_))); + prepare_scanner_pause_backlog_persist(&mut ledger, now + 2).expect("fixture unfinished attempt generation"); + let published = persist_scanner_pause_backlog(Arc::clone(store), &loaded, ledger.clone()) + .await + .expect("publish the complete native expansion fixture"); + assert!(published.durable && published.stable_matches_ledger && !published.requires_reload); + assert!(ledger.has_unfinished_attempt()); + assert_current_native_writer_ledger(store, &ledger).await; + set_runtime_error(None); + ledger + } + + #[test] + #[serial_test::serial] + fn native_expansion_fixture_retains_controller_state_and_complete_disk_proof() { + run_native_retirement_test(async || { + use crate::storage_api::owner::NativeScannerPauseBacklogWriteFault; + + let (_reference_root, reference) = native_pause_backlog_store(1, 2).await; + let now = unix_now(); + let mut controller = ScannerPauseBacklogController::claim(Arc::clone(&reference), now) + .await + .expect("reference native writer claim"); + controller.observe(observation(now + 1, true, 4)).await; + controller.observe(observation(now + 2, false, 4)).await; + assert!(matches!( + controller.begin_attempt(now + 2).await, + ScannerPauseBacklogAttemptDecision::Tracked(_) + )); + let expected = controller.loaded.ledger.clone(); + assert_current_native_writer_ledger(&reference, &expected).await; + + let (_fixture_root, fixture) = native_pause_backlog_store(1, 2).await; + // A third preparation write would fail: the complete native commit + // and stabilization must suffice, without replaying the unrelated + // pause/resume transitions to disk in every recovery matrix cell. + let fault = + NativeScannerPauseBacklogWriteFault::fail_before_write(Arc::clone(&fixture.pools[0].disk_set[0]), "publish", 3); + let actual = seed_native_expansion_ledger(&fixture, now).await; + assert_eq!(actual, expected, "all generations, timestamps and unfinished work must be retained"); + assert_current_native_writer_ledger(&fixture, &expected).await; + drop(fault); + drop(controller); + shutdown_native_retirement_store(reference).await; + shutdown_native_retirement_store(fixture).await; + }); + } + + #[test] + #[should_panic(expected = "original native scenario failure")] + fn native_retirement_runner_preserves_scenario_failure_payload() { + run_native_retirement_test_with_budget(Duration::from_secs(1), async || { + panic!("original native scenario failure"); + }); + } + async fn native_expanded_retirement_store( old_pool_count: usize, unstable_source: bool, diff --git a/crates/scanner/src/scanner/cycle_state.rs b/crates/scanner/src/scanner/cycle_state.rs index 720302608..0ad5d9f2f 100644 --- a/crates/scanner/src/scanner/cycle_state.rs +++ b/crates/scanner/src/scanner/cycle_state.rs @@ -353,6 +353,17 @@ fn publish_scanner_cleanup_failure(reason: String, expected: u64) { let _ = publish_scanner_cleanup_status(status, Some(expected)); } +pub(super) fn clear_scanner_transient_cycle_recovery_status() { + // Snapshot the version before reading status so any concurrent publish rejects the CAS below. + let version = scanner_cleanup_status_version(); + let status = scanner_cycle_recovery_status(); + // A follower can discard its failed leadership observation. Durable recovery + // markers are revalidated before any later leadership claim. + if status.classification.is_none() && matches!(status.state.as_str(), "transient" | "paused") { + let _ = publish_scanner_cleanup_status(recovery_status("healthy", None, false), Some(version)); + } +} + pub(super) fn record_scanner_usage_floor_failure(reason: String) { let previous = scanner_cycle_recovery_status(); let same_failure = previous.classification.as_deref() == Some(USAGE_FLOOR_LOAD_FAILED); @@ -1461,6 +1472,23 @@ async fn mark_cycle_recovery_cleanup_pending( pub(crate) async fn load_scanner_cycle_state_for_startup( storeapi: Arc, +) -> ScannerCycleStateStartup { + let load = load_scanner_cycle_state_inner(storeapi); + #[cfg(feature = "diagnostic-read-version-caller")] + let load = crate::storage_api::ecstore_diagnostic::with_read_version_entry( + crate::storage_api::ecstore_diagnostic::ReadVersionEntrySource::ScannerCycleStateRead, + async { + crate::storage_api::ecstore_diagnostic::observe_read_version_context_boundary( + crate::storage_api::ecstore_diagnostic::ReadVersionContextBoundary::ScannerCycleStateLoad, + ); + load.await + }, + ); + load.await +} + +async fn load_scanner_cycle_state_inner( + storeapi: Arc, ) -> ScannerCycleStateStartup { let marker = match read_cycle_recovery_marker_bytes(storeapi.clone()).await { Ok((None, _)) => None, diff --git a/crates/scanner/src/scanner/tests.rs b/crates/scanner/src/scanner/tests.rs index 6cc15580a..41adad216 100644 --- a/crates/scanner/src/scanner/tests.rs +++ b/crates/scanner/src/scanner/tests.rs @@ -37,6 +37,7 @@ use tokio::time::{Duration, advance}; const TEST_DEFAULT_SCANNER_CYCLE_SECS: u64 = 24 * 60 * 60; +mod cycle_budget_publication; pub(super) mod cycle_persist_failure; mod quota_reset_preservation; @@ -138,6 +139,11 @@ pub(super) async fn setup_scanner_cycle_store_at_path_with_layout_and_disk_prein let endpoint_pools = EndpointServerPools::from(pools); let instance_ctx = Arc::new(InstanceContext::new()); instance_ctx.set_endpoints(endpoint_pools.clone()); + if pool_count == 1 && sets_per_pool == 1 && drives_per_set == 1 { + instance_ctx + .update_erasure_type(crate::storage_api::EcstoreSetupType::ErasureSD) + .await; + } if preinitialize_disks { init_local_disks_with_instance_ctx(&instance_ctx, endpoint_pools.clone()) .await @@ -714,7 +720,7 @@ async fn checkpoint_runtime_handoff(store: Arc, expected_cycl assert_eq!((cycle.next, saved_epoch), (expected_cycle, epoch)); cycle.current = cycle.next; let ctx = CancellationToken::new(); - let mut metrics = ScannerCycleMetricsGuard::new(cycle.clone()).await; + let mut metrics = ScannerCycleMetricsGuard::new(cycle.clone(), Arc::new(RemoteScanProgress::default())).await; tokio::time::pause(); let budget = ScannerCycleBudget::new( &ctx, @@ -883,6 +889,12 @@ impl Drop for ScannerDefaultCycleGuard { #[derive(Debug, Default)] struct MemoryConfigStore { + #[cfg(feature = "diagnostic-read-version-caller")] + entry_sources: Mutex>>, + #[cfg(feature = "diagnostic-read-version-caller")] + write_entry_sources: Mutex>>, + #[cfg(feature = "diagnostic-read-version-caller")] + pending_diagnostic_write: AtomicBool, objects: Mutex>>, revisions: Mutex>, insert_after_gets: Mutex>>, @@ -937,6 +949,11 @@ impl crate::storage_api::scanner_io::ObjectIO for MemoryConfigStore { _h: http::HeaderMap, _opts: &ObjectOptions, ) -> EcstoreResult { + #[cfg(feature = "diagnostic-read-version-caller")] + self.entry_sources + .lock() + .await + .push(crate::storage_api::ecstore_diagnostic::current_read_version_entry_source()); let key = memory_config_key(bucket, object); if let Some(error) = self.read_errors.lock().await.get(&key).cloned() { return Err(error); @@ -983,6 +1000,16 @@ impl crate::storage_api::scanner_io::ObjectIO for MemoryConfigStore { data: &mut PutObjReader, opts: &ObjectOptions, ) -> EcstoreResult { + #[cfg(feature = "diagnostic-read-version-caller")] + { + self.write_entry_sources + .lock() + .await + .push(crate::storage_api::ecstore_diagnostic::current_read_version_entry_source()); + if self.pending_diagnostic_write.load(Ordering::Relaxed) { + std::future::pending::<()>().await; + } + } let mut buf = Vec::new(); data.stream.read_to_end(&mut buf).await?; let key = memory_config_key(bucket, object); @@ -1289,7 +1316,8 @@ async fn test_mark_scan_cycle_idle_clears_published_cycle_state() { }; global_metrics().set_current_scan_mode(HealScanMode::Deep); - let mut cycle_metrics_guard = ScannerCycleMetricsGuard::new(cycle_info.clone()).await; + let mut cycle_metrics_guard = + ScannerCycleMetricsGuard::new(cycle_info.clone(), Arc::new(RemoteScanProgress::default())).await; mark_scan_cycle_idle(&mut cycle_info, &mut cycle_metrics_guard).await; @@ -1307,6 +1335,33 @@ async fn test_mark_scan_cycle_idle_clears_published_cycle_state() { global_metrics().set_cycle(None).await; } +#[tokio::test] +#[serial] +async fn scanner_cycle_metrics_guard_installs_budget_remote_progress() { + let parent = CancellationToken::new(); + let old_budget = ScannerCycleBudget::new(&parent, ScannerCycleBudgetConfig::default()); + let mut guard = ScannerCycleMetricsGuard::new(CurrentCycle::default(), old_budget.remote_scan_progress()).await; + old_budget.record_remote_progress(3, 2); + guard.finish(CurrentCycle::default()).await; + let previous = global_metrics().report().await; + assert_eq!(previous.last_cycle_objects_scanned, 3); + assert_eq!(previous.last_cycle_directories_scanned, 2); + + let budget = ScannerCycleBudget::new(&parent, ScannerCycleBudgetConfig::default()); + let mut guard = ScannerCycleMetricsGuard::new(CurrentCycle::default(), budget.remote_scan_progress()).await; + budget.record_remote_progress(2, 1); + old_budget.record_remote_progress(9, 7); + let active = global_metrics().report().await; + assert_eq!(active.current_cycle_objects_scanned, 2); + assert_eq!(active.current_cycle_directories_scanned, 1); + guard.finish(CurrentCycle::default()).await; + old_budget.record_remote_progress(11, 5); + let finished = global_metrics().report().await; + assert_eq!(finished.last_cycle_objects_scanned, 2); + assert_eq!(finished.last_cycle_directories_scanned, 1); + global_metrics().set_cycle(None).await; +} + #[tokio::test] #[serial] async fn scanner_cycle_metrics_guard_covers_published_first_cycle_lifetime() { @@ -1317,7 +1372,7 @@ async fn scanner_cycle_metrics_guard_covers_published_first_cycle_lifetime() { started: cycle_started, ..Default::default() }; - let mut guard = ScannerCycleMetricsGuard::new(cycle_info.clone()).await; + let mut guard = ScannerCycleMetricsGuard::new(cycle_info.clone(), Arc::new(RemoteScanProgress::default())).await; let setup_report = global_metrics().report().await; assert!(setup_report.current_cycle_active); assert_eq!(setup_report.current_cycle, 0); @@ -1343,7 +1398,7 @@ async fn scanner_cycle_metrics_guard_keeps_active_cycle_published_during_finaliz started: Utc::now(), ..Default::default() }; - let mut guard = ScannerCycleMetricsGuard::new(cycle_info.clone()).await; + let mut guard = ScannerCycleMetricsGuard::new(cycle_info.clone(), Arc::new(RemoteScanProgress::default())).await; cycle_info.current = 0; tokio::task::yield_now().await; @@ -1362,12 +1417,15 @@ async fn scanner_cycle_metrics_guard_keeps_active_cycle_published_during_finaliz #[tokio::test] #[serial] async fn scanner_cycle_metrics_guard_drop_clears_activity() { - let guard = ScannerCycleMetricsGuard::new(CurrentCycle { - current: 12, - next: 13, - started: Utc::now(), - ..Default::default() - }) + let guard = ScannerCycleMetricsGuard::new( + CurrentCycle { + current: 12, + next: 13, + started: Utc::now(), + ..Default::default() + }, + Arc::new(RemoteScanProgress::default()), + ) .await; assert!(global_metrics().report().await.current_cycle_active); @@ -1431,8 +1489,19 @@ async fn run_data_scanner_cycle_publishes_activity_for_owner_lifetime() { #[tokio::test] #[serial] async fn coordinator_persists_only_an_observation_during_pending_put() { + for drives_per_set in [1, 4] { + assert_bootstrap_observation_retry(drives_per_set).await; + } +} + +async fn assert_bootstrap_observation_retry(drives_per_set: usize) { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); - let (_temp_dir, store) = setup_scanner_cycle_store().await; + let temp_dir = tempfile::tempdir().expect("bootstrap directory should be created"); + let store = setup_scanner_cycle_store_at_path_with_layout(temp_dir.path(), false, 1, 1, drives_per_set).await; + initialize_usage_baseline_bootstrap(store.clone()) + .await + .expect("cold startup should persist its bootstrap identity"); + let mut bootstrap = ScannerUsageBootstrapRebuild::from_startup(PersistedUsageFloorStartup::BootstrapPending); let mut pause_backlog = ScannerPauseBacklogController::claim(store.clone(), scanner_pause_backlog_now()) .await .expect("scanner pause backlog should be available"); @@ -1457,6 +1526,25 @@ async fn coordinator_persists_only_an_observation_during_pending_put() { .await .expect("fixture object should finish its rename fanout"); } + let cold_bucket = "scanner-coordinator-cold"; + store + .make_bucket(cold_bucket, &crate::storage_api::scan::MakeBucketOptions::default()) + .await + .expect("cold bucket should be created"); + let mut cold_reader = PutObjReader::from_vec(b"cold".to_vec()); + store.pools[0].disk_set[0] + .put_object( + cold_bucket, + "object", + &mut cold_reader, + &ObjectOptions { + write_completion: WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("cold object should persist before scanning"); + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); crate::scanner_io::record_dirty_usage_bucket(&bucket); let dirty_before = crate::scanner_io::dirty_usage_buckets_for_tests(); let baseline = read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) @@ -1499,7 +1587,7 @@ async fn coordinator_persists_only_an_observation_during_pending_put() { ) .await .expect("the coordinator must finish its namespace walk while a PUT is pending"); - assert_eq!(budget.progress().0, 1, "the coordinator must reach actual object traversal"); + assert_eq!(budget.progress().0, 2, "the coordinator must reach actual object traversal"); assert!( matches!( outcome, @@ -1529,6 +1617,16 @@ async fn coordinator_persists_only_an_observation_during_pending_put() { .expect("the pending cycle should persist its non-authoritative observation"); let observed: DataUsageInfo = serde_json::from_slice(&observed).expect("observed usage should decode"); assert_eq!(observed.usage_snapshot_converged, Some(false)); + assert!(observed.is_complete_bucket_usage_snapshot()); + assert_eq!(observed.objects_total_count, 2); + assert!(!bootstrap.record_cycle(outcome)); + assert!(bootstrap.pending(), "an observation cannot discharge authority bootstrap"); + crate::storage_api::ecstore_invalidate_data_usage_snapshot_cache().await; + let displayed = crate::storage_api::scanner_io::load_admin_data_usage_from_backend_cached(store.clone()) + .await + .expect("admin should load the durable complete observation"); + assert_eq!(displayed.objects_total_count, 2); + assert_eq!(displayed.usage_snapshot_converged, Some(false)); let committed_body = b"committed-after-walk"; let mut reader = PutObjReader::from_vec(committed_body.to_vec()); @@ -1561,7 +1659,12 @@ async fn coordinator_persists_only_an_observation_during_pending_put() { 1, Arc::clone(&retry_budget), ScannerCycleScheduling { - requires_full_scan: true, + requires_full_scan: ScannerMaintenanceFeatures::default().requires_full_scan( + Some(scanner_maintenance_generation()), + scanner_maintenance_generation(), + ScannerCycleWakeReason::Timer, + true, + ), service_cohort: None, }, ), @@ -1570,13 +1673,15 @@ async fn coordinator_persists_only_an_observation_during_pending_put() { .expect("the same cycle must converge after the pending PUT drains"); assert_eq!( retry_budget.progress().0, - 1, - "the same-cycle retry must not reuse the pre-tail bucket cache" + if drives_per_set == 1 { 1 } else { 2 }, + "Normal retries reuse cold inventory; Deep retries still visit every bucket" ); assert!(matches!( outcome, ScannerCycleOutcome::Completed | ScannerCycleOutcome::CompletedWithPendingMaintenance )); + assert!(bootstrap.record_cycle(outcome)); + assert!(!bootstrap.pending()); assert_eq!(cycle_info.next, retry_cycle + 1); assert!(!crate::scanner_io::dirty_usage_buckets_for_tests().contains_key(&bucket)); let usage = read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) @@ -1585,10 +1690,10 @@ async fn coordinator_persists_only_an_observation_during_pending_put() { let usage: DataUsageInfo = serde_json::from_slice(&usage).expect("the persisted usage should decode"); assert_eq!(usage.usage_snapshot_converged, Some(true)); assert_eq!(usage.scanner_cycle, Some(retry_cycle)); - assert_eq!(usage.objects_total_count, 1); + assert_eq!(usage.objects_total_count, 2); assert_eq!( usage.objects_total_size, - u64::try_from(committed_body.len()).expect("fixture body length") + u64::try_from(committed_body.len() + 4).expect("fixture total body length") ); let bucket_usage = usage .buckets_usage @@ -1606,7 +1711,11 @@ async fn complete_cycle_keeps_strict_publication_fence_with_observational_flag() let (_temp_dir, store) = setup_scanner_cycle_store().await; let pending = ecstore_hold_namespace_commit(store.as_ref()); - let complete_uses_observation = scanner_cycle_uses_observational_publication(ScannerCycleStatus::Complete, true); + let uses_observation = |status| { + ScannerCyclePublicationPlan::for_result(&ScannerCycleResult::new(status, None).with_usage_update_emitted(true)) + .is_observational() + }; + let complete_uses_observation = uses_observation(ScannerCycleStatus::Complete); assert!(!complete_uses_observation, "a converged Complete snapshot is authoritative"); assert_eq!( scanner_local_publication_defer_reason(store.as_ref(), complete_uses_observation).await, @@ -1614,16 +1723,30 @@ async fn complete_cycle_keeps_strict_publication_fence_with_observational_flag() "a Complete cycle must remain fenced while an ordinary namespace commit is pending" ); - assert!(scanner_cycle_uses_observational_publication(ScannerCycleStatus::Superseded, true)); - assert!(scanner_cycle_uses_observational_publication( - ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable), - true - )); - assert!(!scanner_cycle_uses_observational_publication(ScannerCycleStatus::Incomplete, true)); - assert!(!scanner_cycle_uses_observational_publication( - ScannerCycleStatus::Deferred(ScannerCycleDeferReason::DataMovement), - true - )); + assert!(uses_observation(ScannerCycleStatus::Superseded)); + assert!(uses_observation(ScannerCycleStatus::Deferred( + ScannerCycleDeferReason::ActivityBaselineUnavailable + ))); + assert!(!uses_observation(ScannerCycleStatus::Deferred(ScannerCycleDeferReason::DataMovement))); + drop(pending); +} + +#[tokio::test] +#[serial] +async fn incomplete_observation_uses_observational_lease_during_pending_namespace_commit() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + let pending = ecstore_hold_namespace_commit(store.as_ref()); + let result = ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None); + assert!(!ScannerCyclePublicationPlan::for_result(&result).is_observational()); + let uses_observation = ScannerCyclePublicationPlan::for_result(&result.with_usage_update_emitted(true)).is_observational(); + + assert!(uses_observation); + assert_eq!( + scanner_local_publication_defer_reason(store.as_ref(), uses_observation).await, + None, + "an incomplete cycle's observation must not be blocked by an ordinary namespace commit" + ); + drop(pending); } @@ -1639,7 +1762,8 @@ async fn test_finalize_partial_scan_cycle_advances_and_persists_counter() { cycle_completed: vec![], started: Utc::now(), }; - let mut cycle_metrics_guard = ScannerCycleMetricsGuard::new(cycle_info.clone()).await; + let mut cycle_metrics_guard = + ScannerCycleMetricsGuard::new(cycle_info.clone(), Arc::new(RemoteScanProgress::default())).await; assert!(finalize_partial_scan_cycle(&ctx, store.clone(), &mut cycle_info, &mut revision, 1, &mut cycle_metrics_guard,).await); @@ -1675,7 +1799,8 @@ async fn scanner_cycle_recovers_to_newer_durable_cache_floor() { cycle_completed: vec![], started: Utc::now(), }; - let mut cycle_metrics_guard = ScannerCycleMetricsGuard::new(cycle_info.clone()).await; + let mut cycle_metrics_guard = + ScannerCycleMetricsGuard::new(cycle_info.clone(), Arc::new(RemoteScanProgress::default())).await; assert!( persist_required_scanner_cycle_floor( @@ -1715,7 +1840,8 @@ async fn scanner_cycle_rejects_invalid_cache_floor() { cycle_completed: vec![], started: Utc::now(), }; - let mut cycle_metrics_guard = ScannerCycleMetricsGuard::new(cycle_info.clone()).await; + let mut cycle_metrics_guard = + ScannerCycleMetricsGuard::new(cycle_info.clone(), Arc::new(RemoteScanProgress::default())).await; assert!( !persist_required_scanner_cycle_floor( @@ -1736,7 +1862,8 @@ async fn scanner_cycle_rejects_invalid_cache_floor() { next: 12, ..Default::default() }; - let mut max_cycle_metrics_guard = ScannerCycleMetricsGuard::new(max_cycle_info.clone()).await; + let mut max_cycle_metrics_guard = + ScannerCycleMetricsGuard::new(max_cycle_info.clone(), Arc::new(RemoteScanProgress::default())).await; assert!( !persist_required_scanner_cycle_floor( &ctx, @@ -1803,6 +1930,140 @@ fn scanner_startup_fails_closed_on_nonempty_corrupt_cycle_state() { assert!(encode_scanner_cycle_state(&exhausted, 7).is_err()); } +#[cfg(feature = "diagnostic-read-version-caller")] +#[tokio::test] +#[serial] +async fn diagnostic_scanner_config_writes_preserve_origin_for_both_helpers_and_cancellation() { + use crate::storage_api::ecstore_diagnostic::{ + ReadVersionEntrySource, current_read_version_entry_source, with_read_version_entry, + }; + + for earlier in [None, Some(ReadVersionEntrySource::TierConfigReload)] { + for shared in [false, true] { + for case in ["success", "error", "cancelled"] { + let store = Arc::new(MemoryConfigStore::default()); + let path = "buckets/diagnostic/write-state"; + let key = memory_config_key(RUSTFS_META_BUCKET, path); + if case == "error" { + store.fail_put_number.lock().await.insert(key.clone(), 1); + } + store.pending_diagnostic_write.store(case == "cancelled", Ordering::Relaxed); + let write = async { + let operation = async { + let preconditions = crate::HTTPPreconditions { + if_none_match: Some("*".to_string()), + ..Default::default() + }; + if shared { + crate::save_config_shared_with_preconditions_and_lease_fence_and_scope( + store.clone(), + path, + bytes::Bytes::from_static(b"state"), + None, + preconditions, + None, + None, + ) + .await + } else { + crate::save_config_with_preconditions(store.clone(), path, b"state".to_vec(), preconditions).await + } + }; + let result = tokio::time::timeout(Duration::from_secs(1), operation).await; + match case { + "success" => assert!(result.expect("write completes").is_ok()), + "error" => assert!(result.expect("error completes").is_err()), + _ => assert!(result.is_err(), "pending write is cancelled"), + } + }; + match earlier { + Some(source) => with_read_version_entry(source, write).await, + None => write.await, + } + assert_eq!( + *store.write_entry_sources.lock().await, + vec![Some(earlier.unwrap_or(ReadVersionEntrySource::ScannerConfigWrite))] + ); + assert_eq!(current_read_version_entry_source(), None, "scope restores: {case}"); + let stored = store.objects.lock().await.get(&key).cloned(); + assert_eq!(stored, (case == "success").then(|| b"state".to_vec()), "write result preserved: {case}"); + } + } + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +#[tokio::test] +#[serial] +async fn diagnostic_cycle_state_load_preserves_entry_for_results_and_cancellation() { + use crate::storage_api::ecstore_diagnostic::{ + ReadVersionEntrySource, current_read_version_entry_source, with_read_version_entry, + }; + + for earlier in [None, Some(ReadVersionEntrySource::TierConfigReload)] { + for case in ["missing", "valid", "error", "cancelled"] { + let store = Arc::new(MemoryConfigStore::default()); + let marker = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()); + let state = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + if case == "valid" { + let cycle = CurrentCycle { + next: 9, + ..Default::default() + }; + store + .objects + .lock() + .await + .insert(state.clone(), encode_scanner_cycle_state(&cycle, 3).expect("cycle fixture")); + store.revisions.lock().await.insert(state, 7); + } else if case == "error" { + store + .read_errors + .lock() + .await + .insert(marker.clone(), EcstoreError::ErasureReadQuorum); + } else if case == "cancelled" { + store.delayed_gets.lock().await.insert(marker, Duration::from_secs(1)); + } + let load = async { + if case == "cancelled" { + assert!( + tokio::time::timeout(Duration::from_millis(1), load_scanner_cycle_state_for_startup(store.clone())) + .await + .is_err() + ); + return; + } + let result = load_scanner_cycle_state_for_startup(store.clone()).await; + match (case, result) { + ("missing", ScannerCycleStateStartup::Ready { .. }) => {} + ( + "valid", + ScannerCycleStateStartup::Ready { + cycle, leader_epoch: 3, .. + }, + ) => assert_eq!(cycle.next, 9), + ("error", ScannerCycleStateStartup::Transient(_)) => {} + _ => panic!("unexpected cycle load outcome: {case}"), + } + }; + match earlier { + Some(source) => with_read_version_entry(source, load).await, + None => load.await, + } + let observed = store.entry_sources.lock().await; + assert!(!observed.is_empty(), "load must enter a reader: {case}"); + assert!( + observed + .iter() + .all(|source| *source == Some(earlier.unwrap_or(ReadVersionEntrySource::ScannerCycleStateRead))), + "all readers preserve the original source: {case}" + ); + assert_eq!(current_read_version_entry_source(), None, "scope restores for all outcomes: {case}"); + } + } +} + #[tokio::test] #[serial] async fn corrupt_cycle_state_is_quarantined_once() { @@ -4600,7 +4861,8 @@ async fn test_finalize_partial_scan_cycle_reports_persist_failure() { cycle_completed: vec![], started: Utc::now(), }; - let mut cycle_metrics_guard = ScannerCycleMetricsGuard::new(cycle_info.clone()).await; + let mut cycle_metrics_guard = + ScannerCycleMetricsGuard::new(cycle_info.clone(), Arc::new(RemoteScanProgress::default())).await; assert!(!finalize_partial_scan_cycle(&ctx, store, &mut cycle_info, &mut revision, 1, &mut cycle_metrics_guard,).await); assert_eq!(cycle_info.next, 13); @@ -6344,6 +6606,7 @@ async fn test_observational_usage_defers_when_authoritative_baseline_is_missing( let (sender, receiver) = mpsc::channel(1); let mut observation = complete_usage_with_bucket_count(Some(std::time::SystemTime::UNIX_EPOCH + Duration::from_secs(20)), 1); observation.usage_snapshot_converged = Some(false); + observation.usage_snapshot_observational = true; sender.send(observation).await.expect("observation should enqueue"); drop(sender); @@ -6392,6 +6655,7 @@ async fn test_observational_usage_uses_fenced_backup_when_v2_primary_has_no_iden let (sender, receiver) = mpsc::channel(1); let mut observation = complete_usage_with_bucket_count(Some(std::time::SystemTime::UNIX_EPOCH + Duration::from_secs(20)), 1); observation.usage_snapshot_converged = Some(false); + observation.usage_snapshot_observational = true; sender.send(observation).await.expect("observation should enqueue"); drop(sender); @@ -6433,6 +6697,7 @@ async fn test_observational_usage_uses_bootstrap_pending_primary_as_baseline() { let (sender, receiver) = mpsc::channel(1); let mut observation = complete_usage_with_bucket_count(Some(std::time::SystemTime::UNIX_EPOCH + Duration::from_secs(20)), 1); observation.usage_snapshot_converged = Some(false); + observation.usage_snapshot_observational = true; sender.send(observation).await.expect("observation should enqueue"); drop(sender); @@ -7315,6 +7580,7 @@ async fn test_store_data_usage_in_backend_preserves_superseded_status() { scanner_epoch: Some(7), scanner_cycle: Some(11), usage_snapshot_converged: Some(false), + usage_snapshot_observational: true, ..complete_usage_with_bucket_count(None, 1) }) .await @@ -7342,6 +7608,56 @@ async fn test_store_data_usage_in_backend_preserves_superseded_status() { ); } +#[tokio::test] +async fn test_store_data_usage_in_backend_publishes_pending_bucket_certification_as_authoritative() { + let store = Arc::new(MemoryConfigStore::default()); + let authoritative_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_OBJ_NAME_PATH.as_str()); + let observed_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str()); + let authoritative = DataUsageInfo { + scanner_epoch: Some(7), + scanner_cycle: Some(10), + usage_snapshot_converged: Some(true), + ..complete_usage_with_bucket_count(Some(std::time::SystemTime::UNIX_EPOCH), 1) + }; + store.objects.lock().await.insert( + authoritative_key.clone(), + serde_json::to_vec(&authoritative).expect("authoritative snapshot should encode"), + ); + store.revisions.lock().await.insert(authoritative_key.clone(), 1); + + let (sender, receiver) = mpsc::channel(1); + sender + .send(DataUsageInfo { + last_update: Some(std::time::SystemTime::UNIX_EPOCH + Duration::from_secs(1)), + scanner_epoch: Some(7), + scanner_cycle: Some(11), + usage_snapshot_converged: Some(false), + usage_snapshot_pending_buckets: vec!["bucket-0".to_string()], + usage_snapshot_pending_bucket_count: 1, + ..complete_usage_with_bucket_count(None, 1) + }) + .await + .expect("certified usage snapshot should enqueue"); + drop(sender); + + let outcome = store_data_usage_in_backend_with_outcome(CancellationToken::new(), store.clone(), receiver).await; + assert_eq!(outcome, DataUsagePersistOutcome::Saved); + let objects = store.objects.lock().await; + assert!( + !objects.contains_key(&observed_key), + "pending buckets do not make a snapshot observational" + ); + let saved = serde_json::from_slice::(objects.get(&authoritative_key).expect("authoritative snapshot")) + .expect("persisted usage snapshot should decode"); + assert_eq!(saved.scanner_cycle, Some(11)); + assert_eq!(saved.usage_snapshot_converged, Some(false)); + assert_eq!(saved.usage_snapshot_pending_buckets, vec!["bucket-0".to_string()]); + assert_eq!(saved.usage_snapshot_pending_bucket_count, 1); + assert!(!saved.usage_snapshot_observational); + assert_eq!(saved.usage_snapshot_authoritative_baseline, None); + assert!(data_usage_info_has_persisted_baseline_identity(&saved)); +} + #[tokio::test] #[serial] async fn test_store_data_usage_in_backend_removes_observed_after_authoritative_save() { @@ -7365,6 +7681,7 @@ async fn test_store_data_usage_in_backend_removes_observed_after_authoritative_s scanner_epoch: Some(7), scanner_cycle: Some(11), usage_snapshot_converged: Some(false), + usage_snapshot_observational: true, ..complete_usage_with_bucket_count(None, 1) }) .await @@ -7780,7 +8097,7 @@ fn finalizing_post_scan_observation_advances_partially_without_dirty_ack() { ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable), Some(dirty_snapshot), ) - .with_observational_snapshot_published(true); + .with_usage_update_emitted(true); let (outcome, _, acknowledgements) = finalize_scanner_cycle_result(observed, DataUsagePersistOutcome::Saved.into()); @@ -8401,7 +8718,7 @@ fn superseded_retry_preserves_explicit_cycle_cadence() { ..Default::default() }; assert_eq!( - scanner_superseded_retry_interval(backoff, &default_config), + scanner_superseded_retry_interval(backoff, &default_config, ScannerMaintenanceFeatures::default()), Some(SCANNER_RETRY_BASE_INTERVAL), "the default adaptive cadence must retain fast convergence" ); @@ -8417,14 +8734,124 @@ fn superseded_retry_preserves_explicit_cycle_cadence() { ..Default::default() }; + for features in [ + ScannerMaintenanceFeatures::default(), + ScannerMaintenanceFeatures { + lifecycle: true, + ..Default::default() + }, + ] { + assert_eq!( + scanner_superseded_retry_interval(backoff, &runtime_config, features), + Some(runtime_config.cycle_interval), + "{source:?} cycle cadence must not be shortened by convergence retries" + ); + } + } +} + +#[test] +fn superseded_retry_does_not_delay_required_maintenance() { + let features = ScannerMaintenanceFeatures { + lifecycle: true, + ..Default::default() + }; + assert!(features.needs_regular_cycle()); + let config = ScannerRuntimeConfig { + cycle_interval: Duration::from_secs(2), + cycle_interval_source: ScannerRuntimeConfigSource::Env, + ..Default::default() + }; + let mut backoff = ScannerRetryBackoff::default(); + for consecutive_cycles in 1..=12 { + backoff.record_retryable_cycle(true); assert_eq!( - scanner_superseded_retry_interval(backoff, &runtime_config), - Some(runtime_config.cycle_interval), - "{source:?} cycle cadence must not be shortened by convergence retries" + scanner_superseded_retry_interval(backoff, &config, features), + Some(config.cycle_interval), + "superseded cycle {consecutive_cycles} must preserve regular maintenance" ); } } +#[test] +fn superseded_retry_only_caps_regular_maintenance() { + let config = ScannerRuntimeConfig { + cycle_interval: Duration::from_secs(60), + ..Default::default() + }; + let mut backoff = ScannerRetryBackoff::default(); + for _ in 0..6 { + backoff.record_retryable_cycle(true); + } + assert_eq!( + scanner_superseded_retry_interval(backoff, &config, ScannerMaintenanceFeatures::default()), + Some(Duration::from_secs(160)), + ); + for features in [ + ScannerMaintenanceFeatures { + lifecycle: true, + ..Default::default() + }, + ScannerMaintenanceFeatures { + replication: true, + ..Default::default() + }, + ScannerMaintenanceFeatures { + inspection_failed: true, + ..Default::default() + }, + ] { + assert_eq!(scanner_superseded_retry_interval(backoff, &config, features), Some(config.cycle_interval)); + assert_eq!(scanner_superseded_retry_interval(ScannerRetryBackoff::default(), &config, features), None); + } +} + +#[tokio::test(start_paused = true)] +#[serial] +async fn superseded_maintenance_wait_keeps_convergence_fencing() { + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); + let config = ScannerRuntimeConfig { + cycle_interval: Duration::from_secs(60), + ..Default::default() + }; + let features = ScannerMaintenanceFeatures { + lifecycle: true, + ..Default::default() + }; + let mut backoff = ScannerRetryBackoff::default(); + for _ in 0..12 { + backoff.record_retryable_cycle(true); + } + let interval = scanner_superseded_retry_interval(backoff, &config, features); + assert_eq!(interval, Some(config.cycle_interval)); + let generations = ScannerCycleObservedGenerations::for_wait( + &config, + interval, + crate::scanner_io::dirty_usage_generation(), + crate::runtime_config::scanner_runtime_config_generation(), + crate::scanner_io::scanner_maintenance_generation(), + ); + assert!(generations.dirty_usage.is_none()); + assert!(generations.defer_cluster_activity); + let ctx = CancellationToken::new(); + let mut seen = Some(BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3))])); + let changed = BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 8, 3))]); + let started = tokio::time::Instant::now(); + let reason = wait_for_next_scanner_cycle_with_activity( + &ctx, + interval.expect("maintenance must retain its convergence interval"), + Some(Duration::from_secs(1)), + &mut seen, + generations, + || false, + || std::future::ready(Ok(changed.clone())), + ) + .await; + assert_eq!(reason, ScannerCycleWakeReason::Timer); + assert_eq!(started.elapsed(), config.cycle_interval); + assert_eq!(seen, Some(changed)); +} + #[test] fn publication_proof_retry_backoff_reaches_its_short_cap() { for (failures, expected) in [(1, 5), (2, 10), (3, 20), (4, 30), (20, 30)] { @@ -8959,18 +9386,10 @@ fn usage_bootstrap_rebuild_bypasses_clean_idle_until_authoritative_cycle() { cycle_interval: Duration::from_secs(60), ..Default::default() }; - let features = ScannerMaintenanceFeatures::default(); - let generation = Some(scanner_maintenance_generation()); let mut clean_idle_backoff = ScannerCleanIdleBackoff { interval_multiplier: 8 }; let mut rebuild = ScannerUsageBootstrapRebuild::from_startup(PersistedUsageFloorStartup::BootstrapPending); assert!(!rebuild.clean_idle_backoff_enabled(true)); - assert!(rebuild.requires_full_scan( - features, - generation, - scanner_maintenance_generation(), - ScannerCycleWakeReason::DirtyUsage, - )); let plan = rebuild.wait_plan(scanner_cycle_wait_plan(&config, clean_idle_backoff, true, std::convert::identity), None); assert_eq!(plan.delay, Duration::ZERO); @@ -9004,6 +9423,40 @@ fn usage_bootstrap_rebuild_bypasses_clean_idle_until_authoritative_cycle() { ); } +#[test] +fn superseded_retry_preserves_full_maintenance_boundaries() { + let features = ScannerMaintenanceFeatures::default(); + for (wake, ordinary_full, retry_full) in [ + (ScannerCycleWakeReason::Timer, true, false), + (ScannerCycleWakeReason::DirtyUsage, false, false), + (ScannerCycleWakeReason::ClusterActivity, false, false), + (ScannerCycleWakeReason::ClusterMaintenance, true, true), + (ScannerCycleWakeReason::MovementGeneration, true, true), + (ScannerCycleWakeReason::Recovery, true, true), + ] { + assert_eq!(features.requires_full_scan(Some(7), 7, wake, false), ordinary_full, "{wake:?}"); + assert_eq!(features.requires_full_scan(Some(7), 7, wake, true), retry_full, "{wake:?} retry"); + assert!(features.requires_full_scan(None, 7, wake, true)); + assert!(features.requires_full_scan(Some(6), 7, wake, true)); + for unsafe_features in [ + ScannerMaintenanceFeatures { + lifecycle: true, + ..features + }, + ScannerMaintenanceFeatures { + replication: true, + ..features + }, + ScannerMaintenanceFeatures { + inspection_failed: true, + ..features + }, + ] { + assert!(unsafe_features.requires_full_scan(Some(7), 7, wake, true)); + } + } +} + #[tokio::test] #[serial] async fn scoped_scan_explicit_bitrot_keeps_dirty_planning_without_idle_backoff() { @@ -9032,12 +9485,12 @@ async fn scoped_scan_explicit_bitrot_keeps_dirty_planning_without_idle_backoff() (ScannerCycleWakeReason::ClusterMaintenance, true), ] { assert_eq!( - features.requires_full_scan(generation, scanner_maintenance_generation(), wake), + features.requires_full_scan(generation, scanner_maintenance_generation(), wake, false), full, "{wake:?}" ); } - assert!(features.requires_full_scan(None, scanner_maintenance_generation(), ScannerCycleWakeReason::DirtyUsage)); + assert!(features.requires_full_scan(None, scanner_maintenance_generation(), ScannerCycleWakeReason::DirtyUsage, false)); for unsafe_features in [ ScannerMaintenanceFeatures { lifecycle: true, @@ -9055,18 +9508,25 @@ async fn scoped_scan_explicit_bitrot_keeps_dirty_planning_without_idle_backoff() assert!(unsafe_features.requires_full_scan( generation, scanner_maintenance_generation(), - ScannerCycleWakeReason::DirtyUsage + ScannerCycleWakeReason::DirtyUsage, + false )); } crate::scanner_io::record_scanner_maintenance_change("maintenance-proof-change"); - assert!(features.requires_full_scan(generation, scanner_maintenance_generation(), ScannerCycleWakeReason::DirtyUsage)); + assert!(features.requires_full_scan( + generation, + scanner_maintenance_generation(), + ScannerCycleWakeReason::DirtyUsage, + false + )); let (refreshed, refreshed_generation) = detect_stable_scanner_maintenance_features(&ctx, &store) .await .expect("changed maintenance generation should be inspected"); assert!(!refreshed.requires_full_scan( Some(refreshed_generation), scanner_maintenance_generation(), - ScannerCycleWakeReason::DirtyUsage + ScannerCycleWakeReason::DirtyUsage, + false )); crate::scanner_io::clear_dirty_usage_buckets_for_tests(); }) @@ -9604,24 +10064,47 @@ fn scoped_scan_remote_dirty_coverage_invalidates_local_bucket_current() { } #[test] -fn post_lease_activity_proof_rejects_a_put_tail_that_finished_before_lease_acquisition() { - let before = BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3))]); - let expected_digest = Some(scanner_activity_snapshot_digest(&before)); - assert_eq!(scanner_post_lease_activity_defer_reason(expected_digest, Ok(before.clone())), None); +fn post_lease_activity_proof_rejects_a_peer_whose_dirty_usage_moved_before_lease_acquisition() { + let before = BTreeMap::from([ + (LOCAL_SCANNER_ACTIVITY_NODE.to_string(), scanner_node_activity("local", 5, 2)), + ("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3)), + ]); + assert_eq!(scanner_post_lease_activity_defer_reason(Some(&before), Ok(before.clone())), None); + // Local generation movement in the lease window is the next cycle's + // work: its buckets stay in the dirty map and the walk remains certified. + let mut local_shaded = before.clone(); + let local = local_shaded + .get_mut(LOCAL_SCANNER_ACTIVITY_NODE) + .expect("local node should be present"); + local.namespace_generation += 1; + local.dirty_usage_generation += 1; + local.dirty_usage_pending = true; + assert_eq!(scanner_post_lease_activity_defer_reason(Some(&before), Ok(local_shaded)), None); + + // A peer PUT tail that drained before the lease leaves the movement + // generation accepted by the lease unchanged, but the peer's dirty state + // moved and this phase cannot name its buckets. let mut after = before.clone(); - after - .get_mut("node-2") - .expect("writer should be present") - .namespace_generation += 1; + let writer = after.get_mut("node-2").expect("writer should be present"); + writer.namespace_generation += 1; + writer.dirty_usage_generation += 1; assert_eq!( before["node-2"].movement_generation, after["node-2"].movement_generation, "the existing movement-only lease remains valid after a PUT tail drains" ); assert!(scanner_activity_allows_usage_publication(&after)); - let reason = scanner_post_lease_activity_defer_reason(expected_digest, Ok(after)); + let reason = scanner_post_lease_activity_defer_reason(Some(&before), Ok(after)); assert_eq!(reason, Some(ScannerCycleDeferReason::ActivityBaselineUnavailable)); + // A guardrail field moving in the window still defers. + let mut moved = before.clone(); + moved.get_mut("node-2").expect("peer should be present").movement_generation += 1; + assert_eq!( + scanner_post_lease_activity_defer_reason(Some(&before), Ok(moved)), + Some(ScannerCycleDeferReason::ActivityBaselineUnavailable) + ); + let result = ScannerCycleResult::new(ScannerCycleStatus::Complete, None).with_remote_dirty_usage_acknowledgements(vec![ ScannerDirtyUsageAcknowledgement { host: "node-2".to_string(), @@ -9704,15 +10187,13 @@ fn remote_lease_validation_failure_preserves_movement_defer_for_remote_fence_los #[test] fn post_lease_activity_proof_requires_a_complete_matching_baseline() { let before = BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3))]); - let digest = scanner_activity_snapshot_digest(&before); let mut blocked = before.clone(); blocked.get_mut("node-2").expect("peer should be present").publication_blocked = true; - let blocked_digest = scanner_activity_snapshot_digest(&blocked); for (expected, observed) in [ - (None, Ok(before)), - (Some(digest), Err("peer is unavailable".to_string())), - (Some(digest), Ok(BTreeMap::new())), - (Some(blocked_digest), Ok(blocked)), + (None, Ok(before.clone())), + (Some(&before), Err("peer is unavailable".to_string())), + (Some(&before), Ok(BTreeMap::new())), + (Some(&blocked), Ok(blocked.clone())), ] { assert_eq!( scanner_post_lease_activity_defer_reason(expected, observed), @@ -9945,6 +10426,132 @@ fn remote_restart_is_distinct_from_deferred_cluster_activity() { assert!(scanner_activity_observed_work(ScannerActivityObservation::RemoteRestarted)); } +type ActivityFieldMutations = Vec<(&'static str, Box)>; + +#[test] +fn guardrail_veto_is_attributed_to_the_field_and_host_that_moved() { + let before = BTreeMap::from([ + (LOCAL_SCANNER_ACTIVITY_NODE.to_string(), scanner_node_activity("local", 5, 2)), + ("node-2".to_string(), scanner_node_activity("remote", 7, 3)), + ]); + assert_eq!(scanner_activity_guardrail_veto(&before, &before), None); + + let cases: ActivityFieldMutations = vec![ + ("instance_id", Box::new(|activity| activity.instance_id = "restarted".to_string())), + ("protocol_version", Box::new(|activity| activity.protocol_version += 1)), + ("topology_digest", Box::new(|activity| activity.topology_digest = [4; 32])), + ("maintenance_generation", Box::new(|activity| activity.maintenance_generation += 1)), + ("data_movement_active", Box::new(|activity| activity.data_movement_active = true)), + ("movement_generation", Box::new(|activity| activity.movement_generation += 1)), + ("publication_blocked", Box::new(|activity| activity.publication_blocked = true)), + ]; + for host in [LOCAL_SCANNER_ACTIVITY_NODE, "node-2"] { + for (field, mutate) in &cases { + let mut after = before.clone(); + mutate(after.get_mut(host).expect("host should be present")); + let veto = scanner_activity_guardrail_veto(&before, &after).unwrap_or_else(|| panic!("{field} on {host} must veto")); + assert_eq!(veto.field, *field, "{host}"); + assert_eq!(veto.host, host); + assert_eq!(veto.reason(), format!("{field}@{host}")); + assert!(SCANNER_ACTIVITY_GUARDRAIL_VETO_FIELDS.contains(field)); + } + } + + let mut peer_lost = before.clone(); + peer_lost.remove("node-2"); + assert_eq!( + scanner_activity_guardrail_veto(&before, &peer_lost), + Some(ScannerActivityGuardrailVeto::new("node-2", "peer_set")) + ); + let mut peer_added = before.clone(); + peer_added.insert("node-3".to_string(), scanner_node_activity("third", 1, 1)); + assert_eq!( + scanner_activity_guardrail_veto(&before, &peer_added), + Some(ScannerActivityGuardrailVeto::new("node-3", "peer_set")) + ); +} + +#[test] +fn generation_movement_shades_buckets_without_a_guardrail_veto() { + let before = BTreeMap::from([ + (LOCAL_SCANNER_ACTIVITY_NODE.to_string(), scanner_node_activity("local", 5, 2)), + ("node-2".to_string(), scanner_node_activity("remote", 7, 3)), + ]); + let shade: ActivityFieldMutations = vec![ + ("namespace_generation", Box::new(|activity| activity.namespace_generation += 1)), + ("dirty_usage_generation", Box::new(|activity| activity.dirty_usage_generation += 1)), + ("dirty_usage_pending", Box::new(|activity| activity.dirty_usage_pending = true)), + ]; + for host in [LOCAL_SCANNER_ACTIVITY_NODE, "node-2"] { + for (field, mutate) in &shade { + let mut after = before.clone(); + mutate(after.get_mut(host).expect("host should be present")); + assert_ne!(after, before); + assert_eq!(scanner_activity_guardrail_veto(&before, &after), None, "{field} on {host} must not veto"); + assert!(SCANNER_ACTIVITY_SHADE_FIELDS.contains(field)); + assert_eq!( + scanner_activity_structural_digest(&before), + scanner_activity_structural_digest(&after), + "{field} must stay outside the structural digest" + ); + } + } + + // Everything shaded at once on every host is still not a veto. + let mut all_shaded = before.clone(); + for activity in all_shaded.values_mut() { + activity.namespace_generation += 3; + activity.dirty_usage_generation += 3; + activity.dirty_usage_pending = true; + } + assert_eq!(scanner_activity_guardrail_veto(&before, &all_shaded), None); +} + +#[test] +fn peer_dirty_usage_movement_is_reported_for_remote_hosts_only() { + let before = BTreeMap::from([ + (LOCAL_SCANNER_ACTIVITY_NODE.to_string(), scanner_node_activity("local", 5, 2)), + ("node-2".to_string(), scanner_node_activity("remote", 7, 3)), + ]); + assert_eq!(scanner_peer_dirty_usage_moved(&before, &before), None); + + let mut local_moved = before.clone(); + let local = local_moved + .get_mut(LOCAL_SCANNER_ACTIVITY_NODE) + .expect("local activity should exist"); + local.dirty_usage_generation += 1; + local.dirty_usage_pending = true; + local.namespace_generation += 1; + assert_eq!(scanner_peer_dirty_usage_moved(&before, &local_moved), None); + + let mut remote_namespace_only = before.clone(); + remote_namespace_only + .get_mut("node-2") + .expect("remote activity should exist") + .namespace_generation += 1; + assert_eq!(scanner_peer_dirty_usage_moved(&before, &remote_namespace_only), None); + + let mut remote_generation = before.clone(); + remote_generation + .get_mut("node-2") + .expect("remote activity should exist") + .dirty_usage_generation += 1; + assert_eq!( + scanner_peer_dirty_usage_moved(&before, &remote_generation), + Some(ScannerActivityGuardrailVeto::new("node-2", "dirty_usage_generation")) + ); + + let mut remote_pending = before.clone(); + remote_pending + .get_mut("node-2") + .expect("remote activity should exist") + .dirty_usage_pending = true; + assert_eq!( + scanner_peer_dirty_usage_moved(&before, &remote_pending), + Some(ScannerActivityGuardrailVeto::new("node-2", "dirty_usage_pending")) + ); +} + #[test] fn local_maintenance_wakeup_releases_a_remote_maintenance_block() { let blocked = scanner_activity_backoff_blocked_after_wake(false, ScannerCycleWakeReason::ClusterMaintenance); diff --git a/crates/scanner/src/scanner/tests/cycle_budget_publication.rs b/crates/scanner/src/scanner/tests/cycle_budget_publication.rs new file mode 100644 index 000000000..497ff8ee5 --- /dev/null +++ b/crates/scanner/src/scanner/tests/cycle_budget_publication.rs @@ -0,0 +1,417 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use rustfs_utils::path::path_join_buf; + +#[tokio::test] +#[serial] +async fn exhausted_budget_still_proves_peer_movement_without_namespace_authority() { + let (_temp_dir, store) = setup_scanner_cycle_store_with_pool_count(true, 2).await; + let pending = ecstore_hold_namespace_commit(store.as_ref()); + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()); + budget.cancel_for_runtime(); + let generation = store.scanner_data_movement_generation(); + let result = ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None).with_remote_publication_lease_targets(vec![( + "peer".into(), + "session".into(), + generation, + )]); + let plan = ScannerCyclePublicationPlan::for_result(&result); + let calls = AtomicUsize::new(0); + + let proof = plan + .await_leases(&ctx, &budget, 352, |observational| { + calls.fetch_add(1, Ordering::Relaxed); + store.acquire_scanner_publication_lease_with_purpose(generation, Duration::from_secs(60), observational) + }) + .await; + let ScannerPublicationProofWait::Ready((token, admitted_generation)) = proof else { + panic!("a stopped walk must still prove peer movement while the leader context is alive"); + }; + assert_eq!(calls.load(Ordering::Relaxed), 1); + assert!(!ctx.is_cancelled()); + store + .validate_scanner_publication_lease_with_purpose(token, admitted_generation, true) + .await + .expect("progress must retain a movement-only lease"); + assert!( + store + .validate_scanner_publication_lease_with_purpose(token, admitted_generation, false) + .await + .is_err(), + "the lease must not authorize an authoritative snapshot" + ); + assert!(store.release_scanner_publication_lease(token).await); + + let stale = plan + .await_leases(&ctx, &budget, 352, |observational| { + store.acquire_scanner_publication_lease_with_purpose(generation + 1, Duration::from_secs(60), observational) + }) + .await; + assert!(matches!(stale, ScannerPublicationProofWait::Rejected(_))); + drop(pending); + + store.pool_meta.write().await.pools[0].decommission = Some(crate::storage_api::owner::EcstorePoolDecommissionInfo { + failed: true, + ..Default::default() + }); + assert!(store.scanner_data_movement_pause_status().await.paused); + let current_generation = store.scanner_data_movement_generation(); + let moving = plan + .await_leases(&ctx, &budget, 352, |observational| { + store.acquire_scanner_publication_lease_with_purpose(current_generation, Duration::from_secs(60), observational) + }) + .await; + assert!( + matches!(moving, ScannerPublicationProofWait::Rejected(error) if error.to_string().contains("blocked by data movement")) + ); +} + +#[tokio::test] +async fn emitted_updates_keep_budget_cancellation_and_publication_purpose() { + for (status, expected_plan) in [ + (ScannerCycleStatus::Complete, ScannerCyclePublicationPlan::Authoritative), + (ScannerCycleStatus::Incomplete, ScannerCyclePublicationPlan::Observational), + (ScannerCycleStatus::Superseded, ScannerCyclePublicationPlan::Observational), + ( + ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable), + ScannerCyclePublicationPlan::Observational, + ), + ( + ScannerCycleStatus::Deferred(ScannerCycleDeferReason::DataMovement), + ScannerCyclePublicationPlan::Authoritative, + ), + ] { + let result = ScannerCycleResult::new(status, None).with_usage_update_emitted(true); + let plan = ScannerCyclePublicationPlan::for_result(&result); + assert_eq!(plan, expected_plan); + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()); + let ready = plan + .await_leases(&ctx, &budget, 352, |observational| async move { Ok::<_, String>(observational) }) + .await; + assert!(matches!(ready, ScannerPublicationProofWait::Ready(observational) if observational == plan.is_observational())); + budget.cancel_for_runtime(); + let calls = AtomicUsize::new(0); + let cancelled = plan + .await_leases(&ctx, &budget, 352, |_| { + calls.fetch_add(1, Ordering::Relaxed); + async { Ok::<(), String>(()) } + }) + .await; + assert!(matches!(cancelled, ScannerPublicationProofWait::Cancelled)); + assert_eq!(calls.load(Ordering::Relaxed), 0); + } +} + +#[tokio::test] +async fn progress_proof_honors_parent_cancellation() { + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()); + ctx.cancel(); + let calls = AtomicUsize::new(0); + let proof = ScannerCyclePublicationPlan::ProgressOnly + .await_leases(&ctx, &budget, 352, |_| { + calls.fetch_add(1, Ordering::Relaxed); + async { Ok::<_, String>(()) } + }) + .await; + assert!(matches!(proof, ScannerPublicationProofWait::Cancelled)); + assert_eq!(calls.load(Ordering::Relaxed), 0); +} + +#[tokio::test(start_paused = true)] +async fn progress_proof_is_bounded_without_a_runtime_budget() { + for stalled in [false, true] { + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new( + &ctx, + ScannerCycleBudgetConfig { + max_objects: Some(1), + ..Default::default() + }, + ); + budget.record_object_scanned(); + assert!(budget.budget_elapsed()); + let calls = AtomicUsize::new(0); + let started = Instant::now(); + let proof = ScannerCyclePublicationPlan::ProgressOnly + .await_leases(&ctx, &budget, 352, |_| { + calls.fetch_add(1, Ordering::Relaxed); + async move { + if stalled { + std::future::pending::<()>().await; + } + Err::<(), _>("scanner publication lease capacity is exhausted") + } + }) + .await; + assert!(matches!(proof, ScannerPublicationProofWait::Cancelled)); + assert!(calls.load(Ordering::Relaxed) > 0); + assert_eq!(started.elapsed(), SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT); + assert!(!ctx.is_cancelled()); + } +} + +#[tokio::test] +#[serial] +async fn coordinator_persists_budget_progress_without_usage_or_dirty_acknowledgement() { + assert_coordinator_partial_progress(false, None).await; +} + +#[tokio::test] +#[serial] +async fn coordinator_persists_mixed_plan_tail_without_usage_or_dirty_acknowledgement() { + assert_coordinator_partial_progress(true, None).await; +} + +#[tokio::test] +#[serial] +async fn coordinator_partial_progress_requires_successful_cycle_state_save() { + for after_commit in [false, true] { + assert_coordinator_partial_progress(true, Some(after_commit)).await; + } +} + +async fn assert_coordinator_partial_progress(resume_mixed_plan: bool, fail_cycle_save: Option) { + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); + let (_temp_dir, store) = setup_scanner_cycle_store().await; + let bucket = "scanner-budget-progress"; + store + .make_bucket(bucket, &crate::storage_api::scan::MakeBucketOptions::default()) + .await + .expect("fixture bucket"); + for object in ["first", "second", "third"] { + store.pools[0].disk_set[0] + .put_object( + bucket, + object, + &mut PutObjReader::from_vec(b"value".to_vec()), + &ObjectOptions { + write_completion: WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("fixture object"); + } + crate::scanner_io::record_dirty_usage_bucket(bucket); + let dirty_before = crate::scanner_io::dirty_usage_buckets_for_tests(); + let baseline = read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("usage baseline"); + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new_with_progress_tracking( + &ctx, + ScannerCycleBudgetConfig { + max_objects: Some(1), + ..Default::default() + }, + ); + let mut cycle = CurrentCycle { + next: 352, + ..Default::default() + }; + let mut revision = DataUsageCacheRevision::Missing; + let outcome = run_data_scanner_cycle_with_budget( + &ctx, + &store, + &mut cycle, + &mut revision, + 1, + Arc::clone(&budget), + ScannerCycleScheduling { + requires_full_scan: true, + service_cohort: None, + }, + ) + .await; + assert!(matches!(outcome, ScannerCycleOutcome::Partial)); + assert_eq!(budget.reason(), Some(ScannerCycleBudgetReason::Objects)); + assert!(budget.progress().0 > 0); + assert!(budget.cycle_state_persisted()); + assert_eq!(cycle.next, 353); + assert!(cycle.cycle_completed.is_empty()); + assert!(matches!(revision, DataUsageCacheRevision::Etag(_))); + let state = read_config(store.clone(), &DATA_USAGE_BLOOM_NAME_PATH) + .await + .expect("durable cycle progress"); + let (persisted, epoch) = decode_scanner_cycle_state(&state).expect("valid cycle state"); + assert_eq!(persisted.next, 353); + assert_eq!(epoch, 1); + assert_eq!( + read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("unchanged root"), + baseline + ); + assert!( + read_config(store.clone(), DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str()) + .await + .is_err() + ); + assert_eq!(crate::scanner_io::dirty_usage_buckets_for_tests(), dirty_before); + + if resume_mixed_plan { + let set = store.pools[0].disk_set[0].clone(); + let cache_name = path_join_buf(&[bucket, DATA_USAGE_CACHE_NAME]); + let mut checkpoint = DataUsageCache::default(); + let revisions = checkpoint + .load_with_revisions(set.clone(), &cache_name) + .await + .expect("load the bounded sweep's durable checkpoint"); + assert!( + checkpoint.info.scan_checkpoint.is_some() + || checkpoint.info.scan_resume_after.is_some() + || checkpoint.info.scan_raw_enumeration_cursor.is_some() + || checkpoint.info.scan_raw_enumeration_page_index.is_some() + || checkpoint.info.scan_coverage_receipt.is_some(), + "bounded sweep must retain a durable position: {:?}", + checkpoint.info + ); + let progress = checkpoint.info.scan_progress.as_mut().expect("bounded sweep has a plan"); + // A restarted process requests a new plan while retaining the old prefix. + progress.started_plan.0[0] ^= 1; + let epoch = checkpoint.info.scan_identity.expect("checkpoint identity").publication_epoch; + checkpoint + .save_with_revisions_for_epoch(set.clone(), &cache_name, &revisions, epoch) + .await + .expect("persist a prefix from a different process plan"); + + let tail_budget = ScannerCycleBudget::new_with_progress_tracking(&ctx, ScannerCycleBudgetConfig::default()); + let failure_guard = fail_cycle_save.map(|after_commit| super::cycle_persist_failure::fail_next_save(1, after_commit)); + let tail_outcome = run_data_scanner_cycle_with_budget( + &ctx, + &store, + &mut cycle, + &mut revision, + 1, + Arc::clone(&tail_budget), + ScannerCycleScheduling { + requires_full_scan: true, + service_cohort: None, + }, + ) + .await; + assert!(!tail_budget.budget_elapsed(), "this is not the budget-exhaustion fast path"); + checkpoint.load(set, &cache_name).await.expect("reload the saved mixed tail"); + assert!(!checkpoint.info.snapshot_complete); + assert!(checkpoint.info.scan_checkpoint.is_none()); + assert!(checkpoint.info.scan_resume_after.is_none()); + let progress = checkpoint + .info + .scan_progress + .expect("a fresh verification sweep is still required"); + assert_eq!(progress.started_plan, progress.requested_plan); + assert_eq!( + read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("root retained"), + baseline + ); + assert!( + read_config(store.clone(), DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str()) + .await + .is_err() + ); + assert_eq!(crate::scanner_io::dirty_usage_buckets_for_tests(), dirty_before); + if let Some(after_commit) = fail_cycle_save { + assert_eq!(tail_outcome, ScannerCycleOutcome::StatePersistenceFailed); + assert!(!tail_budget.cycle_state_persisted()); + assert!(super::cycle_persist_failure::take_failure(1).is_none(), "injected fault must have fired"); + let state = read_config(store, &DATA_USAGE_BLOOM_NAME_PATH) + .await + .expect("last durable cycle"); + assert_eq!( + decode_scanner_cycle_state(&state) + .expect("cycle after failed acknowledgement") + .0 + .next, + 353 + u64::from(after_commit) + ); + drop(failure_guard); + global_metrics().set_cycle(None).await; + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); + return; + } + assert_eq!( + tail_outcome, + ScannerCycleOutcome::Partial, + "saved recovery progress is not a usage-save failure" + ); + assert_eq!(cycle.next, 354); + assert!(cycle.cycle_completed.is_empty()); + assert!(tail_budget.cycle_state_persisted()); + let state = read_config(store.clone(), &DATA_USAGE_BLOOM_NAME_PATH) + .await + .expect("durable tail cycle"); + assert_eq!(decode_scanner_cycle_state(&state).expect("tail cycle state").0.next, 354); + + let verification = run_data_scanner_cycle_with_budget( + &ctx, + &store, + &mut cycle, + &mut revision, + 1, + ScannerCycleBudget::new_with_progress_tracking(&ctx, ScannerCycleBudgetConfig::default()), + ScannerCycleScheduling { + requires_full_scan: true, + service_cohort: None, + }, + ) + .await; + assert_eq!(verification, ScannerCycleOutcome::Completed, "only the fresh sweep certifies usage"); + let usage: DataUsageInfo = serde_json::from_slice( + &read_config(store, DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("verified root"), + ) + .expect("verified usage JSON"); + assert_eq!((usage.objects_total_count, usage.objects_total_size), (3, 15)); + assert!(!crate::scanner_io::dirty_usage_buckets_pending()); + } + global_metrics().set_cycle(None).await; + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); +} + +#[test] +fn partial_progress_does_not_mask_missing_updates_or_persistence_failure() { + for (persisted, expected) in [ + (DataUsagePersistOutcome::NoUpdate, ScannerCycleOutcome::Partial), + (DataUsagePersistOutcome::Failed, ScannerCycleOutcome::Failed), + (DataUsagePersistOutcome::Current, ScannerCycleOutcome::Failed), + ( + DataUsagePersistOutcome::Deferred(ScannerCycleDeferReason::DataMovement), + ScannerCycleOutcome::Deferred(ScannerCycleDeferReason::DataMovement), + ), + ] { + let result = ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None).with_partial_bucket_progress(true); + assert_eq!(scanner_cycle_completion_outcome_for_result(&result, persisted), expected); + } + for result in [ + ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None), + ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None) + .with_partial_bucket_progress(true) + .with_usage_update_emitted(true), + ScannerCycleResult::new(ScannerCycleStatus::Complete, None).with_partial_bucket_progress(true), + ] { + assert_eq!( + scanner_cycle_completion_outcome_for_result(&result, DataUsagePersistOutcome::NoUpdate), + ScannerCycleOutcome::Failed, + "partial progress cannot stand in for an expected root update: {result:?}" + ); + } +} diff --git a/crates/scanner/src/scanner/tests/cycle_persist_failure.rs b/crates/scanner/src/scanner/tests/cycle_persist_failure.rs index 14345f101..d710ba3c6 100644 --- a/crates/scanner/src/scanner/tests/cycle_persist_failure.rs +++ b/crates/scanner/src/scanner/tests/cycle_persist_failure.rs @@ -24,6 +24,17 @@ static FAILURE: StdMutex> = StdMutex::new(None); struct FailureGuard; +pub(super) fn fail_next_save(epoch: u64, after_commit: bool) -> impl Drop { + let mut slot = FAILURE.lock().expect("failure injection lock"); + assert!(slot.is_none(), "state failure injection must not replace another test's fault"); + *slot = Some(Failure { + epoch, + successful_saves: 0, + after_commit, + }); + FailureGuard +} + impl Drop for FailureGuard { fn drop(&mut self) { *FAILURE.lock().expect("failure injection lock") = None; diff --git a/crates/scanner/src/scanner/tests/recovery_control.rs b/crates/scanner/src/scanner/tests/recovery_control.rs index 61ac96bc5..b721435d1 100644 --- a/crates/scanner/src/scanner/tests/recovery_control.rs +++ b/crates/scanner/src/scanner/tests/recovery_control.rs @@ -81,6 +81,159 @@ async fn persisted_state(store: &Arc) -> Vec<(Option>, DataUsag state } +async fn seed_transient_cycle_read_failure() -> ScannerCycleRecoveryStatus { + let store = Arc::new(MemoryConfigStore::default()); + let error = EcstoreError::InsufficientReadQuorum(RUSTFS_META_BUCKET.to_string(), DATA_USAGE_BLOOM_NAME_PATH.clone()); + let expected_error = format!("Scanner error: failed to inspect scanner cycle state: {error}"); + store + .read_errors + .lock() + .await + .insert(memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()), error); + let ScannerCycleStateStartup::Transient(error) = load_scanner_cycle_state_for_startup(store).await else { + panic!("a quorum outage must produce transient cycle recovery"); + }; + assert_eq!(error.to_string(), expected_error); + let status = scanner_cycle_recovery_status(); + assert_eq!(status.state, "transient"); + assert!(status.retryable); + assert_eq!(status.classification, None); + status +} + +#[tokio::test] +#[serial] +async fn issue_8297_contended_follower_clears_only_transient_cycle_recovery() { + temp_env::async_with_vars([("RUSTFS_LOCK_ACQUIRE_TIMEOUT", Some("1"))], async { + let (_dir, store) = setup_scanner_cycle_store().await; + seed_cleanup(&store, "cleanup-pending").await; + let before = persisted_state(&store).await; + let lock = store + .new_ns_lock(RUSTFS_META_BUCKET, "leader.lock") + .await + .expect("leader lock"); + let guard = lock + .get_write_lock_quiet(Duration::from_secs(1)) + .await + .expect("hold live leader"); + let transient = seed_transient_cycle_read_failure().await; + assert!(!record_scanner_cycle_recovery_retry(MAX_SCANNER_CYCLE_RECOVERY_RETRIES)); + let paused = scanner_cycle_recovery_status(); + assert_eq!(paused.state, "paused"); + let mut cases = vec![(transient, true), (paused, true)]; + for (state, classification, retryable) in [ + ("recovery-required", None, false), + ("blocked", None, false), + ("blocked", Some("corrupt"), false), + ("cleanup-pending", None, true), + ("cleanup-pending", Some("corrupt"), true), + ("usage_floor_load_failed", Some("usage_floor_load_failed"), true), + ("usage_floor_recovery_pending", Some("legacy_empty_usage_floor"), true), + ("paused", Some("usage_floor_load_failed"), false), + ("cache_cycle_ahead", Some("cache_cycle_ahead"), true), + ] { + cases.push(( + ScannerCycleRecoveryStatus { + state: state.to_string(), + classification: classification.map(str::to_string), + retry_count: 5, + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + retryable, + reason: Some("preserve recovery state".to_string()), + ..Default::default() + }, + false, + )); + } + for (status, clear) in cases { + let expected = serde_json::to_value(&status).expect("expected recovery status"); + publish_scanner_cleanup_status(status, None).expect("seed local recovery status"); + tokio::time::timeout( + Duration::from_secs(30), + run_data_scanner_with_maintenance_state( + CancellationToken::new(), + store.clone(), + ScannerMaintenanceFeatures::default(), + None, + ), + ) + .await + .expect("contended scanner iteration must finish") + .expect("lock contention is a normal follower outcome"); + assert_eq!(scanner_cycle_schedule_status().execution_role, "follower"); + let actual = scanner_cycle_recovery_status(); + if clear { + assert_eq!(actual.state, "healthy", "stale follower recovery: {expected}"); + assert_eq!(actual.retry_count, 0); + assert!(!actual.retryable); + } else { + assert_eq!(serde_json::to_value(actual).expect("actual recovery status"), expected); + } + assert!(!guard.is_lock_lost(), "follower must not revoke the leader"); + assert_eq!(persisted_state(&store).await, before, "follower must not rewrite durable recovery state"); + } + }) + .await; +} + +#[tokio::test] +#[serial] +async fn issue_8297_supervisor_recovers_paused_follower_without_restart() { + temp_env::async_with_vars( + [ + ("RUSTFS_LOCK_ACQUIRE_TIMEOUT", Some("1")), + (ENV_SCANNER_START_DELAY_SECS, Some("0")), + (ENV_SCANNER_CYCLE, Some("1")), + ], + async { + let (_dir, store) = setup_scanner_cycle_store().await; + seed_cleanup(&store, "cleanup-pending").await; + let before = persisted_state(&store).await; + let lock = store + .new_ns_lock(RUSTFS_META_BUCKET, "leader.lock") + .await + .expect("leader lock"); + let guard = lock + .get_write_lock_quiet(Duration::from_secs(1)) + .await + .expect("hold live leader"); + for attempt in 1..=MAX_SCANNER_CYCLE_RECOVERY_RETRIES { + seed_transient_cycle_read_failure().await; + assert_eq!(record_scanner_cycle_recovery_retry(attempt), attempt < MAX_SCANNER_CYCLE_RECOVERY_RETRIES); + } + assert_eq!(scanner_cycle_recovery_status().state, "paused"); + assert!(!crate::scanner_runtime_initialized(), "test must own the scanner runtime"); + let ctx = CancellationToken::new(); + let _cancel_on_drop = ctx.clone().drop_guard(); + init_data_scanner_with_storage(ctx.clone(), store.clone()).await; + assert!(crate::scanner_runtime_initialized()); + let recovered = tokio::time::timeout(Duration::from_secs(30), async { + while scanner_cycle_recovery_status().state != "healthy" { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await; + ctx.cancel(); + tokio::time::timeout(Duration::from_secs(15), async { + while crate::scanner_runtime_initialized() { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("cancelled scanner supervisor must stop"); + recovered.expect("follower must recover without another five-minute pause or process restart"); + assert_eq!(scanner_cycle_schedule_status().execution_role, "follower"); + let status = scanner_cycle_recovery_status(); + assert_eq!(status.retry_count, 0); + assert!(!status.retryable); + assert!(!guard.is_lock_lost(), "supervisor must not revoke the leader"); + assert_eq!(persisted_state(&store).await, before, "supervisor must leave durable recovery untouched"); + }, + ) + .await; + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); +} + async fn run_disabled_startup(ctx: CancellationToken, store: Arc) { let initialized_before = crate::scanner_runtime_initialized(); let cleanup = init_scanner_with_recovery(ctx, store, false).await; diff --git a/crates/scanner/src/scanner/tests/scoped_ack_publication.rs b/crates/scanner/src/scanner/tests/scoped_ack_publication.rs index 472e595d7..0acac8514 100644 --- a/crates/scanner/src/scanner/tests/scoped_ack_publication.rs +++ b/crates/scanner/src/scanner/tests/scoped_ack_publication.rs @@ -48,6 +48,7 @@ async fn complete_candidate(store: &Arc, cycle: u64) -> (crate::scanner requires_full_scan: true, service_cohort: None, resolved_scope_observer: None, + post_walk_gate: None, }, ) .await @@ -242,6 +243,7 @@ async fn scoped_ack_publication_observed_candidate_reuse_requires_a_new_root_pro // Seed a complete but unconverged observation from real scanner coverage; // the production writer attaches its authoritative baseline identity. observed_candidate.usage_snapshot_converged = Some(false); + observed_candidate.usage_snapshot_observational = true; let observation = publish_candidate(&store, &prior_scan, observed_candidate, None).await; let (outcome, proof) = observation.into_parts(); assert_eq!(outcome, DataUsagePersistOutcome::Saved); @@ -269,6 +271,7 @@ async fn scoped_ack_publication_observed_candidate_reuse_requires_a_new_root_pro requires_full_scan: false, service_cohort: None, resolved_scope_observer: Some(observer), + post_walk_gate: None, }, ) .await diff --git a/crates/scanner/src/scanner/usage_store.rs b/crates/scanner/src/scanner/usage_store.rs index 21a8a7cad..828999202 100644 --- a/crates/scanner/src/scanner/usage_store.rs +++ b/crates/scanner/src/scanner/usage_store.rs @@ -577,7 +577,10 @@ where outcome = DataUsagePersistOutcome::Deferred(ScannerCycleDeferReason::DataMovement); break 'updates; } - let observational = data_usage_info.usage_snapshot_converged == Some(false); + // Routing keys on the explicit producer mark, not on convergence: a + // `Complete` cycle with pending buckets is authoritative but not + // converged, while a vetoed cycle is observational. + let observational = data_usage_info.usage_snapshot_observational; let target_path = if observational { DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str() } else { @@ -1143,7 +1146,6 @@ where } } if !observational - && data_usage_info.usage_snapshot_converged == Some(true) && matches!(outcome, DataUsagePersistOutcome::Saved | DataUsagePersistOutcome::AlreadyDurable) && (outcome == DataUsagePersistOutcome::AlreadyDurable || write_confirmed) && let (Some(expected), Some(epoch)) = (ack_expectation.as_ref(), ack_epoch) diff --git a/crates/scanner/src/scanner_budget.rs b/crates/scanner/src/scanner_budget.rs index 65743439f..3a084e90c 100644 --- a/crates/scanner/src/scanner_budget.rs +++ b/crates/scanner/src/scanner_budget.rs @@ -12,6 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +use rustfs_scanner_metrics::metrics::RemoteScanProgress; use std::sync::{ Arc, atomic::{AtomicBool, AtomicU8, AtomicU64, Ordering}, @@ -73,6 +74,7 @@ pub struct ScannerCycleBudget { entries_visited: AtomicU64, last_progress_millis: AtomicU64, cycle_state_persisted: AtomicBool, + remote_scan_progress: Arc, } impl ScannerCycleBudget { @@ -137,6 +139,7 @@ impl ScannerCycleBudget { entries_visited: AtomicU64::new(0), last_progress_millis: AtomicU64::new(0), cycle_state_persisted: AtomicBool::new(false), + remote_scan_progress: Arc::new(RemoteScanProgress::default()), }) } @@ -237,7 +240,12 @@ impl ScannerCycleBudget { } } + pub(crate) fn remote_scan_progress(&self) -> Arc { + Arc::clone(&self.remote_scan_progress) + } + pub(crate) fn record_remote_progress(&self, objects_scanned: u64, directories_started: u64) { + self.remote_scan_progress.record(objects_scanned, directories_started); if self.track_progress || self.max_objects.is_some() { let objects = saturating_fetch_add(&self.objects_scanned, objects_scanned); if self.track_progress { diff --git a/crates/scanner/src/scanner_folder.rs b/crates/scanner/src/scanner_folder.rs index fdc7a7f41..9a3e9a829 100644 --- a/crates/scanner/src/scanner_folder.rs +++ b/crates/scanner/src/scanner_folder.rs @@ -21,8 +21,9 @@ use std::time::{Duration, Instant, SystemTime}; use crate::ReplTargetSizeSummary; use crate::data_usage_define::{ DATA_USAGE_SCAN_CHECKPOINT_VERSION, DataUsageCache, DataUsageCacheInfo, DataUsageEntry, DataUsageHash, DataUsageHashMap, - DataUsageRawEnumerationCursor, DataUsageScanCheckpoint, DataUsageScanCheckpointReason, PendingScannerHeal, - PendingScannerHealKind, ScannerSizeSummaryExt, SizeReconciliationEntry, SizeSummary, hash_path, + DataUsageRawEnumerationCursor, DataUsageScanCheckpoint, DataUsageScanCheckpointReason, MAX_DATA_USAGE_CACHE_DEPTH, + PendingScannerHeal, PendingScannerHealKind, ScannerSizeSummaryExt, SizeReconciliationEntry, SizeSummary, hash_path, + scan_frontier_covers_path, }; use crate::error::ScannerError; use crate::raw_page_index::{RawEnumerationPageIndex, RawEnumerationPageWriter}; @@ -121,6 +122,14 @@ const MAX_SIZE_RECONCILIATION_ENTRIES_PER_BUCKET: usize = 10_000; const MAX_SIZE_RECONCILIATION_BYTES_PER_BUCKET: usize = 8 * 1024 * 1024; const MAX_SIZE_RECONCILIATION_AGE_SECS: u64 = 7 * 24 * 60 * 60; +fn checkpoint_interval_for_build_time(build_time: Duration) -> Duration { + // Aim to spend at most a tenth of the interval building snapshots, while + // preserving the existing upper bound on periodic checkpoint attempts. + build_time + .saturating_mul(10) + .clamp(SCANNER_CHECKPOINT_MIN_INTERVAL, SCANNER_CHECKPOINT_INTERVAL) +} + // --- scanner excess alerts as S3 notification events (rustfs/backlog#1868) -- // // The excess-versions / excess-version-size / excess-folders alerts were @@ -472,6 +481,19 @@ fn checkpoint_reason_from_budget(reason: Option) -> Da } } +// Cache entry failures survive diagnostic retry-cache TTL and capacity eviction. +// Index once per scan; resumed subtree checks must not flatten histograms repeatedly. +fn failed_scan_paths(cache: &DataUsageCache) -> Vec { + let mut paths = cache + .cache + .iter() + .filter(|(_, entry)| entry.failed_objects > 0) + .map(|(path, _)| path.clone()) + .collect::>(); + paths.sort_unstable(); + paths +} + fn data_usage_entry_has_progress(entry: &DataUsageEntry) -> bool { data_usage_root_has_progress(entry) } @@ -704,6 +726,12 @@ impl ScannerBucketPrefixScanScope { } /// Folder scanner for scanning directory structures +#[derive(Default)] +struct ScannerCheckpointAncestor { + hash: DataUsageHash, + entry: DataUsageEntry, +} + pub struct FolderScanner { root: String, old_cache: DataUsageCache, @@ -728,10 +756,13 @@ pub struct FolderScanner { updates: Option>, checkpoint_tx: Option>, + checkpoint_ancestors: Vec, + checkpoint_depth: usize, last_update: SystemTime, checkpoint_objects: u64, last_checkpoint_objects: u64, last_checkpoint_at: Instant, + last_checkpoint_build_time: Duration, update_current_path: UpdateCurrentPathFn, @@ -744,6 +775,7 @@ pub struct FolderScanner { pending_heals_changed: bool, coverage_frontier: Option, resume_frontier: Option, + resume_failed_paths: Vec, coverage_gap: bool, raw_enumeration_progress: Vec, pending_heal_sync_deferred: bool, @@ -1099,16 +1131,39 @@ impl FolderScanner { self.update_cache.info.scan_checkpoint = Some(checkpoint); } - fn record_completed_child(&mut self, folder: &str, healthy: bool) { - if self.old_cache.info.scan_progress.is_some() { - self.coverage_gap |= !healthy; - if !self.coverage_gap { - self.coverage_frontier = Some(folder.to_owned()); - } + fn record_completed_child(&mut self, folder: &str) { + if self.old_cache.info.scan_progress.is_some() + && !self.coverage_gap + && !self + .coverage_frontier + .as_deref() + .is_some_and(|frontier| scan_frontier_covers_path(folder, frontier)) + { + self.coverage_frontier = Some(folder.to_owned()); } self.record_scan_resume_hint(folder); } + fn resume_subtree_has_failures(&self, folder: &str) -> bool { + if self.resume_failed_paths.is_empty() { + return false; + } + if self + .resume_failed_paths + .binary_search_by(|path| path.as_str().cmp(folder)) + .is_ok() + { + return true; + } + let prefix = format!("{folder}/"); + let index = self + .resume_failed_paths + .partition_point(|path| path.as_str() < prefix.as_str()); + self.resume_failed_paths + .get(index) + .is_some_and(|path| path.starts_with(&prefix)) + } + fn record_scan_resume_hint_if_not_ancestor(&mut self, folder: &str) { let keep_existing = self .new_cache @@ -1210,11 +1265,43 @@ impl FolderScanner { .map_or((None, None), |progress| (progress.cursor(), progress.page_index())) } - fn maybe_send_checkpoint(&mut self) { + async fn scan_child_folder( + &mut self, + ctx: CancellationToken, + folder: CachedFolder, + parent_hash: &mut DataUsageHash, + parent_entry: &mut DataUsageEntry, + child_entry: &mut DataUsageEntry, + ) -> Result<(), ScannerError> { + if self.checkpoint_tx.is_none() || self.old_cache.info.scan_progress.is_none() { + return Box::pin(self.scan_folder(ctx, folder, child_entry)).await; + } + let depth = self.checkpoint_depth; + if depth >= MAX_DATA_USAGE_CACHE_DEPTH { + return Err(ScannerError::Other("scanner checkpoint ancestor depth limit exceeded".to_string())); + } + if self.checkpoint_ancestors.len() == depth { + self.checkpoint_ancestors.push(ScannerCheckpointAncestor::default()); + } + // Reuse the empty frame slots: moving suspended accumulators avoids + // cloning a growing child set or allocating histogram vectors per child. + std::mem::swap(parent_hash, &mut self.checkpoint_ancestors[depth].hash); + std::mem::swap(parent_entry, &mut self.checkpoint_ancestors[depth].entry); + self.checkpoint_depth += 1; + let result = Box::pin(self.scan_folder(ctx, folder, child_entry)).await; + self.checkpoint_depth -= 1; + // Restore the parent before cancellation/error handling stitches partial work. + std::mem::swap(parent_hash, &mut self.checkpoint_ancestors[depth].hash); + std::mem::swap(parent_entry, &mut self.checkpoint_ancestors[depth].entry); + result + } + + fn maybe_send_checkpoint(&mut self, folder: &CachedFolder, hash: &DataUsageHash, into: &DataUsageEntry) { let Some(checkpoint_tx) = self.checkpoint_tx.clone() else { return }; let elapsed = self.last_checkpoint_at.elapsed(); - if self.new_cache.info.scan_progress.is_none() - || elapsed < SCANNER_CHECKPOINT_MIN_INTERVAL + if into.compacted + || self.new_cache.info.scan_progress.is_none() + || elapsed < checkpoint_interval_for_build_time(self.last_checkpoint_build_time) || (self.checkpoint_objects.saturating_sub(self.last_checkpoint_objects) < SCANNER_CHECKPOINT_OBJECT_INTERVAL && elapsed < SCANNER_CHECKPOINT_INTERVAL) { @@ -1233,34 +1320,55 @@ impl FolderScanner { } let Ok(permit) = checkpoint_tx.try_reserve() else { return }; + // Invalid partial graphs must not trigger a fresh O(N) clone on every object. + self.last_checkpoint_objects = self.checkpoint_objects; + self.last_checkpoint_at = Instant::now(); + let snapshot = self.checkpoint_snapshot(folder, hash, into); + self.last_checkpoint_build_time = self.last_checkpoint_at.elapsed(); + if let Some(snapshot) = snapshot { + permit.send(snapshot); + } + } + + fn checkpoint_snapshot(&self, folder: &CachedFolder, hash: &DataUsageHash, into: &DataUsageEntry) -> Option { + // Heal revisits are not sorted with the forward sweep. An unfinished + // earlier subtree, including a suspended ancestor, must not be signed + // as completed prefix coverage. + if self.coverage_frontier.as_deref().is_some_and(|frontier| { + scan_frontier_covers_path(&folder.name, frontier) + || self.checkpoint_ancestors[..self.checkpoint_depth] + .iter() + .any(|ancestor| scan_frontier_covers_path(&ancestor.hash.0, frontier)) + }) { + return None; + } let mut snapshot = self.new_cache.clone(); + let mut parent = None; + for ancestor in &self.checkpoint_ancestors[..self.checkpoint_depth] { + snapshot.replace_hashed(&ancestor.hash, &parent, &ancestor.entry); + parent = Some(ancestor.hash.clone()); + } + snapshot.replace_hashed(hash, &folder.parent, into); + snapshot.checked_flatten_complete_scope(&snapshot.info.name)?; snapshot.info.last_update = Some(SystemTime::now()); snapshot.info.snapshot_complete = false; let (cursor, page_index) = self.peek_raw_enumeration_resume_state(); - if cursor.is_some() || page_index.is_some() { - snapshot.info.scan_raw_enumeration_cursor = cursor; - snapshot.info.scan_raw_enumeration_page_index = page_index; - snapshot.info.scan_resume_after = None; - snapshot.info.scan_checkpoint = None; - snapshot.info.scan_coverage_receipt = None; - } else if snapshot.seal_scan_frontier(self.coverage_frontier.as_deref()).is_err() { - return; - } - if snapshot.root().is_none() - && snapshot.info.scan_raw_enumeration_cursor.is_none() - && snapshot.info.scan_raw_enumeration_page_index.is_none() - { - return; + snapshot.info.scan_raw_enumeration_cursor = cursor; + snapshot.info.scan_raw_enumeration_page_index = page_index; + snapshot.seal_scan_frontier(self.coverage_frontier.as_deref()).ok()?; + // A resumed readdir can run before covered subtrees are copied. Keep + // its durable receipt until this snapshot can represent that coverage. + if self.coverage_frontier.is_some() && snapshot.info.scan_coverage_receipt.is_none() { + return None; } + // The receipt was just sealed from this snapshot; do not hash it twice. if snapshot.info.scan_raw_enumeration_cursor.is_none() && snapshot.info.scan_raw_enumeration_page_index.is_none() - && snapshot.validated_scan_frontier().is_none() + && snapshot.info.scan_coverage_receipt.is_none() { - return; + return None; } - permit.send(snapshot); - self.last_checkpoint_objects = self.checkpoint_objects; - self.last_checkpoint_at = Instant::now(); + Some(snapshot) } fn carry_forward_old_children(&mut self, parent_hash: &DataUsageHash, entry: &mut DataUsageEntry) { @@ -1297,6 +1405,15 @@ impl FolderScanner { child_entry: &DataUsageEntry, ) { if data_usage_entry_has_progress(child_entry) { + // A cancelled out-of-order heal revisit is not completed prefix work. + if self + .coverage_frontier + .as_deref() + .is_some_and(|frontier| scan_frontier_covers_path(&child_hash.0, frontier)) + { + self.coverage_frontier = None; + self.coverage_gap = true; + } let mut child_entry = child_entry.clone(); self.carry_forward_old_children(child_hash, &mut child_entry); self.record_scan_resume_hint_if_not_ancestor(&child_hash.key()); @@ -1513,7 +1630,7 @@ impl FolderScanner { return Err(ScannerError::Other("Operation cancelled".to_string())); } - let this_hash = hash_path(&folder.name); + let mut this_hash = hash_path(&folder.name); // Store initial compaction state. let was_compacted = into.compacted; @@ -1526,7 +1643,7 @@ impl FolderScanner { let mut abandoned_children: DataUsageHashMap = HashSet::new(); if !into.compacted { - abandoned_children = self.old_cache.find_children_copy(this_hash.clone()); + abandoned_children = self.old_cache.find_children_copy(&this_hash); } debug!( @@ -1587,8 +1704,26 @@ impl FolderScanner { "Scanner folder state updated" ); + // A child prefix can be removed after its parent listed it and before + // it is opened here (e.g. concurrent object deletes). Treat it like a + // directory that vanished during iteration instead of failing the + // whole parent subtree. The bucket root (no parent) still propagates + // NotFound so namespace removal is detected by the caller. let mut dir_reader = match tokio::fs::read_dir(&dir_path).await { - Ok(dir_reader) => dir_reader, + Ok(dir_reader) => Some(dir_reader), + Err(e) if e.kind() == ErrorKind::NotFound && folder.parent.is_some() => { + debug!( + target: "rustfs::scanner::folder", + event = EVENT_SCANNER_FOLDER_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_FOLDER, + dir_path = %dir_path, + state = "dir_missing_before_open", + error = %e, + "Scanner folder state updated" + ); + None + } Err(e) => return Err(ScannerError::Io(e)), }; let mut pending_entry_progress = 0_u64; @@ -1597,7 +1732,11 @@ impl FolderScanner { let raw_enumeration_committed_entries = self.raw_enumeration_committed_entry_oracle(&folder.name); loop { - let entry = match dir_reader.next_entry().await { + let Some(reader) = dir_reader.as_mut() else { + raw_enumeration_complete = true; + break; + }; + let entry = match reader.next_entry().await { Ok(Some(entry)) => entry, Ok(None) => { raw_enumeration_complete = true; @@ -1651,7 +1790,7 @@ impl FolderScanner { } } self.record_raw_enumeration_entry(&folder.name, &file_name); - self.maybe_send_checkpoint(); + self.maybe_send_checkpoint(&folder, &this_hash, into); let is_storage_format_entry = file_name == STORAGE_FORMAT_FILE; let file_path = entry.path().to_string_lossy().to_string(); @@ -1835,7 +1974,7 @@ impl FolderScanner { // A new FolderScanner starts with an empty seen set, so // cached paths are still rechecked once per cycle. Skip // duplicate listings within this same scanner pass. - self.coverage_gap |= self.old_cache.info.scan_progress.is_some(); + into.failed_objects = into.failed_objects.max(1); continue; } @@ -1851,9 +1990,8 @@ impl FolderScanner { if failure_action != GetSizeFailureAction::Skip { self.resolved_failed_object_paths.remove(&item.path); - self.coverage_gap |= self.old_cache.info.scan_progress.is_some(); + into.failed_objects = into.failed_objects.saturating_add(1); if !retry_suppressed { - into.failed_objects += 1; self.record_failed(&item.path); } @@ -1958,7 +2096,7 @@ impl FolderScanner { object_count += 1; self.budget.record_object_scanned(); self.checkpoint_objects = self.checkpoint_objects.saturating_add(1); - self.maybe_send_checkpoint(); + self.maybe_send_checkpoint(&folder, &this_hash, into); timer.sleep().await; @@ -2006,14 +2144,13 @@ impl FolderScanner { } if self.is_erasure_mode && found_erasure_data_directory && !found_object_metadata { - self.coverage_gap |= self.old_cache.info.scan_progress.is_some(); found_object_metadata = true; let metadata_path = path_join_buf(&[&dir_path, STORAGE_FORMAT_FILE]); self.mark_failed_path_seen(&metadata_path); let retry_suppressed = self.failed_retry_suppressed(&metadata_path); + into.failed_objects = into.failed_objects.saturating_add(1); if !retry_suppressed { - into.failed_objects = into.failed_objects.saturating_add(1); self.record_failed(&metadata_path); let failed_cache_entries = self.new_cache.info.failed_objects.len(); @@ -2159,11 +2296,11 @@ impl FolderScanner { let h = hash_path(&folder_item.name); if forward_sweep && !into.compacted - && forward_resume_after.as_deref().is_some_and(|resume| { - folder_item.name.as_str() <= resume - && !matches!(folder_resume_match(&folder_item.name, resume), Some(FolderResumeMatch::Descendant)) - }) + && forward_resume_after + .as_deref() + .is_some_and(|resume| scan_frontier_covers_path(&folder_item.name, resume)) && self.old_cache.find(&folder_item.name).is_some() + && !self.resume_subtree_has_failures(&folder_item.name) { self.new_cache.copy_with_children(&self.old_cache, &h, &folder_item.parent); into.add_child(&h); @@ -2203,6 +2340,12 @@ impl FolderScanner { // Usage sampling is not proof that a Deep check ran. if self.scan_mode != HealScanMode::Deep && !forward_sweep + && !self + .old_cache + .info + .replication + .as_ref() + .is_some_and(|config| config.has_active_rules("", true)) && !into.compacted && self.old_cache.is_compacted(&h) { @@ -2226,15 +2369,17 @@ impl FolderScanner { // In compacted mode child totals are accumulated directly into the parent entry. let fut = Box::pin(self.scan_folder(ctx.clone(), folder_item.clone(), into)); fut.await.map_err(|e| ScannerError::Other(e.to_string()))?; - self.record_completed_child(&folder_item.name, into.failed_objects == 0); + self.record_completed_child(&folder_item.name); self.send_update_for_entry(&this_hash, &folder.parent, into).await; tokio::task::yield_now().await; } else { let mut dst = DataUsageEntry::default(); // Use Box::pin for recursive async call - let fut = Box::pin(self.scan_folder(ctx.clone(), folder_item.clone(), &mut dst)); - if let Err(e) = fut.await { + if let Err(e) = self + .scan_child_folder(ctx.clone(), folder_item.clone(), &mut this_hash, into, &mut dst) + .await + { if ctx.is_cancelled() { self.preserve_partial_child_progress(&folder_item.parent, &h, into, &dst) .await; @@ -2257,8 +2402,8 @@ impl FolderScanner { tokio::task::yield_now().await; into.add_child(&h); - self.record_completed_child(&folder_item.name, dst.failed_objects == 0); - self.maybe_send_checkpoint(); + self.record_completed_child(&folder_item.name); + self.maybe_send_checkpoint(&folder, &this_hash, into); // We scanned a folder, optionally send update. self.update_cache.delete_recursive(&h); self.update_cache.copy_with_children(&self.new_cache, &h, &folder_item.parent); @@ -2635,8 +2780,10 @@ impl FolderScanner { let h = hash_path(&folder_item.name); // Use Box::pin for recursive async call - let fut = Box::pin(self.scan_folder(ctx.clone(), folder_item.clone(), &mut dst)); - if let Err(e) = fut.await { + if let Err(e) = self + .scan_child_folder(ctx.clone(), folder_item.clone(), &mut this_hash, into, &mut dst) + .await + { if ctx.is_cancelled() { self.preserve_partial_child_progress(&folder_item.parent, &h, into, &dst) .await; @@ -2662,7 +2809,7 @@ impl FolderScanner { self.update_cache.delete_recursive(&h); self.update_cache.copy_with_children(&self.new_cache, &h, &folder_item.parent); self.send_update().await; - self.maybe_send_checkpoint(); + self.maybe_send_checkpoint(&folder, &this_hash, into); } } } @@ -2803,8 +2950,12 @@ pub(crate) async fn scan_data_folder_scoped( cache.info.tier_registry_generation = Some(tier_registry.generation); let resume_frontier = cache.validated_scan_frontier().map(str::to_owned); + if resume_frontier.is_some() { + global_metrics().record_scanner_checkpoint_used(); + } let bucket_name = cache.info.name.clone(); let next_cycle = cache.info.next_cycle; + let resume_position_on_entry = cache.has_scan_position(); let had_scan_checkpoint = cache.info.scan_checkpoint.is_some(); let new_cache = DataUsageCache { info: cache.info.clone(), @@ -2814,6 +2965,11 @@ pub(crate) async fn scan_data_folder_scoped( info: cache.info.clone(), ..Default::default() }; + let resume_failed_paths = if resume_frontier.is_some() { + failed_scan_paths(&cache) + } else { + Vec::new() + }; // Create folder scanner let mut scanner = FolderScanner { root: base_path, @@ -2834,10 +2990,13 @@ pub(crate) async fn scan_data_folder_scoped( disks_quorum, updates, checkpoint_tx, + checkpoint_ancestors: Vec::new(), + checkpoint_depth: 0, last_update: SystemTime::UNIX_EPOCH, checkpoint_objects: 0, last_checkpoint_objects: 0, last_checkpoint_at: Instant::now(), + last_checkpoint_build_time: Duration::ZERO, update_current_path, budget: budget.clone(), skip_heal, @@ -2846,6 +3005,7 @@ pub(crate) async fn scan_data_folder_scoped( pending_heals_changed: false, coverage_frontier: resume_frontier.clone(), resume_frontier, + resume_failed_paths, coverage_gap: false, pending_heal_sync_deferred: false, pending_heal_batch_dirty: false, @@ -2889,7 +3049,7 @@ pub(crate) async fn scan_data_folder_scoped( new_cache.info.last_update = Some(SystemTime::now()); new_cache.info.next_cycle = next_cycle; let unresolved_objects = coverage_gap - || root.failed_objects > 0 + || new_cache.cache.values().any(|entry| entry.failed_objects > 0) || !new_cache.info.failed_objects.is_empty() || !new_cache.info.size_reconciliation.is_empty(); let mixed_coverage = new_cache @@ -2920,6 +3080,27 @@ pub(crate) async fn scan_data_folder_scoped( close_disk_guard.close().await; if unresolved_objects || mixed_coverage { + debug!( + target: "rustfs::scanner::folder", + event = EVENT_SCANNER_FOLDER_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_FOLDER, + state = "bucket_sweep_incomplete", + bucket = %bucket_name, + pool = new_cache.info.source.map(|source| source.pool_index), + set = new_cache.info.source.map(|source| source.set_index), + cycle = next_cycle, + leader_epoch = new_cache.info.leader_epoch, + resume_position_on_entry, + resume_position = new_cache.has_scan_position(), + coverage_gap, + unresolved_objects, + root_failed_objects = root.failed_objects, + failed_objects = new_cache.info.failed_objects.len(), + size_reconciliation_entries = new_cache.info.size_reconciliation.len(), + mixed_coverage, + "Scanner bucket sweep requires another verification pass" + ); Err(ScannerError::PartialCache(Box::new(std::mem::take(new_cache)))) } else { Ok(std::mem::take(new_cache)) @@ -2988,6 +3169,13 @@ pub(crate) async fn scan_data_folder_scoped( } if matches!(&e, ScannerError::Io(io) if io.kind() == ErrorKind::NotFound) { let mut partial_cache = scanner.old_cache; + // An empty input still needs a root for the non-publishable + // partial result; retain nonempty durable caches unchanged. + if partial_cache.cache.is_empty() { + partial_cache + .cache + .insert(hash_path(&bucket_name).0, DataUsageEntry::default()); + } partial_cache.info.last_update = Some(SystemTime::now()); partial_cache.info.next_cycle = next_cycle; partial_cache.info.snapshot_complete = false; diff --git a/crates/scanner/src/scanner_folder/tests.rs b/crates/scanner/src/scanner_folder/tests.rs index bde002039..731571965 100644 --- a/crates/scanner/src/scanner_folder/tests.rs +++ b/crates/scanner/src/scanner_folder/tests.rs @@ -343,10 +343,13 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) { disks_quorum: 0, updates: None, checkpoint_tx: None, + checkpoint_ancestors: Vec::new(), + checkpoint_depth: 0, last_update: SystemTime::UNIX_EPOCH, checkpoint_objects: 0, last_checkpoint_objects: 0, last_checkpoint_at: Instant::now(), + last_checkpoint_build_time: Duration::ZERO, update_current_path, budget: ScannerCycleBudget::new(&CancellationToken::new(), Default::default()), skip_heal: Arc::new(AtomicBool::new(false)), @@ -359,6 +362,7 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) { pending_heals_changed: false, coverage_frontier: None, resume_frontier: None, + resume_failed_paths: Vec::new(), coverage_gap: false, raw_enumeration_progress: Vec::new(), pending_heal_sync_deferred: false, @@ -373,15 +377,8 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) { (scanner, temp_dir) } -#[tokio::test] -async fn periodic_checkpoint_emits_at_object_threshold_without_wall_clock_wait() { +async fn build_checkpoint_scanner() -> (FolderScanner, std::path::PathBuf) { let (mut scanner, temp_dir) = build_test_scanner().await; - let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1); - scanner.checkpoint_tx = Some(checkpoint_tx); - scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL; - scanner.last_checkpoint_at = Instant::now() - .checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL) - .expect("test instant subtraction"); scanner.new_cache.info.name = "bucket".to_string(); scanner.new_cache.info.scan_progress = Some(crate::DataUsageScanProgress { started_plan: crate::DataUsageScanPlanDigest([1; 32]), @@ -409,29 +406,196 @@ async fn periodic_checkpoint_emits_at_object_threshold_without_wall_clock_wait() .seal_scan_frontier(Some("bucket/a")) .expect("fixture frontier"); assert_eq!(scanner.new_cache.validated_scan_frontier(), Some("bucket/a")); + (scanner, temp_dir) +} - scanner.maybe_send_checkpoint(); +#[tokio::test] +async fn periodic_checkpoint_emits_at_object_threshold_without_wall_clock_wait() { + let (mut scanner, temp_dir) = build_checkpoint_scanner().await; + let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1); + scanner.checkpoint_tx = Some(checkpoint_tx); + scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL; + scanner.last_checkpoint_at = Instant::now() + .checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL) + .expect("test instant subtraction"); + let checkpoint_folder = CachedFolder { + name: "bucket".to_string(), + parent: None, + object_heal_prob_div: 1, + }; + let checkpoint_hash = hash_path("bucket"); + let checkpoint_entry = scanner.new_cache.root().expect("fixture root"); + scanner.maybe_send_checkpoint(&checkpoint_folder, &checkpoint_hash, &checkpoint_entry); + + let build_time = scanner.last_checkpoint_build_time; scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL * 2; scanner.last_checkpoint_at = Instant::now() .checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL) .expect("test instant subtraction"); - scanner.maybe_send_checkpoint(); + scanner.maybe_send_checkpoint(&checkpoint_folder, &checkpoint_hash, &checkpoint_entry); assert_eq!( scanner.last_checkpoint_objects, SCANNER_CHECKPOINT_OBJECT_INTERVAL, "a full checkpoint queue must reject before cloning or advancing progress" ); + assert_eq!( + scanner.last_checkpoint_build_time, build_time, + "a full queue must not reset the cost estimate" + ); let checkpoint = checkpoint_rx.try_recv().expect("object threshold emits a bounded checkpoint"); assert_eq!(checkpoint.info.name, "bucket"); assert!(!checkpoint.info.snapshot_complete); assert_eq!(scanner.last_checkpoint_objects, SCANNER_CHECKPOINT_OBJECT_INTERVAL); assert!(checkpoint_rx.try_recv().is_err(), "checkpoint queue remains bounded"); + scanner + .new_cache + .cache + .insert("bucket/orphan".to_string(), DataUsageEntry::default()); + scanner.maybe_send_checkpoint(&checkpoint_folder, &checkpoint_hash, &checkpoint_entry); + assert!( + checkpoint_rx.try_recv().is_err(), + "unreachable observations must not become resumable snapshots" + ); + assert_eq!( + scanner.last_checkpoint_objects, + SCANNER_CHECKPOINT_OBJECT_INTERVAL * 2, + "an invalid attempt must still consume the producer interval" + ); + scanner.new_cache.cache.remove("bucket/orphan"); + scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL * 3; + scanner.maybe_send_checkpoint(&checkpoint_folder, &checkpoint_hash, &checkpoint_entry); + assert!(checkpoint_rx.try_recv().is_err(), "a repaired graph still waits for the minimum interval"); + tokio::fs::remove_dir_all(temp_dir) .await .expect("remove test scanner directory"); } +#[test] +fn checkpoint_build_cost_interval_is_bounded() { + for (build_time, interval) in [ + (Duration::ZERO, Duration::from_secs(5)), + (Duration::from_millis(500), Duration::from_secs(5)), + (Duration::from_millis(501), Duration::from_millis(5010)), + (Duration::from_secs(3), Duration::from_secs(30)), + (Duration::from_secs(6), Duration::from_secs(60)), + (Duration::MAX, Duration::from_secs(60)), + ] { + assert_eq!(checkpoint_interval_for_build_time(build_time), interval, "build time: {build_time:?}"); + } +} + +#[tokio::test] +async fn periodic_checkpoint_build_cost_backoff_preserves_deadline_and_validation() { + let (mut scanner, temp_dir) = build_checkpoint_scanner().await; + let _guard = TestGuard { + temp_dir: Some(temp_dir), + }; + let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1); + scanner.checkpoint_tx = Some(checkpoint_tx); + let folder = CachedFolder { + name: "bucket".to_string(), + parent: None, + object_heal_prob_div: 1, + }; + let hash = hash_path("bucket"); + let mut entry = scanner.new_cache.root().expect("checkpoint fixture root"); + scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL; + scanner.last_checkpoint_build_time = Duration::from_secs(3); + scanner.last_checkpoint_at = Instant::now() + .checked_sub(Duration::from_secs(10)) + .expect("backdate checkpoint attempt"); + scanner.maybe_send_checkpoint(&folder, &hash, &entry); + assert!( + checkpoint_rx.try_recv().is_err(), + "expensive checkpoints must not rebuild at the old five-second floor" + ); + assert_eq!(scanner.last_checkpoint_objects, 0, "a deferred checkpoint must not consume progress"); + assert_eq!(scanner.last_checkpoint_build_time, Duration::from_secs(3)); + + scanner.last_checkpoint_at = Instant::now() + .checked_sub(Duration::from_secs(30)) + .expect("reach cost-adjusted interval"); + scanner.maybe_send_checkpoint(&folder, &hash, &entry); + let snapshot = checkpoint_rx + .try_recv() + .expect("checkpoint is admitted at its adjusted interval"); + assert_eq!(snapshot.validated_scan_frontier(), Some("bucket/a")); + assert!(snapshot.checked_flatten_complete_scope("bucket").is_some()); + assert!(!snapshot.info.snapshot_complete); + assert!( + scanner.last_checkpoint_build_time <= scanner.last_checkpoint_at.elapsed(), + "record actual construction time" + ); + + scanner.last_checkpoint_build_time = Duration::MAX; + scanner.last_checkpoint_at = Instant::now() + .checked_sub(SCANNER_CHECKPOINT_INTERVAL) + .expect("reach periodic deadline without another object batch"); + scanner.maybe_send_checkpoint(&folder, &hash, &entry); + assert!( + checkpoint_rx.try_recv().is_ok(), + "even a costly snapshot retains the sixty-second periodic attempt" + ); + + scanner.new_cache.cache.get_mut("bucket/a").expect("fixture child").size = usize::MAX; + entry.size = 1; + scanner.last_checkpoint_build_time = Duration::from_secs(3); + scanner.last_checkpoint_at = Instant::now() + .checked_sub(SCANNER_CHECKPOINT_INTERVAL) + .expect("reach periodic overflow check"); + scanner.maybe_send_checkpoint(&folder, &hash, &entry); + assert!(checkpoint_rx.try_recv().is_err(), "backoff must not bypass overflow validation"); + assert!( + scanner.last_checkpoint_build_time <= scanner.last_checkpoint_at.elapsed(), + "failed construction must also update the cost estimate" + ); +} + +#[tokio::test] +async fn checkpoint_ancestor_depth_limit_preserves_suspended_accounting() { + let (mut scanner, temp_dir) = build_test_scanner().await; + let _guard = TestGuard { + temp_dir: Some(temp_dir), + }; + let (tx, _rx) = mpsc::channel(1); + scanner.checkpoint_tx = Some(tx); + scanner.old_cache.info.scan_progress = Some(crate::DataUsageScanProgress { + started_plan: crate::DataUsageScanPlanDigest([1; 32]), + requested_plan: crate::DataUsageScanPlanDigest([1; 32]), + }); + scanner.checkpoint_depth = MAX_DATA_USAGE_CACHE_DEPTH; + scanner.checkpoint_ancestors = (0..MAX_DATA_USAGE_CACHE_DEPTH) + .map(|_| ScannerCheckpointAncestor::default()) + .collect(); + let mut hash = hash_path("bucket/parent"); + let mut parent = DataUsageEntry { + objects: 3, + size: 7, + ..Default::default() + }; + let mut child = DataUsageEntry::default(); + let error = scanner + .scan_child_folder( + CancellationToken::new(), + CachedFolder { + name: "bucket/parent/child".to_string(), + parent: Some(hash.clone()), + object_heal_prob_div: 1, + }, + &mut hash, + &mut parent, + &mut child, + ) + .await + .expect_err("unsupported depth must stop before recursion"); + assert!(matches!(error, ScannerError::Other(message) if message.contains("depth limit"))); + assert_eq!(hash, hash_path("bucket/parent")); + assert_eq!((parent.objects, parent.size), (3, 7)); + assert_eq!(scanner.checkpoint_depth, MAX_DATA_USAGE_CACHE_DEPTH); +} + struct TestGuard { temp_dir: Option, } @@ -1658,6 +1822,51 @@ fn metadata_for_object_version(bucket: &str, object: &str, version_id: Option partial, + other => panic!("missing namespace must remain incomplete, got {other:?}"), + }; + assert!(!partial.info.snapshot_complete); + assert_eq!(partial.info.next_cycle, 9); + assert_eq!(partial.cache.len(), 1); + assert_eq!(partial.cache.keys().next().expect("retained orphan key").as_ptr(), retained_key_ptr); + assert!( + partial.root().is_none(), + "a disconnected durable cache must not gain an invented zero root" + ); + assert!(partial.checked_flatten_complete_scope("missing-bucket").is_none()); + assert_eq!((partial.cache[&orphan.0].objects, partial.cache[&orphan.0].size), (7, 70)); +} + #[tokio::test] #[serial] async fn test_scan_data_folder_resume_hint_orders_across_new_and_existing_folders() { @@ -3621,6 +3939,15 @@ async fn test_scan_data_folder_success_clears_resume_hint() { #[tokio::test] #[serial] async fn test_scan_data_folder_keeps_unresolved_objects_partial() { + let logs = CapturedLogs::default(); + let subscriber = tracing_subscriber::fmt() + .json() + .with_max_level(tracing::Level::DEBUG) + .with_writer(logs.clone()) + .with_ansi(false) + .without_time() + .finish(); + let _subscriber_guard = tracing::subscriber::set_default(subscriber); let (scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard { temp_dir: Some(temp_dir.clone()), @@ -3671,6 +3998,18 @@ async fn test_scan_data_folder_keeps_unresolved_objects_partial() { "corrupt object failure should remain recorded: {:?}", partial.info.failed_objects ); + let events = logs + .contents() + .lines() + .map(|line| serde_json::from_str::(line).expect("scanner JSON event")) + .filter(|event| event["fields"]["state"] == "bucket_sweep_incomplete") + .collect::>(); + assert_eq!(events.len(), 1, "the incomplete sweep must identify its blocker"); + let fields = &events[0]["fields"]; + assert_eq!(fields["bucket"], "bucket"); + assert_eq!(fields["failed_objects"], 1); + assert_eq!(fields["mixed_coverage"], false); + assert!(!events[0].to_string().contains(&failed_path), "do not log individual object paths"); } #[tokio::test] diff --git a/crates/scanner/src/scanner_folder/tests/checkpoint_fixture.rs b/crates/scanner/src/scanner_folder/tests/checkpoint_fixture.rs index 14fc3c425..e1645fd55 100644 --- a/crates/scanner/src/scanner_folder/tests/checkpoint_fixture.rs +++ b/crates/scanner/src/scanner_folder/tests/checkpoint_fixture.rs @@ -21,6 +21,7 @@ use std::io::Cursor; use tokio::io::AsyncReadExt; mod deep_compacted; +mod periodic_snapshot; mod segment_observation; const CACHE_NAME: &str = "bucket/checkpoint-fixture.bin"; @@ -885,6 +886,25 @@ async fn checkpoint_fixture_hot_digest_retains_partial_progress() { #[tokio::test] #[serial] async fn checkpoint_fixture_hot_bucket_certifies_first_full_sweep_after_mixed_end() { + hot_bucket_certifies_first_full_sweep_after_mixed_end(false).await; +} + +#[tokio::test] +#[serial] +async fn checkpoint_fixture_hot_bucket_certifies_after_mixed_end_and_leader_handoff() { + hot_bucket_certifies_first_full_sweep_after_mixed_end(true).await; +} + +async fn hot_bucket_certifies_first_full_sweep_after_mixed_end(advance_epoch: bool) { + let logs = CapturedLogs::default(); + let subscriber = tracing_subscriber::fmt() + .json() + .with_max_level(tracing::Level::DEBUG) + .with_writer(logs.clone()) + .with_ansi(false) + .without_time() + .finish(); + let _subscriber_guard = tracing::subscriber::set_default(subscriber); let (scanner, root) = build_test_scanner().await; let _guard = TestGuard { temp_dir: Some(root.clone()), @@ -905,6 +925,8 @@ async fn checkpoint_fixture_hot_bucket_certifies_first_full_sweep_after_mixed_en // tail that ends mixed, then fresh unbounded sweeps under new plans. let rounds = [(Some(4), false), (None, false), (None, true), (None, true)]; for (round, (max_objects, expect_complete)) in (0_u64..).zip(rounds) { + let cycle = 11 + round; + let leader_epoch = if advance_epoch { 7 + round } else { 7 }; write_checkpoint_object(&root, "hot/current", &[(None, 1)]).await; let plan = crate::scanner_io::checkpoint_fixture_bucket_digest(PLAN, Some(round)); let mut cache = DataUsageCache::default(); @@ -916,8 +938,8 @@ async fn checkpoint_fixture_hot_bucket_certifies_first_full_sweep_after_mixed_en &mut cache, "bucket", SOURCE, - 11, - 7, + cycle, + leader_epoch, plan, crate::scanner_io::DataUsageCacheReuseOptions { require_source: true, @@ -960,6 +982,18 @@ async fn checkpoint_fixture_hot_bucket_certifies_first_full_sweep_after_mixed_en let saved = store.strict_load().await; assert_eq!(complete, expect_complete, "round {round} outcome"); assert_eq!(saved.info.snapshot_complete, expect_complete, "round {round} saved completion"); + assert_eq!(saved.info.next_cycle, cycle); + assert_eq!(saved.info.leader_epoch, leader_epoch); + if round == 1 { + assert!(saved.info.scan_resume_after.is_none()); + assert!(saved.info.scan_checkpoint.is_none()); + assert!(saved.info.scan_raw_enumeration_cursor.is_none()); + assert!(saved.info.scan_raw_enumeration_page_index.is_none()); + assert!(saved.info.scan_coverage_receipt.is_none()); + let progress = saved.info.scan_progress.expect("mixed tail retains a verification plan"); + assert_eq!(progress.started_plan, plan); + assert_eq!(progress.requested_plan, plan); + } if expect_complete { assert_eq!(saved.info.scan_plan_digest, Some(plan), "round {round} certifies its own plan"); assert!(saved.info.scan_progress.is_none()); @@ -969,6 +1003,53 @@ async fn checkpoint_fixture_hot_bucket_certifies_first_full_sweep_after_mixed_en assert!(saved.info.scan_plan_digest.is_none(), "round {round} must not certify"); } } + let captured = logs.contents(); + let all_events = captured + .lines() + .map(|line| serde_json::from_str::(line).expect("scanner JSON event")) + .collect::>(); + let admissions = all_events + .iter() + .filter(|event| event["fields"]["state"] == "bucket_checkpoint_prepared") + .collect::>(); + assert_eq!(admissions.len(), 4, "every sweep must identify fresh versus resumed admission"); + for (round, event) in admissions.iter().enumerate() { + let fields = &event["fields"]; + let round_cycle = 11 + u64::try_from(round).expect("round fits u64"); + assert_eq!(fields["cycle"], round_cycle); + assert_eq!(fields["leader_epoch"], if advance_epoch { round_cycle - 4 } else { 7 }); + assert_eq!(fields["previous_cycle"], if round == 0 { 0 } else { round_cycle - 1 }); + assert_eq!( + fields["previous_leader_epoch"], + if round == 0 { + 0 + } else if advance_epoch { + round_cycle - 5 + } else { + 7 + } + ); + assert_eq!(fields["resume_position"], round == 1); + assert_eq!(fields["mixed_coverage"], round == 1); + } + let events = all_events + .iter() + .filter(|event| event["fields"]["state"] == "bucket_sweep_incomplete") + .collect::>(); + assert_eq!(events.len(), 1, "only the mixed tail needs a verification-sweep diagnostic"); + let fields = &events[0]["fields"]; + assert_eq!(fields["bucket"], "bucket"); + assert_eq!(fields["pool"], 0); + assert_eq!(fields["set"], 0); + assert_eq!(fields["cycle"], 12); + assert_eq!(fields["leader_epoch"], if advance_epoch { 8 } else { 7 }); + assert_eq!(fields["resume_position_on_entry"], true); + assert_eq!(fields["resume_position"], false); + assert_eq!(fields["mixed_coverage"], true); + assert_eq!(fields["unresolved_objects"], false); + assert_eq!(fields["coverage_gap"], false); + assert_eq!(fields["failed_objects"], 0); + assert_eq!(fields["size_reconciliation_entries"], 0); } async fn write_checkpoint_object(root: &std::path::Path, object: &str, versions: &[(Option, i64)]) { diff --git a/crates/scanner/src/scanner_folder/tests/checkpoint_fixture/periodic_snapshot.rs b/crates/scanner/src/scanner_folder/tests/checkpoint_fixture/periodic_snapshot.rs new file mode 100644 index 000000000..93355f7d2 --- /dev/null +++ b/crates/scanner/src/scanner_folder/tests/checkpoint_fixture/periodic_snapshot.rs @@ -0,0 +1,1104 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Periodic snapshots captured while nested traversal frames are still active. + +use super::*; +use tokio::sync::Notify; + +const OBJECTS: u64 = 8; + +fn prepare_scanner(scanner: &mut FolderScanner, cache: DataUsageCache) { + scanner.resume_frontier = cache.validated_scan_frontier().map(str::to_owned); + scanner.coverage_frontier = scanner.resume_frontier.clone(); + scanner.resume_failed_paths = failed_scan_paths(&cache); + scanner.new_cache = DataUsageCache { + info: cache.info.clone(), + ..Default::default() + }; + scanner.update_cache = DataUsageCache { + info: cache.info.clone(), + ..Default::default() + }; + scanner.old_cache = cache; + scanner.is_erasure_mode = true; + scanner.skip_heal.store(true, Ordering::SeqCst); + scanner.sleeper = DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest); +} + +async fn nested_periodic_snapshot( + corrupt_first: bool, + root_scalars: (usize, usize), +) -> (DataUsageCache, crate::DataUsageScanIdentity, FolderScanner, TestGuard) { + let (mut scanner, root) = build_test_scanner().await; + let guard = TestGuard { + temp_dir: Some(root.clone()), + }; + for branch in ["a", "b"] { + for object in 0..4 { + write_checkpoint_object(&root, &format!("prefix/{branch}/{object:04}"), &[(None, 1)]).await; + } + } + if corrupt_first { + write_test_object_metadata_bytes(&root, "bucket", "prefix/a/0000", b"not-valid-filemeta").await; + } + let identity = crate::DataUsageScanIdentity { + tier_registry_generation: crate::runtime_tier_registry_for_cycle(11, 7).await.generation, + ..bound_checkpoint().1 + }; + let mut cache = DataUsageCache::default(); + cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity); + prepare_scanner(&mut scanner, cache); + let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1); + scanner.checkpoint_tx = Some(checkpoint_tx); + scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL - 2; + scanner.last_checkpoint_at = Instant::now() + .checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL) + .expect("seed elapsed checkpoint interval"); + let paused = Arc::new(Notify::new()); + let release = Arc::new(Notify::new()); + let pause_path = if corrupt_first { + "bucket/prefix/a/0003" + } else { + "bucket/prefix/a/0002" + }; + scanner.update_current_path = Arc::new({ + let paused = paused.clone(); + let release = release.clone(); + move |path: &str| { + let pause_here = path == pause_path; + let paused = paused.clone(); + let release = release.clone(); + Box::pin(async move { + if pause_here { + paused.notify_one(); + release.notified().await; + } + }) + } + }); + let parent = CancellationToken::new(); + scanner.budget = ScannerCycleBudget::new_with_progress_tracking(&parent, Default::default()); + let budget = scanner.budget.clone(); + let mut entry = DataUsageEntry { + objects: root_scalars.0, + size: root_scalars.1, + ..Default::default() + }; + let snapshot = { + let scan = scanner.scan_folder( + parent.clone(), + CachedFolder { + name: "bucket".to_string(), + parent: None, + object_heal_prob_div: 1, + }, + &mut entry, + ); + tokio::pin!(scan); + let snapshot = tokio::time::timeout(Duration::from_secs(10), async { + tokio::select! { + snapshot = checkpoint_rx.recv() => snapshot.expect("real walk emits periodic snapshot"), + result = &mut scan => panic!("walk finished before periodic snapshot: {result:?}"), + } + }) + .await + .expect("bounded nested checkpoint emission"); + tokio::time::timeout(Duration::from_secs(10), async { + tokio::select! { + () = paused.notified() => {}, + result = &mut scan => panic!("nested ancestors unwound before capture: {result:?}"), + } + }) + .await + .expect("walk pauses before third healthy object"); + assert_eq!(budget.progress().0, 2, "capture must precede ancestor unwind and further object reads"); + assert!(!parent.is_cancelled(), "periodic capture must not use cancellation linking"); + snapshot + }; + assert!(!snapshot.info.snapshot_complete); + assert!( + snapshot.validated_raw_enumeration_cursor().is_some() || snapshot.validated_raw_enumeration_page_index().is_some(), + "enumeration progress must coexist with completed subtree coverage" + ); + let totals = snapshot + .checked_flatten_complete_scope("bucket") + .expect("periodic snapshot must connect every active ancestor to the bucket root"); + assert_eq!( + totals.objects, + 2 + root_scalars.0, + "snapshot must retain suspended ancestor and current frame object accounting" + ); + assert_eq!( + totals.size, + 2 + root_scalars.1, + "suspended ancestor and current frame scalar bytes must survive snapshot construction" + ); + let store = FixtureStore::new(); + let revisions = DataUsageCache::default() + .load_with_revisions(store.clone(), CACHE_NAME) + .await + .expect("initial periodic fixture revisions"); + snapshot + .save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0) + .await + .expect("persist real mid-walk periodic snapshot"); + let mut loaded = DataUsageCache::default(); + loaded + .load(store.clone(), CACHE_NAME) + .await + .expect("reload real periodic snapshot"); + assert_eq!(loaded.info.scan_coverage_receipt, snapshot.info.scan_coverage_receipt); + assert_eq!(loaded.info.scan_raw_enumeration_cursor, snapshot.info.scan_raw_enumeration_cursor); + assert_eq!( + loaded + .checked_flatten_complete_scope("bucket") + .expect("reloaded complete scope") + .objects, + 2 + root_scalars.0 + ); + assert_eq!( + loaded.prepare_bucket_checkpoint("bucket", 11, 8, SOURCE, PLAN, identity), + crate::DataUsageCachePrepareOutcome::Reused, + "a higher leader must adopt the persisted periodic snapshot" + ); + (loaded, identity, scanner, guard) +} + +async fn finish_resumed_walk(cache: DataUsageCache, original: &FolderScanner) -> (DataUsageCache, u64, TestGuard) { + let (mut scanner, unused_root) = build_test_scanner().await; + let guard = TestGuard { + temp_dir: Some(unused_root), + }; + scanner.root = original.root.clone(); + scanner.local_disk = original.local_disk.clone(); + prepare_scanner(&mut scanner, cache); + scanner.failed_object_ttl_secs = original.failed_object_ttl_secs; + scanner.failed_objects_max = original.failed_objects_max; + let parent = CancellationToken::new(); + scanner.budget = ScannerCycleBudget::new_with_progress_tracking(&parent, Default::default()); + let mut entry = DataUsageEntry::default(); + scanner + .scan_folder( + parent, + CachedFolder { + name: "bucket".to_string(), + parent: None, + object_heal_prob_div: 1, + }, + &mut entry, + ) + .await + .expect("finish resumed nested periodic walk"); + let visits = scanner.budget.progress().0; + (scanner.new_cache, visits, guard) +} + +#[tokio::test] +#[serial] +async fn periodic_mixed_tail_requires_terminal_persistence_before_fresh_certification() { + let (periodic, identity, scanner, _guard) = nested_periodic_snapshot(false, (0, 0)).await; + assert!(periodic.has_scan_position(), "real periodic snapshot retains traversal position"); + let store = FixtureStore::new(); + let revisions = DataUsageCache::default() + .load_with_revisions(store.clone(), CACHE_NAME) + .await + .expect("initial periodic fixture revisions"); + periodic + .save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0) + .await + .expect("seed durable periodic snapshot"); + + for round in 0..3 { + let cycle = 12 + round; + let leader_epoch = 9 + round; + let plan = crate::scanner_io::checkpoint_fixture_bucket_digest(PLAN, Some(cycle)); + write_checkpoint_object(std::path::Path::new(&scanner.root), "prefix/b/0003", &[(None, 1)]).await; + let mut cache = DataUsageCache::default(); + let revisions = cache + .load_with_revisions(store.clone(), CACHE_NAME) + .await + .expect("reload last successful save"); + cache.prepare_bucket_checkpoint("bucket", cycle, leader_epoch, SOURCE, plan, identity); + assert_eq!( + cache.has_scan_position(), + round < 2, + "only a saved terminal clears the inherited position" + ); + assert_eq!( + cache + .info + .scan_progress + .is_some_and(|progress| progress.started_plan != progress.requested_plan), + round < 2, + "a surviving periodic prefix must remain mixed under a changed plan" + ); + cache.info.skip_healing = true; + let parent = CancellationToken::new(); + let budget = ScannerCycleBudget::new_with_progress_tracking(&parent, Default::default()); + let outcome = scanner + .local_disk + .clone() + .nsscanner_disk( + budget.token(), + budget.clone(), + vec![scanner.local_disk.clone()], + cache, + None, + scan_options(HealScanMode::Normal), + ) + .await + .expect("finish the periodic tail or fresh verification sweep"); + let (terminal, complete) = match outcome { + ScannerDiskScanOutcome::Partial(cache) => (cache, false), + ScannerDiskScanOutcome::Complete(cache) => (cache, true), + ScannerDiskScanOutcome::NamespaceNotFound(_) => panic!("fixture namespace exists"), + }; + assert!(!budget.budget_elapsed()); + assert_eq!(complete, round == 2); + assert!(!terminal.has_scan_position(), "completed traversal clears every position before saving"); + let totals = terminal.checked_flatten("bucket").expect("connected terminal root"); + assert_eq!(totals.objects, usize::try_from(OBJECTS).expect("fixture count")); + assert_eq!(totals.size, usize::try_from(OBJECTS).expect("fixture bytes")); + + store.reject_save.store(round == 0, Ordering::SeqCst); + let saved = terminal + .save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0) + .await; + if round == 0 { + assert!(saved.is_err(), "injected terminal persistence failure must be observable"); + let retained = store.strict_load().await; + assert_eq!(retained.info.scan_coverage_receipt, periodic.info.scan_coverage_receipt); + assert_eq!(retained.info.scan_checkpoint, periodic.info.scan_checkpoint); + assert!( + retained.has_scan_position(), + "failed terminal save must leave periodic recovery state intact" + ); + } else { + saved.expect("persist terminal sweep"); + let retained = store.strict_load().await; + assert!(!retained.has_scan_position()); + assert_eq!(retained.info.snapshot_complete, round == 2); + } + store.reject_save.store(false, Ordering::SeqCst); + } +} + +#[tokio::test] +#[serial] +async fn periodic_nested_snapshot_resumes_after_leader_handoff_with_exact_totals() { + let (loaded, _, original, _guard) = nested_periodic_snapshot(false, (0, 0)).await; + assert_eq!(loaded.validated_scan_frontier(), Some("bucket/prefix/a/0000")); + let (finished, visits, _unused_guard) = finish_resumed_walk(loaded, &original).await; + assert_eq!(visits, OBJECTS - 1, "the completed nested sibling must skip its metadata read"); + let totals = finished + .checked_flatten_complete_scope("bucket") + .expect("finished connected root"); + assert_eq!(u64::try_from(totals.objects).expect("object count"), OBJECTS); + assert_eq!(u64::try_from(totals.size).expect("byte count"), OBJECTS); + assert_eq!(totals.failed_objects, 0); +} + +#[tokio::test] +#[serial] +async fn periodic_nested_snapshot_resumes_healthy_progress_after_a_failed_object() { + let (loaded, _, original, _guard) = nested_periodic_snapshot(true, (0, 0)).await; + assert_eq!( + loaded.validated_scan_frontier(), + Some("bucket/prefix/a/0001"), + "object health debt must not freeze completed traversal coverage" + ); + assert!(!loaded.info.failed_objects.is_empty(), "persisted failure debt must survive handoff"); + let (finished, visits, _unused_guard) = finish_resumed_walk(loaded, &original).await; + assert_eq!(visits, OBJECTS - 2, "the healthy covered sibling must skip its metadata read"); + let totals = finished + .checked_flatten_complete_scope("bucket") + .expect("failed sweep still has connected accounting"); + assert_eq!(u64::try_from(totals.objects).expect("healthy count"), OBJECTS - 1); + assert_eq!(u64::try_from(totals.size).expect("healthy bytes"), OBJECTS - 1); + assert_eq!(totals.failed_objects, 1, "the retried failed subtree must retain its debt"); + assert!(!finished.info.failed_objects.is_empty(), "unrepaired metadata must remain pending"); +} + +#[tokio::test] +#[serial] +async fn periodic_nested_snapshot_preserves_suspended_ancestor_scalars_through_reload() { + let (loaded, _, _, _guard) = nested_periodic_snapshot(false, (3, 21)).await; + let root = loaded.root().expect("reloaded root accumulator"); + assert_eq!(root.objects, 3, "suspended root object accounting must not become an empty topology node"); + assert_eq!(root.size, 21, "suspended root bytes must remain independent of descendant totals"); + let totals = loaded + .checked_flatten_complete_scope("bucket") + .expect("reloaded ancestor accounting"); + assert_eq!(totals.objects, 5); + assert_eq!(totals.size, 23); + assert_eq!(loaded.validated_scan_frontier(), Some("bucket/prefix/a/0000")); +} + +#[tokio::test] +#[serial] +async fn periodic_nested_snapshot_keeps_durable_frontier_across_early_repeated_handoffs() { + let (cache, identity, original, _guard) = nested_periodic_snapshot(false, (0, 0)).await; + let store = FixtureStore::new(); + let revisions = DataUsageCache::default() + .load_with_revisions(store.clone(), CACHE_NAME) + .await + .expect("initial repeated-handoff revisions"); + cache + .save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0) + .await + .expect("save the inherited valid subtree frontier"); + let (mut scanner, unused_root) = build_test_scanner().await; + let _unused_guard = TestGuard { + temp_dir: Some(unused_root), + }; + scanner.root = original.root.clone(); + scanner.local_disk = original.local_disk.clone(); + prepare_scanner(&mut scanner, cache); + let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1); + scanner.checkpoint_tx = Some(checkpoint_tx); + scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL; + scanner.last_checkpoint_at = Instant::now() + .checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL) + .expect("early resumed emitter cadence"); + let folder = CachedFolder { + name: "bucket".to_string(), + parent: None, + object_heal_prob_div: 1, + }; + scanner.record_raw_enumeration_entry("bucket", "prefix"); + scanner.maybe_send_checkpoint(&folder, &hash_path("bucket"), &DataUsageEntry::default()); + assert!( + matches!(checkpoint_rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)), + "an early raw-only snapshot must not overwrite inherited completed coverage" + ); + assert_eq!(scanner.last_checkpoint_objects, SCANNER_CHECKPOINT_OBJECT_INTERVAL); + let mut inherited = DataUsageCache::default(); + let revisions = inherited + .load_with_revisions(store.clone(), CACHE_NAME) + .await + .expect("reload unchanged durable frontier after deferred early emission"); + assert_eq!(inherited.validated_scan_frontier(), Some("bucket/prefix/a/0000")); + assert_eq!( + inherited.prepare_bucket_checkpoint("bucket", 11, 9, SOURCE, PLAN, identity), + crate::DataUsageCachePrepareOutcome::Reused + ); + assert_eq!(inherited.validated_scan_frontier(), Some("bucket/prefix/a/0000")); + prepare_scanner(&mut scanner, inherited); + scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL - 2; + scanner.last_checkpoint_objects = 0; + scanner.last_checkpoint_at = Instant::now() + .checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL) + .expect("later useful resumed emitter cadence"); + let parent = CancellationToken::new(); + scanner.budget = ScannerCycleBudget::new_with_progress_tracking(&parent, Default::default()); + let mut entry = DataUsageEntry::default(); + let candidate = { + let scan = scanner.scan_folder(parent, folder, &mut entry); + tokio::pin!(scan); + tokio::time::timeout(Duration::from_secs(10), async { + tokio::select! { + checkpoint = checkpoint_rx.recv() => checkpoint.expect("resumed walk can later emit useful coverage"), + result = &mut scan => { + result.expect("later resumed walk completes"); + checkpoint_rx.try_recv().expect("completed walk emitted a useful periodic checkpoint") + } + } + }) + .await + .expect("bounded later useful emission") + }; + assert_eq!(candidate.validated_scan_frontier(), Some("bucket/prefix/a/0001")); + assert_eq!( + candidate + .checked_flatten_complete_scope("bucket") + .expect("later snapshot connected accounting") + .objects, + 3, + "inherited completed object and both newly read objects must survive" + ); + candidate + .save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0) + .await + .expect("persist later coverage without losing the inherited prefix"); + let mut latest = DataUsageCache::default(); + latest + .load(store, CACHE_NAME) + .await + .expect("reload later periodic checkpoint"); + assert_eq!( + latest.prepare_bucket_checkpoint("bucket", 11, 10, SOURCE, PLAN, identity), + crate::DataUsageCachePrepareOutcome::Reused + ); + assert_eq!(latest.validated_scan_frontier(), Some("bucket/prefix/a/0001")); +} + +#[tokio::test] +#[serial] +async fn periodic_nested_snapshot_defers_unfinished_paths_at_or_before_completed_frontier() { + let (mut scanner, root) = build_test_scanner().await; + let _guard = TestGuard { temp_dir: Some(root) }; + let (_, identity) = bound_checkpoint(); + let mut cache = DataUsageCache::default(); + cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity); + cache.replace("bucket", "", DataUsageEntry::default()); + cache.replace( + "bucket/z", + "bucket", + DataUsageEntry { + objects: 3, + size: 9, + ..Default::default() + }, + ); + cache + .seal_scan_frontier(Some("bucket/z")) + .expect("completed later sibling coverage"); + prepare_scanner(&mut scanner, cache.clone()); + scanner.new_cache = cache; + let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1); + scanner.checkpoint_tx = Some(checkpoint_tx); + let partial = DataUsageEntry { + objects: 2, + size: 5, + ..Default::default() + }; + for active in ["bucket/a", "bucket/z"] { + scanner.record_raw_enumeration_entry(active, STORAGE_FORMAT_FILE); + scanner.checkpoint_objects += SCANNER_CHECKPOINT_OBJECT_INTERVAL; + scanner.last_checkpoint_at = Instant::now() + .checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL) + .expect("unordered active path cadence"); + scanner.maybe_send_checkpoint( + &CachedFolder { + name: active.to_string(), + parent: Some(hash_path("bucket")), + object_heal_prob_div: 1, + }, + &hash_path(active), + &partial, + ); + assert!( + matches!(checkpoint_rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)), + "unfinished path {active} must not become certified completed coverage" + ); + assert_eq!( + scanner.last_checkpoint_objects, scanner.checkpoint_objects, + "deferred paths still consume cadence" + ); + } + scanner.new_cache.replace("bucket/a", "bucket", partial); + let root_entry = scanner.new_cache.root().expect("completed subtree owner"); + scanner.checkpoint_objects += SCANNER_CHECKPOINT_OBJECT_INTERVAL; + scanner.last_checkpoint_at = Instant::now() + .checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL) + .expect("completed owner cadence"); + scanner.maybe_send_checkpoint( + &CachedFolder { + name: "bucket".to_string(), + parent: None, + object_heal_prob_div: 1, + }, + &hash_path("bucket"), + &root_entry, + ); + let checkpoint = checkpoint_rx.try_recv().expect("completed subtree owner may safely emit"); + assert_eq!(checkpoint.validated_scan_frontier(), Some("bucket/z")); + let totals = checkpoint + .checked_flatten_complete_scope("bucket") + .expect("both completed sibling links remain reachable"); + assert_eq!(totals.objects, 5); + assert_eq!(totals.size, 14); + assert_eq!(checkpoint.cache.len(), 3); +} + +#[tokio::test] +#[serial] +async fn periodic_nested_snapshot_checks_suspended_ancestors_before_certifying_frontier() { + for (ancestor, frontier, may_emit) in [ + ("bucket/z", "bucket/z", false), + ("bucket/a", "bucket/a-", false), + ("bucket/a", "bucket/a/done", true), + ] { + let (mut scanner, root) = build_test_scanner().await; + let _guard = TestGuard { temp_dir: Some(root) }; + let (_, identity) = bound_checkpoint(); + let mut cache = DataUsageCache::default(); + cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity); + cache.replace("bucket", "", DataUsageEntry::default()); + cache.replace(ancestor, "bucket", DataUsageEntry::default()); + let completed = format!("{ancestor}/done"); + cache.replace( + &completed, + ancestor, + DataUsageEntry { + objects: 3, + size: 9, + ..Default::default() + }, + ); + let sibling_frontier = frontier != ancestor && frontier != completed; + if sibling_frontier { + cache.replace( + frontier, + "bucket", + DataUsageEntry { + objects: 3, + size: 9, + ..Default::default() + }, + ); + } + cache.seal_scan_frontier(Some(frontier)).expect("initial completed coverage"); + prepare_scanner(&mut scanner, cache); + scanner.new_cache.replace("bucket", "", DataUsageEntry::default()); + scanner + .new_cache + .copy_with_children(&scanner.old_cache, &hash_path(&completed), &Some(hash_path(ancestor))); + if sibling_frontier { + scanner + .new_cache + .copy_with_children(&scanner.old_cache, &hash_path(frontier), &Some(hash_path("bucket"))); + } + let root_entry = scanner.new_cache.root().expect("suspended root owner").clone(); + let mut ancestor_entry = scanner.new_cache.find(ancestor).expect("suspended subtree owner").clone(); + ancestor_entry.objects = 1; + ancestor_entry.size = 2; + scanner.checkpoint_ancestors = vec![ + ScannerCheckpointAncestor { + hash: hash_path("bucket"), + entry: root_entry, + }, + ScannerCheckpointAncestor { + hash: hash_path(ancestor), + entry: ancestor_entry, + }, + ]; + scanner.checkpoint_depth = 2; + let current = format!("{ancestor}/next"); + assert!( + current.as_str() > frontier, + "the current-folder-only guard must not already reject this fixture" + ); + scanner.record_raw_enumeration_entry(¤t, STORAGE_FORMAT_FILE); + let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1); + scanner.checkpoint_tx = Some(checkpoint_tx); + scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL; + scanner.last_checkpoint_at = Instant::now() + .checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL) + .expect("suspended ancestor emitter cadence"); + scanner.maybe_send_checkpoint( + &CachedFolder { + name: current.clone(), + parent: Some(hash_path(ancestor)), + object_heal_prob_div: 1, + }, + &hash_path(¤t), + &DataUsageEntry { + objects: 2, + size: 5, + ..Default::default() + }, + ); + assert_eq!(scanner.last_checkpoint_objects, SCANNER_CHECKPOINT_OBJECT_INTERVAL); + if may_emit { + let snapshot = checkpoint_rx + .try_recv() + .expect("an ancestor containing the frontier may safely emit"); + assert_eq!(snapshot.validated_scan_frontier(), Some(frontier)); + let totals = snapshot + .checked_flatten_complete_scope("bucket") + .expect("all suspended and current accounting connected"); + assert_eq!(totals.objects, 6); + assert_eq!(totals.size, 16); + assert_eq!(snapshot.cache.len(), 4); + } else { + assert!( + matches!(checkpoint_rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)), + "unfinished ancestor {ancestor} must not be certified through {frontier}" + ); + } + } +} + +#[tokio::test] +#[serial] +async fn periodic_nested_snapshot_partial_preservation_invalidates_covered_child_but_keeps_inner_frontier() { + for (child, frontier, frontier_inside_child, expected_objects) in [ + ("bucket/a", "bucket/z", false, 5), + ("bucket/z", "bucket/z", false, 2), + ("bucket/parent", "bucket/parent/done", true, 5), + ] { + let (mut scanner, root) = build_test_scanner().await; + let _guard = TestGuard { temp_dir: Some(root) }; + let (_, identity) = bound_checkpoint(); + let mut cache = DataUsageCache::default(); + cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity); + cache.replace("bucket", "", DataUsageEntry::default()); + if frontier_inside_child { + cache.replace(child, "bucket", DataUsageEntry::default()); + } + cache.replace( + frontier, + if frontier_inside_child { child } else { "bucket" }, + DataUsageEntry { + objects: 3, + size: 9, + ..Default::default() + }, + ); + cache + .seal_scan_frontier(Some(frontier)) + .expect("initial completed child coverage"); + prepare_scanner(&mut scanner, cache.clone()); + scanner.new_cache = cache; + let mut parent_entry = scanner.new_cache.root().expect("partial child owner"); + let child_entry = DataUsageEntry { + objects: 2, + size: 5, + ..Default::default() + }; + scanner + .preserve_partial_child_progress(&Some(hash_path("bucket")), &hash_path(child), &mut parent_entry, &child_entry) + .await; + let retained_frontier = scanner.coverage_frontier.clone(); + scanner.new_cache.replace_hashed(&hash_path("bucket"), &None, &parent_entry); + scanner + .new_cache + .seal_scan_frontier(retained_frontier.as_deref()) + .expect("seal the final partial cache after child preservation"); + let totals = scanner + .new_cache + .checked_flatten_complete_scope("bucket") + .expect("preserved partial child and completed siblings remain connected"); + assert_eq!(totals.objects, expected_objects, "partial child {child} must be counted once"); + assert_eq!(totals.size, if expected_objects == 2 { 5 } else { 14 }); + if frontier_inside_child { + assert!(!scanner.coverage_gap); + assert_eq!(scanner.new_cache.validated_scan_frontier(), Some(frontier)); + } else { + assert!( + scanner.coverage_gap, + "unfinished covered child {child} must block further frontier advancement" + ); + assert!(retained_frontier.is_none()); + assert!(scanner.new_cache.info.scan_coverage_receipt.is_none()); + assert!(scanner.new_cache.validated_scan_frontier().is_none()); + } + } +} + +async fn assert_nested_frame_restoration(max_objects: Option) { + let (mut scanner, root) = build_test_scanner().await; + let _guard = TestGuard { + temp_dir: Some(root.clone()), + }; + for branch in ["a", "b"] { + for object in 0..4 { + write_checkpoint_object(&root, &format!("prefix/{branch}/{object:04}"), &[(None, 1)]).await; + } + } + let (_, identity) = bound_checkpoint(); + let mut cache = DataUsageCache::default(); + cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity); + prepare_scanner(&mut scanner, cache); + let (checkpoint_tx, _checkpoint_rx) = mpsc::channel(1); + scanner.checkpoint_tx = Some(checkpoint_tx); + let parent = CancellationToken::new(); + scanner.budget = ScannerCycleBudget::new_with_progress_tracking( + &parent, + ScannerCycleBudgetConfig { + max_objects, + ..Default::default() + }, + ); + let mut entry = DataUsageEntry { + objects: usize::from(max_objects.is_some()) * 3, + size: usize::from(max_objects.is_some()) * 21, + ..Default::default() + }; + let result = scanner + .scan_folder( + scanner.budget.token(), + CachedFolder { + name: "bucket".to_string(), + parent: None, + object_heal_prob_div: 1, + }, + &mut entry, + ) + .await; + match max_objects { + Some(_) => { + assert!(result.is_err(), "object budget must interrupt the actual nested walk"); + assert_eq!(scanner.budget.reason(), Some(crate::scanner_budget::ScannerCycleBudgetReason::Objects)); + assert!( + scanner.budget.token().is_cancelled(), + "partial preservation must follow actual cancellation" + ); + assert_eq!( + entry.objects, 3, + "suspended root object accounting must be restored before returning the error" + ); + assert_eq!( + entry.size, 21, + "suspended root byte accounting must be restored before cancellation linking" + ); + } + None => result.expect("complete real nested walk with periodic sender enabled"), + } + let totals = scanner + .new_cache + .checked_flatten_complete_scope("bucket") + .expect("complete nested accounting"); + let expected = max_objects.unwrap_or(OBJECTS); + assert_eq!(u64::try_from(totals.objects).expect("object count"), expected); + assert_eq!(u64::try_from(totals.size).expect("byte count"), expected); + assert_eq!(scanner.checkpoint_depth, 0, "every completed child must restore its parent frame"); + assert_eq!( + scanner.checkpoint_ancestors.len(), + 3, + "sibling count must not increase depth-bounded spare storage" + ); + for spare in &scanner.checkpoint_ancestors { + assert!(spare.hash.0.is_empty(), "reusable frame must release its suspended path"); + assert!( + !data_usage_root_has_progress(&spare.entry), + "completed frame must not retain ancestor accounting" + ); + assert!(!spare.entry.compacted); + assert!(spare.entry.obj_sizes.is_empty()); + assert!(spare.entry.obj_versions.is_empty()); + assert!(spare.entry.tier_accounting_proof.is_none()); + } +} + +#[tokio::test] +#[serial] +async fn periodic_nested_snapshot_reuses_empty_ancestor_frames_after_complete_walk() { + assert_nested_frame_restoration(None).await; +} + +#[tokio::test] +#[serial] +async fn periodic_nested_snapshot_restores_ancestor_frames_before_cancellation_linking() { + assert_nested_frame_restoration(Some(2)).await; +} + +#[tokio::test] +#[serial] +async fn periodic_nested_snapshot_adoption_records_checkpoint_used_on_real_scan_startup() { + let (scanner, root) = build_test_scanner().await; + let _guard = TestGuard { + temp_dir: Some(root.clone()), + }; + tokio::fs::create_dir_all(root.join("bucket/static")) + .await + .expect("create the real cached subtree directory"); + let (mut cache, _) = bound_checkpoint(); + cache.info.skip_healing = true; + let parent = CancellationToken::new(); + let budget = ScannerCycleBudget::new_with_progress_tracking( + &parent, + ScannerCycleBudgetConfig { + max_directories: Some(1), + ..Default::default() + }, + ); + let before = global_metrics().report().await.scan_checkpoint_used; + let completed = scan_data_folder( + budget.token(), + budget.clone(), + vec![scanner.local_disk.clone()], + scanner.local_disk, + cache, + None, + HealScanMode::Normal, + DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest), + ) + .await + .expect("validated frontier must skip the child within a root-only directory budget"); + let after = global_metrics().report().await.scan_checkpoint_used; + assert_eq!(after, before + 1, "adopting a verified forward-sweep frontier must be observable"); + assert!(completed.info.snapshot_complete); + assert_eq!( + completed + .checked_flatten_complete_scope("bucket") + .expect("completed adopted scope") + .objects, + 3 + ); + assert_eq!(budget.progress().1, 1, "adoption must avoid entering the completed subtree"); + assert_eq!(budget.reason(), None); +} + +#[tokio::test] +#[serial] +async fn periodic_nested_snapshot_does_not_publish_shared_compacted_accumulator_under_child_path() { + let (mut scanner, root) = build_test_scanner().await; + let _guard = TestGuard { + temp_dir: Some(root.clone()), + }; + for object in 0..4 { + write_checkpoint_object(&root, &format!("prefix/nested/{object:04}"), &[(None, 1)]).await; + } + let (_, identity) = bound_checkpoint(); + let mut cache = DataUsageCache::default(); + cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity); + prepare_scanner(&mut scanner, cache); + let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1); + scanner.checkpoint_tx = Some(checkpoint_tx); + scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL - 2; + scanner.last_checkpoint_at = Instant::now() + .checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL) + .expect("seed compacted checkpoint interval"); + let mut entry = DataUsageEntry { + compacted: true, + ..Default::default() + }; + scanner + .scan_folder( + CancellationToken::new(), + CachedFolder { + name: "bucket".to_string(), + parent: None, + object_heal_prob_div: 1, + }, + &mut entry, + ) + .await + .expect("walk with shared compacted owner accumulator"); + assert_eq!(entry.objects, 4); + assert_eq!(entry.size, 4); + assert!( + entry.children.is_empty(), + "compacted totals must not acquire independently accounted child links" + ); + assert!( + matches!(checkpoint_rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)), + "a shared compacted parent accumulator must not be snapshotted as the current child" + ); +} + +#[tokio::test] +#[serial] +async fn periodic_failed_object_debt_survives_diagnostic_eviction_and_clears_after_repair() { + let (mut loaded, _, mut original, _guard) = nested_periodic_snapshot(true, (0, 0)).await; + loaded.info.failed_objects.clear(); + original.failed_object_ttl_secs = 0; + let (failed, visits, _unused_guard) = finish_resumed_walk(loaded.clone(), &original).await; + let totals = failed + .checked_flatten_complete_scope("bucket") + .expect("failed resumed accounting"); + assert_eq!(visits, OBJECTS - 2); + assert_eq!(totals.objects, usize::try_from(OBJECTS - 1).expect("healthy objects")); + assert_eq!(totals.failed_objects, 1, "uncapped entry debt must survive without diagnostic entries"); + assert!(failed.info.failed_objects.is_empty(), "TTL zero must not be needed for correctness"); + + write_checkpoint_object(std::path::Path::new(&original.root), "prefix/a/0000", &[(None, 1)]).await; + let (repaired, visits, _repair_guard) = finish_resumed_walk(loaded, &original).await; + let totals = repaired + .checked_flatten_complete_scope("bucket") + .expect("repaired resumed accounting"); + assert_eq!( + visits, + OBJECTS - 1, + "repair must reread the failed object and retain the covered healthy sibling" + ); + assert_eq!(totals.objects, usize::try_from(OBJECTS).expect("all objects")); + assert_eq!(totals.size, usize::try_from(OBJECTS).expect("all bytes")); + assert_eq!(totals.failed_objects, 0); +} + +#[tokio::test] +#[serial] +async fn periodic_failed_subtree_resumes_healthy_descendants_and_compacted_debt() { + let (loaded, identity, original, _guard) = nested_periodic_snapshot(true, (0, 0)).await; + let (mut complete_walk, _, _unused_guard) = finish_resumed_walk(loaded, &original).await; + complete_walk.info.failed_objects.clear(); + complete_walk + .seal_scan_frontier(Some("bucket/prefix/a")) + .expect("seal completed unhealthy parent"); + for compacted in [false, true] { + let mut cache = complete_walk.clone(); + if compacted { + cache.reduce_children_of(&hash_path("bucket/prefix/a"), 1, true); + assert!(cache.find("bucket/prefix/a").expect("compacted parent").compacted); + cache + .seal_scan_frontier(Some("bucket/prefix/a")) + .expect("reseal compacted accounting"); + } + let store = FixtureStore::new(); + let revisions = DataUsageCache::default() + .load_with_revisions(store.clone(), CACHE_NAME) + .await + .expect("fresh revisions"); + cache + .save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0) + .await + .expect("persist failure-bearing coverage"); + let mut adopted = DataUsageCache::default(); + adopted + .load(store, CACHE_NAME) + .await + .expect("reload failure-bearing coverage"); + assert_eq!( + adopted.prepare_bucket_checkpoint("bucket", 11, 9, SOURCE, PLAN, identity), + crate::DataUsageCachePrepareOutcome::Reused + ); + let (finished, visits, _resumed_guard) = finish_resumed_walk(adopted, &original).await; + assert_eq!( + visits, + if compacted { OBJECTS - 1 } else { 4 }, + "only healthy independently cached covered subtrees may skip reads" + ); + let totals = finished + .checked_flatten_complete_scope("bucket") + .expect("resumed failed parent accounting"); + assert_eq!(totals.objects, usize::try_from(OBJECTS - 1).expect("healthy objects")); + assert_eq!(totals.size, usize::try_from(OBJECTS - 1).expect("healthy bytes")); + assert_eq!(totals.failed_objects, 1); + } +} + +#[tokio::test] +#[serial] +async fn periodic_resume_uses_traversal_order_for_punctuation_siblings() { + let (loaded, _, original, _guard) = nested_periodic_snapshot(true, (0, 0)).await; + write_checkpoint_object(std::path::Path::new(&original.root), "prefix/a-", &[(None, 1)]).await; + let (mut cache, _, _walk_guard) = finish_resumed_walk(loaded, &original).await; + cache.info.failed_objects.clear(); + cache + .seal_scan_frontier(Some("bucket/prefix/a-")) + .expect("seal completed punctuation sibling"); + let (finished, visits, _resumed_guard) = finish_resumed_walk(cache, &original).await; + assert_eq!(visits, 4, "healthy children of the earlier failed sibling must remain covered"); + let totals = finished + .checked_flatten_complete_scope("bucket") + .expect("punctuation sibling accounting"); + assert_eq!( + totals.objects, + usize::try_from(OBJECTS).expect("healthy originals plus punctuation sibling") + ); + assert_eq!(totals.size, totals.objects); + assert_eq!(totals.failed_objects, 1); +} + +#[tokio::test] +#[serial] +async fn periodic_failed_nested_entries_cannot_certify_a_bucket_with_diagnostics_disabled() { + let logs = CapturedLogs::default(); + let subscriber = tracing_subscriber::fmt() + .json() + .with_max_level(tracing::Level::DEBUG) + .with_writer(logs.clone()) + .with_ansi(false) + .without_time() + .finish(); + let _subscriber_guard = tracing::subscriber::set_default(subscriber); + let (loaded, _, original, _guard) = nested_periodic_snapshot(true, (0, 0)).await; + let result = temp_env::async_with_vars([(ENV_FAILED_OBJECT_TTL_SECS, Some("0"))], async { + let mut cache = loaded; + cache.info.failed_objects.clear(); + cache.info.skip_healing = true; + let parent = CancellationToken::new(); + let budget = ScannerCycleBudget::new_with_progress_tracking(&parent, Default::default()); + scan_data_folder( + parent, + budget, + vec![original.local_disk.clone()], + original.local_disk.clone(), + cache, + None, + HealScanMode::Normal, + DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest), + ) + .await + }) + .await; + let Err(ScannerError::PartialCache(cache)) = result else { + panic!("nested unresolved object must keep the bucket partial"); + }; + assert!(!cache.info.snapshot_complete); + assert!(cache.info.failed_objects.is_empty()); + assert_eq!( + cache + .checked_flatten_complete_scope("bucket") + .expect("partial accounting") + .failed_objects, + 1 + ); + let events = logs + .contents() + .lines() + .map(|line| serde_json::from_str::(line).expect("scanner JSON event")) + .filter(|event| event["fields"]["state"] == "bucket_sweep_incomplete") + .collect::>(); + assert_eq!(events.len(), 1, "the completed sweep must identify its uncapped failure debt"); + let fields = &events[0]["fields"]; + assert_eq!(fields["coverage_gap"], false); + assert_eq!(fields["root_failed_objects"], 0); + assert_eq!(fields["failed_objects"], 0); + assert_eq!(fields["size_reconciliation_entries"], 0); + assert_eq!(fields["mixed_coverage"], false); + assert_eq!( + fields["unresolved_objects"], true, + "child-entry failure debt remains a certification blocker" + ); +} + +#[tokio::test] +#[serial] +async fn periodic_resume_debt_index_and_frontier_keep_subtree_boundaries() { + let (mut scanner, root) = build_test_scanner().await; + let _guard = TestGuard { temp_dir: Some(root) }; + scanner.resume_failed_paths = vec!["bucket/a-".into(), "bucket/a/z".into(), "bucket/b".into()]; + assert!( + scanner.resume_subtree_has_failures("bucket/a"), + "punctuation sibling must not hide a failing descendant" + ); + assert!(scanner.resume_subtree_has_failures("bucket/b")); + assert!(!scanner.resume_subtree_has_failures("bucket/ab")); + assert!(!scanner.resume_subtree_has_failures("bucket/a/healthy")); + let (cache, _) = bound_checkpoint(); + prepare_scanner(&mut scanner, cache); + scanner.coverage_frontier = Some("bucket/prefix/a".into()); + scanner.record_completed_child("bucket/prefix/a/0000"); + assert_eq!( + scanner.coverage_frontier.as_deref(), + Some("bucket/prefix/a"), + "debt retry must not narrow inherited coverage" + ); + scanner.record_completed_child("bucket/prefix"); + assert_eq!( + scanner.coverage_frontier.as_deref(), + Some("bucket/prefix"), + "completed ancestor widens coverage" + ); + scanner.coverage_frontier = Some("bucket/a-".into()); + assert!(scan_frontier_covers_path("bucket/a/healthy", "bucket/a-")); + scanner.record_completed_child("bucket/a/healthy"); + assert_eq!( + scanner.coverage_frontier.as_deref(), + Some("bucket/a-"), + "component order must preserve a later sibling's frontier" + ); + scanner.coverage_gap = true; + scanner.record_completed_child("bucket/z"); + assert_eq!( + scanner.coverage_frontier.as_deref(), + Some("bucket/a-"), + "structural gaps must still freeze coverage" + ); +} diff --git a/crates/scanner/src/scanner_folder/tests/incremental_enumeration.rs b/crates/scanner/src/scanner_folder/tests/incremental_enumeration.rs index 9390306db..56fc7be53 100644 --- a/crates/scanner/src/scanner_folder/tests/incremental_enumeration.rs +++ b/crates/scanner/src/scanner_folder/tests/incremental_enumeration.rs @@ -91,3 +91,42 @@ fn raw_enumeration_progress_invalid_entry_discards_index_but_retains_diagnostic_ assert!(progress.page_index().is_none()); assert_eq!(progress.cursor().expect("diagnostic cursor").entries_seen, 3); } + +// A flat prefix of tens of thousands of keys must cost the walker a bounded +// amount of bookkeeping per entry. The pre-incremental index re-normalized +// the cumulative entry list on every entry (rustfs#8113: 8,000 entries took +// about 40 s); the page writer hashes each entry once when its page commits. +#[test] +fn raw_enumeration_progress_record_entry_stays_linear_for_large_prefixes() { + const ENTRIES: usize = 20_000; + let committed_page_entries = ENTRIES - ENTRIES % SCANNER_RAW_ENUMERATION_PAGE_ENTRY_LIMIT; + let mut progress = RawEnumerationProgress::new("bucket", None); + RAW_PAGE_DIGEST_ENTRIES.set(0); + let start = Instant::now(); + // Reverse order is the worst case for the sorted insert into the building page. + for i in (0..ENTRIES).rev() { + progress.record_entry(&format!("entry-{i:06}")); + } + let elapsed = start.elapsed(); + assert_eq!(progress.checkpointable_entry_count(), ENTRIES); + assert_eq!( + RAW_PAGE_DIGEST_ENTRIES.get(), + committed_page_entries, + "each entry is hashed exactly once, when its page commits" + ); + assert!( + elapsed < Duration::from_secs(1), + "recording {ENTRIES} entries took {elapsed:?}; per-entry bookkeeping must stay amortised constant" + ); + let checkpoint = progress.page_index().expect("checkpoint with committed pages"); + assert_eq!( + RAW_PAGE_DIGEST_ENTRIES.get(), + ENTRIES, + "a checkpoint hashes only the building page it commits" + ); + assert_eq!(checkpoint.indexed_entries().expect("validated checkpoint").len(), ENTRIES); + assert_eq!( + progress.cursor().expect("cursor").entries_seen, + u64::try_from(ENTRIES).expect("entry count fits") + ); +} diff --git a/crates/scanner/src/scanner_folder/tests/mrf_ownership.rs b/crates/scanner/src/scanner_folder/tests/mrf_ownership.rs index 3e7f9f570..a166a20ef 100644 --- a/crates/scanner/src/scanner_folder/tests/mrf_ownership.rs +++ b/crates/scanner/src/scanner_folder/tests/mrf_ownership.rs @@ -243,6 +243,13 @@ impl HealStorageAPI for NoticeStorage { async fn bucket_incarnation_id(&self, _: &str) -> rustfs_heal::Result> { Ok(Some(self.bucket_incarnation_id)) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> rustfs_heal::Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> rustfs_heal::Result> { Ok(Vec::new()) } diff --git a/crates/scanner/src/scanner_io.rs b/crates/scanner/src/scanner_io.rs index 1af655211..8a1421a4d 100644 --- a/crates/scanner/src/scanner_io.rs +++ b/crates/scanner/src/scanner_io.rs @@ -26,7 +26,7 @@ use futures::future::join_all; use metrics::counter; use rand::seq::SliceRandom as _; #[cfg(test)] -use rustfs_config::{ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS, ENV_SCANNER_MAX_CONCURRENT_SET_SCANS}; +use rustfs_config::{ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS, ENV_SCANNER_MAX_CONCURRENT_SET_SCANS, ENV_SCANNER_WALKERS_PER_DISK}; use rustfs_data_usage::{BucketTargetUsageInfo, BucketUsageInfo, observed_data_usage_is_newer}; use rustfs_filemeta::FileMeta; use rustfs_heal_contracts::heal_channel::HealScanMode; @@ -39,7 +39,7 @@ use s3s::dto::{ BucketLifecycleConfiguration, ObjectLockConfiguration, ObjectLockEnabled, ReplicationConfiguration, VersioningConfiguration, }; use sha2::{Digest as _, Sha256}; -use std::collections::{BTreeMap, HashMap, HashSet}; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; use std::future::Future; use std::path::Path; use std::pin::Pin; @@ -409,14 +409,26 @@ fn resolve_remote_dirty_usage_scope( } } +/// `expected_converged` is `None` for the authoritative object: a `Complete` +/// cycle with pending buckets publishes there unconverged and is still a +/// structural baseline, because the buckets it could not certify stay in the +/// dirty map and are rescanned rather than reused. The convergence marker +/// must still be present; a snapshot without one was not written by a +/// certifying scanner. fn complete_scanner_cache_snapshot_plan_digest( snapshot: &DataUsageInfo, proof: ScannerCacheBaselineProof<'_>, - expected_converged: bool, + expected_converged: Option, ) -> Option { + // The observational mark disqualifies only the authoritative slot: an + // observed candidate carries it by construction. + let marks_accepted = match expected_converged { + Some(expected) => snapshot.usage_snapshot_converged == Some(expected), + None => snapshot.usage_snapshot_converged.is_some() && !snapshot.usage_snapshot_observational, + }; if !snapshot.is_complete_bucket_usage_snapshot() || snapshot.usage_snapshot_partial - || snapshot.usage_snapshot_converged != Some(expected_converged) + || !marks_accepted || snapshot.scanner_epoch != Some(proof.leader_epoch) || snapshot.usage_snapshot_set_states.len() != proof.expected_sources.len() { @@ -455,7 +467,7 @@ fn complete_scanner_cache_snapshot_plan_digest( fn complete_scanner_cache_baseline_plan_digest(proof: ScannerCacheBaselineProof<'_>) -> Option { let authoritative = serde_json::from_slice::(proof.authoritative_data?).ok()?; - if let Some(validated_digest) = complete_scanner_cache_snapshot_plan_digest(&authoritative, proof, true) { + if let Some(validated_digest) = complete_scanner_cache_snapshot_plan_digest(&authoritative, proof, None) { return Some(validated_digest); } @@ -464,7 +476,7 @@ fn complete_scanner_cache_baseline_plan_digest(proof: ScannerCacheBaselineProof< // remains observational: this proof grants bucket-scope reuse only and // never changes authoritative usage publication or dirty acknowledgement. let authoritative_has_identity = (crate::scanner::data_usage_info_has_persisted_baseline_identity(&authoritative) - && authoritative.usage_snapshot_converged != Some(false)) + && !authoritative.usage_snapshot_observational) || crate::scanner::data_usage_info_is_bootstrap_pending(&authoritative); if !authoritative_has_identity { return None; @@ -474,7 +486,7 @@ fn complete_scanner_cache_baseline_plan_digest(proof: ScannerCacheBaselineProof< return None; } - complete_scanner_cache_snapshot_plan_digest(&observed, proof, false) + complete_scanner_cache_snapshot_plan_digest(&observed, proof, Some(false)) } fn scanner_segment_reuse_baseline_producer_evidence( @@ -492,7 +504,9 @@ fn scanner_segment_reuse_baseline_producer_evidence( let Ok(authoritative) = serde_json::from_slice::(authoritative_data) else { return (evidence, false); }; - if complete_scanner_cache_snapshot_plan_digest(&authoritative, baseline_proof, true).is_none() { + // Segment (prefix) reuse keeps requiring a converged baseline: its + // producer evidence is proven against a dirty snapshot that was current. + if complete_scanner_cache_snapshot_plan_digest(&authoritative, baseline_proof, Some(true)).is_none() { return (evidence, false); } let Some(cold_zero_walk_oracle) = @@ -955,6 +969,20 @@ fn scanner_bucket_scan_status(has_failed: bool, has_partial: bool, has_namespace } } +fn scanner_partial_progress_is_usable( + bucket_plan_complete: bool, + scan_scope_matches: bool, + bucket_status: ScannerBucketScanStatus, + has_set_failure: bool, + has_missing_namespace: bool, +) -> bool { + bucket_plan_complete + && scan_scope_matches + && bucket_status == ScannerBucketScanStatus::Partial + && !has_set_failure + && !has_missing_namespace +} + fn scanner_cycle_cold_zero_walk_oracle( scan_scope: &ScannerBucketScanScope, all_buckets: &[BucketInfo], @@ -1000,9 +1028,13 @@ fn classify_nsscanner_cycle( return ScannerCycleStatus::Incomplete; } - match (activity_status, dirty_usage_status) { - (ScannerCycleActivityStatus::Unchanged, DirtyUsageSnapshotStatus::Current) => ScannerCycleStatus::Complete, - _ => ScannerCycleStatus::Superseded, + // Termination is snapshot-at-the-beginning: a structurally complete walk + // that the guardrail did not veto is `Complete`. Buckets whose dirty + // generation moved during the walk stay pending for the next cycle and + // are reported in the publication; they no longer supersede the cycle. + match activity_status { + ScannerCycleActivityStatus::Unchanged => ScannerCycleStatus::Complete, + ScannerCycleActivityStatus::Changed | ScannerCycleActivityStatus::Unverified => ScannerCycleStatus::Superseded, } } @@ -1014,24 +1046,52 @@ fn should_publish_observational_snapshot(status: ScannerCycleStatus) -> bool { matches!(status, ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable)) } +/// Pending buckets of a cycle, sorted so the publication is deterministic. +pub(crate) type ScannerCyclePendingBuckets = BTreeSet; + +/// Stamp the certification outcome of a cycle on its usage snapshot. A +/// `Complete` cycle publishes an authoritative snapshot that is converged +/// only when no bucket is pending; any other publishable status is an +/// observational snapshot routed to the observed object. The marks are part +/// of the bytes the publication expectation digests, so every writer applies +/// exactly this function. +fn apply_scanner_cycle_publication_marks( + data_usage_info: &mut DataUsageInfo, + status: ScannerCycleStatus, + pending_buckets: &ScannerCyclePendingBuckets, +) { + let complete = status == ScannerCycleStatus::Complete; + data_usage_info.usage_snapshot_observational = !complete; + data_usage_info.usage_snapshot_converged = Some(complete && pending_buckets.is_empty()); + if complete { + data_usage_info.usage_snapshot_pending_buckets = pending_buckets.iter().cloned().collect(); + data_usage_info.usage_snapshot_pending_bucket_count = u64::try_from(pending_buckets.len()).unwrap_or(u64::MAX); + } else { + data_usage_info.usage_snapshot_pending_buckets.clear(); + data_usage_info.usage_snapshot_pending_bucket_count = 0; + } +} + fn prepare_usage_snapshot_for_publication( status: ScannerCycleStatus, + pending_buckets: &ScannerCyclePendingBuckets, mut data_usage_info: DataUsageInfo, ) -> Option { if !should_publish_usage_snapshot(status) { return None; } - data_usage_info.usage_snapshot_converged = Some(status == ScannerCycleStatus::Complete); + apply_scanner_cycle_publication_marks(&mut data_usage_info, status, pending_buckets); Some(data_usage_info) } async fn publish_usage_snapshot( updates: &mpsc::Sender, status: ScannerCycleStatus, + pending_buckets: &ScannerCyclePendingBuckets, data_usage_info: DataUsageInfo, ) -> Result { - let Some(data_usage_info) = prepare_usage_snapshot_for_publication(status, data_usage_info) else { + let Some(data_usage_info) = prepare_usage_snapshot_for_publication(status, pending_buckets, data_usage_info) else { return Ok(false); }; send_data_usage_update(updates, data_usage_info).await?; @@ -1045,10 +1105,17 @@ async fn publish_observational_snapshot( data_usage_info.usage_snapshot_complete = false; data_usage_info.usage_snapshot_partial = true; data_usage_info.usage_snapshot_converged = Some(false); + data_usage_info.usage_snapshot_observational = true; + data_usage_info.usage_snapshot_pending_buckets.clear(); + data_usage_info.usage_snapshot_pending_bucket_count = 0; send_data_usage_update(updates, data_usage_info).await?; Ok(true) } +/// Post-walk verdict of the cluster activity guardrail. `Changed` is an +/// attributed veto (or, for peers, the phase-2 dirty-usage fallback); local +/// namespace and dirty-usage generation movement is not a veto and is +/// certified per bucket instead. #[derive(Clone, Copy, Debug, PartialEq, Eq)] enum ScannerCycleActivityStatus { Unchanged, @@ -1069,10 +1136,47 @@ where let publication_blocked = store.scanner_data_usage_publication_blocked().await; match crate::scanner::probe_scanner_activity(store, distributed).await { Ok(after) => { - let status = if !publication_blocked && after == *before { - ScannerCycleActivityStatus::Unchanged + let veto = if publication_blocked { + Some(crate::scanner::ScannerActivityGuardrailVeto::new( + crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE, + "publication_blocked", + )) } else { + crate::scanner::scanner_activity_guardrail_veto(before, &after) + }; + let status = if let Some(veto) = veto { + warn!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_SET_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "cycle_guardrail_veto", + stage = "post_scan_probe", + field = veto.field, + host = %veto.host, + "Scanner cycle publication vetoed by the cluster activity guardrail" + ); + global_metrics().record_scanner_guardrail_veto(veto.reason()); ScannerCycleActivityStatus::Changed + } else if let Some(moved) = crate::scanner::scanner_peer_dirty_usage_moved(before, &after) { + // Per-bucket certification covers the local dirty map only; + // a peer whose dirty state moved keeps today's whole-cycle + // outcome until scoped acknowledgements can name its buckets. + tracing::info!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_SET_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "cycle_peer_dirty_usage_moved", + stage = "post_scan_probe", + field = moved.field, + host = %moved.host, + "Scanner cycle superseded because a peer's dirty usage moved during the walk" + ); + global_metrics().record_scanner_guardrail_veto(moved.reason()); + ScannerCycleActivityStatus::Changed + } else { + ScannerCycleActivityStatus::Unchanged }; (status, crate::scanner::scanner_activity_publication_lease_targets(&after)) } @@ -1134,12 +1238,30 @@ pub(crate) fn cache_root_entry_info(cache: &DataUsageCache) -> std::result::Resu }) } -fn apply_bucket_result_to_cache(cache: &mut DataUsageCache, result: DataUsageEntryInfo, update_time: SystemTime) -> bool { +fn apply_bucket_result_to_cache( + cache: &mut DataUsageCache, + result: DataUsageEntryInfo, + update_time: SystemTime, +) -> Option { if cache.info.tier_registry_generation != result.tier_registry_generation { // A result from another registry generation must never be folded into // this cycle. Leaving it unapplied makes the cycle incomplete and // forces the caller to re-account it under one frozen registry. - return false; + warn!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_CACHE_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "bucket_result_rejected", + bucket = %result.name, + pool = cache.info.source.map(|source| source.pool_index), + set = cache.info.source.map(|source| source.set_index), + cycle = cache.info.next_cycle, + expected_tier_registry_generation = cache.info.tier_registry_generation, + result_tier_registry_generation = result.tier_registry_generation, + "Scanner bucket result rejected because the tier registry generation differs" + ); + return None; } match result.bucket_incarnation { Some(incarnation) if !incarnation.is_nil() => { @@ -1151,7 +1273,7 @@ fn apply_bucket_result_to_cache(cache: &mut DataUsageCache, result: DataUsageEnt } cache.replace(&result.name, &result.parent, result.entry); cache.info.last_update = Some(update_time); - true + Some(result.name) } fn should_publish_completed_snapshot(completed_count: usize, total_count: usize, budget_elapsed: bool, cancelled: bool) -> bool { @@ -1182,6 +1304,42 @@ fn namespace_scanner_workers( workers } +struct NamespaceScannerWalker { + disk: T, + mode: NamespaceScannerWorkerMode, + /// Shared by every walker of one local disk; `None` for disks the + /// coordinator reaches over the network, which keep today's one walker. + local_admission: Option>, +} + +/// Expands each worker into its bucket walkers. A local coordinator disk gets +/// `walkers_per_disk` walkers sharing one disk admission; every other worker +/// keeps a single walker, so the walker count never drops below the disk count. +fn namespace_scanner_walkers( + workers: Vec<(T, NamespaceScannerWorkerMode)>, + walkers_per_disk: usize, + is_local: impl Fn(&T) -> bool, +) -> Vec> { + let mut walkers = Vec::with_capacity(workers.len()); + for (disk, mode) in workers { + if mode == NamespaceScannerWorkerMode::Coordinator && is_local(&disk) { + let local_admission = crate::remote_scanner::LocalDiskAdmissionShare::new(); + walkers.extend((0..walkers_per_disk.max(1)).map(|_| NamespaceScannerWalker { + disk: disk.clone(), + mode, + local_admission: Some(local_admission.clone()), + })); + } else { + walkers.push(NamespaceScannerWalker { + disk, + mode, + local_admission: None, + }); + } + } + walkers +} + fn group_remote_disks_by_peer(disks: Vec, peer_key: impl Fn(&T) -> String) -> Vec> { let mut groups = HashMap::>::new(); for disk in disks { @@ -1426,7 +1584,10 @@ pub(crate) struct ScannerCycleResult { pub(crate) status: ScannerCycleStatus, publication_epoch: Option, activity_digest: Option<[u8; 32]>, - observational_snapshot_published: bool, + activity_baseline: Option>, + pending_dirty_usage_buckets: usize, + usage_update_emitted: bool, + partial_bucket_progress: bool, dirty_usage_clear: Option, remote_dirty_usage_acknowledgements: Vec, distributed_segment_invalidation_evidence: Option, @@ -1444,7 +1605,10 @@ impl ScannerCycleResult { status, publication_epoch: None, activity_digest: None, - observational_snapshot_published: false, + activity_baseline: None, + pending_dirty_usage_buckets: 0, + usage_update_emitted: false, + partial_bucket_progress: false, dirty_usage_clear, remote_dirty_usage_acknowledgements: Vec::new(), distributed_segment_invalidation_evidence: None, @@ -1473,18 +1637,52 @@ impl ScannerCycleResult { self } + #[cfg(test)] pub(crate) fn activity_digest(&self) -> Option<[u8; 32]> { self.activity_digest } - pub(crate) fn with_observational_snapshot_published(mut self, published: bool) -> Self { + fn with_activity_baseline(mut self, baseline: Arc) -> Self { self.publication_expectation = None; - self.observational_snapshot_published = published; + self.activity_baseline = Some(baseline); self } - pub(crate) fn has_observational_snapshot(&self) -> bool { - self.observational_snapshot_published + /// The pre-walk activity probe the post-lease guardrail recheck compares + /// against. + pub(crate) fn activity_baseline(&self) -> Option<&crate::scanner::ScannerActivitySnapshot> { + self.activity_baseline.as_deref() + } + + fn with_pending_dirty_usage_buckets(mut self, pending: usize) -> Self { + self.publication_expectation = None; + self.pending_dirty_usage_buckets = pending; + self + } + + /// Buckets a `Complete` cycle left pending for the next cycle. + pub(crate) fn pending_dirty_usage_buckets(&self) -> usize { + self.pending_dirty_usage_buckets + } + + pub(crate) fn with_usage_update_emitted(mut self, emitted: bool) -> Self { + self.publication_expectation = None; + self.usage_update_emitted = emitted; + self + } + + pub(crate) fn has_usage_update(&self) -> bool { + self.usage_update_emitted + } + + pub(crate) fn with_partial_bucket_progress(mut self, partial_bucket_progress: bool) -> Self { + self.publication_expectation = None; + self.partial_bucket_progress = partial_bucket_progress; + self + } + + pub(crate) fn has_partial_bucket_progress(&self) -> bool { + self.partial_bucket_progress } fn with_failed_dirty_usage(mut self, failed_dirty_usage: bool) -> Self { @@ -1611,7 +1809,8 @@ use guards::*; pub(crate) use cache::{ DataUsageCacheReuseOptions, DataUsageCacheScanState, ScannerCheckpointPersistContext, ScannerCheckpointPersistResult, - acquire_scanner_cache_locks, current_cache_root_or_prepare_with_generation, persist_scanner_checkpoint, + acquire_scanner_cache_locks, current_cache_root_or_prepare_with_generation, log_saved_partial_bucket_cache, + persist_scanner_checkpoint, }; pub use dirty_usage::{ ScannerDirtyUsageAckError, ScannerDirtyUsageBucket, ScannerDirtyUsageClearObserver, ScannerDirtyUsageMutationObserver, @@ -1620,9 +1819,9 @@ pub use dirty_usage::{ acknowledge_scoped_dirty_usage, clear_dirty_usage_bucket, encode_durable_dirty_usage_producer_replay_record, record_dirty_usage_bucket, record_dirty_usage_bucket_from_producer, record_dirty_usage_bucket_from_producers, record_dirty_usage_object, record_dirty_usage_object_from_producer, record_scanner_maintenance_change, - replay_durable_dirty_usage_producer_record, scanner_activity_epoch, scanner_dirty_usage_bucket_generation, - scanner_dirty_usage_snapshot, scanner_dirty_usage_state, scanner_maintenance_generation, - set_scanner_dirty_usage_clear_observer, set_scanner_dirty_usage_mutation_observer, + record_status_class_mutation, replay_durable_dirty_usage_producer_record, scanner_activity_epoch, + scanner_dirty_usage_bucket_generation, scanner_dirty_usage_snapshot, scanner_dirty_usage_state, + scanner_maintenance_generation, set_scanner_dirty_usage_clear_observer, set_scanner_dirty_usage_mutation_observer, }; #[cfg(test)] pub(crate) use dirty_usage::{clear_dirty_usage_buckets_for_tests, dirty_usage_buckets_for_tests}; diff --git a/crates/scanner/src/scanner_io/cache.rs b/crates/scanner/src/scanner_io/cache.rs index 4ae95d6a0..d11c2d000 100644 --- a/crates/scanner/src/scanner_io/cache.rs +++ b/crates/scanner/src/scanner_io/cache.rs @@ -15,6 +15,24 @@ use super::*; use crate::{ScannerConfigObjectDelete, ScannerObjectIO}; +pub(crate) fn log_saved_partial_bucket_cache(cache: &DataUsageCache, cache_name: &str) { + debug!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_CACHE_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "partial_saved_not_published", + bucket = %cache.info.name, + pool = cache.info.source.map(|source| source.pool_index), + set = cache.info.source.map(|source| source.set_index), + cycle = cache.info.next_cycle, + leader_epoch = cache.info.leader_epoch, + cache_name, + resume_position = cache.has_scan_position(), + "Scanner partial bucket cache saved without publishing usage aggregate" + ); +} + pub(crate) fn scanner_cache_lock_resource(cache_name: &str, source: DataUsageCacheSource) -> String { let lock_name = format!("{SCANNER_CACHE_LOCK_SUFFIX}.pool-{}.set-{}", source.pool_index, source.set_index); path_join_buf(&[crate::BUCKET_META_PREFIX, cache_name, &lock_name]) @@ -28,7 +46,8 @@ pub(crate) fn scanner_cache_lock_timeout() -> Duration { pub(crate) enum ScannerCheckpointPersistResult { Saved, FenceChanged, - Failed(StorageError), + Skipped, + RetryBucket(StorageError), } pub(crate) struct ScannerCheckpointPersistContext<'a> { @@ -80,7 +99,12 @@ where ) .await; if let Err(error) = checkpoint_save { - return ScannerCheckpointPersistResult::Failed(error); + record_checkpoint_save_failure(cache_name, checkpoint, &error); + return if matches!(error, StorageError::PreconditionFailed) { + ScannerCheckpointPersistResult::RetryBucket(error) + } else { + ScannerCheckpointPersistResult::Skipped + }; } if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(context.cycle, context.leader_epoch, fence_store) @@ -101,12 +125,48 @@ where { Ok(next_revisions) => { *revisions = next_revisions; + if let Some(source) = checkpoint.info.source { + let saved = checkpoint.info.scan_checkpoint.as_ref().map(|cursor| { + rustfs_scanner_metrics::metrics::ScannerCheckpointReport { + version: cursor.version, + resume_after: cursor.resume_after.clone(), + reason: cursor.reason.as_str().to_string(), + last_event: "saved".to_string(), + } + }); + global_metrics().record_scanner_checkpoint_saved( + source.pool_index, + source.set_index, + &checkpoint.info.name, + saved, + ); + } ScannerCheckpointPersistResult::Saved } - Err(error) => ScannerCheckpointPersistResult::Failed(error), + Err(error) => { + // A successful PUT consumed the old CAS revision. Reload the bucket + // before another write instead of rebasing this scan over a newer writer. + record_checkpoint_save_failure(cache_name, checkpoint, &error); + ScannerCheckpointPersistResult::RetryBucket(error) + } } } +fn record_checkpoint_save_failure(cache_name: &str, checkpoint: &DataUsageCache, error: &StorageError) { + global_metrics().record_scanner_checkpoint_save_failed(); + error!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_CACHE_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "periodic_checkpoint_save_failed", + bucket = %checkpoint.info.name, + cache_name = %cache_name, + error = %error, + "Scanner periodic checkpoint save failed" + ); +} + #[derive(Debug)] pub(crate) struct ScannerCacheLockGuards { scoped: NamespaceLockGuard, @@ -277,6 +337,24 @@ pub(crate) fn current_cache_root_or_prepare_with_generation( // an older registry cannot be reused under the new cycle generation. cache.info.scan_plan_digest = None; } + if cache.info.name == name + && cache.info.next_cycle <= next_cycle + && cache.info.leader_epoch <= leader_epoch + && cache.info.snapshot_complete + && cache + .info + .replication + .as_ref() + .is_some_and(|config| config.has_active_rules("", true)) + { + // Replication writebacks leave namespace generations unchanged. A + // completed cache must start a fresh coverage sweep, even on retry. + cache.info.scan_progress = Some(crate::DataUsageScanProgress { + started_plan: scan_plan_digest, + requested_plan: scan_plan_digest, + }); + cache.info.scan_plan_digest = None; + } match current_cache_root_entry_with_generation( cache, name, @@ -314,6 +392,72 @@ pub(super) fn cache_snapshot_is_current( ) } +/// Replication statistics are observational and may change within one +/// namespace execution. Bind them without relaxing same-execution reuse. +pub(super) fn scanner_replication_execution_digest( + execution_digest: DataUsageScanPlanDigest, + cache: &DataUsageCache, +) -> DataUsageScanPlanDigest { + let mut entries = cache + .cache + .iter() + .filter_map(|(key, entry)| { + entry + .replication_stats + .as_ref() + .filter(|stats| stats.replica_size != 0 || stats.replica_count != 0 || !stats.targets.is_empty()) + .map(|stats| (key, stats)) + }) + .collect::>(); + if entries.is_empty() { + return execution_digest; + } + entries.sort_unstable_by(|left, right| left.0.cmp(right.0)); + let mut hasher = Sha256::new(); + hasher.update(b"scanner-replication-execution-v1"); + hasher.update(execution_digest.0); + for (key, stats) in entries { + hasher.update(u64::try_from(key.len()).unwrap_or(u64::MAX).to_be_bytes()); + hasher.update(key.as_bytes()); + hasher.update(stats.replica_size.to_be_bytes()); + hasher.update(stats.replica_count.to_be_bytes()); + let mut targets = stats.targets.iter().collect::>(); + targets.sort_unstable_by(|left, right| left.0.cmp(right.0)); + hasher.update(u64::try_from(targets.len()).unwrap_or(u64::MAX).to_be_bytes()); + for (arn, target) in targets { + hasher.update(u64::try_from(arn.len()).unwrap_or(u64::MAX).to_be_bytes()); + hasher.update(arn.as_bytes()); + let rustfs_data_usage::ReplicationTargetUsage { + pending_size, + replicated_size, + failed_size, + failed_count, + pending_count, + missed_threshold_size, + after_threshold_size, + missed_threshold_count, + after_threshold_count, + replicated_count, + } = target; + for value in [ + pending_size, + replicated_size, + failed_size, + failed_count, + pending_count, + missed_threshold_size, + after_threshold_size, + missed_threshold_count, + after_threshold_count, + replicated_count, + ] { + hasher.update(value.to_be_bytes()); + } + } + } + DataUsageScanPlanDigest(hasher.finalize().into()) +} + #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(super) struct ScannerSnapshotIdentity { pub(super) cycle: u64, @@ -452,8 +596,12 @@ pub(super) fn empty_namespace_usage_candidate( } impl ValidatedUsageCandidate { - pub(super) fn prepare(mut self, status: ScannerCycleStatus) -> (DataUsageInfo, Option) { - self.data.usage_snapshot_converged = Some(status == ScannerCycleStatus::Complete); + pub(super) fn prepare( + mut self, + status: ScannerCycleStatus, + pending_buckets: &ScannerCyclePendingBuckets, + ) -> (DataUsageInfo, Option) { + apply_scanner_cycle_publication_marks(&mut self.data, status, pending_buckets); let expectation = if status == ScannerCycleStatus::Complete { struct DigestWriter(Sha256); impl std::io::Write for DigestWriter { diff --git a/crates/scanner/src/scanner_io/dirty_usage.rs b/crates/scanner/src/scanner_io/dirty_usage.rs index 43af3e1df..5e549d6c5 100644 --- a/crates/scanner/src/scanner_io/dirty_usage.rs +++ b/crates/scanner/src/scanner_io/dirty_usage.rs @@ -457,6 +457,88 @@ fn apply_scoped_dirty_usage_ack( Ok(cleared) } +#[cfg(test)] +mod status_class_mutation_tests { + use super::*; + use crate::segment_invalidation::SegmentInvalidationProducerIdentity; + use serial_test::serial; + + fn bucket(name: &str) -> BucketInfo { + BucketInfo { + name: name.to_string(), + created: None, + deleted: None, + versioning: false, + object_locking: false, + } + } + + #[test] + #[serial] + fn status_class_mutation_is_not_namespace_activity() { + clear_dirty_usage_buckets_for_tests(); + let generation_before = dirty_usage_generation(); + let snapshot = snapshot_dirty_usage_buckets(&[bucket("photos")], generation_before); + assert_eq!(dirty_usage_snapshot_status(&snapshot), DirtyUsageSnapshotStatus::Current); + + record_status_class_mutation("photos", "2026/image.jpg", SegmentInvalidationProducerIdentity::Replication); + + assert_eq!( + dirty_usage_generation(), + generation_before, + "a status-class writeback must not advance the dirty generation" + ); + assert!( + !dirty_usage_buckets_pending(), + "a status-class writeback must not count as pending dirty usage" + ); + assert!( + dirty_usage_buckets_for_tests().is_empty(), + "a status-class writeback must not enter the dirty map" + ); + assert_eq!( + dirty_usage_snapshot_status(&snapshot), + DirtyUsageSnapshotStatus::Current, + "a status-class writeback during a walk must not supersede the cycle" + ); + assert_ne!( + DIRTY_USAGE_PRODUCER_COVERAGE.load(Ordering::Acquire) + & SegmentInvalidationProducerIdentity::Replication + .production_coverage_bit() + .expect("replication has a coverage bit"), + 0, + "producer coverage keeps recording the reclassified producer" + ); + clear_dirty_usage_buckets_for_tests(); + } + + #[test] + #[serial] + fn status_class_mutation_does_not_reach_the_mutation_observer() { + use std::sync::{Arc, Mutex}; + + clear_dirty_usage_buckets_for_tests(); + let observed = Arc::new(Mutex::new(Vec::new())); + let sink = observed.clone(); + let previous = set_scanner_dirty_usage_mutation_observer(Some(Arc::new(move |bucket, object, producer| { + sink.lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .push((bucket.to_string(), object.to_string(), producer)); + }))); + + record_status_class_mutation("photos", "2026/image.jpg", SegmentInvalidationProducerIdentity::Replication); + record_status_class_mutation("", "2026/image.jpg", SegmentInvalidationProducerIdentity::Replication); + record_status_class_mutation("photos", "", SegmentInvalidationProducerIdentity::Replication); + + set_scanner_dirty_usage_mutation_observer(previous); + assert!( + observed.lock().unwrap_or_else(|poisoned| poisoned.into_inner()).is_empty(), + "the durable dirty journal must never see a status-class writeback" + ); + clear_dirty_usage_buckets_for_tests(); + } +} + #[cfg(test)] mod scoped_dirty_usage_tests { use super::*; @@ -934,6 +1016,31 @@ where DIRTY_USAGE_BUCKET_NOTIFY.notify_one(); } +/// Record a status-class barrier hit: a committed writeback that changed only +/// replication or restore state on an existing version. +/// +/// Such a writeback changes no size, version count or delete marker, so it is +/// not namespace activity: it must not advance the dirty-usage generation, +/// enter the activity snapshot, wake the scanner or reach the durable dirty +/// journal, because any of those makes a cycle `Superseded` for as long as +/// replication runs. It needs no rescan queue either: a deployment with an +/// active replication rule walks every bucket on every cycle +/// (`ScannerMaintenanceFeatures::needs_regular_cycle`), and that walk is what +/// refreshes `replication_stats`. Only the process-local producer coverage is +/// recorded so segment-reuse evidence keeps its meaning. +pub fn record_status_class_mutation( + bucket: &str, + object: &str, + producer: crate::segment_invalidation::SegmentInvalidationProducerIdentity, +) { + if bucket.is_empty() || object.is_empty() { + return; + } + if let Some(coverage_bit) = producer.production_coverage_bit() { + DIRTY_USAGE_PRODUCER_COVERAGE.fetch_or(coverage_bit, Ordering::AcqRel); + } +} + fn record_segment_invalidation_producer_identities_for_generation( identities: &mut DirtyUsageProducerIdentities, bucket: &str, @@ -1218,21 +1325,41 @@ pub(super) fn dirty_usage_buckets_excluding_failed( .collect() } +/// Plan the per-bucket certification of a walk. The returned snapshot is +/// applied by `clear_dirty_usage_buckets`, which removes only the buckets +/// whose generation still equals the start snapshot: a bucket written during +/// the walk keeps its newer generation and stays pending. The dirty-usage +/// generation therefore does not need to be unchanged; only the guardrail +/// must have passed. pub(super) fn should_clear_dirty_usage_snapshot( result_ok: bool, completed_all_sets: bool, budget_elapsed: bool, - activity_and_generation_current: bool, + guardrail_passed: bool, dirty_buckets: &DirtyUsageBuckets, failed_buckets: &HashSet, ) -> Option { - if result_ok && completed_all_sets && !budget_elapsed && activity_and_generation_current { + if result_ok && completed_all_sets && !budget_elapsed && guardrail_passed { return Some(dirty_usage_buckets_excluding_failed(dirty_buckets, failed_buckets)); } None } +/// Buckets the walk cannot certify against its start snapshot: every dirty +/// bucket whose generation differs from the snapshot's, including buckets +/// dirtied after the snapshot was taken. Only a `Complete` cycle reads this, +/// and a failed bucket never reaches `Complete`, so the failed set that +/// `dirty_usage_buckets_excluding_failed` removes is empty here. +pub(super) fn pending_dirty_usage_buckets(snapshot: &DirtyUsageBuckets) -> BTreeSet { + let dirty_buckets = dirty_usage_buckets(); + dirty_buckets + .iter() + .filter(|(bucket, generation)| snapshot.get(bucket.as_str()) != Some(*generation)) + .map(|(bucket, _)| bucket.clone()) + .collect() +} + pub(super) async fn record_failed_dirty_bucket(failed_buckets: &Arc>>, bucket: &str) { failed_buckets.lock().await.insert(bucket.to_string()); } diff --git a/crates/scanner/src/scanner_io/guards.rs b/crates/scanner/src/scanner_io/guards.rs index bc1e0e4a2..3a5cb4fa9 100644 --- a/crates/scanner/src/scanner_io/guards.rs +++ b/crates/scanner/src/scanner_io/guards.rs @@ -546,15 +546,13 @@ pub(super) fn reset_disk_bucket_scan_gauges(pool: &str, set: &str) { record_disk_bucket_scans_active(0, pool, set); } +// Foreground pressure is handled during work by disk admission and the sleeper. +// Dispatch limits must not freeze a momentary request count for an entire cycle. pub(super) fn scanner_concurrency_limit(configured: usize, available: usize) -> usize { if available == 0 { return 0; } - if crate::workload_admission::foreground_workload_activity() > 0 { - return 1; - } - if configured == 0 { available } else { diff --git a/crates/scanner/src/scanner_io/io_cache.rs b/crates/scanner/src/scanner_io/io_cache.rs index e8920915e..07eddd249 100644 --- a/crates/scanner/src/scanner_io/io_cache.rs +++ b/crates/scanner/src/scanner_io/io_cache.rs @@ -14,6 +14,28 @@ /// ScannerIOCache implementation for SetDisks: bucket ordering, worker fan-out, merge, and publish. use super::*; +pub(super) async fn collect_completed_bucket_results( + ctx: CancellationToken, + mut cache: DataUsageCache, + mut results: mpsc::Receiver, +) -> (DataUsageCache, HashSet) { + let mut cancelled = false; + let mut completed = HashSet::new(); + loop { + tokio::select! { + _ = ctx.cancelled(), if !cancelled => cancelled = true, + result = results.recv() => { + let Some(result) = result else { + return (cache, completed); + }; + if let Some(bucket) = apply_bucket_result_to_cache(&mut cache, result, SystemTime::now()) { + completed.insert(bucket); + } + } + } + } +} + #[derive(Clone, Copy)] pub(super) struct ScannerSetCacheGeneration { pub(super) want_cycle: u64, @@ -294,7 +316,8 @@ impl ScannerIOCache for SetDisks { }; cache.info.last_update = Some(now); cache.info.snapshot_complete = true; - cache.info.scan_execution_digest = Some(execution_digest); + let replication_execution = super::cache::scanner_replication_execution_digest(execution_digest, &cache); + cache.info.scan_execution_digest = Some(replication_execution); cache.info.segment_invalidation_proof = completed_segment_invalidation_proof; cache.info.lkg_snapshot_complete = false; cache.info.lkg_next_cycle = None; @@ -469,21 +492,34 @@ impl ScannerIOCache for SetDisks { "Scanner set assigned remote disks without namespace scanner support to coordinator-driven workers" ); } - let disk_scan_limit = scanner_budgeted_concurrency_limit( - scanner_max_concurrent_disk_scans(workers.len()), - budget.requires_serial_progress_accounting(), - ); + let online_disks = workers.len(); + let walkers_per_disk = crate::runtime_config::scanner_walkers_per_disk_configured(); + let walkers = namespace_scanner_walkers(workers, walkers_per_disk, |disk| disk.is_local()); + let dispatch_limit = scanner_max_concurrent_disk_scans(walkers.len()); + let configured_disk_scans = crate::runtime_config::scanner_max_concurrent_disk_scans_configured(); + let serial_progress_accounting = budget.requires_serial_progress_accounting(); + let disk_scan_limit = scanner_budgeted_concurrency_limit(dispatch_limit, serial_progress_accounting); record_disk_scan_concurrency_limit(&pool_label, &set_label, disk_scan_limit); - debug!( + tracing::info!( target: "rustfs::scanner::io", event = EVENT_SCANNER_SET_STATE, component = LOG_COMPONENT_SCANNER, subsystem = LOG_SUBSYSTEM_IO, + state = "disk_concurrency_budget", + reason = if serial_progress_accounting { + "count_budget" + } else if configured_disk_scans != 0 && configured_disk_scans < walkers.len() { + "configured_cap" + } else { + "available_walkers" + }, pool = self.pool_index, set = self.set_index, - online_disks = workers.len(), + online_disks, + walkers_per_disk, + walkers = walkers.len(), + dispatch_limit, concurrency_limit = disk_scan_limit, - state = "disk_concurrency_budget", "Scanner disk concurrency budget resolved" ); let disk_scan_semaphore = Arc::new(Semaphore::new(disk_scan_limit)); @@ -625,42 +661,21 @@ impl ScannerIOCache for SetDisks { } } - let cache_mutex: Arc> = Arc::new(Mutex::new(cache)); - - let (bucket_result_tx, mut bucket_result_rx) = mpsc::channel::(workers.len()); - - let cache_mutex_clone = cache_mutex.clone(); - let ctx_clone = ctx.clone(); - let completed_bucket_count = Arc::new(AtomicUsize::new(0)); - let completed_bucket_count_clone = completed_bucket_count.clone(); - let collect_bucket_results_fut = AbortOnDropHandle::new(tokio::spawn(async move { - let mut cancelled = false; - - loop { - tokio::select! { - _ = ctx_clone.cancelled(), if !cancelled => { - cancelled = true; - } - result = bucket_result_rx.recv() => { - let Some(result) = result else { - return; - }; - - let mut cache = cache_mutex_clone.lock().await; - if apply_bucket_result_to_cache(&mut cache, result, SystemTime::now()) { - completed_bucket_count_clone.fetch_add(1, Ordering::Relaxed); - } - } - } - } - })); + let (bucket_result_tx, bucket_result_rx) = mpsc::channel::(walkers.len()); + let collect_bucket_results_fut = + AbortOnDropHandle::new(tokio::spawn(collect_completed_bucket_results(ctx.clone(), cache, bucket_result_rx))); let mut futs = Vec::new(); let bucket_rx_mutex: Arc>> = Arc::new(Mutex::new(bucket_rx)); let remaining_bucket_work = Arc::new(AtomicUsize::new(buckets.len())); let bucket_work_complete = CancellationToken::new(); - for (disk, worker_mode) in workers { + for NamespaceScannerWalker { + disk, + mode: worker_mode, + local_admission, + } in walkers + { let service_cohort_clone = service_cohort.clone(); let bucket_rx_mutex_clone = bucket_rx_mutex.clone(); let bucket_tx_clone = bucket_tx.clone(); @@ -813,6 +828,19 @@ impl ScannerIOCache for SetDisks { } Ok(crate::remote_scanner::RemoteScannerOutcome::Partial) => { record_partial_dirty_bucket(&partial_dirty_buckets_clone, &bucket.name).await; + debug!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_DISK_BUCKET_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "remote_scan_partial", + bucket = %bucket.name, + pool = source.pool_index, + set = source.set_index, + cycle = want_cycle, + disk = %disk.path().display(), + "Remote scanner bucket returned partial coverage" + ); } Ok(crate::remote_scanner::RemoteScannerOutcome::NamespaceNotFound) => { if requeue_bucket_work(&bucket_tx_clone, &bucket, &mut work_guard).await { @@ -905,8 +933,8 @@ impl ScannerIOCache for SetDisks { continue; } - let _local_admission = if disk.is_local() { - match crate::remote_scanner::try_admit_remote_scanner(&disk) { + let _local_admission = if let Some(share) = &local_admission { + match share.acquire(&disk) { Ok(admission) => Some(admission), Err(e) => { if requeue_bucket_work(&bucket_tx_clone, &bucket, &mut work_guard).await { @@ -934,6 +962,8 @@ impl ScannerIOCache for SetDisks { } else { None }; + #[cfg(test)] + super::tests::observe_coordinator_bucket_scan_admitted(&bucket.name).await; // Lock order: scanner leader fence -> set-scoped per-bucket cache lock -> // cache object read/write. @@ -1209,7 +1239,7 @@ impl ScannerIOCache for SetDisks { ) .await { - ScannerCheckpointPersistResult::Saved => { + ScannerCheckpointPersistResult::Saved | ScannerCheckpointPersistResult::Skipped => { if cache_guard.is_lock_lost() || ctx_clone.is_cancelled() { scan_ctx.cancel(); await_scanner_disk_shutdown(scan.as_mut()).await; @@ -1221,19 +1251,10 @@ impl ScannerIOCache for SetDisks { await_scanner_disk_shutdown(scan.as_mut()).await; break Err(Error::other("scanner bucket cache fence changed during checkpoint save")); } - ScannerCheckpointPersistResult::Failed(error) => { - error!( - target: "rustfs::scanner::io", - event = EVENT_SCANNER_CACHE_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_IO, - bucket = %bucket.name, - cache_name = %cache_name, - state = "periodic_checkpoint_save_failed", - error = %error, - "Scanner periodic checkpoint save failed" - ); - checkpoint_channel_closed = true; + ScannerCheckpointPersistResult::RetryBucket(error) => { + scan_ctx.cancel(); + await_scanner_disk_shutdown(scan.as_mut()).await; + break Err(error); } } } @@ -1380,16 +1401,7 @@ impl ScannerIOCache for SetDisks { record_failed_dirty_bucket(&failed_dirty_buckets_clone, &bucket.name).await; } if partial_saved { - debug!( - target: "rustfs::scanner::io", - event = EVENT_SCANNER_CACHE_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_IO, - bucket = %bucket.name, - cache_name = %cache_name, - state = "partial_saved_not_published", - "Scanner partial bucket cache saved without publishing usage aggregate" - ); + log_saved_partial_bucket_cache(&partial_cache, &cache_name); } continue; @@ -1557,46 +1569,45 @@ impl ScannerIOCache for SetDisks { record_disk_bucket_scans_queued(0, &pool_label, &set_label); record_disk_bucket_scans_active(0, &pool_label, &set_label); - if let Err(err) = collect_bucket_results_fut.await { - return Err(scanner_task_join_error("scanner bucket result collector", err)); - } + let (mut cache, completed_buckets) = collect_bucket_results_fut + .await + .map_err(|err| scanner_task_join_error("scanner bucket result collector", err))?; if let Some(err) = first_join_err { return Err(err); } - let completed_count = completed_bucket_count.load(Ordering::Relaxed); - if should_publish_completed_snapshot(completed_count, buckets.len(), budget.budget_elapsed(), ctx.is_cancelled()) { + let completed_count = completed_buckets.len(); + let coverage_complete = + should_publish_completed_snapshot(completed_count, buckets.len(), budget.budget_elapsed(), ctx.is_cancelled()); + if coverage_complete { let completed_segment_invalidation_proof = scanner_completed_set_segment_invalidation_proof(&segment_invalidation_proof, cold_zero_walk_reuse_candidate); - let cache_snapshot = { - let mut cache = cache_mutex.lock().await; - cache.info.next_cycle = want_cycle; - cache.info.last_update.get_or_insert_with(SystemTime::now); - cache.info.snapshot_complete = true; - cache.info.scan_execution_digest = Some(execution_digest); - cache.info.segment_invalidation_proof = completed_segment_invalidation_proof; - cache.info.lkg_snapshot_complete = false; - cache.info.lkg_next_cycle = None; - cache.info.lkg_last_update = None; - cache.info.lkg_leader_epoch = None; - cache.info.lkg_scan_plan_digest = None; - if cold_zero_walk_reuse_candidate { - cold_zero_walk_reuse_observed.store(true, Ordering::Release); - } - cache.clone() - }; + cache.info.next_cycle = want_cycle; + cache.info.last_update.get_or_insert_with(SystemTime::now); + cache.info.snapshot_complete = true; + let replication_execution = super::cache::scanner_replication_execution_digest(execution_digest, &cache); + cache.info.scan_execution_digest = Some(replication_execution); + cache.info.segment_invalidation_proof = completed_segment_invalidation_proof; + cache.info.lkg_snapshot_complete = false; + cache.info.lkg_next_cycle = None; + cache.info.lkg_last_update = None; + cache.info.lkg_leader_epoch = None; + cache.info.lkg_scan_plan_digest = None; + if cold_zero_walk_reuse_candidate { + cold_zero_walk_reuse_observed.store(true, Ordering::Release); + } let _ = persist_and_publish_cache_snapshot( self.clone(), &updates, - cache_snapshot, + cache, initial_revisions.as_ref(), cache_cycle_floor.as_ref(), expected_publication_epoch, ) .await; } else { - let mut incomplete_scope = cache_mutex.lock().await.clone(); + let mut incomplete_scope = cache; incomplete_scope.info.name = DATA_USAGE_ROOT.to_string(); incomplete_scope.info.next_cycle = want_cycle; incomplete_scope.info.last_update = None; @@ -1625,11 +1636,20 @@ impl ScannerIOCache for SetDisks { event = EVENT_SCANNER_CACHE_PERSIST_STATE, component = LOG_COMPONENT_SCANNER, subsystem = LOG_SUBSYSTEM_IO, + state = "set_cache_publish_skipped", + pool = source.pool_index, + set = source.set_index, + cycle = want_cycle, completed_buckets = completed_count, total_buckets = buckets.len(), + first_incomplete_bucket = buckets + .iter() + .map(|bucket| bucket.name.as_str()) + .filter(|bucket| !completed_buckets.contains(*bucket)) + .min(), + unprocessed_buckets, budget_elapsed = budget.budget_elapsed(), cancelled = ctx.is_cancelled(), - state = "set_cache_publish_skipped", "Scanner set cache publish skipped because cycle did not complete cleanly" ); } @@ -1640,7 +1660,13 @@ impl ScannerIOCache for SetDisks { component = LOG_COMPONENT_SCANNER, subsystem = LOG_SUBSYSTEM_IO, state = "set_scan_completed", - "Scanner set scan completed" + pool = source.pool_index, + set = source.set_index, + cycle = want_cycle, + completed_buckets = completed_count, + total_buckets = buckets.len(), + coverage_complete, + "Scanner set workers finished" ); Ok(()) diff --git a/crates/scanner/src/scanner_io/io_cycle.rs b/crates/scanner/src/scanner_io/io_cycle.rs index b77411e09..58e3150e1 100644 --- a/crates/scanner/src/scanner_io/io_cycle.rs +++ b/crates/scanner/src/scanner_io/io_cycle.rs @@ -77,6 +77,8 @@ where service_cohort: None, #[cfg(test)] resolved_scope_observer: None, + #[cfg(test)] + post_walk_gate: None, }; nsscanner_with_storage_status_scoped(store, request).await } @@ -96,6 +98,11 @@ pub(crate) struct ScannerCycleRequest { pub(crate) service_cohort: Option>>, #[cfg(test)] pub(crate) resolved_scope_observer: Option>, + /// Holds the post-walk guardrail probe until the test releases it, so a + /// write issued after `resolved_scope_observer` fired is ordered before + /// certification regardless of how fast the walk finishes. + #[cfg(test)] + pub(crate) post_walk_gate: Option>, } pub(super) struct ScannerBucketScopeResolution<'a> { @@ -287,6 +294,8 @@ where service_cohort, #[cfg(test)] resolved_scope_observer, + #[cfg(test)] + post_walk_gate, } = request; let child_token = ctx.child_token(); let _tier_cycle_guard = begin_tier_registry_cycle(want_cycle, leader_epoch); @@ -325,7 +334,7 @@ where let distributed = store.setup_is_dist_erasure().await; let activity_before = match scanner_activity_preflight(crate::scanner::probe_scanner_activity(store, distributed).await) { - ScannerActivityPreflight::Ready(snapshot) => snapshot, + ScannerActivityPreflight::Ready(snapshot) => Arc::new(snapshot), ScannerActivityPreflight::ActivityBaselineUnavailable(err) => { warn!( target: "rustfs::scanner::io", @@ -442,6 +451,11 @@ where dirty_usage_status, activity_status, ); + let pending_buckets = if status == ScannerCycleStatus::Complete { + pending_dirty_usage_buckets(&dirty_usage_snapshot.buckets) + } else { + ScannerCyclePendingBuckets::new() + }; let Some(candidate) = empty_namespace_usage_candidate( &all_buckets, &expected_sources, @@ -456,13 +470,13 @@ where ) else { return Ok(ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None).with_publication_epoch(publication_epoch)); }; - let (empty_usage, publication_expectation) = candidate.prepare(status); - let observational_snapshot_published = if should_publish_observational_snapshot(status) { + let (empty_usage, publication_expectation) = candidate.prepare(status, &pending_buckets); + let usage_update_emitted = if should_publish_observational_snapshot(status) { publish_observational_snapshot(&updates, empty_usage).await? } else { - publish_usage_snapshot(&updates, status, empty_usage).await? + publish_usage_snapshot(&updates, status, &pending_buckets, empty_usage).await? }; - if !observational_snapshot_published { + if !usage_update_emitted { return Ok(ScannerCycleResult::new(status, None).with_publication_epoch(publication_epoch)); } if status == ScannerCycleStatus::Complete { @@ -484,7 +498,9 @@ where return Ok(ScannerCycleResult::new(status, dirty_usage_clear) .with_publication_epoch(publication_epoch) .with_activity_digest(activity_digest) - .with_observational_snapshot_published(observational_snapshot_published) + .with_activity_baseline(activity_before) + .with_pending_dirty_usage_buckets(pending_buckets.len()) + .with_usage_update_emitted(usage_update_emitted) .with_remote_publication_lease_targets(remote_publication_lease_targets) .with_remote_dirty_usage_acknowledgements(remote_dirty_usage_acknowledgements) .with_segment_reuse_activation_preflight(segment_reuse_activation_preflight) @@ -506,22 +522,23 @@ where return Ok(ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None).with_publication_epoch(publication_epoch)); } - let set_scan_limit = scanner_budgeted_concurrency_limit( - scanner_max_concurrent_set_scans(total_results), - budget.requires_serial_progress_accounting(), - ); + let dispatch_limit = scanner_max_concurrent_set_scans(total_results); + let serial_progress_accounting = budget.requires_serial_progress_accounting(); + let set_scan_limit = scanner_budgeted_concurrency_limit(dispatch_limit, serial_progress_accounting); let bucket_failures = ScannerBucketFailureState::default(); let pending_maintenance_work = Arc::new(AtomicBool::new(false)); let cold_zero_walk_reuse_observed = Arc::new(AtomicBool::new(false)); record_set_scan_concurrency_limit(set_scan_limit); - debug!( + tracing::info!( target: "rustfs::scanner::io", event = EVENT_SCANNER_SET_STATE, component = LOG_COMPONENT_SCANNER, subsystem = LOG_SUBSYSTEM_IO, - total_sets = total_results, - concurrency_limit = set_scan_limit, state = "concurrency_budget", + reason = if serial_progress_accounting { "count_budget" } else { "configured_or_available_work" }, + total_sets = total_results, + dispatch_limit, + concurrency_limit = set_scan_limit, "Scanner set concurrency budget resolved" ); let set_scan_semaphore = Arc::new(Semaphore::new(set_scan_limit)); @@ -697,8 +714,13 @@ where record_set_scan_concurrency_limit(0); record_set_scans_queued(0); record_set_scans_active(0); + #[cfg(test)] + if let Some(gate) = post_walk_gate { + let _ = gate.await; + } let first_err = first_err_mutex.lock().await.take(); + let has_set_failure = first_err.is_some(); let results = results_mutex.lock().await.clone(); let completed_all_sets = bucket_plan_complete && scanner_results_form_complete_snapshot(&results, &expected_sources); let result = finalize_nsscanner_result(&results, first_err); @@ -730,8 +752,19 @@ where let observed_cycle_floor = cache_cycle_floor.load(Ordering::Acquire); let required_cycle_floor = (observed_cycle_floor > want_cycle).then_some(observed_cycle_floor); let budget_elapsed = budget.budget_elapsed(); + // A mixed-plan tail can save its checkpoint without producing a root + // candidate. Do not infer progress from an arbitrary missing update, or + // let a successful set hide another set's worker/persistence failure. + let partial_bucket_progress = !budget_elapsed + && !ctx.is_cancelled() + && scanner_partial_progress_is_usable( + bucket_plan_complete, + scan_scope_matches, + bucket_scan_status, + has_set_failure, + !namespace_not_found_buckets.is_empty(), + ); let dirty_usage_status = dirty_usage_snapshot_status(&dirty_usage_snapshot); - let dirty_usage_current = dirty_usage_status == DirtyUsageSnapshotStatus::Current; let (activity_status, remote_publication_lease_targets) = scanner_cycle_activity_status(store, distributed, &activity_before).await; let all_bucket_names = all_buckets.iter().map(|bucket| bucket.name.clone()).collect::>(); @@ -776,14 +809,22 @@ where dirty_usage_status, activity_status, ); + // Certification is per bucket: the pending set is read after the + // guardrail verdict so a bucket written during the walk is reported in + // the same publication that certifies the others. + let pending_buckets = if cycle_status == ScannerCycleStatus::Complete { + pending_dirty_usage_buckets(&dirty_usage_snapshot.buckets) + } else { + ScannerCyclePendingBuckets::new() + }; let mut publication_expectation = None; - let observational_snapshot_published = if let Some(candidate) = completed_usage { - let (data_usage_info, expectation) = candidate.prepare(cycle_status); + let usage_update_emitted = if let Some(candidate) = completed_usage { + let (data_usage_info, expectation) = candidate.prepare(cycle_status, &pending_buckets); publication_expectation = expectation; if should_publish_observational_snapshot(cycle_status) { publish_observational_snapshot(&updates, data_usage_info).await? } else { - publish_usage_snapshot(&updates, cycle_status, data_usage_info).await? + publish_usage_snapshot(&updates, cycle_status, &pending_buckets, data_usage_info).await? } } else if !ctx.is_cancelled() && let Some((data_usage_info, _)) = observational_usage @@ -796,7 +837,7 @@ where result.is_ok(), structurally_complete_snapshot, budget_elapsed, - activity_status == ScannerCycleActivityStatus::Unchanged && dirty_usage_current, + activity_status == ScannerCycleActivityStatus::Unchanged, &dirty_usage_snapshot.buckets, &failed_buckets, ); @@ -815,7 +856,10 @@ where Ok(ScannerCycleResult::new(cycle_status, dirty_usage_clear) .with_publication_epoch(publication_epoch) .with_activity_digest(activity_digest) - .with_observational_snapshot_published(observational_snapshot_published) + .with_activity_baseline(activity_before) + .with_pending_dirty_usage_buckets(pending_buckets.len()) + .with_usage_update_emitted(usage_update_emitted) + .with_partial_bucket_progress(partial_bucket_progress) .with_remote_publication_lease_targets(remote_publication_lease_targets) .with_remote_dirty_usage_acknowledgements(remote_dirty_usage_acknowledgements) .with_distributed_segment_invalidation_evidence(distributed_segment_invalidation_evidence) diff --git a/crates/scanner/src/scanner_io/io_disk.rs b/crates/scanner/src/scanner_io/io_disk.rs index a77639c49..58d8dc36a 100644 --- a/crates/scanner/src/scanner_io/io_disk.rs +++ b/crates/scanner/src/scanner_io/io_disk.rs @@ -180,9 +180,9 @@ impl ScannerIODisk for Disk { .await .unwrap_or_else(|_| (BucketLifecycleConfiguration::default(), OffsetDateTime::now_utc())); - if lifecycle_config.has_active_rules("") { - cache.info.lifecycle = Some(Arc::new(lifecycle_config)); - } + // Persisted usage caches are not an authority for lifecycle policy. + // Clear their old snapshot when configuration is absent, disabled, or unreadable. + cache.info.lifecycle = lifecycle_config.has_active_rules("").then(|| Arc::new(lifecycle_config)); let (replication_config, _) = crate::storage_api::ecstore_with_background_disk_io(ctx.clone(), get_replication_config(&cache.info.name)) diff --git a/crates/scanner/src/scanner_io/publish_gate_tests.rs b/crates/scanner/src/scanner_io/publish_gate_tests.rs index 564ea2baf..c44a16a33 100644 --- a/crates/scanner/src/scanner_io/publish_gate_tests.rs +++ b/crates/scanner/src/scanner_io/publish_gate_tests.rs @@ -19,6 +19,41 @@ use rustfs_data_usage::{ReplicationAllStats, ReplicationTargetUsage, TierAccount const TEST_PLAN_DIGEST: DataUsageScanPlanDigest = DataUsageScanPlanDigest([7; 32]); const TEST_COVERAGE_DIGEST: DataUsageScanPlanDigest = DataUsageScanPlanDigest([6; 32]); +#[test] +fn partial_progress_requires_complete_scope_and_no_hard_failure() { + for plan_complete in [false, true] { + for scope_matches in [false, true] { + for set_failed in [false, true] { + for missing_namespace in [false, true] { + for status in [ + ScannerBucketScanStatus::Complete, + ScannerBucketScanStatus::Partial, + ScannerBucketScanStatus::Failed, + ScannerBucketScanStatus::NamespaceNotFound, + ] { + let expected = plan_complete + && scope_matches + && !set_failed + && !missing_namespace + && status == ScannerBucketScanStatus::Partial; + assert_eq!( + scanner_partial_progress_is_usable( + plan_complete, + scope_matches, + status, + set_failed, + missing_namespace + ), + expected, + "plan={plan_complete}, scope={scope_matches}, status={status:?}, set_failed={set_failed}, missing_namespace={missing_namespace}" + ); + } + } + } + } + } +} + #[test] fn scanner_bucket_inventory_requires_exact_unique_set_union() { let first = BucketInfo { @@ -899,12 +934,14 @@ fn completed_data_usage_info_publishes_confirmed_empty_namespace() { } #[test] -fn complete_usage_candidate_with_changed_generation_is_superseded() { +fn complete_usage_candidate_with_changed_generation_is_certified_per_bucket() { let all_buckets = vec!["bucket".to_string()]; let completed_set = completed_root_cache("bucket", 2, 10, DataUsageCacheSource::new(0, 0)); let completed_usage = completed_data_usage_info_for_test(&[completed_set], &all_buckets, false, false); assert!(completed_usage.is_some()); + // A moved dirty generation shades buckets for the next cycle; only a + // guardrail veto supersedes a structurally complete candidate. assert_eq!( classify_nsscanner_cycle( completed_usage.is_some(), @@ -914,6 +951,17 @@ fn complete_usage_candidate_with_changed_generation_is_superseded() { DirtyUsageSnapshotStatus::Changed, ScannerCycleActivityStatus::Unchanged, ), + ScannerCycleStatus::Complete + ); + assert_eq!( + classify_nsscanner_cycle( + completed_usage.is_some(), + false, + false, + ScannerBucketScanStatus::Complete, + DirtyUsageSnapshotStatus::Changed, + ScannerCycleActivityStatus::Changed, + ), ScannerCycleStatus::Superseded ); } @@ -1528,3 +1576,55 @@ async fn requeued_bucket_remains_pending_for_another_worker() { assert!(!complete.is_cancelled()); assert_eq!(rx.recv().await.expect("requeued bucket").name, "photos"); } + +#[test] +fn replication_execution_digest_is_canonical_and_tracks_statistics() { + use rustfs_data_usage::{ReplicationAllStats, ReplicationTargetUsage}; + let execution = DataUsageScanPlanDigest([9; 32]); + let mut cache = DataUsageCache::default(); + assert_eq!(super::cache::scanner_replication_execution_digest(execution, &cache), execution); + let first = ReplicationTargetUsage { + replicated_size: 7, + replicated_count: 1, + ..Default::default() + }; + let second = ReplicationTargetUsage { + pending_size: 3, + pending_count: 1, + ..Default::default() + }; + cache.cache.insert( + "photos".to_string(), + DataUsageEntry { + replication_stats: Some(ReplicationAllStats { + replica_size: 0, + replica_count: 0, + targets: HashMap::from([("arn:a".to_string(), first.clone()), ("arn:b".to_string(), second.clone())]), + }), + ..Default::default() + }, + ); + let digest = super::cache::scanner_replication_execution_digest(execution, &cache); + assert_ne!(digest, execution); + let stats = cache + .cache + .get_mut("photos") + .expect("cached bucket") + .replication_stats + .as_mut() + .expect("replication stats"); + stats.targets = HashMap::from([("arn:b".to_string(), second), ("arn:a".to_string(), first)]); + assert_eq!(super::cache::scanner_replication_execution_digest(execution, &cache), digest); + cache + .cache + .get_mut("photos") + .expect("cached bucket") + .replication_stats + .as_mut() + .expect("replication stats") + .targets + .get_mut("arn:a") + .expect("target stats") + .failed_count = 1; + assert_ne!(super::cache::scanner_replication_execution_digest(execution, &cache), digest); +} diff --git a/crates/scanner/src/scanner_io/tests.rs b/crates/scanner/src/scanner_io/tests.rs index 05f51fbe3..a9cd0cd21 100644 --- a/crates/scanner/src/scanner_io/tests.rs +++ b/crates/scanner/src/scanner_io/tests.rs @@ -44,6 +44,7 @@ use temp_env::with_var; use time::OffsetDateTime; use uuid::Uuid; +mod per_bucket_certification; mod scoped_entry_fallback; mod service_cohort; mod usage_expiration; @@ -717,7 +718,7 @@ async fn checkpoint_uses_global_cycle_fence_with_set_scoped_cache_revisions() { &mut revisions, ) .await, - ScannerCheckpointPersistResult::Failed(StorageError::PreconditionFailed) + ScannerCheckpointPersistResult::RetryBucket(StorageError::PreconditionFailed) ), "matching global fence must not authorize stale cache CAS revisions" ); @@ -734,6 +735,175 @@ async fn checkpoint_uses_global_cycle_fence_with_set_scoped_cache_revisions() { assert!(mismatched_sets > 0, "multi-pool fixture must expose a set-scoped cycle fence mismatch"); } +#[derive(Debug)] +struct CheckpointFaultStore { + set: Arc, + fail_put: AtomicBool, + fail_revision_read: AtomicBool, +} + +#[async_trait::async_trait] +impl crate::storage_api::owner::ObjectIO for CheckpointFaultStore { + type Error = crate::EcstoreError; + type RangeSpec = crate::storage_api::owner::HTTPRangeSpec; + type HeaderMap = http::HeaderMap; + type ObjectOptions = crate::ScannerObjectOptions; + type ObjectInfo = crate::ScannerObjectInfo; + type GetObjectReader = crate::ScannerGetObjectReader; + type PutObjectReader = crate::ScannerPutObjReader; + + async fn get_object_reader( + &self, + bucket: &str, + object: &str, + range: Option, + headers: Self::HeaderMap, + opts: &Self::ObjectOptions, + ) -> crate::EcstoreResult { + if self.fail_revision_read.swap(false, Ordering::SeqCst) { + return Err(StorageError::Io(std::io::Error::new( + std::io::ErrorKind::Interrupted, + "injected revision read failure", + ))); + } + self.set.get_object_reader(bucket, object, range, headers, opts).await + } + + async fn put_object( + &self, + bucket: &str, + object: &str, + data: &mut Self::PutObjectReader, + opts: &Self::ObjectOptions, + ) -> crate::EcstoreResult { + if self.fail_put.swap(false, Ordering::SeqCst) { + return Err(StorageError::Lock(LockError::Timeout { + resource: object.to_string(), + timeout: Duration::from_secs(5), + })); + } + self.set.put_object(bucket, object, data, opts).await + } +} + +#[async_trait::async_trait] +impl crate::ScannerConfigObjectDelete for CheckpointFaultStore { + async fn delete_config_object( + &self, + bucket: &str, + object: &str, + opts: ScannerObjectOptions, + ) -> crate::EcstoreResult { + crate::ScannerConfigObjectDelete::delete_config_object(self.set.as_ref(), bucket, object, opts).await + } + + async fn scanner_data_usage_publication_admission(&self) -> Option { + crate::ScannerConfigObjectDelete::scanner_data_usage_publication_admission(self.set.as_ref()).await + } +} + +#[tokio::test] +#[serial] +async fn checkpoint_retries_next_snapshot_after_lock_failure_and_reloads_after_revision_failure() { + let (_temp_dir, fence_store) = setup_local_scanner_store(1, 4).await; + let cycle = 8457; + let leader_epoch = 19; + save_config( + fence_store.clone(), + DATA_USAGE_BLOOM_NAME_PATH.as_str(), + crate::scanner::encode_scanner_cycle_fence_for_test(cycle, leader_epoch), + ) + .await + .expect("seed global cycle fence"); + let set = fence_store.pools[0].disk_set[0].clone(); + let publication_epoch = scanner_publication_epoch(set.clone()).await.expect("read publication epoch"); + let cache_name = "checkpoint-recovery/.usage-cache.bin"; + DataUsageCache::default() + .save(set.clone(), cache_name) + .await + .expect("seed cache"); + let mut revisions = DataUsageCache::read_revisions(set.clone(), cache_name) + .await + .expect("read initial revisions"); + let initial_revisions = revisions.clone(); + let store = Arc::new(CheckpointFaultStore { + set: set.clone(), + fail_put: AtomicBool::new(true), + fail_revision_read: AtomicBool::new(false), + }); + let ctx = CancellationToken::new(); + let context = || ScannerCheckpointPersistContext { + ctx: &ctx, + expected_publication_epoch: publication_epoch, + cycle, + leader_epoch, + }; + let mut checkpoint = DataUsageCache::default(); + checkpoint.info.name = "checkpoint-recovery".to_string(); + checkpoint.info.source = Some(DataUsageCacheSource::new(0, 0)); + checkpoint.info.scan_checkpoint = Some(crate::DataUsageScanCheckpoint::new( + "checkpoint-recovery/first".to_string(), + crate::DataUsageScanCheckpointReason::Objects, + )); + assert!(matches!( + persist_scanner_checkpoint(store.clone(), fence_store.clone(), context(), cache_name, &checkpoint, &mut revisions).await, + ScannerCheckpointPersistResult::Skipped + )); + assert_eq!(revisions, initial_revisions); + assert_eq!( + DataUsageCache::read_revisions(set.clone(), cache_name) + .await + .expect("read after failed save"), + initial_revisions + ); + + checkpoint + .info + .scan_checkpoint + .as_mut() + .expect("checkpoint cursor") + .resume_after = "checkpoint-recovery/second".to_string(); + assert!(matches!( + persist_scanner_checkpoint(store.clone(), fence_store.clone(), context(), cache_name, &checkpoint, &mut revisions).await, + ScannerCheckpointPersistResult::Saved + )); + assert_ne!(revisions, initial_revisions); + let mut loaded = DataUsageCache::default(); + loaded.load(set.clone(), cache_name).await.expect("load recovered checkpoint"); + assert_eq!( + loaded.info.scan_checkpoint, checkpoint.info.scan_checkpoint, + "next snapshot must become durable" + ); + + let saved_revisions = revisions.clone(); + checkpoint + .info + .scan_checkpoint + .as_mut() + .expect("checkpoint cursor") + .resume_after = "checkpoint-recovery/third".to_string(); + store.fail_revision_read.store(true, Ordering::SeqCst); + assert!(matches!( + persist_scanner_checkpoint(store, fence_store, context(), cache_name, &checkpoint, &mut revisions).await, + ScannerCheckpointPersistResult::RetryBucket(StorageError::Io(_)) + )); + assert_eq!(revisions, saved_revisions, "indeterminate revision refresh must not rebase the writer"); + loaded + .load(set.clone(), cache_name) + .await + .expect("load committed checkpoint after refresh failure"); + assert_eq!( + loaded.info.scan_checkpoint, checkpoint.info.scan_checkpoint, + "PUT completed before revision refresh failed" + ); + assert_ne!( + DataUsageCache::read_revisions(set, cache_name) + .await + .expect("read committed revisions"), + saved_revisions + ); +} + #[derive(Debug)] struct CheckpointFenceStore { before: Vec, @@ -1149,6 +1319,7 @@ async fn scoped_scan_production_entry_preserves_deep_and_full_maintenance_work() observed_usage_candidate: None, requires_full_scan, resolved_scope_observer: Some(observer), + post_walk_gate: None, service_cohort: None, }, ), @@ -1239,6 +1410,7 @@ async fn scoped_scan_same_cycle_maintenance_rewalks_after_root_delivery_failure( observed_usage_candidate: None, requires_full_scan: false, resolved_scope_observer: None, + post_walk_gate: None, service_cohort: None, }, ), @@ -1292,6 +1464,7 @@ async fn scoped_scan_same_cycle_maintenance_rewalks_after_root_delivery_failure( observed_usage_candidate: None, requires_full_scan, resolved_scope_observer: None, + post_walk_gate: None, service_cohort: None, }, ), @@ -1335,17 +1508,20 @@ async fn data_usage_publish_rejects_a_second_terminal_update_without_blocking() ), ] { let (updates, mut receiver) = mpsc::channel(1); + let pending = ScannerCyclePendingBuckets::new(); assert!( - publish_usage_snapshot(&updates, status, data_usage_info) + publish_usage_snapshot(&updates, status, &pending, data_usage_info) .await .expect("first terminal update should be accepted") ); - let err = - tokio::time::timeout(Duration::from_secs(1), publish_usage_snapshot(&updates, status, DataUsageInfo::default())) - .await - .expect("a full terminal update must fail without waiting") - .expect_err("a second terminal update must be rejected"); + let err = tokio::time::timeout( + Duration::from_secs(1), + publish_usage_snapshot(&updates, status, &pending, DataUsageInfo::default()), + ) + .await + .expect("a full terminal update must fail without waiting") + .expect_err("a second terminal update must be rejected"); assert!(err.to_string().contains("already queued")); assert!(receiver.try_recv().is_ok(), "the first terminal update must remain owned by the receiver"); assert!(receiver.try_recv().is_err(), "the rejected second update must not enter the channel"); @@ -2222,7 +2398,7 @@ fn complete_segment_reuse_baseline( } #[test] -fn scoped_scan_requires_a_converged_complete_baseline_with_exact_set_provenance() { +fn scoped_scan_requires_a_complete_authoritative_baseline_with_exact_set_provenance() { let source = DataUsageCacheSource::new(1, 2); let expected_sources = HashSet::from([source]); let scan_plan_digest = DataUsageScanPlanDigest([9; 32]); @@ -2240,19 +2416,41 @@ fn scoped_scan_requires_a_converged_complete_baseline_with_exact_set_provenance( Some(scan_plan_digest) ); - let mut incomplete = serde_json::from_slice::(&baseline).expect("test baseline should decode"); - incomplete.usage_snapshot_converged = Some(false); - let incomplete = bytes::Bytes::from(serde_json::to_vec(&incomplete).expect("test baseline should encode")); + // A complete cycle that left buckets pending publishes unconverged on + // the authoritative object and is still the structural baseline: its + // pending buckets stay dirty and are rescanned, the rest reuse. + let mut pending = serde_json::from_slice::(&baseline).expect("test baseline should decode"); + pending.usage_snapshot_converged = Some(false); + pending.usage_snapshot_pending_buckets = vec!["bucket-a".to_string()]; + pending.usage_snapshot_pending_bucket_count = 1; + let pending = bytes::Bytes::from(serde_json::to_vec(&pending).expect("test baseline should encode")); assert_eq!( complete_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof { - authoritative_data: Some(&incomplete), + authoritative_data: Some(&pending), observed_candidate_data: None, expected_sources: &expected_sources, leader_epoch: 11, want_cycle: 8, scan_plan_digest, }), - None + Some(scan_plan_digest) + ); + + let mut observational = serde_json::from_slice::(&baseline).expect("test baseline should decode"); + observational.usage_snapshot_converged = Some(false); + observational.usage_snapshot_observational = true; + let observational = bytes::Bytes::from(serde_json::to_vec(&observational).expect("test baseline should encode")); + assert_eq!( + complete_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof { + authoritative_data: Some(&observational), + observed_candidate_data: None, + expected_sources: &expected_sources, + leader_epoch: 11, + want_cycle: 8, + scan_plan_digest, + }), + None, + "an observational snapshot in the authoritative slot is not a baseline" ); let mut wrong_provenance = serde_json::from_slice::(&baseline).expect("test baseline should decode"); @@ -2359,10 +2557,18 @@ fn scoped_scan_accepts_only_a_complete_observation_tied_to_the_authoritative_bas observation_of_nonconverged_authoritative.usage_snapshot_set_states[0].scanner_cycle = Some(8); observation_of_nonconverged_authoritative.usage_snapshot_authoritative_baseline = Some(nonconverged_authoritative.snapshot_identity()); + observation_of_nonconverged_authoritative.usage_snapshot_observational = true; + let mut observational_in_authoritative_slot = nonconverged_authoritative.clone(); + observational_in_authoritative_slot.usage_snapshot_observational = true; let nonconverged_authoritative = bytes::Bytes::from(serde_json::to_vec(&nonconverged_authoritative).expect("nonconverged baseline should encode")); + let observational_in_authoritative_slot = bytes::Bytes::from( + serde_json::to_vec(&observational_in_authoritative_slot).expect("observational baseline should encode"), + ); let observation_of_nonconverged_authoritative = bytes::Bytes::from(serde_json::to_vec(&observation_of_nonconverged_authoritative).expect("observation should encode")); + // The unconverged authoritative snapshot is itself the baseline, so the + // observation chained on it is not consulted. assert_eq!( complete_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof { authoritative_data: Some(&nonconverged_authoritative), @@ -2372,6 +2578,19 @@ fn scoped_scan_accepts_only_a_complete_observation_tied_to_the_authoritative_bas want_cycle: 9, scan_plan_digest, }), + Some(scan_plan_digest) + ); + // An observational snapshot that somehow sits in the authoritative slot + // is neither a baseline nor an anchor for a chained observation. + assert_eq!( + complete_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof { + authoritative_data: Some(&observational_in_authoritative_slot), + observed_candidate_data: Some(&observation_of_nonconverged_authoritative), + expected_sources: &expected_sources, + leader_epoch: 11, + want_cycle: 9, + scan_plan_digest, + }), None ); } @@ -3414,6 +3633,8 @@ fn scanner_cycle_status_requires_a_clean_complete_snapshot() { ), ScannerCycleStatus::Complete ); + // Dirty-usage movement during the walk shades buckets; it is certified + // per bucket and no longer supersedes a walk the guardrail accepted. assert_eq!( classify_nsscanner_cycle( true, @@ -3423,19 +3644,22 @@ fn scanner_cycle_status_requires_a_clean_complete_snapshot() { DirtyUsageSnapshotStatus::Changed, ScannerCycleActivityStatus::Unchanged, ), - ScannerCycleStatus::Superseded - ); - assert_eq!( - classify_nsscanner_cycle( - true, - false, - false, - ScannerBucketScanStatus::Complete, - DirtyUsageSnapshotStatus::Current, - ScannerCycleActivityStatus::Changed, - ), - ScannerCycleStatus::Superseded + ScannerCycleStatus::Complete ); + for dirty_usage_status in [DirtyUsageSnapshotStatus::Current, DirtyUsageSnapshotStatus::Changed] { + assert_eq!( + classify_nsscanner_cycle( + true, + false, + false, + ScannerBucketScanStatus::Complete, + dirty_usage_status, + ScannerCycleActivityStatus::Changed, + ), + ScannerCycleStatus::Superseded, + "a guardrail veto supersedes regardless of dirty state ({dirty_usage_status:?})" + ); + } assert_eq!( classify_nsscanner_cycle( true, @@ -3508,11 +3732,26 @@ fn checkpoint_fixture_superseded_is_distinct_from_partial_and_cancel() { cancelled, bucket, DirtyUsageSnapshotStatus::Changed, - ScannerCycleActivityStatus::Unchanged + ScannerCycleActivityStatus::Changed ), expected, ); } + // Budget and cancellation stay `Incomplete` even when the guardrail + // passed and only dirty usage moved. + for (budget, cancelled) in [(true, false), (false, true)] { + assert_eq!( + classify_nsscanner_cycle( + true, + budget, + cancelled, + ScannerBucketScanStatus::Complete, + DirtyUsageSnapshotStatus::Changed, + ScannerCycleActivityStatus::Unchanged + ), + ScannerCycleStatus::Incomplete, + ); + } } #[test] @@ -3545,20 +3784,28 @@ fn unverified_activity_defers_partial_and_floor_cycles() { #[tokio::test] async fn structurally_complete_superseded_cycles_publish_without_claiming_convergence() { - let (updates, mut receiver) = mpsc::channel(2); + let (updates, mut receiver) = mpsc::channel(3); + let no_pending = ScannerCyclePendingBuckets::new(); + let pending = ScannerCyclePendingBuckets::from(["hot-b".to_string(), "hot-a".to_string()]); assert!( - publish_usage_snapshot(&updates, ScannerCycleStatus::Complete, DataUsageInfo::default()) + publish_usage_snapshot(&updates, ScannerCycleStatus::Complete, &no_pending, DataUsageInfo::default()) .await .expect("complete snapshot publication should succeed") ); assert!( - publish_usage_snapshot(&updates, ScannerCycleStatus::Superseded, DataUsageInfo::default()) + publish_usage_snapshot(&updates, ScannerCycleStatus::Complete, &pending, DataUsageInfo::default()) + .await + .expect("complete snapshot with pending buckets should publish") + ); + // A vetoed cycle never carries a pending list even when one is supplied. + assert!( + publish_usage_snapshot(&updates, ScannerCycleStatus::Superseded, &pending, DataUsageInfo::default()) .await .expect("superseded snapshot publication should succeed") ); assert!( - !publish_usage_snapshot(&updates, ScannerCycleStatus::Incomplete, DataUsageInfo::default()) + !publish_usage_snapshot(&updates, ScannerCycleStatus::Incomplete, &no_pending, DataUsageInfo::default()) .await .expect("incomplete snapshot suppression should succeed") ); @@ -3566,28 +3813,33 @@ async fn structurally_complete_superseded_cycles_publish_without_claiming_conver !publish_usage_snapshot( &updates, ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable), + &no_pending, DataUsageInfo::default(), ) .await .expect("unverified activity suppression should succeed") ); - assert_eq!( - receiver - .recv() - .await - .expect("complete update should be sent") - .usage_snapshot_converged, - Some(true) - ); - assert_eq!( - receiver - .recv() - .await - .expect("superseded update should be sent") - .usage_snapshot_converged, - Some(false) - ); + let converged = receiver.recv().await.expect("complete update should be sent"); + assert_eq!(converged.usage_snapshot_converged, Some(true)); + assert!(!converged.usage_snapshot_observational); + assert!(converged.usage_snapshot_pending_buckets.is_empty()); + assert_eq!(converged.usage_snapshot_pending_bucket_count, 0); + + let certified = receiver + .recv() + .await + .expect("complete update with pending buckets should be sent"); + assert_eq!(certified.usage_snapshot_converged, Some(false)); + assert!(!certified.usage_snapshot_observational, "pending buckets keep the authoritative route"); + assert_eq!(certified.usage_snapshot_pending_buckets, vec!["hot-a".to_string(), "hot-b".to_string()]); + assert_eq!(certified.usage_snapshot_pending_bucket_count, 2); + + let superseded = receiver.recv().await.expect("superseded update should be sent"); + assert_eq!(superseded.usage_snapshot_converged, Some(false)); + assert!(superseded.usage_snapshot_observational); + assert!(superseded.usage_snapshot_pending_buckets.is_empty()); + assert_eq!(superseded.usage_snapshot_pending_bucket_count, 0); } #[tokio::test] @@ -3790,39 +4042,63 @@ fn scanner_concurrency_limit_handles_no_available_work() { #[test] #[serial] -fn scanner_concurrency_limit_yields_to_foreground_reads() { +fn scanner_concurrency_limit_is_stable_across_foreground_read_transitions() { crate::reset_foreground_read_activity_for_test(); crate::workload_admission::clear_scanner_workload_admission_snapshot_provider_for_test(); - crate::set_foreground_read_activity(8); - assert_eq!(scanner_concurrency_limit(0, 4), 1); - assert_eq!(scanner_concurrency_limit(3, 4), 1); + for foreground_activity in [0, 1, 8, 0] { + crate::set_foreground_read_activity(foreground_activity); + assert_eq!(scanner_concurrency_limit(0, 4), 4); + assert_eq!(scanner_concurrency_limit(3, 4), 3); + assert_eq!(scanner_concurrency_limit(1, 4), 1); + assert_eq!(scanner_concurrency_limit(8, 0), 0); + assert_eq!(scanner_budgeted_concurrency_limit(scanner_concurrency_limit(3, 4), true), 1); + assert_eq!(scanner_budgeted_concurrency_limit(scanner_concurrency_limit(3, 4), false), 3); + } crate::reset_foreground_read_activity_for_test(); } #[test] #[serial] -fn scanner_concurrency_limit_yields_to_shared_foreground_pressure() { +fn scanner_concurrency_limit_is_stable_across_shared_foreground_pressure() { crate::reset_foreground_read_activity_for_test(); crate::workload_admission::clear_scanner_workload_admission_snapshot_provider_for_test(); - install_scanner_workload_provider(WorkloadAdmissionRegistrySnapshot::new(vec![ - WorkloadAdmissionSnapshot::new(WorkloadClass::ForegroundWrite, AdmissionState::Open).with_counts(Some(2), None, Some(16)), - ])); - - assert_eq!(scanner_concurrency_limit(0, 4), 1); - assert_eq!(scanner_concurrency_limit(3, 4), 1); - + for (state, active) in [ + (AdmissionState::Open, Some(0)), + (AdmissionState::Open, Some(2)), + (AdmissionState::Saturated, None), + (AdmissionState::Open, Some(0)), + ] { + install_scanner_workload_provider(WorkloadAdmissionRegistrySnapshot::new(vec![ + WorkloadAdmissionSnapshot::new(WorkloadClass::ForegroundWrite, state).with_counts(active, None, Some(16)), + ])); + assert_eq!(scanner_concurrency_limit(0, 4), 4); + assert_eq!(scanner_concurrency_limit(3, 4), 3); + } crate::workload_admission::clear_scanner_workload_admission_snapshot_provider_for_test(); } #[test] #[serial] -fn scanner_concurrency_limit_yields_to_streaming_reads() { +fn scanner_dispatch_limits_preserve_configured_caps_during_streaming_reads() { crate::reset_foreground_read_activity_for_test(); crate::workload_admission::clear_scanner_workload_admission_snapshot_provider_for_test(); - let _guard = crate::ForegroundReadGuard::new(); - - assert_eq!(scanner_concurrency_limit(0, 4), 1); - assert_eq!(scanner_concurrency_limit(3, 4), 1); + temp_env::with_vars( + [ + (ENV_SCANNER_MAX_CONCURRENT_SET_SCANS, Some("3")), + (ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS, Some("2")), + ], + || { + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + for streaming in [false, true, false] { + let _guard = streaming.then(crate::ForegroundReadGuard::new); + assert_eq!(scanner_max_concurrent_set_scans(4), 3); + assert_eq!(scanner_max_concurrent_disk_scans(4), 2); + assert_eq!(scanner_budgeted_concurrency_limit(scanner_max_concurrent_set_scans(4), true), 1); + assert_eq!(scanner_budgeted_concurrency_limit(scanner_max_concurrent_disk_scans(4), true), 1); + } + }, + ); + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); } #[test] @@ -4200,6 +4476,53 @@ fn cache_root_entry_info_rejects_missing_or_dangling_roots() { assert!(cache_root_entry_info(&compacted_with_child).is_err()); } +#[tokio::test] +async fn completed_bucket_results_count_distinct_current_roots_even_after_cancellation() { + for cancelled in [false, true] { + let ctx = CancellationToken::new(); + if cancelled { + ctx.cancel(); + } + let mut cache = DataUsageCache { + info: DataUsageCacheInfo { + name: DATA_USAGE_ROOT.to_string(), + tier_registry_generation: Some(7), + ..Default::default() + }, + ..Default::default() + }; + cache.replace( + "missing", + DATA_USAGE_ROOT, + DataUsageEntry { + size: 3, + ..Default::default() + }, + ); + let (tx, rx) = mpsc::channel(3); + for (bucket, generation, size) in [("complete", 7, 1), ("complete", 7, 2), ("missing", 8, 11)] { + tx.send(DataUsageEntryInfo { + name: bucket.to_string(), + parent: DATA_USAGE_ROOT.to_string(), + entry: DataUsageEntry { + size, + ..Default::default() + }, + bucket_incarnation: None, + tier_registry_generation: Some(generation), + }) + .await + .expect("fixture result channel"); + } + drop(tx); + let (cache, complete) = super::io_cache::collect_completed_bucket_results(ctx, cache, rx).await; + assert_eq!(complete, HashSet::from(["complete".to_string()])); + assert!(!should_publish_completed_snapshot(complete.len(), 2, false, false)); + assert_eq!(cache.find("complete").expect("applied root").size, 2); + assert_eq!(cache.find("missing").expect("retained old root").size, 3); + } +} + #[test] fn apply_bucket_result_to_cache_updates_bucket_entry() { let mut cache = DataUsageCache { @@ -4220,10 +4543,12 @@ fn apply_bucket_result_to_cache_updates_bucket_entry() { ); let update_time = SystemTime::now(); - assert!(apply_bucket_result_to_cache( + let bucket = "bucket".to_string(); + let bucket_name_ptr = bucket.as_ptr(); + let applied_bucket = apply_bucket_result_to_cache( &mut cache, DataUsageEntryInfo { - name: "bucket".to_string(), + name: bucket, parent: DATA_USAGE_ROOT.to_string(), entry: DataUsageEntry { size: 10, @@ -4234,7 +4559,10 @@ fn apply_bucket_result_to_cache_updates_bucket_entry() { tier_registry_generation: None, }, update_time, - )); + ) + .expect("accepted result returns its bucket name"); + assert_eq!(applied_bucket, "bucket"); + assert_eq!(applied_bucket.as_ptr(), bucket_name_ptr, "reuse the result's name allocation"); assert_eq!(cache.info.last_update, Some(update_time)); let entry = cache.find("bucket").expect("bucket entry should remain present"); @@ -4277,7 +4605,357 @@ fn apply_bucket_result_to_cache_rejects_a_different_tier_generation() { SystemTime::now(), ); - assert!(!applied); + assert!(applied.is_none()); assert_eq!(cache.find("bucket").map(|entry| entry.size), Some(3)); assert!(cache.info.last_update.is_none()); } + +#[tokio::test] +#[serial] +async fn replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries() { + use crate::storage_api::owner::ObjectOperations as _; + for same_cycle in [false, true] { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + clear_dirty_usage_buckets_for_tests(); + let bucket = "status-bucket"; + let object = "cold/object"; + let set = store.pools[0].disk_set[0].clone(); + store + .make_bucket( + bucket, + &MakeBucketOptions { + versioning_enabled: true, + ..Default::default() + }, + ) + .await + .expect("create status bucket"); + let mut reader = ScannerPutObjReader::from_vec(b"initial".to_vec()); + set.put_object( + bucket, + object, + &mut reader, + &ScannerObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect("persist initial object"); + wait_for_namespace_commit_tails(store.as_ref()).await; + + let ctx = CancellationToken::new(); + let (updates, mut receiver) = mpsc::channel(1); + let initial = nsscanner_with_storage_status_scoped( + store.as_ref(), + ScannerCycleRequest { + budget: ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()), + ctx, + updates, + want_cycle: 1, + leader_epoch: 11, + scan_mode: HealScanMode::Normal, + scan_scope: ScannerBucketScanScope::default(), + persisted_usage_baseline: None, + observed_usage_candidate: None, + requires_full_scan: true, + service_cohort: None, + resolved_scope_observer: None, + post_walk_gate: None, + }, + ) + .await + .expect("initial full usage scan"); + assert_eq!(initial.status, ScannerCycleStatus::Complete); + let baseline = receiver.recv().await.expect("initial usage publication"); + assert!(baseline.buckets_usage[bucket].replication_info.is_empty()); + + let cache_name = path_join_buf(&[bucket, DATA_USAGE_CACHE_NAME]); + let mut cache = DataUsageCache::default(); + cache + .load(set.clone(), &cache_name) + .await + .expect("load complete bucket cache"); + assert!(cache.info.snapshot_complete); + // Keep an accounting context with no remote work in this local fixture. + let replication: ReplicationConfiguration = serde_json::from_value(serde_json::json!({ + "role": "", + "rules": [{ + "destination": { "bucket": "arn:test" }, + "id": "status", + "prefix": "", + "priority": 1, + "status": "Enabled" + }] + })) + .expect("decode the local replication accounting fixture"); + cache.info.replication = Some(Arc::new(ReplicationConfig::new( + Some(replication), + Some(crate::storage_api::EcstoreBucketTargets::default()), + ))); + // A complete compacted prefix is eligible for normal usage sampling. + let mut compacted = cache.checked_flatten(bucket).expect("complete cached usage"); + compacted.children.clear(); + compacted.compacted = true; + cache.cache.clear(); + cache.replace(bucket, "", DataUsageEntry::default()); + cache.replace("status-bucket/cold", bucket, compacted); + cache + .save(set.clone(), &cache_name) + .await + .expect("persist compacted prefix fixture"); + let cycle = (2..18) + .find(|cycle| { + !rustfs_data_usage::hash_path("status-bucket/cold") + .mod_(*cycle, crate::scanner_folder::data_usage_update_dir_cycles()) + }) + .expect("choose a cycle outside the compacted prefix sampling window"); + + let mut metadata = HashMap::new(); + rustfs_utils::http::metadata_compat::insert_str( + &mut metadata, + rustfs_utils::http::metadata_compat::SUFFIX_REPLICATION_STATUS, + "arn:test=COMPLETED;".to_string(), + ); + let updated = set + .put_object_metadata( + bucket, + object, + &ScannerObjectOptions { + eval_metadata: Some(metadata), + versioned: true, + ..Default::default() + }, + ) + .await + .expect("persist replication-only metadata update"); + assert_eq!(updated.replication_status.as_str(), "COMPLETED"); + assert!(updated.version_id.is_some_and(|version| !version.is_nil())); + let persisted = set + .get_object_info( + bucket, + object, + &ScannerObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect("read back status writeback"); + assert_eq!(persisted.replication_status.as_str(), "COMPLETED"); + let namespace_generation = dirty_usage_generation(); + record_status_class_mutation( + bucket, + object, + crate::segment_invalidation::SegmentInvalidationProducerIdentity::Replication, + ); + assert_eq!(dirty_usage_generation(), namespace_generation); + + let ctx = CancellationToken::new(); + let (updates, mut receiver) = mpsc::channel(1); + let result = nsscanner_with_storage_status_scoped( + store.as_ref(), + ScannerCycleRequest { + budget: ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()), + ctx, + updates, + want_cycle: if same_cycle { 1 } else { u64::from(cycle) }, + leader_epoch: 11, + scan_mode: HealScanMode::Normal, + scan_scope: ScannerBucketScanScope::default(), + persisted_usage_baseline: None, + observed_usage_candidate: None, + requires_full_scan: true, + service_cohort: None, + resolved_scope_observer: None, + post_walk_gate: None, + }, + ) + .await + .expect("normal scan must refresh the status bucket"); + assert_eq!(result.status, ScannerCycleStatus::Complete); + let refreshed = receiver.recv().await.expect("refreshed usage publication"); + assert_eq!(refreshed.buckets_usage[bucket].objects_count, 1); + assert_eq!(refreshed.buckets_usage[bucket].size, 7); + assert_eq!( + refreshed.buckets_usage[bucket] + .replication_info + .get("arn:test") + .map_or(0, |target| target.replicated_size), + 7, + "a replication maintenance walk must refresh compacted statistics before publishing" + ); + clear_dirty_usage_buckets_for_tests(); + } +} + +pub(super) struct CoordinatorBucketScanProbe { + active: AtomicUsize, + peak: AtomicUsize, + hold: Duration, +} + +static COORDINATOR_BUCKET_SCAN_PROBE: std::sync::Mutex>> = std::sync::Mutex::new(None); + +/// Called by the coordinator walker loop once a bucket scan is admitted on +/// its disk. The hook is absent from non-test builds and a no-op until a test +/// installs a probe; the probe then holds the walker so overlapping +/// admissions become observable as a peak. +pub(super) async fn observe_coordinator_bucket_scan_admitted(_bucket: &str) { + let probe = COORDINATOR_BUCKET_SCAN_PROBE + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone(); + let Some(probe) = probe else { return }; + let active = probe.active.fetch_add(1, Ordering::SeqCst) + 1; + probe.peak.fetch_max(active, Ordering::SeqCst); + tokio::time::sleep(probe.hold).await; + probe.active.fetch_sub(1, Ordering::SeqCst); +} + +fn install_coordinator_bucket_scan_probe(hold: Duration) -> Arc { + let probe = Arc::new(CoordinatorBucketScanProbe { + active: AtomicUsize::new(0), + peak: AtomicUsize::new(0), + hold, + }); + *COORDINATOR_BUCKET_SCAN_PROBE + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(probe.clone()); + probe +} + +fn clear_coordinator_bucket_scan_probe() { + *COORDINATOR_BUCKET_SCAN_PROBE + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) = None; +} + +#[test] +fn namespace_scanner_walkers_multiply_local_disks_and_share_their_admission() { + let server_epoch = Uuid::from_u128(7); + let workers = vec![ + ("local-a", NamespaceScannerWorkerMode::Coordinator), + ("local-b", NamespaceScannerWorkerMode::Coordinator), + ("legacy-peer", NamespaceScannerWorkerMode::Coordinator), + ("v4-peer", NamespaceScannerWorkerMode::RemoteV4(server_epoch)), + ]; + let is_local = |disk: &&str| disk.starts_with("local"); + + let walkers = namespace_scanner_walkers(workers.clone(), 3, is_local); + assert_eq!(walkers.len(), 2 * 3 + 2, "two local disks fan out, both remote disks keep one walker"); + let local_a: Vec<_> = walkers.iter().filter(|walker| walker.disk == "local-a").collect(); + assert_eq!(local_a.len(), 3); + let share = local_a[0] + .local_admission + .as_ref() + .expect("local walkers carry an admission share"); + assert!( + local_a + .iter() + .all(|walker| walker.local_admission.as_ref().is_some_and(|other| Arc::ptr_eq(other, share))), + "walkers of one disk share one admission" + ); + let local_b = walkers + .iter() + .find(|walker| walker.disk == "local-b") + .expect("local-b walker"); + assert!( + !Arc::ptr_eq(local_b.local_admission.as_ref().expect("local-b share"), share), + "each disk owns its own admission share" + ); + assert!( + walkers + .iter() + .filter(|walker| !is_local(&walker.disk)) + .all(|walker| walker.local_admission.is_none()), + "remote disks admit through the peer, not a local share" + ); + assert_eq!( + walkers + .iter() + .filter(|walker| walker.mode == NamespaceScannerWorkerMode::RemoteV4(server_epoch)) + .count(), + 1 + ); + // The walker count, not the disk count, is the available work the + // configured cap applies to; the disk count stays the floor. + assert_eq!(scanner_concurrency_limit(0, walkers.len()), 8); + assert_eq!(scanner_concurrency_limit(4, walkers.len()), 4); + for factor in [0, 1] { + let walkers = namespace_scanner_walkers(workers.clone(), factor, is_local); + assert_eq!(walkers.len(), workers.len(), "factor {factor} keeps one walker per disk"); + assert_eq!(scanner_concurrency_limit(0, walkers.len()), workers.len()); + } +} + +async fn run_full_cycle(store: &Arc, cycle: u64) -> ScannerCycleStatus { + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()); + let (updates, _receiver) = mpsc::channel(8); + tokio::time::timeout( + Duration::from_secs(60), + nsscanner_with_storage_status_scoped( + store.as_ref(), + ScannerCycleRequest { + ctx, + budget, + updates, + want_cycle: cycle, + leader_epoch: 11, + scan_mode: HealScanMode::Normal, + scan_scope: ScannerBucketScanScope::default(), + persisted_usage_baseline: None, + observed_usage_candidate: None, + requires_full_scan: true, + resolved_scope_observer: None, + service_cohort: None, + post_walk_gate: None, + }, + ), + ) + .await + .expect("cycle should finish within the test deadline") + .expect("cycle should succeed") + .status +} + +#[tokio::test] +#[serial] +async fn walkers_per_disk_lets_one_disk_walk_two_buckets_concurrently() { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + clear_dirty_usage_buckets_for_tests(); + for bucket in ["walk-a", "walk-b"] { + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("bucket should be created"); + let mut reader = ScannerPutObjReader::from_vec(b"object".to_vec()); + store + .put_object(bucket, "object", &mut reader, &ScannerObjectOptions::default()) + .await + .expect("object should persist"); + } + wait_for_namespace_commit_tails(store.as_ref()).await; + + // The probe holds every admitted bucket scan long enough that two walkers + // on the one disk must overlap; a single walker cannot overlap with itself. + let hold = Duration::from_millis(750); + for (cycle, factor, expected_peak) in [(1, "1", 1), (2, "2", 2)] { + let probe = install_coordinator_bucket_scan_probe(hold); + let status = temp_env::async_with_vars([(ENV_SCANNER_WALKERS_PER_DISK, Some(factor))], async { + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + run_full_cycle(&store, cycle).await + }) + .await; + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + clear_coordinator_bucket_scan_probe(); + assert_eq!(status, ScannerCycleStatus::Complete, "walkers_per_disk={factor}"); + assert_eq!( + probe.peak.load(Ordering::SeqCst), + expected_peak, + "walkers_per_disk={factor}: concurrent bucket scans admitted on the single disk" + ); + assert_eq!(probe.active.load(Ordering::SeqCst), 0, "every admitted scan left the probe"); + } +} diff --git a/crates/scanner/src/scanner_io/tests/per_bucket_certification.rs b/crates/scanner/src/scanner_io/tests/per_bucket_certification.rs new file mode 100644 index 000000000..3b504c2e6 --- /dev/null +++ b/crates/scanner/src/scanner_io/tests/per_bucket_certification.rs @@ -0,0 +1,245 @@ +// Copyright 2026 RustFS Team +// SPDX-License-Identifier: Apache-2.0 + +//! Per-bucket certification on the single-disk fixture: a bucket written +//! during the walk is reported pending while the other buckets are certified, +//! the publication stays authoritative, and the next cycle is scoped to the +//! pending bucket. A guardrail veto still supersedes the whole cycle. + +use super::*; + +const BUCKETS: [&str; 3] = ["cold-bucket", "hot-bucket", "idle-bucket"]; + +async fn put_fixture_object(store: &Arc, bucket: &str, object: &str) { + let mut reader = ScannerPutObjReader::from_vec(vec![7; 16]); + store + .put_object(bucket, object, &mut reader, &ScannerObjectOptions::default()) + .await + .expect("fixture object should persist"); +} + +struct CycleHooks { + resolved_scope_observer: Option>, + post_walk_gate: Option>, +} + +async fn run_cycle( + store: &Arc, + cycle: u64, + baseline: Option, + requires_full_scan: bool, + hooks: CycleHooks, +) -> (ScannerCycleResult, DataUsageInfo) { + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()); + let (updates, mut receiver) = mpsc::channel(1); + let result = tokio::time::timeout( + Duration::from_secs(60), + nsscanner_with_storage_status_scoped( + store.as_ref(), + ScannerCycleRequest { + ctx, + budget, + updates, + want_cycle: cycle, + leader_epoch: 11, + scan_mode: HealScanMode::Normal, + scan_scope: ScannerBucketScanScope::default(), + persisted_usage_baseline: baseline, + observed_usage_candidate: None, + requires_full_scan, + service_cohort: None, + resolved_scope_observer: hooks.resolved_scope_observer, + post_walk_gate: hooks.post_walk_gate, + }, + ), + ) + .await + .expect("cycle should finish within the test deadline") + .expect("cycle should succeed"); + let snapshot = receiver.recv().await.expect("cycle should publish a snapshot"); + assert!(receiver.recv().await.is_none(), "a cycle publishes one terminal snapshot"); + (result, snapshot) +} + +fn baseline_bytes(snapshot: &DataUsageInfo) -> Bytes { + Bytes::from(serde_json::to_vec(snapshot).expect("snapshot should encode")) +} + +#[tokio::test] +#[serial] +async fn write_during_walk_leaves_only_that_bucket_pending_and_scopes_the_next_cycle() { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + clear_dirty_usage_buckets_for_tests(); + for bucket in BUCKETS { + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("bucket should be created"); + put_fixture_object(&store, bucket, "initial").await; + record_dirty_usage_bucket(bucket); + } + wait_for_namespace_commit_tails(store.as_ref()).await; + + // Cycle 1: a quiet full walk certifies every bucket. + let (result, first) = run_cycle( + &store, + 1, + None, + true, + CycleHooks { + resolved_scope_observer: None, + post_walk_gate: None, + }, + ) + .await; + assert_eq!(result.status, ScannerCycleStatus::Complete); + assert_eq!(result.pending_dirty_usage_buckets(), 0); + assert_eq!(first.usage_snapshot_converged, Some(true)); + assert!(!first.usage_snapshot_observational); + assert!(first.usage_snapshot_pending_buckets.is_empty()); + assert!(result.clear_verified_usage().is_empty()); + assert!(dirty_usage_buckets_for_tests().is_empty(), "a converged cycle clears the dirty map"); + + // Cycle 2: cold and idle are dirty at the start snapshot; hot is written + // after the scope resolved and before the post-walk proof runs. + for bucket in ["cold-bucket", "idle-bucket"] { + put_fixture_object(&store, bucket, "second").await; + record_dirty_usage_bucket(bucket); + } + wait_for_namespace_commit_tails(store.as_ref()).await; + let (observer, resolved_scope) = tokio::sync::oneshot::channel(); + let (release, gate) = tokio::sync::oneshot::channel(); + let writer_store = store.clone(); + let writer = tokio::spawn(async move { + let scope = resolved_scope.await.expect("the cycle should report its resolved scope"); + put_fixture_object(&writer_store, "hot-bucket", "during-walk").await; + record_dirty_usage_bucket("hot-bucket"); + wait_for_namespace_commit_tails(writer_store.as_ref()).await; + release.send(()).expect("the cycle should still be waiting on the gate"); + scope + }); + let (result, second) = run_cycle( + &store, + 2, + Some(baseline_bytes(&first)), + false, + CycleHooks { + resolved_scope_observer: Some(observer), + post_walk_gate: Some(gate), + }, + ) + .await; + let scope = writer.await.expect("writer task should finish"); + assert_eq!( + scope.selected_buckets_for_tests(), + Some(&HashSet::from(["cold-bucket".to_string(), "idle-bucket".to_string()])), + "the converged baseline scopes cycle 2 to the buckets dirty at its start" + ); + + assert_eq!( + result.status, + ScannerCycleStatus::Complete, + "a write during the walk no longer supersedes" + ); + assert_eq!(result.pending_dirty_usage_buckets(), 1); + assert!(!second.usage_snapshot_observational, "pending buckets publish on the authoritative route"); + assert_eq!(second.usage_snapshot_converged, Some(false)); + assert_eq!(second.usage_snapshot_pending_buckets, vec!["hot-bucket".to_string()]); + assert_eq!(second.usage_snapshot_pending_bucket_count, 1); + assert!(second.is_complete_bucket_usage_snapshot()); + assert_eq!(second.buckets_usage["cold-bucket"].objects_count, 2); + assert_eq!(second.buckets_usage["idle-bucket"].objects_count, 2); + assert_eq!( + second.buckets_usage["hot-bucket"].objects_count, 1, + "the pending bucket carries its last walked value" + ); + assert!(result.has_dirty_usage_to_acknowledge()); + assert!(result.clear_verified_usage().is_empty()); + let dirty = dirty_usage_buckets_for_tests(); + assert_eq!( + dirty.keys().cloned().collect::>(), + vec!["hot-bucket".to_string()], + "certification clears the unchanged buckets and keeps the written one" + ); + + // Cycle 3: the unconverged authoritative snapshot is still a structural + // baseline, so the walk is scoped to the pending bucket. + let (observer, resolved_scope) = tokio::sync::oneshot::channel(); + let (result, third) = run_cycle( + &store, + 3, + Some(baseline_bytes(&second)), + false, + CycleHooks { + resolved_scope_observer: Some(observer), + post_walk_gate: None, + }, + ) + .await; + let scope = resolved_scope.await.expect("the cycle should report its resolved scope"); + assert_eq!( + scope.selected_buckets_for_tests(), + Some(&HashSet::from(["hot-bucket".to_string()])), + "the next cycle walks only the pending bucket" + ); + assert_eq!(result.status, ScannerCycleStatus::Complete); + assert_eq!(result.pending_dirty_usage_buckets(), 0); + assert_eq!(third.usage_snapshot_converged, Some(true)); + assert!(third.usage_snapshot_pending_buckets.is_empty()); + assert_eq!(third.buckets_usage["hot-bucket"].objects_count, 2); + assert_eq!(third.buckets_usage["cold-bucket"].objects_count, 2); + assert!(result.clear_verified_usage().is_empty()); + assert!(dirty_usage_buckets_for_tests().is_empty()); + clear_dirty_usage_buckets_for_tests(); +} + +#[tokio::test] +#[serial] +async fn guardrail_veto_still_supersedes_the_cycle_onto_the_observed_route() { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + clear_dirty_usage_buckets_for_tests(); + for bucket in BUCKETS { + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("bucket should be created"); + put_fixture_object(&store, bucket, "initial").await; + record_dirty_usage_bucket(bucket); + } + wait_for_namespace_commit_tails(store.as_ref()).await; + let vetoes_before = global_metrics().report().await.usage_freshness.guardrail_veto_total; + + // A PUT tail held across the walk flips the local publication barrier, + // which is a guardrail veto regardless of the dirty map. + let pending_commit = ecstore_hold_namespace_commit(store.as_ref()); + assert!(store.scanner_data_usage_publication_blocked().await); + let (result, snapshot) = run_cycle( + &store, + 1, + None, + true, + CycleHooks { + resolved_scope_observer: None, + post_walk_gate: None, + }, + ) + .await; + drop(pending_commit); + + assert_eq!(result.status, ScannerCycleStatus::Superseded); + assert_eq!(result.pending_dirty_usage_buckets(), 0); + assert!(snapshot.usage_snapshot_observational, "a vetoed cycle routes to the observed object"); + assert_eq!(snapshot.usage_snapshot_converged, Some(false)); + assert!(snapshot.usage_snapshot_pending_buckets.is_empty()); + assert!(snapshot.is_complete_bucket_usage_snapshot()); + assert!(!result.has_dirty_usage_to_acknowledge(), "nothing is certified under a veto"); + assert_eq!(dirty_usage_buckets_for_tests().len(), BUCKETS.len(), "the dirty map is untouched"); + let freshness = global_metrics().report().await.usage_freshness; + assert_eq!(freshness.guardrail_veto_total, vetoes_before + 1); + assert_eq!( + freshness.last_guardrail_veto_reason, + format!("publication_blocked@{}", crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE) + ); + clear_dirty_usage_buckets_for_tests(); +} diff --git a/crates/scanner/src/scanner_io/tests/scoped_entry_fallback.rs b/crates/scanner/src/scanner_io/tests/scoped_entry_fallback.rs index d974bd923..f31ae0f0d 100644 --- a/crates/scanner/src/scanner_io/tests/scoped_entry_fallback.rs +++ b/crates/scanner/src/scanner_io/tests/scoped_entry_fallback.rs @@ -138,6 +138,7 @@ async fn run_entry( requires_full_scan: false, service_cohort: None, resolved_scope_observer: Some(observer), + post_walk_gate: None, }, ), ) @@ -366,7 +367,7 @@ async fn scoped_entry_fallback_rejects_invalid_persisted_baseline_at_the_walker_ let baseline = run_entry(&store, 1, None, true, false, false).await; for (index, kind) in [ "malformed", - "unconverged", + "observational", "missing-set", "wrong-source", "mixed-plan", @@ -378,7 +379,12 @@ async fn scoped_entry_fallback_rejects_invalid_persisted_baseline_at_the_walker_ let mut candidate = baseline.clone(); candidate.usage_snapshot_converged = Some(true); match kind { - "unconverged" => candidate.usage_snapshot_converged = Some(false), + // An unconverged authoritative snapshot is a valid baseline since + // per-bucket certification; only the observational mark rejects. + "observational" => { + candidate.usage_snapshot_converged = Some(false); + candidate.usage_snapshot_observational = true; + } "missing-set" => { candidate.usage_snapshot_set_states.pop(); } diff --git a/crates/scanner/src/scanner_io/tests/service_cohort.rs b/crates/scanner/src/scanner_io/tests/service_cohort.rs index deb1da547..1deb5d6d1 100644 --- a/crates/scanner/src/scanner_io/tests/service_cohort.rs +++ b/crates/scanner/src/scanner_io/tests/service_cohort.rs @@ -73,6 +73,7 @@ async fn run_cohort_cycle( requires_full_scan: false, service_cohort: Some(cohort), resolved_scope_observer: None, + post_walk_gate: None, }, ), ) diff --git a/crates/scanner/src/scanner_io/tests/usage_expiration.rs b/crates/scanner/src/scanner_io/tests/usage_expiration.rs index c914378b1..d3dcd8b45 100644 --- a/crates/scanner/src/scanner_io/tests/usage_expiration.rs +++ b/crates/scanner/src/scanner_io/tests/usage_expiration.rs @@ -9,6 +9,77 @@ use crate::storage_api::scanner_io::{ record_bucket_object_write_memory, replace_bucket_usage_memory_from_info, seed_persisted_usage, update_bucket_metadata, }; +#[tokio::test] +#[serial] +async fn disk_scan_withdraws_persisted_lifecycle_policy() { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + let bucket = "withdrawn-lifecycle"; + store.make_bucket(bucket, &MakeBucketOptions::default()).await.unwrap(); + let mut reader = ScannerPutObjReader::from_vec(b"retained".to_vec()); + store + .put_object(bucket, "object", &mut reader, &ScannerObjectOptions::default()) + .await + .unwrap(); + let xml = br#"expireEnabled0"#; + update_bucket_metadata(bucket, BUCKET_LIFECYCLE_CONFIG, xml.to_vec()) + .await + .unwrap(); + let (policy, _) = get_lifecycle_config(bucket).await.unwrap(); + let set = Arc::clone(&store.pools[0].disk_set[0]); + let disks = scanner_set_disk_inventory(&set).await; + let cache_name = format!("{bucket}/{DATA_USAGE_CACHE_NAME}"); + for config in [ + Vec::new(), + String::from_utf8(xml.to_vec()) + .unwrap() + .replace("Enabled", "Disabled") + .into_bytes(), + ] { + let mut old_cache = DataUsageCache::default(); + old_cache.info.name = bucket.to_string(); + old_cache.info.lifecycle = Some(Arc::new(policy.clone())); + old_cache.replace(bucket, "", DataUsageEntry::default()); + old_cache.save(Arc::clone(&set), &cache_name).await.unwrap(); + update_bucket_metadata(bucket, BUCKET_LIFECYCLE_CONFIG, config).await.unwrap(); + + // A fresh cache instance models restart: the obsolete policy survives + // serialization and must be withdrawn by the actual disk scan. + let mut loaded = DataUsageCache::default(); + loaded.load(Arc::clone(&set), &cache_name).await.unwrap(); + assert!(loaded.info.lifecycle.is_some()); + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()); + let outcome = disks[0] + .clone() + .nsscanner_disk( + ctx, + budget, + disks.clone(), + loaded, + None, + ScannerDiskScanOptions { + scan_mode: HealScanMode::Normal, + prefix_scan_scope: None, + checkpoint_tx: None, + }, + ) + .await + .unwrap(); + let ScannerDiskScanOutcome::Complete(refreshed) = outcome else { + panic!("the single-drive scan must complete"); + }; + assert!(refreshed.info.lifecycle.is_none()); + refreshed.save(Arc::clone(&set), &cache_name).await.unwrap(); + let mut restarted = DataUsageCache::default(); + restarted.load(Arc::clone(&set), &cache_name).await.unwrap(); + assert!(restarted.info.lifecycle.is_none(), "another restart must not resurrect policy"); + store + .get_object_info(bucket, "object", &ScannerObjectOptions::default()) + .await + .unwrap(); + } +} + // Lifecycle expiry goes through the process-global store, which the first // store created in the process claims. Nextest runs this test in its own // process; the lifecycle scenarios share the store that owns expiry processing. diff --git a/crates/scanner/src/storage_api.rs b/crates/scanner/src/storage_api.rs index c0ff85a96..a333e0faa 100644 --- a/crates/scanner/src/storage_api.rs +++ b/crates/scanner/src/storage_api.rs @@ -101,6 +101,7 @@ pub(crate) use rustfs_ecstore::api::event::{EventArgs as EcstoreEventArgs, send_ #[cfg(test)] pub(crate) use rustfs_ecstore::api::layout::{ EndpointServerPools as EcstoreEndpointServerPools, Endpoints as EcstoreEndpoints, PoolEndpoints as EcstorePoolEndpoints, + SetupType as EcstoreSetupType, }; pub(crate) use rustfs_ecstore::api::notification::NotificationSys as EcstoreNotificationSys; pub(crate) use rustfs_ecstore::api::notification::scanner_peer_transport_error_message_is_retryable; @@ -140,10 +141,21 @@ pub(crate) use rustfs_lifecycle::{ }; use rustfs_storage_api as storage_contracts; +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) mod ecstore_diagnostic { + #[cfg(test)] + pub(crate) use rustfs_ecstore::api::diagnostic::current_read_version_entry_source; + pub(crate) use rustfs_ecstore::api::diagnostic::{ + ReadVersionContextBoundary, ReadVersionEntrySource, observe_read_version_context_boundary, with_read_version_entry, + }; +} + #[cfg(test)] pub(crate) type EcstoreHealResultItem = ::HealResultItem; pub(crate) mod owner { + #[cfg(test)] + pub(crate) use super::storage_contracts::DiskSetSelector; pub(crate) use super::{ MAX_SCANNER_PAUSE_BACKLOG_BYTES, ScannerPauseBacklogRetirementError, ScannerPauseBacklogRetirementPlan, ScannerPauseBacklogRetirementReplica, register_scanner_pause_backlog_retirement_planner, diff --git a/crates/scanner/tests/lifecycle_integration_test.rs b/crates/scanner/tests/lifecycle_integration_test.rs index f913d114a..713f00edc 100644 --- a/crates/scanner/tests/lifecycle_integration_test.rs +++ b/crates/scanner/tests/lifecycle_integration_test.rs @@ -663,8 +663,17 @@ mod serial_tests { ); ExpiryState::resize_workers(1, ecstore.clone()).await; + set_bucket_lifecycle(bucket_name.as_str()) + .await + .expect("install the expiration policy"); let lc_event = LcEvent { action: IlmAction::DeleteAction, + expiry_policy: get_bucket_metadata(bucket_name.as_str()) + .await + .expect("load expiration policy") + .lifecycle_config + .clone() + .map(|configuration| Arc::new(configuration).into()), ..Default::default() }; let bucket_incarnation_id = ecstore @@ -672,6 +681,25 @@ mod serial_tests { .await .expect("read bucket incarnation"); + update_bucket_metadata(bucket_name.as_str(), BUCKET_LIFECYCLE_CONFIG, Vec::new()) + .await + .expect("withdraw the queued expiration policy"); + assert!( + expire_transitioned_object(ecstore.clone(), &oi, &lc_event, &LcEventSrc::Scanner, bucket_incarnation_id) + .await + .is_err(), + "a withdrawn policy must retain both local metadata and remote data" + ); + assert_eq!(backend.remove_count().await, 0); + assert!(backend.contains(&remote_object).await); + ecstore + .get_object_info(bucket_name.as_str(), object_name, &ObjectOptions::default()) + .await + .expect("withdrawn expiration must retain transition metadata"); + set_bucket_lifecycle(bucket_name.as_str()) + .await + .expect("restore the expiration policy"); + // Pause one real tier GET after it has resolved local transition // metadata. The reader still owns the object read lock, so local expiry // cannot commit until this GET finishes. @@ -2451,6 +2479,9 @@ mod serial_tests { // The scanner's restore-expiry action removes only the local restored // copy (same direct-drive pattern as the ilm-2 expiry test). + update_bucket_metadata(bucket_name.as_str(), BUCKET_LIFECYCLE_CONFIG, Vec::new()) + .await + .expect("restore-copy cleanup must not require a lifecycle policy"); let lc_event = LcEvent { action: IlmAction::DeleteRestoredAction, ..Default::default() diff --git a/crates/storage-api/src/error.rs b/crates/storage-api/src/error.rs index b24fedfe4..ad6c62335 100644 --- a/crates/storage-api/src/error.rs +++ b/crates/storage-api/src/error.rs @@ -107,6 +107,8 @@ pub enum StorageErrorCode { QuotaExceeded, RemoteClientUnavailable, RemoteNotInitialized, + ConditionalDeleteConflict, + GuardedDeleteUnsupported, } impl StorageErrorCode { @@ -196,6 +198,8 @@ impl StorageErrorCode { Self::QuotaExceeded => 0x53, Self::RemoteClientUnavailable => 0x54, Self::RemoteNotInitialized => 0x55, + Self::ConditionalDeleteConflict => 0x57, + Self::GuardedDeleteUnsupported => 0x58, } } @@ -285,6 +289,8 @@ impl StorageErrorCode { 0x53 => Some(Self::QuotaExceeded), 0x54 => Some(Self::RemoteClientUnavailable), 0x55 => Some(Self::RemoteNotInitialized), + 0x57 => Some(Self::ConditionalDeleteConflict), + 0x58 => Some(Self::GuardedDeleteUnsupported), _ => None, } } @@ -382,6 +388,8 @@ mod tests { (StorageErrorCode::SourceStalled, 0x50), (StorageErrorCode::Timeout, 0x51), (StorageErrorCode::InvalidPath, 0x52), + (StorageErrorCode::ConditionalDeleteConflict, 0x57), + (StorageErrorCode::GuardedDeleteUnsupported, 0x58), ]; #[test] diff --git a/crates/storage-api/src/object.rs b/crates/storage-api/src/object.rs index b7aaf28dc..9b1d40315 100644 --- a/crates/storage-api/src/object.rs +++ b/crates/storage-api/src/object.rs @@ -199,10 +199,79 @@ pub struct ObjectToDeleteIdentity { pub delete_marker: bool, } +/// A single ETag condition on the current live object of a bulk delete. +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub enum DeleteObjectCondition { + Exists, + MatchEtag(String), +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum DeleteObjectConditionError { + InvalidEtag, +} + +impl fmt::Display for DeleteObjectConditionError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::InvalidEtag => f.write_str("a delete ETag condition must contain one non-empty ETag"), + } + } +} + +impl std::error::Error for DeleteObjectConditionError {} + +impl DeleteObjectCondition { + pub fn parse(value: &str) -> Result { + let value = value.trim(); + let value = match (value.starts_with('"'), value.ends_with('"')) { + (true, true) if value.len() >= 2 => &value[1..value.len() - 1], + (false, false) => value, + _ => return Err(DeleteObjectConditionError::InvalidEtag), + }; + if value.trim().is_empty() || value.contains('"') { + return Err(DeleteObjectConditionError::InvalidEtag); + } + if value == "*" { + Ok(Self::Exists) + } else { + Ok(Self::MatchEtag(value.to_string())) + } + } + + /// Reject malformed values constructed directly or decoded from a peer. + pub fn validate(&self) -> Result<(), DeleteObjectConditionError> { + match self { + Self::Exists => Ok(()), + Self::MatchEtag(value) if !value.trim().is_empty() && value != "*" && !value.contains('"') => Ok(()), + Self::MatchEtag(_) => Err(DeleteObjectConditionError::InvalidEtag), + } + } + + pub fn requires_read_permission(&self) -> bool { + matches!(self, Self::MatchEtag(_)) + } + + /// The caller must separately require a live, existing object. + pub fn matches(&self, object_etag: &str) -> bool { + match self { + Self::Exists => true, + Self::MatchEtag(expected) => { + let object_etag = object_etag + .strip_prefix('"') + .and_then(|value| value.strip_suffix('"')) + .unwrap_or(object_etag); + self.validate().is_ok() && expected == object_etag + } + } + } +} + #[derive(Debug, Default, Clone)] pub struct ObjectToDelete { pub object_name: String, pub version_id: Option, + pub delete_condition: Option, /// RustFS-only precondition checked under the object write lock. #[doc(hidden)] pub expected_identity: Option, @@ -761,6 +830,41 @@ fn is_modified_since(mod_time: &OffsetDateTime, given_time: &OffsetDateTime) -> mod tests { use super::*; + #[test] + fn delete_object_condition_rejects_empty_or_malformed_etags() { + for value in ["", " \t ", "\"\"", "\" \t \"", "\"", "\"etag", "etag\"", "\"\"etag\"\""] { + assert_eq!(DeleteObjectCondition::parse(value), Err(DeleteObjectConditionError::InvalidEtag)); + } + assert!(DeleteObjectCondition::MatchEtag(String::new()).validate().is_err()); + assert!(!DeleteObjectCondition::MatchEtag(String::new()).matches("")); + assert!(DeleteObjectCondition::MatchEtag("*".to_string()).validate().is_err()); + } + + #[test] + fn delete_object_condition_is_a_case_sensitive_opaque_single_etag() { + let condition = DeleteObjectCondition::parse(" \"opaque-ETag-7\" ").expect("quoted ETag should parse"); + assert_eq!(condition, DeleteObjectCondition::MatchEtag("opaque-ETag-7".to_string())); + assert!(condition.matches("opaque-ETag-7")); + assert!(condition.matches("\"opaque-ETag-7\"")); + assert!(!condition.matches("opaque-etag-7")); + assert!(condition.requires_read_permission()); + assert!(condition.validate().is_ok()); + + let condition = DeleteObjectCondition::parse("one,two").expect("ETags are opaque values"); + assert!(condition.matches("one,two")); + assert!(!condition.matches("one")); + assert!(!condition.matches("two")); + } + + #[test] + fn delete_object_condition_exists_does_not_require_read_permission() { + for value in ["*", " \"*\" "] { + let condition = DeleteObjectCondition::parse(value).expect("wildcard condition should parse"); + assert_eq!(condition, DeleteObjectCondition::Exists); + assert!(!condition.requires_read_permission()); + } + } + #[test] fn version_marker_parse_folds_null_and_nil_uuid_into_null() { assert_eq!(VersionMarker::parse("null"), Ok(VersionMarker::Null)); diff --git a/crates/tls-runtime/Cargo.toml b/crates/tls-runtime/Cargo.toml index fb8920547..443be5ea0 100644 --- a/crates/tls-runtime/Cargo.toml +++ b/crates/tls-runtime/Cargo.toml @@ -46,6 +46,7 @@ sha2.workspace = true thiserror.workspace = true tokio = { workspace = true, features = ["fs", "rt", "sync", "time"] } tracing.workspace = true +x509-parser.workspace = true [dev-dependencies] rcgen.workspace = true diff --git a/crates/tls-runtime/src/certs.rs b/crates/tls-runtime/src/certs.rs index 20260768f..a79c8e9c0 100644 --- a/crates/tls-runtime/src/certs.rs +++ b/crates/tls-runtime/src/certs.rs @@ -429,6 +429,209 @@ pub fn load_all_certs_from_directory( Ok(cert_key_pairs) } +/// Strict runtime loader: a discovered identity must never disappear through +/// best-effort filtering. The diagnostic inspection API remains independent. +pub fn load_all_certs_from_directory_strict( + options: CertDirectoryLoadOptions, +) -> Result>, PrivateKeyDer<'static>)>, crate::error::TlsRuntimeError> { + use crate::error::TlsRuntimeError; + options.validate()?; + let dir = &options.dir_path; + let mut pairs = HashMap::new(); + if let Some(pair) = read_cert_key_pair_strict(dir, &options.cert_filename, &options.key_filename, "default", true)? { + pairs.insert("default".to_string(), pair); + } + for entry in fs::read_dir(dir).map_err(|source| TlsRuntimeError::SourceIo { + path: dir.clone(), + source, + })? { + let entry = entry.map_err(|source| TlsRuntimeError::SourceIo { + path: dir.clone(), + source, + })?; + let path = entry.path(); + let name = entry.file_name(); + let name = name.to_str().ok_or_else(|| TlsRuntimeError::InvalidCertificate { + name: "SNI directory".to_string(), + reason: "directory name is not UTF-8".to_string(), + })?; + if !is_discoverable_cert_domain_dir(name) { + continue; + } + let metadata = fs::metadata(&path).map_err(|source| TlsRuntimeError::SourceIo { + path: path.clone(), + source, + })?; + if !metadata.is_dir() { + continue; + } + if let Some(pair) = read_cert_key_pair_strict(&path, &options.cert_filename, &options.key_filename, name, true)? { + pairs.insert(name.to_string(), pair); + } + } + Ok(pairs) +} + +pub(crate) fn read_optional_tls_file(path: &Path) -> Result>, crate::error::TlsRuntimeError> { + use crate::error::TlsRuntimeError; + match fs::symlink_metadata(path) { + Ok(_) => fs::read(path).map(Some).map_err(|source| TlsRuntimeError::SourceIo { + path: path.to_path_buf(), + source, + }), + Err(error) if error.kind() == io::ErrorKind::NotFound => { + // A disappearing pinned generation is an error, not optional absence. + if let Some(parent) = path.parent() { + fs::read_dir(parent).map_err(|source| TlsRuntimeError::SourceIo { + path: parent.to_path_buf(), + source, + })?; + } + Ok(None) + } + Err(source) => Err(TlsRuntimeError::SourceIo { + path: path.to_path_buf(), + source, + }), + } +} + +pub(crate) fn parse_certificate_pem( + pem: &[u8], + name: &str, +) -> Result>, crate::error::TlsRuntimeError> { + use crate::error::TlsRuntimeError; + let certs = CertificateDer::pem_slice_iter(pem) + .collect::, _>>() + .map_err(|error| TlsRuntimeError::InvalidCertificate { + name: name.to_string(), + reason: error.to_string(), + })?; + if certs.is_empty() { + return Err(TlsRuntimeError::InvalidCertificate { + name: name.to_string(), + reason: "empty certificate PEM".to_string(), + }); + } + for cert in &certs { + let (remaining, _) = + x509_parser::parse_x509_certificate(cert.as_ref()).map_err(|error| TlsRuntimeError::InvalidCertificate { + name: name.to_string(), + reason: error.to_string(), + })?; + if !remaining.is_empty() { + return Err(TlsRuntimeError::InvalidCertificate { + name: name.to_string(), + reason: "trailing certificate DER".to_string(), + }); + } + } + Ok(certs) +} + +pub(crate) fn parse_private_key_pem(pem: &[u8], name: &str) -> Result, crate::error::TlsRuntimeError> { + PrivateKeyDer::from_pem_slice(pem).map_err(|error| crate::error::TlsRuntimeError::InvalidPrivateKey { + name: name.to_string(), + reason: error.to_string(), + }) +} + +pub(crate) fn read_cert_key_pair_strict( + dir: &Path, + cert_filename: &str, + key_filename: &str, + name: &str, + server: bool, +) -> Result>, PrivateKeyDer<'static>)>, crate::error::TlsRuntimeError> { + let cert = read_optional_tls_file(&dir.join(cert_filename))?; + let key = read_optional_tls_file(&dir.join(key_filename))?; + match (cert, key) { + (None, None) => Ok(None), + (Some(cert), Some(key)) => { + let certs = parse_certificate_pem(&cert, name)?; + let key = parse_private_key_pem(&key, name)?; + validate_certified_key(&certs, &key, name, server)?; + Ok(Some((certs, key))) + } + _ => Err(crate::error::TlsRuntimeError::IncompleteIdentity { name: name.to_string() }), + } +} + +/// Validate before publication, including the default certificate that bypasses +/// rustls' SNI name checks. +pub fn validate_certified_key( + certs: &[CertificateDer<'static>], + key: &PrivateKeyDer<'static>, + name: &str, + server: bool, +) -> Result { + use crate::error::TlsRuntimeError; + if certs.is_empty() { + return Err(TlsRuntimeError::InvalidCertificate { + name: name.to_string(), + reason: "empty certificate chain".to_string(), + }); + } + for (index, cert) in certs.iter().enumerate() { + let (remaining, parsed) = + x509_parser::parse_x509_certificate(cert.as_ref()).map_err(|error| TlsRuntimeError::InvalidCertificate { + name: name.to_string(), + reason: error.to_string(), + })?; + if !remaining.is_empty() { + return Err(TlsRuntimeError::InvalidCertificate { + name: name.to_string(), + reason: "trailing certificate DER".to_string(), + }); + } + if index == 0 { + if !parsed.validity().is_valid() { + return Err(TlsRuntimeError::CertificateValidity { name: name.to_string() }); + } + if parsed + .extended_key_usage() + .map_err(|error| TlsRuntimeError::InvalidCertificate { + name: name.to_string(), + reason: error.to_string(), + })? + .is_some_and(|eku| { + if server { + !eku.value.server_auth + } else { + !eku.value.client_auth + } + }) + { + return Err(if server { + TlsRuntimeError::InvalidServerEku { name: name.to_string() } + } else { + TlsRuntimeError::InvalidClientEku { name: name.to_string() } + }); + } + } + } + let certified = CertifiedKey::from_der(certs.to_vec(), key.clone_key(), &rustls::crypto::aws_lc_rs::default_provider()) + .map_err(|source| TlsRuntimeError::InvalidCertifiedKey { + name: name.to_string(), + source, + })?; + certified + .keys_match() + .map_err(|source| TlsRuntimeError::InvalidCertifiedKey { + name: name.to_string(), + source, + })?; + if server && name != "default" { + let mut sni = ResolvesServerCertUsingSni::new(); + sni.add(name, certified.clone()) + .map_err(|source| TlsRuntimeError::InvalidCertifiedKey { + name: name.to_string(), + source, + })?; + } + Ok(certified) +} + fn load_cert_key_pair(cert_path: &str, key_path: &str) -> io::Result<(Vec>, PrivateKeyDer<'static>)> { let certs = load_certs(cert_path)?; let key = load_private_key(key_path)?; @@ -494,10 +697,7 @@ pub fn create_multi_cert_resolver( let mut default_cert = None; for (domain, (certs, key)) in cert_key_pairs { - let signing_key = rustls::crypto::aws_lc_rs::sign::any_supported_type(&key) - .map_err(|e| certs_error(format!("unsupported private key types:{domain}, err:{e:?}")))?; - - let certified_key = CertifiedKey::new(certs, signing_key); + let certified_key = validate_certified_key(&certs, &key, &domain, true).map_err(io::Error::other)?; if domain == "default" { default_cert = Some(Arc::new(certified_key.clone())); } else { diff --git a/crates/tls-runtime/src/error.rs b/crates/tls-runtime/src/error.rs index ac7d50691..601624d6a 100644 --- a/crates/tls-runtime/src/error.rs +++ b/crates/tls-runtime/src/error.rs @@ -23,6 +23,38 @@ pub enum TlsRuntimeError { DirectoryNotFound { path: PathBuf }, #[error("TLS path is not a directory: {path}")] NotADirectory { path: PathBuf }, + #[error("TLS source I/O error at {path}: {source}")] + SourceIo { + path: PathBuf, + #[source] + source: std::io::Error, + }, + #[error("incomplete TLS certificate/private key pair for {name}")] + IncompleteIdentity { name: String }, + #[error("invalid TLS certificate material for {name}: {reason}")] + InvalidCertificate { name: String, reason: String }, + #[error("invalid TLS private key for {name}: {reason}")] + InvalidPrivateKey { name: String, reason: String }, + #[error("TLS certificate/private key validation failed for {name}: {source}")] + InvalidCertifiedKey { + name: String, + #[source] + source: rustls::Error, + }, + #[error("invalid TLS trust anchor in {name}: {source}")] + InvalidTrustAnchor { + name: String, + #[source] + source: rustls::Error, + }, + #[error("TLS certificate for {name} is outside its validity interval")] + CertificateValidity { name: String }, + #[error("TLS certificate for {name} does not permit server authentication")] + InvalidServerEku { name: String }, + #[error("TLS certificate for {name} does not permit client authentication")] + InvalidClientEku { name: String }, + #[error("TLS source loading task failed: {0}")] + SourceTaskJoin(#[from] tokio::task::JoinError), #[error("TLS material error: {0}")] Material(String), #[error("TLS publication error: {0}")] diff --git a/crates/tls-runtime/src/lib.rs b/crates/tls-runtime/src/lib.rs index 1d5bcf232..ae25ed86b 100644 --- a/crates/tls-runtime/src/lib.rs +++ b/crates/tls-runtime/src/lib.rs @@ -28,7 +28,8 @@ pub mod state; pub use certs::{ CertDirectoryLoadOptions, TlsCertPairInspection, TlsCertPairStatus, TlsDirectoryInspection, TlsDomainInspection, WebPkiClientVerifierOptions, build_webpki_client_verifier, create_multi_cert_resolver, inspect_cert_directory, - load_all_certs_from_directory, load_cert_bundle_der_bytes, load_certs, load_private_key, + load_all_certs_from_directory, load_all_certs_from_directory_strict, load_cert_bundle_der_bytes, load_certs, + load_private_key, validate_certified_key, }; pub use config::{ReloadApplyHint, ReloadDetectMode, TlsReloadOptions}; pub use coordinator::{TlsConsumer, TlsReloadCoordinator}; @@ -45,7 +46,7 @@ pub use outbound::{ load_global_outbound_tls_state, publish_global_outbound_tls_state, summarize_global_outbound_tls_state, }; pub use server::{ReloadableServerCertResolver, spawn_server_cert_reload_loop}; -pub use source::{TlsFileLayout, TlsSource, TlsSourceKind}; +pub use source::{TlsFileLayout, TlsSource, TlsSourceKind, TlsSourceSnapshot}; pub use state::OutboundOnlySnapshotArgs; pub use state::{TlsGeneration, TlsPublishedState, TlsReloadRuntimeState, TlsRuntimeStatusSnapshot}; pub use state::{TlsRuntimeConsumerSection, TlsRuntimeOutboundSection, TlsRuntimeRuntimeSection, TlsRuntimeServerSection}; @@ -111,6 +112,8 @@ mod tests { let coordinator = TlsReloadCoordinator::new(source.clone(), TlsReloadOptions::default()); let snapshot = TlsMaterialSnapshot { source, + server_client_ca_pem: None, + material_files: std::collections::BTreeSet::new(), server: Some(ServerTlsMaterial::MultiCert { cert_key_pairs: HashMap::new(), }), diff --git a/crates/tls-runtime/src/material.rs b/crates/tls-runtime/src/material.rs index f6b3fab99..7046f0174 100644 --- a/crates/tls-runtime/src/material.rs +++ b/crates/tls-runtime/src/material.rs @@ -12,15 +12,16 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::certs::{CertDirectoryLoadOptions, load_all_certs_from_directory, load_certs, load_private_key}; +use crate::certs::{ + CertDirectoryLoadOptions, load_all_certs_from_directory_strict, parse_certificate_pem, parse_private_key_pem, + read_optional_tls_file, validate_certified_key, +}; use crate::error::TlsRuntimeError; use crate::fingerprint::TlsFingerprint; -use crate::source::TlsSource; +use crate::source::{TlsSource, TlsSourceSnapshot}; use rustfs_common::MtlsIdentityPem; -use rustls::pki_types::{CertificateDer, PrivateKeyDer, pem::PemObject}; -use std::collections::HashMap; -use std::io::Cursor; -use std::io::ErrorKind; +use rustls::pki_types::{CertificateDer, PrivateKeyDer}; +use std::collections::{BTreeSet, HashMap}; use std::path::Path; #[derive(Debug, Clone)] @@ -40,115 +41,196 @@ pub enum ServerTlsMaterial { }, } +impl ServerTlsMaterial { + /// Revalidate an accepted generation without reading the filesystem. + pub fn validate(&self) -> Result<(), TlsRuntimeError> { + match self { + Self::SingleCert { certs, key } => { + crate::certs::validate_certified_key(certs, key, "default", true)?; + } + Self::MultiCert { cert_key_pairs } => { + for (name, (certs, key)) in cert_key_pairs { + crate::certs::validate_certified_key(certs, key, name, true)?; + } + } + } + Ok(()) + } +} + +impl Clone for ServerTlsMaterial { + fn clone(&self) -> Self { + match self { + Self::SingleCert { certs, key } => Self::SingleCert { + certs: certs.clone(), + key: key.clone_key(), + }, + Self::MultiCert { cert_key_pairs } => Self::MultiCert { + cert_key_pairs: cert_key_pairs + .iter() + .map(|(name, (certs, key))| (name.clone(), (certs.clone(), key.clone_key()))) + .collect(), + }, + } + } +} + #[derive(Debug)] pub struct TlsMaterialSnapshot { + /// The resolved directory generation used by every consumer of this snapshot. pub source: TlsSource, pub server: Option, pub outbound: OutboundTlsMaterial, + pub server_client_ca_pem: Option>, + pub material_files: BTreeSet, pub fingerprint: TlsFingerprint, } impl TlsMaterialSnapshot { pub async fn load(source: &TlsSource) -> Result { - let base_dir = source.validate_directory()?.to_path_buf(); + let source = source.clone(); + tokio::task::spawn_blocking(move || Self::load_from_snapshot(&source.snapshot()?)).await? + } - let server = load_server_material(source, &base_dir)?; + /// Load all consumers' material from one previously pinned directory generation. + pub fn load_from_snapshot(snapshot: &TlsSourceSnapshot) -> Result { + Self::load_from_snapshot_with_server_directory(snapshot, None) + } - let public_ca_path = base_dir.join(&source.layout.public_ca_filename); - let client_ca_path = base_dir.join(&source.fallback_ca_filename); - let client_cert_path = base_dir.join(&source.layout.client_cert_filename); - let client_key_path = base_dir.join(&source.layout.client_key_filename); + /// Load a server-only publication without replacing the pinned trust or client identity source. + pub fn load_from_snapshot_with_server_directory( + snapshot: &TlsSourceSnapshot, + server_directory: Option<&Path>, + ) -> Result { + snapshot.validate_available()?; + let source = snapshot.resolved_source(); + let base_dir = snapshot.base_dir(); + let server = load_server_material(source, server_directory.unwrap_or(base_dir))?; + Self::load_from_snapshot_with_loaded_server(snapshot, server) + } - let public_ca_pem = tokio::fs::read(&public_ca_path).await.ok(); - let client_ca_pem = tokio::fs::read(&client_ca_path).await.ok(); - let root_ca_pem = combine_optional_pem(public_ca_pem.as_deref(), client_ca_pem.as_deref()); + /// Validate an already decoded default server chain while reading trust and client identity from the pinned source. + /// Serving validity, EKU and key matching are checked here even when the publication ledger was verified earlier. + pub fn load_from_snapshot_with_server_identity( + snapshot: &TlsSourceSnapshot, + certificates: Vec>, + private_key_pem: &[u8], + ) -> Result { + snapshot.validate_available()?; + let key = parse_private_key_pem(private_key_pem, "default")?; + validate_certified_key(&certificates, &key, "default", true)?; + Self::load_from_snapshot_with_loaded_server( + snapshot, + Some(ServerTlsMaterial::SingleCert { + certs: certificates, + key, + }), + ) + } - let mtls_identity = match ( - tokio::fs::read(&client_cert_path).await.ok(), - tokio::fs::read(&client_key_path).await.ok(), - ) { - (Some(cert_pem), Some(key_pem)) => { - let mut cert_reader = Cursor::new(&cert_pem); - if CertificateDer::pem_reader_iter(&mut cert_reader).next().is_none() { - return Err(TlsRuntimeError::Material("no valid certificate in client cert PEM".to_string())); + fn load_from_snapshot_with_loaded_server( + snapshot: &TlsSourceSnapshot, + server: Option, + ) -> Result { + snapshot.validate_available()?; + let source = snapshot.resolved_source(); + let base_dir = snapshot.base_dir(); + let public_ca_pem = read_optional_tls_file(&base_dir.join(&source.layout.public_ca_filename))?; + let fallback_ca_pem = read_optional_tls_file(&base_dir.join(&source.fallback_ca_filename))?; + let client_ca_pem = read_optional_tls_file(&base_dir.join(&source.layout.client_ca_filename))?; + let mut material_files = BTreeSet::new(); + for (name, present) in [ + (&source.layout.public_ca_filename, public_ca_pem.is_some()), + (&source.fallback_ca_filename, fallback_ca_pem.is_some()), + (&source.layout.client_ca_filename, client_ca_pem.is_some()), + ] { + if present { + material_files.insert(name.clone()); + } + } + for (name, pem) in [ + (&source.layout.public_ca_filename, &public_ca_pem), + (&source.fallback_ca_filename, &fallback_ca_pem), + (&source.layout.client_ca_filename, &client_ca_pem), + ] { + if let Some(pem) = pem { + let certs = parse_certificate_pem(pem, name)?; + let mut roots = rustls::RootCertStore::empty(); + for cert in certs { + roots.add(cert).map_err(|source| TlsRuntimeError::InvalidTrustAnchor { + name: name.clone(), + source, + })?; } - - let mut key_reader = Cursor::new(&key_pem); - PrivateKeyDer::from_pem_reader(&mut key_reader) - .map_err(|e| TlsRuntimeError::Material(format!("invalid client key PEM: {e}")))?; - + } + } + let server_client_ca_pem = client_ca_pem.or_else(|| fallback_ca_pem.clone()); + let root_ca_pem = combine_optional_pem(public_ca_pem.as_deref(), fallback_ca_pem.as_deref()); + let client_cert_pem = read_optional_tls_file(&base_dir.join(&source.layout.client_cert_filename))?; + let client_key_pem = read_optional_tls_file(&base_dir.join(&source.layout.client_key_filename))?; + if client_cert_pem.is_some() { + material_files.insert(source.layout.client_cert_filename.clone()); + } + if client_key_pem.is_some() { + material_files.insert(source.layout.client_key_filename.clone()); + } + let mtls_identity = match (client_cert_pem, client_key_pem) { + (Some(cert_pem), Some(key_pem)) => { + let certs = parse_certificate_pem(&cert_pem, "client identity")?; + let key = parse_private_key_pem(&key_pem, "client identity")?; + validate_certified_key(&certs, &key, "client identity", false)?; Some(MtlsIdentityPem { cert_pem, key_pem }) } - _ => None, + (None, None) => None, + _ => { + return Err(TlsRuntimeError::IncompleteIdentity { + name: "client identity".to_string(), + }); + } }; - + snapshot.validate_available()?; let outbound = OutboundTlsMaterial { - root_ca_pem: root_ca_pem.clone(), + root_ca_pem, mtls_identity: mtls_identity.clone(), }; - - let server_fingerprint_bytes = server.as_ref().map(serialize_server_material_for_fingerprint); + let mut server_fingerprint_bytes = server.as_ref().map(serialize_server_material_for_fingerprint); + if let Some(pem) = &server_client_ca_pem { + let bytes = server_fingerprint_bytes.get_or_insert_with(Vec::new); + bytes.extend_from_slice(b"\0server-client-ca\0"); + bytes.extend_from_slice(pem); + } let fingerprint = TlsFingerprint::from_optional_bytes( server_fingerprint_bytes.as_deref(), public_ca_pem.as_deref(), - client_ca_pem.as_deref(), + fallback_ca_pem.as_deref(), mtls_identity.as_ref().map(|identity| identity.cert_pem.as_slice()), mtls_identity.as_ref().map(|identity| identity.key_pem.as_slice()), ); - Ok(Self { - source: source.clone(), + source: snapshot.resolved_source().clone(), server, outbound, + server_client_ca_pem, + material_files, fingerprint, }) } } fn load_server_material(source: &TlsSource, base_dir: &Path) -> Result, TlsRuntimeError> { - let root_cert = base_dir.join(&source.layout.server_cert_filename); - let root_key = base_dir.join(&source.layout.server_key_filename); - - let has_root_pair = root_cert.exists() && root_key.exists(); - let cert_key_pairs = load_all_certs_from_directory( + let mut cert_key_pairs = load_all_certs_from_directory_strict( CertDirectoryLoadOptions::builder(base_dir, &source.layout.server_cert_filename, &source.layout.server_key_filename) .build(), - ); - - match cert_key_pairs { - Ok(cert_key_pairs) if cert_key_pairs.len() > 1 || cert_key_pairs.keys().any(|key| key != "default") => { - Ok(Some(ServerTlsMaterial::MultiCert { cert_key_pairs })) - } - Ok(cert_key_pairs) if !cert_key_pairs.is_empty() => { - if let Some((certs, key)) = cert_key_pairs.get("default") { - return Ok(Some(ServerTlsMaterial::SingleCert { - certs: certs.clone(), - key: key.clone_key(), - })); - } - Ok(Some(ServerTlsMaterial::MultiCert { cert_key_pairs })) - } - Ok(_) => Ok(None), - Err(_err) if has_root_pair => { - let root_cert_path = path_to_utf8_str(&root_cert, "root TLS certificate")?; - let root_key_path = path_to_utf8_str(&root_key, "root TLS private key")?; - let certs = load_certs(root_cert_path) - .map_err(|e| TlsRuntimeError::Material(format!("load root TLS certificate {}: {e}", root_cert.display())))?; - let key = load_private_key(root_key_path) - .map_err(|e| TlsRuntimeError::Material(format!("load root TLS private key {}: {e}", root_key.display())))?; - Ok(Some(ServerTlsMaterial::SingleCert { certs, key })) - } - Err(err) if err.kind() == ErrorKind::NotFound => Ok(None), - Err(err) => Err(TlsRuntimeError::Material(format!( - "discover server TLS certificates under '{}': {err}", - base_dir.display() - ))), + )?; + if cert_key_pairs.is_empty() { + return Ok(None); } -} - -fn path_to_utf8_str<'a>(path: &'a Path, description: &str) -> Result<&'a str, TlsRuntimeError> { - path.to_str() - .ok_or_else(|| TlsRuntimeError::Material(format!("{description} path '{}' is not valid UTF-8", path.display()))) + if cert_key_pairs.len() == 1 + && let Some((certs, key)) = cert_key_pairs.remove("default") + { + return Ok(Some(ServerTlsMaterial::SingleCert { certs, key })); + } + Ok(Some(ServerTlsMaterial::MultiCert { cert_key_pairs })) } fn combine_optional_pem(primary: Option<&[u8]>, fallback: Option<&[u8]>) -> Vec { @@ -198,3 +280,425 @@ pub(crate) fn server_material_fingerprint(material: &ServerTlsMaterial) -> TlsFi let bytes = serialize_server_material_for_fingerprint(material); TlsFingerprint::from_optional_bytes(Some(&bytes), None, None, None, None) } + +#[cfg(test)] +mod tests { + use super::*; + use rcgen::{CertificateParams, ExtendedKeyUsagePurpose, KeyPair}; + use std::fs; + use tempfile::TempDir; + + fn write_pair(dir: &Path, domain: &str) { + let pair = rcgen::generate_simple_self_signed(vec![domain.to_string()]).expect("generate certificate"); + fs::write(dir.join("rustfs_cert.pem"), pair.cert.pem()).expect("write certificate"); + fs::write(dir.join("rustfs_key.pem"), pair.signing_key.serialize_pem()).expect("write private key"); + } + + #[test] + fn provided_server_identity_does_not_reread_server_files_and_preserves_trust() { + let base = TempDir::new().expect("base source"); + let serving = rcgen::generate_simple_self_signed(vec!["example.com".into()]).expect("server identity"); + let trust = rcgen::generate_simple_self_signed(vec!["trust.example".into()]).expect("trust source"); + fs::write(base.path().join("rustfs_cert.pem"), "unusable server certificate").expect("poison base certificate"); + fs::write(base.path().join("rustfs_key.pem"), "unusable server key").expect("poison base key"); + fs::write(base.path().join(rustfs_config::RUSTFS_CA_CERT), trust.cert.pem()).expect("trust file"); + let source = TlsSource::from_directory(base.path()).snapshot().expect("source"); + let loaded = TlsMaterialSnapshot::load_from_snapshot_with_server_identity( + &source, + vec![serving.cert.der().clone()], + serving.signing_key.serialize_pem().as_bytes(), + ) + .expect("provided identity"); + assert_eq!(loaded.outbound.root_ca_pem, trust.cert.pem().as_bytes()); + assert!( + matches!(loaded.server, Some(ServerTlsMaterial::SingleCert { certs, .. }) if certs == vec![serving.cert.der().clone()]) + ); + fs::write(base.path().join(rustfs_config::RUSTFS_CA_CERT), "corrupt trust").expect("corrupt independent trust"); + assert!( + TlsMaterialSnapshot::load_from_snapshot_with_server_identity( + &source, + vec![serving.cert.der().clone()], + serving.signing_key.serialize_pem().as_bytes() + ) + .is_err(), + "providing server material must not bypass trust validation" + ); + } + + #[test] + fn provided_server_identity_preserves_key_time_and_eku_validation() { + let base = TempDir::new().expect("base source"); + let source = TlsSource::from_directory(base.path()).snapshot().expect("source"); + let serving = rcgen::generate_simple_self_signed(vec!["example.com".into()]).expect("server identity"); + let wrong_key = KeyPair::generate().expect("wrong key"); + assert!( + TlsMaterialSnapshot::load_from_snapshot_with_server_identity( + &source, + vec![serving.cert.der().clone()], + wrong_key.serialize_pem().as_bytes() + ) + .is_err(), + "key matching remains mandatory" + ); + let mut expired = CertificateParams::new(vec!["example.com".into()]).expect("parameters"); + expired.not_before = rcgen::date_time_ymd(2000, 1, 1); + expired.not_after = rcgen::date_time_ymd(2001, 1, 1); + let cert = expired.self_signed(&wrong_key).expect("expired certificate"); + assert!( + TlsMaterialSnapshot::load_from_snapshot_with_server_identity( + &source, + vec![cert.der().clone()], + wrong_key.serialize_pem().as_bytes() + ) + .is_err(), + "serving validity remains mandatory" + ); + let mut client = CertificateParams::new(vec!["example.com".into()]).expect("parameters"); + client.extended_key_usages = vec![ExtendedKeyUsagePurpose::ClientAuth]; + let cert = client.self_signed(&wrong_key).expect("client certificate"); + assert!( + TlsMaterialSnapshot::load_from_snapshot_with_server_identity( + &source, + vec![cert.der().clone()], + wrong_key.serialize_pem().as_bytes() + ) + .is_err(), + "server EKU remains mandatory" + ); + } + + #[tokio::test] + async fn server_overlay_survives_expired_bootstrap_without_replacing_trust() { + let base = TempDir::new().expect("base source"); + let overlay = TempDir::new().expect("server overlay"); + let key = KeyPair::generate().expect("bootstrap key"); + let mut params = CertificateParams::new(vec!["example.com".into()]).expect("bootstrap params"); + params.not_before = rcgen::date_time_ymd(2000, 1, 1); + params.not_after = rcgen::date_time_ymd(2001, 1, 1); + let expired = params.self_signed(&key).expect("expired bootstrap"); + fs::write(base.path().join("rustfs_cert.pem"), expired.pem()).expect("bootstrap certificate"); + fs::write(base.path().join("rustfs_key.pem"), key.serialize_pem()).expect("bootstrap key"); + let ca = rcgen::generate_simple_self_signed(vec!["trust.example".into()]).expect("trust source"); + fs::write(base.path().join(rustfs_config::RUSTFS_CA_CERT), ca.cert.pem()).expect("trust file"); + write_pair(overlay.path(), "example.com"); + let source = TlsSource::from_directory(base.path()); + assert!( + TlsMaterialSnapshot::load(&source).await.is_err(), + "the expired base alone must remain invalid" + ); + let loaded = TlsMaterialSnapshot::load_from_snapshot_with_server_directory( + &source.snapshot().expect("pinned base"), + Some(overlay.path()), + ) + .expect("valid overlay"); + assert_eq!(loaded.outbound.root_ca_pem, ca.cert.pem().as_bytes()); + assert!(matches!(loaded.server, Some(ServerTlsMaterial::SingleCert { .. }))); + } + + #[tokio::test] + async fn snapshot_rejects_mismatched_default_private_key() { + let dir = TempDir::new().expect("create temp directory"); + write_pair(dir.path(), "example.com"); + let other = KeyPair::generate().expect("generate other key"); + fs::write(dir.path().join("rustfs_key.pem"), other.serialize_pem()).expect("replace key"); + assert!( + TlsMaterialSnapshot::load(&TlsSource::from_directory(dir.path())) + .await + .is_err() + ); + } + + #[tokio::test] + async fn snapshot_rejects_partial_sni_pair_even_with_valid_default() { + let dir = TempDir::new().expect("create temp directory"); + write_pair(dir.path(), "example.com"); + let domain = dir.path().join("sni.example.com"); + fs::create_dir(&domain).expect("create SNI directory"); + fs::write(domain.join("rustfs_cert.pem"), "invalid certificate").expect("write partial SNI pair"); + assert!( + TlsMaterialSnapshot::load(&TlsSource::from_directory(dir.path())) + .await + .is_err() + ); + } + + #[tokio::test] + async fn snapshot_rejects_invalid_existing_ca() { + let dir = TempDir::new().expect("create temp directory"); + write_pair(dir.path(), "example.com"); + let source = TlsSource::from_directory(dir.path()); + fs::write(dir.path().join(&source.layout.public_ca_filename), "invalid CA").expect("write invalid CA"); + assert!(TlsMaterialSnapshot::load(&source).await.is_err()); + } + + #[tokio::test] + async fn snapshot_rejects_incomplete_client_identity() { + let dir = TempDir::new().expect("create temp directory"); + write_pair(dir.path(), "example.com"); + let source = TlsSource::from_directory(dir.path()); + fs::write(dir.path().join(&source.layout.client_cert_filename), "invalid client certificate").expect("write client cert"); + assert!(TlsMaterialSnapshot::load(&source).await.is_err()); + } + + #[tokio::test] + async fn snapshot_rejects_expired_server_certificate() { + let dir = TempDir::new().expect("create temp directory"); + let key = KeyPair::generate().expect("generate key"); + let mut params = CertificateParams::new(vec!["example.com".to_string()]).expect("certificate params"); + params.not_before = rcgen::date_time_ymd(2000, 1, 1); + params.not_after = rcgen::date_time_ymd(2001, 1, 1); + let cert = params.self_signed(&key).expect("generate expired certificate"); + fs::write(dir.path().join("rustfs_cert.pem"), cert.pem()).expect("write certificate"); + fs::write(dir.path().join("rustfs_key.pem"), key.serialize_pem()).expect("write key"); + assert!( + TlsMaterialSnapshot::load(&TlsSource::from_directory(dir.path())) + .await + .is_err() + ); + } + + #[tokio::test] + async fn snapshot_rejects_client_only_server_eku() { + let dir = TempDir::new().expect("create temp directory"); + let key = KeyPair::generate().expect("generate key"); + let mut params = CertificateParams::new(vec!["example.com".to_string()]).expect("certificate params"); + params.extended_key_usages = vec![ExtendedKeyUsagePurpose::ClientAuth]; + let cert = params.self_signed(&key).expect("generate client-only certificate"); + fs::write(dir.path().join("rustfs_cert.pem"), cert.pem()).expect("write certificate"); + fs::write(dir.path().join("rustfs_key.pem"), key.serialize_pem()).expect("write key"); + assert!( + TlsMaterialSnapshot::load(&TlsSource::from_directory(dir.path())) + .await + .is_err() + ); + } + + #[tokio::test] + async fn snapshot_rejects_server_only_client_eku() { + let dir = TempDir::new().expect("client identity directory"); + write_pair(dir.path(), "server.example"); + let key = KeyPair::generate().expect("client key"); + let mut params = CertificateParams::new(vec!["client.example".to_string()]).expect("client params"); + params.extended_key_usages = vec![ExtendedKeyUsagePurpose::ServerAuth]; + let cert = params.self_signed(&key).expect("server-only certificate"); + fs::write(dir.path().join("client_cert.pem"), cert.pem()).expect("client certificate"); + fs::write(dir.path().join("client_key.pem"), key.serialize_pem()).expect("client key"); + assert!(matches!( + TlsMaterialSnapshot::load(&TlsSource::from_directory(dir.path())).await, + Err(TlsRuntimeError::InvalidClientEku { .. }) + )); + } + #[tokio::test] + async fn snapshot_rejects_future_server_certificate() { + let dir = TempDir::new().expect("create directory"); + let key = KeyPair::generate().expect("generate key"); + let mut params = CertificateParams::new(vec!["example.com".to_string()]).expect("certificate params"); + params.not_before = rcgen::date_time_ymd(2100, 1, 1); + params.not_after = rcgen::date_time_ymd(2101, 1, 1); + let cert = params.self_signed(&key).expect("generate future certificate"); + fs::write(dir.path().join("rustfs_cert.pem"), cert.pem()).expect("write certificate"); + fs::write(dir.path().join("rustfs_key.pem"), key.serialize_pem()).expect("write key"); + assert!(matches!( + TlsMaterialSnapshot::load(&TlsSource::from_directory(dir.path())).await, + Err(TlsRuntimeError::CertificateValidity { .. }) + )); + } + + #[tokio::test] + async fn snapshot_rejects_wrong_sni_certificate_name() { + let dir = TempDir::new().expect("create directory"); + write_pair(dir.path(), "example.com"); + let domain = dir.path().join("other.example.com"); + fs::create_dir(&domain).expect("create domain directory"); + write_pair(&domain, "wrong.example.com"); + assert!( + TlsMaterialSnapshot::load(&TlsSource::from_directory(dir.path())) + .await + .is_err() + ); + } + + #[tokio::test] + async fn snapshot_rejects_malformed_complete_sni_pair() { + let dir = TempDir::new().expect("create directory"); + write_pair(dir.path(), "example.com"); + let domain = dir.path().join("sni.example.com"); + fs::create_dir(&domain).expect("create domain directory"); + write_pair(&domain, "sni.example.com"); + fs::write(domain.join("rustfs_cert.pem"), "invalid certificate").expect("corrupt domain certificate"); + assert!(matches!( + TlsMaterialSnapshot::load(&TlsSource::from_directory(dir.path())).await, + Err(TlsRuntimeError::InvalidCertificate { .. }) + )); + } + + #[tokio::test] + async fn snapshot_rejects_cert_only_root() { + let dir = TempDir::new().expect("create directory"); + write_pair(dir.path(), "example.com"); + fs::remove_file(dir.path().join("rustfs_key.pem")).expect("remove key"); + assert!(matches!( + TlsMaterialSnapshot::load(&TlsSource::from_directory(dir.path())).await, + Err(TlsRuntimeError::IncompleteIdentity { .. }) + )); + } + + #[tokio::test] + async fn snapshot_fingerprint_is_stable_and_certificate_only_renewal_changes_it() { + let dir = TempDir::new().expect("create directory"); + write_pair(dir.path(), "example.com"); + let source = TlsSource::from_directory(dir.path()); + let initial = TlsMaterialSnapshot::load(&source) + .await + .expect("load valid initial generation"); + let unchanged = TlsMaterialSnapshot::load(&source).await.expect("load unchanged generation"); + assert_eq!(initial.fingerprint, unchanged.fingerprint); + let key = KeyPair::from_pem(&fs::read_to_string(dir.path().join("rustfs_key.pem")).expect("read key")) + .expect("parse existing key"); + let mut params = CertificateParams::new(vec!["example.com".to_string()]).expect("certificate params"); + params.serial_number = Some(123_u64.into()); + let renewed = params.self_signed(&key).expect("renew certificate with existing key"); + fs::write(dir.path().join("rustfs_cert.pem"), renewed.pem()).expect("write renewed certificate"); + let changed = TlsMaterialSnapshot::load(&source).await.expect("load renewed generation"); + assert_ne!(initial.fingerprint, changed.fingerprint); + } + + #[tokio::test] + async fn snapshot_server_client_ca_rotation_changes_fingerprint() { + let dir = TempDir::new().expect("create directory"); + write_pair(dir.path(), "example.com"); + let source = TlsSource::from_directory(dir.path()); + let first_ca = rcgen::generate_simple_self_signed(vec!["first-ca".to_string()]).expect("first CA"); + fs::write(dir.path().join(&source.layout.client_ca_filename), first_ca.cert.pem()).expect("write first CA"); + let initial = TlsMaterialSnapshot::load(&source).await.expect("load first CA"); + let second_ca = rcgen::generate_simple_self_signed(vec!["second-ca".to_string()]).expect("second CA"); + fs::write(dir.path().join(&source.layout.client_ca_filename), second_ca.cert.pem()).expect("write second CA"); + let changed = TlsMaterialSnapshot::load(&source).await.expect("load second CA"); + assert_ne!(initial.fingerprint, changed.fingerprint); + assert_eq!(changed.server_client_ca_pem.as_deref(), Some(second_ca.cert.pem().as_bytes())); + } + + #[cfg(unix)] + #[test] + fn snapshot_kubernetes_switch_loads_all_material_from_pinned_generation() { + use std::os::unix::fs::symlink; + let dir = TempDir::new().expect("create projection"); + let source = TlsSource::from_directory(dir.path()); + for generation in ["..a", "..b"] { + let path = dir.path().join(generation); + fs::create_dir(&path).expect("create generation"); + write_pair(&path, "example.com"); + let cert = fs::read(path.join(&source.layout.server_cert_filename)).expect("read generated certificate"); + let key = fs::read(path.join(&source.layout.server_key_filename)).expect("read generated key"); + for filename in [ + &source.layout.public_ca_filename, + &source.fallback_ca_filename, + &source.layout.client_ca_filename, + &source.layout.client_cert_filename, + ] { + fs::write(path.join(filename), &cert).expect("write generation CA/client certificate"); + } + fs::write(path.join(&source.layout.client_key_filename), key).expect("write generation client key"); + } + symlink("..a", dir.path().join("..data")).expect("link initial generation"); + for filename in [ + &source.layout.server_cert_filename, + &source.layout.server_key_filename, + &source.layout.public_ca_filename, + &source.fallback_ca_filename, + &source.layout.client_ca_filename, + &source.layout.client_cert_filename, + &source.layout.client_key_filename, + ] { + symlink(format!("..data/{filename}"), dir.path().join(filename)).expect("link projected file"); + } + let pinned = source.snapshot().expect("pin initial generation"); + let expected = TlsMaterialSnapshot::load_from_snapshot(&pinned).expect("load initial generation"); + symlink("..b", dir.path().join("..next")).expect("link next generation"); + fs::rename(dir.path().join("..next"), dir.path().join("..data")).expect("switch projection"); + let actual = TlsMaterialSnapshot::load_from_snapshot(&pinned).expect("load pinned generation after switch"); + assert_eq!(actual.fingerprint, expected.fingerprint); + assert_eq!(actual.source.base_dir, pinned.base_dir()); + assert_eq!(actual.outbound.root_ca_pem, expected.outbound.root_ca_pem); + assert_eq!(actual.server_client_ca_pem, expected.server_client_ca_pem); + assert_eq!( + actual.outbound.mtls_identity.expect("client identity").key_pem, + expected.outbound.mtls_identity.expect("expected identity").key_pem + ); + let current = TlsMaterialSnapshot::load_from_snapshot(&source.snapshot().expect("pin new generation")) + .expect("load new generation"); + assert_ne!(actual.fingerprint, current.fingerprint); + } + + #[cfg(unix)] + #[test] + fn snapshot_rejects_disappeared_projection_source_and_dangling_optional_ca() { + use std::os::unix::fs::symlink; + let dir = TempDir::new().expect("create directory"); + let source = TlsSource::from_directory(dir.path()); + write_pair(dir.path(), "example.com"); + symlink("missing-ca", dir.path().join(&source.layout.public_ca_filename)).expect("link missing CA"); + let pinned = source.snapshot().expect("pin directory"); + assert!(matches!( + TlsMaterialSnapshot::load_from_snapshot(&pinned), + Err(TlsRuntimeError::SourceIo { .. }) + )); + fs::remove_file(dir.path().join(&source.layout.public_ca_filename)).expect("remove CA link"); + fs::remove_dir_all(dir.path()).expect("remove pinned source"); + assert!(matches!( + TlsMaterialSnapshot::load_from_snapshot(&pinned), + Err(TlsRuntimeError::SourceIo { .. }) + )); + } + #[tokio::test] + async fn snapshot_rejects_mismatched_sni_key_and_malformed_chain_member() { + let dir = TempDir::new().expect("create directory"); + write_pair(dir.path(), "example.com"); + let domain = dir.path().join("sni.example.com"); + fs::create_dir(&domain).expect("create SNI directory"); + write_pair(&domain, "sni.example.com"); + let source = TlsSource::from_directory(dir.path()); + let valid = TlsMaterialSnapshot::load(&source) + .await + .expect("load valid multi certificate material"); + assert!(matches!(valid.server, Some(ServerTlsMaterial::MultiCert { ref cert_key_pairs }) if cert_key_pairs.len() == 2)); + fs::write( + domain.join("rustfs_key.pem"), + KeyPair::generate().expect("generate different key").serialize_pem(), + ) + .expect("replace SNI key"); + assert!(matches!( + TlsMaterialSnapshot::load(&source).await, + Err(TlsRuntimeError::InvalidCertifiedKey { .. }) + )); + write_pair(&domain, "sni.example.com"); + let cert_path = dir.path().join("rustfs_cert.pem"); + let mut cert = fs::read_to_string(&cert_path).expect("read leaf certificate"); + cert.push_str("-----BEGIN CERTIFICATE-----\nAQID\n-----END CERTIFICATE-----\n"); + fs::write(cert_path, cert).expect("append malformed chain member"); + assert!(matches!( + TlsMaterialSnapshot::load(&source).await, + Err(TlsRuntimeError::InvalidCertificate { .. }) + )); + } + + #[tokio::test] + async fn snapshot_rejects_invalid_private_key_pem_and_invalid_primary_client_ca() { + let dir = TempDir::new().expect("create directory"); + write_pair(dir.path(), "example.com"); + let source = TlsSource::from_directory(dir.path()); + fs::write(dir.path().join("rustfs_key.pem"), "invalid key").expect("corrupt key"); + assert!(matches!( + TlsMaterialSnapshot::load(&source).await, + Err(TlsRuntimeError::InvalidPrivateKey { .. }) + )); + write_pair(dir.path(), "example.com"); + let valid_ca = fs::read(dir.path().join("rustfs_cert.pem")).expect("read valid certificate"); + fs::write(dir.path().join(&source.fallback_ca_filename), valid_ca).expect("write valid fallback CA"); + fs::write(dir.path().join(&source.layout.client_ca_filename), "invalid primary CA").expect("write invalid primary CA"); + assert!(matches!( + TlsMaterialSnapshot::load(&source).await, + Err(TlsRuntimeError::InvalidCertificate { .. }) + )); + } +} diff --git a/crates/tls-runtime/src/outbound.rs b/crates/tls-runtime/src/outbound.rs index 11c9c5916..fac3a65cb 100644 --- a/crates/tls-runtime/src/outbound.rs +++ b/crates/tls-runtime/src/outbound.rs @@ -16,8 +16,8 @@ use crate::material::OutboundTlsMaterial; use crate::metrics::record_outbound_tls_publication; use crate::state::TlsGeneration; use rustfs_common::{ - MtlsIdentityPem, clear_global_root_cert, get_global_mtls_identity, get_global_outbound_tls_generation, get_global_root_cert, - set_global_mtls_identity, set_global_outbound_tls_generation, set_global_root_cert, + MtlsIdentityPem, OutboundTlsSnapshot, get_global_outbound_tls_generation, get_global_outbound_tls_snapshot, + publish_global_outbound_tls_snapshot, }; #[derive(Debug, Clone)] @@ -35,21 +35,21 @@ pub struct GlobalOutboundTlsStateSummary { } pub async fn publish_global_outbound_tls_state(generation: TlsGeneration, material: &OutboundTlsMaterial) { - if !material.root_ca_pem.is_empty() { - set_global_root_cert(material.root_ca_pem.clone()).await; - } else { - clear_global_root_cert().await; - } - set_global_mtls_identity(material.mtls_identity.clone()).await; - set_global_outbound_tls_generation(generation.0); - record_outbound_tls_publication(generation.0, !material.root_ca_pem.is_empty(), material.mtls_identity.is_some()); + let accepted = publish_global_outbound_tls_snapshot(OutboundTlsSnapshot { + generation: generation.0, + root_ca_pem: (!material.root_ca_pem.is_empty()).then(|| material.root_ca_pem.clone()), + mtls_identity: material.mtls_identity.clone(), + }) + .await; + record_outbound_tls_publication(accepted, !material.root_ca_pem.is_empty(), material.mtls_identity.is_some()); } pub async fn load_global_outbound_tls_state() -> GlobalPublishedOutboundTlsState { + let snapshot = get_global_outbound_tls_snapshot(); GlobalPublishedOutboundTlsState { - generation: TlsGeneration(get_global_outbound_tls_generation()), - root_ca_pem: get_global_root_cert().await, - mtls_identity: get_global_mtls_identity().await, + generation: TlsGeneration(snapshot.generation), + root_ca_pem: snapshot.root_ca_pem.clone(), + mtls_identity: snapshot.mtls_identity.clone(), } } diff --git a/crates/tls-runtime/src/source.rs b/crates/tls-runtime/src/source.rs index dee0b0277..a35b68001 100644 --- a/crates/tls-runtime/src/source.rs +++ b/crates/tls-runtime/src/source.rs @@ -92,3 +92,125 @@ impl TlsSource { Ok(self.base_dir.as_path()) } } + +/// A directory generation resolved once before loading any TLS material. +/// +/// Projected Kubernetes volumes are read through their physical `..data` +/// directory. Ordinary directory and certificate-file symlinks remain supported +/// for layouts such as Certbot. Writers must keep pinned generations immutable. +#[derive(Debug, Clone)] +pub struct TlsSourceSnapshot { + source: TlsSource, + resolved_source: TlsSource, +} + +impl TlsSource { + pub fn snapshot(&self) -> Result { + self.validate_directory()?; + let mut resolved_source = self.clone(); + let root = std::fs::canonicalize(&self.base_dir).map_err(|source| TlsRuntimeError::SourceIo { + path: self.base_dir.clone(), + source, + })?; + let projection = root.join("..data"); + resolved_source.base_dir = match std::fs::symlink_metadata(&projection) { + Ok(metadata) if metadata.file_type().is_symlink() => { + std::fs::canonicalize(&projection).map_err(|source| TlsRuntimeError::SourceIo { + path: projection, + source, + })? + } + Ok(_) => root, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => root, + Err(source) => { + return Err(TlsRuntimeError::SourceIo { + path: projection, + source, + }); + } + }; + resolved_source.validate_directory()?; + Ok(TlsSourceSnapshot { + source: self.clone(), + resolved_source, + }) + } +} + +impl TlsSourceSnapshot { + pub fn source(&self) -> &TlsSource { + &self.source + } + + pub fn resolved_source(&self) -> &TlsSource { + &self.resolved_source + } + + pub fn base_dir(&self) -> &Path { + &self.resolved_source.base_dir + } + + /// Missing optional files are allowed only while the pinned source exists. + pub(crate) fn validate_available(&self) -> Result<(), TlsRuntimeError> { + std::fs::read_dir(self.base_dir()).map_err(|source| TlsRuntimeError::SourceIo { + path: self.base_dir().to_path_buf(), + source, + })?; + Ok(()) + } +} + +#[cfg(all(test, unix))] +mod tests { + use super::*; + use std::fs; + use std::os::unix::fs::symlink; + use tempfile::TempDir; + + #[test] + fn source_snapshot_pins_kubernetes_projection_generation() { + let dir = TempDir::new().expect("create projection directory"); + for generation in ["..generation-a", "..generation-b"] { + fs::create_dir(dir.path().join(generation)).expect("create generation"); + fs::write(dir.path().join(generation).join("marker"), generation).expect("write generation marker"); + } + symlink("..generation-a", dir.path().join("..data")).expect("link initial generation"); + symlink("..data/marker", dir.path().join("marker")).expect("link projected file"); + let pinned = TlsSource::from_directory(dir.path()).snapshot().expect("pin projection"); + symlink("..generation-b", dir.path().join("..next-data")).expect("link next generation"); + fs::rename(dir.path().join("..next-data"), dir.path().join("..data")).expect("swap projection"); + assert_eq!( + fs::read_to_string(pinned.base_dir().join("marker")).expect("read pinned file"), + "..generation-a" + ); + assert_eq!( + fs::read_to_string(dir.path().join("marker")).expect("read current projection"), + "..generation-b" + ); + } + + #[test] + fn source_snapshot_pins_tls_path_directory_symlink() { + let dir = TempDir::new().expect("create TLS directory"); + fs::create_dir(dir.path().join("a")).expect("create generation A"); + fs::create_dir(dir.path().join("b")).expect("create generation B"); + symlink("a", dir.path().join("live")).expect("link generation A"); + let pinned = TlsSource::from_directory(dir.path().join("live")) + .snapshot() + .expect("pin TLS source"); + fs::remove_file(dir.path().join("live")).expect("unlink source"); + symlink("b", dir.path().join("live")).expect("link generation B"); + assert_eq!(pinned.base_dir(), fs::canonicalize(dir.path().join("a")).expect("resolve generation A")); + } + + #[test] + fn source_snapshot_rejects_disappeared_generation() { + let dir = TempDir::new().expect("create TLS directory"); + let generation = dir.path().join("..generation"); + fs::create_dir(&generation).expect("create generation"); + symlink("..generation", dir.path().join("..data")).expect("link generation"); + let pinned = TlsSource::from_directory(dir.path()).snapshot().expect("pin TLS source"); + fs::remove_dir(&generation).expect("remove pinned generation"); + assert!(matches!(pinned.validate_available(), Err(TlsRuntimeError::SourceIo { .. }))); + } +} diff --git a/crates/utils/src/dunce.rs b/crates/utils/src/dunce.rs index b6405057c..42133cde1 100644 --- a/crates/utils/src/dunce.rs +++ b/crates/utils/src/dunce.rs @@ -21,6 +21,7 @@ use std::ffi::OsStr; #[cfg(windows)] use std::path::{Component, Prefix}; use std::{ + borrow::Cow, fs, io, path::{Path, PathBuf}, }; @@ -106,6 +107,115 @@ fn canonicalize_win(path: &Path) -> io::Result { Ok(try_simplified(&real_path).map(PathBuf::from).unwrap_or(real_path)) } +/// Returns a form of `path` that Win32 file APIs accept beyond `MAX_PATH`. +/// +/// `std::fs` adds the `\\?\` prefix internally, but crates that call Win32 +/// directly with the caller's path (for example `tempfile`'s `persist`, which +/// uses `MoveFileExW`) fail with `ERROR_PATH_NOT_FOUND` once the path reaches +/// 260 UTF-16 units, unless the process opted into long paths through both the +/// `LongPathsEnabled` policy and an application manifest. +/// +/// The path is first made absolute with `GetFullPathNameW` (via +/// [`std::path::absolute`]), so separators, `.`/`..` and trailing dots or spaces +/// are normalized exactly as a non-verbatim Win32 call would see them, and only +/// then prefixed with `\\?\` (or `\\?\UNC\`). Verbatim and device paths are +/// returned unchanged. +/// +/// On non-Windows this is a no-op. +pub fn extended_length_path(path: &Path) -> io::Result> { + #[cfg(not(windows))] + { + Ok(Cow::Borrowed(path)) + } + #[cfg(windows)] + { + extended_length_path_win(path).map(Cow::Owned) + } +} + +#[cfg(windows)] +fn extended_length_path_win(path: &Path) -> io::Result { + use std::{ + ffi::OsString, + os::windows::ffi::{OsStrExt, OsStringExt}, + }; + + let absolute = std::path::absolute(path)?; + let unc = match absolute.components().next() { + Some(Component::Prefix(prefix)) => match prefix.kind() { + Prefix::Disk(_) => false, + Prefix::UNC(..) => true, + // Verbatim and `\\.\` device paths already bypass Win32 normalization. + _ => return Ok(absolute), + }, + _ => return Ok(absolute), + }; + let wide = absolute.as_os_str().encode_wide().collect::>(); + let (prefix, rest) = if unc { + // `\\server\share\x` becomes `\\?\UNC\server\share\x`. + (r"\\?\UNC", &wide[1..]) + } else { + (r"\\?\", &wide[..]) + }; + let mut extended = prefix.encode_utf16().collect::>(); + extended.extend_from_slice(rest); + Ok(PathBuf::from(OsString::from_wide(&extended))) +} + +/// Like [`simplified`], but without the `MAX_PATH` limit, and also turning +/// `\\?\UNC\server\share\...` into `\\server\share\...`. +/// +/// Use it for a root that callers extend by joining `/`-separated relative +/// paths as strings: inside a verbatim path `/` is not a separator, so every +/// such join fails with `ERROR_INVALID_NAME`. `std::fs` re-adds the +/// extended-length prefix to long paths, and code handing paths to Win32 +/// directly goes through [`extended_length_path`]. A path with a component +/// that Win32 would rewrite (reserved names, trailing dots or spaces, ...) or +/// that is not valid UTF-8 is returned unchanged, as are volume GUID and +/// device paths. +/// +/// On non-Windows this is a no-op. +#[must_use] +pub fn non_verbatim(path: &Path) -> Cow<'_, Path> { + #[cfg(not(windows))] + { + Cow::Borrowed(path) + } + #[cfg(windows)] + { + try_non_verbatim(path).map_or(Cow::Borrowed(path), Cow::Owned) + } +} + +#[cfg(windows)] +fn try_non_verbatim(path: &Path) -> Option { + let mut components = path.components(); + let unc = match components.next() { + Some(Component::Prefix(prefix)) => match prefix.kind() { + Prefix::VerbatimDisk(..) => false, + Prefix::VerbatimUNC(..) => true, + _ => return None, + }, + _ => return None, + }; + for component in components { + match component { + Component::RootDir => {} + Component::Normal(file_name) if is_valid_filename(file_name) && !is_reserved(file_name) => {} + _ => return None, + } + } + + let s = path.to_str()?; + Some(if unc { + // `\\?\UNC\server\share` (8-character prefix) becomes `\\server\share`. + PathBuf::from(format!(r"\\{}", s.get(8..)?)) + } else { + // `\\?\C:\x` (4-character prefix) becomes `C:\x`. + PathBuf::from(s.get(4..)?) + }) +} + /// Returns `true` if the path is relative or starts with a disk prefix (like "C:"). /// /// This checks if the path is in a "simplified" format. On Windows, this means it @@ -134,6 +244,64 @@ pub fn is_simplified(path: &Path) -> bool { #[doc(hidden)] pub use self::canonicalize as realpath; +/// Why a single path segment cannot be used as a regular Win32 file name. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum WindowsFileNameError { + /// The segment is empty. + Empty, + /// The segment exceeds 255 UTF-16 code units. + TooLong, + /// The segment contains one of `< > : " / \ | ? *`. + ReservedCharacter, + /// The segment contains a control character (0x00-0x1F). + ControlCharacter, + /// The segment ends with `.` or a space, which Win32 path parsing strips. + TrailingDotOrSpace, + /// The segment names a DOS device (e.g. `NUL`, `COM1`, `CONIN$`), bare or + /// with an extension. + ReservedDeviceName, +} + +/// Checks whether `name` can be created *and* addressed again as a single +/// path segment through the Win32 API. +/// +/// This is the one rule set for Windows file-name validity: it backs +/// [`simplified()`] and the object-key admission checks in ecstore. The rules +/// apply on every platform so callers can evaluate them without a Windows +/// host; whether to enforce them is the caller's decision. +pub fn check_windows_file_name(name: &str) -> Result<(), WindowsFileNameError> { + // Windows filename limit is 255 characters (UTF-16 code units) + // We check byte length first as a fast path optimization. + if name.len() > 255 && name.encode_utf16().count() > 255 { + return Err(WindowsFileNameError::TooLong); + } + + if name.is_empty() { + return Err(WindowsFileNameError::Empty); + } + + // Only ASCII subset is checked, and WTF-8/UTF-8 is safe for that + // Invalid chars: < > : " / \ | ? * and control chars (0-31) + let byte_str = name.as_bytes(); + if byte_str + .iter() + .any(|&c| matches!(c, b'<' | b'>' | b':' | b'"' | b'/' | b'\\' | b'|' | b'?' | b'*')) + { + return Err(WindowsFileNameError::ReservedCharacter); + } + if byte_str.iter().any(|&c| c <= 31) { + return Err(WindowsFileNameError::ControlCharacter); + } + // Filename can't end with . or space + if matches!(byte_str.last(), Some(b' ' | b'.')) { + return Err(WindowsFileNameError::TrailingDotOrSpace); + } + if is_reserved_name(name) { + return Err(WindowsFileNameError::ReservedDeviceName); + } + Ok(()) +} + /// Checks if the filename is valid for a regular Windows path. /// /// This checks for: @@ -149,30 +317,7 @@ fn is_valid_filename(file_name: &OsStr) -> bool { None => return false, }; - // Windows filename limit is 255 characters (UTF-16 code units) - // We check byte length first as a fast path optimization. - if s.len() > 255 && s.encode_utf16().count() > 255 { - return false; - } - - if s.is_empty() { - return false; - } - - // Only ASCII subset is checked, and WTF-8/UTF-8 is safe for that - // Invalid chars: < > : " / \ | ? * and control chars (0-31) - let byte_str = s.as_bytes(); - if byte_str - .iter() - .any(|&c| matches!(c, 0..=31 | b'<' | b'>' | b':' | b'"' | b'/' | b'\\' | b'|' | b'?' | b'*')) - { - return false; - } - // Filename can't end with . or space - if matches!(byte_str.last(), Some(b' ' | b'.')) { - return false; - } - true + matches!(check_windows_file_name(s), Ok(()) | Err(WindowsFileNameError::ReservedDeviceName)) } /// Checks if the filename is a reserved DOS device name (e.g., CON, PRN, COM1). @@ -180,11 +325,13 @@ fn is_valid_filename(file_name: &OsStr) -> bool { /// Reserved names are case-insensitive and apply even if an extension is present (e.g., "CON.txt" is reserved). #[cfg(any(windows, test))] fn is_reserved>(file_name: P) -> bool { - let s = match file_name.as_ref().to_str() { - Some(s) => s, - None => return false, // Reserved names are all ASCII - }; + match file_name.as_ref().to_str() { + Some(s) => is_reserved_name(s), + None => false, // Reserved names are all ASCII + } +} +fn is_reserved_name(s: &str) -> bool { if s.is_empty() { return false; } @@ -200,25 +347,26 @@ fn is_reserved>(file_name: P) -> bool { // Trim trailing spaces (Windows ignores them, so "CON " is "CON") let stem = stem.trim_end(); - // Reserved names are 3 or 4 chars - if stem.len() != 3 && stem.len() != 4 { + if ["CON", "PRN", "AUX", "NUL", "CONIN$", "CONOUT$"] + .iter() + .any(|name| stem.eq_ignore_ascii_case(name)) + { + return true; + } + + // Check COM1-9, LPT1-9, and the ISO 8859-1 superscript digits Windows + // also accepts as port numbers (COM¹, LPT³, ...). + let (Some(base), Some(suffix)) = (stem.get(..3), stem.get(3..)) else { + return false; + }; + if !base.eq_ignore_ascii_case("COM") && !base.eq_ignore_ascii_case("LPT") { return false; } - - if stem.len() == 3 { - return stem.eq_ignore_ascii_case("CON") - || stem.eq_ignore_ascii_case("PRN") - || stem.eq_ignore_ascii_case("AUX") - || stem.eq_ignore_ascii_case("NUL"); + let mut digits = suffix.chars(); + match (digits.next(), digits.next()) { + (Some(digit), None) => matches!(digit, '1'..='9' | '\u{b9}' | '\u{b2}' | '\u{b3}'), + _ => false, } - - // Check COM1-9, LPT1-9 - let (base, suffix) = stem.split_at(3); - if base.eq_ignore_ascii_case("COM") || base.eq_ignore_ascii_case("LPT") { - return suffix.chars().next().map(|c| c.is_ascii_digit() && c != '0').unwrap_or(false); - } - - false } #[cfg(not(windows))] @@ -306,14 +454,39 @@ mod tests { "CON.tar.gz", "con.", "con .", + "COM\u{b9}", + "com\u{b2}.txt", + "LPT\u{b3}", + "CONIN$", + "conout$", + "CONIN$.txt", ]; for name in reserved { assert!(is_reserved(name), "Should be reserved: {}", name); } let not_reserved = [ - "CON0", "COM0", "LPT0", "COM10", "LPT10", "CONsole", "PrNter", "NuLly", "not_con", "con_not", ".CON", "@CON", - "CON。", "foo.con", + "CON0", + "COM0", + "LPT0", + "COM10", + "LPT10", + "CONsole", + "PrNter", + "NuLly", + "not_con", + "con_not", + ".CON", + "@CON", + "CON。", + "foo.con", + "COM\u{b9}0", + "COM\u{bd}", + "CONIN", + "CONIN$$", + "中文名", + "é12", + "COMé", ]; for name in not_reserved { assert!(!is_reserved(name), "Should NOT be reserved: {}", name); @@ -367,6 +540,25 @@ mod tests { assert!(is_valid_filename(OsStr::new(&ok_name))); } + #[test] + fn test_check_windows_file_name_reports_the_violated_rule() { + assert_eq!(check_windows_file_name("file.txt"), Ok(())); + assert_eq!(check_windows_file_name(""), Err(WindowsFileNameError::Empty)); + assert_eq!(check_windows_file_name(&"a".repeat(256)), Err(WindowsFileNameError::TooLong)); + for name in ["a:b", "a\\b", "a/b", "a*b", "a?b", "a\"b", "ab", "a|b"] { + assert_eq!(check_windows_file_name(name), Err(WindowsFileNameError::ReservedCharacter), "{name:?}"); + } + for name in ["a\0b", "a\x01b", "tab\there", "a\x1fb"] { + assert_eq!(check_windows_file_name(name), Err(WindowsFileNameError::ControlCharacter), "{name:?}"); + } + for name in ["dir.", "file ", "..", "CON."] { + assert_eq!(check_windows_file_name(name), Err(WindowsFileNameError::TrailingDotOrSpace), "{name:?}"); + } + for name in ["NUL", "nul.txt", "COM\u{b9}", "CONOUT$"] { + assert_eq!(check_windows_file_name(name), Err(WindowsFileNameError::ReservedDeviceName), "{name:?}"); + } + } + #[test] #[cfg(windows)] fn test_simplified_windows() { @@ -390,6 +582,70 @@ mod tests { assert_eq!(simplified(Path::new(r"\\.\C:\Windows")), Path::new(r"\\.\C:\Windows")); } + #[test] + #[cfg(windows)] + fn extended_length_path_prefixes_normalized_windows_paths() { + assert_eq!( + &*extended_length_path(Path::new(r"C:/data/./bucket/../bucket/object")).unwrap(), + Path::new(r"\\?\C:\data\bucket\object") + ); + assert_eq!( + &*extended_length_path(Path::new(r"\\server\share\data\object")).unwrap(), + Path::new(r"\\?\UNC\server\share\data\object") + ); + for unchanged in [r"\\?\C:\data\object", r"\\?\UNC\server\share\object", r"\\.\C:\data"] { + assert_eq!(&*extended_length_path(Path::new(unchanged)).unwrap(), Path::new(unchanged)); + } + + let long = format!(r"C:\data\{}\xl.meta.bkp", "a".repeat(300)); + let extended = extended_length_path(Path::new(&long)).unwrap(); + assert_eq!(&*extended, Path::new(&format!(r"\\?\{long}"))); + assert_eq!(simplified(&extended), &*extended, "a long path must stay verbatim"); + } + + #[test] + #[cfg(windows)] + fn non_verbatim_strips_verbatim_prefixes_without_the_max_path_limit() { + let long = format!(r"C:\data\{}\root", "a".repeat(250)); + assert_eq!(&*non_verbatim(Path::new(&format!(r"\\?\{long}"))), Path::new(&long)); + assert_eq!( + &*non_verbatim(Path::new(r"\\?\UNC\localhost\C$\rfs\smb")), + Path::new(r"\\localhost\C$\rfs\smb") + ); + let long_unc = format!(r"\\?\UNC\server\share\{}", "b".repeat(250)); + assert_eq!(&*non_verbatim(Path::new(&long_unc)), Path::new(&format!(r"\\{}", &long_unc[8..]))); + + for unchanged in [ + r"C:\data\root", + r"\\server\share\root", + r"\\?\C:\data\CON", + r"\\?\C:\data\trailing.", + r"\\?\C:\data\..\root", + r"\\?\Volume{00000000-0000-0000-0000-000000000000}\root", + r"\\?\GLOBALROOT\Device\HarddiskVolume1\root", + r"\\.\C:\data", + ] { + assert!( + matches!(non_verbatim(Path::new(unchanged)), Cow::Borrowed(p) if p == Path::new(unchanged)), + "{unchanged} must be returned unchanged" + ); + } + } + + #[test] + #[cfg(not(windows))] + fn non_verbatim_is_a_noop_off_windows() { + let path = Path::new("/data/root"); + assert!(matches!(non_verbatim(path), Cow::Borrowed(p) if p == path)); + } + + #[test] + #[cfg(not(windows))] + fn extended_length_path_is_a_noop_off_windows() { + let path = Path::new("/data/./bucket/object"); + assert!(matches!(extended_length_path(path).unwrap(), Cow::Borrowed(p) if p == path)); + } + #[test] #[cfg(not(windows))] fn test_simplified_unix() { diff --git a/crates/utils/src/http/metadata_compat.rs b/crates/utils/src/http/metadata_compat.rs index 305933cd8..8ae681427 100644 --- a/crates/utils/src/http/metadata_compat.rs +++ b/crates/utils/src/http/metadata_compat.rs @@ -223,6 +223,19 @@ pub fn insert_str(map: &mut HashMap, suffix: &str, value: String map.insert(k2, value); } +fn eq_prefix_suffix_ignore_ascii_case(key: &str, prefix: &str, suffix: &str) -> bool { + if key.len() != prefix.len() + suffix.len() { + return false; + } + let (key_prefix, key_suffix) = key.as_bytes().split_at(prefix.len()); + key_prefix.eq_ignore_ascii_case(prefix.as_bytes()) && key_suffix.eq_ignore_ascii_case(suffix.as_bytes()) +} + +fn matches_either_internal_key_ignore_ascii_case(key: &str, suffix: &str) -> bool { + eq_prefix_suffix_ignore_ascii_case(key, RUSTFS_INTERNAL_PREFIX, suffix) + || eq_prefix_suffix_ignore_ascii_case(key, MINIO_INTERNAL_PREFIX, suffix) +} + pub fn get_str(map: &HashMap, suffix: &str) -> Option { if let Some(v) = with_internal_key(RUSTFS_INTERNAL_PREFIX, suffix, |k1| map.get(k1).cloned()) { return Some(v); @@ -231,17 +244,15 @@ pub fn get_str(map: &HashMap, suffix: &str) -> Option { return Some(v); } // Rare fallback: case-insensitive scan for non-canonical key casing. - let (k1, k2) = both_keys(suffix); map.iter() - .find(|(key, _)| key.eq_ignore_ascii_case(&k1) || key.eq_ignore_ascii_case(&k2)) + .find(|(key, _)| matches_either_internal_key_ignore_ascii_case(key, suffix)) .map(|(_, value)| value.clone()) } fn get_consistent_value<'a, V: AsRef<[u8]>>(map: &'a HashMap, suffix: &str) -> Option<&'a V> { - let (rustfs_key, minio_key) = both_keys(suffix); let mut value = None; for (key, candidate) in map { - if !key.eq_ignore_ascii_case(&rustfs_key) && !key.eq_ignore_ascii_case(&minio_key) { + if !matches_either_internal_key_ignore_ascii_case(key, suffix) { continue; } if candidate.as_ref().is_empty() || value.is_some_and(|current: &V| current.as_ref() != candidate.as_ref()) { @@ -266,16 +277,14 @@ pub fn contains_key_str(map: &HashMap, suffix: &str) -> bool { if with_internal_key(MINIO_INTERNAL_PREFIX, suffix, |k2| map.contains_key(k2)) { return true; } - let (k1, k2) = both_keys(suffix); map.keys() - .any(|key| key.eq_ignore_ascii_case(&k1) || key.eq_ignore_ascii_case(&k2)) + .any(|key| matches_either_internal_key_ignore_ascii_case(key, suffix)) } pub fn remove_str(map: &mut HashMap, suffix: &str) { with_internal_key(RUSTFS_INTERNAL_PREFIX, suffix, |k1| map.remove(k1)); with_internal_key(MINIO_INTERNAL_PREFIX, suffix, |k2| map.remove(k2)); - let (k1, k2) = both_keys(suffix); - map.retain(|key, _| !key.eq_ignore_ascii_case(&k1) && !key.eq_ignore_ascii_case(&k2)); + map.retain(|key, _| !matches_either_internal_key_ignore_ascii_case(key, suffix)); } // === Vec type (meta_sys) === @@ -458,6 +467,16 @@ mod tests { assert_eq!(get_consistent_str(&metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID), None); } + #[test] + fn test_consistent_str_rejects_conflicting_mixed_case_aliases() { + let metadata = HashMap::from([ + ("X-RustFS-Internal-transition-tier-destination-id".to_string(), "identity-a".to_string()), + ("X-Minio-Internal-Transition-Tier-Destination-Id".to_string(), "identity-b".to_string()), + ]); + + assert_eq!(get_consistent_str(&metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID), None); + } + #[test] fn test_consistent_bytes_accepts_single_or_matching_values_and_rejects_conflicts() { let rustfs_key = internal_key_rustfs(SUFFIX_TRANSITION_TIER_DESTINATION_ID); diff --git a/crates/utils/src/os/mod.rs b/crates/utils/src/os/mod.rs index 0d5ed3c22..1708dae36 100644 --- a/crates/utils/src/os/mod.rs +++ b/crates/utils/src/os/mod.rs @@ -64,6 +64,13 @@ pub struct DiskInfo { pub files: u64, pub ffree: u64, pub fstype: String, + /// The volume is reached through a network redirector. + /// + /// Windows names the server's filesystem (usually `NTFS`) in `fstype` for + /// SMB shares, so remote volumes are reported here instead. On Unix the + /// client filesystem type (`nfs`, `cifs`, `smb2`, ...) already identifies + /// network mounts and this stays `false`. + pub remote: bool, pub major: u64, pub minor: u64, pub name: String, diff --git a/crates/utils/src/os/windows.rs b/crates/utils/src/os/windows.rs index 237adbed3..84d1b1023 100644 --- a/crates/utils/src/os/windows.rs +++ b/crates/utils/src/os/windows.rs @@ -13,33 +13,113 @@ // limitations under the License. use crate::os::{DiskInfo, IOStats}; -use std::io::Error; -use std::path::Path; -use windows::Win32::Foundation::MAX_PATH; -use windows::Win32::Storage::FileSystem::{GetDiskFreeSpaceExW, GetDiskFreeSpaceW, GetVolumeInformationW, GetVolumePathNameW}; +use std::io::{Error, ErrorKind}; +use std::os::windows::ffi::OsStrExt; +use std::os::windows::fs::OpenOptionsExt; +use std::os::windows::io::AsRawHandle; +use std::path::{Component, Path, Prefix}; +use windows::Win32::Foundation::{HANDLE, MAX_PATH}; +use windows::Win32::Storage::FileSystem::{ + FILE_FLAG_BACKUP_SEMANTICS, FILE_REMOTE_PROTOCOL_INFO, FileRemoteProtocolInfo, GetDiskFreeSpaceExW, GetDiskFreeSpaceW, + GetDriveTypeW, GetFileInformationByHandleEx, GetFinalPathNameByHandleW, GetVolumeInformationW, GetVolumePathNameW, + VOLUME_NAME_NT, +}; +use windows::core::PCWSTR; + +/// Documented maximum for the `GetVolumeInformationW` name buffers. +const VOLUME_INFORMATION_BUFFER_LEN: usize = MAX_PATH as usize + 1; + +/// `GetDriveTypeW` result for network drives; defined locally because the +/// binding lives behind the unrelated `Win32_System_WindowsProgramming` feature. +const DRIVE_REMOTE: u32 = 4; + +const BACKSLASH: u16 = b'\\' as u16; +const SLASH: u16 = b'/' as u16; + +/// A NUL-terminated UTF-16 path that is safe to hand to Win32 as a `PCWSTR`. +#[derive(Debug, PartialEq, Eq)] +struct WidePath(Vec); + +impl WidePath { + /// Terminates `units` with NUL, rejecting interior NULs that would silently + /// truncate the path seen by Win32. + fn new(mut units: Vec) -> std::io::Result { + if units.contains(&0) { + return Err(Error::new(ErrorKind::InvalidInput, "path contains an interior NUL character")); + } + units.push(0); + Ok(Self(units)) + } + + /// Returns `path` in the extended-length (`\\?\` or `\\?\UNC\`) form, so + /// the volume APIs keep working beyond `MAX_PATH`. + /// + /// The path is first made absolute with `GetFullPathNameW` (via + /// [`std::path::absolute`]) so separators, `.`/`..` and trailing dots or + /// spaces are normalized as a non-verbatim Win32 call would see them. + /// Verbatim and `\\.\` device paths are kept unchanged. + fn extended(path: &Path) -> std::io::Result { + let absolute = std::path::absolute(path)?; + let wide = absolute.as_os_str().encode_wide().collect::>(); + let (prefix, rest) = match absolute.components().next() { + Some(Component::Prefix(prefix)) => match prefix.kind() { + Prefix::Disk(_) => (r"\\?\", &wide[..]), + // `\\server\share\x` becomes `\\?\UNC\server\share\x`. + Prefix::UNC(..) => (r"\\?\UNC", &wide[1..]), + _ => return Self::new(wide), + }, + _ => return Self::new(wide), + }; + let mut extended = prefix.encode_utf16().collect::>(); + extended.extend_from_slice(rest); + Self::new(extended) + } + + /// Like [`WidePath::extended`], with the trailing backslash that + /// directory APIs such as `GetDiskFreeSpaceExW` require for UNC paths. + fn directory(path: &Path) -> std::io::Result { + let mut path = Self::extended(path)?; + path.0.pop(); + if !matches!(path.0.last(), Some(&BACKSLASH | &SLASH)) { + path.0.push(BACKSLASH); + } + path.0.push(0); + Ok(path) + } + + fn units(&self) -> &[u16] { + &self.0[..self.0.len() - 1] + } + + fn as_pcwstr(&self) -> PCWSTR { + PCWSTR::from_raw(self.0.as_ptr()) + } +} /// Returns total and free bytes available in a directory, e.g. `C:\`. -// SAFETY: Windows API calls receive null-terminated UTF-16 paths and valid +// SAFETY: Windows API calls receive NUL-terminated UTF-16 paths and valid // pointers to initialized stack output variables. #[allow(unsafe_code)] pub fn get_info(p: impl AsRef) -> std::io::Result { - let path_wide = to_wide_path(p.as_ref()); + let directory = WidePath::directory(p.as_ref())?; let mut free_bytes_available = 0u64; let mut total_number_of_bytes = 0u64; let mut total_number_of_free_bytes = 0u64; + // The directory itself, not its volume root, is queried so per-directory + // quotas (e.g. FSRM) are reflected in the reported capacity. // SAFETY: - // 1. `path_wide` is a valid null-terminated UTF-16 string. + // 1. `directory` is a NUL-terminated UTF-16 string owned for the whole call. // 2. Pointers to `u64` variables are valid and point to initialized stack memory. unsafe { GetDiskFreeSpaceExW( - windows::core::PCWSTR::from_raw(path_wide.as_ptr()), + directory.as_pcwstr(), Some(&mut free_bytes_available), Some(&mut total_number_of_bytes), Some(&mut total_number_of_free_bytes), ) - .map_err(|e| Error::from_raw_os_error(e.code().0))?; + .map_err(win32_error)?; } let total = total_number_of_bytes; @@ -52,23 +132,27 @@ pub fn get_info(p: impl AsRef) -> std::io::Result { ))); } + // `GetDiskFreeSpaceW` and `GetVolumeInformationW` take a volume root, not + // an arbitrary directory. + let volume = get_volume_path(&directory)?; + let mut sectors_per_cluster = 0u32; let mut bytes_per_sector = 0u32; let mut number_of_free_clusters = 0u32; let mut total_number_of_clusters = 0u32; // SAFETY: - // 1. `path_wide` is a valid null-terminated UTF-16 string. + // 1. `volume` is a NUL-terminated UTF-16 string owned for the whole call. // 2. Pointers to `u32` variables are valid and point to initialized stack memory. unsafe { GetDiskFreeSpaceW( - windows::core::PCWSTR::from_raw(path_wide.as_ptr()), + volume.as_pcwstr(), Some(&mut sectors_per_cluster), Some(&mut bytes_per_sector), Some(&mut number_of_free_clusters), Some(&mut total_number_of_clusters), ) - .map_err(|e| Error::from_raw_os_error(e.code().0))?; + .map_err(win32_error)?; } Ok(DiskInfo { @@ -77,63 +161,207 @@ pub fn get_info(p: impl AsRef) -> std::io::Result { used: total - free, files: total_number_of_clusters as u64, ffree: number_of_free_clusters as u64, - fstype: get_windows_fs_type(&path_wide).unwrap_or_default(), + fstype: get_volume_information(&volume) + .map(|info| info.file_system_name) + .unwrap_or_default(), + // A failed handle probe is not conclusive; the volume's drive type + // still covers mapped drives and UNC shares. + remote: is_remote_directory(p.as_ref()).unwrap_or(false) || is_remote_volume(&volume)?, ..Default::default() }) } -// SAFETY: Windows volume APIs receive null-terminated UTF-16 paths and fixed -// stack buffers sized for the documented MAX_PATH outputs used here. -#[allow(unsafe_code)] -fn get_windows_fs_type(p: &[u16]) -> std::io::Result { - let path = get_volume_name(p)?; - - let mut volume_serial_number = 0u32; - let mut maximum_component_length = 0u32; - let mut file_system_flags = 0u32; - let mut volume_name_buffer = [0u16; MAX_PATH as usize]; - let mut file_system_name_buffer = [0u16; MAX_PATH as usize]; - - // SAFETY: - // 1. `path` is a valid null-terminated UTF-16 string (volume root path). - // 2. Buffers are allocated with `MAX_PATH` size, which is sufficient for standard Windows paths. - // 3. Pointers to output variables are valid. - unsafe { - GetVolumeInformationW( - windows::core::PCWSTR::from_raw(path.as_ptr()), - Some(&mut volume_name_buffer), - Some(&mut volume_serial_number), - Some(&mut maximum_component_length), - Some(&mut file_system_flags), - Some(&mut file_system_name_buffer), - ) - .map_err(|e| Error::from_raw_os_error(e.code().0))?; - } - - Ok(utf16_to_string(&file_system_name_buffer)) +struct VolumeInformation { + serial_number: u32, + file_system_name: String, } -// SAFETY: `v` is a null-terminated UTF-16 path and `volume_name_buffer` is a -// writable MAX_PATH-sized stack buffer for the returned volume path. +// SAFETY: `GetVolumeInformationW` receives a NUL-terminated volume root and a +// slice whose length the bindings pass as the buffer size. #[allow(unsafe_code)] -fn get_volume_name(v: &[u16]) -> std::io::Result> { - let mut volume_name_buffer = [0u16; MAX_PATH as usize]; +fn get_volume_information(volume: &WidePath) -> std::io::Result { + let mut serial_number = 0u32; + let mut file_system_name = [0u16; VOLUME_INFORMATION_BUFFER_LEN]; // SAFETY: - // 1. `v` is a valid null-terminated UTF-16 string. - // 2. `volume_name_buffer` is allocated with `MAX_PATH` size. - // 3. `GetVolumePathNameW` writes to the buffer and respects the buffer size (implicitly MAX_PATH for this API context usually, though explicit length param isn't present, it expects a buffer large enough). - // Note: GetVolumePathNameW documentation says "The buffer should be large enough to hold the path". MAX_PATH is generally safe for volume roots. + // 1. `volume` is a NUL-terminated UTF-16 volume root with a trailing backslash, + // as returned by `GetVolumePathNameW`. + // 2. `file_system_name` holds the documented maximum of `MAX_PATH + 1` units and + // its length is passed alongside the pointer. + // 3. The serial number pointer targets an initialized stack variable. unsafe { - GetVolumePathNameW(windows::core::PCWSTR::from_raw(v.as_ptr()), &mut volume_name_buffer) - .map_err(|e| Error::from_raw_os_error(e.code().0))?; + GetVolumeInformationW( + volume.as_pcwstr(), + None, + Some(&mut serial_number), + None, + None, + Some(&mut file_system_name), + ) + .map_err(win32_error)?; } - let len = volume_name_buffer - .iter() - .position(|&x| x == 0) - .unwrap_or(volume_name_buffer.len()); - Ok(volume_name_buffer[..len].to_vec()) + Ok(VolumeInformation { + serial_number, + file_system_name: utf16_to_string(&file_system_name), + }) +} + +/// NT namespace prefix of paths served by the Multiple UNC Provider, which +/// fronts every network redirector (UNC paths and mapped drive letters). +const MUP_DEVICE_PREFIX: &str = r"\Device\Mup\"; + +/// Opens `path` for metadata queries only, following symlinks and junctions. +fn open_directory(path: &Path) -> std::io::Result { + std::fs::OpenOptions::new() + .access_mode(0) + .custom_flags(FILE_FLAG_BACKUP_SEMANTICS.0) + .open(path) +} + +/// Reports whether `path` is served by a network redirector, following +/// symlinks and junctions to their target. +/// +/// Either signal is sufficient: `FileRemoteProtocolInfo` only succeeds on +/// handles opened through a redirector, and the handle's final NT path lies +/// under `\Device\Mup\` for every UNC or mapped-drive target. +fn is_remote_directory(path: &Path) -> std::io::Result { + let directory = open_directory(path)?; + let handle = HANDLE(directory.as_raw_handle()); + if remote_protocol_info(handle).is_ok() { + return Ok(true); + } + let final_path = final_nt_path(handle)?; + Ok(final_path + .get(..MUP_DEVICE_PREFIX.len()) + .is_some_and(|prefix| prefix.eq_ignore_ascii_case(MUP_DEVICE_PREFIX))) +} + +/// Queries the remote protocol behind `handle`; local volumes fail with +/// `ERROR_INVALID_PARAMETER`. +// SAFETY: `GetFileInformationByHandleEx` receives a caller-owned open handle +// and a pointer/size pair describing a local `FILE_REMOTE_PROTOCOL_INFO`. +#[allow(unsafe_code)] +fn remote_protocol_info(handle: HANDLE) -> std::io::Result { + let mut info = FILE_REMOTE_PROTOCOL_INFO::default(); + // SAFETY: + // 1. The caller keeps `handle` open for the whole call. + // 2. `info` is a writable, initialized `FILE_REMOTE_PROTOCOL_INFO` and its + // exact size is passed as the buffer size. + unsafe { + GetFileInformationByHandleEx( + handle, + FileRemoteProtocolInfo, + (&raw mut info).cast(), + size_of::() as u32, + ) + .map_err(win32_error)?; + } + Ok(info) +} + +/// Returns the NT namespace path of `handle`, e.g. `\Device\HarddiskVolume3\x` +/// or `\Device\Mup\server\share\x`. +// SAFETY: `GetFinalPathNameByHandleW` receives a caller-owned open handle and +// a writable slice whose length the bindings pass as the buffer size. +#[allow(unsafe_code)] +fn final_nt_path(handle: HANDLE) -> std::io::Result { + let mut buffer = vec![0u16; MAX_PATH as usize + 1]; + loop { + // SAFETY: + // 1. The caller keeps `handle` open for the whole call. + // 2. `buffer` is writable and its length is passed as `cchFilePath`. + let len = unsafe { GetFinalPathNameByHandleW(handle, &mut buffer, VOLUME_NAME_NT) } as usize; + if len == 0 { + return Err(Error::last_os_error()); + } + // On success the length excludes the NUL; a length that does not fit + // is the required size including it. + if len < buffer.len() { + return Ok(String::from_utf16_lossy(&buffer[..len])); + } + buffer.resize(len, 0); + } +} + +/// Reports whether `volume`, a root returned by [`get_volume_path`], is a +/// network drive such as a mapped drive letter or a UNC share root. +// SAFETY: `GetDriveTypeW` receives a NUL-terminated volume root. +#[allow(unsafe_code)] +fn is_remote_volume(volume: &WidePath) -> std::io::Result { + // `GetDriveTypeW` documents the conventional `C:\` and `\\server\share\` + // root forms, so the extended-length prefix is removed first. + let root = WidePath::new(simplify_extended(volume.units()).into_owned())?; + // SAFETY: `root` is a NUL-terminated UTF-16 string owned for the whole call. + let drive_type = unsafe { GetDriveTypeW(root.as_pcwstr()) }; + Ok(drive_type == DRIVE_REMOTE) +} + +/// Returns the NUL-terminated root of the volume that contains `path`, e.g. +/// `\\?\C:\` or a mounted folder such as `\\?\C:\mnt\disk1\`. +// SAFETY: `GetVolumePathNameW` receives a NUL-terminated path and a writable +// slice whose length the bindings pass as the buffer size. +#[allow(unsafe_code)] +fn get_volume_path(path: &WidePath) -> std::io::Result { + // The volume path is a prefix of the full input path plus a trailing + // backslash, so the input length (including its NUL) plus one always fits. + let mut buffer = vec![0u16; (path.0.len() + 1).max(MAX_PATH as usize + 1)]; + + // SAFETY: + // 1. `path` is a NUL-terminated UTF-16 string owned for the whole call. + // 2. `buffer` is writable and its length is passed as `cchBufferLength`. + unsafe { + GetVolumePathNameW(path.as_pcwstr(), &mut buffer).map_err(win32_error)?; + } + + let len = buffer.iter().position(|&x| x == 0).unwrap_or(buffer.len()); + buffer.truncate(len); + WidePath::new(buffer) +} + +/// Returns the identity of the volume containing `path` in its conventional +/// form (`C:\`, `\\server\share\`), independent of whether the caller used +/// an extended-length path. +fn volume_id(path: &str) -> std::io::Result { + let volume = get_volume_path(&WidePath::extended(Path::new(path))?)?; + Ok(String::from_utf16_lossy(&simplify_extended(volume.units()))) +} + +/// Strips the `\\?\` prefix from a drive path and maps `\\?\UNC\` back to +/// `\\`; other verbatim paths (e.g. `\\?\Volume{GUID}\`) are returned as is. +fn simplify_extended(units: &[u16]) -> std::borrow::Cow<'_, [u16]> { + const VERBATIM: [u16; 4] = [BACKSLASH, BACKSLASH, b'?' as u16, BACKSLASH]; + let Some(rest) = units.strip_prefix(&VERBATIM[..]) else { + return units.into(); + }; + if let [drive, colon, ..] = rest + && *colon == b':' as u16 + && char::from_u32(u32::from(*drive)).is_some_and(|c| c.is_ascii_alphabetic()) + { + return rest.into(); + } + if let [u, n, c, BACKSLASH, share @ ..] = rest + && String::from_utf16_lossy(&[*u, *n, *c]).eq_ignore_ascii_case("UNC") + { + let mut simplified = vec![BACKSLASH, BACKSLASH]; + simplified.extend_from_slice(share); + return simplified.into(); + } + units.into() +} + +/// Converts a failed `windows` call into an `io::Error` carrying the Win32 +/// error code, so `raw_os_error()` and `kind()` match `std::fs` errors. +fn win32_error(err: windows::core::Error) -> Error { + const FACILITY_WIN32_MASK: u32 = 0xFFFF_0000; + const FACILITY_WIN32_FAILURE: u32 = 0x8007_0000; + + let hresult = err.code().0 as u32; + if hresult & FACILITY_WIN32_MASK == FACILITY_WIN32_FAILURE { + Error::from_raw_os_error((hresult & !FACILITY_WIN32_MASK) as i32) + } else { + Error::other(err) + } } fn utf16_to_string(v: &[u16]) -> String { @@ -141,21 +369,8 @@ fn utf16_to_string(v: &[u16]) -> String { String::from_utf16_lossy(&v[..len]) } -fn to_wide_path(path: &Path) -> Vec { - path.as_os_str().encode_wide().chain(std::iter::once(0)).collect() -} - -// Helper trait to access encode_wide which is only available on Windows -use std::os::windows::ffi::OsStrExt; - pub fn same_disk(disk1: &str, disk2: &str) -> std::io::Result { - let path1_wide = to_wide_path(Path::new(disk1)); - let path2_wide = to_wide_path(Path::new(disk2)); - - let volume1 = get_volume_name(&path1_wide)?; - let volume2 = get_volume_name(&path2_wide)?; - - Ok(volume1 == volume2) + Ok(volume_id(disk1)? == volume_id(disk2)?) } /// Automatic replacement formatting is disabled until Windows has an @@ -165,46 +380,16 @@ pub fn is_mount_point(_path: &std::path::Path) -> std::io::Result { } pub fn get_physical_device_ids(disk: &str) -> std::io::Result> { - let path_wide = to_wide_path(Path::new(disk)); - let volume = get_volume_name(&path_wide)?; - - Ok(vec![String::from_utf16_lossy(&volume)]) + Ok(vec![volume_id(disk)?]) } /// Returns the volume serial number for the filesystem backing `path`. /// /// This is the Windows equivalent of Linux's `st_dev` major:minor pair and is /// useful for diagnostic purposes when validating physical disk independence. -// SAFETY: `path_wide` is a valid null-terminated UTF-16 string and all output -// SAFETY: pointers target initialized stack variables with documented MAX_PATH sizing. -#[allow(unsafe_code)] pub fn get_volume_serial_number(path: &str) -> std::io::Result { - let path_wide = to_wide_path(Path::new(path)); - let volume = get_volume_name(&path_wide)?; - - let mut volume_serial_number = 0u32; - let mut maximum_component_length = 0u32; - let mut file_system_flags = 0u32; - let mut volume_name_buffer = [0u16; MAX_PATH as usize]; - let mut file_system_name_buffer = [0u16; MAX_PATH as usize]; - - // SAFETY: - // 1. `volume` is a valid null-terminated UTF-16 string (volume root path). - // 2. Buffers are allocated with `MAX_PATH` size. - // 3. Pointers to output variables are valid. - unsafe { - GetVolumeInformationW( - windows::core::PCWSTR::from_raw(volume.as_ptr()), - Some(&mut volume_name_buffer), - Some(&mut volume_serial_number), - Some(&mut maximum_component_length), - Some(&mut file_system_flags), - Some(&mut file_system_name_buffer), - ) - .map_err(|e| Error::from_raw_os_error(e.code().0))?; - } - - Ok(volume_serial_number) + let volume = get_volume_path(&WidePath::extended(Path::new(path))?)?; + Ok(get_volume_information(&volume)?.serial_number) } pub fn check_cross_device_mounts(_paths: &[String]) -> std::io::Result<()> { @@ -214,3 +399,231 @@ pub fn check_cross_device_mounts(_paths: &[String]) -> std::io::Result<()> { pub fn get_drive_stats(_major: u32, _minor: u32) -> std::io::Result { Ok(IOStats::default()) } + +#[cfg(test)] +mod tests { + use super::*; + use std::path::PathBuf; + + fn wide(s: &str) -> Vec { + s.encode_utf16().collect() + } + + /// Describes both handle signals for `path`, for assertion messages. + fn handle_probe(path: &Path) -> String { + let directory = match open_directory(path) { + Ok(directory) => directory, + Err(err) => return format!("open failed: {err:?}"), + }; + let handle = HANDLE(directory.as_raw_handle()); + let protocol = remote_protocol_info(handle).map(|info| (info.Protocol, info.ProtocolMajorVersion, info.Flags)); + format!("remote protocol {protocol:?}, final NT path {:?}", final_nt_path(handle)) + } + + /// Creates a directory whose absolute path is well beyond `MAX_PATH`. + fn long_directory(root: &Path) -> PathBuf { + let mut dir = root.to_path_buf(); + for i in 0..6 { + dir.push(format!("{i}-{}", "a".repeat(60))); + } + std::fs::create_dir_all(&dir).expect("std::fs creates long directories itself"); + assert!(dir.as_os_str().len() > MAX_PATH as usize); + dir + } + + #[test] + fn windows_wide_path_is_nul_terminated_extended_length() { + let disk = WidePath::extended(Path::new(r"C:/data/./bucket/../bucket")).unwrap(); + assert_eq!(disk.units(), wide(r"\\?\C:\data\bucket")); + assert_eq!(disk.0.last(), Some(&0)); + + let unc = WidePath::extended(Path::new(r"\\server\share\data")).unwrap(); + assert_eq!(unc.units(), wide(r"\\?\UNC\server\share\data")); + + for unchanged in [r"\\?\C:\data", r"\\?\UNC\server\share\data", r"\\.\C:\data"] { + assert_eq!(WidePath::extended(Path::new(unchanged)).unwrap().units(), wide(unchanged)); + } + + let relative = WidePath::extended(Path::new("relative")).unwrap(); + assert!(relative.units().starts_with(&wide(r"\\?\")), "relative paths are made absolute first"); + } + + #[test] + fn windows_directory_path_has_single_trailing_backslash() { + // A UNC share root without the trailing backslash is rejected by + // `GetDiskFreeSpaceExW`. + let share = WidePath::directory(Path::new(r"\\server\share")).unwrap(); + assert_eq!(share.units(), wide(r"\\?\UNC\server\share\")); + let root = WidePath::directory(Path::new(r"C:\")).unwrap(); + assert_eq!(root.units(), wide(r"\\?\C:\")); + assert_eq!(root.0.last(), Some(&0)); + } + + #[test] + fn windows_wide_path_rejects_interior_nul() { + let err = WidePath::new(wide("C:\\a\0b")).unwrap_err(); + assert_eq!(err.kind(), ErrorKind::InvalidInput); + } + + #[test] + fn windows_simplify_extended_restores_conventional_volume_ids() { + for (input, expected) in [ + (r"\\?\C:\", r"C:\"), + (r"\\?\UNC\server\share\", r"\\server\share\"), + (r"\\?\unc\server\share\", r"\\server\share\"), + ( + r"\\?\Volume{0b6e1f0a-0000-0000-0000-100000000000}\", + r"\\?\Volume{0b6e1f0a-0000-0000-0000-100000000000}\", + ), + (r"C:\", r"C:\"), + ] { + assert_eq!(&*simplify_extended(&wide(input)), wide(expected).as_slice(), "{input}"); + } + } + + #[test] + fn windows_win32_error_keeps_os_error_code() { + // HRESULT_FROM_WIN32(ERROR_PATH_NOT_FOUND) + let err = win32_error(windows::core::Error::from_hresult(windows::core::HRESULT(0x8007_0003_u32 as i32))); + assert_eq!(err.raw_os_error(), Some(3)); + assert_eq!(err.kind(), ErrorKind::NotFound); + + let missing = get_info(r"Z:\rustfs-missing\path").unwrap_err(); + assert!(missing.raw_os_error().is_some_and(|code| code < 0x1_0000), "{missing:?}"); + } + + #[test] + fn windows_get_info_reports_fstype() { + let temp_dir = tempfile::tempdir().unwrap(); + let info = get_info(temp_dir.path()).unwrap(); + assert!(info.total > 0); + assert!(!info.fstype.is_empty(), "GetVolumeInformationW must name the filesystem"); + assert!(!info.remote, "the temp dir is on a local volume"); + assert!( + !is_remote_directory(temp_dir.path()).unwrap(), + "local handle: {}", + handle_probe(temp_dir.path()) + ); + + let ids = get_physical_device_ids(temp_dir.path().to_str().unwrap()).unwrap(); + assert_eq!(ids.len(), 1); + assert!(!ids[0].starts_with(r"\\?\"), "volume ids keep their conventional form: {ids:?}"); + assert!(ids[0].ends_with('\\'), "{ids:?}"); + } + + #[test] + fn windows_volume_apis_accept_paths_beyond_max_path() { + let temp_dir = tempfile::tempdir().unwrap(); + let short = temp_dir.path(); + let long = long_directory(short); + let (short_str, long_str) = (short.to_str().unwrap(), long.to_str().unwrap()); + + let short_info = get_info(short).unwrap(); + let long_info = get_info(&long).unwrap(); + assert!(long_info.total > 0); + assert!(!long_info.fstype.is_empty()); + assert_eq!(long_info.fstype, short_info.fstype); + assert_eq!(long_info.files, short_info.files); + + assert!(same_disk(short_str, long_str).unwrap()); + assert_eq!(get_physical_device_ids(long_str).unwrap(), get_physical_device_ids(short_str).unwrap()); + assert_eq!(get_volume_serial_number(long_str).unwrap(), get_volume_serial_number(short_str).unwrap()); + + let verbatim = format!(r"\\?\{long_str}"); + assert_eq!(get_physical_device_ids(&verbatim).unwrap(), get_physical_device_ids(short_str).unwrap()); + } + + #[test] + fn windows_get_info_accepts_unc_share_root_without_trailing_backslash() { + let temp_dir = tempfile::tempdir().unwrap(); + let absolute = std::path::absolute(temp_dir.path()).unwrap(); + let local = absolute.to_str().unwrap(); + let Some((drive, rest)) = local.split_once(":\\") else { + eprintln!("skipping: temp dir {local} is not on a drive letter"); + return; + }; + let share = format!(r"\\localhost\{drive}$"); + if !Path::new(&share).is_dir() { + eprintln!("skipping: administrative share {share} is not reachable"); + return; + } + + let share_info = get_info(&share).unwrap(); + assert!(share_info.total > 0); + assert!(!share_info.fstype.is_empty()); + // The share reports the server's filesystem, so only the drive type + // reveals that it is reached over SMB. + assert!(share_info.remote, "{share} is a network share: {share_info:?}"); + // The handle probe alone must recognize the share, since it is the + // only signal for links whose volume root is local. + assert!( + is_remote_directory(Path::new(&share)).unwrap(), + "share handle: {}", + handle_probe(Path::new(&share)) + ); + + let long = long_directory(&PathBuf::from(format!(r"{share}\{rest}"))); + let long_info = get_info(&long).unwrap(); + assert!(!long_info.fstype.is_empty()); + assert!(long_info.remote, "{long:?} is below a network share"); + assert!(same_disk(&share, long.to_str().unwrap()).unwrap()); + } + + #[test] + fn windows_get_info_follows_directory_symlink_to_network_share() { + let temp_dir = tempfile::tempdir().unwrap(); + let absolute = std::path::absolute(temp_dir.path()).unwrap(); + let Some((drive, rest)) = absolute.to_str().unwrap().split_once(":\\") else { + eprintln!("skipping: temp dir {absolute:?} is not on a drive letter"); + return; + }; + let target = PathBuf::from(format!(r"\\localhost\{drive}$\{rest}")); + if !target.is_dir() { + eprintln!("skipping: administrative share path {target:?} is not reachable"); + return; + } + let link = temp_dir.path().join("share-link"); + if let Err(err) = std::os::windows::fs::symlink_dir(&target, &link) { + eprintln!("skipping: cannot create directory symlinks: {err}"); + return; + } + + // The link itself lives on a local volume, so its volume root alone + // need not reveal that the target is a share. + let info = get_info(&link).unwrap(); + assert!( + info.remote, + "{link:?} -> {target:?} resolves to a network share: {info:?}; link handle: {}", + handle_probe(&link) + ); + } + + /// Set by the Windows filesystem workflow after mapping a drive letter to + /// the administrative share of the temp dir's drive with `net use`. + const MAPPED_DRIVE_ENV: &str = "RUSTFS_TEST_WINDOWS_MAPPED_DRIVE"; + + #[test] + fn windows_get_info_reports_mapped_network_drive_as_remote() { + let Ok(drive) = std::env::var(MAPPED_DRIVE_ENV) else { + eprintln!("skipping: {MAPPED_DRIVE_ENV} is not set"); + return; + }; + let temp_dir = tempfile::tempdir().unwrap(); + let absolute = std::path::absolute(temp_dir.path()).unwrap(); + let Some((_, rest)) = absolute.to_str().unwrap().split_once(":\\") else { + eprintln!("skipping: temp dir {absolute:?} is not on a drive letter"); + return; + }; + let root = format!(r"{}\", drive.trim_end_matches('\\')); + let mapped = PathBuf::from(format!("{root}{rest}")); + if !mapped.is_dir() { + eprintln!("skipping: {mapped:?} is not reachable through the mapped drive"); + return; + } + + for path in [Path::new(&root), mapped.as_path()] { + let info = get_info(path).unwrap(); + assert!(info.remote, "{path:?} is on a mapped network drive: {info:?}"); + } + } +} diff --git a/crates/utils/src/path.rs b/crates/utils/src/path.rs index 8b53e764b..4f52ffd12 100644 --- a/crates/utils/src/path.rs +++ b/crates/utils/src/path.rs @@ -200,6 +200,29 @@ pub fn path_join_buf(elements: &[&str]) -> String { dst } +/// Object name in the segment form the local disk resolves it to. +/// +/// Windows disks treat `\` as a path separator and `path_join_buf` drops the +/// empty segments that leaves behind, so `a\b`, `a\\b`, `\a\b` and `a\b\` +/// all address the xl.meta of `a/b`. In-memory identities that must agree +/// with that file (namespace locks, metadata caches) fold through this. +/// Elsewhere `\` is an ordinary key byte and the name is returned verbatim. +pub fn object_name_disk_identity(object: &str) -> Cow<'_, str> { + fold_object_name_path_separators(object, cfg!(target_os = "windows")) +} + +/// Splits on `/` and, when `backslash_is_separator`, `\`, dropping empty +/// segments. `.` and `..` segments stay as they are: that can only leave two +/// spellings of one file apart, never merge two different files. +pub fn fold_object_name_path_separators(object: &str, backslash_is_separator: bool) -> Cow<'_, str> { + // Names without `\` already passed the `//` and leading-`/` checks. + if !backslash_is_separator || !object.contains('\\') { + return Cow::Borrowed(object); + } + let segments: Vec<&str> = object.split(['/', '\\']).filter(|segment| !segment.is_empty()).collect(); + Cow::Owned(segments.join(SLASH_SEPARATOR)) +} + /// path_needs_clean returns whether path cleaning may change the path. /// Will detect all cases that will be cleaned, /// but may produce false positives on non-trivial paths. @@ -624,6 +647,27 @@ mod tests { assert_eq!(encode_dir_object("prefix/directory/"), encoded); } + #[test] + fn fold_object_name_path_separators_folds_backslash_only_as_separator() { + assert_eq!(fold_object_name_path_separators("a\\b\\c", true), "a/b/c"); + assert_eq!(fold_object_name_path_separators("a/b\\c", true), "a/b/c"); + // Empty segments collapse on disk, so these address `a/b` as well. + assert_eq!(fold_object_name_path_separators("a\\\\b", true), "a/b"); + assert_eq!(fold_object_name_path_separators("a/\\b", true), "a/b"); + assert_eq!(fold_object_name_path_separators("\\a\\b", true), "a/b"); + assert_eq!(fold_object_name_path_separators("a\\b\\", true), "a/b"); + assert_eq!(fold_object_name_path_separators("a\\b__XLDIR__", true), "a/b__XLDIR__"); + assert!(matches!(fold_object_name_path_separators("a/b", true), Cow::Borrowed("a/b"))); + assert!(matches!(fold_object_name_path_separators("a\\b", false), Cow::Borrowed("a\\b"))); + } + + #[test] + fn object_name_disk_identity_follows_platform_separator() { + let expected = if cfg!(target_os = "windows") { "a/b" } else { "a\\b" }; + assert_eq!(object_name_disk_identity("a\\b"), expected); + assert!(matches!(object_name_disk_identity("a/b"), Cow::Borrowed("a/b"))); + } + #[test] fn test_trim_etag() { // Test with quoted ETag diff --git a/deny.toml b/deny.toml index b810318e7..1a84bdbbb 100644 --- a/deny.toml +++ b/deny.toml @@ -19,9 +19,6 @@ no-default-features = false version = 2 yanked = "deny" ignore = [ - # `paste 1.0.15` — unmaintained. No direct dependency. - # owner: rustfs-maintainers review: 2026-07 - { id = "RUSTSEC-2024-0436", reason = "paste unmaintained; transitive only; tracked for upgrade" }, # `rsa` Marvin timing sidechannel (RUSTSEC-2023-0071). Pulled in via # `openidconnect` (transitive) and historically used directly. No upstream # fix is available yet. Tracked separately for follow-up; remove this diff --git a/docs/README.md b/docs/README.md index b64ce1783..c5b301ab5 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,6 +8,12 @@ collection: ## Operations +- [Table maintenance periodic queueing](operations/s3-tables-maintenance-scheduler.md) — opt-in scheduling, limits, worker handoff, and recovery. + +For PUT cleanup admission, blocking-pool isolation, and optional CPU affinity, +see [PUT cleanup isolation](operations/put-cleanup-isolation.md). + +- [External certificate issuance and TLS rotation](operations/tls-certificate-rotation.md) — publication, native reload, trust separation and failure handling. - [Architecture guard troubleshooting](operations/architecture-guard-troubleshooting.md) — diagnose repository guard failures. - [Logging governance](operations/logging-governance.md) — broad event audits, migrations, and guardrail changes. @@ -25,6 +31,9 @@ For legacy protection assessment and bounded protected copies, see [Object integ For encrypted listing compatibility and older stored fingerprints, see [Encrypted object ETags](operations/encrypted-etags.md). +For durable hard quota accounting, node-outage behavior, and fence protocol +upgrades, see [Durable hard quotas](operations/durable-quota-reservation-v2.md). + Operational runbooks live under [`operations/`](operations/). Replication operators should start with: @@ -46,6 +55,9 @@ For concurrent bucket creation and deletion, see For persisted administrator bucket tasks and bucket recreation, see [Bucket heal recovery](operations/bucket-heal-recovery.md). +For interrupted object deletions and returning drives, see +[Delete transaction recovery](operations/delete-transaction-recovery.md). + For disk replacement across VM restarts and schema 5/6 maintenance migration, see [Replacement generation recovery](operations/replacement-generation-recovery.md). @@ -59,6 +71,9 @@ procedure is required. For storage dashboards, see [Storage metrics and observer selection](operations/storage-metrics.md): drive ownership, snapshot freshness, counter queries, and rolling upgrades. +For per-bucket capacity and object counts, see [Bucket usage metrics](operations/bucket-usage-metrics.md): +OTLP export, cluster deduplication, accounting semantics, and scanner freshness. + For optional shard commitments, see [Independent shard integrity rollout](operations/shard-integrity-rollout.md): activation, legacy repair results, multipart mode changes, and rollback limits. diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 6af10a389..fe2e1b29e 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -47,6 +47,7 @@ Required headings and strings in these files are asserted by `scripts/check_arch | [background-services-inventory.md](background-services-inventory.md) | you need one audited background service's desired source, current-status inputs, status surface, and declared side effects | | [scanner-usage-publication.md](scanner-usage-publication.md) | changing scanner data-usage cache publication, quota-visible usage snapshots, scanner cycle recovery, or the persisted scanner usage artifacts | | [scanner-usage-authority-decision.md](scanner-usage-authority-decision.md) | deciding whether quota admission depends on scanner data usage, removing scanner publication layers, or designing a scanner storage boundary | +| [scanner-per-bucket-convergence.md](scanner-per-bucket-convergence.md) | changing what makes a scanner cycle `Complete`, `Superseded`, or converged, adding or re-classifying a dirty-usage producer, deciding whether a background writeback may invalidate a cycle, changing scanner worker concurrency or throttling, or activating scoped dirty acknowledgements | | [config-model-boundary-adr.md](config-model-boundary-adr.md) | touching the server-config model (`Config`, `KV`, `KVS`) or its persistence, or asking which crate owns which part of server configuration | | [admin-route-action-snapshot.md](admin-route-action-snapshot.md) | adding, moving, or re-authorizing an admin route and needing to know where the route → handler → `AdminAction` contract is enforced | | [plugin-delegation.md](plugin-delegation.md) | changing plugin delegation registration, revocation, broker identity, or the scope of attempt credentials | diff --git a/docs/architecture/compat-cleanup-register.md b/docs/architecture/compat-cleanup-register.md index 423ca0bd5..b97bf86e4 100644 --- a/docs/architecture/compat-cleanup-register.md +++ b/docs/architecture/compat-cleanup-register.md @@ -11,6 +11,8 @@ ## Open Items +- `8565` historical rebalance replacement recovery: older schema-seven intents can exhaust their retry budget before format repair when retained rebalance metadata describes removed pools. After validating blank targets, terminal historical topology, and ownership, grant one durable handoff and one rebuild attempt. Remove after supported direct upgrades no longer admit snapshots lacking replacement_preformat_handoff_retry_done and all retained affected intents have been resolved. + - `backlog-2253-metadata-observation-adapter` metadata observations: preserve aligned metadata/error slices for existing layout, shard and exact-version PUT, MPU and delete consumers while read/heal scheduling uses typed disk observations. Pending slots acquire empty placeholders only in this adapter. Remove after those remaining consumers use typed observations; delete the adapter and old facade in a separate cleanup. - `connect-894` pending Connect heartbeats: replay the exact preceding producer-capability list, with or without jobs, when upgrading to the memory-service capability. Preserve request ID, sequence, and persisted body rather than inserting the new capability into a retry. Remove after upgrades from the pre-service-memory capability set are unsupported. diff --git a/docs/architecture/crate-boundaries.md b/docs/architecture/crate-boundaries.md index 09a308d8f..9d940e37b 100644 --- a/docs/architecture/crate-boundaries.md +++ b/docs/architecture/crate-boundaries.md @@ -81,6 +81,16 @@ Every temporary compatibility path carries a `RUSTFS_COMPAT_TODO()` source m The server-config model (`Config`, `KV`, `KVS`) and the global server-config snapshot accessors are owned by `rustfs_config::server_config`; ECStore keeps persistence, storage-class state, and startup wiring, and its public facades must not re-export those symbols. See [config-model-boundary-adr.md](config-model-boundary-adr.md). +## OIDC Ownership + +`crates/iam/src/oidc/` owns provider configuration, discovery, HTTP transport, login state, and the OIDC runtime. `crates/iam/src/federation/` owns verified identity transactions and authorization mapping; its production code does not import concrete OIDC runtime or provider configuration types. Production `FederatedAuthorization` values are constructed by `CoreFederatedAuthorizationMapper` in `crates/iam/src/federation/mapper.rs`. + +`FederatedClaims` is crate-internal. `FederatedAuthorization` has private fields owned by the mapper module and exposes read-only accessors to session binding consumers, including `mapped_iam_groups()` for the IAM groups selected by authorization mapping. Test fixtures are exposed only through the `test-util` feature or unit-test configuration. + +`rustfs/src/startup_auth.rs` constructs `StandardOidcAdapter` and publishes the federation service with its OIDC query as one runtime pair. `AppContext` stores and publishes that pair through `FederatedIdentityRuntimeSnapshot`; consumers receive interfaces rather than constructing a second OIDC runtime. `scripts/check_oidc_architecture_boundaries.sh` enforces these boundaries through the existing architecture guard, including the Keycloak workflow path for `crates/iam/src/oidc/`. + +`crates/iam/src/oidc/config.rs` owns OIDC configuration input, validation, persisted KVS conversion, and `OidcSiteReplicationSnapshot`. `rustfs/src/admin/service/oidc_config.rs` owns stored configuration reads, compare-and-save updates, restart comparison, and provider discovery validation; the OIDC Admin handler translates HTTP requests and errors. `OidcConfigQuery::site_replication_snapshot()` returns only the provider ID, claim name, role policy, client ID, and hashed client secret needed by site replication. `rustfs/src/admin/handlers/site_replication.rs` maps this narrow view to its existing response format without reading raw OIDC credentials. The Keycloak workflow tracks the Admin configuration service path along with IAM OIDC code. + ## Required Architecture Documents The guard requires the documents and section headings listed in its `require_source_contains` entries (`scripts/check_architecture_migration_rules.sh`); the directory index is [README.md](README.md). diff --git a/docs/architecture/runtime-lifecycle.md b/docs/architecture/runtime-lifecycle.md index 9204a21ad..62ef88bcb 100644 --- a/docs/architecture/runtime-lifecycle.md +++ b/docs/architecture/runtime-lifecycle.md @@ -55,3 +55,9 @@ Embedded startup reuses the same phase owners as the binary: server and storage ## AppContext Foundation AppContext is a context-first facade, not a full replacement for every process global. Resolver files are split and covered by compatibility tests before boot extraction or consumer migration, so the old global fallback path keeps working during transition. New migration work keeps fallback reads inside owner-local runtime-source boundaries and follows the target inventory in [global-state-inventory.md](global-state-inventory.md). + +## OIDC Runtime Publication + +`startup_auth.rs` builds the OIDC runtime and publishes its `FederatedIdentityService` and `OidcConfigQuery` together. `FederatedIdentityRuntimeSnapshot` in `app/context/interfaces.rs` and the owning handle in `app/context/handles.rs` retain this pair. Authentication and configuration consumers resolve their respective interfaces from the same published runtime; the OIDC runtime is created only at IAM startup construction and this startup phase. + +The query exposes `site_replication_snapshot()` for active, narrow replication settings. Admin configuration changes use `admin/service/oidc_config.rs` to read and save persisted configuration and report whether a restart is required; they do not replace the active runtime pair during the request. diff --git a/docs/architecture/s3-compatibility-matrix.md b/docs/architecture/s3-compatibility-matrix.md index 83ffc522d..a71d1cd5e 100644 --- a/docs/architecture/s3-compatibility-matrix.md +++ b/docs/architecture/s3-compatibility-matrix.md @@ -32,6 +32,7 @@ Counts ignore blank lines and comments; compute them from the files. The lifecyc | Public access block put/get/delete | Supported | `implemented_tests.txt` | | Presigned GET and PUT URLs | Supported | `implemented_tests.txt` | | Range and conditional reads | Supported | `implemented_tests.txt` | +| Conditional bulk deletion of current objects | Supported with the limits below | `crates/e2e_test/src/conditional_delete_objects_test.rs` | | User metadata | Supported | `implemented_tests.txt` | | SSE-C and selected SSE-KMS edge cases | Supported | `implemented_tests.txt` | | Selected versioning, object-lock, checksum, CORS, raw request, and conditional write behavior | Supported | `implemented_tests.txt` | @@ -44,6 +45,16 @@ Counts ignore blank lines and comments; compute them from the files. The lifecyc The MinIO-style `metadata=true` listing extension returns user metadata names without the HTTP `x-amz-meta-` prefix. It is not the standard S3 `ListObjectsV2` response. Clients that expect canonical HTTP header names in `UserMetadata` must normalize the names at that boundary; ordinary HEAD/GET metadata is unaffected. See `rustfs/src/app/bucket_usecase.rs` and its serialization tests. +## Conditional Bulk Deletion + +`DeleteObjects` accepts an optional `ETag` for each object in the XML body. A specific ETag matches the current live object; `*` requires that a current live object exists. Specific ETags require both `s3:DeleteObject` and `s3:GetObject`, while `*` requires only delete permission. Entries without an ETag retain the unconditional delete behavior. + +An ETag mismatch returns a per-object `PreconditionFailed`; a missing object or a current delete marker returns `NoSuchKey`. Explicit empty or weak ETags return `InvalidArgument`. Supplying both ETag and any explicit `VersionId`, including `null`, returns `InvalidRequest`; this combination is outside the supported conditional-delete scope. Version-enabled and version-suspended buckets are supported when the request omits `VersionId`. + +The response can contain successful and failed entries with HTTP 200. `Quiet=true` hides successful entries and preserves errors. Conditions on repeated keys are evaluated independently against one locked snapshot; another successful entry for the same key can still delete that object. Physical deletion, delete-marker creation and successful side effects are counted once per distinct deletion action. + +Conditional deletion requires namespace locking and every participating storage peer to support guarded metadata mutations. Unsupported peers return `NotImplemented` before mutation, and requests never fall back to unconditional deletion. Complete the upgrade of all S3 ingress and storage nodes before clients use this capability. Conditional entries return `SlowDown` during pool rebalance. A generation change during commit returns the per-object concurrent-operation error `OperationAborted`. Bucket-policy enforcement through `s3:if-match` is outside this support statement. + ## Replication Support Boundary Site replication and bucket replication are not the same compatibility claim. diff --git a/docs/architecture/s3-tables-support-matrix.md b/docs/architecture/s3-tables-support-matrix.md index 620f7dae4..05522badb 100644 --- a/docs/architecture/s3-tables-support-matrix.md +++ b/docs/architecture/s3-tables-support-matrix.md @@ -38,7 +38,7 @@ RustFS S3 Tables is an Iceberg REST Catalog and table-bucket implementation on t | Client or engine | Status | Claim | |---|---|---| -| PyIceberg | Automated | Namespace and table create, append, reload, scan, metadata-location, refs, views, maintenance, diagnostics, optional vended credentials with an exact-prefix data-plane scope check. | +| PyIceberg | Automated | Namespace and table create, append, exact-row reload/scan, metadata-location, refs, views, namespace properties, rename identity and credential negotiation. The smoke discovers the server backing and checks maintenance configuration, scheduler/worker dry-run, diagnostics and export on both backings. Direct metadata maintenance is checked as unsupported on durable-strong. Optional vended credentials include an exact-prefix data-plane scope check. | | Spark Iceberg REST catalog | Manual/live harness | Pinned package inputs, catalog properties, atomic CTAS SQL through staged create, expected `row_count=2`, and a CI opt-in gate for create/refresh/count/cleanup. Live execution and commit-conflict probing remain manual unless enabled in the runner. | | Trino Iceberg REST catalog | Manual/live harness | Catalog properties and a read-only `SELECT COUNT(*)` against a PyIceberg- or Spark-created table. Write compatibility not claimed. | | DuckDB Iceberg 1.5.5 | Automated | `duckdb_smoke.py` covers metadata-location read, single-table create/insert/update/delete/merge, schema evolution, snapshots, concurrent writers, drop, PyIceberg cross-read, and both signing profiles. Purge-on-drop and format v3 are verified fail-closed. Two-table mode runs without claiming cross-table atomicity. AWS `ENDPOINT_TYPE S3_TABLES`, staged-create client integration, and vended-credential integration are not claimed. | @@ -87,6 +87,42 @@ RustFS S3 Tables is an Iceberg REST Catalog and table-bucket implementation on t ## Maintenance Matrix +Object-backed and durable-strong catalogs support the configuration, scheduler, +worker, job inspection, heartbeat, and quarantine endpoints. Durable-strong +maintenance resolves table identity from the authoritative catalog snapshot; +recoverable job and configuration records remain separate objects. The worker +performs metadata and unreferenced-object cleanup without changing the catalog +pointer. The direct `/maintenance/metadata` endpoint, including snapshot +expiration and compaction plan/commit, remains object-backed only. A configured +`background-enabled` table still requires explicit worker ticks. The standalone +server has opt-in, allowlisted periodic queue admission; see the +[scheduler runbook](../operations/s3-tables-maintenance-scheduler.md). + +Renamed tables retain their original metadata paths. Durable-strong scheduling +and cleanup fail closed with a conflict if another table shares a scanned +metadata or warehouse directory, including a recreated source name after rename. +Such tables require disjoint cleanup roots before maintenance can run; enabling +deletion does not override this ownership check. + +Durable-strong rename preserves maintenance configuration, job identifiers, +current/latest aliases, history, audit events, and quarantine state. A persisted +rename intent protects sidecar migration across interruption: the source remains +authoritative until the catalog CAS succeeds, and the destination records are +ready before publication. Retry the same rename to finish pending recovery. +Backup, restore, register, drop, and subsequent rename operations recover a +pending migration before proceeding. Complete pending rename recovery before +rolling back to a version that does not understand maintenance migration intents. + +Durable-strong scheduler reports read an observational history set, not an atomic +snapshot of all maintenance jobs. They fix at most 4096 history-object names +under the publication read fence, scan those objects outside the fence, and +refresh the selected 10-job audit window, effective configuration, and current +job under a final read fence. Exceeding the history budget fails closed with an +unavailable response; retrying does not remove that size limit. A listed object +that disappears during the scan also returns unavailable; retry after concurrent +rename or maintenance activity has settled. No history object is silently omitted +when a rename removes its old path. + | Capability | Status | Claim | |---|---|---| | Metadata retention dry-run | Supported | Reports retained metadata and deletion candidates without moving the pointer. | @@ -103,7 +139,7 @@ RustFS S3 Tables is an Iceberg REST Catalog and table-bucket implementation on t | Compaction planning | Preview / controlled | Partition-local and sort-order-local binpack candidates for Parquet; never mixes partitions or sort orders in one group. | | Delete-file or row-level compaction planning | Preview / controlled | Position or equality delete files force machine-readable planning into manual review. | | Compaction commit | Preview / controlled | Commits a safe partition-local Parquet rewrite preserving sort order IDs. | -| Built-in periodic scheduler | Not claimed | Ticks are operator-triggered; no continuous in-process scheduling. | +| Built-in periodic scheduler | Preview / queue-only | Default off; allowlisted, paged admission with jitter, per-pass quotas, current-job checks, and cooperative drain. Workers and failed/expired-job recovery remain operator-triggered; no automatic execution or hard per-table I/O deadline. | | Delete-file or row-level compaction execution | Not claimed | Manual-review maintenance item. | ## Recovery And Strong Backing Matrix @@ -133,7 +169,7 @@ Do not promote a failure case from live probe or load test to an automated claim ## Unsupported Or Not Claimed -Full AWS S3 Tables control-plane parity; full MinIO AIStor private extensions; full Cloudflare R2 Data Catalog or Alibaba OSS Tables interoperability; built-in periodic maintenance scheduling; active-active multi-region writes; multi-table transactions; no-long-term-data-credential bootstrap; online vendor SDK polling; external catalog policy mirroring; delete-file rewrite or row-level compaction execution; built-in SQL execution; Delta Lake or Hudi; end-to-end SQL row-level DML validation through Spark, Trino, or another engine. +Full AWS S3 Tables control-plane parity; full MinIO AIStor private extensions; full Cloudflare R2 Data Catalog or Alibaba OSS Tables interoperability; automatic maintenance execution and failed/expired-job recovery; durable-strong snapshot-expiration and compaction plan/commit; active-active multi-region writes; multi-table transactions; no-long-term-data-credential bootstrap; online vendor SDK polling; external catalog policy mirroring; delete-file rewrite or row-level compaction execution; built-in SQL execution; Delta Lake or Hudi; end-to-end SQL row-level DML validation through Spark or Trino. DuckDB single-table UPDATE/DELETE/MERGE has its own automated smoke coverage as listed above; this does not establish other engines' DML compatibility. ## Verification diff --git a/docs/architecture/scanner-per-bucket-convergence.md b/docs/architecture/scanner-per-bucket-convergence.md new file mode 100644 index 000000000..f1950981a --- /dev/null +++ b/docs/architecture/scanner-per-bucket-convergence.md @@ -0,0 +1,535 @@ +# Scanner Per-Bucket Convergence Design + +**Use this when:** changing what makes a scanner cycle `Complete`, `Superseded`, or converged; adding or re-classifying a dirty-usage producer; deciding whether a background writeback may invalidate a scanner cycle; changing scanner worker concurrency or throttling; or activating scoped dirty acknowledgements. +**Source of truth:** `classify_nsscanner_cycle`, `scanner_cycle_activity_status`, and `scoped_scan_scope_from_dirty_buckets` in `crates/scanner/src/scanner_io.rs`; `dirty_usage_snapshot_status` and `acknowledge_scoped_dirty_usage` in `crates/scanner/src/scanner_io/dirty_usage.rs`; `scanner_concurrency_limit` in `crates/scanner/src/scanner_io/guards.rs`; the per-disk worker loop in `crates/scanner/src/scanner_io/io_cache.rs`; `DynamicSleeper` in `crates/scanner/src/sleeper.rs`; `ScannerNodeActivity` in `crates/scanner/src/scanner/activity.rs`; `persist_replication_state_if_current` in `crates/ecstore/src/bucket/replication/replication_resyncer.rs`; the publication fences in [scanner-usage-publication.md](scanner-usage-publication.md). + +## Decision + +Date: 2026-10-13. Status: accepted design; phases ship in the order given +under [Delivery](#delivery), each as its own change. + +The scanner is a concurrent collector walking a namespace that mutators keep +changing. It is specified here with the vocabulary of concurrent garbage +collection, because the problems and the known-good answers are the same. + +1. **Tri-color buckets with snapshot-at-the-beginning termination.** A cycle + is `Complete` when every bucket in its start snapshot has been walked and + the publication is durable. Buckets that mutators shaded grey during the + walk stay grey for the next cycle and are reported as pending; they do not + turn the cycle into `Superseded`. +2. **The cluster activity probe stays, as a guardrail only.** It vetoes a + publication when the heap it describes is not the heap that was walked: + topology, peer identity, protocol version, data movement, or a pending PUT + tail changed. Namespace and dirty-usage generation movement alone never + vetoes; it only shades buckets. +3. **Replication status writebacks are status-class barrier hits.** They + shade a bucket for a scoped rescan but are not namespace activity. They + never uncertify a walk and never block cycle completion. The scanner's own + queued replication heals fall under the same class. +4. **Worker concurrency is paced, not derived from disk count.** The number of + bucket walkers per erasure set and the per-object throttle are set from a + pacing target with CPU and I/O ceilings, with disk type as the default + input. Disk count remains a floor, not a cap. +5. **Guardrail vetoes are bounded too.** A probe that fails for one peer + localises to the buckets that peer dirtied; it does not defer the whole + cycle. A veto is always attributed to a named field, retried on a bounded + backoff, and reported with its reason. The guardrail protects publication + correctness; it is not allowed to become a third way of never converging. + +Every liveness rule above is paired with a boundedness rule in +[Boundedness constraints](#boundedness-constraints). A change that improves +one side must restate its effect on the other. + +Backward compatibility is a hard requirement of this design, not a phase. +Every phase must satisfy [Backward compatibility](#backward-compatibility) +on its own: a cluster can be upgraded node by node, and any node can be +rolled back to the previous release, without a scanner reset and without +losing the authoritative usage baseline. + +## Two failure modes + +Scanner defects fall into two opposite classes and a fix for one can create +the other: + +- **Long tail**: part of the namespace is never reached. A cycle keeps being + invalidated, a budget keeps cutting the same buckets, or a queue keeps + reordering new names ahead of old ones. The symptom is usage that never + converges and lifecycle or heal work that never runs for some buckets. +- **Burst amplification**: a sudden wide failure (a disk loss, a mass delete, + a replication resync) makes the scanner discover a large amount of work at + once and the scanner, heal, and replication paths saturate CPU, disk, and + network together. + +A third class sits beside these two and is easy to mistake for the first: + +- **Guardrail false positives**: the cycle is invalidated not by writes but + by the proof machinery itself. An in-flight PUT classified as data + movement, a journal lock timeout reported as an unavailable activity + baseline, or one unreachable peer in a large cluster defers the whole + cycle, and the dirty map is never cleared. Per-bucket certification does + nothing for this class because the walk never reaches certification. The + guardrail therefore carries its own boundedness rules, attributed per + field and localised per peer, so a misclassification can be fixed without + touching the termination rule. + +The current cycle classifier is in the first class. The pre- and post-walk +proofs require the global dirty-usage generation and the full activity +snapshot to be identical, so one write to one bucket during a walk makes the +whole cycle `Superseded`. Scoped rescans then need a converged baseline from +`complete_scanner_cache_baseline_plan_digest`, which only a quiet full cycle +can produce, so every retry is a full walk and cycle time never shrinks. +Since replication writebacks started marking buckets dirty, a resync is a +standing mutator and the scanner's own queued heals write back during its +walk, so the scanner invalidates itself. This is a stop-the-world +termination rule applied to a concurrent collector; it fails on liveness by +construction, not under load. + +The per-bucket data model that a concurrent collector needs already exists: +per-bucket generations are the card table, `notify_scanner_dirty_usage_mutation` +is the write barrier, `DirtyUsageBucketScope::TopLevelEntries` are finer +cards, and `acknowledge_scoped_dirty_usage` is exact per-card clearing. The +cycle-level decision never consumed it. + +## Collector model + +### Colors + +| Color | Meaning | Where it lives | +|---|---|---| +| white | not walked in this cycle | bucket in the start snapshot without a result | +| grey | walked or not, but a mutator touched it since the cycle's start snapshot | bucket present in the dirty map with a generation newer than its start generation | +| black | walked and certified: its result is current with respect to the start snapshot | bucket acknowledged and cleared from the dirty map | + +The write barrier is the existing committed-mutation observer. It shades a +bucket grey by advancing its generation; it never blocks the mutator and it +runs no scanner code inline. Barrier hits carry a class: + +| Class | Producers | Effect on usage | Effect on color | +|---|---|---|---| +| `Namespace` | PUT, multipart complete, copy, delete, delete-objects, bucket create, lifecycle expiry, tier expiration cleanup, heal rewrites that change size or version count | size, versions, delete markers, histograms | shades grey; the bucket is uncertified if hit during its walk | +| `Status` | replication status writeback (`persist_replication_state_if_current` in `Update` mode), scanner-queued replication heal, any writeback that only changes replication or restore state on an existing version | only `replication_stats` folded in by `apply_scanner_size_summary` | none: never shades, never uncertifies a walk, never counts as pending; refreshed by the full walk that every cycle with an active replication rule performs | + +The class is attached in the mutation observer installed by +`init_background_replication` in `rustfs/src/storage/storage_api.rs`. A +`Namespace` hit advances the bucket's generation; a `Status` hit records only +producer coverage. While the scanner is the replication heal driver, +`ScannerMaintenanceFeatures::needs_regular_cycle` forces a full walk on every +cycle of a deployment with an active replication rule, so a `Status` hit needs +no rescan queue: the walk that would consume it always happens. If a later +phase lifts that forcing, `Status` hits must then join the scoped rescan set, +and that set must travel to the coordinator through the peer dirty-usage +snapshot, because hits land on whichever node commits the writeback. + +A replication maintenance walk refreshes compacted prefixes rather than +sampling their old totals. A completed bucket cache with active replication +starts a new coverage sweep even when the cycle is retried; unfinished +checkpoints retain their existing frontier. Set persistence binds the +observed replication statistics to its execution digest, so same-cycle +reuse cannot restore statistics from an earlier walk. None of these cache +requirements changes the namespace generation or publication fences. + +### Snapshot at the beginning + +A cycle is responsible for the namespace as it was at its start snapshot. +Each bucket walk records its start generation for the local node and for +every peer scoped entry resolved by `resolve_remote_dirty_usage_scope`. +Mutations after that point are the next cycle's work. This is what makes +termination possible under continuous writes: the termination condition is +"no grey bucket in the start snapshot", not "no mutation anywhere". + +Usage is a sum, not a reachability fixed point. A certified bucket's value +plus an uncertified bucket's last certified value is a valid snapshot, so the +collector does not need a globally consistent heap to publish. This is why a +`Complete` cycle with pending buckets is sound as an authoritative baseline. + +### Certification + +After the authoritative publication is durable and the root publication +proof holds, the finalize path acknowledges every walked bucket with the +compare-and-remove semantics of `acknowledge_scoped_dirty_usage`: + +- generation unchanged: black, cleared from the dirty map. `Status` hits + never move it; +- generation moved by a `Namespace` hit: stays grey with the newer + generation and counts as pending; +- a peer reported the bucket dirty without a scoped entry the coordinator can + verify: stays grey. This is the mixed-version and partial-peer fallback, + and it is per bucket, not per cycle. + +The whole-cycle generation acknowledgement remains the fallback when no +scoped entries can be built at all. It is never combined with per-bucket +certification in the same cycle. + +### Bounded remark + +Before finalize, the cycle may re-walk grey buckets at most once, within the +remaining cycle budget and with the same per-bucket limits as the main walk. +A bucket still grey after that is handed to the next cycle. There is no loop +until quiet. + +### Cycle classification + +`classify_nsscanner_cycle` keeps its incomplete branches: budget, +cancellation, failed walks, and unverifiable dirty or activity state still +return `Incomplete` or `Deferred`. The final branch changes from a global +equality test to: + +- guardrail vetoed: `Superseded`; +- otherwise: `Complete`, carrying `pending_dirty_buckets` as the count of + buckets left grey by `Namespace` hits. + +`usage_snapshot_converged` becomes `pending_dirty_buckets == 0`. A `Complete` +cycle with pending buckets publishes an authoritative snapshot: certified +buckets are current, pending buckets carry the value the walk produced, +which is at least as recent as the previous certified value and may miss +the write that shaded the bucket; the next cycle rescans them. `DataUsageInfo` +in `crates/data-usage/src/data_usage.rs` gains additive fields beside +`usage_snapshot_set_states`: the sorted list `usage_snapshot_pending_buckets`, +its length `usage_snapshot_pending_bucket_count`, and the routing mark +`usage_snapshot_observational` that a vetoed cycle sets so the observed +object is selected by an explicit mark rather than by the convergence flag. +Readers can tell a certified value from an uncertified one without a schema +break. This replaces the behaviour where a superseded result is published +only as an observed snapshot and never becomes a baseline. + +The pending set is read after the guardrail verdict and before publication. +Certification at finalize still compares each bucket's generation against the +start snapshot, so a write that lands between publication and finalize keeps +its bucket dirty even though the published list does not name it; it is the +next cycle's work, exactly as a write after a converged publication is today. + +### Guardrail + +The probe compares the `ScannerNodeActivity` snapshot before and after the +walk. Field roles: + +| Field | Role | +|---|---| +| `instance_id`, `protocol_version`, `topology_digest` | veto: a peer restarted, changed version, or the topology changed | +| `data_movement_active`, `movement_generation`, `publication_blocked` | veto: rebalance, decommission, or a pending PUT tail can make the walk describe a different generation | +| `maintenance_generation` | veto: no cycle today records whether it published maintenance results, and the field is already part of the structural digest; it becomes a shade signal only once that distinction exists | +| `namespace_generation`, `dirty_usage_generation`, `dirty_usage_pending` | shade signal: consumed per bucket; never a veto on their own | + +`scanner_activity_structural_digest` already separates the structural part +of the snapshot from the generations. The guardrail is the structural digest +plus the movement fields. + +The guardrail is bounded in three ways: + +- **Attributed.** A veto names the field that moved. A whole-snapshot + inequality is never a veto on its own, so a misclassification such as an + in-flight PUT reported as data movement is fixed in that field's producer + and nowhere else. +- **Localised.** When the probe cannot be read from one peer, or that peer + answers with a protocol it cannot be verified under, the coordinator keeps + every bucket that peer dirtied grey (through the peer set resolved by + `resolve_remote_dirty_usage_scope`) and publishes the rest. Only a + movement or topology veto, which describes the whole heap, defers the + cycle. `ActivityBaselineUnavailable` therefore becomes a per-peer + condition, not a cycle outcome, unless no peer answered at all. + Phase 2 ships attribution but not localisation: per-bucket certification + covers the local dirty map, and a remote peer whose `dirty_usage_generation` + or `dirty_usage_pending` moved during the walk still supersedes the whole + cycle (`scanner_peer_dirty_usage_moved`), because without a post-walk scoped + snapshot the coordinator cannot name the buckets that peer dirtied and would + otherwise claim convergence over them. The same rule is applied again after + remote leases are granted. Phase 5 lifts it with scoped acknowledgements. +- **Retried on a bound.** A deferred cycle retries on the existing + superseded backoff schedule with a cap, and `usage_freshness` reports the + field and peer that caused the last deferral so the operator can tell a + rebalance from a broken probe. + +### Generations of buckets + +Most buckets are cold and a few are hot. The scanner keeps two populations +with separate cadences, in the sense of a generational collector: + +- **young**: buckets shaded grey within the last `K` cycles. They are walked + every cycle, scoped to their dirty cards where a scope exists, under a + small per-cycle quantum. +- **old**: buckets black for `K` consecutive cycles. They are walked by a + checkpointed full sweep that is guaranteed to visit every old bucket at + least once every `M` cycles, independent of young-generation load. + +Promotion is `K` consecutive certifications; demotion is any `Namespace` +hit. The split exists so that the long tail has its own cadence instead of +competing with hot buckets for one budget. `K` and `M` are scanner runtime +configuration with defaults; the full sweep keeps today's checkpoint and +resume semantics. + +### Baseline bootstrap + +`complete_scanner_cache_baseline_plan_digest` accepts a `Complete` +publication as the structural baseline whether or not it is converged. +Bucket-cache reuse is governed by color: a black bucket reuses its cache +under the structural plan digest, a grey bucket is rescanned. This removes +the dependency where scoped rescans need a converged baseline and a +converged baseline needs a quiet cycle. The cache execution identity in +[scanner-usage-publication.md](scanner-usage-publication.md) is unchanged +for within-cycle reuse. + +## Pacing and worker concurrency + +Before phase 3 the number of bucket walkers per erasure set equalled the +number of disks in the set, capped by `max_concurrent_disk_scans`, and +`scanner_concurrency_limit` took the minimum of the configured value and the +disk count. Buckets queue on one channel per set and each walker takes one +bucket at a time; inside a bucket the folder walk is sequential and +`DynamicSleeper` sleeps after every object for the work time multiplied by +the speed factor. On a single-disk deployment this is one walker, one bucket, +one object at a time, at roughly one third duty cycle under the default +speed. Walker count is therefore a property of topology, not of the work. + +A collector's pacer sets collector work from the mutator's rate under a +resource ceiling. The scanner adopts the same shape: + +- **Target**: the young generation must be walked within one cycle interval + and the old generation within `M` intervals. The pacer derives a required + object rate from the last cycle's object count and wall time. +- **Walkers**: `available` for `scanner_concurrency_limit` becomes the disk + count multiplied by a per-disk walker factor. The factor defaults from disk + type: 1 for rotational media, higher for flash, where xl.meta decode is CPU + bound and there is no seek penalty. Disk count stays the floor so + distributed deployments keep at least today's parallelism. Shipped as + `scanner.walkers_per_disk` (`RUSTFS_SCANNER_WALKERS_PER_DISK`, `1` through + `16`): each local coordinator disk gets that many walkers, which share one + per-disk scanner admission so a peer still sees one scan session per + physical disk; remote disks keep one walker. The default is `1` everywhere + until a local disk can report its media type + (`rustfs_utils::os::Info::rotational` is never populated today); the flash + default above `1` remains the intent. +- **Throttle**: the `DynamicSleeper` factor is the pacer's output, not a + fixed preset. `ScannerSpeed` presets remain as operator overrides and as + the initial value. Foreground workload backoff in `sleeper.rs` remains the + hard ceiling: the pacer never reduces the sleep below what foreground + admission demands. +- **Ceilings**: a CPU share and a per-disk I/O share bound the pacer's + output. When the target cannot be met inside the ceilings the pacer reports + it in `usage_freshness` rather than exceeding them. + +## Boundedness constraints + +Each rule below bounds a liveness rule above. They are part of the design, +not tuning advice. + +| Liveness rule | Bound | +|---|---| +| Grey buckets are rescanned next cycle | A scoped cycle walks at most a configured number of buckets and objects. Grey buckets beyond that are ordered by waiting age, oldest first, so every bucket is walked within a bounded number of cycles. New names never reorder ahead of an older waiter. | +| `Status` hits are not namespace activity | A `Status` hit costs one atomic OR on the writeback path and no scanner work. It never triggers a dirty-usage wake. A replication resync therefore adds nothing to scanner load beyond the full walk the cycle already performs. | +| A `Complete` cycle with pending buckets is a baseline | When the pending set is the whole namespace (a mass write, a disk loss), the next cycle is declared a full sweep explicitly and uses the existing superseded retry backoff schedule. A full sweep is never run under the scoped cycle's wake cadence. | +| Bounded remark | At most one remark pass per cycle, inside the remaining budget. | +| Old-generation sweep every `M` cycles | The sweep is checkpointed and resumes; it does not restart on a `Namespace` hit. A hit only demotes the bucket to young for the following cycle. | +| Pacer raises walkers and lowers sleep | CPU and I/O ceilings are absolute. Foreground admission backoff cannot be undercut. Walker count never exceeds the ceiling-derived maximum even when the target is missed. | +| Faster convergence discovers heal and replication work sooner | Heal admission keeps its queue bounds and `queue_full` and `dropped` accounting in `crates/scanner/src/scanner_folder/ledger.rs`; replication MRF keeps its pending cap. Convergence speed does not change retry intervals. Discovered work is swept lazily under those bounds, in the sense of a lazy sweeper: discovery and execution are decoupled. | +| Guardrail vetoes publication | A veto is attributed to a field and localised to a peer where the field allows it; only movement and topology defer the whole cycle. Deferred retries use the capped superseded backoff. A probe failure on one peer never costs more than that peer's dirtied buckets. | +| One bucket walk may be large | Inside a bucket, the walk takes a per-prefix quantum: a prefix that exhausts it checkpoints and yields to the next prefix or bucket, and resumes under the existing cursor and page-index fences. No single prefix may consume the whole cycle budget. The walker's own bookkeeping must stay linear in entries per prefix. `RawEnumerationProgress::record_entry` in `crates/scanner/src/scanner_folder.rs` once re-normalized the accumulated entry list on every entry (rustfs#8113: 8,000 entries cost about 40 s); since #8114 the page writer hashes each entry once when its page commits, and a regression test pins 20,000 entries in one prefix to that cost so pacing cannot be built on a quadratic walker. | + +## What stays fail-closed + +- Quota authority is unchanged: an uncertified bucket keeps its last certified + value as the floor, as the usage floor does today. Live writes do not + advance it. +- The storage publication epoch, leadership claim, CAS revision, publication + lease, and tier registry fences are untouched. This design narrows what a + generation movement means; it removes no fence. +- A peer that cannot produce scoped entries keeps every bucket it dirtied + grey. Rolling upgrades degrade to today's behaviour for those buckets and + nothing else. +- Scoped acknowledgements keep the receiver bounds: entry count, byte size, + concurrency, and deadline. A cycle that would exceed them falls back to + whole-cycle acknowledgement for that cycle. + +## Backward compatibility + +Compatibility means four things here, and each phase must prove all four: +an older release reads every artifact the new release writes; the new +release reads every artifact the older release left behind; a mixed-version +cluster keeps publishing wherever it publishes today; and every behaviour an +operator relies on today is reachable by configuration. Rollback must never +require a scanner reset. + +### Persisted artifacts + +| Artifact | Encoding and tolerance today | Rule in this design | Rollback behaviour | +|---|---|---|---| +| `.usage.v2.json` (`DataUsageInfo`) | JSON; every optional field is `serde(default)` and skipped when empty; unknown fields are ignored | The pending-bucket list, its count, and the observational routing mark are additive optional fields. `usage_snapshot_converged` keeps its meaning. No existing field changes type or semantics. | An older reader ignores the pending list and sees a complete snapshot with `usage_snapshot_converged = false`. Its admin and quota readers accept that as an authoritative publication; its scanner does not use it as a scoped-scan baseline and runs full walks until it publishes a converged snapshot of its own, which is today's behaviour, not a reset. | +| `.usage-cache.bin` (`DataUsageCache`) | MessagePack; `DataUsageScanIdentity`, `DataUsageScanProgress`, and `DataUsageRawEnumerationCursor` deny unknown fields; the map-encoded cache metadata tolerates unknown keys | No field is added to a struct that denies unknown fields. Bucket color and start generation are process state in the dirty map, not cache fields. If a per-bucket value must persist, it goes into the map-encoded metadata as `scan_execution_digest` did. The old-generation sweep reuses the existing checkpoint, resume, and cursor fields unchanged. | An older reader ignores any added map key and treats the cache under its existing identity rules. | +| `.bloomcycle.bin` and recovery markers | Versioned by `schema_version`; a mismatch is quarantined | No schema bump. Pending-bucket state is derived from the dirty map and the published snapshot, not stored in cycle state. | Unchanged. | +| `scanner/durable-dirty-producer-replay/.json` | Strict schema; entries deny unknown fields | The barrier-hit class is not persisted. Replayed entries are treated as `Namespace` class, which can only cause an extra scoped rescan, never a missed one. | Unchanged. | +| `.usage.observed.json` | JSON, same tolerance as `.usage.v2.json` | Still written for guardrail-vetoed cycles. A `Complete` cycle with pending buckets writes the authoritative object, not the observed one. | An older release keeps reading observed objects by its existing rules. | + +### Internode protocol + +- `SCANNER_ACTIVITY_PROTOCOL_VERSION` is matched exactly by + `scanner_cycle_activity_status`; a peer on any other version fails the + probe and the cycle is deferred. Phases 1 to 4 therefore change no + `ScannerNodeActivity` wire field and do not bump the version. The + guardrail reinterprets fields that already exist. +- Phase 5 sends scoped acknowledgements over the existing + `SCOPED_DIRTY_USAGE_PROTOCOL_VERSION` receiver, gated per peer by the + existing `probe_only` capability request. A peer without the capability + keeps its dirtied buckets grey and is acknowledged by the whole-cycle path. + No activity protocol bump is required for scoped acknowledgements. +- If a later change does need an activity protocol bump, it inherits the + existing fail-closed mixed-version window and must say so in its + compatibility note; this design does not widen that window. + +### Admin API and console + +- `ScannerUsageFreshnessSnapshot` and the scanner status contract gain only + additive fields with `serde(default)`. No existing field is renamed, + removed, or changes type. +- `last_cycle_result` keeps its existing string values. A `Complete` cycle + with pending buckets reports `complete`; consumers that do not read + `pending_dirty_buckets` see a completed cycle, which is the truthful + reading for them. +- `usage_snapshot_converged` keeps its meaning on every surface that + exposes it, including account info and cluster usage. +- No contract version carried to the console changes. The console gates on + contract versions elsewhere; this design must not be the change that + trips that gate. + +### Configuration + +- New keys (walker factor, pacer ceilings, `K`, `M`, scoped-cycle quantum) + live in the `scanner` subsystem beside the existing keys. The server + config already preserves unknown keys, so an older release leaves them in + place and ignores them. +- `ScannerSpeed` presets and `max_concurrent_disk_scans` keep their names + and accepted values. Today's throttle is reachable by setting the preset + explicitly and the walker factor to 1; the documented operator procedure + for restoring today's I/O profile is that single pair of settings. +- Every new key has a default that keeps rotational-media behaviour equal to + today's. On flash the defaults deliberately change behaviour: a walker + factor above 1 and a pacer-driven throttle are part of the fix, not a side + effect, so they are on by default and called out in the release note of + the phase that ships them. Compatibility is a property of artifacts, + protocols, APIs, and configuration reachability; it is not a promise that + default scanner throughput stays the same. + +### Metrics + +Existing metric names, labels, and meanings are unchanged. New values are +new series. The `superseded_cycles` counter and the superseded code of the +last-cycle-result gauge in `crates/obs/src/metrics/collectors/scanner.rs` +will fall because superseded becomes rare; that is a semantic consequence, +not a rename. + +### Mixed-version behaviour summary + +| Cluster state | Result | +|---|---| +| All nodes on the new release | Full design in effect. | +| Coordinator new, some peers old (same activity protocol) | Per-bucket certification applies to local and capability-bearing peers; buckets dirtied on old peers stay grey and clear through whole-cycle acknowledgement. Publication continues. | +| Coordinator old, some peers new | Old classifier runs; new peers answer the existing activity and acknowledgement protocols unchanged. A `Status` hit on a new peer no longer moves that peer's dirty generation, so it no longer supersedes the old coordinator's cycle; everything else equals today's. | +| Node rolled back after the new release published | The rolled-back node reads the new `.usage.v2.json` as an unconverged authoritative snapshot and continues with today's rules; no reset. | + +No temporary compatibility path is introduced by this design, so no +`RUSTFS_COMPAT_TODO` entry is required. If an implementation phase finds it +needs one, it registers it in +[compat-cleanup-register.md](compat-cleanup-register.md) with a removal +condition before merging. + +## Surfaces + +- `usage_freshness` in the admin scanner status reports + `dirty_pending_buckets` (`Namespace` class only; after certification the + dirty map holds exactly the pending set), the attributed guardrail veto + (`guardrail_veto_total`, `last_guardrail_veto_unix_secs`, + `last_guardrail_veto_reason` as `@`), plus the pacer target, + achieved rate, and whether a ceiling is binding. The console marks usage + stale only for that count or for a guardrail veto. +- `last_cycle_result` reports `success`, its existing label for a completed + cycle, for a cycle with pending buckets; `superseded` is reserved for + guardrail vetoes and the phase-2 peer fallback and should be rare. +- The superseded retry schedule applies only to guardrail vetoes and to + explicit full sweeps. A `Complete` cycle with pending buckets uses the + ordinary dirty-usage wake and the next cycle is scoped to the pending set. +- `rustfs_cluster_usage_objects_snapshot_converged` keeps its meaning. + +## Delivery + +Phases are independent and ship in order; each has an oracle on both failure +modes. + +1. **Status-class barrier hits.** Route replication writebacks, and with + them the scanner-queued heals, off the namespace path so they never + advance the dirty-usage generation, wake the scanner or reach the journal. + Oracle: a single-node site-replication resync reaches a `Complete` cycle + while the resync is running, and `replication_stats` for the bucket + refresh on the following cycle. +2. **Tri-color certification, bounded remark, baseline bootstrap, aging + order, per-prefix quantum, bounded guardrail.** Record start generations, + certify at finalize, publish `Complete` with pending buckets, accept a + non-converged `Complete` publication as the structural baseline, order + grey buckets by waiting age under a per-cycle quantum, give each prefix + inside a bucket its own quantum, and attribute and localise guardrail + vetoes. The first change of this phase ships the termination rule: + certification, `Complete` with pending buckets, the baseline bootstrap + and guardrail attribution. Bounded remark, aging order, the per-prefix + quantum and guardrail localisation follow as their own changes. Oracle: a single-node deployment with one bucket written every + few seconds reaches `Complete` on the first full walk, reports exactly + that bucket pending, and the next cycle walks only that bucket; a + deployment with more buckets than the quantum walks every bucket within + the bounded number of cycles; a bucket holding one flat prefix of roughly + a hundred thousand entries no longer starves the buckets after it; a + multi-node deployment with one peer's probe failing publishes every bucket + that peer did not dirty. +3. **Pacer and walker concurrency.** Pin the entry bookkeeping in + `RawEnumerationProgress::record_entry` to linear cost first (the + clone-per-entry itself was removed by #8114; the regression test and the + in-place page writer are this phase's groundwork), then decouple walker + count from disk count (`walkers_per_disk`, shipped) and drive the throttle + from the pacing target under CPU and I/O ceilings (open). Oracle: on a single flash disk the same namespace + converges in a fraction of today's cycle time with scanner CPU under the + ceiling; on rotational media the default factor keeps today's I/O + profile; a foreground write load at the admission threshold still backs + the scanner off, and latency-sensitive foreground reads on a single NVMe + disk stay within their pre-scanner envelope while a cycle runs. +4. **Generations and the old sweep.** Promote and demote buckets, run the + checkpointed old sweep on its own cadence. Oracle: a cold bucket is walked + at least once every `M` cycles while a hot bucket is written continuously. +5. **Scoped acknowledgement sender.** Activate production of scoped entries + for peers behind the existing receiver so distributed deployments certify + per bucket. Oracle: a multi-node deployment with writes on one node + certifies all buckets written elsewhere and leaves only the hot bucket + grey; a peer on the previous protocol version leaves its dirtied buckets + grey and nothing else. + +Phase 1 lets a replicated single-writer deployment converge; phase 2 makes +continuous business writes converge; phase 3 shortens the cycle itself; +phase 4 protects the long tail; phase 5 extends all of it to distributed +clusters. + +### Validation topologies + +Each phase is validated against the deployment shapes that reported the +defect, not only against synthetic fixtures. The shapes are listed by their +characteristics; the originating reports are the issues referenced from the +tracking discussion. + +| Shape | Exercises | +|---|---| +| single node, single disk, site replication resync in progress | phase 1: `Status`-class hits, self-invalidation by queued heals | +| single node, single disk, parity 0, on the order of 1,500 buckets and 1.4 million objects, continuous writes, no lifecycle or replication | phases 2 and 3: termination rule, baseline bootstrap, aging order, walker count | +| single node, one bucket whose metadata prefix holds on the order of 100,000 keys with noncurrent versions and delete markers under Object Lock | phase 2 per-prefix quantum, phase 3 bookkeeping prerequisite | +| 40 nodes, one pool, 400 drives, continuous log-chunk writes | phase 2 bounded guardrail, phase 5 scoped acknowledgements, journal and lease behaviour under load | +| 4 nodes, 48 drives, on the order of 600 million objects, continuous file-system-backed writes | phase 5 at scale, guardrail attribution under a permanent PUT tail | +| single node, NVMe, latency-sensitive small-object foreground reads | phase 3 ceilings and foreground backoff | + +A phase is not complete until its oracle holds on every shape it names. A +pass on synthetic fixtures alone is evidence for a pull request, not for +release. + +## Non-goals + +- Soft quota or best-effort usage. The authority decision in + [scanner-usage-authority-decision.md](scanner-usage-authority-decision.md) + stands. +- Removing or merging any publication fence. +- Making replication statistics authoritative. They remain observational and + may lag by one cycle after a `Status` hit. +- Parallelising the folder walk inside one bucket. Walker count is raised at + bucket granularity and the walk inside a bucket is bounded by the + per-prefix quantum; intra-bucket parallelism is a separate change with its + own ordering and checkpoint implications. diff --git a/docs/architecture/scanner-usage-publication.md b/docs/architecture/scanner-usage-publication.md index 12c600d21..67c152eb5 100644 --- a/docs/architecture/scanner-usage-publication.md +++ b/docs/architecture/scanner-usage-publication.md @@ -29,10 +29,15 @@ rename work ends, including when the request caller is cancelled. Scan admission remains movement-only so sustained PUTs do not stop namespace walks and scanner-driven lifecycle discovery. The post-walk local publication check and remote publication leases reject pending fanouts. Begin/end namespace generations -invalidate scans and cached plans across the fanout; after acquiring remote -leases, the coordinator rechecks the full activity digest before publishing an +invalidate cached plans across the fanout; after acquiring remote leases, the +coordinator rechecks the guardrail fields of the activity snapshot (the +structural digest and the movement fields, plus the peers' dirty-usage state +while per-bucket certification is local only) before publishing an authoritative aggregate. This catches a tail that finishes between the scan's -last probe and lease acquisition. +last probe and lease acquisition. Local namespace and dirty-usage generation +movement in that window is not a veto: its buckets stay in the dirty map and +are the next cycle's work, as described in +[scanner-per-bucket-convergence.md](scanner-per-bucket-convergence.md). This adds no namespace or movement lock. An already-verified older snapshot may still precede a newly started write. Sustained or stalled PUT tails can delay @@ -121,6 +126,24 @@ reuse without this identity. Older readers can ignore the added map key, but older writers do not enforce its fence; readability is not a mixed-version publication-safety guarantee. +Bucket checkpoints distinguish completed traversal from object health. A failed +object does not stop later completed subtrees from advancing the traversal +frontier. On resume, a sorted index of cached entries with `failed_objects > 0` +excludes their containing subtrees from reuse; healthy covered siblings keep +their exact cached accounting. Failure counters survive retry-cache TTL and +capacity eviction, and any remaining entry failure keeps the bucket partial. +Directory traversal errors and unresolved size reconciliation still stop the +frontier because they do not have the same per-entry coverage proof. + +Coverage follows path-component traversal order, including punctuation in sibling +names. Coverage receipts use digest domain 3 and bind every entry in each covered +subtree using the same predicate as resume, while excluding the unvisited suffix. +No unsigned nested failure can authorize subtree reuse. Current +readers accept and migrate domain 2 and legacy epoch-bound receipts. Older +binaries cannot validate a domain 3 receipt and safely reread the bucket rather +than applying their unconditional covered-subtree skip. The serialized receipt +shape, cache CAS, leader epoch and publication fences are unchanged. + ## Persisted Objects The persisted objects are part of the compatibility contract. Removing one @@ -142,7 +165,11 @@ requires a compatibility window and a dedicated cleanup entry. Observed snapshots are a diagnostic and availability layer. They may be served only when the snapshot explicitly reports that it is partial or observational, and only to consumers that do not make hard quota, durability, or deletion -decisions from it. Admin usage views may expose this state with completeness +decisions from it. The scanner routes a snapshot to the observed object by the +explicit `usage_snapshot_observational` mark that a vetoed or unverifiable cycle +sets, not by `usage_snapshot_converged`: a complete cycle that certified every +bucket but the ones written during its walk publishes an unconverged +authoritative snapshot that names those buckets as pending. Admin usage views may expose this state with completeness flags so operators can see progress while the authoritative publication is blocked. Quota enforcement must not use an observed snapshot as the current usage authority. @@ -195,6 +222,73 @@ Missing, undecodable, or identity-less data is not converted to zero. It is reported as uninitialized, recovery-required, observed-only, or unavailable according to the reader's surface. +## Diagnosing Incomplete Census Progress + +Treat bucket coverage, set-worker completion, cycle certification, and usage +publication as separate stages. A successful save can be observational, and a +worker finishing does not certify its namespace coverage. + +| Signal | Meaning and next evidence | +| --- | --- | +| `bucket_checkpoint_prepared` | Checkpoint admission identifies the bucket, pool/set, cycle/leader epoch, and previous cycle/epoch. `resume_position` distinguishes retained traversal position from a fresh sweep; `mixed_coverage` reports whether that retained prefix began under another plan. | +| `bucket_sweep_incomplete` | A bucket reached the end of a sweep but needs another verification pass. `unresolved_objects` reports the complete coverage/health gate, including failed child-cache entries that can be absent from the root counter and diagnostic failure map. Inspect it with `coverage_gap`, `root_failed_objects`, `failed_objects`, `size_reconciliation_entries`, and `mixed_coverage`, using bucket, pool/set, and cycle/leader epoch. `resume_position_on_entry` identifies a sweep entered with traversal position; `resume_position` reports the returned cache position after terminal cleanup. Multiple blockers can coexist. | +| `partial_saved_not_published` | The local or remote worker successfully saved a terminal Partial cache. Correlate its bucket, pool/set, cycle/leader epoch and `resume_position` with admission and sweep events. This confirms persistence, not complete coverage or authoritative publication. | +| `remote_scan_partial` | The coordinator received a Partial terminal response for the named bucket and disk. Correlate the worker's bucket-sweep or checkpoint events; Partial is not a complete bucket root. | +| `bucket_result_rejected` | A bucket result did not match the cycle's frozen tier-registry generation and was not applied. | +| `set_cache_publish_skipped` | `completed_buckets` counts distinct bucket roots successfully applied to this set cache. Use `first_incomplete_bucket` with pool/set/cycle to locate one actual gap in this set. Inspect `unprocessed_buckets` and budget/cancellation state, then correlate that bucket's worker events. | +| `set_scan_completed` | Set workers have finished. `coverage_complete` reports the structural coverage gate, not durable publication. This event alone does not prove a complete set snapshot. | +| `incomplete_coverage` | The cycle did not obtain complete structural coverage. This does not identify a hot bucket or prove that ordinary writes caused the gap. | +| `publication_proof_pending` | Publication is waiting for a retryable proof. Use cycle/stage, `pending_duration`, the preceding `publication_proof_started` kind, and the peer/purpose in the error. This event does not assert that the walk was Complete. | + +For a report such as 52 completed buckets out of 53, first identify whether +the missing bucket was Partial, failed, never admitted, or rejected at merge. +Only then inspect its worker evidence. Mixed coverage is a resumable sweep +under a newer plan; failed objects and size reconciliation require different +follow-up. A global dirty-generation change alone does not establish which +bucket failed to produce a complete root. + +A mixed tail clears traversal positions before returning Partial. If that +terminal cache is saved, the next fresh sweep binds both coverage plans to its +new request, even if ordinary writes continue between cycles. A periodic +checkpoint saved before terminal cleanup still contains traversal position; +resuming it under a changed plan can legitimately be mixed again. Correlate +admission, sweep completion, and terminal persistence for the same bucket and +pool/set across consecutive cycles. Samples from different sets cannot establish +a repeated mixed loop. Check save failures, remote interruption, and leadership +changes if only periodic progress survives. + +An unknown bucket cannot be quarantined as zero in an authoritative total. +Existing pending-bucket semantics still require complete structural coverage. +A separately labelled subtotal would need proof for each included bucket across +every source, with missing buckets explicitly unavailable or stale; it must not +advance full-census authority or acknowledge their dirty work. + +Use a bounded DEBUG window on the capacity-scanner leader and the workers +serving the affected bucket: + +```text +RUST_LOG=warn,rustfs::scanner=debug,rustfs::scanner::io=debug,rustfs::scanner::folder=debug +``` + +Capture one representative cycle with its effective budget configuration, +source commit on every node, per-set completion summaries, partial-bucket +events, remote errors, publication kind and result, and cycle-state persistence. +The summaries report counts and bounded bucket identifiers, not object-path +inventories. Keep logs access-controlled and redact deployment identifiers +before sharing them. + +An unlimited walk budget can also leave a retryable authoritative publication +waiting for leadership loss or a successful proof. Observational publication +still retains the movement fence, while authoritative publication requires the +strict namespace proof. Do not remove those fences to clear a status. Verify +that a deployment contains the relevant fixes by commit ancestry; a newer +release name alone is not evidence that it includes current main. + +Keep usage objects, checkpoint data, and recovery material intact during +diagnosis. Acceptance requires usable census values, appropriately advancing +dirty acknowledgements after publication, and progress across later cycles or +leadership changes. Worker completion timestamps alone are insufficient. + ## Existing Fixes As Invariants Several prior scanner fixes are consequences of this contract rather than diff --git a/docs/operations/architecture-guard-troubleshooting.md b/docs/operations/architecture-guard-troubleshooting.md index e70a16107..be23906f8 100644 --- a/docs/operations/architecture-guard-troubleshooting.md +++ b/docs/operations/architecture-guard-troubleshooting.md @@ -34,6 +34,16 @@ source lines. If it fails after a doc edit, you reworded or removed a guarded line — restore the wording or update the script deliberately in the same PR, with rationale. +## OIDC architecture boundaries + +`check_architecture_migration_rules.sh` also runs `scripts/check_oidc_architecture_boundaries.sh`. Its rule identifier and `file:line` point to the source boundary that changed. Keep OIDC provider configuration, discovery, transport, state, and runtime under `crates/iam/src/oidc/`; keep verified identity mapping under `crates/iam/src/federation/mapper.rs`; construct the runtime in IAM startup and `rustfs/src/startup_auth.rs`; publish the federation service and OIDC query together through `AppContext`. The Keycloak workflow must include the OIDC module path for pull requests and pushes. + +OIDC production modules must not reference RustFS `Credentials` or `FederatedSessionTransaction`. Federation orchestration creates the transaction, and the Admin session binding issues credentials. + +The Admin OIDC handler delegates persisted configuration reads, updates, and provider validation to `admin/service/oidc_config.rs`. Site replication uses `OidcConfigQuery::site_replication_snapshot()` and receives only the fields required for its response, including a hashed client secret. The Keycloak workflow also tracks the Admin configuration service path for both events. + +Run `bash scripts/check_oidc_architecture_boundaries.sh --self-test` after editing a rule. Its negative and accepted examples under `scripts/fixtures/architecture_migration_rules/oidc/` must produce the exact expected rule identifiers. Then run the full architecture guard. + ## `check_unsafe_code_allowances.sh` Every `#[allow(unsafe_code)]` needs a `SAFETY:` comment within a few lines. diff --git a/docs/operations/bucket-heal-recovery.md b/docs/operations/bucket-heal-recovery.md index 60d10b3ca..b00a5b62b 100644 --- a/docs/operations/bucket-heal-recovery.md +++ b/docs/operations/bucket-heal-recovery.md @@ -1,22 +1,50 @@ # Bucket heal recovery and bucket recreation +**Use this when:** diagnosing incomplete administrator heal coverage, restoring a missing bucket directory, or replaying a bucket heal after restart. + Explicit administrator bucket heals bind the bucket name to its authoritative, non-nil incarnation UUID at admission. The same identity follows the task through queueing, retries, shutdown persistence, and startup replay. Deleting a bucket and creating another with the same name does not transfer the old task to the new bucket. +## Incomplete bucket discovery + +An administrator heal of the cluster or a selected pool/set enumerates bucket candidates from every selected disk, including directories below the ordinary namespace write quorum. Every selected disk must answer before discovery is complete. A pool/set task checks only its selected set; unavailable unrelated pools do not invalidate that scope. Candidate discovery grants no permission to recreate a bucket. Ordinary S3 and scanner listings retain their quorum rules. + +Incomplete discovery uses the existing bounded task retry budget. Once exhausted, the canonical outcome is `execution.state=aborted`, `execution.reason=untraversable`, and `coverage=partial`; the legacy summary is `stopped`. Object counters can remain zero because traversal never started. A genuinely empty, completely enumerated namespace still finishes successfully. Dry runs use the same coverage checks. + +Mutating cluster and pool/set tasks persist their resolved bucket scope only after complete discovery. Resumed checkpoints keep their original frozen scope, including a valid empty scope. A cluster task pins every bucket's incarnation before its first mutation and checkpoints its successful contiguous prefix. Checkpoints are batched into at most 32 intervals per traversal to bound inventory rewriting; an interrupted interval is replayed under the same generations. A fresh task is required to include buckets created after discovery. Cluster dry runs retain their observation-only behavior without requiring bucket incarnation metadata. + +## Restoring a missing bucket directory + +For a bucket that is still intended to exist but has lost its directory on one drive: + +1. Check that the peer drives and authoritative bucket metadata remain available. +2. Start an authenticated administrator heal with `recursive=true`. For the v3 API, use `POST /rustfs/admin/v3/heal/` for the cluster or `POST /rustfs/admin/v3/heal/` for one bucket. A deep scan (`scanMode=2`) additionally checks data integrity. Retain the returned `clientToken` and poll the same endpoint with that query parameter. +3. Inspect both execution and coverage in the terminal outcome. A readable object on a healthy peer alone does not establish that the missing copy was restored. +4. Verify recovered object contents and lengths against known hashes. In an isolated recovery test, stop the original healthy source and read through the repaired target to demonstrate independent recovery. Do not take a production source offline merely to inspect a heal result. + +Default administrator bucket healing can reconstruct missing directories when every selected disk is reachable, each selected set retains a bucket directory, and persisted metadata proves the current generation was committed and has not been retired. The lifecycle fence remains held through reconstruction. Missing, corrupt, uncommitted, retired, or mismatched authority fails closed. Dry runs leave directories unchanged. `recreate=true` remains an explicit recovery option; it is not permission to revive a bucket deleted through S3. Bucket incarnation checks reject stale tasks after deletion and same-name recreation. + +Legacy objects without independent storage commitments can be physically repaired while their canonical dispositions remain `unknown`. Do not infer a verified repair count from progress counters; see [heal terminal reports](../architecture/heal-terminal-reports.md). + ## Recovery records and stale tokens -Pending records use schema 3 in `.rustfs.sys/root-heal-.json` on the existing +Pending records use schema 4 in `.rustfs.sys/root-heal-.json` on the existing coordinator disk. A bucket record includes `bucket_incarnation_id`. Options, task ID, -and the remaining retry budget retain their existing recovery semantics. +and the remaining retry budget retain their existing recovery semantics. A cluster +record additionally carries its frozen bucket generations and completed prefix. +Cluster records with this scope have a 16 MiB limit; other intents and legacy +terminal markers retain the 64 KiB limit. Initial cluster snapshots reserve 1 MiB +for later progress and bounded object outcomes. Oversized inventories fail before repair. A missing bucket, changed incarnation, or legacy bucket record without a usable identity becomes a durable `Failed` terminal result containing `stale_bucket_incarnation`. The original token remains queryable under the existing terminal retention policy. Recovery never fills a legacy task with the identity of -the bucket currently using its name. Existing schema 1 cluster records and schema 2 -non-bucket records keep their recovery behavior. Unsupported, corrupt, and oversized +the bucket currently using its name. Legacy schema 1–3 cluster intents lack a frozen +generation inventory and require a fresh task after upgrade. Other supported legacy +intents and terminal reports remain readable. Unsupported, corrupt, and oversized records remain intact and receive a sibling `quarantined-root-heal-{intent|terminal}-.json` marker. The marker fences the original task ID while allowing an independent compensation heal with a fresh ID. @@ -42,15 +70,16 @@ missing or nil identities and unsigned bodies are rejected. Older nodes return `Unimplemented`; senders do not retry through an unfenced method. Upgrade all participating nodes before expecting these bucket heals to complete. Unscoped RPC bodies retain their prior canonical encoding. Older coordinators do not -understand schema 3 pending records. Finish or cancel pending administrator tasks +understand newer pending record schemas. Finish or cancel pending administrator tasks before downgrading a coordinator; do not rewrite the schema to force replay. The lifecycle fence remains mandatory even when `nolock` disables the optional object namespace lock. Deployments with namespace locking disabled cannot perform incarnation-bound bucket healing. -This contract covers explicit administrator **bucket** tasks. Cluster, prefix, -object, scanner, and replacement tasks retain their existing admission semantics. +The generation-bound repair contract covers explicit administrator **bucket** tasks +and the frozen buckets of mutating administrator cluster tasks. Prefix, object, +scanner, and replacement tasks retain their existing admission semantics. It does not introduce a new object-generation protocol or claim rejection of every already-dispatched native syscall after distributed lock loss; see the existing [heal concurrency model](../architecture/heal-concurrency-model.md). diff --git a/docs/operations/bucket-operation-admission.md b/docs/operations/bucket-operation-admission.md index bab36f986..3e46f1bb7 100644 --- a/docs/operations/bucket-operation-admission.md +++ b/docs/operations/bucket-operation-admission.md @@ -1,10 +1,10 @@ # Bucket creation and deletion admission -`CreateBucket` and `DeleteBucket` share a process-wide execution budget of eight active transactions. Short bursts wait in a FIFO queue with space for 128 waiting requests. Each request can wait up to 30 seconds for an execution slot. These limits protect storage and namespace-lock resources while allowing parallel test workers to create isolated buckets. +`CreateBucket` and `DeleteBucket` share a process-wide execution budget of eight active storage mutations. Short bursts wait in a FIFO queue with space for 128 waiting requests. Each request can wait up to 30 seconds for an execution slot. A separate total budget bounds all unfinished operations to 136, including queued requests, mutations, and post-commit peer reloads. These limits protect storage, namespace-lock, and notification resources while allowing parallel test workers to create isolated buckets. -A request that finds the queue full or exceeds the queue wait returns HTTP 503 with the S3 error code `SlowDown` and `Retry-After: 1`. This is an overload response, not a bucket-name or filesystem error. The active and waiting limits are shared by creation and deletion and are currently fixed. +A request that finds the queue full, exceeds the queue wait, or reaches the total in-flight limit returns HTTP 503 with the S3 error code `SlowDown` and `Retry-After: 1`. Admission rejection happens before storage mutation begins. This is an overload response, not a bucket-name or filesystem error. The limits are shared by creation and deletion and are currently fixed. -Waiting requests do not start storage mutations. Disconnecting while waiting releases the queue slot. Once admitted, the transaction owns its execution slot until the mutation and post-commit hooks finish, including when its caller disconnects. +Waiting requests do not start storage mutations. Disconnecting while waiting releases both the queue slot and its total-operation slot. Once admitted, the detached task continues even when its caller disconnects. Creation releases its execution slot after the storage mutation, site-replication hook, and associated locks finish, then waits for peer metadata reloads before returning. Its total-operation slot remains held until that work finishes, so slow reloads cannot accumulate unbounded background tasks or occupy every storage execution slot. Deletion holds its execution slot through its usecase; its existing asynchronous peer cache invalidation is unchanged. For automated tests and applications: diff --git a/docs/operations/bucket-usage-metrics.md b/docs/operations/bucket-usage-metrics.md new file mode 100644 index 000000000..ed65d05a7 --- /dev/null +++ b/docs/operations/bucket-usage-metrics.md @@ -0,0 +1,122 @@ +# Bucket usage metrics + +Use these metrics to collect every bucket's logical usage through the existing +OpenTelemetry pipeline. RustFS reads the cached scanner usage snapshot once per +collection; exporting metrics does not enumerate objects or run an inline scan. + +## Export and query + +Configure the RustFS nodes with an OTLP/HTTP destination and a stable cluster ID: + +```bash +RUSTFS_OBS_ENDPOINT=http://otel-collector:4318 +RUSTFS_OBS_METRICS_EXPORT_ENABLED=true +OTEL_RESOURCE_ATTRIBUTES=rustfs.cluster.id=production-a +``` + +Append the cluster ID to any existing resource attributes. The shipped +[Collector configuration](../../.docker/observability/otel-collector-config.yaml) +receives OTLP and exposes Prometheus samples on port 8889; its +`resource_to_telemetry_conversion` promotes the cluster ID to +`rustfs_cluster_id`. Prometheus scrapes the Collector. See +[storage metrics](storage-metrics.md#configure-the-pipeline) for pipeline setup +and resource identity. + +The following existing gauges carry a `bucket` label: + +| Metric | Meaning | +|---|---| +| `rustfs_cluster_usage_buckets_total_bytes` | Logical scanner bytes across retained object versions, including noncurrent versions; excludes delete-marker bytes. | +| `rustfs_cluster_usage_buckets_objects_count` | Scanner object-entry/key count, separate from version and delete-marker counts. | + +Logical usage differs from physical drive usage, which includes erasure-coding +overhead and can differ because of compression or remote tiering. Object-entry +count can include keys with only noncurrent versions or delete markers, so it +need not equal the number of currently visible objects from `ListObjectsV2`. + +For monitoring, query all buckets in one cluster: + +```promql +max by (rustfs_cluster_id, job, bucket) ( + rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="production-a"} +) +``` + +```promql +max by (rustfs_cluster_id, job, bucket) ( + rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="production-a"} +) +``` + +Each reporting node exports the whole cluster's logical usage. Summing those +copies would multiply the result by the node count. Keep cluster identity when +grouping: independent clusters can share both a job and a bucket name. The +bundled Bucket Size and Bucket Objects panels use the selected **Storage +cluster** and deduplicate that cluster's reporting nodes. + +Nodes can briefly observe different snapshots. `max` is useful for dashboards +but can retain an older, larger value after deletes. For a consistent accounting +observation, select one reporting node for both bytes and object count: + +```promql +rustfs_cluster_usage_buckets_total_bytes{ + rustfs_cluster_id="production-a", service_instance_id="node1:9000" +} +``` + +Use the same selector with `rustfs_cluster_usage_buckets_objects_count`, and +check the promoted `service_instance_id` in your Collector output. These +floating-point, periodic gauges are usage observations, not an exact billing +ledger or a transactional view of writes and deletes. + +## Freshness and missing values + +Usage collection runs on `RUSTFS_METRICS_CLUSTER_INTERVAL_SEC` (60 seconds by +default), independently of the OTLP export interval +`RUSTFS_OBS_METER_INTERVAL` (30 seconds by default). The separate bucket +collector defaults to 300 seconds. Neither interval forces a scanner cycle. + +Check these gauges on the same reporting node as the bucket values: + +- `rustfs_cluster_usage_objects_since_last_update_seconds`: age of the + persisted usage snapshot when it was collected. A bucket cleanup can advance + the timestamp; it is not necessarily the age of a full object rescan. +- `rustfs_cluster_usage_objects_snapshot_converged`: `1` for a converged + snapshot, `0` when convergence is not confirmed, including observational or + unknown state. Complete observations can still supply usage during sustained writes. + +Before a complete snapshot covers the current bucket namespace, the usage +collector omits its values. A missing series is unknown; do not replace it with +zero for accounting. A bucket confirmed empty in a complete snapshot exports +zero bytes and zero objects. + +A temporary collection failure preserves the last valid export snapshot. +RustFS stops exporting that snapshot once collection stalls beyond three times +the larger collection/export interval. Deleted bucket series receive bounded +zero tombstones and then stop being exported after a valid updated snapshot. +The Collector can retain the last sample after export stops; the shipped +configuration uses a five-minute `metric_expiration`. Prometheus history also +retains past samples. + +If metrics are absent, first check OTLP connectivity and metrics export enablement, +then the scanner leader's `GET /rustfs/admin/v3/scanner/status` and the admin +usage snapshot. A shorter scrape interval cannot repair an incomplete scanner +snapshot. The scanner publication and cache path is shared with the Console's +usage display. + +## Regression checks + +The SDK test uses the production bucket collector and an actual OpenTelemetry +reader to verify two populated buckets, a confirmed-empty bucket, decreasing +usage, and removal from subsequent exports: + +```bash +RUSTUP_TOOLCHAIN=stable cargo test -p rustfs-obs actual_sdk_exports_bucket_size_and_object_count_as_current_gauges --lib +``` + +The PromQL fixture checks same-name buckets in separate clusters and multiple +reporting nodes. Run it from `.docker/observability/tests`: + +```bash +promtool test rules bucket-usage.test.yml +``` diff --git a/docs/operations/cluster-lifecycle-operations.md b/docs/operations/cluster-lifecycle-operations.md index f07056026..87f7c9ab9 100644 --- a/docs/operations/cluster-lifecycle-operations.md +++ b/docs/operations/cluster-lifecycle-operations.md @@ -44,6 +44,14 @@ Rules enforced by `DisksLayout::from_volumes`: with more than one argument every Startup order (`init_startup_storage_foundation` in `rustfs/src/startup_storage.rs`): parse endpoints, validate storage classes against every pool (`ECStore::validate_startup_storage_class`), enforce the unsupported-filesystem policy, initialize local disks, initialize lock clients, then load or create the format quorum and build the store. A storage-class error stops startup before any disk is touched. +### 2.1 Windows endpoints + +Multi-node and multi-pool layouts need URL endpoints, so on Windows the drive path goes after the port with forward slashes: `http://node{1...4}:9000/D:/rustfs/data{1...4}`. `Endpoint::get_file_path` (`crates/ecstore/src/layout/endpoint.rs`) turns the URL path `/D:/rustfs/data1` back into `D:/rustfs/data1` for every scheme; a URL path without a drive, such as `/export1`, stays a URL path. A drive root (`D:\` or `http://node1:9000/D:/`) is accepted in both forms. UNC shares (`\\nas\share\d1`, `//nas/share/d1`, `\\?\UNC\...`) are rejected at parse time: they are not local volumes, and parsing would read the share server as the endpoint host. + +Endpoint parsing then requires distinct physical disks (`validate_local_physical_disk_independence` in `crates/ecstore/src/layout/endpoints.rs`). On Windows the device identity is the volume mount point from `GetVolumePathNameW` (`crates/utils/src/os/windows.rs`), so `D:/rustfs/data1` through `D:/rustfs/data4` are one disk and startup fails with `must use distinct physical disks`. Give each endpoint its own volume (a drive letter or a volume mounted into an NTFS folder). For a single-machine test only, `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` skips the check; it gives no protection against losing that volume. + +A single local path (`rustfs D:\rustfs\data`) is a single-node single-drive deployment and can never gain a pool (§7). If the machine may expand later, start with URL endpoints and at least two drives per pool. + ## 3. Startup configuration | Variable | Effect | Boundary | diff --git a/docs/operations/delete-transaction-recovery.md b/docs/operations/delete-transaction-recovery.md new file mode 100644 index 000000000..69406465c --- /dev/null +++ b/docs/operations/delete-transaction-recovery.md @@ -0,0 +1,209 @@ +# Delete transaction recovery + +An interrupted object deletion can leave metadata backups, reserved shards, or +an unpublished target. Normal S3 listings cannot discover all of these files. +Recovery therefore inspects raw transaction records on every configured drive +in the owning erasure set under the bucket incarnation fence and exact-key +write lock. It does not modify GET or PUT behavior. + +## Decisions and replay + +New deletions assemble the backup, versioned prepared journal, and exact target +in the private system temporary area, then atomically publish the complete +transaction directory before modifying metadata. Interrupted preparation cannot +leave an undiscoverable target-only record or temporary file in the object key. After the metadata write quorum, the coordinator publishes durable +commit receipts to a write quorum before returning success. A failed or lost +publication response is indeterminate: recovery must inspect the persisted +records instead of immediately restoring the backup. + +| Evidence | Action | +| --- | --- | +| New prepared transaction, no decision, intact original shards and matching generation | Restore and sync the original metadata before removing preparation evidence. | +| Structured commit receipt | Publish the certified target to matching returning participants and finish deletion. | +| Missing, unavailable, or unsupported participant | Preserve records and retry when every configured participant can be inspected. | +| Newer generation or partially overlapping version history | Preserve records and report blocked recovery. | +| Historical backup or disk-local marker without a structured set decision | Preserve records during automatic recovery. | + +Missing local metadata does not authorize user-bucket residue cleanup, even +when the current object exists on a metadata quorum. Legacy local cleanup is +restricted to internal system-bucket maintenance. User-bucket legacy residue +uses only the guarded set-wide removal described below. Dry-run and execution +share the same source-generation check; an already-absent explicit non-null +version can converge without touching unrelated live versions. + +Rollback preserves native multipart metadata and integrity sidecars. Protected +parts require their declared proof generation; recovery verifies both the proof +records and actual shard payloads before restoring metadata or removing +recovery evidence. Missing or corrupt required proofs leave the transaction +blocked with its backup intact. + +A disk-local `delete-data.` marker does not prove an erasure-set decision. +Returning drives cannot restore stale data after a certified deletion. Commit +receipts remain until every participant and related transaction for the key +has converged. For consecutive certified version deletions, an older decision +can finish its own data cleanup after later deletion removes current metadata. +A prepared copy needs a matching same-transaction decision before promotion; +absence alone cannot certify it. Cross-transaction null-version supersession also requires causal +predecessor evidence; timestamps alone cannot authorize it. Cleanup respects active snapshots and removes only validated +transaction files, with directory removal performed nonrecursively. + +A live object upgraded from 1.0.1 can still carry an old +`reserve-delete-data.` inside its data directory. A later certified +deletion may reclaim that regular-file reservation together with the payload +only in the directory owned by the current transaction. The old reservation +never authorizes deletion by itself. Foreign commit markers, malformed marker +names, symlinks, unknown files, live versions, and UUID-prefix child objects +remain protected. Snapshot release uses the same checked cleanup path. + +Ordinary heal applies the same boundary to orphan data directories. A UUID +directory name alone is not evidence that a directory contains disposable +shards: it can be a child object's key. Reclamation requires a complete drive +inventory, readable parent metadata that does not reference the directory, no +pending transaction records, and a leaf containing only recognized payload +files. The final mutation rechecks these conditions under the metadata lease, +including after snapshot deferral. It unlinks only the classified files and +removes the empty directory nonrecursively; strict mode synchronizes both the +leaf and its parent. Missing metadata or an unsupported inspection RPC leaves +the directory intact for later recovery. +The evidence check traverses empty pages until completion and owns its iterator +state for the request, so large multipart directories do not block cleanup and +completed or canceled probes do not consume shared discovery cursors. + +## Entry points + +Every startup, including a clean restart, starts incremental raw discovery of +local transaction records. During normal operation, deletion registers a +bounded, coalesced exact-key retry hint before preparing metadata. Synchronous +completion clears only its own hint generation; an uncertain publication or +unfinished cleanup remains eligible for automatic replay without another +DELETE or administrator request. The on-disk journal and receipt remain the +only durable decision authority; the in-memory hints contain no commit decision. +Returning drives and queue overflow request discovery again, and a low-frequency +bounded reconciliation scan covers records written through remote participants. +Discovery advances by pages and retries failed sources separately, rather than +rescanning every healthy bucket on each maintenance tick. +Corrupt record contents remain discoverable by their physical key; exact-key +inspection rejects them without hiding later records. Unsafe paths, symlinks, +and structural filesystem errors still stop that discovery scope. + +Unclean-shutdown erasure-set healing also performs transaction recovery before +ordinary object listing. Administrator bucket, set, prefix, and object heal use the +same recovery core. Existing heal selection, lifecycle admission, namespace +locking, and movement guards still apply. Prefix recovery settles only matching +keys, including transactions hidden from listings. Dry-run inspects and reports without +settling transactions. + +An unconditional DELETE of a missing exact key, or an administrator heal with +`remove` enabled, can authorize historical residue removal only when the +bucket is authoritatively never-versioned and has no Object Lock. Every +configured drive must be inspected, the current object must be absent, and all +residue must match the guarded legacy layout. Versioned/suspended buckets, +retained versions, conditions, unknown entries, symlinks, and prefix children +are not covered by that authorization. The same fresh intent can certify the +absence of an unprotected new prepared null-version transaction when every +configured drive has no current metadata. This is a new authorized decision; +it does not infer an old commit from missing causal evidence. Mixed or +unrecognized historical layouts remain blocked. + +An isolated `xl.meta.bkp` is not enough to distinguish a committed delete from +an interrupted rollback. Automatic recovery does not guess which outcome was +intended. Preserve a copy of the original volume and use an authoritative +application backup or snapshot for recovery when no transaction decision can be +established. `remove` means explicitly discarding eligible residue; it cannot +restore ambiguous data or bypass versioning and Object Lock protections. + +A missing specific-version DELETE first recovers hidden transactions and then +reads the current version again. It must not acknowledge an unresolved hidden +version as an idempotent no-op. + +## Upgrade and failure handling + +Upgrade all participating nodes before relying on the new protocol. Recovery +uses one authenticated additive RPC; unsupported peers cannot silently accept +new flags through the legacy delete endpoint. An online older participant +causes the modern operation to fail before decision publication. Modern nodes +also reject legacy forward transactional user-bucket deletes. A cluster still +controlled by an older majority retains the older behavior. Do not downgrade +while recovery records remain; old binaries cannot interpret the new protocol. +An upgraded initiator reports an unsupported participant as retryable S3 +`ServiceUnavailable` (503), including when the delete aggregates pool results. + +Keep unavailable drives and transaction directories intact. Bring the full +set online, run a scoped heal in dry-run mode, inspect its recovery result, +and then run the same scoped heal without dry-run. Use `remove` for ambiguous +historical residue only after verifying that the exact objects are intended +to be deleted and the bucket meets the restrictions above. Do not recursively +remove transaction directories by hand to make a bucket appear empty. + +Recovery reports discovered, committed, rolled-back, reclaimed historical, +blocked, pending, and failed outcomes. Pending and failed counts can represent +an incomplete set or a failed discovery/inspection scope rather than a known +number of transactions. Bucket and prefix results +include a structured `deleteRecovery` status: `pending` means unavailable +participants or unfinished work, `blocked` means insufficient safe evidence, +and `failed` means corrupt transaction data. A local corruption or pending +transaction does not prevent other buckets or ordinary object healing from +running. The pass still reports incomplete recovery and does not publish a +completed recovery checkpoint. Cancellation, lifecycle/incarnation changes, +movement fences, invalid paths, and other set-level errors retain their +termination semantics. Exact-object recovery still returns a retryable quorum +error for unavailable participants; blocked evidence requires operator +investigation. A successful +S3 delete response does not imply immediate physical cleanup while snapshots +or returning participants still retain necessary proof. + +If ordinary set healing overlaps unresolved transactions, its resume state +durably records that coverage needs refreshing. Pending retries preserve the +completed object-scan progress. Once recovery resolves, the set is scanned +again before reporting completion, including when a background rollback already +restored and finalized formerly hidden objects. This additional scan is required +on that recovery transition; it is not a periodic full-set heal. + +The upgrade CI retains its pinned rc.5 source and independently tests stable +1.0.1. The legacy-reservation case uses metadata and shards written by the +official 1.0.1 binary and injects an empty reservation while it is stopped; +this is a deterministic interruption fixture, not a hardware power-loss test. +Mixed-version stages verify that modern DELETE fails safely while older +participants are online. They do not promise continuous DELETE availability or +arbitrary downgrade while new recovery records exist. + +## Settlement deadline and durability + +`RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS` controls the settlement RPC deadline, +including full shard verification during rollback. Key and bucket lifecycle +ownership stay attached to late local filesystem operations after cancellation +or timeout. The default is 900 seconds; +valid values are 30 through 86400 seconds. Invalid values fail explicitly. +Increase it when intact large objects need more verification time. It does not +change normal object read or delete-apply deadlines. A timeout can conceal a +completed mutation, so rerun inspection and recovery rather than guessing the +outcome from the RPC error. + +The records and restores follow the configured durability policy. Strict mode +syncs the required metadata and directory changes. Process SIGKILL testing +checks restart behavior; it does not establish power-loss behavior on every +filesystem or storage device. Mode `none` intentionally offers weaker sync +guarantees. Do not treat a throughput comparison as a durability substitute. + +## Reproduce the Docker regression suite + +Build the candidate with the test-only barriers enabled: + +```sh +cargo build --locked --release -p rustfs --bin rustfs --no-default-features --features e2e-test-hooks +python3 scripts/test/delete_recovery_docker.py --help +``` + +Pass a native Linux binary with `--binary`, a Linux image containing Python and +curl with `--image`, and a unique Docker namespace. The optional +`--baseline-binary` enables the mixed-version case. The harness uses synthetic +S3 objects, localhost ports, labeled isolated volumes, and a non-root server. +It stops only its own containers and keeps JSON results and logs in the chosen +artifact directory. Production builds must not enable `e2e-test-hooks`. + +Run the preactivation/prepare/apply/decision process-stop cases, interrupted restore, +returning and untouched peers, both generation orderings, version/retention +and prefix protection, administrator dry-run/removal, response-loss, +permission/corruption, and mixed-version cases. Preserve the source identity +of both binaries with the results. A passing process-stop matrix does not +cover hardware power cuts, Windows runtime behavior, or DELETE throughput. diff --git a/docs/operations/durable-quota-reservation-v2.md b/docs/operations/durable-quota-reservation-v2.md index 82200dcd3..250cc84ff 100644 --- a/docs/operations/durable-quota-reservation-v2.md +++ b/docs/operations/durable-quota-reservation-v2.md @@ -1,5 +1,7 @@ # Durable quota reservation protocol v2 +**Use this when:** operating durable hard quotas, upgrading quota-enabled clusters, or diagnosing quota writes during a node outage. + Protocol v2 is an opt-in hard-quota protocol for distributed deployments. It keeps the quota limit authoritative in one allocator ledger while moving request reservations into sixteen independently locked shard ledgers. @@ -66,6 +68,43 @@ The default `BucketQuota::new` remains protocol v1. This keeps existing deployments compatible until an operator has completed the rollout and chosen the sharded protocol explicitly. +## Availability and fence protocol upgrades + +Both reservation protocols use the same write-availability rules. Setting a +durable quota still requires the existing whole-fleet capability proof. Once +configured, a quota write does not require that proof to remain live. It needs +the namespace locks, authoritative bucket configuration, quota ledger, and +object mutation fences to satisfy their respective quorums. A failed node +therefore does not stop quota writes when those quorums remain available. + +Object fences use the target erasure set's write quorum. Ledger, allocator, +and shard fencing retains the existing per-pool set checks. An unavailable +set that owns required quota state can still block a write even if its object +data set is healthy; a majority of nodes across the whole cluster is not a +substitute for these quorums. + +Participating disks must also agree with the coordinator's configured +pool/set/disk topology. The dedicated quota RPCs authenticate that topology +identity and reject a different layout, including a different pool order. +An offline node does not change the configured topology, so this check does +not require every node to respond. + +Quota mutations use the dedicated `AcquireQuotaMutationFence`, +`ReleaseQuotaMutationFence`, and `RenameDataWithQuotaFence` RPCs. A fenced +rename requires a valid token for that disk process and object. A restart +discards the token registry, and revocation waits for existing claims to +drain before returning success. Unsupported peers fail closed; clients never +fall back to ordinary rename or lease RPCs for a quota mutation. + +Upgrade every node before relying on degraded quota writes. During a rolling +upgrade, new coordinators cannot count old peers toward quota fence quorums, +so their quota writes can temporarily fail until enough upgraded disks are +available in every required set. Older coordinators retain their whole-fleet +proof requirement. The same limitation applies when rolling back: avoid +routing quota writes to a newer coordinator that lacks enough compatible +peers. The quota and ledger formats are unchanged, including the zero-quota +encoding that makes pre-reservation coordinators reject positive growth. + ## Performance boundary Normal v2 requests take a shared allocator read lock and one shard write lock; diff --git a/docs/operations/encrypted-etags.md b/docs/operations/encrypted-etags.md index 463ea42f6..854ee3740 100644 --- a/docs/operations/encrypted-etags.md +++ b/docs/operations/encrypted-etags.md @@ -24,6 +24,44 @@ Clients that require an ETag for these older or multipart objects must issue an authorized HeadObject request, including the SSE-C key when required. GET, HEAD, and conditional requests continue to use the object's stored ETag. +## Replication + +Bucket and site replication forward the stored ETag on `x-rustfs-source-etag` +(with its `x-minio-source-*` twin). A target that adopts it (RustFS, MinIO) +reports the fingerprint for the replica; a target that computes its own ETag +(AWS S3 and most compatible stores) reports the MD5 of the plaintext it +received, which never equals the fingerprint. The post-PUT replica ETag audit +therefore skips encrypted sources. Managed-SSE sources forward no checksum on +the replication PUT either, so a target that transforms their payload is not +detected. See +`docs/operations/replication-outbound-transport.md`. + +The same mismatch affects finding an existing replica on a target that mints +its own version ids (AWS S3, Wasabi), which a heal, MRF retry or +existing-object resync does before writing again: + +- A replica recorded in the source's target-version ledger is addressed by + that id, on a 404 and on an AWS-style 400 for the source id alike. For an + encrypted source the replica counts when its size matches the bytes + replication sent (plaintext for SSE-S3 / SSE-KMS, ciphertext for SSE-C) and + it was written no earlier than the source version. +- Without a ledger entry (for example a PUT whose response was lost), the + replica is located through `ListObjectVersions` by the same size and + write-time rule, after removing versions that other generations of the key + already claim. Only a single remaining match is adopted and recorded in the + ledger. Several matches are never resolved by picking one: convergence writes + the replica again, and a version purge is refused with the usual backoff. + The `null` version of an unversioned target is never matched this way, and + a target that refuses the listing keeps the earlier current-version check, + which writes the replica again. +- Plaintext sources keep exact ETag identity. A target that adopts the source + ETag still matches encrypted sources by ETag first. + +Two same-size generations of a key whose own ledgers were both lost can still +be confused, and a version purge with no ledger and no shape match reports the +replica as already gone. Delete a leftover replica on the target by hand if a +`replication-check` VersionFidelity report shows one. + ## Existing data and migration The upgrade does not rewrite existing object metadata or erase fingerprints diff --git a/docs/operations/object-lock-contention.md b/docs/operations/object-lock-contention.md index a8a10e2df..8ee8e131e 100644 --- a/docs/operations/object-lock-contention.md +++ b/docs/operations/object-lock-contention.md @@ -2,6 +2,14 @@ **Use this when:** an object GET returns `ServiceUnavailable` near the namespace lock acquisition deadline while PUT or Heal is active on the same key. +## Client-visible acquisition timeouts + +The canonical `ApiError` conversion in `rustfs/src/error.rs` retains HTTP 503 and the S3 `ServiceUnavailable` code for typed lock acquisition timeouts, and adds `Retry-After: 1`. The header is a minimum retry delay, not a receipt that the competing owner has released its lock. Use bounded exponential backoff with jitter and keep the client's overall request budget. + +S3 data-plane catalog publication-lock timeouts use the same conversion, typed lock-failure counter, and generic public message; catalog resource and peer details remain in the internal error source. Catalog failures represented only as strings do not inherit lock retry advice. + +This advice does not change the lock acquisition budget or shorten a competing writer's commit tail. It is not added to unrelated disk timeouts, quorum failures, or errors that merely contain timeout text. A 503 near five seconds is insufficient to attribute the request to a namespace lock: disk metadata operations also have a five-second default budget. Correlate the typed lock-failure counters and acquisition events below before changing contention policy. + ## Capture one acquisition and its competing owners Enable the existing diagnostic switch before starting the process and add the diagnostic target to the deployment's log filter: diff --git a/docs/operations/oidc-console-integration.md b/docs/operations/oidc-console-integration.md index 3206b7800..3aa3b7637 100644 --- a/docs/operations/oidc-console-integration.md +++ b/docs/operations/oidc-console-integration.md @@ -1,7 +1,7 @@ # OIDC Console integration **Use this when:** connecting RustFS Console login to an OpenID Connect provider (Keycloak, Authing, or any standards-compliant IdP), or debugging an OIDC redirect, token, or policy-mapping failure. -**Source of truth:** `crates/config/src/constants/oidc.rs` (provider keys and `RUSTFS_IDENTITY_OPENID_*`), `crates/iam/src/oidc.rs` (discovery, PKCE, token validation, per-provider env suffixes), `rustfs/src/admin/handlers/oidc.rs` (authorize/callback handlers), `crates/config/src/constants/app.rs` (`ENV_RUSTFS_BROWSER_REDIRECT_URL`), `crates/utils/src/egress.rs` (`ENV_OUTBOUND_ALLOW_ORIGINS`), `crates/policy/src/policy/policy.rs` (built-in policies). +**Source of truth:** `crates/config/src/constants/oidc.rs` (provider keys and `RUSTFS_IDENTITY_OPENID_*`), `crates/iam/src/oidc/config.rs` (per-provider env suffixes), `crates/iam/src/oidc/provider.rs` (discovery and JWKS), `crates/iam/src/oidc/runtime.rs` (PKCE and token validation), `rustfs/src/admin/handlers/oidc.rs` (authorize/callback handlers), `crates/config/src/constants/app.rs` (`ENV_RUSTFS_BROWSER_REDIRECT_URL`), `crates/utils/src/egress.rs` (`ENV_OUTBOUND_ALLOW_ORIGINS`), `crates/policy/src/policy/policy.rs` (built-in policies). The RustFS side is vendor-neutral and is described once; what RustFS requires from any provider is tabulated in [oidc-provider-requirements.md](oidc-provider-requirements.md). The [Keycloak](#keycloak) and [Authing](#authing) sections contain only IdP-side steps and vendor caveats. Examples use provider id `default` and public origin `https://rustfs.example.com`. @@ -16,7 +16,7 @@ Login flow: 3. The IdP redirects back to `/rustfs/admin/v3/oidc/callback/?code=...&state=...`. 4. RustFS exchanges the code at the token endpoint, sending `client_id` and `client_secret` in the request body (`client_secret_post`) together with the PKCE verifier. 5. RustFS validates the ID token signature (JWKS), issuer, audience, expiry, and nonce. -6. RustFS maps claim values to policy names and issues one-hour STS credentials to the Console. +6. RustFS maps claim values to policy names and issues STS credentials to the Console with the lifetime set by `RUSTFS_CONSOLE_OIDC_SESSION_DURATION` (default 3600 seconds, one hour). In-flight `state` and PKCE verifiers are node-local: the authorize and callback requests must reach the same RustFS node. @@ -47,8 +47,9 @@ Process-level settings (environment only, never suffixed per provider): | --- | --- | | `RUSTFS_BROWSER_REDIRECT_URL` | Public browser origin used for callback generation, Console success redirects, and logout fallback. | | `RUSTFS_OUTBOUND_ALLOW_ORIGINS` | Exact `scheme://host[:port]` origins RustFS may contact for discovery, JWKS, and token requests when the IdP resolves to a private, loopback, or container-network address. See [outbound-connection-policy.md](outbound-connection-policy.md). | +| `RUSTFS_CONSOLE_OIDC_SESSION_DURATION` | Credential lifetime in seconds for Console credentials issued by this callback, for every provider. Default `3600`. Values that are missing, not a positive integer, or zero keep the default; values above `604800` (7 days) are clamped to `604800` with a warning. The same value also sets how long the Console logout token stays valid, so RP-initiated logout keeps working for the whole session. Note that `RUSTFS_CONSOLE_AUTH_TIMEOUT` does not control this: it is the Console HTTP request timeout. | -Named providers: to use provider id ``, suffix every provider env var with `_` (for example `RUSTFS_IDENTITY_OPENID_CLIENT_ID_keycloak`) and register the callback `/rustfs/admin/v3/oidc/callback/`. Suffix scanning is `parse_single_provider` in `crates/iam/src/oidc.rs`. +Named providers: to use provider id ``, suffix every provider env var with `_` (for example `RUSTFS_IDENTITY_OPENID_CLIENT_ID_keycloak`) and register the callback `/rustfs/admin/v3/oidc/callback/`. Suffix scanning is `parse_single_provider` in `crates/iam/src/oidc/config.rs`. Restart RustFS after changing any of these settings. @@ -137,6 +138,7 @@ curl -fsS "/.well-known/openid-configuration" | jq '{issuer, authorizati | Login succeeds, access denied | no claim value matches a policy name | Emit `groups` or `roles` as a flat array equal to policy names; check for `/` prefixes. | | Console redirects to an internal host | `RUSTFS_BROWSER_REDIRECT_URL` unset or proxy headers wrong | Set `RUSTFS_BROWSER_REDIRECT_URL` to the public origin. | | Invalid or expired OIDC state | callback reached a different node | Configure load-balancer session affinity for authorize and callback. | +| Console session expires after about an hour | callback issues credentials with the default lifetime | Set `RUSTFS_CONSOLE_OIDC_SESSION_DURATION` to the desired lifetime in seconds (max `604800`, 7 days) and restart RustFS. `RUSTFS_CONSOLE_AUTH_TIMEOUT` is a request timeout and does not extend the session. | ## Production checklist diff --git a/docs/operations/oidc-provider-requirements.md b/docs/operations/oidc-provider-requirements.md index 4bc4c6c14..d2dc487b3 100644 --- a/docs/operations/oidc-provider-requirements.md +++ b/docs/operations/oidc-provider-requirements.md @@ -1,7 +1,7 @@ # OIDC provider requirements **Use this when:** evaluating whether an identity provider (or a vendor's "OIDC-like" OAuth product) can back RustFS console SSO, or debugging why a provider fails discovery, login, or policy mapping. -**Source of truth:** `crates/iam/src/oidc.rs` (`OidcProviderConfig`, `discover_provider`, `trusted_aud`, `map_claims_to_policies`); `crates/config/src/constants/oidc.rs` (`OIDC_DEFAULT_*`, `ENV_IDENTITY_OPENID_*`); `rustfs/src/admin/handlers/oidc.rs` (`derive_callback_uri_with_provider_config`, `browser_redirect_url`). +**Source of truth:** `crates/iam/src/oidc/config.rs` (`OidcProviderConfig`), `crates/iam/src/oidc/provider.rs` (`discover_provider`), `crates/iam/src/oidc/runtime.rs` (`trusted_aud`, `map_claims_to_policies`); `crates/config/src/constants/oidc.rs` (`OIDC_DEFAULT_*`, `ENV_IDENTITY_OPENID_*`); `rustfs/src/admin/handlers/oidc.rs` (`derive_callback_uri_with_provider_config`, `browser_redirect_url`). RustFS is a standard OpenID Connect relying party using the authorization-code flow with PKCE. It reads every claim it uses from the ID token; it never calls the UserInfo endpoint. Provider-side setup steps for Keycloak and Authing, and the RustFS-side configuration keys, are in [oidc-console-integration.md](oidc-console-integration.md). @@ -11,16 +11,16 @@ The table below describes Console login. For endpoint-free STS issuers, use the | # | Requirement | Details | Code anchor | | --- | --- | --- | --- | -| 1 | Discovery document | `RUSTFS_IDENTITY_OPENID_CONFIG_URL` names the provider (issuer base or full discovery URL); RustFS fetches `{issuer}/.well-known/openid-configuration` and needs `issuer`, `authorization_endpoint`, `token_endpoint`, `jwks_uri`, and the standard `*_supported` arrays. When `RUSTFS_IDENTITY_OPENID_ISSUER` is set, the document's `issuer` must equal it exactly; otherwise RustFS tries the issuer candidates derived from the config URL. | `crates/iam/src/oidc.rs` `discover_provider`, `discover_provider_from_config_url` | -| 2 | Outbound reachability | Discovery, JWKS, and token requests go through the shared egress policy. A provider on a private or loopback address needs its exact origin in `RUSTFS_OUTBOUND_ALLOW_ORIGINS`; otherwise startup logs `OIDC provider discovery blocked by outbound policy`. | `crates/iam/src/oidc.rs` `OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY`; [Outbound connection policy](outbound-connection-policy.md) | -| 3 | Signed ID token, verifiable via JWKS | The ID token signature is verified against `jwks_uri`; the key set is refreshed after `OIDC_JWKS_REFRESH_INTERVAL` and once more on a verification failure. A token response without `id_token` fails login — an OAuth-only access token is not sufficient. | `crates/iam/src/oidc.rs` `OIDC_JWKS_REFRESH_INTERVAL`, the `no id_token in token response` error | -| 4 | Authorization-code flow with PKCE (S256) | The authorization request carries `response_type=code`, `scope`, `redirect_uri`, `state`, `nonce`, `code_challenge`, `code_challenge_method=S256`; the token request carries the `code_verifier`. Providers that ignore or reject PKCE, `state`, or `nonce` are not supported. | `crates/iam/src/oidc.rs` `PkceCodeChallenge::new_random_sha256`; `crates/iam/src/oidc_state.rs` `pkce_verifier` | -| 5 | Scopes | Default `openid,profile,email` (`OIDC_DEFAULT_SCOPES`); override with `RUSTFS_IDENTITY_OPENID_SCOPES` (comma-separated). The provider must accept every configured scope. | `crates/config/src/constants/oidc.rs` `OIDC_DEFAULT_SCOPES`; `crates/iam/src/oidc.rs` `OidcProviderConfig::scopes` | -| 6 | Callback returns `state` | The callback must carry both `code` and the original `state`; `state` locates the in-flight session that holds the nonce and PKCE verifier. A callback with only `code` fails. | `rustfs/src/admin/handlers/oidc.rs` `OIDC_CALLBACK_SUFFIX`; `crates/iam/src/oidc_state.rs` | -| 7 | ID token claims | `iss`, `aud`, `exp`, and `nonce` are verified; `sub` identifies the user. `aud` must contain the client id or one of `RUSTFS_IDENTITY_OPENID_OTHER_AUDIENCES`. | `crates/iam/src/oidc.rs` `trusted_aud`, `OidcClaims` | -| 8 | Authorization claims in the ID token | Policy mapping reads the claim named by `RUSTFS_IDENTITY_OPENID_CLAIM_NAME` (default `groups`, `OIDC_DEFAULT_CLAIM_NAME`) together with `RUSTFS_IDENTITY_OPENID_GROUPS_CLAIM` and the optional `RUSTFS_IDENTITY_OPENID_ROLES_CLAIM`, as a string or array of strings, optionally prefixed by `RUSTFS_IDENTITY_OPENID_CLAIM_PREFIX`. Values must equal RustFS policy names (for example `consoleAdmin`, `readwrite`, `readonly`) made only of `[A-Za-z0-9_:.-]`. Groups-claim and roles-claim values with no policy of that name, such as the Active Directory group `DOMAIN\Domain Users` or Keycloak's `offline_access` role, are ignored once storage confirms that no such policy exists. Login is rejected when a policy lookup fails, when no value resolves, when a value names an existing policy whose name is outside that character set (dropping it could remove a Deny), or when a name from `RUSTFS_IDENTITY_OPENID_ROLE_POLICY`, a separate `RUSTFS_IDENTITY_OPENID_CLAIM_NAME` claim, or an explicit IAM policy mapping does not exist. Email and username come from `RUSTFS_IDENTITY_OPENID_EMAIL_CLAIM` (default `email`) and `RUSTFS_IDENTITY_OPENID_USERNAME_CLAIM` (default `preferred_username`). A provider that returns only a user id can authenticate but cannot express RustFS authorization. | `crates/iam/src/oidc.rs` `map_claims_to_policies`, `extract_canonical_group_values`; `OidcSys::group_claim_policy_names`; `rustfs/src/admin/service/federated_identity.rs` `select_oidc_policies` | +| 1 | Discovery document | `RUSTFS_IDENTITY_OPENID_CONFIG_URL` names the provider (issuer base or full discovery URL); RustFS fetches `{issuer}/.well-known/openid-configuration` and needs `issuer`, `authorization_endpoint`, `token_endpoint`, `jwks_uri`, and the standard `*_supported` arrays. When `RUSTFS_IDENTITY_OPENID_ISSUER` is set, the document's `issuer` must equal it exactly; otherwise RustFS tries the issuer candidates derived from the config URL. | `crates/iam/src/oidc/provider.rs` `discover_provider`, `discover_provider_from_config_url` | +| 2 | Outbound reachability | Discovery, JWKS, and token requests go through the shared egress policy. A provider on a private or loopback address needs its exact origin in `RUSTFS_OUTBOUND_ALLOW_ORIGINS`; otherwise startup logs `OIDC provider discovery blocked by outbound policy`. | `crates/iam/src/oidc/provider.rs` `OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY`; [Outbound connection policy](outbound-connection-policy.md) | +| 3 | Signed ID token, verifiable via JWKS | The ID token signature is verified against `jwks_uri`; the key set is refreshed after `OIDC_JWKS_REFRESH_INTERVAL` and once more on a verification failure. A token response without `id_token` fails login — an OAuth-only access token is not sufficient. | `crates/iam/src/oidc/provider.rs` `OIDC_JWKS_REFRESH_INTERVAL`; `crates/iam/src/oidc/runtime.rs` `no id_token in token response` | +| 4 | Authorization-code flow with PKCE (S256) | The authorization request carries `response_type=code`, `scope`, `redirect_uri`, `state`, `nonce`, `code_challenge`, `code_challenge_method=S256`; the token request carries the `code_verifier`. Providers that ignore or reject PKCE, `state`, or `nonce` are not supported. | `crates/iam/src/oidc/runtime.rs` `PkceCodeChallenge::new_random_sha256`; `crates/iam/src/oidc/state.rs` `pkce_verifier` | +| 5 | Scopes | Default `openid,profile,email` (`OIDC_DEFAULT_SCOPES`); override with `RUSTFS_IDENTITY_OPENID_SCOPES` (comma-separated). The provider must accept every configured scope. | `crates/config/src/constants/oidc.rs` `OIDC_DEFAULT_SCOPES`; `crates/iam/src/oidc/config.rs` `OidcProviderConfig::scopes` | +| 6 | Callback returns `state` | The callback must carry both `code` and the original `state`; `state` locates the in-flight session that holds the nonce and PKCE verifier. A callback with only `code` fails. | `rustfs/src/admin/handlers/oidc.rs` `OIDC_CALLBACK_SUFFIX`; `crates/iam/src/oidc/state.rs` | +| 7 | ID token claims | `iss`, `aud`, `exp`, and `nonce` are verified; `sub` identifies the user. `aud` must contain the client id or one of `RUSTFS_IDENTITY_OPENID_OTHER_AUDIENCES`. | `crates/iam/src/oidc/runtime.rs` `trusted_aud`, `OidcClaims` | +| 8 | Authorization claims in the ID token | Policy mapping reads the claim named by `RUSTFS_IDENTITY_OPENID_CLAIM_NAME` (default `groups`, `OIDC_DEFAULT_CLAIM_NAME`) together with `RUSTFS_IDENTITY_OPENID_GROUPS_CLAIM` and the optional `RUSTFS_IDENTITY_OPENID_ROLES_CLAIM`, as a string or array of strings, optionally prefixed by `RUSTFS_IDENTITY_OPENID_CLAIM_PREFIX`. Values must equal RustFS policy names (for example `consoleAdmin`, `readwrite`, `readonly`) made only of `[A-Za-z0-9_:.-]`. Groups-claim and roles-claim values with no policy of that name, such as the Active Directory group `DOMAIN\Domain Users` or Keycloak's `offline_access` role, are ignored once storage confirms that no such policy exists. Login is rejected when a policy lookup fails, when no value resolves, when a value names an existing policy whose name is outside that character set (dropping it could remove a Deny), or when a name from `RUSTFS_IDENTITY_OPENID_ROLE_POLICY`, a separate `RUSTFS_IDENTITY_OPENID_CLAIM_NAME` claim, or an explicit IAM policy mapping does not exist. Email and username come from `RUSTFS_IDENTITY_OPENID_EMAIL_CLAIM` (default `email`) and `RUSTFS_IDENTITY_OPENID_USERNAME_CLAIM` (default `preferred_username`). A provider that returns only a user id can authenticate but cannot express RustFS authorization. | `crates/iam/src/oidc/runtime.rs` `map_claims_to_policies`, `extract_canonical_group_values`, `OidcSys::group_claim_policy_names`; `rustfs/src/admin/service/federated_identity.rs` `select_oidc_policies` | | 9 | Registered redirect URI | The provider must accept the callback `{public-origin}/rustfs/admin/v3/oidc/callback/{provider_id}`. RustFS picks the origin in this order: the provider's `redirect_uri` (`RUSTFS_IDENTITY_OPENID_REDIRECT_URI`), then `RUSTFS_BROWSER_REDIRECT_URL`, then the request's own scheme and host — the last only when `RUSTFS_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC` is enabled. | `rustfs/src/admin/handlers/oidc.rs` `derive_callback_uri_with_provider_config`, `browser_redirect_url` | -| 10 | Logout endpoint (optional) | When discovery advertises `end_session_endpoint`, RustFS builds an RP-initiated logout URL with `id_token_hint`, `client_id`, and `post_logout_redirect_uri`. Without it, logout falls back to the console login page. | `crates/iam/src/oidc.rs` `build_logout_url` (reads `end_session_endpoint` from `ProviderMetadataWithLogout`) | +| 10 | Logout endpoint (optional) | When discovery advertises `end_session_endpoint`, RustFS builds an RP-initiated logout URL with `id_token_hint`, `client_id`, and `post_logout_redirect_uri`. Without it, logout falls back to the console login page. | `crates/iam/src/oidc/runtime.rs` `build_logout_url` (reads `end_session_endpoint` from `ProviderMetadataWithLogout`) | ## STS workload providers diff --git a/docs/operations/outbound-connection-policy.md b/docs/operations/outbound-connection-policy.md index f729a2915..b0a094cbe 100644 --- a/docs/operations/outbound-connection-policy.md +++ b/docs/operations/outbound-connection-policy.md @@ -16,7 +16,7 @@ RustFS validates every operator-configured outbound destination to close a serve | --- | --- | --- | | Event-notification webhooks (`RUSTFS_NOTIFY_WEBHOOK_*`) and audit webhooks (`RUSTFS_AUDIT_WEBHOOK_*`) | Full policy | Proxies disabled and redirects not followed, so the endpoint must be reachable directly (`crates/targets/src/target/webhook.rs`) | | Target configuration validation (startup and admin API) | Full policy | `crates/targets/src/config/common.rs` `validate_outbound_http_url`; `rustfs/src/admin/handlers/target_descriptor.rs` | -| OIDC discovery, JWKS, and token requests | Full policy | A blocked provider logs `OIDC provider discovery blocked by outbound policy` naming the origin to allowlist (`crates/iam/src/oidc.rs`) | +| OIDC discovery, JWKS, and token requests | Full policy | A blocked provider logs `OIDC provider discovery blocked by outbound policy` naming the origin to allowlist (`crates/iam/src/oidc/provider.rs`, `crates/iam/src/oidc/transport.rs`) | | Object Lambda targets | Full policy | `rustfs/src/admin/router.rs` `outbound_policy` | | Bucket replication targets | Literal check, relaxed | Private addresses are always allowed; loopback only with `RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET=true` (`crates/ecstore/src/bucket/remote_s3_client.rs` `validate_remote_endpoint`, shared with on-demand migration sources) | | On-demand migration sources | Literal check, relaxed | Same guard and same escape hatch as replication targets (see below) | diff --git a/docs/operations/pool-metadata-recovery.md b/docs/operations/pool-metadata-recovery.md index d4281530a..cdd48cce6 100644 --- a/docs/operations/pool-metadata-recovery.md +++ b/docs/operations/pool-metadata-recovery.md @@ -51,6 +51,10 @@ An unreadable replica, lost fence, or conditional-write conflict leaves the orig - `rustfs_pool_metadata_blocks_total{reason}` and `rustfs_pool_metadata_recoveries_total` count block and recovery transitions. The original cause and phase remain attached to local typed errors; storage/RPC error numbers and on-disk formats are unchanged. - Node and cluster-write readiness include `pool_meta_write_blocked`. Waiting for the metadata save mutex is bounded to 100 ms and reports `pool_metadata_check_timeout`, not a persistent block. Cluster probes retain their existing cache and overall timeout behavior. Liveness and cluster-read quorum checks are unchanged. The authenticated node-local status and safe error fields are described in [S3 write failure diagnostics](s3-write-failure-diagnostics.md). +Pool metadata CAS explicitly selects `TailDrainedWithOwnerRepair`: it waits for the complete rename result and returns repair responsibility to its existing deployment and transaction owner when write quorum commits but a replica fails or signatures diverge. Cache invalidation and the committed cleanup handoff precede that error. Ordinary system metadata retains quorum-success semantics under `Quorum` and `TailDrained`; draining a tail does not require every disk to succeed. A repair-required completion does not enter rollback cleanup or wait for another temporary-data deletion. The committed bytes are not rolled back. Generic MRF and heal-channel requests cannot supply the deployment and transaction authority needed to repair system metadata; pool CAS retains the uncertain outcome and its existing recovery fence. Ordinary object repair keeps its bucket-incarnation fencing. + +An owning identity writer handles `SystemMetadataRepairRequired` before leaving its transaction: it can retry the exact identity up to three times, waiting one, two, then four seconds for disks to reconnect. Each attempt rereads the committed identity, requires byte-for-byte agreement with its intended payload and a usable ETag, and issues a conditional owner-observed write under the original startup or runtime authority. Quorum readback alone does not complete repair. This lets a transient disk failure during pool expansion converge before startup publishes the new topology. Changed or unreadable identity, a rejected repair CAS, lost authority, cancellation, and exhausted repair attempts retain the transaction-unknown gate. Other write errors and `pool.bin` transactions keep their existing recovery behavior. + If a block persists, inspect the first block and subsequent recovery phase, restore disk/peer readability, and verify every metadata and identity copy before restarting. Do not delete metadata to make readiness green. ## Disk replacement and metadata erasure diff --git a/docs/operations/put-cleanup-isolation.md b/docs/operations/put-cleanup-isolation.md new file mode 100644 index 000000000..a9adaca5b --- /dev/null +++ b/docs/operations/put-cleanup-isolation.md @@ -0,0 +1,195 @@ +# PUT and disk cleanup isolation + +Cleanup can use process-wide admission budgets and a dedicated Tokio runtime. +This is opt-in. It preserves cleanup receipts, eligible-data moves to trash, +PUT temporary workspace cleanup, and periodic physical trash removal. It does +not disable healing or skip cleanup. + +## Configuration + +| Variable | Default | Meaning when isolation is enabled | +| --- | --- | --- | +| `RUSTFS_CLEANUP_ISOLATE_ENABLE` | `false` | Enable admission and execution isolation on this node. | +| `RUSTFS_PUT_RENAME_TAIL_CLEANUP_MAX_PENDING` | `1024` | Maximum admitted PUT ownership groups, including surviving commit and cleanup owners. | +| `RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKERS` | `2` | Maximum active PUT cleanup continuations. Also controls existing deferred queue workers; does not determine OS thread count. | +| `RUSTFS_CLEANUP_ASYNC_THREADS` | up to `2` | Dedicated async worker threads, limited by available parallelism and an explicit CPU mask by default. Independent of logical job concurrency. | +| `RUSTFS_CLEANUP_DISK_MAX_PENDING` | `16 × disk workers` (`64`) | Maximum admitted local disk cleanup operations, including queued and running operations. Shared by local requests, incoming RPC cleanup and periodic GC steps. | +| `RUSTFS_CLEANUP_DISK_WORKERS` | blocking threads (`4`) | Maximum active disk cleanup operations within that budget. | +| `RUSTFS_CLEANUP_GC_WORKERS` | `1` | Maximum periodic disk scans producing GC steps. | +| `RUSTFS_CLEANUP_BLOCKING_THREADS` | `4` | Maximum additional blocking threads on the cleanup runtime. | +| `RUSTFS_CLEANUP_CPUS` | `1-2` on Linux; unbound elsewhere | CPU affinity for cleanup async and blocking threads. Override with a CPU list such as `4-5,8`, or `none` to disable affinity while keeping isolation. | +| `RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ENABLE` | `false` | Use the existing bounded deferred queue before coordinator execution. Isolation also works without this queue. | +| `RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_QUEUE_CAPACITY` | `1024` | Capacity of that existing queue; this is not the total admission budget. | + +Counts must be positive. PUT workers must not exceed PUT pending capacity; +GC workers must not exceed disk workers, which must not exceed disk pending +capacity. Async and blocking thread counts must not exceed 1024. CPU IDs must be supported by the affinity implementation and belong +to the initiating thread's allowed CPU set. Non-Linux hosts reject explicit CPU +masks. Configuration is read once at first use, including disk cleanup or GC; +changes require a restart. Invalid configuration rejects new cleanup admission. +PUT checks configuration before staging data. + +Isolation remains disabled by default. When enabling it on Linux, the default +mask requires logical CPU IDs 1 and 2 to be available inside the process/container +CPU set. On smaller or restricted CPU sets, configure permitted CPU IDs or `none` +before enabling isolation. An unavailable default mask is an error; it does not +silently select different CPUs. Configure each node separately. + +Do not combine isolation with cleanup counterfactuals +`RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP`, +`RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER`, or +`RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP`. These combinations +are rejected because they deliberately suppress cleanup or stop queue progress. + +Do not enable `RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_ENABLE` together with +`RUSTFS_CLEANUP_ISOLATE_ENABLE`. The legacy per-disk lane adds a second queue +inside the isolated coordinator budget, so a slow lane can hold coordinator +capacity while unrelated cleanup waits. This combination is rejected before +PUT admission; either scheduler can still be enabled on its own. + +## Ownership and execution + +PUT attempts admission before staging. Full admission returns `SlowDown`, so +clients should retry with backoff. Admission does not wait: COPY readers and +data-movement callers can already hold locks needed by admitted PUTs. No body +is ingested or tmp data staged by a rejected PUT. + +The request, detached commit owner, and cleanup continuations share one +reservation. An early ACK or cancelled waiter cannot release it while an owner +still needs cleanup. After the existing rename-convergence and guard-release +handoff, deferred workers, queue overflow and direct continuations all acquire +the same coordinator execution semaphore. Queue overflow cannot bypass that +limit. Awaited failure cleanup and no-tail tmp cleanup use the same budget. +Full-tail receipt persistence remains awaited at its existing handoff point; +old-data reclamation remains after the existing guard release. Incomplete +rollback still preserves staging for recovery. + +Each local disk operation has separate pending and active permits. The wrapper +below RPC authentication uses this gate for low-level `delete`, `delete_data_dir` +and old-data cleanup receipt `write_all`. Local PUT fan-out and incoming peer +RPCs therefore share the same node-local disk budget. Receipt classification +uses the internal receipt filename; ordinary writes retain their current path. +RPC schemas and authentication are unchanged. Enable isolation on every node +to isolate peer execution; settings and CPU masks are not transmitted by RPC. + +Coordinator and receiver permits are deliberately distinct: a coordinator can +wait for peer RPCs without occupying the slots those peers need to service disk +cleanup. Full disk admission returns an I/O `WouldBlock` error through the +existing disk error path. A mutation carrying an external publication or +namespace owner also refuses a full active budget instead of waiting while it +holds a lock. Existing cleanup error handling and receipt recovery still apply. + +Ordinary snapshot-lease release stays on the caller and only updates the in-memory +registry. It does not allocate a cleanup task or wait for the cleanup scheduler. +Invalid cleanup configuration also must not strand a reader token: release still +updates the registry, while any deferred reclamation reports the initialization +error and keeps both its intent and data. It does not fall back to unbudgeted I/O. +If the last release triggers deferred old-data +reclamation, that reclamation tries the same pending and active disk permits. +Only this case dispatches an owned task to the cleanup runtime. +Saturation leaves the token released and the cleanup intent/receipt available +for retry; it does not strand a live reader token or bypass the budget. Quota +fence release keeps its original receiver lifecycle and does not acquire disk +execution permits, since it can wait for mutations that already own them. + +An admitted request waits for disk execution capacity in the caller's existing +future. Cancellation or timeout discards this unstarted operation and releases +pending capacity; it cannot later start deleting a reused path. After execution +capacity is acquired, the worker also checks whether its result receiver closed +before its first poll. A job cancelled while waiting for the runtime to schedule +it is discarded when the worker polls it, releasing its permits without starting +I/O. Once the worker starts the operation, the owned job retains its permits and +disk reference until completion. Cancelling or timing out its waiter does not +free disk capacity while that started job still runs. +This also retains the disk's mount ownership during detached execution. Cleanup +waiter timeouts include queue delay and do not by themselves mark a disk faulty; +the existing active disk-health probe retains its timeout policy. + +Periodic trash and stale-tmp scans have a separate scan limit. Each scan acquires +the shared disk budget to open its directory and then for each entry, releasing +it between entries. The steps borrow the scan's iterator and paths and run in +the same owned task, without a spawn/join or path clones per entry. A scan cannot occupy a disk execution slot while waiting +for another scan slot. GC waits for admission without holding object locks; +request admission is bounded without creating waiting producer tasks. A single +trash subtree is still removed as one operation, so a very large subtree can +hold a slot for its entire removal. Queued scans retain the original mount +handle even when the corresponding disk object is dropped. +Missed periodic ticks are skipped so a slow scan does not cause a burst of +catch-up scans. + +Cleanup runs on `rustfs-cleanup` threads. Nested Tokio tasks, Tokio filesystem +work and ordinary `spawn_blocking` use that runtime. Explicit fsync dispatch +also stays on its blocking pool. File/directory group-commit keys separate +cleanup and foreground producers, preventing batches from moving into the +other class's runtime. Fsync durability settings remain unchanged. + +The blocking limit bounds simultaneous blocking execution, not Tokio's internal +queue or kernel I/O. Disk admission bounds selected operations, not bytes, +syscall count or device IOPS. Existing synchronous path probes and disk-full +delete fallbacks can still run on the dedicated async workers, under the disk +operation budget. All classes still share the filesystem, journal, +page cache and devices with foreground work. + +If CPU binding fails at thread startup, new admissions fail and the failure is +logged; already-admitted cleanup can finish. Affinity does not reserve exclusive +cores, move foreground threads elsewhere, change cgroups, or control kernel +workers and interrupt affinity. + +## Scope and recovery + +The limits are per process, not cluster-wide quotas. This is not a durable work +queue. Existing cleanup receipts and tmp/trash recovery retain their role after +process exit. The option does not bound all storage mutations: metadata/version +DELETE transactions, heal/scanner coordination and +multipart orchestration retain their scheduling. Calls from those paths that +reach the selected disk wrapper entries do share the disk budget; internal +LocalDisk operations that bypass those entries do not. + +## Measurement and tuning + +On Linux, `cargo test -p rustfs-ecstore --test cleanup_isolation_test` exercises +the public disk facade, actual thread affinity, receipt/delete execution and +periodic physical trash removal with temporary disks. It does not benchmark +throughput. + +With PUT stage metrics enabled, observe: + +- `rustfs_cleanup_admitted_puts` and `rustfs_cleanup_admission_rejected_total`; +- `rustfs_cleanup_execution_wait_seconds`; +- `rustfs_cleanup_disk_pending`, `rustfs_cleanup_disk_active`, and + `rustfs_cleanup_disk_admission_rejected_total`; +- `rustfs_cleanup_disk_execution_wait_seconds` and `rustfs_cleanup_gc_active`; +- existing deferred queue depth, age, overflow, completion and service time. + +Compare isolation disabled, budget/pool isolation without CPU affinity, and +isolation with affinity. Keep durability, object sizes, overwrite ratio, +concurrency and cluster placement fixed. Measure PUT/GET throughput and p95/p99 +latency, rejections, queue age, receipt/trash growth, per-disk I/O latency, CPU +usage and context switches. Include a long steady-state interval covering +physical GC, and verify backlog drains afterward. + +Start with small worker and blocking-thread limits, then increase only when +cleanup cannot keep up without harming foreground latency. Too few workers or +CPUs can increase rejections and lower PUT throughput. Larger queues do not +increase sustainable device throughput. Affinity helps only when CPU scheduling +contention matters to the measured workload; no throughput gain is guaranteed. + +The defaults protect a node; they are not a claim of optimal production throughput. +The default disk queue permits sixteen service waves, rather than 256 waves with +1024 pending operations and four workers. For illustration, at a uniform 100 ms +per operation, these bounds correspond to roughly 1.6 s versus 25.6 s to drain a +full queue with no new arrivals. Real service times, stalls and disk fan-out vary; +there is no queue latency guarantee. The ordinary disk waiter timeout defaults to +30 s and does not cancel already-admitted work. + +Size pending capacity for measured arrival bursts and local/peer fan-out, not +only for PUT request concurrency. For example, two coordinator jobs each touching +16 local disks can submit 32 disk operations before incoming peer requests and +GC are counted. Sixty-four slots need not absorb every distributed burst. Increase +the explicit limit when measured rejection rates require it, while keeping queue +age comfortably below waiter deadlines. On many-disk nodes, measure whether four +active operations underutilize devices before increasing disk/blocking workers. +The 1024 PUT admission limit includes uploading requests and detached owners; +it is an object-count bound, not a memory or byte budget. The older deferred queue +remains optional and disabled by default; its overflow path cannot bypass the +new coordinator or disk limits. No unbounded foreground fallback is introduced. diff --git a/docs/operations/replication-outbound-transport.md b/docs/operations/replication-outbound-transport.md index 125492735..b99b4bd24 100644 --- a/docs/operations/replication-outbound-transport.md +++ b/docs/operations/replication-outbound-transport.md @@ -9,7 +9,7 @@ - For a single-part object, the checksum the source object was uploaded with, forwarded as its `x-amz-checksum-` header (the value the source verified on upload). A multipart replica is rebuilt through CreateMultipartUpload/UploadPart and carries no object-level checksum header. Managed-SSE objects forward none. - On a PUT that carries Object Lock parameters and no forwarded checksum: `Content-MD5` derived from the source ETag, or an SDK CRC32 checksum when the ETag is not the MD5 of the wire bytes (rustfs#7082). - The source ETag, mtime and version id on `x-rustfs-source-*` headers (with `x-minio-source-*` twins), and the Object Lock mode, retain-until date and legal hold of the source version when present. -- After the PUT, the target's ETag is compared with the source ETag when both are plain single-part MD5s; a mismatch fails the replication instead of reporting a corrupted replica as COMPLETED. +- After the PUT, the target's ETag is compared with the source ETag when both are plain single-part MD5s and the source is unencrypted; a mismatch fails the replication instead of reporting a corrupted replica as COMPLETED. A managed-SSE (SSE-S3 / SSE-KMS) or SSE-C source is not audited this way: its stored ETag is a keyed fingerprint (new writes, see `docs/operations/encrypted-etags.md`) or a legacy plaintext MD5, and the target re-encrypts what it receives, so the two ETags are never comparable. Managed-SSE sources also forward no checksum (above), so a target that transforms their payload is not detected; an SSE-C ciphertext passthrough keeps its HEAD-back evidence audit. ## Target classes and their known requirements @@ -18,7 +18,8 @@ | Rejects or mis-stores `aws-chunked` bodies (SeaweedFS 3.97) | Handled by the plain-payload default above. | Outbound target matrix, `RejectAwsChunked` mode | | Requires `Content-MD5` or `x-amz-checksum-*` on a PutObject with Object Lock parameters (AWS S3, MinIO, Impossible Cloud, most compatible stores) | Satisfied: a locked single PUT carries `Content-MD5` derived from the source ETag (plaintext objects whose ETag is the MD5 of the wire bytes) or an SDK CRC32 checksum (multipart-layout ETags, managed SSE, SSE-C passthrough — this one is an `aws-chunked` trailer, so a target that also rejects that framing cannot take such objects). Releases before this fix (`1.0.0-rc.5`) need `RUSTFS_REPLICATION_STREAMING_CHECKSUMS=true` as a workaround. | Outbound target matrix, `RequireChecksumWithObjectLock` mode | | Stores `x-amz-checksum-*` from a PutObject and returns it on `HEAD ?ChecksumMode=ENABLED` (AWS S3, Wasabi, RustFS) | Satisfied for single-part objects: the replica answers with the source's checksum. Before this fix (`1.0.0-rc.5`) the checksum left the source as `x-amz-meta-` user metadata and no replica carried it (rustfs/backlog#2340). | Outbound target matrix, `Checksummed` shape | -| Mints its own version ids (AWS S3, Wasabi, Impossible Cloud) | Data lands, and later mutations converge through the target-version ledger: the id the target assigned is recorded on the source version (internal key `replication-target-version-`) and version deletes, tag and Object Lock updates address it (tag/retention/legal-hold through the metadata-only APIs, never a re-PUT). A replica written before the ledger existed is located by exact key + ETag, minus the candidates other generations of the key already claim; an ambiguous remainder is refused with a one-hour backoff (`replication_purge_replica_unresolved`). `NoSuchVersion` on a version-addressed DELETE counts as purged. See rustfs/backlog#2340, rustfs/backlog#2085 and `docs/operations/replication-check.md` (VersionFidelity). | `replication-check`, outbound target matrix, `MintOwnVersionIds` mode (also models Wasabi's 404 `NoSuchVersion` on unknown ids) | +| Mints its own version ids (AWS S3, Wasabi, Impossible Cloud) | Data lands, and later mutations converge through the target-version ledger: the id the target assigned is recorded on the source version (internal key `replication-target-version-`) and version deletes, tag and Object Lock updates address it (tag/retention/legal-hold through the metadata-only APIs, never a re-PUT). A replica the ledger does not name (written before the ledger existed, or a PUT whose response was lost) is located by exact key + ETag, minus the candidates other generations of the key already claim; an ambiguous remainder is refused with a one-hour backoff (`replication_purge_replica_unresolved`). An encrypted source whose ETag the target does not report is matched by size and write time instead, and only a unique match is adopted (`docs/operations/encrypted-etags.md`). `NoSuchVersion` on a version-addressed DELETE counts as purged. See rustfs/backlog#2340, rustfs/backlog#2085 and `docs/operations/replication-check.md` (VersionFidelity). | `replication-check`, outbound target matrix, `MintOwnVersionIds` mode (also models Wasabi's 404 `NoSuchVersion` on unknown ids); `matrix_mint_own_version_ids_encrypted_redrive_does_not_duplicate` for SSE-S3 on an ETag-computing target | +| Ignores `x-rustfs-source-etag` / `x-minio-source-etag` and reports the MD5 of the received plaintext as the ETag (AWS S3, generic S3 services) | Plaintext single-part replicas are verified against the source ETag. Encrypted sources skip the comparison (see above). In `1.0.2`, the first release with encrypted-ETag fingerprints, every single-part SSE-S3 object (and SSE-KMS where the target does not echo `aws:kms`) failed with `replica etag mismatch` and was retried indefinitely; the only workaround there is `RUSTFS_REPLICATION_REPLICA_ETAG_VERIFY=false`, which disables the check for every object. RustFS and MinIO targets adopt the source ETag and were not affected. | Outbound target matrix, `ComputesOwnEtags` mode, `SseS3` shape | | Returns an ETag that is not the content MD5 without announcing SSE | Every single-part object fails ETag verification. Set `RUSTFS_REPLICATION_REPLICA_ETAG_VERIFY=false`. | Replication status FAILED with `replica etag mismatch` | ## Environment knobs diff --git a/docs/operations/s3-tables-maintenance-scheduler.md b/docs/operations/s3-tables-maintenance-scheduler.md new file mode 100644 index 000000000..fe2db9cf8 --- /dev/null +++ b/docs/operations/s3-tables-maintenance-scheduler.md @@ -0,0 +1,111 @@ +# Table Maintenance Planning and Periodic Queueing + +**Use this when:** planning table maintenance, enabling automatic job admission, or recovering a stalled queue. + +The server can periodically **queue** maintenance jobs for explicitly selected table buckets. It does not run workers, delete objects, expire snapshots, or compact data automatically. Workers remain an explicit operator action. Both object-backed and durable-strong catalogs use the existing maintenance configuration and job format. + +## Read-only Metadata Maintenance Planning + +Both backings accept authenticated planning requests at: + +```text +POST /iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/metadata +``` + +For example: + +```json +{ + "retain-recent-metadata-files": 2, + "snapshot-expiration": { + "min-snapshots-to-keep": 1, + "max-snapshot-age-ms": 86400000 + }, + "compaction": { + "target-file-size-bytes": 536870912, + "small-file-threshold-bytes": 67108864, + "min-input-files": 2, + "max-rewrite-bytes-per-job": 1073741824 + } +} +``` + +The optional snapshot-expiration and compaction fields request plans, not commits. An empty JSON object requests ordinary metadata planning. A nonzero retention value overrides the effective maintenance configuration for this request; omission or zero uses the effective configured retention. + +In durable-strong mode, successful planning returns HTTP 200 with a completed `DRY_RUN` job and persists its named, current, and latest reports with audit events. Read-only means that table data, metadata pointers, and catalog generations are unchanged; the request still writes maintenance report sidecars and requires the table-scoped `admin:RunTableMaintenance` permission. Inspect the returned plan and any manual-review state before taking further action. + +This direct endpoint rejects `delete=true`, `commit-snapshot-expiration=true`, or `commit-compaction=true` with HTTP 400 in durable-strong mode, without publishing a job or changing table objects. The existing object-backed mutation behavior is unchanged. These restrictions do not change the separate explicit maintenance worker or quarantine APIs. + +A queued, running, or failed job, active quarantine, or inconsistent current/latest aliases or their named report returns HTTP 409 instead of replacing the existing job or discarding its retry budget. Inspect that state before retrying or taking explicit worker/recovery action. A failed publication that saved only a completed named dry-run report leaves harmless history: no alias advertises a running claim, and replanning is safe because there were no table-object effects. Planning revalidates table identity, metadata pointer, configuration, and the prior report before publication; concurrent changes also conflict. It does not introduce automatic workers, snapshot-expiration commits, or compaction commits on durable-strong backing. + +## Enablement + +The standalone server reads these variables at startup; the embedded server does not start this scheduler. + +| Variable | Default | Accepted value | +|---|---|---| +| `RUSTFS_TABLE_MAINTENANCE_SCHEDULER_ENABLED` | `false` | `true` or `false`; invalid values fail startup | +| `RUSTFS_TABLE_MAINTENANCE_SCHEDULER_BUCKETS` | None | Required when enabled: comma-separated, 1–32 distinct valid bucket names | +| `RUSTFS_TABLE_MAINTENANCE_SCHEDULER_INTERVAL_SECONDS` | `300` | Integer from 30 to 86400 | + +Example: + +```sh +export RUSTFS_TABLE_MAINTENANCE_SCHEDULER_ENABLED=true +export RUSTFS_TABLE_MAINTENANCE_SCHEDULER_BUCKETS=analytics-tables +export RUSTFS_TABLE_MAINTENANCE_SCHEDULER_INTERVAL_SECONDS=300 +``` + +The allowlist does not create or enable table buckets. Each bucket must already have an active catalog. The effective bucket default or table override must set `background-enabled=true` and `worker-paused=false`. Deletion remains disabled by default; queueing honors the existing `delete-enabled` setting but never executes the queued operation. + +## Admission And Limits + +- The first pass is delayed by the configured interval plus 0–10% jitter. Each completed pass waits again; missed ticks do not produce catch-up bursts. +- Each process runs one owned pass at a time, visiting at most eight distinct tables and 64 discovery steps. Bucket turns rotate and namespace/table cursors persist between passes. Restart resumes discovery from the beginning; persisted jobs still prevent duplicate admission. +- A 30-second admission deadline stops starting further store operations. An in-flight store call drains without cancellation or overlap. This is **not** a hard wall-clock, memory, or per-table object-I/O limit: existing planning and history inspection costs still depend on the table. +- The scheduler only admits a first job or a new job after a successful previous job and the configured cooldown. It does not reset failed retry budgets, renew abandoned queues, recover expired workers, or clear quarantine. +- Before publishing, the store rechecks table identity, enabled/unpaused configuration, the observed current job, and complete current/latest report publication under its existing locks. A concurrent manual run, rename, or identity change conflicts rather than overwriting that state. + +Use the same interval on enabled nodes. The existing distributed catalog locks coordinate claims; this is not a cluster-wide throughput quota. A busy or corrupt table is reported as a degraded pass and does not remove other buckets from discovery. + +## Inspection And Recovery + +Inspect the existing endpoint: + +```text +GET /iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler +``` + +Use its current job, quarantine status, and recommended actions. An eligible queue can be consumed through the authenticated `POST .../maintenance/worker/run` endpoint. If a job failed, a queue/worker lease expired, or a previous manual invocation left a Disabled/Paused control report, resolve the underlying condition and explicitly invoke `POST .../maintenance/scheduler/run` or the recommended worker/quarantine operation. Automatic admission deliberately leaves these states unchanged. + +A partial current/latest publication requires recovery before periodic admission; repeated ticks do not repair or discard it. A conditional worker replay can repair the narrowly defined terminal-publication case below. Review retained reports for other cases; do not delete catalog sidecars directly. + +## Conditional Worker Invocation + +An operator can bind a worker invocation to the job and attempt observed through the scheduler or job endpoint: + +```text +POST /iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run +``` + +```json +{ + "worker-id": "maintenance-worker-a", + "expected-job-id": "observed-job-id", + "expected-attempt": 0 +} +``` + +Send both expected fields or neither. Missing one, explicit `null`, invalid types, and unknown fields are rejected. Omitting both retains the existing manual worker behavior; an empty JSON object `{}` still uses the default worker ID. Authentication and the table-scoped `admin:RunTableMaintenance` permission are required in either mode. + +The conditional mode consumes an unexpired `QUEUED` job at attempt zero, or retries a `FAILED` job after its persisted `next-retry-after` when its persisted attempt budget allows. It keeps the same job ID, operation, original candidate lists, cleanup watermark, and total attempt budget. The first attempt is allowed even when that budget is zero; zero disables retries. Subsequent failed attempts calculate their delay from the current valid backoff configuration without replacing the saved budget. + +Stale job/attempt, catalog identity or pointer/generation changes, expired worker/queue leases, active quarantine, and changed recorded maintenance policy conflict without claiming another attempt. The recorded policy includes retention, quarantine settings, and configuration source; deletion must still be enabled for a delete job. The job format does not store a complete historical configuration snapshot. This endpoint does not promise detection of changes to fields that were never recorded. + +If the response is lost, repeat the same worker ID, expected job ID, and expected attempt. When that invocation already claimed the current attempt, the endpoint returns its persisted report without rerunning candidates or consuming another attempt. Inspect `job.status`: a returned `RUNNING` or `FAILED` report is not a successful maintenance outcome. To retry a completed failed attempt after its backoff, explicitly use its new observed attempt number. + +All named/current/latest reports must agree before new work starts. If a completed report was persisted but final alias publication was interrupted, the same invocation can reconcile aliases only when the reports prove the same job, attempt, lease, immutable plan, and audit transition. Incomplete claims, conflicting histories, and expired `RUNNING` jobs fail closed for operator inspection; the endpoint does not take over or cancel an abandoned worker. Catalog commits remain protected by the existing publication locks during execution and reconciliation. + +To pause admission for a table, set `worker-paused=true` in its effective maintenance configuration. To stop the process-wide scheduler, disable the startup flag and restart. Shutdown stops new admissions and waits for owned queueing work; if completion cannot be confirmed within the normal shutdown grace, it retains unclean-shutdown markers and logs `drain_unconfirmed`. A deadline is not proof that a mutation was cancelled. + +This scheduler does not claim automatic worker retry, cross-node worker takeover, bounded history retention, or a fully managed maintenance service. diff --git a/docs/operations/scanner-runtime-controls.md b/docs/operations/scanner-runtime-controls.md index 2f497cffe..65a98abce 100644 --- a/docs/operations/scanner-runtime-controls.md +++ b/docs/operations/scanner-runtime-controls.md @@ -43,7 +43,8 @@ Bitrot cycle resolution differs because the canonical persistent key belongs to | `scanner.idle_mode` | `RUSTFS_SCANNER_IDLE_MODE` | boolean | `true` (`DEFAULT_SCANNER_IDLE_MODE`) | Enables preset scanner sleeps and foreground-read backoff. `false` disables those sleeps; per-disk foreground-priority admission remains active. | | `scanner.cache_save_timeout` | `RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS` | seconds | `14` (`DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS`) | Timeout for saving scanner cache; runtime enforces a minimum of `1` and keeps the default persistence budget within the distributed publication lease. | | `scanner.max_concurrent_set_scans` | `RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS` | count | `4` (`DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS`) | Caps concurrent set-level scanner tasks. `0` keeps topology-derived concurrency. | -| `scanner.max_concurrent_disk_scans` | `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` | count | `4` (`DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS`) | Caps concurrent disk bucket walks per set. `0` keeps disk-count-derived concurrency. | +| `scanner.max_concurrent_disk_scans` | `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` | count | `4` (`DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS`) | Caps concurrent disk bucket walks per set. `0` keeps walker-count-derived concurrency. | +| `scanner.walkers_per_disk` | `RUSTFS_SCANNER_WALKERS_PER_DISK` | count | `1` (`DEFAULT_SCANNER_WALKERS_PER_DISK`) | Bucket walkers spawned per local disk of a set. The walkers of one disk share its scanner admission, so a peer still sees one scan session per physical disk; remote disks keep one walker. The resulting walker count is what `scanner.max_concurrent_disk_scans` caps, and the disk count stays the floor. Valid range is `1` through `16` (`MAX_SCANNER_WALKERS_PER_DISK`); `1` keeps today's I/O profile and is the right value for rotational media. The `disk_concurrency_budget` log line reports `walkers_per_disk`, `walkers`, the effective `concurrency_limit`, and its `reason` (`available_walkers`, `configured_cap`, or `count_budget`). | | `scanner.yield_every_n_objects` | `RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS` | objects | `128` (`DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS`) | Controls how often object loops yield to the async runtime. `0` disables this extra yield. | | `scanner.alert_excess_versions` | `RUSTFS_SCANNER_ALERT_EXCESS_VERSIONS` | versions | `100` (`DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS`) | Version count threshold for scanner alerts. | | `scanner.alert_excess_version_size` | `RUSTFS_SCANNER_ALERT_EXCESS_VERSION_SIZE` | bytes | `1099511627776` (`DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE`) | Retained version byte threshold for scanner alerts. | @@ -61,6 +62,16 @@ Speed presets (`crates/config/src/constants/scanner.rs`) set the base sleep mult Use `scanner.delay`, `scanner.max_wait`, and `scanner.cycle` when the preset is close but one axis needs a precise override. With `idle_mode=true`, directory-level sleep is `1ms x factor` and object-level sleep is `time spent on the object x factor`, both capped at `max_wait`; a foreground-read floor of `FOREGROUND_READ_BACKOFF_PER_REQUEST_MS` (10ms) per concurrent GetObject/streaming read, capped at `FOREGROUND_READ_BACKOFF_MAX_MS` (250ms), is applied on top and can exceed the preset's `max_wait` (`crates/scanner/src/sleeper.rs`). The per-disk foreground-priority gate is independent of `idle_mode` and remains active when sleeps are disabled. +### Walkers per disk + +The scanner walks the buckets of an erasure set with one walker per local disk by default, so a single-disk deployment walks one bucket at a time no matter how `scanner.max_concurrent_disk_scans` is set. `scanner.walkers_per_disk` (`RUSTFS_SCANNER_WALKERS_PER_DISK`) multiplies each local disk into that many walkers. + +- Keep the default `1` on rotational media: several walkers on one spindle compete for seeks and slow every walk down. +- On flash, `2` to `4` is the useful range. Raise it when the scanner status shows cycles ending on the cycle budget (`metrics.pacing_pressure.last_cycle_budget_limited`) or when many buckets are never reached within a cycle, and the disk has idle I/O capacity. +- The walkers of one disk share that disk's scanner admission, so a peer still sees one scan session per physical disk; remote disks keep one walker. `scanner.max_concurrent_disk_scans` still caps the resulting walker count, and the disk count stays the floor. +- Confirm the setting took effect from `runtime_config.walkers_per_disk.value` in the admin scanner status and from the `disk_concurrency_budget` log line, which reports `walkers_per_disk`, `walkers`, the effective `concurrency_limit`, and the `reason` it was chosen. +- Helm: `config.rustfs.scanner.walkers_per_disk`; Docker or systemd: the environment variable. The key is additive, so an older binary ignores it. + ### Environment-only scanner knobs These have no persistent key and are read from the environment only. @@ -96,6 +107,10 @@ When a finite deadline expires, RustFS cancels cooperative scanner work and wait An explicit `scanner.cycle` or `RUSTFS_SCANNER_CYCLE` is a minimum inter-cycle cadence: dirty-usage notifications do not bypass that configured interval. The default adaptive policy continues to use dirty-usage notifications to wake the scanner between timer-driven cycles. +Under continuous writes, a structurally complete walk can finish as `superseded`. Its durable observation remains separate from quota authority and leaves dirty work pending. Superseded retries may reuse that complete inventory to refresh dirty buckets, even while authority bootstrap is pending; missing or invalid coverage still forces a full walk. Ordinary maintenance timers, Deep scans, active lifecycle/replication and changed maintenance configuration retain full scans. Server info and datausageinfo can display complete observations, while Last Completed Scan advances only after an authoritative completed cycle. A write-free window is therefore not required to display observed counts, but arbitrary continuous writes do not guarantee authoritative completion. Historically dirty buckets remain eligible for rewalk, so narrower retries do not guarantee shorter cycles. + +Superseded convergence retries retain exponential backoff for usage-only work. Active lifecycle or replication rules, and uncertain maintenance inspection, cap that retry interval at the ordinary configured cadence so complete observations cannot defer regular maintenance. An explicit cycle remains the duty-cycle floor. Convergence mode still suppresses dirty/activity shortcuts and does not grant quota authority; deferred backend/proof retries and pause catch-up limits retain their existing scheduling. Cycle runtime, pacing, and disk admission can still delay completed maintenance. Sustained supersedes with active maintenance may therefore run more full scans than usage-only convergence; use the existing pacing controls to manage that background I/O. + ## Single-disk clean-idle scheduling An erasure single-disk deployment using the built-in cycle and bitrot defaults automatically backs off repeated clean idle scans instead of walking the same unchanged namespace every minute. Each successful timer-driven cycle that finds no dirty usage or unresolved maintenance work doubles the next interval. The status endpoint reports the effective interval and multiplier. @@ -126,6 +141,7 @@ The request must be authenticated with an admin identity that has `ServerInfoAdm |---|---| | `runtime_config` | Effective runtime controls and their value sources. | | `cycle_schedule` | Current effective cycle interval and clean-idle backoff state. | +| `cycle_recovery` | This node's cycle-state recovery observation, retry budget, and reason. | | `metrics` | Scanner work, pressure, checkpoint, lifecycle, replication, heal, bitrot, and alert counters. | | `data_movement_pause` | Global-pause policy, current movement reason, operation epoch, start time, duration, and estimated movement work items. | | `pause_backlog` | Replicated durable pause ledger, post-pause catch-up phase, rate window, retry state, thresholds, and active alert reasons. | @@ -139,6 +155,7 @@ runtime_config.delay.value runtime_config.max_wait_seconds.value runtime_config.cycle_interval_seconds.value runtime_config.bitrot_cycle_seconds.value +runtime_config.walkers_per_disk.value cycle_schedule.effective_interval_seconds cycle_schedule.clean_idle_backoff_enabled cycle_schedule.clean_idle_backoff_multiplier @@ -172,6 +189,26 @@ catch_up_estimate.discovered_expiry_items catch_up_estimate.discovered_transition_items ``` +### Periodic checkpoint persistence + +`metrics.scan_checkpoint` is the in-memory resume hint; it does not confirm a durable save. `metrics.scan_checkpoints_persisted` lists the latest confirmed periodic save for each `(pool_index, set_index)` handled by this process, with its bucket, `saved_unix_secs`, and optional checkpoint cursor. It retains one observation per set, is reset on process restart, and does not include the final bucket save. Query the peer executing the walk for remote-worker observations. `metrics.scan_checkpoint_save_failures` counts failed periodic saves, including failures to refresh CAS revisions after a write; failed attempts do not advance the confirmed save observation. + +An ordinary failed save drops that snapshot and leaves the bounded checkpoint receiver active, so the next producer interval tries again. A CAS conflict or an unreadable revision after a successful write cancels the bucket attempt and reloads persisted state through the existing retry path. Leader, cycle, and publication fencing remain mandatory. This preserves future periodic save attempts for a continuing bucket walk. + +Periodic snapshots include the suspended ancestor accounting needed to connect a nested active branch to the bucket root. Completed coverage is sealed independently of raw enumeration bookkeeping. A valid coverage frontier can therefore skip completed subtrees after a fenced leader handoff without counting unfinished work as completed. Emission remains bounded by the existing interval/object thresholds and capacity-one channel; invalid snapshot attempts also consume the producer interval so they cannot repeatedly clone the cache on every object. + +A resumed walk retains its previous durable coverage while initial enumeration has not recopied the covered subtrees. Unfinished out-of-order heal revisits cannot be certified as completed prefix work, including in cancellation snapshots. + +Shared compacted accumulators defer periodic emission until the owning subtree completes; replay inside that compacted subtree remains possible after an interruption. When bucket checkpoint preparation returns `Reset` for existing resume state, it logs `checkpoint_reset` with the validation reason. The `scan_checkpoint_used` counter includes validated coverage adoption by the forward sweep. + +A confirmed periodic save does not prove restart resumability. Adoption still validates bucket/set identity and, for non-empty caches, connected cache scope. Skipping completed subtrees additionally requires a validated coverage frontier; raw enumeration state alone does not prove completed subtree coverage. + +### Cycle recovery on followers + +Cycle recovery status is local to each node. A transient cycle-state read failure can exhaust the short retry budget and enter `paused`; the enabled scanner then retries leadership every five minutes or when recovery is explicitly woken. If the next lock attempt is contended or times out waiting for `leader.lock`, the node records a follower iteration and clears only its unclassified `transient` or `paused` observation. Normal follower retry cadence resumes without restarting the process. + +This clears local retry bookkeeping, not persisted cycle state or recovery markers. Classified recovery states, cleanup-pending resets, and usage-floor failures remain unchanged. Any later leadership claim revalidates durable cycle state before scanning. A healthy follower recovery status does not prove that leader-dispatched scans are progressing; inspect the work counters separately. + ## Usage State Reset The supported break-glass route for rebuilding scanner usage state is `POST /v3/scanner/usage-state/reset` with body `{"mode":"full-rebuild"}`, authenticated as an admin identity holding `ConfigUpdateAdminAction` (route registered in `rustfs/src/admin/route_registration_test.rs`). Use it only after the scanner status shows a usage-floor load failure, a conflicting persisted usage floor, or an operator decision to discard the durable usage baseline and rebuild it from a full scanner pass. @@ -213,7 +250,7 @@ When a running scanner leader already holds `leader.lock`, submit a durable asyn The async form returns HTTP 202 when a new or replayable intent is accepted. Reusing the same `idempotency_key` is safe for retries of the same operator action; a different payload for the same key is rejected as a conflict. Poll `GET /v3/scanner/usage-state/recovery-intents/{intent_id}` until the intent reaches a terminal state. -A successful reset leaves usage state in `bootstrap-pending` only as a fenced rebuild marker. With the scanner enabled, the leader must treat that marker as pending full-rebuild work: clean-idle backoff must not extend the wait, an empty pause-backlog catch-up ledger must not rate-limit the rebuild, and the next admitted scanner cycle must run as a full scan until an authoritative usage snapshot replaces the marker. +A successful reset leaves usage state in `bootstrap-pending` only as a fenced rebuild marker. With the scanner enabled, the leader must treat that marker as pending full-rebuild work: clean-idle backoff must not extend the wait, an empty pause-backlog catch-up ledger must not rate-limit the rebuild, and the next admitted scanner cycle must rebuild full inventory. After a complete observation is durably saved, superseded retries may refresh dirty buckets using its validated coverage; bootstrap remains pending until authoritative publication replaces the marker. ## Cleanup With The Scanner Disabled @@ -475,7 +512,7 @@ Stale-upload cleanup differs in one crash-recovery detail. RustFS's stale multip Treat replacement recovery as verified only after the repair task has completed for the exact replacement instance and an operator has confirmed the target disk contains the expected `xl.meta` and data parts for every relevant object version. A replacement that is not mounted, is unsafe to format, loses its marker, or returns a partial target outcome must be treated as deferred or incomplete. Do not automate destructive replacement actions from an `idle` observation alone. A new node must not infer replacement completion from an old or unavailable peer; regard that information as unknown or degraded until every required peer can report the same replacement instance and verified completion. -`GET /rustfs/admin/v4/heal/replacement-recovery` reports durable automatic replacement records from survivor disks. `local.records[]` entries distinguish `waiting_for_replacement`, `running`, `incomplete`, `unrecoverable`, `cleanup_pending`, `completed`, and `unknown`; `local.definitive=false` or any `unknown` record means the node could not prove a local replacement state. The `cluster` section queries the replacement-recovery peer RPC and sets `cluster.definitive=true` only when the expected peer topology is complete, every peer supports the RPC, every peer snapshot is locally definitive, and all peers report the same replacement records. Old peers, unavailable peers, malformed peer payloads, topology gaps, and generation disagreements are reported as degraded or unknown rather than complete. +`GET /rustfs/admin/v4/heal/replacement-recovery` reports durable automatic replacement records from survivor disks. `local.records[]` entries distinguish `waiting_for_replacement`, `running`, `incomplete`, `unrecoverable`, `cleanup_pending`, `completed`, and `unknown`; `local.definitive=false` or any `unknown` record means the node could not prove a local replacement state. The `cluster` section queries the replacement-recovery peer RPC and sets `cluster.definitive=true` only when the expected peer topology is complete, every peer supports the RPC, every peer snapshot is locally definitive, and copies of each task agree. Different anchors can report different tasks, and a definitive node may report an empty list or a subset of another node's history. Disjoint target generations merge directly. Every pair of generations sharing a target slot must appear together in at least one definitive local snapshot; this preserves retained completion proofs and committed handoff histories while keeping overlapping records found only on separate peers non-definitive. Shared history is not transitive: observing A with B and B with C does not establish that A and C are compatible. A definitive history can include retired or unfinished generations, so consumers must inspect each record's state rather than treating `cluster.definitive` as proof that every generation completed. Old peers, unavailable peers, malformed peer payloads, topology gaps, and task disagreements are reported as degraded or unknown rather than complete. Replacement resume and checkpoint files use an independent on-disk schema. A newer reader rejects a future schema rather than continuing with data it cannot interpret, while an older binary cannot safely enforce the new generation fence. Do not roll a cluster back after a replacement generation has started; complete that recovery with the current-or-newer release, and if it cannot complete, keep that version for diagnosis rather than deleting its durable records or continuing with an older binary. diff --git a/docs/operations/tls-certificate-rotation.md b/docs/operations/tls-certificate-rotation.md new file mode 100644 index 000000000..750f17c6d --- /dev/null +++ b/docs/operations/tls-certificate-rotation.md @@ -0,0 +1,204 @@ +# External certificate issuance and TLS rotation + +Use an independent certificate controller to obtain and renew certificates. +RustFS consumes certificate material; it does not register ACME accounts or +change DNS records. The independent +[rustfs-cert-controller](https://github.com/houseme/rustfs-cert-controller) +owns issuance, renewal scheduling and publication. `plugin-runner` starts its +container; it does not own certificate schedules or CA credentials. Progress +and deployment acceptance are tracked in +[backlog #2806](https://github.com/rustfs/backlog/issues/2806). + +## Deployment boundaries + +For a distributed Kubernetes deployment, prefer public TLS at an +Ingress/Gateway and private TLS/mTLS between the ingress and RustFS. Public +CA certificates cannot cover internal names such as `*.svc.cluster.local`; +DNS-01 can cover names under a controlled public domain even when the service +is private. A listener +that requires mTLS also requires client certificates from browsers and S3 +clients; selecting a different SNI certificate does not change that policy. + +Native server TLS can consume an externally managed `kubernetes.io/tls` +Secret with only `tls.crt` and `tls.key`: + +```yaml +mode: + distributed: + enabled: false + standalone: + enabled: true +ingress: + enabled: false +tls: + enabled: true + existingSecret: rustfs-serving-tls + reloadInterval: 30 +``` + +Create and renew that Secret using cert-manager or an independently managed +publisher. Configure the issuing resource with the actual server DNS/IP SANs +and explicit private-key rotation. The chart does not install cert-manager, +create an ACME account or grant DNS permissions. For distributed native TLS, +the certificate must also cover the configured peer hostnames; explicitly +configure independent outbound trust when a private server CA is used. + +`tls.trustSecret` optionally names a separate Secret with an explicitly chosen +`ca.crt`, mounted as outbound `public.crt`. It does not enable mTLS or add a +public issuing CA to client authentication trust. Do not write a fabricated +`ca.crt` into the Secret managed by an ACME issuer. + +`tls.enabled` and `mtls.enabled` are mutually exclusive. Existing mTLS mode +keeps its separate server/client certificate Secrets and projects their +contents into one read-only directory. The chart enables certificate reload +for either mode and disables TLS key logging. Explicit container environment +variables can override these settings. + +## Material layout and publication + +`RUSTFS_TLS_PATH` contains `rustfs_cert.pem` and `rustfs_key.pem`. SNI +identities can reside in visible domain subdirectories with the same two +filenames. Outbound trust uses `public.crt` and `ca.crt`; inbound mTLS client +trust prefers `client_ca.crt`, falling back to `ca.crt` only at initial load. +The default outbound client identity is `client_cert.pem`/`client_key.pem`. + +Runtime loading validates existing PEM/DER, key pairing, leaf validity and +explicit authentication EKUs. Each named SNI identity must match its domain. +Chain structure validation does not replace issuer/trust-policy verification +by the issuing controller and client PKI verifier. + +Kubernetes mounts must project the entire directory; single-file `subPath` +mounts do not receive Secret updates. RustFS pins one physical `..data` +generation before loading its material. Separate controllers may update the +serving identity and trust independently, so private CA migration still needs +an old-plus-new trust overlap. + +For VMs, publish immutable version directories and atomically change the +directory pointer after syncing the complete candidate and its containing +directories. Keep current/accepted/in-use versions during cleanup. Certbot +file symlinks remain compatible, but independent file replacements do not +provide an atomic directory generation; key mismatch is rejected, and a +publisher should use a complete immutable bundle for coherent publication. + +`RUSTFS_MTLS_CLIENT_CERT`/`RUSTFS_MTLS_CLIENT_KEY` overrides are validated as +one identity. Explicit missing, malformed, mismatched, expired or unsuitable +override material rejects the update; it does not silently select another +identity. Keep default identity files valid when they are present as well. + +## Controller publication API + +For a VM or Docker installation with one default serving certificate, enable +reload and configure `RUSTFS_TLS_PUBLISH_DIR` to a dedicated persistent directory +owned by the RustFS process with mode `0700`. Keep `RUSTFS_TLS_PATH` configured +for the initial serving identity and independent trust/client material. The +publication directory must be separate from the original TLS source. A valid +initial certificate and trusted management credentials must be provisioned +before a controller can connect to the HTTPS API. That bootstrap identity must +cover the names the configured issuer will continue to renew; publication +preserves existing SAN coverage. + +The controller signs requests using the existing S3 SigV4 authentication: + +- `GET /rustfs/admin/v3/tls/certificate-state` requires + `admin:TLSCertificateStatus`. +- `POST /rustfs/admin/v3/tls/certificate` requires + `admin:TLSCertificatePublish` and a connection terminated by RustFS TLS. + +Grant only these two actions to the controller identity. Read permissions, +general configuration permissions and forwarded HTTPS headers do not authorize +private-key publication. Keep API requests and credentials out of access-body +logs. The API does not add MinIO-compatible aliases. + +The `rustfs.tls-publish/v1alpha1` request fixes `resource_id` to `server-default` +and carries an `issuance_id`, the captured `expected_target_revision`, +`certificate_pem` and `private_key_pem`. Read the target token before issuance; +retain it with that immutable candidate. Repeating the same issuance and payload +recovers a lost receipt. A conflict stops that candidate; fetching a new token +and replaying the old candidate would defeat this concurrency protection. +An HTTP 503 or uncertain network result is retryable with the same issuance, +payload and token; HTTP 409 indicates a revision or payload conflict. + +RustFS validates the complete candidate before publishing an immutable leaf +generation and syncing its atomic pointer. It does not copy or overwrite CA +trust or client credentials. This mode rejects projected Kubernetes sources, +multiple SNI identities and configurations that trust the serving leaf as a CA. +Use the existing external-file/cert-manager integration for those deployments. + +The state reports the visible target revision and public leaf fingerprint. A +successful POST receipt proves the publication was synced; a GET during an +uncertain write does not replace that receipt. Retry the identical POST to +confirm acceptance. `observed_certificate_fingerprint` is present only when +this process's serving reload group has the exact current certificate chain; +matching the leaf alone is insufficient. A durable receipt can precede that +observation. Check convergence and a fresh TLS handshake on every relevant +endpoint. One process's response does not prove another node has converged. + +Use a stable local publication volume with one owner. Captured-token CAS and +local process locking do not implement HA leadership, restore ordering or PITR +protection. Do not remove active generation directories; automated generation +cleanup is not part of this interface. + +## Activation and failure handling + +Native `RUSTFS_TLS_RELOAD_ENABLE` defaults to false. Set it to `1` outside the +chart. `RUSTFS_TLS_RELOAD_INTERVAL` defaults to 30 seconds and is clamped to a +minimum of 5 seconds. + +Listeners sharing the same configured TLS directory use one reload group. +The group prepares TCP and enabled HTTP/3 configurations from the same +snapshot before swapping them. A late listener first joins the accepted +configuration, even if the current files have become invalid. TCP, QUIC and +separate processes still have a short activation window; there is no global +atomic certificate switch. +TCP and QUIC derive their protocol settings from one prepared certificate +resolver and client verifier for that generation. Publication state and retries +load each immutable generation once per operation; the next operation validates +its disk contents again. Reload uses the verified serving identity with the +independently pinned trust/client source, without reopening its certificate and +key files. Candidate staging still performs its separate disk validation. + +Unchanged material is skipped. A serving certificate renewal does not advance +the outbound generation unless outbound trust or identity actually changes. +Outbound state is published as one immutable trust/identity/generation +snapshot. Control and bulk RPC cache acquisition checks that generation; +existing in-flight calls retain their channels. + +Reload rejects the disappearance of accepted CA/identity source components, +SNI entries or accepted DNS/IP SAN coverage. Removing or renaming an identity +or trust source requires an explicit configuration change and restart; +ordinary renewal must preserve those requirements. Rejected updates retain +the last accepted in-memory configuration. New handshakes reject certificates +outside their validity interval. Normal renewal does not terminate existing +uploads, downloads or sessions. + +The chart uses TCP listener liveness for TLS deployments so certificate expiry +does not trigger a restart loop. Readiness continues to check the TLS service. +This liveness probe detects listener availability, not complete application +health. Keep ingress traffic admission separate from internal peer routing +and storage quorum. Do not restart every node because a CA is unavailable. + +There is no automatic persisted private-key fallback in Kubernetes. On +restart, invalid explicitly configured TLS material prevents startup. Key +compromise, explicit certificate revocation, session invalidation, per-purpose +profile coordination and deployment-boundary fencing require the separate +operator/control-plane workflow; ordinary reload does not implement them. + +## Verification + +Check a fresh TLS connection to each endpoint and compare the leaf serial, +public certificate fingerprint, SANs and expiry. Include HTTP/2, enabled +HTTP/3, Console and internal RPC endpoints. Do not infer served certificates +from Secret contents or an outbound generation gauge alone. + +Focused local checks: + +```sh +RUSTUP_TOOLCHAIN=stable cargo test -p rustfs-tls-runtime +RUSTUP_TOOLCHAIN=stable cargo test -p rustfs-protos --lib +bash scripts/test_helm_tls_reload.sh +``` + +Production acceptance also requires real Kubernetes projection, CA/DNS +issuance and renewal, outage/expiry, long-transfer continuity, private CA +overlap and distributed convergence tests. These are separate from protocol +and local loader tests. diff --git a/docs/testing/README.md b/docs/testing/README.md index 208919040..416e8d379 100644 --- a/docs/testing/README.md +++ b/docs/testing/README.md @@ -12,12 +12,13 @@ Pick the lowest layer that can prove the change; add a higher-layer test only wh | Unit & crate integration | Per-crate logic and in-process integration tests | `cargo nextest run --all --exclude e2e_test` (or `-p `); `make test` wraps it | Every PR, required (`Test and Lint`, `ci` profile) | | ecstore black-box | Erasure-coded read/write/recovery validation; profiles `quick` / `full` / `destructive` / `fuzz` | `scripts/run_ecstore_validation_suite.sh --profile quick` | Local and release validation only; not wired into any workflow. Contract: [ecstore-validation-suite-design.md](ecstore-validation-suite-design.md) | | e2e (`e2e_test` crate) | A real `rustfs` binary per test, driven over the S3, admin, and protocol APIs | `python3 scripts/e2e_binary.py build --features e2e-test-hooks`, then `python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test` | PR: `e2e-smoke` (report-only); merge queue / main push: `e2e-full`; nightly: `e2e-repl-nightly`, `e2e-nightly`, `e2e-protocols`, `e2e-distributed`. Guide: [`crates/e2e_test/README.md`](../../crates/e2e_test/README.md); 4-node 4-disk map: [distributed-e2e.md](distributed-e2e.md) | -| Outbound target matrix | Replication of every object shape (empty, plain, retention, legal hold, multipart) against every remote-target failure mode the fake target models; an explicit expectation table pins known-red cells to an open issue | `python3 scripts/e2e_binary.py build`, then `python3 scripts/e2e_binary.py run -- cargo nextest run -p e2e_test -E 'test(/^replication_target_matrix_test::/)'` | With `e2e-repl-nightly`; required locally for any change to outbound client defaults (SOP: [`docs/postmortems/2026-09-03-replication-checksum-default-regression.md`](../postmortems/2026-09-03-replication-checksum-default-regression.md)) | +| Outbound target matrix | Replication of every object shape (empty, plain, retention, legal hold, multipart, checksummed, SSE-S3) against every remote-target failure mode the fake target models; an explicit expectation table pins known-red cells to an open issue | `python3 scripts/e2e_binary.py build`, then `python3 scripts/e2e_binary.py run -- cargo nextest run -p e2e_test -E 'test(/^replication_target_matrix_test::/)'` | With `e2e-repl-nightly`; required locally for any change to outbound client defaults (SOP: [`docs/postmortems/2026-09-03-replication-checksum-default-regression.md`](../postmortems/2026-09-03-replication-checksum-default-regression.md)) | | s3s-e2e conformance | External S3 conformance tool against a live server | `./scripts/e2e-run.sh ./target/debug/rustfs ` | PR, report-only (second half of the `End-to-End Tests` job) | | S3 compatibility | `ceph/s3-tests` (boto3; allow-list `scripts/s3-tests/implemented_tests.txt`) and MinIO `mint` | `scripts/s3-tests/run.sh`; mint via `.github/workflows/mint.yml` | s3-tests: PR report-only plus a weekly full sweep; mint: weekly, report-only | | Chaos / fault-injection | Single-node disk fault injection (`crates/e2e_test/src/chaos.rs`, `crates/e2e_test/src/fault_proxy.rs`) plus the 4-node kill/fresh-drive/blackhole cases in `crates/e2e_test/src/distributed/chaos_test.rs` | Part of the e2e crate (`e2e-reliability` and `e2e-distributed`) | Reliability cases with `e2e-full`; 4-node chaos on storage-sensitive PRs and nightly via `e2e-distributed` | | Fuzz | `cargo-fuzz` targets over untrusted parsing surfaces; isolated sub-workspace under `fuzz/` | `./scripts/fuzz/run.sh` (see [`fuzz/README.md`](../../fuzz/README.md)) | PR smoke on the paths listed in `.github/workflows/fuzz.yml`, plus nightly corpus | | Benchmarks | Criterion benches under each crate's `benches/` | `cargo bench -p ` | On demand; never a gate | +| Conditional reads | Paired ordinary/hit/miss GET latency, throughput and read-lock measurements | `scripts/run_conditional_read_abba.py` | Local conditional-read acceptance; [guide and scope](conditional-read-benchmark.md) | Every script named above is indexed with status and wiring in [`scripts/README.md`](../../scripts/README.md). Fixed GHSA advisories map to named regression tests in [security-regressions.md](security-regressions.md). diff --git a/docs/testing/ci-gates.md b/docs/testing/ci-gates.md index 29cc67db4..f42b5a301 100644 --- a/docs/testing/ci-gates.md +++ b/docs/testing/ci-gates.md @@ -15,10 +15,16 @@ The `main` ruleset (`6436880`) requires exactly these contexts, with `strict_req | `Quick Checks` | `ci.yml` job `quick-checks` | Formatting and repository guard scripts | | `Test and Lint` | `ci.yml` job `required-checks` | Exact expected results for every CI validation job, including workspace checks, critical E2E, feature lanes, and event-specific full suites | -Every PR enters `ci.yml`. The `classify-changes` job uses the base revision of `scripts/ci_gate.py` to select a conservative documentation-only path: root Markdown/licenses, `AGENTS.md`, Markdown under `docs/` or `.agents/skills/`, and documentation images. Unknown paths, unavailable Git history, an empty diff, or a missing base policy select the full matrix. Renames include their deleted source path. Documentation-only PRs still run Quick Checks and Typos; the aggregate requires the expensive jobs to be skipped exactly as selected. +Every PR enters `ci.yml`. The `classify-changes` job uses the base revision of `scripts/ci_gate.py` to select a conservative documentation-only path: root Markdown/licenses, `AGENTS.md`, Markdown under `docs/` or `.agents/skills/`, and documentation images. Unknown paths, unavailable Git history, an empty diff, or a missing base policy select the full scope for that event. Renames include their deleted source path. Documentation-only PRs still run Quick Checks and Typos; the aggregate requires the expensive jobs to be skipped exactly as selected. PRs changing only Rust sources under `crates/e2e_test/src/` and known E2E selection digests (optionally with those documentation paths) use the `e2e` scope. They keep workspace validation, the server build, E2E smoke, and both S3 lanes; independent workflows retain their path selection. Connect boundary, serial ILM, optional protocol/rio-v2, and real io_uring jobs are skipped because their sources and test inputs are unchanged. Manifests, fixtures, shared nextest configuration, scripts, production sources, and unknown paths still select `full`. A dependency on `e2e_test` from another workspace member or reachable local dependency disables this shortcut. All non-PR events retain their full selection. +Alternate-feature unit tests and Clippy for rio-v2, Swift, SFTP, and HTTP/3 run on main pushes, merge groups, and scheduled/manual validation. PRs run the default workspace and retain ILM, S3, real io_uring, and Connect security boundaries in their applicable scopes. The aggregate requires the alternate-feature jobs to be skipped on PRs and successful on non-PR events. This reduces repeated builds across PR revisions; feature-specific regressions can be detected after merging when the merge queue is not used. For changes to these features, dispatch `ci.yml` against the candidate branch before merging when that signal is needed, including PRs targeting `gateway/integration`, which has no automatic push trigger in this workflow. Scheduled results supplement the main baseline and do not replace it. + +Workspace Clippy uses the same non-PR cadence. Every code PR still builds and runs the complete default workspace test suite, documentation tests, and migration proofs; Quick Checks retain formatting and script/workflow linting. Without a merge queue, Clippy diagnostics may first fail main after merge. A manual `ci.yml` dispatch against the candidate branch runs Clippy before merging when needed. + +Serial ILM tests run at the end of `test-and-lint` for the `full` scope, reusing the workspace's Cargo package and feature selection. The existing ignored-test filter and `-j1` remain; `--profile default` keeps the no-retry execution and preserves the separate core CI JUnit report. Any ILM failure fails the workspace job and required aggregate. Logs remain in the `ilm-integration` artifact. There is no separate ILM runner or cache restore. + `required-checks` runs even after failed or skipped dependencies. `scripts/ci_gate.py verify` rejects missing jobs, unexpected jobs, failure, cancellation, and unexpected skips; optional lanes are required only on their declared events. `Workspace Test and Lint` is the ordinary Rust job, while `Test and Lint` uniquely names the aggregate. New validation jobs must update both its direct dependencies and the script contract. Test this wiring and its failure cases with `python3 scripts/ci_gate.py --self-test`. The `gateway/integration` ruleset (`24663286`) requires the same three contexts with the same review rules for the s3s to gateway integration branch (rustfs/backlog#2734). It omits `required_linear_history` because `gateway-integration-sync.yml` lands `--no-ff` merge commits from `main` on that branch every weekday, and it names the `rustfs-cla-assistant` app as the bypass actor that pushes them. `ci.yml` runs on pull requests targeting either branch. @@ -45,10 +51,11 @@ The four site-replication state-writer concurrency proofs reserve the available | Event | Check name | Workflow / job | Merge status | Reproduce | |---|---|---|---|---| | PR, non-doc change | `Quick Checks` | `ci.yml` `quick-checks` | Required | `make pre-commit` | -| PR, non-doc change | `Workspace Test and Lint` | `ci.yml` `test-and-lint` | Via aggregate | `cargo clippy --all-targets -- -D warnings`; `cargo nextest run --profile ci --all --exclude e2e_test`; `cargo test --all --doc`; migration evidence check described below | +| PR, non-doc change | `Workspace Test and Lint` | `ci.yml` `test-and-lint` | Via aggregate | `cargo nextest run --profile ci --all --exclude e2e_test`; `cargo test --all --exclude e2e_test --doc`; migration evidence check described below | +| Main push, merge group, schedule, manual dispatch | `Workspace Test and Lint` | `ci.yml` `test-and-lint` | Via aggregate on these events | workspace tests plus `cargo clippy --all-targets -- -D warnings` | | PR, non-doc change | `Typos` | `ci.yml` `typos` | Via aggregate | `typos` | -| PR, full selection | `ILM Integration (serial)` | `ci.yml` `test-ilm-integration-serial` | Via aggregate | exact command in the job | -| PR, full selection | `Test and Lint (rio-v2)`, `Test and Lint (swift)`, `Test and Lint (sftp)` | `ci.yml` `test-and-lint-rio-v2`, `test-and-lint-protocols` | Via aggregate | `cargo nextest run` with the job's `--features` | +| Full selection | `Workspace Test and Lint` (serial ILM step) | `ci.yml` `test-and-lint` | Via aggregate | exact `Run ignored ILM integration tests serially` command in the job | +| Main push, merge group, schedule, manual dispatch | `Test and Lint (rio-v2)`, `Test and Lint (swift)`, `Test and Lint (sftp)`, `Test and Lint (http3)` | `ci.yml` `test-and-lint-rio-v2`, `test-and-lint-protocols` | Via aggregate on these events; skipped on PRs | `cargo nextest run` with the job's `--features` | | PR, full selection | `Connect Short Credential Boundary` | `ci.yml` `connect-short-credential-boundary` | Via aggregate | `cargo test -p rustfs --test connect_registration --features connect-e2e-short-credentials`; `cargo check -p rustfs --release --features connect-e2e-short-credentials` must fail | | PR, full selection | `Offline Enrollment E2E Root Boundary` | `ci.yml` `offline-enrollment-e2e` | Via aggregate | `scripts/check_offline_enrollment_e2e.sh` | | PR, non-doc change | `Build RustFS Debug Binary` | `ci.yml` `build-rustfs-debug-binary` | Via aggregate; prerequisite for black-box jobs | `python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks` (binary plus its `rustfs.e2e.json` sidecar) | @@ -59,9 +66,9 @@ The four site-replication state-writer concurrency proofs reserve the available | PR touching `paths` in `audit.yml` | `Cargo Deny`, `Workflow Pin Report`, `Dependency Review` | `audit.yml` `cargo-deny`, `workflow-pin-report`, `dependency-review` | Report-only | `cargo deny check`; `scripts/security/check_workflow_pins.sh` | | PR touching `paths` in `nix.yml` | `Nix Build & Check` | `nix.yml` `nix-validation` | Report-only | `nix flake check` | | PR touching `paths` in `fuzz.yml` | `Build Fuzz Harness`, `Smoke / ` | `fuzz.yml` `fuzz-build`, `pr-fuzz-smoke` | Report-only | `MAX_TOTAL_TIME=60 ./scripts/fuzz/run.sh` | -| PR touching `paths` in `windows-filesystem.yml` | `Rename Safety` | `windows-filesystem.yml` `rename-safety` | Report-only | the `cargo test -p rustfs-ecstore --lib ` commands in the job, on Windows | +| PR touching `paths` in `windows-filesystem.yml` | `Rename Safety`, `Startup Smoke` | `windows-filesystem.yml` `rename-safety`, `startup-smoke` | Report-only | the `cargo test -p rustfs-ecstore --lib ` commands in the job, on Windows; `cargo build --locked -p rustfs --bin rustfs`, then `rustfs.exe --version` and a single-volume server polled on `/health/ready` | | PR touching `paths` in `coverage.yml` | `Workspace line coverage` | `coverage.yml` `coverage` | Report-only | `make coverage`; `python3 scripts/check_security_coverage.py target/llvm-cov/coverage.json` | -| PR touching `paths` in `e2e-upgrade.yml` | `Direct upgrade from the previous release`, `Mixed-version rolling upgrade from the previous release`, `Bucket configuration survives the upgrade`, `Rollback reads current bucket metadata` | `e2e-upgrade.yml` `upgrade` matrix | Report-only | `python3 scripts/e2e_binary.py build`, then the job's `python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test` command with `RUSTFS_UPGRADE_SOURCE_BINARY` pointing at the pinned previous release (`UPGRADE_SOURCE_VERSION`) | +| PR touching `paths` in `e2e-upgrade.yml` | `Upgrade compatibility` | `e2e-upgrade.yml` `upgrade` | Report-only | `python3 scripts/e2e_binary.py build`, then the job's `python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test` commands with `RUSTFS_UPGRADE_SOURCE_BINARY` pointing at the pinned previous release (`UPGRADE_SOURCE_VERSION`) | | PR touching `paths` in `oidc-keycloak.yml` | `OIDC Keycloak live gate` | `oidc-keycloak.yml` `oidc-keycloak-live` | Report-only | `cargo build --locked -p rustfs --bin rustfs`, then `bash scripts/test/oidc_keycloak_live.sh ./target/debug/rustfs` | | PR touching `paths` in `targets-integration.yml` | `PostgreSQL, MySQL, AMQP, and NATS` | `targets-integration.yml` `targets-live` | Report-only | start the containers as in the job, export the `RUSTFS_TEST_*` DSNs, then the job's `cargo test --locked -p rustfs-targets --test -- --ignored --test-threads=1` commands | | PR, documentation-only selection | `Quick Checks`, `Typos`, `Test and Lint` | `ci.yml` `quick-checks`, `typos`, `required-checks` | Required directly or via aggregate | Quick Checks commands; `python3 scripts/ci_gate.py --self-test` | @@ -69,7 +76,7 @@ The four site-replication state-writer concurrency proofs reserve the available e2e filters live in `.config/nextest.toml`; extend a profile instead of adding a second selector. Before a profile runs, `scripts/check_test_wiring.py` compares its listing to the committed digest in `.config/e2e--selection.txt`, so a silent test drop fails closed. -Architecture migration rules run once in the required Quick Checks job. Migration proofs reuse the successful workspace run: `scripts/check_migration_gate_count.sh evidence ` checks the unchanged name selection and committed floor, and requires exactly one successful execution without retries for each proof. Missing, filtered, skipped, failed, or duplicate evidence fails the gate. Local `check` and `run` modes remain available. Upgrade compatibility builds one current server and shares its binary and provenance sidecar across the seven cases; each case independently verifies the pinned previous release checksum and runs its existing test. +Architecture migration rules run once in the required Quick Checks job. Migration proofs reuse the successful workspace run: `scripts/check_migration_gate_count.sh evidence ` checks the unchanged name selection and committed floor, and requires exactly one successful execution without retries for each proof. Missing, filtered, skipped, failed, or duplicate evidence fails the gate. Local `check` and `run` modes remain available. Upgrade compatibility builds one current server and shares its binary and provenance sidecar with one test runner. That runner verifies the pinned previous release checksum once, reuses the compiled harness, and executes every existing upgrade case in a separate process. Failed cases do not stop the remaining cases; any failure fails the job. Logs and G09 evidence retain per-case subdirectories in the aggregate artifacts. `scripts/test_upgrade_workflow.py` checks case membership and failure propagation. Scanner usage and heal rebuild coverage are intentionally split by risk and cost. `data_usage_test` runs in the PR `e2e-smoke` lane so changes that affect @@ -99,7 +106,7 @@ Scheduled lanes never block a PR. Their workflow-local gate fails the run, sched | `mint.yml` (weekly) | `mint` | report-only by design; per-suite PASS/FAIL/NA and raw `log.json` | yes | pinned Docker sequence in the workflow | | `coverage.yml` (weekly) | `coverage` | report-only trend; lcov and JSON artifact | yes | `make coverage` | | `runner-hygiene.yml` (monthly) | `check-ephemerality` | runner ephemerality | yes | dispatch | -| `e2e-upgrade.yml` (weekly) | `upgrade` (4-case matrix) | upgrade and rollback gate; server logs | no | see the PR row | +| `e2e-upgrade.yml` (weekly) | `upgrade` (serial cases sharing one harness build) | upgrade and rollback gate; server logs | no | see the PR row | | `oidc-keycloak.yml` (weekly) | `oidc-keycloak-live` | live OIDC gate | no | see the PR row | | `targets-integration.yml` (nightly) | `targets-live` | live target gate; container logs | no | see the PR row | | `gateway-integration-sync.yml` (weekdays) | `sync` | merges `main` into `gateway/integration` as a `--no-ff` merge commit; a conflicting merge is aborted and opens a `sync/-` pull request listing the conflicting files, which keeps the run green; a rejected push or missing pull request fails it | no | dispatch, optionally with `source_ref` | diff --git a/docs/testing/conditional-read-benchmark.md b/docs/testing/conditional-read-benchmark.md new file mode 100644 index 000000000..eef1992e5 --- /dev/null +++ b/docs/testing/conditional-read-benchmark.md @@ -0,0 +1,95 @@ +# Conditional object-read benchmark + +**Use this when:** validating the performance of metadata-based conditional GET +changes or reproducing a paired application-read regression experiment. + +Conditional GET acceptance needs paired performance evidence for ordinary reads, +condition hits, and condition misses. Saving body work on a 304 does not establish +that other reads are unaffected. Keep performance acceptance pending if resource +isolation, baseline stability, or regression budgets fail. + +The manually invoked `conditional_read_paired_benchmark` drives the actual +`DefaultObjectUsecase` against four local EC disks. It consumes and verifies the +entire successful body, validates 304 results, and samples existing read-lock +hold histograms. It covers 4 KiB inline and 1.3 MB sharded objects, body cache off +and warmed, and a 20 ms delay on one metadata disk. It excludes network ingress, +authentication, external tiers, cold-fill contention, and distributed capacity. + +The test-util-only metadata guard bypasses metadata caching in the dedicated +`conditional-bench` bucket so cache hits cannot hide the slow-disk scenario. +Both ordinary GET and prepared GET metadata reads already use `read_data=true` +and therefore exercise the existing delay injector without an additional override. +The guard has no effect in ordinary builds or outside its bucket, and ownership +drop restores the default policy. Both compared revisions must include this +identical benchmark control. + +## Build paired artifacts + +Use a clean detached worktree at the full baseline commit, a separate candidate +worktree, and matching stable toolchain, features, and build profile. The adapter +resolves every source anchor before writing. It installs only the test harness, +test-util controls, and their facade exports; it does not install the conditional +read implementation on the baseline. Review its diff before building. + +```bash +python3 scripts/prepare_conditional_read_baseline.py \ + --baseline-worktree "$BASELINE_WORKTREE" --revision "$BASELINE_REVISION" +``` + +In each worktree, format the adapter and build the same artifact: + +```bash +RUSTUP_TOOLCHAIN=stable cargo fmt --all +RUSTUP_TOOLCHAIN=stable cargo test --locked -p rustfs --lib \ + --no-default-features --features rio-v2 \ + conditional_read_paired_benchmark --no-run +``` + +Save each executable printed by Cargo before another build overwrites it. For +deployment acceptance, use matching optimized profiles and an isolated target +environment. A test-profile run is a local application regression experiment, +not a production throughput estimate. + +## Run and interpret + +Stop concurrent builds and other workloads on the measurement resources. The +runner refuses concurrent Cargo/rustc processes or high host load. It executes +fresh processes serially in A1–B1–B2–A2 order, removes ambient RustFS experimental +settings and proxies, uses independent temporary disks, and records binary, +revision, harness, and profile provenance beside raw logs. + +Executable snapshots are read-only and their hashes are rechecked before and +after every case. Cargo/rustc activity is sampled during each process; host load +and isolation are checked again on exit, including the last A2 case. These checks +do not replace exclusive resource ownership for formal deployment acceptance. + +```bash +python3 scripts/run_conditional_read_abba.py \ + --baseline "$BASELINE_BINARY" --candidate "$CANDIDATE_BINARY" \ + --baseline-revision "$BASELINE_REVISION" \ + --candidate-revision "$CANDIDATE_REVISION" \ + --harness rustfs/src/app/object/conditional_read_bench.rs \ + --profile test --out-dir "$RESULT_DIR" +``` + +The default is 1000 requests per case at concurrency 8. Each case records +throughput, p95/p99 latency, read-lock p95/p99, and a zero metadata-cache-hit +assertion. Body-cache-on cases are warmed; they do not measure cold-fill misses. +Inline shard bytes can be carried by an `xl.meta` read even when no separate +body reader, bitrot verification, or decode is needed. + +Budgets are fixed before the run: + +- Baseline drift: throughput within 10%; latency and lock percentiles within + 15%, with a 0.25 ms absolute noise floor. +- Candidate regression: throughput loss at most 5%; latency and lock percentile + increases at most 10%, with a 0.25 ms absolute noise floor. + +A drift failure produces no candidate attribution. After a stable baseline, the +runner compares geometric means of the paired legs. Any workload over budget +keeps acceptance pending; improving 304 cannot compensate for an ordinary-read +or condition-miss regression. Preserve raw logs and investigate before rerunning; +do not relax budgets after seeing the results. + +Run `python3 scripts/test_conditional_read_abba.py` to verify the acceptance +parser and drift/regression gates. diff --git a/flake.nix b/flake.nix index a58fcd85f..fd9381e4e 100644 --- a/flake.nix +++ b/flake.nix @@ -77,7 +77,7 @@ rustfs = rustPlatform.buildRustPackage { pname = "rustfs"; - version = "1.0.1"; + version = "1.0.3"; src = ./.; diff --git a/helm/README.md b/helm/README.md index b4cec75c0..42d5b3476 100644 --- a/helm/README.md +++ b/helm/README.md @@ -140,6 +140,7 @@ behavior. | config.rustfs.scanner.cache_save_timeout_secs | string | `""` | Override scanner cache save timeout in seconds with `RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS` (minimum `1`). | | config.rustfs.scanner.max_concurrent_set_scans | string | `""` | Cap concurrent scanner set tasks with `RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS` (`0` keeps topology-derived concurrency). | | config.rustfs.scanner.max_concurrent_disk_scans | string | `""` | Cap concurrent scanner disk bucket walks per set with `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` (`0` keeps disk-count-derived concurrency). | +| config.rustfs.scanner.walkers_per_disk | string | `""` | Bucket walkers per local disk with `RUSTFS_SCANNER_WALKERS_PER_DISK` (`1` to `16`; the default `1` keeps one bucket walk per disk, raise it on flash so one disk walks several buckets concurrently). | | config.rustfs.scanner.yield_every_n_objects | string | `""` | Yield to the async runtime every N scanned objects with `RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS` (`0` disables extra yield). | | config.rustfs.scanner.alert_excess_versions | string | `""` | Set version count threshold for scanner alerts with `RUSTFS_SCANNER_ALERT_EXCESS_VERSIONS`. | | config.rustfs.scanner.alert_excess_version_size | string | `""` | Set retained version byte threshold for scanner alerts with `RUSTFS_SCANNER_ALERT_EXCESS_VERSION_SIZE`. | diff --git a/helm/rustfs/Chart.yaml b/helm/rustfs/Chart.yaml index 72ad9abc8..ae3dd7e33 100644 --- a/helm/rustfs/Chart.yaml +++ b/helm/rustfs/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: rustfs description: RustFS helm chart to deploy RustFS on kubernetes cluster. type: application -version: "1.0.1" -appVersion: "1.0.1" +version: "1.0.3" +appVersion: "1.0.3" home: https://rustfs.com icon: https://media.sys.truenas.net/apps/rustfs/icons/icon.svg maintainers: diff --git a/helm/rustfs/templates/_helpers.tpl b/helm/rustfs/templates/_helpers.tpl index 126e07e4f..4c3cac02a 100644 --- a/helm/rustfs/templates/_helpers.tpl +++ b/helm/rustfs/templates/_helpers.tpl @@ -154,6 +154,10 @@ Merges (in order of increasing precedence): }} {{- $annotations = merge $annotations $mtls }} {{- end }} +{{- $tls := .Values.tls | default dict -}} +{{- if and ($tls.enabled | default false) (eq .Values.ingress.className "traefik") -}} +{{- $_ := set $annotations "traefik.ingress.kubernetes.io/service.serversscheme" "https" -}} +{{- end -}} {{- if $annotations }} {{- toYaml $annotations }} {{- end }} @@ -188,6 +192,10 @@ Merges (in order of increasing precedence): {{- end }} {{- $annotations = merge $annotations (default (dict) .Values.ingress.customAnnotations) }} {{- $annotations = merge $annotations (default (dict) .Values.ingress.annotations) }} +{{- $tls := .Values.tls | default dict -}} +{{- if and ($tls.enabled | default false) (eq .Values.ingress.className "nginx") -}} +{{- $_ := set $annotations "nginx.ingress.kubernetes.io/backend-protocol" "HTTPS" -}} +{{- end -}} {{- if $annotations }} {{- toYaml $annotations }} {{- end }} @@ -285,7 +293,8 @@ RUSTFS_VOLUMES on spaces, one pool per expression). */}} {{- define "rustfs.volumes" -}} {{- $protocol := "http" -}} -{{- if .Values.mtls.enabled -}} +{{- $tls := .Values.tls | default dict -}} +{{- if or .Values.mtls.enabled ($tls.enabled | default false) -}} {{- $protocol = "https" -}} {{- end -}} {{- $headless := printf "%s-headless" (include "rustfs.fullname" .) -}} @@ -347,17 +356,17 @@ Render liveness and readiness probe for http and https */}} {{- define "rustfs.probes" -}} +{{- $tls := .Values.tls | default dict -}} {{- if .Values.livenessProbe.enabled }} livenessProbe: - {{- if .Values.mtls.enabled }} - exec: - command: -{{ include "rustfs.probeCommand" (dict "root" . "endpointPath" "/health") | nindent 6 }} + {{- if or .Values.mtls.enabled ($tls.enabled | default false) }} + tcpSocket: + port: {{ .Values.service.endpoint.port | default 9000 }} {{- else }} httpGet: path: /health port: {{ .Values.service.endpoint.port | default 9000 }} - scheme: {{ if .Values.mtls.enabled }}HTTPS{{ else }}HTTP{{ end }} + scheme: {{ if or .Values.mtls.enabled ($tls.enabled | default false) }}HTTPS{{ else }}HTTP{{ end }} {{- end }} initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds | default 60 }} periodSeconds: {{ .Values.livenessProbe.periodSeconds | default 5 }} @@ -376,7 +385,7 @@ readinessProbe: httpGet: path: /health/ready port: {{ .Values.service.endpoint.port | default 9000 }} - scheme: {{ if .Values.mtls.enabled }}HTTPS{{ else }}HTTP{{ end }} + scheme: {{ if or .Values.mtls.enabled ($tls.enabled | default false) }}HTTPS{{ else }}HTTP{{ end }} {{- end }} initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds | default 60 }} periodSeconds: {{ .Values.readinessProbe.periodSeconds | default 5 }} diff --git a/helm/rustfs/templates/configmap.yaml b/helm/rustfs/templates/configmap.yaml index 46e8331f8..f22e296c9 100644 --- a/helm/rustfs/templates/configmap.yaml +++ b/helm/rustfs/templates/configmap.yaml @@ -1,3 +1,8 @@ +{{- $tls := .Values.tls | default dict -}} +{{- $tlsEnabled := $tls.enabled | default false -}} +{{- if and $tlsEnabled .Values.mtls.enabled -}} +{{- fail "tls.enabled and mtls.enabled are mutually exclusive; use ingress TLS for public access to an mTLS backend" -}} +{{- end -}} apiVersion: v1 kind: ConfigMap metadata: @@ -41,8 +46,12 @@ data: {{- end }} {{- if .Values.mtls.enabled }} RUSTFS_SERVER_MTLS_ENABLE: "1" - RUSTFS_TLS_KEYLOG: "1" + {{- end }} + {{- if or .Values.mtls.enabled $tlsEnabled }} + RUSTFS_TLS_KEYLOG: "0" RUSTFS_TLS_PATH: "/opt/tls" + RUSTFS_TLS_RELOAD_ENABLE: "1" + RUSTFS_TLS_RELOAD_INTERVAL: {{ $tls.reloadInterval | default 30 | quote }} {{- end }} {{- if .Values.config.rustfs.obs_endpoint.enabled }} RUSTFS_OBS_ENDPOINT: {{ .Values.config.rustfs.obs_endpoint.base_endpoint | quote }} @@ -114,6 +123,9 @@ data: {{- end }} {{- if ne (toString .max_concurrent_disk_scans) "" }} RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS: {{ .max_concurrent_disk_scans | quote }} + {{- end }} + {{- if ne (toString .walkers_per_disk) "" }} + RUSTFS_SCANNER_WALKERS_PER_DISK: {{ .walkers_per_disk | quote }} {{- end }} {{- if ne (toString .yield_every_n_objects) "" }} RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS: {{ .yield_every_n_objects | quote }} diff --git a/helm/rustfs/templates/deployment.yaml b/helm/rustfs/templates/deployment.yaml index 39effeb24..97b663626 100644 --- a/helm/rustfs/templates/deployment.yaml +++ b/helm/rustfs/templates/deployment.yaml @@ -1,3 +1,5 @@ +{{- $tls := .Values.tls | default dict -}} +{{- $tlsEnabled := $tls.enabled | default false -}} {{- $logDir := .Values.config.rustfs.obs_log_directory }} {{- $logDirEnabled := ne $logDir "" }} @@ -123,25 +125,10 @@ spec: {{- toYaml .Values.resources | nindent 12 }} {{- include "rustfs.probes" . | nindent 10 }} volumeMounts: - {{- if .Values.mtls.enabled }} - - name: server-cert - mountPath: /opt/tls/rustfs_cert.pem - subPath: rustfs_cert.pem - - name: server-cert - mountPath: /opt/tls/rustfs_key.pem - subPath: rustfs_key.pem - - name: server-cert - mountPath: /opt/tls/ca.crt - subPath: ca.crt - - name: client-cert - mountPath: /opt/tls/client_cert.pem - subPath: client_cert.pem - - name: client-cert - mountPath: /opt/tls/client_key.pem - subPath: client_key.pem - - name: client-cert - mountPath: /opt/tls/client_ca.crt - subPath: client_ca.crt + {{- if or .Values.mtls.enabled $tlsEnabled }} + - name: tls-material + mountPath: /opt/tls + readOnly: true {{- end }} {{- with .Values.extraVolumeMounts }} {{- toYaml . | nindent 12 }} @@ -157,27 +144,40 @@ spec: {{- with .Values.extraVolumes }} {{- toYaml . | nindent 8 }} {{- end }} - {{- if .Values.mtls.enabled }} - - name: server-cert - secret: - secretName: {{ include "rustfs.fullname" . }}-server-tls - items: - - key: tls.crt - path: rustfs_cert.pem - - key: tls.key - path: rustfs_key.pem - - key: ca.crt - path: ca.crt - - name: client-cert - secret: - secretName: {{ include "rustfs.fullname" . }}-client-tls - items: - - key: tls.crt - path: client_cert.pem - - key: tls.key - path: client_key.pem - - key: ca.crt - path: client_ca.crt + {{- if or .Values.mtls.enabled $tlsEnabled }} + - name: tls-material + projected: + defaultMode: 0440 + sources: + - secret: + name: {{ if .Values.mtls.enabled }}{{ include "rustfs.fullname" . }}-server-tls{{ else }}{{ required "tls.existingSecret is required when tls.enabled=true" $tls.existingSecret }}{{ end }} + items: + - key: tls.crt + path: rustfs_cert.pem + - key: tls.key + path: rustfs_key.pem + {{- if .Values.mtls.enabled }} + - key: ca.crt + path: ca.crt + {{- end }} + {{- if .Values.mtls.enabled }} + - secret: + name: {{ include "rustfs.fullname" . }}-client-tls + items: + - key: tls.crt + path: client_cert.pem + - key: tls.key + path: client_key.pem + - key: ca.crt + path: client_ca.crt + {{- end }} + {{- if and $tlsEnabled $tls.trustSecret }} + - secret: + name: {{ $tls.trustSecret }} + items: + - key: ca.crt + path: public.crt + {{- end }} {{- end }} {{- if $logDirEnabled }} - name: logs diff --git a/helm/rustfs/templates/statefulset.yaml b/helm/rustfs/templates/statefulset.yaml index f2c90877d..54de97694 100644 --- a/helm/rustfs/templates/statefulset.yaml +++ b/helm/rustfs/templates/statefulset.yaml @@ -1,3 +1,5 @@ +{{- $tls := .Values.tls | default dict -}} +{{- $tlsEnabled := $tls.enabled | default false -}} {{- $logDir := .Values.config.rustfs.obs_log_directory }} {{- $logDirEnabled := ne $logDir "" }} {{- $poolsEnabled := and .Values.pools .Values.pools.enabled }} @@ -217,25 +219,10 @@ spec: {{- toYaml $.Values.resources | nindent 12 }} {{- include "rustfs.probes" $ | nindent 10 }} volumeMounts: - {{- if $.Values.mtls.enabled }} - - name: client-cert - mountPath: /opt/tls/client_cert.pem - subPath: client_cert.pem - - name: client-cert - mountPath: /opt/tls/client_key.pem - subPath: client_key.pem - - name: client-cert - mountPath: /opt/tls/client_ca.crt - subPath: client_ca.crt - - name: server-cert - mountPath: /opt/tls/rustfs_cert.pem - subPath: rustfs_cert.pem - - name: server-cert - mountPath: /opt/tls/rustfs_key.pem - subPath: rustfs_key.pem - - name: server-cert - mountPath: /opt/tls/ca.crt - subPath: ca.crt + {{- if or $.Values.mtls.enabled $tlsEnabled }} + - name: tls-material + mountPath: /opt/tls + readOnly: true {{- end }} {{- with $.Values.extraVolumeMounts }} {{- toYaml . | nindent 12 }} @@ -254,29 +241,42 @@ spec: - name: data mountPath: /data {{- end }} - {{- if or $.Values.mtls.enabled $.Values.extraVolumes }} + {{- if or $.Values.mtls.enabled $tlsEnabled $.Values.extraVolumes }} volumes: - {{- if $.Values.mtls.enabled }} - - name: server-cert - secret: - secretName: {{ include "rustfs.fullname" $ }}-server-tls - items: - - key: tls.crt - path: rustfs_cert.pem - - key: tls.key - path: rustfs_key.pem - - key: ca.crt - path: ca.crt - - name: client-cert - secret: - secretName: {{ include "rustfs.fullname" $ }}-client-tls - items: - - key: tls.crt - path: client_cert.pem - - key: tls.key - path: client_key.pem - - key: ca.crt - path: client_ca.crt + {{- if or $.Values.mtls.enabled $tlsEnabled }} + - name: tls-material + projected: + defaultMode: 0440 + sources: + - secret: + name: {{ if $.Values.mtls.enabled }}{{ include "rustfs.fullname" $ }}-server-tls{{ else }}{{ required "tls.existingSecret is required when tls.enabled=true" $tls.existingSecret }}{{ end }} + items: + - key: tls.crt + path: rustfs_cert.pem + - key: tls.key + path: rustfs_key.pem + {{- if $.Values.mtls.enabled }} + - key: ca.crt + path: ca.crt + {{- end }} + {{- if $.Values.mtls.enabled }} + - secret: + name: {{ include "rustfs.fullname" $ }}-client-tls + items: + - key: tls.crt + path: client_cert.pem + - key: tls.key + path: client_key.pem + - key: ca.crt + path: client_ca.crt + {{- end }} + {{- if and $tlsEnabled $tls.trustSecret }} + - secret: + name: {{ $tls.trustSecret }} + items: + - key: ca.crt + path: public.crt + {{- end }} {{- end }} {{- with $.Values.extraVolumes }} {{- toYaml . | nindent 8 }} diff --git a/helm/rustfs/values.yaml b/helm/rustfs/values.yaml index 88d01e341..b7b622d7e 100644 --- a/helm/rustfs/values.yaml +++ b/helm/rustfs/values.yaml @@ -185,6 +185,8 @@ config: max_concurrent_set_scans: "" # Cap concurrent scanner disk bucket walks per set (0 keeps disk-count-derived concurrency) max_concurrent_disk_scans: "" + # Bucket walkers per local disk (1-16; default 1 keeps one bucket walk per disk, raise on flash) + walkers_per_disk: "" # Yield to the async runtime every N scanned objects (0 disables extra yield) yield_every_n_objects: "" # Version count threshold for scanner alerts @@ -388,6 +390,14 @@ gatewayApi: name: "" namespace: "" +# Native TLS identity managed by an external certificate controller. +# Public TLS and an mTLS backend use separate listeners or an ingress terminator. +tls: + enabled: false + existingSecret: "" # kubernetes.io/tls: tls.crt and tls.key only + trustSecret: "" # Optional independent outbound trust bundle: ca.crt + reloadInterval: 30 + mtls: enabled: false clientCertPath: "/opt/tls/client_cert.pem" diff --git a/rustfs.spec b/rustfs.spec index c3b00b3c2..5b4bef67b 100644 --- a/rustfs.spec +++ b/rustfs.spec @@ -1,7 +1,7 @@ %global _enable_debug_packages 0 %global _empty_manifest_terminate_build 0 Name: rustfs -Version: 1.0.1 +Version: 1.0.3 Release: 1 Summary: High-performance distributed object storage for MinIO alternative @@ -57,6 +57,12 @@ install %_builddir/%{name}-%{version}/target/%_arch/%_arch-unknown-linux-gnu/rel %_bindir/rustfs %changelog +* Sun Oct 18 2026 overtrue +- Update RPM package to RustFS 1.0.3 + +* Fri Oct 16 2026 overtrue +- Update RPM package to RustFS 1.0.2 + * Sat Oct 03 2026 overtrue - Update RPM package to RustFS 1.0.1 diff --git a/rustfs/Cargo.toml b/rustfs/Cargo.toml index 04680f2cb..7338ed8b3 100644 --- a/rustfs/Cargo.toml +++ b/rustfs/Cargo.toml @@ -77,6 +77,10 @@ rio-v2 = ["rustfs-ecstore/rio-v2"] pyroscope = ["rustfs-obs/pyroscope", "dep:pyroscope", "dep:object", "dep:symbolic-demangle"] # Tokio runtime telemetry. Requires `--cfg tokio_unstable`; use `make build-profiling`. dial9 = ["rustfs-obs/dial9"] +# Diagnostic-only Rust GlobalAlloc live-byte accounting; adds per-allocation atomic updates. +diagnostic-allocator-live = [] +# Diagnostic-only S3-operation attribution for internode ReadVersion and list-walker metrics. +diagnostic-read-version-caller = ["rustfs-ecstore/diagnostic-read-version-caller", "rustfs-io-metrics/diagnostic-read-version-caller", "rustfs-scanner/diagnostic-read-version-caller"] http3 = ["dep:h3", "dep:h3-quinn", "dep:quinn"] hotpath = [ "hotpath/hotpath", diff --git a/rustfs/build.rs b/rustfs/build.rs index a302de37d..2f109366f 100644 --- a/rustfs/build.rs +++ b/rustfs/build.rs @@ -12,6 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. +use std::io::Write; + fn main() -> shadow_rs::SdResult<()> { println!("cargo:rerun-if-env-changed=RUSTFS_BUILD_VERSION"); if let Ok(version) = std::env::var("RUSTFS_BUILD_VERSION") @@ -24,6 +26,38 @@ fn main() -> shadow_rs::SdResult<()> { println!("cargo:rustc-env=RUSTFS_BUILD_VERSION={version}"); } - shadow_rs::ShadowBuilder::builder().build()?; + println!("cargo:rerun-if-env-changed=RUSTC"); + let target = std::env::var("TARGET")?; + let target_os = std::env::var("CARGO_CFG_TARGET_OS")?; + let target_arch = std::env::var("CARGO_CFG_TARGET_ARCH")?; + let output = std::process::Command::new(std::env::var("RUSTC")?).arg("-vV").output()?; + if !output.status.success() { + return Err(std::io::Error::other("failed to query the build compiler").into()); + } + let output = String::from_utf8(output.stdout)?; + let release = output + .lines() + .find_map(|line| line.strip_prefix("release: ")) + .ok_or_else(|| std::io::Error::other("build compiler did not report its release"))?; + let channel = match release.split_once('-') { + Some((_, channel)) if matches!(channel.split('.').next(), Some("beta" | "nightly" | "dev")) => { + channel.split('.').next().unwrap() + } + None if release.split('.').count() == 3 && release.split('.').all(|part| part.parse::().is_ok()) => "stable", + _ => return Err(std::io::Error::other("unrecognized build compiler release channel").into()), + }; + + // Generate target metadata once so every shadow-rs consumer agrees, even + // when the build script and compiler run on a different host platform. + let mut deny = shadow_rs::default_deny(); + deny.extend([shadow_rs::BUILD_OS, shadow_rs::RUST_CHANNEL]); + shadow_rs::ShadowBuilder::builder() + .deny_const(deny) + .hook(|file: &std::fs::File| { + writeln!(&*file, "pub const BUILD_OS: &str = {:?};", format!("{target_os}-{target_arch}"))?; + writeln!(&*file, "pub const RUST_CHANNEL: &str = {:?};", format!("{channel}-{target}"))?; + Ok(()) + }) + .build()?; Ok(()) } diff --git a/rustfs/src/admin/console.rs b/rustfs/src/admin/console.rs index 4019c9876..f03a3f3ee 100644 --- a/rustfs/src/admin/console.rs +++ b/rustfs/src/admin/console.rs @@ -1051,10 +1051,12 @@ mod tests { .uri(format!("{CONSOLE_PREFIX}/index.html")) .body(Body::empty()) .expect("failed to build request"); - request.extensions_mut().insert(crate::server::RemoteAddr(SocketAddr::new( - IpAddr::from([198, 51, 100, last_octet]), - 50000, - ))); + request + .extensions_mut() + .insert(Some(crate::server::RemoteAddr(SocketAddr::new( + IpAddr::from([198, 51, 100, last_octet]), + 50000, + )))); request }; diff --git a/rustfs/src/admin/handlers/bucket_meta.rs b/rustfs/src/admin/handlers/bucket_meta.rs index 1a23d64dd..8641d2c05 100644 --- a/rustfs/src/admin/handlers/bucket_meta.rs +++ b/rustfs/src/admin/handlers/bucket_meta.rs @@ -525,10 +525,13 @@ impl Operation for ImportBucketMetadata { // First pass: read all file contents into memory let mut file_contents = Vec::new(); for i in 0..zip_reader.len() { - let mut file = zip_reader + let (mut file, file_path) = zip_reader .by_index(i) + .and_then(|file| { + let path = file.name()?.into_owned(); + Ok((file, path)) + }) .map_err(|e| s3_error!(InternalError, "failed to read archive entry: {e}"))?; - let file_path = file.name().to_string(); let mut content = Vec::new(); file.read_to_end(&mut content) @@ -1564,7 +1567,7 @@ mod backup_zip_compatibility_tests { let mut content = Vec::new(); file.read_to_end(&mut content).expect("read diagnostic config"); assert!(!content.windows(RAW_SECRET.len()).any(|window| window == RAW_SECRET)); - files.insert(file.name().to_string(), content); + files.insert(file.name().expect("diagnostic entry name").into_owned(), content); } assert!(!files.contains_key(&format!("_diagnostic/{BUCKET}/{config_file}"))); assert_eq!(files[&format!("_diagnostic/{BUCKET}/{BUCKET_VERSIONING_CONFIG}")], VERSIONING_XML); @@ -1701,8 +1704,9 @@ mod backup_zip_compatibility_tests { let mut content = Vec::new(); file.read_to_end(&mut content).expect("read diagnostic config"); assert!(!content.windows(SECRET.len()).any(|window| window == SECRET.as_bytes())); - assert!(file.name() == DIAGNOSTIC_EXPORT_MANIFEST || file.name().starts_with("_diagnostic/")); - files.insert(file.name().to_string(), content); + let file_path = file.name().expect("diagnostic entry name").into_owned(); + assert!(file_path == DIAGNOSTIC_EXPORT_MANIFEST || file_path.starts_with("_diagnostic/")); + files.insert(file_path, content); } assert_eq!(files[&format!("_diagnostic/{HEALTHY}/{BUCKET_VERSIONING_CONFIG}")], VERSIONING_XML); assert!(files.contains_key(&format!("_diagnostic/{UNREADABLE}/{BUCKET_VERSIONING_CONFIG}"))); diff --git a/rustfs/src/admin/handlers/config_admin.rs b/rustfs/src/admin/handlers/config_admin.rs index 15bcd7763..3344d4643 100644 --- a/rustfs/src/admin/handlers/config_admin.rs +++ b/rustfs/src/admin/handlers/config_admin.rs @@ -68,15 +68,15 @@ use rustfs_config::{ ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, ENV_SCANNER_CYCLE, ENV_SCANNER_CYCLE_MAX_DIRECTORIES, ENV_SCANNER_CYCLE_MAX_DURATION_SECS, ENV_SCANNER_CYCLE_MAX_OBJECTS, ENV_SCANNER_DELAY, ENV_SCANNER_IDLE_MODE, ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS, ENV_SCANNER_MAX_CONCURRENT_SET_SCANS, ENV_SCANNER_MAX_WAIT_SECS, ENV_SCANNER_SPEED, - ENV_SCANNER_START_DELAY_SECS, ENV_SCANNER_YIELD_EVERY_N_OBJECTS, HEAL_BITROT_CYCLE, HEAL_SUB_SYS, - MAX_ADMIN_REQUEST_BODY_SIZE, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, + ENV_SCANNER_START_DELAY_SECS, ENV_SCANNER_WALKERS_PER_DISK, ENV_SCANNER_YIELD_EVERY_N_OBJECTS, HEAL_BITROT_CYCLE, + HEAL_SUB_SYS, MAX_ADMIN_REQUEST_BODY_SIZE, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TOPIC, MQTT_USERNAME, SCANNER_ALERT_EXCESS_FOLDERS, SCANNER_ALERT_EXCESS_VERSION_SIZE, SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE, SCANNER_MAX_CONCURRENT_DISK_SCANS, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_SPEED, - SCANNER_START_DELAY, SCANNER_SUB_SYS, SCANNER_YIELD_EVERY_N_OBJECTS, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, - WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, - WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, + SCANNER_START_DELAY, SCANNER_SUB_SYS, SCANNER_WALKERS_PER_DISK, SCANNER_YIELD_EVERY_N_OBJECTS, WEBHOOK_AUTH_TOKEN, + WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, + WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, }; use rustfs_credentials::Credentials; use rustfs_policy::policy::action::{Action, AdminAction}; @@ -286,6 +286,12 @@ const SCANNER_HELP_KEYS: &[HelpKeyMetadata] = &[ description: "cap concurrent disk bucket walks per set, 0 uses disk-count defaults", optional: true, }, + HelpKeyMetadata { + key: SCANNER_WALKERS_PER_DISK, + type_name: "number", + description: "bucket walkers per local disk, 1 through 16; remote disks keep one walker", + optional: true, + }, HelpKeyMetadata { key: SCANNER_YIELD_EVERY_N_OBJECTS, type_name: "number", @@ -1619,6 +1625,7 @@ fn env_help_key(sub_system: &str, key: &str) -> String { (SCANNER_SUB_SYS, SCANNER_CACHE_SAVE_TIMEOUT) => ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS.to_string(), (SCANNER_SUB_SYS, SCANNER_MAX_CONCURRENT_SET_SCANS) => ENV_SCANNER_MAX_CONCURRENT_SET_SCANS.to_string(), (SCANNER_SUB_SYS, SCANNER_MAX_CONCURRENT_DISK_SCANS) => ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS.to_string(), + (SCANNER_SUB_SYS, SCANNER_WALKERS_PER_DISK) => ENV_SCANNER_WALKERS_PER_DISK.to_string(), (SCANNER_SUB_SYS, SCANNER_YIELD_EVERY_N_OBJECTS) => ENV_SCANNER_YIELD_EVERY_N_OBJECTS.to_string(), (SCANNER_SUB_SYS, SCANNER_ALERT_EXCESS_VERSIONS) => ENV_SCANNER_ALERT_EXCESS_VERSIONS.to_string(), (SCANNER_SUB_SYS, SCANNER_ALERT_EXCESS_VERSION_SIZE) => ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE.to_string(), diff --git a/rustfs/src/admin/handlers/heal.rs b/rustfs/src/admin/handlers/heal.rs index f6c02aa0c..98dd32666 100644 --- a/rustfs/src/admin/handlers/heal.rs +++ b/rustfs/src/admin/handlers/heal.rs @@ -673,63 +673,80 @@ where } } -fn canonical_replacement_records( - records: &[rustfs_heal::ReplacementRecoveryRecord], -) -> Vec { - let mut records = records.to_vec(); - for record in &mut records { - record.target_slots.sort(); +fn reconcile_replacement_recovery_records( + snapshots: Vec, +) -> (Vec, bool) { + struct Observation { + record: rustfs_heal::ReplacementRecoveryRecord, + sources: Vec, } - records.sort_by(|left, right| { - ( - &left.task_id, - &left.generation, - &left.set_disk_id, - left.state, - &left.target_slots, - &left.verified_at, - &left.reason, - ) - .cmp(&( - &right.task_id, - &right.generation, - &right.set_disk_id, - right.state, - &right.target_slots, - &right.verified_at, - &right.reason, - )) - }); - records -} -fn merge_replacement_recovery_records( - snapshots: &[rustfs_heal::ReplacementRecoverySnapshot], -) -> Vec { - let mut records = BTreeMap::::new(); - for snapshot in snapshots { - for record in canonical_replacement_records(&snapshot.records) { - match records.entry(record.task_id.clone()) { - std::collections::btree_map::Entry::Vacant(entry) => { - entry.insert(record); - } - std::collections::btree_map::Entry::Occupied(entry) if entry.get() == &record => {} + let mut observations = BTreeMap::::new(); + for (source, snapshot) in snapshots.into_iter().enumerate() { + for mut record in snapshot.records { + record.target_slots.sort(); + let observation = match observations.entry(record.task_id.clone()) { + std::collections::btree_map::Entry::Vacant(entry) => entry.insert(Observation { + record, + sources: Vec::new(), + }), std::collections::btree_map::Entry::Occupied(mut entry) => { - let task_id = entry.key().clone(); - entry.insert(rustfs_heal::ReplacementRecoveryRecord { - task_id, - state: rustfs_heal::ReplacementRecoveryState::Unknown, - generation: None, - set_disk_id: None, - target_slots: Vec::new(), - reason: Some("peer replacement recovery records disagree".to_string()), - verified_at: None, - }); + if entry.get().record != record { + entry.get_mut().record = rustfs_heal::ReplacementRecoveryRecord { + task_id: entry.key().clone(), + state: rustfs_heal::ReplacementRecoveryState::Unknown, + generation: None, + set_disk_id: None, + target_slots: Vec::new(), + reason: Some("peer replacement recovery records disagree".to_string()), + verified_at: None, + }; + } + entry.into_mut() } + }; + // Snapshot order keeps sources sorted; repeated copies within one + // snapshot must not duplicate a generation's provenance. + if snapshot.definitive && observation.sources.last() != Some(&source) { + observation.sources.push(source); } } } - records.into_values().collect() + + let conflict = if observations + .values() + .any(|observation| observation.record.state == rustfs_heal::ReplacementRecoveryState::Unknown) + { + true + } else { + let mut target_sources = BTreeMap::<&str, Vec<&[usize]>>::new(); + for observation in observations.values() { + for target in &observation.record.target_slots { + target_sources.entry(target).or_default().push(&observation.sources); + } + } + target_sources.into_values().any(|sources| { + if sources.len() < 2 { + return false; + } + // A complete retained history normally has one common anchor. + // Checking that witness avoids comparing every historical pair. + if sources[0] + .iter() + .any(|source| sources[1..].iter().all(|other| other.binary_search(source).is_ok())) + { + return false; + } + // Pairwise shared history is sufficient even without one anchor + // containing the whole group. A transitive chain alone is not. + sources.iter().enumerate().any(|(index, left)| { + sources[index + 1..] + .iter() + .any(|right| !left.iter().any(|source| right.binary_search(source).is_ok())) + }) + }) + }; + (observations.into_values().map(|observation| observation.record).collect(), conflict) } fn aggregate_replacement_recovery_cluster_status( @@ -762,24 +779,15 @@ fn aggregate_replacement_recovery_cluster_status( reason.get_or_insert("peer_replacement_recovery_status_not_definitive"); } - // Replacement recovery state is anchored on one surviving disk. Nodes - // without that disk legitimately report a definitive empty snapshot, so - // only non-empty snapshots must agree with each other. - let mut non_empty_records = snapshots - .iter() - .filter(|snapshot| !snapshot.records.is_empty()) - .map(|snapshot| canonical_replacement_records(&snapshot.records)); - if let Some(first_records) = non_empty_records.next() - && non_empty_records.any(|records| records != first_records) - { - reason.get_or_insert("peer_replacement_recovery_status_conflict"); - } - - let records = merge_replacement_recovery_records(&snapshots); - if records - .iter() - .any(|record| matches!(record.state, rustfs_heal::ReplacementRecoveryState::Unknown)) - { + // Replacement completion proofs are anchored per generation on one + // surviving disk, so sequential automatic generations legitimately leave + // different completed tasks on different peers, and peers without an + // anchor report definitive empty snapshots. Peers conflict only when they + // disagree about one task (the merge marks those copies `Unknown`) or + // claim overlapping targets without a shared definitive local history. + // Retained histories and disjoint generations can be merged. + let (records, conflict) = reconcile_replacement_recovery_records(snapshots); + if conflict { reason.get_or_insert("peer_replacement_recovery_status_conflict"); } @@ -2195,6 +2203,14 @@ mod tests { } } + fn replacement_snapshot_with_retained_history() -> rustfs_heal::ReplacementRecoverySnapshot { + let mut snapshot = replacement_snapshot("11111111-1111-4111-8111-111111111111"); + snapshot + .records + .push(replacement_record("22222222-2222-4222-8222-222222222222")); + snapshot + } + #[tokio::test] async fn heal_selector_preflight_rejects_before_probe_and_token_creation() { use crate::admin::storage_api::runtime::PoolEndpoints; @@ -2369,7 +2385,9 @@ mod tests { } #[test] - fn replacement_recovery_cluster_requires_matching_peer_records() { + fn replacement_recovery_cluster_rejects_overlapping_targets_without_shared_history() { + // Separate anchors cannot establish that these overlapping records + // describe compatible historical replacement instances. let local = replacement_snapshot("11111111-1111-4111-8111-111111111111"); let peer = replacement_snapshot("22222222-2222-4222-8222-222222222222"); let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); @@ -2379,6 +2397,247 @@ mod tests { assert_eq!(cluster.records.len(), 2); } + #[test] + fn replacement_recovery_cluster_merges_disjoint_generations_across_anchors() { + // rustfs/rustfs#8366: sequential automatic generations anchor their + // completion proofs on different surviving disks, so peers hold + // disjoint completed tasks and the remaining peers hold none at all. + let mut rebuilt_record = replacement_record("11111111-1111-4111-8111-111111111111"); + rebuilt_record.target_slots = vec!["http://node-a:9000/mnt/disk2".to_string()]; + let mut anchor_record = replacement_record("22222222-2222-4222-8222-222222222222"); + anchor_record.target_slots = vec!["http://node-a:9000/mnt/disk1".to_string()]; + let rebuilt = rustfs_heal::ReplacementRecoverySnapshot { + records: vec![rebuilt_record], + definitive: true, + reason: None, + }; + let anchor = rustfs_heal::ReplacementRecoverySnapshot { + records: vec![anchor_record], + definitive: true, + reason: None, + }; + let empty = rustfs_heal::ReplacementRecoverySnapshot { + records: Vec::new(), + definitive: true, + reason: None, + }; + let cluster = + aggregate_replacement_recovery_cluster_status(vec![rebuilt], vec![Ok(Some(anchor)), Ok(Some(empty))], 3, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records.len(), 2); + assert!( + cluster + .records + .iter() + .all(|record| matches!(record.state, rustfs_heal::ReplacementRecoveryState::Completed)) + ); + } + + #[test] + fn replacement_recovery_cluster_rejects_peer_disagreement_about_one_task() { + let local = replacement_snapshot("11111111-1111-4111-8111-111111111111"); + let mut disagreeing = replacement_record("11111111-1111-4111-8111-111111111111"); + disagreeing.set_disk_id = Some("pool_0_set_1".to_string()); + let peer = rustfs_heal::ReplacementRecoverySnapshot { + records: vec![disagreeing], + definitive: true, + reason: None, + }; + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(!cluster.definitive); + assert_eq!(cluster.reason.as_deref(), Some("peer_replacement_recovery_status_conflict")); + assert_eq!(cluster.records[0].state, rustfs_heal::ReplacementRecoveryState::Unknown); + } + + #[test] + fn replacement_recovery_cluster_accepts_retained_history_on_one_anchor() { + let local = replacement_snapshot_with_retained_history(); + let empty = rustfs_heal::ReplacementRecoverySnapshot { + records: Vec::new(), + definitive: true, + reason: None, + }; + let cluster = aggregate_replacement_recovery_cluster_status(vec![local.clone()], vec![Ok(Some(empty))], 2, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, local.records); + } + + #[test] + fn replacement_recovery_cluster_accepts_matching_retained_histories() { + let local = replacement_snapshot_with_retained_history(); + let cluster = aggregate_replacement_recovery_cluster_status(vec![local.clone()], vec![Ok(Some(local.clone()))], 2, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, local.records); + } + + #[test] + fn replacement_recovery_cluster_accepts_committed_handoff_history() { + let mut local = replacement_snapshot_with_retained_history(); + let successor = local.records[1].task_id.clone(); + let predecessor = &mut local.records[0]; + predecessor.state = rustfs_heal::ReplacementRecoveryState::Unrecoverable; + predecessor.reason = Some(format!("replacement responsibility transferred to {successor}")); + predecessor.verified_at = None; + let cluster = aggregate_replacement_recovery_cluster_status(vec![local.clone()], Vec::new(), 1, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, local.records); + } + + #[test] + fn replacement_recovery_cluster_merges_retained_history_with_disjoint_peer_generation() { + let local = replacement_snapshot_with_retained_history(); + let mut peer = replacement_snapshot("33333333-3333-4333-8333-333333333333"); + peer.records[0].target_slots = vec!["http://node-a:9000/mnt/disk2".to_string()]; + let mut expected = local.records.clone(); + expected.extend(peer.records.clone()); + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, expected); + } + + #[test] + fn replacement_recovery_cluster_accepts_retained_history_reported_only_by_peer() { + let local = replacement_snapshot("11111111-1111-4111-8111-111111111111"); + let peer = replacement_snapshot_with_retained_history(); + let expected = peer.records.clone(); + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, expected); + } + + #[test] + fn replacement_recovery_cluster_rejects_overlap_outside_retained_history() { + let local = replacement_snapshot_with_retained_history(); + let peer = replacement_snapshot("33333333-3333-4333-8333-333333333333"); + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(!cluster.definitive); + assert_eq!(cluster.reason.as_deref(), Some("peer_replacement_recovery_status_conflict")); + assert_eq!(cluster.records.len(), 3); + } + + #[test] + fn replacement_recovery_cluster_rejects_disagreeing_copies_in_retained_history() { + let local = replacement_snapshot_with_retained_history(); + let mut peer = local.clone(); + peer.records[0].verified_at = Some(43); + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(!cluster.definitive); + assert_eq!(cluster.reason.as_deref(), Some("peer_replacement_recovery_status_conflict")); + assert_eq!(cluster.records.len(), 2); + assert_eq!(cluster.records[0].state, rustfs_heal::ReplacementRecoveryState::Unknown); + assert_eq!(cluster.records[1].state, rustfs_heal::ReplacementRecoveryState::Completed); + } + + #[test] + fn replacement_recovery_cluster_reconciles_target_and_source_matrix() { + let task_ids = [ + "11111111-1111-4111-8111-111111111111", + "22222222-2222-4222-8222-222222222222", + "33333333-3333-4333-8333-333333333333", + ]; + for targets in 0u8..8 { + let mut records = task_ids.map(replacement_record); + for (task, record) in records.iter_mut().enumerate() { + record.target_slots = vec![format!("http://node-a:9000/mnt/disk{}", (targets >> task) & 1)]; + } + // All non-empty source subsets cover pairwise histories without + // one common anchor, transitive-only histories, and empty peers. + for a in 1u8..8 { + for b in 1u8..8 { + for c in 1u8..8 { + let masks = [a, b, c]; + let mut snapshots = (0..3).map(|source| rustfs_heal::ReplacementRecoverySnapshot { + records: records + .iter() + .enumerate() + .filter(|(task, _)| masks[*task] & (1 << source) != 0) + .map(|(_, record)| record.clone()) + .collect(), + definitive: true, + reason: None, + }); + let expected = (0..3).all(|left| { + (left + 1..3).all(|right| { + records[left].target_slots != records[right].target_slots || masks[left] & masks[right] != 0 + }) + }); + let local = snapshots.next().expect("three source snapshots"); + let cluster = aggregate_replacement_recovery_cluster_status( + vec![local], + snapshots.map(|snapshot| Ok(Some(snapshot))).collect(), + 3, + true, + ); + + assert_eq!(cluster.definitive, expected, "targets={targets}, sources={masks:?}"); + assert_eq!(cluster.reason.is_none(), expected); + assert_eq!(cluster.records, records); + } + } + } + } + } + + #[test] + fn replacement_recovery_cluster_rejects_overlap_on_a_secondary_target() { + let mut local = replacement_snapshot_with_retained_history(); + local.records[0].target_slots.push("http://node-a:9000/mnt/disk2".to_string()); + let mut peer = replacement_snapshot("33333333-3333-4333-8333-333333333333"); + peer.records[0].target_slots = vec!["http://node-a:9000/mnt/disk2".to_string()]; + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(!cluster.definitive); + assert_eq!(cluster.reason.as_deref(), Some("peer_replacement_recovery_status_conflict")); + assert_eq!(cluster.records.len(), 3); + } + + #[test] + fn replacement_recovery_cluster_deduplicates_reordered_targets() { + let mut local = replacement_snapshot("11111111-1111-4111-8111-111111111111"); + local.records[0].target_slots.push("http://node-a:9000/mnt/disk2".to_string()); + let mut peer = local.clone(); + peer.records[0].target_slots.reverse(); + let expected = local.records.clone(); + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, expected); + } + + #[test] + fn replacement_recovery_cluster_accepts_history_from_a_high_index_peer() { + let empty = rustfs_heal::ReplacementRecoverySnapshot { + records: Vec::new(), + definitive: true, + reason: None, + }; + let mut peers = vec![Ok(Some(empty.clone())); 80]; + let history = replacement_snapshot_with_retained_history(); + let expected = history.records.clone(); + peers.push(Ok(Some(history))); + let cluster = aggregate_replacement_recovery_cluster_status(vec![empty], peers, 82, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, expected); + } + #[test] fn replacement_recovery_cluster_accepts_definitive_empty_peers() { let local = rustfs_heal::ReplacementRecoverySnapshot { diff --git a/rustfs/src/admin/handlers/mod.rs b/rustfs/src/admin/handlers/mod.rs index 720c29e04..29125aea7 100644 --- a/rustfs/src/admin/handlers/mod.rs +++ b/rustfs/src/admin/handlers/mod.rs @@ -70,6 +70,7 @@ pub mod system; pub mod table_catalog; mod target_descriptor; pub mod tier; +pub mod tls_certificate; pub mod tls_debug; pub mod trace; pub mod usage_prefix; diff --git a/rustfs/src/admin/handlers/oidc.rs b/rustfs/src/admin/handlers/oidc.rs index b5183ce05..c7151b839 100644 --- a/rustfs/src/admin/handlers/oidc.rs +++ b/rustfs/src/admin/handlers/oidc.rs @@ -13,36 +13,22 @@ // limitations under the License. use crate::admin::auth::authorize_admin_request; -use crate::admin::handlers::supervise_admin_mutation; use crate::admin::router::{AdminOperation, Operation, S3Router}; -use crate::admin::runtime_sources::{ - current_app_context, current_federated_identity_service, current_object_store_handle_for_context, - current_server_config_for_context, -}; +use crate::admin::runtime_sources::current_federated_identity_service; use crate::admin::service::federated_identity::DefaultFederatedSessionBinding; -use crate::admin::storage_api::config::{ - read_admin_config_without_migrate, read_admin_server_config_snapshot, save_admin_server_config_snapshot, -}; +use crate::admin::service::oidc_config; use crate::admin::utils::json_response; use crate::server::{ADMIN_PREFIX, MINIO_ADMIN_PREFIX, console_prefix}; use http::StatusCode; use hyper::Method; use matchit::Params; -use rustfs_config::oidc::{ - IDENTITY_OPENID_SUB_SYS, OIDC_CLAIM_NAME, OIDC_CLAIM_PREFIX, OIDC_CLIENT_ID, OIDC_CLIENT_SECRET, OIDC_CONFIG_URL, - OIDC_DEFAULT_CLAIM_NAME, OIDC_DEFAULT_EMAIL_CLAIM, OIDC_DEFAULT_GROUPS_CLAIM, OIDC_DEFAULT_ROLES_CLAIM, OIDC_DEFAULT_SCOPES, - OIDC_DEFAULT_USERNAME_CLAIM, OIDC_DISPLAY_NAME, OIDC_EMAIL_CLAIM, OIDC_GROUPS_CLAIM, OIDC_HIDE_FROM_UI, OIDC_ISSUER, - OIDC_OTHER_AUDIENCES, OIDC_REDIRECT_URI, OIDC_REDIRECT_URI_DYNAMIC, OIDC_ROLE_POLICY, OIDC_ROLES_CLAIM, OIDC_SCOPES, - OIDC_USERNAME_CLAIM, -}; -use rustfs_config::server_config::Config as ServerConfig; -use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, ENV_RUSTFS_BROWSER_REDIRECT_URL, EnableState, MAX_ADMIN_REQUEST_BODY_SIZE}; +use rustfs_config::{ENV_RUSTFS_BROWSER_REDIRECT_URL, MAX_ADMIN_REQUEST_BODY_SIZE}; use rustfs_iam::federation::{FederatedSessionBindingError, FederationError}; +use rustfs_iam::oidc::{OidcProviderConfigInput, OidcProviderValidationInput, console_session_duration_secs}; use rustfs_policy::policy::action::{Action, AdminAction}; -use rustfs_utils::egress::validate_outbound_url; use s3s::{Body, S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error}; +use serde::Serialize; use serde::de::DeserializeOwned; -use serde::{Deserialize, Serialize}; use time::OffsetDateTime; use tracing::{debug, error, warn}; use url::Url; @@ -78,6 +64,19 @@ fn callback_federation_error(error: FederationError) -> S3Error { } } +fn config_service_error(error: oidc_config::OidcAdminConfigError) -> S3Error { + use oidc_config::OidcAdminConfigError; + let code = match &error { + OidcAdminConfigError::Config(rustfs_iam::oidc::OidcConfigError::EnvironmentManaged) => S3ErrorCode::AccessDenied, + OidcAdminConfigError::Config(_) | OidcAdminConfigError::Validation(_) => S3ErrorCode::InvalidRequest, + OidcAdminConfigError::StorageUnavailable + | OidcAdminConfigError::Load(_) + | OidcAdminConfigError::Save(_) + | OidcAdminConfigError::Task(_) => S3ErrorCode::InternalError, + }; + S3Error::with_message(code, error.to_string()) +} + /// Validate that a provider ID contains only safe characters (alphanumeric, underscore, hyphen). fn is_valid_provider_id(id: &str) -> bool { !id.is_empty() && id.chars().all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-') @@ -198,104 +197,6 @@ struct OidcValidationResponse { token_endpoint: Option, } -#[derive(Debug, Deserialize)] -#[serde(default, deny_unknown_fields)] -struct OidcConfigUpsertRequest { - enabled: bool, - display_name: String, - config_url: String, - issuer: Option, - client_id: String, - client_secret: Option, - scopes: Vec, - other_audiences: Vec, - redirect_uri: Option, - redirect_uri_dynamic: bool, - claim_name: String, - claim_prefix: String, - role_policy: String, - groups_claim: String, - roles_claim: String, - email_claim: String, - username_claim: String, - hide_from_ui: bool, -} - -impl Default for OidcConfigUpsertRequest { - fn default() -> Self { - Self { - enabled: true, - display_name: String::new(), - config_url: String::new(), - issuer: None, - client_id: String::new(), - client_secret: None, - scopes: OIDC_DEFAULT_SCOPES.split(',').map(ToString::to_string).collect(), - other_audiences: Vec::new(), - redirect_uri: None, - redirect_uri_dynamic: true, - claim_name: OIDC_DEFAULT_CLAIM_NAME.to_string(), - claim_prefix: String::new(), - role_policy: String::new(), - groups_claim: OIDC_DEFAULT_GROUPS_CLAIM.to_string(), - roles_claim: OIDC_DEFAULT_ROLES_CLAIM.to_string(), - email_claim: OIDC_DEFAULT_EMAIL_CLAIM.to_string(), - username_claim: OIDC_DEFAULT_USERNAME_CLAIM.to_string(), - hide_from_ui: false, - } - } -} - -#[derive(Debug, Deserialize)] -#[serde(default, deny_unknown_fields)] -struct OidcConfigValidateRequest { - provider_id: String, - enabled: bool, - display_name: String, - config_url: String, - issuer: Option, - client_id: String, - client_secret: Option, - scopes: Vec, - other_audiences: Vec, - redirect_uri: Option, - redirect_uri_dynamic: bool, - claim_name: String, - claim_prefix: String, - role_policy: String, - groups_claim: String, - roles_claim: String, - email_claim: String, - username_claim: String, - hide_from_ui: bool, -} - -impl Default for OidcConfigValidateRequest { - fn default() -> Self { - Self { - provider_id: "default".to_string(), - enabled: true, - display_name: String::new(), - config_url: String::new(), - issuer: None, - client_id: String::new(), - client_secret: None, - scopes: OIDC_DEFAULT_SCOPES.split(',').map(ToString::to_string).collect(), - other_audiences: Vec::new(), - redirect_uri: None, - redirect_uri_dynamic: true, - claim_name: OIDC_DEFAULT_CLAIM_NAME.to_string(), - claim_prefix: String::new(), - role_policy: String::new(), - groups_claim: OIDC_DEFAULT_GROUPS_CLAIM.to_string(), - roles_claim: OIDC_DEFAULT_ROLES_CLAIM.to_string(), - email_claim: OIDC_DEFAULT_EMAIL_CLAIM.to_string(), - username_claim: OIDC_DEFAULT_USERNAME_CLAIM.to_string(), - hide_from_ui: false, - } - } -} - /// Handler: GET /rustfs/admin/v3/oidc/providers /// Returns list of configured OIDC providers for the login page. pub struct ListOidcProvidersHandler {} @@ -323,9 +224,9 @@ impl Operation for GetOidcConfigHandler { async fn call(&self, req: S3Request, _params: Params<'_, '_>) -> S3Result> { authorize_oidc_config_request(&req, AdminAction::ServerInfoAdminAction).await?; - let config = load_server_config_from_store().await?; - let restart_required = oidc_restart_required(&config); - let providers = rustfs_iam::oidc::load_oidc_config_snapshot(Some(&config)) + let config = oidc_config::list_config().await.map_err(config_service_error)?; + let providers = config + .snapshot .into_providers() .into_iter() .map(|provider| OidcConfigView { @@ -357,7 +258,7 @@ impl Operation for GetOidcConfigHandler { StatusCode::OK, &OidcConfigListResponse { providers, - restart_required, + restart_required: config.restart_required, }, ) } @@ -373,22 +274,13 @@ impl Operation for PutOidcConfigHandler { let provider_id = params .get("provider_id") .ok_or_else(|| s3_error!(InvalidRequest, "missing provider_id"))?; - if !is_valid_provider_id(provider_id) { - return Err(s3_error!(InvalidRequest, "invalid provider_id")); - } - if is_env_managed_provider(provider_id) { - return Err(s3_error!(AccessDenied, "provider is managed by environment variables")); - } + oidc_config::ensure_mutable_provider_id(provider_id).map_err(config_service_error)?; let provider_id = provider_id.to_owned(); - let request: OidcConfigUpsertRequest = parse_json_body(&mut req).await?; - update_oidc_server_config(move |config| { - let existing_secret = persisted_provider_secret(config, &provider_id); - let provider_config = build_provider_config_from_upsert(&provider_id, request, existing_secret)?; - upsert_persisted_provider_config(config, &provider_config); - Ok(()) - }) - .await?; + let request: OidcProviderConfigInput = parse_json_body(&mut req).await?; + oidc_config::upsert_config(provider_id, request) + .await + .map_err(config_service_error)?; json_response( StatusCode::OK, @@ -411,19 +303,9 @@ impl Operation for DeleteOidcConfigHandler { let provider_id = params .get("provider_id") .ok_or_else(|| s3_error!(InvalidRequest, "missing provider_id"))?; - if !is_valid_provider_id(provider_id) { - return Err(s3_error!(InvalidRequest, "invalid provider_id")); - } - if is_env_managed_provider(provider_id) { - return Err(s3_error!(AccessDenied, "provider is managed by environment variables")); - } + oidc_config::ensure_mutable_provider_id(provider_id).map_err(config_service_error)?; let provider_id = provider_id.to_owned(); - - update_oidc_server_config(move |config| { - delete_persisted_provider_config(config, &provider_id)?; - Ok(()) - }) - .await?; + oidc_config::delete_config(provider_id).await.map_err(config_service_error)?; json_response( StatusCode::OK, @@ -443,22 +325,8 @@ impl Operation for ValidateOidcConfigHandler { async fn call(&self, mut req: S3Request, _params: Params<'_, '_>) -> S3Result> { authorize_oidc_config_request(&req, AdminAction::ServerInfoAdminAction).await?; - let request: OidcConfigValidateRequest = parse_json_body(&mut req).await?; - let provider_id = if request.provider_id.trim().is_empty() { - "default".to_string() - } else { - request.provider_id.trim().to_string() - }; - let provider_config = build_provider_config_from_validate(request, &provider_id)?; - let oidc_extra_root_ca = crate::startup_auth::current_oidc_extra_root_ca_material() - .await - .map_err(|e| S3Error::with_message(S3ErrorCode::InvalidRequest, format!("validation failed: {e}")))?; - let validation = rustfs_iam::oidc::validate_oidc_provider_config_with_extra_root_ca( - &provider_config, - oidc_extra_root_ca.root_ca_pem.as_deref(), - ) - .await - .map_err(|e| S3Error::with_message(S3ErrorCode::InvalidRequest, format!("validation failed: {e}")))?; + let request: OidcProviderValidationInput = parse_json_body(&mut req).await?; + let validation = oidc_config::validate_config(request).await.map_err(config_service_error)?; json_response( StatusCode::OK, @@ -582,8 +450,9 @@ impl Operation for OidcCallbackHandler { let redirect_uri = derive_callback_uri(&req, provider_id)?; + let session_duration_seconds = console_session_duration_secs(); let login = federation - .complete_authorization_code(&state, &code, &redirect_uri, 3600, &DefaultFederatedSessionBinding) + .complete_authorization_code(&state, &code, &redirect_uri, session_duration_seconds, &DefaultFederatedSessionBinding) .await .map_err(|error| { if let FederationError::CodeExchange(message) = &error { @@ -609,7 +478,7 @@ impl Operation for OidcCallbackHandler { callback_federation_error(error) })?; let authorization = &login.session.authorization; - let actual_provider_id = &authorization.provider_id; + let actual_provider_id = authorization.provider_id(); debug!( event = EVENT_ADMIN_OIDC_STATE, @@ -625,10 +494,10 @@ impl Operation for OidcCallbackHandler { component = LOG_COMPONENT_ADMIN, subsystem = LOG_SUBSYSTEM_OIDC, provider_id = %actual_provider_id, - policy_count = authorization.policies.len(), - group_count = authorization.groups.len(), - policies = ?authorization.policies, - groups = ?authorization.groups, + policy_count = authorization.policies().len(), + group_count = authorization.mapped_iam_groups().len(), + policies = ?authorization.policies(), + groups = ?authorization.mapped_iam_groups(), state = "claims_mapped", "admin oidc state" ); @@ -877,316 +746,6 @@ async fn parse_json_body(req: &mut S3Request) -> S3Re serde_json::from_slice(&body).map_err(|e| s3_error!(InvalidRequest, "invalid JSON: {}", e)) } -async fn load_server_config_from_store() -> S3Result { - let store = oidc_config_store()?; - - read_admin_config_without_migrate(store) - .await - .map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("failed to load server config: {e}"))) -} - -fn oidc_config_store() -> S3Result> { - let context = current_app_context(); - current_object_store_handle_for_context(context.as_deref()) - .ok_or_else(|| s3_error!(InternalError, "storage layer not initialized")) -} - -async fn update_oidc_server_config(modifier: F) -> S3Result<()> -where - F: FnOnce(&mut ServerConfig) -> S3Result<()> + Send + 'static, -{ - let store = oidc_config_store()?; - supervise_admin_mutation("OIDC config update", async move { - let snapshot = read_admin_server_config_snapshot(store.clone()) - .await - .map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("failed to load server config: {e}")))?; - let mut config = snapshot.config.clone(); - modifier(&mut config)?; - save_admin_server_config_snapshot(store, &config, &snapshot) - .await - .map(|_| ()) - .map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("failed to save server config: {e}"))) - }) - .await -} - -fn is_env_managed_provider(provider_id: &str) -> bool { - rustfs_iam::oidc::load_oidc_provider_configs_from_env() - .iter() - .any(|config| config.id == provider_id) -} - -fn provider_instance_key(provider_id: &str) -> String { - if provider_id == "default" { - DEFAULT_DELIMITER.to_string() - } else { - provider_id.to_string() - } -} - -fn oidc_restart_required(config: &ServerConfig) -> bool { - let context = current_app_context(); - let active_config = current_server_config_for_context(context.as_deref()); - oidc_restart_required_from_active_config(config, active_config.as_ref()) -} - -fn oidc_restart_required_from_active_config(config: &ServerConfig, active_config: Option<&ServerConfig>) -> bool { - rustfs_iam::oidc::load_oidc_config_snapshot(Some(config)) != rustfs_iam::oidc::load_oidc_config_snapshot(active_config) -} - -fn default_oidc_kvs() -> s3s::S3Result { - ServerConfig::new() - .get_value(IDENTITY_OPENID_SUB_SYS, DEFAULT_DELIMITER) - .ok_or_else(|| s3_error!(InternalError, "default OIDC configuration missing")) -} - -fn set_kvs_value(kvs: &mut rustfs_config::server_config::KVS, key: &str, value: String) { - if let Some(existing) = kvs.0.iter_mut().find(|kv| kv.key == key) { - existing.value = value; - return; - } - - kvs.insert(key.to_string(), value); -} - -fn normalize_scopes(scopes: &[String]) -> Vec { - scopes - .iter() - .map(|scope| scope.trim().to_string()) - .filter(|scope| !scope.is_empty()) - .collect() -} - -fn normalize_optional(value: Option) -> Option { - value.map(|v| v.trim().to_string()).filter(|v| !v.is_empty()) -} - -fn validate_absolute_http_url(value: &str, field_name: &str) -> S3Result<()> { - let parsed = Url::parse(value).map_err(|_| s3_error!(InvalidRequest, "{} must be an absolute http/https URL", field_name))?; - - if !is_valid_scheme(parsed.scheme()) || parsed.host_str().is_none() { - return Err(s3_error!(InvalidRequest, "{} must be an absolute http/https URL", field_name)); - } - - validate_outbound_url(&parsed).map_err(|err| s3_error!(InvalidRequest, "{} is not allowed: {}", field_name, err))?; - - Ok(()) -} - -fn validate_absolute_http_url_without_outbound_check(value: &str, field_name: &str) -> S3Result<()> { - let parsed = Url::parse(value).map_err(|_| s3_error!(InvalidRequest, "{} must be an absolute http/https URL", field_name))?; - - if !is_valid_scheme(parsed.scheme()) || parsed.host_str().is_none() { - return Err(s3_error!(InvalidRequest, "{} must be an absolute http/https URL", field_name)); - } - - Ok(()) -} - -fn validate_provider_config_fields(config: &rustfs_iam::oidc::OidcProviderConfig) -> S3Result<()> { - if !is_valid_provider_id(&config.id) { - return Err(s3_error!(InvalidRequest, "invalid provider_id")); - } - if config.config_url.trim().is_empty() { - return Err(s3_error!(InvalidRequest, "config_url is required")); - } - validate_absolute_http_url(&config.config_url, "config_url")?; - if let Some(issuer) = config.issuer.as_deref() { - validate_absolute_http_url_without_outbound_check(issuer, "issuer")?; - } - - if config.client_id.trim().is_empty() { - return Err(s3_error!(InvalidRequest, "client_id is required")); - } - - if !config.redirect_uri_dynamic { - let redirect_uri = config - .redirect_uri - .as_deref() - .ok_or_else(|| s3_error!(InvalidRequest, "redirect_uri is required when redirect_uri_dynamic is off"))?; - validate_absolute_http_url(redirect_uri, "redirect_uri")?; - } else if let Some(redirect_uri) = config.redirect_uri.as_deref() { - validate_absolute_http_url(redirect_uri, "redirect_uri")?; - } - - if !config.scopes.iter().any(|scope| scope == "openid") { - return Err(s3_error!(InvalidRequest, "scopes must include openid")); - } - - Ok(()) -} - -fn or_default(value: &str, default: &str) -> String { - if value.trim().is_empty() { - default.to_string() - } else { - value.trim().to_string() - } -} - -/// Normalize an `OidcProviderConfig` by trimming strings and applying defaults. -fn normalize_provider_config(mut config: rustfs_iam::oidc::OidcProviderConfig) -> rustfs_iam::oidc::OidcProviderConfig { - config.config_url = config.config_url.trim().to_string(); - config.issuer = normalize_optional(config.issuer); - config.client_id = config.client_id.trim().to_string(); - config.scopes = normalize_scopes(&config.scopes); - config.redirect_uri = normalize_optional(config.redirect_uri); - config.claim_name = or_default(&config.claim_name, OIDC_DEFAULT_CLAIM_NAME); - config.claim_prefix = config.claim_prefix.trim().to_string(); - config.role_policy = config.role_policy.trim().to_string(); - config.display_name = or_default(&config.display_name, &config.id); - config.groups_claim = or_default(&config.groups_claim, OIDC_DEFAULT_GROUPS_CLAIM); - config.roles_claim = or_default(&config.roles_claim, OIDC_DEFAULT_ROLES_CLAIM); - config.email_claim = or_default(&config.email_claim, OIDC_DEFAULT_EMAIL_CLAIM); - config.username_claim = or_default(&config.username_claim, OIDC_DEFAULT_USERNAME_CLAIM); - config -} - -fn build_provider_config_from_upsert( - provider_id: &str, - request: OidcConfigUpsertRequest, - existing_secret: Option, -) -> S3Result { - let client_secret = match request.client_secret { - Some(value) if !value.trim().is_empty() => Some(value), - _ => existing_secret.filter(|value| !value.trim().is_empty()), - }; - - let config = normalize_provider_config(rustfs_iam::oidc::OidcProviderConfig { - id: provider_id.to_string(), - enabled: request.enabled, - config_url: request.config_url, - issuer: request.issuer, - client_id: request.client_id, - client_secret, - scopes: request.scopes, - other_audiences: request.other_audiences, - redirect_uri: request.redirect_uri, - redirect_uri_dynamic: request.redirect_uri_dynamic, - claim_name: request.claim_name, - claim_prefix: request.claim_prefix, - role_policy: request.role_policy, - display_name: request.display_name, - groups_claim: request.groups_claim, - roles_claim: request.roles_claim, - email_claim: request.email_claim, - username_claim: request.username_claim, - hide_from_ui: request.hide_from_ui, - }); - - validate_provider_config_fields(&config)?; - Ok(config) -} - -fn build_provider_config_from_validate( - request: OidcConfigValidateRequest, - provider_id: &str, -) -> S3Result { - let config = normalize_provider_config(rustfs_iam::oidc::OidcProviderConfig { - id: provider_id.to_string(), - enabled: request.enabled, - config_url: request.config_url, - issuer: request.issuer, - client_id: request.client_id, - client_secret: request.client_secret.filter(|value| !value.trim().is_empty()), - scopes: request.scopes, - other_audiences: request.other_audiences, - redirect_uri: request.redirect_uri, - redirect_uri_dynamic: request.redirect_uri_dynamic, - claim_name: request.claim_name, - claim_prefix: request.claim_prefix, - role_policy: request.role_policy, - display_name: request.display_name, - groups_claim: request.groups_claim, - roles_claim: request.roles_claim, - email_claim: request.email_claim, - username_claim: request.username_claim, - hide_from_ui: request.hide_from_ui, - }); - - validate_provider_config_fields(&config)?; - Ok(config) -} - -fn persisted_provider_secret(config: &ServerConfig, provider_id: &str) -> Option { - config - .0 - .get(IDENTITY_OPENID_SUB_SYS) - .and_then(|subsystem| subsystem.get(&provider_instance_key(provider_id))) - .and_then(|kvs| kvs.lookup(OIDC_CLIENT_SECRET)) - .filter(|value| !value.trim().is_empty()) -} - -fn upsert_persisted_provider_config(config: &mut ServerConfig, provider_config: &rustfs_iam::oidc::OidcProviderConfig) { - let instance_key = provider_instance_key(&provider_config.id); - let mut kvs = default_oidc_kvs().unwrap_or_default(); - - set_kvs_value( - &mut kvs, - ENABLE_KEY, - if provider_config.enabled { - EnableState::On.to_string() - } else { - EnableState::Off.to_string() - }, - ); - set_kvs_value(&mut kvs, OIDC_CONFIG_URL, provider_config.config_url.clone()); - set_kvs_value(&mut kvs, OIDC_ISSUER, provider_config.issuer.clone().unwrap_or_default()); - set_kvs_value(&mut kvs, OIDC_CLIENT_ID, provider_config.client_id.clone()); - set_kvs_value(&mut kvs, OIDC_CLIENT_SECRET, provider_config.client_secret.clone().unwrap_or_default()); - set_kvs_value(&mut kvs, OIDC_SCOPES, provider_config.scopes.join(",")); - set_kvs_value(&mut kvs, OIDC_OTHER_AUDIENCES, provider_config.other_audiences.join(",")); - set_kvs_value(&mut kvs, OIDC_REDIRECT_URI, provider_config.redirect_uri.clone().unwrap_or_default()); - set_kvs_value( - &mut kvs, - OIDC_REDIRECT_URI_DYNAMIC, - if provider_config.redirect_uri_dynamic { - EnableState::On.to_string() - } else { - EnableState::Off.to_string() - }, - ); - set_kvs_value(&mut kvs, OIDC_CLAIM_NAME, provider_config.claim_name.clone()); - set_kvs_value(&mut kvs, OIDC_CLAIM_PREFIX, provider_config.claim_prefix.clone()); - set_kvs_value(&mut kvs, OIDC_ROLE_POLICY, provider_config.role_policy.clone()); - set_kvs_value(&mut kvs, OIDC_DISPLAY_NAME, provider_config.display_name.clone()); - set_kvs_value(&mut kvs, OIDC_GROUPS_CLAIM, provider_config.groups_claim.clone()); - set_kvs_value(&mut kvs, OIDC_ROLES_CLAIM, provider_config.roles_claim.clone()); - set_kvs_value(&mut kvs, OIDC_EMAIL_CLAIM, provider_config.email_claim.clone()); - set_kvs_value(&mut kvs, OIDC_USERNAME_CLAIM, provider_config.username_claim.clone()); - set_kvs_value( - &mut kvs, - OIDC_HIDE_FROM_UI, - if provider_config.hide_from_ui { - EnableState::On.to_string() - } else { - EnableState::Off.to_string() - }, - ); - - config - .0 - .entry(IDENTITY_OPENID_SUB_SYS.to_string()) - .or_default() - .insert(instance_key, kvs); -} - -fn delete_persisted_provider_config(config: &mut ServerConfig, provider_id: &str) -> S3Result<()> { - let Some(subsystem) = config.0.get_mut(IDENTITY_OPENID_SUB_SYS) else { - return Err(s3_error!(InvalidRequest, "provider not found")); - }; - - if subsystem.remove(&provider_instance_key(provider_id)).is_none() { - return Err(s3_error!(InvalidRequest, "provider not found")); - } - - if subsystem.is_empty() { - config.0.remove(IDENTITY_OPENID_SUB_SYS); - } - - Ok(()) -} - fn extract_request_scheme(req: &S3Request) -> S3Result { let raw_scheme = req .headers @@ -1739,191 +1298,22 @@ mod tests { } #[test] - fn test_validate_absolute_http_url_rejects_loopback_targets() { - let err = validate_absolute_http_url("https://127.0.0.1/.well-known/openid-configuration", "config_url") - .expect_err("loopback config URL should be rejected"); - assert_eq!(err.code(), &S3ErrorCode::InvalidRequest); - assert!(err.message().unwrap_or_default().contains("not allowed")); - } + fn oidc_config_errors_keep_admin_codes() { + let error = config_service_error(oidc_config::OidcAdminConfigError::Config( + rustfs_iam::oidc::OidcConfigError::EnvironmentManaged, + )); + assert_eq!(error.code(), &S3ErrorCode::AccessDenied); + assert_eq!(error.message(), Some("provider is managed by environment variables")); - #[test] - fn test_provider_instance_key() { - assert_eq!(provider_instance_key("default"), "_"); - assert_eq!(provider_instance_key("okta"), "okta"); - } + let error = config_service_error(oidc_config::OidcAdminConfigError::Config( + rustfs_iam::oidc::OidcConfigError::ProviderNotFound, + )); + assert_eq!(error.code(), &S3ErrorCode::InvalidRequest); + assert_eq!(error.message(), Some("provider not found")); - #[test] - fn test_build_provider_config_requires_openid_scope() { - let req = OidcConfigUpsertRequest { - scopes: vec!["profile".to_string()], - config_url: "https://example.com/.well-known/openid-configuration".to_string(), - client_id: "client-id".to_string(), - ..Default::default() - }; - - assert!(build_provider_config_from_upsert("default", req, None).is_err()); - } - - #[test] - fn test_build_provider_config_preserves_existing_secret_when_request_is_empty() { - let req = OidcConfigUpsertRequest { - config_url: "https://example.com/.well-known/openid-configuration".to_string(), - client_id: "client-id".to_string(), - client_secret: Some("".to_string()), - ..Default::default() - }; - - let config = - build_provider_config_from_upsert("default", req, Some("existing-secret".to_string())).expect("config should build"); - - assert_eq!(config.client_secret.as_deref(), Some("existing-secret")); - assert_eq!(config.roles_claim, OIDC_DEFAULT_ROLES_CLAIM); - } - - #[test] - fn test_oidc_config_upsert_request_rejects_unknown_fields() { - let err = serde_json::from_str::( - r#"{"config_url":"https://example.com/.well-known/openid-configuration","client_id":"client","unexpected_field":true}"#, - ) - .expect_err("unknown upsert field should fail"); - - assert!(err.to_string().contains("unknown field")); - } - - #[test] - fn test_oidc_config_validate_request_rejects_unknown_fields() { - let err = serde_json::from_str::( - r#"{"provider_id":"default","config_url":"https://example.com/.well-known/openid-configuration","client_id":"client","unexpected_field":true}"#, - ) - .expect_err("unknown validate field should fail"); - - assert!(err.to_string().contains("unknown field")); - } - - #[test] - fn test_build_provider_config_uses_custom_roles_claim() { - let req = OidcConfigUpsertRequest { - config_url: "https://example.com/.well-known/openid-configuration".to_string(), - client_id: "client-id".to_string(), - roles_claim: "app_roles".to_string(), - ..Default::default() - }; - - let config = build_provider_config_from_upsert("default", req, None).expect("config should build"); - assert_eq!(config.roles_claim, "app_roles"); - } - - #[test] - fn test_oidc_restart_required_detects_persisted_changes() { - let active_config = ServerConfig::new(); - let mut persisted_config = ServerConfig::new(); - let provider_config = rustfs_iam::oidc::OidcProviderConfig { - id: "default".to_string(), - enabled: true, - config_url: "https://example.com/.well-known/openid-configuration".to_string(), - issuer: None, - client_id: "console".to_string(), - client_secret: Some("secret".to_string()), - scopes: vec!["openid".to_string(), "profile".to_string()], - other_audiences: vec![], - redirect_uri: None, - redirect_uri_dynamic: true, - claim_name: OIDC_DEFAULT_CLAIM_NAME.to_string(), - claim_prefix: String::new(), - role_policy: String::new(), - display_name: "default".to_string(), - groups_claim: OIDC_DEFAULT_GROUPS_CLAIM.to_string(), - roles_claim: OIDC_DEFAULT_ROLES_CLAIM.to_string(), - email_claim: OIDC_DEFAULT_EMAIL_CLAIM.to_string(), - username_claim: OIDC_DEFAULT_USERNAME_CLAIM.to_string(), - hide_from_ui: false, - }; - - upsert_persisted_provider_config(&mut persisted_config, &provider_config); - - assert!(oidc_restart_required_from_active_config(&persisted_config, Some(&active_config))); - assert!(!oidc_restart_required_from_active_config(&persisted_config, Some(&persisted_config))); - } - - #[test] - fn test_upsert_persists_hide_from_ui_on() { - let mut config = ServerConfig::new(); - let mut provider_config = rustfs_iam::oidc::OidcProviderConfig { - id: "kubernetes".to_string(), - enabled: true, - config_url: "https://example.com/.well-known/openid-configuration".to_string(), - issuer: None, - client_id: "test".to_string(), - client_secret: None, - scopes: vec!["openid".to_string()], - other_audiences: vec![], - redirect_uri: None, - redirect_uri_dynamic: true, - claim_name: "sub".to_string(), - claim_prefix: String::new(), - role_policy: String::new(), - display_name: "Kubernetes".to_string(), - groups_claim: OIDC_DEFAULT_GROUPS_CLAIM.to_string(), - roles_claim: OIDC_DEFAULT_ROLES_CLAIM.to_string(), - email_claim: OIDC_DEFAULT_EMAIL_CLAIM.to_string(), - username_claim: OIDC_DEFAULT_USERNAME_CLAIM.to_string(), - hide_from_ui: true, - }; - - upsert_persisted_provider_config(&mut config, &provider_config); - - let kvs = config - .0 - .get(IDENTITY_OPENID_SUB_SYS) - .and_then(|m| m.get("kubernetes")) - .expect("provider KVS should exist"); - assert_eq!(kvs.get(OIDC_HIDE_FROM_UI), EnableState::On.to_string()); - - // Flip to false and verify - provider_config.hide_from_ui = false; - upsert_persisted_provider_config(&mut config, &provider_config); - - let kvs = config - .0 - .get(IDENTITY_OPENID_SUB_SYS) - .and_then(|m| m.get("kubernetes")) - .expect("provider KVS should exist"); - assert_eq!(kvs.get(OIDC_HIDE_FROM_UI), EnableState::Off.to_string()); - } - - #[test] - fn test_upsert_persists_issuer() { - let mut config = ServerConfig::new(); - let provider_config = rustfs_iam::oidc::OidcProviderConfig { - id: "kubernetes".to_string(), - enabled: true, - config_url: "http://keycloak.ns.svc.cluster.local:8080/realms/app/.well-known/openid-configuration".to_string(), - issuer: Some("https://app.local/realms/app".to_string()), - client_id: "test".to_string(), - client_secret: None, - scopes: vec!["openid".to_string()], - other_audiences: vec![], - redirect_uri: None, - redirect_uri_dynamic: true, - claim_name: "sub".to_string(), - claim_prefix: String::new(), - role_policy: String::new(), - display_name: "Kubernetes".to_string(), - groups_claim: OIDC_DEFAULT_GROUPS_CLAIM.to_string(), - roles_claim: OIDC_DEFAULT_ROLES_CLAIM.to_string(), - email_claim: OIDC_DEFAULT_EMAIL_CLAIM.to_string(), - username_claim: OIDC_DEFAULT_USERNAME_CLAIM.to_string(), - hide_from_ui: false, - }; - - upsert_persisted_provider_config(&mut config, &provider_config); - - let kvs = config - .0 - .get(IDENTITY_OPENID_SUB_SYS) - .and_then(|m| m.get("kubernetes")) - .expect("provider KVS should exist"); - assert_eq!(kvs.get(OIDC_ISSUER), "https://app.local/realms/app"); + let error = config_service_error(oidc_config::OidcAdminConfigError::Task("cancelled")); + assert_eq!(error.code(), &S3ErrorCode::InternalError); + assert_eq!(error.message(), Some("OIDC config update task cancelled")); } /// The OIDC config gate now authorizes through the shared admin gate, which @@ -1986,4 +1376,30 @@ mod tests { ); } } + + #[test] + fn console_callback_uses_the_configured_session_duration() { + let production = include_str!("oidc.rs") + .split("\n#[cfg(test)]\n") + .next() + .expect("production source must precede tests"); + let callback = production + .split_once("impl Operation for OidcCallbackHandler") + .expect("the OIDC callback handler must exist") + .1; + let callback = callback.split_whitespace().collect::(); + + assert!( + callback.contains("session_duration_seconds=console_session_duration_secs()"), + "the callback must resolve the Console session duration from the environment" + ); + assert!( + callback.contains(".complete_authorization_code(&state,&code,&redirect_uri,session_duration_seconds,"), + "the callback must hand the resolved session duration to the code exchange" + ); + assert!( + !callback.contains(".complete_authorization_code(&state,&code,&redirect_uri,3600,"), + "the callback must not hard-code the session duration" + ); + } } diff --git a/rustfs/src/admin/handlers/rebalance.rs b/rustfs/src/admin/handlers/rebalance.rs index 14dc61b64..e59637b2b 100644 --- a/rustfs/src/admin/handlers/rebalance.rs +++ b/rustfs/src/admin/handlers/rebalance.rs @@ -42,12 +42,9 @@ use s3s::{ s3_error, }; use serde::{Deserialize, Serialize}; -use std::{ - future::Future, - sync::Arc, - time::{Duration, Instant}, -}; +use std::{future::Future, sync::Arc, time::Duration}; use time::OffsetDateTime; +use tokio::time::Instant; use tracing::{error, info, warn}; const LOG_COMPONENT_ADMIN: &str = "admin"; @@ -128,6 +125,13 @@ fn is_rebalance_start_fleet_proof_retryable(err: &StorageError) -> bool { && err.to_string().contains(REBALANCE_START_FLEET_PROOF_MARKER) } +fn rebalance_init_failure_error(err: &StorageError) -> S3Error { + if is_rebalance_start_fleet_proof_retryable(err) { + return rebalance_fleet_proof_unavailable_error(); + } + s3_error!(InternalError, "failed to initialize rebalance: {}", err) +} + fn rebalance_start_failure_error(start_err: &StorageError, rollback_result: &Result<(), String>) -> S3Error { if crate::storage_api::capacity::is_pool_activation_fleet_proof_error(start_err) { if rollback_result.is_ok() { @@ -611,7 +615,20 @@ impl Operation for RebalanceStart { let buckets: Vec = bucket_infos.into_iter().map(|bucket| bucket.name).collect(); - let id = match store.init_rebalance_start(buckets).await { + let fleet_proof_deadline = Instant::now() + REBALANCE_START_FLEET_PROOF_RETRY_BUDGET; + // A missing proof fails before init publishes metadata. Retrying the full + // operation reacquires its gates and revalidates competing starts. + let id = match retry_rebalance_start_operation( + "init_rebalance_start", + "", + &request_id, + &actor, + &remote_addr, + fleet_proof_deadline, + || store.init_rebalance_start(buckets.clone()), + ) + .await + { Ok(id) => id, Err(StorageError::DecommissionAlreadyRunning) => { log_rebalance_request_rejected("start", "decommission_in_progress", &request_id, &actor, &remote_addr); @@ -622,7 +639,7 @@ impl Operation for RebalanceStart { return Err(s3_error!(OperationAborted, "rebalance is already in progress")); } Err(e) => { - return Err(s3_error!(InternalError, "failed to initialize rebalance: {}", e)); + return Err(rebalance_init_failure_error(&e)); } }; @@ -639,7 +656,6 @@ impl Operation for RebalanceStart { "admin rebalance state" ); let notification_sys = current_notification_system(); - let fleet_proof_deadline = Instant::now() + REBALANCE_START_FLEET_PROOF_RETRY_BUDGET; for step in rebalance_start_steps(notification_sys.is_some()) { match step { RebalanceStartStep::PropagateFence => { @@ -1170,10 +1186,10 @@ mod rebalance_handler_tests { Body, HeaderMap, Method, Operation, Params, RebalPoolProgress, RebalanceAdminStatus, RebalancePoolStatus, RebalanceStart, RebalanceStartStep, RebalanceStatus, RebalanceStop, RebalanceStopPropagationStatus, S3ErrorCode, S3Request, StatusCode, Uri, build_rebalance_admin_status, build_rebalance_pool_statuses, build_rebalance_stop_propagation_status, - is_rebalance_start_fleet_proof_retryable, rebalance_pool_used, rebalance_query_present, rebalance_remaining_buckets, - rebalance_rollback_failure_message, rebalance_rollback_stop_failure_message, rebalance_start_failure_error, - rebalance_start_rollback_error, rebalance_start_steps, rebalance_stop_target_id, rebalance_used_pct, - retry_rebalance_start_operation, rollback_result_label, stop_rebalance_admission_first, + is_rebalance_start_fleet_proof_retryable, rebalance_init_failure_error, rebalance_pool_used, rebalance_query_present, + rebalance_remaining_buckets, rebalance_rollback_failure_message, rebalance_rollback_stop_failure_message, + rebalance_start_failure_error, rebalance_start_rollback_error, rebalance_start_steps, rebalance_stop_target_id, + rebalance_used_pct, retry_rebalance_start_operation, rollback_result_label, stop_rebalance_admission_first, }; use crate::admin::storage_api::error::StorageError; use crate::admin::storage_api::rebalance::{ @@ -1533,7 +1549,7 @@ mod rebalance_handler_tests { } #[tokio::test(start_paused = true)] - async fn test_rebalance_start_retry_waits_for_fleet_proof() { + async fn test_rebalance_init_retry_waits_for_fleet_proof() { use std::sync::{ Arc, atomic::{AtomicUsize, Ordering}, @@ -1544,12 +1560,12 @@ mod rebalance_handler_tests { let attempts = Arc::clone(&attempts); async move { retry_rebalance_start_operation( - "load_rebalance_meta(start=true)", - "rebalance-id", + "init_rebalance_start", + "", "request-id", "actor", "remote", - std::time::Instant::now() + super::REBALANCE_START_FLEET_PROOF_RETRY_BUDGET, + super::Instant::now() + super::REBALANCE_START_FLEET_PROOF_RETRY_BUDGET, move || { let attempts = Arc::clone(&attempts); async move { @@ -1557,7 +1573,7 @@ mod rebalance_handler_tests { if attempt == 0 { Err(StorageError::other(super::REBALANCE_START_FLEET_PROOF_MARKER)) } else { - Ok(()) + Ok("initialized-once") } } }, @@ -1570,12 +1586,78 @@ mod rebalance_handler_tests { assert_eq!(attempts.load(Ordering::SeqCst), 1); tokio::time::advance(super::REBALANCE_START_FLEET_PROOF_RETRY_DELAY).await; - task.await + let id = task + .await .expect("retry task should not panic") .expect("fleet proof retry should eventually succeed"); + assert_eq!(id, "initialized-once"); assert_eq!(attempts.load(Ordering::SeqCst), 2); } + #[tokio::test(start_paused = true)] + async fn test_rebalance_init_retry_exhaustion_returns_503_with_retry_after() { + let mut attempts = 0; + let started = super::Instant::now(); + let deadline = started + super::REBALANCE_START_FLEET_PROOF_RETRY_BUDGET; + let err = retry_rebalance_start_operation::<(), _, _>( + "init_rebalance_start", + "", + "request-id", + "actor", + "remote", + deadline, + || { + attempts += 1; + std::future::ready(Err(StorageError::other(super::REBALANCE_START_FLEET_PROOF_MARKER))) + }, + ) + .await + .expect_err("a missing proof must exhaust the bounded retry budget"); + + assert_eq!(super::Instant::now() - started, super::REBALANCE_START_FLEET_PROOF_RETRY_BUDGET); + assert_eq!(attempts, 7, "one immediate attempt plus six bounded readiness retries"); + let response = rebalance_init_failure_error(&err); + assert_eq!(response.status_code(), Some(StatusCode::SERVICE_UNAVAILABLE)); + assert_eq!( + response + .headers() + .and_then(|headers| headers.get(http::header::RETRY_AFTER)) + .and_then(|value| value.to_str().ok()), + Some(super::REBALANCE_START_RETRY_AFTER_SECS) + ); + } + + #[tokio::test(start_paused = true)] + async fn test_rebalance_init_does_not_retry_other_failures() { + for source in [ + StorageError::other("disk read failed"), + StorageError::other("pool activation fleet capability proof expired before commit"), + StorageError::RebalanceAlreadyRunning, + ] { + let original = source.to_string(); + let mut source = Some(source); + let started = super::Instant::now(); + let err = retry_rebalance_start_operation::<(), _, _>( + "init_rebalance_start", + "", + "request-id", + "actor", + "remote", + started + super::REBALANCE_START_FLEET_PROOF_RETRY_BUDGET, + || std::future::ready(Err(source.take().expect("non-readiness errors must not be retried"))), + ) + .await + .expect_err("non-readiness errors must be returned unchanged"); + assert_eq!(err.to_string(), original); + assert_eq!(super::Instant::now(), started, "non-readiness failures must not wait"); + if !matches!(err, StorageError::RebalanceAlreadyRunning) { + let response = rebalance_init_failure_error(&err); + assert_eq!(response.code(), &S3ErrorCode::InternalError); + assert!(response.message().is_some_and(|message| message.contains(&original))); + } + } + } + #[test] fn test_rebalance_rollback_stop_failure_message_lists_failures() { let failures = vec![ diff --git a/rustfs/src/admin/handlers/site_replication.rs b/rustfs/src/admin/handlers/site_replication.rs index 7adaf84b6..ec3abcfa8 100644 --- a/rustfs/src/admin/handlers/site_replication.rs +++ b/rustfs/src/admin/handlers/site_replication.rs @@ -2063,34 +2063,32 @@ fn open_id_settings(providers: Vec<(String, OpenIDProviderSettings)>, region: St settings } -fn open_id_settings_from_snapshot(snapshot: &rustfs_iam::oidc::OidcConfigSnapshot, region: String) -> OpenIDSettings { +fn open_id_settings_from_snapshot(snapshot: &rustfs_iam::oidc::OidcSiteReplicationSnapshot, region: String) -> OpenIDSettings { let providers = snapshot .providers() .iter() .map(|provider| { - let config = &provider.config; - (config.id.clone(), open_id_provider_settings(config)) + ( + provider.provider_id.clone(), + OpenIDProviderSettings { + claim_name: provider.claim_name.clone(), + claim_userinfo_enabled: false, + role_policy: provider.role_policy.clone(), + client_id: provider.client_id.clone(), + hashed_client_secret: provider.hashed_client_secret.clone(), + }, + ) }) .collect(); open_id_settings(providers, region) } -fn open_id_provider_settings(config: &rustfs_iam::oidc::OidcProviderConfig) -> OpenIDProviderSettings { - OpenIDProviderSettings { - claim_name: config.claim_name.clone(), - claim_userinfo_enabled: false, - role_policy: config.role_policy.clone(), - client_id: config.client_id.clone(), - hashed_client_secret: hash_client_secret(config.client_secret.as_deref()), - } -} - fn local_idp_settings() -> IDPSettings { let mut settings = IDPSettings::default(); let region = current_region().map(|region| region.to_string()).unwrap_or_default(); settings.open_id = current_oidc_config_query() - .map(|query| open_id_settings_from_snapshot(&query.config_snapshot(), region)) + .map(|query| open_id_settings_from_snapshot(&query.site_replication_snapshot(), region)) .unwrap_or_default(); let (ldap, ldap_configs) = load_ldap_idp_settings(); @@ -11547,16 +11545,18 @@ mod tests { #[test] fn oidc_config_snapshot_preserves_site_replication_settings() { - let snapshot = rustfs_iam::oidc::OidcConfigSnapshot::new(vec![ - oidc_snapshot_provider("corp", "client-b"), - oidc_snapshot_provider("default", "client-a"), - ]); + let snapshot = rustfs_iam::oidc::OidcSiteReplicationSnapshot::from_config_snapshot( + &rustfs_iam::oidc::OidcConfigSnapshot::new(vec![ + oidc_snapshot_provider("corp", "client-b"), + oidc_snapshot_provider("default", "client-a"), + ]), + ); assert_eq!( snapshot .providers() .iter() - .map(|provider| provider.config.id.as_str()) + .map(|provider| provider.provider_id.as_str()) .collect::>(), ["default", "corp"] ); diff --git a/rustfs/src/admin/handlers/sts.rs b/rustfs/src/admin/handlers/sts.rs index 054a1d11c..75211130c 100644 --- a/rustfs/src/admin/handlers/sts.rs +++ b/rustfs/src/admin/handlers/sts.rs @@ -432,7 +432,7 @@ async fn handle_assume_role_with_web_identity(body: AssumeRoleRequest) -> S3Resu })?; let authorization = &session.authorization; let new_cred = &session.credentials; - let subject = authorization.claims.session_identity(); + let subject = authorization.session_identity(); let xml = assume_role_with_web_identity_response_xml(new_cred, &subject); diff --git a/rustfs/src/admin/handlers/table_catalog/maintenance.rs b/rustfs/src/admin/handlers/table_catalog/maintenance.rs index c0dc5fb61..38f3ab525 100644 --- a/rustfs/src/admin/handlers/table_catalog/maintenance.rs +++ b/rustfs/src/admin/handlers/table_catalog/maintenance.rs @@ -27,13 +27,51 @@ impl Operation for RestTableMetadataMaintenanceHandler { ensure_table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?; let request = read_json_body::(req.input).await?; let metadata_backend = table_catalog_backend_from_extensions(&req.extensions)?; - let store = table_catalog_object_store_from_extensions(&req.extensions)?; + let store = table_catalog_store_from_extensions(&req.extensions)?; let response = - table_metadata_maintenance_response(&store, &metadata_backend, &warehouse, &namespace, &table, request).await?; + configured_table_metadata_maintenance_response(&store, &metadata_backend, &warehouse, &namespace, &table, request) + .await?; build_json_response(StatusCode::OK, &response) } } +async fn configured_table_metadata_maintenance_response( + store: &crate::table_catalog::ConfiguredTableCatalogStore, + metadata_backend: &B, + bucket: &str, + namespace: &crate::table_catalog::Namespace, + table: &str, + request: TableMetadataMaintenanceRequest, +) -> S3Result +where + B: crate::table_catalog::TableCatalogObjectBackend + Clone, +{ + match store { + crate::table_catalog::ConfiguredTableCatalogStore::ObjectBacked(store) => { + table_metadata_maintenance_response(store, metadata_backend, bucket, namespace, table, request).await + } + crate::table_catalog::ConfiguredTableCatalogStore::DurableStrong(store) => { + if request.delete || request.commit_snapshot_expiration || request.commit_compaction { + return Err(s3_error!( + InvalidRequest, + "metadata maintenance deletion and snapshot expiration or compaction commits are not supported with durable-strong table catalog backing" + )); + } + store + .run_table_metadata_maintenance_planning( + bucket, + &namespace.public_name(), + table, + request.retain_recent_metadata_files, + request.snapshot_expiration, + request.compaction, + ) + .await + .map_err(catalog_store_error) + } + } +} + pub struct GetTableMaintenanceConfigHandler {} #[async_trait::async_trait] @@ -157,16 +195,32 @@ impl Operation for RunTableMaintenanceWorkerHandler { authorize_table_catalog_resource_request(&req, &resource, AdminAction::RunTableMaintenanceAction).await?; ensure_table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?; let request = read_json_body::(req.input).await?; + let condition = request.condition()?; let store = table_catalog_store_from_extensions(&req.extensions)?; - let response = store - .run_table_metadata_maintenance_worker_once( - &warehouse, - &namespace.public_name(), - &table, - request.worker_id().to_string(), - ) - .await - .map_err(catalog_store_error)?; + let response = match condition { + Some(condition) => { + store + .run_table_metadata_maintenance_worker_once_conditionally( + &warehouse, + &namespace.public_name(), + &table, + request.worker_id().to_string(), + &condition, + ) + .await + } + None => { + store + .run_table_metadata_maintenance_worker_once( + &warehouse, + &namespace.public_name(), + &table, + request.worker_id().to_string(), + ) + .await + } + } + .map_err(catalog_store_error)?; build_json_response(StatusCode::OK, &response) } } diff --git a/rustfs/src/admin/handlers/table_catalog/mod.rs b/rustfs/src/admin/handlers/table_catalog/mod.rs index 976d70532..130cea31d 100644 --- a/rustfs/src/admin/handlers/table_catalog/mod.rs +++ b/rustfs/src/admin/handlers/table_catalog/mod.rs @@ -427,12 +427,52 @@ impl TableMaintenanceSchedulerRunRequest { struct TableMaintenanceWorkerRunRequest { #[serde(default, rename = "worker-id")] worker_id: Option, + #[serde( + default, + rename = "expected-job-id", + deserialize_with = "deserialize_present_worker_condition_field" + )] + expected_job_id: Option, + #[serde( + default, + rename = "expected-attempt", + deserialize_with = "deserialize_present_worker_condition_field" + )] + expected_attempt: Option, +} + +// Explicit null must not turn a conditional request into an unconditional run. +fn deserialize_present_worker_condition_field<'de, D, T>(deserializer: D) -> Result, D::Error> +where + D: serde::Deserializer<'de>, + T: Deserialize<'de>, +{ + T::deserialize(deserializer).map(Some) } impl TableMaintenanceWorkerRunRequest { fn worker_id(&self) -> &str { self.worker_id.as_deref().unwrap_or(DEFAULT_TABLE_MAINTENANCE_WORKER_ID) } + + fn condition(&self) -> S3Result> { + match (&self.expected_job_id, self.expected_attempt) { + (None, None) => Ok(None), + (Some(job_id), Some(attempt)) => { + if job_id.trim().is_empty() || self.worker_id().trim().is_empty() { + return Err(s3_error!(InvalidRequest, "conditional maintenance requires non-empty job and worker ids")); + } + Ok(Some(crate::table_catalog::TableMaintenanceWorkerCondition { + expected_job_id: job_id.clone(), + expected_attempt: attempt, + })) + } + _ => Err(s3_error!( + InvalidRequest, + "expected-job-id and expected-attempt must be supplied together" + )), + } + } } #[derive(Debug, Deserialize)] @@ -1862,6 +1902,16 @@ where .then(|| publication.guards.iter().all(|guard| !guard.is_lock_lost())) } + fn table_commit_object_mutation_fence(&self) -> Option { + let publication = self.publication.lock(); + Some(crate::table_catalog::TableCatalogObjectMutationFence::from_signals( + publication + .guards + .iter() + .map(crate::table_catalog::TableCatalogLockGuard::lock_lost_signal), + )) + } + fn complete_table_commit_publication(&self) { self.complete_publication(); } @@ -5497,7 +5547,8 @@ fn catalog_store_error(err: crate::table_catalog::TableCatalogStoreError) -> S3E crate::table_catalog::TableCatalogStoreError::Unsupported(message) => { iceberg_rest_error(ICEBERG_ERROR_UNSUPPORTED_OPERATION, StatusCode::NOT_ACCEPTABLE, message) } - crate::table_catalog::TableCatalogStoreError::Unavailable(_) => iceberg_rest_error( + crate::table_catalog::TableCatalogStoreError::Unavailable(_) + | crate::table_catalog::TableCatalogStoreError::LockUnavailable { .. } => iceberg_rest_error( ICEBERG_ERROR_REST, StatusCode::SERVICE_UNAVAILABLE, "table catalog is temporarily unavailable", diff --git a/rustfs/src/admin/handlers/table_catalog/tests.rs b/rustfs/src/admin/handlers/table_catalog/tests.rs index 9ce007d95..7580f3ac5 100644 --- a/rustfs/src/admin/handlers/table_catalog/tests.rs +++ b/rustfs/src/admin/handlers/table_catalog/tests.rs @@ -87,6 +87,26 @@ async fn call_load_table_handler(request: S3Request) -> S3Result) -> S3Result> { + let mut router = matchit::Router::new(); + router + .insert("/iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run", ()) + .expect("worker test route should insert"); + let path = request.uri.path().to_string(); + let matched = router.at(&path).expect("worker test route should match"); + RunTableMaintenanceWorkerHandler {}.call(request, matched.params).await +} + +async fn call_metadata_maintenance_handler(request: S3Request) -> S3Result> { + let mut router = matchit::Router::new(); + router + .insert("/iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/metadata", ()) + .expect("maintenance test route should insert"); + let path = request.uri.path().to_string(); + let matched = router.at(&path).expect("maintenance test route should match"); + RestTableMetadataMaintenanceHandler {}.call(request, matched.params).await +} + #[tokio::test] async fn commit_backend_borrowed_put_preserves_owner_authorization_and_precondition() { use crate::admin::storage_api::object::WriteCommitGuard; @@ -246,6 +266,539 @@ async fn table_catalog_authentication_and_credentials_use_the_request_context() let resolved_store = runtime_sources::object_store_from_extensions(&request.extensions).expect("request object store should resolve"); assert!(Arc::ptr_eq(&resolved_store, &store)); + + let mut request = request; + request.method = Method::POST; + request.uri = "/iceberg/v1/request-warehouse/namespaces/analytics/tables/events/maintenance/worker/run" + .parse() + .expect("worker URI"); + request.input = Body::from(br#"{"expected-job-id":"job-a","expected-attempt":0}"#.to_vec()); + let err = call_maintenance_worker_handler(request) + .await + .expect_err("table read permission must not authorize a conditional maintenance worker"); + assert_eq!(err.code(), &S3ErrorCode::AccessDenied); +} + +#[tokio::test] +#[serial_test::serial] +async fn table_maintenance_worker_handler_enforces_conditions_after_authorization() { + use crate::admin::storage_api::RUSTFS_META_BUCKET; + use crate::table_catalog::{ + TableCatalogObject, TableMaintenanceConfig, TableMetadataMaintenanceJobStatus, TableMetadataMaintenanceReport, + }; + + async fn inventory(backend: &B) -> BTreeMap<(String, String), TableCatalogObject> { + let mut objects = BTreeMap::new(); + for bucket in ["warehouse", RUSTFS_META_BUCKET] { + for object in backend.list_objects(bucket, "").await.expect("list isolated fixture objects") { + let stored = backend + .read_object(bucket, &object) + .await + .expect("read fixture object") + .expect("listed object exists"); + objects.insert((bucket.to_string(), object), stored); + } + } + objects + } + + temp_env::async_with_vars( + [( + crate::table_catalog::ENV_TABLE_CATALOG_BACKING, + Some(crate::table_catalog::TABLE_CATALOG_BACKING_OBJECT), + )], + async { + let (_temp_dir, _disk_paths, object_store) = crate::app::gating_test_env::isolated_multi_pool_ecstore().await; + object_store + .make_bucket("warehouse", &MakeBucketOptions::default()) + .await + .expect("create table bucket"); + rustfs_iam::store::object::ObjectStore::new(object_store.clone()) + .save_iam_config( + serde_json::json!({"version": 1}), + format!("{}/format.json", *rustfs_iam::store::object::IAM_CONFIG_PREFIX), + ) + .await + .expect("seed request IAM format"); + let iam = rustfs_iam::build_iam_sys(object_store.clone()) + .await + .expect("initialize request IAM"); + let context = Arc::new(AppContext::new( + object_store.clone(), + Arc::new(RequestIam { handle: iam }), + Arc::new(RequestKms), + )); + let access_key = "maintenance-handler-root"; + let secret_key = "maintenance-handler-root-secret"; + assert!(context.publish_action_credentials(rustfs_credentials::Credentials { + access_key: access_key.to_string(), + secret_key: secret_key.to_string(), + status: "on".to_string(), + ..Default::default() + })); + let request = |body: serde_json::Value| { + let mut request = table_catalog_handler_request(context.clone(), access_key, secret_key); + request.method = Method::POST; + request.uri = "/iceberg/v1/warehouse/namespaces/analytics/tables/events/maintenance/worker/run" + .parse() + .expect("worker URI"); + request.input = Body::from(serde_json::to_vec(&body).expect("worker request JSON")); + request + }; + let setup = request(serde_json::json!({})); + let backend = table_catalog_backend_from_extensions(&setup.extensions).expect("request catalog backend"); + let catalog = table_catalog_store_from_backend(backend.clone()).expect("object-backed catalog"); + enable_table_bucket_marker(object_store.as_ref(), "warehouse") + .await + .expect("enable table bucket"); + ensure_table_bucket_entry(&catalog, "warehouse", true) + .await + .expect("create table bucket entry"); + let namespace = crate::table_catalog::Namespace::parse("analytics").expect("namespace"); + create_namespace_response( + &catalog, + "warehouse", + CreateNamespaceRequest { + namespace: vec!["analytics".to_string()], + properties: BTreeMap::new(), + }, + true, + ) + .await + .expect("create namespace"); + let create_request = serde_json::from_value::(serde_json::json!({ + "name": "events", + "schema": { + "type": "struct", + "schema-id": 0, + "fields": [{"id": 1, "name": "id", "required": true, "type": "long"}] + } + })) + .expect("create table request"); + create_table_response( + &catalog, + &TableCommitObjectBackend::trusted(backend.clone()), + "warehouse", + &namespace, + create_request, + true, + ) + .await + .expect("create table and real metadata"); + catalog + .put_table_maintenance_config( + "warehouse", + "analytics", + "events", + TableMaintenanceConfig { + background_enabled: true, + ..Default::default() + }, + ) + .await + .expect("enable background maintenance"); + let queued = catalog + .run_table_maintenance_scheduler_once("warehouse", "analytics", "events", "handler-scheduler".to_string()) + .await + .expect("queue maintenance") + .report; + assert_eq!(queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(queued.job.attempt, 0); + let entry = catalog + .load_table("warehouse", "analytics", "events") + .await + .expect("load table") + .expect("table exists"); + let before = inventory(&backend).await; + + for body in [ + serde_json::json!({"worker-id":"worker-a", "expected-job-id":"missing-job", "expected-attempt":0}), + serde_json::json!({"worker-id":"worker-a", "expected-job-id":queued.job.job_id, "expected-attempt":1}), + ] { + let error = call_maintenance_worker_handler(request(body)) + .await + .expect_err("stale condition must not dispatch an unconditional worker"); + assert_eq!(error.status_code(), Some(StatusCode::CONFLICT)); + for id in [queued.job.job_id.as_str(), "latest", "current"] { + assert_eq!( + catalog + .get_table_metadata_maintenance_report("warehouse", "analytics", "events", id) + .await + .unwrap(), + Some(queued.clone()), + ); + } + assert_eq!(catalog.load_table("warehouse", "analytics", "events").await.unwrap(), Some(entry.clone())); + assert_eq!(inventory(&backend).await, before); + } + + let claim = serde_json::json!({ + "worker-id":"worker-a", "expected-job-id":queued.job.job_id, "expected-attempt":0 + }); + let response = call_maintenance_worker_handler(request(claim.clone())) + .await + .expect("authorized conditional claim"); + assert_eq!(response.output.0, StatusCode::OK); + let completed: TableMetadataMaintenanceReport = + serde_json::from_slice(&response.output.1.bytes().expect("buffered worker response")).expect("worker report"); + assert_eq!(completed.job.job_id, queued.job.job_id); + assert_eq!(completed.job.attempt, 1); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.worker_id.as_deref(), Some("worker-a")); + for id in [queued.job.job_id.as_str(), "latest", "current"] { + assert_eq!( + catalog + .get_table_metadata_maintenance_report("warehouse", "analytics", "events", id) + .await + .unwrap(), + Some(completed.clone()), + ); + } + assert_eq!(catalog.load_table("warehouse", "analytics", "events").await.unwrap(), Some(entry.clone())); + let committed = inventory(&backend).await; + let response = call_maintenance_worker_handler(request(claim)) + .await + .expect("same-worker replay"); + assert_eq!(response.output.0, StatusCode::OK); + let replay: TableMetadataMaintenanceReport = + serde_json::from_slice(&response.output.1.bytes().expect("buffered replay response")).expect("replayed report"); + assert_eq!(replay, completed); + assert_eq!(inventory(&backend).await, committed); + + let error = call_maintenance_worker_handler(request(serde_json::json!({ + "worker-id":"worker-b", "expected-job-id":queued.job.job_id, "expected-attempt":0 + }))) + .await + .expect_err("another worker cannot replay an old condition"); + assert_eq!(error.status_code(), Some(StatusCode::CONFLICT)); + for id in [queued.job.job_id.as_str(), "latest", "current"] { + assert_eq!( + catalog + .get_table_metadata_maintenance_report("warehouse", "analytics", "events", id) + .await + .unwrap(), + Some(completed.clone()), + ); + } + assert_eq!(catalog.load_table("warehouse", "analytics", "events").await.unwrap(), Some(entry.clone())); + assert_eq!(inventory(&backend).await, committed); + + let response = call_maintenance_worker_handler(request(serde_json::json!({}))) + .await + .expect("legacy unconditional request remains supported"); + assert_eq!(response.output.0, StatusCode::OK); + let legacy: TableMetadataMaintenanceReport = + serde_json::from_slice(&response.output.1.bytes().expect("buffered legacy response")) + .expect("legacy worker report"); + assert_ne!(legacy.job.job_id, completed.job.job_id); + assert_eq!(legacy.job.worker_id.as_deref(), Some(DEFAULT_TABLE_MAINTENANCE_WORKER_ID)); + assert_eq!(legacy.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(catalog.load_table("warehouse", "analytics", "events").await.unwrap(), Some(entry)); + }, + ) + .await; +} + +#[tokio::test] +#[serial_test::serial] +async fn strong_metadata_maintenance_handler_plans_without_changing_table_objects() { + use crate::admin::storage_api::RUSTFS_META_BUCKET; + use crate::table_catalog::{ + TableCatalogObject, TableCatalogObjectPaths, TableMaintenanceAuditAction, TableMetadataMaintenanceJobStatus, + TableMetadataMaintenanceOperation, TableMetadataMaintenanceReport, + }; + + async fn inventory(backend: &B) -> BTreeMap<(String, String), TableCatalogObject> { + let mut objects = BTreeMap::new(); + for bucket in ["warehouse", RUSTFS_META_BUCKET] { + for object in backend.list_objects(bucket, "").await.expect("list isolated fixture objects") { + let stored = backend + .read_object(bucket, &object) + .await + .expect("read fixture object") + .expect("listed object exists"); + objects.insert((bucket.to_string(), object), stored); + } + } + objects + } + + temp_env::async_with_vars( + [( + crate::table_catalog::ENV_TABLE_CATALOG_BACKING, + Some(crate::table_catalog::TABLE_CATALOG_BACKING_DURABLE_STRONG), + )], + async { + let (_temp_dir, _disk_paths, object_store) = crate::app::gating_test_env::isolated_multi_pool_ecstore().await; + object_store + .make_bucket("warehouse", &MakeBucketOptions::default()) + .await + .expect("create table bucket"); + rustfs_iam::store::object::ObjectStore::new(object_store.clone()) + .save_iam_config( + serde_json::json!({"version": 1}), + format!("{}/format.json", *rustfs_iam::store::object::IAM_CONFIG_PREFIX), + ) + .await + .expect("seed request IAM format"); + let iam = rustfs_iam::build_iam_sys(object_store.clone()) + .await + .expect("initialize request IAM"); + let reader_access_key = "planning-handler-reader"; + let reader_secret_key = "planning-handler-reader-secret"; + iam.create_user( + reader_access_key, + &AddOrUpdateUserReq { + secret_key: reader_secret_key.to_string(), + policy: None, + status: AccountStatus::Enabled, + }, + ) + .await + .expect("create read-only user"); + iam.set_policy( + "planning-handler-read", + Policy::parse_config(br#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["admin:GetTable"]}]}"#) + .expect("read-only policy"), + ) + .await + .expect("save read-only policy"); + iam.policy_db_set(reader_access_key, UserType::Reg, false, "planning-handler-read") + .await + .expect("attach read-only policy"); + let context = Arc::new(AppContext::new( + object_store.clone(), + Arc::new(RequestIam { handle: iam }), + Arc::new(RequestKms), + )); + let access_key = "planning-handler-root"; + let secret_key = "planning-handler-root-secret"; + assert!(context.publish_action_credentials(rustfs_credentials::Credentials { + access_key: access_key.to_string(), + secret_key: secret_key.to_string(), + status: "on".to_string(), + ..Default::default() + })); + let request = |body: &str| { + let mut request = table_catalog_handler_request(context.clone(), access_key, secret_key); + request.method = Method::POST; + request.uri = "/iceberg/v1/warehouse/namespaces/analytics/tables/events/maintenance/metadata" + .parse() + .expect("maintenance URI"); + request.input = Body::from(body.as_bytes().to_vec()); + request + }; + let setup = request("{}"); + let backend = table_catalog_backend_from_extensions(&setup.extensions).expect("request catalog backend"); + enable_table_bucket_marker(object_store.as_ref(), "warehouse") + .await + .expect("enable table bucket"); + // The isolated ECStore does not run catalog startup. Seed its snapshot + // before using the production request path, which requires it to exist. + let bootstrap = crate::table_catalog::StrongTableCatalogStore::new(backend.clone()); + ensure_table_bucket_entry(&bootstrap, "warehouse", true) + .await + .expect("seed durable snapshot and bucket entry"); + let catalog = table_catalog_store_from_backend(backend.clone()).expect("strong catalog"); + assert!(matches!(&catalog, crate::table_catalog::ConfiguredTableCatalogStore::DurableStrong(_))); + ensure_table_bucket_entry(&catalog, "warehouse", true) + .await + .expect("read bootstrapped bucket through configured catalog"); + let namespace = crate::table_catalog::Namespace::parse("analytics").expect("namespace"); + let table = crate::table_catalog::IdentifierSegment::parse("events").expect("table"); + create_namespace_response( + &catalog, + "warehouse", + CreateNamespaceRequest { + namespace: vec!["analytics".to_string()], + properties: BTreeMap::new(), + }, + true, + ) + .await + .expect("create namespace"); + let create_request = serde_json::from_value::(serde_json::json!({ + "name": "events", + "schema": {"type": "struct", "schema-id": 0, + "fields": [{"id": 1, "name": "id", "required": true, "type": "long"}]} + })) + .expect("create request"); + create_table_response( + &catalog, + &TableCommitObjectBackend::trusted(backend.clone()), + "warehouse", + &namespace, + create_request, + true, + ) + .await + .expect("create table and metadata"); + let entry = catalog.load_table("warehouse", "analytics", "events").await.unwrap().unwrap(); + let before = inventory(&backend).await; + + for body in [ + r#"{"delete":true}"#, + r#"{"commit-snapshot-expiration":true}"#, + r#"{"commit-compaction":true}"#, + ] { + let error = call_metadata_maintenance_handler(request(body)) + .await + .expect_err("strong mutation must be rejected"); + assert_eq!(error.code(), &S3ErrorCode::InvalidRequest); + assert!( + error + .message() + .unwrap_or_default() + .contains("not supported with durable-strong") + ); + assert_eq!(inventory(&backend).await, before, "{body}"); + } + for body in [ + r#"{"delete":null}"#, + r#"{"retain-recent-metadata-files":-1}"#, + r#"{"snapshot-expiration":{"min-snapshots-to-keep":"1","max-snapshot-age-ms":1000}}"#, + r#"{"compaction":true}"#, + r#"{"force":true}"#, + r#"{"delete":false,"delete":true}"#, + ] { + let error = call_metadata_maintenance_handler(request(body)) + .await + .expect_err("malformed request must be rejected"); + assert_eq!(error.code(), &S3ErrorCode::InvalidRequest); + assert_eq!(inventory(&backend).await, before, "{body}"); + } + let mut anonymous = request("{}"); + anonymous.credentials = None; + let error = call_metadata_maintenance_handler(anonymous) + .await + .expect_err("planning requires authentication"); + assert_eq!(error.code(), &S3ErrorCode::InvalidRequest); + assert!(error.message().unwrap_or_default().contains("authentication required")); + let mut reader = request("not-json"); + reader.credentials = Some(s3s::auth::Credentials { + access_key: reader_access_key.to_string(), + secret_key: s3s::auth::SecretKey::from(reader_secret_key.to_string()), + }); + let error = call_metadata_maintenance_handler(reader) + .await + .expect_err("read-only permission must not authorize planning"); + assert_eq!(error.code(), &S3ErrorCode::AccessDenied); + assert_eq!(inventory(&backend).await, before); + + for body in [ + serde_json::json!({}), + serde_json::json!({ + "retain-recent-metadata-files": 2, + "snapshot-expiration": {"min-snapshots-to-keep": 1, "max-snapshot-age-ms": 1000}, + "compaction": {"target-file-size-bytes": 1024, "small-file-threshold-bytes": 512, + "min-input-files": 2, "max-rewrite-bytes-per-job": 4096} + }), + ] { + let before = inventory(&backend).await; + let response = call_metadata_maintenance_handler(request(&body.to_string())) + .await + .expect("authorized strong planning"); + assert_eq!(response.output.0, StatusCode::OK); + let report: TableMetadataMaintenanceReport = + serde_json::from_slice(&response.output.1.bytes().expect("buffered planning response")) + .expect("planning report"); + assert_eq!(report.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(report.job.operation, TableMetadataMaintenanceOperation::DryRun); + assert_eq!(report.job.table_id, entry.table_id); + assert_eq!(report.current_metadata_location, entry.metadata_location); + assert_eq!(report.job.current_generation, entry.generation); + assert_eq!(report.job.deleted_metadata_file_count, 0); + assert_eq!(report.job.deleted_object_count, 0); + assert_eq!( + report.job.retain_recent_metadata_files, + if body.get("compaction").is_some() { 2 } else { 0 } + ); + assert_eq!(report.snapshot_expiration.is_some(), body.get("snapshot-expiration").is_some()); + assert_eq!(report.compaction.is_some(), body.get("compaction").is_some()); + if let Some(plan) = &report.snapshot_expiration { + assert_eq!(plan.config.min_snapshots_to_keep, 1); + assert_eq!(plan.config.max_snapshot_age_ms, 1000); + assert!(plan.committed_metadata_location.is_none()); + } + if let Some(plan) = &report.compaction { + assert_eq!(plan.config.target_file_size_bytes, 1024); + assert_eq!(plan.config.max_rewrite_bytes_per_job, 4096); + assert!(plan.committed_metadata_location.is_none()); + } + assert!( + report + .audit_events + .iter() + .any(|event| event.action == TableMaintenanceAuditAction::WorkerStarted) + ); + assert!( + report + .audit_events + .iter() + .any(|event| event.action == TableMaintenanceAuditAction::WorkerSucceeded) + ); + let reloaded = table_catalog_store_from_backend(backend.clone()).expect("reload strong catalog"); + for id in [report.job.job_id.as_str(), "latest", "current"] { + assert_eq!( + reloaded + .get_table_metadata_maintenance_report("warehouse", "analytics", "events", id) + .await + .unwrap(), + Some(report.clone()) + ); + } + assert_eq!( + reloaded.load_table("warehouse", "analytics", "events").await.unwrap(), + Some(entry.clone()) + ); + let paths = TableCatalogObjectPaths::default(); + let report_paths = [ + paths.table_maintenance_job_path("warehouse", &namespace, &table, &entry.table_id, &report.job.job_id), + paths.table_maintenance_latest_job_path("warehouse", &namespace, &table, &entry.table_id), + paths.table_maintenance_current_job_path("warehouse", &namespace, &table, &entry.table_id), + ]; + let outside_report = |((bucket, key), _): &((String, String), TableCatalogObject)| { + bucket != RUSTFS_META_BUCKET || !report_paths.contains(key) + }; + assert_eq!( + inventory(&backend) + .await + .into_iter() + .filter(outside_report) + .collect::>(), + before.into_iter().filter(outside_report).collect::>(), + "planning may only publish its named, latest, and current audit reports", + ); + } + + catalog + .put_table_maintenance_config( + "warehouse", + "analytics", + "events", + crate::table_catalog::TableMaintenanceConfig { + background_enabled: true, + ..Default::default() + }, + ) + .await + .expect("enable queue admission"); + let queued = catalog + .run_table_maintenance_scheduler_once("warehouse", "analytics", "events", "planning-test-scheduler".to_string()) + .await + .expect("queue a maintenance job") + .report; + assert_eq!(queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + let before = inventory(&backend).await; + let error = call_metadata_maintenance_handler(request("{}")) + .await + .expect_err("planning must not replace a queued job"); + assert_eq!(error.status_code(), Some(StatusCode::CONFLICT)); + assert_eq!(inventory(&backend).await, before); + assert_eq!(catalog.load_table("warehouse", "analytics", "events").await.unwrap(), Some(entry)); + }, + ) + .await; } #[tokio::test] @@ -646,17 +1199,25 @@ fn catalog_conflicts_use_operation_specific_iceberg_errors() { #[test] fn catalog_unavailable_errors_use_iceberg_503() { - let unavailable = catalog_store_error(crate::table_catalog::TableCatalogStoreError::Unavailable( - "failed to acquire catalog table lock: quorum required 3, achieved 1".to_string(), - )); - assert_eq!(unavailable.code(), &S3ErrorCode::Custom(ICEBERG_ERROR_REST.into())); - assert_eq!(unavailable.status_code(), Some(StatusCode::SERVICE_UNAVAILABLE)); - assert_eq!(unavailable.message(), Some("table catalog is temporarily unavailable")); - assert!( - unavailable - .headers() - .is_none_or(|headers| !headers.contains_key(rustfs_utils::http::RETRY_AFTER)) - ); + for error in [ + crate::table_catalog::TableCatalogStoreError::Unavailable( + "failed to acquire catalog table lock: quorum required 3, achieved 1".to_string(), + ), + crate::table_catalog::TableCatalogStoreError::LockUnavailable { + message: "private publication fence".to_string(), + source: Box::new(rustfs_lock::LockError::timeout("private/lock", std::time::Duration::from_secs(5))), + }, + ] { + let unavailable = catalog_store_error(error); + assert_eq!(unavailable.code(), &S3ErrorCode::Custom(ICEBERG_ERROR_REST.into())); + assert_eq!(unavailable.status_code(), Some(StatusCode::SERVICE_UNAVAILABLE)); + assert_eq!(unavailable.message(), Some("table catalog is temporarily unavailable")); + assert!( + unavailable + .headers() + .is_none_or(|headers| !headers.contains_key(rustfs_utils::http::RETRY_AFTER)) + ); + } let internal = catalog_store_error(crate::table_catalog::TableCatalogStoreError::Internal( "commit state is unknown".to_string(), @@ -724,6 +1285,7 @@ fn table_catalog_handlers_require_table_admin_actions() { ("GetTableMaintenanceJobHandler", "AdminAction::GetTableLifecycleAction"), ("GetTableMaintenanceSchedulerHandler", "AdminAction::GetTableLifecycleAction"), ("RunTableMaintenanceSchedulerHandler", "AdminAction::RunTableMaintenanceAction"), + ("RunTableMaintenanceWorkerHandler", "AdminAction::RunTableMaintenanceAction"), ("TableMaintenanceQuarantineHandler", "AdminAction::RunTableMaintenanceAction"), ("ExportTableCatalogHandler", "AdminAction::GetTableMetadataAction"), ("ImportTableCatalogHandler", "AdminAction::RegisterTableAction"), @@ -810,6 +1372,7 @@ fn table_catalog_handlers_require_table_admin_actions() { ("GetTableMaintenanceJobHandler", "AdminAction::GetTableLifecycleAction"), ("GetTableMaintenanceSchedulerHandler", "AdminAction::GetTableLifecycleAction"), ("RunTableMaintenanceSchedulerHandler", "AdminAction::RunTableMaintenanceAction"), + ("RunTableMaintenanceWorkerHandler", "AdminAction::RunTableMaintenanceAction"), ("TableMaintenanceQuarantineHandler", "AdminAction::RunTableMaintenanceAction"), ("ExportTableCatalogHandler", "AdminAction::GetTableMetadataAction"), ("ImportTableCatalogHandler", "AdminAction::RegisterTableAction"), @@ -1026,6 +1589,99 @@ fn table_commit_publication_acquires_migration_permits_before_publication_lock() ); } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn table_commit_publication_forwards_lease_loss_to_storage_mutation_fence() { + use crate::admin::storage_api::RUSTFS_META_BUCKET; + use crate::table_catalog::test_support::RecordingLeaseClient; + use crate::table_catalog::{TableCatalogLockGuard, TableCatalogObjectPaths, TableCommitPublication}; + use rustfs_lock::LockClient; + + let metadata_backend = TestTableCatalogObjectBackend::content_addressed(); + let commit_backend = TableCommitObjectBackend::trusted(metadata_backend.clone()); + TableCommitPublication::begin_table_bucket(&commit_backend, "analytics") + .await + .unwrap(); + TableCommitPublication::prepare(&commit_backend, "analytics", "sales", "orders") + .await + .unwrap(); + let client = Arc::new(RecordingLeaseClient { + inner: rustfs_lock::LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::Enabled(Arc::new( + rustfs_lock::FastObjectLockManager::new(), + )))), + acquired: std::sync::Mutex::new(None), + }); + let lock = rustfs_lock::NamespaceLockWrapper::new( + rustfs_lock::NamespaceLock::with_clients_and_quorum( + "rest-publication-fence-probe".to_string(), + vec![client.clone() as Arc], + 1, + ), + rustfs_lock::ObjectKey::new( + RUSTFS_META_BUCKET, + TableCatalogObjectPaths::default().backing_migration_global_fence_lock_path(), + ), + "rest-publication-migration".to_string(), + ); + let guard = lock.get_read_lock(StdDuration::from_secs(5)).await.unwrap(); + // The guard exposes an aggregate ID; LocalClient owns the recorded response ID. + let lock_id = client + .acquired + .lock() + .unwrap() + .clone() + .expect("local migration lease has an ID"); + assert!(client.check_status(&lock_id).await.unwrap().is_some()); + { + let mut publication = commit_backend.publication.lock(); + publication.guards.push(TableCatalogLockGuard::test_namespace(guard)); + publication.catalog_migration_read_permit = true; + } + let fence = TableCommitPublication::object_mutation_fence(&commit_backend) + .expect("REST publication must forward its held guards to the storage fence"); + fence.ensure_held().unwrap(); + assert!(TableCommitPublication::holds_table_bucket(&commit_backend, "analytics")); + assert!(TableCommitPublication::holds_table(&commit_backend, "analytics", "sales", "orders")); + assert_eq!(TableCommitPublication::catalog_migration_read_permit_status(&commit_backend), Some(true)); + + let object = "rest-publication-fence-probe.json"; + metadata_backend + .seed_object(RUSTFS_META_BUCKET, object, b"before".to_vec()) + .await; + let before = metadata_backend + .read_object(RUSTFS_META_BUCKET, object) + .await + .unwrap() + .unwrap(); + let pause = metadata_backend.pause_next_put(RUSTFS_META_BUCKET, object).await; + let operation = metadata_backend.put_object_fenced( + RUSTFS_META_BUCKET, + object, + b"after".to_vec(), + crate::table_catalog::TableCatalogPutPrecondition::IfMatch(before.etag.clone().unwrap()), + &fence, + ); + let interfere = async { + tokio::time::timeout(StdDuration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + assert!(client.force_release(&lock_id).await.unwrap()); + tokio::time::timeout(StdDuration::from_secs(15), async { + while fence.ensure_held().is_ok() { + tokio::time::sleep(StdDuration::from_millis(10)).await; + } + }) + .await + .expect("REST publication fence observes migration lease loss during backend await"); + assert!(!TableCommitPublication::holds_table_bucket(&commit_backend, "analytics")); + assert!(!TableCommitPublication::holds_table(&commit_backend, "analytics", "sales", "orders")); + assert_eq!(TableCommitPublication::catalog_migration_read_permit_status(&commit_backend), Some(false)); + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(result.is_err()); + assert_eq!(metadata_backend.read_object(RUSTFS_META_BUCKET, object).await.unwrap(), Some(before)); +} + #[test] fn table_catalog_resource_builds_policy_object_scope() { let namespace = crate::table_catalog::Namespace::parse("analytics.daily_events").expect("namespace should parse"); @@ -1271,6 +1927,7 @@ fn table_maintenance_worker_run_request_uses_stable_default_worker_id() { serde_json::from_value(serde_json::json!({})).expect("worker run request should parse"); assert_eq!(request.worker_id(), "rustfs-maintenance-worker"); + assert!(request.condition().expect("legacy request").is_none()); } #[test] @@ -1281,6 +1938,84 @@ fn table_maintenance_worker_run_request_accepts_worker_id() { .expect("worker run request should parse worker id"); assert_eq!(request.worker_id(), "worker-a"); + assert!(request.condition().expect("legacy request").is_none()); +} + +#[test] +fn table_maintenance_worker_condition_accepts_paired_fields() { + for attempt in [0, 1, u16::MAX] { + let request: TableMaintenanceWorkerRunRequest = serde_json::from_value(serde_json::json!({ + "expected-job-id": "job-a", "expected-attempt": attempt + })) + .expect("paired condition parses"); + let condition = request + .condition() + .expect("valid condition") + .expect("condition remains present"); + assert_eq!(condition.expected_job_id, "job-a"); + assert_eq!(condition.expected_attempt, attempt); + } +} + +#[test] +fn table_maintenance_worker_condition_rejects_incomplete_or_empty_fields() { + for value in [ + serde_json::json!({"expected-job-id":"job-a"}), + serde_json::json!({"expected-attempt":0}), + serde_json::json!({"expected-job-id":"", "expected-attempt":0}), + serde_json::json!({"expected-job-id":" ", "expected-attempt":0}), + serde_json::json!({"worker-id":" ", "expected-job-id":"job-a", "expected-attempt":0}), + ] { + let request: TableMaintenanceWorkerRunRequest = serde_json::from_value(value).expect("well-typed request"); + let err = request + .condition() + .expect_err("partial or empty condition must not run unconditionally"); + assert_eq!(err.code(), &S3ErrorCode::InvalidRequest); + } +} + +#[test] +fn table_maintenance_worker_condition_rejects_null_malformed_duplicate_and_unknown_fields() { + for body in [ + r#"{"expected-job-id":null,"expected-attempt":null}"#, + r#"{"expected-job-id":null}"#, + r#"{"expected-attempt":null}"#, + r#"{"expected-job-id":"job-a","expected-attempt":-1}"#, + r#"{"expected-job-id":"job-a","expected-attempt":65536}"#, + r#"{"expected-job-id":"job-a","expected-attempt":0.5}"#, + r#"{"expected-job-id":"job-a","expected-attempt":"0"}"#, + r#"{"expected-job-id":"job-a","expected-attempt":false}"#, + r#"{"expected-job-id":1,"expected-attempt":0}"#, + r#"{"expected-job-id":"job-a","expected-job-id":"job-b","expected-attempt":0}"#, + r#"{"expected-job-id":"job-a","expected-attempt":0,"expected-attempt":1}"#, + r#"{"expected-job-id":"job-a","expected-attempt":0,"force":true}"#, + ] { + assert!(serde_json::from_str::(body).is_err(), "{body}"); + } +} + +#[tokio::test] +async fn table_maintenance_worker_handler_requires_authentication_for_both_request_forms() { + for body in [r#"{}"#, r#"{"expected-job-id":"job-a","expected-attempt":0}"#] { + let request = S3Request { + input: Body::from(body.as_bytes().to_vec()), + method: Method::POST, + uri: "/iceberg/v1/warehouse/namespaces/analytics/tables/events/maintenance/worker/run" + .parse() + .expect("worker URI"), + headers: HeaderMap::new(), + extensions: http::Extensions::new(), + credentials: None, + region: None, + service: None, + trailing_headers: None, + }; + let err = call_maintenance_worker_handler(request) + .await + .expect_err("worker requires authentication"); + assert_eq!(err.code(), &S3ErrorCode::InvalidRequest); + assert!(err.message().unwrap_or_default().contains("authentication required")); + } } #[test] @@ -4211,24 +4946,46 @@ fn table_metadata_file_name_scoping_is_bounded_and_identity_sensitive() { #[tokio::test] async fn renamed_and_recreated_tables_with_the_same_commit_id_use_disjoint_metadata_files() { + assert_renamed_and_recreated_table_commits(crate::table_catalog::TableCatalogBackingMode::DurableStrong, "curated").await; +} + +#[tokio::test] +async fn object_catalog_cross_namespace_rename_accepts_isolated_standard_commits() { + assert_renamed_and_recreated_table_commits(crate::table_catalog::TableCatalogBackingMode::ObjectBacked, "curated").await; +} + +#[tokio::test] +async fn object_catalog_same_namespace_rename_accepts_isolated_standard_commits() { + assert_renamed_and_recreated_table_commits(crate::table_catalog::TableCatalogBackingMode::ObjectBacked, "analytics").await; +} + +#[tokio::test] +async fn strong_catalog_same_namespace_rename_accepts_isolated_standard_commits() { + assert_renamed_and_recreated_table_commits(crate::table_catalog::TableCatalogBackingMode::DurableStrong, "analytics").await; +} + +async fn assert_renamed_and_recreated_table_commits(mode: crate::table_catalog::TableCatalogBackingMode, destination: &str) { let metadata_backend = TestTableCatalogObjectBackend::content_addressed(); - let store = crate::table_catalog::StrongTableCatalogStore::new(metadata_backend.clone()); + let store = crate::table_catalog::ConfiguredTableCatalogStore::new_for_test(metadata_backend.clone(), mode); let source_namespace = crate::table_catalog::Namespace::parse("analytics").expect("source namespace should parse"); - let destination_namespace = crate::table_catalog::Namespace::parse("curated").expect("destination namespace should parse"); + let destination_namespace = crate::table_catalog::Namespace::parse(destination).expect("destination namespace should parse"); create_standard_events_table(&store, &metadata_backend, &source_namespace).await; - create_namespace_response( - &store, - "warehouse", - CreateNamespaceRequest { - namespace: vec!["curated".to_string()], - properties: BTreeMap::new(), - }, - true, - ) - .await - .expect("destination namespace should be created"); + let original = store.load_table("warehouse", "analytics", "events").await.unwrap().unwrap(); + if source_namespace != destination_namespace { + create_namespace_response( + &store, + "warehouse", + CreateNamespaceRequest { + namespace: vec![destination.to_string()], + properties: BTreeMap::new(), + }, + true, + ) + .await + .expect("destination namespace should be created"); + } store - .rename_table("warehouse", "analytics", "events", "curated", "events_v2") + .rename_table("warehouse", "analytics", "events", destination, "events_v2") .await .expect("table should rename"); @@ -4249,7 +5006,7 @@ async fn renamed_and_recreated_tables_with_the_same_commit_id_use_disjoint_metad .expect("source identifier should be reusable"); let renamed = store - .load_table("warehouse", "curated", "events_v2") + .load_table("warehouse", destination, "events_v2") .await .expect("renamed table lookup should succeed") .expect("renamed table should exist"); @@ -4259,18 +5016,196 @@ async fn renamed_and_recreated_tables_with_the_same_commit_id_use_disjoint_metad .expect("recreated table lookup should succeed") .expect("recreated table should exist"); assert_ne!(renamed.table_id, recreated.table_id); + assert_ne!(renamed.table_uuid, recreated.table_uuid); + assert_ne!(renamed.warehouse_location, recreated.warehouse_location); + assert_eq!(renamed.table_id, original.table_id); + assert_eq!(renamed.table_uuid, original.table_uuid); + assert_eq!(renamed.warehouse_location, original.warehouse_location); + assert_eq!(renamed.metadata_location, original.metadata_location); + assert_eq!(renamed.version_token, original.version_token); + assert_eq!(renamed.generation, original.generation); let commit_id = "11111111-1111-4111-8111-111111111111"; + let renamed_request = || { + let mut request = standard_property_commit_request(commit_id, &renamed.table_uuid, "curated"); + request.idempotency_key = Some("renamed-request".to_string()); + request + }; let renamed_commit = standard_commit_table_response( &store, &trusted_table_commit_backend(&metadata_backend), "warehouse", &destination_namespace, "events_v2", - standard_property_commit_request(commit_id, &renamed.table_uuid, "curated"), + renamed_request(), ) .await .expect("renamed table commit should succeed"); + let committed = store + .load_table("warehouse", destination, "events_v2") + .await + .unwrap() + .unwrap(); + assert_eq!(committed.table_id, renamed.table_id); + assert_eq!(committed.table_uuid, renamed.table_uuid); + assert_eq!(committed.warehouse_location, renamed.warehouse_location); + assert_eq!(committed.generation, renamed.generation + 1); + assert_ne!(committed.version_token, renamed.version_token); + assert_ne!(committed.metadata_location, renamed.metadata_location); + assert_eq!(renamed_commit.metadata["table-uuid"], renamed.table_uuid); + assert_eq!(renamed_commit.metadata["location"], renamed.warehouse_location); + assert_eq!(renamed_commit.metadata["properties"]["owner"], "curated"); + let receipt = store + .get_commit_by_id("warehouse", &renamed.table_id, commit_id) + .await + .unwrap() + .unwrap(); + assert_eq!(receipt.status, crate::table_catalog::CommitLogStatus::Committed); + assert_eq!(receipt.new_version_token, committed.version_token); + assert_eq!(receipt.new_metadata_location, committed.metadata_location); + assert_eq!( + store + .get_commit_by_idempotency_key("warehouse", &renamed.table_id, receipt.idempotency_key.as_deref().unwrap()) + .await + .unwrap(), + Some(receipt.clone()) + ); + let replay = standard_commit_table_response( + &store, + &trusted_table_commit_backend(&metadata_backend), + "warehouse", + &destination_namespace, + "events_v2", + renamed_request(), + ) + .await + .expect("same commit ID should replay after rename"); + assert_eq!(replay.metadata_location, renamed_commit.metadata_location); + assert_eq!(replay.metadata, renamed_commit.metadata); + assert_eq!( + store.load_table("warehouse", destination, "events_v2").await.unwrap(), + Some(committed.clone()) + ); + + let stale_request = crate::table_catalog::TableCommitRequest { + table_bucket: "warehouse".to_string(), + namespace: destination.to_string(), + table: "events_v2".to_string(), + commit_id: "stale-after-rename".to_string(), + idempotency_key: Some("stale-after-rename".to_string()), + operation: "append".to_string(), + expected_version_token: renamed.version_token.clone(), + expected_metadata_location: renamed.metadata_location.clone(), + new_metadata_location: committed.metadata_location.clone(), + requirements: Vec::new(), + writer: None, + }; + for metadata_only in [false, true] { + let mut request = stale_request.clone(); + if metadata_only { + request.expected_version_token = committed.version_token.clone(); + request.commit_id = "stale-metadata-after-rename".to_string(); + request.idempotency_key = Some(request.commit_id.clone()); + } + assert!(matches!( + store.commit_table(request.clone()).await, + Err(crate::table_catalog::TableCatalogStoreError::Conflict(_)) + )); + assert!( + store + .get_commit_by_id("warehouse", &renamed.table_id, &request.commit_id) + .await + .unwrap() + .is_none() + ); + assert!( + store + .get_commit_by_idempotency_key("warehouse", &renamed.table_id, request.idempotency_key.as_deref().unwrap()) + .await + .unwrap() + .is_none() + ); + assert_eq!( + store.load_table("warehouse", destination, "events_v2").await.unwrap(), + Some(committed.clone()) + ); + } + let current_namespace_path = crate::table_catalog::default_table_metadata_dir_path( + &destination_namespace, + &crate::table_catalog::IdentifierSegment::parse("events_v2").unwrap(), + ); + let foreign_namespace_path = crate::table_catalog::default_table_metadata_dir_path( + &source_namespace, + &crate::table_catalog::IdentifierSegment::parse("foreign").unwrap(), + ); + let retained_directory = committed.metadata_location.rsplit_once('/').unwrap().0; + let persisted_metadata = read_table_metadata_json(&metadata_backend, "warehouse", &committed.metadata_location) + .await + .unwrap(); + for (index, location) in [ + format!("{current_namespace_path}/00003.metadata.json"), + format!("{foreign_namespace_path}/00003.metadata.json"), + format!("s3://other-warehouse/{foreign_namespace_path}/00003.metadata.json"), + format!("{retained_directory}/../00003.metadata.json"), + format!("{retained_directory}/child/00003.metadata.json"), + format!("{retained_directory}/%2e%2e.json"), + ] + .into_iter() + .enumerate() + { + if index < 2 { + metadata_backend + .put_json("warehouse", &location, persisted_metadata.clone()) + .await; + } + let mut request = stale_request.clone(); + request.commit_id = format!("invalid-renamed-path-{index}"); + request.idempotency_key = Some(request.commit_id.clone()); + request.expected_version_token = committed.version_token.clone(); + request.expected_metadata_location = committed.metadata_location.clone(); + request.new_metadata_location = location; + assert!(matches!( + store.commit_table(request.clone()).await, + Err(crate::table_catalog::TableCatalogStoreError::Invalid(_)) + )); + assert!( + store + .get_commit_by_id("warehouse", &renamed.table_id, &request.commit_id) + .await + .unwrap() + .is_none() + ); + assert!( + store + .get_commit_by_idempotency_key("warehouse", &renamed.table_id, request.idempotency_key.as_deref().unwrap()) + .await + .unwrap() + .is_none() + ); + assert_eq!( + store.load_table("warehouse", destination, "events_v2").await.unwrap(), + Some(committed.clone()) + ); + assert_eq!( + store.load_table("warehouse", "analytics", "events").await.unwrap(), + Some(recreated.clone()) + ); + } + let wrong_identity = standard_commit_table_response( + &store, + &trusted_table_commit_backend(&metadata_backend), + "warehouse", + &destination_namespace, + "events_v2", + standard_property_commit_request("22222222-2222-4222-8222-222222222222", &recreated.table_uuid, "wrong-identity"), + ) + .await + .expect_err("recreated source identity cannot commit to the renamed table"); + assert_eq!(wrong_identity.status_code(), Some(StatusCode::CONFLICT)); + assert_eq!( + store.load_table("warehouse", destination, "events_v2").await.unwrap(), + Some(committed.clone()) + ); let recreated_commit = standard_commit_table_response( &store, &trusted_table_commit_backend(&metadata_backend), @@ -4300,7 +5235,7 @@ async fn renamed_and_recreated_tables_with_the_same_commit_id_use_disjoint_metad assert!( !renamed_commit .metadata_location - .contains("/namespaces/curated/tables/events_v2/") + .contains(&format!("/namespaces/{destination}/tables/events_v2/")) ); assert!( recreated_commit @@ -4309,8 +5244,8 @@ async fn renamed_and_recreated_tables_with_the_same_commit_id_use_disjoint_metad ); assert_eq!(recreated_commit.metadata["table-uuid"], recreated.table_uuid); assert_eq!(recreated_commit.metadata["properties"]["owner"], "analytics"); - for metadata_location in [renamed_commit.metadata_location, recreated_commit.metadata_location] { - let object_key = test_snapshot_object_key("warehouse", &metadata_location); + for metadata_location in [&renamed_commit.metadata_location, &recreated_commit.metadata_location] { + let object_key = test_snapshot_object_key("warehouse", metadata_location); assert!( metadata_backend .object_exists("warehouse", &object_key) @@ -4318,6 +5253,27 @@ async fn renamed_and_recreated_tables_with_the_same_commit_id_use_disjoint_metad .expect("metadata lookup should succeed") ); } + let restarted = crate::table_catalog::ConfiguredTableCatalogStore::new_for_test(metadata_backend, mode); + assert_eq!( + restarted.load_table("warehouse", destination, "events_v2").await.unwrap(), + Some(committed) + ); + let recreated_after = restarted + .load_table("warehouse", "analytics", "events") + .await + .unwrap() + .unwrap(); + assert_eq!(recreated_after.table_id, recreated.table_id); + assert_eq!(recreated_after.table_uuid, recreated.table_uuid); + assert_eq!(recreated_after.warehouse_location, recreated.warehouse_location); + assert_eq!(recreated_after.generation, recreated.generation + 1); + assert_eq!( + restarted + .get_commit_by_id("warehouse", &renamed.table_id, commit_id) + .await + .unwrap(), + Some(receipt) + ); } #[tokio::test] diff --git a/rustfs/src/admin/handlers/tls_certificate.rs b/rustfs/src/admin/handlers/tls_certificate.rs new file mode 100644 index 000000000..a45762625 --- /dev/null +++ b/rustfs/src/admin/handlers/tls_certificate.rs @@ -0,0 +1,177 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use crate::admin::{ + auth::authorize_admin_request, + router::{AdminOperation, Operation, S3Router}, + storage_api::s3::{Body, S3Error, S3ErrorCode, S3Request, S3Response, S3Result}, +}; +use crate::server::{ + ADMIN_PREFIX, NativeTlsTransport, + tls_publish::{self, PublishError, PublishRequest}, +}; +use http::{HeaderMap, HeaderValue, StatusCode}; +use hyper::Method; +use matchit::Params; +use rustfs_policy::policy::{action::Action, action::AdminAction}; + +pub fn register_tls_certificate_routes(router: &mut S3Router) -> std::io::Result<()> { + router.insert( + Method::GET, + &format!("{ADMIN_PREFIX}/v3/tls/certificate-state"), + AdminOperation(&CertificateStateHandler), + )?; + router.insert( + Method::POST, + &format!("{ADMIN_PREFIX}/v3/tls/certificate"), + AdminOperation(&PublishCertificateHandler), + )?; + Ok(()) +} + +pub struct CertificateStateHandler; +pub struct PublishCertificateHandler; + +fn require_native_tls(req: &S3Request) -> S3Result<()> { + if !req + .extensions + .get::() + .is_some_and(|transport| transport.0) + { + return Err(S3Error::with_message( + S3ErrorCode::AccessDenied, + "certificate management requires TLS on the RustFS listener", + )); + } + Ok(()) +} + +async fn authorize_certificate_request(req: &S3Request, action: AdminAction) -> S3Result<()> { + if req.credentials.is_none() { + return Err(S3Error::with_message( + S3ErrorCode::AccessDenied, + "certificate management requires credentials", + )); + } + authorize_admin_request(req, vec![Action::AdminAction(action)]).await?; + require_native_tls(req) +} + +fn tls_path() -> S3Result { + crate::server::tls_material::active_server_tls_path() + .ok_or_else(|| S3Error::with_message(S3ErrorCode::InvalidRequest, "certificate publication is not configured")) +} + +fn publication_error(error: PublishError) -> S3Error { + let (code, status) = match error { + PublishError::Conflict => (S3ErrorCode::InvalidRequest, StatusCode::CONFLICT), + PublishError::Disabled => (S3ErrorCode::InvalidRequest, StatusCode::SERVICE_UNAVAILABLE), + PublishError::Invalid => (S3ErrorCode::InvalidRequest, StatusCode::BAD_REQUEST), + PublishError::Storage | PublishError::Busy => (S3ErrorCode::InternalError, StatusCode::SERVICE_UNAVAILABLE), + }; + let mut result = S3Error::with_message(code, error.to_string()); + result.set_status_code(status); + result +} + +fn json_response(value: &impl serde::Serialize) -> S3Result> { + let bytes = serde_json::to_vec(value) + .map_err(|_| S3Error::with_message(S3ErrorCode::InternalError, "certificate status serialization failed"))?; + let mut headers = HeaderMap::new(); + headers.insert(http::header::CONTENT_TYPE, HeaderValue::from_static("application/json")); + headers.insert(http::header::CACHE_CONTROL, HeaderValue::from_static("no-store")); + Ok(S3Response::with_headers((StatusCode::OK, Body::from(bytes)), headers)) +} + +#[async_trait::async_trait] +impl Operation for CertificateStateHandler { + async fn call(&self, req: S3Request, _params: Params<'_, '_>) -> S3Result> { + authorize_certificate_request(&req, AdminAction::TLSCertificateStatusAdminAction).await?; + json_response(&tls_publish::certificate_state(tls_path()?).await.map_err(publication_error)?) + } +} + +#[async_trait::async_trait] +impl Operation for PublishCertificateHandler { + async fn call(&self, mut req: S3Request, _params: Params<'_, '_>) -> S3Result> { + authorize_certificate_request(&req, AdminAction::TLSCertificatePublishAdminAction).await?; + let body = req.input.store_all_limited(128 * 1024).await.map_err(|_| { + S3Error::with_message( + S3ErrorCode::InvalidRequest, + "certificate publication body exceeds the limit or is unreadable", + ) + })?; + let request: PublishRequest = serde_json::from_slice(&body) + .map_err(|_| S3Error::with_message(S3ErrorCode::InvalidRequest, "invalid certificate publication JSON"))?; + json_response(&tls_publish::publish(tls_path()?, request).await.map_err(publication_error)?) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn request(method: Method) -> S3Request { + S3Request { + input: Body::empty(), + method, + uri: "/rustfs/admin/v3/tls/certificate".parse().expect("URI"), + headers: HeaderMap::new(), + extensions: http::Extensions::new(), + credentials: None, + region: None, + service: None, + trailing_headers: None, + } + } + + #[tokio::test] + async fn certificate_handlers_reject_missing_credentials() { + assert_eq!( + CertificateStateHandler + .call(request(Method::GET), Params::new()) + .await + .expect_err("unauthenticated GET") + .code(), + &S3ErrorCode::AccessDenied + ); + assert_eq!( + PublishCertificateHandler + .call(request(Method::POST), Params::new()) + .await + .expect_err("unauthenticated POST") + .code(), + &S3ErrorCode::AccessDenied + ); + } + + #[test] + fn forwarded_protocol_never_authorizes_private_key_upload() { + let mut request = request(Method::POST); + request.headers.insert("x-forwarded-proto", HeaderValue::from_static("https")); + assert!(require_native_tls(&request).is_err()); + request.extensions.insert(NativeTlsTransport(false)); + assert!(require_native_tls(&request).is_err()); + request.extensions.insert(NativeTlsTransport(true)); + assert!(require_native_tls(&request).is_ok()); + } + + #[test] + fn unknown_json_fields_are_rejected_without_echoing_material() { + let error = serde_json::from_str::( + r#"{"schema_version":"x","resource_id":"x","expected_target_revision":"x","issuance_id":"00000000-0000-0000-0000-000000000001","certificate_pem":"cert","private_key_pem":"secret","extra":true}"#, + ); + assert!(error.is_err()); + } +} diff --git a/rustfs/src/admin/mod.rs b/rustfs/src/admin/mod.rs index 2721d4034..5ce1e06d3 100644 --- a/rustfs/src/admin/mod.rs +++ b/rustfs/src/admin/mod.rs @@ -107,6 +107,7 @@ fn register_admin_routes(r: &mut S3Router) -> std::io::Result<() inspect_archive::register_inspect_archive_route(r)?; gateway_key_inventory::register_gateway_key_inventory_route(r)?; tls_debug::register_tls_debug_route(r)?; + handlers::tls_certificate::register_tls_certificate_routes(r)?; kms::register_kms_route(r)?; oidc::register_oidc_route(r)?; idp_compat::register_idp_compat_route(r)?; diff --git a/rustfs/src/admin/route_policy.rs b/rustfs/src/admin/route_policy.rs index 2034fac7f..1e5305419 100644 --- a/rustfs/src/admin/route_policy.rs +++ b/rustfs/src/admin/route_policy.rs @@ -27,6 +27,8 @@ const ALL_ADMIN: AdminActionRef = AdminActionRef::new("AllAdminActions"); const ADD_USER_TO_GROUP: AdminActionRef = AdminActionRef::new("AddUserToGroupAdminAction"); const ATTACH_POLICY: AdminActionRef = AdminActionRef::new("AttachPolicyAdminAction"); const CONFIG_UPDATE: AdminActionRef = AdminActionRef::new("ConfigUpdateAdminAction"); +const TLS_CERTIFICATE_PUBLISH: AdminActionRef = AdminActionRef::new("TLSCertificatePublishAdminAction"); +const TLS_CERTIFICATE_STATUS: AdminActionRef = AdminActionRef::new("TLSCertificateStatusAdminAction"); const CONSOLE_LOG: AdminActionRef = AdminActionRef::new("ConsoleLogAdminAction"); const COMMIT_TABLE: AdminActionRef = AdminActionRef::new("CommitTableAction"); const ISSUE_PLUGIN_ATTEMPT: AdminActionRef = AdminActionRef::new("IssuePluginAttemptAction"); @@ -516,6 +518,18 @@ pub const ADMIN_ROUTE_POLICY_SPECS: &[AdminRouteSpec] = &[ ), admin(HttpMethod::Get, "/rustfs/admin/v3/config", CONFIG_UPDATE, RouteRiskLevel::High), admin(HttpMethod::Put, "/rustfs/admin/v3/config", CONFIG_UPDATE, RouteRiskLevel::High), + admin( + HttpMethod::Get, + "/rustfs/admin/v3/tls/certificate-state", + TLS_CERTIFICATE_STATUS, + RouteRiskLevel::Sensitive, + ), + admin( + HttpMethod::Post, + "/rustfs/admin/v3/tls/certificate", + TLS_CERTIFICATE_PUBLISH, + RouteRiskLevel::High, + ), admin(HttpMethod::Get, "/rustfs/admin/v3/scanner/status", SERVER_INFO, RouteRiskLevel::Sensitive), admin( HttpMethod::Post, @@ -2410,6 +2424,14 @@ mod tests { } } + #[test] + fn route_policy_records_dedicated_tls_actions() { + assert_action(HttpMethod::Get, "/rustfs/admin/v3/tls/certificate-state", TLS_CERTIFICATE_STATUS); + assert_action(HttpMethod::Post, "/rustfs/admin/v3/tls/certificate", TLS_CERTIFICATE_PUBLISH); + assert_not_action(HttpMethod::Post, "/rustfs/admin/v3/tls/certificate", CONFIG_UPDATE); + assert_not_action(HttpMethod::Get, "/rustfs/admin/v3/tls/certificate-state", TLS_CERTIFICATE_PUBLISH); + } + /// `Critical` marks the routes whose worst case is permanent loss of user /// data, so the inventory is pinned in both directions: the KMS key /// deletion route may not be quietly downgraded, and no other route may be diff --git a/rustfs/src/admin/route_registration_test.rs b/rustfs/src/admin/route_registration_test.rs index cbfc2bc75..d049fc347 100644 --- a/rustfs/src/admin/route_registration_test.rs +++ b/rustfs/src/admin/route_registration_test.rs @@ -349,6 +349,8 @@ fn expected_admin_route_matrix() -> Vec { admin_route(Method::PUT, "/v3/restore-config-history-kv"), admin_route(Method::GET, "/v3/config"), admin_route(Method::PUT, "/v3/config"), + admin_route(Method::GET, "/v3/tls/certificate-state"), + admin_route(Method::POST, "/v3/tls/certificate"), admin_route(Method::GET, "/v3/scanner/status"), admin_route(Method::POST, "/v3/scanner/cycle-state/reset"), admin_route(Method::POST, "/v3/scanner/usage-state/reset"), diff --git a/rustfs/src/admin/service/federated_identity.rs b/rustfs/src/admin/service/federated_identity.rs index 7417a7322..1641a9447 100644 --- a/rustfs/src/admin/service/federated_identity.rs +++ b/rustfs/src/admin/service/federated_identity.rs @@ -30,6 +30,10 @@ use std::collections::{BTreeMap, HashMap}; use time::{Duration, OffsetDateTime}; use tracing::{debug, warn}; +const LOG_COMPONENT_ADMIN: &str = "admin"; +const LOG_SUBSYSTEM_OIDC: &str = "oidc"; +const EVENT_ADMIN_OIDC_POLICY: &str = "admin_oidc_policy"; + pub(crate) struct DefaultFederatedSessionBinding; // Legacy receivers reject an empty parsed mapping; current receivers ignore it when the virtual-parent marker is present. @@ -104,34 +108,33 @@ fn select_oidc_policies( fn build_oidc_token_claims(transaction: &FederatedSessionTransaction) -> HashMap { let authorization = &transaction.authorization; - let claims = &authorization.claims; let mut token_claims = HashMap::new(); - token_claims.insert("sub".to_string(), Value::String(claims.sub.clone())); + token_claims.insert("sub".to_string(), Value::String(authorization.subject().to_string())); token_claims.insert("iss".to_string(), Value::String("rustfs-oidc".to_string())); - token_claims.insert("oidc_provider".to_string(), Value::String(authorization.provider_id.clone())); + token_claims.insert("oidc_provider".to_string(), Value::String(authorization.provider_id().to_string())); - if !claims.email.is_empty() { - token_claims.insert("email".to_string(), Value::String(claims.email.clone())); + if !authorization.email().is_empty() { + token_claims.insert("email".to_string(), Value::String(authorization.email().to_string())); } - if !claims.username.is_empty() { - token_claims.insert("preferred_username".to_string(), Value::String(claims.username.clone())); + if !authorization.username().is_empty() { + token_claims.insert("preferred_username".to_string(), Value::String(authorization.username().to_string())); } - if !authorization.groups.is_empty() { + if !authorization.mapped_iam_groups().is_empty() { token_claims.insert( "groups".to_string(), Value::Array( authorization - .groups + .mapped_iam_groups() .iter() .map(|group| Value::String(group.clone())) .collect(), ), ); } - if !authorization.roles.is_empty() { + if !authorization.roles().is_empty() { token_claims.insert( "roles".to_string(), - Value::Array(authorization.roles.iter().cloned().map(Value::String).collect()), + Value::Array(authorization.roles().iter().cloned().map(Value::String).collect()), ); } @@ -242,7 +245,6 @@ fn issue_credentials( secret: Option<&str>, ) -> Result { let authorization = &transaction.authorization; - let claims = &authorization.claims; let parent_user = authorization.oidc_virtual_parent().ok_or_else(|| { FederatedSessionBindingError::InvalidRequest("verified OIDC identity is missing issuer or subject".to_string()) })?; @@ -262,16 +264,19 @@ fn issue_credentials( )); } debug!( - provider_id = %authorization.provider_id, + event = EVENT_ADMIN_OIDC_POLICY, + component = LOG_COMPONENT_ADMIN, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), parent_user = %parent_user, - email = %claims.email, - username = %claims.username, - sub = %claims.sub, - policy_count = authorization.policies.len(), - group_count = authorization.groups.len(), - policies = ?authorization.policies, - groups = ?authorization.groups, - roles_claim_key = ?authorization.roles_claim_key, + email = %authorization.email(), + username = %authorization.username(), + sub = %authorization.subject(), + policy_count = authorization.policies().len(), + group_count = authorization.mapped_iam_groups().len(), + policies = ?authorization.policies(), + groups = ?authorization.mapped_iam_groups(), + roles_claim_key = ?authorization.roles_claim_key(), has_session_policy = transaction.session_policy.is_some(), "OIDC STS credential claims prepared" ); @@ -291,7 +296,7 @@ fn issue_credentials( let mut credentials = get_new_credentials_with_metadata(&token_claims, secret) .map_err(|error| FederatedSessionBindingError::Internal(format!("credential generation failed: {error}")))?; credentials.parent_user = parent_user; - credentials.groups = Some(authorization.groups.clone()); + credentials.groups = Some(authorization.mapped_iam_groups().to_vec()); Ok(credentials) } @@ -325,12 +330,12 @@ impl FederatedSessionBinding for DefaultFederatedSessionBinding { FederatedSessionBindingError::InvalidRequest("verified OIDC identity is missing issuer or subject".to_string()) })?; let (selected_policy_names, ignorable_policy_names) = match iam_store - .sts_policy_db_get(&parent_user, &Some(authorization.groups.clone())) + .sts_policy_db_get(&parent_user, &Some(authorization.mapped_iam_groups().to_vec())) .await .map_err(|_| FederatedSessionBindingError::Internal("failed to resolve OIDC policy mapping".to_string()))? { mapped_policy_names if !mapped_policy_names.is_empty() => (mapped_policy_names, &[][..]), - _ => (authorization.policies.clone(), authorization.group_claim_policies.as_slice()), + _ => (authorization.policies().to_vec(), authorization.group_claim_policies()), }; // A name may be skipped only after storage confirms the policy is absent. A failed lookup // must not be mistaken for absence, or an unreadable Deny policy would be dropped from the @@ -339,7 +344,10 @@ impl FederatedSessionBinding for DefaultFederatedSessionBinding { for policy_name in &selected_policy_names { let exists = iam_store.policy_exists(policy_name).await.map_err(|err| { warn!( - provider_id = %authorization.provider_id, + event = EVENT_ADMIN_OIDC_POLICY, + component = LOG_COMPONENT_ADMIN, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), parent_user = %parent_user, policy = %policy_name, error = %err, @@ -367,7 +375,10 @@ impl FederatedSessionBinding for DefaultFederatedSessionBinding { .map_err(|rejection| match rejection { OidcPolicyRejection::Unsupported(policy_names) => { warn!( - provider_id = %authorization.provider_id, + event = EVENT_ADMIN_OIDC_POLICY, + component = LOG_COMPONENT_ADMIN, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), parent_user = %parent_user, unsupported_policies = ?policy_names, "OIDC STS rejected existing policies whose names are not allowed in session claims" @@ -376,7 +387,10 @@ impl FederatedSessionBinding for DefaultFederatedSessionBinding { } OidcPolicyRejection::Unresolved(policy_names) => { debug!( - provider_id = %authorization.provider_id, + event = EVENT_ADMIN_OIDC_POLICY, + component = LOG_COMPONENT_ADMIN, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), parent_user = %parent_user, unresolved_policies = ?policy_names, "OIDC STS rejected required policies without a current policy" @@ -389,7 +403,10 @@ impl FederatedSessionBinding for DefaultFederatedSessionBinding { })?; if !selection.ignored.is_empty() { debug!( - provider_id = %authorization.provider_id, + event = EVENT_ADMIN_OIDC_POLICY, + component = LOG_COMPONENT_ADMIN, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), parent_user = %parent_user, ignored_group_policies = ?selection.ignored, "OIDC STS ignoring group claim values without a matching policy" @@ -402,10 +419,10 @@ impl FederatedSessionBinding for DefaultFederatedSessionBinding { if tracing::enabled!(tracing::Level::DEBUG) { log_oidc_policy_diagnostics( &iam_store, - &authorization.provider_id, + authorization.provider_id(), &credentials.parent_user, &selected_policy_names, - &authorization.groups, + authorization.mapped_iam_groups(), ) .await; } @@ -428,7 +445,7 @@ mod tests { use super::*; use crate::admin::runtime_sources::{AppContext, current_object_store_handle, publish_test_app_context}; use hmac::{Hmac, KeyInit, Mac}; - use rustfs_iam::federation::{FederatedAuthorization, FederatedClaims}; + use rustfs_iam::federation::FederatedAuthorizationFixture; use rustfs_iam::store::{ Store, UserType, object::{IAM_CONFIG_POLICIES_PREFIX, IAM_CONFIG_PREFIX}, @@ -443,31 +460,37 @@ mod tests { use sha2::Sha512; use std::sync::Arc; - fn transaction() -> FederatedSessionTransaction { + fn authorization_fixture() -> FederatedAuthorizationFixture { + FederatedAuthorizationFixture { + provider_id: "default".to_string(), + sub: "subject".to_string(), + email: "user@example.com".to_string(), + username: "user".to_string(), + source_groups: vec!["source-group".to_string()], + attributes: HashMap::from([ + ("iss".to_string(), serde_json::json!("https://idp.example.test")), + ("raw_poison".to_string(), serde_json::json!("must-not-leak")), + ]), + policies: vec!["readwrite".to_string()], + group_claim_policies: Vec::new(), + mapped_iam_groups: vec!["devs".to_string()], + roles_claim_key: Some("roles".to_string()), + roles: vec!["admin".to_string(), "reader".to_string()], + } + } + + fn transaction_from_fixture(authorization: FederatedAuthorizationFixture) -> FederatedSessionTransaction { FederatedSessionTransaction { - authorization: FederatedAuthorization { - provider_id: "default".to_string(), - claims: FederatedClaims { - sub: "subject".to_string(), - email: "user@example.com".to_string(), - username: "user".to_string(), - groups: vec!["source-group".to_string()], - raw: HashMap::from([ - ("iss".to_string(), serde_json::json!("https://idp.example.test")), - ("raw_poison".to_string(), serde_json::json!("must-not-leak")), - ]), - }, - policies: vec!["readwrite".to_string()], - group_claim_policies: Vec::new(), - groups: vec!["devs".to_string()], - roles_claim_key: Some("roles".to_string()), - roles: vec!["admin".to_string(), "reader".to_string()], - }, + authorization: authorization.into_authorization(), duration_seconds: 3600, session_policy: None, } } + fn transaction() -> FederatedSessionTransaction { + transaction_from_fixture(authorization_fixture()) + } + async fn ready_test_iam() -> Arc> { let _ = rustfs_credentials::init_global_action_credentials( Some("TESTROOTACCESSKEY".to_string()), @@ -531,11 +554,12 @@ mod tests { #[test] fn token_claims_omit_empty_optional_oidc_fields() { - let mut transaction = transaction(); - transaction.authorization.claims.email.clear(); - transaction.authorization.claims.username.clear(); - transaction.authorization.groups.clear(); - transaction.authorization.roles.clear(); + let mut authorization = authorization_fixture(); + authorization.email.clear(); + authorization.username.clear(); + authorization.mapped_iam_groups.clear(); + authorization.roles.clear(); + let transaction = transaction_from_fixture(authorization); assert_eq!( build_oidc_token_claims(&transaction), @@ -547,6 +571,36 @@ mod tests { ); } + #[test] + #[serial] + fn console_session_duration_bounds_signed_credential_expiration() { + let secret = "console-session-expiration-test-secret"; + for (configured, expected_seconds) in [(None, 3600), (Some("7200"), 7200), (Some("604801"), 604_800)] { + temp_env::with_var(rustfs_config::ENV_CONSOLE_OIDC_SESSION_DURATION, configured, || { + let mut transaction = transaction(); + transaction.duration_seconds = rustfs_iam::oidc::console_session_duration_secs(); + let before = OffsetDateTime::now_utc().unix_timestamp(); + let credentials = issue_credentials(&transaction, &["readonly".to_string()], Some(secret)) + .expect("configured Console session should issue credentials"); + let after = OffsetDateTime::now_utc().unix_timestamp(); + let claims = rustfs_iam::sys::get_claims_from_token_with_secret(&credentials.session_token, secret) + .expect("Console credential signature should verify"); + let expiration = claims["exp"].as_i64().expect("signed expiration should be an integer"); + assert!( + (before + expected_seconds..=after + expected_seconds).contains(&expiration), + "signed expiration must reflect the bounded Console session for {configured:?}" + ); + assert_eq!( + credentials + .expiration + .expect("Console credential must expire") + .unix_timestamp(), + expiration + ); + }); + } + } + #[test] fn issued_credentials_and_replication_item_use_minio_parent_shape() { let transaction = transaction(); @@ -634,9 +688,10 @@ mod tests { #[serial] async fn binding_uses_sts_policy_when_regular_mapping_collides() { let iam = ready_test_iam().await; - let mut transaction = transaction(); - transaction.authorization.claims.sub = "binding-sts-policy-subject".to_string(); - transaction.authorization.policies = vec!["readwrite".to_string()]; + let mut authorization = authorization_fixture(); + authorization.sub = "binding-sts-policy-subject".to_string(); + authorization.policies = vec!["readwrite".to_string()]; + let transaction = transaction_from_fixture(authorization); let parent = transaction .authorization .oidc_virtual_parent() @@ -750,7 +805,7 @@ mod tests { let mut transaction = transaction(); transaction.session_policy = Some("not-json".to_string()); - let error = issue_credentials(&transaction, &transaction.authorization.policies, None) + let error = issue_credentials(&transaction, transaction.authorization.policies(), None) .expect_err("invalid policy should fail first"); assert!(matches!(error, FederatedSessionBindingError::InvalidRequest(_))); } @@ -782,10 +837,11 @@ mod tests { #[test] fn missing_oidc_issuer_fails_closed_before_credential_issuance() { - let mut transaction = transaction(); - transaction.authorization.claims.raw.clear(); + let mut authorization = authorization_fixture(); + authorization.attributes.clear(); + let transaction = transaction_from_fixture(authorization); - let error = issue_credentials(&transaction, &transaction.authorization.policies, Some("signing-secret")) + let error = issue_credentials(&transaction, transaction.authorization.policies(), Some("signing-secret")) .expect_err("credential issuance should reject a missing issuer"); assert!(matches!(error, FederatedSessionBindingError::InvalidRequest(_))); } @@ -865,12 +921,12 @@ mod tests { } fn group_claim_transaction(subject: &str, groups: &[&str]) -> FederatedSessionTransaction { - let mut transaction = transaction(); - transaction.authorization.claims.sub = subject.to_string(); - transaction.authorization.groups = names(groups); - transaction.authorization.policies = names(groups); - transaction.authorization.group_claim_policies = names(groups); - transaction + let mut authorization = authorization_fixture(); + authorization.sub = subject.to_string(); + authorization.mapped_iam_groups = names(groups); + authorization.policies = names(groups); + authorization.group_claim_policies = names(groups); + transaction_from_fixture(authorization) } fn assert_binding_rejected(error: FederatedSessionBindingError, expected_message: &str) { @@ -898,7 +954,7 @@ mod tests { .expect("issued session token should verify"); assert_eq!(claims.get("policy"), Some(&serde_json::json!("readonly"))); - assert_eq!(credentials.groups.as_deref(), Some(transaction.authorization.groups.as_slice())); + assert_eq!(credentials.groups.as_deref(), Some(transaction.authorization.mapped_iam_groups())); let groups = credentials.groups.clone(); let conditions = HashMap::new(); @@ -987,10 +1043,11 @@ mod tests { #[serial] async fn binding_keeps_role_policy_strict() { ready_test_iam().await; - let mut transaction = transaction(); - transaction.authorization.claims.sub = "binding-strict-role-policy-subject".to_string(); - transaction.authorization.policies = names(&["readonly", "missing-role-policy"]); - transaction.authorization.group_claim_policies = Vec::new(); + let mut authorization = authorization_fixture(); + authorization.sub = "binding-strict-role-policy-subject".to_string(); + authorization.policies = names(&["readonly", "missing-role-policy"]); + authorization.group_claim_policies = Vec::new(); + let transaction = transaction_from_fixture(authorization); let error = DefaultFederatedSessionBinding .bind(&transaction) diff --git a/rustfs/src/admin/service/mod.rs b/rustfs/src/admin/service/mod.rs index 403530f5c..4c1dc8177 100644 --- a/rustfs/src/admin/service/mod.rs +++ b/rustfs/src/admin/service/mod.rs @@ -15,5 +15,6 @@ pub(crate) mod caller_identity; pub mod config; pub(crate) mod federated_identity; +pub(crate) mod oidc_config; pub(crate) mod session_policy; pub mod site_replication; diff --git a/rustfs/src/admin/service/oidc_config.rs b/rustfs/src/admin/service/oidc_config.rs new file mode 100644 index 000000000..a3eea052e --- /dev/null +++ b/rustfs/src/admin/service/oidc_config.rs @@ -0,0 +1,158 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use crate::admin::runtime_sources::{ + current_app_context, current_object_store_handle_for_context, current_server_config_for_context, +}; +use crate::admin::storage_api::config::{ + read_admin_config_without_migrate, read_admin_server_config_snapshot, save_admin_server_config_snapshot, +}; +use crate::admin::storage_api::error::Error as StorageError; +use rustfs_config::server_config::Config as ServerConfig; +use rustfs_iam::oidc::{ + OidcConfigError, OidcConfigSnapshot, OidcProviderConfigInput, OidcProviderValidationInput, OidcProviderValidationResult, + build_upsert_provider_config, build_validation_provider_config, delete_persisted_provider_config, load_oidc_config_snapshot, + persisted_provider_secret, upsert_persisted_provider_config, validate_mutable_provider_id, + validate_oidc_provider_config_with_extra_root_ca, +}; + +#[derive(Debug, thiserror::Error)] +pub(crate) enum OidcAdminConfigError { + #[error("storage layer not initialized")] + StorageUnavailable, + #[error("failed to load server config: {0}")] + Load(#[source] StorageError), + #[error("failed to save server config: {0}")] + Save(#[source] StorageError), + #[error("validation failed: {0}")] + Validation(String), + #[error("OIDC config update task {0}")] + Task(&'static str), + #[error(transparent)] + Config(#[from] OidcConfigError), +} + +type Result = std::result::Result; + +pub(crate) struct OidcConfigList { + pub snapshot: OidcConfigSnapshot, + pub restart_required: bool, +} + +pub(crate) fn ensure_mutable_provider_id(provider_id: &str) -> Result<()> { + validate_mutable_provider_id(provider_id)?; + Ok(()) +} + +fn oidc_config_store() -> Result> { + let context = current_app_context(); + current_object_store_handle_for_context(context.as_deref()).ok_or(OidcAdminConfigError::StorageUnavailable) +} + +async fn load_server_config_from_store() -> Result { + let store = oidc_config_store()?; + read_admin_config_without_migrate(store) + .await + .map_err(OidcAdminConfigError::Load) +} + +fn restart_required_from_active_config(config: &ServerConfig, active_config: Option<&ServerConfig>) -> bool { + load_oidc_config_snapshot(Some(config)) != load_oidc_config_snapshot(active_config) +} + +pub(crate) async fn list_config() -> Result { + let config = load_server_config_from_store().await?; + let context = current_app_context(); + let active_config = current_server_config_for_context(context.as_deref()); + Ok(OidcConfigList { + snapshot: load_oidc_config_snapshot(Some(&config)), + restart_required: restart_required_from_active_config(&config, active_config.as_ref()), + }) +} + +async fn update_server_config(modifier: F) -> Result<()> +where + F: FnOnce(&mut ServerConfig) -> Result<()> + Send + 'static, +{ + let store = oidc_config_store()?; + tokio::spawn(async move { + let snapshot = read_admin_server_config_snapshot(store.clone()) + .await + .map_err(OidcAdminConfigError::Load)?; + let mut config = snapshot.config.clone(); + modifier(&mut config)?; + save_admin_server_config_snapshot(store, &config, &snapshot) + .await + .map(|_| ()) + .map_err(OidcAdminConfigError::Save) + }) + .await + .map_err(|error| OidcAdminConfigError::Task(if error.is_cancelled() { "cancelled" } else { "panicked" }))? +} + +pub(crate) async fn upsert_config(provider_id: String, input: OidcProviderConfigInput) -> Result<()> { + ensure_mutable_provider_id(&provider_id)?; + update_server_config(move |config| { + let existing_secret = persisted_provider_secret(config, &provider_id); + let provider = build_upsert_provider_config(&provider_id, input, existing_secret)?; + upsert_persisted_provider_config(config, &provider); + Ok(()) + }) + .await +} + +pub(crate) async fn delete_config(provider_id: String) -> Result<()> { + ensure_mutable_provider_id(&provider_id)?; + update_server_config(move |config| { + delete_persisted_provider_config(config, &provider_id)?; + Ok(()) + }) + .await +} + +pub(crate) async fn validate_config(input: OidcProviderValidationInput) -> Result { + let provider = build_validation_provider_config(input)?; + let extra_root_ca = crate::startup_auth::current_oidc_extra_root_ca_material() + .await + .map_err(OidcAdminConfigError::Validation)?; + validate_oidc_provider_config_with_extra_root_ca(&provider, extra_root_ca.root_ca_pem.as_deref()) + .await + .map_err(OidcAdminConfigError::Validation) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn restart_required_detects_persisted_changes() { + let active_config = ServerConfig::new(); + let mut persisted_config = ServerConfig::new(); + let provider = build_upsert_provider_config( + "default", + OidcProviderConfigInput { + config_url: "https://example.com/.well-known/openid-configuration".to_string(), + client_id: "console".to_string(), + client_secret: Some("secret".to_string()), + ..Default::default() + }, + None, + ) + .expect("provider config should be valid"); + upsert_persisted_provider_config(&mut persisted_config, &provider); + + assert!(restart_required_from_active_config(&persisted_config, Some(&active_config))); + assert!(!restart_required_from_active_config(&persisted_config, Some(&persisted_config))); + } +} diff --git a/rustfs/src/app/bucket_usecase.rs b/rustfs/src/app/bucket_usecase.rs index 074ec0c66..69d06c455 100644 --- a/rustfs/src/app/bucket_usecase.rs +++ b/rustfs/src/app/bucket_usecase.rs @@ -69,7 +69,8 @@ use crate::app::object::shared::mark_on_demand_migration_list_local_only; use crate::app::object_data_cache::invalidate_object_data_cache_bucket_after_delete; use crate::app::runtime_sources::{ AppContext, current_app_context, current_encryption_service, current_notification_system, - current_notify_interface_for_context, current_object_data_cache_for_context, current_object_store_handle_for_context, + current_notification_system_for_context, current_notify_interface_for_context, current_object_data_cache_for_context, + current_object_store_handle_for_context, }; use crate::app::table_list_isolation; use crate::auth::get_condition_values_with_client_info; @@ -151,6 +152,7 @@ static BUCKET_OPERATION_ADMISSION: LazyLock> = Laz struct BucketOperationAdmission { active: Arc, waiting: Semaphore, + in_flight: Arc, } impl BucketOperationAdmission { @@ -158,12 +160,19 @@ impl BucketOperationAdmission { Self { active: Arc::new(Semaphore::new(concurrency)), waiting: Semaphore::new(queue_capacity), + in_flight: Arc::new(Semaphore::new(concurrency + queue_capacity)), } } - async fn acquire(&self, operation: &'static str) -> S3Result { + async fn acquire(&self, operation: &'static str) -> S3Result<(OwnedSemaphorePermit, OwnedSemaphorePermit)> { + // Reserve the entire operation before mutation, including post-commit + // work that can outlive both its execution slot and the request. + let in_flight = self.in_flight.clone().try_acquire_owned().map_err(|err| match err { + TryAcquireError::NoPermits => bucket_admission_slow_down(operation, "in-flight limit reached"), + TryAcquireError::Closed => bucket_admission_closed(operation), + })?; match self.active.clone().try_acquire_owned() { - Ok(permit) => return Ok(permit), + Ok(permit) => return Ok((permit, in_flight)), Err(TryAcquireError::Closed) => return Err(bucket_admission_closed(operation)), Err(TryAcquireError::NoPermits) => {} } @@ -174,10 +183,11 @@ impl BucketOperationAdmission { TryAcquireError::NoPermits => bucket_admission_slow_down(operation, "queue is full"), TryAcquireError::Closed => bucket_admission_closed(operation), })?; - tokio::time::timeout(BUCKET_OPERATION_QUEUE_TIMEOUT, self.active.clone().acquire_owned()) + let active = tokio::time::timeout(BUCKET_OPERATION_QUEUE_TIMEOUT, self.active.clone().acquire_owned()) .await .map_err(|_| bucket_admission_slow_down(operation, "queue wait timed out"))? - .map_err(|_| bucket_admission_closed(operation)) + .map_err(|_| bucket_admission_closed(operation))?; + Ok((active, in_flight)) } } @@ -615,12 +625,27 @@ async fn notify_bucket_metadata_reload( operation: &'static str, request_context: Option, scanner_maintenance_change: bool, +) { + notify_bucket_metadata_reload_with_context(bucket, operation, request_context, scanner_maintenance_change, None).await; +} + +async fn notify_bucket_metadata_reload_with_context( + bucket: String, + operation: &'static str, + request_context: Option, + scanner_maintenance_change: bool, + app_context: Option>, ) { record_local_scanner_maintenance_reload(&bucket, scanner_maintenance_change); // Keep reload detached across request cancellation, but wait before a healthy peer can serve the previous config. let (completed_tx, completed_rx) = tokio::sync::oneshot::channel(); spawn_background_with_context(request_context, async move { - if let Some(notification_sys) = current_notification_system() { + let notification_sys = if let Some(context) = app_context.as_deref() { + current_notification_system_for_context(Some(context)) + } else { + current_notification_system() + }; + if let Some(notification_sys) = notification_sys { let result = if scanner_maintenance_change { notification_sys.load_bucket_metadata_for_scanner_maintenance(&bucket).await } else { @@ -1232,15 +1257,19 @@ fn build_list_objects_v2_metadata_output( } } -fn create_bucket_exists_response(is_owner: bool) -> S3Result> { - if is_owner { +fn create_bucket_exists_response(is_owner: bool, region: &str) -> S3Result> { + if is_owner && region == RUSTFS_REGION { + // AWS preserves the legacy idempotent response for buckets recreated in + // us-east-1 (the default RustFS region). return Ok(S3Response::new(CreateBucketOutput::default())); } - Err(s3_error!( - BucketAlreadyExists, - "The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again." - )) + let error_code = if is_owner { + S3ErrorCode::BucketAlreadyOwnedByYou + } else { + S3ErrorCode::BucketAlreadyExists + }; + Err(S3Error::with_message(error_code, "The requested bucket already exists.")) } fn resolve_notification_region(global_region: Option, request_region: Option) -> String { @@ -1357,28 +1386,30 @@ pub struct DefaultBucketUsecase { context: Option>, } -async fn await_bucket_usecase_on_fresh_task(operation: &'static str, future: F) -> S3Result +async fn await_bucket_usecase_on_fresh_task(operation: &'static str, make_future: MakeFuture) -> S3Result where T: Send + 'static, F: Future> + Send + 'static, + MakeFuture: FnOnce(OwnedSemaphorePermit) -> F + Send + 'static, { - await_bucket_usecase_on_fresh_task_with_admission(operation, Arc::clone(&BUCKET_OPERATION_ADMISSION), future).await + await_bucket_usecase_on_fresh_task_with_admission(operation, Arc::clone(&BUCKET_OPERATION_ADMISSION), make_future).await } -async fn await_bucket_usecase_on_fresh_task_with_admission( +async fn await_bucket_usecase_on_fresh_task_with_admission( operation: &'static str, admission: Arc, - future: F, + make_future: MakeFuture, ) -> S3Result where T: Send + 'static, F: Future> + Send + 'static, + MakeFuture: FnOnce(OwnedSemaphorePermit) -> F + Send + 'static, { - let permit = admission.acquire(operation).await?; + let (mutation_permit, in_flight) = admission.acquire(operation).await?; tokio::spawn( async move { - let _permit = permit; - future.await + let _in_flight = in_flight; + make_future(mutation_permit).await } .in_current_span(), ) @@ -1421,11 +1452,19 @@ impl DefaultBucketUsecase { )] pub async fn execute_create_bucket(&self, req: S3Request) -> S3Result> { let usecase = self.clone(); - await_bucket_usecase_on_fresh_task("bucket creation", async move { usecase.execute_create_bucket_inner(req).await }).await + await_bucket_usecase_on_fresh_task("bucket creation", move |mutation_permit| async move { + usecase.execute_create_bucket_inner(req, mutation_permit).await + }) + .await } - async fn execute_create_bucket_inner(&self, req: S3Request) -> S3Result> { + async fn execute_create_bucket_inner( + &self, + req: S3Request, + mutation_permit: OwnedSemaphorePermit, + ) -> S3Result> { let helper = OperationHelper::new(&req, EventName::BucketCreated, S3Operation::CreateBucket); + let request_context = req.extensions.get::().cloned(); let requester_is_owner = match req_info_ref(&req) { Ok(r) => r.is_owner, Err(_) => { @@ -1469,12 +1508,35 @@ impl DefaultBucketUsecase { }) .await?; + // The storage/site mutation and its lock fences have finished. Keep + // the operation's in-flight slot while peers reload, without holding + // an execution slot needed by other creates and deletes. + drop(mutation_permit); + match make_result { - Ok(()) => {} + Ok(()) => { + // A peer may have cached this bucket as missing before the + // create. Refresh peer metadata after the namespace and + // metadata commit fences are released so the peer reload can + // acquire its bucket namespace read lock without an ABBA + // interleaving with creation. + notify_bucket_metadata_reload_with_context( + bucket.clone(), + "create bucket", + request_context, + false, + self.context.clone(), + ) + .await; + } Err(StorageError::BucketExists(_)) => { - // Per S3 spec: bucket namespace is global. Owner recreating returns 200 OK; - // non-owner gets 409 BucketAlreadyExists. - let result = create_bucket_exists_response(requester_is_owner); + // Match AWS CreateBucket semantics: an owned bucket is idempotent in + // us-east-1 and returns BucketAlreadyOwnedByYou elsewhere. + let region = self + .global_region() + .map(|region| region.to_string()) + .unwrap_or_else(|| RUSTFS_REGION.to_string()); + let result = create_bucket_exists_response(requester_is_owner, ®ion); let _ = helper.complete(&result); return result; } @@ -1495,7 +1557,11 @@ impl DefaultBucketUsecase { #[instrument(level = "debug", skip(self, req))] pub async fn execute_delete_bucket(&self, req: S3Request) -> S3Result> { let usecase = self.clone(); - await_bucket_usecase_on_fresh_task("bucket deletion", async move { usecase.execute_delete_bucket_inner(req).await }).await + await_bucket_usecase_on_fresh_task("bucket deletion", move |mutation_permit| async move { + let _mutation_permit = mutation_permit; + usecase.execute_delete_bucket_inner(req).await + }) + .await } async fn execute_delete_bucket_inner( @@ -3308,7 +3374,8 @@ mod tests { let parent = tokio::spawn(await_bucket_usecase_on_fresh_task_with_admission( "test bucket operation", admission.clone(), - async move { + move |mutation_permit| async move { + drop(mutation_permit); committed_for_task.notify_one(); release_hook_for_task.notified().await; hook_ran_for_task.store(true, Ordering::SeqCst); @@ -3320,6 +3387,8 @@ mod tests { parent.abort(); let _ = parent.await; + assert_eq!(admission.active.available_permits(), 1); + assert_eq!(admission.in_flight.available_permits(), 1, "cancelled post-commit work stays bounded"); release_hook.notify_waiters(); tokio::time::timeout(Duration::from_secs(1), finished_wait) .await @@ -3330,6 +3399,137 @@ mod tests { "the fresh task must own both the storage mutation and its post-commit hooks" ); assert_eq!(admission.active.available_permits(), 1); + assert_eq!(admission.in_flight.available_permits(), 2); + } + + #[tokio::test(start_paused = true)] + async fn bucket_usecase_admission_releases_execution_during_slow_post_commit_work() { + let admission = Arc::new(BucketOperationAdmission::new( + BUCKET_OPERATION_CONCURRENCY, + BUCKET_OPERATION_QUEUE_CAPACITY, + )); + let active_mutations = Arc::new(AtomicUsize::new(0)); + let peak_mutations = Arc::new(AtomicUsize::new(0)); + let (committed_tx, mut committed_rx) = tokio::sync::mpsc::unbounded_channel(); + let mut tasks = Vec::new(); + for _ in 0..100 { + let active = active_mutations.clone(); + let peak = peak_mutations.clone(); + let committed = committed_tx.clone(); + tasks.push(tokio::spawn(await_bucket_usecase_on_fresh_task_with_admission( + "bucket creation", + admission.clone(), + move |mutation_permit| async move { + let count = active.fetch_add(1, Ordering::SeqCst) + 1; + peak.fetch_max(count, Ordering::SeqCst); + tokio::time::sleep(Duration::from_millis(1)).await; + active.fetch_sub(1, Ordering::SeqCst); + drop(mutation_permit); + committed.send(()).expect("record commit"); + tokio::time::sleep(Duration::from_secs(4)).await; + Ok(()) + }, + ))); + } + tokio::time::timeout(Duration::from_secs(1), async { + for _ in 0..100 { + committed_rx.recv().await.expect("every mutation commits"); + } + }) + .await + .expect("peer reloads must not serialize subsequent mutations"); + assert_eq!(peak_mutations.load(Ordering::SeqCst), BUCKET_OPERATION_CONCURRENCY); + assert!( + tasks.iter().all(|task| !task.is_finished()), + "requests must still await their post-commit work" + ); + assert_eq!(admission.active.available_permits(), BUCKET_OPERATION_CONCURRENCY); + assert_eq!( + admission.in_flight.available_permits(), + BUCKET_OPERATION_CONCURRENCY + BUCKET_OPERATION_QUEUE_CAPACITY - 100 + ); + + await_bucket_usecase_on_fresh_task_with_admission("bucket deletion", admission.clone(), |mutation_permit| async move { + let _mutation_permit = mutation_permit; + Ok(()) + }) + .await + .expect("deletion can execute while creates await peer reloads"); + for task in tasks { + task.await.expect("request joins").expect("queued creation succeeds"); + } + assert_eq!( + admission.in_flight.available_permits(), + BUCKET_OPERATION_CONCURRENCY + BUCKET_OPERATION_QUEUE_CAPACITY + ); + } + + #[tokio::test] + async fn bucket_usecase_admission_bounds_post_commit_work_after_request_cancellation() { + let admission = Arc::new(BucketOperationAdmission::new(1, 1)); + let release = Arc::new(Semaphore::new(0)); + let (committed_tx, mut committed_rx) = tokio::sync::mpsc::unbounded_channel(); + let (finished_tx, mut finished_rx) = tokio::sync::mpsc::unbounded_channel(); + let mut parents = Vec::new(); + for _ in 0..2 { + let release = release.clone(); + let committed = committed_tx.clone(); + let finished = finished_tx.clone(); + parents.push(tokio::spawn(await_bucket_usecase_on_fresh_task_with_admission( + "bucket creation", + admission.clone(), + move |mutation_permit| async move { + drop(mutation_permit); + committed.send(()).expect("record commit"); + release.acquire().await.expect("release hook").forget(); + finished.send(()).expect("record completion"); + Ok(()) + }, + ))); + } + tokio::time::timeout(Duration::from_secs(5), async { + for _ in 0..2 { + committed_rx.recv().await.expect("operation commits"); + } + }) + .await + .expect("both operations reach post-commit work"); + for parent in parents { + parent.abort(); + assert!(parent.await.expect_err("caller cancelled").is_cancelled()); + } + assert_eq!(admission.active.available_permits(), 1); + assert_eq!(admission.in_flight.available_permits(), 0); + let started = Arc::new(AtomicBool::new(false)); + let started_for_task = started.clone(); + let error = await_bucket_usecase_on_fresh_task_with_admission( + "bucket creation", + admission.clone(), + move |mutation_permit| async move { + let _mutation_permit = mutation_permit; + started_for_task.store(true, Ordering::SeqCst); + Ok(()) + }, + ) + .await + .expect_err("unfinished hooks consume the total operation budget"); + assert_eq!(error.code(), &S3ErrorCode::SlowDown); + assert_eq!(error.message(), Some("bucket creation in-flight limit reached; retry later")); + assert_eq!(error.headers().expect("retry headers")[http::header::RETRY_AFTER], "1"); + assert!(!started.load(Ordering::SeqCst)); + release.add_permits(2); + tokio::time::timeout(Duration::from_secs(5), async { + for _ in 0..2 { + finished_rx.recv().await.expect("detached hook completes"); + } + while admission.in_flight.available_permits() != 2 { + tokio::task::yield_now().await; + } + }) + .await + .expect("capacity recovers after hooks complete"); + let permits = admission.acquire("bucket deletion").await.expect("new operation is admitted"); + drop(permits); } #[tokio::test(start_paused = true)] @@ -3338,16 +3538,22 @@ mod tests { let held = admission.active.clone().acquire_owned().await.expect("hold active slot"); let started = Arc::new(AtomicBool::new(false)); let started_for_task = started.clone(); - let result = await_bucket_usecase_on_fresh_task_with_admission("test bucket operation", admission.clone(), async move { - started_for_task.store(true, Ordering::SeqCst); - Ok(()) - }) + let result = await_bucket_usecase_on_fresh_task_with_admission( + "test bucket operation", + admission.clone(), + move |mutation_permit| async move { + let _mutation_permit = mutation_permit; + started_for_task.store(true, Ordering::SeqCst); + Ok(()) + }, + ) .await; let error = result.expect_err("queued request must time out"); assert_eq!(error.code(), &S3ErrorCode::SlowDown); assert_eq!(error.headers().expect("retry headers")[http::header::RETRY_AFTER], "1"); assert!(!started.load(Ordering::SeqCst), "timed-out work must not start"); assert_eq!(admission.waiting.available_permits(), 1); + assert_eq!(admission.in_flight.available_permits(), 2); drop(held); assert_eq!(admission.active.available_permits(), 1); } @@ -3368,7 +3574,8 @@ mod tests { tasks.push(tokio::spawn(await_bucket_usecase_on_fresh_task_with_admission( "test bucket operation", admission_for_task, - async move { + move |mutation_permit| async move { + let _mutation_permit = mutation_permit; started_for_task.fetch_add(1, Ordering::SeqCst); release_for_task.acquire().await.expect("release gate").forget(); Ok(()) @@ -3403,7 +3610,8 @@ mod tests { let queued = tokio::spawn(await_bucket_usecase_on_fresh_task_with_admission( "test bucket operation", admission.clone(), - async move { + move |mutation_permit| async move { + let _mutation_permit = mutation_permit; started_for_task.store(true, Ordering::SeqCst); Ok(()) }, @@ -3430,35 +3638,58 @@ mod tests { queued.abort(); assert!(queued.await.expect_err("queued request was cancelled").is_cancelled()); assert_eq!(admission.waiting.available_permits(), 1); + assert_eq!(admission.in_flight.available_permits(), 2); drop(held); assert!(!started.load(Ordering::SeqCst), "cancelled waiter must not mutate storage"); - await_bucket_usecase_on_fresh_task_with_admission("test bucket operation", admission, async { Ok(()) }) - .await - .expect("admission recovers after cancellation"); + await_bucket_usecase_on_fresh_task_with_admission("test bucket operation", admission, |mutation_permit| async move { + let _mutation_permit = mutation_permit; + Ok(()) + }) + .await + .expect("admission recovers after cancellation"); } #[tokio::test] async fn bucket_usecase_admission_maps_closed_semaphore_to_internal_error() { - let admission = BucketOperationAdmission::new(1, 1); - admission.active.close(); - let error = admission - .acquire("test bucket operation") - .await - .expect_err("closed admission must fail"); - assert_eq!(error.code(), &S3ErrorCode::InternalError); + for close_in_flight in [false, true] { + let admission = BucketOperationAdmission::new(1, 1); + if close_in_flight { + admission.in_flight.close(); + } else { + admission.active.close(); + } + let error = admission + .acquire("test bucket operation") + .await + .expect_err("closed admission must fail"); + assert_eq!(error.code(), &S3ErrorCode::InternalError); + assert_eq!(admission.in_flight.available_permits(), 2); + } } #[tokio::test] async fn bucket_usecase_task_maps_panics_to_internal_errors() { - async fn panicking_bucket_operation() -> S3Result<()> { - panic!("injected bucket task panic"); + for post_commit in [false, true] { + let admission = Arc::new(BucketOperationAdmission::new(1, 1)); + let result: S3Result<()> = await_bucket_usecase_on_fresh_task_with_admission( + "test bucket operation", + admission.clone(), + move |mutation_permit| async move { + let mut mutation_permit = Some(mutation_permit); + if post_commit { + drop(mutation_permit.take()); + } + panic!("injected bucket task panic"); + }, + ) + .await; + + let err = result.expect_err("a bucket task panic must be returned as an error"); + assert_eq!(err.code(), &S3ErrorCode::InternalError); + assert!(err.message().is_some_and(|message| message.contains("task failed"))); + assert_eq!(admission.active.available_permits(), 1); + assert_eq!(admission.in_flight.available_permits(), 2); } - - let result = await_bucket_usecase_on_fresh_task("test bucket operation", panicking_bucket_operation()).await; - - let err = result.expect_err("a bucket task panic must be returned as an error"); - assert_eq!(err.code(), &S3ErrorCode::InternalError); - assert!(err.message().is_some_and(|message| message.contains("task failed"))); } fn build_request(input: T, method: Method) -> S3Request { @@ -3482,30 +3713,46 @@ mod tests { } fn usecase_method_source<'a>(source: &'a str, method: &str) -> &'a str { - let start_marker = format!("pub async fn {method}"); - let start = source.find(&start_marker).expect("method should exist"); - let rest = &source[start + start_marker.len()..]; - let end = rest.find("\n pub async fn ").unwrap_or(rest.len()); + let public_start_marker = format!("pub async fn {method}"); + let private_start_marker = format!("async fn {method}"); + let (start, marker_len) = if let Some(start) = source.find(&public_start_marker) { + (start, public_start_marker.len()) + } else { + let start = source.find(&private_start_marker).expect("method should exist"); + (start, private_start_marker.len()) + }; + let rest = &source[start + marker_len..]; + let end = [rest.find("\n pub async fn "), rest.find("\n async fn ")] + .into_iter() + .flatten() + .min() + .unwrap_or(rest.len()); &rest[..end] } #[test] fn bucket_create_and_delete_use_admitted_fresh_tasks() { let source = include_str!("bucket_usecase.rs"); - for (method, operation, inner_method) in [ - ("execute_create_bucket", "bucket creation", "execute_create_bucket_inner"), - ("execute_delete_bucket", "bucket deletion", "execute_delete_bucket_inner"), + for (method, operation, inner_call) in [ + ( + "execute_create_bucket", + "bucket creation", + "execute_create_bucket_inner(req, mutation_permit).await", + ), + ("execute_delete_bucket", "bucket deletion", "execute_delete_bucket_inner(req).await"), ] { let body = usecase_method_source(source, method); assert!( body.contains(&format!("await_bucket_usecase_on_fresh_task(\"{operation}\"")), "{method} must use bounded detached-task admission" ); - assert!( - body.contains(&format!("{inner_method}(req).await")), - "{method} must run its mutation inside the admitted task" - ); + assert!(body.contains(inner_call), "{method} must run its mutation inside the admitted task"); } + let create_body = usecase_method_source(source, "execute_create_bucket_inner"); + assert!( + create_body.contains("notify_bucket_metadata_reload_with_context("), + "a successful bucket creation must refresh peer metadata caches" + ); } #[test] @@ -4060,16 +4307,22 @@ mod tests { #[test] fn create_bucket_exists_response_returns_ok_for_owner() { - let response = create_bucket_exists_response(true).expect("owner recreate should succeed"); + let response = create_bucket_exists_response(true, RUSTFS_REGION).expect("owner recreate in us-east-1 should succeed"); assert_eq!(response.output.location, None); } #[test] fn create_bucket_exists_response_returns_bucket_already_exists_for_non_owner() { - let err = create_bucket_exists_response(false).expect_err("non-owner recreate should fail"); + let err = create_bucket_exists_response(false, RUSTFS_REGION).expect_err("non-owner recreate should fail"); assert_eq!(err.code(), &S3ErrorCode::BucketAlreadyExists); } + #[test] + fn create_bucket_exists_response_returns_owned_by_you_outside_us_east_1() { + let err = create_bucket_exists_response(true, "eu-west-1").expect_err("owner recreate outside us-east-1 should fail"); + assert_eq!(err.code(), &S3ErrorCode::BucketAlreadyOwnedByYou); + } + #[test] fn build_request_with_req_info_preserves_owner_state() { let input = CreateBucketInput::builder() diff --git a/rustfs/src/app/context.rs b/rustfs/src/app/context.rs index f1373cb46..a9bdcd025 100644 --- a/rustfs/src/app/context.rs +++ b/rustfs/src/app/context.rs @@ -39,9 +39,7 @@ use crate::config::RustFSBufferConfig; use rustfs_config::server_config::Config; use rustfs_credentials::Credentials; use rustfs_iam::{ - error::Error as IamError, - federation::{FederatedIdentityService, oidc::OidcConfigQuery}, - store::object::ObjectStore, + error::Error as IamError, federation::FederatedIdentityService, oidc::OidcConfigQuery, store::object::ObjectStore, sys::IamSys, }; use rustfs_io_metrics::{PerformanceMetrics, internode_metrics::InternodeMetrics}; @@ -544,11 +542,8 @@ mod tests { use crate::config::{RustFSBufferConfig, WorkloadProfile}; use async_trait::async_trait; use rustfs_iam::{ - federation::{ - CoreFederatedAuthorizationMapper, FederatedIdentityService, - oidc::{OidcConfigQuery, StandardOidcAdapter}, - }, - oidc::OidcSys, + federation::{CoreFederatedAuthorizationMapper, FederatedIdentityService}, + oidc::{OidcConfigQuery, OidcSys, StandardOidcAdapter}, store::object::ObjectStore, sys::IamSys, }; diff --git a/rustfs/src/app/context/global.rs b/rustfs/src/app/context/global.rs index fe52e40db..c4ea0edc6 100644 --- a/rustfs/src/app/context/global.rs +++ b/rustfs/src/app/context/global.rs @@ -35,11 +35,7 @@ use super::interfaces::{ }; use crate::app::object_data_cache::ObjectDataCacheAdapter; use crate::app::object_traffic_health::ObjectTrafficHealth; -use rustfs_iam::{ - federation::{FederatedIdentityService, oidc::OidcConfigQuery}, - store::object::ObjectStore, - sys::IamSys, -}; +use rustfs_iam::{federation::FederatedIdentityService, oidc::OidcConfigQuery, store::object::ObjectStore, sys::IamSys}; use rustfs_kms::KmsServiceManager; use std::sync::{Arc, OnceLock}; diff --git a/rustfs/src/app/context/handles.rs b/rustfs/src/app/context/handles.rs index 6fbbddc50..2968af533 100644 --- a/rustfs/src/app/context/handles.rs +++ b/rustfs/src/app/context/handles.rs @@ -32,11 +32,7 @@ use crate::config::RustFSBufferConfig; use async_trait::async_trait; use rustfs_config::server_config::Config; use rustfs_credentials::Credentials; -use rustfs_iam::{ - federation::{FederatedIdentityService, oidc::OidcConfigQuery}, - store::object::ObjectStore, - sys::IamSys, -}; +use rustfs_iam::{federation::FederatedIdentityService, oidc::OidcConfigQuery, store::object::ObjectStore, sys::IamSys}; use rustfs_io_metrics::{PerformanceMetrics, internode_metrics::InternodeMetrics}; use rustfs_kms::KmsServiceManager; use rustfs_lock::LockClient; @@ -654,11 +650,8 @@ mod tests { use crate::app::context::interfaces::{KmsRuntimeInterface, ServerConfigInterface}; use rustfs_config::server_config::Config; use rustfs_iam::{ - federation::{ - CoreFederatedAuthorizationMapper, FederatedIdentityService, - oidc::{OidcConfigQuery, StandardOidcAdapter}, - }, - oidc::OidcSys, + federation::{CoreFederatedAuthorizationMapper, FederatedIdentityService}, + oidc::{OidcConfigQuery, OidcSys, StandardOidcAdapter}, }; use std::collections::HashMap; use std::sync::Arc; diff --git a/rustfs/src/app/context/interfaces.rs b/rustfs/src/app/context/interfaces.rs index 8fd9eb7bf..43052f5ad 100644 --- a/rustfs/src/app/context/interfaces.rs +++ b/rustfs/src/app/context/interfaces.rs @@ -22,11 +22,7 @@ use crate::config::RustFSBufferConfig; use async_trait::async_trait; use rustfs_config::server_config::Config; use rustfs_credentials::Credentials; -use rustfs_iam::{ - federation::{FederatedIdentityService, oidc::OidcConfigQuery}, - store::object::ObjectStore, - sys::IamSys, -}; +use rustfs_iam::{federation::FederatedIdentityService, oidc::OidcConfigQuery, store::object::ObjectStore, sys::IamSys}; use rustfs_io_metrics::{PerformanceMetrics, internode_metrics::InternodeMetrics}; use rustfs_kms::KmsServiceManager; use rustfs_lock::LockClient; diff --git a/rustfs/src/app/lifecycle_transition_api_test.rs b/rustfs/src/app/lifecycle_transition_api_test.rs index a35a12446..854ff88c5 100644 --- a/rustfs/src/app/lifecycle_transition_api_test.rs +++ b/rustfs/src/app/lifecycle_transition_api_test.rs @@ -589,26 +589,47 @@ async fn put_object_if_none_match_existing_object_returns_precondition_failed() .await .expect("Failed to fetch existing object info"); let existing_etag = existing_info.etag.expect("existing object should have an ETag"); + let initial_versions = live_object_version_count(&ecstore, bucket.as_str(), object).await; - let replacement_input = PutObjectInput::builder() - .bucket(bucket.clone()) - .key(object.to_string()) - .body(Some(streaming_blob_from_bytes(replacement_payload))) - .content_length(Some(replacement_payload.len() as i64)) - .build() - .unwrap(); - let mut req = build_request(replacement_input, Method::PUT); - req.headers - .insert(IF_NONE_MATCH, HeaderValue::from_str(existing_etag.as_str()).unwrap()); + for (condition, expected_code) in [ + (ETagCondition::Any, s3s::S3ErrorCode::PreconditionFailed), + (ETagCondition::ETag(ETag::Strong(existing_etag)), s3s::S3ErrorCode::NotImplemented), + ( + ETagCondition::ETag(ETag::Strong("different-etag".to_string())), + s3s::S3ErrorCode::NotImplemented, + ), + ] { + let header = condition.to_http_header().expect("conditional PUT header should encode"); + // Cover header-only callers and the DTO-plus-header shape of parsed HTTP requests. + for if_none_match in [None, Some(condition)] { + let has_dto = if_none_match.is_some(); + let replacement_input = PutObjectInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .body(Some(streaming_blob_from_bytes(replacement_payload))) + .content_length(Some(replacement_payload.len() as i64)) + .if_none_match(if_none_match) + .build() + .expect("conditional replacement PUT input should build"); + let mut req = build_request(replacement_input, Method::PUT); + req.headers.insert(IF_NONE_MATCH, header.clone()); - let err = Box::pin(usecase.execute_put_object(&fs, req)).await.unwrap_err(); - - assert_eq!(err.code(), &s3s::S3ErrorCode::PreconditionFailed); - assert_eq!( - read_object_bytes(&ecstore, bucket.as_str(), object).await, - initial_payload, - "failed conditional PutObject must not overwrite the current object" - ); + let err = Box::pin(usecase.execute_put_object(&fs, req)) + .await + .expect_err("conditional PUT must reject the replacement"); + assert_eq!(err.code(), &expected_code, "header={header:?}, has_dto={has_dto}"); + assert_eq!( + read_object_bytes(&ecstore, bucket.as_str(), object).await, + initial_payload, + "failed conditional PutObject must not overwrite the current object" + ); + assert_eq!( + live_object_version_count(&ecstore, bucket.as_str(), object).await, + initial_versions, + "failed conditional PutObject must not create a version" + ); + } + } } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] @@ -635,29 +656,50 @@ async fn copy_object_if_none_match_existing_destination_returns_precondition_fai .await .expect("Failed to fetch destination object info"); let dst_etag = dst_info.etag.expect("destination object should have an ETag"); + let initial_versions = live_object_version_count(&ecstore, dst_bucket.as_str(), dst_object).await; - let copy_input = CopyObjectInput::builder() - .copy_source(CopySource::Bucket { - bucket: src_bucket.clone().into(), - key: src_object.to_string().into(), - version_id: None, - }) - .bucket(dst_bucket.clone()) - .key(dst_object.to_string()) - .build() - .unwrap(); - let mut req = build_request(copy_input, Method::PUT); - req.headers - .insert(IF_NONE_MATCH, HeaderValue::from_str(dst_etag.as_str()).unwrap()); + for (condition, expected_code) in [ + (ETagCondition::Any, s3s::S3ErrorCode::PreconditionFailed), + (ETagCondition::ETag(ETag::Strong(dst_etag)), s3s::S3ErrorCode::NotImplemented), + ( + ETagCondition::ETag(ETag::Strong("different-etag".to_string())), + s3s::S3ErrorCode::NotImplemented, + ), + ] { + let header = condition.to_http_header().expect("conditional COPY header should encode"); + // Cover header-only callers and the DTO-plus-header shape of parsed HTTP requests. + for if_none_match in [None, Some(condition)] { + let has_dto = if_none_match.is_some(); + let copy_input = CopyObjectInput::builder() + .copy_source(CopySource::Bucket { + bucket: src_bucket.clone().into(), + key: src_object.to_string().into(), + version_id: None, + }) + .bucket(dst_bucket.clone()) + .key(dst_object.to_string()) + .if_none_match(if_none_match) + .build() + .expect("conditional COPY input should build"); + let mut req = build_request(copy_input, Method::PUT); + req.headers.insert(IF_NONE_MATCH, header.clone()); - let err = Box::pin(usecase.execute_copy_object(req)).await.unwrap_err(); - - assert_eq!(err.code(), &s3s::S3ErrorCode::PreconditionFailed); - assert_eq!( - read_object_bytes(&ecstore, dst_bucket.as_str(), dst_object).await, - dst_payload, - "failed conditional CopyObject must not overwrite the current destination object" - ); + let err = Box::pin(usecase.execute_copy_object(req)) + .await + .expect_err("conditional COPY must reject the replacement"); + assert_eq!(err.code(), &expected_code, "header={header:?}, has_dto={has_dto}"); + assert_eq!( + read_object_bytes(&ecstore, dst_bucket.as_str(), dst_object).await, + dst_payload, + "failed conditional CopyObject must not overwrite the current destination object" + ); + assert_eq!( + live_object_version_count(&ecstore, dst_bucket.as_str(), dst_object).await, + initial_versions, + "failed conditional CopyObject must not create a version" + ); + } + } } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] @@ -964,7 +1006,11 @@ async fn copy_object_allows_new_version_for_locked_destination_but_blocks_explic .await .expect_err("explicit CopyObject overwrite of a locked version should be blocked"); - assert_eq!(err.code(), &s3s::S3ErrorCode::AccessDenied); + assert_eq!(err.code(), &s3s::S3ErrorCode::InvalidArgument); + assert_eq!( + err.message(), + Some("Invalid argument: A destination versionId requires replication authorization") + ); assert_eq!(read_object_bytes(&ecstore, bucket.as_str(), dst_object).await, source_payload); assert_eq!(live_object_version_count(&ecstore, bucket.as_str(), dst_object).await, 2); } diff --git a/rustfs/src/app/metadata_route.rs b/rustfs/src/app/metadata_route.rs index 4cfd6b601..53615a8bb 100644 --- a/rustfs/src/app/metadata_route.rs +++ b/rustfs/src/app/metadata_route.rs @@ -271,7 +271,7 @@ async fn call_website( website_error_with_document(&req, &server_ctx, &target.bucket, config.error_document.as_ref(), StatusCode::NOT_FOUND) .await } - Err(err) if err.code() == &S3ErrorCode::NotModified => Ok(website_error(StatusCode::NOT_MODIFIED, true)), + Err(err) if err.code() == &S3ErrorCode::NotModified => Ok(website_not_modified(&err)), Err(_) if status.is_client_error() => { website_error_with_document(&req, &server_ctx, &target.bucket, config.error_document.as_ref(), status).await } @@ -520,6 +520,18 @@ fn website_error(status: StatusCode, head: bool) -> S3Response { } } +fn website_not_modified(error: &S3Error) -> S3Response { + let mut response = S3Response::new(Body::empty()); + response.status = Some(StatusCode::NOT_MODIFIED); + if let Some(headers) = error.headers() { + response.headers = headers.clone(); + } + response.headers.remove(http::header::CONTENT_LENGTH); + response.headers.remove(http::header::CONTENT_TYPE); + response.headers.remove(http::header::TRANSFER_ENCODING); + response +} + fn website_location(location: String, status: StatusCode) -> S3Result> { let mut response = S3Response::new(Body::empty()); response.status = Some(status); @@ -826,6 +838,39 @@ fn header_value(headers: &HeaderMap, name: &str) -> S3Result> { #[cfg(test)] mod tests { + #[tokio::test] + async fn website_304_preserves_cache_headers_without_content() { + use http_body_util::BodyExt; + let mut error = s3s::S3Error::new(s3s::S3ErrorCode::NotModified); + let mut headers = http::HeaderMap::new(); + for (name, value) in [ + ("etag", "\"abc\""), + ("cache-control", "max-age=60"), + ("expires", "Tue, 14 Nov 2023 22:13:20 GMT"), + ("content-length", "123"), + ("content-type", "application/xml"), + ] { + headers.insert(name, http::HeaderValue::from_static(value)); + } + error.set_headers(headers); + let response = super::website_not_modified(&error); + assert_eq!(response.status, Some(http::StatusCode::NOT_MODIFIED)); + assert_eq!(response.headers["etag"], "\"abc\""); + assert_eq!(response.headers["cache-control"], "max-age=60"); + assert!(response.headers.contains_key("expires")); + assert!(!response.headers.contains_key("content-length")); + assert!(!response.headers.contains_key("content-type")); + assert!( + response + .output + .collect() + .await + .expect("empty website body") + .to_bytes() + .is_empty() + ); + } + use super::{ MetadataOperation, list_object_versions_input, list_objects_v2_input, metadata_operation, website_redirect, website_relative_redirect, website_rule_matches, website_target, diff --git a/rustfs/src/app/multipart_usecase.rs b/rustfs/src/app/multipart_usecase.rs index 149e427bf..c5d368b35 100644 --- a/rustfs/src/app/multipart_usecase.rs +++ b/rustfs/src/app/multipart_usecase.rs @@ -72,7 +72,7 @@ use super::storage_api::multipart_usecase::{ use super::trailer_adapter::trailer_source; use crate::app::object::{ ConcurrencyManager, ForegroundWriteAdmission, get_concurrency_manager, guard_put_object_body_read_timeout, - put_object_body_read_timeout, reject_oversize_single_upload, + put_object_body_read_timeout, reject_non_wildcard_write_if_none_match, reject_oversize_single_upload, }; use crate::app::object_data_cache::{ ObjectDataCacheAdapter, invalidate_object_data_cache_after_complete_multipart_success, @@ -548,7 +548,7 @@ impl DefaultMultipartUsecase { #[hotpath::measure(impl_type = "MultipartUsecase")] pub async fn execute_complete_multipart_upload( &self, - req: S3Request, + mut req: S3Request, ) -> S3Result> { reject_presigned_multipart_max_total_object_size_for_other_operation( &req.headers, @@ -566,7 +566,7 @@ impl DefaultMultipartUsecase { S3Operation::CompleteMultipartUpload, ); let replication_authorized = replication_request_authorized(&req); - let input = req.input.clone(); + let input = std::mem::take(&mut req.input); let CompleteMultipartUploadInput { multipart_upload, bucket, @@ -580,6 +580,7 @@ impl DefaultMultipartUsecase { .. } = input; + reject_non_wildcard_write_if_none_match(if_none_match.as_ref(), &req.headers)?; validate_table_catalog_object_mutation(&bucket, &key).await?; if if_match.is_some() || if_none_match.is_some() { @@ -589,23 +590,22 @@ impl DefaultMultipartUsecase { match store.get_object_info(&bucket, &key, &ObjectOptions::default()).await { Ok(info) => { - if !info.delete_marker { - if let Some(ifmatch) = if_match - && let Some(strong_etag) = ifmatch.into_etag() - && info - .etag - .as_ref() - .is_some_and(|etag| ETag::Strong(etag.clone()) != strong_etag) + if info.delete_marker { + if if_match.is_some() { + return Err(ApiError::from(StorageError::ObjectNotFound(bucket.clone(), key.clone())).into()); + } + } else { + if let Some(ifmatch) = &if_match + && !ifmatch.is_any() + && ifmatch + .as_etag() + .and_then(ETag::as_strong) + .is_none_or(|expected| info.etag.as_deref() != Some(expected)) { return Err(s3_error!(PreconditionFailed)); } - if let Some(ifnonematch) = if_none_match - && let Some(strong_etag) = ifnonematch.into_etag() - && info - .etag - .as_ref() - .is_some_and(|etag| ETag::Strong(etag.clone()) == strong_etag) - { + // The request guard permits only the wildcard here. + if if_none_match.is_some() { return Err(s3_error!(PreconditionFailed)); } } @@ -1124,9 +1124,11 @@ impl DefaultMultipartUsecase { ); } - let current_opts: ObjectOptions = get_opts(&bucket, &key, opts.version_id.clone(), None, &req.headers) - .await - .map_err(ApiError::from)?; + let current_opts: ObjectOptions = internal_object_info_lookup_opts( + get_opts(&bucket, &key, opts.version_id.clone(), None, &req.headers) + .await + .map_err(ApiError::from)?, + ); match store.get_object_info(&bucket, &key, ¤t_opts).await { Ok(existing_obj_info) => { validate_existing_object_lock_for_write(&object_lock_config_state, &existing_obj_info, &opts)? @@ -2892,6 +2894,351 @@ mod tests { ); } + #[test] + #[serial_test::serial] + fn conditional_multipart_upload_if_none_match_returns_precondition_failed_when_object_exists() { + crate::app::gating_test_env::run_large_stack_test( + "conditional-mpu-if-none-match", + conditional_multipart_upload_if_none_match_returns_precondition_failed_when_object_exists_inner, + ); + } + + async fn conditional_multipart_upload_if_none_match_returns_precondition_failed_when_object_exists_inner() { + use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions}; + use crate::app::storage_api::test::contract::object::ObjectOperations as _; + + let (store, bucket) = + crate::app::gating_test_env::durable_quota_test_bucket("conditional-mpu-if-none-match", 16_384).await; + let object = "conditional-object"; + let usecase = DefaultMultipartUsecase::from_global(); + let initial_body = b"initial multipart body"; + + let create_input = CreateMultipartUploadInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .build() + .expect("create multipart input should build"); + let mut initial_create_req = build_request(create_input.clone(), Method::POST); + initial_create_req + .headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*")); + let first_upload_id = usecase + .execute_create_multipart_upload(initial_create_req) + .await + .expect("initial conditional create multipart upload should succeed") + .output + .upload_id + .expect("create response should contain upload id"); + + let mut reader = PutObjReader::from_vec(initial_body.to_vec()); + let first_part = store + .put_object_part(&bucket, object, &first_upload_id, 1, &mut reader, &ObjectOptions::default()) + .await + .expect("initial multipart part should be staged"); + let first_complete_input = CompleteMultipartUploadInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .upload_id(first_upload_id) + .if_none_match(Some(s3s::dto::ETagCondition::Any)) + .multipart_upload(Some(CompletedMultipartUpload { + parts: Some(vec![CompletedPart { + part_number: Some(1), + e_tag: first_part.etag.map(|etag| to_s3s_etag(&etag)), + ..Default::default() + }]), + })) + .build() + .expect("initial complete multipart input should build"); + let mut first_complete_req = build_request(first_complete_input, Method::POST); + first_complete_req + .headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*")); + usecase + .execute_complete_multipart_upload(first_complete_req) + .await + .expect("initial conditional complete multipart upload should succeed"); + + let committed_info = store + .get_object_info(&bucket, object, &ObjectOptions::default()) + .await + .expect("committed object info should be readable"); + let committed_etag = committed_info.etag.clone().expect("committed object should have an ETag"); + assert_eq!(committed_info.size, initial_body.len() as i64); + + let mut second_create_req = build_request(create_input.clone(), Method::POST); + second_create_req + .headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*")); + let create_err = usecase + .execute_create_multipart_upload(second_create_req) + .await + .expect_err("create multipart upload with If-None-Match: * on existing key must fail"); + assert_eq!(create_err.code(), &S3ErrorCode::PreconditionFailed); + + let second_upload_id = usecase + .execute_create_multipart_upload(build_request(create_input, Method::POST)) + .await + .expect("unconditional create multipart upload should succeed") + .output + .upload_id + .expect("second create response should contain upload id"); + let mut second_reader = PutObjReader::from_vec(b"replacement multipart body".to_vec()); + let second_part = store + .put_object_part(&bucket, object, &second_upload_id, 1, &mut second_reader, &ObjectOptions::default()) + .await + .expect("second multipart part should be staged"); + let completed_parts = CompletedMultipartUpload { + parts: Some(vec![CompletedPart { + part_number: Some(1), + e_tag: second_part.etag.map(|etag| to_s3s_etag(&etag)), + ..Default::default() + }]), + }; + + // DTO-only If-None-Match: * exercises the early precheck in execute_complete_multipart_upload. + let dto_only_complete_input = CompleteMultipartUploadInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .upload_id(second_upload_id.clone()) + .if_none_match(Some(s3s::dto::ETagCondition::Any)) + .multipart_upload(Some(completed_parts.clone())) + .build() + .expect("DTO-only complete multipart input should build"); + let dto_complete_err = usecase + .execute_complete_multipart_upload(build_request(dto_only_complete_input, Method::POST)) + .await + .expect_err("DTO-only If-None-Match: * on existing key must fail"); + assert_eq!(dto_complete_err.code(), &S3ErrorCode::PreconditionFailed); + + for condition in [ETag::Weak(committed_etag.clone()), ETag::Strong("different-etag".to_string())] { + let input = CompleteMultipartUploadInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .upload_id(second_upload_id.clone()) + .if_match(Some(s3s::dto::ETagCondition::ETag(condition))) + .multipart_upload(Some(completed_parts.clone())) + .build() + .expect("If-Match complete input should build"); + let err = usecase + .execute_complete_multipart_upload(build_request(input, Method::POST)) + .await + .expect_err("weak or nonmatching If-Match must reject the overwrite"); + assert_eq!(err.code(), &S3ErrorCode::PreconditionFailed); + } + + // Write requests reject ETag-valued If-None-Match before checking object state. + let weak_if_none_match_input = CompleteMultipartUploadInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .upload_id(second_upload_id.clone()) + .if_none_match(Some(s3s::dto::ETagCondition::ETag(ETag::Weak(committed_etag.clone())))) + .multipart_upload(Some(completed_parts.clone())) + .build() + .expect("weak If-None-Match complete multipart input should build"); + let weak_complete_err = usecase + .execute_complete_multipart_upload(build_request(weak_if_none_match_input, Method::POST)) + .await + .expect_err("weak If-None-Match ETag must be rejected as unsupported"); + assert_eq!(weak_complete_err.code(), &S3ErrorCode::NotImplemented); + + let second_complete_input = CompleteMultipartUploadInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .upload_id(second_upload_id) + .if_none_match(Some(s3s::dto::ETagCondition::Any)) + .multipart_upload(Some(completed_parts)) + .build() + .expect("second complete multipart input should build"); + let mut second_complete_req = build_request(second_complete_input, Method::POST); + second_complete_req + .headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*")); + let complete_err = usecase + .execute_complete_multipart_upload(second_complete_req) + .await + .expect_err("complete multipart upload with If-None-Match: * on existing key must fail"); + assert_eq!(complete_err.code(), &S3ErrorCode::PreconditionFailed); + + let after_info = store + .get_object_info(&bucket, object, &ObjectOptions::default()) + .await + .expect("original object must remain readable after rejected conditional completions"); + assert_eq!(after_info.etag.as_deref(), Some(committed_etag.as_str())); + assert_eq!(after_info.size, initial_body.len() as i64); + + // Two-pool regression: when Pool 0 holds an incomplete multipart upload and Pool 1 + // holds the committed object, conditional new_multipart_upload and complete_multipart_upload + // must still reject If-None-Match: * with PreconditionFailed. + let (_multi_temp_dir, _pool_paths, multi_store) = crate::app::gating_test_env::isolated_multi_pool_ecstore().await; + let multi_bucket = format!("multi-pool-cond-mpu-{}", Uuid::new_v4().simple()); + multi_store + .make_bucket(&multi_bucket, &MakeBucketOptions::default()) + .await + .expect("create multi-pool test bucket"); + let multi_object = "cross-pool-conditional-object"; + let staged_pool_0 = multi_store.pools[0] + .new_multipart_upload(&multi_bucket, multi_object, &ObjectOptions::default()) + .await + .expect("stage incomplete upload in pool 0"); + let mut pool0_part_reader = PutObjReader::from_vec(b"pool-0-staged-part".to_vec()); + let pool0_part = multi_store.pools[0] + .put_object_part( + &multi_bucket, + multi_object, + &staged_pool_0.upload_id, + 1, + &mut pool0_part_reader, + &ObjectOptions::default(), + ) + .await + .expect("write staged part in pool 0"); + let mut pool1_reader = PutObjReader::from_vec(b"pool-1-committed-object".to_vec()); + multi_store.pools[1] + .put_object(&multi_bucket, multi_object, &mut pool1_reader, &ObjectOptions::default()) + .await + .expect("commit object in pool 1"); + + let multi_cond_opts = ObjectOptions { + http_preconditions: Some(HTTPPreconditions { + if_none_match: Some("*".to_string()), + ..Default::default() + }), + ..Default::default() + }; + let multi_create_err = multi_store + .new_multipart_upload(&multi_bucket, multi_object, &multi_cond_opts) + .await + .expect_err("multi-pool new_multipart_upload with If-None-Match: * must fail when object is in pool 1"); + assert_eq!(multi_create_err, StorageError::PreconditionFailed); + + let multi_complete_err = Arc::clone(&multi_store) + .complete_multipart_upload( + &multi_bucket, + multi_object, + &staged_pool_0.upload_id, + vec![CompletePart { + part_num: 1, + etag: pool0_part.etag, + ..Default::default() + }], + &multi_cond_opts, + ) + .await + .expect_err("multi-pool complete_multipart_upload with If-None-Match: * must fail when object is in pool 1"); + assert_eq!(multi_complete_err, StorageError::PreconditionFailed); + metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + + // Delete-marker regression: when the current version in a versioned bucket is a delete + // marker, If-Match must fail with NoSuchKey while If-None-Match: * succeeds. + let versioned_bucket = format!("versioned-cond-mpu-{}", Uuid::new_v4().simple()); + store + .make_bucket( + &versioned_bucket, + &MakeBucketOptions { + versioning_enabled: true, + ..Default::default() + }, + ) + .await + .expect("create versioned bucket"); + let mut v_reader = PutObjReader::from_vec(b"v1-body".to_vec()); + store + .put_object( + &versioned_bucket, + object, + &mut v_reader, + &ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect("write initial versioned object"); + let marker = store + .delete_object( + &versioned_bucket, + object, + ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect("delete versioned object to create delete marker"); + assert!(marker.delete_marker); + + let dm_create_input = CreateMultipartUploadInput::builder() + .bucket(versioned_bucket.clone()) + .key(object.to_string()) + .build() + .expect("delete-marker create input"); + let mut dm_create_req = build_request(dm_create_input, Method::POST); + dm_create_req + .headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*")); + let dm_upload_id = usecase + .execute_create_multipart_upload(dm_create_req) + .await + .expect("If-None-Match: * create must succeed past a delete marker") + .output + .upload_id + .expect("upload id"); + let mut dm_part_reader = PutObjReader::from_vec(b"after-delete-marker".to_vec()); + let dm_part = store + .put_object_part( + &versioned_bucket, + object, + &dm_upload_id, + 1, + &mut dm_part_reader, + &ObjectOptions::default(), + ) + .await + .expect("stage part past delete marker"); + let dm_parts = CompletedMultipartUpload { + parts: Some(vec![CompletedPart { + part_number: Some(1), + e_tag: dm_part.etag.map(|etag| to_s3s_etag(&etag)), + ..Default::default() + }]), + }; + + let dm_if_match_input = CompleteMultipartUploadInput::builder() + .bucket(versioned_bucket.clone()) + .key(object.to_string()) + .upload_id(dm_upload_id.clone()) + .if_match(Some(s3s::dto::ETagCondition::Any)) + .multipart_upload(Some(dm_parts.clone())) + .build() + .expect("delete-marker If-Match complete input"); + let mut dm_if_match_req = build_request(dm_if_match_input, Method::POST); + dm_if_match_req + .headers + .insert(http::header::IF_MATCH, HeaderValue::from_static("*")); + let dm_if_match_err = usecase + .execute_complete_multipart_upload(dm_if_match_req) + .await + .expect_err("If-Match against a current delete marker must fail with NoSuchKey"); + assert_eq!(dm_if_match_err.code(), &S3ErrorCode::NoSuchKey); + + let dm_if_none_match_input = CompleteMultipartUploadInput::builder() + .bucket(versioned_bucket) + .key(object.to_string()) + .upload_id(dm_upload_id) + .if_none_match(Some(s3s::dto::ETagCondition::Any)) + .multipart_upload(Some(dm_parts)) + .build() + .expect("delete-marker If-None-Match complete input"); + let mut dm_if_none_match_req = build_request(dm_if_none_match_input, Method::POST); + dm_if_none_match_req + .headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*")); + usecase + .execute_complete_multipart_upload(dm_if_none_match_req) + .await + .expect("If-None-Match: * completion must succeed past a delete marker"); + } + #[tokio::test] #[serial_test::serial] async fn create_multipart_rejects_ciphertext_replication_before_parts_are_staged() { @@ -2966,7 +3313,7 @@ mod tests { #[tokio::test] #[serial_test::serial] - async fn multipart_completion_rejects_rotated_quota_capability_before_rename() { + async fn multipart_completion_commits_after_quota_fleet_proof_loss_before_rename() { use crate::app::storage_api::test::set_disk::{MultipartCommitBarrier, MultipartCommitPause}; let (store, bucket) = crate::app::gating_test_env::durable_quota_test_bucket("rotated-proof-mpu-quota", 4096).await; @@ -3000,27 +3347,19 @@ mod tests { .await }); barrier.wait_until_paused().await; - assert!( - crate::storage::storage_api::ecstore_notification::rotate_cross_pool_fence_fleet_proof_for_test(), - "the gating environment must have a current fleet proof" - ); + let _without_proof = crate::storage::storage_api::ecstore_notification::without_cross_pool_fence_fleet_proof_for_test(); + assert!(crate::storage::storage_api::ecstore_notification::acquire_cross_pool_fence_fleet_proof().is_none()); barrier.release(); - let err = complete + complete .await .expect("completion task should not panic") - .expect_err("a replaced fleet proof must fence multipart rename"); - assert!(matches!( - err, - StorageError::NamespaceLockQuorumUnavailable { - mode: "quota_reservation", - .. - } - )); - store - .get_multipart_info(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .expect("fleet proof loss must not invalidate multipart quota reservation and disk fences"); + let stored = store + .get_object_info(&bucket, object, &ObjectOptions::default()) .await - .expect("proof rotation must preserve the multipart upload for retry"); + .expect("completed multipart object must remain readable after fleet proof loss"); + assert_eq!(stored.size, 4096); } #[tokio::test] diff --git a/rustfs/src/app/object/conditional_read_bench.rs b/rustfs/src/app/object/conditional_read_bench.rs new file mode 100644 index 000000000..e0acc3839 --- /dev/null +++ b/rustfs/src/app/object/conditional_read_bench.rs @@ -0,0 +1,148 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Manually invoked, paired application-read benchmark on real local EC disks. +//! Run the identical harness on both revisions in separate processes. Environment +//! configuration must be set before either process initializes the store. + +use super::*; +use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions}; +use futures::stream; +use http::{HeaderValue, Method}; +use metrics_util::debugging::{DebugValue, DebuggingRecorder}; +use std::time::Instant; + +fn percentile(samples: &mut [f64], percent: usize) -> f64 { + assert!(!samples.is_empty(), "benchmark must collect samples"); + samples.sort_by(f64::total_cmp); + samples[(samples.len() * percent).div_ceil(100).saturating_sub(1)] +} + +async fn read_once(usecase: &DefaultObjectUsecase, object: &str, etag: &str, kind: &str, expected: &[u8]) -> f64 { + let start = Instant::now(); + let mut req = build_request( + GetObjectInput::builder() + .bucket("conditional-bench".to_owned()) + .key(object.to_owned()) + .build() + .expect("benchmark input"), + Method::GET, + ); + if kind != "unconditional" { + let tag = if kind == "hit" { etag } else { "unmatched-benchmark-etag" }; + req.headers.insert( + http::header::IF_NONE_MATCH, + HeaderValue::from_str(&format!("\"{tag}\"")).expect("benchmark condition"), + ); + } + match usecase.execute_get_object(req).await { + Err(error) => { + assert_eq!(kind, "hit"); + assert_eq!(error.code(), &S3ErrorCode::NotModified); + } + Ok(mut response) => { + assert_ne!(kind, "hit"); + let mut body = response.output.body.take().expect("benchmark body"); + let mut actual = Vec::with_capacity(expected.len()); + while let Some(chunk) = body.next().await { + actual.extend_from_slice(&chunk.expect("benchmark body chunk")); + } + assert_eq!(actual, expected, "benchmark must consume and validate the whole body"); + } + } + start.elapsed().as_secs_f64() * 1000.0 +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "manual benchmark: use the paired runner with --ignored --nocapture"] +async fn conditional_read_paired_benchmark() { + let iterations = std::env::var("RUSTFS_CONDITIONAL_BENCH_ITERATIONS") + .unwrap_or_else(|_| "1000".to_owned()) + .parse::() + .expect("benchmark iteration count"); + let concurrency = std::env::var("RUSTFS_CONDITIONAL_BENCH_CONCURRENCY") + .unwrap_or_else(|_| "8".to_owned()) + .parse::() + .expect("benchmark concurrency"); + assert!(iterations >= 100 && concurrency > 0); + let recorder = DebuggingRecorder::new(); + let snapshotter = recorder.snapshotter(); + metrics::set_global_recorder(recorder).expect("run the benchmark alone in a fresh process"); + let _metadata_profile = crate::app::storage_api::test::set_disk::ConditionalReadBenchmarkMetadataGuard::acquire(); + let (store, context) = real_store_test_context().await; + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let cache_enabled = !context.object_data_cache().is_disabled(); + let usecase = DefaultObjectUsecase::with_context(Some(context)); + store + .make_bucket("conditional-bench", &MakeBucketOptions::default()) + .await + .expect("benchmark bucket"); + for size in [4096, 1_300_000] { + let object = format!("bench-{size}"); + let expected = (0..size).map(|index| (index % 251) as u8).collect::>(); + let info = put_real_cold_fill_object(&store, "conditional-bench", &object, &expected).await; + let etag = info.etag.as_deref().expect("benchmark ETag"); + for kind in ["unconditional", "hit", "miss"] { + // Prime the body cache and filesystem for every case, including 304. + for _ in 0..20 { + read_once(&usecase, &object, etag, "unconditional", &expected).await; + } + snapshotter.snapshot(); + let start = Instant::now(); + let mut latencies = stream::iter(0..iterations) + .map(|_| read_once(&usecase, &object, etag, kind, &expected)) + .buffer_unordered(concurrency) + .collect::>() + .await; + let elapsed = start.elapsed().as_secs_f64(); + let mut lock_ms = Vec::new(); + let mut metadata_cache_hits = 0; + for (key, _, _, value) in snapshotter.snapshot().into_vec() { + if key.key().name() == "rustfs_io_get_object_metadata_cache_total" + && key + .key() + .labels() + .any(|label| label.key() == "decision" && label.value() == "hit") + && let DebugValue::Counter(count) = &value + { + metadata_cache_hits += count; + } + if key.key().name() == "rustfs_object_lock_diag_hold_duration_seconds" + && key + .key() + .labels() + .any(|label| label.key() == "mode" && label.value() == "read") + && let DebugValue::Histogram(values) = value + { + lock_ms.extend(values.iter().map(|sample| sample.0 * 1000.0)); + } + } + assert!(!lock_ms.is_empty(), "enable RUSTFS_OBJECT_LOCK_DIAG_ENABLE before startup"); + assert_eq!(metadata_cache_hits, 0, "slow-disk measurements require a metadata-cache bypass"); + println!( + "CONDITIONAL_BENCH {}", + serde_json::json!({ + "size": size, "kind": kind, "cache": cache_enabled, + "iterations": iterations, "concurrency": concurrency, + "metadata_cache_hits": metadata_cache_hits, "metadata_cache_bypassed": true, + "slowtail_ms": std::env::var("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS").unwrap_or_default(), + "ops_per_sec": iterations as f64 / elapsed, + "p95_ms": percentile(&mut latencies, 95), "p99_ms": percentile(&mut latencies, 99), + "lock_samples": lock_ms.len(), "lock_p95_ms": percentile(&mut lock_ms, 95), + "lock_p99_ms": percentile(&mut lock_ms, 99), + }) + ); + } + } +} diff --git a/rustfs/src/app/object/copy.rs b/rustfs/src/app/object/copy.rs index 6dcc5acef..350f7443c 100644 --- a/rustfs/src/app/object/copy.rs +++ b/rustfs/src/app/object/copy.rs @@ -15,6 +15,7 @@ //! CopyObject path. use super::*; +use rustfs_utils::path::fold_object_name_path_separators; use crate::auth::{VerifiedPresignedRequest, reject_presigned_put_max_content_length_for_other_operation}; use crate::error::ServerSideSourceReadError; @@ -96,6 +97,16 @@ fn copy_namespace_lock_error(bucket: &str, object: &str, mode: &'static str, err } } +/// Object name the namespace lock of a copy endpoint is keyed by: the disk +/// identity of the stored name. On Windows that folds spellings +/// `path_join_buf` keeps apart (`a\b\` vs `a/b`), so a copy that is not a +/// same-object copy can still have its source read guard and destination +/// write lock on one lock, and must take the self-copy lock to not wait on +/// itself. +fn copy_namespace_lock_object(key: &str, backslash_is_separator: bool) -> String { + fold_object_name_path_separators(&encode_dir_object(key), backslash_is_separator).into_owned() +} + async fn acquire_self_copy_namespace_lock( store: &S, bucket: &str, @@ -161,6 +172,7 @@ impl DefaultObjectUsecase { req.uri.query(), req.extensions.get::().is_some(), )?; + reject_non_wildcard_write_if_none_match(req.input.if_none_match.as_ref(), &req.headers)?; if let Some(context) = &self.context { let _ = context.object_store(); } @@ -332,6 +344,10 @@ impl DefaultObjectUsecase { apply_bucket_generation_guard(&req, &bucket, &mut dst_opts)?; let cp_src_dst_same = path_join_buf(&[&src_bucket, &src_key]) == path_join_buf(&[&bucket, &key]); + let backslash_is_separator = cfg!(target_os = "windows"); + let dst_lock_object = copy_namespace_lock_object(&key, backslash_is_separator); + let self_copy_lock = cp_src_dst_same + || (src_bucket == bucket && copy_namespace_lock_object(&src_key, backslash_is_separator) == dst_lock_object); let expected_current_version_id = expected_current_version_id(&req.headers)?; if expected_current_version_id.is_some() && (!cp_src_dst_same || version_id.is_none() || dest_version_id.is_some() || !dst_opts.versioned) @@ -393,7 +409,7 @@ impl DefaultObjectUsecase { // with that key (for example, copying `bucket/bucket` onto itself). let bucket_sse_config = load_bucket_default_sse_config(&bucket).await?; let object_lock_config_state = load_bucket_object_lock_config_state(&bucket).await?; - if cp_src_dst_same && key == bucket { + if self_copy_lock && dst_lock_object == bucket { dst_opts.object_lock_config_snapshot = Some(store.object_lock_config_snapshot(&bucket).await.map_err(ApiError::from)?); } @@ -407,7 +423,7 @@ impl DefaultObjectUsecase { // expected-current historical copies. With lock optimization disabled, the source // stream retains its read guard until EOF; taking the write lock later in ECStore // would self-deadlock before the copy can consume that stream. - let _self_copy_lock_guard = if cp_src_dst_same { + let _self_copy_lock_guard = if self_copy_lock { let guard = acquire_self_copy_namespace_lock(store.as_ref(), &bucket, &key).await?; src_opts.no_lock = true; src_get_opts.no_lock = true; @@ -923,6 +939,21 @@ mod tests { use std::sync::Arc; use tokio::io::AsyncReadExt; + #[test] + fn copy_namespace_lock_object_folds_backslash_spellings_only_as_separators() { + // `a\b\` is not a same-object copy of `a/b` by `path_join_buf` (the + // trailing separator survives), yet on Windows both are the xl.meta + // of `a/b` and share one namespace lock. + assert_eq!(copy_namespace_lock_object("a\\b\\", true), copy_namespace_lock_object("a/b", true)); + assert_eq!(copy_namespace_lock_object("\\a\\b", true), "a/b"); + assert_ne!(copy_namespace_lock_object("a\\b\\", false), copy_namespace_lock_object("a/b", false)); + // Directory objects keep their encoded identity. + assert_eq!(copy_namespace_lock_object("a/b/", true), "a/b__XLDIR__"); + assert_ne!(copy_namespace_lock_object("a/b/", true), copy_namespace_lock_object("a/b", true)); + assert_eq!(copy_namespace_lock_object("bucket", true), "bucket"); + assert_eq!(copy_namespace_lock_object("\\bucket", true), "bucket"); + } + #[test] fn local_copy_does_not_inherit_source_replication_generation() { let mut metadata = HashMap::from([ diff --git a/rustfs/src/app/object/delete.rs b/rustfs/src/app/object/delete.rs index f231fc904..c0202e388 100644 --- a/rustfs/src/app/object/delete.rs +++ b/rustfs/src/app/object/delete.rs @@ -15,6 +15,9 @@ //! DeleteObject / DeleteObjects path. use super::*; +use crate::app::storage_api::object_usecase::DeleteObjectCondition; +use s3s::dto::ObjectIdentifier; +use std::collections::HashSet; async fn authorize_recursive_delete( req: &mut S3Request, @@ -110,6 +113,48 @@ fn normalize_delete_objects_version_id( } } +fn parse_delete_objects_condition(object: &ObjectIdentifier) -> Result, s3s::dto::Error> { + let Some(etag) = object.e_tag.as_ref() else { + return Ok(None); + }; + let (code, message) = if object.version_id.is_some() { + ( + S3ErrorCode::InvalidRequest, + "An ETag condition cannot be combined with an explicit VersionId".to_string(), + ) + } else if let Some(etag) = etag.as_strong() { + match DeleteObjectCondition::parse(etag) { + Ok(condition) => return Ok(Some(condition)), + Err(error) => (S3ErrorCode::InvalidArgument, error.to_string()), + } + } else { + (S3ErrorCode::InvalidArgument, "A delete ETag condition must use a strong ETag".to_string()) + }; + Err(s3s::dto::Error { + code: Some(code.as_str().to_string()), + key: Some(object.key.clone()), + message: Some(message), + version_id: object.version_id.clone(), + }) +} + +fn delete_objects_action_identity( + deleted: &StorageDeletedObject, + requested_version_id: Option, +) -> (String, Option, bool) { + let creates_delete_marker = deleted.delete_marker && requested_version_id.is_none(); + let target_version = if creates_delete_marker { + deleted.delete_marker_version_id + } else { + deleted.version_id.or(requested_version_id) + }; + ( + deleted.object_name.clone(), + target_version.filter(|version| !version.is_nil()), + creates_delete_marker, + ) +} + #[cfg(test)] type DeleteSnapshotTestHook = (String, Arc, Arc); @@ -391,6 +436,17 @@ fn delete_response_version_id(version_id: Option, synthetic_version_id: bo } } +fn delete_marker_response_version_id( + committed_version_id: Option, + creates_null_delete_marker: bool, + synthetic_version_id: bool, +) -> Option { + // The suspended-marker write shape uses None on disk. Only its response + // identity is null; changing the write shape would change delete semantics. + let version_id = committed_version_id.or_else(|| creates_null_delete_marker.then(Uuid::nil)); + delete_response_version_id(version_id, synthetic_version_id) +} + fn project_delete_objects_pre_stat_error( object: ObjectToDelete, synthetic_version_id: bool, @@ -437,7 +493,7 @@ fn reduce_delete_objects_result<'a>( ) -> Result<&'a StorageDeletedObject, s3s::dto::Error> { match error { None => Ok(deleted), - Some(error) if is_delete_objects_not_found(error) => Ok(deleted), + Some(error) if object.delete_condition.is_none() && is_delete_objects_not_found(error) => Ok(deleted), Some(error) => { let api_error = ApiError::from(error.clone()); Err(s3s::dto::Error { @@ -490,7 +546,9 @@ impl DefaultObjectUsecase { struct DeleteResult { delete_object: Option, error: Option, + emit_side_effects: bool, synthetic_version_id: bool, + version_suspended: bool, // The version identity the request addressed, before any // synthetic-directory synthesis. `None` means the request carried // no version id, i.e. a committed delete marker was created, not @@ -520,6 +578,13 @@ impl DefaultObjectUsecase { }); continue; } + let delete_condition = match parse_delete_objects_condition(obj_id) { + Ok(condition) => condition, + Err(error) => { + delete_results[idx].error = Some(error); + continue; + } + }; let raw_version_id = obj_id.version_id.clone(); let (version_id, version_uuid) = match normalize_delete_objects_version_id(raw_version_id.clone()) { Ok(parsed) => parsed, @@ -541,7 +606,14 @@ impl DefaultObjectUsecase { req_info.version_id = version_id.clone(); } - let auth_res = authorize_request(&mut req, delete_object_authorize_action(version_id.as_deref())).await; + let mut auth_res = authorize_request(&mut req, delete_object_authorize_action(version_id.as_deref())).await; + if auth_res.is_ok() + && delete_condition + .as_ref() + .is_some_and(DeleteObjectCondition::requires_read_permission) + { + auth_res = authorize_request(&mut req, Action::S3Action(S3Action::GetObjectAction)).await; + } if auth_res.is_err() { if !bulk_denial_logged { bulk_denial_logged = true; @@ -583,10 +655,11 @@ impl DefaultObjectUsecase { continue; } - let synthetic_version_id = version_id.is_none() && is_dir_object(&obj_id.key); + let synthetic_version_id = delete_condition.is_none() && version_id.is_none() && is_dir_object(&obj_id.key); let object = ObjectToDelete { object_name: obj_id.key.clone(), version_id: version_uuid, + delete_condition, synthetic_version_id, ..Default::default() }; @@ -598,7 +671,7 @@ impl DefaultObjectUsecase { if authorized_deletes.is_empty() { let output = DeleteObjectsOutput { - deleted: Some(Vec::new()), + deleted: (!delete.quiet.unwrap_or_default()).then(Vec::new), errors: Some(delete_results.into_iter().filter_map(|result| result.error).collect()), ..Default::default() }; @@ -687,7 +760,7 @@ impl DefaultObjectUsecase { opts, skip_stat, } = prepared; - let synthetic_version_id = object.version_id.is_none() && is_dir_object(&object.object_name); + let synthetic_version_id = object.synthetic_version_id; let error = if skip_stat { None } else { @@ -763,6 +836,7 @@ impl DefaultObjectUsecase { return result; } + let mut committed_actions = HashSet::new(); for (i, err) in errs.iter().enumerate() { let didx = object_to_delete_idx[i]; @@ -773,8 +847,15 @@ impl DefaultObjectUsecase { delete_results[didx].synthetic_version_id, ) { Ok(deleted_object) => { - delete_results[didx].delete_object = Some(deleted_object.clone()); let (versioned, version_suspended) = object_versioning[i]; + delete_results[didx].delete_object = Some(deleted_object.clone()); + delete_results[didx].version_suspended = version_suspended; + if !committed_actions + .insert(delete_objects_action_identity(deleted_object, delete_results[didx].requested_version_id)) + { + continue; + } + delete_results[didx].emit_side_effects = true; let creates_delete_marker = object_to_delete[i].version_id.is_none() && versioned && !version_suspended; let committed_delete_marker = dobjs[i].delete_marker; let delete_accounting = accounting.get(i).and_then(Option::as_ref); @@ -804,8 +885,9 @@ impl DefaultObjectUsecase { .filter_map(|result| result.delete_object.as_ref().map(|object| (result, object))) .map(|(result, object)| DeletedObject { delete_marker: { if object.delete_marker { Some(true) } else { None } }, - delete_marker_version_id: delete_response_version_id( + delete_marker_version_id: delete_marker_response_version_id( object.delete_marker_version_id, + object.delete_marker && result.version_suspended && result.requested_version_id.is_none(), result.synthetic_version_id, ), key: Some(object.object_name.clone()), @@ -828,7 +910,7 @@ impl DefaultObjectUsecase { .filter_map(|v| v.error.clone()) .collect::>(); let output = DeleteObjectsOutput { - deleted: Some(deleted), + deleted: (!delete.quiet.unwrap_or_default()).then_some(deleted), errors: Some(errors), ..Default::default() }; @@ -843,6 +925,7 @@ impl DefaultObjectUsecase { let replication_deletes = if replicate_deletes { delete_results .iter() + .filter(|result| result.emit_side_effects) .filter_map(|result| result.delete_object.as_ref()) .filter(|dobj| deleted_object_has_pending_replication_delete(dobj)) .cloned() @@ -870,6 +953,7 @@ impl DefaultObjectUsecase { let deleted_any = delete_results.iter().any(|result| result.delete_object.is_some()); let delete_producers = delete_results .iter() + .filter(|result| result.emit_side_effects) .filter_map(|result| { result.delete_object.as_ref().map(|deleted_object| { if deleted_object.delete_marker && result.requested_version_id.is_none() { @@ -884,6 +968,9 @@ impl DefaultObjectUsecase { spawn_background_with_context(Some(request_context), async move { let _activity_guard = DeleteTailActivityGuard::new(DeleteTailStage::Notify); for res in delete_results { + if !res.emit_side_effects { + continue; + } if let Some(dobj) = res.delete_object { // `DeleteMarkerCreated` is a creation event; a delete // marker removed by an explicit version id is a plain @@ -924,8 +1011,10 @@ impl DefaultObjectUsecase { rustfs_scanner::record_dirty_usage_bucket_from_producers(&bucket, delete_producers); } // Record write operation for capacity management (inline to avoid per-request tokio::spawn overhead) - let manager = get_capacity_manager(); - manager.record_write_operation().await; + if deleted_any { + let manager = get_capacity_manager(); + manager.record_write_operation().await; + } result } @@ -1951,6 +2040,26 @@ mod tests { assert_eq!(delete_response_version_id(None, false), None); } + #[test] + fn delete_objects_created_null_marker_response_uses_the_committed_marker_identity() { + let uuid = Uuid::new_v4(); + assert_eq!(delete_marker_response_version_id(None, true, false), Some("null".to_string())); + assert_eq!(delete_marker_response_version_id(None, false, false), None); + assert_eq!(delete_marker_response_version_id(None, true, true), None); + assert_eq!(delete_marker_response_version_id(Some(uuid), false, false), Some(uuid.to_string())); + assert_eq!( + delete_marker_response_version_id(Some(uuid), true, false), + Some(uuid.to_string()), + "the committed identity takes precedence over any null fallback" + ); + assert_eq!( + delete_marker_response_version_id(Some(Uuid::nil()), false, false), + Some("null".to_string()), + "explicit null-marker removal keeps its existing response identity" + ); + assert_eq!(delete_marker_response_version_id(Some(Uuid::nil()), true, true), None); + } + #[test] fn delete_entry_response_version_id_echoes_requested_identity_for_marker_removal() { let version_id = Uuid::new_v4(); @@ -2290,6 +2399,84 @@ mod tests { assert_eq!(internal_version_id, None); } + #[test] + fn delete_objects_condition_parser_preserves_presence_and_rejects_version_combinations() { + let object = ObjectIdentifier { + key: "key".to_string(), + ..Default::default() + }; + assert_eq!(parse_delete_objects_condition(&object).expect("missing ETag is unconditional"), None); + for etag in ["", "\"\"", " \t "] { + let error = parse_delete_objects_condition(&ObjectIdentifier { + e_tag: Some(ETag::Strong(etag.to_string())), + ..object.clone() + }) + .expect_err("an explicit empty ETag must never become an unconditional delete"); + assert_eq!(error.code.as_deref(), Some("InvalidArgument")); + assert_eq!(error.key.as_deref(), Some("key")); + } + for version in ["null", "", "bad-version"] { + let error = parse_delete_objects_condition(&ObjectIdentifier { + e_tag: Some(ETag::Strong("*".to_string())), + version_id: Some(version.to_string()), + ..object.clone() + }) + .expect_err("any explicit VersionId is unsupported with a condition"); + assert_eq!(error.code.as_deref(), Some("InvalidRequest")); + assert_eq!(error.version_id.as_deref(), Some(version)); + } + } + + #[test] + fn delete_objects_action_identity_deduplicates_actions_without_losing_historical_targets() { + let current = StorageDeletedObject { + object_name: "key".to_string(), + ..Default::default() + }; + assert_eq!( + delete_objects_action_identity(¤t, None), + delete_objects_action_identity(¤t, Some(Uuid::nil())) + ); + let historical = Uuid::new_v4(); + assert_ne!( + delete_objects_action_identity(¤t, None), + delete_objects_action_identity(¤t, Some(historical)) + ); + let marker = StorageDeletedObject { + delete_marker: true, + delete_marker_version_id: Some(historical), + ..current + }; + assert_ne!( + delete_objects_action_identity(&marker, None), + delete_objects_action_identity(&marker, Some(historical)) + ); + } + + #[test] + fn delete_objects_condition_parser_rejects_weak_validators_without_dropping_the_condition() { + for value in ["current-etag", "*", ""] { + let object = ObjectIdentifier { + key: "key".to_string(), + e_tag: Some(ETag::Weak(value.to_string())), + ..Default::default() + }; + let error = parse_delete_objects_condition(&object) + .expect_err("a weak validator must fail instead of falling back to an unconditional delete"); + assert_eq!(error.code.as_deref(), Some("InvalidArgument")); + assert_eq!(error.key.as_deref(), Some("key")); + } + let object = ObjectIdentifier { + key: "key".to_string(), + e_tag: Some(ETag::Strong("current-etag".to_string())), + ..Default::default() + }; + assert_eq!( + parse_delete_objects_condition(&object).expect("strong validator retains its condition"), + Some(DeleteObjectCondition::MatchEtag("current-etag".to_string())) + ); + } + #[test] fn delete_objects_treats_raw_io_not_found_as_idempotent() { assert!(is_delete_objects_not_found(&StorageError::FileNotFound)); @@ -2319,6 +2506,230 @@ mod tests { assert_eq!(deleted.object_name, "missing-key"); } + #[test] + fn delete_objects_result_reducer_preserves_conditional_not_found_errors() { + let object = ObjectToDelete { + object_name: "missing-key".to_string(), + delete_condition: Some(DeleteObjectCondition::Exists), + ..Default::default() + }; + let deleted = StorageDeletedObject { + object_name: object.object_name.clone(), + ..Default::default() + }; + let error = reduce_delete_objects_result(&object, &deleted, Some(&StorageError::FileNotFound), false) + .expect_err("a failed existence condition is not an idempotent successful delete"); + assert_eq!(error.key.as_deref(), Some("missing-key")); + assert_eq!(error.code.as_deref(), Some("NoSuchKey")); + } + + #[test] + #[serial_test::serial] + fn execute_delete_objects_conditions_authorize_per_entry_and_keep_quiet_errors() { + crate::app::gating_test_env::run_large_stack_test("conditional-bulk-delete-api", || async { + use crate::app::storage_api::test::contract::bucket::{ + BucketOperations as _, DeleteBucketOptions, MakeBucketOptions, + }; + + let store = crate::app::gating_test_env::shared_gating_ecstore().await; + if current_app_context().is_none() { + crate::app::runtime_sources::install_test_app_context(Arc::clone(&store)).await; + } + let bucket = format!("conditional-bulk-api-{}", Uuid::new_v4().simple()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create conditional bulk-delete bucket"); + let mut readable_etag = String::new(); + let mut denied_etag = String::new(); + let mut total_size = 0; + for key in [ + "readable", + "denied", + "wildcard", + "unconditional", + "invalid", + "version-combination", + ] { + let object = store + .put_object( + &bucket, + key, + &mut PutObjReader::from_vec(format!("payload-{key}").into_bytes()), + &ObjectOptions::default(), + ) + .await + .expect("seed object before request authorization"); + if key == "readable" { + readable_etag = object.etag.expect("readable fixture has an ETag"); + } else if key == "denied" { + denied_etag = object.etag.expect("denied fixture has an ETag"); + } + total_size += u64::try_from(object.size).expect("fixture size is nonnegative"); + } + crate::app::storage_api::test::data_usage::seed_bucket_usage_memory_for_test(&bucket, total_size).await; + let policy_json = format!( + r#"{{"Version":"2012-10-17","Statement":[{{"Effect":"Allow","Principal":{{"AWS":"*"}},"Action":["s3:DeleteObject"],"Resource":["arn:aws:s3:::{bucket}/*"]}},{{"Effect":"Allow","Principal":{{"AWS":"*"}},"Action":["s3:GetObject"],"Resource":["arn:aws:s3:::{bucket}/readable"]}},{{"Effect":"Deny","Principal":{{"AWS":"*"}},"Action":["s3:GetObject"],"Resource":["arn:aws:s3:::{bucket}/denied","arn:aws:s3:::{bucket}/wildcard"]}}]}}"# + ); + let mut metadata = (*crate::storage::get_bucket_metadata(&bucket) + .await + .expect("test bucket metadata")) + .clone(); + metadata.policy_config = Some(serde_json::from_str(&policy_json).expect("parse conditional delete policy")); + metadata.policy_config_json = policy_json.into_bytes(); + crate::storage::storage_api::set_bucket_metadata(bucket.clone(), metadata) + .await + .expect("publish conditional delete authorization policy"); + let requests = vec![ + ObjectIdentifier { + key: "readable".to_string(), + e_tag: Some(ETag::Strong(readable_etag.clone())), + ..Default::default() + }, + ObjectIdentifier { + key: "readable".to_string(), + e_tag: Some(ETag::Strong("stale-etag".to_string())), + ..Default::default() + }, + ObjectIdentifier { + key: "readable".to_string(), + e_tag: Some(ETag::Strong(readable_etag)), + ..Default::default() + }, + ObjectIdentifier { + key: "denied".to_string(), + e_tag: Some(ETag::Strong(denied_etag)), + ..Default::default() + }, + ObjectIdentifier { + key: "wildcard".to_string(), + e_tag: Some(ETag::Strong("*".to_string())), + ..Default::default() + }, + ObjectIdentifier { + key: "missing".to_string(), + e_tag: Some(ETag::Strong("*".to_string())), + ..Default::default() + }, + ObjectIdentifier { + key: "invalid".to_string(), + e_tag: Some(ETag::Strong("".to_string())), + ..Default::default() + }, + ObjectIdentifier { + key: "version-combination".to_string(), + e_tag: Some(ETag::Strong("*".to_string())), + version_id: Some("null".to_string()), + ..Default::default() + }, + ObjectIdentifier { + key: "unconditional".to_string(), + ..Default::default() + }, + ]; + let input = DeleteObjectsInput::builder() + .bucket(bucket.clone()) + .delete(Delete { + objects: requests, + quiet: Some(false), + }) + .build() + .expect("build mixed conditional batch"); + let mut req = build_request(input, Method::POST); + req.extensions.insert(crate::storage::access::ReqInfo::default()); + let usecase = DefaultObjectUsecase::from_global(); + let response = usecase + .execute_delete_objects(req) + .await + .expect("mixed batch returns per-entry outcomes"); + let deleted = response + .output + .deleted + .expect("non-quiet batch reports every successful input entry"); + assert_eq!( + deleted.iter().map(|object| object.key.as_deref()).collect::>(), + vec![Some("readable"), Some("readable"), Some("wildcard"), Some("unconditional")], + "duplicate actions preserve both original input results" + ); + let errors = response.output.errors.expect("failed entries remain visible in Quiet mode"); + let actual = errors + .iter() + .map(|error| (error.key.as_deref(), error.code.as_deref())) + .collect::>(); + assert_eq!( + actual, + vec![ + (Some("readable"), Some("PreconditionFailed")), + (Some("denied"), Some("AccessDenied")), + (Some("missing"), Some("NoSuchKey")), + (Some("invalid"), Some("InvalidArgument")), + (Some("version-combination"), Some("InvalidRequest")), + ] + ); + for key in ["denied", "invalid", "version-combination"] { + store + .get_object_info(&bucket, key, &ObjectOptions::default()) + .await + .expect("rejected entry preserves data"); + } + for key in ["readable", "wildcard", "unconditional"] { + let error = store + .get_object_info(&bucket, key, &ObjectOptions::default()) + .await + .expect_err("successful entry deletes data"); + assert!(is_err_object_not_found(&error)); + } + let mut usage = rustfs_data_usage::DataUsageInfo::default(); + crate::app::storage_api::test::data_usage::apply_bucket_usage_memory_overlay(&mut usage).await; + let retained_size = ["denied", "invalid", "version-combination"] + .into_iter() + .map(|key| u64::try_from(format!("payload-{key}").len()).expect("fixture sizes fit u64")) + .sum::(); + assert_eq!( + usage.buckets_usage.get(&bucket).map(|value| value.size), + Some(retained_size), + "duplicate success and failed conditions must not subtract additional object bytes" + ); + + let quiet_input = DeleteObjectsInput::builder() + .bucket(bucket.clone()) + .delete(Delete { + objects: vec![ + ObjectIdentifier { + key: "unconditional".to_string(), + ..Default::default() + }, + ObjectIdentifier { + key: "missing".to_string(), + e_tag: Some(ETag::Strong("*".to_string())), + ..Default::default() + }, + ], + quiet: Some(true), + }) + .build() + .expect("build quiet mixed batch"); + let mut quiet_req = build_request(quiet_input, Method::POST); + quiet_req.extensions.insert(crate::storage::access::ReqInfo::default()); + let quiet_response = usecase.execute_delete_objects(quiet_req).await.expect("quiet batch succeeds"); + assert!(quiet_response.output.deleted.is_none(), "Quiet hides only successful entries"); + let quiet_errors = quiet_response.output.errors.expect("Quiet preserves errors"); + assert_eq!(quiet_errors.len(), 1); + assert_eq!(quiet_errors[0].key.as_deref(), Some("missing")); + assert_eq!(quiet_errors[0].code.as_deref(), Some("NoSuchKey")); + store + .delete_bucket( + &bucket, + &DeleteBucketOptions { + force: true, + ..Default::default() + }, + ) + .await + .expect("remove fixture bucket"); + }); + } + #[test] #[serial_test::serial] fn recursive_delete_holds_writer_exclusion_after_authorization() { diff --git a/rustfs/src/app/object/get.rs b/rustfs/src/app/object/get.rs index bdcb79d26..eab0a72bf 100644 --- a/rustfs/src/app/object/get.rs +++ b/rustfs/src/app/object/get.rs @@ -15,7 +15,30 @@ //! GetObject / GetObjectAttributes read path: cold fill, resume, stream tuning. use super::*; +use crate::app::storage_api::object_usecase::object_cache::{ + GetObjectReadCondition, GetObjectReadMetadata, evaluate_read_preconditions, has_read_preconditions, + with_get_object_read_condition, +}; use crate::on_demand_migration::WriteBackBody; + +struct ConditionalRead { + bucket: String, + object: String, + headers: HeaderMap, + part_number: Option, +} + +impl GetObjectReadCondition for ConditionalRead { + fn is_terminal(&self, metadata: GetObjectReadMetadata<'_>) -> bool { + if metadata.bucket != self.bucket || metadata.object != self.object || metadata.delete_marker { + return false; + } + evaluate_read_preconditions(&self.headers, metadata.etag, metadata.mod_time).is_err() + || self + .part_number + .is_some_and(|number| number > 1 && !metadata.parts.iter().any(|part| part.number == number)) + } +} use crate::on_demand_migration::{ BucketOdmState, OdmLookup, OdmOp, OdmOutcome, OnDemandMigrationSys, PullError, PullLeader, PullOutcome, PullReason, PullSlot, RangeGetPolicy, SourceBody, SourceClient, SourceError, SourceGet, SourceHead, commit_inline, idle_guarded_body, @@ -49,6 +72,39 @@ type ColdFillReaderOpenProbeState = Option<(rustfs_object_data_cache::ObjectData #[cfg(test)] static COLD_FILL_READER_OPEN_PROBE: OnceLock> = OnceLock::new(); +#[cfg(test)] +#[derive(Default)] +struct BodyCacheFillCalls { + reservations: AtomicU64, + fills: AtomicU64, +} + +#[cfg(test)] +type BodyCacheFillProbeState = Option<(rustfs_object_data_cache::ObjectDataCacheKey, Arc)>; + +#[cfg(test)] +static BODY_CACHE_FILL_PROBE: OnceLock> = OnceLock::new(); + +#[cfg(test)] +fn record_body_cache_fill_call_for_test(plan: &rustfs_object_data_cache::ObjectDataCacheGetPlan, reservation: bool) { + let Some(key) = plan.key() else { return }; + let counters = BODY_CACHE_FILL_PROBE + .get_or_init(|| Mutex::new(None)) + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .as_ref() + .filter(|(probe_key, _)| probe_key == key) + .map(|(_, counts)| Arc::clone(counts)); + if let Some(counters) = counters { + let counter = if reservation { + &counters.reservations + } else { + &counters.fills + }; + counter.fetch_add(1, Ordering::Relaxed); + } +} + fn adjust_cold_fill_disk_permit_metric(owner: ColdFillDiskPermitOwner, acquired: bool) { macro_rules! adjust_gauge { ($name:literal) => {{ @@ -221,6 +277,16 @@ const GET_OBJECT_STAGE_LIFECYCLE_EXPIRATION: &str = "lifecycle_expiration"; const GET_OBJECT_STAGE_METADATA_FILTER: &str = "metadata_filter"; +const GET_OBJECT_STAGE_BODY_CACHE_HOOK_HIT: &str = "body_cache_hook_hit"; +const GET_OBJECT_STAGE_BODY_CACHE_LOOKUP: &str = "body_cache_lookup"; +const GET_OBJECT_STAGE_BODY_CACHE_FILL_HANDOFF: &str = "body_cache_fill_handoff"; +const GET_OBJECT_STAGE_BODY_BUFFERED_HANDOFF: &str = "body_buffered_handoff"; +const GET_OBJECT_STAGE_OUTPUT_STRUCT_BUILD: &str = "output_struct_build"; +const GET_OBJECT_STAGE_PREPARE_READ_EXECUTION: &str = "prepare_read_execution"; +const GET_OBJECT_STAGE_RESPONSE_CORS_WRAP: &str = "response_cors_wrap"; +const GET_OBJECT_STAGE_RESPONSE_HEADER_INJECT: &str = "response_header_inject"; +const GET_OBJECT_STAGE_RESPONSE_EVENT_COMPLETE: &str = "response_event_complete"; + const GET_OBJECT_STREAM_WARN_THRESHOLD: Duration = Duration::from_secs(5); static GET_OBJECT_BUFFER_THRESHOLD_WARNED: AtomicBool = AtomicBool::new(false); @@ -499,10 +565,10 @@ fn get_object_stream_failure_reason(error_class: &'static str) -> &'static str { } } -/// A mid-stream reopen can recover only when another representation of the -/// committed object may exist. A single-disk, no-parity object has no alternate -/// shard after a bitrot/read-quorum failure, so retrying the same read only -/// repeats the failure and adds latency. +/// A mid-stream reopen can only recover when another representation of the +/// committed object may exist. Single-disk, no-parity objects have no alternate +/// shard to discover after a bitrot/read-quorum failure, so arming the normal +/// three-attempt resume loop only repeats the same failing read and adds delay. fn should_attach_get_object_resume(info: &ObjectInfo) -> bool { info.parity_blocks > 0 || info.data_blocks > 1 || !info.transitioned_object.tier.is_empty() } @@ -1075,6 +1141,8 @@ async fn start_cold_fill_producer( + req: &S3Request, + part_number: Option, + read: F, + ) -> impl std::future::Future { + // The concrete reader future is large; keep it out of the request's stack frame. + let read = Box::pin(read); + let condition = if has_read_preconditions(&req.headers) { + let mut headers = HeaderMap::with_capacity(4); + for name in [ + http::header::IF_MATCH, + http::header::IF_UNMODIFIED_SINCE, + http::header::IF_NONE_MATCH, + http::header::IF_MODIFIED_SINCE, + ] { + if let Some(value) = req.headers.get(&name) { + headers.insert(name, value.clone()); + } + } + Some(Arc::new(ConditionalRead { + bucket: req.input.bucket.clone(), + object: rustfs_utils::path::encode_dir_object_ref(&req.input.key).into_owned(), + headers, + part_number, + })) + } else { + None + }; + async move { + if let Some(condition) = condition { + with_get_object_read_condition(condition, read).await + } else { + read.await + } + } + } fn build_memory_bytes_blob( bytes: Bytes, response_content_length: i64, @@ -2279,9 +2383,29 @@ impl DefaultObjectUsecase { Ok(()) } - fn validate_get_object_before_cold_fill(headers: &HeaderMap, part_number: Option, info: &ObjectInfo) -> S3Result<()> { - check_preconditions(headers, info)?; - Self::validate_get_object_part_number(part_number, info) + async fn validate_get_object_before_body( + req: &S3Request, + bucket: &str, + key: &str, + part_number: Option, + info: &ObjectInfo, + ) -> S3Result<()> { + if info.delete_marker { + let code = if req.input.version_id.is_some() { + S3ErrorCode::MethodNotAllowed + } else { + S3ErrorCode::NoSuchKey + }; + return Err(with_delete_marker_read_headers(S3Error::new(code), info, req.input.version_id.as_deref())); + } + let result = + check_preconditions(&req.headers, info).and_then(|()| Self::validate_get_object_part_number(part_number, info)); + if result.is_err() { + let principal = SseKmsPrincipal::from_request(req); + crate::storage::sse::validate_conditional_read(bucket, key, &info.user_defined, &req.headers, principal.as_ref()) + .await?; + } + result } /// The `x-amz-mp-parts-count` value for a response, if one is owed. @@ -2524,6 +2648,7 @@ impl DefaultObjectUsecase { let read_stage_start = rustfs_io_metrics::get_stage_metrics_enabled().then_some(read_start); let store_headers = project_ssec_transport_headers(&req.headers); let cache_adapter = self.object_data_cache(); + let conditional = has_read_preconditions(&req.headers); if cache_adapter.is_disabled() || !cache_adapter.materialize_fill_enabled() { let io_planning = Self::acquire_get_object_io_planning( manager, @@ -2537,7 +2662,11 @@ impl DefaultObjectUsecase { .await?; let reader = track_object_read_setup( object_traffic_health.as_deref(), - store.get_object_reader(bucket, key, rs.clone(), store_headers, opts), + Self::with_get_object_conditions( + req, + part_number, + store.get_object_reader(bucket, key, rs.clone(), store_headers, opts), + ), ) .await .map_err(map_get_object_reader_error)?; @@ -2560,16 +2689,18 @@ impl DefaultObjectUsecase { ) .await?, ); - let mut prepared = Some( - track_object_read_setup( - object_traffic_health.as_deref(), - store.prepare_get_object_reader(bucket, key, rs.clone(), HeaderMap::new(), opts), - ) - .await - .map_err(map_get_object_reader_error)?, - ); + let initial_metadata = track_object_read_setup( + object_traffic_health.as_deref(), + store.prepare_get_object_reader(bucket, key, rs.clone(), HeaderMap::new(), opts), + ) + .await + .map_err(map_get_object_reader_error)?; let mut cache_fill_allowed = true; let mut legacy_hook_missed = false; + if conditional { + Self::validate_get_object_before_body(req, bucket, key, part_number, initial_metadata.object_info()).await?; + } + let mut prepared = Some(initial_metadata); 'snapshot: { let info = prepared .as_ref() @@ -2609,7 +2740,6 @@ impl DefaultObjectUsecase { if matches!(legacy_probe, GetObjectBodyCacheHookLookup::Ineligible) { break 'snapshot; } - Self::validate_get_object_before_cold_fill(&req.headers, part_number, info)?; if let GetObjectBodyCacheHookLookup::Hit(body) = legacy_probe { drop(metadata_admission.take()); let info = prepared @@ -2770,6 +2900,8 @@ impl DefaultObjectUsecase { return; }; + #[cfg(test)] + record_body_cache_fill_call_for_test(&engine_plan, true); let reservation = adapter.reserve_body(&engine_plan); #[cfg(test)] let reader_open_plan = engine_plan.clone(); @@ -2826,10 +2958,12 @@ impl DefaultObjectUsecase { let io_planning = metadata_admission .take() .ok_or_else(|| s3_error!(InternalError, "prepared metadata admission is unavailable"))?; - let reader = - track_object_read_setup(object_traffic_health.as_deref(), prepared.with_headers(store_headers).into_reader()) - .await - .map_err(map_get_object_reader_error)?; + let reader = track_object_read_setup( + object_traffic_health.as_deref(), + Self::with_get_object_conditions(req, part_number, prepared.with_headers(store_headers).into_reader()), + ) + .await + .map_err(map_get_object_reader_error)?; (io_planning, reader) } else { let io_planning = Self::acquire_get_object_io_planning( @@ -2849,13 +2983,20 @@ impl DefaultObjectUsecase { ) .await .map_err(map_get_object_reader_error)?; - track_object_read_setup(object_traffic_health.as_deref(), prepared.with_headers(store_headers).into_reader()) - .await - .map_err(map_get_object_reader_error)? + track_object_read_setup( + object_traffic_health.as_deref(), + Self::with_get_object_conditions(req, part_number, prepared.with_headers(store_headers).into_reader()), + ) + .await + .map_err(map_get_object_reader_error)? } else { track_object_read_setup( object_traffic_health.as_deref(), - store.get_object_reader(bucket, key, rs.clone(), store_headers, opts), + Self::with_get_object_conditions( + req, + part_number, + store.get_object_reader(bucket, key, rs.clone(), store_headers, opts), + ), ) .await .map_err(map_get_object_reader_error)? @@ -2900,6 +3041,7 @@ impl DefaultObjectUsecase { let stream = reader.stream; let buffered_body = reader.buffered_body; + Self::validate_get_object_before_body(req, bucket, key, part_number, &info).await?; let read_duration = read_start.elapsed(); // Conditional metrics recording to reduce overhead @@ -2909,9 +3051,6 @@ impl DefaultObjectUsecase { manager.record_disk_operation(info.size as u64, read_duration, true).await; } - check_preconditions(&req.headers, &info)?; - Self::validate_get_object_part_number(part_number, &info)?; - debug!(object_size = info.size, part_count = info.parts.len(), "GET object metadata snapshot"); for part in info.parts.iter() { debug!( @@ -3341,12 +3480,15 @@ impl DefaultObjectUsecase { // already supplied this body, the request-level plan was built before // the authoritative lookup. Serve it without planning a second time. if cache_hook_served && let Some(bytes) = buffered_body.take() { - return Ok(Self::build_memory_bytes_blob( + let hook_hit_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); + let body = Self::build_memory_bytes_blob( bytes, response_content_length, GET_MEMORY_BODY_SOURCE_OBJECT_DATA_CACHE, lifecycle, - )); + ); + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_BODY_CACHE_HOOK_HIT, hook_hit_start); + return Ok(body); } if !cache_fill_allowed { @@ -3387,20 +3529,27 @@ impl DefaultObjectUsecase { // authoritative because it ran after fresh metadata resolution, so the // app layer skips its own lookup and only uses the plan to fill. if !cache_hook_probed { - match lookup_get_object_body_cache_hit(cache_adapter, &cache_plan).await { + let cache_lookup_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); + let cache_lookup = lookup_get_object_body_cache_hit(cache_adapter, &cache_plan).await; + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_BODY_CACHE_LOOKUP, cache_lookup_start); + match cache_lookup { GetObjectBodyCacheLookup::Hit(bytes) => { - return Ok(Self::build_memory_bytes_blob( + let cache_hit_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); + let body = Self::build_memory_bytes_blob( bytes, response_content_length, GET_MEMORY_BODY_SOURCE_OBJECT_DATA_CACHE, lifecycle, - )); + ); + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_BODY_CACHE_HOOK_HIT, cache_hit_start); + return Ok(body); } GetObjectBodyCacheLookup::Disabled | GetObjectBodyCacheLookup::Skip | GetObjectBodyCacheLookup::Miss => {} } } if let Some(buffered_body) = buffered_body { + let cache_fill_handoff_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); // ODC-15: the body is already fully in hand, so keep the fill off the // response's critical path. For a cacheable plan, run the fill in a // detached task (Bytes is a cheap clone) and return immediately. For @@ -3411,18 +3560,30 @@ impl DefaultObjectUsecase { let cache_plan = cache_plan.clone(); let fill_bytes = buffered_body.clone(); tokio::spawn(async move { + #[cfg(test)] + if let GetObjectBodyCachePlan::Cacheable(plan) = &cache_plan { + record_body_cache_fill_call_for_test(plan, false); + } let _ = fill_get_object_body_cache_from_buffered_body(&cache_adapter, &cache_plan, &fill_bytes).await; }); } else if cache_fill_allowed { + #[cfg(test)] + if let GetObjectBodyCachePlan::Cacheable(plan) = &cache_plan { + record_body_cache_fill_call_for_test(plan, false); + } let _ = fill_get_object_body_cache_from_buffered_body(cache_adapter, &cache_plan, &buffered_body).await; } + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_BODY_CACHE_FILL_HANDOFF, cache_fill_handoff_start); - return Ok(Self::build_memory_bytes_blob( + let buffered_handoff_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); + let body = Self::build_memory_bytes_blob( buffered_body, response_content_length, GET_MEMORY_BODY_SOURCE_BUFFERED_BODY, lifecycle, - )); + ); + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_BODY_BUFFERED_HANDOFF, buffered_handoff_start); + return Ok(body); } let should_materialize_for_cache = cache_adapter.materialize_fill_enabled() @@ -3476,14 +3637,23 @@ impl DefaultObjectUsecase { { Ok(buf) => { let bytes = Bytes::from(buf); + let cache_fill_handoff_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); // ODC-15: fill off the response's critical path (see the // buffered-body branch above). let cache_adapter = cache_adapter.clone(); let cache_plan = cache_plan.clone(); let fill_bytes = bytes.clone(); tokio::spawn(async move { + #[cfg(test)] + if let GetObjectBodyCachePlan::Cacheable(plan) = &cache_plan { + record_body_cache_fill_call_for_test(plan, false); + } let _ = fill_get_object_body_cache_from_materialized_body(&cache_adapter, &cache_plan, &fill_bytes).await; }); + record_get_object_s3_handler_stage_duration( + GET_OBJECT_STAGE_BODY_CACHE_FILL_HANDOFF, + cache_fill_handoff_start, + ); return Ok(Self::build_memory_bytes_blob( bytes, @@ -3622,13 +3792,19 @@ impl DefaultObjectUsecase { None => helper, }; let helper = helper.version_id(version_id_for_event); + let cors_wrap_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); let mut response = wrap_response_with_cors(bucket, method, headers, output).await; + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_RESPONSE_CORS_WRAP, cors_wrap_start); + let header_inject_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); inject_accept_ranges_header(&mut response.headers); // Emit XXHash3/64/128 and SHA-512 checksums that s3s GetObjectOutput cannot // carry (#1257). This is the download-side integrity path AWS SDKs verify. inject_additional_checksum_headers(&mut response.headers, &extra_checksum_headers); + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_RESPONSE_HEADER_INJECT, header_inject_start); let result = Ok(response); + let event_complete_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); let _ = helper.complete(&result); + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_RESPONSE_EVENT_COMPLETE, event_complete_start); result } @@ -3738,9 +3914,8 @@ impl DefaultObjectUsecase { let metadata_filter_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); let metadata = filter_object_metadata(&info.user_defined); record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_METADATA_FILTER, metadata_filter_start); - + let output_struct_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); let parts_count = Self::get_object_parts_count(part_number, &info); - let output = GetObjectOutput { body: Some(body), content_length: Some(response_content_length), @@ -3774,6 +3949,7 @@ impl DefaultObjectUsecase { storage_class, ..Default::default() }; + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_OUTPUT_STRUCT_BUILD, output_struct_start); Ok(GetObjectOutputContext { output, @@ -4076,6 +4252,7 @@ impl DefaultObjectUsecase { let manager = get_concurrency_manager(); + let prepare_read_start = stage_metrics_enabled.then(std::time::Instant::now); let mut prepared_read = self .prepare_get_object_read_execution( &req, @@ -4091,6 +4268,7 @@ impl DefaultObjectUsecase { object_traffic_health.clone(), ) .await; + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_PREPARE_READ_EXECUTION, prepare_read_start); // An object missing locally (and only missing — other errors keep // their semantics) may still be served by a remote copy. if let Err(err) = &prepared_read @@ -4134,6 +4312,7 @@ impl DefaultObjectUsecase { return Self::complete_get_object_error(helper.version_id(version_id_for_event), err); } Some(OdmGetOutcome::RetryLocal) => { + let prepare_read_retry_start = stage_metrics_enabled.then(std::time::Instant::now); prepared_read = self .prepare_get_object_read_execution( &req, @@ -4149,6 +4328,10 @@ impl DefaultObjectUsecase { object_traffic_health, ) .await; + record_get_object_s3_handler_stage_duration( + GET_OBJECT_STAGE_PREPARE_READ_EXECUTION, + prepare_read_retry_start, + ); } } } @@ -5365,7 +5548,11 @@ mod on_demand_migration_tests { let data = payload(512); let etag = source_head(&data).etag.expect("etag"); - let source = ScriptedSource::for_object(&data); + let head = source_head(&data); + let source = ScriptedSource::new( + vec![Ok(head.clone()), Ok(head.clone()), Ok(head.clone())], + vec![Ok((head, data.clone(), None))], + ); let mut headers = HeaderMap::new(); headers.insert(http::header::IF_NONE_MATCH, HeaderValue::from_str(&format!("\"{etag}\"")).unwrap()); let err = failed(odm_get_from_source(&state, &source, &headers, KEY, None, None).await); @@ -5373,7 +5560,6 @@ mod on_demand_migration_tests { assert_eq!(source.head_calls(), 1); assert_eq!(source.get_calls(), 0, "a 304 never pulls"); - let source = ScriptedSource::for_object(&data); let mut headers = HeaderMap::new(); headers.insert(http::header::IF_MATCH, HeaderValue::from_static("\"another-etag\"")); let err = failed(odm_get_from_source(&state, &source, &headers, KEY, None, None).await); @@ -5382,6 +5568,15 @@ mod on_demand_migration_tests { assert_eq!(get_count(&state, OdmOutcome::SourceHit), 2); assert_eq!(state.inflight_keys(), 0); assert!(rt.write_back.puts().is_empty()); + + let (mut output, _) = served(odm_get_from_source(&state, &source, &HeaderMap::new(), KEY, None, None).await); + assert_eq!( + collect_body(&mut output).await, + data, + "positive control still fetches the source representation" + ); + assert_eq!(source.get_calls(), 1, "the same SourceClient counter observes the positive-control GET"); + assert_eq!(source.head_calls(), 3, "each conditional and ordinary request reads source metadata"); } #[tokio::test] @@ -5671,6 +5866,28 @@ mod tests { use std::task::{Context, Poll}; use tokio::io::{AsyncRead, ReadBuf}; + #[tokio::test] + async fn conditional_reader_scope_keeps_large_futures_off_the_request_stack() { + let mut req = build_request(GetObjectInput::default(), Method::GET); + for conditional in [false, true] { + if conditional { + req.headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("\"etag\"")); + } + let payload = [0u8; 64 * 1024]; + let read = async move { + std::hint::black_box(payload); + std::future::ready(()).await; + }; + let read = DefaultObjectUsecase::with_get_object_conditions(&req, None, read); + assert!( + std::mem::size_of_val(&read) < 2048, + "reader scope must not embed the concrete reader future" + ); + read.await; + } + } + #[tokio::test(start_paused = true)] async fn cold_fill_disk_admission_preserves_slow_down() { let manager = Box::leak(Box::new(ConcurrencyManager::with_disk_read_caps_for_test(1, 1))); @@ -5690,6 +5907,194 @@ mod tests { drop(primary); } + #[tokio::test] + #[serial_test::serial(body_cache_hook)] + async fn conditional_get_304_uses_metadata_without_decoding_damaged_data() { + use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions}; + let (store, ambient) = real_cold_fill_test_context().await; + let context = temp_env::with_vars([(rustfs_config::ENV_OBJECT_DATA_CACHE_ENABLE, Some("false"))], || { + Arc::new(AppContext::new(Arc::clone(&store), ambient.iam(), ambient.kms())) + }); + assert!(context.object_data_cache().is_disabled()); + let usecase = DefaultObjectUsecase::with_context(Some(context)); + let (disk_paths, _) = crate::app::gating_test_env::shared_gating_ecstore_and_disk_paths().await; + let bucket = format!("conditional-metadata-{}", Uuid::new_v4()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create test bucket"); + for (object, length) in [("inline.bin", 4096), ("shards.bin", 1_300_000)] { + let payload = vec![b'c'; length]; + let info = put_real_cold_fill_object(&store, &bucket, object, &payload).await; + let etag = info.etag.as_deref().expect("committed ETag"); + for disk in &disk_paths { + let object_path = disk.join(&bucket).join(object); + if length == 4096 { + let meta_path = object_path.join("xl.meta"); + let mut metadata = tokio::fs::read(&meta_path).await.expect("read inline metadata"); + *metadata.last_mut().expect("inline data bytes") ^= 1; + tokio::fs::write(&meta_path, metadata).await.expect("damage inline payload"); + } else { + // Discover version data dirs by listing the object directory + // (matching delete_object_part_shards) instead of constructing + // the path from info.data_dir, whose UUID string form may not + // match the on-disk directory name. + let mut entries = tokio::fs::read_dir(&object_path).await.expect("read object dir"); + while let Some(entry) = entries.next_entry().await.expect("read object dir entry") { + if entry.file_type().await.expect("entry file type").is_dir() { + tokio::fs::remove_dir_all(entry.path()) + .await + .expect("remove object data shards"); + } + } + } + } + let resolved = store + .get_object_info(&bucket, object, &ObjectOptions::default()) + .await + .expect("metadata remains readable"); + assert_eq!(resolved.etag.as_deref(), Some(etag)); + let mut req = build_request( + GetObjectInput::builder() + .bucket(bucket.clone()) + .key(object.to_owned()) + .build() + .expect("GET input"), + Method::GET, + ); + req.headers.insert( + http::header::IF_NONE_MATCH, + HeaderValue::from_str(&format!("W/\"{etag}\"")).expect("ETag header"), + ); + let error = usecase + .execute_get_object(req) + .await + .expect_err("matching condition returns 304"); + assert_eq!(error.code(), &S3ErrorCode::NotModified); + assert_eq!(error.headers().expect("304 validators")["etag"], format!("\"{etag}\"")); + + let mut req = build_request( + HeadObjectInput::builder() + .bucket(bucket.clone()) + .key(object.to_owned()) + .build() + .expect("HEAD input"), + Method::HEAD, + ); + req.headers.insert( + http::header::IF_NONE_MATCH, + HeaderValue::from_str(&format!("\"{etag}\"")).expect("ETag header"), + ); + let error = usecase + .execute_head_object(req) + .await + .expect_err("conditional HEAD returns 304"); + assert_eq!(error.code(), &S3ErrorCode::NotModified); + assert_eq!(error.headers().expect("HEAD validators")["etag"], format!("\"{etag}\"")); + + let req = build_request( + GetObjectInput::builder() + .bucket(bucket.clone()) + .key(object.to_owned()) + .build() + .expect("GET input"), + Method::GET, + ); + if let Ok(mut response) = usecase.execute_get_object(req).await { + let mut body = response.output.body.take().expect("body for damaged GET"); + let mut failed = false; + while let Some(chunk) = body.next().await { + if chunk.is_err() { + failed = true; + break; + } + } + assert!(failed, "ordinary GET must still reject damaged data"); + } + } + } + + #[tokio::test] + async fn conditional_read_checks_encryption_before_not_modified() { + let mut info = ObjectInfo { + etag: Some("abc".to_owned()), + ..Default::default() + }; + info.user_defined = std::collections::HashMap::from([( + "x-amz-server-side-encryption-customer-algorithm".to_owned(), + "AES256".to_owned(), + )]) + .into(); + let mut req = build_request( + GetObjectInput::builder() + .bucket("bucket".to_owned()) + .key("key".to_owned()) + .build() + .expect("GET input"), + Method::GET, + ); + req.headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("\"abc\"")); + let error = DefaultObjectUsecase::validate_get_object_before_body(&req, "bucket", "key", None, &info) + .await + .expect_err("SSE-C key is mandatory before 304"); + assert_eq!(error.code(), &S3ErrorCode::InvalidRequest); + } + + #[tokio::test] + #[serial_test::serial(body_cache_hook)] + async fn conditional_get_miss_preserves_full_and_range_bodies() { + use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions}; + let (store, ambient) = real_cold_fill_test_context().await; + let context = temp_env::with_vars([(rustfs_config::ENV_OBJECT_DATA_CACHE_ENABLE, Some("false"))], || { + Arc::new(AppContext::new(Arc::clone(&store), ambient.iam(), ambient.kms())) + }); + let usecase = DefaultObjectUsecase::with_context(Some(context)); + let bucket = format!("conditional-miss-{}", Uuid::new_v4()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let payload: Vec = (0..1_300_000) + .map(|index| u8::try_from(index % 251).expect("pattern byte")) + .collect(); + put_real_cold_fill_object(&store, &bucket, "object", &payload).await; + for ranged in [false, true] { + let input = GetObjectInput::builder() + .bucket(bucket.clone()) + .key("object".to_owned()) + .range(ranged.then_some(Range::Int { + first: 10, + last: Some(29), + })) + .build() + .expect("GET input"); + let mut req = build_request(input, Method::GET); + req.headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("\"other\"")); + if ranged { + req.headers + .insert(http::header::RANGE, HeaderValue::from_static("bytes=10-29")); + } + let mut response = usecase + .execute_get_object(req) + .await + .expect("unmatched condition serves body"); + let expected = if ranged { &payload[10..30] } else { payload.as_slice() }; + assert_eq!( + response.output.content_length, + Some(i64::try_from(expected.len()).expect("test body length")) + ); + assert_eq!(response.output.content_range.as_deref(), ranged.then_some("bytes 10-29/1300000")); + let mut body = response.output.body.take().expect("GET body"); + let mut bytes = Vec::new(); + while let Some(chunk) = body.next().await { + bytes.extend_from_slice(&chunk.expect("body chunk")); + } + assert_eq!(bytes, expected); + } + } + #[tokio::test] async fn cold_fill_closed_disk_admission_is_not_slow_down() { let manager = Box::leak(Box::new(ConcurrencyManager::with_disk_read_caps_for_test(1, 1))); @@ -6665,6 +7070,100 @@ mod tests { assert_eq!(coordinator.active_session_count_for_test(), 0); } + #[tokio::test] + #[serial_test::serial(body_cache_hook)] + async fn conditional_get_terminal_response_skips_cache_fill_calls_with_live_positive_control() { + use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions}; + struct ResetProbes; + impl Drop for ResetProbes { + fn drop(&mut self) { + *BODY_CACHE_FILL_PROBE + .get_or_init(|| Mutex::new(None)) + .lock() + .expect("cache-fill probe") = None; + *COLD_FILL_READER_OPEN_PROBE + .get_or_init(|| Mutex::new(None)) + .lock() + .expect("reader-open probe") = None; + } + } + let (store, context) = real_cold_fill_test_context().await; + let bucket = format!("conditional-fill-calls-{}", Uuid::new_v4()); + let object = "object.bin"; + let payload: Vec<_> = (0..1_300_000) + .map(|index| u8::try_from(index % 251).expect("pattern byte")) + .collect(); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let info = put_real_cold_fill_object(&store, &bucket, object, &payload).await; + let adapter = context.object_data_cache(); + let plan = real_cold_fill_plan(&adapter, &bucket, object, &info); + let key = plan.key().expect("cacheable key").clone(); + let fills = Arc::new(BodyCacheFillCalls::default()); + let reader_opens = Arc::new(AtomicU64::new(0)); + let _reset = ResetProbes; + *BODY_CACHE_FILL_PROBE + .get_or_init(|| Mutex::new(None)) + .lock() + .expect("cache-fill probe") = Some((key.clone(), Arc::clone(&fills))); + *COLD_FILL_READER_OPEN_PROBE + .get_or_init(|| Mutex::new(None)) + .lock() + .expect("reader-open probe") = Some((key, Arc::clone(&reader_opens))); + let usecase = DefaultObjectUsecase::with_context(Some(context)); + let input = GetObjectInput::builder() + .bucket(bucket) + .key(object.to_owned()) + .build() + .expect("GET input"); + for (header, value, expected) in [ + ( + http::header::IF_NONE_MATCH, + format!("\"{}\"", info.etag.as_deref().expect("ETag")), + S3ErrorCode::NotModified, + ), + (http::header::IF_MATCH, "\"other-etag\"".to_owned(), S3ErrorCode::PreconditionFailed), + ] { + let mut req = build_request(input.clone(), Method::GET); + req.headers.insert(header, HeaderValue::from_str(&value).expect("condition")); + let error = usecase.execute_get_object(req).await.expect_err("terminal condition"); + assert_eq!(error.code(), &expected); + assert_eq!(reader_opens.load(Ordering::Relaxed), 0, "no cold-fill body factory call"); + assert_eq!(fills.reservations.load(Ordering::Relaxed), 0, "no cache-fill reservation call"); + assert_eq!(fills.fills.load(Ordering::Relaxed), 0, "no cache-fill publish call"); + assert_eq!(adapter.cold_fill_coordinator().active_session_count_for_test(), 0); + } + + let mut response = usecase + .execute_get_object(build_request(input, Method::GET)) + .await + .expect("positive-control GET"); + let mut body = response.output.body.take().expect("body"); + let mut actual = Vec::new(); + while let Some(chunk) = body.next().await { + actual.extend_from_slice(&chunk.expect("positive-control chunk")); + } + assert_eq!(actual, payload); + assert!( + reader_opens.load(Ordering::Relaxed) > 0, + "the same factory probe observes the positive control" + ); + assert!( + fills.reservations.load(Ordering::Relaxed) > 0, + "the same reservation probe observes the positive control" + ); + assert!( + fills.fills.load(Ordering::Relaxed) > 0, + "the same fill probe observes the positive control" + ); + assert!( + matches!(adapter.lookup_body(&plan).await, rustfs_object_data_cache::ObjectDataCacheLookup::Hit(_)), + "the positive control really populated the cache" + ); + } + #[tokio::test] #[serial_test::serial(body_cache_hook)] async fn execute_get_object_maps_cold_fill_session_rejection_to_slow_down_without_opening_reader() { @@ -6732,6 +7231,16 @@ mod tests { #[tokio::test] #[serial_test::serial(body_cache_hook)] async fn execute_get_object_generation_change_bypasses_old_cold_fill_plan() { + run_conditional_generation_change(false).await; + } + + #[tokio::test] + #[serial_test::serial(body_cache_hook)] + async fn conditional_get_rechecks_generation_before_fallback_body() { + run_conditional_generation_change(true).await; + } + + async fn run_conditional_generation_change(conditional: bool) { use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions}; let (store, context) = real_cold_fill_test_context().await; @@ -6743,6 +7252,16 @@ mod tests { .expect("real cold-fill generation bucket must be created"); let initial_body = vec![b'a'; 1_300_000]; let changed_body = vec![b'b'; initial_body.len()]; + let condition_etag = if conditional { + Some( + put_real_cold_fill_object(&store, &bucket, object, &changed_body) + .await + .etag + .expect("replacement ETag"), + ) + } else { + None + }; let initial_info = put_real_cold_fill_object(&store, &bucket, object, &initial_body).await; let adapter = context.object_data_cache(); let initial_plan = real_cold_fill_plan(&adapter, &bucket, object, &initial_info); @@ -6769,7 +7288,14 @@ mod tests { ..GetObjectTimeoutPolicy::default() }, ); - let request = tokio::spawn(async move { usecase.execute_get_object(build_request(input, Method::GET)).await }); + let mut req = build_request(input, Method::GET); + if let Some(etag) = &condition_etag { + req.headers.insert( + http::header::IF_NONE_MATCH, + HeaderValue::from_str(&format!("\"{etag}\"")).expect("replacement condition"), + ); + } + let request = tokio::spawn(async move { usecase.execute_get_object(req).await }); tokio::time::timeout(Duration::from_secs(2), async { while coordinator.global_waiter_count_for_test() != 1 { tokio::task::yield_now().await; @@ -6780,13 +7306,32 @@ mod tests { let changed_info = put_real_cold_fill_object(&store, &bucket, object, &changed_body).await; assert_ne!(initial_info.etag, changed_info.etag); + if conditional { + let (paths, _) = crate::app::gating_test_env::shared_gating_ecstore_and_disk_paths().await; + for path in paths { + tokio::fs::remove_dir_all( + path.join(&bucket) + .join(object) + .join(changed_info.data_dir.expect("replacement data directory").to_string()), + ) + .await + .expect("remove replacement shards before fallback"); + } + } producer.relinquish_or_finish(ColdFillError::Storage(StorageError::Timeout)); - let mut response = tokio::time::timeout(Duration::from_secs(10), request) + let result = tokio::time::timeout(Duration::from_secs(10), request) .await .expect("generation-changing GET must complete") - .expect("generation-changing GET task must join") - .expect("generation-changing GET must fall back successfully"); + .expect("generation-changing GET task must join"); + if conditional { + let error = result.expect_err("fallback must recheck the replacement ETag before body access"); + assert_eq!(error.code(), &S3ErrorCode::NotModified); + assert_eq!(coordinator.global_waiter_count_for_test(), 0); + assert_eq!(coordinator.active_session_count_for_test(), 0); + return; + } + let mut response = result.expect("generation-changing GET must fall back successfully"); let mut response_body = response.output.body.take().expect("GET response must include a body"); let mut actual = Vec::with_capacity(changed_body.len()); while let Some(chunk) = response_body.next().await { @@ -10977,53 +11522,6 @@ mod tests { assert_eq!(DefaultObjectUsecase::get_object_parts_count(Some(1), &single_part_multipart), Some(1)); } - #[test] - fn cold_fill_conditions_fail_before_phase_probe_advances() { - fn run_phase_probe(headers: &HeaderMap, info: &ObjectInfo) -> (S3Result<()>, [usize; 3]) { - let coordination = AtomicUsize::new(0); - let permit = AtomicUsize::new(0); - let reader = AtomicUsize::new(0); - let result = DefaultObjectUsecase::validate_get_object_before_cold_fill(headers, None, info); - if result.is_ok() { - coordination.fetch_add(1, AtomicOrdering::Relaxed); - permit.fetch_add(1, AtomicOrdering::Relaxed); - reader.fetch_add(1, AtomicOrdering::Relaxed); - } - ( - result, - [ - coordination.load(AtomicOrdering::Relaxed), - permit.load(AtomicOrdering::Relaxed), - reader.load(AtomicOrdering::Relaxed), - ], - ) - } - - let info = ObjectInfo { - etag: Some("phase-etag".to_string()), - parts: Arc::new(vec![rustfs_filemeta::ObjectPartInfo { - number: 1, - ..Default::default() - }]), - ..Default::default() - }; - - let mut not_modified = HeaderMap::new(); - not_modified.insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("\"phase-etag\"")); - let (result, phases) = run_phase_probe(¬_modified, &info); - assert_eq!(result.expect_err("matching If-None-Match must reject").code(), &S3ErrorCode::NotModified); - assert_eq!(phases, [0, 0, 0]); - - let mut precondition_failed = HeaderMap::new(); - precondition_failed.insert(http::header::IF_MATCH, HeaderValue::from_static("\"other-etag\"")); - let (result, phases) = run_phase_probe(&precondition_failed, &info); - assert_eq!( - result.expect_err("mismatched If-Match must reject").code(), - &S3ErrorCode::PreconditionFailed - ); - assert_eq!(phases, [0, 0, 0]); - } - #[tokio::test] async fn execute_get_object_rejects_range_with_part_number() { let input = GetObjectInput::builder() diff --git a/rustfs/src/app/object/head.rs b/rustfs/src/app/object/head.rs index 4d53df5ad..35537850b 100644 --- a/rustfs/src/app/object/head.rs +++ b/rustfs/src/app/object/head.rs @@ -312,10 +312,6 @@ impl DefaultObjectUsecase { version_id, part_number, range, - if_none_match, - if_match, - if_modified_since, - if_unmodified_since, .. } = req.input.clone(); @@ -343,6 +339,7 @@ impl DefaultObjectUsecase { let opts: ObjectOptions = get_opts(&bucket, &key, version_id, part_number, &req.headers) .await .map_err(ApiError::from)?; + let opts = internal_object_info_lookup_opts(opts); prepare_odm_read_generation(&store, &mut req, &bucket).await; @@ -413,41 +410,6 @@ impl DefaultObjectUsecase { opts.version_id.as_deref(), )); } - if let Some(match_etag) = if_none_match - && let Some(strong_etag) = match_etag.into_etag() - && info - .etag - .as_ref() - .is_some_and(|etag| ETag::Strong(etag.clone()) == strong_etag) - { - return Err(S3Error::new(S3ErrorCode::NotModified)); - } - if let Some(modified_since) = if_modified_since { - // obj_time < givenTime + 1s - if info.mod_time.is_some_and(|mod_time| { - let give_time: OffsetDateTime = modified_since.into(); - mod_time < give_time.add(time::Duration::seconds(1)) - }) { - return Err(S3Error::new(S3ErrorCode::NotModified)); - } - } - if let Some(match_etag) = if_match { - if let Some(strong_etag) = match_etag.into_etag() - && info - .etag - .as_ref() - .is_some_and(|etag| ETag::Strong(etag.clone()) != strong_etag) - { - return Err(S3Error::new(S3ErrorCode::PreconditionFailed)); - } - } else if let Some(unmodified_since) = if_unmodified_since - && info.mod_time.is_some_and(|mod_time| { - let give_time: OffsetDateTime = unmodified_since.into(); - mod_time > give_time.add(time::Duration::seconds(1)) - }) - { - return Err(S3Error::new(S3ErrorCode::PreconditionFailed)); - } // An authorized replication convergence check only needs etag/size/mtime // to compare source and replica; it holds no customer key, so the SSE-C // read validation is skipped for it (and only it). @@ -465,6 +427,14 @@ impl DefaultObjectUsecase { )?; } + let conditions = check_preconditions(&req.headers, &info); + if conditions.is_err() && !replication_check { + let principal = SseKmsPrincipal::from_request(&req); + crate::storage::sse::validate_conditional_read(&bucket, &key, &info.user_defined, &req.headers, principal.as_ref()) + .await?; + } + conditions?; + // Compute x-amz-expiration header from lifecycle prediction (before info is partially moved) let expiration_header = resolve_put_object_expiration(&bucket, &info).await; // Clone ObjectInfo for event notification only when an event will diff --git a/rustfs/src/app/object/internal_put.rs b/rustfs/src/app/object/internal_put.rs index 45dca295a..c27d6e3bf 100644 --- a/rustfs/src/app/object/internal_put.rs +++ b/rustfs/src/app/object/internal_put.rs @@ -37,7 +37,7 @@ use crate::app::storage_api::object_usecase::sse::{ use crate::capacity::record_capacity_write; use crate::runtime_sources::NotifyInterface; use crate::table_catalog::{ - EcStoreTableCatalogObjectBackend, TableCatalogLockGuard, TableCatalogObjectBackend, TableCatalogStoreError, + EcStoreTableCatalogObjectBackend, TableCatalogLockGuard, TableCatalogObjectBackend, default_table_bucket_publication_lock_path, }; use http::HeaderName; @@ -250,13 +250,7 @@ impl DefaultObjectUsecase { let guard = backend .acquire_read_lock(bucket, &default_table_bucket_publication_lock_path()) .await - .map_err(|err: TableCatalogStoreError| { - let code = match err { - TableCatalogStoreError::Unavailable(_) => S3ErrorCode::ServiceUnavailable, - _ => S3ErrorCode::InternalError, - }; - api_error_from_s3(S3Error::with_message(code, "failed to acquire table publication guard")) - })?; + .map_err(ApiError::from)?; Ok(Some(guard)) } diff --git a/rustfs/src/app/object/mod.rs b/rustfs/src/app/object/mod.rs index d26ee4a80..29086df8f 100644 --- a/rustfs/src/app/object/mod.rs +++ b/rustfs/src/app/object/mod.rs @@ -204,6 +204,8 @@ fn remove_source_replication_bookkeeping(user_defined: &mut HashMap Option { + PUT_STORAGE_TEST_FAILURE + .try_with(|(configured_stage, error)| (*configured_stage == stage).then(|| error.clone())) + .ok() + .flatten() +} + const DEFAULT_PUT_LARGE_CONCURRENCY_TUNING_MIN_SIZE_BYTES: i64 = 32 * 1024 * 1024; const ENV_ZERO_COPY_EAGER_PUT_MAX_SIZE_BYTES: &str = "RUSTFS_ZERO_COPY_EAGER_PUT_MAX_SIZE_BYTES"; @@ -1098,6 +1119,11 @@ impl PutObjectCompletion { /// Record the failed completion and hand the error back to the caller. fn fail_put_object(completion: PutObjectCompletion, err: S3Error) -> S3Error { + let err = if matches!(&completion, PutObjectCompletion::S3(_)) { + normalize_put_read_quorum_error(err) + } else { + err + }; let result: S3Result> = Err(err); let _ = completion.complete(&result); match result { @@ -1185,7 +1211,9 @@ impl DefaultObjectUsecase { #[instrument(name = "execute_put_object", level = "info", skip(self, _fs, req))] #[hotpath::measure(impl_type = "DefaultObjectUsecase", future = true)] pub async fn execute_put_object(&self, _fs: &FS, req: S3Request) -> S3Result> { - self.execute_put_object_boxed(_fs, req).await + self.execute_put_object_boxed(_fs, req) + .await + .map_err(normalize_put_read_quorum_error) } fn execute_put_object_boxed<'a>( @@ -1233,6 +1261,7 @@ impl DefaultObjectUsecase { { return Err(s3_error!(InvalidStorageClass)); } + reject_non_wildcard_write_if_none_match(req.input.if_none_match.as_ref(), &req.headers)?; // An authorized inbound replication PUT must store the replica verbatim. // Legacy snowball-extracted members may still carry the auto-extract // metadata, which replication replays as a header. Do not interpret that @@ -1549,6 +1578,9 @@ impl DefaultObjectUsecase { let sse_config_stage_start = put_stage_metrics_enabled.then(Instant::now); let bucket_sse_config = load_bucket_default_sse_config(&bucket).await; + #[cfg(test)] + let bucket_sse_config = injected_put_storage_failure(PutStorageTestStage::DefaultSse) + .map_or(bucket_sse_config, |err| Err(ApiError::from(err).into())); rustfs_io_metrics::record_put_object_stage_duration_from("app_sse_config_lookup", sse_config_stage_start); let bucket_sse_config = bucket_sse_config?; debug!( @@ -1699,6 +1731,8 @@ impl DefaultObjectUsecase { // the dst xl.meta that rename_data already reads, saving a full-disk // metadata fanout per PUT. let prelookup_required = version_id.is_some() || object_lock_checks_required_for_state(&object_lock_config_state); + #[cfg(test)] + let prelookup_required = prelookup_required || injected_put_storage_failure(PutStorageTestStage::Prelookup).is_some(); // Outer None = prelookup skipped (accounting comes from the commit // backfill); Some(inner) = the previous current size as observed by the // lookup, with the pre-#1009 semantics kept bit-for-bit. @@ -1713,6 +1747,9 @@ impl DefaultObjectUsecase { crate::hp_guard!("S3::put_object_prelookup"); store.get_object_info(&bucket, &key, ¤t_opts).await }; + #[cfg(test)] + let previous_current_info = + injected_put_storage_failure(PutStorageTestStage::Prelookup).map_or(previous_current_info, Err); Some(match previous_current_info { Ok(existing_obj_info) => { validate_existing_object_lock_for_write(&object_lock_config_state, &existing_obj_info, &opts)?; @@ -1751,6 +1788,7 @@ impl DefaultObjectUsecase { )?; } + let checksum_input_stage_start = put_stage_metrics_enabled.then(Instant::now); let mut md5hex = match content_md5 { Some(PutObjectContentMd5::Base64(base64_md5)) => { let md5 = base64_simd::STANDARD @@ -1763,12 +1801,15 @@ impl DefaultObjectUsecase { }; let mut sha256hex = get_content_sha256_with_query(headers, query); + rustfs_io_metrics::record_put_object_stage_duration_from("app_checksum_input_prepare", checksum_input_stage_start); let mut write_plan = WritePlan::new(); // Additional-checksum (XXHash3/64/128, SHA-512) values to echo on the PutObject // response (#1256); captured at want_checksum set points before opts is moved. let mut put_extra_checksum_headers: Vec<(&'static str, String)> = Vec::new(); + let reader_prepare_stage_start = put_stage_metrics_enabled.then(Instant::now); let mut reader = if should_compress { + let compressed_reader_stage_start = put_stage_metrics_enabled.then(Instant::now); let body = tokio::io::BufReader::with_capacity( buffer_size, StreamReader::new(body.map(|f| f.map_err(s3s_body_error_to_io))), @@ -1791,51 +1832,98 @@ impl DefaultObjectUsecase { size = HashReader::SIZE_PRESERVE_LAYER; write_plan = write_plan.with_compression(algorithm); + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_reader_prepare_compressed", + compressed_reader_stage_start, + ); hrd } else { if use_zero_copy_eager_put_path { let zero_copy_start = std::time::Instant::now(); + let eager_read_stage_start = put_stage_metrics_enabled.then(Instant::now); let eager_body = read_zero_copy_put_body_exact(body, body_size as usize).await?; + rustfs_io_metrics::record_put_object_stage_duration_from("app_body_eager_read_zero_copy", eager_read_stage_start); rustfs_io_metrics::record_zero_copy_write(body_size as usize, zero_copy_start.elapsed().as_secs_f64() * 1000.0); - HashReader::from_stream(eager_body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)? + let hash_reader_stage_start = put_stage_metrics_enabled.then(Instant::now); + let reader = + HashReader::from_stream(eager_body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)?; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_hash_reader_setup_zero_copy", + hash_reader_stage_start, + ); + reader } else if use_empty_or_small_eager_put_path { if (body_size as usize) <= POOL_BYPASS_MAX_SIZE { // Bypass BytesPool for very small objects to avoid Small-tier // Mutex contention under high concurrency. Direct allocation // for ≤4KiB is negligible cost. + let eager_read_stage_start = put_stage_metrics_enabled.then(Instant::now); let eager_body = read_small_put_body_exact_direct( StreamReader::new(body.map(|f| f.map_err(s3s_body_error_to_io))), body_size as usize, ) .await?; - HashReader::from_stream(eager_body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)? + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_body_eager_read_small_direct", + eager_read_stage_start, + ); + let hash_reader_stage_start = put_stage_metrics_enabled.then(Instant::now); + let reader = HashReader::from_stream(eager_body, size, actual_size, md5hex, sha256hex, false) + .map_err(ApiError::from)?; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_hash_reader_setup_small_direct", + hash_reader_stage_start, + ); + reader } else { let pool = get_concurrency_manager().bytes_pool(); + let eager_read_stage_start = put_stage_metrics_enabled.then(Instant::now); let eager_body = read_small_put_body_exact_pooled( StreamReader::new(body.map(|f| f.map_err(s3s_body_error_to_io))), body_size as usize, pool.as_ref(), ) .await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_body_eager_read_small_pooled", + eager_read_stage_start, + ); + let hash_reader_stage_start = put_stage_metrics_enabled.then(Instant::now); let eager_reader = PooledBufferReader::new(eager_body, body_size as usize); - HashReader::from_stream(eager_reader, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)? + let reader = HashReader::from_stream(eager_reader, size, actual_size, md5hex, sha256hex, false) + .map_err(ApiError::from)?; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_hash_reader_setup_small_pooled", + hash_reader_stage_start, + ); + reader } } else { + let streaming_reader_stage_start = put_stage_metrics_enabled.then(Instant::now); let body = tokio::io::BufReader::with_capacity( buffer_size, StreamReader::new(body.map(|f| f.map_err(s3s_body_error_to_io))), ); - HashReader::from_stream(body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)? + let reader = + HashReader::from_stream(body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)?; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_reader_prepare_streaming", + streaming_reader_stage_start, + ); + reader } }; + rustfs_io_metrics::record_put_object_stage_duration_from("app_reader_prepare", reader_prepare_stage_start); if size >= 0 { + let checksum_setup_stage_start = put_stage_metrics_enabled.then(Instant::now); if let Err(err) = reader.add_checksum(headers, trailer_source(trailing_headers.clone()), false) { return Err(ApiError::from(err).into()); } opts.want_checksum = reader.checksum(); put_extra_checksum_headers = additional_checksum_echo_pairs(&opts.want_checksum); + rustfs_io_metrics::record_put_object_stage_duration_from("app_reader_checksum_setup", checksum_setup_stage_start); } rustfs_io_metrics::record_put_object_path(put_path); rustfs_io_metrics::record_put_object_stage_duration_from("ingress_prepare", ingress_stage_start); @@ -1987,7 +2075,9 @@ impl DefaultObjectUsecase { } else { None }; - let put_commit = spawn_traced_join({ + #[cfg(test)] + let injected_commit_error = injected_put_storage_failure(PutStorageTestStage::Commit); + let put_commit_future = { let store = Arc::clone(&store); let bucket = bucket.clone(); let key = key.clone(); @@ -2007,6 +2097,11 @@ impl DefaultObjectUsecase { .and_then(ObjectTrafficHealth::track_write_storage); let mut reader = reader; let store_put_stage_start = put_stage_metrics_enabled.then(Instant::now); + #[cfg(test)] + if let Some(err) = injected_commit_error { + store_put_watchdog.cancel(); + return Err(ApiError::from(err).into()); + } let (obj_info, backfilled_old_current_size) = match store .put_object_with_old_current_size(&bucket, &key, &mut reader, &opts) .await @@ -2069,17 +2164,36 @@ impl DefaultObjectUsecase { wait_for_put_post_store_test_hook(&bucket).await; let post_store_stage_start = put_stage_metrics_enabled.then(Instant::now); + let transition_enqueue_stage_start = put_stage_metrics_enabled.then(Instant::now); maybe_enqueue_transition_immediate(&obj_info, LcEventSrc::S3PutObject).await; - let _ = invalidate_object_data_cache_after_put_success(&cache_adapter, &bucket, &key).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_post_store_transition_enqueue", + transition_enqueue_stage_start, + ); + let cache_invalidate_stage_start = put_stage_metrics_enabled.then(Instant::now); + let _ = invalidate_object_data_cache_after_put_success(&cache_adapter, &bucket, &key).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_post_store_data_cache_invalidate", + cache_invalidate_stage_start, + ); + + let versioning_lookup_stage_start = put_stage_metrics_enabled.then(Instant::now); let put_versioned = BucketVersioningSys::prefix_enabled(&bucket, &key).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_post_store_versioning_lookup", + versioning_lookup_stage_start, + ); // Fast in-memory update for immediate quota and admin usage consistency. // The previous current size comes from the prelookup when it ran, // otherwise from the rename_data backfill (rustfs/backlog#1009); the // backfill reproduces the lookup's observation bit for bit (latest // version's ObjectInfo.size — 0 for a delete-marker latest — or // not-found → None). + let quota_size_stage_start = put_stage_metrics_enabled.then(Instant::now); let committed_size = quota_accounting_object_size(&obj_info, quota_enabled)?; + rustfs_io_metrics::record_put_object_stage_duration_from("app_post_store_quota_size", quota_size_stage_start); + let usage_memory_stage_start = put_stage_metrics_enabled.then(Instant::now); match prelookup_previous_current_size.or_else(|| previous_current_size_from_backfill(backfilled_old_current_size)) { Some(previous_current_size) => { @@ -2105,16 +2219,27 @@ impl DefaultObjectUsecase { record_bucket_object_write_unknown_previous_memory(&bucket, committed_size, put_versioned).await; } } + rustfs_io_metrics::record_put_object_stage_duration_from("app_post_store_usage_memory", usage_memory_stage_start); if dsc.replicate_any() { + let replication_schedule_stage_start = put_stage_metrics_enabled.then(Instant::now); schedule_object_replication(obj_info.clone(), store, dsc).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_post_store_replication_schedule", + replication_schedule_stage_start, + ); } + let dirty_usage_stage_start = put_stage_metrics_enabled.then(Instant::now); rustfs_scanner::record_dirty_usage_object_from_producer( &bucket, &key, rustfs_scanner::SegmentInvalidationProducerIdentity::PutObject, ); + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_post_store_dirty_usage_record", + dirty_usage_stage_start, + ); rustfs_io_metrics::record_put_object_stage_duration_from("app_post_store_bookkeeping", post_store_stage_start); let capacity_update_stage_start = put_stage_metrics_enabled.then(Instant::now); @@ -2124,7 +2249,14 @@ impl DefaultObjectUsecase { Ok::<_, S3Error>(PutObjectCommitResult { obj_info, put_versioned }) } - }); + }; + #[cfg(feature = "diagnostic-read-version-caller")] + let put_commit_future = crate::app::storage_api::diagnostic::with_read_version_spawn_context( + crate::app::storage_api::diagnostic::ReadVersionEntrySource::AppPutOwner, + crate::app::storage_api::diagnostic::ReadVersionContextBoundary::AppPutOwner, + put_commit_future, + ); + let put_commit = spawn_traced_join(put_commit_future); let commit_stage_start = put_stage_metrics_enabled.then(Instant::now); let put_commit_result = if let Some(cancellation) = eager_put_commit_cancellation { EagerPutCommitOwner::new(put_commit, cancellation, EAGER_PUT_COMMIT_CANCELLATION_GRACE) @@ -2203,6 +2335,130 @@ mod tests { use super::*; use futures::StreamExt; use http::{HeaderMap, HeaderName, HeaderValue, Method}; + + #[tokio::test] + #[serial_test::serial] + async fn put_read_quorum_failures_are_normalized_at_each_stage() { + use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions}; + + let (store, context) = real_store_test_context().await; + let bucket = format!("put-quorum-{}", Uuid::new_v4()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("read-quorum PUT bucket must be created"); + let usecase = DefaultObjectUsecase::with_context(Some(context)); + + for (index, stage) in [ + PutStorageTestStage::DefaultSse, + PutStorageTestStage::Prelookup, + PutStorageTestStage::Commit, + ] + .into_iter() + .enumerate() + { + for method in [Method::PUT, Method::POST] { + let key = format!("refused-{index}-{method}"); + let input = PutObjectInput::builder() + .bucket(bucket.clone()) + .key(key.clone()) + .body(Some(StreamingBlob::from_bytes(Bytes::from_static(b"quorum probe")))) + .content_length(Some(12)) + .build() + .expect("quorum probe input must build"); + let mut req = build_request(input, method.clone()); + if method == Method::POST { + req.extensions.insert(PostObjectRequestMarker); + } + let err = PUT_STORAGE_TEST_FAILURE + .scope( + (stage, StorageError::InsufficientReadQuorum("bucket".into(), "object".into())), + usecase.execute_put_object(&FS::new(), req), + ) + .await + .expect_err("read quorum failure must refuse the upload"); + assert_eq!(err.code(), &S3ErrorCode::ServiceUnavailable, "stage {index}, method {method}"); + assert_eq!(err.status_code(), Some(http::StatusCode::SERVICE_UNAVAILABLE)); + assert_eq!(err.message(), Some("The service is unavailable. Please retry.")); + assert!(matches!( + err.source().and_then(|source| source.downcast_ref::()), + Some(StorageError::InsufficientReadQuorum(_, _)) + )); + let response = err.to_http_response().expect("PUT error must serialize"); + assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE); + let xml = http_body_util::BodyExt::collect(response.into_body()) + .await + .expect("PUT error XML must be readable") + .to_bytes(); + let xml = std::str::from_utf8(&xml).expect("PUT error XML must be UTF-8"); + assert!(xml.contains("ServiceUnavailable"), "{xml}"); + assert!(!xml.contains("SlowDownRead"), "{xml}"); + let lookup = store.get_object_info(&bucket, &key, &ObjectOptions::default()).await; + assert!( + lookup.as_ref().is_err_and(is_err_object_not_found), + "failed PUT must leave no object: {lookup:?}" + ); + } + } + } + + #[tokio::test] + #[serial_test::serial] + async fn put_capacity_failure_returns_unavailable_without_publishing_object() { + use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions}; + + let (store, context) = real_store_test_context().await; + let bucket = format!("put-capacity-{}", Uuid::new_v4()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("capacity PUT bucket must be created"); + let usecase = DefaultObjectUsecase::with_context(Some(context)); + for (index, failure) in [StorageError::DiskFull, StorageError::StorageFull].into_iter().enumerate() { + let key = format!("refused-{index}"); + let input = PutObjectInput::builder() + .bucket(bucket.clone()) + .key(key.clone()) + .body(Some(StreamingBlob::from_bytes(Bytes::from_static(b"capacity probe")))) + .content_length(Some(14)) + .build() + .expect("capacity PUT input must build"); + let request = build_request(input, Method::PUT); + let error = PUT_STORAGE_TEST_FAILURE + .scope((PutStorageTestStage::Commit, failure), usecase.execute_put_object(&FS::new(), request)) + .await + .expect_err("capacity refusal must fail PUT"); + assert_eq!(error.code(), &S3ErrorCode::ServiceUnavailable); + assert!(matches!( + error.source().and_then(|source| source.downcast_ref::()), + Some(StorageError::DiskFull | StorageError::StorageFull) + )); + let response = error.to_http_response().expect("PUT error must serialize"); + assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE); + assert!(!response.headers().contains_key(http::header::RETRY_AFTER)); + let body = http_body_util::BodyExt::collect(response.into_body()) + .await + .expect("PUT error XML must be readable") + .to_bytes(); + let xml = std::str::from_utf8(&body).expect("PUT error XML must be UTF-8"); + assert!(xml.contains("ServiceUnavailable")); + let lookup = store.get_object_info(&bucket, &key, &ObjectOptions::default()).await; + assert!( + lookup.as_ref().is_err_and(is_err_object_not_found), + "failed PUT must publish no object: {lookup:?}" + ); + } + } + + #[test] + fn failed_put_completion_normalizes_only_s3_errors() { + let error = || S3Error::from(ApiError::from(StorageError::ErasureReadQuorum)); + let err = fail_put_object(PutObjectCompletion::S3(OperationHelper::Disabled), error()); + assert_eq!(err.code(), &S3ErrorCode::ServiceUnavailable); + assert_eq!(err.message(), Some("The service is unavailable. Please retry.")); + let internal = fail_put_object(PutObjectCompletion::Internal(None), error()); + assert_eq!(internal.code(), &S3ErrorCode::Custom("SlowDownRead".into())); + } use s3s::dto::{ DefaultRetention, ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRule, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule, @@ -3317,6 +3573,25 @@ mod tests { assert!(should_use_zero_copy(2 * 1024 * 1024, &headers)); } + #[tokio::test] + async fn execute_put_object_rejects_post_object_etag_if_none_match_header() { + let input = PutObjectInput::builder() + .bucket("test-bucket".to_string()) + .key("test-key".to_string()) + .build() + .expect("POST-to-PUT input should build without conditional DTO fields"); + let mut req = build_request(input, Method::POST); + req.extensions.insert(PostObjectRequestMarker); + req.headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("\"different-etag\"")); + + let usecase = DefaultObjectUsecase::without_context(); + let err = Box::pin(usecase.execute_put_object(&FS::new(), req)) + .await + .expect_err("POST header-only ETag precondition must fail before storage access"); + assert_eq!(err.code(), &S3ErrorCode::NotImplemented); + } + #[tokio::test] async fn execute_put_object_rejects_post_object_sse_kms_from_input() { let input = PutObjectInput::builder() @@ -3913,7 +4188,7 @@ mod tests { #[tokio::test] #[serial_test::serial] - async fn put_rejects_rotated_quota_capability_before_rename() { + async fn put_commits_after_quota_fleet_proof_loss_before_rename() { use crate::app::storage_api::test::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause}; let (store, bucket) = crate::app::gating_test_env::durable_quota_test_bucket("rotated-proof-put-quota", 4096).await; @@ -3927,27 +4202,18 @@ mod tests { .await }); barrier.wait_until_paused().await; - assert!( - crate::storage::storage_api::ecstore_notification::rotate_cross_pool_fence_fleet_proof_for_test(), - "the gating environment must have a current fleet proof" - ); + let _without_proof = crate::storage::storage_api::ecstore_notification::without_cross_pool_fence_fleet_proof_for_test(); + assert!(crate::storage::storage_api::ecstore_notification::acquire_cross_pool_fence_fleet_proof().is_none()); barrier.release(); - let err = put - .await + put.await .expect("PUT task should not panic") - .expect_err("a replaced fleet proof must fence the authoritative rename"); - assert!(matches!( - err, - StorageError::NamespaceLockQuorumUnavailable { - mode: "quota_reservation", - .. - } - )); - store + .expect("fleet proof loss must not invalidate a live quota reservation and its disk fences"); + let stored = store .get_object_info(&bucket, "object", &ObjectOptions::default()) .await - .expect_err("proof rotation before rename must leave no committed object"); + .expect("the committed object must remain readable after fleet proof loss"); + assert_eq!(stored.size, 4096); } #[tokio::test] @@ -4230,6 +4496,83 @@ mod tests { assert!(is_err_object_not_found(&lookup_err), "{lookup_err}"); } + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + #[serial_test::serial] + fn diagnostic_s3_put_preserves_api_through_app_owner_and_tier_lookup() { + use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions}; + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + use rustfs_io_metrics::s3_http_metrics::{S3HttpRequestGuard, current_s3_http_operation_name}; + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let (store, context) = runtime.block_on(Box::pin(real_store_test_context())); + let bucket = format!("put-context-{}", Uuid::new_v4()); + runtime + .block_on(Box::pin(store.make_bucket(&bucket, &MakeBucketOptions::default()))) + .expect("test bucket"); + let payload = Bytes::from_static(b"diagnostic PUT context"); + let input = PutObjectInput::builder() + .bucket(bucket.clone()) + .key("object.bin".to_string()) + .body(Some(StreamingBlob::from(s3s::Body::from(payload.clone())))) + .content_length(Some(i64::try_from(payload.len()).expect("payload length"))) + .build() + .expect("PUT input"); + let usecase = DefaultObjectUsecase::with_context(Some(context)); + let recorder = DebuggingRecorder::new(); + let snapshotter = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let mut request = S3HttpRequestGuard::new("PUT"); + metrics::with_local_recorder(&recorder, || { + request.in_scope(|| { + runtime.block_on(async { + Box::pin(usecase.execute_put_object(&FS::new(), build_request(input, Method::PUT))) + .await + .expect("S3 PUT must commit"); + assert_eq!(current_s3_http_operation_name(), Some("s3:PutObject")); + assert_eq!(crate::app::storage_api::diagnostic::current_read_version_entry_source(), None); + }) + }); + }); + let context_rows = snapshotter + .snapshot() + .into_vec() + .into_iter() + .filter(|(key, _, _, _)| key.key().name() == "rustfs_storage_diagnostic_context_transitions_total") + .map(|(key, _, _, value)| { + ( + key.key() + .labels() + .map(|l| (l.key().to_string(), l.value().to_string())) + .collect::>(), + value, + ) + }) + .collect::>(); + for boundary in ["app_put_owner", "put_commit_owner", "tier_free_version_source_lookup"] { + assert!( + context_rows.iter().any(|(labels, value)| { + labels.get("entry_source").map(String::as_str) == Some("app_put_owner") + && labels.get("boundary").map(String::as_str) == Some(boundary) + && labels.get("parent_api_operation").map(String::as_str) == Some("s3:PutObject") + && labels.get("child_api_operation").map(String::as_str) == Some("s3:PutObject") + && labels.get("api_context").map(String::as_str) == Some("preserved") + && matches!(value, DebugValue::Counter(count) if *count > 0) + }), + "actual S3 PUT boundary must preserve the parent API: {boundary}; {context_rows:?}" + ); + } + let info = runtime + .block_on(Box::pin(store.get_object_info(&bucket, "object.bin", &ObjectOptions::default()))) + .expect("committed object"); + assert_eq!(info.size, i64::try_from(payload.len()).expect("payload length")); + assert_eq!(current_s3_http_operation_name(), None); + assert_eq!(crate::app::storage_api::diagnostic::current_read_version_entry_source(), None); + } + #[tokio::test] #[serial_test::serial] async fn execute_put_object_still_writes_plaintext_without_bucket_encryption() { diff --git a/rustfs/src/app/object/shared.rs b/rustfs/src/app/object/shared.rs index 421c47cdf..ac8ddeddf 100644 --- a/rustfs/src/app/object/shared.rs +++ b/rustfs/src/app/object/shared.rs @@ -1092,33 +1092,43 @@ pub(crate) fn odm_check_source_preconditions(headers: &HeaderMap, head: &SourceH if (needs_etag && head.etag.is_none()) || (needs_mtime && head.last_modified.is_none()) { return Err(odm_source_unavailable_error("missing_source_validator")); } + let mut metadata = HashMap::new(); + for (name, value) in [("cache-control", &head.cache_control), ("expires", &head.expires)] { + if let Some(value) = value { + metadata.insert(name.to_owned(), value.clone()); + } + } let info = ObjectInfo { etag: head.etag.clone(), mod_time: head.last_modified.map(OffsetDateTime::from), + user_defined: metadata.into(), ..Default::default() }; - // A successful source read establishes wildcard existence, but the - // remaining conditions must still run in their ordinary precedence. - if head.etag.is_none() && (if_match == Some("*") || if_none_match == Some("*")) { - let mut remaining = headers.clone(); - if if_match == Some("*") { - remaining.remove(http::header::IF_MATCH); - remaining.remove(http::header::IF_UNMODIFIED_SINCE); - } - if if_none_match == Some("*") { - remaining.remove(http::header::IF_NONE_MATCH); - remaining.remove(http::header::IF_MODIFIED_SINCE); - } - check_preconditions(&remaining, &info)?; - return if if_none_match == Some("*") { - Err(S3Error::new(S3ErrorCode::NotModified)) - } else { - Ok(()) - }; - } check_preconditions(headers, &info) } +/// S3 documents write-side `If-None-Match` only as `*` (create-only). +/// Reject unsupported ETag comparisons with 501 NotImplemented. +/// Treating it as an ETag comparison would let a write overwrite an existing key +/// while still satisfying a policy that requires `s3:if-none-match`. +pub(crate) fn reject_non_wildcard_write_if_none_match( + if_none_match: Option<&s3s::dto::ETagCondition>, + headers: &HeaderMap, +) -> S3Result<()> { + // POST parses conditional DTO fields from the form, so HTTP headers also need validation. + if matches!(if_none_match, Some(s3s::dto::ETagCondition::ETag(_))) + || headers.get_all(http::header::IF_NONE_MATCH).iter().any(|value| { + value.to_str().map_or(true, |value| { + let value = value.trim(); + !value.is_empty() && value != "*" + }) + }) + { + return Err(S3Error::new(S3ErrorCode::NotImplemented)); + } + Ok(()) +} + #[cfg(test)] mod tests { use super::*; @@ -1129,6 +1139,54 @@ mod tests { }; use std::sync::Arc; + #[test] + fn reject_non_wildcard_write_if_none_match_allows_only_wildcard() { + let headers = HeaderMap::new(); + assert!(reject_non_wildcard_write_if_none_match(None, &headers).is_ok()); + assert!(reject_non_wildcard_write_if_none_match(Some(&s3s::dto::ETagCondition::Any), &headers).is_ok()); + + let etag = s3s::dto::ETagCondition::ETag(ETag::Strong("9b2cf535f27731c974343645a3985328".to_owned())); + let err = + reject_non_wildcard_write_if_none_match(Some(&etag), &headers).expect_err("an ETag If-None-Match must be rejected"); + assert_eq!(*err.code(), S3ErrorCode::NotImplemented); + assert_eq!(err.status_code(), Some(http::StatusCode::NOT_IMPLEMENTED)); + assert_eq!( + err.message(), + Some("A header you provided implies functionality that is not implemented.") + ); + } + + #[test] + fn reject_non_wildcard_write_if_none_match_checks_header_without_dto() { + for value in ["*", " * ", "", " "] { + let mut headers = HeaderMap::new(); + headers.insert(http::header::IF_NONE_MATCH, http::HeaderValue::from_str(value).expect("valid header")); + assert!(reject_non_wildcard_write_if_none_match(None, &headers).is_ok(), "{value:?}"); + } + for value in ["\"different-etag\"", "W/\"weak-etag\"", "unquoted-etag"] { + let mut headers = HeaderMap::new(); + headers.insert(http::header::IF_NONE_MATCH, http::HeaderValue::from_str(value).expect("valid header")); + let err = reject_non_wildcard_write_if_none_match(None, &headers) + .expect_err("header-only ETag precondition must be rejected"); + assert_eq!(err.code(), &S3ErrorCode::NotImplemented); + } + } + + #[test] + fn reject_non_wildcard_write_if_none_match_rejects_conflicting_sources() { + for value in [ + http::HeaderValue::from_static("\"different-etag\""), + http::HeaderValue::from_bytes(b"\xff").expect("obs-text is a valid HTTP header"), + ] { + let mut headers = HeaderMap::new(); + headers.append(http::header::IF_NONE_MATCH, http::HeaderValue::from_static("*")); + headers.append(http::header::IF_NONE_MATCH, value); + let err = reject_non_wildcard_write_if_none_match(Some(&s3s::dto::ETagCondition::Any), &headers) + .expect_err("a form wildcard must not hide an unsupported raw HTTP precondition"); + assert_eq!(err.code(), &S3ErrorCode::NotImplemented); + } + } + #[test] fn s3_response_version_id_distinguishes_absent_null_and_uuid_versions() { let version = Uuid::parse_str("9341ae04-d4ce-468c-a4e1-6501d58cd6b7").unwrap(); @@ -1513,7 +1571,7 @@ mod tests { #[test] fn build_put_like_object_lock_metadata_rejects_retain_until_date_without_mode() { - let retain_until = Timestamp::from(OffsetDateTime::now_utc().add(time::Duration::days(1))); + let retain_until = Timestamp::from(OffsetDateTime::now_utc() + time::Duration::days(1)); let err = build_put_like_object_lock_metadata( "test-bucket", &metadata_sys::ObjectLockConfigState::ConfirmedAbsent, @@ -1870,6 +1928,7 @@ mod tests { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: String::new(), + expiry_policy: None, }; assert!(build_put_object_expiration_header(&event).is_none()); @@ -1885,6 +1944,7 @@ mod tests { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: String::new(), + expiry_policy: None, }; // HTTP-date, not RFC 3339: the AWS SDKs parse this field with an @@ -1902,6 +1962,7 @@ mod tests { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: String::new(), + expiry_policy: None, }; assert!(build_put_object_expiration_header(&event).is_none()); @@ -1913,6 +1974,7 @@ mod tests { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: String::new(), + expiry_policy: None, }; assert!(build_put_object_expiration_header(&event).is_none()); diff --git a/rustfs/src/app/runtime_sources.rs b/rustfs/src/app/runtime_sources.rs index ad7c7f949..389d97517 100644 --- a/rustfs/src/app/runtime_sources.rs +++ b/rustfs/src/app/runtime_sources.rs @@ -19,7 +19,8 @@ use crate::runtime_sources as root_runtime_sources; pub(crate) use crate::runtime_sources::ServerContextSlot; pub(crate) use crate::runtime_sources::{ AppContext, current_encryption_service, current_endpoints_handle, current_notification_system, - current_object_data_cache_handle_for_context, current_object_store_handle_for_context, + current_notification_system_for_context, current_object_data_cache_handle_for_context, + current_object_store_handle_for_context, }; use rustfs_s3select_api::{QueryResult, server::dbms::DatabaseManagerSystem}; use s3s::dto::SelectObjectContentInput; diff --git a/rustfs/src/app/storage_api.rs b/rustfs/src/app/storage_api.rs index fc0c51131..4666a403f 100644 --- a/rustfs/src/app/storage_api.rs +++ b/rustfs/src/app/storage_api.rs @@ -24,6 +24,15 @@ pub(crate) fn EndpointServerPools( crate::storage::storage_api::EndpointServerPools::from(pools) } +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) mod diagnostic { + #[cfg(test)] + pub(crate) use crate::storage::storage_api::ecstore_diagnostic::current_read_version_entry_source; + pub(crate) use crate::storage::storage_api::ecstore_diagnostic::{ + ReadVersionContextBoundary, ReadVersionEntrySource, with_read_version_spawn_context, + }; +} + /// S3 wire types for app-layer modules, funneled here so new files stay off /// the direct s3s surface (s3s footprint ratchet, `scripts/check_s3s_footprint.sh`). pub(crate) mod s3 { @@ -1181,6 +1190,8 @@ pub(crate) mod bucket_usecase { } pub(crate) mod object_usecase { + pub(crate) use super::storage_contracts::BUCKET_LIFECYCLE_LOCK_OBJECT; + pub(crate) use super::storage_contracts::object::DeleteObjectCondition; pub(crate) use crate::storage::storage_api::ecstore_object::{WriteCommitGuard, WriteCompletion}; pub(crate) mod object_cache { @@ -1189,8 +1200,10 @@ pub(crate) mod object_usecase { #[cfg(test)] pub(crate) use crate::storage::storage_api::ecstore_object::lookup_get_object_body_cache_hook; pub(crate) use crate::storage::storage_api::ecstore_object::{ - GetObjectBodyCacheHookLookup, get_object_body_cache_plaintext_len, + GetObjectBodyCacheHookLookup, GetObjectReadCondition, GetObjectReadMetadata, get_object_body_cache_plaintext_len, + with_get_object_read_condition, }; + pub(crate) use crate::storage::storage_api::{evaluate_read_preconditions, has_read_preconditions}; } pub(crate) mod contract { @@ -1321,8 +1334,8 @@ pub(crate) mod test { }; pub(crate) mod set_disk { pub(crate) use crate::storage::storage_api::ecstore_set_disk::{ - MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause, - fail_next_quota_ledger_save_for_test, + ConditionalReadBenchmarkMetadataGuard, MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, + PutObjectCommitPause, fail_next_quota_ledger_save_for_test, }; } diff --git a/rustfs/src/auth.rs b/rustfs/src/auth.rs index 68e17da96..f29fa0a82 100644 --- a/rustfs/src/auth.rs +++ b/rustfs/src/auth.rs @@ -15,6 +15,7 @@ use crate::runtime_sources::{AppContext, ServerContextSlot, current_action_credentials, current_ready_iam_handle}; use http::HeaderMap; use http::Uri; +use http::header::{IF_MATCH, IF_NONE_MATCH}; use rustfs_credentials::Credentials; use rustfs_iam::error::Error as IamError; use rustfs_iam::sys::{ @@ -730,10 +731,8 @@ pub fn get_condition_values_with_query_and_client_info( args.insert("LocationConstraint".to_owned(), vec![lc.to_string()]); } - let mut clone_header = header.clone(); - if let Some(v) = clone_header.get("x-amz-signature-age") { + if let Some(v) = header.get("x-amz-signature-age") { args.insert("signatureAge".to_string(), vec![v.to_str().unwrap_or("").to_string()]); - clone_header.remove("x-amz-signature-age"); } for obj_lock in &[ @@ -741,7 +740,7 @@ pub fn get_condition_values_with_query_and_client_info( AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, ] { - let values = clone_header + let values = header .get_all(*obj_lock) .iter() .map(|v| v.to_str().unwrap_or("").to_string()) @@ -749,7 +748,6 @@ pub fn get_condition_values_with_query_and_client_info( if !values.is_empty() { args.insert(obj_lock.trim_start_matches("x-amz-").to_string(), values); } - clone_header.remove(*obj_lock); } // S3 policy condition keys use "x-amz-grant-*" (policy key s3:x-amz-grant-* -> name() returns x-amz-grant-*) @@ -760,7 +758,7 @@ pub fn get_condition_values_with_query_and_client_info( "x-amz-grant-read-acp", "x-amz-grant-write-acp", ] { - let values = clone_header + let values = header .get_all(*grant_header) .iter() .map(|v| v.to_str().unwrap_or("").to_string()) @@ -768,7 +766,6 @@ pub fn get_condition_values_with_query_and_client_info( if !values.is_empty() { args.insert((*grant_header).to_string(), values); } - clone_header.remove(*grant_header); } // Claims and group membership are part of the verified identity, so they are @@ -776,7 +773,15 @@ pub fn get_condition_values_with_query_and_client_info( if let Some(claims) = &cred.claims { for (k, v) in claims { if let Some(v_str) = v.as_str() { - args.insert(k.trim_start_matches("ldap").to_lowercase(), vec![v_str.to_string()]); + let key = k.trim_start_matches("ldap"); + // Request conditions cannot originate from identity claims, even without a header. + if key.eq_ignore_ascii_case(OBJECT_CREATION_OPERATION_CONDITION) + || key.eq_ignore_ascii_case(IF_MATCH.as_str()) + || key.eq_ignore_ascii_case(IF_NONE_MATCH.as_str()) + { + continue; + } + args.insert(key.to_lowercase(), vec![v_str.to_string()]); } } @@ -804,9 +809,27 @@ pub fn get_condition_values_with_query_and_client_info( // both the ones already populated above and the well-known identity keys that are // absent for this credential, since an absent key is exactly what a spoofed header // would fill in. - for key in clone_header.keys() { - if key.as_str().eq_ignore_ascii_case("x-amz-tagging") { - continue; + for key in header.keys() { + match key.as_str() { + // These headers were consumed under their policy names above, or by tagging authorization. + "x-amz-signature-age" + | AMZ_OBJECT_LOCK_MODE_LOWER + | AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER + | AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER + | "x-amz-tagging" => continue, + "if-match" | "if-none-match" => { + // A blank header must not satisfy `Null: false` while storage ignores it. + if let Some(value) = header + .get(key) + .and_then(|value| value.to_str().ok()) + .map(str::trim) + .filter(|value| !value.is_empty()) + { + args.insert(key.as_str().to_string(), vec![value.to_string()]); + } + continue; + } + _ => {} } if is_reserved_condition_key(key.as_str(), &args) { continue; @@ -824,6 +847,9 @@ pub fn get_condition_values_with_query_and_client_info( args } +/// The S3 operation, rather than identity claims or request input, determines this value. +pub(crate) const OBJECT_CREATION_OPERATION_CONDITION: &str = "ObjectCreationOperation"; + /// Condition name for `s3:object-lock-remaining-retention-days`. Storage access /// derives it from the requested retain-until date, so no header may set it. pub(crate) const OBJECT_LOCK_REMAINING_RETENTION_DAYS_CONDITION: &str = "object-lock-remaining-retention-days"; @@ -2039,6 +2065,131 @@ mod tests { ); } + #[test] + fn test_get_condition_values_with_conditional_write_headers() { + let cred = create_test_credentials(); + let mut headers = HeaderMap::new(); + headers.insert("If-None-Match", HeaderValue::from_static(" * ")); + headers.insert("If-Match", HeaderValue::from_static("\"9b2cf535f27731c974343645a3985328\"")); + + let conditions = get_condition_values(&headers, &cred, None, None, None); + + assert_eq!(conditions.get("if-none-match"), Some(&vec!["*".to_string()])); + assert_eq!( + conditions.get("if-match"), + Some(&vec!["\"9b2cf535f27731c974343645a3985328\"".to_string()]) + ); + } + + /// Storage ignores a blank precondition, and s3s rejects a non-UTF-8 one, so + /// neither may satisfy a `Null: false` check on `s3:if-none-match` / `s3:if-match`. + #[test] + fn test_get_condition_values_without_conditional_write_headers() { + let cred = create_test_credentials(); + let mut blank = HeaderMap::new(); + blank.insert("If-None-Match", HeaderValue::from_static("")); + blank.insert("If-Match", HeaderValue::from_static("")); + let mut non_utf8 = HeaderMap::new(); + non_utf8.insert( + "If-None-Match", + HeaderValue::from_bytes(b"\xff").expect("obs-text is a valid header value"), + ); + non_utf8.insert("If-Match", HeaderValue::from_bytes(b"\xff").expect("obs-text is a valid header value")); + + for headers in [HeaderMap::new(), blank, non_utf8] { + let conditions = get_condition_values(&headers, &cred, None, None, None); + + assert!(!conditions.contains_key("if-none-match"), "headers {headers:?}"); + assert!(!conditions.contains_key("if-match"), "headers {headers:?}"); + } + } + + #[test] + fn test_get_condition_values_conditional_write_headers_cannot_be_overridden_by_claims() { + let mut cred = create_service_account_credentials(); + cred.claims = Some(HashMap::from([ + ("if-none-match".to_string(), json!("*")), + ("if-match".to_string(), json!("\"claim-etag\"")), + ("IF-NONE-MATCH".to_string(), json!("*")), + ("ldapIF-MATCH".to_string(), json!("\"claim-etag\"")), + ])); + + for value in [None, Some(""), Some(" ")] { + let mut headers = HeaderMap::new(); + if let Some(value) = value { + headers.insert(IF_NONE_MATCH, HeaderValue::from_str(value).expect("valid header")); + headers.insert(IF_MATCH, HeaderValue::from_str(value).expect("valid header")); + } + let conditions = get_condition_values(&headers, &cred, None, None, None); + assert!( + !conditions.contains_key("if-none-match"), + "claim must not fabricate an absent precondition" + ); + assert!(!conditions.contains_key("if-match"), "claim must not fabricate an absent precondition"); + } + + let mut headers = HeaderMap::new(); + headers.insert(IF_NONE_MATCH, HeaderValue::from_static("*")); + headers.insert(IF_MATCH, HeaderValue::from_static("\"request-etag\"")); + let conditions = get_condition_values(&headers, &cred, None, None, None); + assert_eq!(conditions.get("if-none-match"), Some(&vec!["*".to_string()])); + assert_eq!(conditions.get("if-match"), Some(&vec!["\"request-etag\"".to_string()])); + } + + #[test] + fn test_get_condition_values_preserves_consumed_headers_and_multivalues() { + let cred = create_test_credentials(); + let mut headers = HeaderMap::new(); + headers.insert("x-amz-signature-age", HeaderValue::from_static("300")); + headers.insert("x-amz-object-lock-mode", HeaderValue::from_static("GOVERNANCE")); + headers.insert("x-amz-object-lock-legal-hold", HeaderValue::from_static("ON")); + headers.insert("x-amz-object-lock-retain-until-date", HeaderValue::from_static("2030-01-01T00:00:00Z")); + headers.insert("x-amz-tagging", HeaderValue::from_static("project=blue")); + for name in [ + "x-amz-grant-full-control", + "x-amz-grant-read", + "x-amz-grant-write", + "x-amz-grant-read-acp", + "x-amz-grant-write-acp", + "x-custom-header", + ] { + headers.append(name, HeaderValue::from_static("first")); + headers.append(name, HeaderValue::from_static("second")); + } + headers.append(IF_MATCH, HeaderValue::from_static(" \"first-etag\" ")); + headers.append(IF_MATCH, HeaderValue::from_static("\"second-etag\"")); + + let conditions = get_condition_values(&headers, &cred, None, None, None); + for (name, expected) in [ + ("signatureAge", "300"), + ("object-lock-mode", "GOVERNANCE"), + ("object-lock-legal-hold", "ON"), + ("object-lock-retain-until-date", "2030-01-01T00:00:00Z"), + ("if-match", "\"first-etag\""), + ] { + assert_eq!(conditions.get(name), Some(&vec![expected.to_string()]), "{name}"); + } + for name in [ + "x-amz-signature-age", + "x-amz-object-lock-mode", + "x-amz-object-lock-legal-hold", + "x-amz-object-lock-retain-until-date", + "x-amz-tagging", + ] { + assert!(!conditions.contains_key(name), "consumed header {name} must not be merged again"); + } + for name in [ + "x-amz-grant-full-control", + "x-amz-grant-read", + "x-amz-grant-write", + "x-amz-grant-read-acp", + "x-amz-grant-write-acp", + "x-custom-header", + ] { + assert_eq!(conditions.get(name), Some(&vec!["first".to_string(), "second".to_string()]), "{name}"); + } + } + #[test] fn test_get_condition_values_with_signature_age() { let cred = create_test_credentials(); diff --git a/rustfs/src/config/cli.rs b/rustfs/src/config/cli.rs index 867006bdf..106bca3cb 100644 --- a/rustfs/src/config/cli.rs +++ b/rustfs/src/config/cli.rs @@ -38,6 +38,7 @@ pub(super) const LONG_VERSION: &str = concat!( concat!("build time : ", build::BUILD_TIME, "\n"), concat!("build profile: ", build::BUILD_RUST_CHANNEL, "\n"), concat!("build os : ", build::BUILD_OS, "\n"), + concat!("build target : ", build::BUILD_TARGET, "\n"), concat!("rust version : ", build::RUST_VERSION, "\n"), concat!("rust channel : ", build::RUST_CHANNEL, "\n"), concat!("git branch : ", build::BRANCH, "\n"), diff --git a/rustfs/src/connect/diagnostics/perf_object.rs b/rustfs/src/connect/diagnostics/perf_object.rs index 6487c4d21..09d93a9ed 100644 --- a/rustfs/src/connect/diagnostics/perf_object.rs +++ b/rustfs/src/connect/diagnostics/perf_object.rs @@ -66,6 +66,9 @@ const MAX_OBJECT_RESPONSE_BYTES: usize = 16_384; const CLEANUP_RESERVE_MAX: Duration = Duration::from_millis(250); const CLEANUP_TIMEOUT: Duration = Duration::from_secs(5); const OUTPUT_MODE: u32 = 0o600; +const LOG_COMPONENT_CONNECT: &str = "connect"; +const LOG_SUBSYSTEM_PERFORMANCE_OBJECT: &str = "performance_object"; +const EVENT_OBJECT_CLEANUP_FAILED: &str = "connect_object_cleanup_failed"; static OBJECT_COLLECTOR_ACTIVE: AtomicBool = AtomicBool::new(false); @@ -483,17 +486,61 @@ impl S3ObjectProbe { } async fn cleanup(&self, object_url: Url, bucket_url: Url, deadline: Instant) -> Result<(), ObjectProbeError> { - let object = self.send(Method::DELETE, object_url, Bytes::new(), deadline, None).await?; + let started = Instant::now(); + let object = self + .send(Method::DELETE, object_url, Bytes::new(), deadline, None) + .await + .inspect_err(|reason| Self::log_cleanup_failure("object_delete", None, *reason, started, deadline))?; if !object.status().is_success() && object.status() != StatusCode::NOT_FOUND { + Self::log_cleanup_failure( + "object_delete", + Some(object.status().as_u16()), + ObjectProbeError::CleanupFailed, + started, + deadline, + ); return Err(ObjectProbeError::CleanupFailed); } - let bucket = self.send(Method::DELETE, bucket_url, Bytes::new(), deadline, None).await?; + let bucket = self + .send(Method::DELETE, bucket_url, Bytes::new(), deadline, None) + .await + .inspect_err(|reason| Self::log_cleanup_failure("bucket_delete", None, *reason, started, deadline))?; if !bucket.status().is_success() { + Self::log_cleanup_failure( + "bucket_delete", + Some(bucket.status().as_u16()), + ObjectProbeError::CleanupFailed, + started, + deadline, + ); return Err(ObjectProbeError::CleanupFailed); } Ok(()) } + fn log_cleanup_failure( + stage: &'static str, + http_status: Option, + reason: ObjectProbeError, + started: Instant, + deadline: Instant, + ) { + // Only bounded local classifications are logged, never the signed URL, + // credentials, response body, or a transport error containing them. + tracing::error!( + event = EVENT_OBJECT_CLEANUP_FAILED, + component = LOG_COMPONENT_CONNECT, + subsystem = LOG_SUBSYSTEM_PERFORMANCE_OBJECT, + result = "failed", + stage, + reason = ?reason, + http_status, + elapsed_ms = started.elapsed().as_millis(), + remaining_ms = deadline.saturating_duration_since(Instant::now()).as_millis(), + "Object diagnostic cleanup failed" + ); + } + async fn send( &self, method: Method, diff --git a/rustfs/src/connect/diagnostics/trace_runtime.rs b/rustfs/src/connect/diagnostics/trace_runtime.rs index 04aa165ff..76360179a 100644 --- a/rustfs/src/connect/diagnostics/trace_runtime.rs +++ b/rustfs/src/connect/diagnostics/trace_runtime.rs @@ -1911,6 +1911,14 @@ mod tests { .expect("telemetry subscriber"); } + async fn prepare_profile_test_provenance() { + // Cold hashing of the test executable is fixture setup, not IPC response time. + let provenance = super::super::job_delivery::executable_provenance() + .await + .expect("test executable provenance should be available"); + assert!(provenance.is_valid(), "profile capture requires valid executable provenance"); + } + #[tokio::test] #[serial] async fn local_runtime_captures_server_process_events() { @@ -2869,12 +2877,78 @@ mod tests { assert!(!state.path().join("identity").exists()); } + #[cfg(target_os = "linux")] + const NATIVE_THREADS_PREPARED_FILE: &str = "native-threads-provenance-ready"; + + #[cfg(target_os = "linux")] + const NATIVE_THREADS_FIXTURE_STARTED_FILE: &str = "native-threads-fixture-started"; + + #[cfg(target_os = "linux")] + fn spawn_native_threads_test_child(state: &std::path::Path, mode: &str) -> std::io::Result { + tokio::process::Command::new(std::env::current_exe()?) + .args([ + "connect::diagnostics::trace_runtime::tests::local_native_threads_service_child", + "--exact", + "--nocapture", + ]) + .env("RUSTFS_TEST_NATIVE_THREADS_STATE", state) + .env("RUSTFS_TEST_NATIVE_THREADS_FIXTURE_MODE", mode) + .stdin(std::process::Stdio::piped()) + .kill_on_drop(true) + .spawn() + } + + #[cfg(target_os = "linux")] + async fn wait_for_profile_child_marker( + child: &mut tokio::process::Child, + path: &std::path::Path, + budget: Duration, + ) -> std::io::Result<()> { + tokio::time::timeout(budget, async { + loop { + if let Some(status) = child.try_wait()? { + return Err(std::io::Error::new( + std::io::ErrorKind::UnexpectedEof, + format!("profile fixture exited before readiness: {status}"), + )); + } + if path.try_exists()? { + return Ok(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .map_err(|_| std::io::Error::new(std::io::ErrorKind::TimedOut, "profile fixture readiness timed out"))? + } + + #[cfg(target_os = "linux")] + async fn stop_profile_test_child( + child: &mut tokio::process::Child, + budget: Duration, + ) -> std::io::Result { + drop(child.stdin.take()); + match tokio::time::timeout(budget, child.wait()).await { + Ok(status) => status, + Err(_) => { + child.kill().await?; + child.wait().await + } + } + } + #[cfg(target_os = "linux")] #[test] fn local_native_threads_service_child() { let Some(state) = std::env::var_os("RUSTFS_TEST_NATIVE_THREADS_STATE") else { return; }; + let mode = std::env::var("RUSTFS_TEST_NATIVE_THREADS_FIXTURE_MODE").unwrap_or_else(|_| "capture".to_owned()); + match mode.as_str() { + "exit-before-ready" => return, + "capture" | "wait-before-ready" | "ignore-stop" => {} + _ => panic!("unknown native threads fixture mode"), + } let stop = CancellationToken::new(); let input_stop = stop.clone(); std::thread::spawn(move || { @@ -2882,6 +2956,14 @@ mod tests { let _ = std::io::stdin().read(&mut [0u8]); input_stop.cancel(); }); + if mode != "capture" { + std::fs::write(std::path::Path::new(&state).join(NATIVE_THREADS_FIXTURE_STARTED_FILE), b"started") + .expect("fixture started marker"); + while mode == "ignore-stop" || !stop.is_cancelled() { + std::thread::park_timeout(Duration::from_millis(10)); + } + return; + } let sleepers: Vec<_> = (0..16) .map(|_| { let stop = stop.clone(); @@ -2893,6 +2975,9 @@ mod tests { }) .collect(); tokio::runtime::Runtime::new().unwrap().block_on(async { + prepare_profile_test_provenance().await; + std::fs::write(std::path::Path::new(&state).join(NATIVE_THREADS_PREPARED_FILE), b"ready") + .expect("profile provenance ready marker"); let runtime = spawn_local_trace_capture_runtime(std::path::Path::new(&state), &stop).unwrap(); stop.cancelled().await; runtime.shutdown().await; @@ -2909,31 +2994,26 @@ mod tests { use std::io::Read as _; let state = tempfile::tempdir().unwrap(); std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o700)).unwrap(); - let (request, key) = runtime_request(state.path()); - let mut child = std::process::Command::new(std::env::current_exe().unwrap()) - .args([ - "connect::diagnostics::trace_runtime::tests::local_native_threads_service_child", - "--exact", - "--nocapture", - ]) - .env("RUSTFS_TEST_NATIVE_THREADS_STATE", state.path()) - .stdin(std::process::Stdio::piped()) - .spawn() - .unwrap(); - let ready = tokio::time::timeout(Duration::from_secs(5), async { - while !state.path().join(super::SOCKET_FILE).exists() { - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) + let mut child = spawn_native_threads_test_child(state.path(), "capture").unwrap(); + let capture = async { + // This setup budget does not extend the socket or production request deadlines. + wait_for_profile_child_marker(&mut child, &state.path().join(NATIVE_THREADS_PREPARED_FILE), Duration::from_secs(120)) + .await + .map_err(LocalTraceCaptureError::Io)?; + wait_for_profile_child_marker(&mut child, &state.path().join(super::SOCKET_FILE), Duration::from_secs(5)) + .await + .map_err(LocalTraceCaptureError::Io)?; + let (request, key) = runtime_request(state.path()); + let export = + super::request_local_native_threads_profile(state.path(), request.clone(), &CancellationToken::new()).await?; + Ok::<_, LocalTraceCaptureError>((export, request, key)) + } .await; - let export = if ready.is_ok() { - super::request_local_native_threads_profile(state.path(), request.clone(), &CancellationToken::new()).await - } else { - Err(LocalTraceCaptureError::Protocol) - }; - drop(child.stdin.take()); - assert!(child.wait().unwrap().success()); - let export = export.unwrap(); + let status = stop_profile_test_child(&mut child, Duration::from_secs(10)) + .await + .expect("native threads fixture must be reaped"); + assert!(status.success(), "fixture exit: {status}; capture error: {:?}", capture.as_ref().err()); + let (export, request, key) = capture.unwrap(); let mut zip = zip::ZipArchive::new(std::io::Cursor::new(&export.archive_bytes)).unwrap(); let result: serde_json::Value = serde_json::from_reader(zip.by_name("result.json").unwrap()).unwrap(); assert_eq!(result["toolId"], "profile.threads"); @@ -2955,6 +3035,63 @@ mod tests { assert!(!state.path().join("identity").exists()); } + #[cfg(target_os = "linux")] + #[tokio::test] + async fn local_native_threads_preparation_detects_early_child_exit() { + let state = tempfile::tempdir().unwrap(); + let mut child = spawn_native_threads_test_child(state.path(), "exit-before-ready").unwrap(); + let ready = + wait_for_profile_child_marker(&mut child, &state.path().join(NATIVE_THREADS_PREPARED_FILE), Duration::from_secs(5)) + .await; + let status = stop_profile_test_child(&mut child, Duration::from_secs(1)).await.unwrap(); + assert_eq!(ready.unwrap_err().kind(), std::io::ErrorKind::UnexpectedEof); + assert!(status.success()); + assert!(child.try_wait().unwrap().is_some()); + } + + #[cfg(target_os = "linux")] + #[tokio::test] + async fn local_native_threads_preparation_timeout_reaps_child() { + let state = tempfile::tempdir().unwrap(); + let mut child = spawn_native_threads_test_child(state.path(), "wait-before-ready").unwrap(); + let started = wait_for_profile_child_marker( + &mut child, + &state.path().join(NATIVE_THREADS_FIXTURE_STARTED_FILE), + Duration::from_secs(5), + ) + .await; + let ready = wait_for_profile_child_marker( + &mut child, + &state.path().join(NATIVE_THREADS_PREPARED_FILE), + Duration::from_millis(20), + ) + .await; + let status = stop_profile_test_child(&mut child, Duration::from_secs(1)).await.unwrap(); + started.unwrap(); + assert_eq!(ready.unwrap_err().kind(), std::io::ErrorKind::TimedOut); + assert!(status.success()); + assert!(child.try_wait().unwrap().is_some()); + } + + #[cfg(target_os = "linux")] + #[tokio::test] + async fn local_native_threads_cleanup_kills_and_reaps_unresponsive_child() { + use std::os::unix::process::ExitStatusExt as _; + + let state = tempfile::tempdir().unwrap(); + let mut child = spawn_native_threads_test_child(state.path(), "ignore-stop").unwrap(); + let started = wait_for_profile_child_marker( + &mut child, + &state.path().join(NATIVE_THREADS_FIXTURE_STARTED_FILE), + Duration::from_secs(5), + ) + .await; + let status = stop_profile_test_child(&mut child, Duration::from_millis(20)).await.unwrap(); + started.unwrap(); + assert_eq!(status.signal(), Some(9)); + assert!(child.try_wait().unwrap().is_some()); + } + #[tokio::test] async fn local_native_threads_rejects_consent_scope_and_missing_identity() { let state = tempfile::tempdir().unwrap(); @@ -2996,14 +3133,14 @@ mod tests { use std::io::Read as _; let state = tempfile::tempdir().unwrap(); std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o700)).unwrap(); + prepare_profile_test_provenance().await; let (mut request, key) = runtime_request(state.path()); request.capability = super::super::profile_cpu::MEMORY_PROFILE_CAPABILITY.to_owned(); let runtime = spawn_local_trace_capture_runtime(state.path(), &CancellationToken::new()).unwrap(); - let export = super::request_local_runtime_profile(state.path(), request.clone(), &CancellationToken::new()) - .await - .unwrap(); - assert_eq!(export.tool, super::ProfileTool::Memory); + let export = super::request_local_runtime_profile(state.path(), request.clone(), &CancellationToken::new()).await; runtime.shutdown().await; + let export = export.unwrap(); + assert_eq!(export.tool, super::ProfileTool::Memory); let mut zip = zip::ZipArchive::new(std::io::Cursor::new(&export.archive_bytes)).unwrap(); let result: serde_json::Value = serde_json::from_reader(zip.by_name("result.json").unwrap()).unwrap(); assert_eq!(result["toolId"], "profile.memory"); @@ -3029,6 +3166,7 @@ mod tests { use std::io::Read as _; let state = tempfile::tempdir().unwrap(); std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o700)).unwrap(); + prepare_profile_test_provenance().await; let (mut request, _key) = runtime_request(state.path()); request.capability = super::super::profile_cpu::CPU_PROFILE_CAPABILITY.to_owned(); request.duration_millis = 1_000; @@ -3212,10 +3350,7 @@ mod tests { async fn local_runtime_profile_cancellation_waits_for_lease_release() { let state = tempfile::tempdir().unwrap(); std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o700)).unwrap(); - let provenance = super::super::job_delivery::executable_provenance() - .await - .expect("test executable provenance should be available"); - assert!(provenance.is_valid(), "profile capture requires valid executable provenance"); + prepare_profile_test_provenance().await; let (mut request, _) = runtime_request(state.path()); request.duration_millis = 5_000; let server_state_root = state.path().to_path_buf(); diff --git a/rustfs/src/connect/report_bundle.rs b/rustfs/src/connect/report_bundle.rs index 890629ac2..454626261 100644 --- a/rustfs/src/connect/report_bundle.rs +++ b/rustfs/src/connect/report_bundle.rs @@ -76,7 +76,11 @@ pub(crate) fn upload_source( Err(_) => return original_source(path, generated_bundle_uid), }; let names = (0..archive.len()) - .map(|index| archive.by_index(index).map(|entry| entry.name().to_owned())) + .map(|index| { + archive + .by_index(index) + .and_then(|entry| entry.name().map(|name| name.into_owned())) + }) .collect::, _>>()?; if names diff --git a/rustfs/src/error.rs b/rustfs/src/error.rs index 6f7242b91..57afc8208 100644 --- a/rustfs/src/error.rs +++ b/rustfs/src/error.rs @@ -23,6 +23,13 @@ const MAX_VERSIONS_EXCEEDED_CODE: &str = "MaxVersionsExceeded"; const MAX_VERSIONS_EXCEEDED_MESSAGE: &str = "You've exceeded the limit on the number of versions you can create on this object"; const SLOW_DOWN_READ_CODE: &str = "SlowDownRead"; const SLOW_DOWN_READ_MESSAGE: &str = "Resource requested is unreadable, please reduce your request rate"; +/// Public message for `StorageError::UnsupportedDisk` on a request path. The +/// only request-time producer is the delete-transaction protocol when a +/// participant answers Unimplemented (a peer still on a release without the +/// recovery RPC, or a protocol-version mismatch); the variant's Display text +/// ("drive does not support O_DIRECT") describes the startup drive probe and +/// would send an operator mid-upgrade after the wrong cause. +const UNSUPPORTED_PARTICIPANT_MESSAGE: &str = "A storage participant does not support this operation's protocol version. If the cluster is being upgraded, finish upgrading every node and retry."; /// S3 error code for a request that names a KMS key the KMS does not hold. pub const KMS_KEY_NOT_FOUND_ERROR_CODE: &str = "KMS.NotFoundException"; @@ -173,6 +180,40 @@ pub(crate) fn slow_down_read_api_error(err: StorageError) -> ApiError { } } +/// Normalize a read-quorum failure that escaped through a PutObject boundary. +/// +/// Storage-level read quorum failures keep their `SlowDownRead` code for GET +/// compatibility. A write request that encounters the same failure must expose +/// the write-side retry contract instead, while retaining the typed source for +/// diagnostics and audit consumers. +pub(crate) fn normalize_put_read_quorum_error(mut err: S3Error) -> S3Error { + let is_slow_down_read = matches!( + err.code(), + S3ErrorCode::Custom(code) if &**code == SLOW_DOWN_READ_CODE + ); + if !is_slow_down_read { + return err; + } + let has_read_quorum_source = err.source().is_some_and(|source| { + error_chain_find(source, |error| { + matches!( + error.downcast_ref::(), + Some(StorageError::ErasureReadQuorum | StorageError::InsufficientReadQuorum(_, _)) + ) + .then_some(()) + }) + .is_some() + }); + + if has_read_quorum_source { + err.set_code(S3ErrorCode::ServiceUnavailable); + err.set_message(ApiError::error_code_to_message(&S3ErrorCode::ServiceUnavailable)); + err.set_status_code(StatusCode::SERVICE_UNAVAILABLE); + } + + err +} + /// Marks a request body that exceeded a presigned upload size capability. /// /// This marker must survive the body-reader and storage layers so the client @@ -565,7 +606,21 @@ fn error_chain_has_hyper_body_eof(err: &(dyn std::error::Error + 'static)) -> bo impl From for S3Error { fn from(err: ApiError) -> Self { let status = custom_error_status(&err.code); + // A typed lock acquisition timeout is transient contention. Keep its + // existing S3 code while advising a minimum delay before another attempt. + let lock_timeout = err.code == S3ErrorCode::ServiceUnavailable + && err.source.as_deref().is_some_and(|source| { + error_chain_find(source, |error| { + matches!(error.downcast_ref::(), Some(LockError::Timeout { .. })).then_some(()) + }) + .is_some() + }); let mut s3e = S3Error::with_message(err.code, err.message); + if lock_timeout { + let mut headers = http::HeaderMap::new(); + headers.insert(http::header::RETRY_AFTER, http::HeaderValue::from_static("1")); + s3e.set_headers(headers); + } if let Some(status) = status { s3e.set_status_code(status); } @@ -576,6 +631,26 @@ impl From for S3Error { } } +impl From for ApiError { + fn from(err: crate::table_catalog::TableCatalogStoreError) -> Self { + if let crate::table_catalog::TableCatalogStoreError::LockUnavailable { source, .. } = err { + // Use the storage boundary once so typed lock failures share its + // bounded metrics, public message, and retry advice. + return Self::from(StorageError::Lock(*source)); + } + let code = if err.is_unavailable() { + S3ErrorCode::ServiceUnavailable + } else { + S3ErrorCode::InternalError + }; + Self { + message: Self::error_code_to_message(&code), + code, + source: Some(Box::new(err)), + } + } +} + impl From for ApiError { fn from(err: StorageError) -> Self { record_s3_lock_failure(&err); @@ -690,11 +765,15 @@ impl From for ApiError { StorageError::BucketNameInvalid(_) => S3ErrorCode::InvalidBucketName, StorageError::ObjectNameInvalid(_, _) => S3ErrorCode::InvalidArgument, StorageError::BucketExists(_) => S3ErrorCode::BucketAlreadyOwnedByYou, - StorageError::StorageFull => S3ErrorCode::ServiceUnavailable, + StorageError::DiskFull | StorageError::StorageFull => S3ErrorCode::ServiceUnavailable, + // Retryable on an upgraded node (or once the cluster upgrade + // completes); see UNSUPPORTED_PARTICIPANT_MESSAGE. + StorageError::UnsupportedDisk => S3ErrorCode::ServiceUnavailable, StorageError::SlowDown => S3ErrorCode::SlowDown, StorageError::FaultyDisk | StorageError::FaultyRemoteDisk | StorageError::DiskNotFound + | StorageError::Timeout | StorageError::TooManyOpenFiles => S3ErrorCode::ServiceUnavailable, StorageError::ErasureReadQuorum | StorageError::InsufficientReadQuorum(_, _) => { S3ErrorCode::Custom(SLOW_DOWN_READ_CODE.into()) @@ -726,6 +805,8 @@ impl From for ApiError { StorageError::EntityTooSmall(_, _, _) => S3ErrorCode::EntityTooSmall, StorageError::EntityTooLarge(_, _) => S3ErrorCode::EntityTooLarge, StorageError::PreconditionFailed => S3ErrorCode::PreconditionFailed, + StorageError::ConditionalDeleteConflict => S3ErrorCode::OperationAborted, + StorageError::GuardedDeleteUnsupported => S3ErrorCode::NotImplemented, StorageError::NotModified => S3ErrorCode::NotModified, StorageError::InvalidRangeSpec(_) => S3ErrorCode::InvalidRange, _ => S3ErrorCode::InternalError, @@ -733,6 +814,8 @@ impl From for ApiError { let message = if matches!(&err, StorageError::QuotaExceeded { .. }) { err.to_string() + } else if matches!(&err, StorageError::UnsupportedDisk) { + UNSUPPORTED_PARTICIPANT_MESSAGE.to_string() } else if matches!(&err, StorageError::MaxVersionsExceeded) || (code == S3ErrorCode::InternalError && matches!(&err, StorageError::Io(_))) { @@ -885,6 +968,137 @@ mod tests { use s3s::{S3Error, S3ErrorCode}; use std::io::{Error as IoError, ErrorKind}; + #[tokio::test] + async fn lock_timeout_response_advises_retry_without_exposing_resource() { + use http_body_util::BodyExt as _; + + let storage_error = + StorageError::Lock(LockError::timeout("private-bucket/private-key@latest", std::time::Duration::from_secs(5))); + let error = S3Error::from(ApiError::from(storage_error)); + assert_eq!(error.code(), &S3ErrorCode::ServiceUnavailable); + let source = error.source().expect("retain the typed storage cause"); + assert!(source.downcast_ref::().is_some()); + assert!(error_chain_has_type::(source)); + let response = error.to_http_response().expect("serialize lock timeout response"); + assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE); + assert_eq!(response.headers()[http::header::RETRY_AFTER], "1"); + let body = response + .into_body() + .collect() + .await + .expect("read timeout error XML") + .to_bytes(); + let body = String::from_utf8(body.to_vec()).expect("S3 XML is UTF-8"); + assert!(body.contains("ServiceUnavailable")); + assert!(!body.contains("private-bucket")); + assert!(!body.contains("private-key")); + } + + #[tokio::test] + async fn unsupported_participant_is_retryable_with_an_actionable_message() { + use http_body_util::BodyExt as _; + + let error = S3Error::from(ApiError::from(StorageError::UnsupportedDisk)); + assert_eq!(error.code(), &S3ErrorCode::ServiceUnavailable); + assert_eq!(error.message(), Some(UNSUPPORTED_PARTICIPANT_MESSAGE)); + assert!(matches!( + error.source().and_then(|source| source.downcast_ref::()), + Some(StorageError::UnsupportedDisk) + )); + + let response = error + .to_http_response() + .expect("unsupported participant error must serialize"); + assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE); + assert!( + !response.headers().contains_key(http::header::RETRY_AFTER), + "retrying the same node does not help until it or its peers are upgraded" + ); + let body = response + .into_body() + .collect() + .await + .expect("error XML must be readable") + .to_bytes(); + let xml = std::str::from_utf8(&body).expect("error XML must be UTF-8"); + assert!(xml.contains("ServiceUnavailable")); + assert!( + !xml.contains("O_DIRECT"), + "the drive-probe Display text must not leak as the public message" + ); + } + + #[tokio::test] + async fn capacity_failures_share_service_unavailable_without_lock_retry_advice() { + use crate::storage_api::error::DiskError; + use http_body_util::BodyExt as _; + + for failure in [ + StorageError::DiskFull, + StorageError::StorageFull, + StorageError::from(DiskError::DiskFull), + ] { + let expected_code = failure.code(); + let error = S3Error::from(ApiError::from(failure)); + assert_eq!(error.code(), &S3ErrorCode::ServiceUnavailable); + assert_eq!(error.message(), Some("The service is unavailable. Please retry.")); + let source = error + .source() + .and_then(|source| source.downcast_ref::()) + .expect("retain the original capacity failure"); + assert_eq!(source.code(), expected_code); + let response = error.to_http_response().expect("serialize capacity refusal"); + assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE); + assert!(!response.headers().contains_key(http::header::RETRY_AFTER)); + let body = response.into_body().collect().await.expect("read S3 error XML").to_bytes(); + let xml = std::str::from_utf8(&body).expect("S3 XML is UTF-8"); + assert!(xml.contains("ServiceUnavailable")); + assert!(!xml.contains("InternalError")); + } + } + + #[test] + fn lock_timeout_response_preserves_wrapped_source_and_requires_503() { + for code in [S3ErrorCode::ServiceUnavailable, S3ErrorCode::RequestTimeout] { + let should_advise_retry = code == S3ErrorCode::ServiceUnavailable; + let error = S3Error::from(ApiError { + message: ApiError::error_code_to_message(&code), + code: code.clone(), + source: Some(Box::new(IoError::other(LockError::timeout( + "bucket/object@latest", + std::time::Duration::from_secs(5), + )))), + }); + assert_eq!(error.code(), &code); + let source = error.source().expect("preserve wrapped lock timeout"); + assert!(error_chain_has_type::(source)); + let response = error.to_http_response().expect("serialize wrapped timeout"); + assert_eq!(response.headers().contains_key(http::header::RETRY_AFTER), should_advise_retry); + } + } + + #[test] + fn unrelated_unavailable_responses_do_not_inherit_lock_retry_advice() { + for failure in [ + StorageError::Timeout, + StorageError::DiskNotFound, + StorageError::FileCorrupt, + StorageError::NamespaceLockQuorumUnavailable { + mode: "read", + bucket: "bucket".to_owned(), + object: "object".to_owned(), + required: 2, + achieved: 1, + }, + StorageError::Io(IoError::other("Lock acquisition timeout for bucket/object after 5s")), + StorageError::SlowDown, + ] { + let error = S3Error::from(ApiError::from(failure)); + let response = error.to_http_response().expect("serialize unrelated storage failure"); + assert!(!response.headers().contains_key(http::header::RETRY_AFTER)); + } + } + #[test] fn lock_503_mapping_records_operation_and_resource_class_without_resource_name() { let recorder = DebuggingRecorder::new(); @@ -918,6 +1132,42 @@ mod tests { assert!(!labels.values().any(|value| value.contains("durable-dirty-producer-replay"))); } + #[test] + fn catalog_lock_failure_uses_canonical_metrics_and_retry_advice_once() { + let recorder = DebuggingRecorder::new(); + let error = with_local_recorder(&recorder, || { + let mut request = S3HttpRequestGuard::new("PUT"); + request.in_scope(|| { + rustfs_io_metrics::record_s3_op(S3Operation::PutObject); + S3Error::from(ApiError::from(crate::table_catalog::TableCatalogStoreError::LockUnavailable { + message: "private catalog operation".to_owned(), + source: Box::new(LockError::timeout( + "private-bucket/.rustfs-table/warehouses/default/publication.lock@latest", + std::time::Duration::from_secs(5), + )), + })) + }) + }); + let response = error.to_http_response().expect("serialize catalog lock timeout"); + assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE); + assert_eq!(response.headers()[http::header::RETRY_AFTER], "1"); + let metrics = recorder.snapshotter().snapshot().into_vec(); + let (key, _, _, value) = metrics + .iter() + .find(|(key, _, _, _)| key.key().name() == "rustfs_s3_lock_failures_total") + .expect("catalog lock failures should use the canonical counter"); + assert!(matches!(value, DebugValue::Counter(1))); + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + assert_eq!(labels["op"], S3Operation::PutObject.as_str()); + assert_eq!(labels["reason"], "timeout"); + assert_eq!(labels["resource_class"], "object_namespace"); + assert!(!labels.values().any(|value| value.contains("private"))); + } + #[test] fn multipart_lock_resources_have_a_distinct_bounded_class() { assert_eq!(lock_resource_class(Some(".rustfs.sys/multipart/upload-1")), "multipart_metadata"); @@ -1601,6 +1851,90 @@ mod tests { assert_eq!(S3Error::from(api_error).status_code(), Some(StatusCode::SERVICE_UNAVAILABLE)); } + #[test] + fn put_read_quorum_error_preserves_context_and_is_idempotent() { + for source in [ + StorageError::ErasureReadQuorum, + StorageError::InsufficientReadQuorum("bucket".into(), "object".into()), + ] { + for nested in [false, true] { + let mut err: S3Error = ApiError::from(source.clone()).into(); + if nested { + err.set_source(Box::new(IoError::other(source.clone()))); + } + err.set_request_id("put-quorum-request"); + let mut headers = http::HeaderMap::new(); + headers.insert("retry-after", http::HeaderValue::from_static("2")); + err.set_headers(headers); + + let err = normalize_put_read_quorum_error(normalize_put_read_quorum_error(err)); + assert_eq!(err.code(), &S3ErrorCode::ServiceUnavailable); + assert_eq!(err.status_code(), Some(StatusCode::SERVICE_UNAVAILABLE)); + assert_eq!(err.message(), Some("The service is unavailable. Please retry.")); + assert_eq!(err.request_id(), Some("put-quorum-request")); + assert_eq!(err.headers().expect("retry headers must survive")["retry-after"], "2"); + let retained = error_chain_find(err.source().expect("typed source must survive"), |error| { + error.downcast_ref::().cloned() + }); + assert_eq!(retained, Some(source.clone())); + } + } + } + + #[test] + fn put_read_quorum_error_preserves_other_error_contracts() { + for code in [ + S3ErrorCode::SlowDown, + S3ErrorCode::AccessDenied, + S3ErrorCode::InvalidRequest, + S3ErrorCode::PreconditionFailed, + S3ErrorCode::BadDigest, + S3ErrorCode::IncompleteBody, + S3ErrorCode::NoSuchKey, + S3ErrorCode::NoSuchBucket, + S3ErrorCode::InternalError, + S3ErrorCode::ServiceUnavailable, + ] { + let mut err = S3Error::with_message(code.clone(), "original message"); + err.set_source(Box::new(StorageError::ErasureReadQuorum)); + let original_status = err.status_code(); + let err = normalize_put_read_quorum_error(err); + assert_eq!(err.code(), &code); + assert_eq!(err.message(), Some("original message")); + assert_eq!(err.status_code(), original_status); + } + for source in [ + None, + Some(StorageError::ErasureWriteQuorum), + Some(StorageError::PartMissingOrCorrupt), + ] { + let mut err = S3Error::with_message(S3ErrorCode::Custom(SLOW_DOWN_READ_CODE.into()), "original message"); + if let Some(source) = source { + err.set_source(Box::new(source)); + } + let err = normalize_put_read_quorum_error(err); + assert_eq!(err.code(), &S3ErrorCode::Custom(SLOW_DOWN_READ_CODE.into())); + assert_eq!(err.message(), Some("original message")); + } + } + + #[test] + fn conditional_delete_storage_errors_preserve_api_contract() { + for (source, expected) in [ + (StorageError::ConditionalDeleteConflict, S3ErrorCode::OperationAborted), + (StorageError::GuardedDeleteUnsupported, S3ErrorCode::NotImplemented), + ] { + let error = ApiError::from(source.clone()); + assert_eq!(error.code, expected); + assert!( + error + .source + .as_ref() + .is_some_and(|value| value.downcast_ref::() == Some(&source)) + ); + } + } + #[test] fn test_unknown_authoritative_quota_usage_maps_to_retryable_error() { let api_error = ApiError::from(QuotaError::UsageUnavailable { @@ -1668,7 +2002,9 @@ mod tests { S3ErrorCode::InvalidArgument, ), (StorageError::BucketExists("test".into()), S3ErrorCode::BucketAlreadyOwnedByYou), + (StorageError::DiskFull, S3ErrorCode::ServiceUnavailable), (StorageError::StorageFull, S3ErrorCode::ServiceUnavailable), + (StorageError::UnsupportedDisk, S3ErrorCode::ServiceUnavailable), (StorageError::SlowDown, S3ErrorCode::SlowDown), (StorageError::FaultyDisk, S3ErrorCode::ServiceUnavailable), (StorageError::FaultyRemoteDisk, S3ErrorCode::ServiceUnavailable), @@ -1719,6 +2055,37 @@ mod tests { } } + #[test] + fn disk_quorum_timeouts_map_to_retryable_service_unavailable() { + use crate::storage_api::error::{DiskError, OBJECT_OP_IGNORED_ERRS, reduce_read_quorum_errs, reduce_write_quorum_errs}; + + let errors = [ + Some(DiskError::Timeout), + Some(DiskError::Timeout), + Some(DiskError::Timeout), + None, + ]; + for reduced in [ + reduce_read_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 2), + reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 3), + ] { + let reduced = reduced.expect("quorum of timeouts must retain its typed cause"); + assert_eq!(reduced, DiskError::Timeout); + let error = ApiError::from(StorageError::from(reduced)); + assert_eq!(error.code, S3ErrorCode::ServiceUnavailable); + let response = S3Error::from(error); + assert_eq!(response.status_code(), Some(http::StatusCode::SERVICE_UNAVAILABLE)); + } + let degraded = [None, None, None, Some(DiskError::Timeout)]; + assert!(reduce_write_quorum_errs(°raded, OBJECT_OP_IGNORED_ERRS, 3).is_none()); + for failure in [DiskError::FileCorrupt, DiskError::Io(IoError::other("unknown disk failure"))] { + let errors = [Some(failure.clone()), Some(failure.clone()), Some(failure), None]; + let reduced = + reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 3).expect("permanent failures must remain errors"); + assert_eq!(ApiError::from(StorageError::from(reduced)).code, S3ErrorCode::InternalError); + } + } + #[test] fn test_api_error_from_unexpected_eof_maps_to_incomplete_body() { let io_error = IoError::new(ErrorKind::UnexpectedEof, rustfs_rio::IncompleteBody { remaining: 7 }); diff --git a/rustfs/src/protocols/client.rs b/rustfs/src/protocols/client.rs index dfea6d36a..9b982356d 100644 --- a/rustfs/src/protocols/client.rs +++ b/rustfs/src/protocols/client.rs @@ -18,7 +18,7 @@ use crate::runtime_sources::current_object_store_handle; use crate::shared_types::RemoteAddr; #[cfg(feature = "webdav")] use crate::storage_api::protocols::client::capacity; -use crate::storage_api::protocols::client::{FS, ReqInfo, RequestContext, authorize_request}; +use crate::storage_api::protocols::client::{FS, MultipartStagingOperation, ReqInfo, RequestContext, authorize_request}; use http::{HeaderMap, Method}; use percent_encoding::{AsciiSet, CONTROLS, utf8_percent_encode}; use rustfs_credentials; @@ -286,6 +286,10 @@ impl ProtocolStorageClient { client_info.forwarded_proto = Some(if session.secure_transport { "https" } else { "http" }.to_owned()); request.extensions.insert(client_info); request.headers = session.request_headers.clone(); + // Preserve staging semantics before distinct operations collapse to PutObject. + if matches!(action, S3Action::CreateMultipartUpload | S3Action::UploadPart) { + request.extensions.insert(MultipartStagingOperation); + } let policy_action = if matches!(action, S3Action::HeadBucket) { S3Action::ListBucket.into() } else { diff --git a/rustfs/src/protocols/policy_test.rs b/rustfs/src/protocols/policy_test.rs index 7d4bdcb79..318e25436 100644 --- a/rustfs/src/protocols/policy_test.rs +++ b/rustfs/src/protocols/policy_test.rs @@ -561,3 +561,121 @@ fn protocol_policy_denials_and_request_conditions() { } }); } + +#[test] +#[serial_test::serial] +fn protocol_object_creation_operation_preserves_multipart_staging() { + crate::app::gating_test_env::run_large_stack_test("protocol-object-creation", || async { + let root = Credentials { + access_key: "creation-policy-root".into(), + secret_key: "creation-policy-root-secret".into(), + status: "on".into(), + ..Default::default() + }; + let (_temp, context, iam) = protocol_test_context(root).await; + let store = context.object_store(); + let slot = ServerContextSlot::new(); + assert!(slot.install(context)); + let backend = crate::init::protocol_storage_client_for_server(slot); + let bucket = "protocol-object-creation"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + iam.create_user( + "creation-writer", + &rustfs_madmin::AddOrUpdateUserReq { + secret_key: "creation-writer-secret".into(), + policy: None, + status: rustfs_madmin::AccountStatus::Enabled, + }, + ) + .await + .expect("create protocol writer"); + let identity = iam + .check_key("creation-writer") + .await + .expect("load writer") + .0 + .expect("writer exists"); + let sessions: Vec<_> = [Protocol::Ftps, Protocol::Sftp] + .into_iter() + .map(|protocol| { + SessionContext::new( + ProtocolPrincipal::new(Arc::new(identity.clone())), + protocol, + "127.0.0.1".parse().expect("loopback"), + ) + }) + .collect(); + let resource = format!("arn:aws:s3:::{bucket}/*"); + let allow = json!({"Effect":"Allow", "Action":"s3:PutObject", "Resource": resource}); + iam.set_policy( + "creation-policy", + decode_policy(json!({"Version":"2012-10-17","Statement":[allow.clone()]})).expect("parse initial allow"), + ) + .await + .expect("save initial allow"); + iam.policy_db_set("creation-writer", UserType::Reg, false, "creation-policy") + .await + .expect("attach writer policy"); + for source in ["IAM", "bucket"] { + for denied_creation in [false, true] { + let deny = json!({"Effect":"Deny", "Action":"s3:PutObject", "Resource": resource, + "Condition":{"Bool":{"s3:ObjectCreationOperation":denied_creation.to_string()}}}); + let mut statements = vec![allow.clone()]; + let mut bucket_statements = Vec::new(); + if source == "IAM" { + statements.push(deny); + } else { + let mut deny = deny; + deny["Principal"] = json!("*"); + bucket_statements.push(deny); + } + iam.set_policy( + "creation-policy", + decode_policy(json!({"Version":"2012-10-17","Statement":statements})).expect("parse IAM policy"), + ) + .await + .expect("save IAM policy"); + store + .update_bucket_metadata_config( + bucket, + "policy.json", + serde_json::to_vec(&json!({"Version":"2012-10-17","Statement":bucket_statements})) + .expect("encode bucket policy"), + ) + .await + .expect("save bucket policy"); + for session in &sessions { + for (action, creates_object) in [ + (S3Action::CreateMultipartUpload, false), + (S3Action::UploadPart, false), + (S3Action::CompleteMultipartUpload, true), + (S3Action::PutObject, true), + (S3Action::CopyObject, true), + ] { + if session.protocol == Protocol::Ftps && action == S3Action::CopyObject { + continue; + } + let result = backend.authorize_operation(session, &action, bucket, Some("key")).await; + if creates_object == denied_creation { + assert!( + matches!(result, Err(AuthorizationError::AccessDenied)), + "{source} deny={denied_creation}, {:?}, {action:?}: {result:?}", + session.protocol + ); + } else { + result.unwrap_or_else(|error| { + panic!( + "{source} deny={denied_creation}, {:?}, {action:?} should allow: {error}", + session.protocol + ) + }); + } + } + } + } + } + }); +} diff --git a/rustfs/src/runtime_sources.rs b/rustfs/src/runtime_sources.rs index fc211890b..5fdf451ab 100644 --- a/rustfs/src/runtime_sources.rs +++ b/rustfs/src/runtime_sources.rs @@ -14,7 +14,7 @@ use crate::app::context; use rustfs_config::ENV_RUSTFS_BROWSER_REDIRECT_URL; -use rustfs_iam::federation::{FederatedIdentityService, oidc::OidcConfigQuery}; +use rustfs_iam::{federation::FederatedIdentityService, oidc::OidcConfigQuery}; use std::sync::Arc; use tracing::warn; @@ -104,11 +104,8 @@ pub(crate) fn current_app_context() -> Option> { mod tests { use super::*; use rustfs_iam::{ - federation::{ - CoreFederatedAuthorizationMapper, - oidc::{OidcConfigQuery, StandardOidcAdapter}, - }, - oidc::{OidcSys, make_test_sys, test_config}, + federation::CoreFederatedAuthorizationMapper, + oidc::{OidcConfigQuery, OidcSys, StandardOidcAdapter, make_test_sys, test_config}, }; use std::{ io::{self, Write}, diff --git a/rustfs/src/server/http.rs b/rustfs/src/server/http.rs index 72303bfdc..f54aa5187 100644 --- a/rustfs/src/server/http.rs +++ b/rustfs/src/server/http.rs @@ -32,12 +32,12 @@ use crate::server::{ ssec_transport::SsecTransportLayer, strip_valid_port_suffix, tls_material::{ - TlsAcceptFailure, TlsAcceptorHolder, TlsHandshakeFailureKind, accept_tls_with_deadline, build_acceptor_from_loaded, + PreparedServerTls, TlsAcceptFailure, TlsAcceptorHolder, TlsHandshakeFailureKind, accept_tls_with_deadline, load_tls_material, spawn_reload_loop, }, }; #[cfg(feature = "http3")] -use crate::server::{http3, hybrid::HybridBody, tls_material::build_http3_server_config_from_loaded}; +use crate::server::{http3, hybrid::HybridBody}; use crate::storage_api::server::http as storage; use crate::storage_api::server::http::rpc::InternodeRpcService; #[cfg(test)] @@ -1448,6 +1448,7 @@ pub async fn start_http_server( // Load TLS materials and build server acceptor in a single pass. // Outbound material (root CAs, mTLS identity) was already published in main.rs; // this load is needed for the server-side TLS acceptor and reload loop. + let mut tls_initial_snapshot = None; let tls_acceptor = if tls_path_configured { let snapshot = load_tls_material(tls_path).await.map_err(|e| { Error::other(format!( @@ -1456,27 +1457,30 @@ pub async fn start_http_server( )) })?; - #[cfg(feature = "http3")] - if http3_enabled { - http3_server_config = build_http3_server_config_from_loaded(snapshot.server.as_ref(), std::path::Path::new(tls_path)) - .await - .map_err(|e| Error::other(format!("HTTP/3 TLS configuration failed: {e}")))?; - } - - let acceptor = build_acceptor_from_loaded(snapshot.server, std::path::Path::new(tls_path)) - .await - .map_err(|e| Error::other(e.to_string()))?; + let prepared = PreparedServerTls::from_snapshot(&snapshot).map_err(|e| Error::other(e.to_string()))?; // Fail closed: if TLS was explicitly configured but no server certificates // were found, refuse to start rather than silently falling back to plain HTTP. - match acceptor { + match prepared { None => { return Err(Error::other(format!( "TLS is explicitly configured via RUSTFS_TLS_PATH/tls_path='{}' but no server certificates were found", tls_path ))); } - Some(a) => Some(a), + Some(prepared) => { + #[cfg(feature = "http3")] + if http3_enabled { + http3_server_config = Some( + prepared + .http3_config() + .map_err(|e| Error::other(format!("HTTP/3 TLS configuration failed: {e}")))?, + ); + } + let acceptor = prepared.tcp_acceptor().map_err(|e| Error::other(e.to_string()))?; + tls_initial_snapshot = Some((snapshot, prepared)); + Some(acceptor) + } } } else { None @@ -1755,14 +1759,20 @@ pub async fn start_http_server( } else { None }; - let tls_reload_task = tls_acceptor.as_ref().and_then(|holder| { + let tls_reload_task = if let (Some(holder), Some((initial, prepared))) = (tls_acceptor.as_ref(), tls_initial_snapshot.take()) + { spawn_reload_loop( tls_path.to_string(), holder.clone(), + initial, + prepared, #[cfg(feature = "http3")] http3_server.as_ref().map(|(endpoint, _)| endpoint.clone()), ) - }); + .map_err(|e| Error::other(e.to_string()))? + } else { + None + }; let is_console = config.console_enable; let server_domains_configured = !config.server_domains.is_empty(); @@ -2644,7 +2654,10 @@ fn check_auth(req: Request<()>) -> std::result::Result, Status> { .unwrap_or("unknown"); let peer_addr = req .extensions() - .get::() + .get::>() + .copied() + .flatten() + .or_else(|| req.extensions().get::().copied()) .map(|addr| addr.0.to_string()) .unwrap_or_else(|| "unknown".to_string()); let failure_reason = storage::tonic_rpc_auth_failure_reason(&e); @@ -2660,25 +2673,41 @@ fn check_auth(req: Request<()>) -> std::result::Result, Status> { INTERNODE_TRANSPORT_BACKEND_GRPC, failure_reason, ); - error!( - event = EVENT_RPC_SIGNATURE_VERIFICATION_FAILED, - component = LOG_COMPONENT_SERVER, - subsystem = LOG_SUBSYSTEM_HTTP, - failure_reason, - rpc_path, - rpc_service, - rpc_method, - expected_audience = %audience, - peer_addr = %peer_addr, - replay_scope_bootstrap_allowed = allow_replay_scope_bootstrap, - error = %e, - "RPC signature verification failed" - ); if failure_reason == "stale_boot_epoch" { // The signature is valid, but the peer restarted before this request. Reject it // before execution and let the client retry after its authenticated epoch refresh. + warn!( + event = EVENT_RPC_SIGNATURE_VERIFICATION_FAILED, + component = LOG_COMPONENT_SERVER, + subsystem = LOG_SUBSYSTEM_HTTP, + result = "retryable_rejection", + failure_reason, + rpc_path, + rpc_service, + rpc_method, + expected_audience = %audience, + peer_addr = %peer_addr, + replay_scope_bootstrap_allowed = allow_replay_scope_bootstrap, + error = %e, + "RPC boot epoch changed; request rejected for retry" + ); Status::unavailable("RPC boot epoch changed") } else { + error!( + event = EVENT_RPC_SIGNATURE_VERIFICATION_FAILED, + component = LOG_COMPONENT_SERVER, + subsystem = LOG_SUBSYSTEM_HTTP, + result = "rejected", + failure_reason, + rpc_path, + rpc_service, + rpc_method, + expected_audience = %audience, + peer_addr = %peer_addr, + replay_scope_bootstrap_allowed = allow_replay_scope_bootstrap, + error = %e, + "RPC signature verification failed" + ); Status::unauthenticated("No valid auth token") } })?; @@ -2806,6 +2835,163 @@ mod tests { use tokio::sync::{Notify, mpsc}; use tower::{Layer, Service, ServiceBuilder}; + type RateLimitTestCore = tower::util::BoxCloneService< + HttpRequest, + Response>>, + Box, + >; + type RateLimitTestStack = tower::util::BoxCloneService< + HttpRequest, + Response>>, + Box, + >; + + fn rate_limited_external_stack( + peer: SocketAddr, + proxy_layer: Option, + inner: RateLimitTestCore, + ) -> RateLimitTestStack { + use crate::server::rate_limit::RateLimitQuota; + + let readiness = Arc::new(GlobalReadiness::new()); + readiness.mark_stage(rustfs_common::SystemStage::FullReady); + RateLimitTestStack::new(external_service_stack!( + inner, + Some(RemoteAddr(peer)), + proxy_layer, + false, + Some(RateLimitLayer::new( + Some(RateLimitQuota { + requests_per_minute: 1, + burst: 1 + }), + None, + vec![] + )), + false, + readiness, + None::>, + HttpCompressionConfig::default(), + ServerContextSlot::new(), + false + )) + } + + #[tokio::test] + async fn external_stack_limits_socket_peers_without_trusted_proxies() { + use crate::server::hybrid::HybridBody; + use tower::ServiceExt; + + for peer in ["192.0.2.10:51000", "[2001:db8::10]:51000"] { + let peer: SocketAddr = peer.parse().expect("socket peer"); + for version in [Version::HTTP_11, Version::HTTP_2, Version::HTTP_3] { + let calls = Arc::new(AtomicUsize::new(0)); + let inner_calls = calls.clone(); + let inner = tower::service_fn(move |request: HttpRequest| { + assert_eq!( + request + .extensions() + .get::>() + .copied() + .flatten() + .map(|addr| addr.0), + Some(peer) + ); + inner_calls.fetch_add(1, Ordering::SeqCst); + async { + Ok::<_, Box>(Response::new( + HybridBody::>::Rest { + rest_body: s3s::Body::empty(), + }, + )) + } + }); + let mut service = rate_limited_external_stack(peer, None, RateLimitTestCore::new(inner)); + + for (forwarded, expected) in [ + ("203.0.113.1", StatusCode::OK), + ("203.0.113.2", StatusCode::TOO_MANY_REQUESTS), + ] { + let request = HttpRequest::builder() + .uri("/bucket/object") + .version(version) + .header("x-forwarded-for", forwarded) + .header("x-real-ip", forwarded) + .body(s3s::Body::empty()) + .expect("request"); + let response = service + .ready() + .await + .expect("service ready") + .call(request) + .await + .expect("response"); + assert_eq!(response.status(), expected, "peer={peer}, version={version:?}"); + if expected == StatusCode::TOO_MANY_REQUESTS { + assert!(response.headers().contains_key(http::header::RETRY_AFTER)); + let body = response.into_body().collect().await.expect("body").to_bytes(); + assert!(String::from_utf8_lossy(&body).contains("TooManyRequests")); + } + } + assert_eq!(calls.load(Ordering::SeqCst), 1, "throttled requests must not reach the inner service"); + } + } + } + + #[tokio::test] + async fn external_stack_limits_validated_proxy_clients_independently() { + use crate::server::hybrid::HybridBody; + use rustfs_trusted_proxies::{ + LegacyTrustedProxyLayer, TrustedProxy, TrustedProxyConfig, TrustedProxyLayer, ValidationMode, + }; + use tower::ServiceExt; + + let peer: SocketAddr = "192.0.2.100:51000".parse().expect("proxy peer"); + for version in [Version::HTTP_11, Version::HTTP_2, Version::HTTP_3] { + let proxy_config = + TrustedProxyConfig::new(vec![TrustedProxy::Single(peer.ip())], ValidationMode::HopByHop, false, 3, false, vec![]); + let proxy_layer = TrustedProxyLayer::legacy(LegacyTrustedProxyLayer::new(proxy_config, None, true)); + let calls = Arc::new(AtomicUsize::new(0)); + let inner_calls = calls.clone(); + let inner = tower::service_fn(move |request: HttpRequest| { + let info = request.extensions().get::().expect("validated client context"); + assert!(info.is_from_trusted_proxy); + assert_ne!(info.real_ip, peer.ip()); + inner_calls.fetch_add(1, Ordering::SeqCst); + async { + Ok::<_, Box>(Response::new( + HybridBody::>::Rest { + rest_body: s3s::Body::empty(), + }, + )) + } + }); + let mut service = rate_limited_external_stack(peer, Some(proxy_layer), RateLimitTestCore::new(inner)); + + for (client, expected) in [ + ("203.0.113.1", StatusCode::OK), + ("203.0.113.2", StatusCode::OK), + ("203.0.113.1", StatusCode::TOO_MANY_REQUESTS), + ] { + let request = HttpRequest::builder() + .uri("/bucket/object") + .version(version) + .header("x-forwarded-for", client) + .body(s3s::Body::empty()) + .expect("request"); + let response = service + .ready() + .await + .expect("service ready") + .call(request) + .await + .expect("response"); + assert_eq!(response.status(), expected, "client={client}, version={version:?}"); + } + assert_eq!(calls.load(Ordering::SeqCst), 2); + } + } + #[test] fn peer_transport_close_kind_uses_io_error_sources() { #[derive(Debug)] @@ -3278,6 +3464,291 @@ mod tests { assert_eq!(bytes_polled.load(Ordering::Relaxed), 0); } + mod conditional_object_http { + use super::*; + use crate::app::storage_api::object_usecase::{StorageObjectInfo, check_preconditions}; + use crate::server::hybrid::HybridBody; + + #[derive(Clone)] + struct ConditionalObjectS3 { + calls: Arc, + } + + #[derive(Debug)] + struct FixtureTransportError(String); + + impl std::fmt::Display for FixtureTransportError { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str(&self.0) + } + } + + impl std::error::Error for FixtureTransportError {} + + type HttpConditionCase<'a> = (&'a str, &'a [(&'a str, &'a str)], StatusCode, bool); + + impl<'a> From> for FixtureTransportError { + fn from(error: Box) -> Self { + Self(error.to_string()) + } + } + + fn object_info() -> StorageObjectInfo { + StorageObjectInfo { + etag: Some("abc".to_owned()), + mod_time: Some(time::OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("fixture timestamp")), + user_defined: HashMap::from([("cache-control".to_owned(), "max-age=60".to_owned())]).into(), + ..Default::default() + } + } + + #[async_trait::async_trait] + impl s3s::S3 for ConditionalObjectS3 { + async fn get_object( + &self, + req: s3s::S3Request, + ) -> s3s::S3Result> { + self.calls.fetch_add(1, Ordering::Relaxed); + assert_eq!(req.input.bucket, "bucket"); + assert_eq!(req.input.key, "object"); + assert_eq!(req.input.if_match.is_some(), req.headers.contains_key(http::header::IF_MATCH)); + assert_eq!(req.input.if_none_match.is_some(), req.headers.contains_key(http::header::IF_NONE_MATCH)); + check_preconditions(&req.headers, &object_info())?; + Ok(s3s::S3Response::new(s3s::dto::GetObjectOutput { + body: Some(s3s::dto::StreamingBlob::from_bytes(Bytes::from_static(b"payload"))), + content_length: Some(7), + e_tag: Some(s3s::dto::ETag::Strong("abc".to_owned())), + ..Default::default() + })) + } + + async fn head_object( + &self, + req: s3s::S3Request, + ) -> s3s::S3Result> { + self.calls.fetch_add(1, Ordering::Relaxed); + assert_eq!(req.input.bucket, "bucket"); + assert_eq!(req.input.key, "object"); + assert_eq!(req.input.if_match.is_some(), req.headers.contains_key(http::header::IF_MATCH)); + assert_eq!(req.input.if_none_match.is_some(), req.headers.contains_key(http::header::IF_NONE_MATCH)); + check_preconditions(&req.headers, &object_info())?; + Ok(s3s::S3Response::new(s3s::dto::HeadObjectOutput { + content_length: Some(7), + e_tag: Some(s3s::dto::ETag::Strong("abc".to_owned())), + ..Default::default() + })) + } + } + + #[tokio::test] + async fn conditional_get_head_http1_http2_parse_headers_and_apply_precedence() { + let cases: &[HttpConditionCase<'_>] = &[ + ("strong list", &[("if-none-match", "\"other\", \"abc\"")], StatusCode::NOT_MODIFIED, true), + ("weak list", &[("if-none-match", "\"other\", W/\"abc\"")], StatusCode::NOT_MODIFIED, true), + ("weak tag", &[("if-none-match", "W/\"abc\"")], StatusCode::NOT_MODIFIED, true), + ("wildcard", &[("if-none-match", "*")], StatusCode::NOT_MODIFIED, true), + ("strong match list", &[("if-match", "\"other\", \"abc\"")], StatusCode::OK, true), + ("weak match fails", &[("if-match", "W/\"abc\"")], StatusCode::PRECONDITION_FAILED, true), + ("wildcard match", &[("if-match", "*")], StatusCode::OK, true), + ( + "If-Match precedes matching If-None-Match", + &[("if-match", "\"other\""), ("if-none-match", "\"abc\"")], + StatusCode::PRECONDITION_FAILED, + true, + ), + ( + "matching If-Match does not suppress If-None-Match", + &[("if-match", "\"abc\""), ("if-none-match", "W/\"abc\"")], + StatusCode::NOT_MODIFIED, + true, + ), + ( + "If-Match suppresses If-Unmodified-Since", + &[ + ("if-match", "\"abc\""), + ("if-unmodified-since", "Tue, 14 Nov 2023 22:13:19 GMT"), + ], + StatusCode::OK, + true, + ), + ( + "If-None-Match suppresses If-Modified-Since", + &[ + ("if-none-match", "\"other\""), + ("if-modified-since", "Tue, 14 Nov 2023 22:13:21 GMT"), + ], + StatusCode::OK, + true, + ), + ( + "If-Unmodified-Since precedes wildcard If-None-Match", + &[ + ("if-unmodified-since", "Tue, 14 Nov 2023 22:13:19 GMT"), + ("if-none-match", "*"), + ], + StatusCode::PRECONDITION_FAILED, + true, + ), + ( + "date matches at same second", + &[("if-modified-since", "Tue, 14 Nov 2023 22:13:20 GMT")], + StatusCode::NOT_MODIFIED, + true, + ), + ("unmatched list", &[("if-none-match", "\"other\", W/\"different\"")], StatusCode::OK, true), + ( + "empty list member is allowed", + &[("if-none-match", "\"other\", , \"abc\"")], + StatusCode::NOT_MODIFIED, + true, + ), + // s3s accepts these quoted fields; the canonical matcher must validate every member. + ( + "adjacent tags cannot validate cache", + &[("if-none-match", "\"other\"\"abc\"")], + StatusCode::OK, + true, + ), + ( + "adjacent tags cannot satisfy If-Match", + &[("if-match", "\"other\"\"abc\"")], + StatusCode::PRECONDITION_FAILED, + true, + ), + ( + "matching prefix with trailing garbage", + &[("if-none-match", "\"abc\"garbage\"")], + StatusCode::OK, + true, + ), + ( + "matching prefix cannot satisfy If-Match", + &[("if-match", "\"abc\"garbage\"")], + StatusCode::PRECONDITION_FAILED, + true, + ), + ( + "matching first tag with malformed later member", + &[("if-none-match", "\"abc\", \"other\"\"different\"")], + StatusCode::OK, + true, + ), + ( + "If-Match validates malformed later member", + &[("if-match", "\"abc\", \"other\"\"different\"")], + StatusCode::PRECONDITION_FAILED, + true, + ), + // These existing s3s parse boundaries reject the request before the handler. + ("malformed ETag", &[("if-none-match", "\"abc")], StatusCode::BAD_REQUEST, false), + ("invalid date", &[("if-modified-since", "not-a-date")], StatusCode::BAD_REQUEST, false), + ( + "obsolete date", + &[("if-modified-since", "Tuesday, 14-Nov-23 22:13:20 GMT")], + StatusCode::BAD_REQUEST, + false, + ), + ( + "duplicate condition", + &[("if-none-match", "\"other\""), ("if-none-match", "\"abc\"")], + StatusCode::BAD_REQUEST, + false, + ), + ]; + for http2 in [false, true] { + for method in [Method::GET, Method::HEAD] { + for (name, headers, expected_status, reaches_handler) in cases { + let calls = Arc::new(AtomicUsize::new(0)); + let inner = S3ServiceBuilder::new(ConditionalObjectS3 { + calls: Arc::clone(&calls), + }) + .build(); + let service = ServiceBuilder::new() + .layer(BodylessStatusFixLayer) + .layer(HeadRequestBodyFixLayer) + .map_response(|response: Response| { + response.map(|rest_body| HybridBody::>::Rest { rest_body }) + }) + .map_err(|error: s3s::HttpError| FixtureTransportError(error.to_string())) + .service(inner); + let (client_io, server_io) = tokio::io::duplex(64 * 1024); + let server = tokio::spawn(async move { + let service = TowerToHyperService::new(service); + if http2 { + hyper::server::conn::http2::Builder::new(TokioExecutor::new()) + .serve_connection(TokioIo::new(server_io), service) + .await + .expect("HTTP/2 server"); + } else { + hyper::server::conn::http1::Builder::new() + .serve_connection(TokioIo::new(server_io), service) + .await + .expect("HTTP/1 server"); + } + }); + let mut request = HttpRequest::builder() + .method(method.clone()) + .uri("http://localhost/bucket/object"); + for (header, value) in *headers { + request = request.header(*header, *value); + } + let request = request.body(Empty::::new()).expect("conditional request"); + let (response, client) = if http2 { + let (mut sender, connection) = + hyper::client::conn::http2::handshake(TokioExecutor::new(), TokioIo::new(client_io)) + .await + .expect("HTTP/2 handshake"); + let client = tokio::spawn(async move { + connection.await.expect("HTTP/2 client"); + }); + let response = tokio::time::timeout(Duration::from_secs(5), sender.send_request(request)) + .await + .expect("HTTP/2 response deadline") + .expect("HTTP/2 response"); + (response, client) + } else { + let (mut sender, connection) = hyper::client::conn::http1::handshake(TokioIo::new(client_io)) + .await + .expect("HTTP/1 handshake"); + let client = tokio::spawn(async move { + connection.await.expect("HTTP/1 client"); + }); + let response = tokio::time::timeout(Duration::from_secs(5), sender.send_request(request)) + .await + .expect("HTTP/1 response deadline") + .expect("HTTP/1 response"); + (response, client) + }; + assert_eq!(response.status(), *expected_status, "{name}: {method}, HTTP/2={http2}"); + assert_eq!(calls.load(Ordering::Relaxed), usize::from(*reaches_handler), "{name}: parser routing"); + if *expected_status == StatusCode::NOT_MODIFIED { + assert_eq!(response.headers()[http::header::ETAG], "\"abc\"", "{name}"); + assert_eq!(response.headers()[http::header::CACHE_CONTROL], "max-age=60", "{name}"); + assert!(response.headers().contains_key(http::header::LAST_MODIFIED), "{name}"); + assert!(response.headers().contains_key(http::header::DATE), "{name}"); + assert!(!response.headers().contains_key(CONTENT_LENGTH), "{name}"); + assert!(!response.headers().contains_key(http::header::TRANSFER_ENCODING), "{name}"); + } + let bytes = tokio::time::timeout(Duration::from_secs(5), response.into_body().collect()) + .await + .expect("body deadline") + .expect("response body") + .to_bytes(); + if method == Method::HEAD || *expected_status == StatusCode::NOT_MODIFIED { + assert!(bytes.is_empty(), "{name}: {method}, HTTP/2={http2}"); + } else if *expected_status == StatusCode::OK { + assert_eq!(&bytes[..], b"payload", "{name}"); + } else if *expected_status == StatusCode::PRECONDITION_FAILED { + assert!(String::from_utf8_lossy(&bytes).contains("PreconditionFailed"), "{name}"); + } + client.abort(); + server.abort(); + } + } + } + } + } + #[derive(Clone, Copy)] struct UploadPartTimeoutS3 { timeout: Duration, @@ -3976,14 +4447,14 @@ mod tests { let _ = rustfs_credentials::set_global_rpc_secret("rpc-http-test-secret".to_string()); let previous_node_name = rustfs_common::get_global_local_node_name().await; let audience = "127.0.0.1:9000"; - let path = "/node_service.NodeService/ReadVersion"; + let path = "/node_service.NodeService/ReadAll"; rustfs_common::set_global_local_node_name(audience).await; let challenge = uuid::Uuid::new_v4(); let proof = storage::tonic_boot_epoch_response_headers(audience, challenge).expect("signed epoch proof"); let epoch = storage::verify_tonic_boot_epoch_response(audience, challenge, &proof).expect("authenticated epoch"); let stale_epoch = uuid::Uuid::from_u128(epoch.as_u128() ^ 1); let signed_headers = |audience: &str, epoch| { - let mut headers = storage::gen_tonic_signature_headers(audience, "node_service.NodeService", "ReadVersion", None) + let mut headers = storage::gen_tonic_signature_headers(audience, "node_service.NodeService", "ReadAll", None) .expect("method-bound signature"); let replay = storage::gen_tonic_replay_scope_headers( audience, @@ -3996,34 +4467,198 @@ mod tests { headers.extend(replay); headers }; - let request = |headers: HeaderMap| { + let request = |headers: HeaderMap, peer: Option| { let mut request = Request::new(()); request.metadata_mut().as_mut().extend(headers); request.extensions_mut().insert(RpcRequestTarget { uri: format!("http://{audience}{path}").parse().expect("RPC URI"), method: Method::POST, }); + // The connection stack stores this optional wrapper, not a bare RemoteAddr. + request.extensions_mut().insert(peer.map(RemoteAddr)); request }; - + let log_file = tempfile::NamedTempFile::new().expect("create RPC auth log capture"); + let subscriber = tracing_subscriber::fmt() + .json() + .with_ansi(false) + .with_max_level(Level::TRACE) + .with_writer(Arc::new(log_file.reopen().expect("open RPC auth log writer"))) + .finish(); + let peer: SocketAddr = "192.0.2.10:51000".parse().expect("test peer address"); let stale = signed_headers(audience, stale_epoch); - let error = check_auth(request(stale.clone())).expect_err("a stale epoch must still reject the request"); - assert_eq!(error.code(), tonic::Code::Unavailable); - assert_eq!(error.message(), "RPC boot epoch changed"); + tracing::subscriber::with_default(subscriber, || { + let error = check_auth(request(stale.clone(), Some(peer))).expect_err("a stale epoch must still reject the request"); + assert_eq!(error.code(), tonic::Code::Unavailable); + assert_eq!(error.message(), "RPC boot epoch changed"); + let mut forged = stale.clone(); + forged.insert("x-rustfs-rpc-signature-v3", HeaderValue::from_static("00")); + let error = check_auth(request(forged, Some(peer))).expect_err("a forged stale-epoch signature must remain terminal"); + assert_eq!(error.code(), tonic::Code::Unauthenticated); + let error = check_auth(request(signed_headers("127.0.0.1:9001", stale_epoch), Some(peer))) + .expect_err("a stale epoch must not hide an audience mismatch"); + assert_eq!(error.code(), tonic::Code::Unauthenticated); - let mut forged = stale; - forged.insert("x-rustfs-rpc-signature-v3", HeaderValue::from_static("00")); - let error = check_auth(request(forged)).expect_err("a forged stale-epoch signature must remain terminal"); - assert_eq!(error.code(), tonic::Code::Unauthenticated); - let error = check_auth(request(signed_headers("127.0.0.1:9001", stale_epoch))) - .expect_err("a stale epoch must not hide an audience mismatch"); - assert_eq!(error.code(), tonic::Code::Unauthenticated); - - let current = signed_headers(audience, epoch); - assert!(check_auth(request(current.clone())).is_ok(), "a fresh authenticated scope must succeed"); - let error = check_auth(request(current)).expect_err("a same-epoch nonce replay must remain terminal"); - assert_eq!(error.code(), tonic::Code::Unauthenticated); + let current = signed_headers(audience, epoch); + assert!( + check_auth(request(current.clone(), Some(peer))).is_ok(), + "a fresh authenticated scope must succeed" + ); + let error = check_auth(request(current, Some(peer))).expect_err("a same-epoch nonce replay must remain terminal"); + assert_eq!(error.code(), tonic::Code::Unauthenticated); + let error = + check_auth(request(stale.clone(), None)).expect_err("a missing peer address must not affect authentication"); + assert_eq!(error.code(), tonic::Code::Unavailable); + }); rustfs_common::set_global_local_node_name(&previous_node_name).await; + + let logs = std::fs::read_to_string(log_file.path()).expect("read RPC auth logs"); + let events: Vec = logs + .lines() + .map(|line| serde_json::from_str::(line).expect("valid JSON log event")) + .filter(|event| event["fields"]["event"] == EVENT_RPC_SIGNATURE_VERIFICATION_FAILED) + .collect(); + assert_eq!(events.len(), 5, "each authentication rejection must retain its diagnostic event"); + let stale_events: Vec<_> = events + .iter() + .filter(|event| event["fields"]["failure_reason"] == "stale_boot_epoch") + .collect(); + assert_eq!(stale_events.len(), 2); + for event in &stale_events { + assert_eq!(event["level"], "WARN"); + assert_eq!(event["fields"]["result"], "retryable_rejection"); + assert_eq!(event["fields"]["rpc_method"], "ReadAll"); + } + assert_eq!(stale_events[0]["fields"]["peer_addr"], peer.to_string()); + assert_eq!(stale_events[1]["fields"]["peer_addr"], "unknown"); + for event in events + .iter() + .filter(|event| event["fields"]["failure_reason"] != "stale_boot_epoch") + { + assert_eq!(event["level"], "ERROR"); + assert_eq!(event["fields"]["result"], "rejected"); + assert_eq!(event["fields"]["peer_addr"], peer.to_string()); + } + assert!(!logs.contains("rpc-http-test-secret"), "logs must not expose the RPC secret"); + for header in ["x-rustfs-signature", "x-rustfs-rpc-signature-v2", "x-rustfs-rpc-signature-v3"] { + if let Some(signature) = stale.get(header) { + assert!( + !logs.contains(signature.to_str().expect("ASCII signature")), + "logs must not expose {header}" + ); + } + } + } + + #[tokio::test] + #[serial_test::serial] + async fn rpc_auth_read_all_recovers_after_stale_boot_epoch() { + use storage::DiskAPI; + + temp_env::async_with_vars( + [ + ("RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES", Some("1")), + ("RUSTFS_INTERNODE_PREWARM", Some("false")), + ], + async { + let _ = rustfs_credentials::set_global_rpc_secret("rpc-http-test-secret".to_string()); + let listener = TcpListener::bind("127.0.0.1:0") + .await + .expect("bind ReadAll epoch recovery peer"); + let addr = listener.local_addr().expect("ReadAll recovery address"); + let audience = addr.to_string(); + let previous_node_name = rustfs_common::get_global_local_node_name().await; + rustfs_common::set_global_local_node_name(&audience).await; + let challenge = uuid::Uuid::new_v4(); + let proof = storage::tonic_boot_epoch_response_headers(&audience, challenge).expect("current epoch proof"); + let epoch = storage::verify_tonic_boot_epoch_response(&audience, challenge, &proof).expect("current epoch"); + let stale_epoch = uuid::Uuid::from_u128(epoch.as_u128() ^ 1); + let admitted = Arc::new(AtomicUsize::new(0)); + let admitted_by_server = Arc::clone(&admitted); + let node = InterceptedService::new(NodeServiceServer::new(make_server()), move |request: Request<()>| { + let request = check_auth(request)?; + assert!( + request.extensions().get::>().copied().flatten().is_some(), + "production auth must receive the connection's optional peer address" + ); + admitted_by_server.fetch_add(1, Ordering::SeqCst); + Ok(request) + }); + let scopes = Arc::new(Mutex::new(Vec::new())); + let observed_scopes = Arc::clone(&scopes); + let service = ServiceBuilder::new() + .map_request(move |mut request: HttpRequest| { + assert_eq!(request.uri().path(), "/node_service.NodeService/ReadAll"); + let mut scopes = observed_scopes.lock().expect("captured replay scopes"); + if scopes.is_empty() { + assert!( + !request.headers().contains_key("x-rustfs-rpc-boot-epoch"), + "the fresh peer must start without a cached epoch" + ); + // Inject a correctly signed stale scope at the transport boundary. + // The retry must learn its scope from the real server response proof. + let replay = storage::gen_tonic_replay_scope_headers( + &audience, + request.uri().path(), + request.headers()["x-rustfs-timestamp"].to_str().expect("timestamp"), + request.headers()["x-rustfs-content-sha256"].to_str().expect("body digest"), + stale_epoch, + ) + .expect("signed stale scope"); + request.headers_mut().extend(replay); + } + let parse_uuid = |name: &str| { + uuid::Uuid::parse_str(request.headers()[name].to_str().expect("UUID header")) + .expect("valid replay scope UUID") + }; + scopes.push((parse_uuid("x-rustfs-rpc-boot-epoch"), parse_uuid("x-rustfs-rpc-replay-nonce"))); + request + }) + .service(RpcRequestPathService::new(Routes::new(node).prepare())); + let server = tokio::spawn(async move { + let (socket, peer) = listener.accept().await.expect("accept ReadAll recovery connection"); + let service = ServiceBuilder::new() + .layer(AddExtensionLayer::new(Some(RemoteAddr(peer)))) + .service(service); + ConnBuilder::new(TokioExecutor::new()) + .serve_connection(TokioIo::new(socket), TowerToHyperService::new(service)) + .await + .expect("serve ReadAll recovery connection"); + }); + let mut endpoint = + Endpoint::try_from(format!("http://{addr}/missing-read-all-disk").as_str()).expect("ReadAll remote endpoint"); + endpoint.is_local = false; + let disk = storage::new_disk( + &endpoint, + &storage::DiskOption { + cleanup: false, + health_check: false, + }, + ) + .await + .expect("production remote disk client"); + let result = tokio::time::timeout(Duration::from_secs(10), disk.read_all("bucket", "object")).await; + server.abort(); + let _ = server.await; + rustfs_common::set_global_local_node_name(&previous_node_name).await; + + let error = result + .expect("bounded ReadAll retry must complete") + .expect_err("missing disk must remain an error"); + assert_eq!( + error.to_string(), + "io error io error cannot find disk", + "the retry must reach the real ReadAll missing-disk handler" + ); + let scopes = scopes.lock().expect("captured replay scopes"); + assert_eq!(scopes.len(), 2, "stale epoch recovery must use one bounded retry"); + assert_eq!(scopes[0].0, stale_epoch); + assert_eq!(scopes[1].0, epoch, "the retry must use the authenticated response epoch"); + assert_ne!(scopes[0].1, scopes[1].1, "the retry must carry a fresh replay nonce"); + assert_eq!(admitted.load(Ordering::SeqCst), 1, "the stale request must not reach the handler"); + }, + ) + .await; } #[tokio::test] @@ -4155,6 +4790,51 @@ mod tests { ); } + #[tokio::test] + #[serial_test::serial] + async fn cold_delete_recovery_discovery_uses_production_body_auth() { + use storage::DiskAPI; + temp_env::async_with_vars([("RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT", Some("false"))], async { + let _ = rustfs_credentials::set_global_rpc_secret("rpc-http-test-secret".to_string()); + let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind recovery RPC peer"); + let addr = listener.local_addr().expect("recovery RPC address"); + let previous_node_name = rustfs_common::get_global_local_node_name().await; + rustfs_common::set_global_local_node_name(&addr.to_string()).await; + let node = InterceptedService::new(NodeServiceServer::new(make_server()), check_auth); + let service = RpcRequestPathService::new(Routes::new(node).prepare()); + let server = tokio::spawn(async move { + let (socket, _) = listener.accept().await.expect("accept recovery RPC connection"); + ConnBuilder::new(TokioExecutor::new()) + .serve_connection(TokioIo::new(socket), TowerToHyperService::new(service)) + .await + .expect("serve recovery RPC connection"); + }); + let mut endpoint = + Endpoint::try_from(format!("http://{addr}/cold-recovery-disk").as_str()).expect("remote recovery endpoint"); + endpoint.is_local = false; + let disk = storage::new_disk( + &endpoint, + &storage::DiskOption { + cleanup: false, + health_check: false, + }, + ) + .await + .expect("production remote disk client"); + // Recovery may race a heal-control peer-state refresh, before another + // authenticated response has repopulated the shared peer state. + let result = disk.discover_delete_transactions("cold-recovery-bucket", None, None, 1).await; + server.abort(); + let _ = server.await; + rustfs_common::set_global_local_node_name(&previous_node_name).await; + assert!( + matches!(result, Err(crate::storage_api::error::DiskError::DiskNotFound)), + "the cold request must pass production authentication and reach the missing-disk handler: {result:?}" + ); + }) + .await; + } + #[tokio::test] #[serial_test::serial] async fn peer_rest_heal_control_uses_production_auth_and_keeps_validation_errors_online() { diff --git a/rustfs/src/server/hybrid.rs b/rustfs/src/server/hybrid.rs index c20d47215..170f7e6cf 100644 --- a/rustfs/src/server/hybrid.rs +++ b/rustfs/src/server/hybrid.rs @@ -19,6 +19,7 @@ use hyper::{Request, Response}; use pin_project_lite::pin_project; use std::pin::Pin; use std::task::{Context, Poll}; +use std::time::Instant; use tower::Service; type BoxError = Box; @@ -35,6 +36,15 @@ pub(crate) fn is_grpc_request(req: &Request) -> bool { ) } +fn rename_data_grpc_stage(path: &str) -> bool { + matches!( + path, + "/node_service.NodeService/RenameData" + | "/node_service.NodeService/RenameDataAtIncarnation" + | "/node_service.NodeService/RenameDataWithQuotaFence" + ) +} + /// The service that can serve both gRPC and REST HTTP Requests #[derive(Clone)] pub struct HybridService { @@ -71,8 +81,11 @@ where /// as a REST request fn call(&mut self, req: Request) -> Self::Future { if is_grpc_request(&req) { + let observe_rename_data = rustfs_io_metrics::put_stage_metrics_enabled() && rename_data_grpc_stage(req.uri().path()); HybridFuture::Grpc { grpc_future: self.grpc.call(req), + observe_rename_data, + service_future_started: observe_rename_data.then(Instant::now), } } else { HybridFuture::Rest { @@ -93,7 +106,10 @@ pin_project! { }, Grpc { #[pin] - grpc_body: GrpcBody + grpc_body: GrpcBody, + first_frame_started: Option, + body_started: Option, + first_frame_recorded: bool, }, } } @@ -123,21 +139,44 @@ where fn is_end_stream(&self) -> bool { match self { Self::Rest { rest_body } => rest_body.is_end_stream(), - Self::Grpc { grpc_body } => grpc_body.is_end_stream(), + Self::Grpc { grpc_body, .. } => grpc_body.is_end_stream(), } } fn poll_frame(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll, Self::Error>>> { match self.project() { HybridBodyProj::Rest { rest_body } => rest_body.poll_frame(cx).map_err(Into::into), - HybridBodyProj::Grpc { grpc_body } => grpc_body.poll_frame(cx).map_err(Into::into), + HybridBodyProj::Grpc { + mut grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + } => { + let poll = grpc_body.as_mut().poll_frame(cx).map_err(Into::into); + if !*first_frame_recorded && let Poll::Ready(_) = &poll { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_FIRST_FRAME, + first_frame_started.take(), + ); + *first_frame_recorded = true; + } + let body_complete = matches!(&poll, Poll::Ready(None)) + || (matches!(&poll, Poll::Ready(Some(Ok(_)))) && grpc_body.as_ref().get_ref().is_end_stream()); + if body_complete { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_BODY_COMPLETE, + body_started.take(), + ); + } + poll + } } } fn size_hint(&self) -> http_body::SizeHint { match self { Self::Rest { rest_body } => rest_body.size_hint(), - Self::Grpc { grpc_body } => grpc_body.size_hint(), + Self::Grpc { grpc_body, .. } => grpc_body.size_hint(), } } } @@ -154,6 +193,8 @@ pin_project! { Grpc { #[pin] grpc_future: GrpcFuture, + observe_rename_data: bool, + service_future_started: Option, }, } } @@ -174,8 +215,27 @@ where Poll::Ready(Err(err)) => Poll::Ready(Err(err.into())), Poll::Pending => Poll::Pending, }, - HybridFutureProj::Grpc { grpc_future } => match grpc_future.poll(cx) { - Poll::Ready(Ok(res)) => Poll::Ready(Ok(res.map(|grpc_body| HybridBody::Grpc { grpc_body }))), + HybridFutureProj::Grpc { + grpc_future, + observe_rename_data, + service_future_started, + } => match grpc_future.poll(cx) { + Poll::Ready(Ok(res)) => { + if *observe_rename_data { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_SERVICE_FUTURE, + service_future_started.take(), + ); + } + let first_frame_started = (*observe_rename_data).then(Instant::now); + let body_started = (*observe_rename_data).then(Instant::now); + Poll::Ready(Ok(res.map(|grpc_body| HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded: !*observe_rename_data, + }))) + } Poll::Ready(Err(err)) => Poll::Ready(Err(err.into())), Poll::Pending => Poll::Pending, }, diff --git a/rustfs/src/server/layer.rs b/rustfs/src/server/layer.rs index c6657cddf..bcc21fb37 100644 --- a/rustfs/src/server/layer.rs +++ b/rustfs/src/server/layer.rs @@ -587,6 +587,7 @@ where let watchdog_context = context.clone(); async move { tokio::select! { + biased; _ = watchdog.cancelled() => {} _ = tokio::time::sleep(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD) => { watchdog_context.log_slow_inflight(); @@ -596,11 +597,11 @@ where }); } + // An unpolled response future must also cancel its watchdog on drop. + let watchdog_guard = watchdog.map(CancellationToken::drop_guard); Box::pin(async move { let result = inner.call(req).await; - if let Some(watchdog) = watchdog { - watchdog.cancel(); - } + drop(watchdog_guard); match &result { Ok(response) => context.log_response(response), Err(error) => context.log_failure(error), @@ -863,7 +864,20 @@ where } Response::from_parts(parts, HybridBody::Rest { rest_body }) } - HybridBody::Grpc { grpc_body } => Response::from_parts(parts, HybridBody::Grpc { grpc_body }), + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + } => Response::from_parts( + parts, + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + }, + ), }; Ok(response) @@ -1012,7 +1026,20 @@ where } Response::from_parts(parts, HybridBody::Rest { rest_body }) } - HybridBody::Grpc { grpc_body } => Response::from_parts(parts, HybridBody::Grpc { grpc_body }), + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + } => Response::from_parts( + parts, + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + }, + ), }; Ok(response) @@ -1163,7 +1190,20 @@ where Response::from_parts(parts, HybridBody::Rest { rest_body }) } - HybridBody::Grpc { grpc_body } => Response::from_parts(parts, HybridBody::Grpc { grpc_body }), + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + } => Response::from_parts( + parts, + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + }, + ), }; Ok(response) @@ -1175,15 +1215,9 @@ where /// responses whose HTTP status code MUST NOT carry a body per RFC 9110 §6.4.1 /// and §15 (1xx, 204, 205, 304). /// -/// The inner s3s layer serializes every `S3Error` — including 304 `NotModified` -/// preconditions — as an XML body. Returning that body for a 304 is a protocol -/// violation: hyper's HTTP/1.1 encoder forces the body to zero length but -/// preserves the response, while the HTTP/2 path fills in `content-length` -/// from the body's size hint and writes DATA frames after a HEADERS frame that -/// should have carried END_STREAM. h2 clients (curl, browsers) and proxies see -/// the malformed response as a connection-level failure — in the wild this -/// surfaces as `GOAWAY error=0` on h2 and as an upstream-disconnect 5xx from -/// reverse proxies like ngrok (`ERR_NGROK_3004`). +/// s3s already suppresses bodyless error responses. This layer also protects +/// REST routes and custom responses: unlike the HTTP/1.1 encoder, hyper's +/// HTTP/2 sender relies on the body rather than the status to end the stream. #[derive(Clone)] pub struct BodylessStatusFixLayer; @@ -1240,7 +1274,20 @@ where }, ) } - HybridBody::Grpc { grpc_body } => Response::from_parts(parts, HybridBody::Grpc { grpc_body }), + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + } => Response::from_parts( + parts, + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + }, + ), }; Ok(response) @@ -1308,7 +1355,20 @@ where rest_body: RestBody::from(Bytes::new()), }, ), - HybridBody::Grpc { grpc_body } => Response::from_parts(parts, HybridBody::Grpc { grpc_body }), + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + } => Response::from_parts( + parts, + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + }, + ), }; Ok(response) @@ -5312,6 +5372,62 @@ mod tests { assert!(bytes.is_empty(), "304 response body must be empty"); } + #[tokio::test] + async fn bodyless_304_is_valid_on_http1_and_http2() { + use hyper_util::rt::{TokioExecutor, TokioIo}; + use hyper_util::service::TowerToHyperService; + for http2 in [false, true] { + let (client_io, server_io) = tokio::io::duplex(4096); + let service = TowerToHyperService::new(BodylessStatusFixLayer.layer(FixedResponse { + status: StatusCode::NOT_MODIFIED, + body: Bytes::from_static(b"NotModified"), + content_type: Some("application/xml"), + })); + let server = tokio::spawn(async move { + if http2 { + hyper::server::conn::http2::Builder::new(TokioExecutor::new()) + .serve_connection(TokioIo::new(server_io), service) + .await + .expect("HTTP/2 server"); + } else { + hyper::server::conn::http1::Builder::new() + .serve_connection(TokioIo::new(server_io), service) + .await + .expect("HTTP/1 server"); + } + }); + let request = Request::builder() + .uri("http://localhost/object") + .body(Empty::::new()) + .expect("request"); + let (response, client) = if http2 { + let (mut sender, connection) = + hyper::client::conn::http2::handshake(TokioExecutor::new(), TokioIo::new(client_io)) + .await + .expect("HTTP/2 handshake"); + let client = tokio::spawn(async move { + connection.await.expect("HTTP/2 client"); + }); + (sender.send_request(request).await.expect("HTTP/2 response"), client) + } else { + let (mut sender, connection) = hyper::client::conn::http1::handshake(TokioIo::new(client_io)) + .await + .expect("HTTP/1 handshake"); + let client = tokio::spawn(async move { + connection.await.expect("HTTP/1 client"); + }); + (sender.send_request(request).await.expect("HTTP/1 response"), client) + }; + assert_eq!(response.status(), StatusCode::NOT_MODIFIED); + assert_eq!(response.headers()[http::header::ETAG], "\"abc123\""); + assert!(!response.headers().contains_key(http::header::CONTENT_LENGTH)); + assert!(!response.headers().contains_key(http::header::CONTENT_TYPE)); + assert!(collect_body(response.into_body()).await.is_empty()); + client.abort(); + server.abort(); + } + } + #[tokio::test] async fn strips_body_for_204() { let mut svc = BodylessStatusFixLayer.layer(FixedResponse { @@ -5632,6 +5748,154 @@ mod tests { assert_eq!(redact_sensitive_uri_query(&uri), "/rustfs/admin/v3/users?token=not-a-download-token"); } + #[tokio::test(start_paused = true)] + async fn request_logging_watchdog_cancels_dropped_futures() { + for poll_request in [false, true] { + for deadline_ready in [false, true] { + let attempts = if deadline_ready { 32 } else { 1 }; + for attempt in 0..attempts { + let writer = SharedWriter::default(); + let captured = writer.buffer.clone(); + let _subscriber = tracing::subscriber::set_default( + tracing_subscriber::fmt() + .without_time() + .with_ansi(false) + .with_writer(writer) + .finish(), + ); + assert!(tracing::enabled!(target: HTTP_SERVER_LOG_TARGET, Level::WARN)); + let inner = + tower::service_fn(|_: HttpRequest<()>| std::future::pending::, Infallible>>()); + let mut service = RequestLoggingLayer.layer(inner); + let mut response = service.call(HttpRequest::new(())); + if poll_request { + assert!(futures::poll!(&mut response).is_pending()); + } + tokio::task::yield_now().await; + if deadline_ready { + tokio::time::advance(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD).await; + } + drop(response); + tokio::time::advance(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD + Duration::from_millis(1)).await; + tokio::task::yield_now().await; + + let output = String::from_utf8(captured.lock().expect("logs").clone()).expect("UTF-8 logs"); + assert!( + !output.contains(HTTP_REQUEST_INFLIGHT_SLOW_EVENT), + "polled={poll_request}, deadline_ready={deadline_ready}, attempt={attempt}: {output}" + ); + } + } + } + } + + #[tokio::test(start_paused = true)] + async fn request_logging_watchdog_cancels_panicking_future() { + let writer = SharedWriter::default(); + let captured = writer.buffer.clone(); + let _subscriber = tracing::subscriber::set_default( + tracing_subscriber::fmt() + .without_time() + .with_ansi(false) + .with_writer(writer) + .finish(), + ); + assert!(tracing::enabled!(target: HTTP_SERVER_LOG_TARGET, Level::WARN)); + let inner = tower::service_fn(|_: HttpRequest<()>| { + std::future::poll_fn(|_| -> Poll, Infallible>> { panic!("test service panic") }) + }); + let mut service = RequestLoggingLayer.layer(inner); + let mut response = service.call(HttpRequest::new(())); + tokio::task::yield_now().await; + let panic = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + response.as_mut().poll(&mut Context::from_waker(std::task::Waker::noop())) + })); + assert!(panic.is_err()); + tokio::time::advance(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD + Duration::from_millis(1)).await; + tokio::task::yield_now().await; + + let output = String::from_utf8(captured.lock().expect("logs").clone()).expect("UTF-8 logs"); + assert!(!output.contains(HTTP_REQUEST_INFLIGHT_SLOW_EVENT), "{output}"); + drop(response); + } + + #[tokio::test(start_paused = true)] + async fn request_logging_watchdog_cancels_completed_futures() { + for fail_request in [false, true] { + let writer = SharedWriter::default(); + let captured = writer.buffer.clone(); + let _subscriber = tracing::subscriber::set_default( + tracing_subscriber::fmt() + .without_time() + .with_ansi(false) + .with_writer(writer) + .finish(), + ); + assert!(tracing::enabled!(target: HTTP_SERVER_LOG_TARGET, Level::WARN)); + let inner = tower::service_fn(move |_: HttpRequest<()>| { + std::future::ready(if fail_request { + Err(io::Error::other("test service failure")) + } else { + Ok(Response::new(())) + }) + }); + let mut service = RequestLoggingLayer.layer(inner); + let mut response = service.call(HttpRequest::new(())); + tokio::task::yield_now().await; + let Poll::Ready(result) = futures::poll!(&mut response) else { + panic!("service should complete immediately"); + }; + assert_eq!(result.is_err(), fail_request); + // Retaining a completed future must not retain an active watchdog. + tokio::time::advance(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD + Duration::from_millis(1)).await; + tokio::task::yield_now().await; + + let output = String::from_utf8(captured.lock().expect("logs").clone()).expect("UTF-8 logs"); + assert!(!output.contains(HTTP_REQUEST_INFLIGHT_SLOW_EVENT), "{output}"); + let expected_event = if fail_request { + HTTP_REQUEST_FAILED_EVENT + } else { + HTTP_REQUEST_COMPLETED_EVENT + }; + assert_eq!(output.matches(expected_event).count(), 1, "{output}"); + drop(response); + } + } + + #[tokio::test(start_paused = true)] + async fn request_logging_watchdog_warns_once_for_pending_request() { + let writer = SharedWriter::default(); + let captured = writer.buffer.clone(); + let _subscriber = tracing::subscriber::set_default( + tracing_subscriber::fmt() + .without_time() + .with_ansi(false) + .with_writer(writer) + .finish(), + ); + assert!(tracing::enabled!(target: HTTP_SERVER_LOG_TARGET, Level::WARN)); + let inner = tower::service_fn(|_: HttpRequest<()>| std::future::pending::, Infallible>>()); + let mut service = RequestLoggingLayer.layer(inner); + let mut response = service.call(HttpRequest::new(())); + assert!(futures::poll!(&mut response).is_pending()); + tokio::task::yield_now().await; + tokio::time::advance(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD - Duration::from_millis(1)).await; + tokio::task::yield_now().await; + assert!(captured.lock().expect("logs").is_empty()); + + tokio::time::advance(Duration::from_millis(1)).await; + tokio::task::yield_now().await; + let output = String::from_utf8(captured.lock().expect("logs").clone()).expect("UTF-8 logs"); + assert_eq!(output.matches(HTTP_REQUEST_INFLIGHT_SLOW_EVENT).count(), 1, "{output}"); + assert!(output.contains("response_pending"), "{output}"); + + tokio::time::advance(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD).await; + tokio::task::yield_now().await; + let output = String::from_utf8(captured.lock().expect("logs").clone()).expect("UTF-8 logs"); + assert_eq!(output.matches(HTTP_REQUEST_INFLIGHT_SLOW_EVENT).count(), 1, "{output}"); + drop(response); + } + #[tokio::test] async fn request_logging_bounds_s3_failure_bursts_without_losing_counts_or_leaking_queries() { use rustfs_io_metrics::s3_http_metrics::s3_http_metrics_snapshot; diff --git a/rustfs/src/server/mod.rs b/rustfs/src/server/mod.rs index 7f919d6f7..4841bf4fe 100644 --- a/rustfs/src/server/mod.rs +++ b/rustfs/src/server/mod.rs @@ -29,6 +29,7 @@ pub(crate) mod runtime_sources; mod service_state; mod ssec_transport; pub mod tls_material; +pub(crate) mod tls_publish; #[cfg(feature = "http3")] mod http3; @@ -85,6 +86,7 @@ pub(crate) use readiness::collect_node_readiness_report; pub use readiness::publish_ready_when_runtime_ready; pub(crate) use readiness::snapshot_dependency_readiness_report; pub(crate) use readiness::{collect_cluster_read_health_report, collect_cluster_write_health_report}; +pub(crate) use ssec_transport::NativeTlsTransport; pub use crate::shared_types::RemoteAddr; diff --git a/rustfs/src/server/rate_limit.rs b/rustfs/src/server/rate_limit.rs index 483b805a0..ca39b8d28 100644 --- a/rustfs/src/server/rate_limit.rs +++ b/rustfs/src/server/rate_limit.rs @@ -301,14 +301,16 @@ impl RateLimiter { /// Resolve the client identity for rate limiting. /// /// Only trusted sources are consulted: `ClientInfo` is inserted by the -/// trusted-proxy layer after validating the proxy chain, and `RemoteAddr` is -/// the raw socket peer. Spoofable headers (`X-Forwarded-For`, `X-Real-IP`) -/// are deliberately never read here. +/// trusted-proxy layer after validating the proxy chain, and `Option` +/// carries the socket peer injected by the external HTTP stack. Spoofable +/// headers (`X-Forwarded-For`, `X-Real-IP`) are deliberately never read here. pub(crate) fn client_ip(req: &Request) -> Option { if let Some(info) = req.extensions().get::() { return Some(info.real_ip); } - req.extensions().get::().map(|remote| remote.0.ip()) + req.extensions() + .get::>() + .and_then(|remote| remote.map(|remote| remote.0.ip())) } /// Resolve the host a request addresses, for virtual-hosted-style bucket @@ -929,7 +931,7 @@ mod tests { fn request_from(peer: IpAddr, path: &str) -> Request<()> { let mut req = Request::builder().uri(path).body(()).expect("request"); - req.extensions_mut().insert(RemoteAddr(SocketAddr::new(peer, 51000))); + req.extensions_mut().insert(Some(RemoteAddr(SocketAddr::new(peer, 51000)))); req } @@ -1047,7 +1049,11 @@ mod tests { async fn missing_client_identity_fails_open() { let mut service = service_with_quota(60, 1); for _ in 0..3 { - let req = Request::builder().uri("/bucket/object").body(()).expect("request"); + let mut req = Request::builder().uri("/bucket/object").body(()).expect("request"); + req.extensions_mut().insert(None::); + req.headers_mut() + .insert("x-forwarded-for", HeaderValue::from_static("203.0.113.1")); + req.headers_mut().insert("x-real-ip", HeaderValue::from_static("203.0.113.2")); assert_eq!(service.call(req).await.expect("ok").status(), StatusCode::OK); } } diff --git a/rustfs/src/server/readiness.rs b/rustfs/src/server/readiness.rs index 712522e1a..5de55bb7e 100644 --- a/rustfs/src/server/readiness.rs +++ b/rustfs/src/server/readiness.rs @@ -237,7 +237,10 @@ pub async fn publish_ready_when_runtime_ready( wait_for_runtime_readiness_with( startup_runtime_readiness_max_wait(), STARTUP_RUNTIME_READINESS_POLL_INTERVAL, - collect_node_readiness, + // Publish service readiness at the same per-set read and shared-lock + // boundary reported by the node probe. Mutation paths independently + // require write quorum and a writable pool-metadata state. + || async { collect_node_readiness_report().await.readiness }, |dependency_readiness| { readiness.mark_stage(rustfs_common::SystemStage::FullReady); mark_get_metadata_read_version_coalescing_service_ready(); @@ -1060,20 +1063,6 @@ fn cluster_health_timeout_report() -> DependencyReadinessReport { } } -async fn collect_node_readiness() -> DependencyReadiness { - // Startup publication retains its local dependency gate. Runtime probes - // additionally report storage quorum without changing S3 admission. - let storage = node_pool_meta_write_readiness().await; - let readiness = DependencyReadiness { - storage_ready: storage.ready, - iam_ready: runtime_sources::current_iam_ready(), - lock_quorum_ready: collect_lock_quorum_status().await.ready, - peer_health_ready: collect_peer_health_readiness(), - }; - record_readiness_report(&dependency_readiness_report_from_write_status(readiness, storage)); - readiness -} - pub async fn collect_cluster_read_dependency_readiness_report() -> DependencyReadinessReport { let iam_ready_raw = runtime_sources::current_iam_ready(); let storage_ready = collect_storage_read_readiness_uncached().await; @@ -1116,14 +1105,6 @@ async fn collect_lock_quorum_observation() -> LockQuorumObservation { } } -async fn node_pool_meta_write_readiness() -> StorageWriteReadinessStatus { - if let Some(store) = runtime_sources::current_object_store_handle() { - return pool_metadata_write_readiness(store.pool_meta_write_status().await); - } - - StorageWriteReadinessStatus::default() -} - async fn collect_storage_write_readiness_uncached() -> StorageWriteReadinessStatus { if let Some(store) = runtime_sources::current_object_store_handle() { let metadata = pool_metadata_write_readiness(store.pool_meta_write_status().await); diff --git a/rustfs/src/server/ssec_transport.rs b/rustfs/src/server/ssec_transport.rs index bcda8f08a..bbe7adaba 100644 --- a/rustfs/src/server/ssec_transport.rs +++ b/rustfs/src/server/ssec_transport.rs @@ -63,6 +63,10 @@ pub(crate) const METRIC_SSEC_PLAINTEXT_REQUESTS_TOTAL: &str = "rustfs_ssec_plain type BoxError = Box; type BoxBody = http_body_util::combinators::UnsyncBoxBody; +/// Set only by the listener stack; proxy protocol assertions do not establish this marker. +#[derive(Clone, Copy)] +pub(crate) struct NativeTlsTransport(pub(crate) bool); + /// Every header that carries, or announces, a customer-provided key: the /// object's own triple and the copy-source triple that CopyObject and /// UploadPartCopy use to read an SSE-C source. @@ -191,7 +195,8 @@ where self.inner.poll_ready(cx) } - fn call(&mut self, req: Request) -> Self::Future { + fn call(&mut self, mut req: Request) -> Self::Future { + req.extensions_mut().insert(NativeTlsTransport(self.connection_is_tls)); if !carries_ssec_headers(req.headers()) || is_secure_transport(self.connection_is_tls, req.extensions().get::()) { diff --git a/rustfs/src/server/tls_material.rs b/rustfs/src/server/tls_material.rs index 4d6239eb2..4d5522409 100644 --- a/rustfs/src/server/tls_material.rs +++ b/rustfs/src/server/tls_material.rs @@ -26,19 +26,20 @@ use rustfs_config::{ DEFAULT_SERVER_MTLS_ENABLE, DEFAULT_TLS_KEYLOG, DEFAULT_TLS_RELOAD_ENABLE, DEFAULT_TLS_RELOAD_INTERVAL, DEFAULT_TRUST_LEAF_CERT_AS_CA, DEFAULT_TRUST_SYSTEM_CA, ENV_MTLS_CLIENT_CERT, ENV_MTLS_CLIENT_KEY, ENV_RUSTFS_EXTRA_CA_CERT, ENV_SERVER_MTLS_ENABLE, ENV_TLS_KEYLOG, ENV_TLS_RELOAD_ENABLE, ENV_TLS_RELOAD_INTERVAL, ENV_TRUST_LEAF_CERT_AS_CA, - ENV_TRUST_SYSTEM_CA, RUSTFS_CA_CERT, RUSTFS_CLIENT_CA_CERT_FILENAME, RUSTFS_CLIENT_CERT_FILENAME, RUSTFS_CLIENT_KEY_FILENAME, - RUSTFS_TLS_CERT, -}; -use rustfs_tls_runtime::{ - ServerTlsMaterial as RuntimeServerTlsMaterial, TlsGeneration, TlsSource, WebPkiClientVerifierOptions, - build_webpki_client_verifier, create_multi_cert_resolver, + ENV_TRUST_SYSTEM_CA, RUSTFS_CLIENT_CERT_FILENAME, RUSTFS_CLIENT_KEY_FILENAME, }; +#[cfg(test)] +use rustfs_config::{RUSTFS_CA_CERT, RUSTFS_CLIENT_CA_CERT_FILENAME}; +use rustfs_tls_runtime::{ServerTlsMaterial as RuntimeServerTlsMaterial, TlsGeneration, TlsSource, create_multi_cert_resolver}; +#[cfg(test)] +use rustfs_tls_runtime::{WebPkiClientVerifierOptions, build_webpki_client_verifier}; use rustfs_utils::{get_env_bool, get_env_opt_str}; use rustls::pki_types::{CertificateDer, PrivateKeyDer, pem::PemObject}; +use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::io::Cursor; use std::path::{Path, PathBuf}; -use std::sync::Arc; use std::sync::RwLock; +use std::sync::{Arc, LazyLock, Mutex, Weak}; use std::time::Duration; use tokio_rustls::TlsAcceptor; use tracing::{debug, info, warn}; @@ -75,11 +76,20 @@ pub async fn load_tls_material(tls_path: &str) -> Result Result, tls_dir: &Path, @@ -100,76 +111,166 @@ pub(crate) async fn build_acceptor_from_loaded( ) .map_err(|e| TlsMaterialError::Io(format!("build mTLS verifier: {e}")))?; - match server { - Some(RuntimeServerTlsMaterial::SingleCert { certs, key }) => { - let config = build_server_config(ServerCertSource::SingleCert { certs, key }, mtls_verifier, ServerProtocol::Tcp)?; - info!( - component = LOG_COMPONENT_TLS, - subsystem = LOG_SUBSYSTEM_TLS, - event = "tls_acceptor_created", - mode = "single_cert", - "TLS acceptor created" - ); - let acceptor = Arc::new(TlsAcceptor::from(Arc::new(config))); - Ok(Some(Arc::new(TlsAcceptorHolder::new(acceptor)))) + build_acceptor_with_verifier(server, mtls_verifier) +} + +fn snapshot_client_verifier( + snapshot: &rustfs_tls_runtime::TlsMaterialSnapshot, +) -> Result>, TlsMaterialError> { + if !get_env_bool(ENV_SERVER_MTLS_ENABLE, DEFAULT_SERVER_MTLS_ENABLE) { + return Ok(None); + } + let pem = snapshot + .server_client_ca_pem + .as_deref() + .ok_or_else(|| TlsMaterialError::Parse("mTLS client trust is missing".into()))?; + let mut reader = Cursor::new(pem); + let mut roots = rustls::RootCertStore::empty(); + for cert in CertificateDer::pem_reader_iter(&mut reader) { + roots + .add(cert.map_err(|e| TlsMaterialError::Parse(e.to_string()))?) + .map_err(|e| TlsMaterialError::Parse(e.to_string()))?; + } + let verifier = rustls::server::WebPkiClientVerifier::builder(Arc::new(roots)) + .build() + .map_err(|e| TlsMaterialError::Parse(e.to_string()))?; + #[cfg(test)] + TLS_PREPARE_COUNTS.with(|counts| { + let (roots, keys) = counts.get(); + counts.set((roots + 1, keys)); + }); + Ok(Some(verifier)) +} + +#[cfg(test)] +thread_local! { + static TLS_PREPARE_COUNTS: std::cell::Cell<(usize, usize)> = const { std::cell::Cell::new((0, 0)) }; +} + +/// Expensive certificate signing keys and client trust are prepared once per generation. +/// TCP and QUIC retain this shared resolver while owning their protocol settings. +#[derive(Clone)] +pub(crate) struct PreparedServerTls { + resolver: Arc, + verifier: Option>, + provider: Arc, +} + +impl PreparedServerTls { + pub(crate) fn from_snapshot(snapshot: &rustfs_tls_runtime::TlsMaterialSnapshot) -> Result, TlsMaterialError> { + Self::from_material(snapshot.server.as_ref(), snapshot_client_verifier(snapshot)?) + } + + fn from_material( + server: Option<&RuntimeServerTlsMaterial>, + verifier: Option>, + ) -> Result, TlsMaterialError> { + let Some(server) = server else { + return Ok(None); + }; + let provider = rustls::ServerConfig::builder().crypto_provider().clone(); + let (resolver, validity): (Arc, _) = match server { + RuntimeServerTlsMaterial::SingleCert { certs, key } => { + let validity = certificate_validities(std::iter::once(certs.as_slice()))?; + let certified = rustls::sign::CertifiedKey::from_der(certs.clone(), key.clone_key(), &provider) + .map_err(|e| TlsMaterialError::Io(format!("configure single cert: {e}")))?; + (Arc::new(rustls::sign::SingleCertAndKey::from(certified)), validity) + } + RuntimeServerTlsMaterial::MultiCert { cert_key_pairs } => { + let validity = certificate_validities(cert_key_pairs.values().map(|(certs, _)| certs.as_slice()))?; + let pairs = cert_key_pairs + .iter() + .map(|(name, (certs, key))| (name.clone(), (certs.clone(), key.clone_key()))) + .collect(); + let resolver = create_multi_cert_resolver(pairs) + .map_err(|e| TlsMaterialError::Parse(format!("build multi-cert resolver: {e}")))?; + (Arc::new(resolver), validity) + } + }; + #[cfg(test)] + TLS_PREPARE_COUNTS.with(|counts| { + let (roots, keys) = counts.get(); + counts.set((roots, keys + 1)); + }); + Ok(Some(Self { + resolver: Arc::new(ValidityCheckedResolver { + inner: resolver, + validity, + }), + verifier, + provider, + })) + } + + pub(crate) fn tcp_acceptor(&self) -> Result, TlsMaterialError> { + let config = self.server_config(ServerProtocol::Tcp)?; + Ok(Arc::new(TlsAcceptorHolder::new(Arc::new(TlsAcceptor::from(Arc::new(config)))))) + } + + #[cfg(feature = "http3")] + pub(crate) fn http3_config(&self) -> Result { + let crypto = quinn::crypto::rustls::QuicServerConfig::try_from(self.server_config(ServerProtocol::Http3)?) + .map_err(|e| TlsMaterialError::Io(format!("configure QUIC TLS: {e}")))?; + Ok(quinn::ServerConfig::with_crypto(Arc::new(crypto))) + } + + fn server_config(&self, protocol: ServerProtocol) -> Result { + let versions = match protocol { + ServerProtocol::Tcp => rustls::DEFAULT_VERSIONS, + #[cfg(feature = "http3")] + ServerProtocol::Http3 => &[&rustls::version::TLS13], + }; + let builder = rustls::ServerConfig::builder_with_provider(self.provider.clone()) + .with_protocol_versions(versions) + .map_err(|e| TlsMaterialError::Io(format!("configure TLS protocol: {e}")))?; + let mut config = if let Some(verifier) = &self.verifier { + builder + .with_client_cert_verifier(verifier.clone()) + .with_cert_resolver(self.resolver.clone()) + } else { + builder.with_no_client_auth().with_cert_resolver(self.resolver.clone()) + }; + config.alpn_protocols = match protocol { + ServerProtocol::Tcp => vec![b"h2".to_vec(), b"http/1.1".to_vec(), b"http/1.0".to_vec()], + #[cfg(feature = "http3")] + ServerProtocol::Http3 => vec![b"h3".to_vec()], + }; + #[cfg(feature = "http3")] + if matches!(protocol, ServerProtocol::Http3) { + config.max_early_data_size = 0; } - Some(RuntimeServerTlsMaterial::MultiCert { cert_key_pairs }) => { - let resolver = create_multi_cert_resolver(cert_key_pairs) - .map_err(|e| TlsMaterialError::Parse(format!("build multi-cert resolver: {e}")))?; - let config = build_server_config(ServerCertSource::Resolver(Arc::new(resolver)), mtls_verifier, ServerProtocol::Tcp)?; - info!( - component = LOG_COMPONENT_TLS, - subsystem = LOG_SUBSYSTEM_TLS, - event = "tls_acceptor_created", - mode = "sni_resolver", - "TLS acceptor created" - ); - let acceptor = Arc::new(TlsAcceptor::from(Arc::new(config))); - Ok(Some(Arc::new(TlsAcceptorHolder::new(acceptor)))) + config.session_storage = rustls::server::ServerSessionMemoryCache::new(10000); + if tls_key_log() { + config.key_log = Arc::new(rustls::KeyLogFile::new()); } - None => Ok(None), + Ok(config) } } -#[cfg(feature = "http3")] +#[cfg(test)] +fn build_acceptor_with_verifier( + server: Option, + mtls_verifier: Option>, +) -> Result>, TlsMaterialError> { + PreparedServerTls::from_material(server.as_ref(), mtls_verifier)? + .map(|prepared| prepared.tcp_acceptor()) + .transpose() +} + +#[cfg(all(test, feature = "http3"))] pub(crate) async fn build_http3_server_config_from_loaded( server: Option<&RuntimeServerTlsMaterial>, tls_dir: &Path, ) -> Result, TlsMaterialError> { - let Some(server) = server else { - return Ok(None); - }; - let mtls_verifier = build_webpki_client_verifier( WebPkiClientVerifierOptions::builder(tls_dir, RUSTFS_CLIENT_CA_CERT_FILENAME, RUSTFS_CA_CERT) .enabled(get_env_bool(ENV_SERVER_MTLS_ENABLE, DEFAULT_SERVER_MTLS_ENABLE)) .build(), ) .map_err(|e| TlsMaterialError::Io(format!("build mTLS verifier: {e}")))?; - - let source = match server { - RuntimeServerTlsMaterial::SingleCert { certs, key } => ServerCertSource::SingleCert { - certs: certs.clone(), - key: key.clone_key(), - }, - RuntimeServerTlsMaterial::MultiCert { cert_key_pairs } => { - let pairs = cert_key_pairs - .iter() - .map(|(domain, (certs, key))| (domain.clone(), (certs.clone(), key.clone_key()))) - .collect(); - - let resolver = create_multi_cert_resolver(pairs) - .map_err(|e| TlsMaterialError::Parse(format!("build HTTP/3 SNI resolver: {e}")))?; - - ServerCertSource::Resolver(Arc::new(resolver)) - } - }; - - let tls = build_server_config(source, mtls_verifier, ServerProtocol::Http3)?; - let crypto = quinn::crypto::rustls::QuicServerConfig::try_from(tls) - .map_err(|e| TlsMaterialError::Io(format!("configure QUIC TLS: {e}")))?; - - Ok(Some(quinn::ServerConfig::with_crypto(Arc::new(crypto)))) + PreparedServerTls::from_material(server, mtls_verifier)? + .map(|prepared| prepared.http3_config()) + .transpose() } // ── Outbound Enrichment ── @@ -179,18 +280,30 @@ pub(crate) async fn build_http3_server_config_from_loaded( /// 1. Optional: server leaf cert as root CA (`RUSTFS_TRUST_LEAF_CERT_AS_CA`) /// 2. Optional: system root CAs from platform paths (`RUSTFS_TRUST_SYSTEM_CA`) /// 3. Optional: mTLS identity from env-var-overridden paths -async fn enrich_outbound(outbound: &mut rustfs_tls_runtime::OutboundTlsMaterial, tls_dir: &Path) -> Result<(), TlsMaterialError> { - // 1. Optional: load leaf certs as root CAs - if get_env_bool(ENV_TRUST_LEAF_CERT_AS_CA, DEFAULT_TRUST_LEAF_CERT_AS_CA) - && load_cert_file_by_name(tls_dir, RUSTFS_TLS_CERT, &mut outbound.root_ca_pem).await - { - info!( - component = LOG_COMPONENT_TLS, - subsystem = LOG_SUBSYSTEM_TLS, - event = "tls_root_ca_enriched", - source = "leaf_certificate", - "TLS root CA set enriched" - ); +async fn enrich_outbound(snapshot: &mut rustfs_tls_runtime::TlsMaterialSnapshot) -> Result<(), TlsMaterialError> { + let tls_dir = &snapshot.source.base_dir; + let outbound = &mut snapshot.outbound; + if get_env_bool(ENV_TRUST_LEAF_CERT_AS_CA, DEFAULT_TRUST_LEAF_CERT_AS_CA) { + let chains: Vec<_> = match &snapshot.server { + Some(RuntimeServerTlsMaterial::SingleCert { certs, .. }) => vec![certs], + Some(RuntimeServerTlsMaterial::MultiCert { cert_key_pairs }) => { + cert_key_pairs.values().map(|(certs, _)| certs).collect() + } + None => Vec::new(), + }; + let mut certificates = BTreeSet::new(); + for chain in chains { + certificates.extend(chain.iter().map(|cert| cert.as_ref().to_vec())); + } + for certificate in certificates { + outbound.root_ca_pem.extend_from_slice( + format!( + "-----BEGIN CERTIFICATE-----\n{}\n-----END CERTIFICATE-----\n", + base64_simd::STANDARD.encode_to_string(&certificate), + ) + .as_bytes(), + ); + } } // 2. Optional: load system root CAs @@ -260,19 +373,6 @@ async fn load_mtls_identity_from_overridden_paths( None => tls_dir.join(RUSTFS_CLIENT_KEY_FILENAME), }; - if !client_cert_path.exists() || !client_key_path.exists() { - info!( - component = LOG_COMPONENT_TLS, - subsystem = LOG_SUBSYSTEM_TLS, - event = "mtls_identity_state", - state = "not_configured", - cert_path = ?client_cert_path, - key_path = ?client_key_path, - "mTLS client identity not configured" - ); - return Ok(None); - } - let cert_pem = tokio::fs::read(&client_cert_path) .await .map_err(|e| TlsMaterialError::Io(format!("read client cert {client_cert_path:?}: {e}")))?; @@ -281,11 +381,13 @@ async fn load_mtls_identity_from_overridden_paths( .map_err(|e| TlsMaterialError::Io(format!("read client key {client_key_path:?}: {e}")))?; let mut reader = std::io::Cursor::new(&cert_pem); - if CertificateDer::pem_reader_iter(&mut reader).next().is_none() { - return Err(TlsMaterialError::Parse("no valid certificate in client cert PEM".into())); - } + let certs = CertificateDer::pem_reader_iter(&mut reader) + .collect::, _>>() + .map_err(|e| TlsMaterialError::Parse(format!("invalid client certificate: {e}")))?; let mut reader = std::io::Cursor::new(&key_pem); - PrivateKeyDer::from_pem_reader(&mut reader).map_err(|e| TlsMaterialError::Parse(format!("invalid client key PEM: {e}")))?; + let key = + PrivateKeyDer::from_pem_reader(&mut reader).map_err(|e| TlsMaterialError::Parse(format!("invalid client key: {e}")))?; + rustfs_tls_runtime::validate_certified_key(&certs, &key, "client identity override", false).map_err(map_runtime_tls_error)?; info!( component = LOG_COMPONENT_TLS, @@ -410,57 +512,47 @@ async fn load_cert_file(path: &Path, pem_data: &mut Vec, desc: &str) -> bool } } -/// Search for and load certificate files matching `cert_name` in the directory -/// and one level of subdirectories. -/// Returns `true` if at least one matching file was loaded. -async fn load_cert_file_by_name(dir: &Path, cert_name: &str, pem_data: &mut Vec) -> bool { - let Ok(mut rd) = tokio::fs::read_dir(dir).await else { - debug!( - component = LOG_COMPONENT_TLS, - subsystem = LOG_SUBSYSTEM_TLS, - event = "tls_material_directory_state", - path = %dir.display(), - state = "missing", - "TLS material directory state changed" +/// Validity windows indexed by leaf certificate DER. +type CertificateValidities = HashMap, (i64, i64)>; + +fn certificate_validities<'a>( + chains: impl Iterator]>, +) -> Result { + let mut validity = HashMap::new(); + for chain in chains { + let leaf = chain + .first() + .ok_or_else(|| TlsMaterialError::Parse("empty certificate chain".into()))?; + let (_, parsed) = x509_parser::parse_x509_certificate(leaf.as_ref()) + .map_err(|e| TlsMaterialError::Parse(format!("invalid certificate: {e}")))?; + validity.insert( + leaf.as_ref().to_vec(), + (parsed.validity().not_before.timestamp(), parsed.validity().not_after.timestamp()), ); - return false; - }; - - let mut loaded = false; - while let Ok(Some(entry)) = rd.next_entry().await { - let Ok(ft) = entry.file_type().await else { continue }; - - if ft.is_file() { - let fname = entry.file_name().to_string_lossy().to_string(); - if fname == cert_name && load_cert_file(&entry.path(), pem_data, "certificate").await { - loaded = true; - } - } else if ft.is_dir() - && let Ok(mut sub_rd) = tokio::fs::read_dir(&entry.path()).await - { - while let Ok(Some(sub_entry)) = sub_rd.next_entry().await { - if let Ok(sub_ft) = sub_entry.file_type().await - && sub_ft.is_file() - { - let fname = sub_entry.file_name().to_string_lossy().to_string(); - if fname == cert_name && load_cert_file(&sub_entry.path(), pem_data, "certificate").await { - loaded = true; - } - } - } - } } - loaded + Ok(validity) } -// ── Server Config Construction ── +#[derive(Debug)] +struct ValidityCheckedResolver { + inner: Arc, + validity: CertificateValidities, +} -enum ServerCertSource { - Resolver(Arc), - SingleCert { - certs: Vec>, - key: PrivateKeyDer<'static>, - }, +impl rustls::server::ResolvesServerCert for ValidityCheckedResolver { + fn resolve(&self, hello: rustls::server::ClientHello<'_>) -> Option> { + let certificate = self.inner.resolve(hello)?; + let leaf = certificate.cert.first()?; + let (not_before, not_after) = self.validity.get(leaf.as_ref())?; + let now = i64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .ok()? + .as_secs(), + ) + .ok()?; + (*not_before <= now && now < *not_after).then_some(certificate) + } } enum ServerProtocol { @@ -469,62 +561,6 @@ enum ServerProtocol { Http3, } -fn build_server_config( - cert_source: ServerCertSource, - mtls_verifier: Option>, - protocol: ServerProtocol, -) -> Result { - let builder = match protocol { - ServerProtocol::Tcp => rustls::ServerConfig::builder(), - #[cfg(feature = "http3")] - ServerProtocol::Http3 => rustls::ServerConfig::builder_with_protocol_versions(&[&rustls::version::TLS13]), - }; - - let mut config = match cert_source { - ServerCertSource::Resolver(resolver) => { - if let Some(verifier) = mtls_verifier { - builder.with_client_cert_verifier(verifier).with_cert_resolver(resolver) - } else { - builder.with_no_client_auth().with_cert_resolver(resolver) - } - } - ServerCertSource::SingleCert { certs, key } => { - if let Some(verifier) = mtls_verifier { - builder - .with_client_cert_verifier(verifier) - .with_single_cert(certs, key) - .map_err(|e| TlsMaterialError::Io(format!("configure single cert with mTLS: {e}")))? - } else { - builder - .with_no_client_auth() - .with_single_cert(certs, key) - .map_err(|e| TlsMaterialError::Io(format!("configure single cert: {e}")))? - } - } - }; - - config.alpn_protocols = match protocol { - ServerProtocol::Tcp => { - vec![b"h2".to_vec(), b"http/1.1".to_vec(), b"http/1.0".to_vec()] - } - #[cfg(feature = "http3")] - ServerProtocol::Http3 => vec![b"h3".to_vec()], - }; - - #[cfg(feature = "http3")] - if matches!(protocol, ServerProtocol::Http3) { - config.max_early_data_size = 0; - } - - config.session_storage = rustls::server::ServerSessionMemoryCache::new(10000); - - if tls_key_log() { - config.key_log = Arc::new(rustls::KeyLogFile::new()); - } - - Ok(config) -} - fn tls_key_log() -> bool { get_env_bool(ENV_TLS_KEYLOG, DEFAULT_TLS_KEYLOG) } @@ -647,26 +683,30 @@ impl TlsAcceptorHolder { } // Build both configurations before publishing either listener's new certificate. +#[cfg(all(test, feature = "http3"))] async fn reload_acceptors( server: Option, tls_dir: &Path, holder: &TlsAcceptorHolder, #[cfg(feature = "http3")] http3_endpoint: Option<&quinn::Endpoint>, ) -> Result { - #[cfg(feature = "http3")] + let verifier = build_webpki_client_verifier( + WebPkiClientVerifierOptions::builder(tls_dir, RUSTFS_CLIENT_CA_CERT_FILENAME, RUSTFS_CA_CERT) + .enabled(get_env_bool(ENV_SERVER_MTLS_ENABLE, DEFAULT_SERVER_MTLS_ENABLE)) + .build(), + ) + .map_err(|e| TlsMaterialError::Io(format!("build mTLS verifier: {e}")))?; + let Some(prepared) = PreparedServerTls::from_material(server.as_ref(), verifier)? else { + return Ok(false); + }; let http3_config = if http3_endpoint.is_some() { - let config = build_http3_server_config_from_loaded(server.as_ref(), tls_dir).await?; - let Some(mut config) = config else { - return Ok(false); - }; + let mut config = prepared.http3_config()?; config.migration(false); Some(config) } else { None }; - let Some(new_holder) = build_acceptor_from_loaded(server, tls_dir).await? else { - return Ok(false); - }; + let new_holder = prepared.tcp_acceptor()?; #[cfg(feature = "http3")] if let (Some(endpoint), Some(config)) = (http3_endpoint, http3_config) { endpoint.set_server_config(Some(config)); @@ -680,146 +720,327 @@ async fn reload_acceptors( /// Spawn a background task that periodically checks for TLS certificate changes. /// Single load per tick: loads once via tls-runtime, enriches, publishes outbound, /// and builds acceptor — no double reads. +#[derive(Clone)] +struct ReloadTarget { + holder: Arc, + #[cfg(feature = "http3")] + http3_endpoint: Option, +} + +struct ReloadState { + targets: HashMap, + accepted: Arc, + tcp: Arc, + prepared: PreparedServerTls, + #[cfg(feature = "http3")] + quic: Option, +} + +struct ReloadGroup { + required_startup_files: BTreeSet, + state: Mutex, + changed: tokio::sync::Notify, + next_id: std::sync::atomic::AtomicU64, + #[cfg(test)] + pass_completed: tokio::sync::Notify, +} + +// Lock order is registry, then group state, then acceptor. Neither guard crosses an await. +static RELOAD_GROUPS: LazyLock>>> = LazyLock::new(|| Mutex::new(HashMap::new())); + +pub(crate) fn observed_server_snapshot(tls_path: &str) -> Option> { + let registry = RELOAD_GROUPS.lock().unwrap_or_else(|e| e.into_inner()); + let group = registry.get(tls_path)?.upgrade()?; + let state = group.state.lock().unwrap_or_else(|e| e.into_inner()); + (!state.targets.is_empty()).then(|| state.accepted.clone()) +} + +/// Resolve the source actually registered by listener configuration, including CLI overrides. +pub(crate) fn active_server_tls_path() -> Option { + let registry = RELOAD_GROUPS.lock().unwrap_or_else(|e| e.into_inner()); + let mut paths = registry.iter().filter_map(|(path, group)| { + let group = group.upgrade()?; + let state = group.state.lock().unwrap_or_else(|e| e.into_inner()); + (!state.targets.is_empty()).then(|| path.clone()) + }); + let path = paths.next()?; + paths.next().is_none().then_some(path) +} + +pub(crate) fn notify_server_publication(tls_path: &str) { + let registry = RELOAD_GROUPS.lock().unwrap_or_else(|e| e.into_inner()); + if let Some(group) = registry.get(tls_path).and_then(Weak::upgrade) { + group.changed.notify_one(); + } +} + +struct ReloadRegistration { + group: Arc, + id: u64, +} + +impl Drop for ReloadRegistration { + fn drop(&mut self) { + self.group + .state + .lock() + .unwrap_or_else(|e| e.into_inner()) + .targets + .remove(&self.id); + self.group.changed.notify_one(); + } +} + +fn server_names(server: &Option) -> Result>, TlsMaterialError> { + let chains: Vec<_> = match server { + Some(RuntimeServerTlsMaterial::SingleCert { certs, .. }) => vec![("default", certs)], + Some(RuntimeServerTlsMaterial::MultiCert { cert_key_pairs }) => cert_key_pairs + .iter() + .map(|(selector, (certs, _))| (selector.as_str(), certs)) + .collect(), + None => Vec::new(), + }; + let mut selectors = BTreeMap::new(); + for (selector, chain) in chains { + let leaf = chain + .first() + .ok_or_else(|| TlsMaterialError::Parse("empty certificate chain".into()))?; + selectors.insert(selector.to_string(), certificate_names(leaf)?); + } + Ok(selectors) +} + +pub(crate) fn certificate_names(leaf: &CertificateDer<'_>) -> Result, TlsMaterialError> { + let mut names = BTreeSet::new(); + let (_, parsed) = x509_parser::parse_x509_certificate(leaf.as_ref()) + .map_err(|e| TlsMaterialError::Parse(format!("invalid certificate: {e}")))?; + if let Some(san) = parsed + .subject_alternative_name() + .map_err(|e| TlsMaterialError::Parse(format!("invalid SAN: {e}")))? + { + for name in &san.value.general_names { + match name { + x509_parser::extensions::GeneralName::DNSName(name) => { + names.insert(format!("dns:{}", name.to_ascii_lowercase())); + } + x509_parser::extensions::GeneralName::IPAddress(bytes) => { + names.insert(format!("ip:{bytes:?}")); + } + _ => {} + } + } + } + Ok(names) +} + +pub(crate) fn validate_required_material_files( + required: &BTreeSet, + candidate: &rustfs_tls_runtime::TlsMaterialSnapshot, +) -> Result<(), TlsMaterialError> { + if !required.is_subset(&candidate.material_files) { + return Err(TlsMaterialError::Parse("required TLS source components disappeared".into())); + } + Ok(()) +} + +pub(crate) fn validate_candidate_replacement( + accepted: &rustfs_tls_runtime::TlsMaterialSnapshot, + candidate: &rustfs_tls_runtime::TlsMaterialSnapshot, +) -> Result<(), TlsMaterialError> { + validate_required_material_files(&accepted.material_files, candidate)?; + let old_names = server_names(&accepted.server)?; + let new_names = server_names(&candidate.server)?; + for (selector, names) in old_names { + if !new_names.get(&selector).is_some_and(|new| names.is_subset(new)) { + return Err(TlsMaterialError::Parse("required server certificate names disappeared".into())); + } + } + Ok(()) +} + +fn outbound_material_matches(a: &rustfs_tls_runtime::OutboundTlsMaterial, b: &rustfs_tls_runtime::OutboundTlsMaterial) -> bool { + a.root_ca_pem == b.root_ca_pem && a.mtls_identity == b.mtls_identity +} + pub(crate) fn spawn_reload_loop( tls_path: String, holder: Arc, + initial: rustfs_tls_runtime::TlsMaterialSnapshot, + prepared: PreparedServerTls, #[cfg(feature = "http3")] http3_endpoint: Option, -) -> Option> { - let enabled = get_env_bool(ENV_TLS_RELOAD_ENABLE, DEFAULT_TLS_RELOAD_ENABLE); - if !enabled { - debug!( - component = LOG_COMPONENT_TLS, - subsystem = LOG_SUBSYSTEM_TLS, - event = "tls_reload_state", - state = "disabled", - env_var = ENV_TLS_RELOAD_ENABLE, - "TLS reload state changed" - ); - return None; +) -> Result>, TlsMaterialError> { + if !get_env_bool(ENV_TLS_RELOAD_ENABLE, DEFAULT_TLS_RELOAD_ENABLE) { + return Ok(None); } + let required_startup_files = crate::startup_tls_material::required_startup_material_files(&tls_path); + let initial = Arc::new(initial); + let mut registry = RELOAD_GROUPS.lock().unwrap_or_else(|e| e.into_inner()); + registry.retain(|_, group| group.strong_count() != 0); + let existing = registry.get(&tls_path).and_then(Weak::upgrade); + let first = existing.is_none(); + let group = existing.unwrap_or_else(|| { + Arc::new(ReloadGroup { + required_startup_files, + state: Mutex::new(ReloadState { + targets: HashMap::new(), + accepted: initial.clone(), + tcp: holder.get(), + prepared, + #[cfg(feature = "http3")] + quic: None, + }), + changed: tokio::sync::Notify::new(), + next_id: std::sync::atomic::AtomicU64::new(1), + #[cfg(test)] + pass_completed: tokio::sync::Notify::new(), + }) + }); + let id = group.next_id.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + { + let mut state = group.state.lock().unwrap_or_else(|e| e.into_inner()); + #[cfg(feature = "http3")] + if let Some(endpoint) = &http3_endpoint { + if state.quic.is_none() { + let mut config = state.prepared.http3_config()?; + config.migration(false); + state.quic = Some(config); + } + endpoint.set_server_config(state.quic.clone()); + } + holder.swap(&TlsAcceptorHolder::new(state.tcp.clone())); + state.targets.insert( + id, + ReloadTarget { + holder, + #[cfg(feature = "http3")] + http3_endpoint, + }, + ); + } + registry.insert(tls_path.clone(), Arc::downgrade(&group)); + drop(registry); + if first { + tokio::spawn(run_reload_group(tls_path, group.clone())); + } + group.changed.notify_one(); + let registration = ReloadRegistration { group, id }; + Ok(Some(tokio::spawn(async move { + let _registration = registration; + std::future::pending::<()>().await; + }))) +} +async fn run_reload_group(tls_path: String, group: Arc) { let interval_secs = rustfs_utils::get_env_u64(ENV_TLS_RELOAD_INTERVAL, DEFAULT_TLS_RELOAD_INTERVAL).max(5); - let tls_source = TlsSource::from_directory(&tls_path); - - info!( - component = LOG_COMPONENT_TLS, - subsystem = LOG_SUBSYSTEM_TLS, - event = "tls_reload_state", - state = "enabled", - interval_secs, - "TLS reload state changed" - ); - - Some(tokio::spawn(async move { - let mut interval = tokio::time::interval(Duration::from_secs(interval_secs)); - let tls_dir = PathBuf::from(&tls_path); - loop { - interval.tick().await; - - match rustfs_tls_runtime::TlsMaterialSnapshot::load(&tls_source).await { - Ok(mut snapshot) => { - if let Err(e) = enrich_outbound(&mut snapshot.outbound, &tls_dir).await { - startup_runtime_sources::record_tls_reload_result("rustfs_server_reload_loop", "enrich_err", None, None); - warn!( - component = LOG_COMPONENT_TLS, - subsystem = LOG_SUBSYSTEM_TLS, - event = "tls_reload_failed", - stage = "enrich_outbound", - error = %e, - "TLS reload failed" - ); - continue; - } - - let generation = startup_runtime_sources::current_outbound_tls_generation().saturating_add(1); - startup_runtime_sources::publish_outbound_tls_state(TlsGeneration(generation), &snapshot.outbound).await; - startup_runtime_sources::record_tls_generation("rustfs_server_reload_loop", generation); - if !snapshot.outbound.root_ca_pem.is_empty() { - info!( - component = LOG_COMPONENT_TLS, - subsystem = LOG_SUBSYSTEM_TLS, - event = "tls_root_ca_enriched", - source = "reload_snapshot", - generation, - "TLS root CA set updated" - ); - } - - match reload_acceptors( - snapshot.server, - &tls_dir, - &holder, - #[cfg(feature = "http3")] - http3_endpoint.as_ref(), - ) - .await - { - Ok(true) => { - info!( - component = LOG_COMPONENT_TLS, - subsystem = LOG_SUBSYSTEM_TLS, - event = "tls_reload_state", - state = "reloaded", - generation, - "TLS reload state changed" - ); - startup_runtime_sources::record_tls_reload_result( - "rustfs_server_reload_loop", - "ok", - None, - Some(generation), - ); - } - Ok(false) => { - startup_runtime_sources::record_tls_reload_skipped("rustfs_server_reload_loop", "no_acceptor"); - warn!( - component = LOG_COMPONENT_TLS, - subsystem = LOG_SUBSYSTEM_TLS, - event = "tls_reload_failed", - stage = "build_acceptor", - reason = "no_acceptor", - "TLS reload returned no acceptor" - ) - } - Err(e) => { - startup_runtime_sources::record_tls_reload_result( - "rustfs_server_reload_loop", - "acceptor_err", - None, - Some(generation), - ); - warn!( - component = LOG_COMPONENT_TLS, - subsystem = LOG_SUBSYSTEM_TLS, - event = "tls_reload_failed", - stage = "build_acceptor", - error = %e, - generation, - "TLS reload failed" - ) - } + let mut interval = tokio::time::interval(Duration::from_secs(interval_secs)); + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + let mut generation = 1_u64; + loop { + tokio::select! { + _ = interval.tick() => {}, + _ = group.changed.notified() => {}, + } + let (accepted, empty) = { + let state = group.state.lock().unwrap_or_else(|e| e.into_inner()); + (state.accepted.clone(), state.targets.is_empty()) + }; + if empty { + let mut registry = RELOAD_GROUPS.lock().unwrap_or_else(|e| e.into_inner()); + if group.state.lock().unwrap_or_else(|e| e.into_inner()).targets.is_empty() { + registry.remove(&tls_path); + return; + } + continue; + } + let result = async { + let candidate = load_tls_material(&tls_path).await?; + validate_candidate_replacement(&accepted, &candidate)?; + validate_required_material_files(&group.required_startup_files, &candidate)?; + // Listener initialization can load a newer bundle than startup published. + let published = startup_runtime_sources::current_outbound_tls_state().await; + let required_roots = + !accepted.outbound.root_ca_pem.is_empty() || published.root_ca_pem.as_ref().is_some_and(|pem| !pem.is_empty()); + let required_identity = accepted.outbound.mtls_identity.is_some() || published.mtls_identity.is_some(); + if (required_roots && candidate.outbound.root_ca_pem.is_empty()) + || (required_identity && candidate.outbound.mtls_identity.is_none()) + { + return Err(TlsMaterialError::Parse("required outbound TLS material disappeared".into())); + } + let outbound_changed = !outbound_material_matches(&accepted.outbound, &candidate.outbound) + || published.root_ca_pem.as_deref().unwrap_or_default() != candidate.outbound.root_ca_pem.as_slice() + || published.mtls_identity != candidate.outbound.mtls_identity; + if accepted.fingerprint == candidate.fingerprint && !outbound_changed { + startup_runtime_sources::record_tls_reload_skipped("rustfs_server_reload_loop", "unchanged"); + return Ok(None); + } + let prepared = PreparedServerTls::from_snapshot(&candidate)? + .ok_or_else(|| TlsMaterialError::Parse("server certificate is missing".into()))?; + let tcp = prepared.tcp_acceptor()?; + #[cfg(feature = "http3")] + let quic = { + let mut config = prepared.http3_config()?; + config.migration(false); + config + }; + let candidate = Arc::new(candidate); + let listeners = { + let mut state = group.state.lock().unwrap_or_else(|e| e.into_inner()); + // Newly registered listeners join the same prepared generation under this lock. + for target in state.targets.values() { + #[cfg(feature = "http3")] + if let Some(endpoint) = &target.http3_endpoint { + endpoint.set_server_config(Some(quic.clone())); } + target.holder.swap(&tcp); } - Err(e) => { - startup_runtime_sources::record_tls_reload_result("rustfs_server_reload_loop", "load_err", None, None); - warn!( - component = LOG_COMPONENT_TLS, - subsystem = LOG_SUBSYSTEM_TLS, - event = "tls_reload_failed", - stage = "load_snapshot", - error = %e, - "TLS reload failed" - ); + state.tcp = tcp.get(); + state.prepared = prepared; + state.accepted = candidate.clone(); + #[cfg(feature = "http3")] + { + state.quic = Some(quic); } + state.targets.len() + }; + if outbound_changed { + let next = startup_runtime_sources::current_outbound_tls_generation().saturating_add(1); + startup_runtime_sources::publish_outbound_tls_state(TlsGeneration(next), &candidate.outbound).await; + } + Ok(Some(listeners)) + } + .await; + match result { + Ok(Some(listeners)) => { + generation = generation.saturating_add(1); + startup_runtime_sources::record_tls_generation("rustfs_server_reload_loop", generation); + startup_runtime_sources::record_tls_reload_result("rustfs_server_reload_loop", "ok", None, Some(generation)); + info!( + event = "tls_reload_state", + component = LOG_COMPONENT_TLS, + subsystem = LOG_SUBSYSTEM_TLS, + state = "reloaded", + generation, + listeners, + "TLS material updated" + ); + } + Ok(None) => {} + Err(error) => { + startup_runtime_sources::record_tls_reload_result("rustfs_server_reload_loop", "rejected", None, None); + warn!(event = "tls_reload_failed", component = LOG_COMPONENT_TLS, subsystem = LOG_SUBSYSTEM_TLS, + result = "rejected", error = %error, "TLS material update rejected"); } } - })) + #[cfg(test)] + group.pass_completed.notify_one(); + } } #[cfg(test)] mod tests { use super::*; use rcgen::CertifiedKey; + use rustfs_config::RUSTFS_TLS_CERT; use std::fs; use std::sync::Once; use tempfile::TempDir; @@ -839,6 +1060,507 @@ mod tests { fs::write(dir.join(rustfs_config::RUSTFS_TLS_KEY), signing_key.serialize_pem()).unwrap(); } + #[tokio::test] + #[serial_test::serial] + async fn prepared_generation_shares_crypto_and_serves_sni_with_mtls() { + ensure_rustls_crypto_provider(); + temp_env::async_with_vars([(ENV_SERVER_MTLS_ENABLE, Some("1"))], async { + let dir = TempDir::new().expect("TLS directory"); + write_test_cert_pair(dir.path(), "default.local"); + let sni_dir = dir.path().join("sni.example"); + fs::create_dir(&sni_dir).expect("SNI directory"); + write_test_cert_pair(&sni_dir, "sni.example"); + let mut ca_params = rcgen::CertificateParams::new(Vec::::new()).expect("CA parameters"); + ca_params.is_ca = rcgen::IsCa::Ca(rcgen::BasicConstraints::Unconstrained); + let ca_key = rcgen::KeyPair::generate().expect("CA key"); + let ca = ca_params.self_signed(&ca_key).expect("CA certificate"); + fs::write(dir.path().join(RUSTFS_CLIENT_CA_CERT_FILENAME), ca.pem()).expect("client trust"); + let mut client_params = rcgen::CertificateParams::new(vec!["client.example".into()]).expect("client parameters"); + client_params.extended_key_usages = vec![rcgen::ExtendedKeyUsagePurpose::ClientAuth]; + let client_key = rcgen::KeyPair::generate().expect("client key"); + let issuer = rcgen::Issuer::from_params(&ca_params, &ca_key); + let client_cert = client_params.signed_by(&client_key, &issuer).expect("client identity"); + let snapshot = rustfs_tls_runtime::TlsMaterialSnapshot::load(&TlsSource::from_directory(dir.path())) + .await + .expect("validated TLS snapshot"); + let before = TLS_PREPARE_COUNTS.with(std::cell::Cell::get); + let prepared = PreparedServerTls::from_snapshot(&snapshot) + .expect("prepare generation") + .expect("server TLS"); + let tcp = prepared.server_config(ServerProtocol::Tcp).expect("TCP config"); + let another_tcp = prepared + .server_config(ServerProtocol::Tcp) + .expect("additional listener config"); + assert!(Arc::ptr_eq(&tcp.cert_resolver, &another_tcp.cert_resolver)); + #[cfg(feature = "http3")] + { + let quic_tls = prepared.server_config(ServerProtocol::Http3).expect("QUIC TLS config"); + assert!(Arc::ptr_eq(&tcp.cert_resolver, &quic_tls.cert_resolver)); + } + let after = TLS_PREPARE_COUNTS.with(std::cell::Cell::get); + assert_eq!( + (after.0 - before.0, after.1 - before.1), + (1, 1), + "client verifier and signing resolver must each be prepared once for all protocols and listeners" + ); + + let RuntimeServerTlsMaterial::MultiCert { cert_key_pairs } = snapshot.server.as_ref().expect("SNI material") else { + panic!("multiple selectors expected"); + }; + let mut roots = rustls::RootCertStore::empty(); + for (certificates, _) in cert_key_pairs.values() { + roots.add(certificates[0].clone()).expect("server trust"); + } + for (name, selector) in [("default.local", "default"), ("sni.example", "sni.example")] { + let expected = cert_key_pairs[selector].0[0].clone(); + let client_tls = rustls::ClientConfig::builder() + .with_root_certificates(roots.clone()) + .with_client_auth_cert( + vec![client_cert.der().clone()], + PrivateKeyDer::try_from(client_key.serialize_der()).expect("client private key"), + ) + .expect("mTLS client"); + let (server_io, client_io) = tokio::io::duplex(65536); + let acceptor = prepared.tcp_acceptor().expect("acceptor").get(); + let connector = tokio_rustls::TlsConnector::from(Arc::new(client_tls.clone())); + let (server, client) = tokio::join!( + acceptor.accept(server_io), + connector.connect(rustls::pki_types::ServerName::try_from(name.to_owned()).expect("server name"), client_io) + ); + let server = server.expect("authenticated server handshake"); + let client = client.expect("authenticated client handshake"); + assert_eq!(client.get_ref().1.peer_certificates().expect("server identity")[0], expected); + assert_eq!(server.get_ref().1.peer_certificates().expect("client identity")[0], *client_cert.der()); + + #[cfg(feature = "http3")] + { + let endpoint = quinn::Endpoint::server( + prepared.http3_config().expect("QUIC config"), + "127.0.0.1:0".parse().expect("address"), + ) + .expect("QUIC server"); + let mut client_tls = client_tls; + client_tls.alpn_protocols = vec![b"h3".to_vec()]; + let crypto = quinn::crypto::rustls::QuicClientConfig::try_from(client_tls).expect("QUIC client TLS"); + let mut client_endpoint = + quinn::Endpoint::client("127.0.0.1:0".parse().expect("address")).expect("QUIC client"); + client_endpoint.set_default_client_config(quinn::ClientConfig::new(Arc::new(crypto))); + let connection = client_endpoint + .connect(endpoint.local_addr().expect("server address"), name) + .expect("connect"); + let (server, client) = tokio::time::timeout(Duration::from_secs(5), async { + tokio::join!(async { endpoint.accept().await.expect("incoming").await }, connection) + }) + .await + .expect("QUIC handshake deadline"); + let server = server.expect("QUIC mTLS server handshake"); + let client = client.expect("QUIC mTLS client handshake"); + let identity = client + .peer_identity() + .expect("QUIC server identity") + .downcast::>>() + .expect("certificate identity"); + assert_eq!(identity[0], expected); + let identity = server + .peer_identity() + .expect("QUIC client identity") + .downcast::>>() + .expect("certificate identity"); + assert_eq!(identity[0], *client_cert.der()); + client.close(0u32.into(), b"done"); + server.close(0u32.into(), b"done"); + endpoint.close(0u32.into(), b"done"); + client_endpoint.close(0u32.into(), b"done"); + } + } + }) + .await; + } + + #[tokio::test] + async fn candidate_replacement_preserves_ca_components_and_default_names() { + let dir = TempDir::new().expect("TLS directory"); + write_test_cert_pair(dir.path(), "s3.example.com"); + let ca_a = rcgen::generate_simple_self_signed(vec!["ca-a.example".into()]).expect("CA A"); + let ca_b = rcgen::generate_simple_self_signed(vec!["ca-b.example".into()]).expect("CA B"); + fs::write(dir.path().join(RUSTFS_CLIENT_CA_CERT_FILENAME), ca_a.cert.pem()).expect("primary client CA"); + fs::write(dir.path().join(RUSTFS_CA_CERT), ca_b.cert.pem()).expect("fallback CA"); + let source = TlsSource::from_directory(dir.path()); + let accepted = rustfs_tls_runtime::TlsMaterialSnapshot::load(&source) + .await + .expect("accepted generation"); + fs::remove_file(dir.path().join(RUSTFS_CLIENT_CA_CERT_FILENAME)).expect("remove primary client CA"); + let candidate = rustfs_tls_runtime::TlsMaterialSnapshot::load(&source) + .await + .expect("fallback generation parses"); + assert!( + validate_candidate_replacement(&accepted, &candidate).is_err(), + "a fallback must not replace accepted client trust" + ); + + fs::write(dir.path().join(RUSTFS_CLIENT_CA_CERT_FILENAME), ca_a.cert.pem()).expect("restore primary CA"); + write_test_cert_pair(dir.path(), "wrong.example.com"); + let candidate = rustfs_tls_runtime::TlsMaterialSnapshot::load(&source) + .await + .expect("valid but wrong SAN"); + assert!( + validate_candidate_replacement(&accepted, &candidate).is_err(), + "default SAN coverage must survive renewal" + ); + + write_test_cert_pair(dir.path(), "s3.example.com"); + let renewed = rustfs_tls_runtime::TlsMaterialSnapshot::load(&source) + .await + .expect("valid renewal"); + validate_candidate_replacement(&accepted, &renewed).expect("renewal preserving names and trust sources"); + } + + #[tokio::test] + async fn candidate_default_names_cannot_move_into_another_sni_certificate() { + let dir = TempDir::new().expect("TLS directory"); + let sni = dir.path().join("example.com"); + fs::create_dir(&sni).expect("SNI directory"); + write_test_cert_pair(dir.path(), "default.local"); + write_test_cert_pair(&sni, "example.com"); + let source = TlsSource::from_directory(dir.path()); + let accepted = rustfs_tls_runtime::TlsMaterialSnapshot::load(&source) + .await + .expect("accepted selectors"); + write_test_cert_pair(dir.path(), "wrong.local"); + let relocated = rcgen::generate_simple_self_signed(vec!["example.com".into(), "default.local".into()]) + .expect("certificate with relocated SAN"); + fs::write(sni.join(RUSTFS_TLS_CERT), relocated.cert.pem()).expect("SNI certificate"); + fs::write(sni.join(rustfs_config::RUSTFS_TLS_KEY), relocated.signing_key.serialize_pem()).expect("SNI key"); + let candidate = rustfs_tls_runtime::TlsMaterialSnapshot::load(&source) + .await + .expect("individually valid candidate"); + assert!( + validate_candidate_replacement(&accepted, &candidate).is_err(), + "another selector's SAN does not cover the default resolver's hostname" + ); + } + + #[tokio::test] + #[serial_test::serial] + async fn snapshot_leaf_trust_is_encoded_in_canonical_der_order() { + let dir = TempDir::new().expect("TLS directory"); + let a = rcgen::generate_simple_self_signed(vec!["a.example".into()]).expect("certificate A"); + let b = rcgen::generate_simple_self_signed(vec!["b.example".into()]).expect("certificate B"); + let (first, second) = if a.cert.der().as_ref() > b.cert.der().as_ref() { + (&a, &b) + } else { + (&b, &a) + }; + fs::write(dir.path().join(RUSTFS_TLS_CERT), format!("{}{}", first.cert.pem(), second.cert.pem())) + .expect("descending DER chain"); + fs::write(dir.path().join(rustfs_config::RUSTFS_TLS_KEY), first.signing_key.serialize_pem()).expect("leaf key"); + let mut snapshot = rustfs_tls_runtime::TlsMaterialSnapshot::load(&TlsSource::from_directory(dir.path())) + .await + .expect("validated material"); + temp_env::async_with_vars( + [ + (ENV_TRUST_LEAF_CERT_AS_CA, Some("1")), + (ENV_TRUST_SYSTEM_CA, Some("0")), + (ENV_MTLS_CLIENT_CERT, None), + (ENV_MTLS_CLIENT_KEY, None), + ], + async { + enrich_outbound(&mut snapshot).await.expect("derive leaf trust"); + let mut reader = Cursor::new(&snapshot.outbound.root_ca_pem); + let certificates = CertificateDer::pem_reader_iter(&mut reader) + .collect::, _>>() + .expect("derived certificate PEM"); + assert_eq!( + certificates, + vec![second.cert.der().clone(), first.cert.der().clone()], + "leaf trust must not depend on source enumeration order" + ); + }, + ) + .await; + } + + #[tokio::test] + async fn client_identity_override_rejects_mismatch_and_missing_files() { + let dir = TempDir::new().expect("identity directory"); + let a = rcgen::generate_simple_self_signed(vec!["client-a.example".into()]).expect("identity A"); + let b = rcgen::generate_simple_self_signed(vec!["client-b.example".into()]).expect("identity B"); + let cert = dir.path().join(RUSTFS_CLIENT_CERT_FILENAME); + let key = dir.path().join(RUSTFS_CLIENT_KEY_FILENAME); + fs::write(&cert, a.cert.pem()).expect("client cert"); + fs::write(&key, b.signing_key.serialize_pem()).expect("mismatched key"); + assert!( + load_mtls_identity_from_overridden_paths(cert.to_str(), key.to_str(), dir.path()) + .await + .is_err() + ); + fs::remove_file(&key).expect("remove override key"); + assert!( + load_mtls_identity_from_overridden_paths(cert.to_str(), key.to_str(), dir.path()) + .await + .is_err() + ); + fs::write(&key, a.signing_key.serialize_pem()).expect("matching key"); + assert!( + load_mtls_identity_from_overridden_paths(cert.to_str(), key.to_str(), dir.path()) + .await + .expect("valid override") + .is_some() + ); + } + + enum StartupRotation { + Outbound, + RootsOnly, + IdentityOnly, + ServerOnly, + MissingRoots, + PartialRoots, + MissingIdentity, + } + + async fn assert_initial_outbound_reconciliation(rotation: StartupRotation) { + ensure_rustls_crypto_provider(); + temp_env::async_with_vars( + [ + (ENV_TLS_RELOAD_ENABLE, Some("1")), + (ENV_SERVER_MTLS_ENABLE, Some("0")), + (ENV_TRUST_LEAF_CERT_AS_CA, Some("0")), + (ENV_TRUST_SYSTEM_CA, Some("0")), + (ENV_MTLS_CLIENT_CERT, None), + (ENV_MTLS_CLIENT_KEY, None), + (ENV_RUSTFS_EXTRA_CA_CERT, None), + (super::super::tls_publish::ENV_TLS_PUBLISH_DIR, None), + ], + async { + let previous = startup_runtime_sources::current_outbound_tls_state().await; + let previous_files = crate::startup_tls_material::replace_startup_material_files(None); + let dir = TempDir::new().expect("TLS directory"); + write_test_cert_pair(dir.path(), "server.example"); + let identity_a = rcgen::generate_simple_self_signed(vec!["client-a.example".into()]).expect("identity A"); + fs::write(dir.path().join(RUSTFS_CA_CERT), identity_a.cert.pem()).expect("trust A"); + fs::write(dir.path().join(RUSTFS_CLIENT_CERT_FILENAME), identity_a.cert.pem()).expect("client certificate A"); + fs::write(dir.path().join(RUSTFS_CLIENT_KEY_FILENAME), identity_a.signing_key.serialize_pem()) + .expect("client key A"); + if matches!(rotation, StartupRotation::PartialRoots) { + let public_ca = + rcgen::generate_simple_self_signed(vec!["public-ca.example".into()]).expect("distinct public CA"); + fs::write(dir.path().join(rustfs_config::RUSTFS_PUBLIC_CERT), public_ca.cert.pem()) + .expect("additional public trust"); + } + let path = dir.path().to_str().expect("UTF8 path").to_string(); + let mut config = crate::config::Config::new("127.0.0.1:0", Vec::new()); + config.tls_path = Some(path.clone()); + crate::startup_tls_material::init_outbound_tls_material(&config) + .await + .expect("real startup publication"); + let startup = startup_runtime_sources::current_outbound_tls_state().await; + + write_test_cert_pair(dir.path(), "server.example"); + match rotation { + StartupRotation::Outbound | StartupRotation::RootsOnly | StartupRotation::IdentityOnly => { + let identity_b = rcgen::generate_simple_self_signed(vec!["client-b.example".into()]).expect("identity B"); + if !matches!(rotation, StartupRotation::IdentityOnly) { + fs::write(dir.path().join(RUSTFS_CA_CERT), identity_b.cert.pem()).expect("trust B"); + } + if !matches!(rotation, StartupRotation::RootsOnly) { + fs::write(dir.path().join(RUSTFS_CLIENT_CERT_FILENAME), identity_b.cert.pem()) + .expect("client certificate B"); + fs::write(dir.path().join(RUSTFS_CLIENT_KEY_FILENAME), identity_b.signing_key.serialize_pem()) + .expect("client key B"); + } + } + StartupRotation::ServerOnly => {} + StartupRotation::MissingRoots | StartupRotation::PartialRoots => { + fs::remove_file(dir.path().join(RUSTFS_CA_CERT)).expect("remove startup roots"); + } + StartupRotation::MissingIdentity => { + fs::remove_file(dir.path().join(RUSTFS_CLIENT_CERT_FILENAME)).expect("remove startup client certificate"); + fs::remove_file(dir.path().join(RUSTFS_CLIENT_KEY_FILENAME)).expect("remove startup client key"); + } + } + let initial = load_tls_material(&path).await.expect("HTTP initialization snapshot"); + let incomplete = matches!( + rotation, + StartupRotation::MissingRoots | StartupRotation::PartialRoots | StartupRotation::MissingIdentity + ); + let expected_roots = if incomplete { + startup.root_ca_pem.clone().expect("startup trust") + } else { + initial.outbound.root_ca_pem.clone() + }; + let expected_identity = if incomplete { + startup.mtls_identity.clone() + } else { + initial.outbound.mtls_identity.clone() + }; + let prepared = PreparedServerTls::from_snapshot(&initial).expect("config B").expect("TLS B"); + let holder = prepared.tcp_acceptor().expect("TLS B acceptor"); + let initial_acceptor = holder.get(); + let task = spawn_reload_loop( + path.clone(), + holder.clone(), + initial, + prepared, + #[cfg(feature = "http3")] + None, + ) + .expect("register HTTP initialization snapshot") + .expect("reload enabled"); + let group = RELOAD_GROUPS + .lock() + .unwrap_or_else(|e| e.into_inner()) + .get(&path) + .and_then(Weak::upgrade) + .expect("registered reload group"); + notify_server_publication(&path); + tokio::time::timeout(Duration::from_secs(10), group.pass_completed.notified()) + .await + .expect("first reload pass completed"); + let first = startup_runtime_sources::current_outbound_tls_state().await; + notify_server_publication(&path); + tokio::time::timeout(Duration::from_secs(10), group.pass_completed.notified()) + .await + .expect("second reload pass completed"); + let second = startup_runtime_sources::current_outbound_tls_state().await; + + task.abort(); + let _ = task.await; + tokio::time::timeout(Duration::from_secs(10), async { + while RELOAD_GROUPS.lock().unwrap_or_else(|e| e.into_inner()).contains_key(&path) { + tokio::task::yield_now().await; + } + }) + .await + .expect("reload group exits before restoring global state"); + startup_runtime_sources::publish_outbound_tls_state( + TlsGeneration(second.generation.0.saturating_add(1)), + &rustfs_tls_runtime::OutboundTlsMaterial { + root_ca_pem: previous.root_ca_pem.unwrap_or_default(), + mtls_identity: previous.mtls_identity, + }, + ) + .await; + crate::startup_tls_material::replace_startup_material_files(previous_files); + + assert_eq!(first.root_ca_pem.as_deref(), Some(expected_roots.as_slice())); + assert!( + first.mtls_identity == expected_identity, + "published identity must match the initialized listener snapshot" + ); + let rotate_outbound = matches!( + rotation, + StartupRotation::Outbound | StartupRotation::RootsOnly | StartupRotation::IdentityOnly + ); + let expected_generation = startup.generation.0.saturating_add(u64::from(rotate_outbound)); + assert_eq!(first.generation.0, expected_generation); + assert_eq!(second.generation, first.generation, "unchanged material must not churn outbound clients"); + assert!( + rotate_outbound || Arc::ptr_eq(&holder.get(), &initial_acceptor), + "non-publications must retain the initial server configuration" + ); + }, + ) + .await; + } + + #[tokio::test] + #[serial_test::serial] + async fn tls_reload_reconciles_initial_outbound_after_startup_rotation() { + for rotation in [ + StartupRotation::RootsOnly, + StartupRotation::IdentityOnly, + StartupRotation::Outbound, + ] { + assert_initial_outbound_reconciliation(rotation).await; + } + } + + #[tokio::test] + #[serial_test::serial] + async fn tls_reload_initial_server_rotation_preserves_matching_outbound_generation() { + assert_initial_outbound_reconciliation(StartupRotation::ServerOnly).await; + } + + #[tokio::test] + #[serial_test::serial] + async fn tls_reload_initial_missing_roots_retains_startup_material() { + assert_initial_outbound_reconciliation(StartupRotation::MissingRoots).await; + } + + #[tokio::test] + #[serial_test::serial] + async fn tls_reload_initial_partial_missing_roots_retains_startup_material() { + assert_initial_outbound_reconciliation(StartupRotation::PartialRoots).await; + } + + #[tokio::test] + #[serial_test::serial] + async fn tls_reload_initial_missing_identity_retains_startup_material() { + assert_initial_outbound_reconciliation(StartupRotation::MissingIdentity).await; + } + + #[tokio::test] + #[serial_test::serial] + async fn late_listener_joins_accepted_generation_when_source_is_invalid() { + ensure_rustls_crypto_provider(); + temp_env::async_with_vars([(ENV_TLS_RELOAD_ENABLE, Some("1")), (ENV_SERVER_MTLS_ENABLE, Some("0"))], async { + let dir = TempDir::new().expect("TLS directory"); + write_test_cert_pair(dir.path(), "server.example"); + let source = TlsSource::from_directory(dir.path()); + let a = rustfs_tls_runtime::TlsMaterialSnapshot::load(&source) + .await + .expect("generation A"); + let prepared_a = PreparedServerTls::from_snapshot(&a).expect("prepare A").expect("TLS A"); + let first = prepared_a.tcp_acceptor().expect("config A"); + let path = dir.path().to_str().expect("UTF8 path").to_string(); + let task_a = spawn_reload_loop( + path.clone(), + first.clone(), + a, + prepared_a, + #[cfg(feature = "http3")] + None, + ) + .expect("register A") + .expect("reload enabled"); + // No async yield between producing B and registration: the actor cannot accept B. + write_test_cert_pair(dir.path(), "server.example"); + let b = rustfs_tls_runtime::TlsMaterialSnapshot::load_from_snapshot(&source.snapshot().expect("pin B")) + .expect("generation B"); + let prepared_b = PreparedServerTls::from_snapshot(&b).expect("prepare B").expect("TLS B"); + let second = prepared_b.tcp_acceptor().expect("config B"); + assert!(!Arc::ptr_eq(&first.get(), &second.get())); + fs::write(dir.path().join(RUSTFS_TLS_CERT), "invalid certificate").expect("break source"); + let task_b = spawn_reload_loop( + path.clone(), + second.clone(), + b, + prepared_b, + #[cfg(feature = "http3")] + None, + ) + .expect("register B") + .expect("reload enabled"); + assert!( + Arc::ptr_eq(&first.get(), &second.get()), + "late listener must use the group's accepted config" + ); + task_a.abort(); + task_b.abort(); + let _ = task_a.await; + let _ = task_b.await; + tokio::time::timeout(Duration::from_secs(5), async { + while RELOAD_GROUPS.lock().unwrap_or_else(|e| e.into_inner()).contains_key(&path) { + tokio::task::yield_now().await; + } + }) + .await + .expect("group exits when all registrations are dropped"); + }) + .await; + } + #[tokio::test] #[serial_test::serial] async fn oidc_extra_ca_cert_loads_configured_bundle() { @@ -1070,15 +1792,17 @@ mod tests { ensure_rustls_crypto_provider(); let cert = rcgen::generate_simple_self_signed(vec!["localhost".to_string()]).expect("certificate should generate"); - let config = build_server_config( - ServerCertSource::SingleCert { + let config = PreparedServerTls::from_material( + Some(&RuntimeServerTlsMaterial::SingleCert { certs: vec![cert.cert.der().clone()], key: rustls::pki_types::PrivateKeyDer::try_from(cert.signing_key.serialize_der()) .expect("private key should convert"), - }, + }), None, - ServerProtocol::Http3, ) + .expect("prepare TLS") + .expect("server certificate") + .server_config(ServerProtocol::Http3) .expect("HTTP/3 TLS config should build"); assert_eq!(config.alpn_protocols, vec![b"h3".to_vec()]); diff --git a/rustfs/src/server/tls_publish.rs b/rustfs/src/server/tls_publish.rs new file mode 100644 index 000000000..afd6dc2fa --- /dev/null +++ b/rustfs/src/server/tls_publish.rs @@ -0,0 +1,1341 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Durable server-leaf publication. Trust and client identity remain owned by their original source. + +use super::tls_material; +use rustfs_config::{DEFAULT_TLS_RELOAD_ENABLE, ENV_TLS_RELOAD_ENABLE, RUSTFS_TLS_CERT, RUSTFS_TLS_KEY}; +use rustfs_tls_runtime::{ServerTlsMaterial, TlsMaterialSnapshot, TlsSource}; +use rustls::pki_types::{CertificateDer, pem::PemObject}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; +use std::{ + fs::{self, File, OpenOptions}, + io::{Read, Write}, + path::{Component, Path, PathBuf}, +}; +use uuid::Uuid; + +pub(crate) const ENV_TLS_PUBLISH_DIR: &str = "RUSTFS_TLS_PUBLISH_DIR"; +pub(crate) const SCHEMA: &str = "rustfs.tls-publish/v1alpha1"; +pub(crate) const RESOURCE: &str = "server-default"; +const MAX_CERT: usize = 64 * 1024; +const MAX_KEY: usize = 32 * 1024; +const MANIFEST: &str = "manifest.json"; +const CURRENT: &str = "server-current"; +const INITIALIZED: &str = "initialized"; +const INITIALIZED_CONTENT: &[u8] = b"rustfs.tls-publish/v1alpha1\n"; + +#[cfg(test)] +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +struct OperationReads { + certificates: usize, + keys: usize, + manifests: usize, + manifest_decodes: usize, + certificate_pem_traversals: usize, + leaf_der_decodes: usize, +} + +#[cfg(test)] +thread_local! { + static OPERATION_READS: std::cell::Cell = const { std::cell::Cell::new(OperationReads { + certificates: 0, keys: 0, manifests: 0, manifest_decodes: 0, + certificate_pem_traversals: 0, leaf_der_decodes: 0, + }) }; +} + +#[cfg(test)] +fn count_operation(action: impl FnOnce(&mut OperationReads)) { + OPERATION_READS.with(|counts| { + let mut value = counts.get(); + action(&mut value); + counts.set(value); + }); +} + +fn certificate_pem_items( + bytes: &[u8], +) -> impl Iterator, rustls::pki_types::pem::Error>> + '_ { + #[cfg(test)] + count_operation(|counts| counts.certificate_pem_traversals += 1); + CertificateDer::pem_slice_iter(bytes) +} + +#[cfg(test)] +#[derive(Clone, Copy)] +enum FailurePoint { + None, + AfterPointerRename, + AfterPointerRenameUnreadable, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct PublishRequest { + pub schema_version: String, + pub resource_id: String, + pub expected_target_revision: String, + pub issuance_id: Uuid, + pub certificate_pem: String, + pub private_key_pem: String, +} + +#[derive(Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Manifest { + schema_version: String, + issuance_id: Uuid, + target_revision: Uuid, + certificate_fingerprint: String, + not_after_unix: i64, + payload_digest: String, +} + +// A disk operation pins one immutable generation and verifies its envelope once. +// This value is never retained across requests or reload ticks. +struct PublishedGeneration { + manifest: Manifest, + chain: Vec>, + leaf: LeafMetadata, + key_pem: Vec, +} + +struct LeafMetadata { + fingerprint: String, + not_after_unix: i64, + names: std::collections::BTreeSet, + is_ca: bool, +} + +#[derive(Serialize)] +pub(crate) struct CertificateState { + pub schema_version: &'static str, + pub resource_id: &'static str, + pub target_revision: String, + pub certificate_fingerprint: String, + pub not_after_unix: i64, + pub last_issuance_id: Option, + pub observed_certificate_fingerprint: Option, +} + +#[derive(Serialize)] +pub(crate) struct PublishReceipt { + pub schema_version: &'static str, + pub resource_id: &'static str, + pub issuance_id: Uuid, + pub target_revision: String, + pub certificate_fingerprint: String, + pub not_after_unix: i64, + pub observed_certificate_fingerprint: Option, +} + +#[derive(Debug, thiserror::Error)] +pub(crate) enum PublishError { + #[error("certificate publication is disabled or not configured safely")] + Disabled, + #[error("invalid certificate publication request or source")] + Invalid, + #[error("certificate publication revision conflict")] + Conflict, + #[error("certificate publication is busy; retry the identical issuance")] + Busy, + #[error("certificate publication storage failed; retry the identical issuance")] + Storage, +} + +impl From for PublishError { + fn from(_: std::io::Error) -> Self { + Self::Storage + } +} + +fn hex_digest(bytes: &[u8]) -> String { + Sha256::digest(bytes).iter().map(|byte| format!("{byte:02x}")).collect() +} + +fn payload_digest(certificate: &[u8], key: &[u8]) -> String { + let mut digest = Sha256::new(); + // Both inputs have bounded sizes; the fixed-width encoding survives architecture changes. + digest.update((certificate.len() as u128).to_be_bytes()); + digest.update(certificate); + digest.update((key.len() as u128).to_be_bytes()); + digest.update(key); + digest.finalize().iter().map(|byte| format!("{byte:02x}")).collect() +} + +fn parse_leaf(leaf: &[u8]) -> Result { + #[cfg(test)] + count_operation(|counts| counts.leaf_der_decodes += 1); + let (remaining, parsed) = x509_parser::parse_x509_certificate(leaf).map_err(|_| PublishError::Invalid)?; + if !remaining.is_empty() { + return Err(PublishError::Invalid); + } + let mut names = std::collections::BTreeSet::new(); + if let Some(san) = parsed.subject_alternative_name().map_err(|_| PublishError::Invalid)? { + for name in &san.value.general_names { + match name { + x509_parser::extensions::GeneralName::DNSName(name) => { + names.insert(format!("dns:{}", name.to_ascii_lowercase())); + } + x509_parser::extensions::GeneralName::IPAddress(bytes) => { + names.insert(format!("ip:{bytes:?}")); + } + _ => {} + } + } + } + Ok(LeafMetadata { + fingerprint: hex_digest(leaf), + not_after_unix: parsed.validity().not_after.timestamp(), + names, + is_ca: parsed + .basic_constraints() + .map_err(|_| PublishError::Invalid)? + .is_some_and(|constraints| constraints.value.ca), + }) +} + +fn snapshot_leaf(snapshot: &TlsMaterialSnapshot) -> Option { + match &snapshot.server { + Some(ServerTlsMaterial::SingleCert { certs, .. }) => certs.first().map(|cert| hex_digest(cert.as_ref())), + _ => None, + } +} + +fn observed(tls_path: &str, expected_chain: &[CertificateDer<'_>]) -> Option { + let snapshot = tls_material::observed_server_snapshot(tls_path)?; + match &snapshot.server { + Some(ServerTlsMaterial::SingleCert { certs, .. }) if certs == expected_chain => snapshot_leaf(&snapshot), + _ => None, + } +} + +fn open_regular(path: &Path, writable: bool, create: bool) -> Result { + let mut options = OpenOptions::new(); + options.read(true).write(writable).create(create); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.mode(0o600).custom_flags(libc::O_NOFOLLOW | libc::O_CLOEXEC); + } + let file = options.open(path)?; + if !file.metadata()?.is_file() { + return Err(PublishError::Invalid); + } + Ok(file) +} + +fn read_limited(path: &Path, limit: usize) -> Result, PublishError> { + let file = open_regular(path, false, false)?; + #[cfg(test)] + count_operation(|counts| match path.file_name().and_then(|name| name.to_str()) { + Some(RUSTFS_TLS_CERT) => counts.certificates += 1, + Some(RUSTFS_TLS_KEY) => counts.keys += 1, + Some(MANIFEST) => counts.manifests += 1, + _ => {} + }); + let mut bytes = Vec::new(); + file.take(u64::try_from(limit).map_err(|_| PublishError::Invalid)? + 1) + .read_to_end(&mut bytes)?; + if bytes.len() > limit { + return Err(PublishError::Invalid); + } + Ok(bytes) +} + +fn validate_root(root: &Path) -> Result<(), PublishError> { + let metadata = fs::symlink_metadata(root)?; + if !metadata.is_dir() || metadata.file_type().is_symlink() { + return Err(PublishError::Disabled); + } + #[cfg(unix)] + { + use std::os::unix::fs::MetadataExt; + if metadata.mode() & 0o077 != 0 || metadata.uid() != rustix::process::geteuid().as_raw() { + return Err(PublishError::Disabled); + } + } + #[cfg(not(unix))] + { + Err(PublishError::Disabled) + } + #[cfg(unix)] + { + Ok(()) + } +} + +fn configured_root() -> Result, PublishError> { + let root = rustfs_utils::get_env_opt_str(ENV_TLS_PUBLISH_DIR).map(PathBuf::from); + if let Some(root) = &root { + validate_root(root)?; + } + Ok(root) +} + +fn single_default_source(tls_path: &str) -> Result<(), PublishError> { + if fs::symlink_metadata(Path::new(tls_path).join("..data")).is_ok() { + return Err(PublishError::Disabled); + } + for entry in fs::read_dir(tls_path)? { + let entry = entry?; + if !entry.file_name().to_string_lossy().starts_with('.') && entry.path().is_dir() { + return Err(PublishError::Invalid); + } + } + Ok(()) +} + +fn current_generation(root: &Path) -> Result, PublishError> { + let initialized = match fs::symlink_metadata(root.join(INITIALIZED)) { + Ok(_) => { + if read_limited(&root.join(INITIALIZED), 128)? != INITIALIZED_CONTENT { + return Err(PublishError::Invalid); + } + true + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => false, + Err(error) => return Err(error.into()), + }; + let pointer = root.join(CURRENT); + match fs::symlink_metadata(&pointer) { + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + return if initialized { Err(PublishError::Invalid) } else { Ok(None) }; + } + Err(error) => return Err(error.into()), + Ok(metadata) if !metadata.file_type().is_symlink() => return Err(PublishError::Invalid), + _ => {} + } + let target = fs::read_link(pointer)?; + let mut components = target.components(); + if components.next() != Some(Component::Normal("generations".as_ref())) { + return Err(PublishError::Invalid); + } + let Some(Component::Normal(version)) = components.next() else { + return Err(PublishError::Invalid); + }; + let id = version + .to_str() + .and_then(|text| Uuid::parse_str(text).ok()) + .ok_or(PublishError::Invalid)?; + if components.next().is_some() || version != id.to_string().as_str() { + return Err(PublishError::Invalid); + } + let generations = root.join("generations"); + if fs::symlink_metadata(&generations)?.file_type().is_symlink() { + return Err(PublishError::Invalid); + } + let directory = root.join(target); + if !fs::symlink_metadata(&directory)?.is_dir() { + return Err(PublishError::Invalid); + } + PublishedGeneration::load(directory, id).map(Some) +} + +fn validate_source_separation(root: &Path, tls_path: &str) -> Result<(), PublishError> { + let root = fs::canonicalize(root)?; + let source = fs::canonicalize(tls_path)?; + if root.starts_with(&source) || source.starts_with(&root) { + return Err(PublishError::Disabled); + } + Ok(()) +} + +impl PublishedGeneration { + fn into_snapshot(self, source: &rustfs_tls_runtime::TlsSourceSnapshot) -> Result { + TlsMaterialSnapshot::load_from_snapshot_with_server_identity(source, self.chain, &self.key_pem) + .map_err(|_| PublishError::Invalid) + } + + fn load(directory: PathBuf, revision: Uuid) -> Result { + let manifest_bytes = read_limited(&directory.join(MANIFEST), 4096)?; + #[cfg(test)] + count_operation(|counts| counts.manifest_decodes += 1); + let manifest: Manifest = serde_json::from_slice(&manifest_bytes).map_err(|_| PublishError::Invalid)?; + let certificate = read_limited(&directory.join(RUSTFS_TLS_CERT), MAX_CERT)?; + let key = read_limited(&directory.join(RUSTFS_TLS_KEY), MAX_KEY)?; + let chain = certificate_pem_items(&certificate) + .collect::, _>>() + .map_err(|_| PublishError::Invalid)?; + let leaf = parse_leaf(chain.first().ok_or(PublishError::Invalid)?.as_ref())?; + if manifest.schema_version != SCHEMA + || manifest.issuance_id.is_nil() + || manifest.target_revision.is_nil() + || manifest.target_revision != revision + || manifest.certificate_fingerprint != leaf.fingerprint + || manifest.not_after_unix != leaf.not_after_unix + || manifest.payload_digest != payload_digest(&certificate, &key) + { + return Err(PublishError::Invalid); + } + Ok(Self { + manifest, + chain, + leaf, + key_pem: key, + }) + } +} + +#[cfg(test)] +fn current_directory(root: &Path) -> Result, PublishError> { + current_generation(root).map(|generation| { + generation.map(|generation| root.join("generations").join(generation.manifest.target_revision.to_string())) + }) +} + +pub(crate) fn configured_server_snapshot( + tls_path: &str, + source: &rustfs_tls_runtime::TlsSourceSnapshot, +) -> Result, PublishError> { + let Some(root) = configured_root()? else { + return Ok(None); + }; + validate_source_separation(&root, tls_path)?; + single_default_source(tls_path)?; + current_generation(&root)? + .map(|generation| generation.into_snapshot(source)) + .transpose() +} + +fn state_at(root: &Path, tls_path: &str) -> Result { + let generation = current_generation(root)?; + state_from_generation(generation.as_ref(), tls_path) +} + +fn state_from_generation(generation: Option<&PublishedGeneration>, tls_path: &str) -> Result { + if let Some(generation) = generation { + return Ok(CertificateState { + schema_version: SCHEMA, + resource_id: RESOURCE, + target_revision: generation.manifest.target_revision.to_string(), + certificate_fingerprint: generation.leaf.fingerprint.clone(), + not_after_unix: generation.leaf.not_after_unix, + last_issuance_id: Some(generation.manifest.issuance_id), + observed_certificate_fingerprint: observed(tls_path, &generation.chain), + }); + } + let snapshot = tls_material::observed_server_snapshot(tls_path).ok_or(PublishError::Disabled)?; + let Some(ServerTlsMaterial::SingleCert { certs, .. }) = &snapshot.server else { + return Err(PublishError::Invalid); + }; + let leaf = parse_leaf(certs.first().ok_or(PublishError::Invalid)?.as_ref())?; + Ok(CertificateState { + schema_version: SCHEMA, + resource_id: RESOURCE, + target_revision: format!("bootstrap-{}", leaf.fingerprint), + certificate_fingerprint: leaf.fingerprint, + not_after_unix: leaf.not_after_unix, + last_issuance_id: None, + observed_certificate_fingerprint: observed(tls_path, certs), + }) +} + +pub(crate) async fn certificate_state(tls_path: String) -> Result { + tokio::task::spawn_blocking(move || { + let root = configured_root()?.ok_or(PublishError::Disabled)?; + validate_source_separation(&root, &tls_path)?; + single_default_source(&tls_path)?; + state_at(&root, &tls_path) + }) + .await + .map_err(|_| PublishError::Storage)? +} + +fn write_new(path: &Path, bytes: &[u8]) -> Result<(), PublishError> { + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.mode(0o600).custom_flags(libc::O_NOFOLLOW); + } + let mut file = options.open(path)?; + file.write_all(bytes)?; + file.sync_all()?; + Ok(()) +} + +fn receipt(state: CertificateState, issuance_id: Uuid) -> PublishReceipt { + PublishReceipt { + schema_version: SCHEMA, + resource_id: RESOURCE, + issuance_id, + target_revision: state.target_revision, + certificate_fingerprint: state.certificate_fingerprint, + not_after_unix: state.not_after_unix, + observed_certificate_fingerprint: state.observed_certificate_fingerprint, + } +} + +fn mark_initialized(root: &Path) -> Result<(), PublishError> { + let marker = root.join(INITIALIZED); + if fs::symlink_metadata(&marker).is_ok() { + if read_limited(&marker, 128)? != INITIALIZED_CONTENT { + return Err(PublishError::Invalid); + } + } else { + // Partial writes must remain unreferenced so an identical issuance can recover after I/O failure. + let temporary = root.join(format!(".initialized-{}", Uuid::new_v4())); + write_new(&temporary, INITIALIZED_CONTENT)?; + fs::rename(temporary, &marker)?; + } + File::open(root)?.sync_all()?; + Ok(()) +} + +fn publish_at(root: &Path, tls_path: &str, request: PublishRequest) -> Result { + publish_at_inner( + root, + tls_path, + request, + #[cfg(test)] + FailurePoint::None, + ) +} + +fn publish_at_inner( + root: &Path, + tls_path: &str, + request: PublishRequest, + #[cfg(test)] failure: FailurePoint, +) -> Result { + validate_root(root)?; + validate_source_separation(root, tls_path)?; + single_default_source(tls_path)?; + if request.schema_version != SCHEMA + || request.resource_id != RESOURCE + || request.issuance_id.is_nil() + || request.certificate_pem.is_empty() + || request.certificate_pem.len() > MAX_CERT + || request.private_key_pem.is_empty() + || request.private_key_pem.len() > MAX_KEY + || request.expected_target_revision.len() > 128 + { + return Err(PublishError::Invalid); + } + // OS lock is shared by processes using this local volume. Captured CAS tokens are never refreshed on retry. + let lock = open_regular(&root.join("publisher.lock"), true, true)?; + lock.try_lock().map_err(|error| match error { + std::fs::TryLockError::WouldBlock => PublishError::Busy, + std::fs::TryLockError::Error(_) => PublishError::Storage, + })?; + let current = current_generation(root)?; + let state = state_from_generation(current.as_ref(), tls_path)?; + let digest = payload_digest(request.certificate_pem.as_bytes(), request.private_key_pem.as_bytes()); + if state.last_issuance_id == Some(request.issuance_id) { + let generation = current.as_ref().ok_or(PublishError::Invalid)?; + if generation.manifest.payload_digest != digest { + return Err(PublishError::Conflict); + } + // A prior response may have been lost after rename or directory sync. + File::open(root)?.sync_all()?; + mark_initialized(root)?; + return Ok(receipt(state, request.issuance_id)); + } + if request.expected_target_revision != state.target_revision { + return Err(PublishError::Conflict); + } + let accepted = tls_material::observed_server_snapshot(tls_path).ok_or(PublishError::Disabled)?; + let Some(ServerTlsMaterial::SingleCert { certs, .. }) = &accepted.server else { + return Err(PublishError::Invalid); + }; + let old_names = if let Some(generation) = current.as_ref() { + generation.leaf.names.clone() + } else { + if snapshot_leaf(&accepted).as_ref() != Some(&state.certificate_fingerprint) { + return Err(PublishError::Conflict); + } + parse_leaf(certs.first().ok_or(PublishError::Invalid)?.as_ref())?.names + }; + let new_chain = certificate_pem_items(request.certificate_pem.as_bytes()) + .collect::, _>>() + .map_err(|_| PublishError::Invalid)?; + let new_leaf = parse_leaf(new_chain.first().ok_or(PublishError::Invalid)?.as_ref())?; + if new_leaf.is_ca || !old_names.is_subset(&new_leaf.names) { + return Err(PublishError::Invalid); + } + let generations = root.join("generations"); + if !generations.exists() { + let mut builder = fs::DirBuilder::new(); + #[cfg(unix)] + { + use std::os::unix::fs::DirBuilderExt; + builder.mode(0o700); + } + builder.create(&generations)?; + File::open(root)?.sync_all()?; + } + if !fs::symlink_metadata(&generations)?.is_dir() { + return Err(PublishError::Invalid); + } + let revision = Uuid::new_v4(); + let directory = generations.join(revision.to_string()); + let mut builder = fs::DirBuilder::new(); + #[cfg(unix)] + { + use std::os::unix::fs::DirBuilderExt; + builder.mode(0o700); + } + builder.create(&directory)?; + let mut pointer_may_reference = false; + let result = (|| { + write_new(&directory.join(RUSTFS_TLS_CERT), request.certificate_pem.as_bytes())?; + write_new(&directory.join(RUSTFS_TLS_KEY), request.private_key_pem.as_bytes())?; + let source = TlsSource::from_directory(tls_path); + let candidate = TlsMaterialSnapshot::load_from_snapshot_with_server_directory( + &source.snapshot().map_err(|_| PublishError::Invalid)?, + Some(&directory), + ) + .map_err(|_| PublishError::Invalid)?; + if !matches!(&candidate.server, Some(ServerTlsMaterial::SingleCert { certs, .. }) if certs == &new_chain) { + return Err(PublishError::Invalid); + } + tls_material::validate_candidate_replacement(&accepted, &candidate).map_err(|_| PublishError::Invalid)?; + tls_material::validate_required_material_files( + &crate::startup_tls_material::required_startup_material_files(tls_path), + &candidate, + ) + .map_err(|_| PublishError::Invalid)?; + let prepared = tls_material::PreparedServerTls::from_snapshot(&candidate) + .map_err(|_| PublishError::Invalid)? + .ok_or(PublishError::Invalid)?; + prepared.tcp_acceptor().map_err(|_| PublishError::Invalid)?; + #[cfg(feature = "http3")] + prepared.http3_config().map_err(|_| PublishError::Invalid)?; + let ledger = Manifest { + schema_version: SCHEMA.into(), + issuance_id: request.issuance_id, + target_revision: revision, + certificate_fingerprint: new_leaf.fingerprint.clone(), + not_after_unix: new_leaf.not_after_unix, + payload_digest: digest, + }; + write_new( + &directory.join(MANIFEST), + &serde_json::to_vec(&ledger).map_err(|_| PublishError::Invalid)?, + )?; + File::open(&directory)?.sync_all()?; + File::open(&generations)?.sync_all()?; + let temporary = root.join(format!(".pointer-{revision}")); + #[cfg(unix)] + std::os::unix::fs::symlink(Path::new("generations").join(revision.to_string()), &temporary)?; + #[cfg(not(unix))] + let () = Err::<(), _>(PublishError::Disabled)?; + // A rename error can itself have an uncertain outcome on a failing filesystem. + pointer_may_reference = true; + fs::rename(&temporary, root.join(CURRENT))?; + #[cfg(test)] + { + if matches!(failure, FailurePoint::AfterPointerRenameUnreadable) { + fs::remove_file(root.join(CURRENT))?; + write_new(&root.join(CURRENT), b"unreadable pointer")?; + } + if !matches!(failure, FailurePoint::None) { + return Err(PublishError::Storage); + } + } + File::open(root)?.sync_all()?; + mark_initialized(root)?; + let installed = PublishedGeneration { + manifest: ledger, + chain: new_chain, + leaf: new_leaf, + key_pem: request.private_key_pem.into_bytes(), + }; + Ok(receipt(state_from_generation(Some(&installed), tls_path)?, request.issuance_id)) + })(); + // Observation errors must never convert an uncertain publication into deletion of serving material. + if result.is_err() && !pointer_may_reference { + let _ = fs::remove_dir_all(&directory); + } + result +} + +pub(crate) async fn publish(tls_path: String, request: PublishRequest) -> Result { + if !rustfs_utils::get_env_bool(ENV_TLS_RELOAD_ENABLE, DEFAULT_TLS_RELOAD_ENABLE) { + return Err(PublishError::Disabled); + } + if rustfs_utils::get_env_bool(rustfs_config::ENV_TRUST_LEAF_CERT_AS_CA, rustfs_config::DEFAULT_TRUST_LEAF_CERT_AS_CA) { + return Err(PublishError::Disabled); + } + let root = configured_root()?.ok_or(PublishError::Disabled)?; + let notification_path = tls_path.clone(); + let result = tokio::task::spawn_blocking(move || publish_at(&root, &tls_path, request)) + .await + .map_err(|_| PublishError::Storage)?; + tls_material::notify_server_publication(¬ification_path); + result +} + +#[cfg(all(test, unix))] +mod tests { + use super::*; + use std::os::unix::fs::PermissionsExt; + + fn pair(subject: &str) -> rcgen::CertifiedKey { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); + rcgen::generate_simple_self_signed(vec![subject.into()]).expect("certificate") + } + + fn build_request(expected: &str, certificate: &rcgen::CertifiedKey) -> PublishRequest { + PublishRequest { + schema_version: SCHEMA.into(), + resource_id: RESOURCE.into(), + expected_target_revision: expected.into(), + issuance_id: Uuid::new_v4(), + certificate_pem: certificate.cert.pem(), + private_key_pem: certificate.signing_key.serialize_pem(), + } + } + + fn clone_request(request: &PublishRequest) -> PublishRequest { + PublishRequest { + schema_version: request.schema_version.clone(), + resource_id: request.resource_id.clone(), + expected_target_revision: request.expected_target_revision.clone(), + issuance_id: request.issuance_id, + certificate_pem: request.certificate_pem.clone(), + private_key_pem: request.private_key_pem.clone(), + } + } + + fn measure_operation(operation: impl FnOnce() -> T) -> (T, OperationReads) { + OPERATION_READS.with(|counts| counts.set(OperationReads::default())); + let result = operation(); + (result, OPERATION_READS.with(std::cell::Cell::get)) + } + + #[tokio::test] + #[serial_test::serial] + async fn tls_publication_rejects_missing_startup_ca_source() { + temp_env::async_with_vars( + [ + (ENV_TLS_RELOAD_ENABLE, Some("1")), + (rustfs_config::ENV_SERVER_MTLS_ENABLE, Some("0")), + (rustfs_config::ENV_TRUST_LEAF_CERT_AS_CA, Some("0")), + (rustfs_config::ENV_TRUST_SYSTEM_CA, Some("0")), + (rustfs_config::ENV_MTLS_CLIENT_CERT, None), + (rustfs_config::ENV_MTLS_CLIENT_KEY, None), + (rustfs_config::ENV_RUSTFS_EXTRA_CA_CERT, None), + (ENV_TLS_PUBLISH_DIR, None), + ], + async { + let previous = crate::startup_runtime_sources::current_outbound_tls_state().await; + let previous_files = crate::startup_tls_material::replace_startup_material_files(None); + let base = tempfile::TempDir::new().expect("base"); + let root = tempfile::TempDir::new().expect("publication root"); + fs::set_permissions(root.path(), fs::Permissions::from_mode(0o700)).expect("permissions"); + let original = pair("server.example"); + fs::write(base.path().join(RUSTFS_TLS_CERT), original.cert.pem()).expect("certificate"); + fs::write(base.path().join(RUSTFS_TLS_KEY), original.signing_key.serialize_pem()).expect("key"); + fs::write(base.path().join(rustfs_config::RUSTFS_PUBLIC_CERT), pair("public-ca.example").cert.pem()) + .expect("public trust"); + fs::write(base.path().join(rustfs_config::RUSTFS_CA_CERT), pair("fallback-ca.example").cert.pem()) + .expect("fallback trust"); + let path = base.path().to_str().expect("path"); + let mut config = crate::config::Config::new("127.0.0.1:0", Vec::new()); + config.tls_path = Some(path.into()); + crate::startup_tls_material::init_outbound_tls_material(&config) + .await + .expect("real startup publication"); + fs::remove_file(base.path().join(rustfs_config::RUSTFS_CA_CERT)).expect("remove startup CA before listener"); + let initial = tls_material::load_tls_material(path).await.expect("HTTP initial snapshot"); + let prepared = tls_material::PreparedServerTls::from_snapshot(&initial) + .expect("prepared TLS") + .expect("server"); + let holder = prepared.tcp_acceptor().expect("acceptor"); + let task = tls_material::spawn_reload_loop( + path.into(), + holder, + initial, + prepared, + #[cfg(feature = "http3")] + None, + ) + .expect("registration") + .expect("reload enabled"); + let state = state_at(root.path(), path).expect("valid bootstrap CAS state"); + let result = publish_at(root.path(), path, build_request(&state.target_revision, &pair("server.example"))); + let installed = current_directory(root.path()).expect("read durable pointer"); + task.abort(); + let _ = task.await; + tokio::time::timeout(std::time::Duration::from_secs(10), async { + while tls_material::observed_server_snapshot(path).is_some() { + tokio::task::yield_now().await; + } + }) + .await + .expect("reload group exits before restoring startup state"); + crate::startup_runtime_sources::publish_outbound_tls_state( + rustfs_tls_runtime::TlsGeneration( + crate::startup_runtime_sources::current_outbound_tls_generation().saturating_add(1), + ), + &rustfs_tls_runtime::OutboundTlsMaterial { + root_ca_pem: previous.root_ca_pem.unwrap_or_default(), + mtls_identity: previous.mtls_identity, + }, + ) + .await; + crate::startup_tls_material::replace_startup_material_files(previous_files); + assert!( + matches!(result, Err(PublishError::Invalid)), + "source loss must reject an otherwise valid renewal" + ); + assert!(installed.is_none(), "rejected trust must not acquire a durable publication pointer"); + }, + ) + .await; + } + + #[tokio::test] + #[serial_test::serial] + async fn publication_operation_reads_are_bounded() { + temp_env::async_with_vars( + [ + (ENV_TLS_RELOAD_ENABLE, Some("1")), + (rustfs_config::ENV_SERVER_MTLS_ENABLE, Some("0")), + (ENV_TLS_PUBLISH_DIR, None), + ], + async { + let base = tempfile::TempDir::new().expect("base"); + let root = tempfile::TempDir::new().expect("publication root"); + fs::set_permissions(root.path(), fs::Permissions::from_mode(0o700)).expect("permissions"); + let original = pair("server.example"); + fs::write(base.path().join(RUSTFS_TLS_CERT), original.cert.pem()).expect("certificate"); + fs::write(base.path().join(RUSTFS_TLS_KEY), original.signing_key.serialize_pem()).expect("key"); + let path = base.path().to_str().expect("path"); + let initial = TlsMaterialSnapshot::load(&TlsSource::from_directory(base.path())) + .await + .expect("initial"); + let prepared = tls_material::PreparedServerTls::from_snapshot(&initial) + .expect("prepared TLS") + .expect("server"); + let holder = prepared.tcp_acceptor().expect("acceptor"); + let task = tls_material::spawn_reload_loop( + path.into(), + holder, + initial, + prepared, + #[cfg(feature = "http3")] + None, + ) + .expect("registration") + .expect("reload task"); + let bootstrap = state_at(root.path(), path).expect("bootstrap"); + let request = build_request(&bootstrap.target_revision, &pair("server.example")); + let retry = clone_request(&request); + publish_at(root.path(), path, request).expect("first publication"); + let (state, get_counts) = measure_operation(|| state_at(root.path(), path).expect("state")); + let (_, retry_counts) = measure_operation(|| publish_at(root.path(), path, retry).expect("retry")); + let (_, reload_counts) = measure_operation(|| { + let generation = current_generation(root.path()).expect("generation").expect("published"); + generation + .into_snapshot(&TlsSource::from_directory(base.path()).snapshot().expect("source")) + .expect("runtime snapshot") + }); + let (_, publish_counts) = measure_operation(|| { + publish_at(root.path(), path, build_request(&state.target_revision, &pair("server.example"))) + .expect("next publication") + }); + println!("GET={get_counts:?}; RETRY={retry_counts:?}; PUBLISH={publish_counts:?}"); + println!("RELOAD={reload_counts:?}"); + assert_eq!( + reload_counts, + OperationReads { + certificates: 1, + keys: 1, + manifests: 1, + manifest_decodes: 1, + certificate_pem_traversals: 1, + leaf_der_decodes: 1 + } + ); + assert_eq!( + get_counts, + OperationReads { + certificates: 1, + keys: 1, + manifests: 1, + manifest_decodes: 1, + certificate_pem_traversals: 1, + leaf_der_decodes: 1 + } + ); + assert_eq!( + retry_counts, + OperationReads { + certificates: 1, + keys: 1, + manifests: 1, + manifest_decodes: 1, + certificate_pem_traversals: 1, + leaf_der_decodes: 1 + } + ); + assert_eq!( + publish_counts, + OperationReads { + certificates: 1, + keys: 1, + manifests: 1, + manifest_decodes: 1, + certificate_pem_traversals: 2, + leaf_der_decodes: 2 + } + ); + task.abort(); + }, + ) + .await; + } + + #[tokio::test] + #[serial_test::serial] + async fn publication_preserves_trust_rejects_stale_and_recovers_lost_receipt() { + temp_env::async_with_vars( + [ + (ENV_TLS_RELOAD_ENABLE, Some("1")), + (rustfs_config::ENV_SERVER_MTLS_ENABLE, Some("0")), + (ENV_TLS_PUBLISH_DIR, None), + ], + async { + let base = tempfile::TempDir::new().expect("base"); + let root = tempfile::TempDir::new().expect("publication root"); + fs::set_permissions(root.path(), fs::Permissions::from_mode(0o700)).expect("root permissions"); + let original = pair("server.example"); + fs::write(base.path().join(RUSTFS_TLS_CERT), original.cert.pem()).expect("original certificate"); + fs::rename(base.path().join(RUSTFS_TLS_CERT), base.path().join("seed-certificate.pem")) + .expect("symlink seed backing"); + std::os::unix::fs::symlink("seed-certificate.pem", base.path().join(RUSTFS_TLS_CERT)) + .expect("Certbot-style seed symlink"); + fs::write(base.path().join(RUSTFS_TLS_KEY), original.signing_key.serialize_pem()).expect("original key"); + let old_ca = pair("ca-old.example"); + fs::write(base.path().join(rustfs_config::RUSTFS_CA_CERT), old_ca.cert.pem()).expect("trust file"); + let path = base.path().to_str().expect("base path"); + let initial = TlsMaterialSnapshot::load(&TlsSource::from_directory(base.path())) + .await + .expect("initial"); + let prepared = tls_material::PreparedServerTls::from_snapshot(&initial) + .expect("prepared TLS") + .expect("server"); + let holder = prepared.tcp_acceptor().expect("acceptor"); + let task = tls_material::spawn_reload_loop( + path.into(), + holder, + initial, + prepared, + #[cfg(feature = "http3")] + None, + ) + .expect("registration") + .expect("reload task"); + let bootstrap = state_at(root.path(), path).expect("bootstrap state"); + let candidate = pair("server.example"); + let request = build_request(&bootstrap.target_revision, &candidate); + let retry = clone_request(&request); + let mutated = clone_request(&request); + let receipt = publish_at(root.path(), path, request).expect("durable receipt"); + assert_ne!(receipt.target_revision, bootstrap.target_revision); + assert_eq!(receipt.certificate_fingerprint, hex_digest(candidate.cert.der().as_ref())); + assert_eq!( + fs::read(base.path().join(RUSTFS_TLS_CERT)).expect("base certificate"), + original.cert.pem().as_bytes() + ); + let new_ca = pair("ca-new.example"); + fs::write(base.path().join(rustfs_config::RUSTFS_CA_CERT), new_ca.cert.pem()) + .expect("independent trust rotation"); + let directory = current_directory(root.path()).expect("pointer").expect("generation"); + let snapshot = TlsMaterialSnapshot::load_from_snapshot_with_server_directory( + &TlsSource::from_directory(base.path()).snapshot().expect("source"), + Some(&directory), + ) + .expect("overlay snapshot"); + assert_eq!(snapshot.outbound.root_ca_pem, new_ca.cert.pem().as_bytes()); + assert_eq!(snapshot_leaf(&snapshot), Some(receipt.certificate_fingerprint.clone())); + let retry_receipt = publish_at(root.path(), path, retry).expect("lost receipt recovery"); + assert_eq!(retry_receipt.target_revision, receipt.target_revision); + let mut mutated = mutated; + mutated.private_key_pem = pair("server.example").signing_key.serialize_pem(); + assert!(matches!(publish_at(root.path(), path, mutated), Err(PublishError::Conflict))); + assert!(matches!( + publish_at(root.path(), path, build_request(&bootstrap.target_revision, &pair("server.example"))), + Err(PublishError::Conflict) + )); + assert_eq!( + state_at(root.path(), path).expect("accepted state").target_revision, + receipt.target_revision + ); + let expanded = rcgen::generate_simple_self_signed(vec!["server.example".into(), "extra.example".into()]) + .expect("expanded names"); + let expanded_receipt = publish_at(root.path(), path, build_request(&receipt.target_revision, &expanded)) + .expect("pending expanded generation"); + assert!( + matches!( + publish_at(root.path(), path, build_request(&expanded_receipt.target_revision, &candidate)), + Err(PublishError::Invalid) + ), + "a later publication must preserve names of the durable generation even before it is observed" + ); + let barrier = std::sync::Arc::new(std::sync::Barrier::new(2)); + let one = build_request(&expanded_receipt.target_revision, &expanded); + let two = build_request(&expanded_receipt.target_revision, &expanded); + let root_path = root.path(); + let (one, two) = std::thread::scope(|scope| { + let barrier_one = barrier.clone(); + let first = scope.spawn(move || { + barrier_one.wait(); + publish_at(root_path, path, one) + }); + let second = scope.spawn(|| { + barrier.wait(); + publish_at(root_path, path, two) + }); + (first.join().expect("first publisher"), second.join().expect("second publisher")) + }); + assert_eq!(usize::from(one.is_ok()) + usize::from(two.is_ok()), 1, "one concurrent CAS winner"); + assert!( + matches!(one, Err(PublishError::Conflict | PublishError::Busy)) + || matches!(two, Err(PublishError::Conflict | PublishError::Busy)) + ); + let latest = state_at(root.path(), path).expect("concurrent winner"); + let interrupted = build_request(&latest.target_revision, &expanded); + let retry = clone_request(&interrupted); + assert!( + matches!( + publish_at_inner(root.path(), path, interrupted, FailurePoint::AfterPointerRename), + Err(PublishError::Storage) + ), + "failure after rename has an unknown durable outcome" + ); + let recovered = + publish_at(root.path(), path, retry).expect("same issuance recovers and syncs the installed generation"); + assert_ne!(recovered.target_revision, latest.target_revision); + assert!( + current_directory(root.path()) + .expect("installed generation survives error") + .is_some() + ); + let interrupted = build_request(&recovered.target_revision, &expanded); + let before = fs::read_dir(root.path().join("generations")).expect("generations").count(); + assert!(matches!( + publish_at_inner(root.path(), path, interrupted, FailurePoint::AfterPointerRenameUnreadable), + Err(PublishError::Storage) + )); + let after = fs::read_dir(root.path().join("generations")) + .expect("generations after unreadable pointer") + .count(); + assert_eq!(after, before + 1, "an uncertain pointer must not delete the new generation"); + fs::remove_file(root.path().join(CURRENT)).expect("simulate accepted pointer loss"); + assert!( + matches!(current_directory(root.path()), Err(PublishError::Invalid)), + "an acknowledged publisher must never silently revert to bootstrap after pointer loss" + ); + task.abort(); + }, + ) + .await; + } + + #[tokio::test] + #[serial_test::serial] + async fn expired_generation_can_renew_and_next_operation_detects_corruption() { + temp_env::async_with_vars( + [ + (ENV_TLS_RELOAD_ENABLE, Some("1")), + (rustfs_config::ENV_SERVER_MTLS_ENABLE, Some("0")), + (ENV_TLS_PUBLISH_DIR, None), + ], + async { + let base = tempfile::TempDir::new().expect("base"); + let root = tempfile::TempDir::new().expect("publication root"); + fs::set_permissions(root.path(), fs::Permissions::from_mode(0o700)).expect("permissions"); + let original = pair("server.example"); + fs::write(base.path().join(RUSTFS_TLS_CERT), original.cert.pem()).expect("certificate"); + fs::write(base.path().join(RUSTFS_TLS_KEY), original.signing_key.serialize_pem()).expect("key"); + let path = base.path().to_str().expect("path"); + let initial = TlsMaterialSnapshot::load(&TlsSource::from_directory(base.path())) + .await + .expect("initial"); + let prepared = tls_material::PreparedServerTls::from_snapshot(&initial) + .expect("prepared") + .expect("server"); + let holder = prepared.tcp_acceptor().expect("acceptor"); + let task = tls_material::spawn_reload_loop( + path.into(), + holder, + initial, + prepared, + #[cfg(feature = "http3")] + None, + ) + .expect("registration") + .expect("reload task"); + let expired_key = rcgen::KeyPair::generate().expect("expired key"); + let mut params = rcgen::CertificateParams::new(vec!["server.example".into(), "expired-required.example".into()]) + .expect("parameters"); + params.not_before = rcgen::date_time_ymd(2000, 1, 1); + params.not_after = rcgen::date_time_ymd(2001, 1, 1); + let expired = params.self_signed(&expired_key).expect("expired certificate"); + let certificate = expired.pem(); + let key = expired_key.serialize_pem(); + let revision = Uuid::new_v4(); + let generations = root.path().join("generations"); + fs::create_dir(&generations).expect("generations"); + let directory = generations.join(revision.to_string()); + fs::create_dir(&directory).expect("generation"); + write_new(&directory.join(RUSTFS_TLS_CERT), certificate.as_bytes()).expect("expired cert"); + write_new(&directory.join(RUSTFS_TLS_KEY), key.as_bytes()).expect("expired key"); + let leaf = parse_leaf(expired.der().as_ref()).expect("expired metadata"); + let manifest = Manifest { + schema_version: SCHEMA.into(), + issuance_id: Uuid::new_v4(), + target_revision: revision, + certificate_fingerprint: leaf.fingerprint, + not_after_unix: leaf.not_after_unix, + payload_digest: payload_digest(certificate.as_bytes(), key.as_bytes()), + }; + write_new(&directory.join(MANIFEST), &serde_json::to_vec(&manifest).expect("manifest")).expect("manifest file"); + std::os::unix::fs::symlink(Path::new("generations").join(revision.to_string()), root.path().join(CURRENT)) + .expect("pointer"); + mark_initialized(root.path()).expect("initialized"); + let state = state_at(root.path(), path).expect("expired visible head remains readable"); + assert!(state.not_after_unix < 1_000_000_000); + assert!( + current_generation(root.path()) + .expect("expired generation") + .expect("present") + .into_snapshot(&TlsSource::from_directory(base.path()).snapshot().expect("source")) + .is_err(), + "reading an expired ledger must never bypass serving validity checks" + ); + assert!( + matches!( + publish_at(root.path(), path, build_request(&state.target_revision, &pair("server.example"))), + Err(PublishError::Invalid) + ), + "expired durable SAN coverage must survive renewal" + ); + let renewed = + rcgen::generate_simple_self_signed(vec!["server.example".into(), "expired-required.example".into()]) + .expect("renewal"); + let request = build_request(&state.target_revision, &renewed); + let retry = clone_request(&request); + let receipt = publish_at(root.path(), path, request).expect("renew expired generation"); + assert_ne!(receipt.target_revision, state.target_revision); + let generation = current_generation(root.path()).expect("published").expect("generation"); + assert_eq!( + state_at(root.path(), path).expect("first independent read").target_revision, + receipt.target_revision + ); + let directory = root + .path() + .join("generations") + .join(generation.manifest.target_revision.to_string()); + fs::write(directory.join(RUSTFS_TLS_KEY), "corrupted key").expect("corrupt after read"); + let snapshot = generation + .into_snapshot(&TlsSource::from_directory(base.path()).snapshot().expect("source")) + .expect("the pinned operation consumes its already verified material without a second file read"); + assert_eq!(snapshot_leaf(&snapshot), Some(receipt.certificate_fingerprint)); + assert!( + matches!(state_at(root.path(), path), Err(PublishError::Invalid)), + "the next operation must revalidate disk" + ); + assert!( + matches!(publish_at(root.path(), path, retry), Err(PublishError::Invalid)), + "idempotent retry must not bypass disk corruption" + ); + task.abort(); + }, + ) + .await; + } + + #[tokio::test] + #[serial_test::serial] + async fn invalid_candidate_never_changes_pointer() { + temp_env::async_with_vars( + [ + (ENV_TLS_RELOAD_ENABLE, Some("1")), + (rustfs_config::ENV_SERVER_MTLS_ENABLE, Some("0")), + (ENV_TLS_PUBLISH_DIR, None), + ], + async { + let base = tempfile::TempDir::new().expect("base"); + let root = tempfile::TempDir::new().expect("publication root"); + fs::set_permissions(root.path(), fs::Permissions::from_mode(0o700)).expect("root permissions"); + let original = pair("server.example"); + fs::write(base.path().join(RUSTFS_TLS_CERT), original.cert.pem()).expect("certificate"); + fs::write(base.path().join(RUSTFS_TLS_KEY), original.signing_key.serialize_pem()).expect("key"); + let path = base.path().to_str().expect("base path"); + let initial = TlsMaterialSnapshot::load(&TlsSource::from_directory(base.path())) + .await + .expect("initial"); + let prepared = tls_material::PreparedServerTls::from_snapshot(&initial) + .expect("prepared TLS") + .expect("server"); + let holder = prepared.tcp_acceptor().expect("acceptor"); + let task = tls_material::spawn_reload_loop( + path.into(), + holder, + initial, + prepared, + #[cfg(feature = "http3")] + None, + ) + .expect("registration") + .expect("reload task"); + let expected = state_at(root.path(), path).expect("state").target_revision; + assert!(matches!( + publish_at(root.path(), path, build_request(&expected, &pair("wrong.example"))), + Err(PublishError::Invalid) + )); + let valid = pair("server.example"); + let mut mismatch = build_request(&expected, &valid); + mismatch.private_key_pem = pair("server.example").signing_key.serialize_pem(); + assert!(matches!(publish_at(root.path(), path, mismatch), Err(PublishError::Invalid))); + let mut ca_params = rcgen::CertificateParams::new(vec!["server.example".into()]).expect("CA params"); + ca_params.is_ca = rcgen::IsCa::Ca(rcgen::BasicConstraints::Unconstrained); + let ca_key = rcgen::KeyPair::generate().expect("CA key"); + let ca_cert = ca_params.self_signed(&ca_key).expect("CA certificate"); + let mut ca_candidate = build_request(&expected, &valid); + ca_candidate.certificate_pem = ca_cert.pem(); + ca_candidate.private_key_pem = ca_key.serialize_pem(); + assert!( + matches!(publish_at(root.path(), path, ca_candidate), Err(PublishError::Invalid)), + "a CA certificate must not become the serving leaf" + ); + let mut oversized = build_request(&expected, &valid); + oversized.certificate_pem = "x".repeat(MAX_CERT + 1); + assert!(matches!(publish_at(root.path(), path, oversized), Err(PublishError::Invalid))); + assert!(current_directory(root.path()).expect("pointer absent").is_none()); + assert_eq!(state_at(root.path(), path).expect("unchanged").target_revision, expected); + task.abort(); + }, + ) + .await; + } + + #[test] + fn corrupt_or_escaping_pointer_is_rejected() { + let root = tempfile::TempDir::new().expect("root"); + std::os::unix::fs::symlink("../outside", root.path().join(CURRENT)).expect("pointer"); + assert!(matches!(current_directory(root.path()), Err(PublishError::Invalid))); + fs::remove_file(root.path().join(CURRENT)).expect("remove pointer"); + fs::write(root.path().join(CURRENT), "generations/foo").expect("regular pointer"); + assert!(matches!(current_directory(root.path()), Err(PublishError::Invalid))); + } + + #[tokio::test] + #[serial_test::serial] + async fn publication_is_observed_by_live_tls_handshakes() { + let base = tempfile::TempDir::new().expect("base"); + let root = tempfile::TempDir::new().expect("publication root"); + fs::set_permissions(root.path(), fs::Permissions::from_mode(0o700)).expect("root permissions"); + let original = pair("server.example"); + let renewed = pair("server.example"); + fs::write(base.path().join(RUSTFS_TLS_CERT), original.cert.pem()).expect("certificate"); + fs::write(base.path().join(RUSTFS_TLS_KEY), original.signing_key.serialize_pem()).expect("key"); + temp_env::async_with_vars( + [ + (ENV_TLS_RELOAD_ENABLE, Some("1")), + (ENV_TLS_PUBLISH_DIR, root.path().to_str()), + (rustfs_config::ENV_SERVER_MTLS_ENABLE, Some("0")), + (rustfs_config::ENV_TRUST_SYSTEM_CA, Some("0")), + (rustfs_config::ENV_TRUST_LEAF_CERT_AS_CA, Some("0")), + (rustfs_config::ENV_RUSTFS_TLS_PATH, Some("ignored-env-source")), + ], + async { + let path = base.path().to_str().expect("path"); + let initial = tls_material::load_tls_material(path).await.expect("initial snapshot"); + let prepared = tls_material::PreparedServerTls::from_snapshot(&initial) + .expect("prepared TLS") + .expect("server"); + let holder = prepared.tcp_acceptor().expect("acceptor"); + let registration = tls_material::spawn_reload_loop( + path.into(), + holder.clone(), + initial, + prepared, + #[cfg(feature = "http3")] + None, + ) + .expect("registration") + .expect("reload task"); + assert_eq!( + tls_material::active_server_tls_path(), + Some(path.to_string()), + "registered CLI/listener source wins over environment" + ); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("listener"); + let address = listener.local_addr().expect("address"); + let server = tokio::spawn(async move { + for _ in 0..2 { + let (socket, _) = listener.accept().await.expect("TCP connection"); + let _tls = holder.get().accept(socket).await.expect("TLS handshake"); + } + }); + let mut roots = rustls::RootCertStore::empty(); + roots.add(original.cert.der().clone()).expect("bootstrap anchor"); + roots.add(renewed.cert.der().clone()).expect("renewal anchor"); + let config = rustls::ClientConfig::builder() + .with_root_certificates(roots) + .with_no_client_auth(); + let connector = tokio_rustls::TlsConnector::from(std::sync::Arc::new(config)); + async fn handshake(connector: &tokio_rustls::TlsConnector, address: std::net::SocketAddr) -> String { + let socket = tokio::net::TcpStream::connect(address).await.expect("connect"); + let stream = connector + .connect(rustls::pki_types::ServerName::try_from("server.example").expect("SNI"), socket) + .await + .expect("verified handshake"); + hex_digest(stream.get_ref().1.peer_certificates().expect("peer chain")[0].as_ref()) + } + assert_eq!(handshake(&connector, address).await, hex_digest(original.cert.der().as_ref())); + let bootstrap = certificate_state(path.into()).await.expect("bootstrap state"); + let receipt = publish(path.into(), build_request(&bootstrap.target_revision, &renewed)) + .await + .expect("publish"); + let expected = hex_digest(renewed.cert.der().as_ref()); + assert_eq!(receipt.certificate_fingerprint, expected); + tokio::time::timeout(std::time::Duration::from_secs(5), async { + loop { + if certificate_state(path.into()) + .await + .expect("state") + .observed_certificate_fingerprint + == Some(expected.clone()) + { + break; + } + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + }) + .await + .expect("publication observed by serving group"); + assert_eq!(handshake(&connector, address).await, expected); + let mut expected_chain = vec![renewed.cert.der().clone()]; + assert_eq!(observed(path, &expected_chain), Some(expected.clone())); + expected_chain.push(original.cert.der().clone()); + assert!( + observed(path, &expected_chain).is_none(), + "matching leaf alone must not acknowledge a different serving chain" + ); + server.await.expect("TLS server"); + registration.abort(); + }, + ) + .await; + } +} diff --git a/rustfs/src/site_replication/tests.rs b/rustfs/src/site_replication/tests.rs index 4f20ec2b3..e5c38d77e 100644 --- a/rustfs/src/site_replication/tests.rs +++ b/rustfs/src/site_replication/tests.rs @@ -3569,11 +3569,6 @@ fn test_reconcile_site_replication_bucket_targets_allows_peer_on_same_port_as_lo }); } -#[test] -fn test_hash_client_secret_matches_minio_style_base64url_sha256() { - assert_eq!(hash_client_secret(Some("secret")), "K7gNU3sdo-OL0wNhqoVWhr3g6s1xYv72ol_pe_Unols"); -} - #[test] fn test_site_replication_peer_client_cache_hit_generation_mismatch_returns_none() { let cache = Some(SiteReplicationPeerClientCache { diff --git a/rustfs/src/site_replication/transport.rs b/rustfs/src/site_replication/transport.rs index 18d4850b9..8a33c8585 100644 --- a/rustfs/src/site_replication/transport.rs +++ b/rustfs/src/site_replication/transport.rs @@ -374,16 +374,6 @@ pub(crate) fn runtime_tls_enabled() -> bool { runtime_tls_enabled_with(endpoints.as_ref()) } -pub(crate) fn hash_client_secret(secret: Option<&str>) -> String { - let Some(secret) = secret.filter(|secret| !secret.is_empty()) else { - return String::new(); - }; - - let mut hasher = Sha256::new(); - hasher.update(secret.as_bytes()); - URL_SAFE_NO_PAD.encode_to_string(hasher.finalize()) -} - pub(crate) fn loopback_replication_targets_allowed() -> bool { std::env::var(ALLOW_LOOPBACK_REPLICATION_TARGET_ENV) .map(|value| value.eq_ignore_ascii_case("true") || value == "1") diff --git a/rustfs/src/startup_auth.rs b/rustfs/src/startup_auth.rs index b3b10e4f8..c704a9fbf 100644 --- a/rustfs/src/startup_auth.rs +++ b/rustfs/src/startup_auth.rs @@ -13,19 +13,18 @@ // limitations under the License. use rustfs_iam::{ - federation::{ - CoreFederatedAuthorizationMapper, FederatedIdentityService, - oidc::{OidcConfigQuery, StandardOidcAdapter}, - }, - get_oidc, init_oidc_sys_with_extra_root_ca_provider, - oidc::{OidcExtraRootCaMaterial, OidcExtraRootCaProvider, OidcSys}, + build_oidc_sys_with_extra_root_ca_provider, + federation::{CoreFederatedAuthorizationMapper, FederatedIdentityService}, + oidc::{OidcConfigQuery, OidcExtraRootCaMaterial, OidcExtraRootCaProvider, OidcSys, StandardOidcAdapter}, }; use std::{ collections::hash_map::DefaultHasher, + future::Future, hash::{Hash, Hasher}, io::{Error, Result}, sync::Arc, }; +use tokio_util::{sync::CancellationToken, task::AbortOnDropHandle}; use tracing::{error, info, warn}; const LOG_COMPONENT_MAIN: &str = "main"; @@ -34,7 +33,7 @@ const EVENT_KEYSTONE_AUTH_INITIALIZED: &str = "keystone_auth_initialized"; const EVENT_KEYSTONE_AUTH_INITIALIZATION_FAILED: &str = "keystone_auth_initialization_failed"; const EVENT_OIDC_INITIALIZATION_FAILED: &str = "oidc_initialization_failed"; -pub(crate) async fn init_auth_integrations() -> Result<()> { +pub(crate) async fn init_auth_integrations(shutdown: CancellationToken) -> Result>> { let keystone_config = rustfs_keystone::KeystoneConfig::from_env().map_err(Error::other)?; if keystone_config.enable { match crate::auth_keystone::init_keystone_auth(keystone_config).await { @@ -56,11 +55,14 @@ pub(crate) async fn init_auth_integrations() -> Result<()> { } } - match init_oidc_sys_with_extra_root_ca_provider(oidc_extra_root_ca_provider()).await { - Ok(()) => { - if let Some(oidc) = get_oidc() { - let (service, oidc_config_query) = standard_oidc_runtime(oidc); - crate::runtime_sources::publish_federated_identity_runtime(service, oidc_config_query); + match build_oidc_sys_with_extra_root_ca_provider(oidc_extra_root_ca_provider()).await { + Ok(oidc) => { + let (service, oidc_config_query) = standard_oidc_runtime(oidc.clone()); + if crate::runtime_sources::publish_federated_identity_runtime(service, oidc_config_query) { + return Ok(Some(spawn_oidc_discovery_recovery( + async move { oidc.recover_failed_providers().await }, + shutdown, + ))); } } Err(e) => { @@ -74,7 +76,20 @@ pub(crate) async fn init_auth_integrations() -> Result<()> { } } - Ok(()) + Ok(None) +} + +fn spawn_oidc_discovery_recovery( + recovery: impl Future + Send + 'static, + shutdown: CancellationToken, +) -> AbortOnDropHandle<()> { + AbortOnDropHandle::new(tokio::spawn(async move { + tokio::select! { + biased; + () = shutdown.cancelled() => {} + () = recovery => {} + } + })) } fn standard_oidc_runtime(oidc: Arc) -> (Arc, Arc) { @@ -138,4 +153,48 @@ mod tests { drop(oidc_config_query); assert_eq!(Arc::strong_count(&retained), 1, "dropping the runtime should release its OIDC Arc"); } + + #[tokio::test] + async fn oidc_recovery_shutdown_releases_pending_work() { + let shutdown = CancellationToken::new(); + let (started_tx, started_rx) = tokio::sync::oneshot::channel(); + let (released_tx, released_rx) = tokio::sync::oneshot::channel::<()>(); + let recovery = spawn_oidc_discovery_recovery( + async move { + let _release = released_tx; + let _ = started_tx.send(()); + std::future::pending::<()>().await; + }, + shutdown.child_token(), + ); + started_rx.await.expect("recovery started"); + shutdown.cancel(); + tokio::time::timeout(std::time::Duration::from_secs(5), recovery) + .await + .expect("recovery observes shutdown") + .expect("recovery joins cleanly"); + assert!(released_rx.await.is_err(), "shutdown drops in-flight recovery work"); + } + + #[tokio::test] + async fn oidc_recovery_owner_drop_releases_pending_work() { + let (started_tx, started_rx) = tokio::sync::oneshot::channel(); + let (released_tx, released_rx) = tokio::sync::oneshot::channel::<()>(); + let recovery = spawn_oidc_discovery_recovery( + async move { + let _release = released_tx; + let _ = started_tx.send(()); + std::future::pending::<()>().await; + }, + CancellationToken::new(), + ); + started_rx.await.expect("recovery started"); + drop(recovery); + assert!( + tokio::time::timeout(std::time::Duration::from_secs(5), released_rx) + .await + .expect("owner drop aborts recovery") + .is_err() + ); + } } diff --git a/rustfs/src/startup_bucket_metadata.rs b/rustfs/src/startup_bucket_metadata.rs index ad77a0a79..eed955cff 100644 --- a/rustfs/src/startup_bucket_metadata.rs +++ b/rustfs/src/startup_bucket_metadata.rs @@ -15,7 +15,6 @@ use crate::app::object::OnDemandMigrationWriteBack; use crate::module_switches::{on_demand_migration_enabled_from_env, set_on_demand_migration_module_enabled}; use crate::on_demand_migration::OnDemandMigrationSys; -use crate::storage_api::startup::bucket_metadata::contract::bucket::{BucketOperations, BucketOptions}; use crate::storage_api::startup::bucket_metadata::{ ECStore, Error as StorageError, Result as StorageResult, get_global_replication_pool, init_bucket_metadata_sys, reconcile_bucket_resync_target_intents, try_migrate_bucket_metadata, try_migrate_iam_config, @@ -58,19 +57,13 @@ const STARTUP_BACKGROUND_STATUS_CANCELED: f64 = 3.0; pub(crate) async fn init_embedded_bucket_metadata_runtime(store: Arc, ctx: &CancellationToken) -> IoResult> { let buckets_list = retry_metadata_startup(ctx, "list_buckets", || async { - store - .list_bucket(&BucketOptions { - no_metadata: true, - ..Default::default() - }) - .await - .map_err(IoError::other) + store.list_buckets_for_metadata_startup().await.map_err(IoError::other) }) .await?; let buckets: Vec = buckets_list.into_iter().map(|v| v.name).collect(); - retry_metadata_startup(ctx, "bucket_migration", || try_migrate_bucket_metadata(store.clone())).await?; + retry_metadata_startup(ctx, "bucket_migration", || try_migrate_bucket_metadata(store.clone(), &buckets)).await?; init_on_demand_migration_runtime(); init_bucket_metadata_sys(store.clone(), buckets.clone()).await; retry_metadata_startup(ctx, "iam_migration", || try_migrate_iam_config(store.clone())).await?; @@ -81,19 +74,13 @@ pub(crate) async fn init_embedded_bucket_metadata_runtime(store: Arc, c pub(crate) async fn init_bucket_metadata_runtime(store: Arc, ctx: CancellationToken) -> IoResult> { let buckets_list = retry_metadata_startup(&ctx, "list_buckets", || async { - store - .list_bucket(&BucketOptions { - no_metadata: true, - ..Default::default() - }) - .await - .map_err(IoError::other) + store.list_buckets_for_metadata_startup().await.map_err(IoError::other) }) .await?; let buckets: Vec = buckets_list.into_iter().map(|v| v.name).collect(); - retry_metadata_startup(&ctx, "bucket_migration", || try_migrate_bucket_metadata(store.clone())).await?; + retry_metadata_startup(&ctx, "bucket_migration", || try_migrate_bucket_metadata(store.clone(), &buckets)).await?; retry_metadata_startup(&ctx, "iam_migration", || try_migrate_iam_config(store.clone())).await?; init_on_demand_migration_runtime(); diff --git a/rustfs/src/startup_entrypoint.rs b/rustfs/src/startup_entrypoint.rs index e2402b2a4..adf5f056d 100644 --- a/rustfs/src/startup_entrypoint.rs +++ b/rustfs/src/startup_entrypoint.rs @@ -39,7 +39,41 @@ const LOG_SUBSYSTEM_STARTUP: &str = "startup"; const EVENT_SERVER_RUNTIME_FAILED: &str = "server_runtime_failed"; const OBSERVABILITY_INIT_FATAL_ALREADY_REPORTED: &str = "observability initialization failure already reported"; +/// Thread name for the thread that drives the root startup future. +const STARTUP_THREAD_NAME: &str = "rustfs-main"; + +/// Stack reserved for the thread that drives the root startup future. +/// +/// `block_on` polls `async_main` on the calling thread, so the whole startup +/// chain runs on that thread's stack rather than on a Tokio worker. The native +/// main thread only gets what the platform hands out: 8 MiB from `ulimit -s` +/// on Linux and macOS, but the 1 MiB PE default on Windows, where a debug +/// build overflows before it starts listening. `RUST_MIN_STACK` does not apply +/// to the native main thread, so the budget is set here explicitly instead. +/// Startup currently peaks near 2 MiB in a debug build and under 1 MiB in a +/// release build; 8 MiB keeps the budget Linux and macOS already had. +const STARTUP_THREAD_STACK_SIZE: usize = 8 * 1024 * 1024; + pub fn run_process() { + run_on_startup_thread(run_process_on_current_thread); +} + +/// Run `f` on a thread whose stack is [`STARTUP_THREAD_STACK_SIZE`] on every +/// platform, propagating its panic so the process exit status is unchanged. +fn run_on_startup_thread(f: impl FnOnce() -> T + Send + 'static) -> T { + // Failing to spawn the startup thread is a startup fatal boundary. + let handle = std::thread::Builder::new() + .name(STARTUP_THREAD_NAME.to_owned()) + .stack_size(STARTUP_THREAD_STACK_SIZE) + .spawn(f) + .expect("Failed to spawn startup thread"); + match handle.join() { + Ok(value) => value, + Err(panic) => std::panic::resume_unwind(panic), + } +} + +fn run_process_on_current_thread() { // Building the process runtime is a startup fatal boundary. let (runtime, dial9_guard) = crate::server::build_tokio_runtime().expect("Failed to build Tokio runtime"); let result = runtime.block_on(async_main()); @@ -2034,4 +2068,36 @@ mod tests { "[FATAL] Observability initialization failed: collector unavailable" ); } + + /// Consume roughly `bytes` of stack through non-elidable recursive frames. + #[inline(never)] + fn consume_stack(bytes: usize) -> u8 { + const FRAME: usize = 64 * 1024; + let mut frame = [0_u8; FRAME]; + std::hint::black_box(&mut frame); + if bytes <= FRAME { + return frame[0]; + } + frame[0].wrapping_add(consume_stack(bytes - FRAME)) + } + + #[test] + fn startup_thread_provides_documented_stack_budget() { + // Half the budget exceeds both the 2 MiB Rust thread default and the + // 1 MiB Windows main thread, so this overflows unless the startup + // thread really gets its explicit budget. + let used = STARTUP_THREAD_STACK_SIZE / 2; + let name = run_on_startup_thread(move || { + std::hint::black_box(consume_stack(used)); + std::thread::current().name().map(str::to_owned) + }); + assert_eq!(name.as_deref(), Some(STARTUP_THREAD_NAME)); + } + + #[test] + fn startup_thread_propagates_panics() { + let result = std::panic::catch_unwind(|| run_on_startup_thread(|| panic!("startup thread panic"))); + let payload = result.expect_err("startup thread panic must propagate"); + assert_eq!(payload.downcast_ref::<&str>(), Some(&"startup thread panic")); + } } diff --git a/rustfs/src/startup_fs_guard.rs b/rustfs/src/startup_fs_guard.rs index 53964f2e2..918baebcf 100644 --- a/rustfs/src/startup_fs_guard.rs +++ b/rustfs/src/startup_fs_guard.rs @@ -17,6 +17,7 @@ use rustfs_config::{ DEFAULT_RUSTFS_UNSUPPORTED_FS_POLICY, ENV_RUSTFS_UNSUPPORTED_FS_POLICY, RUSTFS_UNSUPPORTED_FS_POLICY_FAIL, RUSTFS_UNSUPPORTED_FS_POLICY_WARN, }; +use rustfs_utils::os::DiskInfo; use std::collections::BTreeSet; use std::io::{Error, Result}; use tracing::warn; @@ -60,6 +61,17 @@ fn is_unsupported_fs_type(fs_type: &str) -> bool { ) || normalized.starts_with("fuse.") } +/// Describes why the filesystem behind `info` is outside the supported +/// boundary, or returns `None` when it is supported. +fn unsupported_fs_label(info: &DiskInfo) -> Option { + if info.remote { + // Windows names the server's filesystem for SMB shares, which would + // otherwise pass as a local NTFS volume. + return Some(format!("{} on network share", info.fstype)); + } + is_unsupported_fs_type(&info.fstype).then(|| info.fstype.clone()) +} + fn collect_local_paths(endpoint_pools: &EndpointServerPools) -> Vec { let mut local_paths = BTreeSet::new(); for pool in endpoint_pools.as_ref() { @@ -92,8 +104,8 @@ pub(crate) fn enforce_unsupported_fs_policy(endpoint_pools: &EndpointServerPools } }; - if is_unsupported_fs_type(&info.fstype) { - unsupported.push((path, info.fstype)); + if let Some(label) = unsupported_fs_label(&info) { + unsupported.push((path, label)); } } @@ -103,7 +115,7 @@ pub(crate) fn enforce_unsupported_fs_policy(endpoint_pools: &EndpointServerPools let detail = unsupported .iter() - .map(|(path, fs_type)| format!("{path} ({fs_type})")) + .map(|(path, label)| format!("{path} ({label})")) .collect::>() .join(", "); @@ -150,6 +162,29 @@ mod tests { assert!(!is_unsupported_fs_type("BTRFS")); } + #[test] + fn remote_volume_is_unsupported_whatever_its_fstype() { + // An SMB share on Windows reports the server's filesystem name. + let share = DiskInfo { + fstype: "NTFS".to_string(), + remote: true, + ..Default::default() + }; + assert_eq!(unsupported_fs_label(&share).as_deref(), Some("NTFS on network share")); + + let local = DiskInfo { + fstype: "NTFS".to_string(), + ..Default::default() + }; + assert_eq!(unsupported_fs_label(&local), None); + + let cifs = DiskInfo { + fstype: "CIFS".to_string(), + ..Default::default() + }; + assert_eq!(unsupported_fs_label(&cifs).as_deref(), Some("CIFS")); + } + #[test] fn policy_parser_handles_valid_and_invalid_values() { assert_eq!(UnsupportedFsPolicy::parse("warn"), Some(UnsupportedFsPolicy::Warn)); diff --git a/rustfs/src/startup_lifecycle.rs b/rustfs/src/startup_lifecycle.rs index 4f9e9fe56..480d07927 100644 --- a/rustfs/src/startup_lifecycle.rs +++ b/rustfs/src/startup_lifecycle.rs @@ -21,7 +21,7 @@ use crate::{ startup_iam::{IamBootstrapDisposition, publish_ready_for_iam_bootstrap}, startup_runtime_sources, startup_services::StartupServiceRuntime, - startup_shutdown::run_startup_shutdown_sequence, + startup_shutdown::{StartupBackgroundTasks, run_startup_shutdown_sequence}, }; use rustfs_common::GlobalReadiness; use rustfs_object_capacity::capacity_manager::CapacityBackgroundTasks; @@ -130,6 +130,8 @@ pub(crate) async fn run_startup_runtime_lifecycle(lifecycle: StartupRuntimeLifec readiness, } = lifecycle; let StartupServiceRuntime { + table_maintenance, + oidc_recovery, optional_runtimes, heartbeat, inventory, @@ -161,10 +163,25 @@ pub(crate) async fn run_startup_runtime_lifecycle(lifecycle: StartupRuntimeLifec shutdown_signal, (s3_shutdown_tx, console_shutdown_tx), optional_runtimes, - capacity_tasks, + StartupBackgroundTasks { + capacity: capacity_tasks, + table_maintenance, + }, shutdown_token, ) .await; + if let Some(recovery) = oidc_recovery + && let Err(error) = recovery.await + { + tracing::warn!( + event = EVENT_SERVER_SHUTDOWN_STATE, + component = LOG_COMPONENT_MAIN, + subsystem = LOG_SUBSYSTEM_STARTUP, + result = "oidc_discovery_recovery_join_failed", + error = %error, + "OIDC discovery recovery task failed to join" + ); + } shutdown_connect_runtimes(heartbeat, inventory).await; if let Some(runtime) = local_trace_capture { runtime.shutdown().await; diff --git a/rustfs/src/startup_observability.rs b/rustfs/src/startup_observability.rs index 01ec050b8..114b1a808 100644 --- a/rustfs/src/startup_observability.rs +++ b/rustfs/src/startup_observability.rs @@ -13,12 +13,16 @@ // limitations under the License. use std::sync::Arc; +use std::time::Duration; use crate::init::{init_auto_tuner, init_update_check, print_server_info}; use crate::startup_runtime_sources; use crate::storage_api::startup::storage::{ECStore, init_compression_total_memory_from_backend}; +use tokio::io::AsyncWriteExt; use tokio_util::sync::CancellationToken; +const GET_STAGE_LOCAL_SUMMARY_FILE: &str = "/var/log/rustfs/get-stage-summary.log"; + pub(crate) async fn init_observability_runtime(store: Arc, ctx: CancellationToken) { print_server_info(); init_update_check(); @@ -27,6 +31,10 @@ pub(crate) async fn init_observability_runtime(store: Arc, ctx: Cancell let metrics_enabled = startup_runtime_sources::observability_metric_enabled(); configure_metric_gates(metrics_enabled); + if rustfs_io_metrics::get_stage_local_summary_enabled() { + init_get_stage_local_summary_runtime(ctx.clone()); + } + if metrics_enabled { // Load persisted compression stats into memory early, before any PUTs can occur. init_compression_total_memory_from_backend(store).await; @@ -37,21 +45,115 @@ pub(crate) async fn init_observability_runtime(store: Arc, ctx: Cancell } fn configure_metric_gates(metrics_enabled: bool) { + let local_summary_enabled = !metrics_enabled + && rustfs_utils::get_env_bool(rustfs_config::observability::ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED, false); + let sample_rate = + rustfs_utils::get_env_u64(rustfs_config::observability::ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE, 64).max(1); + startup_runtime_sources::set_get_stage_local_summary_enabled(local_summary_enabled); + startup_runtime_sources::set_get_stage_local_summary_sample_rate(sample_rate); let put_stage_metrics_enabled = metrics_enabled && rustfs_utils::get_env_bool( rustfs_config::observability::ENV_OBS_PUT_STAGE_METRICS_ENABLED, rustfs_config::DEFAULT_OBS_PUT_STAGE_METRICS_ENABLED, ); startup_runtime_sources::set_put_stage_metrics_enabled(put_stage_metrics_enabled); - startup_runtime_sources::set_get_stage_metrics_enabled(metrics_enabled); + startup_runtime_sources::set_get_stage_metrics_enabled(metrics_enabled || local_summary_enabled); startup_runtime_sources::set_metrics_enabled(metrics_enabled); } +fn init_get_stage_local_summary_runtime(ctx: CancellationToken) { + let interval_secs = + rustfs_utils::get_env_u64(rustfs_config::observability::ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS, 15).max(1); + tokio::spawn(async move { + let mut interval = tokio::time::interval(Duration::from_secs(interval_secs)); + interval.tick().await; + loop { + tokio::select! { + _ = ctx.cancelled() => break, + _ = interval.tick() => { + let rows = rustfs_io_metrics::take_get_stage_local_summary(); + if rows.is_empty() { + continue; + } + let summary = rows.iter().map(|row| { + format!("{}|{}|{}|{}|{}|{}", row.path, row.stage, row.object_class, row.size_bucket, row.sampled_count, row.duration_nanoseconds) + }).collect::>().join(";"); + let timestamp_ms = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_millis(); + let line = format!( + "timestamp_ms={timestamp_ms}\tsample_rate={}\tstage_count={}\tstages={summary}\n", + rustfs_io_metrics::get_stage_local_summary_sample_rate(), + rows.len(), + ); + let Ok(mut file) = tokio::fs::OpenOptions::new() + .create(true) + .append(true) + .open(GET_STAGE_LOCAL_SUMMARY_FILE) + .await + else { + break; + }; + if file.write_all(line.as_bytes()).await.is_err() { + break; + } + } + } + } + }); +} + #[cfg(test)] mod tests { use super::*; const PUT_STAGE_ENV: &str = rustfs_config::observability::ENV_OBS_PUT_STAGE_METRICS_ENABLED; + const LOCAL_SUMMARY_ENV: &str = rustfs_config::observability::ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED; + const LOCAL_SUMMARY_RATE_ENV: &str = rustfs_config::observability::ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE; + + #[test] + #[serial_test::serial] + fn get_stage_local_summary_can_run_without_metrics_export() { + let previous_metrics = rustfs_io_metrics::metrics_enabled(); + let previous_get_stages = rustfs_io_metrics::get_stage_metrics_enabled(); + let previous_local_summary = rustfs_io_metrics::get_stage_local_summary_enabled(); + let previous_sample_rate = rustfs_io_metrics::get_stage_local_summary_sample_rate(); + + temp_env::with_var(LOCAL_SUMMARY_ENV, Some("true"), || { + temp_env::with_var(LOCAL_SUMMARY_RATE_ENV, Some("1"), || { + configure_metric_gates(false); + assert!(!rustfs_io_metrics::metrics_enabled()); + assert!(rustfs_io_metrics::get_stage_metrics_enabled()); + assert!(rustfs_io_metrics::get_stage_local_summary_enabled()); + rustfs_io_metrics::take_get_stage_local_summary(); + + rustfs_io_metrics::record_get_object_stage_duration("legacy_duplex", "metadata", 0.001); + rustfs_io_metrics::record_get_object_stage_duration("legacy_duplex", "metadata", 0.003); + rustfs_io_metrics::record_get_object_stage_duration_by_size( + "legacy_duplex", + "inline_direct.decode", + "plain_single_part", + "le_4kib", + 0.002, + ); + + let rows = rustfs_io_metrics::take_get_stage_local_summary(); + let metadata = rows + .iter() + .find(|row| row.path == "legacy_duplex" && row.stage == "metadata") + .expect("local summary should retain sampled metadata observations"); + assert_eq!(metadata.sampled_count, 2); + assert_eq!(metadata.duration_nanoseconds, 4_000_000); + assert!(rows.iter().any(|row| row.size_bucket == "le_4kib")); + }); + }); + + startup_runtime_sources::set_metrics_enabled(previous_metrics); + startup_runtime_sources::set_get_stage_metrics_enabled(previous_get_stages); + startup_runtime_sources::set_get_stage_local_summary_enabled(previous_local_summary); + startup_runtime_sources::set_get_stage_local_summary_sample_rate(previous_sample_rate); + } #[test] #[serial_test::serial] diff --git a/rustfs/src/startup_runtime_sources.rs b/rustfs/src/startup_runtime_sources.rs index 30b35458f..f69546688 100644 --- a/rustfs/src/startup_runtime_sources.rs +++ b/rustfs/src/startup_runtime_sources.rs @@ -17,6 +17,7 @@ use crate::runtime_sources::current_outbound_tls_generation as runtime_current_o use crate::storage_api::startup::runtime_sources::{InstanceContext, set_global_rustfs_port}; use rustfs_kms::KmsServiceManager; use rustfs_obs::{GlobalError as ObservabilityError, OtelGuard}; +pub(crate) use rustfs_tls_runtime::load_global_outbound_tls_state as current_outbound_tls_state; use rustfs_tls_runtime::{OutboundTlsMaterial, TlsGeneration}; use std::sync::Arc; use tokio_util::sync::CancellationToken; @@ -98,6 +99,14 @@ pub(crate) fn set_get_stage_metrics_enabled(enabled: bool) { rustfs_io_metrics::set_get_stage_metrics_enabled(enabled); } +pub(crate) fn set_get_stage_local_summary_enabled(enabled: bool) { + rustfs_io_metrics::set_get_stage_local_summary_enabled(enabled); +} + +pub(crate) fn set_get_stage_local_summary_sample_rate(sample_rate: u64) { + rustfs_io_metrics::set_get_stage_local_summary_sample_rate(sample_rate); +} + pub(crate) fn set_metrics_enabled(enabled: bool) { rustfs_io_metrics::set_metrics_enabled(enabled); } diff --git a/rustfs/src/startup_services.rs b/rustfs/src/startup_services.rs index 5dd0fe9a7..acf2b7a51 100644 --- a/rustfs/src/startup_services.rs +++ b/rustfs/src/startup_services.rs @@ -36,13 +36,15 @@ use crate::{ }; use rustfs_common::GlobalReadiness; use std::{collections::BTreeSet, io::Result, sync::Arc}; -use tokio_util::sync::CancellationToken; +use tokio_util::{sync::CancellationToken, task::AbortOnDropHandle}; const EVENT_DRIVE_UNAVAILABLE: &str = "drive_unavailable"; const LOG_COMPONENT_CONNECT: &str = "connect"; const LOG_SUBSYSTEM_INVENTORY: &str = "inventory"; pub(crate) struct StartupServiceRuntime { + pub(crate) table_maintenance: Option, + pub(crate) oidc_recovery: Option>, pub(crate) optional_runtimes: OptionalRuntimeServices, pub(crate) heartbeat: Option, pub(crate) inventory: Option, @@ -82,6 +84,7 @@ pub(crate) async fn init_startup_runtime_services( state_manager: Arc, server_ctx: Arc, ) -> Result { + let maintenance_config = crate::table_catalog::PeriodicMaintenanceConfig::from_env().map_err(std::io::Error::other)?; init_kms_system(config, store.clone()).await?; let optional_runtimes = init_optional_runtime_services(server_ctx.clone()).await?; @@ -101,7 +104,7 @@ pub(crate) async fn init_startup_runtime_services( init_deadlock_detector_runtime(); let buckets = init_bucket_metadata_runtime(store.clone(), ctx.clone()).await?; - let iam_bootstrap = init_iam_runtime(store.clone(), ctx.clone(), readiness, state_manager, server_ctx).await?; + let iam_bootstrap = init_iam_runtime(store.clone(), ctx.clone(), readiness, state_manager, server_ctx.clone()).await?; // Audit initialization requires the AppContext (server config + object store) // which is published by ensure_startup_after_iam inside init_iam_runtime. @@ -111,14 +114,19 @@ pub(crate) async fn init_startup_runtime_services( // self-pointing replication rules until the next restart. The task waits for IAM and // bucket metadata itself, and its first pass runs off this path. spawn_site_replication_reconcile_task(ctx.clone()); - init_auth_integrations().await?; + let oidc_recovery = init_auth_integrations(ctx.child_token()).await?; init_notification_runtime(endpoint_pools, buckets).await?; let enable_scanner = init_background_service_runtime(store.clone()).await?; init_observability_runtime(store.clone(), ctx.clone()).await; let heartbeat = start_heartbeat_runtime(heartbeat_config.clone(), heartbeat_nodes, &ctx)?; let inventory = start_inventory_runtime(heartbeat_config, heartbeat_nodes, inventory_drives, store, &ctx)?; + let table_maintenance = + maintenance_config.map(|config| crate::table_catalog::PeriodicMaintenanceRuntime::spawn(config, server_ctx, &ctx)); + Ok(StartupServiceRuntime { + table_maintenance, + oidc_recovery, optional_runtimes, heartbeat, inventory, diff --git a/rustfs/src/startup_shutdown.rs b/rustfs/src/startup_shutdown.rs index 957d8da03..405eb34c1 100644 --- a/rustfs/src/startup_shutdown.rs +++ b/rustfs/src/startup_shutdown.rs @@ -206,10 +206,10 @@ fn background_shutdown_steps(enable_scanner: bool, enable_heal: bool) -> Vec( - heal_handoff_complete: bool, + background_handoff_complete: bool, detached_mutation_timeout: std::time::Duration, wait_for_detached_mutations: W, clear_unclean_shutdown_markers: ClearFut, @@ -233,22 +233,31 @@ where "Detached mutations did not finish before shutdown; retaining unclean-shutdown markers" ); } - if !(heal_handoff_complete && detached_mutations_drained) { + if !(background_handoff_complete && detached_mutations_drained) { return false; } clear_unclean_shutdown_markers.await; true } +pub(crate) struct StartupBackgroundTasks { + pub(crate) capacity: Option, + pub(crate) table_maintenance: Option, +} + pub(crate) async fn run_startup_shutdown_sequence( state_manager: &ServiceStateManager, shutdown_signal: ShutdownSignal, listener_shutdown_handles: (Option, Option), optional_runtimes: OptionalRuntimeServices, - capacity_tasks: Option, + background_tasks: StartupBackgroundTasks, ctx: CancellationToken, ) -> std::io::Result<()> { let (s3_shutdown_handle, console_shutdown_handle) = listener_shutdown_handles; + let StartupBackgroundTasks { + capacity: capacity_tasks, + table_maintenance, + } = background_tasks; ctx.cancel(); // Stop long-lived peer/disk background monitors while the runtime and tracing @@ -463,10 +472,14 @@ pub(crate) async fn run_startup_shutdown_sequence( console_shutdown_handle.shutdown().await; } shutdown_optional_runtime_services(optional_runtime_shutdowns).await; + let maintenance_drained = match table_maintenance { + Some(runtime) => runtime.shutdown(SHUTDOWN_TIMEOUT).await, + None => true, + }; // FTP, SFTP and WebDAV can start deletes too: wait for detached mutations // only after every front end has stopped. finish_detached_mutations_and_clear_unclean_shutdown_markers( - heal_handoff_complete, + heal_handoff_complete && maintenance_drained, DETACHED_MUTATION_SHUTDOWN_TIMEOUT, wait_for_detached_mutations, rustfs_heal::heal::clear_unclean_shutdown_markers(), diff --git a/rustfs/src/startup_tls_material.rs b/rustfs/src/startup_tls_material.rs index 9d770293e..32931c707 100644 --- a/rustfs/src/startup_tls_material.rs +++ b/rustfs/src/startup_tls_material.rs @@ -14,7 +14,9 @@ use crate::config::Config; use crate::startup_runtime_sources; +use std::collections::BTreeSet; use std::io::{Error, Result}; +use std::sync::Mutex; use tracing::{error, info}; const LOG_COMPONENT_MAIN: &str = "main"; @@ -23,6 +25,25 @@ const EVENT_TLS_OUTBOUND_INITIALIZED: &str = "tls_outbound_initialized"; const EVENT_TLS_OUTBOUND_INITIALIZATION_FAILED: &str = "tls_outbound_initialization_failed"; const TLS_STARTUP_GENERATION_CONSUMER: &str = "rustfs_server_startup"; +// Startup completes before listeners register; retain source names independently +// of the later listener snapshot, without pinning renewable certificate bytes. +static STARTUP_MATERIAL_FILES: Mutex)>> = Mutex::new(None); + +pub(crate) fn required_startup_material_files(tls_path: &str) -> BTreeSet { + STARTUP_MATERIAL_FILES + .lock() + .unwrap_or_else(|error| error.into_inner()) + .as_ref() + .filter(|(path, _)| path == tls_path.trim()) + .map(|(_, files)| files.clone()) + .unwrap_or_default() +} + +#[cfg(test)] +pub(crate) fn replace_startup_material_files(files: Option<(String, BTreeSet)>) -> Option<(String, BTreeSet)> { + std::mem::replace(&mut *STARTUP_MATERIAL_FILES.lock().unwrap_or_else(|error| error.into_inner()), files) +} + pub(crate) async fn init_outbound_tls_material(config: &Config) -> Result<()> { crate::server::tls_material::validate_configured_oidc_extra_ca_cert() .await @@ -32,6 +53,8 @@ pub(crate) async fn init_outbound_tls_material(config: &Config) -> Result<()> { match crate::server::tls_material::load_tls_material(tls_path).await { Ok(snapshot) => { publish_outbound_tls_material(&snapshot.outbound, Some(tls_path)).await; + *STARTUP_MATERIAL_FILES.lock().unwrap_or_else(|error| error.into_inner()) = + Some((tls_path.to_string(), snapshot.material_files)); } Err(err) => { error!( @@ -49,6 +72,11 @@ pub(crate) async fn init_outbound_tls_material(config: &Config) -> Result<()> { if startup_runtime_sources::observability_metric_enabled() { startup_runtime_sources::init_tls_metrics(); } + } else { + STARTUP_MATERIAL_FILES + .lock() + .unwrap_or_else(|error| error.into_inner()) + .take(); } Ok(()) diff --git a/rustfs/src/storage/access.rs b/rustfs/src/storage/access.rs index 122ba30fc..c3b971693 100644 --- a/rustfs/src/storage/access.rs +++ b/rustfs/src/storage/access.rs @@ -16,13 +16,13 @@ use super::ObjectOptions; use super::ecfs::FS; use super::{ECStore, PolicySys, ReplicationStatusType, StorageError, get_lock_acquire_timeout, is_err_bucket_not_found}; use crate::auth::{ - AuthType, OBJECT_LOCK_REMAINING_RETENTION_DAYS_CONDITION, RUSTFS_MAX_CONTENT_LENGTH_QUERY, - RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY, VerifiedPresignedRequest, VerifiedSigV4Request, check_key_valid_with_context, - get_condition_values_with_client_info, get_condition_values_with_query_and_client_info, get_request_auth_type_with_query, - get_session_token, parse_presigned_multipart_max_total_object_size, parse_presigned_put_max_content_length, - reject_unsigned_amz_headers_on_sigv4_request, + AuthType, OBJECT_CREATION_OPERATION_CONDITION, OBJECT_LOCK_REMAINING_RETENTION_DAYS_CONDITION, + RUSTFS_MAX_CONTENT_LENGTH_QUERY, RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY, VerifiedPresignedRequest, VerifiedSigV4Request, + check_key_valid_with_context, get_condition_values_with_client_info, get_condition_values_with_query_and_client_info, + get_request_auth_type_with_query, get_session_token, parse_presigned_multipart_max_total_object_size, + parse_presigned_put_max_content_length, reject_unsigned_amz_headers_on_sigv4_request, }; -use crate::error::ApiError; +use crate::error::{ApiError, normalize_put_read_quorum_error}; use crate::license::license_check; use crate::shared_types::RemoteAddr; use crate::storage::request_context::RequestContext; @@ -150,6 +150,17 @@ struct RequestedObjectLockRetainUntil(OffsetDateTime); #[derive(Clone, Debug)] struct RequestedObjectLockRetention(ObjectLockRetention); +/// These operations stage upload state without publishing an object. Keep this +/// separate from the marker controlling multipart publication fences. +#[derive(Clone, Copy, Debug)] +pub(crate) struct MultipartStagingOperation; + +impl MultipartStagingOperation { + fn for_operation(operation: &str) -> Option { + matches!(operation, "CreateMultipartUpload" | "UploadPart" | "UploadPartCopy").then_some(Self) + } +} + #[derive(Clone, Debug)] struct StagedMultipartPartAuthorization; @@ -431,6 +442,11 @@ type ObjectGenerationTestHook = (String, tokio::sync::oneshot::Sender<()>, tokio #[cfg(test)] static OBJECT_GENERATION_TEST_HOOK: OnceLock>> = OnceLock::new(); +#[cfg(test)] +tokio::task_local! { + static PUT_GENERATION_READ_QUORUM_FAILURE: (); +} + #[cfg(test)] fn install_object_generation_test_hook( bucket: String, @@ -889,8 +905,15 @@ fn authorization_conditions( req.uri.query(), client_info, ); - if req.extensions.get::().is_some() { + let internal_object_authorization = req.extensions.get::().is_some(); + if internal_object_authorization { retain_internal_object_authorization_conditions(&mut conditions, cred); + } + if action == Action::S3Action(S3Action::PutObjectAction) { + let creates_object = internal_object_authorization || req.extensions.get::().is_none(); + conditions.insert(OBJECT_CREATION_OPERATION_CONDITION.to_string(), vec![creates_object.to_string()]); + } + if internal_object_authorization { return Ok(conditions); } merge_list_bucket_query_conditions(action, req.uri.query(), &mut conditions); @@ -1751,11 +1774,7 @@ fn table_catalog_store_for_data_plane( } fn table_publication_guard_error(err: crate::table_catalog::TableCatalogStoreError) -> S3Error { - let code = match &err { - crate::table_catalog::TableCatalogStoreError::Unavailable(_) => S3ErrorCode::ServiceUnavailable, - _ => S3ErrorCode::InternalError, - }; - S3Error::with_message(code, format!("failed to acquire table publication guard: {err}")) + S3Error::from(ApiError::from(err)) } async fn retain_table_data_plane_publication_guard( @@ -1891,8 +1910,8 @@ fn table_list_cursor_key(req: &S3Request) -> S3Result<[u8; 32]> { } fn table_list_catalog_error(err: crate::table_catalog::TableCatalogStoreError) -> S3Error { - if matches!(err, crate::table_catalog::TableCatalogStoreError::Unavailable(_)) { - S3Error::from(ApiError::service_unavailable()) + if err.is_unavailable() { + S3Error::from(ApiError::from(err)) } else { S3Error::from(ApiError::access_denied()) } @@ -2034,11 +2053,7 @@ async fn table_data_plane_resource_for_request( error = %err, "failed to resolve table data-plane resource" ); - if matches!(err, crate::table_catalog::TableCatalogStoreError::Unavailable(_)) { - S3Error::from(ApiError::service_unavailable()) - } else { - s3_error!(AccessDenied, "Access Denied") - } + table_list_catalog_error(err) })?; let resource = require_owned_reserved_table_object(object, resource)?; let bucket_fence_key = (bucket.to_string(), crate::table_catalog::default_table_bucket_publication_lock_path()); @@ -2226,12 +2241,16 @@ impl S3Access for FS { let auth_type = get_request_auth_type_with_query(cx.headers(), cx.uri().query()); let verified_presigned = matches!(auth_type, AuthType::Presigned); let verified_sigv4 = matches!(auth_type, AuthType::Presigned | AuthType::Signed); + let multipart_staging_operation = MultipartStagingOperation::for_operation(cx.s3_op().name()); // Publish this server's context slot so downstream data-plane handlers // resolve the same store (backlog#1052 S6). { let ext = cx.extensions_mut(); ext.insert(self.server_ctx().clone()); ext.insert(req_info); + if let Some(operation) = multipart_staging_operation { + ext.insert(operation); + } if verified_presigned { ext.insert(VerifiedPresignedRequest); } @@ -3303,40 +3322,50 @@ impl S3Access for FS { /// This method returns `Ok(())` by default. async fn put_object(&self, req: &mut S3Request) -> S3Result<()> { crate::hp_guard!("S3Access::put_object"); - let req_info = ext_req_info_mut(&mut req.extensions)?; - req_info.bucket = Some(req.input.bucket.clone()); - req_info.object = Some(req.input.key.clone()); - req_info.version_id = req.input.version_id.clone(); + async { + let req_info = ext_req_info_mut(&mut req.extensions)?; + req_info.bucket = Some(req.input.bucket.clone()); + req_info.object = Some(req.input.key.clone()); + req_info.version_id = req.input.version_id.clone(); - if has_write_offset_bytes_header(&req.headers) { - return Err(S3Error::with_message( - S3ErrorCode::NotImplemented, - ApiError::error_code_to_message(&S3ErrorCode::NotImplemented), - )); + if has_write_offset_bytes_header(&req.headers) { + return Err(S3Error::with_message( + S3ErrorCode::NotImplemented, + ApiError::error_code_to_message(&S3ErrorCode::NotImplemented), + )); + } + + let bucket = req.input.bucket.clone(); + // Snapshot before authorization, but preserve AccessDenied precedence + // by exposing any bucket-state error only after authorization succeeds. + let bucket_generation = load_bucket_generation(self, req, &bucket).await; + #[cfg(test)] + let bucket_generation = if PUT_GENERATION_READ_QUORUM_FAILURE.try_with(|_| ()).is_ok() { + Err(ApiError::from(StorageError::ErasureReadQuorum).into()) + } else { + bucket_generation + }; + let requested_retain_until = req.input.object_lock_retain_until_date.clone(); + set_requested_object_lock_retain_until(req, requested_retain_until.as_ref()); + authorize_request(req, Action::S3Action(S3Action::PutObjectAction)).await?; + req.extensions.insert(bucket_generation?); + + if req.method != http::Method::POST { + authorize_replication_only_put_headers(req).await?; + } + + if legal_hold_write_requested(req.input.object_lock_legal_hold_status.as_ref()) { + authorize_request(req, Action::S3Action(S3Action::PutObjectLegalHoldAction)).await?; + } + + if retention_write_requested(req.input.object_lock_mode.as_ref(), req.input.object_lock_retain_until_date.as_ref()) { + authorize_request(req, Action::S3Action(S3Action::PutObjectRetentionAction)).await?; + } + + Ok(()) } - - let bucket = req.input.bucket.clone(); - // Snapshot before authorization, but preserve AccessDenied precedence - // by exposing any bucket-state error only after authorization succeeds. - let bucket_generation = load_bucket_generation(self, req, &bucket).await; - let requested_retain_until = req.input.object_lock_retain_until_date.clone(); - set_requested_object_lock_retain_until(req, requested_retain_until.as_ref()); - authorize_request(req, Action::S3Action(S3Action::PutObjectAction)).await?; - req.extensions.insert(bucket_generation?); - - if req.method != http::Method::POST { - authorize_replication_only_put_headers(req).await?; - } - - if legal_hold_write_requested(req.input.object_lock_legal_hold_status.as_ref()) { - authorize_request(req, Action::S3Action(S3Action::PutObjectLegalHoldAction)).await?; - } - - if retention_write_requested(req.input.object_lock_mode.as_ref(), req.input.object_lock_retain_until_date.as_ref()) { - authorize_request(req, Action::S3Action(S3Action::PutObjectRetentionAction)).await?; - } - - Ok(()) + .await + .map_err(normalize_put_read_quorum_error) } /// Checks whether the PutObjectAcl request has accesses to the resources. @@ -3827,7 +3856,7 @@ mod tests { "", ] { let error = TableCatalogStoreError::Unavailable(message.to_string()); - let expected_message = format!("failed to acquire table publication guard: {error}"); + let expected_message = ApiError::error_code_to_message(&S3ErrorCode::ServiceUnavailable); let err = table_publication_guard_error(error); assert_eq!(err.code(), &S3ErrorCode::ServiceUnavailable, "{message}"); assert_eq!( @@ -3835,6 +3864,8 @@ mod tests { Some(http::StatusCode::SERVICE_UNAVAILABLE) ); assert_eq!(err.message(), Some(expected_message.as_str())); + let response = err.to_http_response().expect("serialize catalog failure"); + assert!(!response.headers().contains_key(http::header::RETRY_AFTER)); } } @@ -3853,7 +3884,7 @@ mod tests { TableCatalogStoreError::AlreadyExists(message.clone()), TableCatalogStoreError::Conflict(message), ] { - let expected_message = format!("failed to acquire table publication guard: {error}"); + let expected_message = ApiError::error_code_to_message(&S3ErrorCode::InternalError); let err = table_publication_guard_error(error); assert_eq!(err.code(), &S3ErrorCode::InternalError); assert_eq!( @@ -3861,13 +3892,16 @@ mod tests { Some(http::StatusCode::INTERNAL_SERVER_ERROR) ); assert_eq!(err.message(), Some(expected_message.as_str())); + let response = err.to_http_response().expect("serialize catalog failure"); + assert!(!response.headers().contains_key(http::header::RETRY_AFTER)); } } #[tokio::test] #[serial] - async fn table_publication_guard_timeout_blocks_put_and_multipart_authorization() { + async fn table_publication_guard_timeout_blocks_put_delete_and_multipart_authorization() { use crate::storage::storage_api::contract::namespace::NamespaceLocking as _; + use http_body_util::BodyExt as _; use std::time::Duration; let store = crate::app::gating_test_env::shared_gating_ecstore().await; @@ -3880,7 +3914,7 @@ mod tests { .await .expect("test bucket should be created"); let policy_json = format!( - r#"{{"Version":"2012-10-17","Statement":[{{"Effect":"Allow","Principal":{{"AWS":"*"}},"Action":["s3:PutObject"],"Resource":["arn:aws:s3:::{bucket}/*"]}}]}}"# + r#"{{"Version":"2012-10-17","Statement":[{{"Effect":"Allow","Principal":{{"AWS":"*"}},"Action":["s3:PutObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::{bucket}/*"]}}]}}"# ); let mut metadata = (*crate::storage::get_bucket_metadata(&bucket) .await @@ -3916,6 +3950,16 @@ mod tests { .expect("multipart input should build"), Method::POST, ); + let mut delete_req = build_request( + DeleteObjectInput::builder() + .bucket(bucket.clone()) + .key("object".to_string()) + .build() + .expect("DELETE input should build"), + Method::DELETE, + ); + ensure_req_info(&mut delete_req); + delete_req.extensions.insert(fs.server_ctx().clone()); ensure_req_info(&mut put_req); ensure_req_info(&mut multipart_req); put_req.extensions.insert(fs.server_ctx().clone()); @@ -3928,6 +3972,12 @@ mod tests { .await .expect_err("PUT must wait for the publication writer"), ), + ( + "DeleteObject", + fs.delete_object(&mut delete_req) + .await + .expect_err("DELETE must wait for the publication writer"), + ), ( "CreateMultipartUpload", fs.create_multipart_upload(&mut multipart_req) @@ -3936,8 +3986,24 @@ mod tests { ), ] { assert_eq!(err.code(), &S3ErrorCode::ServiceUnavailable, "{operation}: {err}"); + let source = err.source().expect("retain the typed publication lock failure"); + let storage = source + .downcast_ref::() + .expect("use the canonical storage error boundary"); + assert!(matches!( + storage, + crate::storage_api::error::StorageError::Lock(rustfs_lock::LockError::Timeout { .. }) + )); + let response = err.to_http_response().expect("serialize publication failure"); + assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE); + assert_eq!(response.headers()[http::header::RETRY_AFTER], "1"); + let body = response.into_body().collect().await.expect("read error XML").to_bytes(); + let body = String::from_utf8(body.to_vec()).expect("S3 XML is UTF-8"); + assert!(body.contains("ServiceUnavailable")); + assert!(!body.contains(&bucket)); + assert!(!body.contains(&lock_object)); } - for extensions in [&put_req.extensions, &multipart_req.extensions] { + for extensions in [&put_req.extensions, &delete_req.extensions, &multipart_req.extensions] { assert!(extensions.get::().is_none()); } @@ -3949,7 +4015,10 @@ mod tests { fs.create_multipart_upload(&mut multipart_req) .await .expect("multipart initialization may retry after publication finishes"); - for extensions in [&put_req.extensions, &multipart_req.extensions] { + fs.delete_object(&mut delete_req) + .await + .expect("DELETE may retry after publication finishes"); + for extensions in [&put_req.extensions, &delete_req.extensions, &multipart_req.extensions] { let retained = extensions .get::() .expect("successful admission retains the guard"); @@ -3960,6 +4029,7 @@ mod tests { } drop(put_req); drop(multipart_req); + drop(delete_req); lock.get_write_lock(Duration::from_secs(1)) .await .expect("dropping the request must release its publication guard"); @@ -4358,15 +4428,8 @@ mod tests { (S3Action::DeleteObjectTaggingAction, S3Action::DeleteObjectVersionTaggingAction), ] { for version in [None, Some("null"), Some("0194e0f1-0000-7000-8000-000000000000")] { - let action = versioned_tagging_action(current.clone(), version); - assert_eq!( - action, - Action::S3Action(if version.is_some() { - versioned.clone() - } else { - current.clone() - }) - ); + let action = versioned_tagging_action(current, version); + assert_eq!(action, Action::S3Action(if version.is_some() { versioned } else { current })); let args = Args { account: "writer", groups: &groups, @@ -4383,6 +4446,170 @@ mod tests { } } + #[test] + fn object_creation_operation_classifies_only_multipart_staging() { + for (operation, staging) in [ + ("CreateMultipartUpload", true), + ("UploadPart", true), + ("UploadPartCopy", true), + ("PutObject", false), + ("CopyObject", false), + ("CompleteMultipartUpload", false), + ("PostObject", false), + ("GetObject", false), + ("InternalObjectWrite", false), + ] { + assert_eq!( + super::MultipartStagingOperation::for_operation(operation).is_some(), + staging, + "{operation}" + ); + } + } + + #[test] + fn object_creation_operation_ignores_request_and_claim_spoofing() { + use crate::auth::{OBJECT_CREATION_OPERATION_CONDITION, get_condition_values}; + for forged in ["false", "true"] { + for claim in [ + "ObjectCreationOperation", + "objectcreationoperation", + "OBJECTCREATIONOPERATION", + "ldapObjectCreationOperation", + "ldapldapObjectCreationOperation", + ] { + let credentials = rustfs_credentials::Credentials { + claims: Some(HashMap::from([(claim.to_string(), serde_json::json!(forged))])), + ..Default::default() + }; + let mut req = build_request((), Method::GET); + req.uri = format!("/bucket/key?ObjectCreationOperation={forged}&uploadId=fake&partNumber=1") + .parse() + .expect("spoofed query should parse"); + req.headers + .insert("objectcreationoperation", HeaderValue::from_str(forged).expect("header should parse")); + assert!( + !get_condition_values(&req.headers, &credentials, None, None, None) + .keys() + .any(|key| key.eq_ignore_ascii_case(OBJECT_CREATION_OPERATION_CONDITION)) + ); + for staging in [false, true] { + if staging { + req.extensions.insert(super::MultipartStagingOperation); + } + for internal in [false, true] { + if internal { + req.extensions.insert(InternalObjectAuthorization); + } else { + req.extensions.remove::(); + } + for action in [ + S3Action::PutObjectAction, + S3Action::GetObjectAction, + S3Action::GetObjectVersionAction, + ] { + let conditions = + authorization_conditions(&req, &credentials, None, None, None, None, Action::S3Action(action)) + .expect("conditions should build"); + let values = conditions + .iter() + .filter(|(key, _)| key.eq_ignore_ascii_case(OBJECT_CREATION_OPERATION_CONDITION)) + .collect::>(); + if action == S3Action::PutObjectAction { + assert_eq!(values.len(), 1, "one canonical server-derived value"); + assert_eq!(values[0].0, OBJECT_CREATION_OPERATION_CONDITION); + assert_eq!( + values[0].1, + &vec![(internal || !staging).to_string()], + "{claim}, internal={internal}, staging={staging}" + ); + } else { + assert!(values.is_empty(), "non-PutObject action {action:?} must have no creation condition"); + } + } + } + } + } + } + } + + #[tokio::test] + async fn object_creation_operation_controls_identity_and_bucket_policies() { + use rustfs_policy::policy::{Args, Policy}; + for effect in ["Allow", "Deny"] { + for expected in [false, true] { + let mut statements = vec![serde_json::json!({ + "Effect": effect, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::bucket/*", + "Condition": { "Bool": { "s3:ObjectCreationOperation": expected.to_string() } } + })]; + if effect == "Deny" { + statements.push(serde_json::json!({ + "Effect": "Allow", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::bucket/*" + })); + } + let identity: Policy = serde_json::from_str( + &serde_json::json!({ + "Version": "2012-10-17", "Statement": statements + }) + .to_string(), + ) + .expect("identity policy should accept the AWS Bool key"); + for statement in &mut statements { + statement["Principal"] = serde_json::json!("*"); + } + let bucket: BucketPolicy = serde_json::from_str( + &serde_json::json!({ + "Version": "2012-10-17", "Statement": statements + }) + .to_string(), + ) + .expect("bucket policy should accept the AWS Bool key"); + for actual in [None, Some(false), Some(true)] { + let conditions = actual + .map(|value| HashMap::from([("ObjectCreationOperation".to_string(), vec![value.to_string()])])) + .unwrap_or_default(); + let allowed = if effect == "Allow" { + actual == Some(expected) + } else { + actual != Some(expected) + }; + assert_eq!( + identity + .is_allowed(&Args { + account: "writer", + groups: &None, + action: Action::S3Action(S3Action::PutObjectAction), + bucket: "bucket", + object: "object", + conditions: &conditions, + is_owner: false, + claims: &HashMap::new(), + deny_only: false, + }) + .await, + allowed, + "identity {effect}, expected={expected}, actual={actual:?}" + ); + assert_eq!( + bucket + .is_allowed(&BucketPolicyArgs { + account: "writer", + groups: &None, + action: Action::S3Action(S3Action::PutObjectAction), + bucket: "bucket", + object: "object", + conditions: &conditions, + is_owner: false, + }) + .await, + allowed, + "bucket {effect}, expected={expected}, actual={actual:?}" + ); + } + } + } + } + fn remaining_retention_days_conditions(req: &S3Request, action: Action) -> HashMap> { let credentials = rustfs_credentials::Credentials::default(); authorization_conditions(req, &credentials, None, None, None, None, action).expect("conditions should build") @@ -5205,6 +5432,60 @@ mod tests { }); } + #[tokio::test] + #[serial] + async fn put_read_quorum_access_keeps_authorization_precedence() { + let store = crate::app::gating_test_env::shared_gating_ecstore().await; + let server_ctx = ServerContextSlot::new(); + assert!(server_ctx.install(Arc::new(AppContext::new(Arc::clone(&store), Arc::new(UnreadyIam), Arc::new(TestKms))))); + let fs = FS::with_server_ctx(server_ctx); + + for effect in ["Allow", "Deny"] { + let bucket = format!("put-quorum-access-{}", uuid::Uuid::new_v4()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("authorization precedence bucket must be created"); + let policy_json = format!( + r#"{{"Version":"2012-10-17","Statement":[{{"Effect":"{effect}","Principal":{{"AWS":"*"}},"Action":["s3:PutObject"],"Resource":["arn:aws:s3:::{bucket}/*"]}}]}}"# + ); + let mut metadata = (*crate::storage::get_bucket_metadata(&bucket) + .await + .expect("test bucket metadata must be cached")) + .clone(); + metadata.policy_config = Some(serde_json::from_str(&policy_json).expect("test policy must parse")); + metadata.policy_config_json = policy_json.into_bytes(); + crate::storage::storage_api::set_bucket_metadata(bucket.clone(), metadata) + .await + .expect("test policy must be installed"); + + let input = PutObjectInput::builder() + .bucket(bucket) + .key("object".into()) + .build() + .expect("access probe input must build"); + let mut req = build_request(input, Method::PUT); + ensure_req_info(&mut req); + req.extensions.insert(fs.server_ctx().clone()); + let err = super::PUT_GENERATION_READ_QUORUM_FAILURE + .scope((), fs.put_object(&mut req)) + .await + .expect_err("a generation failure must never admit a write"); + if effect == "Allow" { + assert_eq!(err.code(), &S3ErrorCode::ServiceUnavailable); + assert_eq!(err.status_code(), Some(http::StatusCode::SERVICE_UNAVAILABLE)); + assert!(matches!( + err.source().and_then(|source| source.downcast_ref::()), + Some(StorageError::ErasureReadQuorum) + )); + } else { + assert_eq!(err.code(), &S3ErrorCode::AccessDenied); + assert_eq!(err.status_code(), Some(http::StatusCode::FORBIDDEN)); + } + assert!(req.extensions.get::().is_none()); + } + } + #[tokio::test] #[serial] async fn put_object_access_captures_authorized_bucket_incarnation() { diff --git a/rustfs/src/storage/ecfs.rs b/rustfs/src/storage/ecfs.rs index f1c045f84..3ac66cacf 100644 --- a/rustfs/src/storage/ecfs.rs +++ b/rustfs/src/storage/ecfs.rs @@ -415,7 +415,12 @@ pub(crate) fn object_lock_retention_from_metadata(metadata: &HashMap S3Result<()> { - let mod_time = info.mod_time; - let etag = info.etag.as_deref(); + match evaluate_read_preconditions(headers, info.etag.as_deref(), info.mod_time) { + Ok(()) => Ok(()), + Err(S3ErrorCode::NotModified) => Err(not_modified_error(info)?), + Err(code) => Err(S3Error::new(code)), + } +} + +pub(crate) fn evaluate_read_preconditions( + headers: &HeaderMap, + etag: Option<&str>, + mod_time: Option, +) -> Result<(), S3ErrorCode> { let if_match = non_empty_header_value(headers, IF_MATCH); let if_none_match = non_empty_header_value(headers, IF_NONE_MATCH); let if_modified_since = non_empty_header_value(headers, IF_MODIFIED_SINCE); let if_unmodified_since = non_empty_header_value(headers, IF_UNMODIFIED_SINCE); - if mod_time.is_none() && etag.is_none() { - return Ok(()); - } - - // If-Match: requires ETag to exist + // The caller has resolved an existing object, so '*' does not require an ETag. if let Some(if_match_val) = if_match { match etag { - Some(e) if is_etag_equal(e, if_match_val) => {} - _ => return Err(S3Error::new(S3ErrorCode::PreconditionFailed)), + _ if if_match_val == "*" => {} + Some(e) if read_etag_matches(e, if_match_val, false) => {} + _ => return Err(S3ErrorCode::PreconditionFailed), } } @@ -553,32 +559,16 @@ pub(crate) fn check_preconditions(headers: &HeaderMap, info: &ObjectInfo) -> S3R && let Some(t) = mod_time && let Some(if_unmodified_since) = if_unmodified_since && let Ok(given_time) = time::PrimitiveDateTime::parse(if_unmodified_since, &RFC1123).map(|dt| dt.assume_utc()) - && t > given_time.add(time::Duration::seconds(1)) + && t.unix_timestamp() > given_time.unix_timestamp() { - return Err(S3Error::new(S3ErrorCode::PreconditionFailed)); + return Err(S3ErrorCode::PreconditionFailed); } // If-None-Match if let Some(if_none_match) = if_none_match - && let Some(e) = etag - && is_etag_equal(e, if_none_match) + && (if_none_match == "*" || etag.is_some_and(|e| read_etag_matches(e, if_none_match, true))) { - let mut error_headers = HeaderMap::new(); - if let Ok(etag_header) = parse_etag(e) { - error_headers.insert("etag", etag_header); - } - if let Some(t) = mod_time - && let Ok(last_modified_str) = t.format(&RFC1123) - && let Ok(last_modified_header) = HeaderValue::from_str(&last_modified_str) - { - error_headers.insert("last-modified", last_modified_header); - } - - let mut s3_error = S3Error::new(S3ErrorCode::NotModified); - s3_error.set_message("Not Modified".to_string()); - s3_error.set_status_code(StatusCode::NOT_MODIFIED); - s3_error.set_headers(error_headers); - return Err(s3_error); + return Err(S3ErrorCode::NotModified); } // If-Modified-Since (only when If-None-Match is absent — semantics per RFC 7232; dates use RFC 1123 format) @@ -586,31 +576,94 @@ pub(crate) fn check_preconditions(headers: &HeaderMap, info: &ObjectInfo) -> S3R && let Some(t) = mod_time && let Some(if_modified_since) = if_modified_since && let Ok(given_time) = time::PrimitiveDateTime::parse(if_modified_since, &RFC1123).map(|dt| dt.assume_utc()) - && t < given_time.add(time::Duration::seconds(1)) + && t.unix_timestamp() <= given_time.unix_timestamp() { - let mut error_headers = HeaderMap::new(); - if let Some(e) = etag - && let Ok(etag_header) = parse_etag(e) - { - error_headers.insert("etag", etag_header); - } - if let Ok(last_modified_str) = t.format(&RFC1123) - && let Ok(last_modified_header) = HeaderValue::from_str(&last_modified_str) - { - error_headers.insert("last-modified", last_modified_header); - } - - let mut s3_error = S3Error::new(S3ErrorCode::NotModified); - s3_error.set_message("Not Modified".to_string()); - s3_error.set_status_code(StatusCode::NOT_MODIFIED); - s3_error.set_headers(error_headers); - - return Err(s3_error); + return Err(S3ErrorCode::NotModified); } Ok(()) } +fn not_modified_error(info: &ObjectInfo) -> S3Result { + let mut headers = HeaderMap::new(); + if let Some(etag) = info.etag.as_deref() { + headers.insert(http::header::ETAG, parse_etag(etag).map_err(S3Error::internal_error)?); + } + if let Some(mod_time) = info.mod_time { + let value = mod_time.format(&RFC1123).map_err(S3Error::internal_error)?; + headers.insert( + http::header::LAST_MODIFIED, + HeaderValue::from_str(&value).map_err(S3Error::internal_error)?, + ); + } + for name in ["cache-control", "content-location", "vary", "expires"] { + if let Some(value) = info.user_defined.get(name) { + headers.insert(name, HeaderValue::from_str(value).map_err(S3Error::internal_error)?); + } + } + if let Some(expires) = info.expires { + let value = expires.format(&RFC1123).map_err(S3Error::internal_error)?; + headers.insert(http::header::EXPIRES, HeaderValue::from_str(&value).map_err(S3Error::internal_error)?); + } + if let Some(version) = info.version_id { + let value = if version.is_nil() { + "null".to_owned() + } else { + version.to_string() + }; + headers.insert("x-amz-version-id", HeaderValue::from_str(&value).map_err(S3Error::internal_error)?); + } + let mut error = S3Error::new(S3ErrorCode::NotModified); + error.set_status_code(StatusCode::NOT_MODIFIED); + error.set_message("Not Modified".to_owned()); + error.set_headers(headers); + Ok(error) +} + +/// GET/HEAD use weak comparison for If-None-Match and strong comparison for If-Match. +fn read_etag_matches(object_etag: &str, condition: &str, allow_weak: bool) -> bool { + let object_etag = object_etag.trim(); + let object_weak = object_etag.starts_with("W/"); + let object_value = object_etag.strip_prefix("W/").unwrap_or(object_etag).trim_matches('"'); + let mut remaining = condition.trim(); + let mut matched = false; + while !remaining.is_empty() { + // Recipients ignore empty list members, but tags still need commas. + if let Some(rest) = remaining.strip_prefix(',') { + remaining = rest.trim(); + continue; + } + let weak = remaining.starts_with("W/"); + let value = remaining.strip_prefix("W/").unwrap_or(remaining); + let (value, rest) = if let Some(quoted) = value.strip_prefix('"') { + let Some(pair) = quoted.split_once('"') else { return false }; + pair + } else { + // Preserve bare tags used by legacy internal callers. + if weak { + return false; + } + let (value, rest) = value.split_at(value.find(',').unwrap_or(value.len())); + let value = value.trim(); + if value.is_empty() || value == "*" { + return false; + } + (value, rest) + }; + if !value.bytes().all(|byte| byte == b'!' || (byte >= b'#' && byte != 0x7f)) { + return false; + } + matched |= value == object_value && (allow_weak || (!weak && !object_weak)); + let rest = rest.trim(); + if rest.is_empty() { + return matched; + } + let Some(rest) = rest.strip_prefix(',') else { return false }; + remaining = rest.trim(); + } + matched +} + fn non_empty_header_value(headers: &HeaderMap, name: http::header::HeaderName) -> Option<&str> { headers .get(name) @@ -619,33 +672,10 @@ fn non_empty_header_value(headers: &HeaderMap, name: http::header::HeaderName) - .filter(|v| !v.is_empty()) } -/// Compares an object ETag with an ETag value from an HTTP header. -/// -/// This helper implements HTTP ETag comparison semantics for headers such as -/// `If-Match` and `If-None-Match`: -/// - Supports the wildcard `*`, which matches any `object_etag`. -/// - Supports comma-separated ETag lists (e.g., `"etag1", "etag2"`), returning -/// `true` if any entry matches `object_etag`. -/// - Automatically trims surrounding whitespace and double quotes from both the -/// header entries and `object_etag` before comparison. -/// -/// # Parameters -/// - `object_etag`: The ETag associated with the stored object. -/// - `header_etag`: The raw ETag header value received in the request, which may -/// be a wildcard, a single ETag, or a comma-separated list of ETags. -/// -/// # Returns -/// `true` if the header value matches the object ETag according to the above -/// HTTP ETag comparison rules, otherwise `false`. -pub(crate) fn is_etag_equal(object_etag: &str, header_etag: &str) -> bool { - let header_etag = header_etag.trim(); - if header_etag == "*" { - return true; - } - header_etag - .split(',') - .map(|s| s.trim().trim_matches('"')) - .any(|e| e == object_etag.trim_matches('"')) +pub(crate) fn has_read_preconditions(headers: &HeaderMap) -> bool { + [IF_MATCH, IF_UNMODIFIED_SINCE, IF_NONE_MATCH, IF_MODIFIED_SINCE] + .into_iter() + .any(|name| non_empty_header_value(headers, name).is_some()) } /// Converts an object ETag string into an HTTP `HeaderValue` for use in response headers. diff --git a/rustfs/src/storage/ecfs_test.rs b/rustfs/src/storage/ecfs_test.rs index 18f96abfa..149ccf387 100644 --- a/rustfs/src/storage/ecfs_test.rs +++ b/rustfs/src/storage/ecfs_test.rs @@ -27,11 +27,11 @@ mod tests { use crate::storage::storage_api::test_consumer::{ BucketMetadata, DEFAULT_READ_BUFFER_SIZE, StorageObjectInfo as ObjectInfo, apply_cors_headers, apply_default_lock_retention_metadata, bucket_metadata_sys_initialized, check_preconditions, decode_tags_to_map, - get_adaptive_buffer_size_with_profile, get_buffer_size_opt_in, get_global_bucket_metadata_sys, is_etag_equal, - matches_origin_pattern, parse_etag, parse_object_lock_legal_hold, parse_object_lock_retention, - process_lambda_configurations, process_queue_configurations, process_topic_configurations, - remove_object_lock_metadata_for_copy, remove_object_lock_retention_metadata, set_bucket_metadata, - validate_bucket_object_lock_enabled, validate_list_object_unordered_with_delimiter, + get_adaptive_buffer_size_with_profile, get_buffer_size_opt_in, get_global_bucket_metadata_sys, matches_origin_pattern, + parse_etag, parse_object_lock_legal_hold, parse_object_lock_retention, process_lambda_configurations, + process_queue_configurations, process_topic_configurations, remove_object_lock_metadata_for_copy, + remove_object_lock_retention_metadata, set_bucket_metadata, validate_bucket_object_lock_enabled, + validate_list_object_unordered_with_delimiter, }; use http::{Extensions, HeaderMap, HeaderValue, Method, StatusCode, Uri}; use rustfs_config::MI_B; @@ -1191,30 +1191,72 @@ mod tests { } #[test] - fn test_is_etag_equal() { - // [1] Header ETag is "*", should return true (match any object ETag) - assert!(is_etag_equal("\"d41d8cd98f00b204e9800998ecf8427e\"", "*")); + fn test_read_etag_comparison() { + for (object_etag, condition, matches) in [ + ("\"d41d8cd98f00b204e9800998ecf8427e\"", "*", true), + ("\"d41d8cd98f00b204e9800998ecf8427e\"", "\"d41d8cd98f00b204e9800998ecf8427e\"", true), + ("\"d41d8cd98f00b204e9800998ecf8427e\"", "d41d8cd98f00b204e9800998ecf8427e", true), + ("\"12345\"", "\"67890\", \"12345\", \"abcde\"", true), + ("\"12345\"", " \"67890\" , \"12345\" , \"abcde\" ", true), + ("\"12345\"", "\"67890\"", false), + ("\"12345\"", "\"67890\", \"abcde\"", false), + ] { + let info = ObjectInfo { + etag: Some(object_etag.to_owned()), + ..Default::default() + }; + let mut headers = HeaderMap::new(); + headers.insert(http::header::IF_NONE_MATCH, HeaderValue::from_str(condition).unwrap()); + let result = check_preconditions(&headers, &info); + if matches { + assert_eq!(result.unwrap_err().code(), &S3ErrorCode::NotModified); + } else { + assert!(result.is_ok()); + } + } + } - // [2] Exact match (both with double quotes) - assert!(is_etag_equal( - "\"d41d8cd98f00b204e9800998ecf8427e\"", - "\"d41d8cd98f00b204e9800998ecf8427e\"" - )); - - // [3] Exact match (object ETag with quotes, header ETag without) - assert!(is_etag_equal("\"d41d8cd98f00b204e9800998ecf8427e\"", "d41d8cd98f00b204e9800998ecf8427e")); - - // [4] Header ETag has multiple values (comma-separated), one matches - assert!(is_etag_equal("\"12345\"", "\"67890\", \"12345\", \"abcde\"")); - - // [5] Header ETag has multiple values with spaces, one matches after trim - assert!(is_etag_equal("\"12345\"", " \"67890\" , \"12345\" , \"abcde\" ")); - - // [6] No match (different ETag) - assert!(!is_etag_equal("\"12345\"", "\"67890\"")); - - // [7] No match in multiple values - assert!(!is_etag_equal("\"12345\"", "\"67890\", \"abcde\"")); + #[test] + fn test_read_etag_comparison_requires_complete_field() { + for (etag, condition, matches) in [ + ("abc", "\"other\"\"abc\"", false), + ("abc", "\"abc\"garbage\"", false), + ("abc", "\"abc\", \"other\"\"different\"", false), + ("abc", "\"abc\", W/ \"other\"", false), + ("abc", "\"abc\", \"contains space\"", false), + ("abc", "\"abc\", *", false), + ("abc", "\"other\", , \"abc\"", true), + ("abc", ", \"abc\",,", true), + ("a,b", "\"other\", \"a,b\"", true), + ("", ",,,", false), + ("", "\"\"", true), + ] { + let info = ObjectInfo { + etag: Some(etag.to_owned()), + ..Default::default() + }; + for name in [http::header::IF_MATCH, http::header::IF_NONE_MATCH] { + let mut headers = HeaderMap::new(); + headers.insert(name.clone(), HeaderValue::from_str(condition).unwrap()); + let result = check_preconditions(&headers, &info); + if name == http::header::IF_MATCH { + if matches { + assert!(result.is_ok(), "{condition}"); + } else { + assert_eq!(result.unwrap_err().code(), &S3ErrorCode::PreconditionFailed, "{condition}"); + } + } else if matches { + // The empty ETag case validates matching via If-Match above; + // emitting that invalid stored response validator fails closed. + assert!(result.is_err(), "{condition}"); + if !etag.is_empty() { + assert_eq!(result.unwrap_err().code(), &S3ErrorCode::NotModified, "{condition}"); + } + } else { + assert!(result.is_ok(), "{condition}"); + } + } + } } #[test] @@ -1409,6 +1451,77 @@ mod tests { assert!(check_preconditions(&headers18, &info18).is_ok()); } + #[test] + fn test_check_preconditions_rfc304() { + let modified = OffsetDateTime::from_unix_timestamp(1700000000).expect("valid timestamp"); + let mut info = ObjectInfo { + etag: Some("abc".to_owned()), + mod_time: Some(modified), + expires: Some(modified + time::Duration::days(1)), + version_id: Some(uuid::Uuid::nil()), + ..Default::default() + }; + info.user_defined = std::collections::HashMap::from([("cache-control".to_owned(), "max-age=60".to_owned())]).into(); + let mut headers = HeaderMap::new(); + headers.insert("if-none-match", HeaderValue::from_static("W/\"abc\"")); + let error = check_preconditions(&headers, &info).expect_err("weak ETag must validate the cache"); + assert_eq!(error.code(), &S3ErrorCode::NotModified); + let response = error.headers().expect("304 carries cache headers"); + assert_eq!(response["etag"], "\"abc\""); + assert_eq!(response["cache-control"], "max-age=60"); + assert!(response.contains_key("expires")); + assert!(response.contains_key("last-modified")); + assert_eq!(response["x-amz-version-id"], "null"); + + headers.insert("if-match", HeaderValue::from_static("\"other\"")); + assert_eq!( + check_preconditions(&headers, &info).expect_err("If-Match is first").code(), + &S3ErrorCode::PreconditionFailed + ); + headers.remove("if-none-match"); + headers.insert("if-match", HeaderValue::from_static("W/\"abc\"")); + assert_eq!( + check_preconditions(&headers, &info) + .expect_err("If-Match requires strong comparison") + .code(), + &S3ErrorCode::PreconditionFailed + ); + + headers.clear(); + headers.insert("if-none-match", HeaderValue::from_static("\"other\"")); + let date = modified.format(&crate::storage::RFC1123).expect("format timestamp"); + headers.insert("if-modified-since", HeaderValue::from_str(&date).expect("valid date header")); + assert!(check_preconditions(&headers, &info).is_ok(), "If-None-Match suppresses the date"); + headers.clear(); + headers.insert("if-unmodified-since", HeaderValue::from_str(&date).expect("valid date header")); + info.mod_time = Some(modified + time::Duration::seconds(1)); + assert_eq!( + check_preconditions(&headers, &info) + .expect_err("next second was modified") + .code(), + &S3ErrorCode::PreconditionFailed + ); + + headers.clear(); + headers.insert("if-none-match", HeaderValue::from_static("*")); + info.etag = None; + assert_eq!( + check_preconditions(&headers, &info) + .expect_err("existing object wildcard needs no ETag") + .code(), + &S3ErrorCode::NotModified + ); + headers.clear(); + headers.insert("if-match", HeaderValue::from_static("*")); + assert!(check_preconditions(&headers, &info).is_ok()); + headers.insert("if-match", HeaderValue::from_static("\"other\", \"abc,def\"")); + info.etag = Some("abc,def".to_owned()); + assert!( + check_preconditions(&headers, &info).is_ok(), + "commas inside an opaque tag are not list delimiters" + ); + } + #[test] fn test_parse_etag() { // [1] ETag without quotes → adds quotes diff --git a/rustfs/src/storage/helper.rs b/rustfs/src/storage/helper.rs index b8eb3c15d..805b5a870 100644 --- a/rustfs/src/storage/helper.rs +++ b/rustfs/src/storage/helper.rs @@ -13,7 +13,7 @@ // limitations under the License. use crate::module_switches::{is_audit_module_enabled, is_notify_module_enabled}; -use crate::shared_types::convert_ecstore_object_info; +use crate::shared_types::{RemoteAddr, convert_ecstore_object_info}; use crate::storage::access::{ReqInfo, request_context_from_req}; use crate::storage::request_context::RequestContext; use crate::storage::sse::KmsRequestAuditScope; @@ -33,6 +33,7 @@ use rustfs_s3_types::EventName; use rustfs_targets::{ extract_params_header, extract_req_params, extract_resp_elements, get_request_host, get_request_port, get_request_user_agent, }; +use rustfs_trusted_proxies::ClientInfo; use rustfs_utils::http::headers::AMZ_REQUEST_ID; use s3s::{S3Request, S3Response, S3Result}; use serde_json::Value; @@ -150,15 +151,6 @@ impl OperationHelper { .filter(|value| !value.is_empty()) .unwrap_or(path_object_key); - // Infer remote address - let remote_host = req - .headers - .get(rustfs_utils::http::X_FORWARDED_FOR) - .and_then(|v| v.to_str().ok()) - .or_else(|| req.headers.get(rustfs_utils::http::X_REAL_IP).and_then(|v| v.to_str().ok())) - .unwrap_or("") - .to_string(); - let trigger = op.as_str(); // Initialize audit builder @@ -179,15 +171,30 @@ impl OperationHelper { let request_id = request_context.request_id.clone(); let audit_builder = if audit_enabled { - Some( - AuditEntryBuilder::new("1.0", event, trigger, api_builder.clone().build()) - .remote_host(remote_host) - .user_agent(get_request_user_agent(&req.headers)) - .req_host(get_request_host(&req.headers)) - .req_path(req.uri.path().to_string()) - .req_query(extract_req_params(req)) - .request_id(&request_id), - ) + let remote_host = req + .extensions + .get::() + .map(|info| info.real_ip.to_string()) + .or_else(|| { + req.extensions + .get::>() + .and_then(|addr| addr.map(|addr| addr.0.ip().to_string())) + }) + .unwrap_or_default(); + let mut builder = AuditEntryBuilder::new("1.0", event, trigger, api_builder.clone().build()) + .remote_host(remote_host) + .user_agent(get_request_user_agent(&req.headers)) + .req_host(get_request_host(&req.headers)) + .req_path(req.uri.path().to_string()) + .req_query(extract_req_params(req)) + .request_id(&request_id); + if let Some(cred) = req_info + .and_then(|info| info.cred.as_ref()) + .filter(|cred| !cred.access_key.is_empty()) + { + builder = builder.access_key(&cred.access_key); + } + Some(builder) } else { None }; @@ -346,10 +353,6 @@ impl OperationHelper { final_builder = final_builder.error(err); } - if let Some(cred) = runtime_sources::current_action_credentials() { - final_builder = final_builder.access_key(&cred.access_key); - } - // Inject OpenTelemetry trace context into audit tags for distributed tracing correlation let mut tags = HashMap::new(); if let Some(ref tid) = state.request_context.trace_id { @@ -572,6 +575,133 @@ mod tests { ); } + #[test] + fn operation_helper_audits_the_authenticated_request_identity() { + with_vars( + [ + (rustfs_config::ENV_NOTIFY_ENABLE, Some("false")), + (rustfs_config::ENV_AUDIT_ENABLE, Some("true")), + ], + || { + refresh_notify_module_enabled(); + refresh_audit_module_enabled(); + + for (access_key, parent_user) in [ + (Some("testuser"), ""), + (Some("rustfsadmin"), ""), + (Some("service-account"), "testuser"), + (Some("sts-session"), "testuser"), + (None, ""), + (Some(""), ""), + ] { + let expected_access_key = access_key.filter(|key| !key.is_empty()); + for outcome in [None, Some(true), Some(false)] { + let input = GetObjectInput::builder() + .bucket("audit-bucket".to_string()) + .key("object.txt".to_string()) + .build() + .expect("get object input should build"); + let mut req = build_request(input, Method::GET, Uri::from_static("/audit-bucket/object.txt")); + req.extensions.insert(ReqInfo { + cred: access_key.map(|key| Credentials { + access_key: key.to_string(), + secret_key: "audit-secret-must-not-leak".to_string(), + session_token: "audit-token-must-not-leak".to_string(), + parent_user: parent_user.to_string(), + ..Default::default() + }), + ..Default::default() + }); + let mut helper = OperationHelper::new(&req, EventName::ObjectAccessedGet, S3Operation::GetObject); + if let Some(success) = outcome { + let result = if success { + Ok(S3Response::new(GetObjectOutput::default())) + } else { + Err(S3Error::new(S3ErrorCode::NoSuchKey)) + }; + helper = helper.complete(&result); + } + let OperationHelper::Enabled(state) = &mut helper else { + panic!("helper should be enabled when the audit switch is on"); + }; + let entry = state.audit_builder.take().expect("audit builder should exist").build(); + assert_eq!(entry.access_key.as_deref(), expected_access_key); + let json = serde_json::to_value(&entry).expect("audit entry should serialize"); + assert_eq!(json.get("accessKey").and_then(|key| key.as_str()), expected_access_key); + let rendered = serde_json::to_string(&entry).expect("audit entry should serialize"); + assert!(!rendered.contains("audit-secret-must-not-leak")); + assert!(!rendered.contains("audit-token-must-not-leak")); + } + } + }, + ); + } + + #[test] + fn operation_helper_audits_verified_client_addresses() { + use crate::shared_types::RemoteAddr; + use rustfs_trusted_proxies::{ClientInfo, ValidationMode}; + use std::net::SocketAddr; + + with_vars( + [ + (rustfs_config::ENV_NOTIFY_ENABLE, Some("false")), + (rustfs_config::ENV_AUDIT_ENABLE, Some("true")), + ], + || { + refresh_notify_module_enabled(); + refresh_audit_module_enabled(); + + let peer: SocketAddr = "192.0.2.10:54321".parse().unwrap(); + let ipv6_peer: SocketAddr = "[2001:db8::10]:54321".parse().unwrap(); + let proxied = ClientInfo::from_trusted_proxy( + "198.51.100.20".parse().unwrap(), + None, + None, + peer.ip(), + 1, + ValidationMode::Strict, + Vec::new(), + ); + for (remote_addr, client_info, expected) in [ + (Some(RemoteAddr(peer)), None, "192.0.2.10"), + (Some(RemoteAddr(ipv6_peer)), None, "2001:db8::10"), + (Some(RemoteAddr(peer)), Some(proxied), "198.51.100.20"), + (Some(RemoteAddr(peer)), Some(ClientInfo::direct(peer)), "192.0.2.10"), + (None, None, ""), + ] { + for spoofed in [false, true] { + let input = GetObjectInput::builder() + .bucket("audit-bucket".to_string()) + .key("object.txt".to_string()) + .build() + .expect("get object input should build"); + let mut req = build_request(input, Method::GET, Uri::from_static("/audit-bucket/object.txt")); + req.extensions.insert(remote_addr); + if let Some(client_info) = client_info.as_ref() { + req.extensions.insert(client_info.clone()); + } + if spoofed { + req.headers + .insert("x-forwarded-for", HeaderValue::from_static("203.0.113.99")); + req.headers.insert("x-real-ip", HeaderValue::from_static("203.0.113.98")); + } + let result = Ok(S3Response::new(GetObjectOutput::default())); + let mut helper = + OperationHelper::new(&req, EventName::ObjectAccessedGet, S3Operation::GetObject).complete(&result); + let OperationHelper::Enabled(state) = &mut helper else { + panic!("helper should be enabled when the audit switch is on"); + }; + let entry = state.audit_builder.take().expect("audit builder should exist").build(); + assert_eq!(entry.remote_host.as_deref(), Some(expected)); + let rendered = serde_json::to_value(&entry).expect("audit entry should serialize"); + assert_eq!(rendered["remotehost"], expected); + } + } + }, + ); + } + #[test] fn operation_helper_adds_objects_to_audit_details() { with_vars( diff --git a/rustfs/src/storage/rpc/node_service.rs b/rustfs/src/storage/rpc/node_service.rs index 81b6c1c7b..f8c178c8d 100644 --- a/rustfs/src/storage/rpc/node_service.rs +++ b/rustfs/src/storage/rpc/node_service.rs @@ -592,6 +592,14 @@ pub struct NodeService { local_peer: LocalPeerS3Client, context: Option>, server_ctx: Option>, + quota_topology_cache: std::sync::Mutex>, + #[cfg(test)] + quota_topology_computations: std::sync::atomic::AtomicUsize, +} + +struct QuotaTopologyCacheEntry { + store: std::sync::Weak, + fingerprint: String, } pub(crate) enum LocalMutationTarget { @@ -711,6 +719,9 @@ pub fn make_server_for_context(context: Option> local_peer, context, server_ctx: None, + quota_topology_cache: std::sync::Mutex::new(None), + #[cfg(test)] + quota_topology_computations: std::sync::atomic::AtomicUsize::new(0), } } @@ -1537,6 +1548,13 @@ impl Node for NodeService { self.handle_rename_part(request).await } + async fn recover_delete_transaction( + &self, + request: Request, + ) -> Result, Status> { + self.handle_recover_delete_transaction(request).await + } + async fn settle_part_transaction( &self, request: Request, @@ -1737,7 +1755,7 @@ impl Node for NodeService { if !request.get_ref().bucket_incarnation_id.is_empty() { return Err(Status::invalid_argument("incarnation-bound rename requires RenameDataAtIncarnation")); } - self.handle_rename_data(request).await + self.handle_rename_data(request, false).await } async fn rename_data_at_incarnation( @@ -1752,7 +1770,14 @@ impl Node for NodeService { { return Err(Status::invalid_argument("bucket incarnation must be a non-nil UUID")); } - self.handle_rename_data(request).await + self.handle_rename_data(request, false).await + } + + async fn rename_data_with_quota_fence( + &self, + request: Request, + ) -> Result, Status> { + self.handle_rename_data(request, true).await } async fn make_volumes(&self, request: Request) -> Result, Status> { @@ -1778,7 +1803,13 @@ impl Node for NodeService { &self, request: Request, ) -> Result, Status> { - self.handle_acquire_snapshot_lease(request).await + self.handle_acquire_snapshot_lease(request, false).await + } + async fn acquire_quota_mutation_fence( + &self, + request: Request, + ) -> Result, Status> { + self.handle_acquire_snapshot_lease(request, true).await } async fn renew_snapshot_lease( &self, @@ -1790,7 +1821,13 @@ impl Node for NodeService { &self, request: Request, ) -> Result, Status> { - self.handle_release_snapshot_lease(request).await + self.handle_release_snapshot_lease(request, false).await + } + async fn release_quota_mutation_fence( + &self, + request: Request, + ) -> Result, Status> { + self.handle_release_snapshot_lease(request, true).await } async fn read_metadata(&self, request: Request) -> Result, Status> { self.handle_read_metadata(request).await @@ -1874,6 +1911,13 @@ impl Node for NodeService { self.handle_delete_version(request, true).await } + async fn guarded_delete_versions( + &self, + request: Request, + ) -> Result, Status> { + self.handle_guarded_delete_versions(request).await + } + async fn delete_versions(&self, request: Request) -> Result, Status> { self.handle_delete_versions(request).await } @@ -3102,12 +3146,13 @@ mod tests { GetAllBucketStatsRequest, GetBucketInfoRequest, GetBucketStatsDataRequest, GetCpusRequest, GetMemInfoRequest, GetMetacacheListingRequest, GetMetricsRequest, GetNetInfoRequest, GetOsInfoRequest, GetPartitionsRequest, GetProcInfoRequest, GetSeLinuxInfoRequest, GetSrMetricsDataRequest, GetSysConfigRequest, GetSysErrorsRequest, - HealBucketRequest, HealControlRequest, HealControlResponse, ListBucketRequest, ListDirRequest, ListVolumesRequest, - LoadBucketMetadataRequest, LoadGroupRequest, LoadPolicyMappingRequest, LoadPolicyRequest, LoadRebalanceMetaRequest, - LoadServiceAccountRequest, LoadTransitionTierConfigRequest, LoadUserRequest, LocalStorageInfoRequest, MakeBucketRequest, - MakeVolumeRequest, MakeVolumesRequest, Mss, PingRequest, PreparePartTransactionRequest, ReadAllRequest, ReadAtRequest, - ReadMultipleRequest, ReadVersionRequest, ReadXlRequest, ReloadPoolMetaRequest, ReloadSiteReplicationConfigRequest, - RenameDataRequest, RenameFileRequest, RenamePartRequest, ScannerActivityRequest, ScannerDirtyUsageSnapshotRequest, + GuardedDeleteVersionsRequest, HealBucketRequest, HealControlRequest, HealControlResponse, ListBucketRequest, + ListDirRequest, ListVolumesRequest, LoadBucketMetadataRequest, LoadGroupRequest, LoadPolicyMappingRequest, + LoadPolicyRequest, LoadRebalanceMetaRequest, LoadServiceAccountRequest, LoadTransitionTierConfigRequest, LoadUserRequest, + LocalStorageInfoRequest, MakeBucketRequest, MakeVolumeRequest, MakeVolumesRequest, Mss, PingRequest, + PreparePartTransactionRequest, ReadAllRequest, ReadAtRequest, ReadMultipleRequest, ReadVersionRequest, ReadXlRequest, + RecoverDeleteTransactionRequest, ReloadPoolMetaRequest, ReloadSiteReplicationConfigRequest, RenameDataRequest, + RenameFileRequest, RenamePartRequest, ScannerActivityRequest, ScannerDirtyUsageSnapshotRequest, ScannerPublicationLeaseReleaseRequest, ScannerPublicationLeaseRequest, SealArtifactAttemptPutsRequest, ServerInfoRequest, SettlePartTransactionRequest, SignalServiceRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest, StartDecommissionRequest, StartProfilingRequest, StatVolumeRequest, StopRebalanceRequest, @@ -3131,7 +3176,7 @@ mod tests { use tonic::{Request, Response, Status}; use uuid::Uuid; - const DISK_MUTATION_RPC_METHODS: [&str; 29] = [ + const DISK_MUTATION_RPC_METHODS: [&str; 34] = [ "artifactrenamedata", "artifactrepairrenamedata", "artifactrepairwritemetadata", @@ -3139,12 +3184,14 @@ mod tests { "sealartifactattemptputs", "renamedata", "renamedataatincarnation", + "renamedatawithquotafence", "writemetadataatincarnation", "deleteatincarnation", "deleteversionatincarnation", "deleteversion", "deleteretiredmarker", "deleteversions", + "guardeddeleteversions", "writemetadata", "updatemetadata", "writeall", @@ -3155,12 +3202,15 @@ mod tests { "renamepart", "prepareparttransaction", "settleparttransaction", + "recoverdeletetransaction", "deletevolume", "makevolume", "makevolumes", "acquiresnapshotlease", + "acquirequotamutationfence", "renewsnapshotlease", "releasesnapshotlease", + "releasequotamutationfence", ]; fn normalized_rpc_method(method: &str) -> String { @@ -3239,6 +3289,13 @@ mod tests { Ok(None) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> rustfs_heal::Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> rustfs_heal::Result> { Ok(Vec::new()) } @@ -4319,12 +4376,56 @@ mod tests { }]) } + fn quota_two_pool_topologies() -> (EndpointServerPools, EndpointServerPools) { + let mut original = heal_control_test_endpoints("node-d").as_ref().to_vec(); + let mut second = heal_control_test_endpoints("node-e").as_ref()[0].clone(); + for (index, endpoint) in second.endpoints.as_mut().iter_mut().enumerate() { + *endpoint = Endpoint::try_from(format!("http://other-node-{index}:9000/disk{}", index + 1).as_str()) + .expect("disjoint second pool endpoint"); + endpoint.set_set_index(index / 2); + endpoint.set_disk_index(index % 2); + } + original.push(second); + let reindex = |pools: &mut Vec| { + for (index, pool) in pools.iter_mut().enumerate() { + for endpoint in pool.endpoints.as_mut() { + endpoint.set_pool_index(index); + } + } + }; + reindex(&mut original); + let mut swapped = original.clone(); + swapped.swap(0, 1); + reindex(&mut swapped); + (EndpointServerPools::from(original), EndpointServerPools::from(swapped)) + } + + #[test] + fn quota_topology_fingerprint_binds_pool_order_after_reindexing() { + let (original, swapped) = quota_two_pool_topologies(); + assert_ne!( + heal_topology_fingerprint(&original).expect("valid original pools"), + heal_topology_fingerprint(&swapped).expect("valid reordered pools"), + "different pool zero lock owners must not share a quota coordinator identity" + ); + } + fn mark_v2_authenticated(request: &mut Request) { request .metadata_mut() .insert("x-rustfs-rpc-auth-version", "2".parse().expect("valid metadata value")); } + fn sign_quota_test_request(request: &mut Request, body: &[u8], topology: &str) { + request.metadata_mut().insert( + rustfs_protos::QUOTA_MUTATION_TOPOLOGY_HEADER, + topology.parse().expect("quota topology metadata"), + ); + let body = rustfs_protos::canonical_quota_mutation_request_body(topology, body).expect("quota topology envelope"); + set_tonic_canonical_body_digest(request, &body).expect("quota request digest"); + mark_v2_authenticated(request); + } + fn signed_tier_prepare_request(mutation_id: uuid::Uuid, canonical_payload: Bytes) -> Request { let mut request = Request::new(TierMutationPrepareRequest { version: rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION, @@ -4425,6 +4526,269 @@ mod tests { assert!(!signed.into_inner().success, "the unknown test disk cannot resolve"); } + #[tokio::test] + async fn delete_recovery_apply_rejects_excessive_batch_cardinality_before_lookup() { + use crate::storage::storage_api::{DeleteOptions, DeleteRecoveryRpcOperation, FileInfoVersions}; + let service = make_server(); + for versions in [ + vec![FileInfoVersions::default(); 1001], + vec![FileInfoVersions { + name: "object".into(), + versions: vec![Default::default(); 1001], + ..Default::default() + }], + ] { + let operation = DeleteRecoveryRpcOperation::ApplyVersions { + versions, + opts: DeleteOptions { + old_data_dir: Some(Uuid::new_v4()), + ..Default::default() + }, + }; + let mut request = Request::new(RecoverDeleteTransactionRequest { + disk: "unknown".into(), + volume: "bucket".into(), + protocol_version: 1, + operation: rmp_serde::to_vec_named(&operation).expect("encode batch").into(), + }); + let body = + rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()).expect("canonical batch"); + set_tonic_canonical_body_digest(&mut request, &body).expect("batch digest"); + mark_v2_authenticated(&mut request); + let error = service + .recover_delete_transaction(request) + .await + .expect_err("oversized batch must reject before disk lookup"); + assert_eq!(error.code(), tonic::Code::InvalidArgument); + } + } + + #[tokio::test] + async fn delete_recovery_apply_rejects_legacy_cleanup_options_and_absolute_keys() { + use crate::storage::storage_api::{DeleteOptions, DeleteRecoveryRpcOperation}; + let service = make_server(); + let valid = DeleteOptions { + old_data_dir: Some(Uuid::new_v4()), + ..Default::default() + }; + let mut invalid = vec![DeleteOptions::default()]; + for mutate in [ + |opts: &mut DeleteOptions| opts.old_data_dir = Some(Uuid::nil()), + |opts: &mut DeleteOptions| opts.undo_delete = true, + |opts: &mut DeleteOptions| opts.undo_write = true, + |opts: &mut DeleteOptions| opts.recursive = true, + |opts: &mut DeleteOptions| opts.immediate = true, + ] { + let mut opts = valid.clone(); + mutate(&mut opts); + invalid.push(opts); + } + let mut operations = Vec::new(); + for opts in invalid { + operations.push(DeleteRecoveryRpcOperation::ApplyVersion { + object: "object".into(), + file_info: Default::default(), + force_del_marker: false, + opts: opts.clone(), + }); + operations.push(DeleteRecoveryRpcOperation::ApplyVersions { + versions: Vec::new(), + opts, + }); + } + operations.push(DeleteRecoveryRpcOperation::ApplyVersion { + object: "/object".into(), + file_info: Default::default(), + force_del_marker: false, + opts: valid.clone(), + }); + operations.push(DeleteRecoveryRpcOperation::ApplyVersions { + versions: vec![crate::storage::storage_api::FileInfoVersions { + name: "/object".into(), + ..Default::default() + }], + opts: valid, + }); + for operation in operations { + let mut request = Request::new(RecoverDeleteTransactionRequest { + disk: "unknown".into(), + volume: "bucket".into(), + protocol_version: 1, + operation: rmp_serde::to_vec_named(&operation).expect("encode apply operation").into(), + }); + let body = + rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()).expect("canonical apply"); + set_tonic_canonical_body_digest(&mut request, &body).expect("apply digest"); + mark_v2_authenticated(&mut request); + let error = service + .recover_delete_transaction(request) + .await + .expect_err("invalid apply must reject before disk lookup"); + assert_eq!(error.code(), tonic::Code::InvalidArgument); + } + } + + #[tokio::test] + async fn orphan_data_dir_probe_requires_body_bound_authentication() { + let service = make_server(); + let operation = crate::storage::storage_api::DeleteRecoveryRpcOperation::InspectOrphanDataDir { + path: format!("object/{}", uuid::Uuid::new_v4()), + }; + let message = RecoverDeleteTransactionRequest { + disk: "unknown".into(), + volume: "bucket".into(), + protocol_version: 1, + operation: rmp_serde::to_vec_named(&operation).expect("encode orphan probe").into(), + }; + let error = service + .recover_delete_transaction(Request::new(message.clone())) + .await + .expect_err("unsigned probe must fail"); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + let mut tampered = Request::new(message.clone()); + let body = rustfs_protos::canonical_recover_delete_transaction_request_body(tampered.get_ref()).expect("canonical probe"); + set_tonic_canonical_body_digest(&mut tampered, &body).expect("probe digest"); + mark_v2_authenticated(&mut tampered); + tampered.get_mut().volume = "other-bucket".into(); + let error = service + .recover_delete_transaction(tampered) + .await + .expect_err("bucket scope must be body-bound"); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + let mut signed = Request::new(message); + set_tonic_canonical_body_digest(&mut signed, &body).expect("probe digest"); + mark_v2_authenticated(&mut signed); + let result = service + .recover_delete_transaction(signed) + .await + .expect("signed probe reaches disk lookup") + .into_inner(); + assert!(!result.success); + assert!(result.error.is_some()); + } + + #[tokio::test] + async fn delete_recovery_exists_requires_body_bound_authentication() { + use crate::storage::storage_api::DeleteRecoveryRpcOperation; + let service = make_server(); + let operation = DeleteRecoveryRpcOperation::Exists { + object: "object".to_owned(), + }; + let message = RecoverDeleteTransactionRequest { + disk: "unknown".into(), + volume: "bucket".into(), + protocol_version: 1, + operation: rmp_serde::to_vec_named(&operation).expect("encode existence probe").into(), + }; + let error = service + .recover_delete_transaction(Request::new(message.clone())) + .await + .expect_err("unsigned existence probe must fail"); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + for change_object in [false, true] { + let mut request = Request::new(message.clone()); + let body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()) + .expect("canonical existence probe"); + set_tonic_canonical_body_digest(&mut request, &body).expect("existence digest"); + mark_v2_authenticated(&mut request); + if change_object { + request.get_mut().operation = rmp_serde::to_vec_named(&DeleteRecoveryRpcOperation::Exists { + object: "other-object".to_owned(), + }) + .expect("encode changed key") + .into(); + } else { + request.get_mut().volume = "other-bucket".into(); + } + let error = service + .recover_delete_transaction(request) + .await + .expect_err("bucket and exact-key scopes must be body-bound"); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + } + let mut request = Request::new(message); + let body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()) + .expect("canonical existence probe"); + set_tonic_canonical_body_digest(&mut request, &body).expect("existence digest"); + mark_v2_authenticated(&mut request); + let result = service + .recover_delete_transaction(request) + .await + .expect("signed probe reaches disk lookup") + .into_inner(); + assert!(!result.success, "an unavailable disk cannot establish absence"); + assert!(result.error.is_some()); + } + + #[tokio::test] + async fn delete_recovery_requires_digest_and_valid_protocol_payload() { + let service = make_server(); + let operation = crate::storage::storage_api::DeleteRecoveryRpcOperation::Discover { + object: None, + cursor: None, + limit: 1, + }; + let message = RecoverDeleteTransactionRequest { + disk: "unknown".into(), + volume: "bucket".into(), + protocol_version: 1, + operation: rmp_serde::to_vec_named(&operation).expect("encode discovery").into(), + }; + let unsigned = service + .recover_delete_transaction(Request::new(message.clone())) + .await + .expect_err("new RPC must reject unsigned requests"); + assert_eq!(unsigned.code(), tonic::Code::PermissionDenied); + for digest in ["", "UNSIGNED-PAYLOAD"] { + let mut request = Request::new(message.clone()); + request + .metadata_mut() + .insert("x-rustfs-content-sha256", digest.parse().expect("metadata")); + mark_v2_authenticated(&mut request); + let error = service + .recover_delete_transaction(request) + .await + .expect_err("missing or unsigned digest"); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + } + let mut invalid = message.clone(); + invalid.protocol_version = 2; + let mut malformed = message.clone(); + malformed.operation = vec![0xc1].into(); + let mut trailing = message.clone(); + let mut payload = trailing.operation.to_vec(); + payload.push(0xc0); + trailing.operation = payload.into(); + for (message, expected) in [ + (invalid, tonic::Code::Unimplemented), + (malformed, tonic::Code::InvalidArgument), + (trailing, tonic::Code::InvalidArgument), + ] { + let mut request = Request::new(message); + let body = + rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()).expect("canonical recovery"); + set_tonic_canonical_body_digest(&mut request, &body).expect("body digest"); + mark_v2_authenticated(&mut request); + let error = service + .recover_delete_transaction(request) + .await + .expect_err("invalid recovery input"); + assert_eq!(error.code(), expected); + } + let mut request = Request::new(message); + let body = + rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()).expect("canonical recovery"); + set_tonic_canonical_body_digest(&mut request, &body).expect("body digest"); + mark_v2_authenticated(&mut request); + let response = service + .recover_delete_transaction(request) + .await + .expect("valid request reaches disk lookup") + .into_inner(); + assert!(!response.success); + assert!(response.error.is_some()); + } + /// Per-handler wiring check for every mutating disk RPC. A mismatched digest must be rejected /// (catches a handler that omits its `verify_disk_mutation_digest` gate) and a correctly /// body-bound digest must pass the gate (catches a handler wired to the wrong @@ -4452,8 +4816,16 @@ mod tests { // lookup, so it must NOT fail with PermissionDenied. let mut ok = Request::new(msg.clone()); let body = $canonical(ok.get_ref()).expect("canonical body should encode"); - set_tonic_canonical_body_digest(&mut ok, &body).expect("digest metadata should encode"); - mark_v2_authenticated(&mut ok); + let quota_method = matches!( + stringify!($method), + "acquire_quota_mutation_fence" | "release_quota_mutation_fence" | "rename_data_with_quota_fence" + ); + if quota_method { + sign_quota_test_request(&mut ok, &body, &"a".repeat(64)); + } else { + set_tonic_canonical_body_digest(&mut ok, &body).expect("digest metadata should encode"); + mark_v2_authenticated(&mut ok); + } if let Err(status) = service.$method(ok).await { assert_ne!( status.code(), @@ -4464,8 +4836,13 @@ mod tests { // Mismatched digest: the gate must reject before any disk work. let mut bad = Request::new(msg); - set_tonic_canonical_body_digest(&mut bad, b"unrelated-canonical-body").expect("digest metadata should encode"); - mark_v2_authenticated(&mut bad); + if quota_method { + sign_quota_test_request(&mut bad, b"unrelated-canonical-body", &"a".repeat(64)); + } else { + set_tonic_canonical_body_digest(&mut bad, b"unrelated-canonical-body") + .expect("digest metadata should encode"); + mark_v2_authenticated(&mut bad); + } let err = service .$method(bad) .await @@ -4549,6 +4926,15 @@ mod tests { }, rustfs_protos::canonical_rename_data_request_body ); + assert_gated!( + rename_data_with_quota_fence, + RenameDataRequest { + disk: disk.clone(), + file_info: "{}".into(), + ..Default::default() + }, + rustfs_protos::canonical_rename_data_request_body + ); assert_gated!( delete_version, DeleteVersionRequest { @@ -4616,6 +5002,16 @@ mod tests { }, rustfs_protos::canonical_delete_request_body ); + assert_gated!( + guarded_delete_versions, + GuardedDeleteVersionsRequest { + disk: disk.clone(), + volume: "v".into(), + payload: vec![0x80].into(), + probe_only: false, + }, + rustfs_protos::canonical_guarded_delete_versions_request_body + ); assert_gated!( delete_versions, DeleteVersionsRequest { @@ -4696,6 +5092,16 @@ mod tests { }, rustfs_protos::canonical_snapshot_lease_request_body ); + assert_gated!( + acquire_quota_mutation_fence, + SnapshotLeaseRequest { + disk: disk.clone(), + volume: ".rustfs.sys".into(), + path: format!("tmp/quota-mutation-fences/{}", "a".repeat(64)), + ttl_ms: 60_000, + }, + rustfs_protos::canonical_snapshot_lease_request_body + ); assert_gated!( renew_snapshot_lease, SnapshotLeaseRenewRequest { @@ -4717,6 +5123,16 @@ mod tests { }, rustfs_protos::canonical_snapshot_lease_release_request_body ); + assert_gated!( + release_quota_mutation_fence, + SnapshotLeaseReleaseRequest { + disk: disk.clone(), + volume: ".rustfs.sys".into(), + path: format!("tmp/quota-mutation-fences/{}", "a".repeat(64)), + token: vec![0; 16].into(), + }, + rustfs_protos::canonical_snapshot_lease_release_request_body + ); assert_gated!( delete_paths, DeletePathsRequest { @@ -4762,6 +5178,16 @@ mod tests { }, rustfs_protos::canonical_prepare_part_transaction_request_body ); + assert_gated!( + recover_delete_transaction, + RecoverDeleteTransactionRequest { + disk: disk.clone(), + volume: "dst".into(), + protocol_version: 1, + operation: vec![0x80].into() + }, + rustfs_protos::canonical_recover_delete_transaction_request_body + ); assert_gated!( settle_part_transaction, SettlePartTransactionRequest { @@ -4875,6 +5301,254 @@ mod tests { ); } + #[tokio::test] + async fn quota_fence_handlers_reject_unsigned_requests_wrong_scopes_and_invalid_tokens() { + use crate::storage::storage_api::ecstore_disk::QUOTA_MUTATION_FENCE_METADATA_SUFFIX; + use rustfs_filemeta::FileInfo; + + let service = super::make_server_for_slot(super::ServerContextSlot::new()); + macro_rules! signed { + ($message:expr, $canonical:path) => {{ + let mut request = Request::new($message); + let body = $canonical(request.get_ref()).expect("quota request canonical body"); + sign_quota_test_request(&mut request, &body, &"a".repeat(64)); + request + }}; + } + assert_eq!( + service + .acquire_quota_mutation_fence(Request::new(SnapshotLeaseRequest::default())) + .await + .expect_err("unsigned acquire") + .code(), + tonic::Code::PermissionDenied + ); + assert_eq!( + service + .release_quota_mutation_fence(Request::new(SnapshotLeaseReleaseRequest::default())) + .await + .expect_err("unsigned release") + .code(), + tonic::Code::PermissionDenied + ); + assert_eq!( + service + .rename_data_with_quota_fence(Request::new(RenameDataRequest::default())) + .await + .expect_err("unsigned rename") + .code(), + tonic::Code::PermissionDenied + ); + + for (volume, path) in [ + ("bucket", format!("tmp/quota-mutation-fences/{}", "a".repeat(64))), + (".rustfs.sys", "tmp/quota-mutation-fences/not-a-digest".to_string()), + ] { + let acquire = SnapshotLeaseRequest { + volume: volume.into(), + path: path.clone(), + ..Default::default() + }; + let release = SnapshotLeaseReleaseRequest { + volume: volume.into(), + path, + token: vec![0; 16].into(), + ..Default::default() + }; + assert_eq!( + service + .acquire_quota_mutation_fence(signed!(acquire, rustfs_protos::canonical_snapshot_lease_request_body)) + .await + .expect_err("invalid acquire scope") + .code(), + tonic::Code::InvalidArgument + ); + assert_eq!( + service + .release_quota_mutation_fence(signed!(release, rustfs_protos::canonical_snapshot_lease_release_request_body)) + .await + .expect_err("invalid release scope") + .code(), + tonic::Code::InvalidArgument + ); + } + + for token in [None, Some("not-a-uuid".to_string()), Some(Uuid::nil().to_string())] { + let mut file_info = FileInfo::default(); + if let Some(token) = token { + rustfs_utils::http::metadata_compat::insert_str( + &mut file_info.metadata, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + token, + ); + } + let message = RenameDataRequest { + file_info: serde_json::to_string(&file_info).expect("file info"), + ..Default::default() + }; + assert_eq!( + service + .rename_data_with_quota_fence(signed!(message, rustfs_protos::canonical_rename_data_request_body)) + .await + .expect_err("invalid quota token") + .code(), + tonic::Code::InvalidArgument + ); + } + let mut file_info = FileInfo::default(); + rustfs_utils::http::metadata_compat::insert_str( + &mut file_info.metadata, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + Uuid::new_v4().to_string(), + ); + file_info.metadata.insert( + format!("x-minio-internal-{QUOTA_MUTATION_FENCE_METADATA_SUFFIX}"), + Uuid::new_v4().to_string(), + ); + let conflicting = RenameDataRequest { + file_info: serde_json::to_string(&file_info).expect("file info"), + ..Default::default() + }; + assert_eq!( + service + .rename_data_with_quota_fence(signed!(conflicting, rustfs_protos::canonical_rename_data_request_body)) + .await + .expect_err("conflicting quota tokens") + .code(), + tonic::Code::InvalidArgument + ); + + let path = format!("tmp/quota-mutation-fences/{}", "a".repeat(64)); + let acquire = SnapshotLeaseRequest { + volume: ".rustfs.sys".into(), + path: path.clone(), + ..Default::default() + }; + assert_eq!( + service + .acquire_quota_mutation_fence(signed!(acquire, rustfs_protos::canonical_snapshot_lease_request_body)) + .await + .expect_err("quota acquire requires a captured ready target") + .code(), + tonic::Code::FailedPrecondition + ); + let release = SnapshotLeaseReleaseRequest { + volume: ".rustfs.sys".into(), + path, + token: vec![0; 16].into(), + ..Default::default() + }; + assert_eq!( + service + .release_quota_mutation_fence(signed!(release, rustfs_protos::canonical_snapshot_lease_release_request_body)) + .await + .expect_err("quota release requires a captured ready target") + .code(), + tonic::Code::FailedPrecondition + ); + rustfs_utils::http::metadata_compat::insert_str( + &mut file_info.metadata, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + Uuid::new_v4().to_string(), + ); + let valid = RenameDataRequest { + file_info: serde_json::to_string(&file_info).expect("file info"), + ..Default::default() + }; + assert_eq!( + service + .rename_data_with_quota_fence(signed!(valid, rustfs_protos::canonical_rename_data_request_body)) + .await + .expect_err("quota rename requires a captured ready target") + .code(), + tonic::Code::FailedPrecondition + ); + } + + #[tokio::test] + async fn quota_fence_handlers_reject_missing_duplicate_malformed_and_tampered_topology() { + let service = super::make_server_for_slot(super::ServerContextSlot::new()); + macro_rules! rejects_invalid_topology { + ($method:ident, $message:expr, $canonical:path) => {{ + for case in [ + "missing", + "duplicate", + "empty", + "short", + "uppercase", + "nonhex", + "tampered", + "v1", + ] { + let mut request = Request::new($message); + let body = $canonical(request.get_ref()).expect("quota canonical body"); + sign_quota_test_request(&mut request, &body, &"a".repeat(64)); + let header = rustfs_protos::QUOTA_MUTATION_TOPOLOGY_HEADER; + match case { + "missing" => { + request.metadata_mut().remove(header); + } + "duplicate" => { + request + .metadata_mut() + .append(header, "a".repeat(64).parse().expect("header")); + } + "empty" => { + request.metadata_mut().insert(header, "".parse().expect("header")); + } + "short" => { + request + .metadata_mut() + .insert(header, "a".repeat(63).parse().expect("header")); + } + "uppercase" => { + request + .metadata_mut() + .insert(header, "A".repeat(64).parse().expect("header")); + } + "nonhex" => { + request + .metadata_mut() + .insert(header, "g".repeat(64).parse().expect("header")); + } + "tampered" => { + request + .metadata_mut() + .insert(header, "b".repeat(64).parse().expect("header")); + } + "v1" => { + request + .metadata_mut() + .insert("x-rustfs-rpc-auth-version", "1".parse().expect("auth version")); + } + _ => unreachable!(), + } + assert_eq!( + service.$method(request).await.expect_err(case).code(), + tonic::Code::PermissionDenied, + "{} must reject {case} topology authentication before resolving a disk", + stringify!($method) + ); + } + }}; + } + rejects_invalid_topology!( + acquire_quota_mutation_fence, + SnapshotLeaseRequest::default(), + rustfs_protos::canonical_snapshot_lease_request_body + ); + rejects_invalid_topology!( + release_quota_mutation_fence, + SnapshotLeaseReleaseRequest::default(), + rustfs_protos::canonical_snapshot_lease_release_request_body + ); + rejects_invalid_topology!( + rename_data_with_quota_fence, + RenameDataRequest::default(), + rustfs_protos::canonical_rename_data_request_body + ); + } + #[tokio::test] async fn snapshot_lease_acquire_and_renew_handlers_fail_closed_for_missing_disk() { let service = make_server(); @@ -6227,6 +6901,124 @@ mod tests { request } + #[tokio::test] + async fn delete_recovery_exists_rpc_reports_exact_key_evidence_without_mutation() { + use crate::storage::storage_api::{DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply}; + let fixture = target_rpc_fixture().await; + let set = fixture + .env + .ecstore + .all_set_disks() + .into_iter() + .next() + .expect("target erasure set"); + let configured = set.disks.read().await.iter().find_map(Clone::clone).expect("configured disk"); + let service = make_server_for_context(Some(fixture.context.clone())); + let disk = service + .find_disk(&configured.endpoint().to_string()) + .await + .expect("RPC target disk"); + let volume = "evidence-rpc"; + disk.make_volume(volume).await.expect("create evidence bucket"); + let record = format!("object/{}/delete-prepared", Uuid::new_v4()); + disk.write_all(volume, &record, Bytes::from_static(b"unresolved recovery record")) + .await + .expect("write evidence"); + for (object, expected) in [("object", true), ("other", false)] { + let mut request = Request::new(RecoverDeleteTransactionRequest { + disk: configured.endpoint().to_string(), + volume: volume.to_owned(), + protocol_version: 1, + operation: rmp_serde::to_vec_named(&DeleteRecoveryRpcOperation::Exists { + object: object.to_owned(), + }) + .expect("encode existence probe") + .into(), + }); + let body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()) + .expect("canonical existence probe"); + set_tonic_canonical_body_digest(&mut request, &body).expect("existence digest"); + mark_v2_authenticated(&mut request); + let response = service + .recover_delete_transaction(request) + .await + .expect("existence RPC response") + .into_inner(); + assert!(response.success, "existence RPC failed: {:?}", response.error); + assert!(matches!( + rustfs_protos::decode_delete_recovery_payload::(&response.result).expect("decode evidence reply"), + DeleteRecoveryRpcReply::Evidence { present } if present == expected + )); + } + assert_eq!( + disk.read_all(volume, &record).await.expect("read retained evidence"), + b"unresolved recovery record"[..] + ); + } + + #[tokio::test] + async fn orphan_data_dir_delete_rpc_reports_snapshot_deferral_as_pending() { + let fixture = target_rpc_fixture().await; + let set = fixture + .env + .ecstore + .all_set_disks() + .into_iter() + .next() + .expect("target erasure set"); + let configured = set.disks.read().await.iter().find_map(Clone::clone).expect("configured disk"); + let service = make_server_for_context(Some(fixture.context.clone())); + let disk = service + .find_disk(&configured.endpoint().to_string()) + .await + .expect("RPC target disk"); + let volume = "orphan-rpc"; + disk.make_volume(volume).await.expect("create orphan bucket"); + let mut info = rustfs_filemeta::FileInfo::new("object", 1, 1); + info.data_dir = Some(Uuid::new_v4()); + info.mod_time = Some(OffsetDateTime::now_utc()); + info.size = 1; + let mut metadata = rustfs_filemeta::FileMeta::default(); + metadata.add_version(info).expect("add parent version"); + disk.write_all(volume, "object/xl.meta", metadata.marshal_msg().expect("encode parent metadata").into()) + .await + .expect("write parent metadata"); + let path = format!("object/{}", Uuid::new_v4()); + disk.write_all(volume, &format!("{path}/part.1"), Bytes::from_static(b"snapshot payload")) + .await + .expect("write orphan payload"); + let token = disk.acquire_snapshot_lease(volume, &path).await.expect("acquire snapshot"); + let mut request = Request::new(DeleteRequest { + disk: configured.endpoint().to_string(), + volume: volume.into(), + path: path.clone(), + options: serde_json::to_string(&crate::storage::storage_api::DeleteOptions { + orphan_data_dir: true, + immediate: true, + ..Default::default() + }) + .expect("encode orphan options"), + ..Default::default() + }); + let body = rustfs_protos::canonical_delete_request_body(request.get_ref()).expect("canonical orphan delete"); + set_tonic_canonical_body_digest(&mut request, &body).expect("delete digest"); + mark_v2_authenticated(&mut request); + let reply = service.delete(request).await.expect("run delete handler").into_inner(); + assert!(!reply.success, "the legacy RPC must not report deferred reclamation as completed"); + let error: DiskError = reply.error.expect("pending error code").into(); + assert_eq!(error, DiskError::DiskOngoingReq); + assert_eq!( + disk.read_all(volume, &format!("{path}/part.1")) + .await + .expect("read retained snapshot"), + b"snapshot payload"[..] + ); + disk.release_snapshot_lease(volume, &path, token) + .await + .expect("release completes safe cleanup"); + assert!(matches!(disk.list_dir("", volume, &path, 1).await, Err(DiskError::FileNotFound))); + } + #[tokio::test] async fn compare_and_update_file_handler_preserves_atomic_outcomes() { use rustfs_protos::proto_gen::node_service::CompareAndUpdateFileOutcome; @@ -6338,6 +7130,251 @@ mod tests { assert!(matches!(disk.read_all(volume, &path).await, Err(DiskError::FileNotFound))); } + #[cfg(not(windows))] + #[tokio::test] + async fn guarded_delete_remote_production_handler_preserves_probe_and_typed_results() { + use crate::storage::storage_api::{ + ecstore_disk::{ + CurrentDeleteGeneration, DeleteOptions, DiskOption, FileInfoVersions, GuardedFileInfoVersions, new_disk, + }, + init_local_disks_with_instance_ctx, + }; + use std::{ + collections::BTreeMap, + path::{Path, PathBuf}, + sync::Mutex, + }; + use tower_http::trace::TraceLayer; + + fn snapshot_tree(root: &Path) -> BTreeMap>> { + let mut entries = BTreeMap::new(); + let mut pending = vec![root.to_path_buf()]; + while let Some(directory) = pending.pop() { + for entry in std::fs::read_dir(&directory).expect("read fixture directory") { + let entry = entry.expect("read fixture entry"); + let path = entry.path(); + let relative = path + .strip_prefix(root) + .expect("fixture path remains beneath root") + .to_path_buf(); + if entry.file_type().expect("fixture entry type").is_dir() { + entries.insert(relative, None); + pending.push(path); + } else { + entries.insert(relative, Some(std::fs::read(path).expect("read fixture file"))); + } + } + } + entries + } + + fn inline_info(object: &str, etag: &str) -> rustfs_filemeta::FileInfo { + let mut info = rustfs_filemeta::FileInfo::new(object, 1, 0); + info.name = object.to_owned(); + info.erasure.index = 1; + info.version_id = Some(Uuid::nil()); + info.data_dir = Some(Uuid::new_v4()); + info.mod_time = Some(OffsetDateTime::now_utc()); + info.size = 7; + info.parts = vec![rustfs_filemeta::ObjectPartInfo { + number: 1, + size: 7, + actual_size: 7, + ..Default::default() + }]; + info.data = Some(Bytes::from_static(b"payload")); + info.metadata.insert("etag".into(), etag.into()); + info.set_inline_data(); + info + } + + fn guarded_group(info: rustfs_filemeta::FileInfo) -> GuardedFileInfoVersions { + let expected_current = CurrentDeleteGeneration::from_file_info(&info).expect("capture fixture generation"); + GuardedFileInfoVersions { + group: FileInfoVersions { + name: info.name.clone(), + versions: vec![info], + ..Default::default() + }, + expected_current, + } + } + + let fixture = target_rpc_fixture().await; + let _ = rustfs_credentials::set_global_rpc_secret(Uuid::new_v4().to_string()); + let listener = TcpListener::bind("127.0.0.1:0") + .await + .expect("bind guarded production RPC target"); + let address = listener.local_addr().expect("guarded RPC target address"); + let mut endpoints = fixture.env.endpoint_pools.as_ref()[0].endpoints.as_ref().clone(); + let mut remote_endpoint = Endpoint::try_from(format!("http://{address}{}", fixture.env.disk_paths[0].display()).as_str()) + .expect("guarded RPC endpoint"); + remote_endpoint.set_pool_index(0); + remote_endpoint.set_set_index(0); + remote_endpoint.set_disk_index(0); + remote_endpoint.is_local = true; + endpoints[0] = remote_endpoint.clone(); + let mut pool = fixture.env.endpoint_pools.as_ref()[0].clone(); + pool.endpoints = Endpoints::from(endpoints); + init_local_disks_with_instance_ctx(&fixture.instance, EndpointServerPools::from(vec![pool])) + .await + .expect("register production local disk at its RPC endpoint"); + let service = make_server_for_context(Some(fixture.context.clone())); + let local = service + .find_disk(&remote_endpoint.to_string()) + .await + .expect("registered production local disk"); + let bucket = "guarded-delete-rpc"; + local.make_volume(bucket).await.expect("create guarded fixture volume"); + let first = inline_info("matching-first", "etag-first"); + let stale = inline_info("stale-middle", "etag-old"); + let last = inline_info("matching-last", "etag-last"); + for info in [&first, &stale, &last] { + local + .write_metadata(bucket, bucket, &info.name, info.clone()) + .await + .expect("write fixture object metadata"); + } + for info in [&first, &stale, &last] { + let published = local + .read_version( + bucket, + bucket, + &info.name, + "", + &crate::storage::storage_api::ecstore_disk::ReadOptions::default(), + ) + .await + .expect("read published fixture generation"); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&published).expect("capture published fixture generation"), + CurrentDeleteGeneration::from_file_info(info).expect("capture original fixture generation"), + "the fixture must capture the exact generation published to xl.meta" + ); + } + let mut replacement = stale.clone(); + replacement.data_dir = Some(Uuid::new_v4()); + replacement.mod_time = stale.mod_time.map(|time| time + time::Duration::seconds(1)); + replacement.metadata.insert("etag".into(), "etag-replacement".into()); + local + .write_metadata(bucket, bucket, &replacement.name, replacement.clone()) + .await + .expect("commit a replacement generation"); + let disk_path = local.path(); + let volume_path = disk_path.join(bucket); + let before_probe = snapshot_tree(&disk_path); + let preserved_replacement = snapshot_tree(&volume_path.join(&stale.name)); + let calls = Arc::new(Mutex::new(Vec::::new())); + let recorded_calls = Arc::clone(&calls); + let trace = TraceLayer::new_for_grpc().on_request(move |request: &http::Request<_>, _: &tracing::Span| { + recorded_calls + .lock() + .expect("RPC method recording lock") + .push(request.uri().path().to_owned()); + }); + let (shutdown, stopped) = tokio::sync::oneshot::channel(); + let server = tokio::spawn(async move { + // This uses the production service and protobuf transport. Signature + // middleware has separate coverage; RemoteDisk still signs each body. + tonic::transport::Server::builder() + .layer(trace) + .add_service(NodeServiceServer::new(service)) + .serve_with_incoming_shutdown(TcpListenerStream::new(listener), async { + let _ = stopped.await; + }) + .await + .expect("serve guarded production RPC target"); + }); + remote_endpoint.is_local = false; + let remote = new_disk( + &remote_endpoint, + &DiskOption { + cleanup: false, + health_check: false, + }, + ) + .await + .expect("construct RemoteDisk client"); + assert!(!remote.is_local(), "the test must exercise the RemoteDisk implementation"); + remote + .supports_guarded_delete_versions() + .await + .expect("production peer supports guarded deletion"); + assert_eq!( + snapshot_tree(&disk_path), + before_probe, + "capability probing creates no metadata or recovery artifacts anywhere on the target disk" + ); + let transaction = Uuid::new_v4(); + let groups = vec![ + guarded_group(first.clone()), + guarded_group(stale.clone()), + guarded_group(last.clone()), + ]; + let payload = crate::storage::storage_api::ecstore_disk::GuardedDeleteVersionsPayload { + versions: groups.clone(), + opts: DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + }; + let encoded = rmp_serde::to_vec_named(&payload).expect("encode guarded fixture payload"); + let decoded: crate::storage::storage_api::ecstore_disk::GuardedDeleteVersionsPayload = + rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode guarded fixture payload"); + for (original, decoded) in groups.iter().zip(&decoded.versions) { + assert_eq!( + decoded.expected_current, original.expected_current, + "guarded generation must survive its RPC codec exactly" + ); + } + let errors = remote + .delete_versions_guarded_recoverable( + bucket, + groups, + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await; + assert_eq!( + errors, + vec![None, Some(DiskError::ConditionalDeleteConflict), None], + "typed per-item results retain the original batch indices" + ); + for info in [&first, &last] { + let object = volume_path.join(&info.name); + assert!(!object.join("xl.meta").exists(), "matching guarded objects are deleted"); + assert!( + object.join(transaction.to_string()).join("xl.meta.bkp").is_file(), + "successful guarded deletion retains its recoverable receipt" + ); + } + assert_eq!( + snapshot_tree(&volume_path.join(&stale.name)), + preserved_replacement, + "the stale guard preserves replacement metadata and creates no recovery artifacts" + ); + let deletion_methods: Vec<_> = calls + .lock() + .expect("RPC method recording lock") + .iter() + .filter(|path| path.contains("Delete")) + .cloned() + .collect(); + assert_eq!( + deletion_methods, + vec!["/node_service.NodeService/GuardedDeleteVersions"; 2], + "probe and application use the additive protocol without legacy fallback" + ); + remote.close().await.expect("close guarded RemoteDisk client"); + let _ = shutdown.send(()); + super::timeout(Duration::from_secs(10), server) + .await + .expect("guarded RPC server shuts down") + .expect("guarded RPC server task"); + } + #[cfg(not(windows))] #[tokio::test] async fn replacement_resume_selection_falls_back_to_a_remote_survivor() { @@ -6532,6 +7569,417 @@ mod tests { request } + fn target_quota_rename_request( + disk: &super::DiskStore, + fi: &rustfs_filemeta::FileInfo, + topology: &str, + ) -> Request { + let mut request = target_rename_request(disk, fi); + let body = rustfs_protos::canonical_rename_data_request_body(request.get_ref()).expect("quota rename body"); + sign_quota_test_request(&mut request, &body, topology); + request + } + + #[tokio::test] + #[serial_test::serial] + async fn artifact_quota_rename_authenticates_topology_and_preserves_physical_fence() { + use crate::storage::storage_api::ecstore_disk::{ + DiskAPI, QUOTA_MUTATION_FENCE_METADATA_SUFFIX, RUSTFS_META_BUCKET, ReadOptions, + }; + use rustfs_utils::http::metadata_compat::{insert_str, remove_str}; + use sha2::{Digest, Sha256}; + + const RUSTFS_META_TMP_BUCKET: &str = ".rustfs.sys/tmp"; + let fixture = target_rpc_fixture().await; + fixture.instance.set_endpoints(fixture.env.endpoint_pools.clone()); + let (disk, mut fi, _) = stage_target_rpc(&fixture).await; + let bucket = "artifact-quota-rpc"; + fixture + .env + .ecstore + .clone() + .make_artifact_bucket(bucket, "outputs/") + .await + .expect("managed authority"); + let incarnation = fixture + .env + .ecstore + .bucket_incarnation_id(bucket) + .await + .expect("managed incarnation"); + let service = make_server_for_context(Some(fixture.context.clone())); + let topology = heal_topology_fingerprint(&fixture.env.ecstore.instance_endpoints().expect("instance topology")) + .expect("topology fingerprint"); + let object = "outputs/74656e616e74/11111111-1111-4111-8111-111111111111/22222222-2222-4222-8222-222222222222/quota"; + let source = "artifact-quota-source"; + fi.volume = bucket.to_owned(); + fi.name = object.to_owned(); + disk.write_metadata(RUSTFS_META_TMP_BUCKET, RUSTFS_META_TMP_BUCKET, source, fi.clone()) + .await + .expect("stage real managed publication"); + let staged_path = disk.path().join(RUSTFS_META_TMP_BUCKET).join(source).join("xl.meta"); + let before = tokio::fs::read(&staged_path).await.expect("staged metadata"); + let request_for = |info: &rustfs_filemeta::FileInfo, destination: &str, quota_topology: Option<&str>| { + let mut request = Request::new(RenameDataRequest { + disk: disk.endpoint().to_string(), + src_volume: RUSTFS_META_TMP_BUCKET.to_owned(), + src_path: source.to_owned(), + dst_volume: bucket.to_owned(), + dst_path: destination.to_owned(), + file_info: serde_json::to_string(info).expect("physical FileInfo"), + bucket_incarnation_id: incarnation.as_bytes().to_vec().into(), + ..Default::default() + }); + let body = rustfs_protos::canonical_artifact_rename_data_request_body(request.get_ref()).expect("artifact body"); + if let Some(topology) = quota_topology { + sign_quota_test_request(&mut request, &body, topology); + } else { + set_tonic_canonical_body_digest(&mut request, &body).expect("artifact digest"); + mark_v2_authenticated(&mut request); + } + request + }; + for case in [ + "missing-token", + "nil-token", + "malformed-token", + "conflicting-token", + "missing-header", + "unenveloped-token", + "malformed-topology", + "duplicate-topology", + "wrong-topology", + "tampered-body", + ] { + let mut info = fi.clone(); + if case != "missing-token" { + let token = match case { + "nil-token" => Uuid::nil().to_string(), + "malformed-token" => "not-a-token".to_owned(), + _ => Uuid::new_v4().to_string(), + }; + insert_str(&mut info.metadata, QUOTA_MUTATION_FENCE_METADATA_SUFFIX, token); + } + if case == "conflicting-token" { + info.metadata.insert( + format!("x-minio-internal-{QUOTA_MUTATION_FENCE_METADATA_SUFFIX}"), + Uuid::new_v4().to_string(), + ); + } + let mut request = request_for(&info, object, (case != "unenveloped-token").then_some(topology.as_str())); + let mut expected = tonic::Code::InvalidArgument; + match case { + "missing-header" => { + request.metadata_mut().remove(rustfs_protos::QUOTA_MUTATION_TOPOLOGY_HEADER); + expected = tonic::Code::PermissionDenied; + } + "malformed-topology" => { + request = request_for(&info, object, Some(&"g".repeat(64))); + expected = tonic::Code::PermissionDenied; + } + "duplicate-topology" => { + request + .metadata_mut() + .append(rustfs_protos::QUOTA_MUTATION_TOPOLOGY_HEADER, topology.parse().expect("header")); + expected = tonic::Code::PermissionDenied; + } + "wrong-topology" => { + let wrong = if topology == "a".repeat(64) { + "b".repeat(64) + } else { + "a".repeat(64) + }; + request = request_for(&info, object, Some(&wrong)); + expected = tonic::Code::FailedPrecondition; + } + "tampered-body" => { + request.get_mut().dst_path.push_str("-tampered"); + expected = tonic::Code::PermissionDenied; + } + _ => {} + } + assert_eq!(service.artifact_rename_data(request).await.expect_err(case).code(), expected, "{case}"); + assert_eq!(tokio::fs::read(&staged_path).await.expect("rejected source remains"), before, "{case}"); + assert!( + !disk + .path() + .join(bucket) + .join("outputs/74656e616e74/11111111-1111-4111-8111-111111111111/22222222-2222-4222-8222-222222222222") + .exists(), + "{case}: no destination mutation" + ); + } + // A syntactically valid but unissued token reaches the physical fence and still cannot publish. + let mut unissued = fi.clone(); + insert_str(&mut unissued.metadata, QUOTA_MUTATION_FENCE_METADATA_SUFFIX, Uuid::new_v4().to_string()); + let response = service + .artifact_rename_data(request_for(&unissued, object, Some(&topology))) + .await + .expect("unissued token response") + .into_inner(); + assert!(!response.success, "quota token must be enforced by the actual physical worker"); + assert_eq!(tokio::fs::read(&staged_path).await.expect("unissued source remains"), before); + assert!(!disk.path().join(bucket).join(object).exists()); + + // Both supported protocols must really publish through the same managed receiver. + let plain = "outputs/74656e616e74/11111111-1111-4111-8111-111111111111/22222222-2222-4222-8222-222222222222/plain"; + fi.name = plain.to_owned(); + let response = service + .artifact_rename_data(request_for(&fi, plain, None)) + .await + .expect("plain artifact response") + .into_inner(); + assert!(response.success, "plain artifact publication: {:?}", response.error); + fi.name = object.to_owned(); + let fence = format!( + "tmp/quota-mutation-fences/{}", + rustfs_utils::crypto::hex(Sha256::digest(format!("{bucket}\0{object}"))) + ); + let token = disk + .acquire_snapshot_lease(RUSTFS_META_BUCKET, &fence) + .await + .expect("issue actual target fence"); + insert_str( + &mut fi.metadata, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + Uuid::from_slice(token.as_bytes()).expect("issued token UUID").to_string(), + ); + disk.write_metadata(RUSTFS_META_TMP_BUCKET, RUSTFS_META_TMP_BUCKET, source, fi.clone()) + .await + .expect("restage actual quota publication"); + let response = service + .artifact_rename_data(request_for(&fi, object, Some(&topology))) + .await + .expect("quota artifact response") + .into_inner(); + assert!(response.success, "quota artifact publication: {:?}", response.error); + remove_str(&mut fi.metadata, QUOTA_MUTATION_FENCE_METADATA_SUFFIX); + for object in [plain, object] { + let read = disk + .read_version( + bucket, + bucket, + object, + &fi.version_id.expect("physical version").to_string(), + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("read committed artifact"); + assert_eq!(read.data, fi.data, "{object}"); + assert!(!rustfs_utils::http::metadata_compat::contains_key_str( + &read.metadata, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX + )); + } + disk.release_snapshot_lease(RUSTFS_META_BUCKET, &fence, token) + .await + .expect("release completed fence"); + } + + #[cfg(not(windows))] + #[tokio::test] + async fn quota_fence_handlers_commit_real_data_and_drain_before_revocation() { + use crate::storage::storage_api::{ + LocalPublicationPause, LocalPublicationStage, + ecstore_disk::{DiskAPI, QUOTA_MUTATION_FENCE_METADATA_SUFFIX, RUSTFS_META_BUCKET, ReadOptions}, + }; + use sha2::{Digest, Sha256}; + + let fixture = target_rpc_fixture().await; + // Mirror server startup: the disk fixture builds the store but leaves + // its write-once instance topology for the listener to initialize. + fixture.instance.set_endpoints(fixture.env.endpoint_pools.clone()); + let (disk, mut fi, _) = stage_target_rpc(&fixture).await; + let service = make_server_for_context(Some(fixture.context.clone())); + let topology = + super::heal::heal_topology_fingerprint(&fixture.env.ecstore.instance_endpoints().expect("target topology")) + .expect("target fingerprint"); + let _without_proof = crate::storage::storage_api::ecstore_notification::without_cross_pool_fence_fleet_proof_for_test(); + let path = format!( + "tmp/quota-mutation-fences/{}", + rustfs_utils::crypto::hex(Sha256::digest(b"target-rpc\0destination")) + ); + assert!(crate::storage::storage_api::ecstore_notification::acquire_cross_pool_fence_fleet_proof().is_none()); + let (_, swapped) = quota_two_pool_topologies(); + let wrong_topology = heal_topology_fingerprint(&swapped).expect("reordered pool fingerprint"); + assert_ne!(wrong_topology, topology); + let set = fixture + .env + .ecstore + .all_set_disks() + .into_iter() + .next() + .expect("quota erasure set"); + let disks = set.disks.read().await.iter().filter_map(Clone::clone).collect::>(); + assert_eq!(disks.len(), 4); + let mut successful = 0; + for participant in &disks { + let message = SnapshotLeaseRequest { + disk: participant.endpoint().to_string(), + volume: RUSTFS_META_BUCKET.to_string(), + path: path.clone(), + ttl_ms: 60_000, + }; + let mut wrong = Request::new(message.clone()); + let body = rustfs_protos::canonical_snapshot_lease_request_body(wrong.get_ref()).expect("acquire body"); + sign_quota_test_request(&mut wrong, &body, &wrong_topology); + assert_eq!( + service + .acquire_quota_mutation_fence(wrong) + .await + .expect_err("wrong layout must not issue a token") + .code(), + tonic::Code::FailedPrecondition + ); + let mut correct = Request::new(message); + sign_quota_test_request(&mut correct, &body, &topology); + let response = service + .acquire_quota_mutation_fence(correct) + .await + .expect("same layout acquire") + .into_inner(); + assert!(response.success, "same layout participant must issue a token: {:?}", response.error); + successful += 1; + } + assert_eq!( + successful, 4, + "correct topology admits every available quorum participant without a fleet proof" + ); + let acquire_message = SnapshotLeaseRequest { + disk: disk.endpoint().to_string(), + volume: RUSTFS_META_BUCKET.to_string(), + path: path.clone(), + ttl_ms: 60_000, + }; + let acquire_request = || { + let mut request = Request::new(acquire_message.clone()); + let body = rustfs_protos::canonical_snapshot_lease_request_body(request.get_ref()).expect("acquire body"); + sign_quota_test_request(&mut request, &body, &topology); + request + }; + let acquired = service + .acquire_quota_mutation_fence(acquire_request()) + .await + .expect("production quota acquire handler") + .into_inner(); + assert!(acquired.success, "real quota fence must be acquired: {:?}", acquired.error); + let token = Uuid::from_slice(&acquired.token).expect("quota token UUID"); + assert!(!token.is_nil()); + rustfs_utils::http::metadata_compat::insert_str( + &mut fi.metadata, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + token.to_string(), + ); + let mut wrong_release = Request::new(SnapshotLeaseReleaseRequest { + disk: disk.endpoint().to_string(), + volume: RUSTFS_META_BUCKET.to_string(), + path: path.clone(), + token: vec![0; 16].into(), + }); + let body = rustfs_protos::canonical_snapshot_lease_release_request_body(wrong_release.get_ref()).expect("release body"); + sign_quota_test_request(&mut wrong_release, &body, &wrong_topology); + assert_eq!( + service + .release_quota_mutation_fence(wrong_release) + .await + .expect_err("wrong topology must not revoke") + .code(), + tonic::Code::FailedPrecondition + ); + assert_eq!( + service + .rename_data_with_quota_fence(target_quota_rename_request(&disk, &fi, &wrong_topology)) + .await + .expect_err("wrong topology must not publish") + .code(), + tonic::Code::FailedPrecondition + ); + assert!(disk.path().join("target-rpc/staged/xl.meta").exists()); + assert!(!disk.path().join("target-rpc/destination/xl.meta").exists()); + // The following publication uses the same token: rejecting the wrong + // topology must neither revoke it nor consume the staged object. + let mut pause = + LocalPublicationPause::install(&disk, "target-rpc", "destination/xl.meta", LocalPublicationStage::PreparedRename) + .expect("pause the real quota publication"); + let mut rename = Box::pin(service.rename_data_with_quota_fence(target_quota_rename_request(&disk, &fi, &topology))); + super::timeout(Duration::from_secs(10), async { + tokio::select! { + result = &mut rename => panic!("quota rename finished before the physical pause: {result:?}"), + entered = pause.entered() => entered.expect("quota rename reached physical publication"), + } + }) + .await + .expect("bounded quota publication entry"); + + let mut request = Request::new(SnapshotLeaseReleaseRequest { + disk: disk.endpoint().to_string(), + volume: RUSTFS_META_BUCKET.to_string(), + path, + token: vec![0; 16].into(), + }); + let body = rustfs_protos::canonical_snapshot_lease_release_request_body(request.get_ref()).expect("release body"); + sign_quota_test_request(&mut request, &body, &topology); + let mut revoke = Box::pin(service.release_quota_mutation_fence(request)); + super::timeout(Duration::from_secs(10), async { + loop { + tokio::select! { + biased; + result = &mut revoke => panic!("revocation returned before the physical commit drained: {result:?}"), + result = service.acquire_quota_mutation_fence(acquire_request()) => { + if !result.expect("probe revoked quota fence").into_inner().success { + break; + } + tokio::task::yield_now().await; + } + } + } + }) + .await + .expect("revocation must reject new tokens while waiting for the active commit"); + drop(pause); + let (renamed, revoked) = super::timeout(Duration::from_secs(10), async { tokio::join!(rename, revoke) }) + .await + .expect("quota publication and revocation drain"); + let renamed = renamed.expect("quota rename response").into_inner(); + assert!(renamed.success, "real quota rename must commit: {:?}", renamed.error); + let revoked = revoked.expect("quota revoke response").into_inner(); + assert!(revoked.success, "real quota revoke must finish: {:?}", revoked.error); + let read = disk + .read_version( + "target-rpc", + "target-rpc", + "destination", + &fi.version_id.expect("version").to_string(), + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("read quota RPC committed object"); + assert_eq!(read.data, fi.data); + assert!( + !rustfs_utils::http::metadata_compat::contains_key_str(&read.metadata, QUOTA_MUTATION_FENCE_METADATA_SUFFIX), + "the transient quota token must not persist in object metadata" + ); + disk.write_metadata("target-rpc", "target-rpc", "staged", fi.clone()) + .await + .expect("restage valid data for a revoked-token replay"); + let replay = service + .rename_data_with_quota_fence(target_quota_rename_request(&disk, &fi, &topology)) + .await + .expect("revoked token replay response") + .into_inner(); + assert!(!replay.success, "revoked quota token must not authorize another real rename"); + assert!( + disk.path().join("target-rpc/staged/xl.meta").exists(), + "rejected replay must retain its staged input" + ); + } + #[tokio::test] async fn target_slot_rejects_mismatched_and_repeated_install_before_global_publication() { let fixture = target_rpc_fixture().await; @@ -6772,7 +8220,7 @@ mod tests { }; let rollback_dir = Uuid::new_v4(); remote - .delete_version( + .delete_version_recoverable( bucket, object, marker.clone(), @@ -7051,6 +8499,7 @@ mod tests { platform: "test".to_string(), }]); let instance = new_instance_ctx(); + instance.set_endpoints(pools.clone()); init_local_disks_with_instance_ctx(&instance, pools.clone()) .await .expect("register this instance's real disks"); @@ -7194,8 +8643,13 @@ mod tests { pool_identity, "A must preserve the initialized cluster identity and epoch" ); - let service = make_server_for_context(Some(context(&store_a).await)); - assert!(Arc::ptr_eq(&service.resolve_object_store().expect("captured store"), &store_a)); + let slot_a = super::ServerContextSlot::new(); + slot_a.try_install(context(&store_a).await).expect("install captured instance A"); + let service = super::make_server_for_slot(slot_a); + let super::LocalMutationTarget::Ready(captured_store) = service.local_mutation_target() else { + panic!("slot A must capture a ready mutation target"); + }; + assert!(Arc::ptr_eq(&captured_store, &store_a)); assert!(Arc::ptr_eq( &crate::runtime_sources::current_object_store_handle().expect("global store"), &store_b @@ -7212,6 +8666,64 @@ mod tests { .expect("global UUID lookup must resolve B before the request"); assert!(Arc::ptr_eq(&global_disk, &disk_b)); + let topology_a = heal_topology_fingerprint(&store_a.instance_endpoints().expect("A layout")).expect("A identity"); + let topology_b = heal_topology_fingerprint(&store_b.instance_endpoints().expect("B layout")).expect("B identity"); + assert_ne!(topology_a, topology_b, "the captured and ambient layouts use different physical paths"); + let target_a = super::LocalMutationTarget::Ready(store_a.clone()); + let target_b = super::LocalMutationTarget::Ready(store_b.clone()); + for _ in 0..4 { + assert!(Arc::ptr_eq(service.quota_store_for_topology(&target_a, &topology_a).expect("cached A"), &store_a)); + } + assert_eq!(service.quota_topology_computations.load(std::sync::atomic::Ordering::Relaxed), 1, + "repeated fanout requests for one captured instance must hash its topology only once"); + assert!(Arc::ptr_eq(service.quota_store_for_topology(&target_b, &topology_b).expect("replacement B"), &store_b)); + assert_eq!(service.quota_topology_computations.load(std::sync::atomic::Ordering::Relaxed), 2, + "a different Arc with the same deployment identity must replace the cache entry"); + assert!(Arc::ptr_eq(service.quota_store_for_topology(&target_a, &topology_a).expect("return to A"), &store_a)); + assert_eq!(service.quota_topology_computations.load(std::sync::atomic::Ordering::Relaxed), 3, + "the bounded single-entry cache must recompute after an instance switch"); + let fence_path = format!("tmp/quota-mutation-fences/{}", "d".repeat(64)); + let acquire = |topology: &str| { + let mut request = Request::new(SnapshotLeaseRequest { + disk: disk_id.to_string(), + volume: RUSTFS_META_BUCKET.to_string(), + path: fence_path.clone(), + ttl_ms: 60_000, + }); + let body = rustfs_protos::canonical_snapshot_lease_request_body(request.get_ref()).expect("acquire body"); + sign_quota_test_request(&mut request, &body, topology); + request + }; + assert_eq!( + service.acquire_quota_mutation_fence(acquire(&topology_b)).await.expect_err("ambient layout must not authorize A").code(), + tonic::Code::FailedPrecondition + ); + let response = service.acquire_quota_mutation_fence(acquire(&topology_a)).await.expect("captured acquire").into_inner(); + assert!(response.success, "A must acquire its own token: {:?}", response.error); + let token_a = crate::storage::storage_api::SnapshotLeaseToken::from_slice(&response.token).expect("A token"); + assert!(disk_b.renew_snapshot_lease(RUSTFS_META_BUCKET, &fence_path, token_a).await.is_err()); + let token_a = disk_a.renew_snapshot_lease(RUSTFS_META_BUCKET, &fence_path, token_a).await.expect("token belongs to A"); + let token_b = disk_b.acquire_snapshot_lease(RUSTFS_META_BUCKET, &fence_path).await.expect("independent B token"); + let release = |topology: &str| { + let mut request = Request::new(SnapshotLeaseReleaseRequest { + disk: disk_id.to_string(), + volume: RUSTFS_META_BUCKET.to_string(), + path: fence_path.clone(), + token: vec![0; 16].into(), + }); + let body = rustfs_protos::canonical_snapshot_lease_release_request_body(request.get_ref()).expect("release body"); + sign_quota_test_request(&mut request, &body, topology); + request + }; + assert_eq!( + service.release_quota_mutation_fence(release(&topology_b)).await.expect_err("ambient layout must not revoke A").code(), + tonic::Code::FailedPrecondition + ); + let token_a = disk_a.renew_snapshot_lease(RUSTFS_META_BUCKET, &fence_path, token_a).await.expect("wrong layout did not revoke A"); + assert!(service.release_quota_mutation_fence(release(&topology_a)).await.expect("captured release").into_inner().success); + assert!(disk_a.renew_snapshot_lease(RUSTFS_META_BUCKET, &fence_path, token_a).await.is_err()); + assert!(disk_b.renew_snapshot_lease(RUSTFS_META_BUCKET, &fence_path, token_b).await.is_ok(), "A release must leave ambient B untouched"); + let volume = VOLUME; let object = "destination"; let staging = "staged"; diff --git a/rustfs/src/storage/rpc/node_service/artifact.rs b/rustfs/src/storage/rpc/node_service/artifact.rs index 751176c5a..2e3ccb4bf 100644 --- a/rustfs/src/storage/rpc/node_service/artifact.rs +++ b/rustfs/src/storage/rpc/node_service/artifact.rs @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::disk::{decode_rename_data_request_file_info, encode_rename_data_response_payloads}; +use super::disk::{decode_rename_data_request_file_info, encode_rename_data_response_payloads, verify_quota_mutation_digest}; use super::{LocalMutationTarget, NodeService}; use crate::storage::storage_api::rpc_consumer::node_service::DiskError; use crate::storage::storage_api::{sign_tonic_rpc_response_proof, verify_tonic_canonical_body_digest}; @@ -45,17 +45,27 @@ impl NodeService { &self, request: Request, ) -> Result, Status> { - let canonical = rustfs_protos::canonical_artifact_rename_data_request_body(request.get_ref()) - .map_err(|_| Status::invalid_argument("artifact rename request is too large to authenticate"))?; - verify_tonic_canonical_body_digest(&request, &canonical) - .map_err(|_| Status::permission_denied("artifact rename requires its body-bound authentication"))?; + let canonical = rustfs_protos::canonical_artifact_rename_data_request_body(request.get_ref()); + let quota_topology = if request.metadata().contains_key(rustfs_protos::QUOTA_MUTATION_TOPOLOGY_HEADER) { + Some(verify_quota_mutation_digest(&request, canonical)?) + } else { + let canonical = + canonical.map_err(|_| Status::invalid_argument("artifact rename request is too large to authenticate"))?; + verify_tonic_canonical_body_digest(&request, &canonical) + .map_err(|_| Status::permission_denied("artifact rename requires its body-bound authentication"))?; + None + }; let incarnation = required_uuid(&request.get_ref().bucket_incarnation_id, "bucket incarnation")?; if !request.get_ref().scanner_publication_lease_token.is_empty() { return Err(Status::invalid_argument("artifact rename cannot carry a scanner publication lease")); } // Capture once: a bootstrap/unbound request cannot switch to a ready // instance after any await, and no global store fallback is permitted. - let LocalMutationTarget::Ready(store) = self.local_mutation_target() else { + let target = self.local_mutation_target(); + if let Some(topology) = "a_topology { + self.quota_store_for_topology(&target, topology)?; + } + let LocalMutationTarget::Ready(store) = target else { return Err(Status::unavailable("artifact rename requires a ready storage instance")); }; let request = request.into_inner(); @@ -63,6 +73,20 @@ impl NodeService { Ok(decoded) => decoded, Err(error) => return Ok(rename_failure(error)), }; + let quota_suffix = crate::storage::storage_api::ecstore_disk::QUOTA_MUTATION_FENCE_METADATA_SUFFIX; + let carries_quota_token = rustfs_utils::http::metadata_compat::contains_key_str(&decoded.value.metadata, quota_suffix); + if quota_topology.is_some() { + let token = rustfs_utils::http::metadata_compat::get_consistent_str(&decoded.value.metadata, quota_suffix) + .and_then(|value| Uuid::parse_str(value).ok()) + .filter(|token| !token.is_nil()); + if token.is_none() { + return Err(Status::invalid_argument( + "quota artifact rename requires a consistent non-nil mutation fence token", + )); + } + } else if carries_quota_token { + return Err(Status::invalid_argument("quota artifact rename requires topology-bound authentication")); + } let result = store .rename_local_artifact_data( &request.disk, diff --git a/rustfs/src/storage/rpc/node_service/bucket.rs b/rustfs/src/storage/rpc/node_service/bucket.rs index e7f8f82ae..40317d898 100644 --- a/rustfs/src/storage/rpc/node_service/bucket.rs +++ b/rustfs/src/storage/rpc/node_service/bucket.rs @@ -17,7 +17,8 @@ use crate::storage::storage_api::rpc_consumer::node_service::contract::bucket::{ BucketOptions, DeleteBucketOptions, MakeBucketOptions, }; use crate::storage::storage_api::rpc_consumer::node_service::{ - DiskError, StoragePeerS3ClientExt as _, decode_heal_bucket_rpc_options, reload_bucket_metadata, remove_bucket_metadata, + DiskError, StoragePeerS3ClientExt as _, decode_heal_bucket_rpc_options, reload_bucket_metadata, + remove_bucket_metadata_if_absent, }; use rustfs_protos::proto_gen::node_service::*; use tonic::{Request, Response, Status}; @@ -38,9 +39,16 @@ impl NodeService { })); } - // Drop the bucket's cached metadata on this peer so it stops serving - // stale configuration after the bucket has been deleted. - match remove_bucket_metadata(&bucket).await { + // Drop the bucket's cached metadata only if the namespace is still + // absent. A delayed delete notification must not erase metadata for a + // same-name bucket that has already been recreated. + let Some(store) = self.resolve_object_store() else { + return Ok(Response::new(DeleteBucketMetadataResponse { + success: false, + error_info: Some("errServerNotInitialized".to_string()), + })); + }; + match remove_bucket_metadata_if_absent(store, &bucket).await { Ok(_) => Ok(Response::new(DeleteBucketMetadataResponse { success: true, error_info: None, diff --git a/rustfs/src/storage/rpc/node_service/disk.rs b/rustfs/src/storage/rpc/node_service/disk.rs index f0a6c8a94..dba488b7b 100644 --- a/rustfs/src/storage/rpc/node_service/disk.rs +++ b/rustfs/src/storage/rpc/node_service/disk.rs @@ -13,15 +13,19 @@ // limitations under the License. use super::{LocalMutationTarget, NodeService}; +use crate::storage::storage_api::ecstore_disk::{ + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, RUSTFS_META_BUCKET, is_quota_mutation_fence_path, +}; use crate::storage::storage_api::rpc_consumer::node_service::{ BatchReadVersionReq, BatchReadVersionResp, ConditionalFileUpdate, DeleteOptions, DiskError, DiskInfoOptions, - FileInfoVersions, ReadMultipleReq, ReadMultipleResp, ReadOptions, StorageDiskRpcExt as _, UpdateMetadataOpts, - validate_batch_read_version_item_count, + FileInfoVersions, GuardedDeleteVersionsPayload, ReadMultipleReq, ReadMultipleResp, ReadOptions, StorageDiskRpcExt as _, + UpdateMetadataOpts, validate_batch_read_version_item_count, }; use crate::storage::storage_api::runtime_sources_consumer::runtime_sources; use crate::storage::storage_api::{ - LegacyFileWrite, LegacyMetadataWrite, LegacyPartMutation, LegacyPartOperation, LegacyPathDelete, LegacyPathDeleteOperation, - LegacyRenameData, LegacyRenameFile, PartTransactionAction, RenameDataResp, SnapshotLeaseToken, + DELETE_RECOVERY_PROTOCOL_VERSION, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, LegacyFileWrite, LegacyMetadataWrite, + LegacyPartMutation, LegacyPartOperation, LegacyPathDelete, LegacyPathDeleteOperation, LegacyRenameData, LegacyRenameFile, + PartTransactionAction, RenameDataResp, SnapshotLeaseToken, verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, }; use bytes::Bytes; @@ -69,6 +73,32 @@ fn startup_cas_rename_observation( } impl LocalMutationTarget { + async fn recover_local_delete_transaction( + &self, + disk_ref: &str, + volume: &str, + operation: DeleteRecoveryRpcOperation, + ) -> Result { + match self { + Self::Ready(store) => store.recover_local_delete_transaction(disk_ref, volume, operation).await, + Self::Bootstrap(target) => target.recover_local_delete_transaction(disk_ref, volume, operation).await, + Self::Unbound => Err(DiskError::other("target disk instance is unavailable")), + } + } + + async fn guarded_local_delete_versions( + &self, + disk_ref: &str, + volume: &str, + payload: GuardedDeleteVersionsPayload, + ) -> Result>, DiskError> { + match self { + Self::Ready(store) => store.guarded_local_delete_versions(disk_ref, volume, payload).await, + Self::Bootstrap(target) => target.guarded_local_delete_versions(disk_ref, volume, payload).await, + Self::Unbound => Err(DiskError::other("target disk instance is unavailable")), + } + } + pub(crate) async fn find_mutation_disk(&self, disk_ref: &str) -> Option { match self { Self::Ready(store) => store.local_mutation_disk(disk_ref).await.ok(), @@ -267,6 +297,16 @@ const MSGPACK_ENCODE_CAPACITY_HINT: usize = 512; const FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT: usize = 1024; const SNAPSHOT_LEASE_PROTOCOL_VERSION: u32 = 1; +fn rejects_legacy_delete_transaction(volume: &str, opts: &DeleteOptions) -> bool { + !(volume == ".rustfs.sys" + || volume.starts_with(".rustfs.sys/") + || volume == ".minio.sys" + || volume.starts_with(".minio.sys/")) + && opts.old_data_dir.is_some() + && !opts.recoverable_delete + && !opts.undo_write +} + fn snapshot_lease_response(result: Result) -> Response { match result { Ok(token) => Response::new(SnapshotLeaseResponse { @@ -422,6 +462,33 @@ fn verify_disk_mutation_digest( .map_err(|err| Status::permission_denied(format!("{op} authentication failed: {err}"))) } +pub(super) fn verify_quota_mutation_digest( + request: &Request, + canonical_body: std::result::Result, std::num::TryFromIntError>, +) -> Result { + let values = request.metadata().get_all(rustfs_protos::QUOTA_MUTATION_TOPOLOGY_HEADER); + let mut values = values.iter(); + let topology = values + .next() + .and_then(|value| value.to_str().ok()) + .filter(|value| { + value.len() == 64 + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + }) + .ok_or_else(|| Status::permission_denied("quota mutation requires a valid topology header"))?; + if values.next().is_some() { + return Err(Status::permission_denied("quota mutation requires exactly one topology header")); + } + let body = canonical_body.map_err(|_| Status::invalid_argument("quota mutation request length cannot be represented"))?; + let body = rustfs_protos::canonical_quota_mutation_request_body(topology, &body) + .map_err(|_| Status::invalid_argument("quota mutation envelope length cannot be represented"))?; + verify_tonic_canonical_body_digest(request, &body) + .map_err(|_| Status::permission_denied("quota mutation requires topology-bound v2 authentication"))?; + Ok(topology.to_owned()) +} + /// JSON compatibility string for a dual-encoded response field. Returns an empty string when /// msgpack-only mode and its explicit fleet confirmation guard are both enabled, or when /// request-local `_bin` payloads prove the caller can consume the paired response `_bin` field, @@ -527,19 +594,71 @@ pub(super) fn encode_rename_data_response_payloads( } impl NodeService { + pub(super) fn quota_store_for_topology<'a>( + &self, + target: &'a LocalMutationTarget, + coordinator_topology: &str, + ) -> Result<&'a Arc, Status> { + let LocalMutationTarget::Ready(store) = target else { + return Err(Status::failed_precondition("quota mutation requires a ready storage instance")); + }; + let mut cache = self + .quota_topology_cache + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let identity = Arc::downgrade(store); + if !cache.as_ref().is_some_and(|entry| entry.store.ptr_eq(&identity)) { + // Instance endpoints are immutable once initialized. Keep one weak + // identity so a replacement instance cannot inherit this layout, + // even when its deployment or disk UUIDs are unchanged. + let endpoints = store + .instance_endpoints() + .ok_or_else(|| Status::failed_precondition("quota mutation target topology is unavailable"))?; + let fingerprint = super::heal::heal_topology_fingerprint(&endpoints) + .map_err(|_| Status::failed_precondition("quota mutation target topology is invalid"))?; + #[cfg(test)] + self.quota_topology_computations + .fetch_add(1, std::sync::atomic::Ordering::Relaxed); + *cache = Some(super::QuotaTopologyCacheEntry { + store: identity, + fingerprint, + }); + } + if cache.as_ref().is_none_or(|entry| entry.fingerprint != coordinator_topology) { + return Err(Status::failed_precondition( + "quota mutation coordinator topology does not match the target", + )); + } + Ok(store) + } + pub(super) async fn handle_acquire_snapshot_lease( &self, request: Request, + quota_fence: bool, ) -> Result, Status> { - verify_disk_mutation_digest( - &request, - rustfs_protos::canonical_snapshot_lease_request_body(request.get_ref()), - "acquire_snapshot_lease", - )?; + let body = rustfs_protos::canonical_snapshot_lease_request_body(request.get_ref()); + let quota_store = if quota_fence { + let topology = verify_quota_mutation_digest(&request, body)?; + if request.get_ref().volume != RUSTFS_META_BUCKET || !is_quota_mutation_fence_path(&request.get_ref().path) { + return Err(Status::invalid_argument("quota fence requires its internal mutation fence path")); + } + Some(Arc::clone(self.quota_store_for_topology(&self.local_mutation_target(), &topology)?)) + } else { + verify_disk_mutation_digest(&request, body, "acquire_snapshot_lease")?; + None + }; let request = request.into_inner(); - let result = match self.find_disk(&request.disk).await { - Some(disk) => disk.acquire_snapshot_lease(&request.volume, &request.path).await, - None => Err(DiskError::other("cannot find disk")), + let disk = match quota_store { + Some(store) => store.quota_mutation_fence_disk(&request.disk).await, + None => self + .find_disk(&request.disk) + .await + .ok_or_else(|| DiskError::other("cannot find disk")), + }; + let result = match disk { + Ok(disk) => disk.acquire_snapshot_lease(&request.volume, &request.path).await, + Err(err) => Err(err), }; Ok(snapshot_lease_response(result)) } @@ -566,25 +685,37 @@ impl NodeService { pub(super) async fn handle_release_snapshot_lease( &self, request: Request, + quota_fence: bool, ) -> Result, Status> { - verify_disk_mutation_digest( - &request, - rustfs_protos::canonical_snapshot_lease_release_request_body(request.get_ref()), - "release_snapshot_lease", - )?; + let body = rustfs_protos::canonical_snapshot_lease_release_request_body(request.get_ref()); + let quota_store = if quota_fence { + let topology = verify_quota_mutation_digest(&request, body)?; + if request.get_ref().volume != RUSTFS_META_BUCKET || !is_quota_mutation_fence_path(&request.get_ref().path) { + return Err(Status::invalid_argument("quota fence requires its internal mutation fence path")); + } + Some(Arc::clone(self.quota_store_for_topology(&self.local_mutation_target(), &topology)?)) + } else { + verify_disk_mutation_digest(&request, body, "release_snapshot_lease")?; + None + }; let request = request.into_inner(); let token = if request.token.as_ref() == SnapshotLeaseToken::revoke_all().as_bytes() { SnapshotLeaseToken::revoke_all() } else { SnapshotLeaseToken::from_slice(&request.token).map_err(|_| Status::invalid_argument("invalid lease token"))? }; - let Some(disk) = self.find_disk(&request.disk).await else { - return Ok(Response::new(SnapshotLeaseMutationResponse { - success: false, - error: Some(DiskError::other("cannot find disk").into()), - })); + let disk = match quota_store { + Some(store) => store.quota_mutation_fence_disk(&request.disk).await, + None => self + .find_disk(&request.disk) + .await + .ok_or_else(|| DiskError::other("cannot find disk")), }; - match disk.release_snapshot_lease(&request.volume, &request.path, token).await { + let result = match disk { + Ok(disk) => disk.release_snapshot_lease(&request.volume, &request.path, token).await, + Err(err) => Err(err), + }; + match result { Ok(()) => Ok(Response::new(SnapshotLeaseMutationResponse { success: true, error: None, @@ -830,6 +961,71 @@ impl NodeService { } } + pub(super) async fn handle_guarded_delete_versions( + &self, + request: Request, + ) -> Result, Status> { + if request.get_ref().payload.len() > rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES { + return Err(Status::resource_exhausted("guarded delete payload exceeds limit")); + } + let body = rustfs_protos::canonical_guarded_delete_versions_request_body(request.get_ref()) + .map_err(|_| Status::invalid_argument("invalid guarded delete body length"))?; + verify_tonic_canonical_body_digest(&request, &body) + .map_err(|_| Status::permission_denied("guarded delete requires body-bound v2 authentication"))?; + let request = request.into_inner(); + let target = self.local_mutation_target(); + let disk = target + .find_mutation_disk(&request.disk) + .await + .ok_or_else(|| Status::not_found("cannot find disk"))?; + if request.probe_only { + if !request.payload.is_empty() || !request.volume.is_empty() { + return Err(Status::invalid_argument("guarded capability probe must have no mutation payload")); + } + if let Err(error) = disk.supports_guarded_delete_versions().await { + return Ok(Response::new(DeleteVersionsResponse { + success: false, + errors: Vec::new(), + error: Some(error.into()), + item_errors: Vec::new(), + })); + } + return Ok(Response::new(DeleteVersionsResponse { + success: true, + errors: Vec::new(), + error: None, + item_errors: Vec::new(), + })); + } + let payload: GuardedDeleteVersionsPayload = rustfs_protos::decode_delete_recovery_payload(&request.payload) + .map_err(|_| Status::invalid_argument("invalid guarded delete payload"))?; + payload + .validate() + .map_err(|_| Status::invalid_argument("invalid guarded delete options or batch"))?; + target.reject_legacy_artifact_mutation(&request.volume).await?; + let result = target + .guarded_local_delete_versions(&request.disk, &request.volume, payload) + .await; + let result = match result { + Ok(result) => result, + Err(error) => { + return Ok(Response::new(DeleteVersionsResponse { + success: false, + errors: Vec::new(), + error: Some(error.into()), + item_errors: Vec::new(), + })); + } + }; + let (errors, item_errors) = encode_delete_versions_errors(result); + Ok(Response::new(DeleteVersionsResponse { + success: true, + errors, + error: None, + item_errors, + })) + } + pub(super) async fn handle_delete_versions( &self, request: Request, @@ -872,6 +1068,14 @@ impl NodeService { } }; + if rejects_legacy_delete_transaction(&request.volume, &opts) { + return Ok(Response::new(DeleteVersionsResponse { + success: false, + errors: Vec::new(), + error: Some(DiskError::UnsupportedDisk.into()), + item_errors: Vec::new(), + })); + } let result = if versions.is_empty() { disk.delete_versions(&request.volume, versions, opts).await } else { @@ -959,6 +1163,13 @@ impl NodeService { })); } }; + if rejects_legacy_delete_transaction(&request.volume, &opts) { + return Ok(Response::new(DeleteVersionResponse { + success: false, + raw_file_info: String::new(), + error: Some(DiskError::UnsupportedDisk.into()), + })); + } if require_marker_condition && opts.expected_delete_marker.is_none() { return Err(Status::invalid_argument("retired marker deletion requires a marker precondition")); } @@ -1604,7 +1815,10 @@ impl NodeService { pub(super) async fn handle_rename_data( &self, request: Request, + quota_fence: bool, ) -> Result, Status> { + let service_return_started = rustfs_io_metrics::put_stage_timer(); + let service_entry_started = rustfs_io_metrics::put_stage_timer(); if !request.get_ref().scanner_publication_lease_token.is_empty() || !request.get_ref().bucket_incarnation_id.is_empty() { let has_body_digest = request .metadata() @@ -1612,14 +1826,40 @@ impl NodeService { .and_then(|value| value.to_str().ok()) .is_some_and(|value| value != "UNSIGNED-PAYLOAD"); if !has_body_digest { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_ENTRY, + service_entry_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + service_return_started, + ); return Err(Status::permission_denied("scanner publication lease rename requires a body-bound digest")); } } - verify_disk_mutation_digest( - &request, - rustfs_protos::canonical_rename_data_request_body(request.get_ref()), - "rename_data", - )?; + let body = rustfs_protos::canonical_rename_data_request_body(request.get_ref()); + let mut quota_topology = None; + let authenticated = if quota_fence { + verify_quota_mutation_digest(&request, body).map(|topology| quota_topology = Some(topology)) + } else { + verify_disk_mutation_digest(&request, body, "rename_data") + }; + if let Err(err) = authenticated { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_ENTRY, + service_entry_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + service_return_started, + ); + return Err(err); + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_ENTRY, + service_entry_started, + ); + let before_handler_started = rustfs_io_metrics::put_stage_timer(); let request = request.into_inner(); let target = self.local_mutation_target(); let bucket_incarnation = if request.bucket_incarnation_id.is_empty() { @@ -1630,6 +1870,14 @@ impl NodeService { .filter(|id| !id.is_nil()) .ok_or_else(|| Status::invalid_argument("bucket incarnation must be a non-nil UUID"))?; if !request.scanner_publication_lease_token.is_empty() { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_BEFORE_HANDLER, + before_handler_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + service_return_started, + ); return Err(Status::invalid_argument( "heal incarnation and scanner publication lease cannot be combined", )); @@ -1641,12 +1889,26 @@ impl NodeService { let decoded_file_info = match decode_rename_data_request_file_info(&request.file_info_bin, &request.file_info) { Ok(file_info) => file_info, Err(err) => { - return Ok(Response::new(RenameDataResponse { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_BEFORE_HANDLER, + before_handler_started, + ); + let response_build_started = rustfs_io_metrics::put_stage_timer(); + let response = Response::new(RenameDataResponse { success: false, rename_data_resp: String::new(), rename_data_resp_bin: Vec::new().into(), error: Some(DiskError::other(format!("decode FileInfo failed: {err}")).into()), - })); + }); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_BUILD, + response_build_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + service_return_started, + ); + return Ok(response); } }; let scanner_publication_lease_token = if request.scanner_publication_lease_token.is_empty() { @@ -1655,10 +1917,34 @@ impl NodeService { let token = Uuid::from_slice(&request.scanner_publication_lease_token) .map_err(|_| Status::invalid_argument("scanner publication lease token must be a UUID"))?; if token.is_nil() { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_BEFORE_HANDLER, + before_handler_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + service_return_started, + ); return Err(Status::invalid_argument("scanner publication lease token must not be nil")); } Some(token) }; + if quota_fence { + let token = rustfs_utils::http::metadata_compat::get_consistent_str( + &decoded_file_info.value.metadata, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + ) + .and_then(|value| Uuid::parse_str(value).ok()) + .filter(|token| !token.is_nil()); + if token.is_none() { + return Err(Status::invalid_argument( + "quota rename requires a consistent non-nil mutation fence token", + )); + } + } + if let Some(topology) = quota_topology { + self.quota_store_for_topology(&target, &topology)?; + } let request_decoded_from_msgpack = decoded_file_info.from_msgpack; // Bind ordinary mutations to the captured instance and both live volumes // before consulting bucket authority; commit consumes those exact locks. @@ -1689,6 +1975,11 @@ impl NodeService { target.reject_legacy_artifact_mutation(&request.dst_volume).await?; #[cfg(feature = "e2e-test-hooks")] let observation = startup_cas_rename_observation(&target, &request, &decoded_file_info.value); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_BEFORE_HANDLER, + before_handler_started, + ); + let handler_await_started = rustfs_io_metrics::put_stage_timer(); let result = if let Some(rename) = prepared { rename.commit().await } else { @@ -1703,6 +1994,11 @@ impl NodeService { ) .await }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_HANDLER_AWAIT, + handler_await_started, + ); + let after_handler_started = rustfs_io_metrics::put_stage_timer(); #[cfg(feature = "e2e-test-hooks")] if let Some(mut observation) = observation { observation["ok"] = serde_json::json!(result.is_ok()); @@ -1710,28 +2006,47 @@ impl NodeService { let line = format!("RUSTFS_E2E_STARTUP_CAS {observation}\n"); let _ = std::io::Write::write_all(&mut std::io::stderr().lock(), line.as_bytes()); } - match result { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_AFTER_HANDLER, + after_handler_started, + ); + let response_build_started = rustfs_io_metrics::put_stage_timer(); + let response_encode_started = rustfs_io_metrics::put_stage_timer(); + let response = match result { Ok(rename_data_resp) => match encode_rename_data_response_payloads(&rename_data_resp, request_decoded_from_msgpack) { - Ok((rename_data_resp, rename_data_resp_bin)) => Ok(Response::new(RenameDataResponse { + Ok((rename_data_resp, rename_data_resp_bin)) => Response::new(RenameDataResponse { success: true, rename_data_resp, rename_data_resp_bin: rename_data_resp_bin.into(), error: None, - })), - Err(err) => Ok(Response::new(RenameDataResponse { + }), + Err(err) => Response::new(RenameDataResponse { success: false, rename_data_resp: String::new(), rename_data_resp_bin: Vec::new().into(), error: Some(err.into()), - })), + }), }, - Err(err) => Ok(Response::new(RenameDataResponse { + Err(err) => Response::new(RenameDataResponse { success: false, rename_data_resp: String::new(), rename_data_resp_bin: Vec::new().into(), error: Some(err.into()), - })), - } + }), + }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_ENCODE, + response_encode_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_BUILD, + response_build_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + service_return_started, + ); + Ok(response) } pub(super) async fn handle_list_dir(&self, request: Request) -> Result, Status> { @@ -1903,6 +2218,106 @@ impl NodeService { } } + pub(super) async fn handle_recover_delete_transaction( + &self, + request: Request, + ) -> Result, Status> { + if request.get_ref().operation.len() > rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES { + return Err(Status::resource_exhausted("delete recovery operation exceeds payload limit")); + } + let body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()) + .map_err(|_| Status::invalid_argument("delete recovery request length cannot be represented"))?; + // This additive protocol has no digestless rolling-upgrade exception. + verify_tonic_canonical_body_digest(&request, &body) + .map_err(|_| Status::permission_denied("delete recovery requires body-bound v2 authentication"))?; + let request = request.into_inner(); + if request.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION { + return Err(Status::unimplemented("incompatible delete recovery protocol")); + } + let operation: DeleteRecoveryRpcOperation = rustfs_protos::decode_delete_recovery_payload(&request.operation) + .map_err(|_| Status::invalid_argument("invalid delete recovery operation"))?; + if request.operation.len() > operation.max_wire_payload_bytes() { + return Err(Status::resource_exhausted("delete recovery operation exceeds payload limit")); + } + match &operation { + DeleteRecoveryRpcOperation::ApplyVersions { versions, .. } + if versions.len() > 1000 + || versions + .iter() + .try_fold(0_usize, |count, versions| count.checked_add(versions.versions.len())) + .is_none_or(|count| count > 1000) => + { + return Err(Status::invalid_argument("recoverable batch delete exceeds 1000 object/version requests")); + } + DeleteRecoveryRpcOperation::Discover { limit, .. } if *limit == 0 || *limit > 256 => { + return Err(Status::invalid_argument("delete recovery page limit must be between 1 and 256")); + } + DeleteRecoveryRpcOperation::Inspect { transaction, .. } | DeleteRecoveryRpcOperation::Settle { transaction, .. } + if transaction.is_nil() => + { + return Err(Status::invalid_argument("delete transaction must be a non-nil UUID")); + } + DeleteRecoveryRpcOperation::ApplyVersion { opts, .. } | DeleteRecoveryRpcOperation::ApplyVersions { opts, .. } + if opts.old_data_dir.is_none_or(|transaction| transaction.is_nil()) + || opts.undo_delete + || opts.undo_write + || opts.recursive + || opts.immediate + || opts.orphan_data_dir + || opts.expected_delete_marker.is_some() + || opts + .delete_recovery_bucket_incarnation + .is_some_and(|incarnation| incarnation.is_nil()) => + { + return Err(Status::invalid_argument( + "recoverable delete requires a non-nil transaction and valid apply options", + )); + } + DeleteRecoveryRpcOperation::ApplyVersion { object, .. } if object.starts_with('/') => { + return Err(Status::invalid_argument("recoverable delete requires a relative object key")); + } + DeleteRecoveryRpcOperation::ApplyVersions { versions, .. } + if versions.iter().any(|version| version.name.starts_with('/')) => + { + return Err(Status::invalid_argument("recoverable delete requires relative object keys")); + } + _ => {} + } + let target = self.local_mutation_target(); + if matches!( + operation, + DeleteRecoveryRpcOperation::ApplyVersion { .. } + | DeleteRecoveryRpcOperation::ApplyVersions { .. } + | DeleteRecoveryRpcOperation::Settle { .. } + ) { + target.reject_legacy_artifact_mutation(&request.volume).await?; + } + let result = target + .recover_local_delete_transaction(&request.disk, &request.volume, operation) + .await; + match result { + Ok(reply) => { + let result = encode_msgpack_named(&reply, "delete recovery reply") + .map_err(|_| Status::internal("cannot encode delete recovery reply"))?; + if result.len() > rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES { + return Err(Status::resource_exhausted("delete recovery reply exceeds payload limit")); + } + Ok(Response::new(RecoverDeleteTransactionResponse { + success: true, + protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION, + result: result.into(), + error: None, + })) + } + Err(error) => Ok(Response::new(RecoverDeleteTransactionResponse { + success: false, + protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION, + result: Bytes::new(), + error: Some(error.into()), + })), + } + } + pub(super) async fn handle_settle_part_transaction( &self, request: Request, @@ -2356,9 +2771,10 @@ impl NodeService { #[cfg(test)] mod tests { use super::{ - compat_response_json, decode_msgpack_or_json, decode_rename_data_request_file_info, + DeleteOptions, compat_response_json, decode_msgpack_or_json, decode_rename_data_request_file_info, encode_batch_read_version_response_payloads, encode_delete_versions_errors, encode_file_info_msgpack, encode_msgpack, encode_msgpack_named, encode_read_multiple_response_payloads, encode_rename_data_response_payloads, + rejects_legacy_delete_transaction, }; use crate::storage::DiskError; use crate::storage::rpc::node_service::make_server; @@ -2371,6 +2787,7 @@ mod tests { use serde::{Deserialize, Serialize}; use serial_test::serial; use tonic::Request; + use uuid::Uuid; #[derive(Debug, PartialEq, Eq, Serialize, Deserialize)] struct SamplePayload { @@ -2378,6 +2795,26 @@ mod tests { count: u32, } + #[test] + fn legacy_delete_transaction_policy_preserves_rollback_and_internal_maintenance() { + let mut opts = DeleteOptions { + old_data_dir: Some(Uuid::new_v4()), + ..Default::default() + }; + assert!(rejects_legacy_delete_transaction("bucket", &opts)); + assert!(!rejects_legacy_delete_transaction(".rustfs.sys", &opts)); + assert!(!rejects_legacy_delete_transaction(".rustfs.sys/tmp", &opts)); + assert!(!rejects_legacy_delete_transaction(".minio.sys", &opts)); + opts.undo_write = true; + assert!(!rejects_legacy_delete_transaction("bucket", &opts)); + opts.undo_write = false; + opts.recoverable_delete = true; + assert!(!rejects_legacy_delete_transaction("bucket", &opts)); + opts.recoverable_delete = false; + opts.old_data_dir = None; + assert!(!rejects_legacy_delete_transaction("bucket", &opts)); + } + #[test] fn response_compat_send_sites_keep_manifest_json_encoders() { // Rolling-upgrade contract (rustfs-protos compat manifest): every @@ -3278,7 +3715,7 @@ mod legacy_artifact_tests { request: Request, ) -> Result, Status> { verify_legacy_rpc_request(&request, "RenameData")?; - service.handle_rename_data(request).await + service.handle_rename_data(request, false).await } #[tokio::test] @@ -3799,7 +4236,9 @@ mod legacy_artifact_tests { Bytes::new() }, opts_bin: if msgpack { - encode_msgpack(opts, "DeleteOptions").expect("msgpack delete options").into() + encode_msgpack_named(opts, "DeleteOptions") + .expect("msgpack delete options") + .into() } else { Bytes::new() }, @@ -4088,6 +4527,22 @@ mod legacy_artifact_tests { tonic::Code::PermissionDenied ); assert_eq!(path_delete_snapshot(&fixture.physical), before); + if matches!(case, LegacyPathCase::Residual(false)) && msgpack && volume != ".rustfs.sys" { + if volume == "path-ordinary" { + let rejected = authenticated_path_mutation(&service, &disk_ref, &fixture, msgpack, false) + .await + .expect("obsolete ordinary transaction envelope") + .expect_err("ordinary legacy transaction is unsupported"); + assert_eq!(rejected, DiskError::UnsupportedDisk); + assert_eq!( + path_delete_snapshot(&fixture.physical), + before, + "obsolete residual transaction must not change physical state" + ); + } + // Exercise ordinary deletion and managed classification with a legal residual request. + fixture.options.old_data_dir = None; + } let result = authenticated_path_mutation(&service, &disk_ref, &fixture, msgpack, false).await; if volume == "path-managed" { assert_eq!(result.expect_err("managed path rejects").code(), tonic::Code::PermissionDenied); @@ -5995,7 +6450,9 @@ mod legacy_artifact_tests { serde_json::to_string(opts).expect("JSON bulk options") }, opts_bin: if msgpack { - encode_msgpack(opts, "DeleteOptions").expect("binary bulk options").into() + encode_msgpack_named(opts, "DeleteOptions") + .expect("binary bulk options") + .into() } else { Bytes::new() }, @@ -6079,6 +6536,322 @@ mod legacy_artifact_tests { } } + #[tokio::test] + #[serial_test::serial] + async fn delete_recovery_and_guarded_receivers_reject_managed_mutations() { + use crate::storage::storage_api::ecstore_disk::{CurrentDeleteGeneration, DeleteRecoveryAction, GuardedFileInfoVersions}; + let root = tempfile::tempdir().expect("managed recovery receiver root"); + let env = rustfs_test_utils::TestECStoreEnv::builder() + .base_dir(root.path()) + .build() + .await; + let bucket = "recovery-managed"; + let object = "outputs/74656e616e74/11111111-1111-4111-8111-111111111111/22222222-2222-4222-8222-222222222222/object"; + env.ecstore + .clone() + .make_artifact_bucket(bucket, "outputs/") + .await + .expect("real managed authority"); + env.put_object_bytes(bucket, object, vec![0x42; 4097]).await; + let service = service_for(&env).await; + let disk_ref = env.disk_paths[0].to_string_lossy().into_owned(); + let disk = service + .local_mutation_target() + .find_mutation_disk(&disk_ref) + .await + .expect("captured target"); + let fi = ecstore_disk::DiskAPI::read_version( + disk.as_ref(), + bucket, + bucket, + object, + "", + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("real published version"); + let group = FileInfoVersions { + volume: bucket.to_owned(), + name: object.to_owned(), + versions: vec![fi.clone()], + ..Default::default() + }; + let opts = DeleteOptions { + old_data_dir: Some(Uuid::new_v4()), + ..Default::default() + }; + let before = path_delete_snapshot(&env.disk_paths[0].join(bucket)); + for operation in [ + DeleteRecoveryRpcOperation::ApplyVersion { + object: object.to_owned(), + file_info: Box::new(fi.clone()), + force_del_marker: false, + opts: opts.clone(), + }, + DeleteRecoveryRpcOperation::ApplyVersions { + versions: vec![group.clone()], + opts: opts.clone(), + }, + DeleteRecoveryRpcOperation::Settle { + object: object.to_owned(), + transaction: opts.old_data_dir.expect("transaction"), + expected_fingerprint: [0; 32], + action: DeleteRecoveryAction::Rollback, + }, + ] { + let mut request = Request::new(RecoverDeleteTransactionRequest { + disk: disk_ref.clone(), + volume: bucket.to_owned(), + protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION, + operation: rmp_serde::to_vec_named(&operation).expect("recovery operation").into(), + }); + let canonical = + rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()).expect("recovery body"); + set_tonic_canonical_body_digest(&mut request, &canonical).expect("recovery digest"); + mark_v2_authenticated(&mut request); + assert_eq!( + service + .handle_recover_delete_transaction(request) + .await + .expect_err("managed recovery mutation") + .code(), + tonic::Code::PermissionDenied + ); + assert_eq!(path_delete_snapshot(&env.disk_paths[0].join(bucket)), before); + } + let payload = GuardedDeleteVersionsPayload { + versions: vec![GuardedFileInfoVersions { + expected_current: CurrentDeleteGeneration::from_file_info(&fi).expect("actual generation"), + group, + }], + opts, + }; + payload.validate().expect("valid guarded mutation fixture"); + let mut request = Request::new(GuardedDeleteVersionsRequest { + disk: disk_ref, + volume: bucket.to_owned(), + probe_only: false, + payload: rmp_serde::to_vec_named(&payload).expect("guarded operation").into(), + }); + let canonical = rustfs_protos::canonical_guarded_delete_versions_request_body(request.get_ref()).expect("guarded body"); + set_tonic_canonical_body_digest(&mut request, &canonical).expect("guarded digest"); + mark_v2_authenticated(&mut request); + assert_eq!( + service + .handle_guarded_delete_versions(request) + .await + .expect_err("managed guarded mutation") + .code(), + tonic::Code::PermissionDenied + ); + assert_eq!(path_delete_snapshot(&env.disk_paths[0].join(bucket)), before); + } + + #[tokio::test] + #[serial_test::serial] + async fn artifact_delete_recovery_receivers_apply_and_rollback_real_ordinary_versions() { + use crate::storage::storage_api::ecstore_disk::{CurrentDeleteGeneration, DeleteRecoveryAction, GuardedFileInfoVersions}; + use crate::storage::storage_api::{ObjectOptions, PutObjReader, contract::object::ObjectIO as _}; + use std::time::Duration; + + async fn recover( + service: &NodeService, + disk: &str, + bucket: &str, + operation: DeleteRecoveryRpcOperation, + ) -> DeleteRecoveryRpcReply { + let mut request = Request::new(RecoverDeleteTransactionRequest { + disk: disk.to_owned(), + volume: bucket.to_owned(), + protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION, + operation: rmp_serde::to_vec_named(&operation).expect("named recovery operation").into(), + }); + let body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()) + .expect("canonical recovery body"); + sign_legacy_rpc_request(&mut request, &body, "RecoverDeleteTransaction"); + verify_legacy_rpc_request(&request, "RecoverDeleteTransaction").expect("real recovery HMAC and replay scope"); + let response = tokio::time::timeout(Duration::from_secs(30), service.handle_recover_delete_transaction(request)) + .await + .expect("bounded recovery operation") + .expect("authenticated recovery response") + .into_inner(); + assert!(response.success, "recovery operation failed: {:?}", response.error); + assert!(response.error.is_none()); + assert_eq!(response.protocol_version, DELETE_RECOVERY_PROTOCOL_VERSION); + rustfs_protos::decode_delete_recovery_payload(&response.result).expect("typed recovery reply") + } + + let _ = rustfs_credentials::set_global_rpc_secret("ordinary-delete-recovery-test-secret".to_owned()); + let root = tempfile::tempdir().expect("ordinary recovery receiver root"); + let env = rustfs_test_utils::TestECStoreEnv::builder() + .base_dir(root.path()) + .build() + .await; + let bucket = "delete-recovery-ordinary"; + env.make_bucket(bucket, true).await; + let incarnation = env + .ecstore + .bucket_incarnation_id(bucket) + .await + .expect("real ordinary incarnation"); + let service = service_for(&env).await; + let disk_ref = env.disk_paths[0].to_string_lossy().into_owned(); + let disk = service + .local_mutation_target() + .find_mutation_disk(&disk_ref) + .await + .expect("local receiver disk"); + for mode in ["single", "batch", "guarded"] { + let object = format!("recovery/{mode}"); + let payload = vec![0x42; 257]; + env.ecstore + .put_object( + bucket, + &object, + &mut PutObjReader::from_vec(payload), + &ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect("real versioned ordinary PUT"); + let fi = disk + .read_version( + bucket, + bucket, + &object, + "", + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("real PUT generation on the receiving disk"); + assert!(fi.inline_data(), "small real PUT must produce a complete inline rollback fixture"); + assert!( + fi.data.as_ref().is_some_and(|data| !data.is_empty()), + "real encoded inline shard is present" + ); + let physical = env.disk_paths[0].join(bucket).join(&object); + let before = path_delete_snapshot(&physical); + let transaction = Uuid::new_v4(); + let opts = DeleteOptions { + old_data_dir: Some(transaction), + delete_recovery_bucket_incarnation: Some(incarnation), + ..Default::default() + }; + let group = FileInfoVersions { + volume: bucket.to_owned(), + name: object.clone(), + versions: vec![fi.clone()], + ..Default::default() + }; + if mode == "guarded" { + let payload = GuardedDeleteVersionsPayload { + versions: vec![GuardedFileInfoVersions { + expected_current: CurrentDeleteGeneration::from_file_info(&fi).expect("actual current generation"), + group, + }], + opts, + }; + payload.validate().expect("valid guarded ordinary request"); + let mut request = Request::new(GuardedDeleteVersionsRequest { + disk: disk_ref.clone(), + volume: bucket.to_owned(), + probe_only: false, + payload: rmp_serde::to_vec_named(&payload).expect("named guarded request").into(), + }); + let body = rustfs_protos::canonical_guarded_delete_versions_request_body(request.get_ref()) + .expect("canonical guarded request"); + sign_legacy_rpc_request(&mut request, &body, "GuardedDeleteVersions"); + verify_legacy_rpc_request(&request, "GuardedDeleteVersions").expect("real guarded HMAC and replay scope"); + let response = tokio::time::timeout(Duration::from_secs(30), service.handle_guarded_delete_versions(request)) + .await + .expect("bounded guarded publication") + .expect("authenticated guarded response") + .into_inner(); + assert_bulk_results(&response, &[None]); + } else { + let operation = if mode == "single" { + DeleteRecoveryRpcOperation::ApplyVersion { + object: object.clone(), + file_info: Box::new(fi.clone()), + force_del_marker: false, + opts, + } + } else { + DeleteRecoveryRpcOperation::ApplyVersions { + versions: vec![group], + opts, + } + }; + match recover(&service, &disk_ref, bucket, operation).await { + DeleteRecoveryRpcReply::AppliedVersion if mode == "single" => {} + DeleteRecoveryRpcReply::AppliedVersions(errors) if mode == "batch" => assert_eq!(errors, vec![None]), + reply => panic!("unexpected apply reply: {reply:?}"), + } + } + let DeleteRecoveryRpcReply::Inspected(state) = recover( + &service, + &disk_ref, + bucket, + DeleteRecoveryRpcOperation::Inspect { + object: object.clone(), + transaction, + }, + ) + .await + else { + panic!("typed inspection reply"); + }; + assert_eq!(serde_json::to_value(&state.state).expect("serialized recovery state"), "NewPrepared"); + assert_eq!(state.current_digest, state.journal.as_ref().expect("durable journal").target_digest); + assert!(!physical.join("xl.meta").exists(), "real selected version was removed"); + assert_ne!(path_delete_snapshot(&physical), before, "forward operation must change physical state"); + let DeleteRecoveryRpcReply::Settled(restored) = recover( + &service, + &disk_ref, + bucket, + DeleteRecoveryRpcOperation::Settle { + object: object.clone(), + transaction, + expected_fingerprint: state.fingerprint, + action: DeleteRecoveryAction::Rollback, + }, + ) + .await + else { + panic!("typed rollback reply"); + }; + assert_eq!(serde_json::to_value(&restored.state).expect("serialized recovery state"), "Absent"); + assert_eq!( + path_delete_snapshot(&physical), + before, + "rollback restores exact metadata and removes journal residue" + ); + let restored = disk + .read_version( + bucket, + bucket, + &object, + &fi.version_id.expect("real version").to_string(), + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("exact restored ordinary version"); + assert_eq!(restored.version_id, fi.version_id); + assert_eq!(restored.data, fi.data); + } + } + #[tokio::test] #[serial_test::serial] async fn legacy_bulk_delete_receiver_preserves_signed_group_results_and_rejects_managed() { @@ -6123,9 +6896,37 @@ mod legacy_artifact_tests { ..Default::default() }, ); - let result = - authenticated_legacy_bulk(&service, signed_legacy_bulk_request(&disk_ref, volume, &input, &opts, msgpack)) - .await; + if volume == "bulk-ordinary" { + let rejected = authenticated_legacy_bulk( + &service, + signed_legacy_bulk_request(&disk_ref, volume, &input, &opts, msgpack), + ) + .await + .expect("obsolete ordinary transaction envelope"); + assert!(!rejected.success); + assert!(rejected.errors.is_empty() && rejected.item_errors.is_empty()); + assert_eq!(DiskError::from(rejected.error.expect("obsolete transaction")), DiskError::UnsupportedDisk); + assert_bulk_bytes(&before).await; + for group in &groups { + assert!( + !env.disk_paths[0] + .join(volume) + .join(&group.name) + .join(rollback.to_string()) + .exists() + ); + } + } + let current_opts = if volume == ".rustfs.sys" { + opts.clone() + } else { + DeleteOptions::default() + }; + let result = authenticated_legacy_bulk( + &service, + signed_legacy_bulk_request(&disk_ref, volume, &input, ¤t_opts, msgpack), + ) + .await; if volume == "bulk-managed" { assert_eq!( result.expect_err("managed batch rejects before writes").code(), @@ -6144,6 +6945,21 @@ mod legacy_artifact_tests { continue; } assert_bulk_results(&result.expect("signed forward"), &[None, Some(DiskError::FileNotFound), None]); + if volume == "bulk-ordinary" { + for (path, _) in &before { + assert!(!path.exists(), "ordinary nontransactional batch removes actual metadata and parts"); + } + for group in &groups { + assert!( + !env.disk_paths[0] + .join(volume) + .join(&group.name) + .join(rollback.to_string()) + .exists() + ); + } + continue; + } for (path, original) in &before { if path.file_name().is_some_and(|name| name == "xl.meta") { assert!(!path.exists(), "all selected versions removed"); @@ -6247,7 +7063,7 @@ mod legacy_artifact_tests { target.reject_legacy_artifact_mutation(&volume).await.expect("still ordinary"); for undo in [false, true] { let opts = DeleteOptions { - old_data_dir: Some(Uuid::new_v4()), + old_data_dir: undo.then(Uuid::new_v4), undo_write: undo, undo_delete: undo, ..Default::default() @@ -6302,7 +7118,7 @@ mod legacy_artifact_tests { for volume in ["bulk-instance", ".rustfs.sys"] { let (groups, before) = prepare_legacy_bulk(&disk, volume, &format!("binding/{msgpack}")).await; let opts = DeleteOptions { - old_data_dir: Some(Uuid::new_v4()), + old_data_dir: (volume == ".rustfs.sys").then(Uuid::new_v4), ..Default::default() }; for service in [&unbound, &wrong] { @@ -6418,7 +7234,6 @@ mod legacy_artifact_tests { .base_dir(root.path()) .build() .await; - env.make_bucket("bulk-remote", true).await; let ready = service_for(&env).await; let instance = crate::storage::storage_api::bootstrap_instance_ctx(); let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("loopback listener"); @@ -6442,7 +7257,7 @@ mod legacy_artifact_tests { .find_mutation_disk(&endpoint.to_string()) .await .expect("loopback receiving disk"); - let (groups, before) = prepare_legacy_bulk(&local, "bulk-remote", "objects").await; + let (groups, before) = prepare_legacy_bulk(&local, ".rustfs.sys", "objects").await; let calls = Arc::new(std::sync::atomic::AtomicUsize::new(0)); let observed = calls.clone(); let audience = addr.to_string(); @@ -6487,14 +7302,14 @@ mod legacy_artifact_tests { input.insert( 1, FileInfoVersions { - volume: "bulk-remote".to_owned(), + volume: ".rustfs.sys".to_owned(), name: "objects/missing".to_owned(), ..Default::default() }, ); let forward = remote .delete_versions( - "bulk-remote", + ".rustfs.sys", input, DeleteOptions { old_data_dir: Some(rollback), @@ -6520,7 +7335,7 @@ mod legacy_artifact_tests { } let undo = remote .delete_versions( - "bulk-remote", + ".rustfs.sys", groups, DeleteOptions { old_data_dir: Some(rollback), @@ -7070,10 +7885,44 @@ mod legacy_artifact_tests { assert_eq!(before.is_none(), case == "marker"); let rollback = Uuid::new_v4(); let rollback_path = object_dir.join(rollback.to_string()); - let opts = DeleteOptions { + let legacy_opts = DeleteOptions { old_data_dir: Some(rollback), ..Default::default() }; + if volume == "staged-delete-ordinary" { + let original = path_delete_snapshot(&object_dir); + let rejected = authenticated_legacy_delete( + &service, + signed_legacy_delete_request( + &disk_ref, + volume, + &path, + &fi, + &legacy_opts, + msgpack, + case == "marker", + ), + ) + .await + .expect("obsolete ordinary staging envelope") + .into_inner(); + assert!(!rejected.success); + assert_eq!( + DiskError::from(rejected.error.expect("obsolete transaction")), + DiskError::UnsupportedDisk + ); + assert_eq!( + path_delete_snapshot(&object_dir), + original, + "obsolete transaction changes no physical bytes" + ); + assert!(!rollback_path.exists()); + } + let opts = if volume == ".rustfs.sys" { + legacy_opts + } else { + DeleteOptions::default() + }; if volume == ".rustfs.sys" && case == "last" { let rejected = authenticated_legacy_delete( &wrong_instance, @@ -7098,6 +7947,57 @@ mod legacy_artifact_tests { assert_eq!(DiskError::from(response.error.expect("bootstrap error")), DiskError::FileAccessDenied); } else if volume == "staged-delete-managed" { assert_eq!(result.expect_err("managed staging rejects").code(), tonic::Code::PermissionDenied); + } else if volume == "staged-delete-ordinary" { + let response = result.expect("ordinary nontransactional delete response").into_inner(); + assert!(response.success, "ordinary current delete succeeds: {:?}", response.error); + assert!(response.error.is_none()); + assert_eq!(response.raw_file_info, "null"); + let current = disk + .read_version( + volume, + volume, + &path, + &fi.version_id.expect("requested version").to_string(), + &ReadOptions::default(), + ) + .await; + if case == "marker" { + let marker = current.expect("ordinary marker is durably created"); + assert!(marker.is_canonical_delete_marker()); + assert_eq!(marker.version_id, fi.version_id); + } else { + assert_eq!( + current.expect_err("ordinary selected version is removed"), + DiskError::FileVersionNotFound + ); + if let Some((previous, _)) = previous { + let retained = disk + .read_version( + volume, + volume, + &path, + &previous.version_id.expect("previous version").to_string(), + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("ordinary delete retains the other version"); + assert_eq!(retained.version_id, previous.version_id); + assert_eq!(retained.data, previous.data); + } else { + assert!(!metadata.exists(), "ordinary final version metadata is removed"); + } + if let Some(data_dir) = fi.data_dir { + assert!( + !object_dir.join(data_dir.to_string()).exists(), + "ordinary final version parts are removed" + ); + } + } + assert!(!rollback_path.exists(), "current direct delete does not stage legacy rollback"); + continue; } else { let response = result.expect("signed staged delete response").into_inner(); assert!( @@ -7255,10 +8155,7 @@ mod legacy_artifact_tests { .reject_legacy_artifact_mutation(&volume) .await .expect("ordinary authority still classifies"); - let opts = DeleteOptions { - old_data_dir: Some(Uuid::new_v4()), - ..Default::default() - }; + let opts = DeleteOptions::default(); let response = authenticated_legacy_delete( &service, signed_legacy_delete_request(&disk_ref, &volume, path, &fi, &opts, msgpack, case == "marker"), diff --git a/rustfs/src/storage/sse.rs b/rustfs/src/storage/sse.rs index b6119c9f1..ee081e25b 100644 --- a/rustfs/src/storage/sse.rs +++ b/rustfs/src/storage/sse.rs @@ -1391,6 +1391,39 @@ fn stored_envelope_master_key_version(metadata: &HashMap) -> Opt pub(crate) struct SseObjectEncryptionResolver; +/// Checks metadata and key access before a conditional read returns without a body. +pub(crate) async fn validate_conditional_read( + bucket: &str, + object: &str, + metadata: &HashMap, + headers: &HeaderMap, + principal: Option<&SseKmsPrincipal>, +) -> Result<(), ApiError> { + let metadata = normalize_encryption_metadata_case(metadata) + .map_err(|error| map_get_object_reader_error(StorageError::Io(std::io::Error::other(error))))?; + validate_sse_headers_for_read(&metadata, headers)?; + let (_, customer_key, customer_key_md5) = extract_ssec_params_from_headers(headers)?; + classify_sse_read_response(DecryptionRequest { + bucket, + key: object, + metadata: &metadata, + sse_customer_key: customer_key.as_ref(), + sse_customer_key_md5: customer_key_md5.as_ref(), + principal, + }) + .await?; + SseObjectEncryptionResolver + .resolve_read_material(ReadEncryptionRequest { + bucket, + object, + metadata: &metadata, + headers, + }) + .await + .map_err(|error| map_get_object_reader_error(StorageError::Io(std::io::Error::other(error))))?; + Ok(()) +} + #[async_trait] impl ObjectEncryptionResolver for SseObjectEncryptionResolver { async fn resolve_read_material( @@ -8588,6 +8621,35 @@ mod tests { assert_eq!(audit_tag(&tags, "kmsErrorClass").as_deref(), Some("access_denied")); } + #[tokio::test] + async fn conditional_read_denies_kms_before_unwrap_with_normalized_metadata() { + let (principal, audit) = audited_principal(true, false); + let metadata = sse_kms_object_metadata() + .into_iter() + .map(|(name, value)| (name.to_ascii_uppercase(), value)) + .collect(); + let error = super::validate_conditional_read("finance", "ledger.csv", &metadata, &HeaderMap::new(), Some(&principal)) + .await + .expect_err("conditional read must authorize the real key before unwrap"); + assert_eq!(error.code, S3ErrorCode::AccessDenied); + assert_eq!(audit_tag(&audit.audit_tags(), "kmsKeyId").as_deref(), Some("finance-key")); + } + + #[cfg(feature = "rio-v2")] + #[tokio::test] + async fn conditional_read_denies_minio_only_kms_before_unwrap() { + let (principal, authorizer) = enforcing_principal(false); + let metadata = HashMap::from([ + (super::MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER.to_owned(), "sealed-key".to_owned()), + (super::MINIO_INTERNAL_ENCRYPTION_KMS_KEY_ID_HEADER.to_owned(), "finance-key".to_owned()), + ]); + let error = super::validate_conditional_read("finance", "ledger.csv", &metadata, &HeaderMap::new(), Some(&principal)) + .await + .expect_err("MinIO metadata must not bypass KMS authorization"); + assert_eq!(error.code, S3ErrorCode::AccessDenied); + assert_eq!(authorizer.calls(), vec![(KmsAction::DecryptAction, "finance-key".to_owned())]); + } + #[test] fn read_response_projection_uses_shared_managed_metadata_compatibility() { let context_json = r#"{"tenant":"s3-select"}"#; diff --git a/rustfs/src/storage/storage_api.rs b/rustfs/src/storage/storage_api.rs index 691c38463..374620c31 100644 --- a/rustfs/src/storage/storage_api.rs +++ b/rustfs/src/storage/storage_api.rs @@ -27,6 +27,15 @@ pub(crate) mod ecstore_integrity { IntegrityError, JobRequest, control_job, create_job, get_job, inventory, readiness, resume_job, }; } + +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) mod ecstore_diagnostic { + #[cfg(test)] + pub(crate) use rustfs_ecstore::api::diagnostic::current_read_version_entry_source; + pub(crate) use rustfs_ecstore::api::diagnostic::{ + ReadVersionContextBoundary, ReadVersionEntrySource, with_read_version_spawn_context, + }; +} use tokio::sync::{Mutex, OwnedMutexGuard}; use tokio_util::sync::CancellationToken; @@ -59,6 +68,8 @@ fn bucket_targets_metadata_lock_shard(bucket: &str) -> usize { pub(crate) mod contract { pub(crate) mod admin { + #[cfg(test)] + pub(crate) use super::super::storage_contracts::DiskSetSelector; pub(crate) use super::super::storage_contracts::StorageAdminApi; } @@ -109,11 +120,12 @@ pub(crate) type StorageSelectObjectSnapshot = ecstore_object::SelectObjectSnapsh pub(crate) type StorageObjectToDelete = contract::object::ObjectToDelete; pub(crate) type StoragePutObjReader = super::PutObjReader; pub(crate) use super::ecfs_extend::{ - RFC1123, apply_bucket_default_lock_retention, apply_cors_headers, check_preconditions, get_buffer_size_opt_in, - get_validated_store, load_bucket_object_lock_config_state, parse_object_lock_legal_hold, parse_object_lock_retention, - parse_part_number_i32_to_usize, process_lambda_configurations, process_queue_configurations, process_topic_configurations, - remove_object_lock_metadata_for_copy, validate_bucket_exists, validate_bucket_object_lock_enabled_state, - validate_list_object_unordered_with_delimiter, validate_object_key, wrap_response_with_cors, + RFC1123, apply_bucket_default_lock_retention, apply_cors_headers, check_preconditions, evaluate_read_preconditions, + get_buffer_size_opt_in, get_validated_store, has_read_preconditions, load_bucket_object_lock_config_state, + parse_object_lock_legal_hold, parse_object_lock_retention, parse_part_number_i32_to_usize, process_lambda_configurations, + process_queue_configurations, process_topic_configurations, remove_object_lock_metadata_for_copy, validate_bucket_exists, + validate_bucket_object_lock_enabled_state, validate_list_object_unordered_with_delimiter, validate_object_key, + wrap_response_with_cors, }; pub(crate) use super::sse::{ DecryptionRequest, EncryptionRequest, ObjectDekRewrapOutcome, PrepareEncryptionRequest, SseKmsPrincipal, @@ -124,10 +136,10 @@ pub(crate) use super::sse::{ pub(crate) mod access_consumer { pub(crate) use super::super::access::{ - PostObjectRequestMarker, ReqInfo, TABLE_DATA_PLANE_LIST_CURSOR_PREFIX, TableDataPlaneListAccess, - TableDataPlaneListCursorPosition, TableDataPlanePublicationGuards, apply_bucket_generation_guard, - apply_copy_source_bucket_generation_guard, authorize_internal_object_request, authorize_request, - bucket_config_mutation_incarnation, delete_object_authorize_action, has_bypass_governance_header, + MultipartStagingOperation, PostObjectRequestMarker, ReqInfo, TABLE_DATA_PLANE_LIST_CURSOR_PREFIX, + TableDataPlaneListAccess, TableDataPlaneListCursorPosition, TableDataPlanePublicationGuards, + apply_bucket_generation_guard, apply_copy_source_bucket_generation_guard, authorize_internal_object_request, + authorize_request, bucket_config_mutation_incarnation, delete_object_authorize_action, has_bypass_governance_header, load_bucket_generation_from_store, log_list_buckets_iam_implicit_deny, odm_read_generation, prepare_list_buckets_iam_authorization, prepare_odm_read_generation, recursive_force_delete_has_authenticated_caller, replication_request_authorized, req_info_mut, req_info_ref, retained_table_data_plane_publication_guards, @@ -269,12 +281,13 @@ pub(crate) mod rpc_consumer { }; pub(crate) use super::super::{ BatchReadVersionReq, BatchReadVersionResp, CollectMetricsOpts, ConditionalFileUpdate, DeleteOptions, DiskError, - DiskInfoOptions, DiskStore, ECStore, Error, FileInfoVersions, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, MetricType, - PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, ReadMultipleReq, ReadMultipleResp, ReadOptions, - SCANNER_PUBLICATION_LEASE_TTL_MS, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, StorageDiskRpcExt, - StoragePeerS3ClientExt, TierDailyStatsWire, UpdateMetadataOpts, all_local_disk_path, collect_local_metrics, - find_local_disk_by_ref, get_global_transition_state, get_local_server_property, reload_bucket_metadata, - reload_transition_tier_config, remove_bucket_metadata, validate_batch_read_version_item_count, + DiskInfoOptions, DiskStore, ECStore, Error, FileInfoVersions, GuardedDeleteVersionsPayload, KMS_SIGNAL_SUBSYSTEM, + LocalPeerS3Client, MetricType, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, ReadMultipleReq, + ReadMultipleResp, ReadOptions, SCANNER_PUBLICATION_LEASE_TTL_MS, SERVICE_SIGNAL_REFRESH_CONFIG, + SERVICE_SIGNAL_RELOAD_DYNAMIC, StorageDiskRpcExt, StoragePeerS3ClientExt, TierDailyStatsWire, UpdateMetadataOpts, + all_local_disk_path, collect_local_metrics, find_local_disk_by_ref, get_global_transition_state, + get_local_server_property, reload_bucket_metadata, reload_transition_tier_config, remove_bucket_metadata_if_absent, + validate_batch_read_version_item_count, }; pub(crate) type StorageResult = super::super::Result; @@ -400,7 +413,7 @@ pub(crate) mod tonic_service_consumer { pub(crate) mod test_consumer { pub(crate) use super::super::{ apply_cors_headers, apply_default_lock_retention_metadata, check_preconditions, decode_tags_to_map, - get_adaptive_buffer_size_with_profile, get_buffer_size_opt_in, is_etag_equal, matches_origin_pattern, parse_etag, + get_adaptive_buffer_size_with_profile, get_buffer_size_opt_in, matches_origin_pattern, parse_etag, parse_object_lock_legal_hold, parse_object_lock_retention, process_lambda_configurations, process_queue_configurations, process_topic_configurations, remove_object_lock_metadata_for_copy, remove_object_lock_retention_metadata, validate_bucket_object_lock_enabled, validate_list_object_unordered_with_delimiter, @@ -478,15 +491,18 @@ pub(crate) mod ecstore_data_usage { #[allow(unused_imports)] pub(crate) mod ecstore_disk { pub(crate) use rustfs_ecstore::api::disk::{ - BUCKET_META_PREFIX, BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DeleteOptions, - DiskAPI, DiskInfo, DiskInfoOptions, DiskStore, FileInfoVersions, FileReader, LegacyFileStream, LegacyFileWrite, - LegacyMetadataWrite, LegacyPartMutation, LegacyPartOperation, LegacyPathDelete, LegacyPathDeleteOperation, - LegacyRenameData, LegacyRenameFile, OldCurrentSize, PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, - ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, - get_object_disk_read_timeout, validate_batch_read_version_item_count, + BUCKET_META_PREFIX, BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, + DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, + DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, + DiskInfoOptions, DiskStore, FileInfoVersions, FileReader, FileWriter, GuardedDeleteVersionsPayload, + GuardedFileInfoVersions, LegacyFileStream, LegacyFileWrite, LegacyMetadataWrite, LegacyPartMutation, LegacyPartOperation, + LegacyPathDelete, LegacyPathDeleteOperation, LegacyRenameData, LegacyRenameFile, OldCurrentSize, PartTransactionAction, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, + SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, get_object_disk_read_timeout, + is_quota_mutation_fence_path, validate_batch_read_version_item_count, }; #[cfg(test)] - pub(crate) use rustfs_ecstore::api::disk::{DiskOption, new_disk}; + pub(crate) use rustfs_ecstore::api::disk::{CurrentDeleteGeneration, DiskOption, new_disk}; pub(crate) use rustfs_ecstore::api::disk::{endpoint, error, error_reduce}; } @@ -527,12 +543,14 @@ pub(crate) mod ecstore_metrics { #[allow(unused_imports)] pub(crate) mod ecstore_notification { - #[cfg(test)] - pub(crate) use rustfs_ecstore::api::notification::rotate_cross_pool_fence_fleet_proof_for_test; pub(crate) use rustfs_ecstore::api::notification::{ ClusterTierDailyStats, CrossPoolFenceFleetProofToken, NotificationSys, acquire_cross_pool_fence_fleet_proof, - cross_pool_fence_fleet_proof_matches, get_global_notification_sys, ilm_recovery_export_local_process_epoch, - new_global_notification_sys, start_remote_version_state_fleet_probe, + configured_fleet_topology_fingerprint, cross_pool_fence_fleet_proof_matches, get_global_notification_sys, + ilm_recovery_export_local_process_epoch, new_global_notification_sys, start_remote_version_state_fleet_probe, + }; + #[cfg(test)] + pub(crate) use rustfs_ecstore::api::notification::{ + rotate_cross_pool_fence_fleet_proof_for_test, without_cross_pool_fence_fleet_proof_for_test, }; } @@ -579,10 +597,11 @@ pub(crate) mod ecstore_object { pub(crate) use rustfs_ecstore::api::object::GetObjectBodySource; pub(crate) use rustfs_ecstore::api::object::{ EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook, GetObjectBodyCacheHookLookup, - ObjectEncryptionResolver, ObjectMutationHook, PrepareSelectObjectSnapshotError, ReadEncryptionMaterial, - ReadEncryptionMode, ReadEncryptionRequest, SelectObjectSnapshot, WriteCommitGuard, WriteCompletion, - get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook, - register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook, + GetObjectReadCondition, GetObjectReadMetadata, ObjectEncryptionResolver, ObjectMutationHook, + PrepareSelectObjectSnapshotError, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest, + SelectObjectSnapshot, WriteCommitGuard, WriteCompletion, get_object_body_cache_plaintext_len, + lookup_get_object_body_cache_hook, register_get_object_body_cache_hook, register_object_mutation_hook, + unregister_get_object_body_cache_hook, unregister_object_mutation_hook, with_get_object_read_condition, }; } @@ -597,8 +616,8 @@ pub(crate) mod ecstore_test_support { pub(crate) mod ecstore_set_disk { #[cfg(test)] pub(crate) use rustfs_ecstore::api::set_disk::test_util::{ - MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause, - fail_next_quota_ledger_save_for_test, + ConditionalReadBenchmarkMetadataGuard, MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, + PutObjectCommitPause, fail_next_quota_ledger_save_for_test, }; pub(crate) use rustfs_ecstore::api::set_disk::{ DEFAULT_READ_BUFFER_SIZE, file_info_quorum_hash, get_lock_acquire_timeout, is_valid_storage_class, @@ -706,6 +725,8 @@ pub(crate) type EndpointServerPools = ecstore_layout::EndpointServerPools; pub(crate) type EventArgs = ecstore_event::EventArgs; pub(crate) type ExpiryState = ecstore_bucket::lifecycle::bucket_lifecycle_ops::ExpiryState; pub(crate) type FileInfoVersions = ecstore_disk::FileInfoVersions; +pub(crate) type GuardedFileInfoVersions = ecstore_disk::GuardedFileInfoVersions; +pub(crate) type GuardedDeleteVersionsPayload = ecstore_disk::GuardedDeleteVersionsPayload; pub(crate) type FileReader = ecstore_disk::FileReader; pub(crate) type SnapshotLeaseToken = ecstore_disk::SnapshotLeaseToken; pub(crate) type FS = super::ecfs::FS; @@ -734,6 +755,14 @@ pub(crate) type ReadMultipleReq = ecstore_disk::ReadMultipleReq; pub(crate) type ReadMultipleResp = ecstore_disk::ReadMultipleResp; pub(crate) type ReadOptions = ecstore_disk::ReadOptions; pub(crate) type OldCurrentSize = ecstore_disk::OldCurrentSize; +pub(crate) type DeleteRecoveryCursor = ecstore_disk::DeleteRecoveryCursor; +pub(crate) type DeleteRecoveryPage = ecstore_disk::DeleteRecoveryPage; +pub(crate) type DeleteRecoveryInspection = ecstore_disk::DeleteRecoveryInspection; +pub(crate) type DeleteRecoveryDigest = ecstore_disk::DeleteRecoveryDigest; +pub(crate) type DeleteRecoveryAction = ecstore_disk::DeleteRecoveryAction; +pub(crate) type DeleteRecoveryRpcOperation = ecstore_disk::DeleteRecoveryRpcOperation; +pub(crate) type DeleteRecoveryRpcReply = ecstore_disk::DeleteRecoveryRpcReply; +pub(crate) const DELETE_RECOVERY_PROTOCOL_VERSION: u32 = ecstore_disk::DELETE_RECOVERY_PROTOCOL_VERSION; pub(crate) type PartTransactionAction = ecstore_disk::PartTransactionAction; pub(crate) type RenameDataResp = ecstore_disk::RenameDataResp; pub(crate) type LegacyRenameData = ecstore_disk::LegacyRenameData; @@ -973,23 +1002,45 @@ pub(crate) async fn get_local_server_property() -> rustfs_madmin::ServerProperti ecstore_admin::get_local_server_property().await } +/// Bridge an ECStore committed replication-path mutation into the scanner's +/// dirty-usage tracking. +/// +/// A replication status writeback changes only replication state on an +/// existing version. It is status class: it must not count as namespace +/// activity, or every scanner cycle is `Superseded` for as long as replication +/// runs (#8375). The scanner's own queued replication heals write back through +/// the same path and are covered by the same rule. Tier-expiration cleanup +/// removes data and stays on the namespace path. +pub(crate) fn record_committed_replication_mutation( + bucket: &str, + object: &str, + source: ecstore_bucket::replication::ScannerDirtyUsageMutationSource, +) { + match source { + ecstore_bucket::replication::ScannerDirtyUsageMutationSource::Replication => { + rustfs_scanner::record_status_class_mutation( + bucket, + object, + rustfs_scanner::SegmentInvalidationProducerIdentity::Replication, + ); + } + ecstore_bucket::replication::ScannerDirtyUsageMutationSource::TierExpiration => { + rustfs_scanner::record_dirty_usage_object_from_producer( + bucket, + object, + rustfs_scanner::SegmentInvalidationProducerIdentity::TierExpiration, + ); + } + } +} + pub(crate) async fn init_background_replication(store: Arc) { let durable_dirty_usage_journal = super::scanner_dirty_journal::start_durable_dirty_usage_journal(store.clone()).await; let mutation_journal = durable_dirty_usage_journal.clone(); rustfs_scanner::set_scanner_dirty_usage_mutation_observer(Some(Arc::new(move |bucket, object, producer| { mutation_journal.record_committed_mutation(bucket, object, producer); }))); - ecstore_bucket::replication::set_scanner_dirty_usage_mutation_observer(Some(Arc::new(move |bucket, object, source| { - let producer = match source { - ecstore_bucket::replication::ScannerDirtyUsageMutationSource::Replication => { - rustfs_scanner::SegmentInvalidationProducerIdentity::Replication - } - ecstore_bucket::replication::ScannerDirtyUsageMutationSource::TierExpiration => { - rustfs_scanner::SegmentInvalidationProducerIdentity::TierExpiration - } - }; - rustfs_scanner::record_dirty_usage_object_from_producer(bucket, object, producer); - }))); + ecstore_bucket::replication::set_scanner_dirty_usage_mutation_observer(Some(Arc::new(record_committed_replication_mutation))); rustfs_scanner::set_scanner_dirty_usage_clear_observer(Some(Arc::new(move |cleared| { durable_dirty_usage_journal.clear_confirmed_buckets(cleared); }))); @@ -1205,8 +1256,8 @@ pub(crate) fn get_global_transition_state() -> Arc { ecstore_bucket::lifecycle::bucket_lifecycle_ops::get_global_transition_state() } -pub(crate) async fn try_migrate_bucket_metadata(store: Arc) -> std::io::Result<()> { - ecstore_bucket::migration::try_migrate_bucket_metadata(store) +pub(crate) async fn try_migrate_bucket_metadata(store: Arc, buckets: &[String]) -> std::io::Result<()> { + ecstore_bucket::migration::try_migrate_bucket_metadata_for_buckets(store, buckets) .await .map_err(ecstore_bucket::migration::migration_startup_error) } @@ -1378,6 +1429,8 @@ pub(crate) async fn try_migrate_server_config(store: Arc) { } pub(crate) trait StorageDiskRpcExt { + async fn supports_guarded_delete_versions(&self) -> DiskResult<()>; + async fn disk_info(&self, opts: &DiskInfoOptions) -> DiskResult; async fn delete_volume(&self, volume: &str, force_delete: bool) -> DiskResult<()>; async fn read_multiple(&self, req: ReadMultipleReq) -> DiskResult>; @@ -1418,6 +1471,7 @@ pub(crate) trait StorageDiskRpcExt { async fn make_volumes(&self, volume: Vec<&str>) -> DiskResult<()>; async fn list_dir(&self, origvolume: &str, volume: &str, dir_path: &str, count: i32) -> DiskResult>; async fn read_file_stream(&self, volume: &str, path: &str, offset: usize, length: usize) -> DiskResult; + async fn verify_file(&self, volume: &str, path: &str, file_info: &rustfs_filemeta::FileInfo) -> DiskResult; async fn check_parts(&self, volume: &str, path: &str, file_info: &rustfs_filemeta::FileInfo) -> DiskResult; async fn read_parts(&self, bucket: &str, paths: &[String]) -> DiskResult>; @@ -1429,6 +1483,10 @@ impl StorageDiskRpcExt for T where T: ecstore_disk::DiskAPI, { + async fn supports_guarded_delete_versions(&self) -> DiskResult<()> { + ecstore_disk::DiskAPI::supports_guarded_delete_versions(self).await + } + async fn disk_info(&self, opts: &DiskInfoOptions) -> DiskResult { ecstore_disk::DiskAPI::disk_info(self, opts).await } @@ -1801,8 +1859,8 @@ pub(crate) async fn reload_bucket_metadata(api: Arc, bucket: &str) -> R ecstore_bucket::metadata_sys::reload_bucket_metadata(api, bucket).await } -pub(crate) async fn remove_bucket_metadata(bucket: &str) -> Result { - ecstore_bucket::metadata_sys::remove_bucket_metadata(bucket).await +pub(crate) async fn remove_bucket_metadata_if_absent(api: Arc, bucket: &str) -> Result { + ecstore_bucket::metadata_sys::remove_bucket_metadata_if_absent(api, bucket).await } pub(crate) async fn update_bucket_metadata_config( @@ -2170,6 +2228,34 @@ mod tests { }; use std::time::Duration; + #[test] + #[serial_test::serial] + fn replication_status_writebacks_are_status_class_and_tier_expiration_is_namespace() { + use ecstore_bucket::replication::ScannerDirtyUsageMutationSource; + let bucket = format!("status-class-{}", uuid::Uuid::new_v4().simple()); + let before = rustfs_scanner::scanner_dirty_usage_state(); + + super::record_committed_replication_mutation(&bucket, "2026/object", ScannerDirtyUsageMutationSource::Replication); + let after_replication = rustfs_scanner::scanner_dirty_usage_state(); + assert_eq!( + after_replication.generation, before.generation, + "a replication status writeback must not advance the scanner dirty-usage generation" + ); + assert!( + rustfs_scanner::scanner_dirty_usage_bucket_generation(&bucket).is_none(), + "a replication status writeback must not mark the bucket dirty" + ); + + super::record_committed_replication_mutation(&bucket, "2026/object", ScannerDirtyUsageMutationSource::TierExpiration); + let after_expiry = rustfs_scanner::scanner_dirty_usage_state(); + assert!( + after_expiry.generation > before.generation, + "tier expiration removes data and must stay on the namespace path" + ); + assert!(rustfs_scanner::scanner_dirty_usage_bucket_generation(&bucket).is_some()); + rustfs_scanner::clear_dirty_usage_bucket(&bucket); + } + #[tokio::test] async fn site_metrics_snapshot_includes_live_failure_windows() { let stats = StorageReplicationStatsHandle::new(); diff --git a/rustfs/src/storage/timeout_wrapper.rs b/rustfs/src/storage/timeout_wrapper.rs index f0ca0f882..94f0bed24 100644 --- a/rustfs/src/storage/timeout_wrapper.rs +++ b/rustfs/src/storage/timeout_wrapper.rs @@ -412,21 +412,21 @@ impl RequestTimeoutWrapper { debug!(timeout_secs = timeout_duration.as_secs(), "Starting timed operation"); - rustfs_io_metrics::record_get_object_request_started(); + rustfs_io_metrics::record_get_object_timeout_operation_start(); let cancel_token_for_op = self.cancel_token.clone(); match tokio::time::timeout(timeout_duration, operation(cancel_token_for_op)).await { Ok(Ok(result)) => { let elapsed = self.elapsed(); - rustfs_io_metrics::record_get_object_request_result("success", elapsed.as_secs_f64()); + rustfs_io_metrics::record_get_object_timeout_operation_result("success", elapsed.as_secs_f64()); debug!(elapsed_ms = elapsed.as_millis(), "Operation completed successfully"); TimedGetObjectResult::Success(result) } Ok(Err(e)) => { let elapsed = self.elapsed(); - rustfs_io_metrics::record_get_object_request_result("error", elapsed.as_secs_f64()); + rustfs_io_metrics::record_get_object_timeout_operation_result("error", elapsed.as_secs_f64()); debug!(elapsed_ms = elapsed.as_millis(), "Operation failed with error"); TimedGetObjectResult::Error(e) @@ -436,7 +436,7 @@ impl RequestTimeoutWrapper { self.cancel_token.cancel(); rustfs_io_metrics::record_get_object_timeout(None, Some(elapsed.as_secs_f64())); - rustfs_io_metrics::record_get_object_request_result("timeout", elapsed.as_secs_f64()); + rustfs_io_metrics::record_get_object_timeout_operation_result("timeout", elapsed.as_secs_f64()); warn!( timeout_secs = timeout_duration.as_secs(), diff --git a/rustfs/src/storage_api.rs b/rustfs/src/storage_api.rs index b7528a860..67e6e9c4b 100644 --- a/rustfs/src/storage_api.rs +++ b/rustfs/src/storage_api.rs @@ -79,6 +79,12 @@ pub(crate) mod config_test { } pub(crate) mod error { + #[cfg(test)] + pub(crate) use crate::storage::storage_api::ecstore_disk::error::Error as DiskError; + #[cfg(test)] + pub(crate) use crate::storage::storage_api::ecstore_disk::error_reduce::{ + OBJECT_OP_IGNORED_ERRS, reduce_read_quorum_errs, reduce_write_quorum_errs, + }; pub(crate) mod contract { pub(crate) use super::super::storage_contracts::error::StorageErrorCode; @@ -128,7 +134,7 @@ pub(crate) mod kms { pub(crate) mod protocols { pub(crate) mod client { - pub(crate) use crate::storage::storage_api::access_consumer::{ReqInfo, authorize_request}; + pub(crate) use crate::storage::storage_api::access_consumer::{MultipartStagingOperation, ReqInfo, authorize_request}; #[cfg(test)] pub(crate) use crate::storage::storage_api::contract::bucket::{BucketOperations, MakeBucketOptions}; #[cfg(test)] @@ -174,6 +180,8 @@ pub(crate) mod server { } pub(crate) mod http { + #[cfg(test)] + pub(crate) use crate::storage::storage_api::ecstore_disk::{DiskAPI, DiskOption, new_disk}; pub(crate) use crate::storage::storage_api::{ ServerContextSlot, TONIC_RPC_PREFIX, normalize_tonic_rpc_audience, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason, verify_tonic_rpc_signature_with_bootstrap, @@ -452,6 +460,7 @@ pub(crate) mod workload { } pub(crate) mod table { + pub(crate) use crate::storage::storage_api::ecstore_bucket::utils::check_valid_bucket_name_strict; pub(crate) use crate::storage::storage_api::ecstore_object::{WriteCommitGuard, WriteCompletion}; pub(crate) mod contract { pub(crate) mod http { diff --git a/rustfs/src/table_catalog/error.rs b/rustfs/src/table_catalog/error.rs index f06bc7c95..db3c93160 100644 --- a/rustfs/src/table_catalog/error.rs +++ b/rustfs/src/table_catalog/error.rs @@ -55,7 +55,7 @@ impl fmt::Display for TableObjectMutationError { } impl std::error::Error for TableObjectMutationError {} -#[derive(Debug, Clone, PartialEq, Eq)] +#[derive(Debug)] pub(crate) enum TableCatalogStoreError { NotFound(String), NamespaceNotFound(String), @@ -65,6 +65,10 @@ pub(crate) enum TableCatalogStoreError { Invalid(String), Unsupported(String), Unavailable(String), + LockUnavailable { + message: String, + source: Box, + }, Internal(String), } @@ -78,12 +82,27 @@ impl fmt::Display for TableCatalogStoreError { Self::Conflict(message) => write!(f, "table catalog conflict: {message}"), Self::Invalid(message) => write!(f, "invalid table catalog entry: {message}"), Self::Unsupported(message) => write!(f, "unsupported table catalog operation: {message}"), - Self::Unavailable(message) => write!(f, "table catalog temporarily unavailable: {message}"), + Self::Unavailable(message) | Self::LockUnavailable { message, .. } => { + write!(f, "table catalog temporarily unavailable: {message}") + } Self::Internal(message) => write!(f, "table catalog store error: {message}"), } } } -impl std::error::Error for TableCatalogStoreError {} +impl TableCatalogStoreError { + pub(crate) fn is_unavailable(&self) -> bool { + matches!(self, Self::Unavailable(_) | Self::LockUnavailable { .. }) + } +} + +impl std::error::Error for TableCatalogStoreError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::LockUnavailable { source, .. } => Some(source.as_ref()), + _ => None, + } + } +} pub(crate) type TableCatalogStoreResult = Result; diff --git a/rustfs/src/table_catalog/iceberg/commit.rs b/rustfs/src/table_catalog/iceberg/commit.rs index 2b91593ad..19ee14ca3 100644 --- a/rustfs/src/table_catalog/iceberg/commit.rs +++ b/rustfs/src/table_catalog/iceberg/commit.rs @@ -277,7 +277,7 @@ fn table_catalog_store_result_label(result: &TableCatalogStoreResult) -> & | TableCatalogStoreError::TableNotFound(_), ) => "not_found", Err(TableCatalogStoreError::Unsupported(_)) => "unsupported", - Err(TableCatalogStoreError::Unavailable(_)) => "unavailable", + Err(TableCatalogStoreError::Unavailable(_) | TableCatalogStoreError::LockUnavailable { .. }) => "unavailable", Err(TableCatalogStoreError::Internal(_)) => "failure", } } @@ -286,7 +286,7 @@ fn duration_millis_u64(duration: StdDuration) -> u64 { u64::try_from(duration.as_millis()).unwrap_or(u64::MAX) } -pub(crate) fn record_table_commit_cas_result(operation: &str, started: Instant, result: &TableCatalogStoreResult<()>) { +pub(crate) fn record_table_commit_cas_result(operation: &str, started: Instant, result: &TableCatalogStoreResult) { let elapsed = started.elapsed(); let result_label = table_catalog_store_result_label(result); counter!( @@ -390,10 +390,15 @@ mod tests { #[test] fn table_catalog_unavailable_result_label_is_stable() { - let result: TableCatalogStoreResult<()> = Err(TableCatalogStoreError::Unavailable( - "catalog publication authority is temporarily unavailable".to_string(), - )); - - assert_eq!(table_catalog_store_result_label(&result), "unavailable"); + for error in [ + TableCatalogStoreError::Unavailable("catalog publication authority is temporarily unavailable".to_string()), + TableCatalogStoreError::LockUnavailable { + message: "catalog publication authority is temporarily unavailable".to_string(), + source: Box::new(rustfs_lock::LockError::timeout("catalog/lock", StdDuration::from_secs(5))), + }, + ] { + let result: TableCatalogStoreResult<()> = Err(error); + assert_eq!(table_catalog_store_result_label(&result), "unavailable"); + } } } diff --git a/rustfs/src/table_catalog/iceberg/manifest.rs b/rustfs/src/table_catalog/iceberg/manifest.rs index 437c156a2..b2c7a8967 100644 --- a/rustfs/src/table_catalog/iceberg/manifest.rs +++ b/rustfs/src/table_catalog/iceberg/manifest.rs @@ -581,6 +581,7 @@ fn avro_i64_value(value: &apache_avro::types::Value) -> Option { #[cfg(test)] mod tests { use super::*; + use std::assert_matches; #[test] fn rejects_v2_manifest_lists_without_required_count_fields() { @@ -609,9 +610,9 @@ mod tests { Ok(_) => panic!("v2 count fields must be declared in the writer schema"), Err(error) => error, }; - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("Iceberg v2 manifest list Avro schema is missing added_files_count".to_string()) + TableCatalogStoreError::Invalid(message) if message == "Iceberg v2 manifest list Avro schema is missing added_files_count" ); } @@ -622,15 +623,15 @@ mod tests { ("added_rows_count".to_string(), apache_avro::types::Value::Long(-1)), ]); - assert_eq!( - avro_nullable_non_negative_i32(&value, "added_files_count", "manifest list") - .expect_err("negative file counts must be rejected"), - TableCatalogStoreError::Invalid("manifest list field added_files_count must be a non-negative int".to_string()) + assert_matches!( + avro_nullable_non_negative_i32(&value, "added_files_count", "manifest list") + .expect_err("negative file counts must be rejected"), + TableCatalogStoreError::Invalid(message) if message == "manifest list field added_files_count must be a non-negative int" ); - assert_eq!( - avro_nullable_non_negative_i64(&value, "added_rows_count", "manifest list") - .expect_err("negative row counts must be rejected"), - TableCatalogStoreError::Invalid("manifest list field added_rows_count must be a non-negative long".to_string()) + assert_matches!( + avro_nullable_non_negative_i64(&value, "added_rows_count", "manifest list") + .expect_err("negative row counts must be rejected"), + TableCatalogStoreError::Invalid(message) if message == "manifest list field added_rows_count must be a non-negative long" ); } @@ -685,9 +686,9 @@ mod tests { Ok(_) => panic!("manifest partitions must preserve their record shape"), Err(error) => error, }; - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("manifest data file partition must be a record".to_string()) + TableCatalogStoreError::Invalid(message) if message == "manifest data file partition must be a record" ); } diff --git a/rustfs/src/table_catalog/iceberg/metadata.rs b/rustfs/src/table_catalog/iceberg/metadata.rs index d8061f441..74955f284 100644 --- a/rustfs/src/table_catalog/iceberg/metadata.rs +++ b/rustfs/src/table_catalog/iceberg/metadata.rs @@ -35,17 +35,6 @@ where .map(Some) } -pub(crate) fn metadata_log_locations( - current_metadata: &serde_json::Value, - table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, -) -> BTreeSet { - metadata_log_locations_matching(current_metadata, table_bucket, |location| { - is_valid_table_metadata_location(namespace, table, location) - }) -} - pub(crate) fn metadata_log_locations_matching( current_metadata: &serde_json::Value, table_bucket: &str, diff --git a/rustfs/src/table_catalog/maintenance/mod.rs b/rustfs/src/table_catalog/maintenance/mod.rs index e38b5a4de..723db7d8d 100644 --- a/rustfs/src/table_catalog/maintenance/mod.rs +++ b/rustfs/src/table_catalog/maintenance/mod.rs @@ -13,11 +13,14 @@ // limitations under the License. mod config; +mod periodic; +mod periodic_runtime; mod planner; mod recovery; mod worker; pub(crate) use config::*; +pub(crate) use periodic_runtime::{PeriodicMaintenanceConfig, PeriodicMaintenanceRuntime}; pub(crate) use planner::*; pub(crate) use recovery::*; pub(crate) use worker::*; diff --git a/rustfs/src/table_catalog/maintenance/periodic.rs b/rustfs/src/table_catalog/maintenance/periodic.rs new file mode 100644 index 000000000..9cfbdd4e0 --- /dev/null +++ b/rustfs/src/table_catalog/maintenance/periodic.rs @@ -0,0 +1,812 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::super::*; +use std::collections::BTreeSet; +use std::num::NonZeroUsize; +use std::time::Duration as StdDuration; +use tokio::time::Instant; +use tokio_util::sync::CancellationToken; + +const PERIODIC_MAINTENANCE_TABLES_PER_PASS: usize = 8; +const PERIODIC_MAINTENANCE_DISCOVERY_STEPS_PER_PASS: usize = 64; + +struct BucketCursor { + bucket: String, + namespace_cursor: Option, + namespace: Option, + table_cursor: Option, +} + +impl BucketCursor { + fn reset(&mut self) { + self.namespace_cursor = None; + self.namespace = None; + self.table_cursor = None; + } +} + +/// Queue-only admission. The caller owns this future until it returns: a +/// deadline stops new operations, never cancels a catalog mutation in flight. +pub(crate) struct PeriodicMaintenanceScheduler { + store: ConfiguredTableCatalogStore, + buckets: Vec, + next_bucket: usize, + interval: StdDuration, + scheduler_id: String, +} + +impl PeriodicMaintenanceScheduler { + /// Bucket names and the interval have already passed runtime configuration validation. + pub(crate) fn new(store: ConfiguredTableCatalogStore, buckets: Vec, interval: StdDuration) -> Self { + Self { + store, + buckets: buckets + .into_iter() + .map(|bucket| BucketCursor { + bucket, + namespace_cursor: None, + namespace: None, + table_cursor: None, + }) + .collect(), + next_bucket: 0, + interval, + scheduler_id: format!("periodic-{}", uuid::Uuid::new_v4()), + } + } + + pub(crate) async fn run_pass( + &mut self, + cancellation: &CancellationToken, + admission_deadline: Instant, + ) -> TableCatalogStoreResult<()> { + if self.buckets.is_empty() { + return Ok(()); + } + let mut visited = BTreeSet::new(); + let mut unavailable_buckets = BTreeSet::new(); + let mut first_error = None; + for _ in 0..PERIODIC_MAINTENANCE_DISCOVERY_STEPS_PER_PASS { + if !admission_open(cancellation, admission_deadline) || visited.len() == PERIODIC_MAINTENANCE_TABLES_PER_PASS { + break; + } + let bucket_index = self.next_bucket; + self.next_bucket = (bucket_index + 1) % self.buckets.len(); + if unavailable_buckets.contains(&bucket_index) { + continue; + } + let table = match self.discover_next_table(bucket_index, cancellation, admission_deadline).await { + Ok(Some(table)) => table, + Ok(None) => continue, + Err(error) => { + if matches!(&error, TableCatalogStoreError::NotFound(_)) { + // A namespace removed between pages must not pin this bucket's cursor forever. + self.buckets[bucket_index].reset(); + } + unavailable_buckets.insert(bucket_index); + first_error.get_or_insert(error); + continue; + } + }; + if !visited.insert((table.table_bucket.clone(), table.table_id.clone())) { + continue; + } + if let Err(error) = self.queue_if_eligible(&table, cancellation, admission_deadline).await { + // A removed/renamed table or one unhealthy bucket must not starve other buckets. + first_error.get_or_insert(error); + } + } + match first_error { + Some(error) => Err(error), + None => Ok(()), + } + } + + async fn discover_next_table( + &mut self, + bucket_index: usize, + cancellation: &CancellationToken, + admission_deadline: Instant, + ) -> TableCatalogStoreResult> { + let cursor = &mut self.buckets[bucket_index]; + if !admission_open(cancellation, admission_deadline) { + return Ok(None); + } + if self + .store + .get_table_bucket(&cursor.bucket) + .await? + .is_none_or(|bucket| bucket.state != TableCatalogEntryState::Active) + { + cursor.reset(); + return Ok(None); + } + if cursor.namespace.is_none() { + if !admission_open(cancellation, admission_deadline) { + return Ok(None); + } + let page = self + .store + .list_namespaces_page(&cursor.bucket, cursor.namespace_cursor.as_deref(), NonZeroUsize::MIN) + .await?; + cursor.namespace_cursor = page.next_cursor; + cursor.namespace = page.entries.into_iter().next().map(|namespace| namespace.namespace); + cursor.table_cursor = None; + } + let Some(namespace) = cursor.namespace.as_deref() else { + return Ok(None); + }; + if !admission_open(cancellation, admission_deadline) { + return Ok(None); + } + let page = self + .store + .list_tables_page(&cursor.bucket, namespace, cursor.table_cursor.as_deref(), NonZeroUsize::MIN) + .await?; + cursor.table_cursor = page.next_cursor; + if cursor.table_cursor.is_none() { + cursor.namespace = None; + } + // Object-backed discovery can return an empty page with a continuation. + Ok(page.entries.into_iter().next()) + } + + async fn queue_if_eligible( + &self, + table: &TableEntry, + cancellation: &CancellationToken, + admission_deadline: Instant, + ) -> TableCatalogStoreResult<()> { + if !admission_open(cancellation, admission_deadline) { + return Ok(()); + } + let scheduler = self + .store + .get_table_maintenance_scheduler_report(&table.table_bucket, &table.namespace, &table.table) + .await?; + if scheduler.table_id != table.table_id { + return Err(TableCatalogStoreError::Conflict( + "table identity changed during periodic maintenance discovery".to_string(), + )); + } + if !scheduler.background_enabled || scheduler.worker_paused || scheduler.quarantine.active { + return Ok(()); + } + if let Some(current) = scheduler.current_job.as_ref() { + if current.status != TableMetadataMaintenanceJobStatus::Successful { + // Periodic admission never renews an abandoned queue or resets a failed job's retry budget. + return Ok(()); + } + let finished_at = current + .finished_at + .as_deref() + .and_then(|timestamp| OffsetDateTime::parse(timestamp, &time::format_description::well_known::Rfc3339).ok()) + .ok_or_else(|| { + TableCatalogStoreError::Invalid( + "periodic maintenance requires a valid successful job completion timestamp".to_string(), + ) + })?; + let interval = time::Duration::try_from(self.interval) + .map_err(|_| TableCatalogStoreError::Invalid("periodic maintenance interval is out of range".to_string()))?; + let next_admission = finished_at.checked_add(interval).ok_or_else(|| { + TableCatalogStoreError::Invalid("periodic maintenance completion timestamp is out of range".to_string()) + })?; + if OffsetDateTime::now_utc() < next_admission { + return Ok(()); + } + } + if !admission_open(cancellation, admission_deadline) { + return Ok(()); + } + self.store + .queue_periodic_table_maintenance(table, self.scheduler_id.clone(), scheduler.current_job.as_ref()) + .await?; + Ok(()) + } +} + +fn admission_open(cancellation: &CancellationToken, deadline: Instant) -> bool { + !cancellation.is_cancelled() && Instant::now() < deadline +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::table_catalog::test_support::TestCatalogObjectBackend; + use crate::table_catalog::tests::{test_bucket_entry, test_namespace_entry, test_table_entry}; + use std::collections::BTreeMap; + + type TestStore = ConfiguredTableCatalogStore; + + fn backings() -> [TableCatalogBackingMode; 2] { + [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] + } + + fn enabled_config() -> TableMaintenanceConfig { + TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + ..Default::default() + } + } + + async fn seed_table( + store: &TestStore, + backend: &TestCatalogObjectBackend, + bucket: &str, + namespace: &str, + table: &str, + config: Option, + ) -> TableEntry { + let namespace = Namespace::parse(namespace).expect("fixture namespace"); + let table = IdentifierSegment::parse(table).expect("fixture table"); + if store.get_table_bucket(bucket).await.expect("read fixture bucket").is_none() { + store + .put_table_bucket(test_bucket_entry(bucket)) + .await + .expect("create fixture bucket"); + } + if store + .get_namespace(bucket, &namespace.public_name()) + .await + .expect("read fixture namespace") + .is_none() + { + store + .create_namespace(test_namespace_entry(bucket, &namespace)) + .await + .expect("create fixture namespace"); + } + let metadata = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let mut entry = test_table_entry(bucket, &namespace, &table, metadata.clone()); + entry.table_id = uuid::Uuid::new_v4().to_string(); + entry.table_uuid = entry.table_id.clone(); + entry.warehouse_location = format!("s3://{bucket}/tables/{}", entry.table_id); + store.create_table(entry.clone()).await.expect("create fixture table"); + backend + .seed_object(bucket, &metadata, br#"{"metadata-log":[]}"#.to_vec()) + .await; + if let Some(config) = config { + store + .put_table_maintenance_config(bucket, &namespace.public_name(), table.as_str(), config) + .await + .expect("configure fixture table"); + } + entry + } + + async fn current(store: &TestStore, table: &TableEntry) -> Option { + store + .get_table_metadata_maintenance_report( + &table.table_bucket, + &table.namespace, + &table.table, + MAINTENANCE_JOB_ALIAS_CURRENT, + ) + .await + .expect("read current fixture job") + } + + async fn seed_report(backend: &TestCatalogObjectBackend, table: &TableEntry, report: &TableMetadataMaintenanceReport) { + let namespace = Namespace::parse(&table.namespace).expect("fixture namespace"); + let name = IdentifierSegment::parse(&table.table).expect("fixture table"); + let paths = TableCatalogObjectPaths::default(); + for path in [ + paths.table_maintenance_job_path(&table.table_bucket, &namespace, &name, &table.table_id, &report.job.job_id), + paths.table_maintenance_latest_job_path(&table.table_bucket, &namespace, &name, &table.table_id), + paths.table_maintenance_current_job_path(&table.table_bucket, &namespace, &name, &table.table_id), + ] { + backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(report).expect("serialize fixture report")) + .await; + } + } + + async fn object_versions(backend: &TestCatalogObjectBackend) -> BTreeMap<(String, String), (Vec, String)> { + backend + .state + .lock() + .await + .objects + .iter() + .map(|(key, value)| (key.clone(), (value.data.clone(), value.etag.clone()))) + .collect() + } + + async fn run_pass(scheduler: &mut PeriodicMaintenanceScheduler) -> TableCatalogStoreResult<()> { + scheduler + .run_pass(&CancellationToken::new(), Instant::now() + StdDuration::from_secs(60)) + .await + } + + #[tokio::test] + async fn periodic_queue_is_idempotent_and_never_runs_delete_worker() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let table = seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let namespace = Namespace::parse(&table.namespace).expect("fixture namespace"); + let name = IdentifierSegment::parse(&table.table).expect("fixture table"); + let orphan = default_table_metadata_file_path(&namespace, &name, "00000.metadata.json"); + backend + .seed_object(&table.table_bucket, &orphan, br#"{"metadata-log":[]}"#.to_vec()) + .await; + let mut scheduler = + PeriodicMaintenanceScheduler::new(store.clone(), vec![table.table_bucket.clone()], StdDuration::from_secs(300)); + + run_pass(&mut scheduler).await.expect("first periodic pass queues work"); + let queued = current(&store, &table).await.expect("queued job"); + assert_eq!(queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(queued.job.operation, TableMetadataMaintenanceOperation::Delete); + assert!(queued.job.worker_id.is_none()); + assert_eq!(queued.job.attempt, 0); + assert!( + backend + .read_object(&table.table_bucket, &orphan) + .await + .expect("read orphan") + .is_some() + ); + assert!(backend.state.lock().await.delete_attempts.is_empty()); + assert_eq!( + store + .load_table(&table.table_bucket, &table.namespace, &table.table) + .await + .expect("read unchanged table"), + Some(table.clone()) + ); + + let versions = object_versions(&backend).await; + run_pass(&mut scheduler).await.expect("repeated pass skips queued work"); + assert_eq!( + object_versions(&backend).await, + versions, + "queue-only repetition must not rewrite history or aliases" + ); + } + } + + #[tokio::test] + async fn periodic_queue_respects_disabled_paused_and_inherited_configuration() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let disabled = seed_table(&store, &backend, "disabled", "sales", "orders", None).await; + let paused = seed_table( + &store, + &backend, + "paused", + "sales", + "orders", + Some(TableMaintenanceConfig { + worker_paused: true, + ..enabled_config() + }), + ) + .await; + let inherited = seed_table(&store, &backend, "inherited", "sales", "orders", None).await; + backend + .seed_object( + RUSTFS_META_BUCKET, + &TableCatalogObjectPaths::default().table_bucket_maintenance_config_path("inherited"), + serde_json::to_vec(&enabled_config()).expect("serialize bucket defaults"), + ) + .await; + let mut scheduler = PeriodicMaintenanceScheduler::new( + store.clone(), + vec!["disabled".to_string(), "paused".to_string(), "inherited".to_string()], + StdDuration::from_secs(300), + ); + run_pass(&mut scheduler).await.expect("configuration-aware pass"); + assert!( + current(&store, &disabled).await.is_none(), + "disabled tables must not gain control history" + ); + assert!(current(&store, &paused).await.is_none(), "paused tables must not gain control history"); + let queued = current(&store, &inherited) + .await + .expect("bucket defaults must enable admission"); + assert_eq!(queued.job.config_source, TableMaintenanceConfigSource::TableBucketDefault); + assert_eq!(queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + } + } + + #[tokio::test] + async fn periodic_queue_never_recovers_failed_expired_or_quarantined_jobs() { + for backing in backings() { + for status in [ + TableMetadataMaintenanceJobStatus::Failed, + TableMetadataMaintenanceJobStatus::Queued, + TableMetadataMaintenanceJobStatus::Running, + TableMetadataMaintenanceJobStatus::Successful, + ] { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let table = seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let mut report = store + .run_table_maintenance_scheduler_once("analytics", "sales", "orders", "fixture".to_string()) + .await + .expect("seed real queued report") + .report; + report.job.status = status.clone(); + report.job.attempt = 4; + report.job.max_retry_attempts = 3; + report.job.scheduled_at = Some(maintenance_timestamp(OffsetDateTime::UNIX_EPOCH)); + report.job.heartbeat_at = report.job.scheduled_at.clone(); + report.job.finished_at = report.job.scheduled_at.clone(); + report.job.next_retry_after = None; + if status == TableMetadataMaintenanceJobStatus::Successful { + report.job.quarantine_enabled = true; + report.job.quarantined_object_count = 1; + report.job.quarantine_retention_seconds = 3600; + } + seed_report(&backend, &table, &report).await; + let versions = object_versions(&backend).await; + let mut scheduler = + PeriodicMaintenanceScheduler::new(store.clone(), vec!["analytics".to_string()], StdDuration::from_secs(300)); + run_pass(&mut scheduler) + .await + .expect("manual recovery boundary is a normal skip"); + assert_eq!( + object_versions(&backend).await, + versions, + "periodic passes must preserve failed budgets, expired leases and quarantine" + ); + } + } + } + + #[tokio::test] + async fn periodic_queue_success_cooldown_requires_valid_persisted_timestamp() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let table = seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let mut report = store + .run_table_maintenance_scheduler_once("analytics", "sales", "orders", "fixture".to_string()) + .await + .expect("seed real report") + .report; + report.job.status = TableMetadataMaintenanceJobStatus::Successful; + report.job.finished_at = Some(maintenance_timestamp(OffsetDateTime::now_utc())); + seed_report(&backend, &table, &report).await; + let mut scheduler = + PeriodicMaintenanceScheduler::new(store.clone(), vec!["analytics".to_string()], StdDuration::from_secs(300)); + let versions = object_versions(&backend).await; + run_pass(&mut scheduler).await.expect("recent success is cooling down"); + assert_eq!(object_versions(&backend).await, versions); + for timestamp in [None, Some("not-a-timestamp".to_string())] { + report.job.finished_at = timestamp; + seed_report(&backend, &table, &report).await; + let versions = object_versions(&backend).await; + assert!(matches!(run_pass(&mut scheduler).await, Err(TableCatalogStoreError::Invalid(_)))); + assert_eq!(object_versions(&backend).await, versions, "invalid cooldown must fail without a new job"); + } + report.job.finished_at = Some(maintenance_timestamp(OffsetDateTime::UNIX_EPOCH)); + seed_report(&backend, &table, &report).await; + run_pass(&mut scheduler).await.expect("expired cooldown permits a new job"); + let queued = current(&store, &table).await.expect("next queue"); + assert_eq!(queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_ne!(queued.job.job_id, report.job.job_id); + } + } + + #[tokio::test] + async fn periodic_queue_pages_and_round_robins_buckets_across_passes() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let buckets = ["first", "second", "third"]; + let mut tables = Vec::new(); + for bucket in buckets { + for index in 0..5 { + tables.push( + seed_table( + &store, + &backend, + bucket, + if index < 2 { "sales" } else { "support" }, + &format!("table{index}"), + Some(enabled_config()), + ) + .await, + ); + } + } + let mut scheduler = PeriodicMaintenanceScheduler::new( + store.clone(), + buckets.iter().map(|bucket| (*bucket).to_string()).collect(), + StdDuration::from_secs(300), + ); + run_pass(&mut scheduler).await.expect("first bounded fair pass"); + let mut per_bucket = BTreeMap::new(); + for table in &tables { + if current(&store, table).await.is_some() { + *per_bucket.entry(table.table_bucket.clone()).or_insert(0_usize) += 1; + } + } + assert_eq!(per_bucket.values().sum::(), PERIODIC_MAINTENANCE_TABLES_PER_PASS); + assert_eq!(per_bucket.len(), buckets.len(), "one bucket must not consume every admission"); + assert!(per_bucket.values().all(|count| *count <= 3)); + run_pass(&mut scheduler).await.expect("resume all retained cursors"); + for table in &tables { + assert_eq!( + current(&store, table) + .await + .expect("every table reached through retained cursors") + .job + .status, + TableMetadataMaintenanceJobStatus::Queued + ); + } + } + } + + #[tokio::test] + async fn periodic_queue_cancelled_or_expired_pass_does_not_start_discovery() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let mut scheduler = + PeriodicMaintenanceScheduler::new(store, vec!["analytics".to_string()], StdDuration::from_secs(300)); + backend.reset_call_counts().await; + let cancelled = CancellationToken::new(); + cancelled.cancel(); + scheduler + .run_pass(&cancelled, Instant::now() + StdDuration::from_secs(60)) + .await + .expect("cancelled pass"); + scheduler + .run_pass(&CancellationToken::new(), Instant::now()) + .await + .expect("expired pass"); + assert_eq!(backend.read_call_count().await, 0); + assert_eq!(backend.list_call_count().await, 0); + } + } + + #[tokio::test(start_paused = true)] + async fn periodic_queue_retains_inflight_publication_after_cancellation_and_deadline() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let first = seed_table(&store, &backend, "analytics", "sales", "a", Some(enabled_config())).await; + let second = seed_table(&store, &backend, "analytics", "sales", "b", Some(enabled_config())).await; + let namespace = Namespace::parse(&first.namespace).expect("fixture namespace"); + let name = IdentifierSegment::parse(&first.table).expect("fixture table"); + let path = TableCatalogObjectPaths::default().table_maintenance_current_job_path( + "analytics", + &namespace, + &name, + &first.table_id, + ); + let pause = backend.pause_next_put(RUSTFS_META_BUCKET, &path).await; + let mut scheduler = + PeriodicMaintenanceScheduler::new(store.clone(), vec!["analytics".to_string()], StdDuration::from_secs(300)); + let cancellation = CancellationToken::new(); + let runner_cancellation = cancellation.clone(); + let deadline = Instant::now() + StdDuration::from_secs(30); + let task = tokio::spawn(async move { scheduler.run_pass(&runner_cancellation, deadline).await }); + tokio::time::timeout(StdDuration::from_secs(5), pause.wait_started()) + .await + .expect("queue reaches publication pause"); + cancellation.cancel(); + tokio::time::advance(StdDuration::from_secs(31)).await; + assert!( + !task.is_finished(), + "cancellation must retain the mutating future until its write returns" + ); + pause.release(); + task.await.expect("runner task").expect("in-flight queue completes"); + assert_eq!( + current(&store, &first).await.expect("complete first publication").job.status, + TableMetadataMaintenanceJobStatus::Queued + ); + assert!( + current(&store, &second).await.is_none(), + "no operation starts after cancellation or admission deadline" + ); + } + } + + #[tokio::test] + async fn periodic_queue_atomic_guard_rejects_a_claim_changed_during_planning() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let table = seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let pause = backend.pause_next_read(&table.table_bucket, &table.metadata_location).await; + let claimant = store.clone(); + let observed = table.clone(); + let task = tokio::spawn(async move { + claimant + .queue_periodic_table_maintenance(&observed, "first".to_string(), None) + .await + }); + tokio::time::timeout(StdDuration::from_secs(5), pause.wait_started()) + .await + .expect("first claimant starts unlocked planning"); + let winner = store + .queue_periodic_table_maintenance(&table, "second".to_string(), None) + .await + .expect("second claimant wins while first is planning"); + let versions = object_versions(&backend).await; + pause.release(); + assert!(matches!( + task.await.expect("first claimant task"), + Err(TableCatalogStoreError::Conflict(_)) + )); + assert_eq!( + current(&store, &table).await.expect("winning current job").job.job_id, + winner.report.job.job_id + ); + assert_eq!(object_versions(&backend).await, versions, "a stale claim must not publish any sidecar"); + assert_eq!( + store + .load_table(&table.table_bucket, &table.namespace, &table.table) + .await + .expect("read table"), + Some(table) + ); + } + } + + #[tokio::test] + async fn periodic_queue_atomic_guard_rejects_changed_identity_config_and_partial_aliases() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let table = seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let mut old_identity = table.clone(); + old_identity.table_id = uuid::Uuid::new_v4().to_string(); + let versions = object_versions(&backend).await; + assert!(matches!( + store + .queue_periodic_table_maintenance(&old_identity, "periodic".to_string(), None) + .await, + Err(TableCatalogStoreError::Conflict(_)) + )); + assert_eq!(object_versions(&backend).await, versions); + + for config in [ + TableMaintenanceConfig::default(), + TableMaintenanceConfig { + worker_paused: true, + ..enabled_config() + }, + ] { + store + .put_table_maintenance_config("analytics", "sales", "orders", config) + .await + .expect("change config after selection"); + let versions = object_versions(&backend).await; + assert!(matches!( + store + .queue_periodic_table_maintenance(&table, "periodic".to_string(), None) + .await, + Err(TableCatalogStoreError::Conflict(_)) + )); + assert_eq!( + object_versions(&backend).await, + versions, + "changed config must not cause Disabled or Paused control writes" + ); + } + store + .put_table_maintenance_config("analytics", "sales", "orders", enabled_config()) + .await + .expect("restore fixture config"); + let mut report = store + .run_table_maintenance_scheduler_once("analytics", "sales", "orders", "fixture".to_string()) + .await + .expect("seed full report") + .report; + report.job.status = TableMetadataMaintenanceJobStatus::Successful; + report.job.finished_at = Some(maintenance_timestamp(OffsetDateTime::UNIX_EPOCH)); + seed_report(&backend, &table, &report).await; + let expected = table_maintenance_scheduler_job_summary(&report); + report.job.status = TableMetadataMaintenanceJobStatus::Failed; + let namespace = Namespace::parse(&table.namespace).expect("fixture namespace"); + let name = IdentifierSegment::parse(&table.table).expect("fixture table"); + let latest = TableCatalogObjectPaths::default().table_maintenance_latest_job_path( + "analytics", + &namespace, + &name, + &table.table_id, + ); + backend + .seed_object( + RUSTFS_META_BUCKET, + &latest, + serde_json::to_vec(&report).expect("serialize partial publication"), + ) + .await; + let versions = object_versions(&backend).await; + assert!(matches!( + store + .queue_periodic_table_maintenance(&table, "periodic".to_string(), Some(&expected)) + .await, + Err(TableCatalogStoreError::Conflict(_)) + )); + assert_eq!( + object_versions(&backend).await, + versions, + "partial report publication requires explicit recovery" + ); + } + } + + #[tokio::test] + async fn periodic_queue_empty_discovery_is_bounded_and_inactive_buckets_are_skipped() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + store + .put_table_bucket(test_bucket_entry("empty")) + .await + .expect("empty active bucket"); + let mut inactive = test_bucket_entry("inactive"); + inactive.state = TableCatalogEntryState::Deleted; + store.put_table_bucket(inactive).await.expect("inactive fixture bucket"); + backend.reset_call_counts().await; + let versions = object_versions(&backend).await; + let mut scheduler = PeriodicMaintenanceScheduler::new( + store, + vec!["empty".to_string(), "inactive".to_string(), "missing".to_string()], + StdDuration::from_secs(300), + ); + run_pass(&mut scheduler) + .await + .expect("bounded discovery with no eligible tables"); + assert_eq!(object_versions(&backend).await, versions); + assert!( + backend.list_call_count().await <= PERIODIC_MAINTENANCE_DISCOVERY_STEPS_PER_PASS, + "empty discovery must stop at the pass budget" + ); + assert_eq!(scheduler.next_bucket, PERIODIC_MAINTENANCE_DISCOVERY_STEPS_PER_PASS % 3); + } + } + + #[tokio::test] + async fn periodic_queue_preserves_empty_object_page_continuations() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), TableCatalogBackingMode::ObjectBacked); + let table = seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let prefix = TableCatalogObjectPaths::default().namespace_entries_prefix("analytics"); + for index in 0..TABLE_CATALOG_LIST_MAX_KEYS { + backend + .seed_object(RUSTFS_META_BUCKET, &format!("{prefix}!non-entry-{index:04}"), b"fixture".to_vec()) + .await; + } + let first_page = store + .list_namespaces_page("analytics", None, NonZeroUsize::MIN) + .await + .expect("bounded empty namespace page"); + assert!(first_page.entries.is_empty()); + assert!(first_page.next_cursor.is_some()); + let mut scheduler = + PeriodicMaintenanceScheduler::new(store.clone(), vec!["analytics".to_string()], StdDuration::from_secs(300)); + run_pass(&mut scheduler).await.expect("continue past an empty page"); + assert_eq!( + current(&store, &table) + .await + .expect("table after the empty page must be reached") + .job + .status, + TableMetadataMaintenanceJobStatus::Queued + ); + } +} diff --git a/rustfs/src/table_catalog/maintenance/periodic_runtime.rs b/rustfs/src/table_catalog/maintenance/periodic_runtime.rs new file mode 100644 index 000000000..088063efc --- /dev/null +++ b/rustfs/src/table_catalog/maintenance/periodic_runtime.rs @@ -0,0 +1,339 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::periodic::PeriodicMaintenanceScheduler; +use crate::storage_api::startup::services::ServerContextSlot; +use crate::storage_api::table::check_valid_bucket_name_strict; +use crate::table_catalog::{ + ConfiguredTableCatalogStore, EcStoreTableCatalogObjectBackend, ObjectTableCatalogStore, StrongTableCatalogStore, + TableCatalogBackingMode, TableCatalogStoreError, TableCatalogStoreResult, +}; +use std::{collections::BTreeSet, future::Future, sync::Arc, time::Duration}; +use tokio::{task::JoinHandle, time::Instant}; +use tokio_util::sync::CancellationToken; +use tracing::{info, warn}; + +const ENV_ENABLED: &str = "RUSTFS_TABLE_MAINTENANCE_SCHEDULER_ENABLED"; +const ENV_BUCKETS: &str = "RUSTFS_TABLE_MAINTENANCE_SCHEDULER_BUCKETS"; +const ENV_INTERVAL: &str = "RUSTFS_TABLE_MAINTENANCE_SCHEDULER_INTERVAL_SECONDS"; +const ADMISSION_BUDGET: Duration = Duration::from_secs(30); +const LOG_COMPONENT: &str = "table_catalog"; +const LOG_SUBSYSTEM: &str = "maintenance"; +const EVENT_PERIODIC_SCHEDULER: &str = "table_maintenance_periodic_scheduler"; + +pub(crate) struct PeriodicMaintenanceConfig { + buckets: Vec, + interval: Duration, + backing: TableCatalogBackingMode, +} + +impl PeriodicMaintenanceConfig { + pub(crate) fn from_env() -> TableCatalogStoreResult> { + Self::parse( + |key| match std::env::var(key) { + Ok(value) => Ok(Some(value)), + Err(std::env::VarError::NotPresent) => Ok(None), + Err(std::env::VarError::NotUnicode(_)) => { + Err(TableCatalogStoreError::Invalid(format!("{key} must be valid UTF-8"))) + } + }, + TableCatalogBackingMode::from_env, + ) + } + + fn parse( + read: impl Fn(&str) -> TableCatalogStoreResult>, + backing: impl FnOnce() -> TableCatalogStoreResult, + ) -> TableCatalogStoreResult> { + match read(ENV_ENABLED)?.as_deref().map(str::trim) { + None | Some("false") => return Ok(None), + Some("true") => {} + _ => return Err(TableCatalogStoreError::Invalid(format!("{ENV_ENABLED} must be true or false"))), + } + let raw = read(ENV_BUCKETS)? + .ok_or_else(|| TableCatalogStoreError::Invalid(format!("{ENV_BUCKETS} is required when scheduling is enabled")))?; + if raw.len() > 2048 { + return Err(TableCatalogStoreError::Invalid(format!("{ENV_BUCKETS} exceeds the allowlist size limit"))); + } + let mut buckets = BTreeSet::new(); + for bucket in raw.split(',').map(str::trim) { + check_valid_bucket_name_strict(bucket) + .map_err(|_| TableCatalogStoreError::Invalid(format!("{ENV_BUCKETS} contains an invalid bucket name")))?; + if !buckets.insert(bucket.to_string()) || buckets.len() > 32 { + return Err(TableCatalogStoreError::Invalid(format!( + "{ENV_BUCKETS} requires 1 to 32 distinct bucket names" + ))); + } + } + let interval = read(ENV_INTERVAL)?.unwrap_or_else(|| "300".to_string()); + let seconds = interval + .trim() + .parse::() + .ok() + .filter(|seconds| (30..=86_400).contains(seconds)) + .ok_or_else(|| TableCatalogStoreError::Invalid(format!("{ENV_INTERVAL} must be between 30 and 86400")))?; + Ok(Some(Self { + buckets: buckets.into_iter().collect(), + interval: Duration::from_secs(seconds), + backing: backing()?, + })) + } +} + +/// Owns queueing until completion; never runs a worker or deletes table objects. +pub(crate) struct PeriodicMaintenanceRuntime { + cancellation: CancellationToken, + task: Option>, +} + +impl Drop for PeriodicMaintenanceRuntime { + fn drop(&mut self) { + // Detach, never abort: in-flight sidecar writes retain their publication owner. + self.cancellation.cancel(); + } +} + +impl PeriodicMaintenanceRuntime { + pub(crate) fn spawn(config: PeriodicMaintenanceConfig, server: Arc, shutdown: &CancellationToken) -> Self { + let cancellation = shutdown.child_token(); + let task = tokio::spawn(run(config, server, cancellation.clone())); + Self { + cancellation, + task: Some(task), + } + } + + pub(crate) async fn shutdown(mut self, grace: Duration) -> bool { + self.cancellation.cancel(); + let Some(task) = self.task.take() else { + return true; + }; + let joined = matches!(tokio::time::timeout(grace, task).await, Ok(Ok(()))); + if !joined { + warn!( + event = EVENT_PERIODIC_SCHEDULER, + component = LOG_COMPONENT, + subsystem = LOG_SUBSYSTEM, + state = "drain_unconfirmed", + "Maintenance scheduler completion is unconfirmed; owned work was not aborted" + ); + } + joined + } +} + +fn tick_delay(interval: Duration) -> Duration { + // Delay after completion avoids catch-up bursts; jitter spreads node scans. + interval + Duration::from_secs(rand::random_range(0..=interval.as_secs() / 10)) +} + +async fn wait_for_tick(cancellation: &CancellationToken, delay: Duration) -> bool { + tokio::select! { + biased; + _ = cancellation.cancelled() => false, + _ = tokio::time::sleep(delay) => true, + } +} + +async fn finish_owned_pass(pass: F, cancellation: &CancellationToken, deadline: Instant) -> F::Output { + tokio::pin!(pass); + tokio::select! { + biased; + result = &mut pass => result, + _ = cancellation.cancelled() => pass.await, + _ = tokio::time::sleep_until(deadline) => { + warn!( + event = EVENT_PERIODIC_SCHEDULER, component = LOG_COMPONENT, subsystem = LOG_SUBSYSTEM, + state = "draining", reason = "admission_deadline", + "Maintenance scheduler stopped admitting work while its store call drains" + ); + pass.await + } + } +} + +async fn run(config: PeriodicMaintenanceConfig, server: Arc, cancellation: CancellationToken) { + let mut scheduler = None; + let mut degraded = false; + while wait_for_tick(&cancellation, tick_delay(config.interval)).await { + if scheduler.is_none() { + // Deferred IAM may not have installed this server yet. Do not use + // another embedded server's ambient context as a fallback. + let Some(context) = server.installed_app_context() else { + continue; + }; + let backend = EcStoreTableCatalogObjectBackend::new_with_strong_runtime( + context.object_store(), + context.table_catalog_strong_runtime(), + ); + let store = match config.backing { + TableCatalogBackingMode::ObjectBacked => { + ConfiguredTableCatalogStore::ObjectBacked(ObjectTableCatalogStore::new(backend)) + } + TableCatalogBackingMode::DurableStrong => { + ConfiguredTableCatalogStore::DurableStrong(StrongTableCatalogStore::new_requiring_snapshot(backend)) + } + }; + scheduler = Some(PeriodicMaintenanceScheduler::new(store, config.buckets.clone(), config.interval)); + } + let deadline = Instant::now() + ADMISSION_BUDGET; + let Some(scheduler) = scheduler.as_mut() else { + continue; + }; + let result = finish_owned_pass(scheduler.run_pass(&cancellation, deadline), &cancellation, deadline).await; + match result { + Ok(()) if degraded => { + info!( + event = EVENT_PERIODIC_SCHEDULER, + component = LOG_COMPONENT, + subsystem = LOG_SUBSYSTEM, + state = "recovered", + "Maintenance scheduler pass recovered" + ); + degraded = false; + } + Err(_) if !degraded => { + warn!( + event = EVENT_PERIODIC_SCHEDULER, + component = LOG_COMPONENT, + subsystem = LOG_SUBSYSTEM, + state = "degraded", + reason = "queue_pass_failed", + "Maintenance scheduler could not complete a queueing pass" + ); + degraded = true; + } + _ => {} + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::atomic::{AtomicBool, Ordering}; + + fn config(values: &[(&str, &str)]) -> TableCatalogStoreResult> { + PeriodicMaintenanceConfig::parse( + |key| { + Ok(values + .iter() + .find(|(name, _)| *name == key) + .map(|(_, value)| value.to_string())) + }, + || Ok(TableCatalogBackingMode::DurableStrong), + ) + } + + #[test] + fn periodic_config_is_opt_in_and_validates_limits() { + assert!(config(&[]).unwrap().is_none()); + assert!( + config(&[(ENV_ENABLED, "false"), (ENV_BUCKETS, "invalid/path")]) + .unwrap() + .is_none() + ); + for value in ["", "1", "yes", "TRUE"] { + assert!(config(&[(ENV_ENABLED, value)]).is_err()); + } + assert!(config(&[(ENV_ENABLED, "true")]).is_err()); + for buckets in ["", "foo,", "foo,foo", "../foo", "127.0.0.1", "Upper", ".minio.sys"] { + assert!(config(&[(ENV_ENABLED, "true"), (ENV_BUCKETS, buckets)]).is_err(), "{buckets}"); + } + let too_many = (0..33).map(|n| format!("bucket-{n}")).collect::>().join(","); + assert!(config(&[(ENV_ENABLED, "true"), (ENV_BUCKETS, &too_many)]).is_err()); + for interval in ["0", "29", "86401", "-1", "NaN", "18446744073709551616"] { + assert!(config(&[(ENV_ENABLED, "true"), (ENV_BUCKETS, "foo"), (ENV_INTERVAL, interval)]).is_err()); + } + let parsed = config(&[(ENV_ENABLED, "true"), (ENV_BUCKETS, "bbb, aaa")]).unwrap().unwrap(); + assert_eq!(parsed.buckets, ["aaa", "bbb"]); + assert_eq!(parsed.interval, Duration::from_secs(300)); + for _ in 0..100 { + assert!((Duration::from_secs(300)..=Duration::from_secs(330)).contains(&tick_delay(parsed.interval))); + } + } + + #[tokio::test(start_paused = true)] + async fn periodic_tick_waits_and_cancels_without_work() { + let cancellation = CancellationToken::new(); + let start = Instant::now(); + assert!(wait_for_tick(&cancellation, Duration::from_secs(300)).await); + assert_eq!(Instant::now() - start, Duration::from_secs(300)); + cancellation.cancel(); + assert!(!wait_for_tick(&cancellation, Duration::from_secs(300)).await); + assert_eq!(Instant::now() - start, Duration::from_secs(300)); + } + + #[tokio::test(start_paused = true)] + async fn periodic_deadline_and_shutdown_retain_owned_work() { + for cancel in [false, true] { + let cancellation = CancellationToken::new(); + let completed = Arc::new(AtomicBool::new(false)); + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, resume) = tokio::sync::oneshot::channel(); + let done = completed.clone(); + let token = cancellation.clone(); + let task = tokio::spawn(async move { + finish_owned_pass( + async move { + entered.send(()).unwrap(); + resume.await.unwrap(); + done.store(true, Ordering::SeqCst); + }, + &token, + Instant::now() + Duration::from_secs(30), + ) + .await; + }); + started.await.unwrap(); + if cancel { + cancellation.cancel(); + } else { + tokio::time::advance(Duration::from_secs(31)).await; + } + tokio::task::yield_now().await; + assert!(!task.is_finished()); + assert!(!completed.load(Ordering::SeqCst)); + release.send(()).unwrap(); + task.await.unwrap(); + assert!(completed.load(Ordering::SeqCst)); + } + } + + #[tokio::test(start_paused = true)] + async fn periodic_shutdown_reports_incomplete_drain_without_aborting() { + let completed = Arc::new(AtomicBool::new(false)); + let done = completed.clone(); + let (release, resume) = tokio::sync::oneshot::channel(); + let (finished, observed) = tokio::sync::oneshot::channel(); + let runtime = PeriodicMaintenanceRuntime { + cancellation: CancellationToken::new(), + task: Some(tokio::spawn(async move { + resume.await.unwrap(); + done.store(true, Ordering::SeqCst); + finished.send(()).unwrap(); + })), + }; + assert!(!runtime.shutdown(Duration::from_secs(30)).await); + assert!(!completed.load(Ordering::SeqCst)); + release.send(()).unwrap(); + observed.await.unwrap(); + assert!(completed.load(Ordering::SeqCst)); + let runtime = PeriodicMaintenanceRuntime { + cancellation: CancellationToken::new(), + task: Some(tokio::spawn(async {})), + }; + assert!(runtime.shutdown(Duration::from_secs(30)).await); + } +} diff --git a/rustfs/src/table_catalog/maintenance/planner.rs b/rustfs/src/table_catalog/maintenance/planner.rs index e6b39354e..fdc6afaf0 100644 --- a/rustfs/src/table_catalog/maintenance/planner.rs +++ b/rustfs/src/table_catalog/maintenance/planner.rs @@ -213,9 +213,7 @@ where Some(manifest_list) => { let planning = match compaction_data_file_candidates( backend, - table_bucket, - namespace, - table, + entry, warehouse_object_prefix.as_deref(), manifest_list, &config, @@ -326,9 +324,7 @@ where async fn compaction_data_file_candidates( backend: &B, - table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, + entry: &TableEntry, warehouse_object_prefix: Option<&str>, manifest_list: &str, config: &TableCompactionPlanningConfig, @@ -336,12 +332,13 @@ async fn compaction_data_file_candidates( where B: TableCatalogObjectBackend, { + let table_bucket = &entry.table_bucket; let Some(manifest_list_key) = table_catalog_object_key_from_location(table_bucket, manifest_list) else { return Err(TableCatalogStoreError::Invalid( "compaction manifest list must be inside the table bucket".to_string(), )); }; - if table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &manifest_list_key) + if table_maintenance_object_kind_for_entry(entry, warehouse_object_prefix, &manifest_list_key) != Some(TableMetadataMaintenanceObjectKind::ManifestList) { return Err(TableCatalogStoreError::Invalid( @@ -367,7 +364,7 @@ where "compaction manifest must be inside the table bucket".to_string(), )); }; - if table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &manifest_key) + if table_maintenance_object_kind_for_entry(entry, warehouse_object_prefix, &manifest_key) != Some(TableMetadataMaintenanceObjectKind::ManifestFile) { return Err(TableCatalogStoreError::Invalid( @@ -384,9 +381,7 @@ where if reference.object_kind != TableMetadataMaintenanceObjectKind::DataFile { record_compaction_row_level_delete_file( backend, - table_bucket, - namespace, - table, + entry, warehouse_object_prefix, &mut planning.row_level_planning, &reference, @@ -400,7 +395,7 @@ where "compaction data file must be inside the table bucket".to_string(), )); }; - if table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &data_key) + if table_maintenance_object_kind_for_entry(entry, warehouse_object_prefix, &data_key) != Some(TableMetadataMaintenanceObjectKind::DataFile) { return Err(TableCatalogStoreError::Invalid( @@ -413,7 +408,7 @@ where let size_bytes = u64::try_from(data_object.data.len()).unwrap_or(u64::MAX); if size_bytes <= config.small_file_threshold_bytes { planning.candidates.push(TableCompactionDataFileCandidate { - rewrite_prefix: compaction_data_file_rewrite_prefix(namespace, table, warehouse_object_prefix, &data_key) + rewrite_prefix: compaction_data_file_rewrite_prefix_for_entry(entry, warehouse_object_prefix, &data_key) .unwrap_or_else(|| data_key.clone()), location: data_key, size_bytes, @@ -427,9 +422,7 @@ where async fn record_compaction_row_level_delete_file( backend: &B, - table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, + entry: &TableEntry, warehouse_object_prefix: Option<&str>, planning: &mut TableRowLevelMaintenancePlanningReport, reference: &ManifestDataFileReference, @@ -437,6 +430,7 @@ async fn record_compaction_row_level_delete_file( where B: TableCatalogObjectBackend, { + let table_bucket = &entry.table_bucket; let Some(content) = row_level_delete_file_content(reference.content) else { return Ok(()); }; @@ -445,7 +439,7 @@ where "compaction delete file must be inside the table bucket".to_string(), )); }; - if table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &delete_key) + if table_maintenance_object_kind_for_entry(entry, warehouse_object_prefix, &delete_key) != Some(TableMetadataMaintenanceObjectKind::DeleteFile) { return Err(TableCatalogStoreError::Invalid( @@ -714,6 +708,29 @@ pub(crate) fn compaction_data_file_rewrite_prefix( None } +fn compaction_data_file_rewrite_prefix_for_entry( + entry: &TableEntry, + warehouse_object_prefix: Option<&str>, + location: &str, +) -> Option { + let metadata_dir = table_metadata_dir_path_for_entry(entry).ok()?; + let table_root = metadata_dir.strip_suffix(&format!("/{METADATA_DIR}"))?; + let entry_data_prefix = format!("{table_root}/{DATA_DIR}/"); + let warehouse_data_prefix = warehouse_object_prefix + .map(|prefix| format!("{prefix}{DATA_DIR}")) + .unwrap_or_else(|| format!("{table_root}/{DATA_DIR}")); + if let Some(relative_path) = location.strip_prefix(&entry_data_prefix) { + return Some(compaction_data_file_output_prefix(&warehouse_data_prefix, relative_path)); + } + if let Some(warehouse_object_prefix) = warehouse_object_prefix { + let warehouse_input_prefix = format!("{warehouse_object_prefix}{DATA_DIR}/"); + if let Some(relative_path) = location.strip_prefix(&warehouse_input_prefix) { + return Some(compaction_data_file_output_prefix(&warehouse_data_prefix, relative_path)); + } + } + None +} + fn compaction_data_file_output_prefix(output_data_prefix: &str, relative_path: &str) -> String { relative_path .rsplit_once('/') diff --git a/rustfs/src/table_catalog/maintenance/recovery.rs b/rustfs/src/table_catalog/maintenance/recovery.rs index fc39cebec..a60b9600c 100644 --- a/rustfs/src/table_catalog/maintenance/recovery.rs +++ b/rustfs/src/table_catalog/maintenance/recovery.rs @@ -72,6 +72,51 @@ fn insert_referenced_object_report( report.reasons.insert(reason); } +#[derive(Clone, Copy)] +enum TableMaintenanceObjectScope<'a> { + Logical { + namespace: &'a Namespace, + table: &'a IdentifierSegment, + }, + Entry(&'a TableEntry), +} + +impl TableMaintenanceObjectScope<'_> { + fn object_kind( + self, + warehouse_object_prefix: Option<&str>, + object_location: &str, + ) -> Option { + match self { + Self::Logical { namespace, table } => { + table_maintenance_object_kind(namespace, table, warehouse_object_prefix, object_location) + } + Self::Entry(entry) => table_maintenance_object_kind_for_entry(entry, warehouse_object_prefix, object_location), + } + } + + fn object_directories(self) -> TableCatalogStoreResult<(String, String, String)> { + match self { + Self::Logical { namespace, table } => Ok(( + default_table_metadata_dir_path(namespace, table), + default_table_data_dir_path(namespace, table), + default_table_delete_dir_path(namespace, table), + )), + Self::Entry(entry) => { + let metadata_dir = table_metadata_dir_path_for_entry(entry)?; + let table_root = metadata_dir.strip_suffix(&format!("/{METADATA_DIR}")).ok_or_else(|| { + TableCatalogStoreError::Invalid( + "current metadata location must be inside a table metadata directory".to_string(), + ) + })?; + let data_dir = format!("{table_root}/{DATA_DIR}"); + let delete_dir = format!("{table_root}/{DELETE_DIR}"); + Ok((metadata_dir, data_dir, delete_dir)) + } + } + } +} + pub(crate) async fn metadata_maintenance_referenced_object_reports( backend: &B, table_bucket: &str, @@ -81,6 +126,50 @@ pub(crate) async fn metadata_maintenance_referenced_object_reports( current_metadata: &serde_json::Value, retained_metadata_locations: &[String], ) -> TableCatalogStoreResult> +where + B: TableCatalogObjectBackend, +{ + metadata_maintenance_referenced_object_reports_in_scope( + backend, + table_bucket, + TableMaintenanceObjectScope::Logical { namespace, table }, + warehouse_object_prefix, + current_metadata, + retained_metadata_locations, + ) + .await +} + +pub(crate) async fn metadata_maintenance_referenced_object_reports_for_entry( + backend: &B, + entry: &TableEntry, + warehouse_object_prefix: Option<&str>, + current_metadata: &serde_json::Value, + retained_metadata_locations: &[String], +) -> TableCatalogStoreResult> +where + B: TableCatalogObjectBackend, +{ + table_metadata_dir_path_for_entry(entry)?; + metadata_maintenance_referenced_object_reports_in_scope( + backend, + &entry.table_bucket, + TableMaintenanceObjectScope::Entry(entry), + warehouse_object_prefix, + current_metadata, + retained_metadata_locations, + ) + .await +} + +async fn metadata_maintenance_referenced_object_reports_in_scope( + backend: &B, + table_bucket: &str, + scope: TableMaintenanceObjectScope<'_>, + warehouse_object_prefix: Option<&str>, + current_metadata: &serde_json::Value, + retained_metadata_locations: &[String], +) -> TableCatalogStoreResult> where B: TableCatalogObjectBackend, { @@ -88,8 +177,7 @@ where metadata_maintenance_referenced_object_reports_for_metadata( backend, table_bucket, - namespace, - table, + scope, warehouse_object_prefix, current_metadata, &mut reports, @@ -114,8 +202,7 @@ where metadata_maintenance_referenced_object_reports_for_metadata( backend, table_bucket, - namespace, - table, + scope, warehouse_object_prefix, &metadata, &mut reports, @@ -137,8 +224,7 @@ where async fn metadata_maintenance_referenced_object_reports_for_metadata( backend: &B, table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, + scope: TableMaintenanceObjectScope<'_>, warehouse_object_prefix: Option<&str>, metadata: &serde_json::Value, reports: &mut BTreeMap, @@ -155,8 +241,7 @@ where metadata_maintenance_referenced_manifest_list( backend, table_bucket, - namespace, - table, + scope, warehouse_object_prefix, manifest_list_location, reports, @@ -182,8 +267,7 @@ where metadata_maintenance_referenced_manifest_file( backend, table_bucket, - namespace, - table, + scope, warehouse_object_prefix, manifest_location, reports, @@ -198,8 +282,7 @@ where async fn metadata_maintenance_referenced_manifest_list( backend: &B, table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, + scope: TableMaintenanceObjectScope<'_>, warehouse_object_prefix: Option<&str>, manifest_list_location: &str, reports: &mut BTreeMap, @@ -217,9 +300,12 @@ where ); return Ok(()); }; - if !table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &manifest_list_key).is_some_and(|kind| { - table_maintenance_object_kind_matches_reference(&kind, &TableMetadataMaintenanceObjectKind::ManifestList) - }) { + if !scope + .object_kind(warehouse_object_prefix, &manifest_list_key) + .is_some_and(|kind| { + table_maintenance_object_kind_matches_reference(&kind, &TableMetadataMaintenanceObjectKind::ManifestList) + }) + { insert_referenced_object_report( reports, manifest_list_key, @@ -272,8 +358,7 @@ where metadata_maintenance_referenced_manifest_file( backend, table_bucket, - namespace, - table, + scope, warehouse_object_prefix, &manifest_location, reports, @@ -287,8 +372,7 @@ where async fn metadata_maintenance_referenced_manifest_file( backend: &B, table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, + scope: TableMaintenanceObjectScope<'_>, warehouse_object_prefix: Option<&str>, manifest_location: &str, reports: &mut BTreeMap, @@ -306,7 +390,7 @@ where ); return Ok(()); }; - if !table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &manifest_key).is_some_and(|kind| { + if !scope.object_kind(warehouse_object_prefix, &manifest_key).is_some_and(|kind| { table_maintenance_object_kind_matches_reference(&kind, &TableMetadataMaintenanceObjectKind::ManifestFile) }) { insert_referenced_object_report( @@ -360,7 +444,8 @@ where ); continue; }; - if !table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &file_key) + if !scope + .object_kind(warehouse_object_prefix, &file_key) .is_some_and(|kind| table_maintenance_object_kind_matches_reference(&kind, &object_kind)) { insert_referenced_object_report( @@ -643,8 +728,50 @@ pub(crate) async fn metadata_maintenance_object_cleanup_reports( where B: TableCatalogObjectBackend, { - let scanned_objects = - table_maintenance_cleanup_objects(backend, table_bucket, namespace, table, warehouse_object_prefix).await?; + metadata_maintenance_object_cleanup_reports_in_scope( + backend, + table_bucket, + TableMaintenanceObjectScope::Logical { namespace, table }, + warehouse_object_prefix, + referenced_object_reports, + now, + ) + .await +} + +pub(crate) async fn metadata_maintenance_object_cleanup_reports_for_entry( + backend: &B, + entry: &TableEntry, + warehouse_object_prefix: Option<&str>, + referenced_object_reports: &[TableMetadataMaintenanceReferencedObjectReport], + now: OffsetDateTime, +) -> TableCatalogStoreResult<(usize, Vec, Vec, Vec)> +where + B: TableCatalogObjectBackend, +{ + metadata_maintenance_object_cleanup_reports_in_scope( + backend, + &entry.table_bucket, + TableMaintenanceObjectScope::Entry(entry), + warehouse_object_prefix, + referenced_object_reports, + now, + ) + .await +} + +async fn metadata_maintenance_object_cleanup_reports_in_scope( + backend: &B, + table_bucket: &str, + scope: TableMaintenanceObjectScope<'_>, + warehouse_object_prefix: Option<&str>, + referenced_object_reports: &[TableMetadataMaintenanceReferencedObjectReport], + now: OffsetDateTime, +) -> TableCatalogStoreResult<(usize, Vec, Vec, Vec)> +where + B: TableCatalogObjectBackend, +{ + let scanned_objects = table_maintenance_cleanup_objects(backend, table_bucket, scope, warehouse_object_prefix).await?; if referenced_object_reports .iter() .any(|report| report.state == TableMetadataMaintenanceObjectState::ManualReviewRequired) @@ -700,17 +827,17 @@ where async fn table_maintenance_cleanup_objects( backend: &B, table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, + scope: TableMaintenanceObjectScope<'_>, warehouse_object_prefix: Option<&str>, ) -> TableCatalogStoreResult> where B: TableCatalogObjectBackend, { let mut objects = BTreeMap::new(); - let mut metadata_prefixes = vec![format!("{}/", default_table_metadata_dir_path(namespace, table))]; - let mut data_prefixes = vec![format!("{}/", default_table_data_dir_path(namespace, table))]; - let mut delete_prefixes = vec![format!("{}/", default_table_delete_dir_path(namespace, table))]; + let (metadata_dir, data_dir, delete_dir) = scope.object_directories()?; + let mut metadata_prefixes = vec![format!("{metadata_dir}/")]; + let mut data_prefixes = vec![format!("{data_dir}/")]; + let mut delete_prefixes = vec![format!("{delete_dir}/")]; if let Some(warehouse_object_prefix) = warehouse_object_prefix { metadata_prefixes.push(format!("{warehouse_object_prefix}{METADATA_DIR}/")); data_prefixes.push(format!("{warehouse_object_prefix}{DATA_DIR}/")); @@ -725,7 +852,7 @@ where for metadata_prefix in metadata_prefixes { for object in backend.list_objects(table_bucket, &metadata_prefix).await? { - if let Some(kind) = table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &object) + if let Some(kind) = scope.object_kind(warehouse_object_prefix, &object) && matches!( kind, TableMetadataMaintenanceObjectKind::ManifestList | TableMetadataMaintenanceObjectKind::ManifestFile @@ -738,9 +865,7 @@ where for data_prefix in data_prefixes { for object in backend.list_objects(table_bucket, &data_prefix).await? { - if table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &object) - == Some(TableMetadataMaintenanceObjectKind::DataFile) - { + if scope.object_kind(warehouse_object_prefix, &object) == Some(TableMetadataMaintenanceObjectKind::DataFile) { objects.insert(object, TableMetadataMaintenanceObjectKind::DataFile); } } @@ -748,9 +873,7 @@ where for delete_prefix in delete_prefixes { for object in backend.list_objects(table_bucket, &delete_prefix).await? { - if table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &object) - == Some(TableMetadataMaintenanceObjectKind::DeleteFile) - { + if scope.object_kind(warehouse_object_prefix, &object) == Some(TableMetadataMaintenanceObjectKind::DeleteFile) { objects.insert(object, TableMetadataMaintenanceObjectKind::DeleteFile); } } diff --git a/rustfs/src/table_catalog/maintenance/worker.rs b/rustfs/src/table_catalog/maintenance/worker.rs index 7c75aa413..834278297 100644 --- a/rustfs/src/table_catalog/maintenance/worker.rs +++ b/rustfs/src/table_catalog/maintenance/worker.rs @@ -223,3 +223,205 @@ pub(crate) fn apply_maintenance_retry_after( let delay_seconds = i64::try_from(delay_seconds).unwrap_or(i64::MAX); job.next_retry_after = Some(maintenance_timestamp(now.saturating_add(Duration::seconds(delay_seconds)))); } + +pub(crate) fn validate_table_maintenance_worker_condition( + worker_id: &str, + condition: &TableMaintenanceWorkerCondition, +) -> TableCatalogStoreResult<()> { + IdentifierSegment::parse(condition.expected_job_id.clone()) + .map_err(|err| TableCatalogStoreError::Invalid(format!("invalid expected maintenance job ID: {err}")))?; + if matches!( + condition.expected_job_id.as_str(), + MAINTENANCE_JOB_ALIAS_CURRENT | MAINTENANCE_JOB_ALIAS_LATEST + ) { + return Err(TableCatalogStoreError::Invalid( + "expected maintenance job ID cannot be an alias".to_string(), + )); + } + if worker_id.trim().is_empty() || condition.expected_attempt == u16::MAX { + return Err(TableCatalogStoreError::Invalid( + "conditional maintenance requires a nonempty worker ID and an incrementable expected attempt".to_string(), + )); + } + Ok(()) +} + +/// True means that this request already claimed the recorded attempt. It must +/// return that report without executing its candidates again. +pub(crate) fn conditional_table_maintenance_is_replay( + report: &TableMetadataMaintenanceReport, + worker_id: &str, + condition: &TableMaintenanceWorkerCondition, +) -> bool { + report.job.job_id == condition.expected_job_id + && condition.expected_attempt.checked_add(1) == Some(report.job.attempt) + && report.job.worker_id.as_deref() == Some(worker_id) + && !report.job.lease_id.is_empty() + && matches!( + report.job.status, + TableMetadataMaintenanceJobStatus::Running + | TableMetadataMaintenanceJobStatus::Successful + | TableMetadataMaintenanceJobStatus::Failed + ) +} + +pub(crate) fn validate_conditional_table_maintenance_claim( + report: &TableMetadataMaintenanceReport, + entry: &TableEntry, + effective: &TableMaintenanceEffectiveConfig, + worker_id: &str, + condition: &TableMaintenanceWorkerCondition, + now: OffsetDateTime, +) -> TableCatalogStoreResult { + let job = &report.job; + if job.job_id != condition.expected_job_id + || job.table_id != entry.table_id + || report.current_metadata_location != entry.metadata_location + || job.current_metadata_location != entry.metadata_location + || job.current_generation != entry.generation + { + return Err(TableCatalogStoreError::Conflict( + "maintenance job identity or catalog state changed before conditional worker claim".to_string(), + )); + } + if conditional_table_maintenance_is_replay(report, worker_id, condition) { + if matches!(job.status, TableMetadataMaintenanceJobStatus::Running) { + if !table_maintenance_job_lease_is_active(job, effective.config.worker_lease_timeout_seconds, now) { + return Err(TableCatalogStoreError::Conflict( + "conditional maintenance does not take over an expired worker; inspect the persisted job".to_string(), + )); + } + } else if job.finished_at.as_deref().and_then(parse_maintenance_timestamp).is_none() { + return Err(TableCatalogStoreError::Conflict( + "maintenance terminal report is incomplete; inspect the persisted job".to_string(), + )); + } + return Ok(true); + } + if job.attempt != condition.expected_attempt { + return Err(TableCatalogStoreError::Conflict( + "maintenance attempt changed before conditional worker claim".to_string(), + )); + } + let config = &effective.config; + if !config.background_enabled || config.worker_paused { + return Err(TableCatalogStoreError::Conflict( + "conditional maintenance is disabled or paused".to_string(), + )); + } + if job.retain_recent_metadata_files != config.retain_recent_metadata_files + || job.quarantine_enabled != config.quarantine_enabled + || job.quarantine_retention_seconds != config.quarantine_retention_seconds + || job.config_source != effective.source + || (matches!(job.operation, TableMetadataMaintenanceOperation::Delete) && !config.delete_enabled) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance policy changed before conditional worker claim".to_string(), + )); + } + if job.quarantined_object_count > 0 || job.max_retry_attempts > 10 { + return Err(TableCatalogStoreError::Conflict( + "maintenance quarantine or invalid persisted attempt budget requires operator recovery".to_string(), + )); + } + match job.status { + TableMetadataMaintenanceJobStatus::Queued + if job.attempt == 0 + && job.worker_id.is_none() + && job.lease_id.is_empty() + && job.started_at.is_none() + && job.finished_at.is_none() + && table_maintenance_scheduler_lease_is_active(job, config.worker_lease_timeout_seconds, now) => {} + TableMetadataMaintenanceJobStatus::Failed if job.attempt > 0 && job.attempt < job.max_retry_attempts => { + let finished_at = job + .finished_at + .as_deref() + .and_then(parse_maintenance_timestamp) + .ok_or_else(|| { + TableCatalogStoreError::Conflict( + "maintenance failed report is incomplete; inspect the persisted job".to_string(), + ) + })?; + let retry_after = job + .next_retry_after + .as_deref() + .and_then(parse_maintenance_timestamp) + .ok_or_else(|| { + TableCatalogStoreError::Conflict("maintenance retry is not scheduled; inspect the persisted job".to_string()) + })?; + if retry_after < finished_at || job.lease_id.is_empty() || job.worker_id.as_deref().is_none_or(str::is_empty) { + return Err(TableCatalogStoreError::Conflict( + "maintenance failed attempt has invalid retry or lease state; inspect the persisted job".to_string(), + )); + } + if retry_after > now { + return Err(TableCatalogStoreError::Conflict("maintenance retry backoff has not elapsed".to_string())); + } + } + _ => { + return Err(TableCatalogStoreError::Conflict( + "maintenance job is not eligible for a conditional claim; inspect its status and persisted attempt budget" + .to_string(), + )); + } + } + Ok(false) +} + +/// Only a completed transition from this exact persisted Running report can +/// repair its aliases. A partial claim is deliberately not resumable here. +pub(crate) fn maintenance_terminal_report_continues_running( + terminal: &TableMetadataMaintenanceReport, + running: &TableMetadataMaintenanceReport, +) -> bool { + let finished = &terminal.job; + let started = &running.job; + let terminal_action = match finished.status { + TableMetadataMaintenanceJobStatus::Successful => TableMaintenanceAuditAction::WorkerSucceeded, + TableMetadataMaintenanceJobStatus::Failed => TableMaintenanceAuditAction::WorkerFailed, + _ => return false, + }; + matches!(started.status, TableMetadataMaintenanceJobStatus::Running) + && finished + .finished_at + .as_deref() + .and_then(parse_maintenance_timestamp) + .is_some() + && finished.job_id == started.job_id + && finished.table_bucket == started.table_bucket + && finished.namespace == started.namespace + && finished.table == started.table + && finished.table_id == started.table_id + && finished.operation == started.operation + && finished.config_source == started.config_source + && finished.worker_id == started.worker_id + && !finished.lease_id.is_empty() + && finished.lease_id == started.lease_id + && finished.attempt == started.attempt + && finished.max_retry_attempts == started.max_retry_attempts + && finished.current_metadata_location == started.current_metadata_location + && finished.current_generation == started.current_generation + && finished.retain_recent_metadata_files == started.retain_recent_metadata_files + && finished.safety_window_seconds == started.safety_window_seconds + && finished.cleanup_watermark_unix_seconds == started.cleanup_watermark_unix_seconds + && finished.quarantine_enabled == started.quarantine_enabled + && finished.quarantine_retention_seconds == started.quarantine_retention_seconds + && finished.started_at == started.started_at + && finished.heartbeat_at == started.heartbeat_at + && finished.scheduler_id == started.scheduler_id + && finished.scheduler_lease_id == started.scheduler_lease_id + && finished.scheduled_at == started.scheduled_at + && terminal.current_metadata_location == running.current_metadata_location + && terminal.cleanup_candidate_locations == running.cleanup_candidate_locations + && terminal.deletable_metadata_locations == running.deletable_metadata_locations + && terminal.cleanup_object_candidate_locations == running.cleanup_object_candidate_locations + && terminal.deletable_object_locations == running.deletable_object_locations + && terminal.audit_events.len() == running.audit_events.len().saturating_add(1) + && terminal.audit_events.starts_with(&running.audit_events) + && terminal.audit_events.last().is_some_and(|event| { + event.actor == TableMaintenanceAuditActor::Worker + && event.action == terminal_action + && event.before_status == Some(TableMetadataMaintenanceJobStatus::Running) + && event.after_status.as_ref() == Some(&finished.status) + }) +} diff --git a/rustfs/src/table_catalog/model.rs b/rustfs/src/table_catalog/model.rs index 56456bed2..5d98d9d38 100644 --- a/rustfs/src/table_catalog/model.rs +++ b/rustfs/src/table_catalog/model.rs @@ -582,6 +582,12 @@ pub(crate) struct TableMaintenanceSchedulerJobSummary { pub audit_events: Vec, } +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct TableMaintenanceWorkerCondition { + pub expected_job_id: String, + pub expected_attempt: u16, +} + #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub(crate) struct TableMetadataMaintenanceJob { pub job_id: String, diff --git a/rustfs/src/table_catalog/store/backup.rs b/rustfs/src/table_catalog/store/backup.rs index fcc1a6559..ff90385fe 100644 --- a/rustfs/src/table_catalog/store/backup.rs +++ b/rustfs/src/table_catalog/store/backup.rs @@ -817,8 +817,8 @@ where } TableCatalogEntryState::Active => {} } - let namespace = parse_namespace_for_store(&entry.namespace)?; - let table = parse_table_for_store(&entry.table)?; + parse_namespace_for_store(&entry.namespace)?; + parse_table_for_store(&entry.table)?; if !is_valid_table_metadata_location_for_entry(entry, &entry.metadata_location) { return Err(TableCatalogStoreError::Invalid(format!( "table backup metadata location is invalid for {}/{}/{}", @@ -892,11 +892,9 @@ where let kind = StrongTableCatalogBackupObjectKind::from_statistics_kind(reference.kind); insert_backup_object_kind(&mut keys, &entry.table_bucket, &object_key, kind)?; } - let referenced = metadata_maintenance_referenced_object_reports( + let referenced = metadata_maintenance_referenced_object_reports_for_entry( &self.object_backend, - &entry.table_bucket, - &namespace, - &table, + &metadata_entry, Some(warehouse_prefix.as_str()), &metadata, std::slice::from_ref(&metadata_location), @@ -1435,6 +1433,8 @@ where let mutation_fence = fence.mutation_fence(); let _write_guard = self.write_lock.lock().await; self.hydrate_state().await?; + self.recover_table_maintenance_rename_unlocked(table_bucket, Some(&mutation_fence)) + .await?; if self.read_restore_intent(table_bucket).await?.is_some() { return Err(TableCatalogStoreError::Conflict( "catalog backup requires pending restore recovery before sealing".to_string(), @@ -1584,6 +1584,8 @@ where Self::validate_maintenance_backup_objects(table_bucket, &backup.snapshot, &backup.maintenance_objects)?; self.hydrate_state().await?; + self.recover_table_maintenance_rename_unlocked(table_bucket, Some(&mutation_fence)) + .await?; let existing_intent = self.read_restore_intent(table_bucket).await?; if let Some((intent, _)) = existing_intent.as_ref() && intent.backup_id != backup_id diff --git a/rustfs/src/table_catalog/store/mod.rs b/rustfs/src/table_catalog/store/mod.rs index 378b01f66..91cc264cc 100644 --- a/rustfs/src/table_catalog/store/mod.rs +++ b/rustfs/src/table_catalog/store/mod.rs @@ -36,6 +36,11 @@ pub(super) use strong::{ }; pub(crate) use strong::{StrongTableCatalogRuntime, StrongTableCatalogStore}; +struct PeriodicTableMaintenanceQueueGuard<'a> { + table_id: &'a str, + current_job: Option<&'a TableMaintenanceSchedulerJobSummary>, +} + pub(in crate::table_catalog) fn catalog_lock_acquisition_error( operation: &str, err: rustfs_lock::LockError, @@ -51,7 +56,10 @@ pub(in crate::table_catalog) fn catalog_lock_acquisition_error( ); let message = format!("failed to {operation}: {err}"); if unavailable { - TableCatalogStoreError::Unavailable(message) + TableCatalogStoreError::LockUnavailable { + message, + source: Box::new(err), + } } else { TableCatalogStoreError::Internal(message) } @@ -468,6 +476,11 @@ pub(crate) trait TableCommitPublication: Send + Sync { None } + /// Storage-level lease signals, available only from publications that retain their guards. + fn object_mutation_fence(&self) -> Option { + None + } + fn complete(&self); } @@ -496,6 +509,7 @@ struct TableCommitLockPublication<'a, B> { struct TableCommitLockPublicationState { table_bucket: Option, table: Option<(String, String, String)>, + migration_read_guard: Option, guards: Vec, } @@ -506,6 +520,45 @@ impl<'a, B> TableCommitLockPublication<'a, B> { state: parking_lot::Mutex::new(TableCommitLockPublicationState::default()), } } + + fn attach_catalog_migration_read_guard(&self, guard: TableCatalogLockGuard) -> TableCatalogStoreResult<()> { + if guard.is_lock_lost() { + return Err(TableCatalogStoreError::Conflict( + "table-bucket catalog migration read permit was lost".to_string(), + )); + } + let mut state = self.state.lock(); + if state.migration_read_guard.is_some() || state.table_bucket.is_some() || state.table.is_some() { + return Err(TableCatalogStoreError::Internal( + "catalog migration read permit must be attached before publication".to_string(), + )); + } + state.migration_read_guard = Some(guard); + Ok(()) + } + + fn attach_catalog_write_guard(&self, guard: TableCatalogLockGuard) -> TableCatalogStoreResult<()> { + if guard.is_lock_lost() { + return Err(TableCatalogStoreError::Conflict("table catalog write lock was lost".to_string())); + } + let mut state = self.state.lock(); + if state.table_bucket.is_none() || state.table.is_some() { + return Err(TableCatalogStoreError::Internal( + "catalog write guard must be attached between bucket and table publication locks".to_string(), + )); + } + state.guards.push(guard); + Ok(()) + } + + fn release_publication_locks(&self) { + // Read-only planning may proceed without excluding unrelated tables' commits. + // The migration permit remains attached until the whole operation completes. + let mut state = self.state.lock(); + state.table = None; + state.table_bucket = None; + state.guards.clear(); + } } #[async_trait::async_trait] @@ -516,6 +569,15 @@ where async fn begin_table_bucket(&self, table_bucket: &str) -> TableCatalogStoreResult<()> { { let mut state = self.state.lock(); + if state + .migration_read_guard + .as_ref() + .is_some_and(TableCatalogLockGuard::is_lock_lost) + { + return Err(TableCatalogStoreError::Conflict( + "table-bucket catalog migration read permit was lost".to_string(), + )); + } if state.table_bucket.as_deref() == Some(table_bucket) { return Ok(()); } @@ -568,7 +630,9 @@ where fn holds_table_bucket(&self, table_bucket: &str) -> bool { let state = self.state.lock(); - state.table_bucket.as_deref() == Some(table_bucket) && state.guards.iter().all(|guard| !guard.is_lock_lost()) + state.table_bucket.as_deref() == Some(table_bucket) + && state.migration_read_guard.as_ref().is_none_or(|guard| !guard.is_lock_lost()) + && state.guards.iter().all(|guard| !guard.is_lock_lost()) } fn holds_table(&self, table_bucket: &str, namespace: &str, table: &str) -> bool { @@ -577,9 +641,29 @@ where .table .as_ref() .is_some_and(|held| held.0 == table_bucket && held.1 == namespace && held.2 == table) + && state.migration_read_guard.as_ref().is_none_or(|guard| !guard.is_lock_lost()) && state.guards.iter().all(|guard| !guard.is_lock_lost()) } + fn catalog_migration_read_permit_status(&self) -> Option { + self.state + .lock() + .migration_read_guard + .as_ref() + .map(|guard| !guard.is_lock_lost()) + } + + fn object_mutation_fence(&self) -> Option { + let state = self.state.lock(); + Some(TableCatalogObjectMutationFence::from_signals( + state + .migration_read_guard + .iter() + .chain(state.guards.iter()) + .map(TableCatalogLockGuard::lock_lost_signal), + )) + } + fn complete(&self) { *self.state.lock() = TableCommitLockPublicationState::default(); } @@ -627,6 +711,11 @@ impl TableCatalogLockGuard { } } + #[cfg(test)] + pub(crate) fn test_namespace(guard: rustfs_lock::NamespaceLockGuard) -> Self { + Self::namespace(guard) + } + fn write_namespace(guard: WriteCommitGuard) -> Self { let lock_lost = guard.lock_lost_signal(); Self { @@ -907,6 +996,10 @@ pub(crate) trait TableCatalogObjectBackend: Clone + Send + Sync + 'static { None } + fn table_commit_object_mutation_fence(&self) -> Option { + None + } + fn complete_table_commit_publication(&self) {} } @@ -1033,6 +1126,10 @@ where TableCatalogObjectBackend::catalog_migration_read_permit_status(self) } + fn object_mutation_fence(&self) -> Option { + self.table_commit_object_mutation_fence() + } + fn complete(&self) { self.complete_table_commit_publication(); } @@ -1762,7 +1859,7 @@ where ) -> TableCatalogStoreResult { match self { Self::ObjectBacked(store) => store.get_table_maintenance_config(table_bucket, namespace, table).await, - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance config")), + Self::DurableStrong(store) => store.get_table_maintenance_config(table_bucket, namespace, table).await, } } @@ -1779,7 +1876,11 @@ where .put_table_maintenance_config(table_bucket, namespace, table, config) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance config")), + Self::DurableStrong(store) => { + store + .put_table_maintenance_config(table_bucket, namespace, table, config) + .await + } } } @@ -1796,7 +1897,11 @@ where .get_table_metadata_maintenance_report(table_bucket, namespace, table, job_id) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance report")), + Self::DurableStrong(store) => { + store + .get_table_metadata_maintenance_report(table_bucket, namespace, table, job_id) + .await + } } } @@ -1812,7 +1917,11 @@ where .get_table_maintenance_scheduler_report(table_bucket, namespace, table) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance scheduler")), + Self::DurableStrong(store) => { + store + .get_table_maintenance_scheduler_report(table_bucket, namespace, table) + .await + } } } @@ -1829,7 +1938,48 @@ where .run_table_maintenance_scheduler_once(table_bucket, namespace, table, scheduler_id) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance scheduler")), + Self::DurableStrong(store) => { + store + .run_table_maintenance_scheduler_once(table_bucket, namespace, table, scheduler_id) + .await + } + } + } + + pub(crate) async fn queue_periodic_table_maintenance( + &self, + entry: &TableEntry, + scheduler_id: String, + expected_job: Option<&TableMaintenanceSchedulerJobSummary>, + ) -> TableCatalogStoreResult { + let guard = PeriodicTableMaintenanceQueueGuard { + table_id: &entry.table_id, + current_job: expected_job, + }; + match self { + Self::ObjectBacked(store) => { + store + .run_table_maintenance_scheduler_once_with_guard_at( + &entry.table_bucket, + &entry.namespace, + &entry.table, + scheduler_id, + OffsetDateTime::now_utc(), + Some(&guard), + ) + .await + } + Self::DurableStrong(store) => { + store + .run_table_maintenance_scheduler_once_with_guard( + &entry.table_bucket, + &entry.namespace, + &entry.table, + scheduler_id, + Some(&guard), + ) + .await + } } } @@ -1847,7 +1997,11 @@ where .apply_table_maintenance_quarantine_operation(table_bucket, namespace, table, job_id, request) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance quarantine")), + Self::DurableStrong(store) => { + store + .apply_table_maintenance_quarantine_operation(table_bucket, namespace, table, job_id, request) + .await + } } } @@ -1864,7 +2018,45 @@ where .run_table_metadata_maintenance_worker_once(table_bucket, namespace, table, worker_id) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance worker")), + Self::DurableStrong(store) => { + store + .run_table_metadata_maintenance_worker_once(table_bucket, namespace, table, worker_id) + .await + } + } + } + + pub(crate) async fn run_table_metadata_maintenance_worker_once_conditionally( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + worker_id: String, + condition: &TableMaintenanceWorkerCondition, + ) -> TableCatalogStoreResult { + match self { + Self::ObjectBacked(store) => { + store + .run_table_metadata_maintenance_worker_once_conditionally( + table_bucket, + namespace, + table, + worker_id, + condition, + ) + .await + } + Self::DurableStrong(store) => { + store + .run_table_metadata_maintenance_worker_once_conditionally( + table_bucket, + namespace, + table, + worker_id, + condition, + ) + .await + } } } @@ -1883,7 +2075,11 @@ where .heartbeat_table_metadata_maintenance_job(table_bucket, namespace, table, job_id, lease_id, worker_id) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance heartbeat")), + Self::DurableStrong(store) => { + store + .heartbeat_table_metadata_maintenance_job(table_bucket, namespace, table, job_id, lease_id, worker_id) + .await + } } } @@ -2448,48 +2644,693 @@ where mod tests { use super::*; use crate::storage_api::protocols::client::{BucketOperations as _, MakeBucketOptions}; - use rustfs_lock::{LockClient, LockId, LockInfo, LockRequest, LockResponse, LockStats}; + use crate::table_catalog::test_support::{RecordingLeaseClient, TestCatalogObjectBackend}; + use crate::table_catalog::tests::{test_bucket_entry, test_namespace_entry, test_table_entry}; + use rustfs_lock::LockClient; use std::sync::{Arc, Mutex}; use std::time::Duration; - #[derive(Debug)] - struct RecordingLeaseClient { - inner: rustfs_lock::LocalClient, - acquired: Mutex>, + #[tokio::test] + async fn table_maintenance_publication_reuses_attached_migration_read_permit() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + store.put_table_bucket(test_bucket_entry(bucket)).await.unwrap(); + store + .create_namespace(test_namespace_entry(bucket, &namespace)) + .await + .unwrap(); + let migration_path = TableCatalogObjectPaths::default().backing_migration_global_fence_lock_path(); + let guard = backend.acquire_read_lock(RUSTFS_META_BUCKET, &migration_path).await.unwrap(); + let publication = TableCommitLockPublication::new(&backend); + publication.attach_catalog_migration_read_guard(guard).unwrap(); + assert_eq!(publication.catalog_migration_read_permit_status(), Some(true)); + assert!( + publication + .attach_catalog_migration_read_guard(TableCatalogLockGuard::stable(())) + .is_err() + ); + let reads = backend.read_lock_acquisition_count(RUSTFS_META_BUCKET, &migration_path).await; + let entry = test_table_entry( + bucket, + &namespace, + &table, + default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"), + ); + store.register_table_with_publication(entry, &publication).await.unwrap(); + assert_eq!(backend.read_lock_acquisition_count(RUSTFS_META_BUCKET, &migration_path).await, reads); + assert_eq!(publication.catalog_migration_read_permit_status(), None); + tokio::time::timeout(Duration::from_secs(5), backend.acquire_write_lock(RUSTFS_META_BUCKET, &migration_path)) + .await + .expect("completion releases the attached migration read permit") + .unwrap(); } - #[async_trait::async_trait] - impl LockClient for RecordingLeaseClient { - async fn acquire_lock(&self, request: &LockRequest) -> rustfs_lock::Result { - let response = self.inner.acquire_lock(request).await?; - if let Some(info) = response.lock_info.as_ref() { - *self.acquired.lock().expect("record lease") = Some(info.id.clone()); + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn table_maintenance_migration_lease_loss_blocks_sidecars_and_cleanup() { + let backend = TestCatalogObjectBackend::default(); + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let current = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let orphan = default_table_metadata_file_path(&namespace, &table, "00000.metadata.json"); + let entry = test_table_entry(bucket, &namespace, &table, current.clone()); + backend + .seed_object(bucket, ¤t, br#"{"metadata-log":[]}"#.to_vec()) + .await; + backend.seed_object(bucket, &orphan, b"{}".to_vec()).await; + let sidecar = ObjectTableCatalogStore::new(backend.clone()); + let config = TableMaintenanceConfig { + version: TABLE_MAINTENANCE_CONFIG_VERSION, + background_enabled: true, + delete_enabled: true, + retain_recent_metadata_files: 0, + ..Default::default() + }; + sidecar + .put_table_maintenance_config_for_entry(&entry, config.clone()) + .await + .unwrap(); + let report = sidecar.plan_table_metadata_maintenance_for_entry(&entry, 0).await.unwrap(); + assert_eq!(report.deletable_metadata_locations, vec![orphan.clone()]); + let client = Arc::new(RecordingLeaseClient { + inner: rustfs_lock::LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::Enabled(Arc::new( + rustfs_lock::FastObjectLockManager::new(), + )))), + acquired: Mutex::new(None), + }); + let lock = rustfs_lock::NamespaceLockWrapper::new( + rustfs_lock::NamespaceLock::with_clients_and_quorum( + "maintenance-probe".to_string(), + vec![client.clone() as Arc], + 1, + ), + rustfs_lock::ObjectKey::new( + RUSTFS_META_BUCKET, + TableCatalogObjectPaths::default().backing_migration_global_fence_lock_path(), + ), + "maintenance-migration-probe".to_string(), + ); + let guard = TableCatalogLockGuard::namespace(lock.get_read_lock(Duration::from_secs(5)).await.unwrap()); + let publication = TableCommitLockPublication::new(&backend); + publication.attach_catalog_migration_read_guard(guard).unwrap(); + publication.begin_table_bucket(bucket).await.unwrap(); + publication.prepare(bucket, "sales", "orders").await.unwrap(); + assert!(publication.holds_table(bucket, "sales", "orders")); + let id = client + .acquired + .lock() + .unwrap() + .clone() + .expect("actual migration lease acquired"); + assert!(client.force_release(&id).await.unwrap()); + tokio::time::timeout(Duration::from_secs(15), async { + while publication.catalog_migration_read_permit_status() != Some(false) { + tokio::time::sleep(Duration::from_millis(10)).await; } - Ok(response) + }) + .await + .expect("heartbeat observes the released migration lease"); + assert!(!publication.holds_table_bucket(bucket)); + assert!(!publication.holds_table(bucket, "sales", "orders")); + assert!(matches!( + publication.begin_table_bucket(bucket).await, + Err(TableCatalogStoreError::Conflict(message)) if message.contains("migration read permit was lost") + )); + let mut changed_config = config.clone(); + changed_config.worker_paused = true; + assert!( + sidecar + .put_table_maintenance_config_for_entry_with_publication(&entry, changed_config, Some(&publication)) + .await + .is_err() + ); + assert_eq!(sidecar.get_table_maintenance_config_for_entry(&entry).await.unwrap(), config); + let effective = TableMaintenanceEffectiveConfig { + config, + source: TableMaintenanceConfigSource::TableOverride, + }; + assert!( + sidecar + .finish_table_metadata_maintenance_run_for_entry_with_publication(&entry, true, &effective, report, &publication) + .await + .is_err() + ); + assert!(backend.object_exists(bucket, ¤t).await.unwrap()); + assert!(backend.object_exists(bucket, &orphan).await.unwrap()); + let paths = TableCatalogObjectPaths::default(); + assert!( + !backend + .object_exists( + RUSTFS_META_BUCKET, + &paths.table_maintenance_current_job_path(bucket, &namespace, &table, &entry.table_id) + ) + .await + .unwrap() + ); + let store = StrongTableCatalogStore::new(backend.clone()); + assert!(matches!( + store.register_table_with_publication(entry, &publication).await, + Err(TableCatalogStoreError::Conflict(message)) if message.contains("migration read permit was lost") + )); + let lost = publication.state.lock().migration_read_guard.take().unwrap(); + let rejected = TableCommitLockPublication::new(&backend); + assert!(matches!( + rejected.attach_catalog_migration_read_guard(lost), + Err(TableCatalogStoreError::Conflict(message)) if message.contains("migration read permit was lost") + )); + assert_eq!(rejected.catalog_migration_read_permit_status(), None); + } + + struct MaintenanceMutationFixture { + backend: TestCatalogObjectBackend, + sidecar: ObjectTableCatalogStore, + entry: TableEntry, + config: TableMaintenanceConfig, + report: TableMetadataMaintenanceReport, + orphan: String, + } + + async fn maintenance_mutation_fixture(data_orphan: bool) -> MaintenanceMutationFixture { + let backend = TestCatalogObjectBackend::default(); + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let current = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let orphan = if data_orphan { + format!("{}/orphan.parquet", default_table_data_dir_path(&namespace, &table)) + } else { + default_table_metadata_file_path(&namespace, &table, "00000.metadata.json") + }; + let entry = test_table_entry("analytics", &namespace, &table, current.clone()); + backend + .seed_object("analytics", ¤t, br#"{"metadata-log":[]}"#.to_vec()) + .await; + backend.seed_object("analytics", &orphan, b"{}".to_vec()).await; + let sidecar = ObjectTableCatalogStore::new(backend.clone()); + let config = TableMaintenanceConfig { + version: TABLE_MAINTENANCE_CONFIG_VERSION, + background_enabled: true, + delete_enabled: true, + retain_recent_metadata_files: 0, + ..Default::default() + }; + sidecar + .put_table_maintenance_config_for_entry(&entry, config.clone()) + .await + .unwrap(); + let report = sidecar.plan_table_metadata_maintenance_for_entry(&entry, 0).await.unwrap(); + assert!(if data_orphan { + report.deletable_object_locations.contains(&orphan) + } else { + report.deletable_metadata_locations.contains(&orphan) + }); + MaintenanceMutationFixture { + backend, + sidecar, + entry, + config, + report, + orphan, } - async fn release(&self, id: &LockId) -> rustfs_lock::Result { - self.inner.release(id).await + } + + async fn maintenance_loss_publication<'a>( + backend: &'a TestCatalogObjectBackend, + entry: &TableEntry, + ) -> (TableCommitLockPublication<'a, TestCatalogObjectBackend>, Arc) { + let client = Arc::new(RecordingLeaseClient { + inner: rustfs_lock::LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::Enabled(Arc::new( + rustfs_lock::FastObjectLockManager::new(), + )))), + acquired: Mutex::new(None), + }); + let lock = rustfs_lock::NamespaceLockWrapper::new( + rustfs_lock::NamespaceLock::with_clients_and_quorum( + "maintenance-await-probe".to_string(), + vec![client.clone() as Arc], + 1, + ), + rustfs_lock::ObjectKey::new( + RUSTFS_META_BUCKET, + TableCatalogObjectPaths::default().backing_migration_global_fence_lock_path(), + ), + "maintenance-await-migration".to_string(), + ); + let publication = TableCommitLockPublication::new(backend); + publication + .attach_catalog_migration_read_guard(TableCatalogLockGuard::namespace( + lock.get_read_lock(Duration::from_secs(5)).await.unwrap(), + )) + .unwrap(); + publication.begin_table_bucket(&entry.table_bucket).await.unwrap(); + publication + .prepare(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap(); + (publication, client) + } + + async fn lose_maintenance_lease( + client: &RecordingLeaseClient, + publication: &TableCommitLockPublication<'_, TestCatalogObjectBackend>, + ) { + let id = client.acquired.lock().unwrap().clone().unwrap(); + assert!(client.force_release(&id).await.unwrap()); + tokio::time::timeout(Duration::from_secs(15), async { + while publication.catalog_migration_read_permit_status() != Some(false) { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("heartbeat observes loss while backend mutation is suspended"); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn table_maintenance_await_loss_prevents_each_sidecar_storage_commit() { + for target in 0..4 { + let fixture = maintenance_mutation_fixture(false).await; + let MaintenanceMutationFixture { + backend, + sidecar, + entry, + config, + mut report, + .. + } = fixture; + sidecar + .put_table_metadata_maintenance_report_for_entry(&report, &entry) + .await + .unwrap(); + let namespace = Namespace::parse(&entry.namespace).unwrap(); + let table = IdentifierSegment::parse(&entry.table).unwrap(); + let paths = TableCatalogObjectPaths::default(); + let sidecars = [ + paths.table_maintenance_config_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + paths.table_maintenance_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id, &report.job.job_id), + paths.table_maintenance_latest_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + paths.table_maintenance_current_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + ]; + let before = backend.read_object(RUSTFS_META_BUCKET, &sidecars[target]).await.unwrap(); + let later_before = if target < 3 { + backend.read_object(RUSTFS_META_BUCKET, &sidecars[3]).await.unwrap() + } else { + None + }; + let (publication, client) = maintenance_loss_publication(&backend, &entry).await; + let pause = backend.pause_next_put(RUSTFS_META_BUCKET, &sidecars[target]).await; + let mut changed_config = config; + changed_config.worker_paused = true; + report.job.status = TableMetadataMaintenanceJobStatus::Running; + report.job.worker_id = Some("new-worker".to_string()); + let operation = async { + if target == 0 { + sidecar + .put_table_maintenance_config_for_entry_with_publication(&entry, changed_config, Some(&publication)) + .await + .map(|_| ()) + } else { + sidecar + .put_table_metadata_maintenance_report_for_entry_with_publication(&report, &entry, Some(&publication)) + .await + } + }; + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + assert!(publication.holds_table(&entry.table_bucket, &entry.namespace, &entry.table)); + lose_maintenance_lease(&client, &publication).await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(result.is_err(), "sidecar target {target} must reject a loss inside backend await"); + assert_eq!(backend.read_object(RUSTFS_META_BUCKET, &sidecars[target]).await.unwrap(), before); + if target < 3 { + assert_eq!(backend.read_object(RUSTFS_META_BUCKET, &sidecars[3]).await.unwrap(), later_before); + } + assert!( + backend + .object_exists(&entry.table_bucket, &entry.metadata_location) + .await + .unwrap() + ); } - async fn refresh(&self, id: &LockId) -> rustfs_lock::Result { - self.inner.refresh(id).await + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn table_maintenance_await_loss_prevents_metadata_and_data_delete() { + for data_orphan in [false, true] { + let MaintenanceMutationFixture { + backend, + sidecar, + entry, + config, + report, + orphan, + } = maintenance_mutation_fixture(data_orphan).await; + let before = backend.read_object(&entry.table_bucket, &orphan).await.unwrap(); + let (publication, client) = maintenance_loss_publication(&backend, &entry).await; + let pause = backend.pause_next_delete(&entry.table_bucket, &orphan).await; + let effective = TableMaintenanceEffectiveConfig { + config, + source: TableMaintenanceConfigSource::TableOverride, + }; + let operation = sidecar.finish_table_metadata_maintenance_run_for_entry_with_publication( + &entry, + true, + &effective, + report, + &publication, + ); + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + assert!(publication.holds_table(&entry.table_bucket, &entry.namespace, &entry.table)); + lose_maintenance_lease(&client, &publication).await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(result.is_err()); + assert_eq!(backend.read_object(&entry.table_bucket, &orphan).await.unwrap(), before); + assert!( + backend + .object_exists(&entry.table_bucket, &entry.metadata_location) + .await + .unwrap() + ); + assert!( + sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, + &Namespace::parse(&entry.namespace).unwrap(), + &IdentifierSegment::parse(&entry.table).unwrap(), + &entry.table_id, + MAINTENANCE_JOB_ALIAS_CURRENT, + ) + .await + .unwrap() + .is_none() + ); } - async fn force_release(&self, id: &LockId) -> rustfs_lock::Result { - self.inner.force_release(id).await + } + + #[tokio::test] + async fn table_maintenance_sidecar_cas_preserves_concurrent_replacement() { + let MaintenanceMutationFixture { + backend, + sidecar, + entry, + config, + .. + } = maintenance_mutation_fixture(false).await; + let namespace = Namespace::parse(&entry.namespace).unwrap(); + let table = IdentifierSegment::parse(&entry.table).unwrap(); + let path = TableCatalogObjectPaths::default().table_maintenance_config_path( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, + ); + let publication = TableCommitLockPublication::new(&backend); + publication + .attach_catalog_migration_read_guard(TableCatalogLockGuard::stable(())) + .unwrap(); + publication.begin_table_bucket(&entry.table_bucket).await.unwrap(); + publication + .prepare(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap(); + let pause = backend.pause_next_put(RUSTFS_META_BUCKET, &path).await; + let mut changed_config = config.clone(); + changed_config.worker_paused = true; + let mut replacement = config; + replacement.delete_enabled = false; + let bytes = serde_json::to_vec(&replacement).unwrap(); + let operation = + sidecar.put_table_maintenance_config_for_entry_with_publication(&entry, changed_config, Some(&publication)); + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + backend.seed_object(RUSTFS_META_BUCKET, &path, bytes.clone()).await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(matches!(result, Err(TableCatalogStoreError::Conflict(_)))); + assert_eq!(backend.read_object(RUSTFS_META_BUCKET, &path).await.unwrap().unwrap().data, bytes); + } + + #[tokio::test] + async fn table_maintenance_delete_cas_preserves_changed_candidates() { + for data_orphan in [false, true] { + let MaintenanceMutationFixture { + backend, + sidecar, + entry, + config, + report, + orphan, + } = maintenance_mutation_fixture(data_orphan).await; + let publication = TableCommitLockPublication::new(&backend); + publication + .attach_catalog_migration_read_guard(TableCatalogLockGuard::stable(())) + .unwrap(); + publication.begin_table_bucket(&entry.table_bucket).await.unwrap(); + publication + .prepare(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap(); + let pause = backend.pause_next_delete(&entry.table_bucket, &orphan).await; + let effective = TableMaintenanceEffectiveConfig { + config, + source: TableMaintenanceConfigSource::TableOverride, + }; + let operation = sidecar.finish_table_metadata_maintenance_run_for_entry_with_publication( + &entry, + true, + &effective, + report, + &publication, + ); + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + backend + .seed_object(&entry.table_bucket, &orphan, b"replacement".to_vec()) + .await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(matches!(result, Err(TableCatalogStoreError::Conflict(_)))); + assert_eq!( + backend.read_object(&entry.table_bucket, &orphan).await.unwrap().unwrap().data, + b"replacement".to_vec() + ); + assert!( + backend + .object_exists(&entry.table_bucket, &entry.metadata_location) + .await + .unwrap() + ); } - async fn check_status(&self, id: &LockId) -> rustfs_lock::Result> { - self.inner.check_status(id).await + } + + fn maintenance_rename_destination(entry: &TableEntry) -> (TableEntry, String, String) { + let mut destination = entry.clone(); + destination.table = "tablearchived".to_string(); + let paths = TableCatalogObjectPaths::default(); + let namespace = Namespace::parse(&entry.namespace).unwrap(); + let source_path = paths.table_maintenance_config_path( + &entry.table_bucket, + &namespace, + &IdentifierSegment::parse(&entry.table).unwrap(), + &entry.table_id, + ); + let destination_path = paths.table_maintenance_config_path( + &destination.table_bucket, + &namespace, + &IdentifierSegment::parse(&destination.table).unwrap(), + &destination.table_id, + ); + (destination, source_path, destination_path) + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn table_maintenance_rename_await_loss_prevents_destination_copy() { + let MaintenanceMutationFixture { + backend, sidecar, entry, .. + } = maintenance_mutation_fixture(false).await; + let (destination, source_path, destination_path) = maintenance_rename_destination(&entry); + let source_before = backend.read_object(RUSTFS_META_BUCKET, &source_path).await.unwrap(); + assert!(source_before.is_some()); + assert!(!backend.object_exists(RUSTFS_META_BUCKET, &destination_path).await.unwrap()); + let (publication, client) = maintenance_loss_publication(&backend, &entry).await; + let fence = publication.object_mutation_fence().unwrap(); + let pause = backend.pause_next_put(RUSTFS_META_BUCKET, &destination_path).await; + let operation = sidecar.prepare_table_maintenance_rename(&entry, &destination, "source-snapshot-etag", &fence); + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + lose_maintenance_lease(&client, &publication).await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(result.is_err()); + assert_eq!(backend.read_object(RUSTFS_META_BUCKET, &source_path).await.unwrap(), source_before); + assert!(!backend.object_exists(RUSTFS_META_BUCKET, &destination_path).await.unwrap()); + assert!( + sidecar + .read_table_maintenance_rename_intent(&entry.table_bucket) + .await + .unwrap() + .is_some() + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn table_maintenance_rename_await_loss_prevents_source_cleanup() { + let MaintenanceMutationFixture { + backend, sidecar, entry, .. + } = maintenance_mutation_fixture(false).await; + let (destination, source_path, destination_path) = maintenance_rename_destination(&entry); + let source_before = backend.read_object(RUSTFS_META_BUCKET, &source_path).await.unwrap(); + assert!(source_before.is_some()); + let (publication, client) = maintenance_loss_publication(&backend, &entry).await; + let fence = publication.object_mutation_fence().unwrap(); + let (intent, intent_etag) = sidecar + .prepare_table_maintenance_rename(&entry, &destination, "source-snapshot-etag", &fence) + .await + .unwrap() + .expect("source maintenance config must be prepared for rename"); + let destination_before = backend.read_object(RUSTFS_META_BUCKET, &destination_path).await.unwrap(); + assert!(destination_before.is_some()); + let pause = backend.pause_next_delete(RUSTFS_META_BUCKET, &source_path).await; + let operation = sidecar.finish_table_maintenance_rename(&intent, &intent_etag, true, &fence); + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + lose_maintenance_lease(&client, &publication).await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(result.is_err()); + assert_eq!(backend.read_object(RUSTFS_META_BUCKET, &source_path).await.unwrap(), source_before); + assert_eq!( + backend.read_object(RUSTFS_META_BUCKET, &destination_path).await.unwrap(), + destination_before + ); + assert!( + sidecar + .read_table_maintenance_rename_intent(&entry.table_bucket) + .await + .unwrap() + .is_some() + ); + } + + #[tokio::test] + async fn table_maintenance_rename_delete_cas_preserves_same_content_replacement() { + for committed in [true, false] { + let MaintenanceMutationFixture { + backend, sidecar, entry, .. + } = maintenance_mutation_fixture(false).await; + let (destination, source_path, destination_path) = maintenance_rename_destination(&entry); + let fence = TableCatalogObjectMutationFence::default(); + let (intent, intent_etag) = sidecar + .prepare_table_maintenance_rename(&entry, &destination, "source-snapshot-etag", &fence) + .await + .unwrap() + .expect("source maintenance config must be prepared for rename"); + let (target_path, other_path) = if committed { + (&source_path, &destination_path) + } else { + (&destination_path, &source_path) + }; + let target_before = backend.read_object(RUSTFS_META_BUCKET, target_path).await.unwrap().unwrap(); + let other_before = backend.read_object(RUSTFS_META_BUCKET, other_path).await.unwrap(); + assert!(other_before.is_some()); + let pause = backend.pause_next_delete(RUSTFS_META_BUCKET, target_path).await; + let operation = sidecar.finish_table_maintenance_rename(&intent, &intent_etag, committed, &fence); + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + backend + .seed_object(RUSTFS_META_BUCKET, target_path, target_before.data.clone()) + .await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(matches!(result, Err(TableCatalogStoreError::Conflict(_)))); + let replacement = backend.read_object(RUSTFS_META_BUCKET, target_path).await.unwrap().unwrap(); + assert_eq!(replacement.data, target_before.data); + assert_ne!(replacement.etag, target_before.etag); + assert_eq!(backend.read_object(RUSTFS_META_BUCKET, other_path).await.unwrap(), other_before); + assert!( + sidecar + .read_table_maintenance_rename_intent(&entry.table_bucket) + .await + .unwrap() + .is_some() + ); } - async fn get_stats(&self) -> rustfs_lock::Result { - self.inner.get_stats().await - } - async fn close(&self) -> rustfs_lock::Result<()> { - self.inner.close().await - } - async fn is_online(&self) -> bool { - self.inner.is_online().await - } - async fn is_local(&self) -> bool { - self.inner.is_local().await + } + + #[tokio::test] + async fn table_maintenance_delete_respects_original_plan_watermark() { + for data_orphan in [false, true] { + let MaintenanceMutationFixture { + backend, + sidecar, + entry, + config, + mut report, + orphan, + } = maintenance_mutation_fixture(data_orphan).await; + let now = OffsetDateTime::now_utc(); + let window = time::Duration::seconds(TABLE_METADATA_CLEANUP_SAFETY_WINDOW_SECONDS); + report.job.cleanup_watermark_unix_seconds = (now - window * 3_i32).unix_timestamp(); + backend + .seed_object_with_mod_time( + &entry.table_bucket, + &orphan, + b"changed since plan".to_vec(), + Some(now - window * 2_i32), + ) + .await; + let before = backend.read_object(&entry.table_bucket, &orphan).await.unwrap(); + let publication = TableCommitLockPublication::new(&backend); + publication + .attach_catalog_migration_read_guard(TableCatalogLockGuard::stable(())) + .unwrap(); + publication.begin_table_bucket(&entry.table_bucket).await.unwrap(); + publication + .prepare(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap(); + let effective = TableMaintenanceEffectiveConfig { + config, + source: TableMaintenanceConfigSource::TableOverride, + }; + let completed = sidecar + .finish_table_metadata_maintenance_run_for_entry_with_publication(&entry, true, &effective, report, &publication) + .await + .unwrap(); + assert_eq!(completed.job.deleted_metadata_file_count, 0); + assert_eq!(completed.job.deleted_object_count, 0); + assert_eq!(backend.read_object(&entry.table_bucket, &orphan).await.unwrap(), before); + assert!( + backend + .object_exists(&entry.table_bucket, &entry.metadata_location) + .await + .unwrap() + ); } } diff --git a/rustfs/src/table_catalog/store/object.rs b/rustfs/src/table_catalog/store/object.rs index f62b2b93f..01690c7bc 100644 --- a/rustfs/src/table_catalog/store/object.rs +++ b/rustfs/src/table_catalog/store/object.rs @@ -14,6 +14,161 @@ use super::*; +const TABLE_MAINTENANCE_RENAME_MAX_OBJECTS: usize = 4096; +const TABLE_MAINTENANCE_SCHEDULER_HISTORY_MAX_OBJECTS: usize = 4096; + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct TableMaintenanceRenameObject { + etag: String, + data: serde_json::Value, +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct TableMaintenanceRenameIntent { + version: u16, + pub(super) source: TableEntry, + pub(super) destination: TableEntry, + pub(super) source_snapshot_etag: String, + source_objects: BTreeMap, + destination_etags: BTreeMap, +} + +fn maintenance_rename_intent_path(table_bucket: &str) -> String { + format!( + "{}maintenance-rename-intent.json", + TableCatalogObjectPaths::default().table_bucket_root_prefix(table_bucket) + ) +} + +pub(super) fn maintenance_rename_recovery_error(mut error: TableCatalogStoreError) -> TableCatalogStoreError { + let message = match &mut error { + TableCatalogStoreError::NotFound(message) + | TableCatalogStoreError::NamespaceNotFound(message) + | TableCatalogStoreError::TableNotFound(message) + | TableCatalogStoreError::AlreadyExists(message) + | TableCatalogStoreError::Conflict(message) + | TableCatalogStoreError::Invalid(message) + | TableCatalogStoreError::Unsupported(message) + | TableCatalogStoreError::Unavailable(message) + | TableCatalogStoreError::LockUnavailable { message, .. } + | TableCatalogStoreError::Internal(message) => message, + }; + *message = format!( + "maintenance rename recovery required; retry the same rename after transient errors, or repair invalid persisted state before retrying: {message}" + ); + error +} + +fn maintenance_rename_prefix(entry: &TableEntry) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let path = TableCatalogObjectPaths::default().table_maintenance_config_path( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, + ); + Ok(path.trim_end_matches(MAINTENANCE_CONFIG_FILE).to_string()) +} + +fn maintenance_rename_suffix_is_valid(suffix: &str) -> bool { + matches!( + suffix, + MAINTENANCE_CONFIG_FILE | MAINTENANCE_CURRENT_JOB_FILE | MAINTENANCE_LATEST_JOB_FILE + ) || suffix + .strip_prefix(&format!("{MAINTENANCE_JOB_ROOT}/")) + .and_then(|suffix| suffix.strip_suffix(".json")) + .is_some_and(|hash| { + hash.len() == 64 + && hash + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + }) +} + +fn maintenance_rename_data( + entry: &TableEntry, + destination: &TableEntry, + suffix: &str, + data: &serde_json::Value, +) -> TableCatalogStoreResult { + if !maintenance_rename_suffix_is_valid(suffix) { + return Err(TableCatalogStoreError::Invalid( + "unexpected table maintenance rename object path".to_string(), + )); + } + if suffix == MAINTENANCE_CONFIG_FILE { + let config = serde_json::from_value::(data.clone()) + .map_err(|err| TableCatalogStoreError::Invalid(format!("invalid maintenance rename config: {err}")))?; + validate_table_maintenance_config(&config)?; + return Ok(data.clone()); + } + let report = serde_json::from_value::(data.clone()) + .map_err(|err| TableCatalogStoreError::Invalid(format!("invalid maintenance rename report: {err}")))?; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + validate_table_maintenance_report_owner(&report, &entry.table_bucket, &namespace, &table, &entry.table_id)?; + if suffix.starts_with(&format!("{MAINTENANCE_JOB_ROOT}/")) + && suffix != format!("{MAINTENANCE_JOB_ROOT}/{}.json", table_catalog_path_hash(&report.job.job_id)) + { + return Err(TableCatalogStoreError::Invalid( + "maintenance rename job does not match its object path".to_string(), + )); + } + // Preserve every report field, including audit records and unknown additive fields. + // Only catalog owner identifiers change; physical locations and generations do not. + let mut renamed = data.clone(); + for field in ["job", "snapshot-expiration", "compaction"] { + let Some(owner) = renamed.get_mut(field).filter(|value| !value.is_null()) else { + continue; + }; + if owner.get("table_bucket").and_then(serde_json::Value::as_str) != Some(entry.table_bucket.as_str()) + || owner.get("namespace").and_then(serde_json::Value::as_str) != Some(entry.namespace.as_str()) + || owner.get("table").and_then(serde_json::Value::as_str) != Some(entry.table.as_str()) + || owner.get("table_id").and_then(serde_json::Value::as_str) != Some(entry.table_id.as_str()) + { + return Err(TableCatalogStoreError::Invalid( + "maintenance rename nested report owner does not match its table".to_string(), + )); + } + owner["namespace"] = serde_json::Value::String(destination.namespace.clone()); + owner["table"] = serde_json::Value::String(destination.table.clone()); + } + Ok(renamed) +} + +fn ensure_table_maintenance_publication_held( + publication: Option<&(dyn TableCommitPublication + Sync)>, + entry: &TableEntry, + stage: &str, +) -> TableCatalogStoreResult<()> { + if publication.is_some_and(|publication| { + !publication.holds_table_bucket(&entry.table_bucket) + || !publication.holds_table(&entry.table_bucket, &entry.namespace, &entry.table) + }) { + return Err(TableCatalogStoreError::Internal(format!( + "table maintenance publication fence was lost {stage}" + ))); + } + Ok(()) +} + +fn table_maintenance_mutation_fence( + publication: Option<&(dyn TableCommitPublication + Sync)>, + entry: &TableEntry, +) -> TableCatalogStoreResult> { + ensure_table_maintenance_publication_held(publication, entry, "before storage mutation")?; + publication + .map(|publication| { + publication.object_mutation_fence().ok_or_else(|| { + TableCatalogStoreError::Internal("table maintenance requires a storage mutation fence".to_string()) + }) + }) + .transpose() +} + pub(super) fn validate_table_bucket_entry_object( paths: &TableCatalogObjectPaths, object: &str, @@ -673,6 +828,299 @@ where Ok(entries) } + async fn collect_table_maintenance_rename_objects( + &self, + entry: &TableEntry, + ) -> TableCatalogStoreResult> { + let prefix = maintenance_rename_prefix(entry)?; + let limit = NonZeroUsize::new(TABLE_CATALOG_LIST_MAX_KEYS) + .ok_or_else(|| TableCatalogStoreError::Internal("maintenance rename page limit must be positive".to_string()))?; + let mut cursor: Option = None; + let mut objects = BTreeMap::new(); + let mut total_bytes = 0usize; + loop { + let page = self + .backend + .list_objects_page(self.catalog_bucket(), &prefix, cursor.as_deref(), limit) + .await?; + if page.objects.len() > limit.get() || (page.is_truncated && page.objects.is_empty()) { + return Err(TableCatalogStoreError::Invalid("invalid maintenance rename listing page".to_string())); + } + for object in page.objects { + if cursor.as_ref().is_some_and(|cursor| object <= *cursor) { + return Err(TableCatalogStoreError::Invalid("maintenance rename listing did not advance".to_string())); + } + let suffix = object + .strip_prefix(&prefix) + .filter(|suffix| maintenance_rename_suffix_is_valid(suffix)) + .ok_or_else(|| { + TableCatalogStoreError::Invalid("maintenance rename listing escaped its owner paths".to_string()) + })?; + if objects.len() >= TABLE_MAINTENANCE_RENAME_MAX_OBJECTS { + return Err(TableCatalogStoreError::Invalid( + "maintenance rename object count exceeds its limit".to_string(), + )); + } + let value = self + .backend + .read_object_limited(self.catalog_bucket(), &object, TABLE_CATALOG_BACKUP_MAX_MAINTENANCE_OBJECT_SIZE) + .await? + .ok_or_else(|| { + TableCatalogStoreError::Conflict("maintenance rename object disappeared during collection".to_string()) + })?; + total_bytes = total_bytes.saturating_add(value.data.len()); + if total_bytes > TABLE_CATALOG_BACKUP_MAX_MAINTENANCE_BYTES { + return Err(TableCatalogStoreError::Invalid( + "maintenance rename content exceeds its byte limit".to_string(), + )); + } + let etag = value + .etag + .filter(|etag| !etag.is_empty()) + .ok_or_else(|| TableCatalogStoreError::Conflict("maintenance rename object has no ETag".to_string()))?; + let data = serde_json::from_slice(&value.data) + .map_err(|err| TableCatalogStoreError::Invalid(format!("invalid maintenance rename JSON: {err}")))?; + maintenance_rename_data(entry, entry, suffix, &data)?; + objects.insert(suffix.to_string(), TableMaintenanceRenameObject { etag, data }); + cursor = Some(object); + } + if !page.is_truncated { + return Ok(objects); + } + } + } + + pub(super) async fn read_table_maintenance_rename_intent( + &self, + table_bucket: &str, + ) -> TableCatalogStoreResult> { + let object = maintenance_rename_intent_path(table_bucket); + let Some(value) = self + .backend + .read_object_limited(self.catalog_bucket(), &object, TABLE_CATALOG_BACKUP_MAX_SIZE) + .await? + else { + return Ok(None); + }; + let intent: TableMaintenanceRenameIntent = serde_json::from_slice(&value.data) + .map_err(|err| TableCatalogStoreError::Invalid(format!("invalid maintenance rename intent: {err}")))?; + let etag = value + .etag + .filter(|etag| !etag.is_empty()) + .ok_or_else(|| TableCatalogStoreError::Conflict("maintenance rename intent has no ETag".to_string()))?; + validate_table_entry_version_and_id(&intent.source)?; + validate_table_entry_version_and_id(&intent.destination)?; + let mut expected_destination = intent.source.clone(); + expected_destination.namespace.clone_from(&intent.destination.namespace); + expected_destination.table.clone_from(&intent.destination.table); + if intent.version != 1 + || intent.source_snapshot_etag.is_empty() + || intent.source.table_bucket != table_bucket + || intent.source.state != TableCatalogEntryState::Active + || intent.destination != expected_destination + || (intent.source.namespace == intent.destination.namespace && intent.source.table == intent.destination.table) + || intent.source_objects.len() > TABLE_MAINTENANCE_RENAME_MAX_OBJECTS + || intent.destination_etags.len() > TABLE_MAINTENANCE_RENAME_MAX_OBJECTS + { + return Err(TableCatalogStoreError::Invalid( + "invalid maintenance rename intent identity, version, or count".to_string(), + )); + } + maintenance_rename_prefix(&intent.source)?; + maintenance_rename_prefix(&intent.destination)?; + let mut total_bytes = 0usize; + for (suffix, value) in &intent.source_objects { + let size = serde_json::to_vec(&value.data) + .map_err(|err| TableCatalogStoreError::Invalid(format!("invalid maintenance rename data: {err}")))? + .len(); + total_bytes = total_bytes.saturating_add(size); + if value.etag.is_empty() + || size > TABLE_CATALOG_BACKUP_MAX_MAINTENANCE_OBJECT_SIZE + || total_bytes > TABLE_CATALOG_BACKUP_MAX_MAINTENANCE_BYTES + { + return Err(TableCatalogStoreError::Invalid( + "maintenance rename intent object exceeds its limits".to_string(), + )); + } + maintenance_rename_data(&intent.source, &intent.destination, suffix, &value.data)?; + } + if intent + .destination_etags + .iter() + .any(|(suffix, etag)| !maintenance_rename_suffix_is_valid(suffix) || etag.is_empty()) + { + return Err(TableCatalogStoreError::Invalid( + "invalid maintenance rename destination watermark".to_string(), + )); + } + Ok(Some((intent, etag))) + } + + pub(super) async fn prepare_table_maintenance_rename( + &self, + source: &TableEntry, + destination: &TableEntry, + source_snapshot_etag: &str, + fence: &TableCatalogObjectMutationFence, + ) -> TableCatalogStoreResult> { + fence.ensure_held()?; + let source_objects = self.collect_table_maintenance_rename_objects(source).await?; + let destination_objects = self.collect_table_maintenance_rename_objects(destination).await?; + if source_objects.is_empty() && destination_objects.is_empty() { + return Ok(None); + } + let intent = TableMaintenanceRenameIntent { + version: 1, + source: source.clone(), + destination: destination.clone(), + source_snapshot_etag: source_snapshot_etag.to_string(), + source_objects, + destination_etags: destination_objects + .into_iter() + .map(|(key, value)| (key, value.etag)) + .collect(), + }; + let object = maintenance_rename_intent_path(&source.table_bucket); + let data = serde_json::to_vec(&intent) + .map_err(|err| TableCatalogStoreError::Internal(format!("failed to serialize maintenance rename intent: {err}")))?; + if data.len() > TABLE_CATALOG_BACKUP_MAX_SIZE { + return Err(TableCatalogStoreError::Invalid( + "maintenance rename intent exceeds its byte limit".to_string(), + )); + } + self.backend + .put_object_fenced(self.catalog_bucket(), &object, data, TableCatalogPutPrecondition::IfAbsent, fence) + .await?; + let (persisted_intent, intent_etag) = self + .read_table_maintenance_rename_intent(&source.table_bucket) + .await? + .ok_or_else(|| TableCatalogStoreError::Conflict("maintenance rename intent disappeared".to_string()))?; + if persisted_intent != intent { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename intent changed during preparation".to_string(), + )); + } + let destination_prefix = maintenance_rename_prefix(destination)?; + for (suffix, value) in &intent.source_objects { + let renamed = maintenance_rename_data(source, destination, suffix, &value.data)?; + let data = serde_json::to_vec(&renamed).map_err(|err| { + TableCatalogStoreError::Internal(format!("failed to serialize renamed maintenance object: {err}")) + })?; + let precondition = intent + .destination_etags + .get(suffix) + .map_or(TableCatalogPutPrecondition::IfAbsent, |etag| { + TableCatalogPutPrecondition::IfMatch(etag.clone()) + }); + self.backend + .put_object_fenced(self.catalog_bucket(), &format!("{destination_prefix}{suffix}"), data, precondition, fence) + .await?; + } + // Inactive leftovers from a former visit to this name must not resurrect jobs or overrides. + for (suffix, etag) in &intent.destination_etags { + if !intent.source_objects.contains_key(suffix) { + self.backend + .delete_object_if_match_fenced(self.catalog_bucket(), &format!("{destination_prefix}{suffix}"), etag, fence) + .await?; + } + } + let current_source = self.collect_table_maintenance_rename_objects(source).await?; + if current_source.len() != intent.source_objects.len() + || current_source + .iter() + .any(|(suffix, value)| intent.source_objects.get(suffix).is_none_or(|saved| saved.etag != value.etag)) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename source changed before catalog publication".to_string(), + )); + } + let current_destination = self.collect_table_maintenance_rename_objects(destination).await?; + if current_destination.len() != intent.source_objects.len() { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename destination is not a complete mirror".to_string(), + )); + } + for (suffix, value) in &intent.source_objects { + if current_destination.get(suffix).map(|value| &value.data) + != Some(&maintenance_rename_data(source, destination, suffix, &value.data)?) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename destination changed before publication".to_string(), + )); + } + } + fence.ensure_held()?; + Ok(Some((intent, intent_etag))) + } + + pub(super) async fn finish_table_maintenance_rename( + &self, + intent: &TableMaintenanceRenameIntent, + intent_etag: &str, + committed: bool, + fence: &TableCatalogObjectMutationFence, + ) -> TableCatalogStoreResult<()> { + fence.ensure_held()?; + let authoritative = if committed { &intent.destination } else { &intent.source }; + let authoritative_objects = self.collect_table_maintenance_rename_objects(authoritative).await?; + if intent + .source_objects + .keys() + .any(|suffix| !authoritative_objects.contains_key(suffix)) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename authoritative records are incomplete; repair persisted state before retrying rename" + .to_string(), + )); + } + let entry = if committed { &intent.source } else { &intent.destination }; + let prefix = maintenance_rename_prefix(entry)?; + let current = self.collect_table_maintenance_rename_objects(entry).await?; + // Validate the whole cleanup set before the first delete. A failed or interrupted copy + // is recognizable by its exact planned JSON; every deletion still uses the observed ETag. + for (suffix, value) in ¤t { + let allowed = if committed { + intent + .source_objects + .get(suffix) + .is_some_and(|saved| saved.etag == value.etag) + } else if intent.destination_etags.get(suffix) == Some(&value.etag) { + true + } else if let Some(saved) = intent.source_objects.get(suffix) { + value.data == maintenance_rename_data(&intent.source, &intent.destination, suffix, &saved.data)? + } else { + false + }; + if !allowed { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename cleanup encountered a changed object watermark".to_string(), + )); + } + } + for (suffix, value) in current { + self.backend + .delete_object_if_match_fenced(self.catalog_bucket(), &format!("{prefix}{suffix}"), &value.etag, fence) + .await?; + } + let object = maintenance_rename_intent_path(&intent.source.table_bucket); + match self + .backend + .delete_object_if_match_fenced(self.catalog_bucket(), &object, intent_etag, fence) + .await + { + Ok(()) => Ok(()), + Err(err) => { + fence.ensure_held()?; + // A lost delete response after removal must not strand a committed rename. + if self.backend.object_metadata(self.catalog_bucket(), &object).await?.is_none() { + Ok(()) + } else { + Err(err) + } + } + } + } + pub(in crate::table_catalog) async fn read_entry( &self, bucket: &str, @@ -741,6 +1189,34 @@ where self.backend.put_object(bucket, object, data, precondition).await } + async fn write_maintenance_entry( + &self, + object: &str, + value: &T, + entry: &TableEntry, + publication: Option<&(dyn TableCommitPublication + Sync)>, + ) -> TableCatalogStoreResult<()> + where + T: Serialize, + { + let Some(fence) = table_maintenance_mutation_fence(publication, entry)? else { + return self + .write_entry(self.catalog_bucket(), object, value, TableCatalogPutPrecondition::Any) + .await; + }; + let precondition = match self.backend.object_metadata(self.catalog_bucket(), object).await? { + None => TableCatalogPutPrecondition::IfAbsent, + Some(current) => TableCatalogPutPrecondition::IfMatch(current.etag.ok_or_else(|| { + TableCatalogStoreError::Conflict(format!("maintenance sidecar has no conditional-write ETag: {object}")) + })?), + }; + let data = serde_json::to_vec(value) + .map_err(|err| TableCatalogStoreError::Internal(format!("failed to serialize catalog entry {object}: {err}")))?; + self.backend + .put_object_fenced(self.catalog_bucket(), object, data, precondition, &fence) + .await + } + async fn write_entry_unlocked( &self, bucket: &str, @@ -2660,9 +3136,19 @@ where ))); }; + self.get_table_maintenance_config_for_entry(&entry).await + } + + pub(super) async fn get_table_maintenance_config_for_entry( + &self, + entry: &TableEntry, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let config_path = self .paths - .table_maintenance_config_path(table_bucket, &namespace, &table, &entry.table_id); + .table_maintenance_config_path(&entry.table_bucket, &namespace, &table, &entry.table_id); let config = self .read_entry::(self.catalog_bucket(), &config_path) .await? @@ -2711,7 +3197,7 @@ where .await } - async fn get_effective_table_maintenance_config_for_entry_unlocked( + pub(super) async fn get_effective_table_maintenance_config_for_entry_unlocked( &self, table_bucket: &str, namespace: &Namespace, @@ -2770,11 +3256,34 @@ where ))); }; + self.put_table_maintenance_config_for_entry(&entry, config).await + } + + pub(super) async fn put_table_maintenance_config_for_entry( + &self, + entry: &TableEntry, + config: TableMaintenanceConfig, + ) -> TableCatalogStoreResult { + self.put_table_maintenance_config_for_entry_with_publication(entry, config, None) + .await + } + + pub(super) async fn put_table_maintenance_config_for_entry_with_publication( + &self, + entry: &TableEntry, + config: TableMaintenanceConfig, + publication: Option<&(dyn TableCommitPublication + Sync)>, + ) -> TableCatalogStoreResult { + validate_table_maintenance_config(&config)?; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; let config_path = self .paths - .table_maintenance_config_path(table_bucket, &namespace, &table, &entry.table_id); - self.write_entry(self.catalog_bucket(), &config_path, &config, TableCatalogPutPrecondition::Any) + .table_maintenance_config_path(&entry.table_bucket, &namespace, &table, &entry.table_id); + ensure_table_maintenance_publication_held(publication, entry, "before config write")?; + self.write_maintenance_entry(&config_path, &config, entry, publication) .await?; + ensure_table_maintenance_publication_held(publication, entry, "after config write")?; Ok(config) } @@ -2808,10 +3317,20 @@ where self.put_table_metadata_maintenance_report_for_entry(report, &entry).await } - async fn put_table_metadata_maintenance_report_for_entry( + pub(super) async fn put_table_metadata_maintenance_report_for_entry( &self, report: &TableMetadataMaintenanceReport, entry: &TableEntry, + ) -> TableCatalogStoreResult<()> { + self.put_table_metadata_maintenance_report_for_entry_with_publication(report, entry, None) + .await + } + + pub(super) async fn put_table_metadata_maintenance_report_for_entry_with_publication( + &self, + report: &TableMetadataMaintenanceReport, + entry: &TableEntry, + publication: Option<&(dyn TableCommitPublication + Sync)>, ) -> TableCatalogStoreResult<()> { let report = table_maintenance_report_with_recommended_actions(report.clone()); let namespace = parse_namespace_for_store(&entry.namespace)?; @@ -2826,12 +3345,17 @@ where let current_job_path = self.paths .table_maintenance_current_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id); - self.write_entry(self.catalog_bucket(), &job_path, &report, TableCatalogPutPrecondition::Any) + ensure_table_maintenance_publication_held(publication, entry, "before job report write")?; + self.write_maintenance_entry(&job_path, &report, entry, publication).await?; + ensure_table_maintenance_publication_held(publication, entry, "after job report write")?; + ensure_table_maintenance_publication_held(publication, entry, "before latest report write")?; + self.write_maintenance_entry(&latest_job_path, &report, entry, publication) .await?; - self.write_entry(self.catalog_bucket(), &latest_job_path, &report, TableCatalogPutPrecondition::Any) + ensure_table_maintenance_publication_held(publication, entry, "after latest report write")?; + ensure_table_maintenance_publication_held(publication, entry, "before current report write")?; + self.write_maintenance_entry(¤t_job_path, &report, entry, publication) .await?; - self.write_entry(self.catalog_bucket(), ¤t_job_path, &report, TableCatalogPutPrecondition::Any) - .await + ensure_table_maintenance_publication_held(publication, entry, "after current report write") } pub(crate) async fn get_table_metadata_maintenance_report( @@ -2856,7 +3380,7 @@ where .await } - async fn get_table_metadata_maintenance_report_for_entry_unlocked( + pub(super) async fn get_table_metadata_maintenance_report_for_entry_unlocked( &self, table_bucket: &str, namespace: &Namespace, @@ -2925,20 +3449,45 @@ where table.as_str() ))); }; + self.get_table_maintenance_scheduler_report_for_entry_at(&entry, now).await + } + + pub(super) async fn get_table_maintenance_scheduler_report_for_entry_at( + &self, + entry: &TableEntry, + now: OffsetDateTime, + ) -> TableCatalogStoreResult { + self.get_table_maintenance_scheduler_report_with_audit_reports_for_entry_at(entry, now, None) + .await + } + + pub(super) async fn get_table_maintenance_scheduler_report_with_audit_reports_for_entry_at( + &self, + entry: &TableEntry, + now: OffsetDateTime, + reports: Option>, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; let effective = self - .get_effective_table_maintenance_config(table_bucket, &namespace.public_name(), table.as_str()) + .get_effective_table_maintenance_config_for_entry_unlocked(&entry.table_bucket, &namespace, &table, entry) .await?; let current = self - .get_table_metadata_maintenance_report( - table_bucket, - &namespace.public_name(), - table.as_str(), + .get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, MAINTENANCE_JOB_ALIAS_CURRENT, ) .await?; - let reports = self - .list_table_metadata_maintenance_audit_reports(table_bucket, &namespace, &table, &entry.table_id) - .await?; + let reports = match reports { + Some(reports) => reports, + None => { + self.list_table_metadata_maintenance_audit_reports(&entry.table_bucket, &namespace, &table, &entry.table_id) + .await? + } + }; let quarantine = table_maintenance_scheduler_quarantine_boundary(&effective.config, &reports); let mut recommended_actions = Vec::new(); @@ -2978,10 +3527,10 @@ where }; Ok(TableMaintenanceSchedulerReport { - table_bucket: table_bucket.to_string(), + table_bucket: entry.table_bucket.clone(), namespace: namespace.public_name(), table: table.as_str().to_string(), - table_id: entry.table_id, + table_id: entry.table_id.clone(), status, config_source: effective.source, background_enabled: effective.config.background_enabled, @@ -3016,6 +3565,19 @@ where table: &str, scheduler_id: String, now: OffsetDateTime, + ) -> TableCatalogStoreResult { + self.run_table_maintenance_scheduler_once_with_guard_at(table_bucket, namespace, table, scheduler_id, now, None) + .await + } + + pub(super) async fn run_table_maintenance_scheduler_once_with_guard_at( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + scheduler_id: String, + now: OffsetDateTime, + periodic_guard: Option<&PeriodicTableMaintenanceQueueGuard<'_>>, ) -> TableCatalogStoreResult { let namespace = parse_namespace_for_store(namespace)?; let table = parse_table_for_store(table)?; @@ -3035,6 +3597,19 @@ where let effective = self .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) .await?; + if let Some(periodic_guard) = periodic_guard { + self.validate_periodic_table_maintenance_queue_guard( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &effective, + periodic_guard, + ) + .await?; + } self.table_metadata_maintenance_scheduler_preflight( TableMaintenancePreflightContext { table_bucket, @@ -3081,6 +3656,19 @@ where let effective = self .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) .await?; + if let Some(periodic_guard) = periodic_guard { + self.validate_periodic_table_maintenance_queue_guard( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &effective, + periodic_guard, + ) + .await?; + } match self .table_metadata_maintenance_scheduler_preflight( TableMaintenancePreflightContext { @@ -3278,26 +3866,105 @@ where let jobs_prefix = self .paths .table_maintenance_jobs_prefix(table_bucket, namespace, table, table_id); + let objects = self.backend.list_objects(self.catalog_bucket(), &jobs_prefix).await?; + self.read_table_metadata_maintenance_audit_reports(table_bucket, namespace, table, table_id, &objects, false) + .await + } + + pub(super) async fn list_table_metadata_maintenance_audit_objects_for_entry( + &self, + entry: &TableEntry, + ) -> TableCatalogStoreResult> { + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let prefix = self + .paths + .table_maintenance_jobs_prefix(&entry.table_bucket, &namespace, &table, &entry.table_id); + let mut objects = Vec::new(); + let mut cursor: Option = None; + loop { + let remaining = TABLE_MAINTENANCE_SCHEDULER_HISTORY_MAX_OBJECTS.saturating_sub(objects.len()); + let limit = NonZeroUsize::new(TABLE_CATALOG_LIST_MAX_KEYS.min(remaining + 1)).ok_or_else(|| { + TableCatalogStoreError::Internal("maintenance scheduler page limit must be positive".to_string()) + })?; + let page = self + .backend + .list_objects_page(self.catalog_bucket(), &prefix, cursor.as_deref(), limit) + .await?; + if page.objects.len() > limit.get() || (page.is_truncated && page.objects.is_empty()) { + return Err(TableCatalogStoreError::Invalid( + "invalid maintenance scheduler history listing page".to_string(), + )); + } + for object in page.objects { + if !object.starts_with(&prefix) || cursor.as_ref().is_some_and(|cursor| object <= *cursor) { + return Err(TableCatalogStoreError::Invalid( + "maintenance scheduler history listing escaped its prefix or did not advance".to_string(), + )); + } + if objects.len() == TABLE_MAINTENANCE_SCHEDULER_HISTORY_MAX_OBJECTS { + return Err(TableCatalogStoreError::Unavailable( + "maintenance scheduler history exceeds the 4096-object read budget".to_string(), + )); + } + cursor = Some(object.clone()); + objects.push(object); + } + if !page.is_truncated { + return Ok(objects); + } + } + } + + pub(super) async fn read_table_metadata_maintenance_audit_reports_for_entry( + &self, + entry: &TableEntry, + objects: &[String], + ) -> TableCatalogStoreResult> { + self.read_table_metadata_maintenance_audit_reports( + &entry.table_bucket, + &parse_namespace_for_store(&entry.namespace)?, + &parse_table_for_store(&entry.table)?, + &entry.table_id, + objects, + true, + ) + .await + } + + async fn read_table_metadata_maintenance_audit_reports( + &self, + table_bucket: &str, + namespace: &Namespace, + table: &IdentifierSegment, + table_id: &str, + objects: &[String], + require_present: bool, + ) -> TableCatalogStoreResult> { let mut reports = Vec::new(); - for object in self.backend.list_objects(self.catalog_bucket(), &jobs_prefix).await? { + for object in objects { if !object.ends_with(".json") { continue; } if let Some((report, _)) = self - .read_entry::(self.catalog_bucket(), &object) + .read_entry::(self.catalog_bucket(), object) .await? { validate_table_maintenance_report_owner(&report, table_bucket, namespace, table, table_id)?; if self .paths .table_maintenance_job_path(table_bucket, namespace, table, table_id, &report.job.job_id) - != object + != *object { return Err(TableCatalogStoreError::Invalid( "table maintenance report identity does not match its object path".to_string(), )); } reports.push(table_maintenance_report_with_recommended_actions(report)); + } else if require_present { + return Err(TableCatalogStoreError::Unavailable( + "maintenance scheduler history changed during the read; retry the scheduler report".to_string(), + )); } } reports.sort_by(|left, right| { @@ -3309,42 +3976,115 @@ where Ok(reports) } - async fn table_metadata_maintenance_scheduler_preflight( + pub(super) async fn validate_periodic_table_maintenance_queue_guard( + &self, + context: TableMaintenancePreflightContext<'_>, + effective: &TableMaintenanceEffectiveConfig, + expected: &PeriodicTableMaintenanceQueueGuard<'_>, + ) -> TableCatalogStoreResult<()> { + if context.entry.table_id != expected.table_id { + return Err(TableCatalogStoreError::Conflict( + "table identity changed before periodic maintenance claim".to_string(), + )); + } + if !effective.config.background_enabled || effective.config.worker_paused { + return Err(TableCatalogStoreError::Conflict("periodic maintenance is disabled or paused".to_string())); + } + let current = self + .get_table_metadata_maintenance_report_for_entry_unlocked( + context.table_bucket, + context.namespace, + context.table, + &context.entry.table_id, + MAINTENANCE_JOB_ALIAS_CURRENT, + ) + .await?; + if current.as_ref().map(table_maintenance_scheduler_job_summary).as_ref() != expected.current_job + || current + .as_ref() + .is_some_and(|report| report.job.status != TableMetadataMaintenanceJobStatus::Successful) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance job changed or requires explicit operator action before periodic scheduling".to_string(), + )); + } + let latest = self + .get_table_metadata_maintenance_report_for_entry_unlocked( + context.table_bucket, + context.namespace, + context.table, + &context.entry.table_id, + MAINTENANCE_JOB_ALIAS_LATEST, + ) + .await?; + // Report publication writes latest before current. A partial write must + // not let an automatic pass replace an unresolved job or quarantine. + if latest != current + || current + .as_ref() + .is_some_and(|report| report.job.quarantine_enabled && report.job.quarantined_object_count > 0) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance report publication or quarantine requires recovery before periodic scheduling".to_string(), + )); + } + Ok(()) + } + + pub(super) async fn table_metadata_maintenance_scheduler_preflight( &self, context: TableMaintenancePreflightContext<'_>, scheduler_id: &str, now: OffsetDateTime, effective: TableMaintenanceEffectiveConfig, + ) -> TableCatalogStoreResult { + self.table_metadata_maintenance_scheduler_preflight_with_publication(context, scheduler_id, now, effective, None) + .await + } + + pub(super) async fn table_metadata_maintenance_scheduler_preflight_with_publication( + &self, + context: TableMaintenancePreflightContext<'_>, + scheduler_id: &str, + now: OffsetDateTime, + effective: TableMaintenanceEffectiveConfig, + publication: Option<&(dyn TableCommitPublication + Sync)>, ) -> TableCatalogStoreResult { if !effective.config.background_enabled { let report = self - .put_table_metadata_maintenance_scheduler_control_report(TableMaintenanceSchedulerControlReport { - table_bucket: context.table_bucket, - namespace: context.namespace, - table: context.table, - entry: context.entry, - scheduler_id: scheduler_id.to_string(), - effective: &effective, - status: TableMetadataMaintenanceJobStatus::Disabled, - reason: "background maintenance is disabled", - now, - }) + .put_table_metadata_maintenance_scheduler_control_report( + TableMaintenanceSchedulerControlReport { + table_bucket: context.table_bucket, + namespace: context.namespace, + table: context.table, + entry: context.entry, + scheduler_id: scheduler_id.to_string(), + effective: &effective, + status: TableMetadataMaintenanceJobStatus::Disabled, + reason: "background maintenance is disabled", + now, + }, + publication, + ) .await?; return Ok(TableMaintenanceSchedulerPreflight::Complete(Box::new(report))); } if effective.config.worker_paused { let report = self - .put_table_metadata_maintenance_scheduler_control_report(TableMaintenanceSchedulerControlReport { - table_bucket: context.table_bucket, - namespace: context.namespace, - table: context.table, - entry: context.entry, - scheduler_id: scheduler_id.to_string(), - effective: &effective, - status: TableMetadataMaintenanceJobStatus::Paused, - reason: "background maintenance worker is paused", - now, - }) + .put_table_metadata_maintenance_scheduler_control_report( + TableMaintenanceSchedulerControlReport { + table_bucket: context.table_bucket, + namespace: context.namespace, + table: context.table, + entry: context.entry, + scheduler_id: scheduler_id.to_string(), + effective: &effective, + status: TableMetadataMaintenanceJobStatus::Paused, + reason: "background maintenance worker is paused", + now, + }, + publication, + ) .await?; return Ok(TableMaintenanceSchedulerPreflight::Complete(Box::new(report))); } @@ -3369,6 +4109,7 @@ where now, "maintenance worker lease expired", TableMaintenanceAuditAction::WorkerLeaseExpired, + publication, ) .await?; } else if matches!(current.job.status, TableMetadataMaintenanceJobStatus::Queued) { @@ -3381,6 +4122,7 @@ where now, "maintenance scheduler lease expired", TableMaintenanceAuditAction::SchedulerLeaseExpired, + publication, ) .await?; } else if table_maintenance_job_retry_is_pending(¤t.job, now) { @@ -3391,6 +4133,169 @@ where Ok(TableMaintenanceSchedulerPreflight::Ready(effective)) } + pub(crate) async fn run_table_metadata_maintenance_worker_once_conditionally( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + worker_id: String, + condition: &TableMaintenanceWorkerCondition, + ) -> TableCatalogStoreResult { + self.run_table_metadata_maintenance_worker_once_conditionally_with_clock( + table_bucket, + namespace, + table, + worker_id, + condition, + OffsetDateTime::now_utc, + ) + .await + } + + pub(in crate::table_catalog) async fn run_table_metadata_maintenance_worker_once_conditionally_with_clock( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + worker_id: String, + condition: &TableMaintenanceWorkerCondition, + clock: impl FnOnce() -> OffsetDateTime + Send, + ) -> TableCatalogStoreResult { + validate_table_maintenance_worker_condition(&worker_id, condition)?; + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = TableCommitLockPublication::new(&self.backend); + publication.attach_catalog_migration_read_guard(self.acquire_object_backed_catalog_write_permit(table_bucket).await?)?; + publication.begin_table_bucket(table_bucket).await?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + self.ensure_no_active_table_rename(table_bucket).await?; + let table_path = self.paths.table_entry_path(table_bucket, &namespace, &table); + let guard = self.backend.acquire_write_lock(self.catalog_bucket(), &table_path).await?; + publication.attach_catalog_write_guard(guard)?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + let Some((entry, _)) = self.read_table_with_etag_unlocked(table_bucket, &namespace, &table).await? else { + return Err(TableCatalogStoreError::NotFound(format!( + "table {}/{}/{}", + table_bucket, + namespace.public_name(), + table.as_str() + ))); + }; + self.run_conditional_table_maintenance_worker_for_entry(&entry, worker_id, condition, clock, &publication) + .await + } + + pub(super) async fn run_conditional_table_maintenance_worker_for_entry( + &self, + entry: &TableEntry, + worker_id: String, + condition: &TableMaintenanceWorkerCondition, + clock: impl FnOnce() -> OffsetDateTime + Send, + publication: &(dyn TableCommitPublication + Sync), + ) -> TableCatalogStoreResult { + validate_table_maintenance_worker_condition(&worker_id, condition)?; + ensure_table_maintenance_publication_held(Some(publication), entry, "before conditional worker claim")?; + if entry.state != TableCatalogEntryState::Active { + return Err(TableCatalogStoreError::Conflict( + "conditional maintenance requires an active table".to_string(), + )); + } + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let effective = self + .get_effective_table_maintenance_config_for_entry_unlocked(&entry.table_bucket, &namespace, &table, entry) + .await?; + let mut reports = Vec::with_capacity(3); + for job_id in [ + condition.expected_job_id.as_str(), + MAINTENANCE_JOB_ALIAS_LATEST, + MAINTENANCE_JOB_ALIAS_CURRENT, + ] { + let report = self.get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, &namespace, &table, &entry.table_id, job_id, + ).await?.ok_or_else(|| TableCatalogStoreError::Conflict( + "conditional maintenance requires fully published named, latest, and current reports; inspect the persisted job".to_string(), + ))?; + reports.push(report); + } + let mut report = reports.remove(0); + // Lock acquisition and report reads can outlast the queued job's lease. + // Sample the claim time only after those waits have completed. + let now = clock(); + let replay = validate_conditional_table_maintenance_claim(&report, entry, &effective, &worker_id, condition, now)?; + if reports.iter().any(|alias| alias != &report) { + // A complete terminal receipt can finish publishing aliases after a + // lost response. Never resume a partial Running claim or rerun deletes. + if !replay + || !matches!( + report.job.status, + TableMetadataMaintenanceJobStatus::Successful | TableMetadataMaintenanceJobStatus::Failed + ) + || reports + .iter() + .any(|alias| alias != &report && !maintenance_terminal_report_continues_running(&report, alias)) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance report publication is incomplete; inspect the named job and recover before another claim" + .to_string(), + )); + } + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, entry, Some(publication)) + .await?; + return Ok(report); + } + if replay { + return Ok(report); + } + + // Backoff delays follow the validated current policy; its attempt limit + // must never replace the budget persisted when this job was scheduled. + let mut retry_effective = effective; + retry_effective.config.max_retry_attempts = report.job.max_retry_attempts; + validate_table_maintenance_config(&retry_effective.config)?; + let before_status = Some(report.job.status.clone()); + let before_quarantined_object_count = Some(report.job.quarantined_object_count); + let started_at = maintenance_timestamp(now); + report.job.attempt = report + .job + .attempt + .checked_add(1) + .ok_or_else(|| TableCatalogStoreError::Invalid("maintenance attempt exceeds its supported range".to_string()))?; + report.job.status = TableMetadataMaintenanceJobStatus::Running; + report.job.failure_reason = None; + report.job.worker_id = Some(worker_id); + report.job.lease_id = Uuid::new_v4().to_string(); + report.job.next_retry_after = None; + report.job.heartbeat_at = Some(started_at.clone()); + report.job.started_at = Some(started_at); + report.job.finished_at = None; + refresh_table_maintenance_report_recommended_actions(&mut report); + push_table_maintenance_audit_event( + &mut report, + now, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerStarted, + None, + before_status, + before_quarantined_object_count, + ); + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, entry, Some(publication)) + .await?; + + let delete = matches!(report.job.operation, TableMetadataMaintenanceOperation::Delete); + self.finish_table_metadata_maintenance_run_for_entry_with_optional_publication( + entry, + delete, + &retry_effective, + report, + Some(publication), + true, + ) + .await + } + pub(crate) async fn run_table_metadata_maintenance_worker_once( &self, table_bucket: &str, @@ -3551,42 +4456,60 @@ where .await } - async fn table_metadata_maintenance_worker_preflight( + pub(super) async fn table_metadata_maintenance_worker_preflight( &self, context: TableMaintenancePreflightContext<'_>, worker_id: &str, now: OffsetDateTime, effective: TableMaintenanceEffectiveConfig, + ) -> TableCatalogStoreResult { + self.table_metadata_maintenance_worker_preflight_with_publication(context, worker_id, now, effective, None) + .await + } + + pub(super) async fn table_metadata_maintenance_worker_preflight_with_publication( + &self, + context: TableMaintenancePreflightContext<'_>, + worker_id: &str, + now: OffsetDateTime, + effective: TableMaintenanceEffectiveConfig, + publication: Option<&(dyn TableCommitPublication + Sync)>, ) -> TableCatalogStoreResult { if !effective.config.background_enabled { let report = self - .put_table_metadata_maintenance_worker_control_report(TableMaintenanceWorkerControlReport { - table_bucket: context.table_bucket, - namespace: context.namespace, - table: context.table, - entry: context.entry, - worker_id: worker_id.to_string(), - effective: &effective, - status: TableMetadataMaintenanceJobStatus::Disabled, - reason: "background maintenance is disabled", - now, - }) + .put_table_metadata_maintenance_worker_control_report( + TableMaintenanceWorkerControlReport { + table_bucket: context.table_bucket, + namespace: context.namespace, + table: context.table, + entry: context.entry, + worker_id: worker_id.to_string(), + effective: &effective, + status: TableMetadataMaintenanceJobStatus::Disabled, + reason: "background maintenance is disabled", + now, + }, + publication, + ) .await?; return Ok(TableMaintenanceWorkerPreflight::Complete(Box::new(report))); } if effective.config.worker_paused { let report = self - .put_table_metadata_maintenance_worker_control_report(TableMaintenanceWorkerControlReport { - table_bucket: context.table_bucket, - namespace: context.namespace, - table: context.table, - entry: context.entry, - worker_id: worker_id.to_string(), - effective: &effective, - status: TableMetadataMaintenanceJobStatus::Paused, - reason: "background maintenance worker is paused", - now, - }) + .put_table_metadata_maintenance_worker_control_report( + TableMaintenanceWorkerControlReport { + table_bucket: context.table_bucket, + namespace: context.namespace, + table: context.table, + entry: context.entry, + worker_id: worker_id.to_string(), + effective: &effective, + status: TableMetadataMaintenanceJobStatus::Paused, + reason: "background maintenance worker is paused", + now, + }, + publication, + ) .await?; return Ok(TableMaintenanceWorkerPreflight::Complete(Box::new(report))); } @@ -3611,6 +4534,7 @@ where now, "maintenance worker lease expired", TableMaintenanceAuditAction::WorkerLeaseExpired, + publication, ) .await?; } else if matches!(current.job.status, TableMetadataMaintenanceJobStatus::Queued) { @@ -3626,6 +4550,7 @@ where now, "maintenance scheduler lease expired", TableMaintenanceAuditAction::SchedulerLeaseExpired, + publication, ) .await?; } else if table_maintenance_job_retry_is_pending(¤t.job, now) { @@ -3682,9 +4607,32 @@ where table.as_str() ))); }; + self.heartbeat_table_metadata_maintenance_job_for_entry(&entry, heartbeat, now) + .await + } + + pub(super) async fn heartbeat_table_metadata_maintenance_job_for_entry( + &self, + entry: &TableEntry, + heartbeat: TableMaintenanceHeartbeatRef<'_>, + now: OffsetDateTime, + ) -> TableCatalogStoreResult { + self.heartbeat_table_metadata_maintenance_job_for_entry_with_publication(entry, heartbeat, now, None) + .await + } + + pub(super) async fn heartbeat_table_metadata_maintenance_job_for_entry_with_publication( + &self, + entry: &TableEntry, + heartbeat: TableMaintenanceHeartbeatRef<'_>, + now: OffsetDateTime, + publication: Option<&(dyn TableCommitPublication + Sync)>, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; let Some(mut report) = self .get_table_metadata_maintenance_report_for_entry_unlocked( - heartbeat.table_bucket, + &entry.table_bucket, &namespace, &table, &entry.table_id, @@ -3725,7 +4673,8 @@ where Some(TableMetadataMaintenanceJobStatus::Running), before_quarantined_object_count, ); - self.put_table_metadata_maintenance_report_for_entry(&report, &entry).await?; + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, entry, publication) + .await?; Ok(report) } @@ -3736,6 +4685,7 @@ where now: OffsetDateTime, reason: &str, action: TableMaintenanceAuditAction, + publication: Option<&(dyn TableCommitPublication + Sync)>, ) -> TableCatalogStoreResult { let before_status = Some(report.job.status.clone()); let before_quarantined_object_count = Some(report.job.quarantined_object_count); @@ -3752,13 +4702,15 @@ where before_status, before_quarantined_object_count, ); - self.put_table_metadata_maintenance_report_for_entry(&report, entry).await?; + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, entry, publication) + .await?; Ok(report) } async fn put_table_metadata_maintenance_scheduler_control_report( &self, control: TableMaintenanceSchedulerControlReport<'_>, + publication: Option<&(dyn TableCommitPublication + Sync)>, ) -> TableCatalogStoreResult { let timestamp = maintenance_timestamp(control.now); let cleanup_watermark_unix_seconds = @@ -3829,7 +4781,7 @@ where None, None, ); - self.put_table_metadata_maintenance_report_for_entry(&report, control.entry) + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, control.entry, publication) .await?; Ok(report) } @@ -3837,6 +4789,7 @@ where async fn put_table_metadata_maintenance_worker_control_report( &self, control: TableMaintenanceWorkerControlReport<'_>, + publication: Option<&(dyn TableCommitPublication + Sync)>, ) -> TableCatalogStoreResult { let timestamp = maintenance_timestamp(control.now); let cleanup_watermark_unix_seconds = @@ -3907,7 +4860,7 @@ where None, None, ); - self.put_table_metadata_maintenance_report_for_entry(&report, control.entry) + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, control.entry, publication) .await?; Ok(report) } @@ -3951,7 +4904,30 @@ where )); } - let Some(current_metadata) = read_table_metadata_value(metadata_backend, table_bucket, &entry.metadata_location).await? + self.plan_table_snapshot_expiration_for_entry_with_backend(metadata_backend, &entry, config) + .await + } + + pub(super) async fn plan_table_snapshot_expiration_for_entry_with_backend

( + &self, + metadata_backend: &P, + entry: &TableEntry, + config: TableSnapshotExpirationConfig, + ) -> TableCatalogStoreResult + where + P: TableCatalogObjectBackend, + { + validate_table_snapshot_expiration_config(&config)?; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + if !is_valid_table_metadata_location_for_entry(entry, &entry.metadata_location) { + return Err(TableCatalogStoreError::Invalid( + "current metadata location must be inside the table metadata directory".to_string(), + )); + } + + let Some(current_metadata) = + read_table_metadata_value(metadata_backend, &entry.table_bucket, &entry.metadata_location).await? else { return Err(TableCatalogStoreError::NotFound(format!( "current metadata object {}", @@ -3960,10 +4936,10 @@ where }; Ok(table_snapshot_expiration_report( - table_bucket, + &entry.table_bucket, &namespace, &table, - &entry, + entry, ¤t_metadata, config, OffsetDateTime::now_utc(), @@ -3994,7 +4970,25 @@ where )); } - let Some(current_metadata) = read_table_metadata_value(&self.backend, table_bucket, &entry.metadata_location).await? + self.plan_table_compaction_for_entry(&entry, config).await + } + + pub(super) async fn plan_table_compaction_for_entry( + &self, + entry: &TableEntry, + config: TableCompactionPlanningConfig, + ) -> TableCatalogStoreResult { + validate_table_compaction_planning_config(&config)?; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + if !is_valid_table_metadata_location_for_entry(entry, &entry.metadata_location) { + return Err(TableCatalogStoreError::Invalid( + "current metadata location must be inside the table metadata directory".to_string(), + )); + } + + let Some(current_metadata) = + read_table_metadata_value(&self.backend, &entry.table_bucket, &entry.metadata_location).await? else { return Err(TableCatalogStoreError::NotFound(format!( "current metadata object {}", @@ -4002,7 +4996,8 @@ where ))); }; - table_compaction_planning_report(&self.backend, table_bucket, &namespace, &table, &entry, ¤t_metadata, config).await + table_compaction_planning_report(&self.backend, &entry.table_bucket, &namespace, &table, entry, ¤t_metadata, config) + .await } #[allow( @@ -4301,7 +5296,36 @@ where table.as_str() ))); }; - if !is_valid_table_metadata_location(&namespace, &table, &entry.metadata_location) { + self.plan_table_metadata_maintenance_for_entry_with_scope(&entry, retain_recent_metadata_files, false) + .await + } + + pub(super) async fn plan_table_metadata_maintenance_for_entry( + &self, + entry: &TableEntry, + retain_recent_metadata_files: usize, + ) -> TableCatalogStoreResult { + self.plan_table_metadata_maintenance_for_entry_with_scope(entry, retain_recent_metadata_files, true) + .await + } + + async fn plan_table_metadata_maintenance_for_entry_with_scope( + &self, + entry: &TableEntry, + retain_recent_metadata_files: usize, + entry_aware: bool, + ) -> TableCatalogStoreResult { + let table_bucket = &entry.table_bucket; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let is_valid_location = |location: &str| { + if entry_aware { + is_valid_table_metadata_location_for_entry(entry, location) + } else { + is_valid_table_metadata_location(&namespace, &table, location) + } + }; + if !is_valid_location(&entry.metadata_location) { return Err(TableCatalogStoreError::Invalid( "current metadata location must be inside the table metadata directory".to_string(), )); @@ -4317,7 +5341,7 @@ where let mut retained = BTreeSet::new(); let mut maintenance_reasons = BTreeMap::>::new(); - for metadata_location in metadata_log_locations(¤t_metadata, table_bucket, &namespace, &table) { + for metadata_location in metadata_log_locations_matching(¤t_metadata, table_bucket, is_valid_location) { retained.insert(metadata_location.clone()); insert_metadata_maintenance_reason( &mut maintenance_reasons, @@ -4333,9 +5357,19 @@ where ); let mut metadata_locations = Vec::new(); - let metadata_prefix = format!("{}/", default_table_metadata_dir_path(&namespace, &table)); + let metadata_dir = if entry_aware { + table_metadata_dir_path_for_entry(entry)? + } else { + default_table_metadata_dir_path(&namespace, &table) + }; + let metadata_prefix = format!("{metadata_dir}/"); for object in self.backend.list_objects(table_bucket, &metadata_prefix).await? { - if let Some(metadata_location) = metadata_location_from_metadata_file_path(&namespace, &table, &object) { + let metadata_location = if entry_aware { + is_valid_location(&object).then_some(object) + } else { + metadata_location_from_metadata_file_path(&namespace, &table, &object) + }; + if let Some(metadata_location) = metadata_location { metadata_locations.push(metadata_location); } } @@ -4353,16 +5387,27 @@ where ); } } - for metadata_location in metadata_locations_for_protected_snapshot_refs( - &self.backend, - table_bucket, - &namespace, - &table, - ¤t_metadata, - &metadata_locations, - ) - .await? - { + let protected_refs = if entry_aware { + metadata_locations_for_protected_snapshot_refs_matching( + &self.backend, + table_bucket, + ¤t_metadata, + &metadata_locations, + is_valid_location, + ) + .await? + } else { + metadata_locations_for_protected_snapshot_refs( + &self.backend, + table_bucket, + &namespace, + &table, + ¤t_metadata, + &metadata_locations, + ) + .await? + }; + for metadata_location in protected_refs { retained.insert(metadata_location.clone()); insert_metadata_maintenance_reason( &mut maintenance_reasons, @@ -4408,33 +5453,55 @@ where ); } } - let warehouse_object_prefix = table_warehouse_object_prefix(&entry).ok(); - let current_metadata_location = entry.metadata_location; + let warehouse_object_prefix = table_warehouse_object_prefix(entry).ok(); + let current_metadata_location = entry.metadata_location.clone(); let retained_metadata_locations = retained.into_iter().collect::>(); let object_reports = metadata_maintenance_object_reports(maintenance_reasons); - let referenced_object_reports = metadata_maintenance_referenced_object_reports( - &self.backend, - table_bucket, - &namespace, - &table, - warehouse_object_prefix.as_deref(), - ¤t_metadata, - &retained_metadata_locations, - ) - .await?; - let reachability_graph = - metadata_maintenance_reachability_graph_report(planned_metadata_file_count, &referenced_object_reports); - let (planned_object_file_count, cleanup_object_candidate_locations, deletable_object_locations, object_cleanup_reports) = - metadata_maintenance_object_cleanup_reports( + let referenced_object_reports = if entry_aware { + metadata_maintenance_referenced_object_reports_for_entry( + &self.backend, + entry, + warehouse_object_prefix.as_deref(), + ¤t_metadata, + &retained_metadata_locations, + ) + .await? + } else { + metadata_maintenance_referenced_object_reports( &self.backend, table_bucket, &namespace, &table, warehouse_object_prefix.as_deref(), - &referenced_object_reports, - now, + ¤t_metadata, + &retained_metadata_locations, ) - .await?; + .await? + }; + let reachability_graph = + metadata_maintenance_reachability_graph_report(planned_metadata_file_count, &referenced_object_reports); + let (planned_object_file_count, cleanup_object_candidate_locations, deletable_object_locations, object_cleanup_reports) = + if entry_aware { + metadata_maintenance_object_cleanup_reports_for_entry( + &self.backend, + entry, + warehouse_object_prefix.as_deref(), + &referenced_object_reports, + now, + ) + .await? + } else { + metadata_maintenance_object_cleanup_reports( + &self.backend, + table_bucket, + &namespace, + &table, + warehouse_object_prefix.as_deref(), + &referenced_object_reports, + now, + ) + .await? + }; let mut report = table_maintenance_report_with_recommended_actions(TableMetadataMaintenanceReport { job: TableMetadataMaintenanceJob { @@ -4442,7 +5509,7 @@ where table_bucket: table_bucket.to_string(), namespace: namespace.public_name(), table: table.as_str().to_string(), - table_id: entry.table_id, + table_id: entry.table_id.clone(), operation: TableMetadataMaintenanceOperation::DryRun, status: TableMetadataMaintenanceJobStatus::Successful, failure_reason: None, @@ -4707,6 +5774,134 @@ where Ok(report) } + pub(super) async fn finish_table_metadata_maintenance_run_for_entry_with_publication( + &self, + entry: &TableEntry, + delete: bool, + effective: &TableMaintenanceEffectiveConfig, + report: TableMetadataMaintenanceReport, + publication: &(dyn TableCommitPublication + Sync), + ) -> TableCatalogStoreResult { + self.finish_table_metadata_maintenance_run_for_entry_with_optional_publication( + entry, + delete, + effective, + report, + Some(publication), + false, + ) + .await + } + + async fn finish_table_metadata_maintenance_run_for_entry_with_optional_publication( + &self, + entry: &TableEntry, + delete: bool, + effective: &TableMaintenanceEffectiveConfig, + mut report: TableMetadataMaintenanceReport, + publication: Option<&(dyn TableCommitPublication + Sync)>, + preserve_candidates: bool, + ) -> TableCatalogStoreResult { + if delete && !effective.config.delete_enabled { + let finished_at = OffsetDateTime::now_utc(); + let before_status = Some(report.job.status.clone()); + let before_quarantined_object_count = Some(report.job.quarantined_object_count); + report.job.status = TableMetadataMaintenanceJobStatus::Failed; + report.job.failure_reason = Some(TABLE_MAINTENANCE_DELETE_DISABLED_REASON.to_string()); + apply_maintenance_retry_after(&mut report.job, &effective.config, finished_at); + report.job.finished_at = Some(maintenance_timestamp(finished_at)); + refresh_table_maintenance_report_recommended_actions(&mut report); + push_table_maintenance_audit_event( + &mut report, + finished_at, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerFailed, + Some(TABLE_MAINTENANCE_DELETE_DISABLED_REASON.to_string()), + before_status, + before_quarantined_object_count, + ); + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, entry, publication) + .await?; + return Ok(report); + } + + if delete { + let running_report = report.clone(); + let mut deleted = match self + .delete_table_metadata_maintenance_report_for_entry_with_publication(entry, report, publication) + .await + { + Ok(report) => report, + Err(err) => { + let finished_at = OffsetDateTime::now_utc(); + let mut failed = running_report; + let before_status = Some(failed.job.status.clone()); + let before_quarantined_object_count = Some(failed.job.quarantined_object_count); + let reason = err.to_string(); + failed.job.status = TableMetadataMaintenanceJobStatus::Failed; + failed.job.failure_reason = Some(reason.clone()); + apply_maintenance_retry_after(&mut failed.job, &effective.config, finished_at); + failed.job.finished_at = Some(maintenance_timestamp(finished_at)); + refresh_table_maintenance_report_recommended_actions(&mut failed); + push_table_maintenance_audit_event( + &mut failed, + finished_at, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerFailed, + Some(reason), + before_status, + before_quarantined_object_count, + ); + self.put_table_metadata_maintenance_report_for_entry_with_publication(&failed, entry, publication) + .await?; + return Err(err); + } + }; + if preserve_candidates { + deleted.cleanup_candidate_locations = running_report.cleanup_candidate_locations; + deleted.deletable_metadata_locations = running_report.deletable_metadata_locations; + deleted.cleanup_object_candidate_locations = running_report.cleanup_object_candidate_locations; + deleted.deletable_object_locations = running_report.deletable_object_locations; + } + let finished_at = OffsetDateTime::now_utc(); + let before_status = Some(TableMetadataMaintenanceJobStatus::Running); + let before_quarantined_object_count = Some(0); + deleted.job.finished_at = Some(maintenance_timestamp(finished_at)); + refresh_table_maintenance_report_recommended_actions(&mut deleted); + push_table_maintenance_audit_event( + &mut deleted, + finished_at, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerSucceeded, + None, + before_status, + before_quarantined_object_count, + ); + self.put_table_metadata_maintenance_report_for_entry_with_publication(&deleted, entry, publication) + .await?; + return Ok(deleted); + } + + let finished_at = OffsetDateTime::now_utc(); + let before_status = Some(report.job.status.clone()); + let before_quarantined_object_count = Some(report.job.quarantined_object_count); + report.job.status = TableMetadataMaintenanceJobStatus::Successful; + report.job.finished_at = Some(maintenance_timestamp(finished_at)); + refresh_table_maintenance_report_recommended_actions(&mut report); + push_table_maintenance_audit_event( + &mut report, + finished_at, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerSucceeded, + None, + before_status, + before_quarantined_object_count, + ); + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, entry, publication) + .await?; + Ok(report) + } + #[allow( dead_code, reason = "exercised by table_catalog/tests.rs; the lib target cannot see test-only consumers (backlog#1823)" @@ -4750,12 +5945,47 @@ where table.as_str() ))); }; + self.delete_table_metadata_maintenance_report_for_entry(&entry, report).await + } + + pub(super) async fn delete_table_metadata_maintenance_report_for_entry( + &self, + entry: &TableEntry, + report: TableMetadataMaintenanceReport, + ) -> TableCatalogStoreResult { + self.delete_table_metadata_maintenance_report_for_entry_with_publication(entry, report, None) + .await + } + + async fn delete_table_metadata_maintenance_report_for_entry_with_publication( + &self, + entry: &TableEntry, + report: TableMetadataMaintenanceReport, + publication: Option<&(dyn TableCommitPublication + Sync)>, + ) -> TableCatalogStoreResult { + let mutation_fence = table_maintenance_mutation_fence(publication, entry)?; + let table_bucket = &entry.table_bucket; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let is_valid_location = |location: &str| { + if publication.is_some() { + is_valid_table_metadata_location_for_entry(entry, location) + } else { + is_valid_table_metadata_location(&namespace, &table, location) + } + }; + if !is_valid_location(&report.current_metadata_location) { + return Err(TableCatalogStoreError::Invalid( + "maintenance report current metadata location must be inside the table metadata directory".to_string(), + )); + } + if entry.metadata_location != report.current_metadata_location { return Err(TableCatalogStoreError::Conflict( "current metadata location changed before maintenance delete".to_string(), )); } - let warehouse_object_prefix = table_warehouse_object_prefix(&entry).ok(); + let warehouse_object_prefix = table_warehouse_object_prefix(entry).ok(); let Some(current_metadata) = read_table_metadata_value(&self.backend, table_bucket, &entry.metadata_location).await? else { @@ -4765,17 +5995,16 @@ where ))); }; - let mut protected = metadata_log_locations(¤t_metadata, table_bucket, &namespace, &table); + let mut protected = metadata_log_locations_matching(¤t_metadata, table_bucket, is_valid_location); protected.insert(entry.metadata_location.clone()); protected.extend(report.retained_metadata_locations.iter().cloned()); protected.extend( - metadata_locations_for_protected_snapshot_refs( + metadata_locations_for_protected_snapshot_refs_matching( &self.backend, table_bucket, - &namespace, - &table, ¤t_metadata, &report.cleanup_candidate_locations, + is_valid_location, ) .await?, ); @@ -4783,9 +6012,10 @@ where let cleanup_candidate_count = report.cleanup_candidate_locations.len(); let planned_deletable_locations = report.deletable_metadata_locations.iter().cloned().collect::>(); let mut cleanup_candidate_locations = BTreeSet::new(); + let mut delete_etags = BTreeMap::new(); let now = OffsetDateTime::now_utc(); for metadata_location in &report.cleanup_candidate_locations { - if !is_valid_table_metadata_location(&namespace, &table, metadata_location) { + if !is_valid_location(metadata_location) { return Err(TableCatalogStoreError::Invalid(format!( "cleanup candidate {metadata_location} must be inside the table metadata directory" ))); @@ -4804,25 +6034,58 @@ where if !metadata_candidate_is_past_safety_window(candidate_object.mod_time, now) { continue; } + if mutation_fence.is_some() { + if candidate_object + .mod_time + .is_none_or(|modified| modified.unix_timestamp() > report.job.cleanup_watermark_unix_seconds) + { + continue; + } + let etag = candidate_object.etag.ok_or_else(|| { + TableCatalogStoreError::Conflict(format!( + "maintenance candidate has no conditional-delete ETag: {metadata_location}" + )) + })?; + delete_etags.insert(metadata_location.clone(), etag); + } cleanup_candidate_locations.insert(metadata_location.clone()); } let cleanup_candidate_locations = cleanup_candidate_locations.into_iter().collect::>(); let deleted_locations = cleanup_candidate_locations.iter().cloned().collect::>(); for metadata_location in &cleanup_candidate_locations { - self.backend.delete_object(table_bucket, metadata_location).await?; + ensure_table_maintenance_publication_held(publication, entry, "before metadata deletion")?; + if let Some(fence) = &mutation_fence { + self.backend + .delete_object_if_match_fenced(table_bucket, metadata_location, &delete_etags[metadata_location], fence) + .await?; + } else { + self.backend.delete_object(table_bucket, metadata_location).await?; + } + ensure_table_maintenance_publication_held(publication, entry, "after metadata deletion")?; } - let referenced_object_reports = metadata_maintenance_referenced_object_reports( - &self.backend, - table_bucket, - &namespace, - &table, - warehouse_object_prefix.as_deref(), - ¤t_metadata, - &report.retained_metadata_locations, - ) - .await?; + let referenced_object_reports = if publication.is_some() { + metadata_maintenance_referenced_object_reports_for_entry( + &self.backend, + entry, + warehouse_object_prefix.as_deref(), + ¤t_metadata, + &report.retained_metadata_locations, + ) + .await? + } else { + metadata_maintenance_referenced_object_reports( + &self.backend, + table_bucket, + &namespace, + &table, + warehouse_object_prefix.as_deref(), + ¤t_metadata, + &report.retained_metadata_locations, + ) + .await? + }; let referenced_object_locations = if referenced_object_reports .iter() .any(|report| report.state == TableMetadataMaintenanceObjectState::ManualReviewRequired) @@ -4841,9 +6104,12 @@ where .any(|report| report.state == TableMetadataMaintenanceObjectState::ManualReviewRequired) { for object_location in &report.cleanup_object_candidate_locations { - if table_maintenance_object_kind(&namespace, &table, warehouse_object_prefix.as_deref(), object_location) - .is_none() - { + let object_kind = if publication.is_some() { + table_maintenance_object_kind_for_entry(entry, warehouse_object_prefix.as_deref(), object_location) + } else { + table_maintenance_object_kind(&namespace, &table, warehouse_object_prefix.as_deref(), object_location) + }; + if object_kind.is_none() { return Err(TableCatalogStoreError::Invalid(format!( "cleanup object candidate {object_location} must be inside table metadata, data, or delete directories" ))); @@ -4862,6 +6128,20 @@ where if !metadata_candidate_is_past_safety_window(candidate_object.mod_time, now) { continue; } + if mutation_fence.is_some() { + if candidate_object + .mod_time + .is_none_or(|modified| modified.unix_timestamp() > report.job.cleanup_watermark_unix_seconds) + { + continue; + } + let etag = candidate_object.etag.ok_or_else(|| { + TableCatalogStoreError::Conflict(format!( + "maintenance candidate has no conditional-delete ETag: {object_location}" + )) + })?; + delete_etags.insert(object_location.clone(), etag); + } cleanup_object_candidate_locations.insert(object_location.clone()); } } @@ -4869,7 +6149,15 @@ where let cleanup_object_candidate_locations = cleanup_object_candidate_locations.into_iter().collect::>(); let deleted_object_locations = cleanup_object_candidate_locations.iter().cloned().collect::>(); for object_location in &cleanup_object_candidate_locations { - self.backend.delete_object(table_bucket, object_location).await?; + ensure_table_maintenance_publication_held(publication, entry, "before object deletion")?; + if let Some(fence) = &mutation_fence { + self.backend + .delete_object_if_match_fenced(table_bucket, object_location, &delete_etags[object_location], fence) + .await?; + } else { + self.backend.delete_object(table_bucket, object_location).await?; + } + ensure_table_maintenance_publication_held(publication, entry, "after object deletion")?; } let retained_metadata_locations = protected.into_iter().collect::>(); @@ -4891,7 +6179,7 @@ where Ok(table_maintenance_report_with_recommended_actions(TableMetadataMaintenanceReport { job, - current_metadata_location: entry.metadata_location, + current_metadata_location: entry.metadata_location.clone(), retained_metadata_locations, cleanup_candidate_locations: cleanup_candidate_locations.clone(), deletable_metadata_locations: cleanup_candidate_locations, @@ -5875,7 +7163,7 @@ where )), ); } - if !is_valid_table_metadata_location(&namespace, &table, &request.new_metadata_location) { + if !is_valid_table_metadata_location_for_entry(¤t, &request.new_metadata_location) { return table_commit_result( &request.table_bucket, &request.namespace, @@ -6410,3 +7698,22 @@ where .await } } + +#[cfg(test)] +#[test] +fn catalog_lock_context_preserves_original_network_cause() { + let error = super::catalog_lock_acquisition_error( + "acquire catalog migration lock", + rustfs_lock::LockError::network("peer unavailable", std::io::Error::from(std::io::ErrorKind::TimedOut)), + ); + let error = maintenance_rename_recovery_error(error); + assert!(error.is_unavailable()); + assert!(error.to_string().contains("maintenance rename recovery required")); + let source = std::error::Error::source(&error).expect("retain the typed lock cause"); + assert!(source.is::()); + let io = source + .source() + .and_then(|cause| cause.downcast_ref::()) + .expect("retain the original network cause without cloning it"); + assert_eq!(io.kind(), std::io::ErrorKind::TimedOut); +} diff --git a/rustfs/src/table_catalog/store/strong.rs b/rustfs/src/table_catalog/store/strong.rs index f70f180fe..4412cad9a 100644 --- a/rustfs/src/table_catalog/store/strong.rs +++ b/rustfs/src/table_catalog/store/strong.rs @@ -321,6 +321,127 @@ impl StrongTableCatalogStore where B: TableCatalogObjectBackend, { + async fn table_maintenance_publication(&self) -> TableCatalogStoreResult> { + let publication = TableCommitLockPublication::new(&self.object_backend); + let migration_guard = self.acquire_snapshot_write_permit().await?; + // The publication checks must cover migration lease loss throughout sidecar writes and cleanup. + publication.attach_catalog_migration_read_guard(migration_guard)?; + Ok(publication) + } + + async fn table_maintenance_read_guards( + &self, + table_bucket: &str, + ) -> TableCatalogStoreResult<(TableCatalogLockGuard, TableCatalogLockGuard)> { + // Keep the catalog identity and its sidecars in one publication interval, + // including an A -> B -> A rename that would evade an entry-only recheck. + let migration = self.acquire_snapshot_write_permit().await?; + let publication = self + .object_backend + .acquire_read_lock(table_bucket, &default_table_bucket_publication_lock_path()) + .await?; + let guards = (migration, publication); + Self::ensure_table_maintenance_read_guards(&guards)?; + Ok(guards) + } + + fn ensure_table_maintenance_read_guards( + guards: &(TableCatalogLockGuard, TableCatalogLockGuard), + ) -> TableCatalogStoreResult<()> { + if guards.0.is_lock_lost() || guards.1.is_lock_lost() { + return Err(TableCatalogStoreError::Unavailable( + "table maintenance read lost its migration or publication fence; retry the read".to_string(), + )); + } + Ok(()) + } + + fn require_table_maintenance_publication( + publication: &(dyn TableCommitPublication + Sync), + table_bucket: &str, + namespace: &str, + table: &str, + operation: &str, + ) -> TableCatalogStoreResult<()> { + if !publication.holds_table_bucket(table_bucket) || !publication.holds_table(table_bucket, namespace, table) { + return Err(TableCatalogStoreError::Internal(format!( + "table maintenance {operation} publication fence was lost" + ))); + } + Ok(()) + } + + // Call only with migration -> bucket publication -> local write locks held. + // Reads remain side-effect free: a prepared rename never removes its source sidecars, + // and a published rename already has a complete destination before this cleanup. + pub(super) async fn recover_table_maintenance_rename_unlocked( + &self, + table_bucket: &str, + fence: Option<&TableCatalogObjectMutationFence>, + ) -> TableCatalogStoreResult> { + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let Some((intent, etag)) = sidecar + .read_table_maintenance_rename_intent(table_bucket) + .await + .map_err(object::maintenance_rename_recovery_error)? + else { + return Ok(None); + }; + let fence = fence.ok_or_else(|| { + TableCatalogStoreError::Conflict("maintenance rename recovery requires a storage mutation fence".to_string()) + })?; + fence.ensure_held()?; + let source_key = Self::table_key( + table_bucket, + &parse_namespace_for_store(&intent.source.namespace)?, + &parse_table_for_store(&intent.source.table)?, + ); + let destination_key = Self::table_key( + table_bucket, + &parse_namespace_for_store(&intent.destination.namespace)?, + &parse_table_for_store(&intent.destination.table)?, + ); + let committed = { + let state = self.state.lock().await; + let source = state.tables.get(&source_key); + let destination = state.tables.get(&destination_key); + let owned_tables = state + .tables + .values() + .filter(|entry| entry.table_bucket == table_bucket && entry.table_id == intent.source.table_id) + .count(); + if owned_tables != 1 { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename recovery table identity is not unique; repair persisted state before retrying rename" + .to_string(), + )); + } + let identity_matches = |entry: &TableEntry| { + entry.table_id == intent.source.table_id + && entry.table_uuid == intent.source.table_uuid + && entry.state == TableCatalogEntryState::Active + }; + if source.is_some_and(identity_matches) && destination.is_none() { + false + } else if source.is_none() + && destination.is_some_and(identity_matches) + && state + .snapshot_etag + .as_deref() + .is_some_and(|etag| etag != intent.source_snapshot_etag) + { + true + } else { + return Err(TableCatalogStoreError::Conflict("maintenance rename recovery does not match the catalog publication state; repair persisted state before retrying rename".to_string())); + } + }; + sidecar + .finish_table_maintenance_rename(&intent, &etag, committed, fence) + .await + .map_err(object::maintenance_rename_recovery_error)?; + Ok(Some((intent, committed))) + } + pub fn new(object_backend: B) -> Self { Self::new_with_snapshot_requirement(object_backend, false) } @@ -1881,6 +2002,1022 @@ where Ok(TableCommitResult { table: next, commit_log }) } + async fn maintenance_table_entry( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + ) -> TableCatalogStoreResult { + self.hydrate_state().await?; + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let key = Self::table_key(table_bucket, &namespace, &table); + let state = self.state.lock().await; + Self::ensure_identifier_is_unambiguous_locked(&state, &key)?; + state + .tables + .get(&key) + .filter(|entry| entry.state == TableCatalogEntryState::Active) + .cloned() + .ok_or_else(|| { + TableCatalogStoreError::NotFound(format!("table {}/{}/{}", table_bucket, namespace.public_name(), table.as_str())) + }) + } + + fn maintenance_cleanup_roots(entry: &TableEntry) -> TableCatalogStoreResult> { + let metadata_dir = table_metadata_dir_path_for_entry(entry)?; + let table_root = metadata_dir + .strip_suffix(&format!("/{METADATA_DIR}")) + .ok_or_else(|| TableCatalogStoreError::Invalid("maintenance metadata directory has no table root".to_string()))?; + let warehouse = table_warehouse_object_prefix(entry)?; + Ok(vec![ + format!("{metadata_dir}/"), + format!("{table_root}/{DATA_DIR}/"), + format!("{table_root}/{DELETE_DIR}/"), + format!("{warehouse}{METADATA_DIR}/"), + format!("{warehouse}{DATA_DIR}/"), + format!("{warehouse}{DELETE_DIR}/"), + ]) + } + + async fn maintenance_cleanup_entry( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + ) -> TableCatalogStoreResult { + let entry = self.maintenance_table_entry(table_bucket, namespace, table).await?; + let roots = Self::maintenance_cleanup_roots(&entry)?; + let state = self.state.lock().await; + for other in state.tables.values().filter(|other| { + other.table_bucket == table_bucket + && other.state != TableCatalogEntryState::Deleted + && (other.namespace != entry.namespace || other.table != entry.table) + }) { + let other_roots = Self::maintenance_cleanup_roots(other)?; + if roots.iter().any(|root| { + other_roots + .iter() + .any(|other_root| warehouse_object_prefixes_overlap(root, other_root)) + }) { + return Err(TableCatalogStoreError::Conflict(format!( + "table maintenance requires exclusive metadata and warehouse roots; shared with {}/{}", + other.namespace, other.table + ))); + } + } + Ok(entry) + } + + pub(crate) async fn get_table_maintenance_config( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + ) -> TableCatalogStoreResult { + let guards = self.table_maintenance_read_guards(table_bucket).await?; + let entry = self.maintenance_table_entry(table_bucket, namespace, table).await?; + let result = ObjectTableCatalogStore::new(self.object_backend.clone()) + .get_table_maintenance_config_for_entry(&entry) + .await; + Self::ensure_table_maintenance_read_guards(&guards)?; + result + } + + pub(crate) async fn put_table_maintenance_config( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + config: TableMaintenanceConfig, + ) -> TableCatalogStoreResult { + validate_table_maintenance_config(&config)?; + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "config update", + )?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + let entry = self + .maintenance_table_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + ObjectTableCatalogStore::new(self.object_backend.clone()) + .put_table_maintenance_config_for_entry_with_publication(&entry, config, Some(&publication)) + .await + } + + pub(crate) async fn get_table_metadata_maintenance_report( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + job_id: &str, + ) -> TableCatalogStoreResult> { + let guards = self.table_maintenance_read_guards(table_bucket).await?; + let entry = self.maintenance_table_entry(table_bucket, namespace, table).await?; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let result = ObjectTableCatalogStore::new(self.object_backend.clone()) + .get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, + job_id, + ) + .await; + Self::ensure_table_maintenance_read_guards(&guards)?; + result + } + + pub(crate) async fn get_table_maintenance_scheduler_report( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + ) -> TableCatalogStoreResult { + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let (entry, objects) = { + let guards = self.table_maintenance_read_guards(table_bucket).await?; + let entry = self.maintenance_table_entry(table_bucket, namespace, table).await?; + // Fix membership while rename cannot remove the source directory. The + // bounded listing has no per-job reads under the publication lock. + let objects = sidecar + .list_table_metadata_maintenance_audit_objects_for_entry(&entry) + .await?; + Self::ensure_table_maintenance_read_guards(&guards)?; + (entry, objects) + }; + // History is observational, not a transactionally atomic snapshot. Read + // every fixed member outside the bucket lock, but never skip a member + // that disappeared during rename (including an A -> B -> A round trip). + let reports = sidecar + .read_table_metadata_maintenance_audit_reports_for_entry(&entry, &objects) + .await?; + + let guards = self.table_maintenance_read_guards(table_bucket).await?; + if self.maintenance_table_entry(table_bucket, namespace, table).await? != entry { + return Err(TableCatalogStoreError::Unavailable( + "table changed during the maintenance scheduler read; retry the scheduler report".to_string(), + )); + } + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let paths = TableCatalogObjectPaths::default(); + let selected_objects = reports + .iter() + .map(|report| paths.table_maintenance_job_path(table_bucket, &namespace, &table, &entry.table_id, &report.job.job_id)) + .collect::>(); + // Only the fixed-size audit window, effective config and current alias + // are refreshed under the final read fence. + let reports = sidecar + .read_table_metadata_maintenance_audit_reports_for_entry(&entry, &selected_objects) + .await?; + let result = sidecar + .get_table_maintenance_scheduler_report_with_audit_reports_for_entry_at( + &entry, + OffsetDateTime::now_utc(), + Some(reports), + ) + .await; + Self::ensure_table_maintenance_read_guards(&guards)?; + result + } + + // A dry-run may publish its audit record, but must not replace pending work + // or hide an incomplete sidecar publication. + async fn maintenance_planning_predecessor( + sidecar: &ObjectTableCatalogStore, + entry: &TableEntry, + ) -> TableCatalogStoreResult> { + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let current = sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, + MAINTENANCE_JOB_ALIAS_CURRENT, + ) + .await?; + let latest = sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, + MAINTENANCE_JOB_ALIAS_LATEST, + ) + .await?; + if current != latest { + return Err(TableCatalogStoreError::Conflict( + "maintenance report publication requires recovery before planning".to_string(), + )); + } + if let Some(report) = ¤t { + let persisted = sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, + &report.job.job_id, + ) + .await?; + if persisted.as_ref() != Some(report) { + return Err(TableCatalogStoreError::Conflict( + "maintenance job publication requires recovery before planning".to_string(), + )); + } + if matches!( + report.job.status, + TableMetadataMaintenanceJobStatus::Queued + | TableMetadataMaintenanceJobStatus::Running + | TableMetadataMaintenanceJobStatus::Failed + ) || report.job.quarantined_object_count > 0 + { + return Err(TableCatalogStoreError::Conflict( + "pending maintenance work or quarantine must be resolved before planning".to_string(), + )); + } + } + Ok(current) + } + + pub(crate) async fn run_table_metadata_maintenance_planning( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + retain_recent_metadata_files: usize, + snapshot_expiration: Option, + compaction: Option, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "planning", + )?; + let _completion = TableCommitPublicationCompletion::new(&publication); + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let entry = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + let planned_effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + let planned_job = Self::maintenance_planning_predecessor(&sidecar, &entry).await?; + let mut effective = planned_effective.clone(); + if retain_recent_metadata_files > 0 { + effective.config.retain_recent_metadata_files = retain_recent_metadata_files; + } + + // Keep the migration permit while reading potentially large manifests; + // unrelated catalog commits need not wait for the object traversal. + publication.release_publication_locks(); + let mut report = sidecar + .plan_table_metadata_maintenance_for_entry(&entry, effective.config.retain_recent_metadata_files) + .await?; + report.snapshot_expiration = match snapshot_expiration { + Some(config) => Some( + sidecar + .plan_table_snapshot_expiration_for_entry_with_backend(&self.object_backend, &entry, config) + .await?, + ), + None => None, + }; + report.compaction = match compaction { + Some(config) => Some(sidecar.plan_table_compaction_for_entry(&entry, config).await?), + None => None, + }; + + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "planning revalidation", + )?; + if self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await? + != entry + { + return Err(TableCatalogStoreError::Conflict("table changed during maintenance planning".to_string())); + } + let current_effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + if current_effective != planned_effective { + return Err(TableCatalogStoreError::Conflict("maintenance config changed during planning".to_string())); + } + if Self::maintenance_planning_predecessor(&sidecar, &entry).await? != planned_job { + return Err(TableCatalogStoreError::Conflict( + "current maintenance job changed during planning".to_string(), + )); + } + + let now = OffsetDateTime::now_utc(); + let timestamp = maintenance_timestamp(now); + let before_status = Some(report.job.status.clone()); + report.job.operation = TableMetadataMaintenanceOperation::DryRun; + report.job.status = TableMetadataMaintenanceJobStatus::Running; + report.job.failure_reason = None; + report.job.config_source = effective.source; + report.job.worker_id = Some("rustfs-admin".to_string()); + report.job.lease_id = Uuid::new_v4().to_string(); + report.job.attempt = 1; + report.job.max_retry_attempts = effective.config.max_retry_attempts; + report.job.next_retry_after = None; + report.job.quarantine_enabled = effective.config.quarantine_enabled; + report.job.quarantine_retention_seconds = effective.config.quarantine_retention_seconds; + report.job.heartbeat_at = Some(timestamp.clone()); + report.job.started_at = Some(timestamp); + report.job.finished_at = None; + push_table_maintenance_audit_event( + &mut report, + now, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerStarted, + None, + before_status, + Some(0), + ); + // Planning has no object side effects, so publish only the completed + // report. There is no Running lease to strand if the request is lost. + sidecar + .finish_table_metadata_maintenance_run_for_entry_with_publication(&entry, false, &effective, report, &publication) + .await + } + + pub(crate) async fn run_table_maintenance_scheduler_once( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + scheduler_id: String, + ) -> TableCatalogStoreResult { + self.run_table_maintenance_scheduler_once_with_guard(table_bucket, namespace, table, scheduler_id, None) + .await + } + + pub(super) async fn run_table_maintenance_scheduler_once_with_guard( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + scheduler_id: String, + periodic_guard: Option<&PeriodicTableMaintenanceQueueGuard<'_>>, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "scheduler", + )?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let entry = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + let now = OffsetDateTime::now_utc(); + let effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + if let Some(periodic_guard) = periodic_guard { + sidecar + .validate_periodic_table_maintenance_queue_guard( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &effective, + periodic_guard, + ) + .await?; + } + let preflight = sidecar + .table_metadata_maintenance_scheduler_preflight_with_publication( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &scheduler_id, + now, + effective, + Some(&publication), + ) + .await?; + let effective = match preflight { + TableMaintenanceSchedulerPreflight::Ready(effective) => effective, + TableMaintenanceSchedulerPreflight::Complete(report) => { + publication.release_publication_locks(); + let scheduler = sidecar + .get_table_maintenance_scheduler_report_for_entry_at(&entry, now) + .await?; + return Ok(TableMaintenanceSchedulerRunResult { + report: *report, + scheduler, + }); + } + }; + + // Planning only reads objects. Keep the migration permit, not the bucket-wide writer. + let planned_effective = effective; + publication.release_publication_locks(); + let mut report = sidecar + .plan_table_metadata_maintenance_for_entry(&entry, planned_effective.config.retain_recent_metadata_files) + .await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "scheduler revalidation", + )?; + let current = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + if current != entry { + return Err(TableCatalogStoreError::Conflict("table changed during maintenance planning".to_string())); + } + let now = OffsetDateTime::now_utc(); + let effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + if effective.config != planned_effective.config || effective.source != planned_effective.source { + return Err(TableCatalogStoreError::Conflict("maintenance config changed during planning".to_string())); + } + if let Some(periodic_guard) = periodic_guard { + sidecar + .validate_periodic_table_maintenance_queue_guard( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &effective, + periodic_guard, + ) + .await?; + } + match sidecar + .table_metadata_maintenance_scheduler_preflight_with_publication( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &scheduler_id, + now, + effective, + Some(&publication), + ) + .await? + { + TableMaintenanceSchedulerPreflight::Ready(effective) => { + if report.job.retain_recent_metadata_files != effective.config.retain_recent_metadata_files { + return Err(TableCatalogStoreError::Conflict( + "maintenance config changed before scheduler claim".to_string(), + )); + } + if entry.metadata_location != report.current_metadata_location { + return Err(TableCatalogStoreError::Conflict( + "current metadata location changed before maintenance scheduler claim".to_string(), + )); + } + let before_status = Some(report.job.status.clone()); + let before_quarantined_object_count = Some(report.job.quarantined_object_count); + let scheduled_at = maintenance_timestamp(now); + report.job.operation = if effective.config.delete_enabled { + TableMetadataMaintenanceOperation::Delete + } else { + TableMetadataMaintenanceOperation::DryRun + }; + report.job.status = TableMetadataMaintenanceJobStatus::Queued; + report.job.failure_reason = None; + report.job.config_source = effective.source; + report.job.scheduler_id = Some(scheduler_id); + report.job.scheduler_lease_id = Uuid::new_v4().to_string(); + report.job.scheduled_at = Some(scheduled_at); + report.job.worker_id = None; + report.job.lease_id = String::new(); + report.job.attempt = 0; + report.job.max_retry_attempts = effective.config.max_retry_attempts; + report.job.next_retry_after = None; + report.job.quarantine_enabled = effective.config.quarantine_enabled; + report.job.quarantine_retention_seconds = effective.config.quarantine_retention_seconds; + report.job.heartbeat_at = None; + report.job.started_at = None; + report.job.finished_at = None; + refresh_table_maintenance_report_recommended_actions(&mut report); + push_table_maintenance_audit_event( + &mut report, + now, + TableMaintenanceAuditActor::Scheduler, + TableMaintenanceAuditAction::SchedulerQueued, + None, + before_status, + before_quarantined_object_count, + ); + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "scheduler report", + )?; + sidecar + .put_table_metadata_maintenance_report_for_entry_with_publication(&report, &entry, Some(&publication)) + .await?; + } + TableMaintenanceSchedulerPreflight::Complete(completed) => report = *completed, + } + + publication.release_publication_locks(); + let scheduler = sidecar + .get_table_maintenance_scheduler_report_for_entry_at(&entry, now) + .await?; + Ok(TableMaintenanceSchedulerRunResult { report, scheduler }) + } + + pub(crate) async fn run_table_metadata_maintenance_worker_once_conditionally( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + worker_id: String, + condition: &TableMaintenanceWorkerCondition, + ) -> TableCatalogStoreResult { + self.run_table_metadata_maintenance_worker_once_conditionally_with_clock( + table_bucket, + namespace, + table, + worker_id, + condition, + OffsetDateTime::now_utc, + ) + .await + } + + pub(in crate::table_catalog) async fn run_table_metadata_maintenance_worker_once_conditionally_with_clock( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + worker_id: String, + condition: &TableMaintenanceWorkerCondition, + clock: impl FnOnce() -> OffsetDateTime + Send, + ) -> TableCatalogStoreResult { + validate_table_maintenance_worker_condition(&worker_id, condition)?; + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "conditional worker", + )?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + let entry = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + ObjectTableCatalogStore::new(self.object_backend.clone()) + .run_conditional_table_maintenance_worker_for_entry(&entry, worker_id, condition, clock, &publication) + .await + } + + pub(crate) async fn run_table_metadata_maintenance_worker_once( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + worker_id: String, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "worker", + )?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let entry = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + let now = OffsetDateTime::now_utc(); + let effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + let (effective, queued) = match sidecar + .table_metadata_maintenance_worker_preflight_with_publication( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &worker_id, + now, + effective, + Some(&publication), + ) + .await? + { + TableMaintenanceWorkerPreflight::Ready { effective, queued } => (effective, queued), + TableMaintenanceWorkerPreflight::Complete(report) => return Ok(*report), + }; + + let mut report = if let Some(queued) = queued { + *queued + } else { + let planned_job = sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + table_bucket, + &namespace, + &table, + &entry.table_id, + MAINTENANCE_JOB_ALIAS_CURRENT, + ) + .await?; + // Only fresh planning is read-only; queued execution and deletion keep publication held. + publication.release_publication_locks(); + let report = sidecar + .plan_table_metadata_maintenance_for_entry(&entry, effective.config.retain_recent_metadata_files) + .await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "worker revalidation", + )?; + let current = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + if current != entry { + return Err(TableCatalogStoreError::Conflict( + "table changed during maintenance worker planning".to_string(), + )); + } + let current_effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + if current_effective != effective { + return Err(TableCatalogStoreError::Conflict( + "maintenance config changed during worker planning".to_string(), + )); + } + let current_job = sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + table_bucket, + &namespace, + &table, + &entry.table_id, + MAINTENANCE_JOB_ALIAS_CURRENT, + ) + .await?; + if current_job != planned_job { + return Err(TableCatalogStoreError::Conflict( + "current maintenance job changed during planning".to_string(), + )); + } + report + }; + let now = OffsetDateTime::now_utc(); + let effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + let (effective, queued) = match sidecar + .table_metadata_maintenance_worker_preflight_with_publication( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &worker_id, + now, + effective, + Some(&publication), + ) + .await? + { + TableMaintenanceWorkerPreflight::Ready { effective, queued } => (effective, queued), + TableMaintenanceWorkerPreflight::Complete(report) => return Ok(*report), + }; + if let Some(queued) = queued { + if queued.job.job_id != report.job.job_id { + return Err(TableCatalogStoreError::Conflict( + "queued maintenance job changed before worker claim".to_string(), + )); + } + report = *queued; + } + if report.job.retain_recent_metadata_files != effective.config.retain_recent_metadata_files { + return Err(TableCatalogStoreError::Conflict( + "maintenance config changed before worker claim".to_string(), + )); + } + if entry.metadata_location != report.current_metadata_location || entry.generation != report.job.current_generation { + return Err(TableCatalogStoreError::Conflict( + "current metadata location changed before maintenance worker claim".to_string(), + )); + } + + let current = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + if current != entry { + return Err(TableCatalogStoreError::Conflict( + "table changed before maintenance worker claim".to_string(), + )); + } + let was_queued_claim = matches!(report.job.status, TableMetadataMaintenanceJobStatus::Queued); + let before_status = Some(report.job.status.clone()); + let before_quarantined_object_count = Some(report.job.quarantined_object_count); + let started_at = maintenance_timestamp(now); + if !was_queued_claim { + report.job.operation = if effective.config.delete_enabled { + TableMetadataMaintenanceOperation::Delete + } else { + TableMetadataMaintenanceOperation::DryRun + }; + } + report.job.status = TableMetadataMaintenanceJobStatus::Running; + report.job.failure_reason = None; + report.job.config_source = effective.source; + report.job.worker_id = Some(worker_id); + report.job.lease_id = Uuid::new_v4().to_string(); + report.job.attempt = 1; + report.job.max_retry_attempts = effective.config.max_retry_attempts; + report.job.next_retry_after = None; + report.job.quarantine_enabled = effective.config.quarantine_enabled; + report.job.quarantine_retention_seconds = effective.config.quarantine_retention_seconds; + report.job.heartbeat_at = Some(started_at.clone()); + report.job.started_at = Some(started_at); + report.job.finished_at = None; + refresh_table_maintenance_report_recommended_actions(&mut report); + push_table_maintenance_audit_event( + &mut report, + now, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerStarted, + None, + before_status, + before_quarantined_object_count, + ); + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "worker report", + )?; + sidecar + .put_table_metadata_maintenance_report_for_entry_with_publication(&report, &entry, Some(&publication)) + .await?; + + let delete = matches!(report.job.operation, TableMetadataMaintenanceOperation::Delete); + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "worker finish", + )?; + sidecar + .finish_table_metadata_maintenance_run_for_entry_with_publication(&entry, delete, &effective, report, &publication) + .await + } + + pub(crate) async fn heartbeat_table_metadata_maintenance_job( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + job_id: &str, + lease_id: &str, + worker_id: &str, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "heartbeat", + )?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + let entry = self + .maintenance_table_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + sidecar + .heartbeat_table_metadata_maintenance_job_for_entry_with_publication( + &entry, + TableMaintenanceHeartbeatRef { + table_bucket, + namespace: &namespace.public_name(), + table: table.as_str(), + job_id, + lease_id, + worker_id, + }, + OffsetDateTime::now_utc(), + Some(&publication), + ) + .await + } + + pub(crate) async fn apply_table_maintenance_quarantine_operation( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + job_id: &str, + request: TableMaintenanceQuarantineOperationRequest, + ) -> TableCatalogStoreResult { + let action = request.action.clone(); + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "quarantine", + )?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + let entry = self + .maintenance_table_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let report_lookup = if matches!(action, TableMaintenanceQuarantineAction::Inspect) { + job_id + } else { + MAINTENANCE_JOB_ALIAS_CURRENT + }; + let mut report = sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + table_bucket, + &namespace, + &table, + &entry.table_id, + report_lookup, + ) + .await? + .ok_or_else(|| { + TableCatalogStoreError::NotFound(format!( + "maintenance job {}/{}/{}/{}", + table_bucket, + namespace.public_name(), + table.as_str(), + job_id + )) + })?; + + if !matches!(action, TableMaintenanceQuarantineAction::Inspect) { + if report.job.job_id != job_id { + return Err(TableCatalogStoreError::Conflict("maintenance job is not current".to_string())); + } + if !matches!(report.job.status, TableMetadataMaintenanceJobStatus::Failed) { + return Err(TableCatalogStoreError::Conflict( + "maintenance quarantine operation requires a failed job".to_string(), + )); + } + if !report.job.quarantine_enabled || report.job.quarantined_object_count == 0 { + return Err(TableCatalogStoreError::Conflict( + "maintenance job has no active quarantine boundary".to_string(), + )); + } + let before_status = Some(report.job.status.clone()); + let before_quarantined_object_count = Some(report.job.quarantined_object_count); + report.job.quarantined_object_count = 0; + let audit_action = match &action { + TableMaintenanceQuarantineAction::Inspect => unreachable!(), + TableMaintenanceQuarantineAction::Release => { + report.job.failure_reason = + Some(table_maintenance_quarantine_operator_reason("released", request.reason.as_deref())); + TableMaintenanceAuditAction::QuarantineRelease + } + TableMaintenanceQuarantineAction::Retry => { + report.job.next_retry_after = None; + report.job.failure_reason = Some(table_maintenance_quarantine_operator_reason( + "released for retry", + request.reason.as_deref(), + )); + TableMaintenanceAuditAction::QuarantineRetry + } + TableMaintenanceQuarantineAction::Abandon => { + report.job.next_retry_after = None; + report.job.failure_reason = + Some(table_maintenance_quarantine_operator_reason("abandoned", request.reason.as_deref())); + TableMaintenanceAuditAction::QuarantineAbandon + } + }; + refresh_table_maintenance_report_recommended_actions(&mut report); + push_table_maintenance_audit_event( + &mut report, + OffsetDateTime::now_utc(), + TableMaintenanceAuditActor::Operator, + audit_action, + request.reason, + before_status, + before_quarantined_object_count, + ); + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "quarantine update", + )?; + sidecar + .put_table_metadata_maintenance_report_for_entry_with_publication(&report, &entry, Some(&publication)) + .await?; + } + + let scheduler = sidecar + .get_table_maintenance_scheduler_report_for_entry_at(&entry, OffsetDateTime::now_utc()) + .await?; + Ok(TableMaintenanceQuarantineOperationResult { + action, + report, + scheduler, + }) + } + pub(crate) async fn plan_table_commit_recovery( &self, table_bucket: &str, @@ -2439,6 +3576,14 @@ where self.hydrate_state().await?; let key = Self::table_key(&entry.table_bucket, &namespace, &table); let publication_identity = (entry.table_bucket.clone(), entry.namespace.clone(), entry.table.clone()); + let mut recovery_fence = publication.object_mutation_fence(); + if let Some(fence) = recovery_fence.as_mut() { + fence + .signals + .extend(_migration_guard.as_ref().and_then(TableCatalogLockGuard::lock_lost_signal)); + } + self.recover_table_maintenance_rename_unlocked(&entry.table_bucket, recovery_fence.as_ref()) + .await?; let (snapshot, precondition, postcondition) = { let state = self.state.lock().await; Self::require_table_bucket_in_state(&state, &entry.table_bucket)?; @@ -2573,9 +3718,9 @@ where destination_namespace: &str, destination_table: &str, ) -> TableCatalogStoreResult<()> { - let publication = TableCommitLockPublication::new(&self.object_backend); - let _migration_guard = self.acquire_snapshot_write_permit().await?; + let publication = self.table_maintenance_publication().await?; publication.begin_table_bucket(table_bucket).await?; + publication.prepare(table_bucket, source_namespace, source_table).await?; if !publication.holds_table_bucket(table_bucket) { return Err(TableCatalogStoreError::Internal( "table rename requires a table-bucket publication fence".to_string(), @@ -2591,8 +3736,21 @@ where let destination_table = parse_table_for_store(destination_table)?; let source_key = Self::table_key(table_bucket, &source_namespace, &source_table); let destination_key = Self::table_key(table_bucket, &destination_namespace, &destination_table); + let fence = publication + .object_mutation_fence() + .ok_or_else(|| TableCatalogStoreError::Internal("table rename requires a storage mutation fence".to_string()))?; + if let Some((recovered, true)) = self + .recover_table_maintenance_rename_unlocked(table_bucket, Some(&fence)) + .await? + && recovered.source.namespace == source_namespace.public_name() + && recovered.source.table == source_table.as_str() + && recovered.destination.namespace == destination_namespace.public_name() + && recovered.destination.table == destination_table.as_str() + { + return Ok(()); + } - let (snapshot, precondition, postcondition) = { + let (snapshot, precondition, postcondition, source, destination) = { let state = self.state.lock().await; Self::require_table_bucket_in_state(&state, table_bucket)?; Self::ensure_identifier_is_unambiguous_locked(&state, &source_key)?; @@ -2637,7 +3795,7 @@ where let (precondition, mut draft_state) = Self::snapshot_draft_context_locked(&state); draft_state.tables.remove(&source_key); - let mut destination = source; + let mut destination = source.clone(); destination.namespace = destination_namespace.public_name(); destination.table = destination_table.as_str().to_string(); draft_state.tables.insert(destination_key.clone(), destination.clone()); @@ -2647,11 +3805,31 @@ where StrongSnapshotWritePostcondition::TableRenamed { source_key, destination_key, - table_id: destination.table_id, + table_id: destination.table_id.clone(), }, + source, + destination, ) }; - self.finalize_snapshot_write(snapshot, precondition, postcondition).await + let TableCatalogPutPrecondition::IfMatch(source_snapshot_etag) = &precondition else { + return Err(TableCatalogStoreError::Internal( + "table rename requires an existing snapshot watermark".to_string(), + )); + }; + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let pending = sidecar + .prepare_table_maintenance_rename(&source, &destination, source_snapshot_etag, &fence) + .await + .map_err(object::maintenance_rename_recovery_error)?; + self.finalize_snapshot_write_with_fence(snapshot, precondition, postcondition, Some(&fence)) + .await?; + if let Some((intent, etag)) = pending { + sidecar + .finish_table_maintenance_rename(&intent, &etag, true, &fence) + .await + .map_err(object::maintenance_rename_recovery_error)?; + } + Ok(()) } async fn resolve_table_data_plane_resource( @@ -2913,8 +4091,7 @@ where } Err(err) => Err(err), }; - let cas_result = result.as_ref().map(|_| ()).map_err(Clone::clone); - record_table_commit_cas_result(&request.operation, cas_started, &cas_result); + record_table_commit_cas_result(&request.operation, cas_started, &result); table_commit_result( &request.table_bucket, &request.namespace, @@ -2927,8 +4104,7 @@ where } async fn drop_table(&self, table_bucket: &str, namespace: &str, table: &str) -> TableCatalogStoreResult<()> { - let publication = TableCommitLockPublication::new(&self.object_backend); - let _migration_guard = self.acquire_snapshot_write_permit().await?; + let publication = self.table_maintenance_publication().await?; publication.begin_table_bucket(table_bucket).await?; publication.prepare(table_bucket, namespace, table).await?; if !publication.holds_table_bucket(table_bucket) || !publication.holds_table(table_bucket, namespace, table) { @@ -2939,6 +4115,9 @@ where let _publication_completion = TableCommitPublicationCompletion::new(&publication); let _write_guard = self.write_lock.lock().await; self.hydrate_state().await?; + let recovery_fence = publication.object_mutation_fence(); + self.recover_table_maintenance_rename_unlocked(table_bucket, recovery_fence.as_ref()) + .await?; let namespace = parse_namespace_for_store(namespace)?; let table = parse_table_for_store(table)?; let key = Self::table_key(table_bucket, &namespace, &table); diff --git a/rustfs/src/table_catalog/test_support.rs b/rustfs/src/table_catalog/test_support.rs index 3c0878b1c..5972c60df 100644 --- a/rustfs/src/table_catalog/test_support.rs +++ b/rustfs/src/table_catalog/test_support.rs @@ -22,12 +22,54 @@ //! fixed values (sequence 7 / snapshot 20), which keeps every produced byte //! identical to the pre-extraction fixtures. +use rustfs_lock::{LockClient, LockId, LockInfo, LockRequest, LockResponse, LockStats}; use std::collections::{BTreeMap, BTreeSet}; -use std::sync::Arc; +use std::sync::{Arc, Mutex}; use time::OffsetDateTime; use super::*; +#[derive(Debug)] +pub(crate) struct RecordingLeaseClient { + pub(crate) inner: rustfs_lock::LocalClient, + pub(crate) acquired: Mutex>, +} + +#[async_trait::async_trait] +impl LockClient for RecordingLeaseClient { + async fn acquire_lock(&self, request: &LockRequest) -> rustfs_lock::Result { + let response = self.inner.acquire_lock(request).await?; + if let Some(info) = response.lock_info.as_ref() { + *self.acquired.lock().expect("record lease") = Some(info.id.clone()); + } + Ok(response) + } + async fn release(&self, id: &LockId) -> rustfs_lock::Result { + self.inner.release(id).await + } + async fn refresh(&self, id: &LockId) -> rustfs_lock::Result { + self.inner.refresh(id).await + } + async fn force_release(&self, id: &LockId) -> rustfs_lock::Result { + self.inner.force_release(id).await + } + async fn check_status(&self, id: &LockId) -> rustfs_lock::Result> { + self.inner.check_status(id).await + } + async fn get_stats(&self) -> rustfs_lock::Result { + self.inner.get_stats().await + } + async fn close(&self) -> rustfs_lock::Result<()> { + self.inner.close().await + } + async fn is_online(&self) -> bool { + self.inner.is_online().await + } + async fn is_local(&self) -> bool { + self.inner.is_local().await + } +} + pub(crate) fn table_metadata_json(table_uuid: &str, location: &str) -> serde_json::Value { serde_json::json!({ "format-version": 2, @@ -495,6 +537,7 @@ pub(crate) struct TestCatalogObjectState { pub(crate) pause_put_attempts: BTreeMap<(String, String), BTreeMap>, pub(crate) fail_delete_attempts: BTreeMap<(String, String), BTreeSet>, pub(crate) fail_after_delete_attempts: BTreeMap<(String, String), BTreeSet>, + pub(crate) pause_delete_attempts: BTreeMap<(String, String), BTreeMap>, pub(crate) put_attempts: BTreeMap<(String, String), usize>, pub(crate) delete_attempts: BTreeMap<(String, String), usize>, pub(crate) write_lock_acquisitions: BTreeMap<(String, String), usize>, @@ -689,6 +732,27 @@ impl TestCatalogObjectBackend { pause } + pub(crate) async fn pause_next_delete(&self, bucket: &str, object: &str) -> TestCatalogObjectPause { + let mut state = self.state.lock().await; + let key = (bucket.to_string(), object.to_string()); + let next_attempt = state.delete_attempts.get(&key).copied().unwrap_or_default() + 1; + Self::pause_delete_attempt_unlocked(&mut state, key, next_attempt) + } + + fn pause_delete_attempt_unlocked( + state: &mut TestCatalogObjectState, + key: (String, String), + attempt: usize, + ) -> TestCatalogObjectPause { + let pause = TestCatalogObjectPause::default(); + state + .pause_delete_attempts + .entry(key) + .or_default() + .insert(attempt, pause.clone()); + pause + } + pub(crate) async fn put_attempt_count(&self, bucket: &str, object: &str) -> usize { self.state .lock() @@ -700,6 +764,188 @@ impl TestCatalogObjectBackend { } } +impl TestCatalogObjectBackend { + async fn put_object_with_fence( + &self, + bucket: &str, + object: &str, + data: Vec, + precondition: TableCatalogPutPrecondition, + fence: Option<&TableCatalogObjectMutationFence>, + ) -> TableCatalogStoreResult<()> { + if let Some(fence) = fence { + fence.ensure_held()?; + } + let mut fail_put_object_path = self.fail_put_object_path.lock().await; + if fail_put_object_path.as_deref() == Some(object) { + fail_put_object_path.take(); + return Err(TableCatalogStoreError::Internal("injected metadata write failure".to_string())); + } + drop(fail_put_object_path); + + let mut corrupt_put_object_path = self.corrupt_put_object_path.lock().await; + let data = if corrupt_put_object_path.as_deref() == Some(object) { + corrupt_put_object_path.take(); + b"{}".to_vec() + } else { + data + }; + drop(corrupt_put_object_path); + + let key = (bucket.to_string(), object.to_string()); + let (attempt, pause) = { + let mut state = self.state.lock().await; + let attempt = { + let attempts = state.put_attempts.entry(key.clone()).or_default(); + *attempts += 1; + *attempts + }; + if state + .fail_put_attempts + .get(&key) + .is_some_and(|attempts| attempts.contains(&attempt)) + { + return Err(TableCatalogStoreError::Internal(format!( + "injected put failure for {object} attempt {attempt}" + ))); + } + let pause = state + .pause_put_attempts + .get_mut(&key) + .and_then(|attempts| attempts.remove(&attempt)); + (attempt, pause) + }; + if let Some(pause) = pause { + pause.started.notify_one(); + pause.release.notified().await; + } + + let result = { + let mut state = self.state.lock().await; + let precondition_failure = match &precondition { + TableCatalogPutPrecondition::IfAbsent if state.objects.contains_key(&key) => { + Some(TableCatalogStoreError::Conflict(format!("object already exists: {object}"))) + } + TableCatalogPutPrecondition::IfMatch(expected) => match state.objects.get(&key) { + None => Some(TableCatalogStoreError::Conflict(format!("object is missing: {object}"))), + Some(current) if ¤t.etag != expected => { + Some(TableCatalogStoreError::Conflict(format!("object changed: {object}"))) + } + Some(_) => None, + }, + _ => None, + }; + if let Some(err) = precondition_failure { + Err(err) + } else { + if let Some(fence) = fence { + fence.ensure_held()?; + } + let etag = if self.content_addressed_etags { + content_etag(&data) + } else { + state.next_etag() + }; + state.objects.insert( + key.clone(), + TestCatalogObjectRecord { + data, + etag, + mod_time: Some(OffsetDateTime::now_utc()), + }, + ); + if state + .fail_after_put_attempts + .get(&key) + .is_some_and(|attempts| attempts.contains(&attempt)) + { + Err(TableCatalogStoreError::Internal(format!( + "injected post-commit put failure for {object} attempt {attempt}" + ))) + } else { + Ok(()) + } + } + }; + if let Some(barrier) = &self.put_object_barrier { + barrier.wait().await; + } + result?; + if let Some(fence) = fence { + fence.ensure_held()?; + } + Ok(()) + } + + async fn delete_object_if_match_with_fence( + &self, + bucket: &str, + object: &str, + expected_etag: &str, + fence: Option<&TableCatalogObjectMutationFence>, + ) -> TableCatalogStoreResult<()> { + if let Some(fence) = fence { + fence.ensure_held()?; + } + let mut state = self.state.lock().await; + let key = (bucket.to_string(), object.to_string()); + let attempt = { + let attempts = state.delete_attempts.entry(key.clone()).or_default(); + *attempts += 1; + *attempts + }; + if state + .fail_delete_attempts + .get(&key) + .is_some_and(|attempts| attempts.contains(&attempt)) + { + return Err(TableCatalogStoreError::Internal(format!( + "injected delete failure for {object} attempt {attempt}" + ))); + } + let pause = state + .pause_delete_attempts + .get_mut(&key) + .and_then(|attempts| attempts.remove(&attempt)); + if let Some(pause) = pause { + drop(state); + pause.started.notify_one(); + pause.release.notified().await; + state = self.state.lock().await; + } + match state.objects.get(&key) { + None => { + return Err(TableCatalogStoreError::Conflict(format!( + "conditionally delete catalog object: {bucket}/{object} disappeared before cleanup" + ))); + } + Some(record) if record.etag != expected_etag => { + return Err(TableCatalogStoreError::Conflict(format!( + "conditionally delete catalog object: {bucket}/{object} changed before cleanup" + ))); + } + Some(_) => {} + } + if let Some(fence) = fence { + fence.ensure_held()?; + } + state.objects.remove(&key); + if state + .fail_after_delete_attempts + .get(&key) + .is_some_and(|attempts| attempts.contains(&attempt)) + { + return Err(TableCatalogStoreError::Internal(format!( + "injected post-commit delete failure for {object} attempt {attempt}" + ))); + } + if let Some(fence) = fence { + fence.ensure_held()?; + } + Ok(()) + } +} + impl TestCatalogObjectState { pub(crate) fn next_etag(&mut self) -> String { self.next_etag += 1; @@ -835,98 +1081,19 @@ impl TableCatalogObjectBackend for TestCatalogObjectBackend { data: Vec, precondition: TableCatalogPutPrecondition, ) -> TableCatalogStoreResult<()> { - let mut fail_put_object_path = self.fail_put_object_path.lock().await; - if fail_put_object_path.as_deref() == Some(object) { - fail_put_object_path.take(); - return Err(TableCatalogStoreError::Internal("injected metadata write failure".to_string())); - } - drop(fail_put_object_path); + self.put_object_with_fence(bucket, object, data, precondition, None).await + } - let mut corrupt_put_object_path = self.corrupt_put_object_path.lock().await; - let data = if corrupt_put_object_path.as_deref() == Some(object) { - corrupt_put_object_path.take(); - b"{}".to_vec() - } else { - data - }; - drop(corrupt_put_object_path); - - let key = (bucket.to_string(), object.to_string()); - let (attempt, pause) = { - let mut state = self.state.lock().await; - let attempt = { - let attempts = state.put_attempts.entry(key.clone()).or_default(); - *attempts += 1; - *attempts - }; - if state - .fail_put_attempts - .get(&key) - .is_some_and(|attempts| attempts.contains(&attempt)) - { - return Err(TableCatalogStoreError::Internal(format!( - "injected put failure for {object} attempt {attempt}" - ))); - } - let pause = state - .pause_put_attempts - .get_mut(&key) - .and_then(|attempts| attempts.remove(&attempt)); - (attempt, pause) - }; - if let Some(pause) = pause { - pause.started.notify_one(); - pause.release.notified().await; - } - - let result = { - let mut state = self.state.lock().await; - let precondition_failure = match &precondition { - TableCatalogPutPrecondition::IfAbsent if state.objects.contains_key(&key) => { - Some(TableCatalogStoreError::Conflict(format!("object already exists: {object}"))) - } - TableCatalogPutPrecondition::IfMatch(expected) => match state.objects.get(&key) { - None => Some(TableCatalogStoreError::Conflict(format!("object is missing: {object}"))), - Some(current) if ¤t.etag != expected => { - Some(TableCatalogStoreError::Conflict(format!("object changed: {object}"))) - } - Some(_) => None, - }, - _ => None, - }; - if let Some(err) = precondition_failure { - Err(err) - } else { - let etag = if self.content_addressed_etags { - content_etag(&data) - } else { - state.next_etag() - }; - state.objects.insert( - key.clone(), - TestCatalogObjectRecord { - data, - etag, - mod_time: Some(OffsetDateTime::now_utc()), - }, - ); - if state - .fail_after_put_attempts - .get(&key) - .is_some_and(|attempts| attempts.contains(&attempt)) - { - Err(TableCatalogStoreError::Internal(format!( - "injected post-commit put failure for {object} attempt {attempt}" - ))) - } else { - Ok(()) - } - } - }; - if let Some(barrier) = &self.put_object_barrier { - barrier.wait().await; - } - result + async fn put_object_fenced( + &self, + bucket: &str, + object: &str, + data: Vec, + precondition: TableCatalogPutPrecondition, + fence: &TableCatalogObjectMutationFence, + ) -> TableCatalogStoreResult<()> { + self.put_object_with_fence(bucket, object, data, precondition, Some(fence)) + .await } async fn delete_object(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<()> { @@ -960,46 +1127,19 @@ impl TableCatalogObjectBackend for TestCatalogObjectBackend { } async fn delete_object_if_match(&self, bucket: &str, object: &str, expected_etag: &str) -> TableCatalogStoreResult<()> { - let mut state = self.state.lock().await; - let key = (bucket.to_string(), object.to_string()); - let attempt = { - let attempts = state.delete_attempts.entry(key.clone()).or_default(); - *attempts += 1; - *attempts - }; - if state - .fail_delete_attempts - .get(&key) - .is_some_and(|attempts| attempts.contains(&attempt)) - { - return Err(TableCatalogStoreError::Internal(format!( - "injected delete failure for {object} attempt {attempt}" - ))); - } - match state.objects.get(&key) { - None => { - return Err(TableCatalogStoreError::Conflict(format!( - "conditionally delete catalog object: {bucket}/{object} disappeared before cleanup" - ))); - } - Some(record) if record.etag != expected_etag => { - return Err(TableCatalogStoreError::Conflict(format!( - "conditionally delete catalog object: {bucket}/{object} changed before cleanup" - ))); - } - Some(_) => {} - } - state.objects.remove(&key); - if state - .fail_after_delete_attempts - .get(&key) - .is_some_and(|attempts| attempts.contains(&attempt)) - { - return Err(TableCatalogStoreError::Internal(format!( - "injected post-commit delete failure for {object} attempt {attempt}" - ))); - } - Ok(()) + self.delete_object_if_match_with_fence(bucket, object, expected_etag, None) + .await + } + + async fn delete_object_if_match_fenced( + &self, + bucket: &str, + object: &str, + expected_etag: &str, + fence: &TableCatalogObjectMutationFence, + ) -> TableCatalogStoreResult<()> { + self.delete_object_if_match_with_fence(bucket, object, expected_etag, Some(fence)) + .await } async fn list_objects(&self, bucket: &str, prefix: &str) -> TableCatalogStoreResult> { diff --git a/rustfs/src/table_catalog/tests.rs b/rustfs/src/table_catalog/tests.rs index 2f61241c3..dbc8b4a0f 100644 --- a/rustfs/src/table_catalog/tests.rs +++ b/rustfs/src/table_catalog/tests.rs @@ -18,6 +18,9 @@ use datafusion::{ use std::assert_matches; use std::sync::Arc; +mod strong_maintenance_planning; +mod strong_maintenance_planning_paths; + const TABLE_CATALOG_TEST_TIMEOUT: StdDuration = StdDuration::from_secs(30); #[test] @@ -71,7 +74,7 @@ fn catalog_lock_authority_failures_are_typed_as_unavailable() { ] { assert_matches!( super::store::catalog_lock_acquisition_error("acquire catalog table lock", error), - TableCatalogStoreError::Unavailable(_) + TableCatalogStoreError::LockUnavailable { .. } ); } @@ -131,7 +134,7 @@ fn catalog_storage_retryable_lock_failures_are_typed_as_unavailable() { ] { assert_matches!( storage_error_to_catalog("stat catalog object", StorageError::Lock(error)), - TableCatalogStoreError::Unavailable(_) + TableCatalogStoreError::LockUnavailable { .. } ); } @@ -734,6 +737,79 @@ async fn strong_table_registration_and_drop_acquire_migration_read_lock_before_p drop_task.await.expect("drop task should join").expect("drop should succeed"); } +#[tokio::test] +async fn strong_table_maintenance_acquires_migration_read_lock_before_publication() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let bucket = "analytics"; + let namespace = Namespace::parse("sales").expect("namespace should parse"); + let table = IdentifierSegment::parse("orders").expect("table should parse"); + let metadata_location = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + store.put_table_bucket(test_bucket_entry(bucket)).await.unwrap(); + store + .create_namespace(test_namespace_entry(bucket, &namespace)) + .await + .unwrap(); + store + .create_table(test_table_entry(bucket, &namespace, &table, metadata_location)) + .await + .unwrap(); + + let publication_lock = default_table_bucket_publication_lock_path(); + let migration_lock = TableCatalogObjectPaths::default().backing_migration_global_fence_lock_path(); + let publication_guard = backend + .acquire_write_lock(bucket, &publication_lock) + .await + .expect("publication lock should be acquired"); + let publication_attempts = backend.write_lock_acquisition_count(bucket, &publication_lock).await; + let migration_reads = backend.read_lock_acquisition_count(RUSTFS_META_BUCKET, &migration_lock).await; + let maintenance_store = store.clone(); + let migration_guard = backend + .acquire_write_lock(RUSTFS_META_BUCKET, &migration_lock) + .await + .expect("migration lock should be acquired"); + let maintenance = tokio::spawn(async move { + maintenance_store + .put_table_maintenance_config( + bucket, + "sales", + "orders", + TableMaintenanceConfig { + version: TABLE_MAINTENANCE_CONFIG_VERSION, + background_enabled: true, + ..Default::default() + }, + ) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, async { + while backend.read_lock_acquisition_count(RUSTFS_META_BUCKET, &migration_lock).await == migration_reads { + tokio::task::yield_now().await; + } + }) + .await + .expect("maintenance should request the migration read lock first"); + assert_eq!( + backend.write_lock_acquisition_count(bucket, &publication_lock).await, + publication_attempts, + "maintenance must not request publication while waiting for the migration fence" + ); + drop(migration_guard); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, async { + while backend.write_lock_acquisition_count(bucket, &publication_lock).await == publication_attempts { + tokio::task::yield_now().await; + } + }) + .await + .expect("maintenance should request publication after the migration fence"); + assert!(!maintenance.is_finished()); + drop(publication_guard); + maintenance + .await + .expect("maintenance task should join") + .expect("maintenance config should persist"); +} + async fn assert_direct_drop_uses_publication_locks(store: S, backend: &TestCatalogObjectBackend) where S: TableCatalogStore + Clone + Send + Sync + 'static, @@ -1527,9 +1603,9 @@ fn iceberg_metadata_validation_enforces_schema_evolution() { let mut incompatible = promoted; incompatible["schemas"][1]["fields"][0]["type"] = serde_json::Value::from("string"); let error = validate_supported_table_metadata(&incompatible).expect_err("field IDs must retain compatible types"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("schema field 1 has an incompatible type evolution".to_string()) + TableCatalogStoreError::Invalid(message) if message == "schema field 1 has an incompatible type evolution" ); let mut moved_into_collection = table_metadata_json_for_validation(); @@ -1571,20 +1647,18 @@ fn iceberg_metadata_validation_enforces_schema_evolution() { ]); reused["current-schema-id"] = serde_json::Value::from(2); let error = validate_supported_table_metadata(&reused).expect_err("removed Iceberg field IDs must never be reused"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("schema field 1 cannot be reused after removal".to_string()) + TableCatalogStoreError::Invalid(message) if message == "schema field 1 cannot be reused after removal" ); let mut stale_last_column_id = table_metadata_json_for_validation(); stale_last_column_id["last-column-id"] = serde_json::Value::from(0); let error = validate_supported_table_metadata(&stale_last_column_id) .expect_err("last-column-id must cover nested and top-level assigned field IDs"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid( - "last-column-id must be non-negative and cover every assigned schema field id".to_string() - ) + TableCatalogStoreError::Invalid(message) if message == "last-column-id must be non-negative and cover every assigned schema field id" ); } @@ -1596,18 +1670,18 @@ fn iceberg_metadata_transition_preserves_assignment_watermarks_and_schema_histor lower_column_watermark["last-column-id"] = serde_json::Value::from(0); let error = validate_table_metadata_transition(¤t, &lower_column_watermark) .expect_err("last-column-id must not decrease across commits"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("last-column-id must not decrease across table metadata commits".to_string()) + TableCatalogStoreError::Invalid(message) if message == "last-column-id must not decrease across table metadata commits" ); let mut lower_partition_watermark = current.clone(); lower_partition_watermark["last-partition-id"] = serde_json::Value::from(998); let error = validate_table_metadata_transition(¤t, &lower_partition_watermark) .expect_err("last-partition-id must not decrease across commits"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("last-partition-id must not decrease across table metadata commits".to_string()) + TableCatalogStoreError::Invalid(message) if message == "last-partition-id must not decrease across table metadata commits" ); let mut current_sequence = current.clone(); @@ -1616,9 +1690,9 @@ fn iceberg_metadata_transition_preserves_assignment_watermarks_and_schema_histor lower_sequence_watermark["last-sequence-number"] = serde_json::Value::from(1); let error = validate_table_metadata_transition(¤t_sequence, &lower_sequence_watermark) .expect_err("last-sequence-number must not decrease across commits"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("last-sequence-number must not decrease across table metadata commits".to_string()) + TableCatalogStoreError::Invalid(message) if message == "last-sequence-number must not decrease across table metadata commits" ); let mut current_partitioned = current.clone(); @@ -1636,9 +1710,9 @@ fn iceberg_metadata_transition_preserves_assignment_watermarks_and_schema_histor modified_partition["partition-specs"][0]["fields"][0]["name"] = serde_json::Value::from("renamed_id"); let error = validate_table_metadata_transition(¤t_partitioned, &modified_partition) .expect_err("published partition specs must be immutable"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("existing partition spec 0 must not be modified".to_string()) + TableCatalogStoreError::Invalid(message) if message == "existing partition spec 0 must not be modified" ); let mut current_sorted = current.clone(); @@ -1659,9 +1733,9 @@ fn iceberg_metadata_transition_preserves_assignment_watermarks_and_schema_histor modified_sort["sort-orders"][1]["fields"][0]["direction"] = serde_json::Value::from("desc"); let error = validate_table_metadata_transition(¤t_sorted, &modified_sort).expect_err("published sort orders must be immutable"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("existing sort order 1 must not be modified".to_string()) + TableCatalogStoreError::Invalid(message) if message == "existing sort order 1 must not be modified" ); let mut current_with_snapshot = current.clone(); @@ -1677,18 +1751,18 @@ fn iceberg_metadata_transition_preserves_assignment_watermarks_and_schema_histor modified_snapshot["snapshots"][0]["timestamp-ms"] = serde_json::Value::from(2); let error = validate_table_metadata_transition(¤t_with_snapshot, &modified_snapshot) .expect_err("published snapshots must be immutable"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("existing snapshot 10 must not be modified".to_string()) + TableCatalogStoreError::Invalid(message) if message == "existing snapshot 10 must not be modified" ); let mut modified_history = current.clone(); modified_history["schemas"][0]["fields"][0]["type"] = serde_json::Value::from("string"); let error = validate_table_metadata_transition(¤t, &modified_history) .expect_err("published schema definitions must be immutable"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("existing schema 0 must not be modified".to_string()) + TableCatalogStoreError::Invalid(message) if message == "existing schema 0 must not be modified" ); let mut current_with_retired_id = current.clone(); @@ -1713,9 +1787,9 @@ fn iceberg_metadata_transition_preserves_assignment_watermarks_and_schema_histor reused_id["current-schema-id"] = serde_json::Value::from(2); let error = validate_table_metadata_transition(¤t_with_retired_id, &reused_id) .expect_err("new schemas must not reuse a previously assigned field ID"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("schema field 1 cannot reuse a previously assigned field id".to_string()) + TableCatalogStoreError::Invalid(message) if message == "schema field 1 cannot reuse a previously assigned field id" ); let mut valid = current.clone(); @@ -2057,9 +2131,9 @@ fn iceberg_metadata_validation_rejects_duplicate_snapshot_statistics() { metadata["statistics"] = serde_json::json!([statistics.clone(), statistics]); let error = validate_supported_table_metadata(&metadata).expect_err("a snapshot must not have duplicate statistics entries"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("statistics contains duplicate entries for snapshot 10".to_string()) + TableCatalogStoreError::Invalid(message) if message == "statistics contains duplicate entries for snapshot 10" ); } @@ -2118,9 +2192,9 @@ async fn snapshot_validation_bounds_changed_statistics_object_fanout() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("statistics object fanout must be bounded before storage lookups"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("statistics object count exceeds the commit limit".to_string()) + TableCatalogStoreError::Invalid(message) if message == "statistics object count exceeds the commit limit" ); } @@ -2179,9 +2253,9 @@ async fn snapshot_validation_bounds_changed_statistics_object_bytes() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("statistics bytes must be bounded before storage lookups"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("statistics bytes exceed the commit validation limit".to_string()) + TableCatalogStoreError::Invalid(message) if message == "statistics bytes exceed the commit validation limit" ); } @@ -2215,9 +2289,9 @@ async fn snapshot_validation_rechecks_retained_statistics_locations() { let error = validate_table_snapshot_changes(&context, Some(¤t), &target) .await .expect_err("retained statistics must remain inside the table warehouse"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("statistics object is outside the table warehouse".to_string()) + TableCatalogStoreError::Invalid(message) if message == "statistics object is outside the table warehouse" ); } @@ -2255,9 +2329,9 @@ async fn snapshot_validation_rejects_non_puffin_table_statistics() { let error = validate_table_snapshot_changes(&context, Some(¤t), &target) .await .expect_err("table statistics must be a Puffin file"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("table statistics object is not a Puffin file".to_string()) + TableCatalogStoreError::Invalid(message) if message == "table statistics object is not a Puffin file" ); } @@ -2300,9 +2374,9 @@ async fn snapshot_validation_accepts_non_parquet_partition_statistics() { let error = validate_table_snapshot_changes(&context, Some(¤t), &target) .await .expect_err("partition statistics must use a supported table data-file envelope"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("partition statistics object is not a supported table data file".to_string()) + TableCatalogStoreError::Invalid(message) if message == "partition statistics object is not a supported table data file" ); } @@ -2340,9 +2414,9 @@ async fn snapshot_validation_rejects_statistics_size_mismatch() { let error = validate_table_snapshot_changes(&context, Some(¤t), &target) .await .expect_err("statistics lengths must match the published object"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("statistics file-size-in-bytes does not match the object".to_string()) + TableCatalogStoreError::Invalid(message) if message == "statistics file-size-in-bytes does not match the object" ); } @@ -2358,11 +2432,9 @@ fn iceberg_metadata_validation_bounds_snapshot_sequence_numbers() { "summary": {"operation": "append"} }]); let error = validate_supported_table_metadata(&v2).expect_err("a v2 snapshot sequence must not exceed the table sequence"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid( - "Iceberg v2 snapshot sequence-number must be between zero and last-sequence-number".to_string() - ) + TableCatalogStoreError::Invalid(message) if message == "Iceberg v2 snapshot sequence-number must be between zero and last-sequence-number" ); v2["last-sequence-number"] = serde_json::Value::from(0); v2["snapshots"][0] @@ -2370,9 +2442,9 @@ fn iceberg_metadata_validation_bounds_snapshot_sequence_numbers() { .expect("snapshot should be an object") .remove("sequence-number"); let error = validate_supported_table_metadata(&v2).expect_err("Iceberg v2 snapshots must include sequence-number"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("Iceberg v2 snapshot sequence-number is required".to_string()) + TableCatalogStoreError::Invalid(message) if message == "Iceberg v2 snapshot sequence-number is required" ); let mut v1 = serde_json::json!({ @@ -2391,9 +2463,9 @@ fn iceberg_metadata_validation_bounds_snapshot_sequence_numbers() { }] }); let error = validate_supported_table_metadata(&v1).expect_err("v1 metadata must not carry a non-zero snapshot sequence"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("Iceberg v1 snapshot sequence-number must be zero when present".to_string()) + TableCatalogStoreError::Invalid(message) if message == "Iceberg v1 snapshot sequence-number must be zero when present" ); v1["snapshots"][0]["sequence-number"] = serde_json::Value::from(0); validate_supported_table_metadata(&v1).expect("a zero v1 compatibility sequence should validate"); @@ -2434,7 +2506,9 @@ fn metadata_log_locations_normalize_same_bucket_s3_uris() { }); assert_eq!( - metadata_log_locations(&metadata, "warehouse", &namespace, &table), + metadata_log_locations_matching(&metadata, "warehouse", |location| { + is_valid_table_metadata_location(&namespace, &table, location) + }), BTreeSet::from([metadata_location]) ); } @@ -2601,9 +2675,9 @@ async fn iceberg_snapshot_graph_rejects_unknown_partition_specs() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("manifest partition specs absent from table metadata must be rejected"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("snapshot manifest references missing partition spec 7".to_string()) + TableCatalogStoreError::Invalid(message) if message == "snapshot manifest references missing partition spec 7" ); } @@ -2684,9 +2758,9 @@ async fn iceberg_snapshot_graph_revalidates_unchanged_snapshots_after_spec_remov let error = validate_table_snapshot_changes(&context, Some(¤t), &target) .await .expect_err("removing a spec referenced by an unchanged snapshot must fail"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("snapshot manifest references missing partition spec 7".to_string()) + TableCatalogStoreError::Invalid(message) if message == "snapshot manifest references missing partition spec 7" ); } @@ -2904,9 +2978,9 @@ async fn iceberg_snapshot_registration_validates_all_retained_snapshots() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("registration must validate every retained snapshot"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("snapshot manifest-list object is missing".to_string()) + TableCatalogStoreError::Invalid(message) if message == "snapshot manifest-list object is missing" ); } @@ -3016,9 +3090,9 @@ async fn iceberg_snapshot_graph_revalidates_cached_manifest_declarations() { let error = validate_table_snapshot_changes(&context, Some(¤t_metadata), &metadata) .await .expect_err("every manifest-list declaration must match the cached manifest object"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("manifest-list manifest_length does not match the manifest object".to_string()) + TableCatalogStoreError::Invalid(message) if message == "manifest-list manifest_length does not match the manifest object" ); backend @@ -3036,9 +3110,9 @@ async fn iceberg_snapshot_graph_revalidates_cached_manifest_declarations() { let error = validate_table_snapshot_changes(&context, Some(¤t_metadata), &metadata) .await .expect_err("every cached manifest must match each manifest-list partition spec declaration"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("manifest partition-spec-id does not match its manifest-list entry".to_string()) + TableCatalogStoreError::Invalid(message) if message == "manifest partition-spec-id does not match its manifest-list entry" ); } @@ -3090,9 +3164,9 @@ async fn iceberg_snapshot_graph_bounds_shared_manifest_traversals() { let error = validate_table_snapshot_changes(&context, Some(¤t_metadata), &metadata) .await .expect_err("logical manifest traversals must remain bounded across shared snapshots"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("snapshot manifest traversal count exceeds the commit limit".to_string()) + TableCatalogStoreError::Invalid(message) if message == "snapshot manifest traversal count exceeds the commit limit" ); } @@ -3184,9 +3258,9 @@ async fn iceberg_v2_snapshot_graph_rejects_embedded_v2_manifests() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("new v2 snapshots must use a manifest list"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("new Iceberg v2 snapshots require manifest-list".to_string()) + TableCatalogStoreError::Invalid(message) if message == "new Iceberg v2 snapshots require manifest-list" ); } @@ -3229,9 +3303,9 @@ async fn iceberg_snapshot_graph_rejects_delete_files_in_data_manifest() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("a data manifest must not contain delete files"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("manifest-list content does not match manifest file content".to_string()) + TableCatalogStoreError::Invalid(message) if message == "manifest-list content does not match manifest file content" ); } @@ -3315,9 +3389,9 @@ async fn iceberg_snapshot_graph_rejects_manifest_length_mismatch() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("manifest-list lengths must match the published manifest object"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("manifest-list manifest_length does not match the manifest object".to_string()) + TableCatalogStoreError::Invalid(message) if message == "manifest-list manifest_length does not match the manifest object" ); } @@ -3360,9 +3434,9 @@ async fn iceberg_snapshot_graph_rejects_manifest_partition_spec_mismatch() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("manifest headers must agree with their manifest-list entry"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("manifest partition-spec-id does not match its manifest-list entry".to_string()) + TableCatalogStoreError::Invalid(message) if message == "manifest partition-spec-id does not match its manifest-list entry" ); } @@ -3574,7 +3648,7 @@ fn parquet_i32_values(data: Vec) -> Vec { values } -fn test_bucket_entry(bucket: &str) -> TableBucketEntry { +pub(super) fn test_bucket_entry(bucket: &str) -> TableBucketEntry { TableBucketEntry { version: TABLE_CATALOG_ENTRY_VERSION, table_bucket: bucket.to_string(), @@ -3588,7 +3662,7 @@ fn test_bucket_entry(bucket: &str) -> TableBucketEntry { } } -fn test_namespace_entry(bucket: &str, namespace: &Namespace) -> NamespaceEntry { +pub(super) fn test_namespace_entry(bucket: &str, namespace: &Namespace) -> NamespaceEntry { NamespaceEntry { version: TABLE_CATALOG_ENTRY_VERSION, table_bucket: bucket.to_string(), @@ -3601,7 +3675,12 @@ fn test_namespace_entry(bucket: &str, namespace: &Namespace) -> NamespaceEntry { } } -fn test_table_entry(bucket: &str, namespace: &Namespace, table: &IdentifierSegment, metadata_location: String) -> TableEntry { +pub(super) fn test_table_entry( + bucket: &str, + namespace: &Namespace, + table: &IdentifierSegment, + metadata_location: String, +) -> TableEntry { TableEntry { version: TABLE_CATALOG_ENTRY_VERSION, table_bucket: bucket.to_string(), @@ -4145,9 +4224,9 @@ async fn durable_catalog_backup_rejects_incomplete_snapshot_graph() { .create_durable_catalog_backup(bucket, None) .await .expect_err("backup must reject a snapshot without a manifest graph"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("snapshot manifest-list object is missing".to_string()) + TableCatalogStoreError::Invalid(message) if message == "snapshot manifest-list object is missing" ); } @@ -4620,6 +4699,2159 @@ async fn durable_catalog_restore_retries_after_ambiguous_maintenance_replacement assert_eq!(restored.data, serde_json::to_vec(&backup_report).unwrap()); } +#[tokio::test] +async fn durable_strong_table_maintenance_uses_strong_entry_and_recoverable_sidecars() { + let backend = TestCatalogObjectBackend::default(); + let store = ConfiguredTableCatalogStore::new_for_test(backend.clone(), TableCatalogBackingMode::DurableStrong); + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let metadata_location = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let old_metadata_location = default_table_metadata_file_path(&namespace, &table, "00000.metadata.json"); + + store.put_table_bucket(test_bucket_entry(bucket)).await.unwrap(); + store + .create_namespace(test_namespace_entry(bucket, &namespace)) + .await + .unwrap(); + store + .create_table(test_table_entry(bucket, &namespace, &table, metadata_location.clone())) + .await + .unwrap(); + backend + .seed_object(bucket, &metadata_location, br#"{"metadata-log":[]}"#.to_vec()) + .await; + backend.seed_object(bucket, &old_metadata_location, b"{}".to_vec()).await; + + store + .put_table_maintenance_config( + bucket, + &namespace.public_name(), + table.as_str(), + TableMaintenanceConfig { + version: TABLE_MAINTENANCE_CONFIG_VERSION, + background_enabled: true, + delete_enabled: true, + retain_recent_metadata_files: 0, + ..Default::default() + }, + ) + .await + .unwrap(); + assert!( + store + .get_table_maintenance_config(bucket, &namespace.public_name(), table.as_str()) + .await + .unwrap() + .background_enabled + ); + + let scheduled = store + .run_table_maintenance_scheduler_once(bucket, &namespace.public_name(), table.as_str(), "scheduler-1".to_string()) + .await + .unwrap(); + assert_eq!(scheduled.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(scheduled.report.job.operation, TableMetadataMaintenanceOperation::Delete); + + let completed = store + .run_table_metadata_maintenance_worker_once(bucket, &namespace.public_name(), table.as_str(), "worker-1".to_string()) + .await + .unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.table_id, "table-id"); + assert_eq!(completed.current_metadata_location, metadata_location); + assert_eq!(completed.job.deleted_metadata_file_count, 1); + assert!(!backend.object_exists(bucket, &old_metadata_location).await.unwrap()); + assert!( + store + .get_table_metadata_maintenance_report(bucket, &namespace.public_name(), table.as_str(), &completed.job.job_id) + .await + .unwrap() + .is_some() + ); + + let mut running = completed.clone(); + running.job.status = TableMetadataMaintenanceJobStatus::Running; + running.job.worker_id = Some("worker-heartbeat".to_string()); + running.job.lease_id = "lease-heartbeat".to_string(); + running.job.heartbeat_at = None; + running.job.finished_at = None; + let paths = TableCatalogObjectPaths::default(); + let current_job_path = paths.table_maintenance_current_job_path(bucket, &namespace, &table, &running.job.table_id); + backend + .seed_object(RUSTFS_META_BUCKET, ¤t_job_path, serde_json::to_vec(&running).unwrap()) + .await; + let heartbeat = store + .heartbeat_table_metadata_maintenance_job( + bucket, + &namespace.public_name(), + table.as_str(), + &running.job.job_id, + "lease-heartbeat", + "worker-heartbeat", + ) + .await + .unwrap(); + assert_eq!(heartbeat.job.status, TableMetadataMaintenanceJobStatus::Running); + assert!(heartbeat.job.heartbeat_at.is_some()); + assert!( + heartbeat + .audit_events + .iter() + .any(|event| event.action == TableMaintenanceAuditAction::WorkerHeartbeat) + ); + + let reloaded = ConfiguredTableCatalogStore::new_for_test(backend.clone(), TableCatalogBackingMode::DurableStrong); + assert!( + reloaded + .get_table_maintenance_config(bucket, &namespace.public_name(), table.as_str()) + .await + .unwrap() + .background_enabled + ); + let reloaded_report = reloaded + .get_table_metadata_maintenance_report(bucket, &namespace.public_name(), table.as_str(), &completed.job.job_id) + .await + .unwrap() + .expect("maintenance report should survive strong catalog reload"); + assert_eq!(reloaded_report.job.status, TableMetadataMaintenanceJobStatus::Running); + assert_eq!( + reloaded + .get_table_maintenance_scheduler_report(bucket, &namespace.public_name(), table.as_str()) + .await + .unwrap() + .current_job + .expect("reloaded scheduler report should expose the current job") + .job_id, + completed.job.job_id + ); +} + +async fn seed_strong_maintenance_table( + store: &StrongTableCatalogStore, + backend: &TestCatalogObjectBackend, +) -> TableEntry { + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let current = default_table_metadata_file_path(&namespace, &table, "00003.metadata.json"); + let entry = test_table_entry(bucket, &namespace, &table, current.clone()); + store.put_table_bucket(test_bucket_entry(bucket)).await.unwrap(); + store + .create_namespace(test_namespace_entry(bucket, &namespace)) + .await + .unwrap(); + store.create_table(entry.clone()).await.unwrap(); + backend + .seed_object(bucket, ¤t, br#"{"metadata-log":[]}"#.to_vec()) + .await; + store + .put_table_maintenance_config( + bucket, + "sales", + "orders", + TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + retain_recent_metadata_files: 0, + ..Default::default() + }, + ) + .await + .unwrap(); + entry +} + +async fn maintenance_test_object_versions(backend: &TestCatalogObjectBackend) -> BTreeMap<(String, String), TableCatalogObject> { + backend + .state + .lock() + .await + .objects + .iter() + .map(|(key, object)| { + ( + key.clone(), + TableCatalogObject { + data: object.data.clone(), + etag: Some(object.etag.clone()), + mod_time: object.mod_time, + }, + ) + }) + .collect() +} + +#[tokio::test] +async fn durable_strong_maintenance_follows_renamed_metadata_and_manifest_roots() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = entry.table_bucket.as_str(); + let namespace = Namespace::parse("sales").unwrap(); + let source = IdentifierSegment::parse("orders").unwrap(); + let destination = IdentifierSegment::parse("archived_orders").unwrap(); + let metadata_dir = default_table_metadata_dir_path(&namespace, &source); + let logged = format!("{metadata_dir}/00002.metadata.json"); + let orphan = format!("{metadata_dir}/00001.metadata.json"); + let manifest_list = format!("{metadata_dir}/snap-10.avro"); + let manifest = format!("{metadata_dir}/manifest-10.avro"); + let orphan_manifest = format!("{metadata_dir}/manifest-orphan.avro"); + let data = format!("{}/referenced.parquet", default_table_data_dir_path(&namespace, &source)); + let delete = format!("{}/referenced.parquet", default_table_delete_dir_path(&namespace, &source)); + let orphan_data = format!("{}/orphan.parquet", default_table_data_dir_path(&namespace, &source)); + let unrelated = default_table_metadata_file_path(&namespace, &destination, "00000.metadata.json"); + let manifest_bytes = manifest_avro_bytes(&[(&data, 0), (&delete, 1)]); + backend + .seed_object(bucket, &manifest_list, manifest_list_avro_bytes(&[(&manifest, manifest_bytes.len())])) + .await; + backend.seed_object(bucket, &manifest, manifest_bytes).await; + for location in [&logged, &orphan, &orphan_manifest, &data, &delete, &orphan_data, &unrelated] { + backend.seed_object(bucket, location, b"{}".to_vec()).await; + } + backend + .seed_object( + bucket, + &entry.metadata_location, + serde_json::to_vec(&serde_json::json!({ + "metadata-log": [{"metadata-file": format!("s3://{bucket}/{logged}")}], + "snapshots": [{"snapshot-id": 10, "manifest-list": manifest_list}] + })) + .unwrap(), + ) + .await; + store + .rename_table(bucket, "sales", "orders", "sales", "archived_orders") + .await + .unwrap(); + store + .put_table_maintenance_config( + bucket, + "sales", + "archived_orders", + TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + ..Default::default() + }, + ) + .await + .unwrap(); + + let queued = store + .run_table_maintenance_scheduler_once(bucket, "sales", "archived_orders", "rename-scheduler".to_string()) + .await + .unwrap(); + assert_eq!(queued.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert!(queued.report.retained_metadata_locations.contains(&logged)); + assert_eq!(queued.report.deletable_metadata_locations, vec![orphan.clone()]); + assert_eq!(queued.report.reachability_graph.status, TableMaintenanceReachabilityGraphStatus::Complete); + for location in [&manifest_list, &manifest, &data, &delete] { + assert!(queued.report.referenced_object_reports.iter().any(|report| { + &report.object_location == location && report.state == TableMetadataMaintenanceObjectState::Retained + })); + } + for location in [&orphan_manifest, &orphan_data] { + assert!(queued.report.deletable_object_locations.contains(location)); + } + let completed = store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "archived_orders", "rename-worker".to_string()) + .await + .unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.deleted_metadata_file_count, 1); + assert_eq!(completed.job.deleted_object_count, 2); + for location in [ + &entry.metadata_location, + &logged, + &manifest_list, + &manifest, + &data, + &delete, + &unrelated, + ] { + assert!(backend.object_exists(bucket, location).await.unwrap(), "retained object {location}"); + } + for location in [&orphan, &orphan_manifest, &orphan_data] { + assert!(!backend.object_exists(bucket, location).await.unwrap(), "orphan object {location}"); + } + + let reloaded = StrongTableCatalogStore::new(backend.clone()); + let renamed = reloaded + .load_table(bucket, "sales", "archived_orders") + .await + .unwrap() + .unwrap(); + assert_eq!(renamed.table_id, entry.table_id); + assert_eq!(renamed.metadata_location, entry.metadata_location); + assert!(reloaded.load_table(bucket, "sales", "orders").await.unwrap().is_none()); + assert_eq!( + reloaded + .get_table_metadata_maintenance_report(bucket, "sales", "archived_orders", &completed.job.job_id) + .await + .unwrap() + .unwrap(), + completed + ); +} + +#[tokio::test] +async fn durable_strong_maintenance_rejects_shared_roots_before_successful_job_preflight() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = entry.table_bucket.as_str(); + let namespace = Namespace::parse("sales").unwrap(); + let source = IdentifierSegment::parse("orders").unwrap(); + let destination = IdentifierSegment::parse("archived_orders").unwrap(); + store + .rename_table(bucket, "sales", "orders", "sales", "archived_orders") + .await + .unwrap(); + let config = TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + ..Default::default() + }; + store + .put_table_maintenance_config(bucket, "sales", "archived_orders", config.clone()) + .await + .unwrap(); + let completed = store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "archived_orders", "before-share".to_string()) + .await + .unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + + let replacement_metadata = default_table_metadata_file_path(&namespace, &source, "00004.metadata.json"); + let mut replacement = test_table_entry(bucket, &namespace, &source, replacement_metadata.clone()); + replacement.table_id = "replacement-table-id".to_string(); + replacement.table_uuid = "replacement-table-uuid".to_string(); + replacement.warehouse_location = format!("s3://{bucket}/tables/replacement-table-id"); + backend + .seed_object(bucket, &replacement_metadata, br#"{"metadata-log":[]}"#.to_vec()) + .await; + store.create_table(replacement.clone()).await.unwrap(); + store + .put_table_maintenance_config(bucket, "sales", "orders", config) + .await + .unwrap(); + let renamed = store.load_table(bucket, "sales", "archived_orders").await.unwrap().unwrap(); + assert_ne!(renamed.table_id, replacement.table_id); + assert_ne!(renamed.warehouse_location, replacement.warehouse_location); + assert_eq!( + table_metadata_dir_path_for_entry(&renamed).unwrap(), + table_metadata_dir_path_for_entry(&replacement).unwrap() + ); + + let paths = TableCatalogObjectPaths::default(); + let mut replacement_report = completed.clone(); + replacement_report.job.table = "orders".to_string(); + replacement_report.job.table_id = replacement.table_id.clone(); + replacement_report.job.current_metadata_location = replacement_metadata.clone(); + replacement_report.current_metadata_location = replacement_metadata; + for path in [ + paths.table_maintenance_current_job_path(bucket, &namespace, &source, &replacement.table_id), + paths.table_maintenance_latest_job_path(bucket, &namespace, &source, &replacement.table_id), + paths.table_maintenance_job_path(bucket, &namespace, &source, &replacement.table_id, &replacement_report.job.job_id), + ] { + backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(&replacement_report).unwrap()) + .await; + } + for status in [ + TableMetadataMaintenanceJobStatus::Successful, + TableMetadataMaintenanceJobStatus::Running, + ] { + for (table, report) in [(&source, &replacement_report), (&destination, &completed)] { + let mut current = report.clone(); + current.job.status = status.clone(); + if status == TableMetadataMaintenanceJobStatus::Running { + current.job.worker_id = Some("active-worker".to_string()); + current.job.lease_id = "active-lease".to_string(); + current.job.heartbeat_at = Some(maintenance_timestamp(OffsetDateTime::now_utc())); + current.job.finished_at = None; + } + let path = paths.table_maintenance_current_job_path(bucket, &namespace, table, ¤t.job.table_id); + backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(¤t).unwrap()) + .await; + } + let before = maintenance_test_object_versions(&backend).await; + for table in [&source, &destination] { + assert_matches!( + store.run_table_maintenance_scheduler_once(bucket, "sales", table.as_str(), "shared-scheduler".to_string()).await, + Err(TableCatalogStoreError::Conflict(message)) if message.contains("exclusive metadata and warehouse roots") + ); + assert_matches!( + store.run_table_metadata_maintenance_worker_once(bucket, "sales", table.as_str(), "shared-worker".to_string()).await, + Err(TableCatalogStoreError::Conflict(message)) if message.contains("exclusive metadata and warehouse roots") + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before); + } + } +} + +#[tokio::test] +async fn durable_strong_maintenance_rejects_queued_cleanup_after_rename_and_recreate() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = entry.table_bucket.as_str(); + let namespace = Namespace::parse("sales").unwrap(); + let source = IdentifierSegment::parse("orders").unwrap(); + let orphan = default_table_metadata_file_path(&namespace, &source, "00001.metadata.json"); + backend.seed_object(bucket, &orphan, b"{}".to_vec()).await; + let queued = store + .run_table_maintenance_scheduler_once(bucket, "sales", "orders", "before-rename".to_string()) + .await + .unwrap(); + assert_eq!(queued.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(queued.report.deletable_metadata_locations, vec![orphan.clone()]); + store + .rename_table(bucket, "sales", "orders", "sales", "archived_orders") + .await + .unwrap(); + store + .put_table_maintenance_config( + bucket, + "sales", + "archived_orders", + TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + ..Default::default() + }, + ) + .await + .unwrap(); + let renamed_queued = store + .run_table_maintenance_scheduler_once(bucket, "sales", "archived_orders", "before-recreate".to_string()) + .await + .unwrap(); + assert_eq!(renamed_queued.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + let mut replacement = test_table_entry( + bucket, + &namespace, + &source, + default_table_metadata_file_path(&namespace, &source, "00004.metadata.json"), + ); + replacement.table_id = "replacement-table-id".to_string(); + replacement.table_uuid = "replacement-table-uuid".to_string(); + replacement.warehouse_location = format!("s3://{bucket}/tables/replacement-table-id"); + backend + .seed_object(bucket, &replacement.metadata_location, br#"{"metadata-log":[]}"#.to_vec()) + .await; + store.create_table(replacement).await.unwrap(); + let before = maintenance_test_object_versions(&backend).await; + for table in ["orders", "archived_orders"] { + assert_matches!( + store.run_table_metadata_maintenance_worker_once(bucket, "sales", table, "after-share".to_string()).await, + Err(TableCatalogStoreError::Conflict(message)) if message.contains("exclusive metadata and warehouse roots") + ); + } + assert_eq!(maintenance_test_object_versions(&backend).await, before); + assert!(backend.object_exists(bucket, &orphan).await.unwrap()); + let queued_path = + TableCatalogObjectPaths::default().table_maintenance_current_job_path(bucket, &namespace, &source, &entry.table_id); + assert!(backend.read_object(RUSTFS_META_BUCKET, &queued_path).await.unwrap().is_none()); + assert_eq!(renamed_queued.report.job.job_id, queued.report.job.job_id); + assert_eq!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "archived_orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(), + renamed_queued.report + ); +} + +#[tokio::test] +async fn durable_strong_scheduler_plans_without_bucket_lock_and_revalidates_full_config() { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let other = IdentifierSegment::parse("payments").unwrap(); + let other_current = default_table_metadata_file_path(&namespace, &other, "00001.metadata.json"); + let other_next = default_table_metadata_file_path(&namespace, &other, "00002.metadata.json"); + let mut other_entry = test_table_entry(bucket, &namespace, &other, other_current.clone()); + other_entry.table_id = "payments-id".to_string(); + other_entry.table_uuid = "payments-uuid".to_string(); + other_entry.warehouse_location = format!("s3://{bucket}/tables/payments-id"); + store.create_table(other_entry).await.unwrap(); + backend.seed_object(bucket, &other_next, b"{}".to_vec()).await; + let pause = backend.pause_next_read(bucket, &entry.metadata_location).await; + let scheduler_store = Arc::clone(&store); + let scheduler = tokio::spawn(async move { + scheduler_store + .run_table_maintenance_scheduler_once(bucket, "sales", "orders", "paused-plan".to_string()) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("planner must reach current metadata read"); + let committed = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.commit_table(TableCommitRequest { + table_bucket: bucket.to_string(), + namespace: "sales".to_string(), + table: "payments".to_string(), + commit_id: "unrelated-during-maintenance".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: "token-v1".to_string(), + expected_metadata_location: other_current, + new_metadata_location: other_next.clone(), + requirements: Vec::new(), + writer: Some("maintenance-test".to_string()), + }), + ) + .await + .expect("planning must release the bucket publication writer") + .unwrap(); + assert_eq!(committed.table.metadata_location, other_next); + assert!(!scheduler.is_finished()); + let mut changed = store.get_table_maintenance_config(bucket, "sales", "orders").await.unwrap(); + changed.max_retry_attempts = 1; + tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.put_table_maintenance_config(bucket, "sales", "orders", changed), + ) + .await + .expect("planning must release the table publication writer") + .unwrap(); + let winning_claim = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.run_table_maintenance_scheduler_once(bucket, "sales", "orders", "new-config-claim".to_string()), + ) + .await + .expect("a second scheduler can claim while the first planner is paused") + .unwrap(); + assert_eq!(winning_claim.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(winning_claim.report.job.max_retry_attempts, 1); + let before_claim = maintenance_test_object_versions(&backend).await; + pause.release(); + assert_matches!( + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, scheduler).await.unwrap().unwrap(), + Err(TableCatalogStoreError::Conflict(message)) if message.contains("config changed during planning") + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before_claim); + assert_eq!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(), + winning_claim.report + ); +} + +#[tokio::test] +async fn durable_strong_scheduler_revalidates_entry_after_unlocked_planning() { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let next = default_table_metadata_file_path(&namespace, &table, "00004.metadata.json"); + backend.seed_object(bucket, &next, b"{}".to_vec()).await; + let pause = backend.pause_next_read(bucket, &entry.metadata_location).await; + let scheduler_store = Arc::clone(&store); + let scheduler = tokio::spawn(async move { + scheduler_store + .run_table_maintenance_scheduler_once(bucket, "sales", "orders", "paused-plan".to_string()) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("planner must reach current metadata read"); + let committed = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.commit_table(TableCommitRequest { + table_bucket: bucket.to_string(), + namespace: "sales".to_string(), + table: "orders".to_string(), + commit_id: "target-during-maintenance".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: entry.version_token, + expected_metadata_location: entry.metadata_location, + new_metadata_location: next.clone(), + requirements: Vec::new(), + writer: Some("maintenance-test".to_string()), + }), + ) + .await + .expect("planning must release the table publication writer") + .unwrap(); + assert_eq!(committed.table.metadata_location, next); + let before_claim = maintenance_test_object_versions(&backend).await; + pause.release(); + assert_matches!( + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, scheduler).await.unwrap().unwrap(), + Err(TableCatalogStoreError::Conflict(message)) if message.contains("table changed during maintenance planning") + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before_claim); + assert!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .is_none() + ); +} + +#[tokio::test] +async fn durable_strong_rename_preserves_complete_maintenance_state_across_namespaces_and_reload() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let curated = Namespace::parse("curated").unwrap(); + store + .create_namespace(test_namespace_entry(&source.table_bucket, &curated)) + .await + .unwrap(); + let mut current = source.clone(); + for (namespace, table) in [("curated", "orders_v2"), ("curated", "orders_v3"), ("sales", "orders")] { + let reloaded = StrongTableCatalogStore::new(backend.clone()); + reloaded + .rename_table(&source.table_bucket, ¤t.namespace, ¤t.table, namespace, table) + .await + .unwrap(); + assert!(rename_maintenance_objects(&backend, ¤t).await.is_empty()); + assert!( + reloaded + .load_table(&source.table_bucket, ¤t.namespace, ¤t.table) + .await + .unwrap() + .is_none() + ); + current = reloaded + .load_table(&source.table_bucket, namespace, table) + .await + .unwrap() + .unwrap(); + assert_eq!(current.table_id, source.table_id); + assert_eq!(current.table_uuid, source.table_uuid); + assert_renamed_maintenance_objects(&backend, ¤t, &expected).await; + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path(&source.table_bucket)) + .await + .unwrap() + ); + } + let reloaded = StrongTableCatalogStore::new(backend.clone()); + assert_eq!( + reloaded + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap() + .max_retry_attempts, + 2 + ); + assert_eq!( + reloaded + .get_table_metadata_maintenance_report("analytics", "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap() + .job + .job_id, + "running-rename" + ); + assert_eq!( + reloaded + .get_table_metadata_maintenance_report("analytics", "sales", "orders", MAINTENANCE_JOB_ALIAS_LATEST) + .await + .unwrap() + .unwrap() + .job + .job_id, + "quarantined-rename" + ); +} + +fn rename_maintenance_root(entry: &TableEntry) -> String { + let config = TableCatalogObjectPaths::default().table_maintenance_config_path( + &entry.table_bucket, + &Namespace::parse(&entry.namespace).unwrap(), + &IdentifierSegment::parse(&entry.table).unwrap(), + &entry.table_id, + ); + config.strip_suffix(MAINTENANCE_CONFIG_FILE).unwrap().to_string() +} + +fn rename_maintenance_intent_path(bucket: &str) -> String { + let entry_path = TableCatalogObjectPaths::default().table_bucket_entry_path(bucket); + format!( + "{}maintenance-rename-intent.json", + entry_path.strip_suffix(TABLE_BUCKET_ENTRY_FILE).unwrap() + ) +} + +async fn rename_maintenance_objects( + backend: &TestCatalogObjectBackend, + entry: &TableEntry, +) -> BTreeMap { + let prefix = rename_maintenance_root(entry); + backend + .state + .lock() + .await + .objects + .iter() + .filter_map(|((bucket, path), object)| { + if bucket != RUSTFS_META_BUCKET { + return None; + } + path.strip_prefix(&prefix) + .map(|suffix| (suffix.to_string(), serde_json::from_slice(&object.data).unwrap())) + }) + .collect() +} + +async fn assert_renamed_maintenance_objects( + backend: &TestCatalogObjectBackend, + entry: &TableEntry, + expected: &BTreeMap, +) { + let renamed = expected + .iter() + .map(|(suffix, value)| { + let mut value = value.clone(); + for field in ["job", "snapshot-expiration", "compaction"] { + if let Some(owner) = value.get_mut(field).and_then(serde_json::Value::as_object_mut) { + owner.insert("namespace".to_string(), serde_json::Value::String(entry.namespace.clone())); + owner.insert("table".to_string(), serde_json::Value::String(entry.table.clone())); + } + } + (suffix.clone(), value) + }) + .collect::>(); + assert_eq!(rename_maintenance_objects(backend, entry).await, renamed); +} + +async fn seed_strong_rename_maintenance_state( + store: &StrongTableCatalogStore, + backend: &TestCatalogObjectBackend, +) -> TableEntry { + let entry = seed_strong_maintenance_table(store, backend).await; + let namespace = Namespace::parse(&entry.namespace).unwrap(); + let table = IdentifierSegment::parse(&entry.table).unwrap(); + store + .put_table_maintenance_config( + &entry.table_bucket, + &entry.namespace, + &entry.table, + TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + max_retry_attempts: 2, + quarantine_enabled: true, + quarantine_retention_seconds: 86_400, + ..Default::default() + }, + ) + .await + .unwrap(); + let mut successful = store + .run_table_metadata_maintenance_worker_once( + &entry.table_bucket, + &entry.namespace, + &entry.table, + "seed-rename".to_string(), + ) + .await + .unwrap(); + let metadata = serde_json::json!({"metadata-log": []}); + successful.snapshot_expiration = Some(table_snapshot_expiration_report( + &entry.table_bucket, + &namespace, + &table, + &entry, + &metadata, + TableSnapshotExpirationConfig { + min_snapshots_to_keep: 1, + max_snapshot_age_ms: 86_400_000, + }, + OffsetDateTime::now_utc(), + )); + successful.compaction = Some( + table_compaction_planning_report( + backend, + &entry.table_bucket, + &namespace, + &table, + &entry, + &metadata, + TableCompactionPlanningConfig { + target_file_size_bytes: 512 * 1024 * 1024, + small_file_threshold_bytes: 128 * 1024 * 1024, + min_input_files: 2, + max_rewrite_bytes_per_job: 1024 * 1024 * 1024, + }, + ) + .await + .unwrap(), + ); + let mut queued = successful.clone(); + queued.job.job_id = "queued-rename".to_string(); + queued.job.status = TableMetadataMaintenanceJobStatus::Queued; + queued.job.scheduler_id = Some("scheduler-rename".to_string()); + queued.job.scheduler_lease_id = "scheduler-lease-rename".to_string(); + queued.job.scheduled_at = Some(maintenance_timestamp(OffsetDateTime::now_utc())); + queued.job.finished_at = None; + push_table_maintenance_audit_event( + &mut queued, + OffsetDateTime::now_utc(), + TableMaintenanceAuditActor::Scheduler, + TableMaintenanceAuditAction::SchedulerQueued, + Some("preserve queued audit".to_string()), + None, + None, + ); + let mut running = queued.clone(); + running.job.job_id = "running-rename".to_string(); + running.job.status = TableMetadataMaintenanceJobStatus::Running; + running.job.worker_id = Some("worker-rename".to_string()); + running.job.lease_id = "worker-lease-rename".to_string(); + running.job.heartbeat_at = Some(maintenance_timestamp(OffsetDateTime::now_utc())); + push_table_maintenance_audit_event( + &mut running, + OffsetDateTime::now_utc(), + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerHeartbeat, + Some("preserve heartbeat audit".to_string()), + None, + None, + ); + let mut quarantined = running.clone(); + quarantined.job.job_id = "quarantined-rename".to_string(); + quarantined.job.status = TableMetadataMaintenanceJobStatus::Failed; + quarantined.job.failure_reason = Some("preserve quarantine details".to_string()); + quarantined.job.quarantined_object_count = 3; + quarantined.job.next_retry_after = Some(maintenance_timestamp(OffsetDateTime::now_utc() + Duration::hours(1))); + push_table_maintenance_audit_event( + &mut quarantined, + OffsetDateTime::now_utc(), + TableMaintenanceAuditActor::Operator, + TableMaintenanceAuditAction::QuarantineRetry, + Some("preserve operator audit".to_string()), + None, + Some(3), + ); + let paths = TableCatalogObjectPaths::default(); + for report in [&successful, &queued, &running, &quarantined] { + let path = paths.table_maintenance_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id, &report.job.job_id); + backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(report).unwrap()) + .await; + } + for (path, report) in [ + ( + paths.table_maintenance_current_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + &running, + ), + ( + paths.table_maintenance_latest_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + &quarantined, + ), + ] { + backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(report).unwrap()) + .await; + } + assert_eq!(rename_maintenance_objects(backend, &entry).await.len(), 7); + entry +} + +#[tokio::test] +async fn durable_strong_maintenance_gets_hold_publication_read_lock_through_sidecar_reads() { + for read_kind in ["config", "report"] { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected_objects = rename_maintenance_objects(&backend, &source).await; + let expected = match read_kind { + "config" => { + let config = store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + assert_ne!(config, TableMaintenanceConfig::default()); + serde_json::to_value(config).unwrap() + } + "report" => { + let report = store + .get_table_metadata_maintenance_report("analytics", "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(); + assert_eq!(report.job.job_id, "running-rename"); + serde_json::to_value(report).unwrap() + } + _ => unreachable!(), + }; + let suffix = if read_kind == "report" { + MAINTENANCE_CURRENT_JOB_FILE + } else { + MAINTENANCE_CONFIG_FILE + }; + let pause = backend + .pause_next_read(RUSTFS_META_BUCKET, &format!("{}{suffix}", rename_maintenance_root(&source))) + .await; + let read_store = Arc::clone(&store); + let read = tokio::spawn(async move { + match read_kind { + "config" => serde_json::to_value( + read_store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(), + ) + .unwrap(), + "report" => serde_json::to_value( + read_store + .get_table_metadata_maintenance_report("analytics", "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(), + ) + .unwrap(), + _ => unreachable!(), + } + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("GET must reach its sidecar read"); + let publication_path = default_table_bucket_publication_lock_path(); + let attempts = backend.write_lock_acquisition_count("analytics", &publication_path).await; + let rename_store = Arc::clone(&store); + let mut rename = tokio::spawn(async move { + rename_store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, async { + while backend.write_lock_acquisition_count("analytics", &publication_path).await == attempts { + tokio::task::yield_now().await; + } + }) + .await + .expect("rename must attempt the bucket publication write lock"); + assert!( + tokio::time::timeout(StdDuration::from_millis(50), &mut rename).await.is_err(), + "rename must wait for the {read_kind} GET" + ); + assert_eq!(rename_maintenance_objects(&backend, &source).await, expected_objects); + pause.release(); + assert_eq!(tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, read).await.unwrap().unwrap(), expected); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, rename) + .await + .unwrap() + .unwrap() + .unwrap(); + assert!(store.load_table("analytics", "sales", "orders").await.unwrap().is_none()); + let destination = store.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(); + assert_renamed_maintenance_objects(&backend, &destination, &expected_objects).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + } +} + +#[tokio::test] +async fn durable_strong_scheduler_history_scan_allows_same_bucket_commit() { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let namespace = Namespace::parse("sales").unwrap(); + let payments = IdentifierSegment::parse("payments").unwrap(); + let payments_current = default_table_metadata_file_path(&namespace, &payments, "00001.metadata.json"); + let payments_next = default_table_metadata_file_path(&namespace, &payments, "00002.metadata.json"); + let mut payments_entry = test_table_entry("analytics", &namespace, &payments, payments_current.clone()); + payments_entry.table_id = "payments-id".to_string(); + payments_entry.table_uuid = "payments-uuid".to_string(); + payments_entry.warehouse_location = "s3://analytics/tables/payments-id".to_string(); + store.create_table(payments_entry).await.unwrap(); + backend.seed_object("analytics", &payments_next, b"{}".to_vec()).await; + let expected = store + .get_table_maintenance_scheduler_report("analytics", "sales", "orders") + .await + .unwrap(); + assert_eq!(expected.audit_timeline.len(), 4); + let job_path = TableCatalogObjectPaths::default().table_maintenance_job_path( + "analytics", + &namespace, + &IdentifierSegment::parse("orders").unwrap(), + &source.table_id, + "queued-rename", + ); + let pause = backend.pause_next_read(RUSTFS_META_BUCKET, &job_path).await; + let read_store = Arc::clone(&store); + let read = tokio::spawn(async move { + read_store + .get_table_maintenance_scheduler_report("analytics", "sales", "orders") + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("scheduler GET must reach a history read"); + let committed = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.commit_table(TableCommitRequest { + table_bucket: "analytics".to_string(), + namespace: "sales".to_string(), + table: "payments".to_string(), + commit_id: "payments-during-scheduler-history".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: "token-v1".to_string(), + expected_metadata_location: payments_current, + new_metadata_location: payments_next.clone(), + requirements: Vec::new(), + writer: Some("maintenance-test".to_string()), + }), + ) + .await + .expect("history scanning must not hold the bucket publication reader") + .unwrap(); + assert_eq!(committed.table.metadata_location, payments_next); + assert!(!read.is_finished()); + let before_read_completion = maintenance_test_object_versions(&backend).await; + pause.release(); + assert_eq!( + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, read) + .await + .unwrap() + .unwrap() + .unwrap(), + expected + ); + assert_eq!( + store.load_table("analytics", "sales", "payments").await.unwrap().unwrap(), + committed.table + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before_read_completion); +} + +#[tokio::test] +async fn durable_strong_scheduler_history_missing_during_rename_rejects_even_after_content_etag_aba() { + for round_trip in [false, true] { + let backend = TestCatalogObjectBackend::content_addressed(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected_objects = rename_maintenance_objects(&backend, &source).await; + let root = rename_maintenance_root(&source); + for (suffix, value) in &expected_objects { + backend + .put_json(RUSTFS_META_BUCKET, &format!("{root}{suffix}"), value.clone()) + .await; + } + let snapshot_path = StrongTableCatalogStore::::snapshot_object_path(); + let snapshot_before = backend + .read_object(RUSTFS_META_BUCKET, &snapshot_path) + .await + .unwrap() + .unwrap(); + let job_path = TableCatalogObjectPaths::default().table_maintenance_job_path( + "analytics", + &Namespace::parse("sales").unwrap(), + &IdentifierSegment::parse("orders").unwrap(), + &source.table_id, + "queued-rename", + ); + let job_before = backend.read_object(RUSTFS_META_BUCKET, &job_path).await.unwrap().unwrap(); + let before_read = backend.pause_before_next_read(RUSTFS_META_BUCKET, &job_path).await; + let after_read = backend.pause_next_read(RUSTFS_META_BUCKET, &job_path).await; + let read_store = Arc::clone(&store); + let read = tokio::spawn(async move { + read_store + .get_table_maintenance_scheduler_report("analytics", "sales", "orders") + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, before_read.wait_started()) + .await + .expect("fixed history membership must precede reading its job"); + tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.rename_table("analytics", "sales", "orders", "sales", "orders_v2"), + ) + .await + .expect("rename must proceed while history reading is paused") + .unwrap(); + assert!(!backend.object_exists(RUSTFS_META_BUCKET, &job_path).await.unwrap()); + before_read.release(); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, after_read.wait_started()) + .await + .expect("history read must capture the missing object before returning"); + if round_trip { + tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.rename_table("analytics", "sales", "orders_v2", "sales", "orders"), + ) + .await + .expect("reverse rename must proceed before the missing read is returned") + .unwrap(); + assert_eq!(store.load_table("analytics", "sales", "orders").await.unwrap().unwrap(), source); + let snapshot_after = backend + .read_object(RUSTFS_META_BUCKET, &snapshot_path) + .await + .unwrap() + .unwrap(); + assert_eq!(snapshot_after.data, snapshot_before.data); + assert_eq!(snapshot_after.etag, snapshot_before.etag); + let job_after = backend.read_object(RUSTFS_META_BUCKET, &job_path).await.unwrap().unwrap(); + assert_eq!(job_after.data, job_before.data); + assert_eq!(job_after.etag, job_before.etag); + assert_renamed_maintenance_objects(&backend, &source, &expected_objects).await; + } + let before_read_completion = maintenance_test_object_versions(&backend).await; + after_read.release(); + assert_matches!( + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, read).await.unwrap().unwrap(), + Err(TableCatalogStoreError::Unavailable(message)) if message.contains("maintenance scheduler history changed") + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before_read_completion); + } +} + +#[tokio::test] +async fn durable_strong_scheduler_history_membership_budget_precedes_report_reads() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_maintenance_table(&store, &backend).await; + let prefix = TableCatalogObjectPaths::default().table_maintenance_jobs_prefix( + "analytics", + &Namespace::parse("sales").unwrap(), + &IdentifierSegment::parse("orders").unwrap(), + &source.table_id, + ); + for index in 0..4097 { + backend + .seed_object(RUSTFS_META_BUCKET, &format!("{prefix}{index:064x}.json"), b"not-json".to_vec()) + .await; + } + let before = maintenance_test_object_versions(&backend).await; + assert_matches!( + store.get_table_maintenance_scheduler_report("analytics", "sales", "orders").await, + Err(TableCatalogStoreError::Unavailable(message)) if message.contains("maintenance scheduler history exceeds") + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before); +} + +#[tokio::test] +async fn durable_strong_maintenance_gets_preserve_defaults_without_sidecars() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let metadata = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let entry = test_table_entry("analytics", &namespace, &table, metadata); + store.put_table_bucket(test_bucket_entry("analytics")).await.unwrap(); + store + .create_namespace(test_namespace_entry("analytics", &namespace)) + .await + .unwrap(); + store.create_table(entry.clone()).await.unwrap(); + let before = maintenance_test_object_versions(&backend).await; + assert_eq!( + store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(), + TableMaintenanceConfig::default() + ); + assert!( + store + .get_table_metadata_maintenance_report("analytics", "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .is_none() + ); + let scheduler = store + .get_table_maintenance_scheduler_report("analytics", "sales", "orders") + .await + .unwrap(); + assert_eq!(scheduler.status, TableMaintenanceSchedulerStatus::Disabled); + assert_eq!(scheduler.config_source, TableMaintenanceConfigSource::Default); + assert!(!scheduler.background_enabled); + assert!(!scheduler.delete_enabled); + assert!(scheduler.current_job.is_none()); + assert!(scheduler.audit_timeline.is_empty()); + assert!(rename_maintenance_objects(&backend, &entry).await.is_empty()); + assert_eq!(maintenance_test_object_versions(&backend).await, before); +} + +#[tokio::test] +async fn durable_strong_rename_replaces_same_identity_destination_sidecars_and_removes_extra_history() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let mut destination = source.clone(); + destination.table = "orders_v2".to_string(); + let root = rename_maintenance_root(&destination); + for (suffix, value) in &expected { + let mut value = value.clone(); + if let Some(job) = value.get_mut("job") { + job["table"] = serde_json::json!("orders_v2"); + job["failure_reason"] = serde_json::json!("stale destination state"); + for field in ["snapshot-expiration", "compaction"] { + value[field]["table"] = serde_json::json!("orders_v2"); + } + } else { + value["delete-enabled"] = serde_json::json!(false); + } + backend + .seed_object(RUSTFS_META_BUCKET, &format!("{root}{suffix}"), serde_json::to_vec(&value).unwrap()) + .await; + } + let mut extra = expected[MAINTENANCE_CURRENT_JOB_FILE].clone(); + extra["job"]["job_id"] = serde_json::json!("extra-destination-history"); + for field in ["job", "snapshot-expiration", "compaction"] { + extra[field]["table"] = serde_json::json!("orders_v2"); + } + let extra_path = TableCatalogObjectPaths::default().table_maintenance_job_path( + "analytics", + &Namespace::parse("sales").unwrap(), + &IdentifierSegment::parse("orders_v2").unwrap(), + &source.table_id, + "extra-destination-history", + ); + backend + .seed_object(RUSTFS_META_BUCKET, &extra_path, serde_json::to_vec(&extra).unwrap()) + .await; + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + assert_renamed_maintenance_objects(&backend, &destination, &expected).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + assert!(!backend.object_exists(RUSTFS_META_BUCKET, &extra_path).await.unwrap()); +} + +async fn assert_strong_rename_recovers_each_sidecar_copy_failure(after_write: bool) { + for failed_index in 0..7 { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let mut destination = source.clone(); + destination.table = "orders_v2".to_string(); + let suffix = expected.keys().nth(failed_index).unwrap(); + let target = format!("{}{suffix}", rename_maintenance_root(&destination)); + if after_write { + backend.fail_after_next_put(RUSTFS_META_BUCKET, &target).await; + } else { + backend.fail_next_put(RUSTFS_META_BUCKET, &target).await; + } + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err(), + "copy failure {suffix}" + ); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + assert_eq!(reloaded.load_table("analytics", "sales", "orders").await.unwrap().unwrap(), source); + assert!( + reloaded + .load_table("analytics", "sales", "orders_v2") + .await + .unwrap() + .is_none() + ); + assert_eq!(rename_maintenance_objects(&backend, &source).await, expected); + assert!( + backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + assert_renamed_maintenance_objects(&backend, &destination, &expected).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + } +} + +#[tokio::test] +async fn durable_strong_rename_recovers_each_sidecar_copy_failure() { + assert_strong_rename_recovers_each_sidecar_copy_failure(false).await; +} + +#[tokio::test] +async fn durable_strong_rename_recovers_each_sidecar_copy_response_loss() { + assert_strong_rename_recovers_each_sidecar_copy_failure(true).await; +} + +#[tokio::test] +async fn durable_strong_rename_recovers_snapshot_failure_and_response_loss() { + for after_write in [false, true] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let snapshot = StrongTableCatalogStore::::snapshot_object_path(); + if after_write { + backend.fail_after_next_put(RUSTFS_META_BUCKET, &snapshot).await; + } else { + backend.fail_next_put(RUSTFS_META_BUCKET, &snapshot).await; + } + let result = store.rename_table("analytics", "sales", "orders", "sales", "orders_v2").await; + let reloaded = StrongTableCatalogStore::new(backend.clone()); + if after_write { + assert!(reloaded.load_table("analytics", "sales", "orders").await.unwrap().is_none()); + let destination = reloaded.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(); + assert_renamed_maintenance_objects(&backend, &destination, &expected).await; + if result.is_err() { + reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + } + } else { + assert!(result.is_err()); + assert_eq!(reloaded.load_table("analytics", "sales", "orders").await.unwrap().unwrap(), source); + assert_eq!(rename_maintenance_objects(&backend, &source).await, expected); + reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + } + reloaded + .rename_table("analytics", "sales", "orders_v2", "sales", "orders_v3") + .await + .unwrap(); + let final_entry = reloaded.load_table("analytics", "sales", "orders_v3").await.unwrap().unwrap(); + assert_renamed_maintenance_objects(&backend, &final_entry, &expected).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + } +} + +#[tokio::test] +async fn durable_strong_rename_retries_source_cleanup_failure_and_response_loss() { + for after_delete in [false, true] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let path = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&source)); + if after_delete { + backend.fail_after_next_delete(RUSTFS_META_BUCKET, &path).await; + } else { + let attempt = backend + .state + .lock() + .await + .delete_attempts + .get(&(RUSTFS_META_BUCKET.to_string(), path.clone())) + .copied() + .unwrap_or_default() + + 1; + backend.fail_delete_attempt(RUSTFS_META_BUCKET, &path, attempt).await; + } + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + assert!(reloaded.load_table("analytics", "sales", "orders").await.unwrap().is_none()); + let destination = reloaded.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(); + assert_renamed_maintenance_objects(&backend, &destination, &expected).await; + assert!( + backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + let recovered_destination = if after_delete { + reloaded + .rename_table("analytics", "sales", "orders_v2", "sales", "orders_v3") + .await + .unwrap(); + assert!(rename_maintenance_objects(&backend, &destination).await.is_empty()); + reloaded.load_table("analytics", "sales", "orders_v3").await.unwrap().unwrap() + } else { + reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + destination + }; + assert_renamed_maintenance_objects(&backend, &recovered_destination, &expected).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + } +} + +#[tokio::test] +async fn durable_strong_rename_rejects_corrupt_recovery_journal_without_mutation() { + for corruption in ["json", "version", "owner", "path", "report-owner", "unknown-field"] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let mut destination = source.clone(); + destination.table = "orders_v2".to_string(); + let target = format!("{}{MAINTENANCE_CONFIG_FILE}", rename_maintenance_root(&destination)); + backend.fail_next_put(RUSTFS_META_BUCKET, &target).await; + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + let journal_path = rename_maintenance_intent_path("analytics"); + let journal = backend.read_object(RUSTFS_META_BUCKET, &journal_path).await.unwrap().unwrap(); + let mut value: serde_json::Value = serde_json::from_slice(&journal.data).unwrap(); + match corruption { + "json" => {} + "version" => value["version"] = serde_json::json!(999), + "owner" => value["destination"]["table_uuid"] = serde_json::json!("other-table-uuid"), + "path" => { + let objects = value["source_objects"].as_object_mut().unwrap(); + let object = objects.remove(MAINTENANCE_CONFIG_FILE).unwrap(); + objects.insert("../outside.json".to_string(), object); + } + "report-owner" => { + value["source_objects"][MAINTENANCE_CURRENT_JOB_FILE]["data"]["job"]["namespace"] = serde_json::json!("other") + } + "unknown-field" => value["source_objects"][MAINTENANCE_CONFIG_FILE]["unexpected"] = serde_json::json!(true), + _ => unreachable!(), + } + let data = if corruption == "json" { + b"not json".to_vec() + } else { + serde_json::to_vec(&value).unwrap() + }; + backend.seed_object(RUSTFS_META_BUCKET, &journal_path, data).await; + let before = maintenance_test_object_versions(&backend).await; + let reloaded = StrongTableCatalogStore::new(backend.clone()); + assert_matches!( + reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await, + Err(TableCatalogStoreError::Invalid(_)) + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before, "corruption {corruption}"); + } +} + +#[tokio::test] +async fn durable_strong_create_table_recovers_an_uncommitted_maintenance_rename() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let mut inactive_destination = source.clone(); + inactive_destination.table = "orders_v2".to_string(); + let target = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&inactive_destination)); + backend.fail_after_next_put(RUSTFS_META_BUCKET, &target).await; + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + assert!(!rename_maintenance_objects(&backend, &inactive_destination).await.is_empty()); + + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders_v2").unwrap(); + let metadata = default_table_metadata_file_path(&namespace, &table, "recreated.metadata.json"); + let mut recreated = test_table_entry("analytics", &namespace, &table, metadata.clone()); + recreated.table_id = "recreated-table-id".to_string(); + recreated.table_uuid = "recreated-table-uuid".to_string(); + recreated.warehouse_location = "s3://analytics/tables/recreated-table-id".to_string(); + backend.seed_object("analytics", &metadata, b"{}".to_vec()).await; + let reloaded = StrongTableCatalogStore::new(backend.clone()); + reloaded.create_table(recreated.clone()).await.unwrap(); + assert_eq!(reloaded.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(), recreated); + assert_eq!(reloaded.load_table("analytics", "sales", "orders").await.unwrap().unwrap(), source); + assert_eq!(rename_maintenance_objects(&backend, &source).await, expected); + assert!(rename_maintenance_objects(&backend, &inactive_destination).await.is_empty()); + assert_eq!( + reloaded + .get_table_maintenance_config("analytics", "sales", "orders_v2") + .await + .unwrap(), + TableMaintenanceConfig::default() + ); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); +} + +#[tokio::test] +async fn durable_strong_drop_table_recovers_committed_maintenance_rename_cleanup() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let current = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&source)); + backend.fail_after_next_delete(RUSTFS_META_BUCKET, ¤t).await; + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + let destination = reloaded.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(); + assert_renamed_maintenance_objects(&backend, &destination, &expected).await; + reloaded.drop_table("analytics", "sales", "orders_v2").await.unwrap(); + assert!(reloaded.load_table("analytics", "sales", "orders").await.unwrap().is_none()); + assert!( + reloaded + .load_table("analytics", "sales", "orders_v2") + .await + .unwrap() + .is_none() + ); + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + assert_renamed_maintenance_objects(&backend, &destination, &expected).await; + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); +} + +#[tokio::test] +async fn durable_catalog_backup_recovers_pending_maintenance_rename_before_collecting_sidecars() { + for committed in [false, true] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_maintenance_table(&store, &backend).await; + backend + .seed_object( + "analytics", + &source.metadata_location, + serde_json::to_vec(&table_metadata_json_for_backup("analytics", &source.table_id, &source.table_uuid)).unwrap(), + ) + .await; + let completed = store + .run_table_metadata_maintenance_worker_once("analytics", "sales", "orders", "backup-recovery".to_string()) + .await + .unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + let expected = rename_maintenance_objects(&backend, &source).await; + let mut destination = source.clone(); + destination.table = "orders_v2".to_string(); + if committed { + let path = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&source)); + backend.fail_after_next_delete(RUSTFS_META_BUCKET, &path).await; + } else { + let path = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&destination)); + backend.fail_after_next_put(RUSTFS_META_BUCKET, &path).await; + } + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + assert!( + backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + let backup = reloaded.create_durable_catalog_backup("analytics", None).await.unwrap(); + assert_eq!(backup.maintenance_object_count, expected.len()); + let (active, inactive) = if committed { + (&destination, &source) + } else { + (&source, &destination) + }; + assert_eq!( + reloaded + .load_table("analytics", &active.namespace, &active.table) + .await + .unwrap() + .unwrap(), + *active + ); + assert!( + reloaded + .load_table("analytics", &inactive.namespace, &inactive.table) + .await + .unwrap() + .is_none() + ); + assert_renamed_maintenance_objects(&backend, active, &expected).await; + assert!(rename_maintenance_objects(&backend, inactive).await.is_empty()); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + } +} + +#[tokio::test] +async fn durable_strong_rename_retry_preserves_source_commit_and_config_progress() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let mut destination = source.clone(); + destination.table = "orders_v2".to_string(); + let destination_current = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&destination)); + backend.fail_after_next_put(RUSTFS_META_BUCKET, &destination_current).await; + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let next = default_table_metadata_file_path(&namespace, &table, "00004.metadata.json"); + backend.seed_object("analytics", &next, b"{}".to_vec()).await; + let committed = store + .commit_table(TableCommitRequest { + table_bucket: "analytics".to_string(), + namespace: "sales".to_string(), + table: "orders".to_string(), + commit_id: "source-progress-during-rename-recovery".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: source.version_token.clone(), + expected_metadata_location: source.metadata_location.clone(), + new_metadata_location: next.clone(), + requirements: Vec::new(), + writer: Some("maintenance-test".to_string()), + }) + .await + .unwrap(); + assert_eq!(committed.table.metadata_location, next); + let mut config = store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + config.max_retry_attempts = 1; + store + .put_table_maintenance_config("analytics", "sales", "orders", config.clone()) + .await + .unwrap(); + let expected = rename_maintenance_objects(&backend, &source).await; + assert!( + backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + let mut expected_entry = committed.table; + expected_entry.table = "orders_v2".to_string(); + assert_eq!( + reloaded.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(), + expected_entry + ); + assert_eq!( + reloaded + .get_table_maintenance_config("analytics", "sales", "orders_v2") + .await + .unwrap(), + config + ); + assert_renamed_maintenance_objects(&backend, &expected_entry, &expected).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); +} + +#[tokio::test] +async fn durable_strong_rename_recovery_validates_authoritative_sidecars_before_cleanup() { + for committed in [false, true] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let mut destination = source.clone(); + destination.table = "orders_v2".to_string(); + let source_current = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&source)); + let destination_current = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&destination)); + if committed { + backend.fail_after_next_delete(RUSTFS_META_BUCKET, &source_current).await; + } else { + backend.fail_after_next_put(RUSTFS_META_BUCKET, &destination_current).await; + } + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + assert!( + backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + let authoritative_current = if committed { &destination_current } else { &source_current }; + let object = backend + .read_object(RUSTFS_META_BUCKET, authoritative_current) + .await + .unwrap() + .unwrap(); + let mut report: serde_json::Value = serde_json::from_slice(&object.data).unwrap(); + report["job"]["table_id"] = serde_json::json!("wrong-authoritative-owner"); + backend + .seed_object(RUSTFS_META_BUCKET, authoritative_current, serde_json::to_vec(&report).unwrap()) + .await; + let before = maintenance_test_object_versions(&backend).await; + let reloaded = StrongTableCatalogStore::new(backend.clone()); + let retry = reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await; + if committed { + assert_matches!(retry, Err(TableCatalogStoreError::Invalid(_) | TableCatalogStoreError::Conflict(_))); + } else { + assert_matches!(retry, Err(TableCatalogStoreError::Invalid(_))); + } + assert_eq!(maintenance_test_object_versions(&backend).await, before); + } +} + +#[tokio::test] +async fn durable_strong_rename_rejects_unrecognized_or_misowned_sidecars_before_journaling() { + for invalid_path in [true, false] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let root = rename_maintenance_root(&source); + if invalid_path { + backend + .seed_object(RUSTFS_META_BUCKET, &format!("{root}unexpected.json"), b"{}".to_vec()) + .await; + } else { + let path = format!("{root}{MAINTENANCE_CURRENT_JOB_FILE}"); + let mut report = rename_maintenance_objects(&backend, &source).await[MAINTENANCE_CURRENT_JOB_FILE].clone(); + report["snapshot-expiration"]["table_id"] = serde_json::json!("other-table-id"); + backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(&report).unwrap()) + .await; + } + let before = maintenance_test_object_versions(&backend).await; + assert_matches!( + store.rename_table("analytics", "sales", "orders", "sales", "orders_v2").await, + Err(TableCatalogStoreError::Invalid(_)) + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + } +} + +#[tokio::test] +async fn durable_strong_rename_keeps_maintenance_backup_and_restore_complete() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_maintenance_table(&store, &backend).await; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let metadata_dir = default_table_metadata_dir_path(&namespace, &table); + let manifest_list = format!("{metadata_dir}/snap-20.avro"); + let manifest = format!("{metadata_dir}/manifest-20.avro"); + let data_file = format!("{}/part-00001.parquet", default_table_data_dir_path(&namespace, &table)); + let manifest_bytes = manifest_avro_bytes(&[(&data_file, 0)]); + backend + .seed_object( + "analytics", + &manifest_list, + manifest_list_avro_bytes(&[(&manifest, manifest_bytes.len())]), + ) + .await; + backend.seed_object("analytics", &manifest, manifest_bytes).await; + backend.seed_object("analytics", &data_file, b"d".to_vec()).await; + let mut metadata = table_metadata_json_for_backup("analytics", &source.table_id, &source.table_uuid); + metadata["last-sequence-number"] = serde_json::json!(7); + metadata["current-snapshot-id"] = serde_json::json!(20); + metadata["snapshots"] = serde_json::json!([{ + "snapshot-id": 20, "sequence-number": 7, "timestamp-ms": 1, + "manifest-list": format!("s3://analytics/{manifest_list}"), "summary": {"operation": "append"} + }]); + metadata["snapshot-log"] = serde_json::json!([{"timestamp-ms": 1, "snapshot-id": 20}]); + metadata["refs"] = serde_json::json!({"main": {"type": "branch", "snapshot-id": 20}}); + backend + .seed_object("analytics", &source.metadata_location, serde_json::to_vec(&metadata).unwrap()) + .await; + store + .run_table_metadata_maintenance_worker_once("analytics", "sales", "orders", "backup-rename".to_string()) + .await + .unwrap(); + let expected = rename_maintenance_objects(&backend, &source).await; + let before_rename = store.create_durable_catalog_backup("analytics", None).await.unwrap(); + assert_eq!(before_rename.object_count, 4); + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + let renamed = store.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(); + let backup = store.create_durable_catalog_backup("analytics", None).await.unwrap(); + assert_eq!(backup.object_count, before_rename.object_count); + assert_eq!(backup.maintenance_object_count, expected.len()); + store + .put_table_maintenance_config("analytics", "sales", "orders_v2", TableMaintenanceConfig::default()) + .await + .unwrap(); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + reloaded + .restore_durable_catalog_backup("analytics", &backup.backup_id, None, true) + .await + .unwrap(); + assert_renamed_maintenance_objects(&backend, &renamed, &expected).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); +} + +#[tokio::test] +async fn durable_strong_direct_worker_plans_without_bucket_publication_lock() { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let other = IdentifierSegment::parse("payments").unwrap(); + let other_current = default_table_metadata_file_path(&namespace, &other, "00001.metadata.json"); + let other_next = default_table_metadata_file_path(&namespace, &other, "00002.metadata.json"); + let mut other_entry = test_table_entry(bucket, &namespace, &other, other_current.clone()); + other_entry.table_id = "payments-id".to_string(); + other_entry.table_uuid = "payments-uuid".to_string(); + other_entry.warehouse_location = format!("s3://{bucket}/tables/payments-id"); + store.create_table(other_entry).await.unwrap(); + backend.seed_object(bucket, &other_next, b"{}".to_vec()).await; + assert!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .is_none() + ); + let pause = backend.pause_next_read(bucket, &entry.metadata_location).await; + let worker_store = Arc::clone(&store); + let worker = tokio::spawn(async move { + worker_store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "orders", "direct-paused-plan".to_string()) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("direct worker planner must reach current metadata read"); + let committed = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.commit_table(TableCommitRequest { + table_bucket: bucket.to_string(), + namespace: "sales".to_string(), + table: "payments".to_string(), + commit_id: "unrelated-during-direct-worker".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: "token-v1".to_string(), + expected_metadata_location: other_current, + new_metadata_location: other_next.clone(), + requirements: Vec::new(), + writer: Some("maintenance-test".to_string()), + }), + ) + .await + .expect("direct worker planning must release the bucket publication writer") + .unwrap(); + assert_eq!(committed.table.metadata_location, other_next); + assert!(!worker.is_finished()); + pause.release(); + let completed = tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, worker) + .await + .expect("direct worker must finish after planning resumes") + .unwrap() + .unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.worker_id.as_deref(), Some("direct-paused-plan")); + assert_eq!(completed.current_metadata_location, entry.metadata_location); + assert_eq!(store.load_table(bucket, "sales", "payments").await.unwrap().unwrap(), committed.table); + assert_eq!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(), + completed + ); +} + +#[tokio::test] +async fn durable_strong_direct_worker_does_not_overwrite_new_scheduler_claim() { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = "analytics"; + let previous = store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "orders", "previous-worker".to_string()) + .await + .unwrap(); + assert_eq!(previous.job.status, TableMetadataMaintenanceJobStatus::Successful); + let pause = backend.pause_next_read(bucket, &entry.metadata_location).await; + let worker_store = Arc::clone(&store); + let worker = tokio::spawn(async move { + worker_store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "orders", "stale-direct-plan".to_string()) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("direct worker planner must reach current metadata read"); + assert_eq!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(), + previous + ); + let winning_claim = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.run_table_maintenance_scheduler_once(bucket, "sales", "orders", "winning-scheduler".to_string()), + ) + .await + .expect("scheduler must be able to claim while the direct worker plans") + .unwrap(); + assert_eq!(winning_claim.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_ne!(winning_claim.report.job.job_id, previous.job.job_id); + let before_claim = maintenance_test_object_versions(&backend).await; + pause.release(); + assert_matches!( + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, worker).await.unwrap().unwrap(), + Err(TableCatalogStoreError::Conflict(message)) if message.contains("current maintenance job changed during planning") + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before_claim); + assert_eq!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(), + winning_claim.report + ); + assert_eq!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", &previous.job.job_id) + .await + .unwrap() + .unwrap(), + previous + ); +} + +#[tokio::test] +async fn durable_strong_direct_worker_revalidates_entry_and_full_config_after_planning() { + for change_config in [false, true] { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = "analytics"; + let pause = backend.pause_next_read(bucket, &entry.metadata_location).await; + let worker_store = Arc::clone(&store); + let worker = tokio::spawn(async move { + worker_store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "orders", "stale-direct-plan".to_string()) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("direct worker planner must reach current metadata read"); + let expected_conflict = if change_config { + let mut config = store.get_table_maintenance_config(bucket, "sales", "orders").await.unwrap(); + config.max_retry_attempts = 1; + tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.put_table_maintenance_config(bucket, "sales", "orders", config), + ) + .await + .expect("direct worker planning must permit config changes") + .unwrap(); + "maintenance config changed during worker planning" + } else { + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let next = default_table_metadata_file_path(&namespace, &table, "00004.metadata.json"); + backend.seed_object(bucket, &next, b"{}".to_vec()).await; + let committed = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.commit_table(TableCommitRequest { + table_bucket: bucket.to_string(), + namespace: "sales".to_string(), + table: "orders".to_string(), + commit_id: "target-during-direct-worker".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: entry.version_token, + expected_metadata_location: entry.metadata_location, + new_metadata_location: next.clone(), + requirements: Vec::new(), + writer: Some("maintenance-test".to_string()), + }), + ) + .await + .expect("direct worker planning must permit target table commits") + .unwrap(); + assert_eq!(committed.table.metadata_location, next); + "table changed during maintenance worker planning" + }; + let before_claim = maintenance_test_object_versions(&backend).await; + pause.release(); + assert_matches!( + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, worker).await.unwrap().unwrap(), + Err(TableCatalogStoreError::Conflict(message)) if message.contains(expected_conflict) + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before_claim); + assert!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .is_none() + ); + } +} + +#[tokio::test] +async fn durable_strong_quarantine_mutation_rejects_non_current_job() { + let backend = TestCatalogObjectBackend::default(); + let object_store = ObjectTableCatalogStore::new(backend.clone()); + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let metadata_location = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + + seed_table_for_metadata_maintenance(&object_store, bucket, &namespace, &table, metadata_location.clone()).await; + let store = ConfiguredTableCatalogStore::new_for_test(backend.clone(), TableCatalogBackingMode::DurableStrong); + store.put_table_bucket(test_bucket_entry(bucket)).await.unwrap(); + store + .create_namespace(test_namespace_entry(bucket, &namespace)) + .await + .unwrap(); + store + .create_table(test_table_entry(bucket, &namespace, &table, metadata_location.clone())) + .await + .unwrap(); + backend + .seed_object(bucket, &metadata_location, br#"{"metadata-log":[]}"#.to_vec()) + .await; + store + .put_table_maintenance_config( + bucket, + &namespace.public_name(), + table.as_str(), + TableMaintenanceConfig { + version: TABLE_MAINTENANCE_CONFIG_VERSION, + background_enabled: true, + quarantine_enabled: true, + quarantine_retention_seconds: 86_400, + ..Default::default() + }, + ) + .await + .unwrap(); + + let mut old = object_store + .plan_table_metadata_maintenance(bucket, &namespace.public_name(), table.as_str(), 0) + .await + .unwrap(); + old.job.status = TableMetadataMaintenanceJobStatus::Failed; + old.job.quarantine_enabled = true; + old.job.quarantined_object_count = 1; + object_store.put_table_metadata_maintenance_report(&old).await.unwrap(); + + let mut current = object_store + .plan_table_metadata_maintenance(bucket, &namespace.public_name(), table.as_str(), 0) + .await + .unwrap(); + current.job.status = TableMetadataMaintenanceJobStatus::Failed; + current.job.quarantine_enabled = true; + current.job.quarantined_object_count = 1; + object_store.put_table_metadata_maintenance_report(¤t).await.unwrap(); + + assert_matches!( + store + .apply_table_maintenance_quarantine_operation( + bucket, + &namespace.public_name(), + table.as_str(), + &old.job.job_id, + TableMaintenanceQuarantineOperationRequest { + action: TableMaintenanceQuarantineAction::Release, + reason: None, + }, + ) + .await, + Err(TableCatalogStoreError::Conflict(message)) if message.contains("not current") + ); + + let released = store + .apply_table_maintenance_quarantine_operation( + bucket, + &namespace.public_name(), + table.as_str(), + ¤t.job.job_id, + TableMaintenanceQuarantineOperationRequest { + action: TableMaintenanceQuarantineAction::Release, + reason: Some("strong quarantine release".to_string()), + }, + ) + .await + .unwrap(); + assert_eq!(released.report.job.quarantined_object_count, 0); + assert!( + released + .report + .job + .failure_reason + .as_deref() + .is_some_and(|reason| reason.contains("strong quarantine release")) + ); +} + #[tokio::test] async fn object_table_catalog_store_persists_view_entries_and_blocks_non_empty_namespace_drop() { let backend = TestCatalogObjectBackend { @@ -7908,6 +10140,67 @@ async fn maintenance_worker_run_honors_paused_config() { assert!(backend.object_exists(bucket, &old).await.unwrap()); } +#[tokio::test] +async fn object_maintenance_worker_delete_waits_for_table_publication() { + let backend = TestCatalogObjectBackend::default(); + let store = ObjectTableCatalogStore::new(backend.clone()); + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let old = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let current = default_table_metadata_file_path(&namespace, &table, "00002.metadata.json"); + seed_table_for_metadata_maintenance(&store, bucket, &namespace, &table, current.clone()).await; + backend.seed_object(bucket, &old, b"{}".to_vec()).await; + backend + .seed_object(bucket, ¤t, br#"{"metadata-log":[]}"#.to_vec()) + .await; + store + .put_table_maintenance_config( + bucket, + "sales", + "orders", + TableMaintenanceConfig { + version: TABLE_MAINTENANCE_CONFIG_VERSION, + background_enabled: true, + delete_enabled: true, + retain_recent_metadata_files: 0, + ..Default::default() + }, + ) + .await + .unwrap(); + let table_path = store.paths.table_entry_path(bucket, &namespace, &table); + let table_locks = backend.write_lock_acquisition_count(RUSTFS_META_BUCKET, &table_path).await; + let publication_path = default_table_publication_lock_path(&namespace, &table); + let publication_guard = backend.acquire_write_lock(bucket, &publication_path).await.unwrap(); + let publication_locks = backend.write_lock_acquisition_count(bucket, &publication_path).await; + let worker = tokio::spawn(async move { + store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "orders", "worker-a".to_string()) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, async { + while backend.write_lock_acquisition_count(bucket, &publication_path).await == publication_locks { + tokio::task::yield_now().await; + } + }) + .await + .expect("object maintenance deletion must acquire the table publication lock"); + assert!(!worker.is_finished()); + assert_eq!( + backend.write_lock_acquisition_count(RUSTFS_META_BUCKET, &table_path).await, + table_locks + 3 + ); + assert!(backend.object_exists(bucket, &old).await.unwrap()); + assert!(backend.object_exists(bucket, ¤t).await.unwrap()); + drop(publication_guard); + let report = worker.await.unwrap().unwrap(); + assert_eq!(report.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(report.job.deleted_metadata_file_count, 1); + assert!(!backend.object_exists(bucket, &old).await.unwrap()); + assert!(backend.object_exists(bucket, ¤t).await.unwrap()); +} + #[tokio::test] async fn maintenance_worker_run_claims_table_entry_write_lock() { let backend = TestCatalogObjectBackend::default(); @@ -8123,6 +10416,927 @@ async fn maintenance_scheduler_run_reuses_active_queued_job() { assert_eq!(second.scheduler.audit_timeline.len(), 1); } +struct ConditionalMaintenanceFixture { + backend: TestCatalogObjectBackend, + store: ConfiguredTableCatalogStore, + queued: TableMetadataMaintenanceReport, + entry: TableEntry, + old_metadata: String, + now: OffsetDateTime, +} + +impl ConditionalMaintenanceFixture { + async fn new(mode: TableCatalogBackingMode, max_retry_attempts: u16) -> Self { + let backend = TestCatalogObjectBackend::default(); + let store = ConfiguredTableCatalogStore::new_for_test(backend.clone(), mode); + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let current = default_table_metadata_file_path(&namespace, &table, "00002.metadata.json"); + let old_metadata = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + store.put_table_bucket(test_bucket_entry("analytics")).await.unwrap(); + store + .create_namespace(test_namespace_entry("analytics", &namespace)) + .await + .unwrap(); + store + .create_table(test_table_entry("analytics", &namespace, &table, current.clone())) + .await + .unwrap(); + let entry = store.load_table("analytics", "sales", "orders").await.unwrap().unwrap(); + backend + .seed_object("analytics", ¤t, br#"{"metadata-log":[]}"#.to_vec()) + .await; + backend.seed_object("analytics", &old_metadata, b"{}".to_vec()).await; + store + .put_table_maintenance_config( + "analytics", + "sales", + "orders", + TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + max_retry_attempts, + retry_initial_backoff_seconds: 1, + retry_max_backoff_seconds: 1, + ..Default::default() + }, + ) + .await + .unwrap(); + let queued = store + .run_table_maintenance_scheduler_once("analytics", "sales", "orders", "scheduler-a".to_string()) + .await + .unwrap() + .report; + assert_eq!(queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(queued.deletable_metadata_locations, vec![old_metadata.clone()]); + Self { + backend, + store, + queued, + entry, + old_metadata, + now: OffsetDateTime::now_utc(), + } + } + + async fn run( + &self, + worker: &str, + attempt: u16, + now: OffsetDateTime, + ) -> TableCatalogStoreResult { + let condition = TableMaintenanceWorkerCondition { + expected_job_id: self.queued.job.job_id.clone(), + expected_attempt: attempt, + }; + match &self.store { + ConfiguredTableCatalogStore::ObjectBacked(store) => { + store + .run_table_metadata_maintenance_worker_once_conditionally_with_clock( + "analytics", + "sales", + "orders", + worker.to_string(), + &condition, + || now, + ) + .await + } + ConfiguredTableCatalogStore::DurableStrong(store) => { + store + .run_table_metadata_maintenance_worker_once_conditionally_with_clock( + "analytics", + "sales", + "orders", + worker.to_string(), + &condition, + || now, + ) + .await + } + } + } + + fn report_paths(&self) -> [String; 3] { + let paths = TableCatalogObjectPaths::default(); + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + [ + paths.table_maintenance_job_path("analytics", &namespace, &table, &self.entry.table_id, &self.queued.job.job_id), + paths.table_maintenance_latest_job_path("analytics", &namespace, &table, &self.entry.table_id), + paths.table_maintenance_current_job_path("analytics", &namespace, &table, &self.entry.table_id), + ] + } + + async fn seed_report(&self, report: &TableMetadataMaintenanceReport) { + for path in self.report_paths() { + self.backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(report).unwrap()) + .await; + } + } + + async fn report(&self, id: &str) -> TableMetadataMaintenanceReport { + self.store + .get_table_metadata_maintenance_report("analytics", "sales", "orders", id) + .await + .unwrap() + .unwrap() + } + + async fn assert_report_views(&self, expected: &TableMetadataMaintenanceReport) { + for path in self.report_paths() { + let object = self.backend.read_object(RUSTFS_META_BUCKET, &path).await.unwrap().unwrap(); + let actual: TableMetadataMaintenanceReport = serde_json::from_slice(&object.data).unwrap(); + assert_eq!(&actual, expected, "report view {path}"); + } + assert_eq!( + self.store.load_table("analytics", "sales", "orders").await.unwrap(), + Some(self.entry.clone()) + ); + assert!( + self.backend + .object_exists("analytics", &self.entry.metadata_location) + .await + .unwrap() + ); + } + + async fn delete_attempts(&self) -> usize { + self.backend + .state + .lock() + .await + .delete_attempts + .get(&("analytics".to_string(), self.old_metadata.clone())) + .copied() + .unwrap_or_default() + } + + async fn fail_first_attempt(&self) -> TableMetadataMaintenanceReport { + self.backend.fail_delete_attempt("analytics", &self.old_metadata, 1).await; + self.run("worker-a", 0, self.now) + .await + .expect_err("injected delete failure must be surfaced"); + let failed = self.report(&self.queued.job.job_id).await; + assert_eq!(failed.job.status, TableMetadataMaintenanceJobStatus::Failed); + assert_eq!(failed.job.attempt, 1); + self.assert_report_views(&failed).await; + failed + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_claims_queued_job_and_replays_without_writes() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + let fixture = ConditionalMaintenanceFixture::new(mode, 0).await; + let completed = fixture.run("worker-a", 0, fixture.now).await.unwrap(); + assert_eq!(completed.job.job_id, fixture.queued.job.job_id); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.attempt, 1); + assert_eq!(completed.job.max_retry_attempts, 0); + assert_eq!(completed.job.worker_id.as_deref(), Some("worker-a")); + assert_eq!(completed.job.scheduler_id, fixture.queued.job.scheduler_id); + assert_eq!(completed.job.deleted_metadata_file_count, 1); + assert_eq!(completed.cleanup_candidate_locations, fixture.queued.cleanup_candidate_locations); + assert_eq!( + completed.job.cleanup_watermark_unix_seconds, + fixture.queued.job.cleanup_watermark_unix_seconds + ); + assert!( + !fixture + .backend + .object_exists("analytics", &fixture.old_metadata) + .await + .unwrap() + ); + fixture.assert_report_views(&completed).await; + let before = maintenance_test_object_versions(&fixture.backend).await; + let replayed = fixture.run("worker-a", 0, fixture.now + Duration::seconds(1)).await.unwrap(); + assert_eq!(replayed, completed); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 1); + assert_eq!( + completed + .audit_events + .iter() + .filter(|event| event.action == TableMaintenanceAuditAction::WorkerStarted) + .count(), + 1 + ); + assert_eq!( + completed + .audit_events + .iter() + .filter(|event| event.action == TableMaintenanceAuditAction::WorkerSucceeded) + .count(), + 1 + ); + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_samples_time_after_publication_wait_and_report_reads() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for expired in [false, true] { + let mut fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let config = fixture + .store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + let deadline = fixture.now; + let lease_timeout = Duration::seconds(i64::try_from(config.worker_lease_timeout_seconds).unwrap()); + fixture.queued.job.scheduled_at = Some(maintenance_timestamp(deadline - lease_timeout)); + fixture.seed_report(&fixture.queued).await; + let fixture = Arc::new(fixture); + let before = maintenance_test_object_versions(&fixture.backend).await; + let publication_path = default_table_bucket_publication_lock_path(); + let publication_guard = fixture + .backend + .acquire_write_lock("analytics", &publication_path) + .await + .unwrap(); + let attempts = fixture + .backend + .write_lock_acquisition_count("analytics", &publication_path) + .await; + let final_read = fixture + .backend + .pause_next_read(RUSTFS_META_BUCKET, &fixture.report_paths()[2]) + .await; + let clock = Arc::new(std::sync::Mutex::new((deadline - Duration::seconds(2), 0_usize))); + let worker_clock = Arc::clone(&clock); + let worker_fixture = Arc::clone(&fixture); + let worker = tokio::spawn(async move { + let condition = TableMaintenanceWorkerCondition { + expected_job_id: worker_fixture.queued.job.job_id.clone(), + expected_attempt: 0, + }; + let clock = move || { + let mut state = worker_clock.lock().unwrap(); + state.1 += 1; + state.0 + }; + match &worker_fixture.store { + ConfiguredTableCatalogStore::ObjectBacked(store) => { + store + .run_table_metadata_maintenance_worker_once_conditionally_with_clock( + "analytics", + "sales", + "orders", + "worker-a".to_string(), + &condition, + clock, + ) + .await + } + ConfiguredTableCatalogStore::DurableStrong(store) => { + store + .run_table_metadata_maintenance_worker_once_conditionally_with_clock( + "analytics", + "sales", + "orders", + "worker-a".to_string(), + &condition, + clock, + ) + .await + } + } + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, async { + while fixture + .backend + .write_lock_acquisition_count("analytics", &publication_path) + .await + == attempts + { + tokio::task::yield_now().await; + } + }) + .await + .expect("conditional worker must reach publication lock contention"); + assert!(!worker.is_finished(), "worker must wait for publication ownership"); + assert_eq!(clock.lock().unwrap().1, 0, "claim time must not precede publication ownership"); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + let claim_time = if expired { + deadline + } else { + deadline - Duration::nanoseconds(1) + }; + clock.lock().unwrap().0 = claim_time; + drop(publication_guard); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, final_read.wait_started()) + .await + .expect("conditional worker must read its final persisted report"); + assert_eq!(clock.lock().unwrap().1, 0, "claim time must follow every persisted report read"); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + final_read.release(); + let result = tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, worker) + .await + .expect("conditional worker must finish after publication and report reads") + .unwrap(); + assert_eq!(clock.lock().unwrap().1, 1, "claim validation must sample the clock once"); + if expired { + assert_matches!(result, Err(TableCatalogStoreError::Conflict(reason)) + if reason == "maintenance job is not eligible for a conditional claim; inspect its status and persisted attempt budget"); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + fixture.assert_report_views(&fixture.queued).await; + assert_eq!(fixture.queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(fixture.queued.job.attempt, 0); + assert_eq!(fixture.delete_attempts().await, 0); + assert!( + fixture + .backend + .object_exists("analytics", &fixture.old_metadata) + .await + .unwrap() + ); + } else { + let completed = result.expect("a queued lease remains claimable immediately before its deadline"); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.attempt, 1); + assert_eq!(completed.job.started_at, Some(maintenance_timestamp(claim_time))); + assert_eq!(completed.job.heartbeat_at, Some(maintenance_timestamp(claim_time))); + assert_eq!( + completed + .audit_events + .iter() + .filter(|event| event.action == TableMaintenanceAuditAction::WorkerStarted) + .map(|event| event.timestamp.clone()) + .collect::>(), + vec![maintenance_timestamp(claim_time)], + ); + assert_eq!(completed.job.deleted_metadata_file_count, 1); + assert_eq!(fixture.delete_attempts().await, 1); + assert!( + !fixture + .backend + .object_exists("analytics", &fixture.old_metadata) + .await + .unwrap() + ); + fixture.assert_report_views(&completed).await; + } + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_retries_failed_job_with_persisted_plan_and_budget() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + let mut fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let failed = fixture.fail_first_attempt().await; + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_eq!(fixture.run("worker-a", 0, fixture.now).await.unwrap(), failed); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + let retry_at = OffsetDateTime::parse( + failed.job.next_retry_after.as_deref().unwrap(), + &time::format_description::well_known::Rfc3339, + ) + .unwrap(); + assert_matches!( + fixture.run("worker-b", 1, retry_at - Duration::seconds(1)).await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + let mut config = fixture + .store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + config.max_retry_attempts = 10; + fixture + .store + .put_table_maintenance_config("analytics", "sales", "orders", config) + .await + .unwrap(); + fixture.store = ConfiguredTableCatalogStore::new_for_test(fixture.backend.clone(), mode); + let completed = fixture.run("worker-b", 1, retry_at).await.unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.job_id, failed.job.job_id); + assert_eq!(completed.job.attempt, 2); + assert_eq!(completed.job.max_retry_attempts, 2); + assert_ne!(completed.job.lease_id, failed.job.lease_id); + assert_eq!(completed.cleanup_candidate_locations, failed.cleanup_candidate_locations); + assert_eq!(completed.deletable_metadata_locations, failed.deletable_metadata_locations); + assert_eq!(completed.job.cleanup_watermark_unix_seconds, failed.job.cleanup_watermark_unix_seconds); + assert_eq!(fixture.delete_attempts().await, 2); + fixture.assert_report_views(&completed).await; + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_eq!(fixture.run("worker-b", 1, retry_at).await.unwrap(), completed); + assert_matches!(fixture.run("worker-a", 0, retry_at).await, Err(TableCatalogStoreError::Conflict(_))); + assert_matches!( + fixture + .store + .heartbeat_table_metadata_maintenance_job( + "analytics", + "sales", + "orders", + &failed.job.job_id, + &failed.job.lease_id, + "worker-a", + ) + .await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_does_not_reset_exhausted_budget_after_reload() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for budget in [0, 1, 2] { + let mut fixture = ConditionalMaintenanceFixture::new(mode, budget).await; + let mut failed = fixture.fail_first_attempt().await; + if budget == 2 { + fixture + .backend + .fail_delete_attempt("analytics", &fixture.old_metadata, 2) + .await; + let due = OffsetDateTime::parse( + failed.job.next_retry_after.as_deref().unwrap(), + &time::format_description::well_known::Rfc3339, + ) + .unwrap(); + fixture + .run("worker-b", 1, due) + .await + .expect_err("second delete failure must be surfaced"); + failed = fixture.report(&failed.job.job_id).await; + assert_eq!(failed.job.attempt, 2); + } + assert_eq!(failed.job.next_retry_after, None); + let mut config = fixture + .store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + config.max_retry_attempts = 10; + fixture + .store + .put_table_maintenance_config("analytics", "sales", "orders", config) + .await + .unwrap(); + fixture.store = ConfiguredTableCatalogStore::new_for_test(fixture.backend.clone(), mode); + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!( + fixture + .run("worker-c", failed.job.attempt, fixture.now + Duration::hours(1)) + .await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, usize::from(budget.max(1))); + fixture.assert_report_views(&failed).await; + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_rejects_unsafe_persisted_states_without_effects() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for scenario in [ + "job", + "attempt", + "expired-queued", + "expired-running", + "quarantine", + "pointer", + "generation", + "retention", + "paused", + "disabled", + "delete-disabled", + ] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let mut report = fixture.queued.clone(); + let mut attempt = 0; + match scenario { + "job" => report.job.job_id = "different-job".to_string(), + "attempt" => attempt = 2, + "expired-queued" => report.job.scheduled_at = Some(maintenance_timestamp(fixture.now - Duration::hours(1))), + "expired-running" => { + report.job.status = TableMetadataMaintenanceJobStatus::Running; + report.job.attempt = 1; + report.job.worker_id = Some("worker-a".to_string()); + report.job.lease_id = "expired-lease".to_string(); + report.job.heartbeat_at = Some(maintenance_timestamp(fixture.now - Duration::hours(1))); + } + "quarantine" => report.job.quarantined_object_count = 1, + "pointer" => report.job.current_metadata_location.push_str(".stale"), + "generation" => report.job.current_generation += 1, + "retention" => report.job.retain_recent_metadata_files += 1, + "paused" | "disabled" | "delete-disabled" => { + let mut config = fixture + .store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + match scenario { + "paused" => config.worker_paused = true, + "disabled" => config.background_enabled = false, + _ => config.delete_enabled = false, + } + fixture + .store + .put_table_maintenance_config("analytics", "sales", "orders", config) + .await + .unwrap(); + } + _ => unreachable!(), + } + fixture.seed_report(&report).await; + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!( + fixture.run("worker-a", attempt, fixture.now).await, + Err(TableCatalogStoreError::Conflict(_)), + "{mode:?}: {scenario}" + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before, "{mode:?}: {scenario}"); + assert_eq!(fixture.delete_attempts().await, 0, "{mode:?}: {scenario}"); + assert_eq!( + fixture.store.load_table("analytics", "sales", "orders").await.unwrap(), + Some(fixture.entry) + ); + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_claim_publication_failures_do_not_repeat_effects() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for target in 0..3 { + for after_write in [false, true] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let path = &fixture.report_paths()[target]; + if after_write { + fixture.backend.fail_after_next_put(RUSTFS_META_BUCKET, path).await; + } else { + fixture.backend.fail_next_put(RUSTFS_META_BUCKET, path).await; + } + fixture + .run("worker-a", 0, fixture.now) + .await + .expect_err("claim publication fault must be surfaced"); + assert_eq!(fixture.delete_attempts().await, 0); + let before = maintenance_test_object_versions(&fixture.backend).await; + let replay = fixture.run("worker-a", 0, fixture.now).await; + if target == 0 && !after_write { + let completed = replay.expect("unpublished claim can safely retry"); + assert_eq!(completed.job.attempt, 1); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(fixture.delete_attempts().await, 1); + fixture.assert_report_views(&completed).await; + } else if target == 2 && after_write { + let running = replay.expect("fully published running claim can be replayed"); + assert_eq!(running.job.attempt, 1); + assert_eq!(running.job.status, TableMetadataMaintenanceJobStatus::Running); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + fixture.assert_report_views(&running).await; + } else { + assert_matches!(replay, Err(TableCatalogStoreError::Conflict(_))); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 0); + } + } + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_terminal_publication_faults_never_repeat_deletion() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for target in 0..3 { + for after_write in [false, true] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let path = fixture.report_paths()[target].clone(); + let attempt = fixture.backend.put_attempt_count(RUSTFS_META_BUCKET, &path).await + 2; + if after_write { + fixture + .backend + .state + .lock() + .await + .fail_after_put_attempts + .entry((RUSTFS_META_BUCKET.to_string(), path)) + .or_default() + .insert(attempt); + } else { + fixture.backend.fail_put_attempt(RUSTFS_META_BUCKET, &path, attempt).await; + } + fixture + .run("worker-a", 0, fixture.now) + .await + .expect_err("terminal publication fault must be surfaced"); + assert_eq!(fixture.delete_attempts().await, 1); + assert!( + !fixture + .backend + .object_exists("analytics", &fixture.old_metadata) + .await + .unwrap() + ); + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!(fixture.run("worker-b", 0, fixture.now).await, Err(TableCatalogStoreError::Conflict(_))); + assert_matches!(fixture.run("worker-a", 1, fixture.now).await, Err(TableCatalogStoreError::Conflict(_))); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + let replay = fixture.run("worker-a", 0, fixture.now).await.unwrap(); + assert_eq!(replay.job.job_id, fixture.queued.job.job_id); + assert_eq!(replay.job.attempt, 1); + if target == 0 && !after_write { + assert_eq!(replay.job.status, TableMetadataMaintenanceJobStatus::Running); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + } else { + assert_eq!(replay.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(replay.job.deleted_metadata_file_count, 1); + } + fixture.assert_report_views(&replay).await; + let repaired = maintenance_test_object_versions(&fixture.backend).await; + assert_eq!(fixture.run("worker-a", 0, fixture.now).await.unwrap(), replay); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, repaired); + assert_eq!(fixture.delete_attempts().await, 1); + } + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_terminal_repair_requires_exact_running_predecessor() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for mismatch in ["lease", "attempt", "worker", "candidate", "audit", "terminal"] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let paths = fixture.report_paths(); + let attempt = fixture.backend.put_attempt_count(RUSTFS_META_BUCKET, &paths[1]).await + 2; + fixture.backend.fail_put_attempt(RUSTFS_META_BUCKET, &paths[1], attempt).await; + fixture + .run("worker-a", 0, fixture.now) + .await + .expect_err("latest terminal write must fail"); + let object = fixture + .backend + .read_object(RUSTFS_META_BUCKET, &paths[2]) + .await + .unwrap() + .unwrap(); + let mut running: TableMetadataMaintenanceReport = serde_json::from_slice(&object.data).unwrap(); + assert_eq!(running.job.status, TableMetadataMaintenanceJobStatus::Running); + match mismatch { + "lease" => running.job.lease_id = "other-lease".to_string(), + "attempt" => running.job.attempt += 1, + "worker" => running.job.worker_id = Some("worker-b".to_string()), + "candidate" => running.cleanup_candidate_locations.clear(), + "audit" => { + running.audit_events.pop(); + } + "terminal" => running.job.status = TableMetadataMaintenanceJobStatus::Failed, + _ => unreachable!(), + } + fixture + .backend + .seed_object(RUSTFS_META_BUCKET, &paths[2], serde_json::to_vec(&running).unwrap()) + .await; + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!( + fixture.run("worker-a", 0, fixture.now).await, + Err(TableCatalogStoreError::Conflict(_)), + "{mode:?}: {mismatch}" + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 1); + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_failed_terminal_publication_can_be_repaired_and_retried() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let latest = fixture.report_paths()[1].clone(); + let attempt = fixture.backend.put_attempt_count(RUSTFS_META_BUCKET, &latest).await + 2; + fixture.backend.fail_put_attempt(RUSTFS_META_BUCKET, &latest, attempt).await; + fixture + .backend + .fail_delete_attempt("analytics", &fixture.old_metadata, 1) + .await; + fixture + .run("worker-a", 0, fixture.now) + .await + .expect_err("delete and terminal publication faults must surface"); + let repaired = fixture.run("worker-a", 0, fixture.now).await.unwrap(); + assert_eq!(repaired.job.status, TableMetadataMaintenanceJobStatus::Failed); + assert_eq!(repaired.job.attempt, 1); + assert_eq!(fixture.delete_attempts().await, 1); + fixture.assert_report_views(&repaired).await; + let due = OffsetDateTime::parse( + repaired.job.next_retry_after.as_deref().unwrap(), + &time::format_description::well_known::Rfc3339, + ) + .unwrap(); + let completed = fixture.run("worker-b", 1, due).await.unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.attempt, 2); + assert_eq!(fixture.delete_attempts().await, 2); + fixture.assert_report_views(&completed).await; + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_serializes_competing_claims_and_same_worker_replays() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for same_worker in [false, true] { + let fixture = Arc::new(ConditionalMaintenanceFixture::new(mode, 2).await); + let second_worker = if same_worker { "worker-a" } else { "worker-b" }; + let named_path = fixture.report_paths()[0].clone(); + let before = maintenance_test_object_versions(&fixture.backend).await; + let puts_before = fixture.backend.put_attempt_count(RUSTFS_META_BUCKET, &named_path).await; + let pause = fixture.backend.pause_next_put(RUSTFS_META_BUCKET, &named_path).await; + let first_fixture = Arc::clone(&fixture); + let first = tokio::spawn(async move { first_fixture.run("worker-a", 0, first_fixture.now).await }); + let (first, second) = tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, async { + pause.wait_started().await; + // The first Running write has not committed: both workers must + // contend while every durable report still describes Queued. + let publication_path = default_table_bucket_publication_lock_path(); + let attempts = fixture + .backend + .write_lock_acquisition_count("analytics", &publication_path) + .await; + let second_fixture = Arc::clone(&fixture); + let second = tokio::spawn(async move { second_fixture.run(second_worker, 0, second_fixture.now).await }); + while fixture + .backend + .write_lock_acquisition_count("analytics", &publication_path) + .await + == attempts + { + tokio::task::yield_now().await; + } + assert_eq!( + fixture + .backend + .write_lock_acquisition_count("analytics", &publication_path) + .await, + attempts + 1, + ); + assert!(!first.is_finished(), "the first claim must remain paused before publication"); + assert!(!second.is_finished(), "the competing claim must wait for publication ownership"); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.backend.put_attempt_count(RUSTFS_META_BUCKET, &named_path).await, puts_before + 1); + assert_eq!(fixture.delete_attempts().await, 0); + pause.release(); + (first.await.unwrap(), second.await.unwrap()) + }) + .await + .expect("concurrent claims must complete without deadlock"); + let winner = if same_worker { + let first = first.unwrap(); + assert_eq!(second.unwrap(), first); + first + } else { + match (first, second) { + (Ok(report), Err(TableCatalogStoreError::Conflict(_))) + | (Err(TableCatalogStoreError::Conflict(_)), Ok(report)) => report, + other => panic!("exactly one competing claim must win: {other:?}"), + } + }; + assert_eq!(winner.job.attempt, 1); + assert_eq!(winner.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(fixture.delete_attempts().await, 1); + fixture.assert_report_views(&winner).await; + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_running_retry_rejects_previous_lease_and_takeover() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let failed = fixture.fail_first_attempt().await; + let due = OffsetDateTime::parse( + failed.job.next_retry_after.as_deref().unwrap(), + &time::format_description::well_known::Rfc3339, + ) + .unwrap(); + let current = fixture.report_paths()[2].clone(); + fixture.backend.fail_after_next_put(RUSTFS_META_BUCKET, ¤t).await; + fixture + .run("worker-b", 1, due) + .await + .expect_err("response after complete Running claim is lost"); + let running = fixture.run("worker-b", 1, due).await.unwrap(); + assert_eq!(running.job.status, TableMetadataMaintenanceJobStatus::Running); + assert_eq!(running.job.attempt, 2); + assert_ne!(running.job.lease_id, failed.job.lease_id); + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!( + fixture + .store + .heartbeat_table_metadata_maintenance_job( + "analytics", + "sales", + "orders", + &failed.job.job_id, + &failed.job.lease_id, + "worker-a", + ) + .await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_matches!(fixture.run("worker-a", 1, due).await, Err(TableCatalogStoreError::Conflict(_))); + assert_matches!(fixture.run("worker-c", 2, due).await, Err(TableCatalogStoreError::Conflict(_))); + assert_matches!( + fixture.run("worker-b", 1, due + Duration::hours(1)).await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 1); + fixture.assert_report_views(&running).await; + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_rejects_invalid_conditions_before_writes() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for invalid in ["empty-worker", "overflow", "current", "latest", "../job", ""] { + let mut fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let mut worker = "worker-a"; + let mut attempt = 0; + match invalid { + "empty-worker" => worker = " \t ", + "overflow" => attempt = u16::MAX, + job_id => fixture.queued.job.job_id = job_id.to_string(), + } + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!( + fixture.run(worker, attempt, fixture.now).await, + Err(TableCatalogStoreError::Invalid(_)), + "{mode:?}: {invalid}" + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 0); + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_requires_every_persisted_report_view() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for missing in 0..3 { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + fixture + .backend + .state + .lock() + .await + .objects + .remove(&(RUSTFS_META_BUCKET.to_string(), fixture.report_paths()[missing].clone())); + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!(fixture.run("worker-a", 0, fixture.now).await, Err(TableCatalogStoreError::Conflict(_))); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 0); + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_rejects_invalid_failed_retry_state() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for invalid in [ + "missing-backoff", + "malformed-backoff", + "zero-attempt", + "invalid-budget", + "exhausted", + ] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let mut failed = fixture.fail_first_attempt().await; + match invalid { + "missing-backoff" => failed.job.next_retry_after = None, + "malformed-backoff" => failed.job.next_retry_after = Some("not-a-timestamp".to_string()), + "zero-attempt" => failed.job.attempt = 0, + "invalid-budget" => failed.job.max_retry_attempts = 11, + "exhausted" => failed.job.attempt = failed.job.max_retry_attempts, + _ => unreachable!(), + } + fixture.seed_report(&failed).await; + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!( + fixture + .run("worker-b", failed.job.attempt, fixture.now + Duration::hours(1)) + .await, + Err(TableCatalogStoreError::Conflict(_)), + "{mode:?}: {invalid}" + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 1); + } + } +} + #[tokio::test] async fn maintenance_worker_claims_queued_job_before_running_it() { let backend = TestCatalogObjectBackend::default(); @@ -8677,7 +11891,10 @@ async fn maintenance_quarantine_rejects_mutating_non_current_job() { .await .expect_err("mutating a non-current quarantine job should fail"); - assert_eq!(error, TableCatalogStoreError::Conflict("maintenance job is not current".to_string())); + assert_matches!( + error, + TableCatalogStoreError::Conflict(message) if message == "maintenance job is not current" + ); } #[tokio::test] diff --git a/rustfs/src/table_catalog/tests/strong_maintenance_planning.rs b/rustfs/src/table_catalog/tests/strong_maintenance_planning.rs new file mode 100644 index 000000000..d0f864017 --- /dev/null +++ b/rustfs/src/table_catalog/tests/strong_maintenance_planning.rs @@ -0,0 +1,363 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; + +fn planning_report_paths(entry: &TableEntry, job_id: &str) -> [String; 3] { + let paths = TableCatalogObjectPaths::default(); + let namespace = Namespace::parse(&entry.namespace).unwrap(); + let table = IdentifierSegment::parse(&entry.table).unwrap(); + [ + paths.table_maintenance_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id, job_id), + paths.table_maintenance_current_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + paths.table_maintenance_latest_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + ] +} + +async fn assert_only_planning_sidecars_changed( + backend: &TestCatalogObjectBackend, + before: BTreeMap<(String, String), TableCatalogObject>, + entry: &TableEntry, + report: &TableMetadataMaintenanceReport, +) { + let permitted = planning_report_paths(entry, &report.job.job_id) + .map(|path| (RUSTFS_META_BUCKET.to_string(), path)) + .into_iter() + .collect::>(); + let after = maintenance_test_object_versions(backend).await; + for key in &permitted { + let persisted = after.get(key).expect("all three report views must be published"); + assert_eq!( + serde_json::from_slice::(&persisted.data).unwrap(), + *report + ); + } + let unchanged_before = before + .into_iter() + .filter(|(key, _)| !permitted.contains(key)) + .collect::>(); + let unchanged_after = after + .into_iter() + .filter(|(key, _)| !permitted.contains(key)) + .collect::>(); + assert_eq!( + unchanged_after, unchanged_before, + "planning must not change catalog, metadata, manifests or data" + ); +} + +#[tokio::test] +async fn durable_strong_planning_uses_authoritative_entry_and_persists_only_audited_sidecars() { + for requested_retention in [0, 2] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let namespace = Namespace::parse(&entry.namespace).unwrap(); + let table = IdentifierSegment::parse(&entry.table).unwrap(); + let orphan = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let recent = default_table_metadata_file_path(&namespace, &table, "00002.metadata.json"); + for path in [&orphan, &recent] { + backend + .seed_object(&entry.table_bucket, path, br#"{"metadata-log":[]}"#.to_vec()) + .await; + } + let config = TableMaintenanceConfig { + retain_recent_metadata_files: 3, + delete_enabled: true, + max_retry_attempts: 2, + ..Default::default() + }; + store + .put_table_maintenance_config(&entry.table_bucket, &entry.namespace, &entry.table, config.clone()) + .await + .unwrap(); + assert!( + ObjectTableCatalogStore::new(backend.clone()) + .load_table(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap() + .is_none() + ); + let before = maintenance_test_object_versions(&backend).await; + let report = store + .run_table_metadata_maintenance_planning( + &entry.table_bucket, + &entry.namespace, + &entry.table, + requested_retention, + None, + None, + ) + .await + .unwrap(); + assert_eq!(report.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(report.job.operation, TableMetadataMaintenanceOperation::DryRun); + assert_eq!(report.job.table_id, entry.table_id); + assert_eq!(report.current_metadata_location, entry.metadata_location); + assert_eq!(report.job.current_generation, entry.generation); + assert_eq!(report.job.config_source, TableMaintenanceConfigSource::TableOverride); + assert_eq!(report.job.retain_recent_metadata_files, if requested_retention == 0 { 3 } else { 2 }); + assert_eq!(report.job.max_retry_attempts, 2); + assert_eq!(report.job.worker_id.as_deref(), Some("rustfs-admin")); + assert_eq!(report.job.attempt, 1); + assert!(report.job.started_at.is_some() && report.job.finished_at.is_some()); + assert_eq!(report.job.deleted_metadata_file_count, 0); + assert_eq!(report.job.deleted_object_count, 0); + assert_eq!( + report.cleanup_candidate_locations, + if requested_retention == 0 { vec![] } else { vec![orphan] } + ); + for action in [ + TableMaintenanceAuditAction::Planned, + TableMaintenanceAuditAction::WorkerStarted, + TableMaintenanceAuditAction::WorkerSucceeded, + ] { + assert!(report.audit_events.iter().any(|event| event.action == action)); + } + assert_only_planning_sidecars_changed(&backend, before, &entry, &report).await; + let reloaded = StrongTableCatalogStore::new(backend.clone()); + assert_eq!( + reloaded + .load_table(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap() + .unwrap(), + entry + ); + assert_eq!( + reloaded + .get_table_maintenance_config(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap(), + config + ); + assert_eq!( + reloaded + .get_table_metadata_maintenance_report(&entry.table_bucket, &entry.namespace, &entry.table, &report.job.job_id) + .await + .unwrap() + .unwrap(), + report + ); + } +} + +#[tokio::test] +async fn durable_strong_planning_rejects_pending_work_quarantine_and_incomplete_report_views() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let previous = store + .run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None) + .await + .unwrap(); + let paths = planning_report_paths(&entry, &previous.job.job_id); + for (status, quarantined) in [ + (TableMetadataMaintenanceJobStatus::Queued, 0), + (TableMetadataMaintenanceJobStatus::Running, 0), + (TableMetadataMaintenanceJobStatus::Failed, 0), + (TableMetadataMaintenanceJobStatus::Successful, 1), + ] { + let mut pending = previous.clone(); + pending.job.status = status; + pending.job.quarantined_object_count = quarantined; + pending.job.quarantine_enabled = quarantined > 0; + pending.job.quarantine_retention_seconds = if quarantined > 0 { 86_400 } else { 0 }; + for path in &paths { + backend + .seed_object(RUSTFS_META_BUCKET, path, serde_json::to_vec(&pending).unwrap()) + .await; + } + let before = maintenance_test_object_versions(&backend).await; + assert_matches!( + store + .run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None) + .await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before); + } + for mismatched_view in [0, 2] { + for path in &paths { + backend + .seed_object(RUSTFS_META_BUCKET, path, serde_json::to_vec(&previous).unwrap()) + .await; + } + let mut inconsistent = previous.clone(); + inconsistent.job.attempt += 1; + backend + .seed_object(RUSTFS_META_BUCKET, &paths[mismatched_view], serde_json::to_vec(&inconsistent).unwrap()) + .await; + let before = maintenance_test_object_versions(&backend).await; + assert_matches!( + store + .run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None) + .await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before); + } +} + +#[tokio::test] +async fn durable_strong_planning_revalidates_config_entry_and_job_after_unlocked_reads() { + for change in ["config", "entry", "job", "completed-job"] { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let pause = backend.pause_next_read("analytics", &entry.metadata_location).await; + let planning_store = Arc::clone(&store); + let planning = tokio::spawn(async move { + planning_store + .run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("planning must reach metadata read"); + match change { + "config" => { + let mut config = store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + config.max_retry_attempts += 1; + tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.put_table_maintenance_config("analytics", "sales", "orders", config), + ) + .await + .expect("planning must release config publication lock") + .unwrap(); + } + "entry" => { + let next = default_table_metadata_file_path( + &Namespace::parse("sales").unwrap(), + &IdentifierSegment::parse("orders").unwrap(), + "00004.metadata.json", + ); + backend.seed_object("analytics", &next, b"{}".to_vec()).await; + tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.commit_table(TableCommitRequest { + table_bucket: "analytics".to_string(), + namespace: "sales".to_string(), + table: "orders".to_string(), + commit_id: "concurrent-planning-commit".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: entry.version_token.clone(), + expected_metadata_location: entry.metadata_location.clone(), + new_metadata_location: next, + requirements: Vec::new(), + writer: Some("planning-test".to_string()), + }), + ) + .await + .expect("planning must release catalog publication lock") + .unwrap(); + } + "job" => { + let winning = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.run_table_maintenance_scheduler_once( + "analytics", + "sales", + "orders", + "concurrent-scheduler".to_string(), + ), + ) + .await + .expect("planning must release job publication lock") + .unwrap(); + assert_eq!(winning.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + } + "completed-job" => { + let winner = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None), + ) + .await + .expect("a second planner must complete while the first reads metadata") + .unwrap(); + assert_eq!(winner.job.status, TableMetadataMaintenanceJobStatus::Successful); + } + _ => unreachable!(), + } + let before_publish = maintenance_test_object_versions(&backend).await; + pause.release(); + let result = tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, planning) + .await + .unwrap() + .unwrap(); + if change == "completed-job" { + assert_matches!(result, Err(TableCatalogStoreError::Conflict(reason)) + if reason == "current maintenance job changed during planning"); + } else { + assert_matches!(result, Err(TableCatalogStoreError::Conflict(_))); + } + assert_eq!( + maintenance_test_object_versions(&backend).await, + before_publish, + "stale {change} planning must publish nothing" + ); + } +} + +#[tokio::test] +async fn durable_strong_planning_partial_terminal_publication_preserves_safe_retry_boundary() { + // A named-only completed plan is harmless history. A published latest alias + // without its current alias is observable incomplete publication and conflicts. + for failed_view in [1, 2] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let paths = planning_report_paths(&entry, "unused"); + let before = maintenance_test_object_versions(&backend).await; + backend.fail_next_put(RUSTFS_META_BUCKET, &paths[failed_view]).await; + let result = store + .run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None) + .await; + assert!(result.is_err(), "an interrupted report publication must not return success"); + let interrupted = maintenance_test_object_versions(&backend).await; + for (key, value) in &before { + assert_eq!(interrupted.get(key), Some(value)); + } + let (named_key, named_value) = interrupted + .iter() + .find(|(key, _)| !before.contains_key(*key) && key.1 != paths[1] && key.1 != paths[2]) + .expect("the completed named report must be retained"); + let original: TableMetadataMaintenanceReport = serde_json::from_slice(&named_value.data).unwrap(); + assert_eq!(original.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(original.job.operation, TableMetadataMaintenanceOperation::DryRun); + assert_eq!(original.job.deleted_object_count, 0); + assert_eq!(original.job.deleted_metadata_file_count, 0); + let retry = store + .run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None) + .await; + if failed_view == 2 { + assert_eq!(interrupted.len(), before.len() + 1); + let retried = retry.expect("named-only completed dry-run history permits safe replanning"); + assert_ne!(retried.job.job_id, original.job.job_id); + assert_eq!(retried.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(maintenance_test_object_versions(&backend).await.get(named_key), Some(named_value)); + assert_only_planning_sidecars_changed(&backend, interrupted.clone(), &entry, &retried).await; + } else { + assert_eq!(interrupted.len(), before.len() + 2); + assert_matches!(retry, Err(TableCatalogStoreError::Conflict(_))); + assert_eq!(maintenance_test_object_versions(&backend).await, interrupted); + } + assert_eq!(store.load_table("analytics", "sales", "orders").await.unwrap().unwrap(), entry); + } +} diff --git a/rustfs/src/table_catalog/tests/strong_maintenance_planning_paths.rs b/rustfs/src/table_catalog/tests/strong_maintenance_planning_paths.rs new file mode 100644 index 000000000..5116c495b --- /dev/null +++ b/rustfs/src/table_catalog/tests/strong_maintenance_planning_paths.rs @@ -0,0 +1,166 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; + +#[tokio::test] +async fn durable_strong_planning_renamed_paths_preserve_identity_and_fail_closed() { + for (target_namespace, delete_content, foreign_manifest) in [ + ("sales", None, false), + ("archive", None, false), + ("archive", Some(1), false), + ("archive", Some(2), false), + ("archive", None, true), + ] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = entry.table_bucket.as_str(); + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let metadata_dir = default_table_metadata_dir_path(&namespace, &table); + let data_dir = default_table_data_dir_path(&namespace, &table); + let manifest_list = format!("{metadata_dir}/snap-20.avro"); + let manifest = if foreign_manifest { + default_table_metadata_file_path(&namespace, &IdentifierSegment::parse("foreign").unwrap(), "manifest.avro") + } else { + format!("{metadata_dir}/manifest-20.avro") + }; + let left = format!("{data_dir}/left.parquet"); + let right = format!("{data_dir}/right.parquet"); + let delete = format!("{}/delete.parquet", default_table_delete_dir_path(&namespace, &table)); + let mut files = vec![(left.as_str(), 0), (right.as_str(), 0)]; + if let Some(content) = delete_content { + files.push((delete.as_str(), content)); + backend.seed_object(bucket, &delete, b"delete fixture".to_vec()).await; + } + let manifest_bytes = manifest_avro_bytes(&files); + backend + .seed_object(bucket, &manifest_list, manifest_list_avro_bytes(&[(&manifest, manifest_bytes.len())])) + .await; + backend.seed_object(bucket, &manifest, manifest_bytes).await; + backend.seed_object(bucket, &left, parquet_i32_bytes(&[1])).await; + backend.seed_object(bucket, &right, parquet_i32_bytes(&[2])).await; + backend + .seed_object( + bucket, + &entry.metadata_location, + serde_json::to_vec(&serde_json::json!({ + "format-version": 2, + "table-uuid": entry.table_uuid, + "location": entry.warehouse_location, + "metadata-log": [], + "snapshots": [ + {"snapshot-id": 10, "timestamp-ms": 1000, "manifest-list": manifest_list}, + {"snapshot-id": 20, "timestamp-ms": 2000, "manifest-list": manifest_list} + ], + "current-snapshot-id": 20, + "refs": {"main": {"snapshot-id": 20, "type": "branch"}} + })) + .unwrap(), + ) + .await; + if target_namespace != "sales" { + store + .create_namespace(test_namespace_entry(bucket, &Namespace::parse(target_namespace).unwrap())) + .await + .unwrap(); + } + store + .rename_table(bucket, "sales", "orders", target_namespace, "renamed") + .await + .unwrap(); + let renamed = store.load_table(bucket, target_namespace, "renamed").await.unwrap().unwrap(); + let before = maintenance_test_object_versions(&backend).await; + let report = store + .run_table_metadata_maintenance_planning( + bucket, + target_namespace, + "renamed", + 0, + Some(TableSnapshotExpirationConfig { + min_snapshots_to_keep: 1, + max_snapshot_age_ms: 1000, + }), + Some(TableCompactionPlanningConfig { + target_file_size_bytes: 1024 * 1024, + small_file_threshold_bytes: 64 * 1024, + min_input_files: 2, + max_rewrite_bytes_per_job: 2 * 1024 * 1024, + }), + ) + .await + .unwrap(); + assert_eq!(report.job.operation, TableMetadataMaintenanceOperation::DryRun); + assert_eq!(report.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(report.job.deleted_metadata_file_count, 0); + assert_eq!(report.job.deleted_object_count, 0); + let expiration = report.snapshot_expiration.as_ref().unwrap(); + assert_eq!(expiration.namespace, target_namespace); + assert_eq!(expiration.table, "renamed"); + assert_eq!(expiration.table_id, entry.table_id); + assert_eq!(expiration.current_metadata_location, entry.metadata_location); + assert_eq!(expiration.expiration_candidate_count, 1); + assert!(expiration.committed_metadata_location.is_none()); + let compaction = report.compaction.as_ref().unwrap(); + assert_eq!(compaction.namespace, target_namespace); + assert_eq!(compaction.table, "renamed"); + assert_eq!(compaction.table_id, entry.table_id); + assert_eq!(compaction.current_metadata_location, entry.metadata_location); + assert_eq!(compaction.current_snapshot_id, Some(20)); + assert!(compaction.committed_metadata_location.is_none()); + if delete_content.is_some() || foreign_manifest { + assert_eq!(compaction.status, TableCompactionPlanningStatus::ManualReviewRequired); + assert!(compaction.rewrite_groups.is_empty()); + assert!(compaction.manual_review_count > 0); + if delete_content.is_some() { + assert_eq!(compaction.row_level_planning.delete_file_count, 1); + } + } else { + assert_eq!(compaction.status, TableCompactionPlanningStatus::RewriteCandidates); + assert_eq!(compaction.candidate_file_count, 2); + assert_eq!(compaction.rewrite_group_count, 1); + assert_eq!(compaction.rewrite_groups[0].input_file_locations, vec![left, right]); + assert!(compaction.rewrite_groups[0].output_file_location.is_none()); + } + let after = maintenance_test_object_versions(&backend).await; + for (key, value) in &before { + assert_eq!(after.get(key), Some(value), "planning changed an existing object: {key:?}"); + } + assert_eq!(after.len(), before.len() + 3, "only named/current/latest reports may be added"); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + assert_eq!( + reloaded + .load_table(bucket, target_namespace, "renamed") + .await + .unwrap() + .unwrap(), + renamed + ); + for job in [ + report.job.job_id.as_str(), + MAINTENANCE_JOB_ALIAS_CURRENT, + MAINTENANCE_JOB_ALIAS_LATEST, + ] { + assert_eq!( + reloaded + .get_table_metadata_maintenance_report(bucket, target_namespace, "renamed", job) + .await + .unwrap() + .unwrap(), + report + ); + } + } +} diff --git a/rustfs/tests/connect_license.rs b/rustfs/tests/connect_license.rs index d5ad32bed..bbe55e583 100644 --- a/rustfs/tests/connect_license.rs +++ b/rustfs/tests/connect_license.rs @@ -126,8 +126,8 @@ fn tamper_foreign_scope_and_expiry_are_structured_and_preserve_installed_state() let payload = URL_SAFE_NO_PAD .decode_to_vec(envelope["payload"].as_str().expect("payload").as_bytes()) .expect("decode payload"); - let mut claims: Value = serde_json::from_slice(&payload).expect("decode claims"); - claims["deployment"] = Value::String(OTHER_DEPLOYMENT.to_owned()); + let mut claims: LicenseClaims = serde_json::from_slice(&payload).expect("decode claims"); + claims.deployment = OTHER_DEPLOYMENT.to_owned(); envelope["payload"] = Value::String(URL_SAFE_NO_PAD.encode_to_string(serde_json::to_vec(&claims).unwrap())); let tampered = write_artifact(temporary.path(), "tampered.json", &serde_json::to_string(&envelope).unwrap()); let error = apply_license_artifact(&tampered, &state, &valid).expect_err("tampered artifact must fail"); diff --git a/rustfs/tests/connect_offline_bundle.rs b/rustfs/tests/connect_offline_bundle.rs index 9146e25b1..88cb394bf 100644 --- a/rustfs/tests/connect_offline_bundle.rs +++ b/rustfs/tests/connect_offline_bundle.rs @@ -155,7 +155,7 @@ fn connect_offline_bundle_is_deterministic_bounded_and_signed_over_exact_manifes assert_eq!(archive.len(), PATHS.len()); for (index, expected_path) in PATHS.into_iter().enumerate() { let file = archive.by_index(index).expect("archive member"); - assert_eq!(file.name(), expected_path); + assert_eq!(file.name().expect("archive entry name"), expected_path); assert_eq!(file.compression(), CompressionMethod::Stored); assert_eq!(file.last_modified(), Some(zip::DateTime::DEFAULT)); assert_eq!(file.unix_mode().map(|mode| mode & 0o777), Some(0o600)); diff --git a/rustfs/tests/connect_perf_drive.rs b/rustfs/tests/connect_perf_drive.rs index 43a3ba919..a3c88ec81 100644 --- a/rustfs/tests/connect_perf_drive.rs +++ b/rustfs/tests/connect_perf_drive.rs @@ -389,9 +389,15 @@ async fn successful_result_has_signed_bounded_private_offline_export() { let mut archive = zip::ZipArchive::new(Cursor::new(&export.archive_bytes)).expect("three-file archive"); assert_eq!(archive.len(), 3); - assert_eq!(archive.by_index(0).expect("envelope").name(), "envelope.json"); - assert_eq!(archive.by_index(1).expect("signature").name(), "envelope.sig"); - assert_eq!(archive.by_index(2).expect("result").name(), "result.json"); + assert_eq!( + archive.by_index(0).expect("envelope").name().expect("archive entry name"), + "envelope.json" + ); + assert_eq!( + archive.by_index(1).expect("signature").name().expect("archive entry name"), + "envelope.sig" + ); + assert_eq!(archive.by_index(2).expect("result").name().expect("archive entry name"), "result.json"); let envelope: serde_json::Value = serde_json::from_slice(&export.envelope_json).expect("envelope JSON"); assert_eq!(envelope["toolId"], "performance.drive"); @@ -535,7 +541,14 @@ fn production_cli_measures_and_signs_exact_binary_provenance() { let archive_bytes = fs::read(&output).expect("saved archive"); let mut archive = zip::ZipArchive::new(Cursor::new(archive_bytes.as_slice())).expect("signed archive"); let names = (0..archive.len()) - .map(|index| archive.by_index(index).expect("archive member").name().to_owned()) + .map(|index| { + archive + .by_index(index) + .expect("archive member") + .name() + .expect("archive entry name") + .into_owned() + }) .collect::>(); assert_eq!(names, ["envelope.json", "envelope.sig", "result.json"]); let envelope_bytes = read_archive_member(&mut archive, "envelope.json"); diff --git a/rustfs/tests/connect_perf_network.rs b/rustfs/tests/connect_perf_network.rs index 4136a9ca8..6ada066b8 100644 --- a/rustfs/tests/connect_perf_network.rs +++ b/rustfs/tests/connect_perf_network.rs @@ -669,9 +669,26 @@ async fn successful_result_has_signed_bounded_private_offline_export() { assert_eq!(export.archive_sha256, hex(&Sha256::digest(&export.archive_bytes))); let mut archive = zip::ZipArchive::new(Cursor::new(&export.archive_bytes)).expect("three-file archive"); assert_eq!(archive.len(), 3); - assert_eq!(archive.by_index(0).expect("envelope entry").name(), "envelope.json"); - assert_eq!(archive.by_index(1).expect("signature entry").name(), "envelope.sig"); - assert_eq!(archive.by_index(2).expect("result entry").name(), "result.json"); + assert_eq!( + archive + .by_index(0) + .expect("envelope entry") + .name() + .expect("archive entry name"), + "envelope.json" + ); + assert_eq!( + archive + .by_index(1) + .expect("signature entry") + .name() + .expect("archive entry name"), + "envelope.sig" + ); + assert_eq!( + archive.by_index(2).expect("result entry").name().expect("archive entry name"), + "result.json" + ); let envelope: serde_json::Value = serde_json::from_slice(&export.envelope_json).expect("envelope JSON"); assert_eq!(envelope["toolId"], "performance.network"); diff --git a/rustfs/tests/connect_perf_object.rs b/rustfs/tests/connect_perf_object.rs index 62dd99d18..292d74634 100644 --- a/rustfs/tests/connect_perf_object.rs +++ b/rustfs/tests/connect_perf_object.rs @@ -19,20 +19,22 @@ use std::os::unix::fs::PermissionsExt as _; use std::path::Path; use std::process::Command; use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; use std::time::{Duration, SystemTime, UNIX_EPOCH}; use base64_simd::URL_SAFE_NO_PAD; use p256::ecdsa::{Signature, VerifyingKey, signature::Verifier as _}; use p256::pkcs8::DecodePublicKey as _; use rustfs::connect::{ - DeviceIdentity, LocalObjectConsent, OBJECT_CAPABILITY, ObjectOperation, ObjectOutcome, ObjectPerformanceError, - ObjectPerformanceRequest, ObjectProbe, ObjectProbeError, ObjectProbeFuture, ObjectProbeMeasurement, ObjectProvenance, - ObjectReasonCode, ObjectTargetReasonCode, S3ObjectProbe, measure_object, save_signed_object_export, sign_object_export, - validate_object_limits, + DeviceIdentity, LocalObjectConsent, OBJECT_CAPABILITY, ObjectMeasurement, ObjectOperation, ObjectOutcome, + ObjectPerformanceError, ObjectPerformanceRequest, ObjectProbe, ObjectProbeError, ObjectProbeFuture, ObjectProbeMeasurement, + ObjectProvenance, ObjectReasonCode, ObjectTargetReasonCode, S3ObjectProbe, measure_object, save_signed_object_export, + sign_object_export, validate_object_limits, }; use rustfs::embedded::{RustFSServerBuilder, find_available_port}; use sha2::{Digest as _, Sha256}; use tokio_util::sync::CancellationToken; +use tracing::instrument::WithSubscriber as _; use zeroize::Zeroizing; mod common; @@ -500,9 +502,9 @@ async fn cleanup_uses_a_budget_independent_of_the_measurement_window() { "target result: {:?}", measurement.target ); - // The cleanup budget is two sequential requests, each allowed to use the - // probe's bounded cleanup timeout. Leave scheduler and CI runner headroom - // here while keeping the assertion independent from the one-second + // Both sequential cleanup requests share one bounded cleanup deadline. + // Leave scheduler and CI runner headroom while keeping this assertion + // independent from the one-second // measurement window above. tokio::time::timeout(Duration::from_secs(10), server) .await @@ -510,6 +512,251 @@ async fn cleanup_uses_a_budget_independent_of_the_measurement_window() { .expect("response server"); } +#[derive(Clone, Default)] +struct CleanupLog(Arc>>); + +impl std::io::Write for CleanupLog { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + self.0.lock().expect("cleanup log").extend_from_slice(bytes); + Ok(bytes.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + +#[derive(Clone, Copy)] +enum CleanupReply { + Status(u16), + Disconnect, + Delay(Duration), +} + +struct CleanupObservation { + measurement: ObjectMeasurement, + log: String, + deletes: Vec, + elapsed: Duration, +} + +async fn observe_cleanup(replies: [CleanupReply; 2]) -> CleanupObservation { + use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; + + const PRIVATE_BODY: &str = "private-cleanup-response-body"; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("cleanup listener"); + let endpoint = format!("http://{}", listener.local_addr().expect("listener address")); + let deletes = Arc::new(Mutex::new(Vec::new())); + let seen_deletes = Arc::clone(&deletes); + let server = tokio::spawn(async move { + for index in 0..4 { + let (mut socket, _) = listener.accept().await.expect("probe connection"); + let mut bytes = Vec::new(); + let (header_end, content_length) = loop { + let mut buffer = [0_u8; 4096]; + let count = socket.read(&mut buffer).await.expect("request headers"); + assert!(count > 0, "complete request headers"); + bytes.extend_from_slice(&buffer[..count]); + assert!(bytes.len() <= 64 * 1024, "bounded request"); + if let Some(offset) = bytes.windows(4).position(|window| window == b"\r\n\r\n") { + let headers = std::str::from_utf8(&bytes[..offset]).expect("ASCII headers"); + let content_length = headers + .lines() + .filter_map(|line| line.split_once(':')) + .find(|(name, _)| name.eq_ignore_ascii_case("content-length")) + .map_or(0, |(_, value)| value.trim().parse::().expect("content length")); + break (offset + 4, content_length); + } + }; + assert!(content_length <= 4, "bounded probe body"); + while bytes.len() < header_end + content_length { + let mut buffer = [0_u8; 4]; + let count = socket.read(&mut buffer).await.expect("request body"); + assert!(count > 0, "complete request body"); + bytes.extend_from_slice(&buffer[..count]); + } + let request_line = std::str::from_utf8(&bytes[..header_end]) + .expect("ASCII request") + .lines() + .next() + .expect("request line") + .to_owned(); + let reply = if index < 2 { + assert!(request_line.starts_with("PUT ")); + CleanupReply::Status(200) + } else { + assert!(request_line.starts_with("DELETE ")); + seen_deletes.lock().expect("seen deletes").push(request_line); + replies[index - 2] + }; + let status = match reply { + CleanupReply::Status(status) => status, + CleanupReply::Disconnect => return, + CleanupReply::Delay(delay) => { + tokio::time::sleep(delay).await; + 204 + } + }; + let body = if status >= 400 { PRIVATE_BODY } else { "" }; + let response = format!( + "HTTP/1.1 {status} Fixture\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + socket.write_all(response.as_bytes()).await.expect("fixture response"); + } + }); + let probe = S3ObjectProbe::new( + &endpoint, + None, + None, + Zeroizing::new("private-cleanup-access".to_owned()), + Zeroizing::new("private-cleanup-secret".to_owned()), + Zeroizing::new("private-cleanup-token".to_owned()), + Duration::from_secs(1), + ) + .expect("cleanup probe"); + let mut request = request(ObjectOperation::PutObject); + request.traffic_bytes = 4; + let log = CleanupLog::default(); + let writer = log.clone(); + let subscriber = tracing_subscriber::fmt() + .with_max_level(tracing::Level::ERROR) + .with_ansi(false) + .without_time() + .with_target(false) + .with_writer(move || writer.clone()) + .finish(); + let started = std::time::Instant::now(); + let measurement = tokio::time::timeout( + Duration::from_secs(8), + measure_object(&request, &probe, &CancellationToken::new()).with_subscriber(subscriber), + ) + .await; + let elapsed = started.elapsed(); + // Reap the fixture before asserting, including stalled or early-failure cases. + server.abort(); + if let Err(error) = server.await { + assert!(error.is_cancelled(), "cleanup fixture failed: {error}"); + } + let measurement = measurement.expect("bounded cleanup probe").expect("typed cleanup result"); + let log = String::from_utf8(log.0.lock().expect("cleanup log").clone()).expect("UTF-8 cleanup log"); + for sensitive in [ + endpoint.as_str(), + "127.0.0.1", + "private-cleanup-access", + "private-cleanup-secret", + "private-cleanup-token", + PRIVATE_BODY, + "rustfs-connect-perf-", + "synthetic-object", + "Authorization", + "X-Amz-Security-Token", + ] { + assert!(!log.contains(sensitive), "cleanup diagnostic exposed private context"); + } + let deletes = deletes.lock().expect("seen deletes").clone(); + CleanupObservation { + measurement, + log, + deletes, + elapsed, + } +} + +fn assert_cleanup_failure(observed: &CleanupObservation, stage: &str, reason: &str, status: Option) { + assert_eq!(observed.measurement.result.outcome(), ObjectOutcome::Failed); + assert_eq!(observed.measurement.target.reason_code, ObjectTargetReasonCode::CleanupFailed); + let events: Vec<_> = observed + .log + .lines() + .filter(|line| line.contains("event=\"connect_object_cleanup_failed\"")) + .collect(); + assert_eq!(events.len(), 1, "one cleanup diagnostic: {}", observed.log); + let event = events[0]; + for field in [ + "component=\"connect\"".to_owned(), + "subsystem=\"performance_object\"".to_owned(), + "result=\"failed\"".to_owned(), + format!("stage=\"{stage}\""), + format!("reason={reason}"), + ] { + assert!(event.contains(&field), "missing {field}: {event}"); + } + match status { + Some(status) => assert!(event.contains(&format!("http_status={status}")), "{event}"), + None => assert!(!event.contains("http_status="), "transport failure has no HTTP status: {event}"), + } + for field in ["elapsed_ms=", "remaining_ms="] { + let value = event + .split_whitespace() + .find_map(|part| part.strip_prefix(field)) + .expect("bounded timing field") + .parse::() + .expect("integer milliseconds"); + if field == "remaining_ms=" { + assert!(value <= 5_000, "cleanup deadline must not be extended"); + } + } +} + +#[tokio::test] +async fn cleanup_diagnostics_identify_each_failed_http_stage_without_private_context() { + let _guard = TEST_LOCK.lock().await; + for (replies, stage, status) in [ + ([CleanupReply::Status(503), CleanupReply::Status(204)], "object_delete", 503), + ([CleanupReply::Status(204), CleanupReply::Status(403)], "bucket_delete", 403), + ([CleanupReply::Status(204), CleanupReply::Status(404)], "bucket_delete", 404), + ] { + let observed = observe_cleanup(replies).await; + assert_cleanup_failure(&observed, stage, "CleanupFailed", Some(status)); + assert_eq!(observed.deletes.len(), if stage == "object_delete" { 1 } else { 2 }); + } +} + +#[tokio::test] +async fn cleanup_diagnostics_identify_each_disconnected_transport_stage() { + let _guard = TEST_LOCK.lock().await; + for (replies, stage) in [ + ([CleanupReply::Disconnect, CleanupReply::Status(204)], "object_delete"), + ([CleanupReply::Status(204), CleanupReply::Disconnect], "bucket_delete"), + ] { + let observed = observe_cleanup(replies).await; + assert_cleanup_failure(&observed, stage, "ProtocolFailure", None); + } +} + +#[tokio::test] +async fn cleanup_accepts_missing_object_and_still_deletes_the_bucket() { + let _guard = TEST_LOCK.lock().await; + let observed = observe_cleanup([CleanupReply::Status(404), CleanupReply::Status(204)]).await; + assert_eq!(observed.measurement.result.outcome(), ObjectOutcome::Succeeded); + assert_eq!(observed.measurement.target.reason_code, ObjectTargetReasonCode::Complete); + assert_eq!(observed.deletes.len(), 2); + assert!(observed.deletes[0].ends_with("/synthetic-object HTTP/1.1")); + assert!(observed.deletes[1].ends_with("/ HTTP/1.1")); + assert!(!observed.log.contains("connect_object_cleanup_failed")); +} + +#[tokio::test] +async fn cleanup_diagnostics_identify_timeouts_without_resetting_the_shared_deadline() { + let _guard = TEST_LOCK.lock().await; + for (replies, stage) in [ + ([CleanupReply::Delay(Duration::from_secs(6)), CleanupReply::Status(204)], "object_delete"), + ( + [ + CleanupReply::Delay(Duration::from_secs(3)), + CleanupReply::Delay(Duration::from_secs(3)), + ], + "bucket_delete", + ), + ] { + let observed = observe_cleanup(replies).await; + assert_cleanup_failure(&observed, stage, "TimedOut", None); + assert!(observed.elapsed >= Duration::from_secs(5)); + assert!(observed.log.contains("remaining_ms=0"), "{}", observed.log); + } +} + #[tokio::test] async fn signed_result_is_saved_without_overwrite() { let _guard = TEST_LOCK.lock().await; @@ -610,6 +857,14 @@ async fn real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put_bod request.artifact_uid = "019e3ae0-0000-7000-8000-000000000016".to_owned(); } let measurement = measure_object(&request, &probe, &CancellationToken::new()) + .with_subscriber( + tracing_subscriber::fmt() + .with_max_level(tracing::Level::ERROR) + .with_ansi(false) + .without_time() + .with_writer(std::io::stderr) + .finish(), + ) .await .expect("real object measurement"); assert_eq!( diff --git a/rustfs/tests/connect_top_net.rs b/rustfs/tests/connect_top_net.rs index 294cd3642..0da443a39 100644 --- a/rustfs/tests/connect_top_net.rs +++ b/rustfs/tests/connect_top_net.rs @@ -399,7 +399,14 @@ fn production_cli_exports_top_net_and_fails_closed_for_unavailable_and_invalid_r assert_eq!(fs::metadata(&net_output).expect("output metadata").permissions().mode() & 0o777, 0o600); let mut archive = ZipArchive::new(Cursor::new(bytes)).expect("top.net archive"); let names = (0..archive.len()) - .map(|index| archive.by_index(index).expect("archive member").name().to_owned()) + .map(|index| { + archive + .by_index(index) + .expect("archive member") + .name() + .expect("archive entry name") + .into_owned() + }) .collect::>(); assert_eq!(names, ["envelope.json", "envelope.sig", "result.json"]); diff --git a/scripts/README.md b/scripts/README.md index 1ea493f2c..78b21a70e 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -108,6 +108,9 @@ their issue closes. | `run_object_batch_bench.sh` | dev-tool | Batch object benchmark runner (warp/s3bench) | internode + scanner runbooks | | `run_object_batch_bench_enhanced.sh` | dev-tool | Enhanced batch benchmark runner; hub used by the smoke rigs | hotpath runbook | | `run_pinned_paired_abba_bench.sh` | dev-tool | Pinned RustFS/MinIO paired ABBA benchmark orchestrator for backlog#1432 | `test_pinned_paired_abba_bench.sh` | +| `prepare_conditional_read_baseline.py` | dev-tool | Installs only the matching conditional-read benchmark harness in a clean detached baseline | `docs/testing/conditional-read-benchmark.md` | +| `run_conditional_read_abba.py` | dev-tool | Serial ordinary/hit/miss GET ABBA with cache/slow-disk controls and drift/regression gates | `docs/testing/conditional-read-benchmark.md` | +| `test_conditional_read_abba.py` | dev-tool | Parser, drift rejection, and throughput/lock regression tests for conditional-read evidence | `run_conditional_read_abba.py` | | `run_get_codec_streaming_smoke.sh` | dev-tool | Local GET benchmark harness for the codec streaming read path | `docs/testing/ecstore-validation-suite-design.md` | | `run_get_1mib_abba_stage_metrics.sh` | dev-tool | Exact-1MiB isolated-host GET ABBA/stage-metrics harness for backlog#1434 | `test_get_1mib_abba_stage_metrics.sh` | | `issue_2007_coalescer_prometheus_report.py` | dev-tool | Read-only Prometheus report for GET metadata coalescer delay cost validation; usage in the module docstring | `test_issue_2007_coalescer_prometheus_report.sh` | diff --git a/scripts/check_architecture_migration_rules.sh b/scripts/check_architecture_migration_rules.sh index 8f3983269..f0b9fad84 100755 --- a/scripts/check_architecture_migration_rules.sh +++ b/scripts/check_architecture_migration_rules.sh @@ -59,12 +59,14 @@ require_source_contains "docs/architecture/overview.md" "## Phase Order" "archit require_source_contains "docs/architecture/runtime-lifecycle.md" "## Startup And Readiness" "runtime lifecycle startup readiness section" require_source_contains "docs/architecture/runtime-lifecycle.md" "## Shutdown Lifecycle Boundary" "runtime lifecycle shutdown boundary section" require_source_contains "docs/architecture/runtime-lifecycle.md" "## AppContext Foundation" "runtime lifecycle AppContext section" +require_source_contains "docs/architecture/runtime-lifecycle.md" "## OIDC Runtime Publication" "runtime lifecycle OIDC publication section" require_source_contains "docs/architecture/storage-control-data-plane.md" "## Storage API Contracts" "storage control/data plane contracts section" require_source_contains "docs/architecture/storage-control-data-plane.md" "## Cluster Control Plane" "storage control/data plane cluster section" require_source_contains "docs/architecture/storage-control-data-plane.md" "## Background Controllers" "storage control/data plane background controllers section" require_source_contains "docs/architecture/crate-boundaries.md" "## Dependency Direction" "crate boundaries dependency direction section" require_source_contains "docs/architecture/crate-boundaries.md" "storage-api -> ecstore" "crate boundaries storage dependency rule" require_source_contains "docs/architecture/crate-boundaries.md" "extension-schema -> rustfs" "crate boundaries extension dependency rule" +require_source_contains "docs/architecture/crate-boundaries.md" "## OIDC Ownership" "crate boundaries OIDC ownership section" require_source_contains "docs/architecture/readiness-matrix.md" "## Request Behavior Matrix" "readiness matrix request behavior section" require_source_contains "docs/architecture/readiness-matrix.md" "## Runtime Dependency Matrix" "readiness matrix runtime dependency section" require_source_contains "docs/architecture/readiness-matrix.md" "## Probe Semantics" "readiness matrix probe semantics section" @@ -5544,6 +5546,10 @@ else done <"$ECSTORE_LINT_EXPECTED" fi +if ! bash "${ROOT_DIR}/scripts/check_oidc_architecture_boundaries.sh"; then + report_failure "OIDC architecture boundary check" +fi + if (( FAILURES > 0 )); then exit 1 fi diff --git a/scripts/check_oidc_architecture_boundaries.sh b/scripts/check_oidc_architecture_boundaries.sh new file mode 100755 index 000000000..b251492a7 --- /dev/null +++ b/scripts/check_oidc_architecture_boundaries.sh @@ -0,0 +1,365 @@ +#!/usr/bin/env bash + +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +exec python3 - "$ROOT_DIR" "$@" <<'PY' +import json +import re +import sys +from pathlib import Path + + +root = Path(sys.argv[1]) +arguments = sys.argv[2:] +if arguments not in ([], ["--self-test"]): + raise SystemExit("usage: check_oidc_architecture_boundaries.sh [--self-test]") + +fixture_file = root / "scripts/fixtures/architecture_migration_rules/oidc/cases.json" +raw_string = re.compile(r'r(#+)?"') +char_literal = re.compile(r"'(?:\\.|[^'\\\n])'") +required_modules = ( + "crates/iam/src/oidc/mod.rs", + "crates/iam/src/oidc/config.rs", + "crates/iam/src/oidc/provider.rs", + "crates/iam/src/oidc/runtime.rs", + "crates/iam/src/oidc/state.rs", + "crates/iam/src/oidc/transport.rs", +) +removed_modules = ( + "crates/iam/src/oidc.rs", + "crates/iam/src/oidc_state.rs", + "crates/iam/src/federation/registry.rs", + "crates/iam/src/federation/oidc", +) +admin_config_service = "rustfs/src/admin/service/oidc_config.rs" +site_replication_handler = "rustfs/src/admin/handlers/site_replication.rs" +admin_oidc_handler = "rustfs/src/admin/handlers/oidc.rs" +admin_idp_compat_handler = "rustfs/src/admin/handlers/idp_compat.rs" + + +def mask(source, start, end): + return "".join("\n" if character == "\n" else " " for character in source[start:end]) + + +def code_only(source): + """Preserve offsets while ignoring Rust comments, strings, and test items.""" + output = list(source) + index = 0 + while index < len(source): + start = index + if source.startswith("//", index): + end = source.find("\n", index) + index = len(source) if end < 0 else end + elif source.startswith("/*", index): + depth = 1 + index += 2 + while index < len(source) and depth: + if source.startswith("/*", index): + depth += 1 + index += 2 + elif source.startswith("*/", index): + depth -= 1 + index += 2 + else: + index += 1 + else: + raw = raw_string.match(source, index) + if raw: + delimiter = '"' + (raw.group(1) or "") + end = source.find(delimiter, index + len(raw.group())) + index = len(source) if end < 0 else end + len(delimiter) + elif source[index] == '"': + index += 1 + while index < len(source): + if source[index] == "\\": + index += 2 + elif source[index] == '"': + index += 1 + break + else: + index += 1 + elif source[index] == "'" and (literal := char_literal.match(source, index)): + index = literal.end() + else: + index += 1 + continue + output[start:index] = mask(source, start, index) + + production = "".join(output) + test_item = re.compile( + r"#\s*\[\s*cfg\s*\(\s*test\s*\)\s*\]\s*(?:#\s*\[[^]]*\]\s*)*" + r"(?:pub(?:\([^)]*\))?\s+)?(?:async\s+)?(?:mod|fn|impl|use|struct|enum|trait|const|static)\b" + ) + search_from = 0 + while match := test_item.search(production, search_from): + delimiter = re.search(r"[;{]", production[match.end():]) + if not delimiter: + break + end = match.end() + delimiter.end() + if delimiter.group() == "{": + depth = 1 + while end < len(production) and depth: + if production[end] == "{": + depth += 1 + elif production[end] == "}": + depth -= 1 + end += 1 + production = production[: match.start()] + mask(production, match.start(), end) + production[end:] + search_from = end + return production + + +def find_matches(rule, path, source, pattern): + return [ + (rule, path, source.count("\n", 0, match.start()) + 1) + for match in re.finditer(pattern, source, re.MULTILINE | re.DOTALL) + ] + + +def is_struct_pattern(source, opening_brace): + depth = 1 + index = opening_brace + 1 + while index < len(source) and depth: + if source[index] == "{": + depth += 1 + elif source[index] == "}": + depth -= 1 + index += 1 + if depth: + return False + suffix = source[index:].lstrip() + typed_let_pattern = suffix.startswith(":") and re.search( + r"\blet\s+(?:\w+\s*::\s*)*FederatedAuthorization\s*$", source[:opening_brace] + ) + return bool(typed_let_pattern) or suffix.startswith("=>") or (suffix.startswith("=") and not suffix.startswith("==")) + + +def function_body(source, name): + signature = re.search(rf"\bfn\s+{name}\s*\([^)]*\)[^{{]*\{{", source) + if not signature: + return "" + depth = 1 + end = signature.end() + while end < len(source) and depth: + if source[end] == "{": + depth += 1 + elif source[end] == "}": + depth -= 1 + end += 1 + return source[signature.end(): end - 1] + + +def workflow_event_paths(source, event): + section = re.search(rf"^ {event}:\s*\n(.*?)(?=^ [a-z_]+:|\Z)", source, re.MULTILINE | re.DOTALL) + if not section: + return set() + paths = set() + in_paths = False + for line in section.group(1).splitlines(): + if re.fullmatch(r" paths:\s*", line): + in_paths = True + continue + if not in_paths: + continue + if not line.strip() or line.lstrip().startswith("#"): + continue + entry = re.fullmatch(r" -\s+(?:[\"']([^\"']+)[\"']|([^\s#]+))\s*(?:#.*)?", line) + if not entry: + break + paths.add(entry.group(1) or entry.group(2)) + return paths + + +def source_findings(path, source): + findings = [] + if path in removed_modules or path.startswith("crates/iam/src/federation/oidc/"): + return [("oidc-module-owner", path, 1)] + if path == ".github/workflows/oidc-keycloak.yml": + expected_paths = {"crates/iam/src/oidc/**", admin_config_service} + if any(not expected_paths.issubset(workflow_event_paths(source, event)) for event in ("pull_request", "push")): + findings.append(("oidc-live-workflow", path, 1)) + return findings + if not path.endswith(".rs") or path.endswith("_tests.rs") or path.endswith("test_support.rs") or "/tests/" in path: + return findings + if not (path.startswith("crates/iam/src/") or path.startswith("rustfs/src/")): + return findings + + code = code_only(source) + findings += find_matches( + "oidc-legacy-runtime", path, code, + r"\b(?:FederatedIdentityRegistry|FederatedIdentityProvider|FederatedCodeExchange|OIDC_SYS|get_oidc|init_oidc_sys|init_oidc_sys_with_extra_root_ca_provider)\b", + ) + + if path.startswith("crates/iam/src/federation/"): + findings += find_matches( + "federation-oidc-detail", path, code, + r"\b(?:crate|super)::oidc\b|\buse\b[^;]*\boidc\b[^;]*;|\boidc::|\b(?:OidcSys|OidcClaims|OidcStateStore|OidcProviderConfig|OidcConfigSnapshot|OidcExtraRootCaProvider|ProviderRuntime|ReqwestHttpClient|openidconnect|reqwest)\b", + ) + + if path.startswith("crates/iam/src/oidc/"): + findings += find_matches( + "oidc-iam-session-boundary", path, code, + r"\b(?:Credentials|FederatedSessionTransaction)\b", + ) + + if path.startswith("crates/iam/src/") and path != "crates/iam/src/lib.rs" and not path.startswith( + ("crates/iam/src/oidc/", "crates/iam/src/federation/") + ): + findings += find_matches( + "oidc-runtime-owner", path, code, + r"\b(?:OidcSys|StandardOidcAdapter|build_oidc_sys_with_extra_root_ca_provider)\b", + ) + + if path == "crates/iam/src/oidc/config.rs": + findings += find_matches( + "oidc-config-direction", path, code, + r"\b(?:provider|runtime|state|transport)::|\b(?:ProviderRuntime|OidcSys|OidcStateStore|ReqwestHttpClient|CoreClient)\b", + ) + query = re.search(r"pub\s+trait\s+OidcConfigQuery\b[^\{]*\{([^}]*)\}", code, re.DOTALL) + narrow_query = r"\s*fn\s+site_replication_snapshot\s*\(\s*&self\s*\)\s*->\s*OidcSiteReplicationSnapshot\s*;\s*" + if not query or not re.fullmatch(narrow_query, query.group(1)): + findings.append(("oidc-query-shape", path, 1)) + provider = re.search(r"pub\s+struct\s+OidcSiteReplicationProvider\s*\{([^}]*)\}", code, re.DOTALL) + provider_shape = ( + r"\s*pub\s+provider_id\s*:\s*String\s*," + r"\s*pub\s+claim_name\s*:\s*String\s*," + r"\s*pub\s+role_policy\s*:\s*String\s*," + r"\s*pub\s+client_id\s*:\s*String\s*," + r"\s*pub\s+hashed_client_secret\s*:\s*String\s*,?\s*" + ) + if not provider or not re.fullmatch(provider_shape, provider.group(1)): + findings.append(("oidc-site-replication-shape", path, 1)) + snapshot = re.search(r"pub\s+struct\s+OidcSiteReplicationSnapshot\s*\{([^}]*)\}", code, re.DOTALL) + if not snapshot or not re.fullmatch(r"\s*providers\s*:\s*Vec\s*<\s*OidcSiteReplicationProvider\s*>\s*,?\s*", snapshot.group(1)): + findings.append(("oidc-site-replication-shape", path, 1)) + + if path.startswith("rustfs/src/") and path != "rustfs/src/startup_auth.rs": + findings += find_matches( + "oidc-runtime-owner", path, code, + r"\b(?:OidcSys|StandardOidcAdapter|build_oidc_sys_with_extra_root_ca_provider)\b", + ) + + if path == admin_oidc_handler: + findings += find_matches( + "oidc-admin-config-owner", path, code, + r"\b(?:read_admin_config_without_migrate|read_admin_server_config_snapshot|save_admin_server_config_snapshot|current_object_store_handle_for_context|ServerConfig|KVS|build_upsert_provider_config|upsert_persisted_provider_config|delete_persisted_provider_config|persisted_provider_secret|validate_oidc_provider_config_with_extra_root_ca|load_oidc_config_snapshot|current_oidc_extra_root_ca_material)\b", + ) + if path == admin_idp_compat_handler: + findings += find_matches( + "oidc-admin-config-owner", path, code, + r"\b(?:read_admin_config_without_migrate|read_admin_server_config_snapshot|save_admin_server_config_snapshot|current_object_store_handle_for_context|build_upsert_provider_config|upsert_persisted_provider_config|delete_persisted_provider_config|persisted_provider_secret|validate_oidc_provider_config_with_extra_root_ca|current_oidc_extra_root_ca_material)\b", + ) + if path == admin_config_service: + findings += find_matches("oidc-admin-service-layer", path, code, r"\b(?:s3s|S3Error|S3Result|s3_error)\b") + for operation in ("list_config", "upsert_config", "delete_config", "validate_config"): + if not re.search(rf"pub\(crate\)\s+async\s+fn\s+{operation}\s*\(", code): + findings.append(("oidc-admin-config-owner", path, 1)) + for storage_operation in ("read_admin_server_config_snapshot", "save_admin_server_config_snapshot"): + if not re.search(rf"\b{storage_operation}\s*\(", code): + findings.append(("oidc-admin-config-owner", path, 1)) + if path == site_replication_handler: + findings += find_matches( + "oidc-site-replication-consumer", path, code, + r"\b(?:OidcConfigSnapshot|OidcProviderConfig|load_oidc_config_snapshot)\b", + ) + oidc_mapping = function_body(code, "open_id_settings_from_snapshot") + if not oidc_mapping or re.search(r"\b(?:hash_client_secret|Sha256)\b|\.\s*(?:config_snapshot|client_secret)\b", oidc_mapping): + findings.append(("oidc-site-replication-consumer", path, 1)) + if not re.search(r"\.\s*site_replication_snapshot\s*\(", code): + findings.append(("oidc-site-replication-consumer", path, 1)) + + for match in re.finditer(r"\bFederatedAuthorization\s*\{", code): + preceding_word = re.search(r"\b(\w+)\s*$", code[: match.start()]) + if preceding_word and preceding_word.group(1) in ("struct", "impl"): + continue + if is_struct_pattern(code, match.end() - 1): + continue + if path != "crates/iam/src/federation/mapper.rs": + findings.append(("federation-authorization-owner", path, code.count("\n", 0, match.start()) + 1)) + + if path == "rustfs/src/app/context/interfaces.rs": + paired_snapshot = ( + r"pub\s+type\s+FederatedIdentityRuntimeSnapshot\s*=\s*\(\s*" + r"Arc\s*<\s*FederatedIdentityService\s*>\s*,\s*" + r"Arc\s*<\s*dyn\s+OidcConfigQuery\s*>\s*\)\s*;" + ) + if not re.search(paired_snapshot, code): + findings.append(("oidc-runtime-pair", path, 1)) + if path == "rustfs/src/app/context/handles.rs": + runtime = re.search(r"struct\s+FederatedIdentityRuntime\s*\{([^}]*)\}", code, re.DOTALL) + if not runtime or not re.search(r"\bservice\s*:\s*Arc\s*<\s*FederatedIdentityService\s*>", runtime.group(1)) or not re.search( + r"\boidc_config_query\s*:\s*Arc\s*<\s*dyn\s+OidcConfigQuery\s*>", runtime.group(1) + ): + findings.append(("oidc-runtime-pair", path, 1)) + return findings + + +def layout_findings(exists=None, service_modules=None): + if exists is None: + exists = lambda path: (root / path).exists() + if service_modules is None: + service_modules = (root / "rustfs/src/admin/service/mod.rs").read_text() + findings = [] + for path in required_modules: + if not exists(path): + findings.append(("oidc-module-owner", path, 1)) + for path in removed_modules: + if exists(path): + findings.append(("oidc-module-owner", path, 1)) + if not exists(admin_config_service): + findings.append(("oidc-admin-config-owner", admin_config_service, 1)) + if not re.search(r"\bmod\s+oidc_config\s*;", service_modules): + findings.append(("oidc-admin-config-owner", "rustfs/src/admin/service/mod.rs", 1)) + return findings + + +def fixture_findings(): + cases = json.loads(fixture_file.read_text()) + actual = [] + expected = [] + for case in cases: + case_id = case["id"] + expected.extend(f"{rule}|{case_id}" for rule in case["expected"]) + if "layout" in case: + layout = case["layout"] + present = set(required_modules) | {admin_config_service} + present.difference_update(layout.get("missing", [])) + present.update(layout.get("restored", [])) + findings = layout_findings(present.__contains__, layout.get("service_modules", "mod oidc_config;")) + else: + findings = source_findings(case["path"], case["source"]) + actual.extend(f"{rule}|{case_id}" for rule, _, _ in findings) + return sorted(set(actual)), sorted(set(expected)) + + +def repository_findings(): + findings = layout_findings() + for base in ("crates/iam/src", "rustfs/src"): + for file in sorted((root / base).rglob("*.rs")): + path = file.relative_to(root).as_posix() + findings.extend(source_findings(path, file.read_text())) + workflow = ".github/workflows/oidc-keycloak.yml" + findings.extend(source_findings(workflow, (root / workflow).read_text())) + return findings + + +actual, expected = fixture_findings() +if actual != expected: + print("OIDC architecture guard fixture mismatch:", file=sys.stderr) + print(f" expected: {expected}", file=sys.stderr) + print(f" actual: {actual}", file=sys.stderr) + raise SystemExit(1) + +if arguments == ["--self-test"]: + print("OIDC architecture guard fixtures passed.") + raise SystemExit(0) + +findings = repository_findings() +for rule, path, line in findings: + print(f"OIDC architecture boundary failed: {rule}: {path}:{line}", file=sys.stderr) +if findings: + raise SystemExit(1) +print("OIDC architecture boundaries passed.") +PY diff --git a/scripts/check_test_wiring.py b/scripts/check_test_wiring.py index bd9f7013f..3a8251775 100755 --- a/scripts/check_test_wiring.py +++ b/scripts/check_test_wiring.py @@ -684,12 +684,12 @@ def check_ilm_build_budget(root: Path) -> list[str]: return [".github/workflows/ci.yml: missing workflow"] workflow = workflow_path.read_text() job = re.search( - r"^ test-ilm-integration-serial:\n(?P.*?)(?=^ [a-z0-9-]+:\n)", + r"^ test-and-lint:\n(?P.*?)(?=^ [a-z0-9-]+:\n)", workflow, re.MULTILINE | re.DOTALL, ) if job is None: - return [".github/workflows/ci.yml: missing test-ilm-integration-serial job"] + return [".github/workflows/ci.yml: ILM integration must share the workspace test job"] step = re.search( r"^ - name: Run ignored ILM integration tests serially\n(?P.*?)(?=^ - name:|\Z)", @@ -705,6 +705,16 @@ def check_ilm_build_budget(root: Path) -> list[str]: ) if budget not in step.group("body"): return [".github/workflows/ci.yml: ILM integration must keep the measured 3/2 Cargo build budget"] + for required in ( + " if: needs.classify-changes.outputs.mode == 'full'", + "cargo nextest run --profile default -j1 --run-ignored ignored-only", + "--all --exclude e2e_test", + "-E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'", + 'status=${PIPESTATUS[0]}', + 'exit "${status}"', + ): + if required not in step.group("body"): + return [f".github/workflows/ci.yml: ILM integration is missing {required.strip()}"] return [] @@ -5566,12 +5576,17 @@ class SelfTests(unittest.TestCase): workflow.parent.mkdir(parents=True) valid = ( "jobs:\n" - " test-ilm-integration-serial:\n" + " test-and-lint:\n" " steps:\n" " - name: Run ignored ILM integration tests serially\n" + " if: needs.classify-changes.outputs.mode == 'full'\n" " env:\n" " CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}\n" - " run: cargo nextest run\n" + " run: |\n" + " cargo nextest run --profile default -j1 --run-ignored ignored-only --all --exclude e2e_test \\\n" + " -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'\n" + " status=${PIPESTATUS[0]}\n" + ' exit "${status}"\n' " next-job:\n" " steps: []\n" ) @@ -5580,6 +5595,18 @@ class SelfTests(unittest.TestCase): workflow.write_text(valid.replace(" CARGO_BUILD_JOBS:", " BUILD_JOBS:")) self.assertEqual(len(check_ilm_build_budget(root)), 1) + for old, new in ( + ("test-and-lint:", "test-ilm-integration-serial:"), + ("outputs.mode == 'full'", "outputs.mode == 'docs'"), + ("--profile default", "--profile ci"), + ("-j1", "-j2"), + ("ignored-only", "default"), + ("--all --exclude e2e_test", "-p rustfs-scanner -p rustfs"), + ("binary(lifecycle_integration_test)", "none()"), + ('exit "${status}"', "exit 0"), + ): + workflow.write_text(valid.replace(old, new)) + self.assertTrue(check_ilm_build_budget(root), (old, new)) def test_vault_tests_require_cross_process_serialization(self) -> None: with tempfile.TemporaryDirectory() as tmp: diff --git a/scripts/ci_gate.py b/scripts/ci_gate.py index 3979991f7..b99c10927 100644 --- a/scripts/ci_gate.py +++ b/scripts/ci_gate.py @@ -16,11 +16,14 @@ import unittest ROOT = Path(__file__).resolve().parent.parent ALWAYS_JOBS = ("classify-changes", "typos", "quick-checks") CODE_JOBS = ( - "test-and-lint", "offline-enrollment-e2e", "test-ilm-integration-serial", "test-and-lint-rio-v2", + "test-and-lint", "offline-enrollment-e2e", "test-and-lint-rio-v2", "connect-short-credential-boundary", "test-and-lint-protocols", "build-rustfs-debug-binary", "uring-integration", "e2e-tests", "s3-implemented-tests", "s3-lifecycle-behavior-tests", ) +# Full alternate-feature tests remain mandatory on main, merge groups, and +# scheduled/manual runs, without repeating four builds on every PR revision. +EXTENDED_FEATURE_JOBS = ("test-and-lint-rio-v2", "test-and-lint-protocols") # These jobs still exercise the server and its black-box harness when only E2E # Rust sources change. Production code, manifests and shared test configuration # always select the full matrix. No production package depends on e2e_test. @@ -127,6 +130,8 @@ def expected_results(mode: str, event: str, ref: str) -> dict[str, str]: raise ValueError(f"invalid CI selection: {mode!r} for {event!r}") expected = {job: "success" for job in ALWAYS_JOBS} expected.update({job: "success" if mode == "full" or (mode == "e2e" and job in E2E_JOBS) else "skipped" for job in CODE_JOBS}) + if event == "pull_request": + expected.update({job: "skipped" for job in EXTENDED_FEATURE_JOBS}) rio = mode == "full" and event in ("schedule", "workflow_dispatch") expected.update({job: "success" if rio else "skipped" for job in OPTIONAL_JOBS[:2]}) full = mode == "full" and (event in ("merge_group", "workflow_dispatch") or (event == "push" and ref == "refs/heads/main")) @@ -215,6 +220,14 @@ def check_workflow(root: Path) -> list[str]: errors.append("Test and Lint must uniquely name the aggregate gate") if scalar(gate, "if", 4) != "always() && (github.event_name != 'pull_request' || github.event.action != 'closed')": errors.append("required-checks must run after failed or skipped dependencies") + for job in EXTENDED_FEATURE_JOBS: + block = yaml_block(jobs, job, 2) or [] + if scalar(block, "if", 4) != "needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'": + errors.append(f"{job} must retain full validation on every non-PR event") + workspace = "\n".join(yaml_block(jobs, "test-and-lint", 2) or []) + clippy = re.search(r"^ - name: Run clippy lints\n(?P.*?)(?=^ - name:|\Z)", workspace, re.MULTILINE | re.DOTALL) + if clippy is None or scalar(clippy["body"].splitlines(), "if", 8) != "github.event_name != 'pull_request'": + errors.append("workspace Clippy must run on every non-PR event") if scalar(gate, "shell", 8) != "bash" or scalar(gate, "run", 8) != "python3 scripts/ci_gate.py verify" or scalar(gate, "CI_NEEDS", 10) != "${{ toJSON(needs) }}": errors.append("required-checks must verify the actual needs results") if any(re.match(r'''\s+(?:- )?(?:["']?continue-on-error["']?\s*:|["']?if["']?\s*:)''', line) and not line.startswith(" if:") for line in gate): @@ -315,13 +328,14 @@ class SelfTests(unittest.TestCase): def test_event_contract_requires_complete_candidate_and_optional_lanes(self): ordinary = expected_results("full", "pull_request", "refs/pull/1/merge") - self.assertEqual({job for job, state in ordinary.items() if state == "skipped"}, set(OPTIONAL_JOBS)) + self.assertEqual({job for job, state in ordinary.items() if state == "skipped"}, set(OPTIONAL_JOBS + EXTENDED_FEATURE_JOBS)) docs = expected_results("docs", "pull_request", "refs/pull/1/merge") self.assertEqual({job for job, state in docs.items() if state == "success"}, set(ALWAYS_JOBS)) e2e = expected_results("e2e", "pull_request", "refs/pull/1/merge") self.assertEqual({job for job, state in e2e.items() if state == "success"}, set(ALWAYS_JOBS + E2E_JOBS)) for event in ("schedule", "workflow_dispatch", "merge_group", "push"): result = expected_results("full", event, "refs/heads/main") + self.assertTrue(all(result[job] == "success" for job in CODE_JOBS)) self.assertEqual(result["e2e-full"], "skipped" if event == "schedule" else "success") self.assertEqual(result["e2e-tests-rio-v2"], "success" if event in ("schedule", "workflow_dispatch") else "skipped") with self.assertRaises(ValueError): @@ -330,7 +344,7 @@ class SelfTests(unittest.TestCase): expected_results("e2e", event, "refs/heads/main") def test_every_wrong_result_missing_job_or_selection_fails_closed(self): - for mode, event in (("full", "pull_request"), ("docs", "pull_request"), ("e2e", "pull_request"), ("full", "schedule"), ("full", "workflow_dispatch"), ("full", "merge_group")): + for mode, event in (("full", "pull_request"), ("docs", "pull_request"), ("e2e", "pull_request"), ("full", "push"), ("full", "schedule"), ("full", "workflow_dispatch"), ("full", "merge_group")): good = {job: {"result": value} for job, value in expected_results(mode, event, "refs/heads/main").items()} good["classify-changes"]["outputs"] = {"mode": mode} self.assertEqual(verify_results(good, event, "refs/heads/main"), []) @@ -377,6 +391,15 @@ class SelfTests(unittest.TestCase): (root / ".github/workflows").mkdir(parents=True) source = (ROOT / ".github/workflows/ci.yml").read_text() path = root / ".github/workflows/ci.yml" + for condition in ("false", "github.event_name == 'schedule'", "github.event_name == 'pull_request'"): + path.write_text(source.replace(" - name: Run clippy lints\n if: github.event_name != 'pull_request'", f" - name: Run clippy lints\n if: {condition}")) + self.assertIn("workspace Clippy must run on every non-PR event", check_workflow(root)) + for job in EXTENDED_FEATURE_JOBS: + before, block = source.split(f" {job}:\n", 1) + for condition in ("false", "github.event_name == 'schedule'", "github.event_name != 'pull_request'"): + changed = re.sub(r"^ if: .*", f" if: {condition}", block, count=1, flags=re.MULTILINE) + path.write_text(before + f" {job}:\n" + changed) + self.assertIn(f"{job} must retain full validation on every non-PR event", check_workflow(root)) for job in ALWAYS_JOBS + CODE_JOBS + OPTIONAL_JOBS: before, gate = source.split(" required-checks:\n", 1) path.write_text(before + " required-checks:\n" + gate.replace(f" - {job}\n", "", 1)) @@ -446,6 +469,39 @@ class SelfTests(unittest.TestCase): self.assertEqual(result.returncode, code, result.stderr) self.assertIn("ERROR:" if code else "CI contract passed", result.stderr if code else result.stdout) + def test_merged_ilm_step_preserves_exit_status_and_core_report(self): + from check_test_wiring import yaml_block + from test_security_workflow import named_steps, shell_body + + workflow = (ROOT / ".github/workflows/ci.yml").read_text().splitlines() + step = named_steps(yaml_block(workflow, "test-and-lint", 2))["Run ignored ILM integration tests serially"] + body = shell_body(step) + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + commands = root / "bin" + commands.mkdir() + # Exercise the actual pipeline without building the workspace. + fake_timeout = commands / "timeout" + fake_timeout.write_text('#!/bin/sh\nshift 4\nexec "$@"\n') + fake_timeout.chmod(0o755) + cargo = commands / "cargo" + cargo.write_text('#!/bin/sh\nprintf "%s\\n" "$@" > "$ILM_ARGS"\nexit "$ILM_STATUS"\n') + cargo.chmod(0o755) + report = root / "target/nextest/ci/junit.xml" + report.parent.mkdir(parents=True) + report.write_text("existing workspace report") + args = root / "args.txt" + for status in (0, 23, 124): + env = dict(os.environ, PATH=f"{commands}{os.pathsep}{os.environ['PATH']}", + ILM_ARGS=str(args), ILM_STATUS=str(status)) + result = subprocess.run(["bash", "-e", "-o", "pipefail", "-c", body], cwd=root, env=env, capture_output=True, text=True) + self.assertEqual(result.returncode, status, result.stderr) + self.assertEqual(report.read_text(), "existing workspace report") + arguments = args.read_text().splitlines() + self.assertEqual(arguments[:9], ["nextest", "run", "--profile", "default", "-j1", "--run-ignored", "ignored-only", "--all", "--exclude"]) + self.assertEqual(arguments[9], "e2e_test") + self.assertIn(f"exit_status={status}", (root / "artifacts/ilm-integration/diagnostics.txt").read_text()) + def test_actual_selector_bootstrap_uses_base_policy_and_fails_closed(self): from check_test_wiring import yaml_block jobs = yaml_block((ROOT / ".github/workflows/ci.yml").read_text().splitlines(), "jobs", 0) diff --git a/scripts/error-other-format-baseline.txt b/scripts/error-other-format-baseline.txt index 967a87143..2151ae440 100644 --- a/scripts/error-other-format-baseline.txt +++ b/scripts/error-other-format-baseline.txt @@ -59,7 +59,7 @@ 7|crates/ecstore/src/set_disk/core/io_primitives.rs 1|crates/ecstore/src/set_disk/mod.rs 3|crates/ecstore/src/set_disk/ops/bitrot_self_verify.rs -2|crates/ecstore/src/set_disk/ops/heal.rs +1|crates/ecstore/src/set_disk/ops/heal.rs 8|crates/ecstore/src/set_disk/ops/multipart.rs 2|crates/ecstore/src/set_disk/ops/object.rs 3|crates/ecstore/src/set_disk/read.rs @@ -68,4 +68,4 @@ 10|crates/ecstore/src/store/init.rs 3|crates/ecstore/src/store/multipart.rs 6|crates/ecstore/src/store/object.rs -5|crates/ecstore/src/store/rebalance/support.rs +3|crates/ecstore/src/store/rebalance/support.rs diff --git a/scripts/fixtures/architecture_migration_rules/oidc/cases.json b/scripts/fixtures/architecture_migration_rules/oidc/cases.json new file mode 100644 index 000000000..9c9772527 --- /dev/null +++ b/scripts/fixtures/architecture_migration_rules/oidc/cases.json @@ -0,0 +1,381 @@ +[ + { + "id": "legacy_registry", + "path": "rustfs/src/admin/handlers/sts.rs", + "source": "let registry = FederatedIdentityRegistry::new(provider);\n", + "expected": ["oidc-legacy-runtime"] + }, + { + "id": "legacy_global_initializer", + "path": "rustfs/src/startup_auth.rs", + "source": "init_oidc_sys_with_extra_root_ca_provider(provider).await?;\n", + "expected": ["oidc-legacy-runtime"] + }, + { + "id": "federation_imports_oidc_runtime", + "path": "crates/iam/src/federation/transaction.rs", + "source": "use crate::oidc::OidcSys;\n", + "expected": ["federation-oidc-detail"] + }, + { + "id": "federation_imports_oidc_module", + "path": "crates/iam/src/federation/provider.rs", + "source": "use crate::oidc::config::OidcProviderConfig;\n", + "expected": ["federation-oidc-detail"] + }, + { + "id": "federation_aliases_oidc_module", + "path": "crates/iam/src/federation/provider.rs", + "source": "use crate::{oidc as oidc_impl};\n", + "expected": ["federation-oidc-detail"] + }, + { + "id": "federation_aliases_oidc_after_nested_import", + "path": "crates/iam/src/federation/provider.rs", + "source": "use crate::{other::{Foo}, oidc as oidc_impl};\n", + "expected": ["federation-oidc-detail"] + }, + { + "id": "federation_constructs_authorization", + "path": "crates/iam/src/federation/transaction.rs", + "source": "let authorization = FederatedAuthorization { policies, groups };\n", + "expected": ["federation-authorization-owner"] + }, + { + "id": "accepted_federation_destructures_authorization", + "path": "crates/iam/src/federation/transaction.rs", + "source": "let FederatedAuthorization { policies, groups, .. } = &authorization;\n", + "expected": [] + }, + { + "id": "accepted_federation_typed_borrow_pattern", + "path": "crates/iam/src/federation/transaction.rs", + "source": "let FederatedAuthorization { policies, groups, .. }: &FederatedAuthorization = &authorization;\n", + "expected": [] + }, + { + "id": "accepted_federation_qualified_typed_borrow_pattern", + "path": "crates/iam/src/federation/transaction.rs", + "source": "let crate::federation::FederatedAuthorization { policies, groups, .. }: &crate::federation::FederatedAuthorization = &authorization;\n", + "expected": [] + }, + { + "id": "federation_constructs_typed_authorization", + "path": "crates/iam/src/federation/transaction.rs", + "source": "let authorization: FederatedAuthorization = FederatedAuthorization { policies, groups };\n", + "expected": ["federation-authorization-owner"] + }, + { + "id": "accepted_federation_matches_authorization", + "path": "crates/iam/src/federation/transaction.rs", + "source": "match authorization { FederatedAuthorization { policies, .. } => log(policies) }\n", + "expected": [] + }, + { + "id": "federation_constructs_authorization_for_comparison", + "path": "crates/iam/src/federation/transaction.rs", + "source": "let matches = FederatedAuthorization { policies, groups } == expected;\n", + "expected": ["federation-authorization-owner"] + }, + { + "id": "oidc_adapter_imports_credentials", + "path": "crates/iam/src/oidc/runtime.rs", + "source": "use rustfs_credentials::Credentials;\n", + "expected": ["oidc-iam-session-boundary"] + }, + { + "id": "oidc_adapter_constructs_qualified_credentials", + "path": "crates/iam/src/oidc/runtime.rs", + "source": "let credentials = rustfs_credentials::Credentials { access_key };\n", + "expected": ["oidc-iam-session-boundary"] + }, + { + "id": "oidc_adapter_imports_binding_transaction", + "path": "crates/iam/src/oidc/runtime.rs", + "source": "use crate::federation::FederatedSessionTransaction as BindingRequest;\n", + "expected": ["oidc-iam-session-boundary"] + }, + { + "id": "oidc_adapter_constructs_binding_transaction", + "path": "crates/iam/src/oidc/runtime.rs", + "source": "let transaction = FederatedSessionTransaction { authorization, duration_seconds, session_policy };\n", + "expected": ["oidc-iam-session-boundary"] + }, + { + "id": "oidc_adapter_test_module_uses_credentials", + "path": "crates/iam/src/oidc/runtime.rs", + "source": "#[cfg(test)]\nmod tests { use rustfs_credentials::Credentials; }\n", + "expected": [] + }, + { + "id": "accepted_federation_transaction_owner", + "path": "crates/iam/src/federation/transaction.rs", + "source": "let transaction = FederatedSessionTransaction { authorization, duration_seconds, session_policy };\n", + "expected": [] + }, + { + "id": "accepted_binding_credentials_owner", + "path": "rustfs/src/admin/service/federated_identity.rs", + "source": "let credentials = Credentials { access_key };\n", + "expected": [] + }, + { + "id": "config_imports_provider_runtime", + "path": "crates/iam/src/oidc/config.rs", + "source": "pub struct OidcSiteReplicationProvider { pub provider_id: String, pub claim_name: String, pub role_policy: String, pub client_id: String, pub hashed_client_secret: String }\npub struct OidcSiteReplicationSnapshot { providers: Vec }\npub trait OidcConfigQuery: Send + Sync { fn site_replication_snapshot(&self) -> OidcSiteReplicationSnapshot; }\nuse super::provider::ProviderRuntime;\n", + "expected": ["oidc-config-direction"] + }, + { + "id": "broad_oidc_config_query", + "path": "crates/iam/src/oidc/config.rs", + "source": "pub struct OidcSiteReplicationProvider { pub provider_id: String, pub claim_name: String, pub role_policy: String, pub client_id: String, pub hashed_client_secret: String }\npub struct OidcSiteReplicationSnapshot { providers: Vec }\npub trait OidcConfigQuery: Send + Sync { fn config_snapshot(&self) -> OidcConfigSnapshot; }\n", + "expected": ["oidc-query-shape"] + }, + { + "id": "replication_view_contains_raw_secret", + "path": "crates/iam/src/oidc/config.rs", + "source": "pub struct OidcSiteReplicationProvider { pub provider_id: String, pub claim_name: String, pub role_policy: String, pub client_id: String, pub hashed_client_secret: String, pub client_secret: String }\npub struct OidcSiteReplicationSnapshot { providers: Vec }\npub trait OidcConfigQuery: Send + Sync { fn site_replication_snapshot(&self) -> OidcSiteReplicationSnapshot; }\n", + "expected": ["oidc-site-replication-shape"] + }, + { + "id": "replication_view_contains_private_secret", + "path": "crates/iam/src/oidc/config.rs", + "source": "pub struct OidcSiteReplicationProvider { pub provider_id: String, pub claim_name: String, pub role_policy: String, pub client_id: String, pub hashed_client_secret: String, client_secret: String }\npub struct OidcSiteReplicationSnapshot { providers: Vec }\npub trait OidcConfigQuery: Send + Sync { fn site_replication_snapshot(&self) -> OidcSiteReplicationSnapshot; }\n", + "expected": ["oidc-site-replication-shape"] + }, + { + "id": "query_adds_full_config_method", + "path": "crates/iam/src/oidc/config.rs", + "source": "pub struct OidcSiteReplicationProvider { pub provider_id: String, pub claim_name: String, pub role_policy: String, pub client_id: String, pub hashed_client_secret: String }\npub struct OidcSiteReplicationSnapshot { providers: Vec }\npub trait OidcConfigQuery: Send + Sync { fn site_replication_snapshot(&self) -> OidcSiteReplicationSnapshot; fn raw_snapshot(&self) -> OidcConfigSnapshot; }\n", + "expected": ["oidc-query-shape"] + }, + { + "id": "admin_constructs_runtime", + "path": "rustfs/src/admin/handlers/oidc.rs", + "source": "let adapter = StandardOidcAdapter::new(oidc);\n", + "expected": ["oidc-runtime-owner"] + }, + { + "id": "admin_builds_runtime", + "path": "rustfs/src/admin/handlers/oidc.rs", + "source": "let oidc = build_oidc_sys_with_extra_root_ca_provider(ca).await?;\n", + "expected": ["oidc-runtime-owner"] + }, + { + "id": "iam_sys_builds_oidc_runtime", + "path": "crates/iam/src/sys/mod.rs", + "source": "let oidc = OidcSys::empty()?;\n", + "expected": ["oidc-runtime-owner"] + }, + { + "id": "handler_reads_config_storage", + "path": "rustfs/src/admin/handlers/oidc.rs", + "source": "let snapshot = read_admin_server_config_snapshot(store).await?;\n", + "expected": ["oidc-admin-config-owner"] + }, + { + "id": "handler_validates_provider_directly", + "path": "rustfs/src/admin/handlers/oidc.rs", + "source": "let result = validate_oidc_provider_config_with_extra_root_ca(&provider, ca).await?;\n", + "expected": ["oidc-admin-config-owner"] + }, + { + "id": "compat_handler_saves_oidc_config", + "path": "rustfs/src/admin/handlers/idp_compat.rs", + "source": "save_admin_server_config_snapshot(store, &config, &snapshot).await?;\n", + "expected": ["oidc-admin-config-owner"] + }, + { + "id": "service_imports_http_error", + "path": "rustfs/src/admin/service/oidc_config.rs", + "source": "use s3s::S3Error;\npub(crate) async fn list_config() {}\npub(crate) async fn upsert_config() {}\npub(crate) async fn delete_config() {}\npub(crate) async fn validate_config() {}\nfn storage() { read_admin_server_config_snapshot(store); save_admin_server_config_snapshot(store); }\n", + "expected": ["oidc-admin-service-layer"] + }, + { + "id": "service_omits_config_write", + "path": "rustfs/src/admin/service/oidc_config.rs", + "source": "pub(crate) async fn list_config() {}\npub(crate) async fn upsert_config() {}\npub(crate) async fn delete_config() {}\npub(crate) async fn validate_config() {}\nfn storage() { read_admin_server_config_snapshot(store); }\n", + "expected": ["oidc-admin-config-owner"] + }, + { + "id": "replication_reads_full_config", + "path": "rustfs/src/admin/handlers/site_replication.rs", + "source": "let providers = query.config_snapshot().providers();\n", + "expected": ["oidc-site-replication-consumer"] + }, + { + "id": "replication_reads_raw_secret", + "path": "rustfs/src/admin/handlers/site_replication.rs", + "source": "fn open_id_settings_from_snapshot(snapshot: &OidcSiteReplicationSnapshot) -> OpenIDSettings { let secret = provider.client_secret.clone(); }\nlet snapshot = query.site_replication_snapshot();\n", + "expected": ["oidc-site-replication-consumer"] + }, + { + "id": "single_runtime_snapshot", + "path": "rustfs/src/app/context/interfaces.rs", + "source": "pub type FederatedIdentityRuntimeSnapshot = Arc;\n", + "expected": ["oidc-runtime-pair"] + }, + { + "id": "single_runtime_handle", + "path": "rustfs/src/app/context/handles.rs", + "source": "struct FederatedIdentityRuntime { service: Arc }\n", + "expected": ["oidc-runtime-pair"] + }, + { + "id": "old_single_file", + "path": "crates/iam/src/oidc.rs", + "source": "pub struct OidcSys;\n", + "expected": ["oidc-module-owner"] + }, + { + "id": "old_federation_directory", + "path": "crates/iam/src/federation/oidc/flow.rs", + "source": "pub fn exchange_code() {}\n", + "expected": ["oidc-module-owner"] + }, + { + "id": "missing_oidc_owner_module", + "layout": { "missing": ["crates/iam/src/oidc/provider.rs"] }, + "expected": ["oidc-module-owner"] + }, + { + "id": "restored_legacy_module", + "layout": { "restored": ["crates/iam/src/federation/oidc"] }, + "expected": ["oidc-module-owner"] + }, + { + "id": "missing_admin_config_service", + "layout": { "missing": ["rustfs/src/admin/service/oidc_config.rs"] }, + "expected": ["oidc-admin-config-owner"] + }, + { + "id": "missing_admin_service_registration", + "layout": { "service_modules": "mod other;" }, + "expected": ["oidc-admin-config-owner"] + }, + { + "id": "missing_live_workflow_path", + "path": ".github/workflows/oidc-keycloak.yml", + "source": "on:\n pull_request:\n paths:\n - \"crates/iam/src/oidc/**\"\n - \"rustfs/src/admin/service/oidc_config.rs\"\n push:\n paths:\n - \"rustfs/src/admin/service/oidc_config.rs\"\n", + "expected": ["oidc-live-workflow"] + }, + { + "id": "service_path_missing_from_push", + "path": ".github/workflows/oidc-keycloak.yml", + "source": "on:\n pull_request:\n paths:\n - \"crates/iam/src/oidc/**\"\n - \"rustfs/src/admin/service/oidc_config.rs\"\n push:\n paths:\n - \"crates/iam/src/oidc/**\"\n", + "expected": ["oidc-live-workflow"] + }, + { + "id": "commented_workflow_path", + "path": ".github/workflows/oidc-keycloak.yml", + "source": "on:\n pull_request:\n paths:\n - \"crates/iam/src/oidc/**\"\n # - \"rustfs/src/admin/service/oidc_config.rs\"\n push:\n paths:\n - \"crates/iam/src/oidc/**\"\n - \"rustfs/src/admin/service/oidc_config.rs\"\n", + "expected": ["oidc-live-workflow"] + }, + { + "id": "workflow_path_in_paths_ignore", + "path": ".github/workflows/oidc-keycloak.yml", + "source": "on:\n pull_request:\n paths-ignore:\n - \"crates/iam/src/oidc/**\"\n - \"rustfs/src/admin/service/oidc_config.rs\"\n push:\n paths:\n - \"crates/iam/src/oidc/**\"\n - \"rustfs/src/admin/service/oidc_config.rs\"\n", + "expected": ["oidc-live-workflow"] + }, + { + "id": "comment_and_string_literals", + "path": "rustfs/src/admin/handlers/oidc.rs", + "source": "// StandardOidcAdapter and OIDC_SYS are inert here.\n/* FederatedIdentityRegistry /* OidcSys */ */\nlet message = r#\"FederatedCodeExchange StandardOidcAdapter\"#;\n", + "expected": [] + }, + { + "id": "test_module_exclusion_and_recovery", + "path": "rustfs/src/admin/handlers/oidc.rs", + "source": "#[cfg(test)]\nmod tests {\n fn test() { let _ = OidcSys::empty(); }\n}\nfn production() { let _ = OidcSys::empty(); }\n", + "expected": ["oidc-runtime-owner"] + }, + { + "id": "test_module_with_extra_attribute", + "path": "rustfs/src/admin/handlers/oidc.rs", + "source": "#[cfg(test)]\n#[allow(dead_code)]\nmod tests { fn test() { let _ = OidcSys::empty(); } }\n", + "expected": [] + }, + { + "id": "test_function_exclusion_and_recovery", + "path": "rustfs/src/app/context.rs", + "source": "#[cfg(test)]\nfn test_runtime() { let _ = OidcSys::empty(); }\nfn production() { let _ = OidcSys::empty(); }\n", + "expected": ["oidc-runtime-owner"] + }, + { + "id": "test_crate_visible_function", + "path": "rustfs/src/app/context.rs", + "source": "#[cfg(test)]\npub(crate) fn test_runtime() { let _ = OidcSys::empty(); }\n", + "expected": [] + }, + { + "id": "accepted_runtime_owner", + "path": "rustfs/src/startup_auth.rs", + "source": "let adapter = StandardOidcAdapter::new(oidc);\n", + "expected": [] + }, + { + "id": "accepted_iam_builder_owner", + "path": "crates/iam/src/lib.rs", + "source": "let oidc = OidcSys::empty()?;\n", + "expected": [] + }, + { + "id": "accepted_mapper_owner", + "path": "crates/iam/src/federation/mapper.rs", + "source": "return FederatedAuthorization { policies, groups };\n", + "expected": [] + }, + { + "id": "accepted_runtime_snapshot", + "path": "rustfs/src/app/context/interfaces.rs", + "source": "pub type FederatedIdentityRuntimeSnapshot = (Arc, Arc);\n", + "expected": [] + }, + { + "id": "accepted_runtime_handle", + "path": "rustfs/src/app/context/handles.rs", + "source": "struct FederatedIdentityRuntime { service: Arc, oidc_config_query: Arc }\n", + "expected": [] + }, + { + "id": "accepted_replication_query", + "path": "crates/iam/src/oidc/config.rs", + "source": "pub struct OidcSiteReplicationProvider { pub provider_id: String, pub claim_name: String, pub role_policy: String, pub client_id: String, pub hashed_client_secret: String }\npub struct OidcSiteReplicationSnapshot { providers: Vec }\npub trait OidcConfigQuery: Send + Sync { fn site_replication_snapshot(&self) -> OidcSiteReplicationSnapshot; }\n", + "expected": [] + }, + { + "id": "accepted_admin_service", + "path": "rustfs/src/admin/service/oidc_config.rs", + "source": "pub(crate) async fn list_config() {}\npub(crate) async fn upsert_config() {}\npub(crate) async fn delete_config() {}\npub(crate) async fn validate_config() {}\nfn storage() { read_admin_server_config_snapshot(store); save_admin_server_config_snapshot(store); }\n", + "expected": [] + }, + { + "id": "accepted_replication_consumer", + "path": "rustfs/src/admin/handlers/site_replication.rs", + "source": "fn open_id_settings_from_snapshot(snapshot: &OidcSiteReplicationSnapshot) -> OpenIDSettings { let secret_hash = provider.hashed_client_secret.clone(); }\nlet snapshot = query.site_replication_snapshot();\n", + "expected": [] + }, + { + "id": "accepted_unrelated_replication_secret", + "path": "rustfs/src/admin/handlers/site_replication.rs", + "source": "fn open_id_settings_from_snapshot(snapshot: &OidcSiteReplicationSnapshot) -> OpenIDSettings { let secret_hash = provider.hashed_client_secret.clone(); }\nfn ldap_settings() { let secret = ldap_provider.client_secret.clone(); }\nlet snapshot = query.site_replication_snapshot();\n", + "expected": [] + }, + { + "id": "accepted_live_workflow_paths", + "path": ".github/workflows/oidc-keycloak.yml", + "source": "on:\n pull_request:\n paths:\n - \"crates/iam/src/oidc/**\"\n - \"rustfs/src/admin/service/oidc_config.rs\"\n push:\n paths:\n - \"crates/iam/src/oidc/**\"\n - \"rustfs/src/admin/service/oidc_config.rs\"\n", + "expected": [] + }, + { + "id": "accepted_workflow_comment_between_paths", + "path": ".github/workflows/oidc-keycloak.yml", + "source": "on:\n pull_request:\n paths:\n - \"crates/iam/src/oidc/**\"\n # Keep configuration changes in the live gate.\n - \"rustfs/src/admin/service/oidc_config.rs\"\n push:\n paths:\n - \"crates/iam/src/oidc/**\"\n - \"rustfs/src/admin/service/oidc_config.rs\"\n", + "expected": [] + }, + { + "id": "accepted_owner_layout", + "layout": {}, + "expected": [] + } +] diff --git a/scripts/prepare_conditional_read_baseline.py b/scripts/prepare_conditional_read_baseline.py new file mode 100644 index 000000000..ce25889b0 --- /dev/null +++ b/scripts/prepare_conditional_read_baseline.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +"""Install only the paired benchmark harness in a clean detached baseline.""" + +import argparse +from pathlib import Path +import re +import subprocess + + +def replace_once(source, old, new, path): + if source.count(old) != 1: + raise ValueError(f"baseline anchor changed in {path}; inspect it before measuring") + return source.replace(old, new) + + +def prepare(root, baseline, revision): + if baseline.resolve() == root.resolve(): + raise ValueError("baseline must be a separate worktree") + if not re.fullmatch(r"[0-9a-f]{40}", revision): + raise ValueError("supply the full baseline commit SHA") + def git(*args): + return subprocess.check_output(["git", "-C", str(baseline), *args], text=True).strip() + if git("rev-parse", "HEAD") != revision or git("status", "--porcelain"): + raise ValueError("baseline must be clean at the requested commit") + if subprocess.run(["git", "-C", str(baseline), "symbolic-ref", "-q", "HEAD"], + stdout=subprocess.DEVNULL).returncode == 0: + raise ValueError("use a detached baseline worktree") + edits = {} + def edit(path, old, new): + previous = edits.get(path, (baseline / path).read_text()) + edits[path] = replace_once(previous, old, new, path) + edit("rustfs/src/app/object/mod.rs", "mod test_support;", + "mod test_support;\n#[cfg(test)]\nmod conditional_read_bench;") + edit("crates/ecstore/src/set_disk/mod.rs", + 'const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS";', + '#[cfg(feature = "test-util")]\nmod conditional_read_bench;\n' + '#[cfg(feature = "test-util")]\npub use conditional_read_bench::ConditionalReadBenchmarkMetadataGuard;\n\n' + 'const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS";') + edit("crates/ecstore/src/set_disk/read.rs", + "fn get_object_metadata_cache_request_bypass_reason(bucket: &str, opts: &ObjectOptions, read_data: bool) -> Option<&'static str> {", + "fn get_object_metadata_cache_request_bypass_reason(bucket: &str, opts: &ObjectOptions, read_data: bool) -> Option<&'static str> {\n" + ' #[cfg(feature = "test-util")]\n if super::conditional_read_bench::cache_bypass_applies(bucket) {\n' + ' return Some("benchmark");\n }') + edit("crates/ecstore/src/api/mod.rs", + "pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;", + "pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;\n" + " pub use crate::set_disk::ConditionalReadBenchmarkMetadataGuard;") + for path in ("rustfs/src/storage/storage_api.rs", "rustfs/src/app/storage_api.rs"): + # The facades also expose these types elsewhere; target the test-util block. + source = (baseline / path).read_text() + marker = ("pub(crate) use rustfs_ecstore::api::set_disk::test_util::{" if "/storage/" in path else + "pub(crate) use crate::storage::storage_api::ecstore_set_disk::{") + replacement = marker + "\n ConditionalReadBenchmarkMetadataGuard," + edits[path] = replace_once(source, marker, replacement, path) + for path in ("rustfs/src/app/object/conditional_read_bench.rs", + "crates/ecstore/src/set_disk/conditional_read_bench.rs"): + if (baseline / path).exists(): + raise ValueError(f"baseline already contains {path}") + edits[path] = (root / path).read_text() + # Resolve every anchor before writing; never apply a partial adapter. + for path, source in edits.items(): + (baseline / path).write_text(source) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--baseline-worktree", type=Path, required=True) + parser.add_argument("--revision", required=True) + args = parser.parse_args() + prepare(Path(__file__).resolve().parent.parent, args.baseline_worktree, args.revision) + + +if __name__ == "__main__": + main() diff --git a/scripts/release/check_glibc.py b/scripts/release/check_glibc.py new file mode 100644 index 000000000..7bf618d36 --- /dev/null +++ b/scripts/release/check_glibc.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +"""Reject GNU artifacts whose ELF version needs exceed the package ABI floor.""" + +from __future__ import annotations + +import argparse +import os +import re +import subprocess +from pathlib import Path + + +def version(value: str) -> tuple[int, ...]: + if not re.fullmatch(r"[0-9]+(?:\.[0-9]+)+", value): + raise ValueError(f"invalid glibc version: {value}") + parts = tuple(map(int, value.split("."))) + return parts + (0,) * max(0, 3 - len(parts)) + + +def check(binary: Path, baseline: str) -> str: + floor = version(baseline) + result = subprocess.run( + ["readelf", "--version-info", "--wide", str(binary)], + check=True, capture_output=True, text=True, env={**os.environ, "LC_ALL": "C"}, + ) + # Only loader requirements matter. Definitions and arbitrary strings in + # the executable do not describe the libc that it needs to run. + needs = result.stdout.partition("Version needs section")[2] + symbols = re.findall(r"\bName:\s+(GLIBC_\S+)", needs) + if not symbols: + raise ValueError(f"{binary}: no GLIBC version needs found in GNU artifact") + highest = "GLIBC_0.0" + for symbol in symbols: + required = version(symbol[len("GLIBC_"):]) + if required > floor: + raise ValueError(f"{binary}: requires {symbol}, above GLIBC_{baseline}") + if required > version(highest[len("GLIBC_"):]): + highest = symbol + return highest + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("baseline") + parser.add_argument("binaries", type=Path, nargs="+") + args = parser.parse_args() + try: + for binary in args.binaries: + print(f"{binary}: highest required symbol {check(binary, args.baseline)} <= GLIBC_{args.baseline}") + except (OSError, ValueError, subprocess.CalledProcessError) as error: + parser.exit(1, f"{error}\n") + + +if __name__ == "__main__": + main() diff --git a/scripts/run_conditional_read_abba.py b/scripts/run_conditional_read_abba.py new file mode 100644 index 000000000..ab7a3e57c --- /dev/null +++ b/scripts/run_conditional_read_abba.py @@ -0,0 +1,205 @@ +#!/usr/bin/env python3 +"""Run the identical application-read test harness in serial ABBA processes. + +This measures local EC reads, not network/authentication or deployment capacity. +Use matching toolchain, features and profiles for both executable artifacts. +""" + +import argparse +import hashlib +import json +import math +import os +from pathlib import Path +import shutil +import subprocess +import time + + +def digest(path): + result = hashlib.sha256() + with path.open("rb") as source: + for chunk in iter(lambda: source.read(1024 * 1024), b""): + result.update(chunk) + return result.hexdigest() + + +def parse_rows(output): + rows = {} + prefix = "CONDITIONAL_BENCH " + for line in output.splitlines(): + if prefix in line: + row = json.loads(line.split(prefix, 1)[1]) + key = (row["size"], row["kind"], row["cache"], row["slowtail_ms"]) + if key in rows: + raise ValueError(f"duplicate benchmark row {key}") + rows[key] = row + if len(rows) != 6: + raise ValueError(f"expected six benchmark rows, received {len(rows)}") + if {(key[0], key[1]) for key in rows} != { + (size, kind) for size in (4096, 1_300_000) for kind in ("unconditional", "hit", "miss") + }: + raise ValueError("benchmark workload matrix is incomplete") + for row in rows.values(): + for metric in ("ops_per_sec", "p95_ms", "p99_ms", "lock_p95_ms", "lock_p99_ms"): + if not math.isfinite(row[metric]) or row[metric] <= 0: + raise ValueError(f"invalid benchmark metric {metric}") + return rows + + +def check_resource_isolation(check_load=True): + commands = subprocess.check_output(["ps", "-A", "-o", "comm="], text=True) + if any(Path(command.strip()).name in ("cargo", "rustc") for command in commands.splitlines()): + raise RuntimeError("stop concurrent Cargo builds before measuring") + load = os.getloadavg() + if check_load and load[0] > (os.cpu_count() or 1) * 0.5: + raise RuntimeError(f"host load too high for paired measurements: {load[0]}") + return load + + +def run_case(executable, expected_digest, env, log_path=None): + if digest(executable) != expected_digest: + raise RuntimeError("benchmark executable changed before execution") + # Poll for concurrent builds without attributing this benchmark's own CPU + # use to external host load. Load gates run before and after the process. + with subprocess.Popen([str(executable.resolve()), "conditional_read_paired_benchmark", + "--ignored", "--nocapture", "--test-threads=1"], + env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) as run: + try: + while True: + try: + output, _ = run.communicate(timeout=1) + break + except subprocess.TimeoutExpired: + check_resource_isolation(check_load=False) + except BaseException: + run.kill() + output, _ = run.communicate() + if log_path is not None: + log_path.write_text(output) + raise + returncode = run.returncode + if log_path is not None: + log_path.write_text(output) + check_resource_isolation() + if digest(executable) != expected_digest: + raise RuntimeError("benchmark executable changed during execution") + return returncode, output + + +def drift_passes(first, last): + if abs(last["ops_per_sec"] / first["ops_per_sec"] - 1) > 0.10: + return False + return all( + abs(last[metric] - first[metric]) <= max(first[metric] * 0.15, 0.25) + for metric in ("p95_ms", "p99_ms", "lock_p95_ms", "lock_p99_ms") + ) + + +def compare(rows): + if not all(rows[0].keys() == leg.keys() for leg in rows): + raise ValueError("ABBA legs have different workload matrices") + unstable = [key for key in rows[0] if not drift_passes(rows[0][key], rows[3][key])] + if unstable: + return {"accepted": False, "reason": "baseline_drift", "unstable_cases": unstable} + results = [] + for key, first in rows[0].items(): + baseline = {metric: math.sqrt(first[metric] * rows[3][key][metric]) for metric in + ("ops_per_sec", "p95_ms", "p99_ms", "lock_p95_ms", "lock_p99_ms")} + candidate = {metric: math.sqrt(rows[1][key][metric] * rows[2][key][metric]) for metric in baseline} + # Protect ordinary reads and condition misses; 304 savings alone do not pass. + passed = candidate["ops_per_sec"] >= baseline["ops_per_sec"] * 0.95 and all( + candidate[metric] <= baseline[metric] + max(baseline[metric] * 0.10, 0.25) + for metric in ("p95_ms", "p99_ms", "lock_p95_ms", "lock_p99_ms") + ) + results.append({"case": key, "baseline": baseline, "candidate": candidate, + "within_regression_budget": passed}) + return {"accepted": all(row["within_regression_budget"] for row in results), "cases": results} + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--baseline", type=Path, required=True) + parser.add_argument("--candidate", type=Path, required=True) + parser.add_argument("--baseline-revision", required=True) + parser.add_argument("--candidate-revision", required=True) + parser.add_argument("--harness", type=Path, required=True) + parser.add_argument("--profile", required=True) + parser.add_argument("--out-dir", type=Path, required=True) + parser.add_argument("--iterations", type=int, default=1000) + parser.add_argument("--concurrency", type=int, default=8) + parser.add_argument("--cooldown-seconds", type=float, default=5) + args = parser.parse_args() + if args.iterations < 100 or args.concurrency < 1 or args.cooldown_seconds < 0: + parser.error("iterations must be >=100, concurrency >0, cooldown >=0") + args.out_dir.mkdir(parents=True, exist_ok=False) + executables = {} + for name, source in (("baseline", args.baseline), ("candidate", args.candidate)): + original_digest = digest(source) + snapshot = args.out_dir / f"{name}.binary" + shutil.copyfile(source, snapshot) + snapshot.chmod(0o555) + if digest(snapshot) != original_digest: + raise RuntimeError("source executable changed while taking its snapshot") + executables[name] = snapshot + manifest = { + "baseline_revision": args.baseline_revision, "candidate_revision": args.candidate_revision, + "baseline_sha256": digest(executables["baseline"]), "candidate_sha256": digest(executables["candidate"]), + "harness_sha256": digest(args.harness), "profile": args.profile, + "iterations": args.iterations, "concurrency": args.concurrency, + "order": ["A1", "B1", "B2", "A2"], "slowtail_ms": [0, 20], "cache": [False, True], + "drift_budget": "throughput 10%; latency and lock percentiles 15% or 0.25ms", + "regression_budget": "throughput -5%; latency and lock percentiles +10% or 0.25ms", + "cpu_count": os.cpu_count(), "load_at_start": check_resource_isolation(), + } + (args.out_dir / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n") + legs = [] + for name, artifact in (("A1", "baseline"), ("B1", "candidate"), + ("B2", "candidate"), ("A2", "baseline")): + executable = executables[artifact] + cases = {} + for cache in (False, True): + for delay in (0, 20): + check_resource_isolation() + env = os.environ.copy() + # Keep ambient experiments and proxies out of localhost-sensitive runs. + for variable in tuple(env): + if variable.startswith("RUSTFS_") or variable.lower().endswith("_proxy"): + del env[variable] + env.update({ + "NO_PROXY": "*", "no_proxy": "*", "RUSTFS_OBJECT_LOCK_DIAG_ENABLE": "true", + "RUSTFS_OBJECT_DATA_CACHE_ENABLE": str(cache).lower(), + "RUSTFS_OBJECT_DATA_CACHE_MODE": "fill_materialize_enabled", + "RUSTFS_OBJECT_DATA_CACHE_MAX_BYTES": "8388608", + "RUSTFS_OBJECT_DATA_CACHE_MAX_ENTRY_BYTES": "2097152", + "RUSTFS_OBJECT_DATA_CACHE_MIN_FREE_MEMORY_PERCENT": "0", + "RUSTFS_CONDITIONAL_BENCH_ITERATIONS": str(args.iterations), + "RUSTFS_CONDITIONAL_BENCH_CONCURRENCY": str(args.concurrency), + "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS": str(delay), + "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DISKS": "3", + "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_BUCKET": "conditional-bench", + "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_OBJECT_PREFIX": "bench-", + }) + returncode, output = run_case(executable, manifest[f"{artifact}_sha256"], env, + args.out_dir / f"{name}-cache{int(cache)}-delay{delay}.log") + if returncode: + raise RuntimeError(f"{name} cache={cache} delay={delay} failed; see saved log") + for key, row in parse_rows(output).items(): + if (row["cache"] != cache or row["slowtail_ms"] != str(delay) + or row["iterations"] != args.iterations or row["concurrency"] != args.concurrency + or row["metadata_cache_hits"] != 0 or not row["metadata_cache_bypassed"] + or row["lock_samples"] <= 0): + raise ValueError("benchmark did not observe requested environment") + cases[key] = row + print(f"{name}: cache={cache} slowtail={delay}ms complete", flush=True) + legs.append(cases) + time.sleep(args.cooldown_seconds) + result = compare(legs) + result["load_at_end"] = os.getloadavg() + (args.out_dir / "summary.json").write_text(json.dumps(result, indent=2) + "\n") + print(json.dumps(result, indent=2)) + return 0 if result["accepted"] else 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/s3-tests/README.md b/scripts/s3-tests/README.md index c7bebde87..ed6aefb12 100644 --- a/scripts/s3-tests/README.md +++ b/scripts/s3-tests/README.md @@ -260,6 +260,17 @@ server's default. Set both key IDs to keys that already exist, and set templates should include `kms_keyid`, `kms_keyid2`, and `kms_default_keyid`, or the commented placeholders from the bundled template. +### Conditional write contract + +The conditional-write patch keeps the upstream tests in the gate while aligning +their requests with the [S3 write contract](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html): +`If-None-Match` accepts `*`, and an existing current object returns 412. RustFS +returns 501 `NotImplemented` for unsupported ETag-valued `If-None-Match`; the S3 +documentation does not specify that error code for PutObject or +CompleteMultipartUpload. The patched tests assert rejection without replacing +object contents, successful creation after a delete marker, and the existing +`If-Match` checks. No conditional-write tests are excluded. + ### Custom Configuration Files ```bash diff --git a/scripts/s3-tests/patches/0006-conditional-write-wildcard-contract.patch b/scripts/s3-tests/patches/0006-conditional-write-wildcard-contract.patch new file mode 100644 index 000000000..7ab4f0139 --- /dev/null +++ b/scripts/s3-tests/patches/0006-conditional-write-wildcard-contract.patch @@ -0,0 +1,163 @@ +diff --git a/s3tests/functional/test_s3.py b/s3tests/functional/test_s3.py +index e28021325..b1187e44e 100644 +--- a/s3tests/functional/test_s3.py ++++ b/s3tests/functional/test_s3.py +@@ -3203,11 +3203,12 @@ def test_put_object_ifnonmatch_good(): + + lf = (lambda **kwargs: kwargs['params']['headers'].update({'If-None-Match': 'ABCORZ'})) + client.meta.events.register('before-call.s3.PutObject', lf) +- response = client.put_object(Bucket=bucket_name,Key='foo', Body='zar') ++ e = assert_raises(ClientError, client.put_object, Bucket=bucket_name, Key='foo', Body='zar') ++ assert (501, 'NotImplemented') == _get_status_and_error_code(e.response) + + response = client.get_object(Bucket=bucket_name, Key='foo') + body = _get_body(response) +- assert body == 'zar' ++ assert body == 'bar' + + @pytest.mark.fails_on_aws + @pytest.mark.fails_on_dbstore +@@ -3227,8 +3228,8 @@ def test_put_object_ifnonmatch_failed(): + e = assert_raises(ClientError, client.put_object, Bucket=bucket_name, Key='foo', Body='zar') + + status, error_code = _get_status_and_error_code(e.response) +- assert status == 412 +- assert error_code == 'PreconditionFailed' ++ assert status == 501 ++ assert error_code == 'NotImplemented' + + response = client.get_object(Bucket=bucket_name, Key='foo') + body = _get_body(response) +@@ -19498,10 +19499,9 @@ def test_put_object_if_match(): + + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch='*') + assert (412, 'PreconditionFailed') == _get_status_and_error_code(e.response) +- e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch=etag) +- assert (412, 'PreconditionFailed') == _get_status_and_error_code(e.response) +- response = client.put_object(Bucket=bucket, Key=key, IfNoneMatch='badetag') +- assert 200 == response['ResponseMetadata']['HTTPStatusCode'] ++ for unsupported in (etag, 'badetag'): ++ e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch=unsupported) ++ assert (501, 'NotImplemented') == _get_status_and_error_code(e.response) + + client.put_object(Bucket=bucket, Key=key, IfMatch=etag) + +@@ -19512,7 +19512,9 @@ def test_put_object_if_match(): + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfMatch='badetag') + assert (404, 'NoSuchKey') == _get_status_and_error_code(e.response) + +- response = client.put_object(Bucket=bucket, Key=key, IfNoneMatch=etag) ++ e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch=etag) ++ assert (501, 'NotImplemented') == _get_status_and_error_code(e.response) ++ response = client.put_object(Bucket=bucket, Key=key, IfNoneMatch='*') + assert 200 == response['ResponseMetadata']['HTTPStatusCode'] + response = client.put_object(Bucket=bucket, Key=key, IfMatch='*') + assert 200 == response['ResponseMetadata']['HTTPStatusCode'] +@@ -19560,11 +19562,8 @@ def test_multipart_put_object_if_match(): + etag = response['ETag'] + + failing_conditional_multipart_upload((412, 'PreconditionFailed'), client, bucket, key, IfNoneMatch='*') +- failing_conditional_multipart_upload((412, 'PreconditionFailed'), client, bucket, key, IfNoneMatch=etag) +- +- response = successful_conditional_multipart_upload(client, bucket, key, IfNoneMatch='badetag') +- etag = response['ETag'] +- assert 200 == response['ResponseMetadata']['HTTPStatusCode'] ++ for unsupported in (etag, 'badetag'): ++ failing_conditional_multipart_upload((501, 'NotImplemented'), client, bucket, key, IfNoneMatch=unsupported) + + response = successful_conditional_multipart_upload(client, bucket, key, IfMatch=etag) + etag = response['ETag'] +@@ -19574,7 +19573,8 @@ def test_multipart_put_object_if_match(): + failing_conditional_multipart_upload((404, 'NoSuchKey'), client, bucket, key, IfMatch='*') + failing_conditional_multipart_upload((404, 'NoSuchKey'), client, bucket, key, IfMatch='badetag') + +- response = successful_conditional_multipart_upload(client, bucket, key, IfNoneMatch=etag) ++ failing_conditional_multipart_upload((501, 'NotImplemented'), client, bucket, key, IfNoneMatch=etag) ++ response = successful_conditional_multipart_upload(client, bucket, key, IfNoneMatch='*') + etag = response['ETag'] + assert 200 == response['ResponseMetadata']['HTTPStatusCode'] + response = successful_conditional_multipart_upload(client, bucket, key, IfMatch='*') +@@ -19602,16 +19602,16 @@ def test_put_current_object_if_none_match(): + + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch='*') + assert (412, 'PreconditionFailed') == _get_status_and_error_code(e.response) +- e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch=etag2) +- assert (412, 'PreconditionFailed') == _get_status_and_error_code(e.response) +- +- # we can't specify a version, so we only check against current object +- response = client.put_object(Bucket=bucket, Key=key, IfNoneMatch=etag) +- assert 200 == response['ResponseMetadata']['HTTPStatusCode'] +- response = client.put_object(Bucket=bucket, Key=key, IfNoneMatch='badetag') +- assert 200 == response['ResponseMetadata']['HTTPStatusCode'] ++ for unsupported in (etag2, etag, 'badetag'): ++ e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch=unsupported) ++ assert (501, 'NotImplemented') == _get_status_and_error_code(e.response) ++ assert _get_body(client.get_object(Bucket=bucket, Key=key)) == data2 + ++ # If-None-Match checks the current version; a delete marker permits creation. + client.delete_object(Bucket=bucket, Key=key) ++ response = client.put_object(Bucket=bucket, Key=key, Body=data1, IfNoneMatch='*') ++ assert 200 == response['ResponseMetadata']['HTTPStatusCode'] ++ assert _get_body(client.get_object(Bucket=bucket, Key=key)) == data1 + + @pytest.mark.conditional_write + @pytest.mark.fails_on_dbstore +@@ -19630,15 +19630,15 @@ def test_multipart_put_current_object_if_none_match(): + etag2 = response['ETag'] + + failing_conditional_multipart_upload((412, 'PreconditionFailed'), client, bucket, key, IfNoneMatch='*') +- failing_conditional_multipart_upload((412, 'PreconditionFailed'), client, bucket, key, IfNoneMatch=etag2) +- +- # we can't specify a version, so we only check against current object +- response = successful_conditional_multipart_upload(client, bucket, key, IfNoneMatch=etag) +- assert 200 == response['ResponseMetadata']['HTTPStatusCode'] +- response = successful_conditional_multipart_upload(client, bucket, key, IfNoneMatch='badetag') +- assert 200 == response['ResponseMetadata']['HTTPStatusCode'] ++ for unsupported in (etag2, etag, 'badetag'): ++ failing_conditional_multipart_upload((501, 'NotImplemented'), client, bucket, key, IfNoneMatch=unsupported) ++ assert _get_body(client.get_object(Bucket=bucket, Key=key)) == data2 + ++ # If-None-Match checks the current version; a delete marker permits creation. + client.delete_object(Bucket=bucket, Key=key) ++ response = successful_conditional_multipart_upload(client, bucket, key, data1, IfNoneMatch='*') ++ assert 200 == response['ResponseMetadata']['HTTPStatusCode'] ++ assert _get_body(client.get_object(Bucket=bucket, Key=key)) == data1 + + @pytest.mark.conditional_write + @pytest.mark.fails_on_dbstore +@@ -19656,7 +19656,7 @@ def test_put_current_object_if_match(): + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfMatch='badetag') + assert (404, 'NoSuchKey') == _get_status_and_error_code(e.response) + +- response = client.put_object(Bucket=bucket, Key=key, Body=data1, IfNoneMatch=etag) ++ response = client.put_object(Bucket=bucket, Key=key, Body=data1, IfNoneMatch='*') + assert 200 == response['ResponseMetadata']['HTTPStatusCode'] + etag = response['ETag'] + versionId = response['VersionId'] +@@ -19685,7 +19685,7 @@ def test_multipart_put_current_object_if_match(): + failing_conditional_multipart_upload((404, 'NoSuchKey'), client, bucket, key, IfMatch='*') + failing_conditional_multipart_upload((404, 'NoSuchKey'), client, bucket, key, IfMatch='badetag') + +- response = successful_conditional_multipart_upload(client, bucket, key, data1, IfNoneMatch=etag) ++ response = successful_conditional_multipart_upload(client, bucket, key, data1, IfNoneMatch='*') + assert 200 == response['ResponseMetadata']['HTTPStatusCode'] + etag = response['ETag'] + versionId = response['VersionId'] +@@ -19711,7 +19711,7 @@ def test_put_object_current_if_match(): + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch='*') + assert (412, 'PreconditionFailed') == _get_status_and_error_code(e.response) + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch=etag) +- assert (412, 'PreconditionFailed') == _get_status_and_error_code(e.response) ++ assert (501, 'NotImplemented') == _get_status_and_error_code(e.response) + + client.put_object(Bucket=bucket, Key=key, IfMatch=etag) + +@@ -19723,7 +19723,7 @@ def test_put_object_current_if_match(): + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfMatch='badetag') + assert (404, 'NoSuchKey') == _get_status_and_error_code(e.response) + +- client.put_object(Bucket=bucket, Key=key, IfNoneMatch=etag) ++ client.put_object(Bucket=bucket, Key=key, IfNoneMatch='*') + + @pytest.mark.fails_on_aws # only supported for directory buckets + @pytest.mark.conditional_write diff --git a/scripts/security/check_preview_release_workflow.sh b/scripts/security/check_preview_release_workflow.sh index 54bf2ef50..7f6b4d05b 100755 --- a/scripts/security/check_preview_release_workflow.sh +++ b/scripts/security/check_preview_release_workflow.sh @@ -32,10 +32,10 @@ require_absent() { fi } -require_line "$build_workflow" " cargo zigbuild --release --target \${{ matrix.target }} \"\${FEATURE_ARGS[@]}\" \"\${PROFILE_ARGS[@]}\" -p rustfs --bin \"\$binary\"" "cross builds must keep matrix features" -require_line "$build_workflow" " cargo build --release --target \${{ matrix.target }} \"\${FEATURE_ARGS[@]}\" \"\${PROFILE_ARGS[@]}\" -p rustfs --bin \"\$binary\"" "native builds must keep matrix features" -require_absent "$build_workflow" " cargo zigbuild --release --target \${{ matrix.target }} -p rustfs --bin \"\$binary\"" "cross builds must not drop matrix features" -require_absent "$build_workflow" " cargo build --release --target \${{ matrix.target }} -p rustfs --bin \"\$binary\"" "native builds must not drop matrix features" +require_line "$build_workflow" " cargo zigbuild --release --locked --target \"\$TARGET_ARG\" \"\${FEATURE_ARGS[@]}\" \"\${PROFILE_ARGS[@]}\" -p rustfs --bin rustfs" "cross builds must keep matrix features" +require_line "$build_workflow" " cargo build --release --locked --target \"\$TARGET_ARG\" \"\${FEATURE_ARGS[@]}\" \"\${PROFILE_ARGS[@]}\" -p rustfs --bin rustfs" "native builds must keep matrix features" +require_absent "$build_workflow" " cargo zigbuild --release --locked --target \"\$TARGET_ARG\" -p rustfs --bin rustfs" "cross builds must not drop matrix features" +require_absent "$build_workflow" " cargo build --release --locked --target \"\$TARGET_ARG\" -p rustfs --bin rustfs" "native builds must not drop matrix features" extract_job_if() { local file="$1" diff --git a/scripts/table-catalog/README.md b/scripts/table-catalog/README.md index b1b42df83..5a4a16244 100644 --- a/scripts/table-catalog/README.md +++ b/scripts/table-catalog/README.md @@ -55,6 +55,29 @@ metadata, mismatched expected and observed status, invalid row counts, or claim promotion beyond the client boundary are treated as evidence failures instead of support-matrix proof. +The smoke reads `rustfs.catalog-backing` from `/v1/config`, applying overrides +after defaults, before creating objects. `--catalog-backing` is an optional +expected value, not a way to relabel the deployment. A mismatch or an unknown +server backing aborts the run. Object mode uses the wire value `object`; the +legacy operator label `object-backed` is accepted as an expected-value alias. +Durable-strong must already be provisioned through the +[cutover procedure](../../docs/operations/s3-tables-cutover-runbook.md). +`--client-version` likewise checks the installed PyIceberg version rather than +overriding the version recorded in evidence. A mismatch aborts before setup. + +New PyIceberg evidence includes `catalog_probes`, `rest_signing_name`, and +`credential_vending_required`. Probe results distinguish `pass`, +`expected-unsupported`, and `skipped`. Both backings exercise maintenance +configuration, scheduler/worker dry-run, job inspection, quarantine inspection, +diagnostics, and export. Only the direct metadata-maintenance endpoint is +expected to reject durable-strong requests. A completed dry-run must retain its +queued job identity, report zero deletions, and leave the commit state unchanged; +the smoke restores its disabled, paused configuration after execution. +New full runs use `create-append-reload-scan-backing-contracts-v2`; historical +scenario names retain their original scope. +Explicitly skipping REST probes records only `create-append-reload-scan-core`; +it cannot establish extended REST compatibility. + The smoke test covers: - create or reuse the S3 bucket @@ -62,10 +85,18 @@ The smoke test covers: - load the PyIceberg REST catalog - create namespace and table - append two rows through PyIceberg -- reload and scan the table +- reload and compare the complete scanned rows, not only the row count - probe direct REST catalog endpoints for metadata-location, table refs, Iceberg views, maintenance config, metadata maintenance, worker run, and catalog diagnostics +- update namespace properties, reject overlapping update/removal keys without + changing persisted properties, and verify rename preserves table metadata +- round-trip an encoded multi-level namespace and reject double-encoded aliases +- check LoadTable does not vend credentials for missing or unknown delegation + tokens; when vending is required, check exact-token negotiation and scope +- on both backings, check a successful maintenance dry-run and export against + the current pointer, token, and generation; in durable-strong mode, also + verify the direct metadata-maintenance rejection leaves commit state unchanged - optionally drop the table and namespace The default profile uses the canonical RustFS catalog URI: @@ -139,7 +170,9 @@ temporary credentials: rejected. The direct REST catalog probes run by default after the PyIceberg append and -scan. For deployments that intentionally expose only the core Iceberg REST +scan. Both backings run maintenance control, diagnostics, and export probes. +Durable-strong checks the direct metadata-maintenance rejection separately. +For deployments that intentionally expose only the core Iceberg REST Catalog table path, skip those probes explicitly: ```bash diff --git a/scripts/table-catalog/duckdb_smoke.py b/scripts/table-catalog/duckdb_smoke.py index 5b89ce399..0459dd78c 100755 --- a/scripts/table-catalog/duckdb_smoke.py +++ b/scripts/table-catalog/duckdb_smoke.py @@ -41,6 +41,7 @@ class DuckDBSmokeResult: row_count: int cleanup_result: str checks: dict[str, str] + catalog_backing: str def parse_args(argv: list[str] | None = None) -> argparse.Namespace: @@ -335,6 +336,7 @@ def pyiceberg_args(args: argparse.Namespace) -> argparse.Namespace: require_vended_credentials=False, timeout=args.timeout, insecure=args.insecure, + catalog_backing=args.catalog_backing, ) @@ -417,6 +419,7 @@ def run_smoke(args: argparse.Namespace, deps: pyiceberg_smoke.RuntimeDeps) -> Du pyiceberg_smoke.ensure_local_proxy_bypass(endpoint) pyiceberg_smoke.ensure_aws_env(args.access_key, args.secret_key, args.region) iceberg_args = pyiceberg_args(args) + backing = pyiceberg_smoke.discover_catalog_backing(iceberg_args, deps) pyiceberg_smoke.ensure_bucket(iceberg_args, deps) pyiceberg_smoke.enable_table_bucket(iceberg_args, deps) @@ -581,7 +584,7 @@ def run_smoke(args: argparse.Namespace, deps: pyiceberg_smoke.RuntimeDeps) -> Du drop_namespace=not namespace_preexisting, ) - return DuckDBSmokeResult(client_version, metadata_location, 2, cleanup_result, checks) + return DuckDBSmokeResult(client_version, metadata_location, 2, cleanup_result, checks, backing) def current_utc_timestamp() -> str: @@ -598,7 +601,7 @@ def write_live_evidence(args: argparse.Namespace, result: DuckDBSmokeResult) -> scenario="rest-catalog-single-table-read-write-cross-engine-negative-boundaries", rustfs_build=args.rustfs_build, git_sha=args.git_sha, - catalog_backing=args.catalog_backing, + catalog_backing=result.catalog_backing, endpoint=args.endpoint, warehouse=args.bucket, rest_path="/iceberg", diff --git a/scripts/table-catalog/engine_compatibility.py b/scripts/table-catalog/engine_compatibility.py index 5b7de88c0..4743bcba0 100644 --- a/scripts/table-catalog/engine_compatibility.py +++ b/scripts/table-catalog/engine_compatibility.py @@ -18,6 +18,8 @@ DEFAULT_ICEBERG_VERSION = "1.7.1" DEFAULT_SCALA_VERSION = "2.12" DEFAULT_TRINO_VERSION = "477" DEFAULT_DUCKDB_VERSION = "1.5.5" +PYICEBERG_BACKING_SCENARIO = "create-append-reload-scan-backing-contracts-v2" +PYICEBERG_CORE_SCENARIO = "create-append-reload-scan-core" DEFAULT_SNOWFLAKE_CLIENT_VERSION = "operator-recorded" DEFAULT_DATABEND_VERSION = "operator-recorded" DEFAULT_TRINO_SERVER = "http://127.0.0.1:8080" @@ -738,6 +740,21 @@ def live_conformance_evidence( ) +def pyiceberg_catalog_probe_results(backing: str, *, skipped: bool, vended: bool) -> dict[str, str]: + if backing not in {"object", "durable-strong"}: + raise ValueError("PyIceberg probe evidence requires an observed RustFS backing") + if skipped: + return {"direct-rest": "skipped"} + probes = dict.fromkeys(("metadata-location", "refs", "views-and-namespace-properties", "rename", "load-table-delegation"), "pass") + probes["load-table-vended-credentials"] = "pass" if vended else "skipped" + probes.update(dict.fromkeys(( + "maintenance-config", "maintenance-scheduler", "maintenance-worker-dry-run", + "maintenance-job", "maintenance-quarantine-inspect", "diagnostics", "export", + ), "pass")) + probes["metadata-maintenance"] = "expected-unsupported" if backing == "durable-strong" else "pass" + return probes + + def live_conformance_evidence_schema() -> OrderedDict[str, Any]: return OrderedDict( [ @@ -879,6 +896,18 @@ def validate_live_conformance_evidence(record: dict[str, Any]) -> OrderedDict[st if expected_status == "pass" and client_name in LIVE_EVIDENCE_ROW_COUNT_CLIENTS and row_count != 2: raise ValueError("live conformance evidence row_count must be 2 for successful table smoke/read probes") + if client_name == "PyIceberg" and record["scenario"] in {PYICEBERG_CORE_SCENARIO, PYICEBERG_BACKING_SCENARIO}: + vended = record.get("credential_vending_required") + if not isinstance(vended, bool): + raise ValueError("PyIceberg probe evidence requires the credential negotiation mode") + expected_probes = pyiceberg_catalog_probe_results( + record["catalog_backing"], skipped=record["scenario"] == PYICEBERG_CORE_SCENARIO, vended=vended + ) + if record.get("catalog_probes") != expected_probes: + raise ValueError("PyIceberg probe evidence does not match the scenario and backing capabilities") + if record.get("rest_signing_name") not in {"s3", "s3tables"}: + raise ValueError("PyIceberg probe evidence requires the REST signing name") + return OrderedDict( [ ("status", "accepted"), diff --git a/scripts/table-catalog/pyiceberg_smoke.py b/scripts/table-catalog/pyiceberg_smoke.py index ebf7b5de2..911c994bc 100755 --- a/scripts/table-catalog/pyiceberg_smoke.py +++ b/scripts/table-catalog/pyiceberg_smoke.py @@ -27,6 +27,9 @@ import failure_coverage DEFAULT_PROFILE = "rustfs" CATALOG_VENDED_PROFILE = "rustfs-vended-credentials" VENDED_CREDENTIAL_PROFILES = {CATALOG_VENDED_PROFILE} +CATALOG_BACKING_KEY = "rustfs.catalog-backing" +CATALOG_BACKINGS = {"object", "durable-strong"} +SMOKE_ROWS = [{"id": 1, "payload": "alpha"}, {"id": 2, "payload": "beta"}] REQUIRED_STORAGE_CREDENTIAL_KEYS = ( "s3.access-key-id", "s3.secret-access-key", @@ -200,7 +203,7 @@ UNSUPPORTED_INVENTORY: list[dict[str, str]] = [ "status": "controlled-run-once-supported", "roadmap_area": "maintenance-worker", "catalog_endpoint": "GET /v1/{prefix}/namespaces/{namespace}/tables/{table}/maintenance/scheduler", - "expected_behavior": "background-enabled maintenance can be queued by the scheduler run endpoint, claimed by the worker run endpoint, and inspected through the scheduler status endpoint with disabled/paused state, queued-job handoff, current-job backpressure, retry deferral, quarantine boundary, audit timeline, per-job audit events, lease expiry recovery, heartbeat updates, and operator quarantine inspect/release/retry/abandon actions; built-in periodic scheduling is not claimed", + "expected_behavior": "background-enabled maintenance can be queued by the scheduler run endpoint, claimed by the worker run endpoint, and inspected through the scheduler status endpoint with disabled/paused state, queued-job handoff, current-job backpressure, retry deferral, quarantine boundary, audit timeline, per-job audit events, lease expiry recovery, heartbeat updates, and operator quarantine inspect/release/retry/abandon actions; opt-in periodic admission can queue jobs for allowlisted buckets, while workers still require an explicit operator invocation", }, { "capability": "manifest-data-reachability-cleanup", @@ -212,13 +215,13 @@ UNSUPPORTED_INVENTORY: list[dict[str, str]] = [ "capability": "snapshot-expiration", "status": "manual-maintenance-supported", "roadmap_area": "snapshot-maintenance", - "expected_behavior": "metadata maintenance can report snapshot expiration plans and manually commit safe snapshot expiration through the catalog; background scheduling and manifest/data cleanup are not claimed", + "expected_behavior": "metadata maintenance can report snapshot expiration plans and manually commit safe snapshot expiration through the catalog; automatic snapshot expiration and background manifest/data cleanup are not claimed", }, { "capability": "compaction-rewrite", "status": "controlled-run-once-supported", "roadmap_area": "snapshot-maintenance", - "expected_behavior": "metadata maintenance can plan binpack candidates and commit a safe partition-local and sort-order-preserving Parquet rewrite through the catalog; manifests with position or equality delete files produce machine-readable row-level planning and fail closed before rewrite; built-in periodic scheduling, delete-file rewrite, and row-level compaction execution are not claimed", + "expected_behavior": "metadata maintenance can plan binpack candidates and commit a safe partition-local and sort-order-preserving Parquet rewrite through the catalog; manifests with position or equality delete files produce machine-readable row-level planning and fail closed before rewrite; automatic compaction execution, delete-file rewrite, and row-level compaction execution are not claimed", }, { "capability": "row-level-delete-update-merge", @@ -283,6 +286,8 @@ class SmokeResult: row_count: int cleanup_result: str table_warehouse_location: str + catalog_backing: str + catalog_probes: dict[str, str] class RestRequestError(RuntimeError): @@ -536,7 +541,15 @@ def sign_rest_request(args: argparse.Namespace, deps: RuntimeDeps, request: Any) deps.botocore_auth(credentials, args.rest_signing_name, args.region).add_auth(request) -def signed_rest_request(args: argparse.Namespace, deps: RuntimeDeps, method: str, path: str, body: dict[str, Any] | None = None) -> dict[str, Any]: +def signed_rest_request( + args: argparse.Namespace, + deps: RuntimeDeps, + method: str, + path: str, + body: dict[str, Any] | None = None, + *, + access_delegation: str | None = None, +) -> dict[str, Any]: endpoint = normalized_endpoint(args.endpoint) url = f"{endpoint}{path}" payload = b"" if body is None else json.dumps(body, separators=(",", ":")).encode("utf-8") @@ -546,6 +559,8 @@ def signed_rest_request(args: argparse.Namespace, deps: RuntimeDeps, method: str } if body is not None: headers["content-type"] = "application/json" + if access_delegation is not None: + headers["X-Iceberg-Access-Delegation"] = access_delegation aws_request = deps.botocore_awsrequest(method=method, url=url, data=payload, headers=headers) sign_rest_request(args, deps, aws_request) @@ -586,6 +601,21 @@ def signed_rest_request_expect_error( raise RuntimeError(f"{method} {path} unexpectedly succeeded") +def discover_catalog_backing(args: argparse.Namespace, deps: RuntimeDeps) -> str: + query = urllib.parse.urlencode({"warehouse": profile_warehouse(args)}) + response = signed_rest_request(args, deps, "GET", f"{args.rest_path}/v1/config?{query}") + defaults, overrides = response.get("defaults"), response.get("overrides") + if not isinstance(defaults, dict) or not isinstance(overrides, dict): + raise RuntimeError("catalog config must return defaults and overrides") + backing = overrides.get(CATALOG_BACKING_KEY, defaults.get(CATALOG_BACKING_KEY)) + if not isinstance(backing, str) or backing not in CATALOG_BACKINGS: + raise RuntimeError("catalog config did not identify a supported RustFS backing") + expected = "object" if args.catalog_backing == "object-backed" else args.catalog_backing + if expected != "operator-recorded" and expected != backing: + raise RuntimeError("configured catalog backing does not match the server config") + return backing + + def ensure_bucket(args: argparse.Namespace, deps: RuntimeDeps) -> None: client = configured_s3_client(args, deps) try: @@ -972,12 +1002,13 @@ def current_utc_timestamp() -> str: def pyiceberg_client_version(args: argparse.Namespace) -> str: - if args.client_version: - return args.client_version try: - return importlib.metadata.version("pyiceberg") + version = importlib.metadata.version("pyiceberg") except importlib.metadata.PackageNotFoundError: - return "operator-recorded" + version = "operator-recorded" + if args.client_version and args.client_version != version: + raise RuntimeError(f"expected PyIceberg {args.client_version}, found {version}") + return version def redacted_command(argv: list[str]) -> str: @@ -1012,10 +1043,12 @@ def pyiceberg_live_evidence_record( operator: str, command: str, ) -> dict[str, Any]: - return engine_compatibility.live_conformance_evidence_record( + if catalog_backing != result.catalog_backing: + raise ValueError("evidence backing must match the observed server backing") + record = engine_compatibility.live_conformance_evidence_record( client_name="PyIceberg", client_version=client_version, - scenario="create-append-reload-scan-direct-rest-probes", + scenario=engine_compatibility.PYICEBERG_CORE_SCENARIO if args.skip_catalog_api_probes else engine_compatibility.PYICEBERG_BACKING_SCENARIO, rustfs_build=rustfs_build, git_sha=git_sha, catalog_backing=catalog_backing, @@ -1034,6 +1067,10 @@ def pyiceberg_live_evidence_record( claim="automated-smoke", command=command, ) + record["catalog_probes"] = result.catalog_probes.copy() + record["rest_signing_name"] = args.rest_signing_name + record["credential_vending_required"] = bool(args.require_vended_credentials) + return record def write_live_evidence(args: argparse.Namespace, result: SmokeResult) -> None: @@ -1045,7 +1082,7 @@ def write_live_evidence(args: argparse.Namespace, result: SmokeResult) -> None: client_version=pyiceberg_client_version(args), rustfs_build=args.rustfs_build, git_sha=args.git_sha, - catalog_backing=args.catalog_backing, + catalog_backing=result.catalog_backing, run_timestamp_utc=args.run_timestamp_utc or current_utc_timestamp(), operator=args.operator, command=redacted_command(sys.argv), @@ -1145,12 +1182,12 @@ def drop_table_if_present(catalog: Any, identifier: tuple[str, str]) -> None: def cleanup_catalog(catalog: Any, identifier: tuple[str, str]) -> None: drop_table_if_present(catalog, identifier) - try: + if catalog.table_exists(identifier): + raise RuntimeError("smoke table remains after cleanup") + if catalog.namespace_exists(identifier[0]): catalog.drop_namespace(identifier[0]) - except Exception as error: - message = str(error).lower() - if "not found" not in message and "does not exist" not in message: - print(f"warning: failed to drop namespace {identifier[0]}: {error}", file=sys.stderr) + if catalog.namespace_exists(identifier[0]): + raise RuntimeError("smoke namespace remains after cleanup") def current_snapshot_id_from_table_response(response: dict[str, Any]) -> int: @@ -1262,6 +1299,8 @@ def run_view_probe(args: argparse.Namespace, deps: RuntimeDeps) -> None: cleanup_namespace = False raise + run_namespace_properties_probe(args, deps, probe_namespace) + run_encoded_namespace_probe(args, deps, probe_namespace) view_path = view_endpoint_path(args, namespace=probe_namespace) for view_name in view_names: cleanup_views.append(view_name) @@ -1315,6 +1354,138 @@ def run_view_probe(args: argparse.Namespace, deps: RuntimeDeps) -> None: print(f"warning: failed to clean up view pagination probe: {error}", file=sys.stderr) +def run_encoded_namespace_probe(args: argparse.Namespace, deps: RuntimeDeps, parent: str) -> None: + segments = [parent, "encoded"] + path = namespace_endpoint_path(args, "\x1f".join(segments)) + signed_rest_request(args, deps, "POST", namespace_endpoint_path(args), {"namespace": segments}) + try: + response = signed_rest_request(args, deps, "GET", path) + if response.get("namespace") != segments: + raise RuntimeError("encoded namespace path did not preserve its component identity") + signed_rest_request_expect_error(args, deps, "GET", path.replace("%1F", "%251F"), expected_statuses={400}) + if signed_rest_request(args, deps, "GET", path).get("namespace") != segments: + raise RuntimeError("double-encoded lookup changed the namespace identity") + finally: + signed_rest_request(args, deps, "DELETE", path) + + +def run_namespace_properties_probe(args: argparse.Namespace, deps: RuntimeDeps, namespace: str) -> None: + path = namespace_endpoint_path(args, namespace) + before = signed_rest_request(args, deps, "GET", path) + properties = before.get("properties") + if not isinstance(properties, dict): + raise RuntimeError("namespace response did not include properties") + update = signed_rest_request( + args, deps, "POST", f"{path}/properties", {"updates": {"rustfs.smoke.updated": "true"}, "removals": ["rustfs.smoke"]} + ) + if update.get("updated") != ["rustfs.smoke.updated"] or update.get("removed") != ["rustfs.smoke"]: + raise RuntimeError("namespace property update did not report the applied keys") + expected = {**properties, "rustfs.smoke.updated": "true"} + expected.pop("rustfs.smoke", None) + if signed_rest_request(args, deps, "GET", path).get("properties") != expected: + raise RuntimeError("namespace property update was not persisted") + signed_rest_request_expect_error( + args, deps, "POST", f"{path}/properties", + {"updates": {"rustfs.smoke.updated": "false"}, "removals": ["rustfs.smoke.updated"]}, + expected_statuses={422}, + ) + if signed_rest_request(args, deps, "GET", path).get("properties") != expected: + raise RuntimeError("rejected namespace property update changed persisted properties") + + +def run_table_rename_probe(args: argparse.Namespace, deps: RuntimeDeps, backing: str) -> None: + if backing not in CATALOG_BACKINGS: + raise RuntimeError("table rename probe requires a known backing") + source = {"namespace": [args.namespace], "name": args.table} + destination = {"namespace": [args.namespace], "name": f"renamed-{uuid.uuid4().hex}"} + rename_path = f"{args.rest_path}/v1/{urllib.parse.quote(args.bucket, safe='')}/tables/rename" + source_path = table_endpoint_path(args) + destination_path = source_path.rsplit("/", 1)[0] + "/" + destination["name"] + before = signed_rest_request(args, deps, "GET", source_path) + signed_rest_request(args, deps, "POST", rename_path, {"source": source, "destination": destination}) + try: + signed_rest_request_expect_error(args, deps, "GET", source_path, expected_statuses={404}) + after = signed_rest_request(args, deps, "GET", destination_path) + for key in ("metadata-location", "metadata"): + if key not in before or after.get(key) != before[key]: + raise RuntimeError("table rename changed the Iceberg metadata or warehouse identity") + finally: + signed_rest_request(args, deps, "POST", rename_path, {"source": destination, "destination": source}) + restored = signed_rest_request(args, deps, "GET", source_path) + if any(restored.get(key) != before[key] for key in ("metadata-location", "metadata")): + raise RuntimeError("restored table rename changed the metadata or identity") + if backing == "object": + # Object-backed rename retains an inactive source entry. Consume only + # this probe's random identifier through normal create/drop APIs. + metadata = restored["metadata"] + schema = next((schema for schema in metadata.get("schemas", []) + if schema.get("schema-id") == metadata.get("current-schema-id")), None) + if schema is None: + raise RuntimeError("rename cleanup requires the original table schema") + pointer_path = table_endpoint_path(args, "/metadata-location") + pointer = signed_rest_request(args, deps, "GET", pointer_path) + if any(key not in pointer for key in ("metadata-location", "version-token", "generation")): + raise RuntimeError("rename cleanup requires the original table commit state") + replacement = signed_rest_request(args, deps, "POST", source_path.rsplit("/", 1)[0], + {"name": destination["name"], "schema": schema}) + fresh = replacement.get("metadata", {}) + if (any(not isinstance(fresh.get(key), str) or not fresh[key] or fresh[key] == metadata.get(key) + for key in ("table-uuid", "location")) + or fresh.get("snapshots", []) or fresh.get("current-snapshot-id") not in (None, -1)): + raise RuntimeError("rename cleanup replacement is not an isolated empty table") + signed_rest_request(args, deps, "DELETE", destination_path) + signed_rest_request_expect_error(args, deps, "GET", destination_path, expected_statuses={404}) + final = signed_rest_request(args, deps, "GET", source_path) + if (any(final.get(key) != restored[key] for key in ("metadata-location", "metadata")) + or signed_rest_request(args, deps, "GET", pointer_path) != pointer): + raise RuntimeError("rename cleanup changed the original table metadata or commit state") + + +def run_load_table_delegation_probe(args: argparse.Namespace, deps: RuntimeDeps) -> None: + path = table_endpoint_path(args) + for token in (None, "remote-signing", "not-vended-credentials", "VENDED-CREDENTIALS"): + response = signed_rest_request(args, deps, "GET", path, access_delegation=token) + config = response.get("config", {}) + if response.get("storage-credentials") or any(config.get(key) for key in REQUIRED_STORAGE_CREDENTIAL_KEYS): + raise RuntimeError("LoadTable returned credentials without exact delegation negotiation") + if args.require_vended_credentials: + response = signed_rest_request(args, deps, "GET", path, access_delegation="remote-signing, vended-credentials") + warehouse = table_warehouse_location(response.get("metadata")) + credentials = response.get("storage-credentials") + if not isinstance(credentials, list) or len(credentials) != 2 or any(not isinstance(item, dict) for item in credentials): + raise RuntimeError("LoadTable must vend exactly the warehouse and metadata scopes") + by_prefix = {item.get("prefix"): item for item in credentials} + warehouse_prefix = warehouse.rstrip("/") + "/" + metadata_location = response.get("metadata-location") + if not isinstance(metadata_location, str) or set(by_prefix) != {warehouse_prefix, metadata_location}: + raise RuntimeError("LoadTable vended scopes do not match the warehouse and current metadata") + credential = storage_credential_from_response({"storage-credentials": [by_prefix[warehouse_prefix]]}) + metadata_credential = storage_credential_from_response({"storage-credentials": [by_prefix[metadata_location]]}) + if any(credential.config[key] != metadata_credential.config[key] for key in REQUIRED_STORAGE_CREDENTIAL_KEYS): + raise RuntimeError("LoadTable scopes do not share the same temporary session") + verify_vended_credential_data_plane_scope(args, deps, credential, warehouse) + + +def run_strong_backing_boundaries(args: argparse.Namespace, deps: RuntimeDeps) -> None: + pointer_path = table_endpoint_path(args, "/metadata-location") + before = signed_rest_request(args, deps, "GET", pointer_path) + for key in ("metadata-location", "version-token", "generation"): + if key not in before: + raise RuntimeError("metadata-location response is missing the commit state") + path = table_endpoint_path(args, "/maintenance/metadata") + error = signed_rest_request_expect_error( + args, deps, "POST", path, + {"retain-recent-metadata-files": 1, "delete": False}, expected_statuses={400}, + ) + envelope = json.loads(error.response_body).get("error") + if not isinstance(envelope, dict) or (envelope.get("code"), envelope.get("type"), envelope.get("message")) != ( + 400, "BadRequestException", "operation is not supported with durable-strong table catalog backing" + ): + raise RuntimeError("strong metadata maintenance did not return the expected Iceberg error envelope") + if signed_rest_request(args, deps, "GET", pointer_path) != before: + raise RuntimeError("unsupported strong backing operation changed the table commit state") + + def maintenance_job_id(job: object) -> str | None: if not isinstance(job, dict): return None @@ -1324,93 +1495,125 @@ def maintenance_job_id(job: object) -> str | None: return value -def run_maintenance_probe(args: argparse.Namespace, deps: RuntimeDeps) -> None: +def run_maintenance_probe(args: argparse.Namespace, deps: RuntimeDeps, backing: str = "object") -> None: config_path = table_endpoint_path(args, "/maintenance/config") scheduler_path = table_endpoint_path(args, "/maintenance/scheduler") - scheduler_run_path = table_endpoint_path(args, "/maintenance/scheduler/run") - signed_rest_request(args, deps, "PUT", config_path, default_maintenance_config()) + pointer_path = table_endpoint_path(args, "/metadata-location") + before = signed_rest_request(args, deps, "GET", pointer_path) + for key in ("metadata-location", "version-token", "generation"): + if key not in before: + raise RuntimeError("metadata-location response is missing the commit state") + safe_config = default_maintenance_config() + signed_rest_request(args, deps, "PUT", config_path, safe_config) config = signed_rest_request(args, deps, "GET", config_path) - if config.get("version") != TABLE_MAINTENANCE_CONFIG_VERSION: + if any(config.get(key) != value for key, value in safe_config.items()): raise RuntimeError("maintenance config endpoint did not persist the smoke config") - report = signed_rest_request( - args, - deps, - "POST", - table_endpoint_path(args, "/maintenance/metadata"), - {"retain-recent-metadata-files": 1, "delete": False}, - ) - job = report.get("job") - job_id = maintenance_job_id(job) - if job_id is None: - raise RuntimeError("maintenance metadata endpoint did not return a job id") - audit_events = report.get("audit-events") - if not isinstance(audit_events, list) or not audit_events: - raise RuntimeError("maintenance metadata endpoint did not return audit events") - if not any(isinstance(event, dict) and event.get("action") == "PLANNED" for event in audit_events): - raise RuntimeError("maintenance metadata endpoint did not report a planning audit event") - job_report = signed_rest_request(args, deps, "GET", table_endpoint_path(args, f"/maintenance/jobs/{job_id}")) - if not isinstance(job_report.get("audit-events"), list): - raise RuntimeError("maintenance job endpoint did not return audit events") - quarantine = signed_rest_request( - args, - deps, - "POST", - table_endpoint_path(args, f"/maintenance/jobs/{job_id}/quarantine"), - {"action": "INSPECT"}, - ) - if quarantine.get("action") != "INSPECT" or not isinstance(quarantine.get("report"), dict): - raise RuntimeError("maintenance quarantine endpoint did not return an inspection report") + if backing == "durable-strong": + run_strong_backing_boundaries(args, deps) + elif backing == "object": + report = signed_rest_request( + args, deps, "POST", table_endpoint_path(args, "/maintenance/metadata"), + {"retain-recent-metadata-files": 1, "delete": False}, + ) + if maintenance_job_id(report.get("job")) is None: + raise RuntimeError("maintenance metadata endpoint did not return a job id") + audit_events = report.get("audit-events") + if not isinstance(audit_events, list) or not any( + isinstance(event, dict) and event.get("action") == "PLANNED" for event in audit_events + ): + raise RuntimeError("maintenance metadata endpoint did not report a planning audit event") + else: + raise RuntimeError("maintenance probes require a known backing") + scheduler = signed_rest_request(args, deps, "GET", scheduler_path) - expected_scheduler_statuses = {"READY", "QUEUED", "DISABLED", "PAUSED", "BACKPRESSURED", "RETRY_DEFERRED", "QUARANTINED"} - if scheduler.get("status") not in expected_scheduler_statuses: + expected_statuses = {"READY", "QUEUED", "DISABLED", "PAUSED", "BACKPRESSURED", "RETRY_DEFERRED", "QUARANTINED"} + if scheduler.get("status") not in expected_statuses: raise RuntimeError("maintenance scheduler endpoint did not return a stable scheduler status") - scheduler_timeline = scheduler.get("audit_timeline") - if not isinstance(scheduler_timeline, list): + timeline = scheduler.get("audit_timeline") + if not isinstance(timeline, list) or any( + not isinstance(item, dict) or not isinstance(item.get("audit-events"), list) for item in timeline + ): raise RuntimeError("maintenance scheduler endpoint did not return an audit timeline") - if scheduler_timeline and not isinstance(scheduler_timeline[0].get("audit-events"), list): - raise RuntimeError("maintenance scheduler audit timeline did not include job audit events") - scheduler_config = default_maintenance_config() - scheduler_config["background-enabled"] = True - scheduler_config["worker-paused"] = False - signed_rest_request(args, deps, "PUT", config_path, scheduler_config) - scheduler_run = signed_rest_request(args, deps, "POST", scheduler_run_path, {"scheduler-id": "pyiceberg-smoke-scheduler"}) - scheduler_run_job = scheduler_run.get("report", {}).get("job") - if not isinstance(scheduler_run_job, dict) or scheduler_run_job.get("status") != "QUEUED": - raise RuntimeError("maintenance scheduler run endpoint did not queue a maintenance job") - if scheduler_run_job.get("scheduler-id") != "pyiceberg-smoke-scheduler": - raise RuntimeError("maintenance scheduler run endpoint did not preserve scheduler identity") - scheduler_run_report = scheduler_run.get("scheduler") - if not isinstance(scheduler_run_report, dict) or scheduler_run_report.get("status") != "QUEUED": - raise RuntimeError("maintenance scheduler run endpoint did not return queued scheduler state") + scheduler_id = f"pyiceberg-smoke-{uuid.uuid4().hex}" + scheduler_config = {**safe_config, "background-enabled": True, "worker-paused": False} + try: + signed_rest_request(args, deps, "PUT", config_path, scheduler_config) + scheduler_run = signed_rest_request( + args, deps, "POST", f"{scheduler_path}/run", {"scheduler-id": scheduler_id} + ) + queued = scheduler_run.get("report", {}).get("job", {}) + job_id = maintenance_job_id(queued) + if job_id is None or queued.get("status") != "QUEUED" or queued.get("operation") != "DRY_RUN": + raise RuntimeError("maintenance scheduler run endpoint did not queue a dry-run job") + if queued.get("scheduler-id") != scheduler_id: + raise RuntimeError("maintenance scheduler run endpoint did not preserve scheduler identity") + if scheduler_run.get("scheduler", {}).get("status") != "QUEUED": + raise RuntimeError("maintenance scheduler run endpoint did not return queued scheduler state") + job_path = table_endpoint_path(args, f"/maintenance/jobs/{urllib.parse.quote(job_id, safe='')}") + job_report = signed_rest_request(args, deps, "GET", job_path) + if maintenance_job_id(job_report.get("job")) != job_id or not isinstance(job_report.get("audit-events"), list): + raise RuntimeError("maintenance job endpoint did not return the queued job and audit events") + quarantine = signed_rest_request(args, deps, "POST", f"{job_path}/quarantine", {"action": "INSPECT"}) + if quarantine.get("action") != "INSPECT" or maintenance_job_id(quarantine.get("report", {}).get("job")) != job_id: + raise RuntimeError("maintenance quarantine endpoint did not return the queued job inspection") - worker = signed_rest_request(args, deps, "POST", table_endpoint_path(args, "/maintenance/worker/run"), {}) - worker_job = worker.get("job") - expected_statuses = {"DISABLED", "PAUSED", "FAILED", "SUCCESSFUL", "RUNNING", "QUEUED"} - if not isinstance(worker_job, dict) or worker_job.get("status") not in expected_statuses: - raise RuntimeError("maintenance worker endpoint did not return a stable job status") - if not isinstance(worker.get("audit-events"), list): - raise RuntimeError("maintenance worker endpoint did not return audit events") + worker = signed_rest_request(args, deps, "POST", table_endpoint_path(args, "/maintenance/worker/run"), {}) + job = worker.get("job") + if not isinstance(job, dict) or job.get("status") != "SUCCESSFUL": + raise RuntimeError("maintenance worker did not complete the dry-run job") + if maintenance_job_id(job) != job_id or job.get("operation") != "DRY_RUN": + raise RuntimeError("maintenance worker changed the queued job identity or operation") + for key in ("deleted_metadata_file_count", "deleted_object_count"): + if type(job.get(key)) is not int or job[key] != 0: + raise RuntimeError("maintenance dry-run did not report zero deletions") + if not isinstance(worker.get("audit-events"), list) or not worker["audit-events"]: + raise RuntimeError("maintenance worker endpoint did not return audit events") + if signed_rest_request(args, deps, "GET", pointer_path) != before: + raise RuntimeError("maintenance dry-run changed the table commit state") + finally: + # Leave smoke-owned tables paused even if admission or execution fails. + signed_rest_request(args, deps, "PUT", config_path, safe_config) -def run_catalog_api_probes(args: argparse.Namespace, deps: RuntimeDeps) -> dict[str, Any]: +def run_catalog_api_probes(args: argparse.Namespace, deps: RuntimeDeps, backing: str) -> tuple[dict[str, Any], dict[str, str]]: + if backing not in CATALOG_BACKINGS: + raise RuntimeError("catalog probes require a known backing") table_response = signed_rest_request(args, deps, "GET", table_endpoint_path(args)) snapshot_id = current_snapshot_id_from_table_response(table_response) run_metadata_location_probe(args, deps, table_response) run_table_ref_probe(args, deps, snapshot_id) run_view_probe(args, deps) - run_maintenance_probe(args, deps) + run_table_rename_probe(args, deps, backing) + run_load_table_delegation_probe(args, deps) + run_maintenance_probe(args, deps, backing) diagnostics = signed_rest_request(args, deps, "GET", table_endpoint_path(args, "/catalog/diagnostics")) if not isinstance(diagnostics, dict) or not diagnostics: raise RuntimeError("catalog diagnostics endpoint returned an empty response") - return table_response + exported = signed_rest_request(args, deps, "GET", table_endpoint_path(args, "/catalog/export")) + pointer = signed_rest_request(args, deps, "GET", table_endpoint_path(args, "/metadata-location")) + entry = exported.get("table", {}) + location = entry.get("metadata_location") + if not isinstance(location, str) or not location: + raise RuntimeError("catalog export did not include its metadata location") + client_location = location if location.startswith("s3://") else f"s3://{args.bucket}/{location}" + if pointer.get("metadata-location") != client_location: + raise RuntimeError("catalog export does not match the current metadata pointer") + for public, stored in (("version-token", "version_token"), ("generation", "generation")): + if public not in pointer or entry.get(stored) != pointer[public]: + raise RuntimeError("catalog export does not match the current table commit state") + probes = engine_compatibility.pyiceberg_catalog_probe_results(backing, skipped=False, vended=bool(args.require_vended_credentials)) + return signed_rest_request(args, deps, "GET", table_endpoint_path(args)), probes def run_smoke(args: argparse.Namespace, deps: RuntimeDeps) -> SmokeResult: + if args.client_version: + pyiceberg_client_version(args) endpoint = normalized_endpoint(args.endpoint) ensure_local_proxy_bypass(endpoint) ensure_aws_env(args.access_key, args.secret_key, args.region) + backing = discover_catalog_backing(args, deps) print(f"[1/10] ensuring S3 bucket {args.bucket}") ensure_bucket(args, deps) @@ -1457,10 +1660,7 @@ def run_smoke(args: argparse.Namespace, deps: RuntimeDeps) -> SmokeResult: print(f"[8/10] appending rows through PyIceberg") table = catalog.load_table(identifier) rows = deps.pyarrow.Table.from_pylist( - [ - {"id": 1, "payload": "alpha"}, - {"id": 2, "payload": "beta"}, - ], + SMOKE_ROWS, schema=arrow_schema, ) table.append(rows) @@ -1470,9 +1670,12 @@ def run_smoke(args: argparse.Namespace, deps: RuntimeDeps) -> SmokeResult: scanned = loaded.scan().to_arrow() if scanned.num_rows != 2: raise RuntimeError(f"expected 2 rows after append, got {scanned.num_rows}") + if sorted(scanned.to_pylist(), key=lambda row: row["id"]) != SMOKE_ROWS: + raise RuntimeError("scan returned different data after append") loaded_table_location = table_warehouse_location(loaded) loaded_metadata_location = table_metadata_location(loaded) table_response = None + catalog_probes = {"direct-rest": "skipped"} if args.skip_catalog_api_probes: print("[10/10] skipping direct REST catalog API probes") @@ -1480,7 +1683,7 @@ def run_smoke(args: argparse.Namespace, deps: RuntimeDeps) -> SmokeResult: table_response = signed_rest_request(args, deps, "GET", table_endpoint_path(args)) else: print("[10/10] probing direct REST catalog APIs") - table_response = run_catalog_api_probes(args, deps) + table_response, catalog_probes = run_catalog_api_probes(args, deps, backing) metadata_location = loaded_metadata_location if isinstance(table_response, dict): @@ -1500,6 +1703,8 @@ def run_smoke(args: argparse.Namespace, deps: RuntimeDeps) -> SmokeResult: row_count=scanned.num_rows, cleanup_result="dropped-table-and-namespace" if args.cleanup else "not-requested", table_warehouse_location=loaded_table_location, + catalog_backing=backing, + catalog_probes=catalog_probes, ) diff --git a/scripts/table-catalog/test_duckdb_smoke.py b/scripts/table-catalog/test_duckdb_smoke.py index 768a95e8a..e5fb05afc 100644 --- a/scripts/table-catalog/test_duckdb_smoke.py +++ b/scripts/table-catalog/test_duckdb_smoke.py @@ -180,15 +180,28 @@ class DuckDBSmokeTest(unittest.TestCase): self.assertEqual(args.rest_path, "/iceberg") self.assertEqual(args.rest_signing_name, "s3") self.assertEqual(args.bucket, "rustfs-duckdb-smoke") + self.assertEqual(args.catalog_backing, "object") + + def test_backing_mismatch_prevents_duckdb_table_setup(self) -> None: + with mock.patch.object(duckdb_smoke, "duckdb_path", return_value="duckdb"), mock.patch.object( + duckdb_smoke, "duckdb_client_version", return_value="1.5.5" + ), mock.patch.object(duckdb_smoke.pyiceberg_smoke, "ensure_aws_env"), mock.patch.object( + duckdb_smoke.pyiceberg_smoke, "discover_catalog_backing", side_effect=RuntimeError("backing mismatch") + ), mock.patch.object(duckdb_smoke.pyiceberg_smoke, "ensure_bucket") as bucket: + with self.assertRaisesRegex(RuntimeError, "backing mismatch"): + duckdb_smoke.run_smoke(self.args(), mock.Mock()) + bucket.assert_not_called() def test_live_evidence_records_automated_duckdb_claim(self) -> None: args = self.args() + args.catalog_backing = "operator-recorded" result = duckdb_smoke.DuckDBSmokeResult( client_version="1.5.5", metadata_location="s3://rustfs-duckdb-smoke/metadata/00001.json", row_count=2, cleanup_result="dropped-tables-and-namespace", checks={"canonical_rest_catalog": "pass"}, + catalog_backing="durable-strong", ) with tempfile.TemporaryDirectory() as temp_dir: output = Path(temp_dir) / "evidence.json" @@ -200,6 +213,7 @@ class DuckDBSmokeTest(unittest.TestCase): evidence = document["live_conformance_evidence"] self.assertEqual(evidence["client_name"], "DuckDB Iceberg") self.assertEqual(evidence["claim"], "automated-rest-catalog-smoke") + self.assertEqual(evidence["catalog_backing"], "durable-strong") self.assertIn("--secret-key ''", evidence["command"]) self.assertNotIn("--secret-key secret", evidence["command"]) self.assertEqual(document["validation"]["status"], "accepted") diff --git a/scripts/table-catalog/test_engine_compatibility.py b/scripts/table-catalog/test_engine_compatibility.py index 65a766f40..27214e120 100644 --- a/scripts/table-catalog/test_engine_compatibility.py +++ b/scripts/table-catalog/test_engine_compatibility.py @@ -544,6 +544,53 @@ class EngineCompatibilityTest(unittest.TestCase): self.assertEqual(validation["claim"], "automated-smoke") self.assertEqual(validation["client_name"], "PyIceberg") + for legacy in ("create-append-reload-scan", "create-append-reload-scan-backing-contracts"): + with self.subTest(scenario=legacy): + historical = dict(record, scenario=legacy) + validated = engine_compatibility.validate_live_conformance_evidence(historical) + self.assertEqual(validated["scenario"], legacy) + self.assertNotIn("catalog_probes", historical) + + def test_backing_probe_evidence_rejects_missing_or_overstated_results(self) -> None: + for backing in ("object", "durable-strong"): + for skipped in (False, True): + for vended in (False, True): + with self.subTest(backing=backing, skipped=skipped, vended=vended): + scenario = (engine_compatibility.PYICEBERG_CORE_SCENARIO if skipped + else engine_compatibility.PYICEBERG_BACKING_SCENARIO) + record = engine_compatibility.live_conformance_evidence_record( + client_name="PyIceberg", client_version="0.11.1", scenario=scenario, + rustfs_build="test", git_sha="abc123", catalog_backing=backing, + endpoint="http://127.0.0.1:9000", warehouse="lake", rest_path="/iceberg", + namespace="sales", table="orders", metadata_location="s3://lake/metadata/v2.json", + run_timestamp_utc="2026-10-14T00:00:00Z", operator="test", + expected_status="pass", observed_status="pass", row_count=2, + cleanup_result="not-requested", claim="automated-smoke", command="smoke", + ) + probes = engine_compatibility.pyiceberg_catalog_probe_results( + backing, skipped=skipped, vended=vended + ) + record.update(catalog_probes=probes, rest_signing_name="s3", + credential_vending_required=vended) + validate = engine_compatibility.validate_live_conformance_evidence + self.assertEqual(validate(record)["status"], "accepted") + for changes in ( + {"catalog_probes": {}}, + {"catalog_probes": {**probes, "automatic-worker": "pass"}}, + {"rest_signing_name": "unknown"}, + {"credential_vending_required": "false"}, + ): + with self.subTest(changes=changes), self.assertRaises(ValueError): + validate({**record, **changes}) + if skipped: + with self.assertRaises(ValueError): + validate({**record, "scenario": engine_compatibility.PYICEBERG_BACKING_SCENARIO}) + else: + for name in ("metadata-maintenance", "maintenance-worker-dry-run", "diagnostics", "export"): + broken = {**probes, name: "skipped"} + with self.subTest(probe=name), self.assertRaises(ValueError): + validate({**record, "catalog_probes": broken}) + def test_live_conformance_evidence_rejects_missing_or_mismatched_results(self) -> None: record = engine_compatibility.live_conformance_evidence_record( client_name="Spark Iceberg REST catalog", diff --git a/scripts/table-catalog/test_pyiceberg_signing.py b/scripts/table-catalog/test_pyiceberg_signing.py index c326ea636..95ec7ebe2 100644 --- a/scripts/table-catalog/test_pyiceberg_signing.py +++ b/scripts/table-catalog/test_pyiceberg_signing.py @@ -104,6 +104,22 @@ class PyIcebergSigningTest(unittest.TestCase): SigV4Auth(Credentials(args.access_key, args.secret_key), args.rest_signing_name, args.region).add_auth(expected) self.assertEqual(request.headers["Authorization"], expected.headers["Authorization"]) + def test_delegation_header_is_included_in_the_signed_request(self) -> None: + args = self.args() + response = mock.MagicMock() + response.__enter__.return_value.read.return_value = b"{}" + with mock.patch.object(pyiceberg_smoke.urllib.request, "urlopen", return_value=response) as send: + pyiceberg_smoke.signed_rest_request( + args, self.deps, "GET", "/iceberg/v1/warehouse/namespaces/sales/tables/orders", + access_delegation="remote-signing, vended-credentials", + ) + wire = send.call_args.args[0] + request = Request(wire.method, wire.full_url, headers=dict(wire.header_items())).prepare() + self.assertEqual(request.headers["X-Iceberg-Access-Delegation"], "remote-signing, vended-credentials") + self.assertEqual(request.headers["Authorization"], self.expected_signature(request, args)) + request.headers["X-Iceberg-Access-Delegation"] = "remote-signing" + self.assertNotEqual(request.headers["Authorization"], self.expected_signature(request, args)) + def test_initial_config_and_recreated_sessions_are_signed(self) -> None: for profile in sorted(RUSTFS_PROFILES): args = self.args(profile) diff --git a/scripts/table-catalog/test_pyiceberg_smoke.py b/scripts/table-catalog/test_pyiceberg_smoke.py index 9c321f180..8f67684d6 100644 --- a/scripts/table-catalog/test_pyiceberg_smoke.py +++ b/scripts/table-catalog/test_pyiceberg_smoke.py @@ -3,12 +3,13 @@ from __future__ import annotations +import copy import json import os import re import sys import unittest -from contextlib import redirect_stdout +from contextlib import redirect_stderr, redirect_stdout from io import StringIO from types import SimpleNamespace from pathlib import Path @@ -286,7 +287,9 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json"}, ) - def test_view_probe_drops_smoke_view_after_load_failure(self) -> None: + @mock.patch.object(pyiceberg_smoke, "run_namespace_properties_probe") + @mock.patch.object(pyiceberg_smoke, "run_encoded_namespace_probe") + def test_view_probe_drops_smoke_view_after_load_failure(self, _encoded: object, _properties: object) -> None: args = self.parse_with_args(["--namespace", "sales", "--table", "orders"]) probe_namespace = "smoke-sales-orders-views-12345678" namespace_path = pyiceberg_smoke.namespace_endpoint_path(args) @@ -331,7 +334,9 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): self.assertIn(("DELETE", f"{view_path}/view-b", None), calls) self.assertIn(("DELETE", pyiceberg_smoke.namespace_endpoint_path(args, probe_namespace), None), calls) - def test_view_probe_cleans_candidate_after_create_timeout(self) -> None: + @mock.patch.object(pyiceberg_smoke, "run_namespace_properties_probe") + @mock.patch.object(pyiceberg_smoke, "run_encoded_namespace_probe") + def test_view_probe_cleans_candidate_after_create_timeout(self, _encoded: object, _properties: object) -> None: args = self.parse_with_args(["--namespace", "sales", "--table", "orders"]) probe_namespace = "smoke-sales-orders-views-12345678" namespace_path = pyiceberg_smoke.namespace_endpoint_path(args) @@ -364,7 +369,9 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): self.assertIn(("DELETE", f"{view_path}/view-a", None), calls) self.assertIn(("DELETE", pyiceberg_smoke.namespace_endpoint_path(args, probe_namespace), None), calls) - def test_view_probe_continues_cleanup_after_delete_failure(self) -> None: + @mock.patch.object(pyiceberg_smoke, "run_namespace_properties_probe") + @mock.patch.object(pyiceberg_smoke, "run_encoded_namespace_probe") + def test_view_probe_continues_cleanup_after_delete_failure(self, _encoded: object, _properties: object) -> None: args = self.parse_with_args(["--namespace", "sales", "--table", "orders"]) probe_namespace = "smoke-sales-orders-views-12345678" namespace_path = pyiceberg_smoke.namespace_endpoint_path(args) @@ -440,47 +447,104 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): with self.assertRaisesRegex(RuntimeError, message): pyiceberg_smoke.paginated_identifier_names(args, mock.Mock(), view_path, page_size=1) - def test_maintenance_probe_rejects_unknown_worker_status(self) -> None: - args = self.parse_with_args(["--namespace", "sales", "--table", "orders"]) - config_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/config") - maintenance_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/metadata") - job_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/jobs/job-1") - quarantine_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/jobs/job-1/quarantine") - scheduler_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/scheduler") - scheduler_run_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/scheduler/run") - worker_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/worker/run") + def maintenance_responses(self, backing="object", *, worker_changes=None, mutate_pointer=False): + config = {} + worker_ran = False + pointer = {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json", "version-token": "token-2", "generation": 2} + job = {"job_id": "job-2", "status": "QUEUED", "operation": "DRY_RUN"} + audit = [{"action": "PLANNED"}] - def fake_signed_request( - _args: object, - _deps: object, - method: str, - path: str, - body: object = None, - ) -> dict[str, object]: - if (method, path) == ("PUT", config_path): - return {} - if (method, path) == ("GET", config_path): - return {"version": 1} - if (method, path) == ("POST", maintenance_path): - return {"job": {"job_id": "job-1"}, "audit-events": [{"action": "PLANNED"}]} - if (method, path) == ("GET", job_path): - return {"job": {"job-id": "job-1", "status": "SUCCESSFUL"}, "audit-events": [{"action": "PLANNED"}]} - if (method, path) == ("POST", quarantine_path): - return {"action": "INSPECT", "report": {"job": {"job_id": "job-1"}}} - if (method, path) == ("GET", scheduler_path): - return {"status": "DISABLED", "audit_timeline": [{"job_id": "job-1", "audit-events": [{"action": "PLANNED"}]}]} - if (method, path) == ("POST", scheduler_run_path): - return { - "report": {"job": {"job-id": "job-2", "status": "QUEUED", "scheduler-id": "pyiceberg-smoke-scheduler"}}, - "scheduler": {"status": "QUEUED"}, - } - if (method, path) == ("POST", worker_path): - return {"job": {"status": "UNKNOWN"}, "audit-events": [{"action": "WORKER_CONTROL"}]} - raise AssertionError(f"unexpected REST request: {method} {path}") + def request(_args, _deps, method, path, body=None): + nonlocal config, worker_ran + if (method, path.rsplit("/", 1)[-1]) == ("GET", "metadata-location"): + return {**pointer, "generation": 3} if worker_ran and mutate_pointer else pointer.copy() + if path.endswith("/maintenance/config"): + if method == "PUT": + config = body.copy() + return config.copy() + if path.endswith("/maintenance/metadata"): + if backing == "durable-strong": + envelope = {"error": {"code": 400, "type": "BadRequestException", + "message": "operation is not supported with durable-strong table catalog backing"}} + raise pyiceberg_smoke.RestRequestError(method, path, 400, json.dumps(envelope)) + return {"job": {"job_id": "job-1"}, "audit-events": audit} + if path.endswith("/maintenance/scheduler"): + return {"status": "DISABLED", "audit_timeline": []} + if path.endswith("/maintenance/scheduler/run"): + return {"report": {"job": {**job, "scheduler-id": body["scheduler-id"]}}, + "scheduler": {"status": "QUEUED"}} + if path.endswith("/maintenance/jobs/job-2"): + return {"job": job.copy(), "audit-events": audit} + if path.endswith("/maintenance/jobs/job-2/quarantine"): + return {"action": "INSPECT", "report": {"job": job.copy()}} + if path.endswith("/maintenance/worker/run"): + worker_ran = True + completed = {**job, "status": "SUCCESSFUL", "deleted_metadata_file_count": 0, "deleted_object_count": 0} + completed.update(worker_changes or {}) + return {"job": completed, "audit-events": audit} + raise AssertionError(f"unexpected maintenance request: {method} {path}") - with mock.patch.object(pyiceberg_smoke, "signed_rest_request", side_effect=fake_signed_request): - with self.assertRaisesRegex(RuntimeError, "stable job status"): - pyiceberg_smoke.run_maintenance_probe(args, mock.Mock()) + return request + + def test_maintenance_dry_run_exercises_both_backings_and_leaves_worker_paused(self) -> None: + args = self.parse_with_args([]) + for backing in ("object", "durable-strong"): + with self.subTest(backing=backing), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=self.maintenance_responses(backing) + ) as request: + pyiceberg_smoke.run_maintenance_probe(args, mock.Mock(), backing) + paths = [call.args[3] for call in request.call_args_list] + for suffix in ("/maintenance/scheduler/run", "/maintenance/jobs/job-2", + "/maintenance/jobs/job-2/quarantine", "/maintenance/worker/run"): + self.assertTrue(any(path.endswith(suffix) for path in paths), suffix) + self.assertEqual(request.call_args.args[2], "PUT") + self.assertEqual(request.call_args.args[4], pyiceberg_smoke.default_maintenance_config()) + for call in request.call_args_list: + if call.args[2] == "PUT": + self.assertFalse(call.args[4]["delete-enabled"]) + + def test_maintenance_failure_never_counts_as_success_and_restores_safe_config(self) -> None: + args = self.parse_with_args([]) + failures = [ + ({"status": status}, "complete the dry-run") + for status in ("FAILED", "PAUSED", "RUNNING", "QUEUED", "DISABLED", "UNKNOWN", None) + ] + failures.extend([ + ({"job_id": "other-job"}, "job identity"), + ({"operation": "DELETE"}, "job identity"), + ({"deleted_object_count": 1}, "zero deletions"), + ({"deleted_metadata_file_count": 1}, "zero deletions"), + ({"deleted_object_count": None}, "zero deletions"), + ({"deleted_object_count": False}, "zero deletions"), + ]) + for backing in ("object", "durable-strong"): + for changes, message in failures: + with self.subTest(backing=backing, changes=changes), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", + side_effect=self.maintenance_responses(backing, worker_changes=changes) + ) as request: + with self.assertRaisesRegex(RuntimeError, message): + pyiceberg_smoke.run_maintenance_probe(args, mock.Mock(), backing) + self.assertEqual(request.call_args.args[4], pyiceberg_smoke.default_maintenance_config()) + + def test_maintenance_detects_pointer_advance_and_pauses_after_transport_failure(self) -> None: + args = self.parse_with_args([]) + for backing in ("object", "durable-strong"): + for transport_failure in (False, True): + base = self.maintenance_responses(backing, mutate_pointer=not transport_failure) + + def request(*args, **kwargs): + if transport_failure and args[3].endswith("/maintenance/worker/run"): + raise OSError("worker connection lost") + return base(*args, **kwargs) + + with self.subTest(backing=backing, transport_failure=transport_failure), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=request + ) as calls: + expected = OSError if transport_failure else RuntimeError + with self.assertRaises(expected): + pyiceberg_smoke.run_maintenance_probe(args, mock.Mock(), backing) + self.assertEqual(calls.call_args.args[4], pyiceberg_smoke.default_maintenance_config()) def test_catalog_api_probe_exercises_extended_rest_surfaces(self) -> None: args = self.parse_with_args(["--namespace", "sales", "--table", "orders"]) @@ -490,13 +554,12 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): metadata_location_path = pyiceberg_smoke.table_endpoint_path(args, "/metadata-location") refs_path = pyiceberg_smoke.table_ref_endpoint_path(args) ref_name = pyiceberg_smoke.safe_ref_segment(args.namespace, args.table) - config_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/config") - maintenance_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/metadata") quarantine_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/jobs/job-1/quarantine") scheduler_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/scheduler") scheduler_run_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/scheduler/run") worker_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/worker/run") diagnostics_path = pyiceberg_smoke.table_endpoint_path(args, "/catalog/diagnostics") + maintenance_request = self.maintenance_responses() def fake_signed_request( _args: object, @@ -509,7 +572,9 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): if (method, path) == ("GET", table_path): return {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json", "metadata": {"current-snapshot-id": 7}} if (method, path) == ("GET", metadata_location_path): - return {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json", "version-token": "token-2"} + return {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json", "version-token": "token-2", "generation": 2} + if (method, path) == ("GET", pyiceberg_smoke.table_endpoint_path(args, "/catalog/export")): + return {"table": {"metadata_location": "s3://lake/tables/id/metadata/v2.metadata.json", "version_token": "token-2", "generation": 2}} if method == "PUT" and path.startswith(f"{refs_path}/"): return {} if (method, path) == ("GET", refs_path): @@ -518,31 +583,16 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): return {"refs": {}} if method == "DELETE" and path.startswith(f"{refs_path}/"): return {} - if (method, path) == ("PUT", config_path): - return {} - if (method, path) == ("GET", config_path): - return {"version": 1} - if (method, path) == ("POST", maintenance_path): - return {"job": {"job_id": "job-1"}, "audit-events": [{"action": "PLANNED"}]} - if (method, path) == ("GET", pyiceberg_smoke.table_endpoint_path(args, "/maintenance/jobs/job-1")): - return {"job": {"job-id": "job-1", "status": "SUCCESSFUL"}, "audit-events": [{"action": "PLANNED"}]} - if (method, path) == ("POST", quarantine_path): - return {"action": "INSPECT", "report": {"job": {"job_id": "job-1"}}} - if (method, path) == ("GET", scheduler_path): - return {"status": "DISABLED", "audit_timeline": [{"job_id": "job-1", "audit-events": [{"action": "PLANNED"}]}]} - if (method, path) == ("POST", scheduler_run_path): - return { - "report": {"job": {"job-id": "job-2", "status": "QUEUED", "scheduler-id": "pyiceberg-smoke-scheduler"}}, - "scheduler": {"status": "QUEUED"}, - } - if (method, path) == ("POST", worker_path): - return {"job": {"status": "PAUSED"}, "audit-events": [{"action": "WORKER_CONTROL"}]} + if "/maintenance/" in path: + return maintenance_request(_args, _deps, method, path, body) if (method, path) == ("GET", diagnostics_path): return {"status": "ok"} raise AssertionError(f"unexpected REST request: {method} {path}") with mock.patch.object(pyiceberg_smoke, "signed_rest_request", side_effect=fake_signed_request): - with mock.patch.object(pyiceberg_smoke, "run_view_probe") as view_probe: + with mock.patch.object(pyiceberg_smoke, "run_view_probe") as view_probe, mock.patch.object( + pyiceberg_smoke, "run_table_rename_probe" + ), mock.patch.object(pyiceberg_smoke, "run_load_table_delegation_probe"): with mock.patch.object( pyiceberg_smoke, "signed_rest_request_expect_error", @@ -553,8 +603,10 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): "snapshot ref has retention policy; force is required", ), ) as expect_error: - pyiceberg_smoke.run_catalog_api_probes(args, deps) + _, probes = pyiceberg_smoke.run_catalog_api_probes(args, deps, "object") + self.assertEqual(probes["maintenance-worker-dry-run"], "pass") + self.assertEqual(probes["diagnostics"], "pass") view_probe.assert_called_once_with(args, deps) expect_error.assert_called_once_with( args, @@ -567,8 +619,8 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): self.assertIn(("GET", metadata_location_path, None), calls) self.assertIn(("GET", diagnostics_path, None), calls) self.assertIn(("GET", scheduler_path, None), calls) - self.assertIn(("POST", scheduler_run_path, {"scheduler-id": "pyiceberg-smoke-scheduler"}), calls) - self.assertIn(("POST", quarantine_path, {"action": "INSPECT"}), calls) + self.assertTrue(any(method == "POST" and path == scheduler_run_path for method, path, _ in calls)) + self.assertIn(("POST", quarantine_path.replace("job-1", "job-2"), {"action": "INSPECT"}), calls) self.assertIn(("POST", worker_path, {}), calls) def test_table_ref_probe_force_deletes_smoke_ref_after_validation_failure(self) -> None: @@ -601,12 +653,15 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): calls, ) - def run_smoke_with_fakes(self, args: object, probe_calls: list[str]) -> list[str]: + def run_smoke_with_fakes(self, args: object, probe_calls: list[str], scanned_rows: list[dict[str, object]] | None = None) -> list[str]: events: list[str] = [] class FakeArrowTable: num_rows = 2 + def to_pylist(self) -> list[dict[str, object]]: + return pyiceberg_smoke.SMOKE_ROWS if scanned_rows is None else scanned_rows + class FakePyArrowTableFactory: @staticmethod def from_pylist(rows: list[dict[str, object]], *, schema: object) -> object: @@ -665,7 +720,9 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): }, ) - with mock.patch.object(pyiceberg_smoke, "ensure_local_proxy_bypass"): + with mock.patch.object(pyiceberg_smoke, "ensure_local_proxy_bypass"), mock.patch.object( + pyiceberg_smoke, "discover_catalog_backing", return_value="object" + ): with mock.patch.object(pyiceberg_smoke, "ensure_aws_env"): with mock.patch.object(pyiceberg_smoke, "ensure_bucket"): with mock.patch.object(pyiceberg_smoke, "enable_table_bucket"): @@ -687,12 +744,20 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): side_effect=lambda *_args: ( events.append("catalog-probes"), probe_calls.append("catalog-probes"), - ), + ({}, {"direct-rest": "pass"}), + )[-1], ): with redirect_stdout(StringIO()): pyiceberg_smoke.run_smoke(args, deps) return events + def test_run_smoke_rejects_equal_row_count_with_different_data(self) -> None: + args = self.parse_with_args(["--bucket", "lake", "--namespace", "sales", "--table", "orders"]) + calls: list[str] = [] + with self.assertRaisesRegex(RuntimeError, "different data"): + self.run_smoke_with_fakes(args, calls, [{"id": 1, "payload": "wrong"}, {"id": 2, "payload": "beta"}]) + self.assertEqual(calls, []) + def test_run_smoke_probes_extended_catalog_apis_by_default(self) -> None: args = self.parse_with_args(["--bucket", "lake", "--namespace", "sales", "--table", "orders"]) probe_calls: list[str] = [] @@ -1052,6 +1117,8 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): row_count=2, cleanup_result="not-requested", table_warehouse_location="s3://lake/tables/table-id", + catalog_backing="durable-strong", + catalog_probes=pyiceberg_smoke.engine_compatibility.pyiceberg_catalog_probe_results("durable-strong", skipped=False, vended=False), ) record = pyiceberg_smoke.pyiceberg_live_evidence_record( @@ -1072,6 +1139,217 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): self.assertEqual(record["metadata_location"], "s3://lake/tables/table-id/metadata/v2.metadata.json") self.assertEqual(record["row_count"], 2) self.assertEqual(record["claim"], "automated-smoke") + self.assertEqual(record["catalog_probes"]["metadata-maintenance"], "expected-unsupported") + self.assertEqual(record["catalog_probes"]["maintenance-worker-dry-run"], "pass") + self.assertEqual(record["rest_signing_name"], "s3") + validate = pyiceberg_smoke.engine_compatibility.validate_live_conformance_evidence + self.assertEqual(validate(record)["status"], "accepted") + record["catalog_probes"]["metadata-maintenance"] = "pass" + with self.assertRaisesRegex(ValueError, "capabilities"): + validate(record) + del record["catalog_probes"] + with self.assertRaisesRegex(ValueError, "capabilities"): + validate(record) + + def test_backing_discovery_uses_overrides_and_rejects_mismatches(self) -> None: + args = self.parse_with_args(["--bucket", "lake", "--catalog-backing", "durable-strong"]) + config = {"defaults": {"rustfs.catalog-backing": "object"}, "overrides": {"rustfs.catalog-backing": "durable-strong"}} + with mock.patch.object(pyiceberg_smoke, "signed_rest_request", return_value=config) as request: + self.assertEqual(pyiceberg_smoke.discover_catalog_backing(args, mock.Mock()), "durable-strong") + self.assertEqual(request.call_args.args[2:4], ("GET", "/iceberg/v1/config?warehouse=lake")) + for response in ({}, {"defaults": {}, "overrides": {}}, {"defaults": {"rustfs.catalog-backing": "unknown"}, "overrides": {}}): + with self.subTest(response=response), mock.patch.object(pyiceberg_smoke, "signed_rest_request", return_value=response): + with self.assertRaises(RuntimeError): + pyiceberg_smoke.discover_catalog_backing(args, mock.Mock()) + config["overrides"] = {} + with mock.patch.object(pyiceberg_smoke, "signed_rest_request", return_value=config): + with self.assertRaisesRegex(RuntimeError, "does not match"): + pyiceberg_smoke.discover_catalog_backing(args, mock.Mock()) + args.catalog_backing = "object-backed" + self.assertEqual(pyiceberg_smoke.discover_catalog_backing(args, mock.Mock()), "object") + + def test_client_version_is_observed_and_mismatch_aborts_before_setup(self) -> None: + args = self.parse_with_args(["--client-version", "0.11.1"]) + with mock.patch.object(pyiceberg_smoke.importlib.metadata, "version", return_value="0.11.1"): + self.assertEqual(pyiceberg_smoke.pyiceberg_client_version(args), "0.11.1") + with mock.patch.object(pyiceberg_smoke.importlib.metadata, "version", return_value="0.10.0"), mock.patch.object( + pyiceberg_smoke, "ensure_bucket" + ) as bucket, mock.patch.object(pyiceberg_smoke, "discover_catalog_backing") as config: + with self.assertRaisesRegex(RuntimeError, "expected PyIceberg 0.11.1, found 0.10.0"): + pyiceberg_smoke.run_smoke(args, mock.Mock()) + bucket.assert_not_called() + config.assert_not_called() + + def test_backing_mismatch_fails_before_creating_any_objects(self) -> None: + args = self.parse_with_args([]) + with mock.patch.object(pyiceberg_smoke, "discover_catalog_backing", side_effect=RuntimeError("mismatch")), mock.patch.object( + pyiceberg_smoke, "ensure_bucket" + ) as bucket, mock.patch.object(pyiceberg_smoke, "ensure_aws_env"): + with self.assertRaisesRegex(RuntimeError, "mismatch"): + pyiceberg_smoke.run_smoke(args, mock.Mock()) + bucket.assert_not_called() + + def strong_probe_responses(self, *, status=400, error_type="BadRequestException", mutate=False): + pointer = {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json", "version-token": "t2", "generation": 2} + reads = 0 + + def request(_args, _deps, method, path, body=None): + nonlocal reads + if path.endswith("/metadata-location"): + reads += 1 + return {**pointer, "generation": 3} if mutate and reads > 1 else pointer.copy() + self.assertTrue(path.endswith("/maintenance/metadata"), path) + error = {"error": {"code": status, "type": error_type, + "message": "operation is not supported with durable-strong table catalog backing"}} + raise pyiceberg_smoke.RestRequestError(method, path, status, json.dumps(error)) + + return request + + def test_strong_metadata_boundary_checks_error_and_unchanged_pointer(self) -> None: + args = self.parse_with_args([]) + with mock.patch.object(pyiceberg_smoke, "signed_rest_request", side_effect=self.strong_probe_responses()) as request: + pyiceberg_smoke.run_strong_backing_boundaries(args, mock.Mock()) + self.assertEqual(request.call_count, 3) + for kwargs, message in (({"status": 403}, "expected one of"), ({"status": 500}, "expected one of"), + ({"error_type": "RESTException"}, "error envelope"), ({"mutate": True}, "commit state")): + with self.subTest(kwargs=kwargs), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=self.strong_probe_responses(**kwargs) + ): + with self.assertRaisesRegex(RuntimeError, message): + pyiceberg_smoke.run_strong_backing_boundaries(args, mock.Mock()) + + def test_catalog_export_checks_pointer_token_and_generation_on_both_backings(self) -> None: + args = self.parse_with_args(["--bucket", "lake"]) + location = "s3://lake/tables/id/metadata/v2.metadata.json" + table = {"metadata-location": location, "metadata": {"current-snapshot-id": 7}} + pointer = {"metadata-location": location, "version-token": "token-2", "generation": 2} + entry = {"metadata_location": location, "version_token": "token-2", "generation": 2} + for backing in ("object", "durable-strong"): + for mismatch in (None, "metadata_location", "version_token", "generation"): + exported = {**entry, mismatch: "stale"} if mismatch else entry + with self.subTest(backing=backing, mismatch=mismatch), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", + side_effect=[table, {"status": "HEALTHY"}, {"table": exported}, pointer, table] + ), mock.patch.object(pyiceberg_smoke, "run_metadata_location_probe"), mock.patch.object( + pyiceberg_smoke, "run_table_ref_probe" + ), mock.patch.object(pyiceberg_smoke, "run_view_probe"), mock.patch.object( + pyiceberg_smoke, "run_table_rename_probe" + ), mock.patch.object(pyiceberg_smoke, "run_load_table_delegation_probe"), mock.patch.object( + pyiceberg_smoke, "run_maintenance_probe" + ) as maintenance: + if mismatch: + with self.assertRaisesRegex(RuntimeError, "does not match"): + pyiceberg_smoke.run_catalog_api_probes(args, mock.Mock(), backing) + else: + _, probes = pyiceberg_smoke.run_catalog_api_probes(args, mock.Mock(), backing) + self.assertEqual(probes["diagnostics"], "pass") + self.assertEqual(probes["export"], "pass") + self.assertEqual(probes["maintenance-worker-dry-run"], "pass") + self.assertEqual(probes["metadata-maintenance"], + "expected-unsupported" if backing == "durable-strong" else "pass") + self.assertEqual(maintenance.call_args.args[2], backing) + + def test_namespace_properties_detect_partial_or_rejected_writes(self) -> None: + args = self.parse_with_args([]) + before = {"properties": {"rustfs.smoke": "true", "owner": "unchanged"}} + after = {"properties": {"rustfs.smoke.updated": "true", "owner": "unchanged"}} + update = {"updated": ["rustfs.smoke.updated"], "removed": ["rustfs.smoke"]} + error = pyiceberg_smoke.RestRequestError("POST", "/properties", 422, "overlapping keys") + for final in (after, before): + with self.subTest(final=final), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=[before, update, after, error, final] + ): + if final == after: + pyiceberg_smoke.run_namespace_properties_probe(args, mock.Mock(), "probe") + else: + with self.assertRaisesRegex(RuntimeError, "rejected namespace"): + pyiceberg_smoke.run_namespace_properties_probe(args, mock.Mock(), "probe") + + def test_encoded_namespace_probe_rejects_aliasing_and_cleans_up(self) -> None: + args = self.parse_with_args([]) + namespace = {"namespace": ["probe", "encoded"]} + error = pyiceberg_smoke.RestRequestError("GET", "/namespace", 400, "invalid namespace") + for double_encoded_response in (error, namespace): + with self.subTest(response=double_encoded_response), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=[{}, namespace, double_encoded_response, namespace, {}] + ) as request: + if isinstance(double_encoded_response, Exception): + pyiceberg_smoke.run_encoded_namespace_probe(args, mock.Mock(), "probe") + else: + with self.assertRaisesRegex(RuntimeError, "unexpectedly succeeded"): + pyiceberg_smoke.run_encoded_namespace_probe(args, mock.Mock(), "probe") + self.assertTrue(request.call_args_list[1].args[3].endswith("probe%1Fencoded")) + self.assertTrue(request.call_args_list[2].args[3].endswith("probe%251Fencoded")) + self.assertEqual(request.call_args.args[2], "DELETE") + + def test_rename_probe_restores_name_even_when_metadata_changes(self) -> None: + args = self.parse_with_args(["--bucket", "lake", "--namespace", "sales", "--table", "orders"]) + before = {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json", "metadata": {"location": "s3://lake/tables/id", "table-uuid": "uuid"}} + missing = pyiceberg_smoke.RestRequestError("GET", "/table", 404, "missing") + for changed in (False, True): + after = {**before, "metadata-location": "other"} if changed else before + with self.subTest(changed=changed), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=[before, {}, missing, after, {}, before] + ) as request: + if changed: + with self.assertRaisesRegex(RuntimeError, "warehouse identity"): + pyiceberg_smoke.run_table_rename_probe(args, mock.Mock(), "durable-strong") + else: + pyiceberg_smoke.run_table_rename_probe(args, mock.Mock(), "durable-strong") + renames = [call.args[4] for call in request.call_args_list if call.args[2] == "POST"] + self.assertEqual(renames[0]["source"], renames[1]["destination"]) + self.assertEqual(renames[0]["destination"], renames[1]["source"]) + + def test_delegation_requires_exact_token_and_warehouse_scope(self) -> None: + args = self.parse_with_args(["--require-vended-credentials"]) + credential = {"prefix": "s3://lake/tables/id/", "config": {key: "temporary" for key in pyiceberg_smoke.REQUIRED_STORAGE_CREDENTIAL_KEYS}} + metadata_credential = {"prefix": "s3://lake/catalog/current.metadata.json", "config": credential["config"].copy()} + response = {"metadata": {"location": "s3://lake/tables/id"}, "metadata-location": metadata_credential["prefix"], "storage-credentials": [credential, metadata_credential]} + for prefix in ("s3://lake/tables/id/", "s3://lake/tables/other/"): + credential["prefix"] = prefix + with self.subTest(prefix=prefix), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=[{"storage-credentials": []}] * 4 + [response] + ) as request, mock.patch.object(pyiceberg_smoke, "verify_vended_credential_data_plane_scope") as scope: + if prefix.endswith("/other/"): + with self.assertRaisesRegex(RuntimeError, "do not match"): + pyiceberg_smoke.run_load_table_delegation_probe(args, mock.Mock()) + scope.assert_not_called() + else: + pyiceberg_smoke.run_load_table_delegation_probe(args, mock.Mock()) + scope.assert_called_once() + self.assertEqual(scope.call_args.args[-1], "s3://lake/tables/id") + self.assertEqual(request.call_args.kwargs["access_delegation"], "remote-signing, vended-credentials") + with mock.patch.object(pyiceberg_smoke, "signed_rest_request", return_value=response): + with self.assertRaisesRegex(RuntimeError, "without exact"): + pyiceberg_smoke.run_load_table_delegation_probe(args, mock.Mock()) + response["storage-credentials"] = [credential, metadata_credential, credential] + with mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=[{}] * 4 + [response] + ), mock.patch.object(pyiceberg_smoke, "verify_vended_credential_data_plane_scope") as scope: + with self.assertRaisesRegex(RuntimeError, "exactly the warehouse"): + pyiceberg_smoke.run_load_table_delegation_probe(args, mock.Mock()) + scope.assert_not_called() + + credential["prefix"] = "s3://lake/tables/id/" + metadata_credential["config"]["s3.session-token"] = "different-session" + response["storage-credentials"] = [credential, metadata_credential] + with mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=[{}] * 4 + [response] + ), mock.patch.object(pyiceberg_smoke, "verify_vended_credential_data_plane_scope") as scope: + with self.assertRaisesRegex(RuntimeError, "same temporary session"): + pyiceberg_smoke.run_load_table_delegation_probe(args, mock.Mock()) + scope.assert_not_called() + + def test_skipped_probes_are_not_recorded_as_full_rest_evidence(self) -> None: + args = self.parse_with_args(["--skip-catalog-api-probes"]) + result = pyiceberg_smoke.SmokeResult("s3://lake/tables/id/metadata/v2.metadata.json", 2, "not-requested", "s3://lake/tables/id", "object", {"direct-rest": "skipped"}) + kwargs = dict(client_version="0.11.1", rustfs_build="test", git_sha="abc123", catalog_backing="object", run_timestamp_utc="2026-09-09T00:00:00Z", operator="test", command="smoke") + record = pyiceberg_smoke.pyiceberg_live_evidence_record(args, result, **kwargs) + self.assertEqual(record["scenario"], "create-append-reload-scan-core") + self.assertEqual(record["catalog_probes"], {"direct-rest": "skipped"}) + kwargs["catalog_backing"] = "durable-strong" + with self.assertRaisesRegex(ValueError, "observed server"): + pyiceberg_smoke.pyiceberg_live_evidence_record(args, result, **kwargs) def test_live_evidence_command_redacts_cli_secrets(self) -> None: command = pyiceberg_smoke.redacted_command( @@ -1129,5 +1407,206 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): self.assertIsNone(INTERNAL_ROADMAP_LABEL_RE.search(str(pyiceberg_smoke.failure_coverage.production_failure_matrix()))) +class SmokeCleanupTest(unittest.TestCase): + def setUp(self) -> None: + self.args = SimpleNamespace(bucket="lake", namespace="sales", table="orders", rest_path="/iceberg") + self.original = { + "metadata-location": "s3://lake/original/metadata/v2.metadata.json", + "metadata": {"table-uuid": "original-id", "location": "s3://lake/original", + "current-schema-id": 0, "schemas": [{"schema-id": 0, "type": "struct", "fields": []}], + "current-snapshot-id": 7, "snapshots": [{"snapshot-id": 7}]}, + } + self.pointer = {"metadata-location": self.original["metadata-location"], "version-token": "token", "generation": 4} + self.replacement = {"metadata": {"table-uuid": "fresh-id", "location": "s3://lake/fresh", "snapshots": []}} + self.entries = {"orders": copy.deepcopy(self.original)} + self.namespace_present = True + self.calls = [] + self.backing = "object" + self.after_cleanup = None + self.after_cleanup_pointer = None + + def request(self, args: object, deps: object, method: str, path: str, body: object = None) -> dict[str, object]: + self.calls.append((method, path, body)) + if path.endswith("/tables/rename"): + source, destination = body["source"]["name"], body["destination"]["name"] + value = self.entries.pop(source) + if self.backing == "object": + self.entries[source] = None # Retained inactive entry, hidden from load/list. + self.entries[destination] = value + return {} + if path.endswith("/metadata-location"): + return copy.deepcopy(self.pointer) + if method == "POST" and path.endswith("/tables"): + self.entries[body["name"]] = copy.deepcopy(self.replacement) + return copy.deepcopy(self.replacement) + name = path.rsplit("/", 1)[-1] + if method == "GET": + value = self.entries.get(name) + if value is None: + raise pyiceberg_smoke.RestRequestError(method, path, 404, "missing") + return copy.deepcopy(value) + if method == "DELETE": + del self.entries[name] + if self.after_cleanup is not None: + self.entries["orders"] = self.after_cleanup + if self.after_cleanup_pointer is not None: + self.pointer = self.after_cleanup_pointer + return {} + raise AssertionError(f"unexpected request: {method} {path}") + + def rename(self) -> None: + with mock.patch.object(pyiceberg_smoke.uuid, "uuid4", return_value=SimpleNamespace(hex="owned")), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=self.request + ): + pyiceberg_smoke.run_table_rename_probe(self.args, mock.Mock(), self.backing) + + def catalog(self) -> object: + catalog = mock.Mock() + catalog.table_exists.side_effect = lambda identifier: self.entries.get(identifier[1]) is not None + catalog.drop_table.side_effect = lambda identifier: self.entries.pop(identifier[1]) + catalog.namespace_exists.side_effect = lambda namespace: self.namespace_present + + def drop_namespace(namespace: str) -> None: + if self.entries: + raise RuntimeError("NamespaceNotEmptyException") + self.namespace_present = False + + catalog.drop_namespace.side_effect = drop_namespace + return catalog + + def test_object_rename_consumes_only_its_tombstone_before_namespace_cleanup(self) -> None: + self.rename() + creates = [call for call in self.calls if call[0] == "POST" and call[1].endswith("/tables")] + deletes = [call for call in self.calls if call[0] == "DELETE"] + self.assertEqual(creates, [("POST", "/iceberg/v1/lake/namespaces/sales/tables", + {"name": "renamed-owned", "schema": self.original["metadata"]["schemas"][0]})]) + self.assertEqual(deletes, [("DELETE", "/iceberg/v1/lake/namespaces/sales/tables/renamed-owned", None)]) + self.assertEqual(self.entries, {"orders": self.original}) + self.assertEqual(self.pointer, {"metadata-location": self.original["metadata-location"], + "version-token": "token", "generation": 4}) + pyiceberg_smoke.cleanup_catalog(self.catalog(), ("sales", "orders")) + self.assertFalse(self.namespace_present) + self.assertEqual(self.entries, {}) + + def test_strong_rename_does_not_create_or_drop_a_cleanup_table(self) -> None: + self.backing = "durable-strong" + self.rename() + self.assertFalse(any(method == "DELETE" or (method == "POST" and path.endswith("/tables")) + for method, path, _ in self.calls)) + pyiceberg_smoke.cleanup_catalog(self.catalog(), ("sales", "orders")) + self.assertFalse(self.namespace_present) + + def test_rename_does_not_sweep_unrelated_inactive_entries(self) -> None: + self.entries["unrelated"] = None + self.rename() + self.assertIn("unrelated", self.entries) + with self.assertRaisesRegex(RuntimeError, "NamespaceNotEmptyException"): + pyiceberg_smoke.cleanup_catalog(self.catalog(), ("sales", "orders")) + self.assertTrue(self.namespace_present) + + def test_rename_rejects_unknown_backing_before_mutation(self) -> None: + self.backing = "unknown" + with self.assertRaisesRegex(RuntimeError, "known backing"): + self.rename() + self.assertEqual(self.calls, []) + + def test_replacement_identity_must_be_new_and_empty(self) -> None: + cases = [("table-uuid", "original-id"), ("table-uuid", ""), ("location", "s3://lake/original"), + ("location", None), ("snapshots", [{"snapshot-id": 7}]), ("current-snapshot-id", 7)] + for key, value in cases: + with self.subTest(key=key, value=value): + self.setUp() + self.replacement["metadata"][key] = value + with self.assertRaisesRegex(RuntimeError, "isolated empty table"): + self.rename() + self.assertFalse(any(method == "DELETE" for method, _, _ in self.calls)) + self.assertEqual(self.entries["orders"], self.original) + + def test_missing_schema_or_commit_state_fails_before_create(self) -> None: + for missing in ("schemas", "metadata-location", "version-token", "generation"): + with self.subTest(missing=missing): + self.setUp() + if missing == "schemas": + del self.entries["orders"]["metadata"][missing] + else: + del self.pointer[missing] + with self.assertRaisesRegex(RuntimeError, "rename cleanup requires"): + self.rename() + self.assertFalse(any(method == "POST" and path.endswith("/tables") for method, path, _ in self.calls)) + + def test_cleanup_detects_original_metadata_or_commit_change(self) -> None: + for changed in ("metadata", "metadata-location", "version-token", "generation"): + with self.subTest(changed=changed): + self.setUp() + if changed == "metadata": + self.after_cleanup = copy.deepcopy(self.original) + self.after_cleanup["metadata"]["table-uuid"] = "changed" + else: + self.after_cleanup_pointer = {**self.pointer, changed: "changed"} + with self.assertRaisesRegex(RuntimeError, "changed the original"): + self.rename() + + def test_cleanup_rejects_namespace_drop_failure_and_does_not_write_evidence(self) -> None: + errors = [RuntimeError("NamespaceNotEmptyException"), PermissionError("403"), + RuntimeError("500: object not found during namespace write"), TimeoutError("timeout")] + for error in errors: + with self.subTest(error=error): + catalog = mock.Mock() + catalog.table_exists.side_effect = [True, False] + catalog.namespace_exists.return_value = True + catalog.drop_namespace.side_effect = error + with mock.patch.object(pyiceberg_smoke, "parse_args", return_value=object()), mock.patch.object( + pyiceberg_smoke, "printed_metadata", return_value=False + ), mock.patch.object(pyiceberg_smoke, "load_runtime_deps"), mock.patch.object( + pyiceberg_smoke, "run_smoke", + side_effect=lambda *args: pyiceberg_smoke.cleanup_catalog(catalog, ("sales", "orders")) + ), mock.patch.object(pyiceberg_smoke, "write_live_evidence") as write, redirect_stderr(StringIO()): + self.assertEqual(pyiceberg_smoke.main(), 1) + write.assert_not_called() + + def test_cleanup_rejects_false_successful_table_or_namespace_drop(self) -> None: + for resource in ("table", "namespace"): + with self.subTest(resource=resource): + catalog = mock.Mock() + catalog.table_exists.side_effect = [True, resource == "table"] + catalog.namespace_exists.return_value = True + with self.assertRaisesRegex(RuntimeError, resource + " remains"): + pyiceberg_smoke.cleanup_catalog(catalog, ("sales", "orders")) + + def test_cleanup_accepts_observed_already_absent_resources(self) -> None: + catalog = mock.Mock() + catalog.table_exists.return_value = False + catalog.namespace_exists.return_value = False + pyiceberg_smoke.cleanup_catalog(catalog, ("sales", "orders")) + catalog.drop_table.assert_not_called() + catalog.drop_namespace.assert_not_called() + + def test_cleanup_read_failures_are_not_absence(self) -> None: + for check in ("table_exists", "namespace_exists"): + with self.subTest(check=check): + catalog = mock.Mock() + catalog.table_exists.return_value = False + getattr(catalog, check).side_effect = TimeoutError("read failed") + with self.assertRaises(TimeoutError): + pyiceberg_smoke.cleanup_catalog(catalog, ("sales", "orders")) + catalog.drop_namespace.assert_not_called() + + def test_rename_transport_and_drop_failure_propagate(self) -> None: + for method, suffix in (("POST", "/tables"), ("DELETE", "/renamed-owned")): + with self.subTest(method=method): + self.setUp() + + def fail(args: object, deps: object, verb: str, path: str, body: object = None) -> dict[str, object]: + if verb == method and path.endswith(suffix): + raise TimeoutError("cleanup interrupted") + return self.request(args, deps, verb, path, body) + + with mock.patch.object(pyiceberg_smoke.uuid, "uuid4", return_value=SimpleNamespace(hex="owned")), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=fail + ), self.assertRaises(TimeoutError): + pyiceberg_smoke.run_table_rename_probe(self.args, mock.Mock(), self.backing) + self.assertEqual(self.entries["orders"], self.original) + + if __name__ == "__main__": unittest.main() diff --git a/scripts/test/delete_recovery_docker.py b/scripts/test/delete_recovery_docker.py new file mode 100644 index 000000000..07f581bc7 --- /dev/null +++ b/scripts/test/delete_recovery_docker.py @@ -0,0 +1,1303 @@ +#!/usr/bin/env python3 +"""Run delete recovery regressions in disposable local Docker containers. + +Requires host Python 3.9+, Docker, and curl with --aws-sigv4. Supply a native +Linux binary compiled with e2e-test-hooks and a Linux image containing Python 3 +and that binary's runtime libraries. All S3 requests target localhost with +synthetic credentials. Container processes run as UID 10001. The harness labels +its resources, refuses existing volumes, preserves JSON reports/server logs, +and removes only containers, networks, and volumes it created. + +Example: + python3 scripts/test/delete_recovery_docker.py --binary /absolute/path/to/rustfs-candidate --baseline-binary /absolute/path/to/rustfs-old --image local-rustfs-validation:latest --artifact-dir /absolute/path/to/new-results smoke prepared1 applied1 decision1 restore returning untouched mixed + +Available cases include single/multi-node process termination, inline/empty +objects, legacy explicit cleanup, versioned and retained objects, UUID prefix +children, corrupted journals, real permission errors, lost receipt responses, +admin bucket/object repair, and deep keys. Use --help for runtime options. +Logical disks share the Docker host filesystem; this tests process/RPC failure +and namespace safety, not independent hardware failure or power-loss durability. +""" +from pathlib import Path +import base64, hashlib, ipaddress, json, os, subprocess, tempfile, threading, time, uuid, xml.etree.ElementTree as ET +from urllib.parse import quote +ROOT = None +PREFIX = None +IMAGE = None +PORT = 19010 +CANDIDATE = None +BASELINE = None +ACCESS = 'recovery-test' +SECRET = 'recovery-test-secret' + +def run(*args, check=True, **kwargs): + return subprocess.run(args, check=check, capture_output=True, **kwargs) + +def docker(*args, check=True): + return run('docker', *args, check=check) + +class S3: + + def __init__(self, port): + self.url = f'http://127.0.0.1:{port}' + + def request(self, method, path='', body=None, headers=None, expected=None, timeout=60): + call = ['curl', '--noproxy', '*', '--silent', '--show-error', '--max-time', str(timeout), '--aws-sigv4', 'aws:amz:us-east-1:s3', '--user', ACCESS + ':' + SECRET, '--request', method] + for (key, value) in (headers or {}).items(): + call += ['--header', key + ': ' + value] + if body is not None: + call += ['--data-binary', '@-'] + with tempfile.NamedTemporaryFile(dir=ROOT) as header_file: + response = run(*call, self.url + '/' + path, '--dump-header', header_file.name, '--write-out', '\n%{http_code}', input=body, check=False) + self.last_headers = {} + for line in Path(header_file.name).read_text(errors='replace').splitlines(): + if ':' in line: + (key, value) = line.split(':', 1) + self.last_headers[key.lower()] = value.strip() + if response.returncode: + raise RuntimeError(response.stderr.decode(errors='replace')) + (payload, status) = response.stdout.rsplit(b'\n', 1) + code = int(status) + if expected is not None and code not in ((expected,) if isinstance(expected, int) else expected): + raise AssertionError(f'{method} {path}: {code}, expected {expected}: {payload[:1500]!r}') + return (code, payload) + + def put_bucket(self, bucket, lock=False): + headers = {'x-amz-bucket-object-lock-enabled': 'true'} if lock else {} + return self.request('PUT', bucket, headers=headers, expected=200) + + def put(self, bucket, key, body, headers=None): + return self.request('PUT', bucket + '/' + key, body, headers, 200) + + def get(self, bucket, key): + return self.request('GET', bucket + '/' + key, expected=200)[1] + + def delete(self, bucket, key, expected=(204, 200)): + return self.request('DELETE', bucket + '/' + key, expected=expected) + + def delete_bucket(self, bucket, expected=204): + return self.request('DELETE', bucket, expected=expected) + + def list(self, bucket): + (_, body) = self.request('GET', bucket + '?list-type=2', expected=200) + root = ET.fromstring(body) + return [node.text for node in root.iter() if node.tag.rsplit('}', 1)[-1] == 'Key'] + + def versioning(self, bucket, state='Enabled'): + xml = f'{state}'.encode() + return self.request('PUT', bucket + '?versioning', xml, {'Content-Type': 'application/xml'}, 200) + + def batch_delete(self, bucket, keys): + xml = ET.Element('Delete') + for key in keys: + ET.SubElement(ET.SubElement(xml, 'Object'), 'Key').text = key + data = ET.tostring(xml) + (status, body) = self.request('POST', bucket + '?delete', data, {'Content-Type': 'application/xml', 'Content-MD5': base64.b64encode(hashlib.md5(data).digest()).decode()}, 200) + response = ET.fromstring(body) + errors = [node for node in response.iter() if node.tag.rsplit('}', 1)[-1] == 'Error'] + assert not errors, f'DeleteObjects returned per-key errors: {body!r}' + return (status, body) + + def complete_multipart(self, bucket, key, parts): + path = bucket + '/' + key + (_, response) = self.request('POST', path + '?uploads', expected=200) + upload = next(node.text for node in ET.fromstring(response).iter() if node.tag.rsplit('}', 1)[-1] == 'UploadId') + assert upload, 'native multipart initiation must return an upload ID' + query = '?uploadId=' + quote(upload, safe='') + complete = ET.Element('CompleteMultipartUpload') + for number, body in enumerate(parts, 1): + self.request('PUT', path + query + '&partNumber=' + str(number), body, expected=200) + etag = self.last_headers.get('etag') + assert etag, 'native UploadPart must return its ETag' + part = ET.SubElement(complete, 'Part') + ET.SubElement(part, 'PartNumber').text = str(number) + ET.SubElement(part, 'ETag').text = etag + (_, response) = self.request('POST', path + query, ET.tostring(complete), {'Content-Type': 'application/xml'}, 200) + assert ET.fromstring(response).tag.rsplit('}', 1)[-1] == 'CompleteMultipartUploadResult', response + return upload + +class Cluster: + + def __init__(self, case, nodes=1, drives=4, extra_env=None, binary_by_node=None): + self.case = case + self.nodes = nodes + self.drives = drives + self.port = PORT + self.binary = 'candidate' + self.binary_by_node = binary_by_node or {} + self.names = [f'{PREFIX}-{case}-n{i}' for i in range(1, nodes + 1)] + self.network = f'{PREFIX}-{case}-net' + self.env = extra_env or {} + self.created_names = [] + self.created_network = False + self.clients = [S3(self.port + i) for i in range(nodes)] + self.volumes = [] + self.run_args = {} + self.barriers = [ROOT / 'barriers' / case / f'n{i}' for i in range(1, nodes + 1)] + for directory in self.barriers: + directory.mkdir(parents=True, exist_ok=True) + directory.chmod(511) + + def create(self): + docker('network', 'create', '--label', 'rustfs.delete-recovery-test=' + PREFIX, self.network) + self.created_network = True + configuration = json.loads(docker('network', 'inspect', self.network).stdout)[0]['IPAM']['Config'] + subnet = next(ipaddress.ip_network(value['Subnet']) for value in configuration if ':' not in value['Subnet']) + addresses = [str(subnet[index + 2]) for index in range(self.nodes)] + urls = [] + for name in self.names: + for drive in range(1, self.drives + 1): + urls.append(f'http://{name}:9000/data/d{drive}') + for (idx, name) in enumerate(self.names): + args = ['run', '-d', '--user', '10001:10001', '--name', name, '--network', self.network, '--hostname', name, '-p', f'127.0.0.1:{self.port + idx}:9000', '--label', 'rustfs.delete-recovery-test=' + PREFIX, '-v', str(CANDIDATE) + ':/binaries/candidate:ro', '-v', str(self.barriers[idx]) + ':/barrier'] + # Stopped peers must still resolve while the remaining quorum restarts. + args += ['--ip', addresses[idx]] + for peer, address in zip(self.names, addresses): + args += ['--add-host', peer + ':' + address] + if BASELINE is not None: + args += ['-v', str(BASELINE) + ':/binaries/baseline:ro'] + data_mounts = [] + for drive in range(1, self.drives + 1): + volume = f'{name}-d{drive}' + if docker('volume', 'inspect', volume, check=False).returncode == 0: + raise RuntimeError('refusing existing volume: ' + volume) + docker('volume', 'create', '--label', 'rustfs.delete-recovery-test=' + PREFIX, volume) + self.volumes.append(volume) + args += ['-v', f'{volume}:/data/d{drive}'] + data_mounts += ['-v', f'{volume}:/data/d{drive}'] + docker('run', '--rm', *data_mounts, IMAGE, 'python3', '-c', "from pathlib import Path; import os; [os.chown(p,10001,10001) for p in Path('/data').glob('d*')]") + values = {'RUSTFS_VOLUMES': ' '.join(urls) if self.nodes > 1 else ' '.join((f'/data/d{d}' for d in range(1, self.drives + 1))), 'RUSTFS_ADDRESS': '0.0.0.0:9000', 'RUSTFS_CONSOLE_ENABLE': 'false', 'RUSTFS_ACCESS_KEY': ACCESS, 'RUSTFS_SECRET_KEY': SECRET, 'RUSTFS_OBS_LOG_DIRECTORY': '/tmp/logs', 'RUSTFS_OBS_LOGGER_LEVEL': 'info', 'RUSTFS_OBS_LOG_STDOUT_ENABLED': 'true', 'RUSTFS_DELETE_RECOVERY_BARRIER_DIR': '/barrier', 'RUSTFS_UNSAFE_BYPASS_DISK_CHECK': 'true', 'RUSTFS_DURABILITY_MODE': 'strict', **self.env} + for (key, value) in values.items(): + if value is not None: + args += ['-e', key + '=' + value] + self.run_args[idx] = [*args, IMAGE, '/binaries/' + self.binary_by_node.get(idx, self.binary)] + launch = docker(*self.run_args[idx], check=False) + if launch.returncode: + label = docker('inspect', '--format', '{{index .Config.Labels "rustfs.delete-recovery-test"}}', name, check=False) + if label.returncode == 0 and label.stdout.decode().strip() == PREFIX: + self.created_names.append(name) + raise RuntimeError('Docker launch failed: ' + launch.stderr.decode(errors='replace')) + self.created_names.append(name) + self.ready() + + def ready(self, timeout=180, indices=None): + deadline = time.monotonic() + timeout + pending = set(range(self.nodes) if indices is None else indices) + while pending and time.monotonic() < deadline: + for idx in list(pending): + try: + (code, _) = self.clients[idx].request('GET', '', timeout=2) + if code in (200, 403): + pending.remove(idx) + except Exception: + pass + if pending: + for idx in pending: + state = docker('inspect', '--format', '{{.State.Running}}', self.names[idx], check=False) + if state.returncode == 0 and state.stdout.strip() != b'true': + self.logs() + raise RuntimeError('server exited during startup: ' + self.names[idx]) + time.sleep(0.2) + if pending: + self.logs() + raise TimeoutError('cluster not ready: ' + str(pending)) + + def kill(self, index=0): + docker('kill', '--signal', 'KILL', self.names[index]) + + def restart(self, index=None): + for i in range(self.nodes) if index is None else [index]: + docker('start', self.names[i]) + self.ready() + + def replace_node(self, index, binary): + docker('kill', '--signal', 'KILL', self.names[index], check=False) + docker('rm', self.names[index]) + args = self.run_args[index].copy() + args[-1] = '/binaries/' + binary + docker(*args) + self.run_args[index] = args + self.ready(indices=[index]) + + def logs(self): + for name in self.created_names: + response = docker('logs', name, check=False) + (ROOT / (name + '.log')).write_bytes(response.stdout + response.stderr) + + def inspect_data(self, code, indices=None): + """Access only this cluster's named volumes, including stopped peers.""" + result = [] + for index, name in enumerate(self.names): + if indices is not None and index not in indices: + continue + response = docker('run', '--rm', '--volumes-from', name, IMAGE, 'python3', '-c', code) + result.append(json.loads(response.stdout)) + return result + + def remove(self): + """Keep logs and delete only resources successfully created here.""" + self.logs() + for name in self.created_names: + docker('rm', '-f', name, check=False) + for volume in self.volumes: + docker('volume', 'rm', volume, check=False) + if self.created_network: + docker('network', 'rm', self.network, check=False) + +def smoke(): + cluster = Cluster('smoke', drives=1) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'recovery-smoke' + client.put_bucket(bucket) + for (key, size) in [('zero', 0), ('inline', 4096), ('external', 2 * 1024 * 1024)]: + payload = bytes(range(256)) * (size // 256) + client.put(bucket, key, payload) + assert client.get(bucket, key) == payload + client.batch_delete(bucket, ['zero', 'inline', 'external']) + assert client.list(bucket) == [] + client.delete_bucket(bucket) + return {'case': 's3-smoke', 'status': 'passed', 'topology': '1 node / 1 named local volume'} + finally: + cluster.remove() + +def seed_legacy(cluster, bucket, key, shape): + transaction = str(uuid.uuid4()) + if shape == 'alias': + directories = {path.split('/')[0] for node in residue_inventory(cluster, bucket, key) for drive in node for path in drive['files'] if path.endswith('/part.1')} + assert len(directories) == 1, 'alias fixture requires one shared data-directory generation' + transaction = directories.pop() + code = "from pathlib import Path\nimport hashlib,json,os,uuid\nbucket=%r; key=%r; shape=%r; transaction=%r\nresult=[]\nfor drive in sorted(Path('/data').glob('d*')):\n root=drive/bucket/key\n original=root/'xl.meta'\n if not original.is_file(): continue\n data=original.read_bytes(); backup=root/transaction/'xl.meta.bkp'\n backup.parent.mkdir(exist_ok=shape=='alias'); backup.write_bytes(data); os.chown(backup.parent,10001,10001); os.chown(backup,10001,10001)\n dirs=[p for p in root.iterdir() if p.is_dir() and (p.name!=transaction or shape=='alias') and any(p.glob('part.*'))]\n if shape in ('reserved','alias'):\n assert dirs, 'external fixture must contain erasure parts'\n for d in dirs:\n marker=d/('reserve-delete-data.'+transaction); marker.write_bytes(b''); os.chown(marker,10001,10001)\n elif shape=='backup-only':\n assert not dirs, 'inline fixture must not contain external parts'\n else: raise AssertionError(shape)\n original.unlink()\n owned=[root,*root.rglob('*')]\n assert all(p.stat().st_uid==10001 and p.stat().st_gid==10001 for p in owned), [(str(p),p.stat().st_uid,p.stat().st_gid) for p in owned]\n result.append({'drive':drive.name,'transaction':transaction,'backup_sha256':hashlib.sha256(data).hexdigest(),'data_dirs':[p.name for p in dirs],'fixture_owner':'10001:10001','fixture_modes':{str(p.relative_to(root)):oct(p.stat().st_mode & 0o777) for p in owned}})\nprint(json.dumps(result))\n" % (bucket, key, shape, transaction) + return cluster.inspect_data(code) + +def residue_inventory(cluster, bucket, key): + return cluster.inspect_data("from pathlib import Path\nimport hashlib,json\nresult=[]\nfor drive in sorted(Path('/data').glob('d*')):\n root=drive/%r/%r\n result.append({'drive':drive.name,'exists':root.exists(),'files':{str(p.relative_to(root)):hashlib.sha256(p.read_bytes()).hexdigest() for p in root.rglob('*') if p.is_file()}})\nprint(json.dumps(result))\n" % (bucket, key)) + +def legacy_retry(nodes=1): + case = 'legacy' + str(nodes) + cluster = Cluster(case, nodes=nodes, drives=1 if nodes == 1 else 2) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'legacy-recovery' + client.put_bucket(bucket) + original = {key: bytes(range(256)) * (size // 256) for (key, size) in [('inline', 4096), ('reserved', 2 * 1024 * 1024)]} + for (key, payload) in original.items(): + client.put(bucket, key, payload) + assert client.get(bucket, key) == payload + for i in range(nodes): + cluster.kill(i) + fixtures = {key: seed_legacy(cluster, bucket, key, 'backup-only' if key == 'inline' else 'reserved') for key in original} + before = {key: residue_inventory(cluster, bucket, key) for key in original} + cluster.restart() + time.sleep(2) + after = {key: residue_inventory(cluster, bucket, key) for key in original} + assert before == after, 'automatic recovery must not guess an unjournaled historical outcome' + client.batch_delete(bucket, list(original)) + for key in original: + current = residue_inventory(cluster, bucket, key) + assert all((not drive['exists'] for node in current for drive in node)), f'current explicit DELETE must reclaim {key}: {current}' + assert client.list(bucket) == [] + client.delete_bucket(bucket) + return {'case': case, 'status': 'passed', 'fixtures': fixtures, 'topology': f'{nodes} nodes'} + finally: + cluster.remove() + +def arm_barrier(cluster, index, bucket, key, stage, disk=None): + nonce = str(uuid.uuid4()) + arm = {'nonce': nonce, 'bucket': bucket, 'object': key, 'stage': stage} + if disk is not None: + arm['disk'] = disk + (cluster.barriers[index] / 'arm.json').write_text(json.dumps(arm)) + return nonce + +def reached_barrier(cluster, index, nonce, timeout=45): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + for path in cluster.barriers[index].glob('reached.' + nonce + '.*.json'): + try: + return json.loads(path.read_text()) + except (json.JSONDecodeError, OSError): + pass + time.sleep(0.02) + cluster.logs() + raise TimeoutError('crash stage not reached: ' + nonce) + +def clear_barriers(cluster): + for directory in cluster.barriers: + (directory / 'arm.json').unlink(missing_ok=True) + +def async_delete(client, bucket, key): + result = {} + + def target(): + try: + result['response'] = client.request('DELETE', bucket + '/' + key, timeout=90) + except Exception as error: + result['exception'] = str(error) + thread = threading.Thread(target=target, daemon=True) + thread.start() + return (thread, result) + +def wait_inventory(cluster, bucket, key, predicate, timeout=60): + deadline = time.monotonic() + timeout + last = None + while time.monotonic() < deadline: + last = residue_inventory(cluster, bucket, key) + if predicate(last): + return last + time.sleep(0.1) + raise AssertionError('recovery did not converge: ' + json.dumps(last)) + +def process_crash(stage, nodes=1, size=2 * 1024 * 1024): + case = ('inline-' if size == 4096 else 'zero-' if size == 0 else 'kill-') + stage.lower() + '-' + str(nodes) + cluster = Cluster(case, nodes=nodes, drives=4 if nodes == 1 else 2) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'journal-crash' + key = 'payload' + client.put_bucket(bucket) + payload = bytes(range(256)) * (size // 256) + client.put(bucket, key, payload) + assert client.get(bucket, key) == payload + original = residue_inventory(cluster, bucket, key) + nonce = arm_barrier(cluster, 0, bucket, key, stage, disk='/data/d1') + (thread, request) = async_delete(client, bucket, key) + observation = reached_barrier(cluster, 0, nonce) + for index in range(nodes): + cluster.kill(index) + thread.join(5) + clear_barriers(cluster) + cluster.restart() + if stage == 'DECISION': + wait_inventory(cluster, bucket, key, lambda inventory: all((not drive['exists'] for node in inventory for drive in node))) + assert client.list(bucket) == [] + client.delete_bucket(bucket) + else: + wait_inventory(cluster, bucket, key, lambda inventory: inventory == original) + assert client.get(bucket, key) == payload + client.delete(bucket, key) + client.delete_bucket(bucket) + return {'case': case, 'status': 'passed', 'observation': observation, 'request': {key: str(value) for (key, value) in request.items()}} + finally: + cluster.remove() + +def rollback_crash(): + cluster = Cluster('kill-restore', drives=4) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'rollback-restart' + key = 'payload' + client.put_bucket(bucket) + payload = bytes(range(256)) * 8192 + client.put(bucket, key, payload) + original = residue_inventory(cluster, bucket, key) + first_nonce = arm_barrier(cluster, 0, bucket, key, 'APPLIED', disk='/data/d1') + (thread, request) = async_delete(client, bucket, key) + first = reached_barrier(cluster, 0, first_nonce) + cluster.kill() + thread.join(5) + clear_barriers(cluster) + second_nonce = arm_barrier(cluster, 0, bucket, key, 'RESTORE', disk='/data/d1') + docker('start', cluster.names[0]) + second = reached_barrier(cluster, 0, second_nonce) + cluster.kill() + clear_barriers(cluster) + cluster.restart() + wait_inventory(cluster, bucket, key, lambda inventory: inventory == original) + assert client.get(bucket, key) == payload + client.delete(bucket, key) + client.delete_bucket(bucket) + return {'case': 'rollback-second-process-kill', 'status': 'passed', 'observations': [first, second]} + finally: + cluster.remove() + +def versioned_crash(stage): + cluster = Cluster('versions-' + stage.lower(), drives=4) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'versioned-recovery' + key = 'history' + client.put_bucket(bucket) + client.versioning(bucket) + first = b'old-version-' * 200000 + second = b'new-version-' * 300 + client.put(bucket, key, first) + v1 = client.last_headers.get('x-amz-version-id') + assert v1 and v1 != 'null' + client.put(bucket, key, second) + v2 = client.last_headers.get('x-amz-version-id') + assert v2 and v2 != v1 + assert client.get(bucket, key + '?versionId=' + v1) == first + assert client.get(bucket, key) == second + original = residue_inventory(cluster, bucket, key) + nonce = arm_barrier(cluster, 0, bucket, key, stage, disk='/data/d1') + (thread, request) = async_delete(client, bucket, key + '?versionId=' + v1) + observation = reached_barrier(cluster, 0, nonce) + cluster.kill() + thread.join(5) + clear_barriers(cluster) + cluster.restart() + if stage == 'APPLIED': + wait_inventory(cluster, bucket, key, lambda inventory: inventory == original) + assert client.get(bucket, key + '?versionId=' + v1) == first + client.delete(bucket, key + '?versionId=' + v1) + else: + deadline = time.monotonic() + 60 + while time.monotonic() < deadline: + (code, _) = client.request('GET', bucket + '/' + key + '?versionId=' + v1) + if code == 404: + break + time.sleep(0.1) + else: + raise AssertionError('committed version deletion was resurrected') + assert client.get(bucket, key) == second + assert client.get(bucket, key + '?versionId=' + v2) == second + client.delete(bucket, key + '?versionId=' + v2) + client.delete_bucket(bucket) + return {'case': 'versioned-' + stage.lower(), 'status': 'passed', 'observation': observation, 'deleted_version': v1, 'retained_version': v2} + finally: + cluster.remove() + +def protected_version(): + import datetime + cluster = Cluster('retention', drives=4) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'protected-recovery' + key = 'retained' + client.put_bucket(bucket, lock=True) + client.versioning(bucket) + until = (datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(days=1)).strftime('%Y-%m-%dT%H:%M:%SZ') + payload = b'compliance-and-legal-hold' * 1024 + client.put(bucket, key, payload, {'x-amz-object-lock-mode': 'COMPLIANCE', 'x-amz-object-lock-retain-until-date': until, 'x-amz-object-lock-legal-hold': 'ON'}) + version = client.last_headers.get('x-amz-version-id') + assert version and version != 'null' + original = residue_inventory(cluster, bucket, key) + client.delete(bucket, key + '?versionId=' + version, expected=403) + assert residue_inventory(cluster, bucket, key) == original + cluster.kill() + cluster.restart() + assert client.get(bucket, key + '?versionId=' + version) == payload + assert residue_inventory(cluster, bucket, key) == original + client.delete_bucket(bucket, expected=409) + return {'case': 'compliance-legal-hold-restart', 'status': 'passed', 'retained_version': version} + finally: + cluster.remove() + +def returning_drive_new_generation(): + cluster = Cluster('returning-new-generation', nodes=4, drives=2) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'returning-drive' + key = 'payload' + client.put_bucket(bucket) + old = b'old-payload-' * 200000 + new = b'new-payload-' * 180000 + client.put(bucket, key, old) + assert client.get(bucket, key) == old + nonce = arm_barrier(cluster, 0, bucket, key, 'APPLIED', disk='/data/d1') + (thread, request) = async_delete(client, bucket, key) + observation = reached_barrier(cluster, 0, nonce) + for index in range(4): + cluster.kill(index) + thread.join(5) + clear_barriers(cluster) + for index in range(3): + docker('start', cluster.names[index]) + cluster.ready(indices=range(3)) + client.put(bucket, key, new) + assert client.get(bucket, key) == new + client.delete(bucket, key) + docker('start', cluster.names[3]) + cluster.ready() + wait_inventory(cluster, bucket, key, lambda inventory: all((not drive['exists'] for node in inventory for drive in node)), timeout=90) + client.request('GET', bucket + '/' + key, expected=404) + assert client.list(bucket) == [] + client.delete_bucket(bucket) + return {'case': 'old-prepared-new-put-new-delete-returning-peer', 'status': 'passed', 'observation': observation} + finally: + cluster.remove() + +def mixed_versions(): + cluster = Cluster('mixed', nodes=4, drives=2, binary_by_node={3: 'baseline'}) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'mixed-recovery' + key = 'payload' + payload = b'mixed-version-data' * 100000 + client.put_bucket(bucket) + client.put(bucket, key, payload) + assert client.get(bucket, key) == payload + original = residue_inventory(cluster, bucket, key) + results = [] + for initiator in (0, 3): + (code, body) = cluster.clients[initiator].request('DELETE', bucket + '/' + key) + assert code >= 400, f'mixed initiator{initiator} must fail closed: {code} {body!r}' + if initiator == 0: + assert code == 503 and b'ServiceUnavailable' in body, 'modern mixed-version rejection must remain retryable' + after = residue_inventory(cluster, bucket, key) + (ROOT / f'mixed-initiator-{initiator}-inventory.json').write_text(json.dumps({'before': original, 'after': after}, indent=2)) + assert after == original, 'mixed rejection must preserve exact metadata/shards after rollback' + assert client.get(bucket, key) == payload + results.append({'initiator': initiator, 'delete_status': code}) + cluster.replace_node(3, 'candidate') + client.delete(bucket, key) + client.delete_bucket(bucket) + return {'case': 'three-current-one-pre-extension-peer-both-initiators', 'status': 'passed', 'results': results} + finally: + cluster.remove() + +def permission_fault(): + cluster = Cluster('permission-error', drives=4) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'permission-fault' + key = 'payload' + payload = b'permission-test' * 150000 + client.put_bucket(bucket) + client.put(bucket, key, payload) + assert client.get(bucket, key) == payload + nonce = arm_barrier(cluster, 0, bucket, key, 'APPLIED', disk='/data/d1') + (thread, request) = async_delete(client, bucket, key) + observation = reached_barrier(cluster, 0, nonce) + transaction = str(uuid.UUID(observation['transaction'])) + code = "from pathlib import Path; import json,os; p=Path('/data/d1')/%r/%r/%r; os.chmod(p,0o555); print(json.dumps({'path':str(p),'mode':oct(p.stat().st_mode&0o777)}))" % (bucket, key, transaction) + fault = cluster.inspect_data(code) + (cluster.barriers[0] / ('release.' + nonce)).write_bytes(b'') + thread.join(30) + assert not thread.is_alive(), 'permission failure must return rather than hang' + clear_barriers(cluster) + current = residue_inventory(cluster, bucket, key) + assert any((drive['exists'] for node in current for drive in node)) + client.delete_bucket(bucket, expected=409) + repair = "from pathlib import Path; import json,os; p=Path('/data/d1')/%r/%r/%r; os.chmod(p,0o755); print(json.dumps({'repaired':True}))" % (bucket, key, transaction) + cluster.inspect_data(repair) + cluster.kill() + cluster.restart() + wait_inventory(cluster, bucket, key, lambda inventory: all((not drive['exists'] for node in inventory for drive in node))) + client.request('GET', bucket + '/' + key, expected=404) + client.delete_bucket(bucket) + return {'case': 'actual-eacces-decision-publication', 'status': 'passed', 'fault': fault, 'request': {key: str(value) for (key, value) in request.items()}, 'observation': observation} + finally: + cluster.remove() + +def injected_error(stage): + cluster = Cluster('injected-' + stage.lower(), drives=4) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'injected-fault' + key = 'payload' + payload = b'injected-error' * 170000 + client.put_bucket(bucket) + client.put(bucket, key, payload) + nonce = arm_barrier(cluster, 0, bucket, key, stage, disk='/data/d1') + (thread, request) = async_delete(client, bucket, key) + observation = reached_barrier(cluster, 0, nonce) + (cluster.barriers[0] / ('release.' + nonce)).write_bytes(b'io_error') + thread.join(30) + assert not thread.is_alive(), 'injected error must finish the caller' + clear_barriers(cluster) + cluster.kill() + cluster.restart() + if stage == 'PREPARED': + (code, body) = client.request('GET', bucket + '/' + key) + if code == 200: + assert body == payload + client.delete(bucket, key) + else: + assert code == 404 + wait_inventory(cluster, bucket, key, lambda inventory: all((not drive['exists'] for node in inventory for drive in node))) + client.request('GET', bucket + '/' + key, expected=404) + client.delete_bucket(bucket) + return {'case': 'post-persist-' + stage.lower() + '-io-error', 'status': 'passed', 'observation': observation, 'request': {key: str(value) for (key, value) in request.items()}} + finally: + cluster.remove() + +def missing_version_returning_peer(): + cluster = Cluster('returning-specific-version', nodes=4, drives=2) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'missing-version-recovery' + key = 'history' + client.put_bucket(bucket) + client.versioning(bucket) + old = b'first-version' * 160000 + newer = b'latest-version' * 1000 + client.put(bucket, key, old) + v1 = client.last_headers['x-amz-version-id'] + client.put(bucket, key, newer) + v2 = client.last_headers['x-amz-version-id'] + nonce = arm_barrier(cluster, 0, bucket, key, 'APPLIED', disk='/data/d1') + (thread, request) = async_delete(client, bucket, key + '?versionId=' + v1) + observation = reached_barrier(cluster, 0, nonce) + for index in range(4): + cluster.kill(index) + thread.join(5) + clear_barriers(cluster) + for index in range(3): + docker('start', cluster.names[index]) + cluster.ready(indices=range(3)) + (code, body) = client.request('DELETE', bucket + '/' + key + '?versionId=' + v1) + initial_delete_status = code + assert code >= 400, f'missing version with unresolved offline journal must not acknowledge204: {code} {body!r}' + docker('start', cluster.names[3]) + cluster.ready() + deadline = time.monotonic() + 90 + while time.monotonic() < deadline: + (code, body) = client.request('GET', bucket + '/' + key + '?versionId=' + v1) + if code == 200: + assert body == old + break + time.sleep(0.2) + else: + raise AssertionError('undecided historical version did not recover after peer returned') + client.delete(bucket, key + '?versionId=' + v1) + client.request('GET', bucket + '/' + key + '?versionId=' + v1, expected=404) + assert client.get(bucket, key) == newer + client.delete(bucket, key + '?versionId=' + v2) + client.delete_bucket(bucket) + return {'case': 'missing-specific-version-offline-fence', 'status': 'passed', 'observation': observation, 'initial_delete_status': initial_delete_status} + finally: + cluster.remove() + +def untouched_returning_peer(): + cluster = Cluster('untouched-returning', nodes=4, drives=2) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'untouched-peer' + key = 'payload' + payload = b'untouched-original' * 150000 + client.put_bucket(bucket) + client.put(bucket, key, payload) + assert client.get(bucket, key) == payload + original = residue_inventory(cluster, bucket, key) + cluster.kill(3) + client.delete(bucket, key) + after = residue_inventory(cluster, bucket, key) + assert after[3] == original[3], 'offline unprepared peer must remain byte-for-byte untouched' + client.delete_bucket(bucket, expected=409) + cluster.restart(3) + wait_inventory(cluster, bucket, key, lambda inventory: all((not drive['exists'] for node in inventory for drive in node)), timeout=90) + client.request('GET', bucket + '/' + key, expected=404) + client.delete_bucket(bucket) + return {'case': 'acknowledged-delete-untouched-offline-peer-return', 'status': 'passed'} + finally: + cluster.remove() + +def uuid_prefix_crash(): + cluster = Cluster('uuid-prefix', drives=4) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'uuid-prefix' + key = 'parent' + payload = b'parent-payload' * 160000 + client.put_bucket(bucket) + client.put(bucket, key, payload) + inventory = residue_inventory(cluster, bucket, key) + part = next((name for name in inventory[0][0]['files'] if name.endswith('/part.1'))) + data_dir = str(uuid.UUID(part.split('/')[0])) + child = key + '/' + data_dir + '/child' + child_payload = b'preserve-live-prefix-child' * 800 + client.put(bucket, child, child_payload) + assert client.get(bucket, child) == child_payload + nonce = arm_barrier(cluster, 0, bucket, key, 'DECISION', disk='/data/d1') + (thread, request) = async_delete(client, bucket, key) + observation = reached_barrier(cluster, 0, nonce) + cluster.kill() + thread.join(5) + clear_barriers(cluster) + cluster.restart() + assert client.get(bucket, child) == child_payload, 'recovery must preserve actual data-dir UUID-prefix descendants' + client.request('GET', bucket + '/' + key, expected=404) + client.delete_bucket(bucket, expected=409) + client.delete(bucket, child) + client.delete(bucket, key) + client.delete_bucket(bucket) + return {'case': 'committed-delete-live-uuid-data-prefix', 'status': 'passed', 'observation': observation, 'protected_child': child} + finally: + cluster.remove() + +def corruption_repair(): + cluster = Cluster('corrupt-journal', drives=4) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'corrupt-journal' + key = 'payload' + payload = b'original-protected-data' * 100000 + client.put_bucket(bucket) + client.put(bucket, key, payload) + nonce = arm_barrier(cluster, 0, bucket, key, 'APPLIED', disk='/data/d1') + (thread, request) = async_delete(client, bucket, key) + observation = reached_barrier(cluster, 0, nonce) + cluster.kill() + thread.join(5) + clear_barriers(cluster) + transaction = str(uuid.UUID(observation['transaction'])) + code = "from pathlib import Path; import base64,json; p=Path('/data/d1')/%r/%r/%r/'delete-prepared'; old=p.read_bytes(); p.write_bytes(b'corrupt-record'); print(json.dumps({'original':base64.b64encode(old).decode()}))" % (bucket, key, transaction) + saved = cluster.inspect_data(code)[0]['original'] + before = residue_inventory(cluster, bucket, key) + cluster.restart() + time.sleep(2) + assert residue_inventory(cluster, bucket, key) == before, 'corrupt authority must preserve all backup/payload files' + (code, body) = client.request('DELETE', bucket + '/' + key) + assert code >= 400, f'corrupt hidden transaction must not acknowledge cleanup: {code} {body!r}' + assert residue_inventory(cluster, bucket, key) == before + repair = "from pathlib import Path; import base64,json; p=Path('/data/d1')/%r/%r/%r/'delete-prepared'; p.write_bytes(base64.b64decode(%r)); print(json.dumps({'repaired':True}))" % (bucket, key, transaction, saved) + cluster.inspect_data(repair) + cluster.kill() + cluster.restart() + deadline = time.monotonic() + 60 + while time.monotonic() < deadline: + (code, body) = client.request('GET', bucket + '/' + key) + if code == 200: + assert body == payload + break + time.sleep(0.1) + else: + raise AssertionError('valid original journal must recover original bytes after repair') + client.delete(bucket, key) + client.delete_bucket(bucket) + return {'case': 'corrupt-journal-fails-closed-then-repair', 'status': 'passed', 'observation': observation} + finally: + cluster.remove() + +def admin_heal(client, bucket, dry_run=False, remove=False, key=None, recursive=None): + path = 'rustfs/admin/v3/heal/' + bucket + ('/' + key if key else '') + options = {'recursive': key is None if recursive is None else recursive, 'dryRun': dry_run, 'remove': remove, 'recreate': False, 'scanMode': 1, 'updateParity': False, 'nolock': False, 'readRepair': False} + (_, body) = client.request('POST', path, json.dumps(options).encode(), {'Content-Type': 'application/json'}, 200) + token = json.loads(body)['clientToken'] + deadline = time.monotonic() + 90 + + def states(value): + if isinstance(value, dict): + for (k, v) in value.items(): + if k == 'state' and isinstance(v, str): + yield v.lower() + yield from states(v) + elif isinstance(value, list): + for v in value: + yield from states(v) + while time.monotonic() < deadline: + (_, body) = client.request('POST', path + '?clientToken=' + token, headers={'Content-Type': 'application/json'}, expected=200) + result = json.loads(body) + with (ROOT / 'admin-responses.jsonl').open('a') as report: + report.write(json.dumps({'bucket': bucket, 'key': key, 'dry_run': dry_run, 'remove': remove, 'recursive': options['recursive'], 'response': result}) + '\n') + status = set(states(result)) + if status & {'failed', 'completed_with_errors', 'blocked'}: + raise AssertionError('admin heal failed: ' + json.dumps(result)) + if 'completed' in status or result.get('summary') == 'finished': + return result + time.sleep(0.1) + raise TimeoutError('admin heal did not complete: ' + token) + +def admin_legacy_dry_run(direct=False): + cluster = Cluster('admin-legacy', drives=4) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'admin-legacy' + client.put_bucket(bucket) + for (key, payload) in [('inline', b'i' * 4096), ('external', b'e' * (2 * 1024 * 1024))]: + client.put(bucket, key, payload) + cluster.kill() + seed_legacy(cluster, bucket, 'inline', 'backup-only') + seed_legacy(cluster, bucket, 'external', 'reserved') + before = {key: residue_inventory(cluster, bucket, key) for key in ('inline', 'external')} + cluster.restart() + time.sleep(2) + dry = admin_heal(client, bucket, dry_run=True, remove=True) + assert {key: residue_inventory(cluster, bucket, key) for key in before} == before, 'dry run must never mutate ambiguous residue' + actual = [admin_heal(client, bucket, dry_run=False, remove=True, key=key) for key in before] if direct else admin_heal(client, bucket, dry_run=False, remove=True) + for key in before: + wait_inventory(cluster, bucket, key, lambda inventory: all((not drive['exists'] for node in inventory for drive in node))) + client.delete_bucket(bucket) + return {'case': 'admin-legacy-dry-run-and-explicit-cleanup', 'status': 'passed', 'dry_run_report': dry, 'cleanup_report': actual} + finally: + cluster.remove() + +def admin_prefix_scoped(): + cluster = Cluster('admin-prefix', drives=4) + try: + cluster.create() + client = cluster.clients[0] + bucket, prefix = 'admin-prefix', 'selected/' + selected = {'selected/inline': b'i' * 4096, 'selected/external': b'e' * (2 * 1024 * 1024)} + outside = {'selected-neighbor/hidden': b'n' * 4096, 'outside/hidden': b'o' * 4096} + live_key, live_body = 'selected/live', b'live-prefix-object-must-survive' + client.put_bucket(bucket) + for key, body in {**selected, **outside, live_key: live_body}.items(): + client.put(bucket, key, body) + assert client.get(bucket, key) == body + cluster.kill() + for key in {**selected, **outside}: + seed_legacy(cluster, bucket, key, 'reserved' if key == 'selected/external' else 'backup-only') + before = {key: residue_inventory(cluster, bucket, key) for key in {**selected, **outside}} + cluster.restart() + dry = admin_heal(client, bucket, dry_run=True, remove=True, key=prefix, recursive=True) + assert any('delete recovery (dry run)' in item.get('detail', '') and 'discovered=2' in item.get('detail', '') for item in dry.get('items', [])), 'prefix dry run must inspect the two matching hidden transactions' + assert {key: residue_inventory(cluster, bucket, key) for key in before} == before, 'prefix dry run must preserve every hidden transaction' + actual = admin_heal(client, bucket, remove=True, key=prefix, recursive=True) + after = {key: residue_inventory(cluster, bucket, key) for key in before} + (ROOT / 'prefix-inventory.json').write_text(json.dumps({'before': before, 'after': after}, indent=2)) + for key in selected: + wait_inventory(cluster, bucket, key, lambda inventory: all(not drive['exists'] for node in inventory for drive in node)) + for key in outside: + assert residue_inventory(cluster, bucket, key) == before[key], f'prefix recovery must preserve outside transaction {key}' + assert client.get(bucket, live_key) == live_body, 'prefix remove must preserve the live object' + assert client.list(bucket) == [live_key] + client.delete(bucket, live_key) + client.delete_bucket(bucket, expected=409) + for key in outside: + client.delete(bucket, key) + client.delete_bucket(bucket) + return {'case': 'admin-prefix-hidden-transactions-only', 'status': 'passed', 'prefix': prefix, 'selected_keys': list(selected), 'protected_keys': list(outside), 'dry_run_report': dry, 'cleanup_report': actual} + finally: + cluster.remove() + +def deep_key_crash(large_stack=False): + extra_env = {'RUSTFS_RUNTIME_THREAD_STACK_SIZE': str(32 * 1024 * 1024)} if large_stack else None + cluster = Cluster('deep-key-large-stack' if large_stack else 'deep-key', drives=4, extra_env=extra_env) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'deep-recovery' + key = '/'.join(['a'] * 300) + '/payload' + payload = b'deep-inline-content' * 200 + client.put_bucket(bucket) + client.put(bucket, key, payload) + original = residue_inventory(cluster, bucket, key) + nonce = arm_barrier(cluster, 0, bucket, key, 'APPLIED', disk='/data/d1') + (thread, request) = async_delete(client, bucket, key) + observation = reached_barrier(cluster, 0, nonce) + cluster.kill() + thread.join(5) + clear_barriers(cluster) + cluster.restart() + wait_inventory(cluster, bucket, key, lambda inventory: inventory == original) + assert client.get(bucket, key) == payload + client.delete(bucket, key) + client.delete_bucket(bucket) + return {'case': 'deep-key-recovery-large-stack' if large_stack else 'deep-key-recovery', 'status': 'passed', 'segments': 301, 'runtime_thread_stack_size': 32 * 1024 * 1024 if large_stack else 'default', 'observation': observation} + finally: + cluster.remove() + +def automatic_clean_restart(): + cluster = Cluster('automatic-clean-restart', drives=4) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'automatic-clean-restart' + key = 'payload' + client.put_bucket(bucket) + client.put(bucket, key, b'automatic-recovery' * 150000) + nonce = arm_barrier(cluster, 0, bucket, key, 'DECISION') + thread, request = async_delete(client, bucket, key) + reached_barrier(cluster, 0, nonce) + deadline = time.monotonic() + 40 + while len(list(cluster.barriers[0].glob('reached.' + nonce + '.*.json'))) < 4: + if time.monotonic() > deadline: + raise TimeoutError('all four receipts must persist before injecting response errors') + time.sleep(0.02) + # Prevent the live retry worker from completing before the clean stop. + # The failure is in task-owned payload directories, never in the proof. + deny = "from pathlib import Path; import json,os; roots=list(Path('/data').glob('d*/%s/%s')); dirs=[p for root in roots for p in root.iterdir() if p.is_dir() and (p/'part.1').is_file()]; assert len(dirs)==4; [os.chmod(p,0) for p in dirs]; print(json.dumps({'protected_payload_dirs':len(dirs)}))" % (bucket, key) + cluster.inspect_data(deny) + (cluster.barriers[0] / ('release.' + nonce)).write_bytes(b'io_error') + thread.join(40) + assert not thread.is_alive(), 'uncertain publication must return to the client' + assert 'response' in request and request['response'][0] >= 400, request + clear_barriers(cluster) + before = residue_inventory(cluster, bucket, key) + assert all(any(path.endswith('delete-quorum-committed') for path in drive['files']) for node in before for drive in node) + docker('stop', '--time', '60', cluster.names[0]) + state = json.loads(docker('inspect', '--format', '{{json .State}}', cluster.names[0]).stdout) + assert state['ExitCode'] == 0 and not state['OOMKilled'], state + repair = "from pathlib import Path; import json,os; roots=list(Path('/data').glob('d*/%s/%s')); dirs=[p for root in roots for p in root.iterdir() if p.is_dir() and (p/'part.1').is_file()]; assert len(dirs)==4; [os.chmod(p,0o755) for p in dirs]; print(json.dumps({'restored_payload_dirs':len(dirs)}))" % (bucket, key) + cluster.inspect_data(repair) + cluster.restart() + # Only readiness and read-only inventory are polled after restart. + # No DELETE or administrator heal may manufacture a new retry hint. + wait_inventory(cluster, bucket, key, lambda inventory: all(not drive['exists'] for node in inventory for drive in node), timeout=90) + client.request('GET', bucket + '/' + key, expected=404) + assert client.list(bucket) == [] + client.delete_bucket(bucket) + return {'case': 'uncertain-publication-clean-restart-automatic-recovery', 'status': 'passed', 'clean_exit_code': state['ExitCode']} + finally: + cluster.remove() + +def receipt_response_loss(): + cluster = Cluster('receipt-response-loss', nodes=4, drives=2) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'receipt-response-loss' + key = 'payload' + payload = b'lost-response-data' * 150000 + client.put_bucket(bucket) + client.put(bucket, key, payload) + nonces = {index: arm_barrier(cluster, index, bucket, key, 'DECISION') for index in (1, 2)} + (thread, request) = async_delete(client, bucket, key) + observations = [] + for (index, nonce) in nonces.items(): + observations.append(reached_barrier(cluster, index, nonce)) + deadline = time.monotonic() + 40 + while len(list(cluster.barriers[index].glob('reached.' + nonce + '.*.json'))) < 2: + if time.monotonic() > deadline: + raise TimeoutError('both drive receipts must persist before peer kill') + time.sleep(0.02) + for index in (1, 2): + cluster.kill(index) + thread.join(75) + assert not thread.is_alive(), 'DELETE must return after peer response loss' + if 'response' in request: + assert request['response'][0] >= 400, 'four online responses cannot satisfy five-drive receipt quorum' + before = residue_inventory(cluster, bucket, key) + assert any(('delete-quorum-committed' in path for node in before for drive in node for path in drive['files'])), 'stored decisions must remain while peers are offline' + clear_barriers(cluster) + for index in (1, 2): + docker('start', cluster.names[index]) + cluster.ready() + wait_inventory(cluster, bucket, key, lambda inventory: all((not drive['exists'] for node in inventory for drive in node)), timeout=90) + client.request('GET', bucket + '/' + key, expected=404) + assert client.list(bucket) == [] + client.delete_bucket(bucket) + return {'case': 'receipt-publication-response-loss', 'status': 'passed', 'observations': observations, 'request': {key: str(value) for (key, value) in request.items()}} + finally: + cluster.remove() + +def retained_decision_newer_undecided(): + cluster = Cluster('inverse-null-history', nodes=4, drives=2) + try: + cluster.create() + client = cluster.clients[0] + (bucket, key) = ('inverse-null-history', 'payload') + older = b'older-generation' * 150000 + newer = b'newer-generation' * 150000 + client.put_bucket(bucket) + client.put(bucket, key, older) + cluster.kill(3) + client.delete(bucket, key) + witness = residue_inventory(cluster, bucket, key) + assert any(('delete-quorum-committed' in path for node in witness[:3] for drive in node for path in drive['files'])), 'offline old peer requires retained committed authority' + client.put(bucket, key, newer) + assert client.get(bucket, key) == newer + nonce = arm_barrier(cluster, 0, bucket, key, 'APPLIED', disk='/data/d1') + (thread, request) = async_delete(client, bucket, key) + observation = reached_barrier(cluster, 0, nonce) + for index in range(3): + cluster.kill(index) + thread.join(5) + clear_barriers(cluster) + cluster.restart() + deadline = time.monotonic() + 90 + while time.monotonic() < deadline: + (status, body) = client.request('GET', bucket + '/' + key) + if status == 200: + assert body == newer, 'older committed nil receipt must never promote a newer undecided generation' + break + time.sleep(0.1) + else: + raise AssertionError('newer undecided generation was not restored after complete peer return') + client.delete(bucket, key) + client.delete_bucket(bucket) + return {'case': 'retained-older-decision-newer-undecided-null', 'status': 'passed', 'observation': observation} + finally: + cluster.remove() + + +def completed_multipart_sidecars(protected=False, crash=False): + case = 'multipart-integrity' if protected else 'multipart-sidecars' + if crash: + case += '-applied' + mode = 'true' if protected else 'false' + cluster = Cluster(case, drives=4, extra_env={'RUSTFS_SHARD_INTEGRITY_WRITE': mode, 'RUSTFS_SHARD_INTEGRITY_FLEET_CONFIRMED': mode}) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'completed-multipart' + key = 'payload' + parts = [bytes(range(256)) * 20480, bytes(reversed(range(256))) * 4096] + payload = b''.join(parts) + client.put_bucket(bucket) + upload = client.complete_multipart(bucket, key, parts) + assert client.get(bucket, key) == payload, 'complete MPU must expose all uploaded bytes' + original = residue_inventory(cluster, bucket, key) + (ROOT / 'multipart-original-inventory.json').write_text(json.dumps(original, indent=2)) + for node in original: + for drive in node: + assert drive['exists'] and 'xl.meta' in drive['files'], 'complete MPU must publish metadata on every drive' + for number in (1, 2): + sidecars = [name for name in drive['files'] if Path(name).name == f'part.{number}.meta'] + assert len(sidecars) == 1, 'native completed MPU must retain each real part metadata sidecar' + prefix = f'part.{number}.integrity.' + proofs = [name for name in drive['files'] if Path(name).name.startswith(prefix)] + assert len(proofs) == (1 if protected else 0), 'native completion must retain the selected integrity format' + for proof in proofs: + assert uuid.UUID(Path(proof).name[len(prefix):]).int != 0, 'integrity proof generation must be non-nil' + observation = None + if crash: + nonce = arm_barrier(cluster, 0, bucket, key, 'APPLIED', disk='/data/d1') + (thread, request) = async_delete(client, bucket, key) + observation = reached_barrier(cluster, 0, nonce) + cluster.kill() + thread.join(5) + clear_barriers(cluster) + cluster.restart() + wait_inventory(cluster, bucket, key, lambda inventory: inventory == original) + restored = residue_inventory(cluster, bucket, key) + assert restored == original, 'rollback must preserve exact metadata, all shards, part sidecars and integrity proofs' + (ROOT / 'multipart-restored-inventory.json').write_text(json.dumps(restored, indent=2)) + assert client.get(bucket, key) == payload, 'rolled-back native MPU must retain the complete readable body' + client.batch_delete(bucket, [key]) + wait_inventory(cluster, bucket, key, lambda inventory: all(not drive['exists'] for node in inventory for drive in node)) + client.request('GET', bucket + '/' + key, expected=404) + assert client.list(bucket) == [] + client.delete_bucket(bucket) + return {'case': case, 'status': 'passed', 'native_upload_id': upload, 'integrity': protected, 'part_sizes': [len(part) for part in parts], 'bytes': len(payload), 'observation': observation, 'bucket_deletion': 'ordinary non-force204'} + finally: + cluster.remove() + +def ordinary_missing_with_offline_peer(): + cluster = Cluster('missing-offline', nodes=4, drives=1) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'missing-offline' + client.put_bucket(bucket) + cluster.kill(3) + client.delete(bucket, 'never-existed-single', expected=204) + client.batch_delete(bucket, ['never-existed-batch-a', 'never-existed-batch-b']) + assert client.list(bucket) == [], 'ordinary missing DELETE must leave the existing bucket valid' + payload = b'bucket-remains-usable' * 200 + client.put(bucket, 'live', payload) + assert client.get(bucket, 'live') == payload + docker('start', cluster.names[3]) + cluster.ready() + assert client.get(bucket, 'live') == payload + client.delete(bucket, 'live') + wait_inventory(cluster, bucket, 'live', lambda inventory: all(not drive['exists'] for node in inventory for drive in node), timeout=90) + client.delete_bucket(bucket) + return {'case': 'ordinary-missing-delete-offline-single-and-batch', 'status': 'passed', 'cleanup_claim': 'no residue recovery was requested'} + finally: + cluster.remove() + +def legacy_transaction_data_dir_alias(): + cluster = Cluster('legacy-data-dir-alias', drives=4) + try: + cluster.create() + client = cluster.clients[0] + bucket, key = 'legacy-data-dir-alias', 'payload' + payload = b'legacy-data-dir-content' * 100000 + client.put_bucket(bucket) + client.put(bucket, key, payload) + assert client.get(bucket, key) == payload + cluster.kill() + fixture = seed_legacy(cluster, bucket, key, 'alias') + before = residue_inventory(cluster, bucket, key) + cluster.restart() + time.sleep(2) + assert residue_inventory(cluster, bucket, key) == before, 'automatic recovery must preserve ambiguous alias residue' + client.delete(bucket, key) + wait_inventory(cluster, bucket, key, lambda inventory: all(not drive['exists'] for node in inventory for drive in node)) + client.delete_bucket(bucket) + return {'case': 'legacy-transaction-equals-data-directory', 'status': 'passed', 'fixture': fixture} + finally: + cluster.remove() + +def admin_smoke(): + cluster = Cluster('admin-smoke', drives=4) + try: + cluster.create() + client = cluster.clients[0] + bucket, key = 'admin-smoke', 'payload' + payload = b'healthy-admin-object' * 1000 + client.put_bucket(bucket) + client.put(bucket, key, payload) + bucket_report = admin_heal(client, bucket, dry_run=True) + object_report = admin_heal(client, bucket, key=key) + assert client.get(bucket, key) == payload + client.delete(bucket, key) + client.delete_bucket(bucket) + return {'case': 'admin-healthy-bucket-and-object-client', 'status': 'passed', 'bucket_report': bucket_report, 'object_report': object_report} + finally: + cluster.remove() + +def large_batch_delete(): + cluster = Cluster('batch-1000', nodes=4, drives=2) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'batch-1000' + client.put_bucket(bucket) + payloads = {f'object-{index:04d}': (bytes(range(256)) * 8192 if index % 250 == 0 else f'content-{index:04d}'.encode()) for index in range(1000)} + for key, payload in payloads.items(): + client.put(bucket, key, payload) + for key, payload in payloads.items(): + assert client.get(bucket, key) == payload + client.batch_delete(bucket, list(payloads)) + assert client.list(bucket) == [] + for key in ('object-0000', 'object-0500', 'object-0999'): + assert all(not drive['exists'] for node in residue_inventory(cluster, bucket, key) for drive in node) + client.delete_bucket(bucket) + return {'case': 'maximum-1000-key-remote-delete-batch', 'status': 'passed', 'exact_gets': 1000, 'keys': 1000} + finally: + cluster.remove() + +def large_version_history(): + cluster = Cluster('versions-3000', nodes=4, drives=2) + try: + cluster.create() + client = cluster.clients[0] + bucket, key = 'versions-3000', 'history' + client.put_bucket(bucket) + client.versioning(bucket) + versions = [] + for index in range(3000): + payload = f'version-content-{index:04d}'.encode() + client.put(bucket, key, payload) + versions.append((client.last_headers['x-amz-version-id'], payload)) + for index in (0, 1500, 2999): + version, payload = versions[index] + assert client.request('GET', bucket + '/' + key + '?versionId=' + version, expected=200)[1] == payload + oldest = versions.pop(0)[0] + client.delete(bucket, key + '?versionId=' + oldest) + client.request('GET', bucket + '/' + key + '?versionId=' + oldest, expected=404) + assert client.get(bucket, key) == versions[-1][1] + for offset in range(0, len(versions), 1000): + group = versions[offset:offset + 1000] + xml = ET.Element('Delete') + for version, payload in group: + item = ET.SubElement(xml, 'Object') + ET.SubElement(item, 'Key').text = key + ET.SubElement(item, 'VersionId').text = version + data = ET.tostring(xml) + status, body = client.request('POST', bucket + '?delete', data, {'Content-Type': 'application/xml', 'Content-MD5': base64.b64encode(hashlib.md5(data).digest()).decode()}, 200, timeout=180) + response = ET.fromstring(body) + assert not [node for node in response.iter() if node.tag.rsplit('}', 1)[-1] == 'Error'], body + deleted = [child.text for node in response.iter() if node.tag.rsplit('}', 1)[-1] == 'Deleted' for child in node if child.tag.rsplit('}', 1)[-1] == 'VersionId'] + assert sorted(deleted) == sorted(version for version, payload in group), body + client.request('GET', bucket + '/' + key, expected=404) + client.delete_bucket(bucket) + return {'case': 'large-version-history-remote-single-and-batch-delete', 'status': 'passed', 'versions': 3000, 'survivor_bytes_verified': True} + finally: + cluster.remove() + +def unknown_predecessor_fresh_delete(): + cluster = Cluster('unknown-predecessor', nodes=4, drives=2) + try: + cluster.create() + client = cluster.clients[0] + bucket, key = 'unknown-predecessor', 'payload' + client.put_bucket(bucket) + client.put(bucket, key, b'older-inline' * 300) + nonce = arm_barrier(cluster, 3, bucket, key, 'APPLIED', disk='/data/d1') + thread, request = async_delete(cluster.clients[3], bucket, key) + observation = reached_barrier(cluster, 3, nonce) + transaction = str(uuid.UUID(observation['transaction'])) + verify = "from pathlib import Path; import json; result=[];\nfor drive in sorted(Path('/data').glob('d*')):\n root=drive/%r/%r; tx=root/%r; result.append(not (root/'xl.meta').exists() and (tx/'xl.meta.bkp').is_file())\nprint(json.dumps(result))" % (bucket, key, transaction) + deadline = time.monotonic() + 30 + while not all(cluster.inspect_data(verify, indices=[3])[0]): + if time.monotonic() >= deadline: + raise TimeoutError('both returning-peer preparations must reach metadata absence') + time.sleep(.05) + cluster.kill(3) + for index in range(3): + cluster.kill(index) + thread.join(5) + clear_barriers(cluster) + restore = "from pathlib import Path; import json,os; result=[];\nfor drive in sorted(Path('/data').glob('d*')):\n root=drive/%r/%r; tx=root/%r; backup=tx/'xl.meta.bkp';\n if backup.exists(): os.replace(backup,root/'xl.meta')\n if tx.exists():\n for path in tx.iterdir():\n assert path.is_file() and path.name in ('delete-prepared','delete-target'), path\n path.unlink()\n tx.rmdir()\n assert (root/'xl.meta').is_file(); result.append(True)\nprint(json.dumps(result))" % (bucket, key, transaction) + cluster.inspect_data(restore, indices=range(3)) + for index in range(3): + docker('start', cluster.names[index]) + cluster.ready(indices=range(3)) + newer = b'newer-inline' * 300 + client.put(bucket, key, newer) + assert client.get(bucket, key) == newer + client.delete(bucket, key) + docker('start', cluster.names[3]) + cluster.ready() + time.sleep(2) + pending = residue_inventory(cluster, bucket, key) + assert all(transaction + '/xl.meta.bkp' in drive['files'] for drive in pending[3]), 'automatic recovery must preserve unknown predecessor evidence' + client.request('GET', bucket + '/' + key, expected=404) + client.delete_bucket(bucket, expected=409) + client.delete(bucket, key, expected=204) + wait_inventory(cluster, bucket, key, lambda inventory: all(not drive['exists'] for node in inventory for drive in node)) + client.delete_bucket(bucket) + return {'case': 'unknown-predecessor-preserved-until-fresh-current-delete', 'status': 'passed', 'observation': observation, 'fixture': 'authentic prepared records retained only on stopped returning peer'} + finally: + cluster.remove() + +def main(): + global ROOT, PREFIX, IMAGE, PORT, CANDIDATE, BASELINE + import argparse, re, sys + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument('--binary', required=True, type=Path, help='Linux candidate binary built with e2e-test-hooks') + parser.add_argument('--baseline-binary', type=Path, help='Older Linux binary, required by mixed case') + parser.add_argument('--image', required=True, help='Local Linux image with Python 3 and the binary runtime libraries') + parser.add_argument('--artifact-dir', required=True, type=Path, help='New directory for logs and stage handshakes') + parser.add_argument('--namespace', default='rustfs-delete-' + uuid.uuid4().hex[:12]) + parser.add_argument('--port', type=int, default=19010, help='First of four sequential localhost ports') + parser.add_argument('cases', nargs='*', default=['smoke']) + options = parser.parse_args() + if not re.fullmatch('[a-z][a-z0-9-]{0,45}', options.namespace): + parser.error('namespace must be a short lowercase Docker name') + if not 1024 <= options.port <= 65532: + parser.error('port must leave room for four localhost ports') + CANDIDATE = options.binary.resolve(strict=True) + if not CANDIDATE.is_file(): + parser.error('binary must be a regular file') + BASELINE = options.baseline_binary.resolve(strict=True) if options.baseline_binary else None + if BASELINE is not None and (not BASELINE.is_file()): + parser.error('baseline binary must be a regular file') + ROOT = options.artifact_dir.resolve() + ROOT.mkdir(parents=True, exist_ok=False) + PREFIX = options.namespace + IMAGE = options.image + PORT = options.port + for resource in ('container', 'volume', 'network'): + existing = docker(resource, 'ls', '-q', '--filter', 'label=rustfs.delete-recovery-test=' + PREFIX).stdout.strip() + if existing: + parser.error('namespace already owns Docker resources; choose a new namespace') + cases = {'automatic-clean-restart': automatic_clean_restart, 'unknown-fresh': unknown_predecessor_fresh_delete, 'batch1000': large_batch_delete, 'versions3000': large_version_history, 'admin-smoke': admin_smoke, 'legacy-alias': legacy_transaction_data_dir_alias, 'missing-offline': ordinary_missing_with_offline_peer, 'inverse': retained_decision_newer_undecided, 'deep': deep_key_crash, 'deep-large-stack': lambda : deep_key_crash(True), 'admin-prefix': admin_prefix_scoped, 'response-loss': receipt_response_loss, 'admin-object': lambda : admin_legacy_dry_run(True), 'smoke': smoke, 'legacy1': lambda : legacy_retry(1), 'legacy4': lambda : legacy_retry(4), 'restore': rollback_crash, 'retention': protected_version, 'version-applied': lambda : versioned_crash('APPLIED'), 'version-decision': lambda : versioned_crash('DECISION'), 'returning': returning_drive_new_generation, 'untouched': untouched_returning_peer, 'missing-version': missing_version_returning_peer, 'mixed': mixed_versions, 'admin': admin_legacy_dry_run, 'prefix': uuid_prefix_crash, 'corruption': corruption_repair, 'permission': permission_fault, 'error-prepared': lambda : injected_error('PREPARED'), 'error-applied': lambda : injected_error('APPLIED'), 'error-decision': lambda : injected_error('DECISION')} + cases.update({'multipart-sidecars': completed_multipart_sidecars, 'multipart-integrity': lambda: completed_multipart_sidecars(True), 'multipart-sidecars-applied': lambda: completed_multipart_sidecars(False, True), 'multipart-integrity-applied': lambda: completed_multipart_sidecars(True, True)}) + for stage in ('PREACTIVATION', 'PREPARED', 'APPLIED', 'DECISION'): + for nodes in (1, 4): + cases[f'{stage.lower()}{nodes}'] = lambda stage=stage, nodes=nodes: process_crash(stage, nodes) + cases[f'inline-{stage.lower()}'] = lambda stage=stage: process_crash(stage, 1, 4096) + cases[f'zero-{stage.lower()}'] = lambda stage=stage: process_crash(stage, 1, 0) + if 'mixed' in options.cases and BASELINE is None: + parser.error('mixed requires --baseline-binary') + unknown = set(options.cases) - set(cases) + if unknown: + parser.error('unknown cases: ' + ','.join(sorted(unknown))) + results = [] + for case in options.cases: + started = time.monotonic() + try: + result = cases[case]() + result['elapsed_seconds'] = round(time.monotonic() - started, 3) + except BaseException as error: + result = {'case': case, 'status': 'failed', 'error': str(error), 'elapsed_seconds': round(time.monotonic() - started, 3)} + (ROOT / (case + '.json')).write_text(json.dumps(result, indent=2, sort_keys=True)) + raise + results.append(result) + (ROOT / (case + '.json')).write_text(json.dumps(result, indent=2, sort_keys=True)) + print(json.dumps(result, sort_keys=True), flush=True) + (ROOT / 'results.json').write_text(json.dumps(results, indent=2, sort_keys=True)) +if __name__ == '__main__': + main() diff --git a/scripts/test_conditional_read_abba.py b/scripts/test_conditional_read_abba.py new file mode 100644 index 000000000..c7b53568b --- /dev/null +++ b/scripts/test_conditional_read_abba.py @@ -0,0 +1,116 @@ +#!/usr/bin/env python3 +"""Verify that paired conditional-read evidence cannot bypass acceptance gates.""" + +import copy +import json +from pathlib import Path +import subprocess +import unittest +from unittest.mock import MagicMock, patch + +from run_conditional_read_abba import compare, parse_rows, run_case + + +def row(): + return {"size": 4096, "kind": "miss", "cache": False, "slowtail_ms": "0", + "ops_per_sec": 1000, "p95_ms": 2, "p99_ms": 3, + "lock_p95_ms": 1, "lock_p99_ms": 2} + + +class AcceptanceTests(unittest.TestCase): + def test_stable_equivalent_legs_pass(self): + self.assertTrue(compare([{("case",): row()} for _ in range(4)])["accepted"]) + + def test_baseline_drift_blocks_candidate_attribution(self): + legs = [{("case",): row()} for _ in range(4)] + legs[3][("case",)]["ops_per_sec"] = 800 + result = compare(legs) + self.assertFalse(result["accepted"]) + self.assertEqual(result["reason"], "baseline_drift") + self.assertNotIn("cases", result) + + def test_miss_throughput_and_lock_regressions_fail(self): + for metric, value in [("ops_per_sec", 900), ("lock_p99_ms", 4)]: + legs = [{("case",): row()} for _ in range(4)] + for index in (1, 2): + legs[index][("case",)][metric] = value + self.assertFalse(compare(legs)["accepted"], metric) + + def test_missing_workload_is_rejected(self): + legs = [{("case",): row()} for _ in range(4)] + legs[1] = {} + with self.assertRaises(ValueError): + compare(legs) + + def test_complete_rows_include_first_libtest_output_line(self): + rows = [] + for size in (4096, 1_300_000): + for kind in ("unconditional", "hit", "miss"): + value = copy.copy(row()) + value.update(size=size, kind=kind) + rows.append("CONDITIONAL_BENCH " + json.dumps(value)) + output = "test app::benchmark ... " + "\n".join(rows) + self.assertEqual(len(parse_rows(output)), 6) + with self.assertRaises(ValueError): + parse_rows(output + "\n" + rows[0]) + + def test_wrong_shape_or_nonfinite_metrics_are_rejected(self): + rows = [] + for size in (4096, 1_300_000): + for kind in ("unconditional", "hit", "miss"): + value = copy.copy(row()) + value.update(size=size, kind=kind) + rows.append(value) + for field, value in [("kind", "unknown"), ("p99_ms", float("nan"))]: + changed = copy.deepcopy(rows) + changed[0][field] = value + output = "\n".join("CONDITIONAL_BENCH " + json.dumps(item) for item in changed) + with self.assertRaises(ValueError): + parse_rows(output) + + +class IsolationTests(unittest.TestCase): + def process(self): + process = MagicMock() + process.__enter__.return_value = process + process.communicate.return_value = ("saved output", None) + process.returncode = 0 + return process + + def test_changed_executable_is_rejected_before_launch(self): + with patch("run_conditional_read_abba.digest", return_value="changed"), \ + patch("run_conditional_read_abba.subprocess.Popen") as launch: + with self.assertRaisesRegex(RuntimeError, "before execution"): + run_case(Path("binary"), "expected", {}) + launch.assert_not_called() + + def test_changed_executable_is_rejected_after_launch(self): + with patch("run_conditional_read_abba.digest", side_effect=["expected", "changed"]), \ + patch("run_conditional_read_abba.check_resource_isolation"), \ + patch("run_conditional_read_abba.subprocess.Popen", return_value=self.process()): + with self.assertRaisesRegex(RuntimeError, "during execution"): + run_case(Path("binary"), "expected", {}) + + def test_final_case_checks_resource_isolation_after_exit(self): + with patch("run_conditional_read_abba.digest", return_value="expected"), \ + patch("run_conditional_read_abba.check_resource_isolation", side_effect=RuntimeError("concurrent build")), \ + patch("run_conditional_read_abba.subprocess.Popen", return_value=self.process()): + with self.assertRaisesRegex(RuntimeError, "concurrent build"): + run_case(Path("binary"), "expected", {}) + + def test_concurrent_build_during_execution_kills_case_and_saves_log(self): + process = self.process() + process.communicate.side_effect = [subprocess.TimeoutExpired("binary", 1), ("saved output", None)] + log = MagicMock() + with patch("run_conditional_read_abba.digest", return_value="expected"), \ + patch("run_conditional_read_abba.check_resource_isolation", side_effect=RuntimeError("concurrent build")) as check, \ + patch("run_conditional_read_abba.subprocess.Popen", return_value=process): + with self.assertRaisesRegex(RuntimeError, "concurrent build"): + run_case(Path("binary"), "expected", {}, log) + check.assert_called_once_with(check_load=False) + process.kill.assert_called_once() + log.write_text.assert_called_once_with("saved output") + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/test_helm_templates.sh b/scripts/test_helm_templates.sh index 1b21e68cb..e6e7907d0 100755 --- a/scripts/test_helm_templates.sh +++ b/scripts/test_helm_templates.sh @@ -180,6 +180,7 @@ scanner_config_output=$(render_chart \ --set config.rustfs.scanner.cache_save_timeout_secs=30 \ --set config.rustfs.scanner.max_concurrent_set_scans=2 \ --set config.rustfs.scanner.max_concurrent_disk_scans=1 \ + --set config.rustfs.scanner.walkers_per_disk=2 \ --set config.rustfs.scanner.yield_every_n_objects=128 \ --set config.rustfs.scanner.alert_excess_versions=100 \ --set config.rustfs.scanner.alert_excess_version_size=1099511627776 \ @@ -197,6 +198,7 @@ grep -q 'RUSTFS_SCANNER_IDLE_MODE: "false"' <<<"$scanner_config_output" grep -q 'RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS: "30"' <<<"$scanner_config_output" grep -q 'RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS: "2"' <<<"$scanner_config_output" grep -q 'RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS: "1"' <<<"$scanner_config_output" +grep -q 'RUSTFS_SCANNER_WALKERS_PER_DISK: "2"' <<<"$scanner_config_output" grep -q 'RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS: "128"' <<<"$scanner_config_output" grep -q 'RUSTFS_SCANNER_ALERT_EXCESS_VERSIONS: "100"' <<<"$scanner_config_output" grep -q 'RUSTFS_SCANNER_ALERT_EXCESS_VERSION_SIZE: "1099511627776"' <<<"$scanner_config_output" @@ -568,7 +570,7 @@ nondefault_statefulset=$(awk ' nondefault_local_host=$(statefulset_env_value RUSTFS_LOCAL_ENDPOINT_HOST <<<"$nondefault_statefulset") nondefault_namespace=$(yq eval '.metadata.namespace' - <<<"$nondefault_statefulset") nondefault_endpoint_port=$(yq eval '.spec.template.spec.containers[0].ports[] | select(.name == "endpoint") | .containerPort' - <<<"$nondefault_statefulset") -mtls_liveness_command=$(yq eval '.spec.template.spec.containers[0].livenessProbe.exec.command[-1]' - <<<"$nondefault_statefulset") +mtls_liveness_port=$(yq eval '.spec.template.spec.containers[0].livenessProbe.tcpSocket.port' - <<<"$nondefault_statefulset") mtls_readiness_command=$(yq eval '.spec.template.spec.containers[0].readinessProbe.exec.command[-1]' - <<<"$nondefault_statefulset") if [[ "$nondefault_local_host" != '$(RUSTFS_CHART_POD_NAME).tenant-rustfs-headless.object-storage.svc.cluster.corp' ]]; then echo "Combined generated topology produced an unexpected local endpoint host: $nondefault_local_host" >&2 @@ -578,9 +580,8 @@ if [[ "$nondefault_namespace" != "object-storage" || "$nondefault_endpoint_port" echo "Combined generated topology must retain its namespace and custom endpoint port" >&2 exit 1 fi -if ! grep -Fq 'https://127.0.0.1:9443/health' <<<"$mtls_liveness_command" || - grep -Fq '/health/ready' <<<"$mtls_liveness_command"; then - echo "mTLS liveness must use the process health endpoint, not readiness" >&2 +if [[ "$mtls_liveness_port" != 9443 ]]; then + echo "mTLS liveness must check the listener independently of certificate validity" >&2 exit 1 fi if ! grep -Fq 'https://127.0.0.1:9443/health/ready' <<<"$mtls_readiness_command"; then diff --git a/scripts/test_helm_tls_reload.sh b/scripts/test_helm_tls_reload.sh new file mode 100644 index 000000000..e8ad68d86 --- /dev/null +++ b/scripts/test_helm_tls_reload.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +CHART_DIR="$ROOT_DIR/helm/rustfs" + +for distributed in false true; do + rendered=$(helm template tls-reload "$CHART_DIR" \ + --set secret.allowInsecureDefaults=true \ + --set mode.distributed.enabled="$distributed" \ + --set mode.standalone.enabled="$([[ "$distributed" == false ]] && echo true || echo false)" \ + --set tls.enabled=true --set tls.existingSecret=issued-tls) + workload=$(yq eval 'select(.kind == "Deployment" or .kind == "StatefulSet")' - <<<"$rendered") + mount=$(yq eval '.spec.template.spec.containers[0].volumeMounts[] | select(.name == "tls-material")' - <<<"$workload") + [[ $(yq eval '.mountPath' - <<<"$mount") == /opt/tls ]] + [[ $(yq eval '.readOnly' - <<<"$mount") == true ]] + [[ $(yq eval 'has("subPath")' - <<<"$mount") == false ]] + keys=$(yq eval '.spec.template.spec.volumes[] | select(.name == "tls-material") | .projected.sources[].secret.items[].key' - <<<"$workload") + [[ "$keys" == $'tls.crt\ntls.key' ]] + [[ $(yq eval '.spec.template.spec.containers[0].readinessProbe.httpGet.scheme' - <<<"$workload") == HTTPS ]] + [[ $(yq eval '.spec.template.spec.containers[0].livenessProbe.tcpSocket.port' - <<<"$workload") == 9000 ]] + config=$(yq eval 'select(.kind == "ConfigMap") | .data' - <<<"$rendered") + [[ $(yq eval '.RUSTFS_TLS_RELOAD_ENABLE' - <<<"$config") == 1 ]] + [[ $(yq eval '.RUSTFS_TLS_KEYLOG' - <<<"$config") == 0 ]] + if [[ "$distributed" == true ]]; then + [[ $(yq eval '.RUSTFS_VOLUMES' - <<<"$config") == https://* ]] + fi + + mtls=$(helm template tls-reload "$CHART_DIR" --set secret.allowInsecureDefaults=true \ + --set mode.distributed.enabled="$distributed" \ + --set mode.standalone.enabled="$([[ "$distributed" == false ]] && echo true || echo false)" \ + --set mtls.enabled=true) + workload=$(yq eval 'select(.kind == "Deployment" or .kind == "StatefulSet")' - <<<"$mtls") + [[ $(yq eval '.spec.template.spec.volumes[] | select(.name == "tls-material") | .projected.sources | length' - <<<"$workload") == 2 ]] + [[ $(yq eval '.spec.template.spec.containers[0].volumeMounts[] | select(.name == "tls-material") | has("subPath")' - <<<"$workload") == false ]] + [[ $(yq eval '.spec.template.spec.containers[0].livenessProbe.tcpSocket.port' - <<<"$workload") == 9000 ]] +done + +if helm template tls-reload "$CHART_DIR" --set secret.allowInsecureDefaults=true \ + --set tls.enabled=true --set tls.existingSecret=issued-tls --set mtls.enabled=true >/dev/null 2>&1; then + echo "Public TLS must not silently enable global client authentication" >&2 + exit 1 +fi +if helm template tls-reload "$CHART_DIR" --set secret.allowInsecureDefaults=true \ + --set tls.enabled=true >/dev/null 2>&1; then + echo "Native TLS requires an explicitly configured serving Secret" >&2 + exit 1 +fi +echo "TLS Secret projection, rotation settings, topology and probes passed" diff --git a/scripts/test_release_build_workflow.py b/scripts/test_release_build_workflow.py new file mode 100644 index 000000000..cce983e90 --- /dev/null +++ b/scripts/test_release_build_workflow.py @@ -0,0 +1,137 @@ +#!/usr/bin/env python3 +"""Exercise server-only release commands and strict acceptance archive readers.""" + +from __future__ import annotations + +import io +import json +import os +import re +import shlex +import subprocess +import sys +import tempfile +import unittest +import zipfile +from pathlib import Path + +from test_security_workflow import named_steps, shell_body, yaml_block + +ROOT = Path(__file__).resolve().parents[1] +SOURCE = (ROOT / '.github/workflows/build.yml').read_text() +STEPS = named_steps(yaml_block(SOURCE.splitlines(), 'build-rustfs', 2)) +MATRIX = json.loads(re.search(r"all='(\{.*?\})'", SOURCE, re.S)[1])['include'] +CATALOG = ('rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256') +EXTRACTORS = ( + ('connect-diagnostic-schedule-acceptance.yml', 'diagnostic-schedule', 'Extract the verified official binary'), + ('connect-health-service-acceptance.yml', 'health-service', 'Extract only the verified official service binary'), + ('connect-profile-memory-acceptance.yml', 'profile-memory', 'Extract only the verified official service binary'), + ('connect-top-disk-acceptance.yml', 'top-disk', 'Extract only the verified official service binary'), +) + + +def render(script, target, package='package.zip'): + context = {f'matrix.{key}': str(value).lower() if isinstance(value, bool) else str(value) for key, value in target.items()} + context['steps.package.outputs.package_file'] = package + return re.sub(r'\$\{\{\s*(.*?)\s*\}\}', lambda match: context[match[1]], script) + + +def archive(path, members, binary): + with zipfile.ZipFile(path, 'w') as package: + for name in members: + entry = zipfile.ZipInfo(name) + entry.external_attr = (0o100755 if name == 'rustfs' else 0o100644) << 16 + package.writestr(entry, binary if name == 'rustfs' else b'fixture') + + +class ReleaseBuildTests(unittest.TestCase): + def test_every_matrix_target_builds_only_server(self): + mock_cargo = r''' +cargo() { + printf '%q ' "$@" >> cargo-call + printf '\n' >> cargo-call + local file="target/$BUILD_TARGET/release/rustfs" + if [[ "$BUILD_TARGET" == *-windows-msvc ]]; then file+='.exe'; fi + mkdir -p "$(dirname "$file")" + cat > "$file" <<'BIN' +#!/usr/bin/env bash +printf 'build target : %s\nbuild os : %s\nrust channel : stable-%s\n' "$BUILD_TARGET" "$BUILD_OS" "$BUILD_TARGET" +BIN + chmod +x "$file" +} +''' + for target in MATRIX: + with self.subTest(target=target['target_id']), tempfile.TemporaryDirectory() as directory: + path = Path(directory) + (path / 'rustfs').mkdir() + (path / 'rustfs/build.rs').touch() + (path / 'scripts/release').mkdir(parents=True) + (path / 'scripts/release/check_glibc.py').write_bytes((ROOT / 'scripts/release/check_glibc.py').read_bytes()) + (path / 'tools').mkdir() + readelf = path / 'tools/readelf' + readelf.write_text("#!/bin/sh\nprintf '%s\\n' 'Version needs section:' 'Name: GLIBC_2.30'\n") + readelf.chmod(0o755) + script = mock_cargo + render(shell_body(STEPS['Build RustFS']), target) + environment = {**os.environ, 'BUILD_TARGET': target['target'], + 'BUILD_OS': target['platform'] + '-' + target['target'].split('-')[0], + 'PATH': str(path / 'tools') + os.pathsep + os.environ['PATH']} + result = subprocess.run(['bash', '-e', '-o', 'pipefail', '-c', script], cwd=path, + env=environment, capture_output=True, text=True) + self.assertEqual(result.returncode, 0, result.stderr) + calls = (path / 'cargo-call').read_text().splitlines() + self.assertEqual(len(calls), 1) + args = shlex.split(calls[0]) + self.assertEqual(args[0], 'zigbuild' if target['cross'] else 'build') + self.assertEqual(args[args.index('--bin') + 1], 'rustfs') + self.assertNotIn('rustfs-cli', args) + self.assertIn('--locked', args) + expected_target = target['target'] + ('.' + target['glibc'] if target['glibc'] else '') + self.assertEqual(args[args.index('--target') + 1], expected_target) + if target['features']: + self.assertEqual(args[args.index('--features') + 1], target['features']) + if target['glibc']: + self.assertEqual(args[args.index('--config') + 1], 'profile.release.strip="none"') + + def test_packaged_linux_server_and_catalog_members(self): + binary = b'#!/bin/sh\necho rustfs-fixture\n' + for libc in ('gnu', 'musl'): + target = next(item for item in MATRIX if item['target_id'] == f'linux-x86_64-{libc}') + expected = ['rustfs', *CATALOG] if libc == 'gnu' else ['rustfs'] + for members, valid in ((expected, True), (expected + ['rustfs-cli'], False)): + with self.subTest(libc=libc, members=members), tempfile.TemporaryDirectory() as directory: + path = Path(directory) + archive(path / 'package.zip', members, binary) + script = render(shell_body(STEPS['Verify packaged Linux binary']), target) + result = subprocess.run(['bash', '-e', '-o', 'pipefail', '-c', script], cwd=path, + capture_output=True, text=True) + self.assertEqual(result.returncode == 0, valid, result.stderr) + + def test_actual_acceptance_extractors_keep_strict_members(self): + source_sha = '0123456789abcdef' * 2 + '01234567' + package_name = f'rustfs-linux-x86_64-gnu-dev-{source_sha[:7]}.zip' + header = bytearray(20) + header[:7] = b'\x7fELF\x02\x01\x01' + header[18:20] = b'\x3e\x00' + valid_members = ['rustfs', *CATALOG] + cases = ((valid_members, True), (['rustfs'], False), (valid_members + ['rustfs-cli'], False), + (valid_members + ['../extra'], False)) + for workflow, job_name, step in EXTRACTORS: + source = (ROOT / '.github/workflows' / workflow).read_text() + job = yaml_block(source.splitlines(), job_name, 2) + body = shell_body(named_steps(job)[step]) + python = body.split("python3 - <<'PY'\n", 1)[1].split('\nPY', 1)[0] + for members, valid in cases: + with self.subTest(workflow=workflow, members=members), tempfile.TemporaryDirectory() as directory: + path = Path(directory) + archive(path / package_name, members, bytes(header)) + with zipfile.ZipFile(path / 'official-artifact.zip', 'w') as outer: + outer.write(path / package_name, package_name) + result = subprocess.run([sys.executable, '-c', python], cwd=path, + env={**os.environ, 'SOURCE_SHA': source_sha}, capture_output=True, text=True) + self.assertEqual(result.returncode == 0, valid, result.stderr) + if valid: + self.assertEqual((path / 'binary/rustfs').read_bytes(), bytes(header)) + + +if __name__ == '__main__': + unittest.main() diff --git a/scripts/test_release_glibc.py b/scripts/test_release_glibc.py new file mode 100644 index 000000000..a0359ab11 --- /dev/null +++ b/scripts/test_release_glibc.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 +"""Exercise release ABI checks against loader requirements and malformed inputs.""" + +import importlib.util +import subprocess +import unittest +from pathlib import Path +from unittest.mock import patch + +SPEC = importlib.util.spec_from_file_location("check_glibc", Path(__file__).parent / "release/check_glibc.py") +glibc = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(glibc) + + +class GlibcTests(unittest.TestCase): + def inspect(self, needs: str, baseline: str = "2.31") -> str: + output = "Version definition section: Name: GLIBC_9.99\nVersion needs section:\n" + needs + with patch.object(glibc.subprocess, "run", return_value=subprocess.CompletedProcess([], 0, output)): + return glibc.check(Path("rustfs"), baseline) + + def test_uses_loader_needs_and_numeric_order(self): + self.assertEqual(self.inspect("Name: GLIBC_2.9\nName: GLIBC_2.30\nName: GLIBC_2.2.5"), "GLIBC_2.30") + self.assertEqual(self.inspect("Name: GLIBC_2.31.0"), "GLIBC_2.31.0") + + def test_rejects_newer_requirement(self): + with self.assertRaisesRegex(ValueError, "requires GLIBC_2.34"): + self.inspect("Name: GLIBC_2.34") + + def test_rejects_non_numeric_abi_tags(self): + for tag in ("GLIBC_PRIVATE", "GLIBC_ABI_DT_RELR"): + with self.subTest(tag=tag), self.assertRaises(ValueError): + self.inspect(f"Name: {tag}") + + def test_rejects_missing_needs_and_invalid_baselines(self): + with self.assertRaisesRegex(ValueError, "no GLIBC version needs"): + self.inspect("Name: GCC_3.0") + for baseline in ("", "2", "2.31x", "-2.31"): + with self.subTest(baseline=baseline), self.assertRaises(ValueError): + self.inspect("Name: GLIBC_2.2.5", baseline) + + def test_readelf_failure_is_not_a_pass(self): + with patch.object(glibc.subprocess, "run", side_effect=subprocess.CalledProcessError(1, "readelf")): + with self.assertRaises(subprocess.CalledProcessError): + glibc.check(Path("invalid"), "2.31") + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/test_upgrade_workflow.py b/scripts/test_upgrade_workflow.py new file mode 100644 index 000000000..baa931895 --- /dev/null +++ b/scripts/test_upgrade_workflow.py @@ -0,0 +1,110 @@ +#!/usr/bin/env python3 +"""Exercise both pinned upgrade sources without building RustFS.""" + +from __future__ import annotations + +import json +import os +from pathlib import Path +import re +import subprocess +import sys +import tempfile +import unittest + +from check_test_wiring import yaml_block +from test_security_workflow import named_steps, shell_body + + +ROOT = Path(__file__).resolve().parents[1] + + +class UpgradeWorkflowTests(unittest.TestCase): + def setUp(self): + source = (ROOT / ".github/workflows/e2e-upgrade.yml").read_text().splitlines() + self.job = yaml_block(source, "upgrade", 2) + self.steps = named_steps(self.job) + self.step = self.steps["Run upgrade compatibility tests"] + # Explicit bash enables pipefail in Actions, so tee cannot hide a failure. + self.assertIn(" shell: bash", self.step) + self.assertIn(" fail-fast: false", self.job) + self.assertEqual(re.findall(r"- version: ([^\s]+)", "\n".join(self.job)), ["1.0.0-rc.5", "1.0.1"]) + rust = (ROOT / "crates/e2e_test/src/upgrade_compatibility_test.rs").read_text() + self.expected = set(re.findall(r'#\[ignore[^\n]*\]\s*async fn (\w+)\(', rust)) + self.assertTrue(self.expected) + + def run_cases(self, version, failed_test="", fail_tee=False): + excluded = ("stable_upgrade_deletes_foreign_reservations_without_losing_live_objects" + if version == "1.0.0-rc.5" else "rc5_rollback_requires_restoring_odm_configuration") + expected = self.expected - {excluded} + self.assertEqual(len(expected), 7) + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + commands = root / "commands.jsonl" + fake_python = root / "python3" + fake_python.write_text(f"#!{sys.executable}\n" + '''import json, os, sys +from pathlib import Path +args = sys.argv[1:] +assert args[:8] == ["scripts/e2e_binary.py", "run", "--", "cargo", "test", "--locked", "-p", "e2e_test"] +assert args[9:] == ["--", "--ignored", "--exact", "--nocapture"] +test = args[8].removeprefix("upgrade_compatibility_test::") +assert args[8] != test +assert os.environ["no_proxy"] == "127.0.0.1,localhost,::1" +with open(os.environ["COMMANDS"], "a") as output: + output.write(json.dumps({"test": test, "logs": os.environ["RUSTFS_E2E_LOG_DIR"], + "evidence": os.environ["RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR"]}) + "\\n") +Path(os.environ["RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR"]).mkdir(parents=True) +print("executed " + test) +sys.exit(17 if test == os.environ["FAILED_TEST"] else 0) +''') + fake_python.chmod(0o755) + if fail_tee: + tee = root / "tee" + tee.write_text("#!/bin/sh\ncat >/dev/null\nexit 23\n") + tee.chmod(0o755) + env = dict(os.environ, PATH=f"{root}{os.pathsep}{os.environ['PATH']}", + RUNNER_TEMP=str(root), COMMANDS=str(commands), FAILED_TEST=failed_test, + NO_PROXY="127.0.0.1,localhost,::1", UPGRADE_SOURCE_VERSION=version) + result = subprocess.run(["bash", "--noprofile", "--norc", "-e", "-o", "pipefail", "-c", shell_body(self.step)], + cwd=root, env=env, capture_output=True, text=True) + calls = [json.loads(line) for line in commands.read_text().splitlines()] + self.assertEqual({call["test"] for call in calls}, expected) + self.assertEqual(len(calls), len(expected)) + for field in ("logs", "evidence"): + self.assertEqual(len({call[field] for call in calls}), len(calls)) + self.assertTrue(all(Path(call[field]).is_dir() for call in calls)) + if not fail_tee: + for call in calls: + self.assertIn(call["test"], (Path(call["logs"]) / "test.log").read_text()) + return result + + def test_all_ignored_upgrade_cases_run_once_with_separate_evidence(self): + for version in ("1.0.0-rc.5", "1.0.1"): + with self.subTest(version=version): + result = self.run_cases(version) + self.assertEqual(result.returncode, 0, result.stderr) + + def test_early_or_late_failure_does_not_hide_remaining_cases(self): + for version, last in (("1.0.0-rc.5", "rc5_rollback_requires_restoring_odm_configuration"), + ("1.0.1", "stable_upgrade_deletes_foreign_reservations_without_losing_live_objects")): + for name in ("direct_upgrade_from_previous_release_preserves_object_contracts", last): + with self.subTest(version=version, name=name): + result = self.run_cases(version, failed_test=name) + self.assertNotEqual(result.returncode, 0, result.stdout) + self.assertIn(f"::error::Upgrade compatibility failed: {name}", result.stdout) + + def test_log_write_failure_fails_the_job(self): + for version in ("1.0.0-rc.5", "1.0.1"): + self.assertNotEqual(self.run_cases(version, fail_tee=True).returncode, 0) + + def test_reports_are_uploaded_even_after_failure(self): + for name, directory in (("Upload scanner/heal G09 evidence", "rustfs-upgrade-g09-evidence"), + ("Upload server logs", "rustfs-upgrade-logs")): + step = self.steps[name] + self.assertIn(" if: always()", step) + self.assertIn(" path: ${{ runner.temp }}/" + directory, step) + self.assertTrue(any("name:" in line and "matrix.source.version" in line for line in step)) + + +if __name__ == "__main__": + unittest.main()