Compare commits

..

13 Commits

547 changed files with 14418 additions and 85053 deletions
@@ -45,14 +45,11 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
- `GHSA-9wgh-5r68-49p7`: substituted policy-variable values could recreate `${...}` and loop inside a single resolution pass, never reaching the outer iteration limit. Lesson: bound substitutions within each pass and test cycles formed across otherwise literal resolver values; confirm whether attacker-controlled claims can reach those values before assigning remote impact.
### S3 object actions, copy, multipart, and upload policy validation
- `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
- `GHSA-cg33-pxfp-qcj3` and `GHSA-65wq-gjw8-xr9w`: `HeadObject` and object-tagging paths authorized current-object actions while operating on an explicit historical `versionId`. Lesson: select the corresponding version action for metadata and tag reads and writes too; audit every operation that accepts a version ID.
- `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
@@ -63,14 +60,12 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
- `GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
- `GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
- `GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
- `GHSA-rhg6-jrf2-29c4`: WebDAV, FTP/FTPS, and SFTP used identity-policy checks with empty conditions and bypassed S3 bucket-policy explicit denies and condition-key denies. Lesson: alternate listeners must evaluate the same effective policy and trustworthy request context before storage access.
### Filesystem paths and object key traversal
- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
- `GHSA-g7fq-6xjm-mxf4`: versioned `DeleteObjects` accepted dot-segment keys, authorized the literal prefix, then wrote delete markers under a resolved sibling prefix. Lesson: validate each batch key before authorization and storage normalization, using the same object-name rules as single delete.
### Secrets, defaults, and cryptographic misuse
@@ -111,8 +106,6 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
### Object Lock and retention invariants
- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
- `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
- `GHSA-q23w-fp7p-472v`: `PutObjectRetention` policy conditions read `x-amz-object-lock-*` headers while the handler applied mode and date from the XML body. Lesson: authorize the parsed retention that will be stored, or reject mismatched header/body values; compare retention dates as instants so offsets cannot change the applied bound.
### Serde deserialization and input validation
@@ -142,17 +135,16 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
## Minimum Regression Test Expectations
- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases; test bucket-policy explicit denies and condition-key denies on each enabled listener against an S3 control, and assert denied paths do not reach storage.
- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
- Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, multipart-copy source, `HeadObject`, and tag read/write cases.
- Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, cyclic variable substitutions, plugin mode, and built-in policy mode.
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. For `PutObjectRetention`, test conflicting header/body values and offset dates against policy bounds and read-back retention.
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=1520cd3027c3a6924afc6a204eddd5475bc30f0109d4c2847456a66177263244
sha256-linux=26155b7f1c5bf6bf596ff24a4136e6767ea4ee6461c32db7fbe1400791da2c28
sha256-darwin=98dff0961c6eb21d6128bfd2d4a6100a624ddf1fb7db05dba360a5d3d0a310c2
sha256-linux=9ff0da7406de9b89c0e848ff29f2238b30ba9da89c7e9df97c246ce86631350f
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=347a0d4e5e547d5388703062ce66a66167a845b6e57ec591ccb4c732a567c68d
sha256-linux=4f9efa47194279185bb68249e82c788fa147abe944b6c764050e3d47f31e7926
sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=a2cb67d91eb0d093928371d6e84a84b2dd41b2ba975a28094e81879bd65838ad
sha256-linux=82db0561cf9b111f11782d647d0859c9d2a44c5c68481b5e39c48505828bd392
sha256-darwin=55534a97fbd376f64c8f6c341d319017d11ff77cad6da8629a1a7f6a874e0315
sha256-linux=c06fb8c19aed6f388b9dc61cb8251b7a44f8561a9bf764ad2b9e635598f8dc17
-85
View File
@@ -61,11 +61,6 @@
"suite": "rustfs-ecstore",
"name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_preserves_overwrite_data_dirs_and_staging"
},
{
"invariant": "rollback-closed-inspection-channel",
"suite": "rustfs-ecstore",
"name": "set_disk::core::io_primitives::tests::rename_rollback_closed_heal_receiver_preserves_recovery_material"
},
{
"invariant": "rollback-cancellation",
"suite": "rustfs-ecstore",
@@ -91,11 +86,6 @@
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::put_object_tmp_cleanup_tests::tail_drained_put_waits_for_tail_and_allows_immediate_cas"
},
{
"invariant": "seeded-put-tail-pressure",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::put_object_tmp_cleanup_tests::seeded_put_completion_pressure_drains_every_real_tail"
},
{
"invariant": "borrowed-lock-tail",
"suite": "rustfs-ecstore",
@@ -161,11 +151,6 @@
"suite": "rustfs-ecstore",
"name": "set_disk::core::metadata_quorum::tests::metadata_observation_all_four_slot_states_and_arrival_orders_preserve_reduction"
},
{
"invariant": "metadata-observation-mixed-current-identity-arrivals-preserve-selection",
"suite": "rustfs-ecstore",
"name": "set_disk::core::metadata_quorum::tests::metadata_observation_mixed_current_identity_arrivals_preserve_selection"
},
{
"invariant": "metadata-observation-pending-newer-version-and-same-time-different-directory-force-full-wait",
"suite": "rustfs-ecstore",
@@ -181,21 +166,6 @@
"suite": "rustfs-ecstore",
"name": "set_disk::core::metadata_quorum::tests::metadata_observation_null_versions_markers_and_invalid_success_keep_their_meaning"
},
{
"invariant": "metadata-cache-real-get-and-range-preserve-body-across-read-modes",
"suite": "rustfs-ecstore",
"name": "set_disk::read::metadata_cache_tests::metadata_cache_real_get_and_range_preserve_body_across_read_modes"
},
{
"invariant": "late-metadata-refresh-rejects-changed-generation-and-mapped-shards",
"suite": "rustfs-ecstore",
"name": "set_disk::read::metadata_cache_tests::late_metadata_refresh_rejects_changed_generation_and_mapped_shards"
},
{
"invariant": "late-full-get-rejects-changed-object-generation",
"suite": "rustfs-ecstore",
"name": "set_disk::prepared_get_object_metadata_tests::non_inline_two_phase_get_rejects_generation_change_before_late_refresh"
},
{
"invariant": "list-objects-shared-corpus-static-namespace",
"suite": "rustfs-ecstore",
@@ -290,46 +260,6 @@
"invariant": "logical-prefix-limit",
"suite": "rustfs-ecstore",
"name": "store::list_objects::test::list_path_gather_results_counts_common_prefixes_before_page_limit"
},
{
"invariant": "quota-same-name-owner-and-disk-authority",
"suite": "rustfs-ecstore",
"name": "bucket::metadata_sys::tests::quota_snapshot_reuses_same_name_owner_and_rereads_disk_authority"
},
{
"invariant": "quota-snapshot-scope-and-owner",
"suite": "rustfs-ecstore",
"name": "bucket::metadata_sys::tests::quota_snapshot_rejects_wrong_scope_incarnation_and_transaction_guard"
},
{
"invariant": "quota-snapshot-lifecycle-and-incarnation",
"suite": "rustfs-ecstore",
"name": "bucket::metadata_sys::tests::quota_snapshot_rejects_lifecycle_loss_during_read_and_changed_disk_incarnation"
},
{
"invariant": "marker-purge-receipt-reopen",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::marker_purge_receipt_tests::quorum_marker_purge_receipt_survives_reopen_with_stale_member"
},
{
"invariant": "external-minio-shards",
"suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
"name": "pinned_minio_shards_decode_and_reconstruct_plaintext"
},
{
"invariant": "external-legacy-shards",
"suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
"name": "pinned_legacy_shards_decode_and_reconstruct_plaintext"
},
{
"invariant": "external-shard-bitrot",
"suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
"name": "pinned_erasure_shards_reject_bitrot_corruption"
},
{
"invariant": "external-shard-quorum-codec",
"suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
"name": "pinned_erasure_shards_reject_insufficient_quorum_and_wrong_codec"
}
],
"fixtures": [
@@ -372,21 +302,6 @@
"path": "crates/ecstore/tests/fixtures/list_namespace_keys.json",
"sha256": "5afddb60565deba0938e617470255563f0e15a15fab699645c59bb7ccc84898e",
"source": "Shared static LIST namespace corpus; ECStore differential tests and ODM list_through_static_namespace_boundary_matrix"
},
{
"path": "crates/ecstore/tests/fixtures/erasure-shards/minio/manifest.json",
"sha256": "c398f8dcb76903f6cf8eee07931c76592895ced201ff9d59ed4ff9b0ead2b13c",
"source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/erasure-shards/README.md"
},
{
"path": "crates/ecstore/tests/fixtures/erasure-shards/legacy/manifest.json",
"sha256": "dce397d5b2f42748eb5091b4355bb9d133a50cbba003626fafac28095dbe540b",
"source": "RustFS 1.0.0-alpha.37; crates/ecstore/tests/fixtures/erasure-shards/README.md"
},
{
"path": "crates/ecstore/tests/fixtures/erasure-shards/plaintext.bin",
"sha256": "b5a83332327961a31b65a8731c64559fd04ff8ac9f6124022201dbbdba5ffa01",
"source": "Independent fixed SHA256 counter-stream oracle; crates/ecstore/tests/fixtures/erasure-shards/README.md"
}
]
}
-3
View File
@@ -52,9 +52,6 @@ script-tests: ## Run shell script tests
./scripts/check_embedded_secrets.sh --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
+3 -24
View File
@@ -88,21 +88,10 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
setup = 'ecstore-large-stack'
# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
[[profile.default.scripts]]
filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
setup = 'ecstore-base-stack'
[[profile.default.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
# These handler/store fixtures and the relocated-pool GET compose deep storage
# futures in libtest threads; on Linux they overflow the default 2 MiB stack.
[[profile.default.scripts]]
filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
setup = 'ecstore-base-stack'
[[profile.default.scripts]]
filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
setup = 'lifecycle-large-stack'
@@ -181,11 +170,10 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
test-group = 'embedded-test-ports'
# Registration, inventory delivery, and real drive/object probes include
# filesystem or HTTP IO in short deadlines. Reserve capacity so sibling test
# durable filesystem IO in short deadlines. Reserve capacity so sibling test
# processes cannot exhaust those budgets. Keep deadlines and assertions intact.
# The short HTTP fixtures are tracked in rustfs/backlog#2714.
[[profile.default.overrides]]
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
threads-required = "num-test-threads"
# This cancellation probe hashes the serving executable before acquiring its
@@ -344,19 +332,10 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
setup = 'ecstore-large-stack'
# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
[[profile.ci.scripts]]
filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
setup = 'lifecycle-large-stack'
@@ -449,7 +428,7 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
test-group = 'embedded-test-ports'
[[profile.ci.overrides]]
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
threads-required = "num-test-threads"
# Preserve the cancellation probe's startup budget and assertions in CI too.
@@ -1748,7 +1748,7 @@
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
"expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count{job=~\"$job\", bucket=~\"$bucket\"})",
"legendFormat": "{{bucket}}",
"range": true,
"refId": "A"
@@ -1848,7 +1848,7 @@
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
"expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
"legendFormat": "{{bucket}}",
"range": true,
"refId": "A"
-1
View File
@@ -61,7 +61,6 @@ scrape_configs:
otlp:
promote_resource_attributes:
- service.instance.id
- process.pid
- service.name
- service.namespace
- cloud.availability_zone
@@ -1,48 +0,0 @@
evaluation_interval: 1m
tests:
- name: bucket usage keeps clusters separate and deduplicates reporting nodes
interval: 1m
input_series:
- series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
values: '4096 1024'
- series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
values: '4096 1024'
- series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
values: '65536 65536'
- series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
values: '0 0'
- series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
values: '3 1'
- series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
values: '3 1'
- series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
values: '70 70'
- series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
values: '0 0'
promql_expr_test:
- expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
eval_time: 1m
exp_samples:
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
value: 1024
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
value: 0
- expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
eval_time: 1m
exp_samples:
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
value: 1
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
value: 0
- expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes)
eval_time: 1m
exp_samples:
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
value: 1024
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
value: 0
- labels: '{rustfs_cluster_id="b",job="rustfs",bucket="shared"}'
value: 65536
- expr: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",bucket="unknown"}
eval_time: 1m
exp_samples: []
-4
View File
@@ -98,10 +98,6 @@ runs:
shell: bash
run: make script-tests
- name: Check object generation protocol model
shell: bash
run: python3 ./scripts/test_object_generation_protocol_model.py
- name: Check test wiring
shell: bash
run: python3 ./scripts/check_test_wiring.py
@@ -1,39 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Set up Docker Compose
description: Install the verified Compose CLI used by native Linux x86_64 Connect acceptance
runs:
using: composite
steps:
- name: Install verified Docker Compose CLI plugin
shell: bash
run: |
set -euo pipefail
[[ $(uname -s) == Linux && $(uname -m) == x86_64 ]]
plugin_dir="${DOCKER_CONFIG:-$HOME/.docker}/cli-plugins"
mkdir -p "$plugin_dir"
compose_download=$(mktemp "${RUNNER_TEMP:?}/connect-compose.XXXXXX")
compose_staged=$(mktemp "$plugin_dir/.docker-compose.XXXXXX")
trap 'rm -f "$compose_download" "$compose_staged"' EXIT
curl --fail --location --retry 3 \
https://github.com/docker/compose/releases/download/v2.40.3/docker-compose-linux-x86_64 \
--output "$compose_download"
printf '%s %s\n' \
dba9d98e1ba5bfe11d88c99b9bd32fc4a0624a30fafe68eea34d61a3e42fd372 \
"$compose_download" | sha256sum --check --strict
install -m 0755 "$compose_download" "$compose_staged"
mv -f "$compose_staged" "$plugin_dir/docker-compose"
[[ $(docker compose version --short) == 2.40.3 ]]
+1 -15
View File
@@ -36,16 +36,6 @@ inputs:
description: "Install cross-compilation tools"
required: false
default: "false"
zig-version:
description: >-
Zig version for cargo-zigbuild builds. Pinned so the `<target>.<glibc>`
baseline semantics cannot drift with an untested zig release.
required: false
default: "0.17.0"
cargo-zigbuild-version:
description: "cargo-zigbuild version installed for cross builds."
required: false
default: "0.23.4"
target:
description: "Target architecture to add"
required: false
@@ -109,14 +99,10 @@ runs:
- name: Install Zig
if: inputs.install-cross-tools == 'true'
uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2
with:
version: ${{ inputs.zig-version }}
- name: Install cargo-zigbuild
if: inputs.install-cross-tools == 'true'
uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
with:
tool: cargo-zigbuild@${{ inputs.cargo-zigbuild-version }}
- name: Install cargo-nextest
if: inputs.install-test-tools == 'true'
@@ -140,5 +126,5 @@ runs:
# normalises, so cargo fingerprints stay valid.
cache-all-crates: false
cache-on-failure: true
shared-key: ${{ inputs.cache-shared-key }}${{ inputs.install-cross-tools == 'true' && format('-zig-{0}-zigbuild-{1}', inputs.zig-version, inputs.cargo-zigbuild-version) || '' }}
shared-key: ${{ inputs.cache-shared-key }}
save-if: ${{ inputs.cache-save-if }}
+3 -3
View File
@@ -88,7 +88,7 @@ jobs:
# database lookup with a second ignore list to maintain.
cargo-deny:
name: Cargo Deny
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: sm-standard-2
timeout-minutes: 20
steps:
@@ -131,7 +131,7 @@ jobs:
workflow-pin-report:
name: Workflow Pin Report
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: sm-standard-2
timeout-minutes: 5
steps:
@@ -203,7 +203,7 @@ jobs:
# workflow produces, and until now it was only visible to whoever happened to
# open the Actions tab. Same ci-8 mechanism coverage.yml and
# e2e-replication-nightly.yml already use.
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
+39 -67
View File
@@ -213,12 +213,12 @@ jobs:
fi
all='{"include":[
{"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
{"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
{"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
{"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
{"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
{"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
{"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""}
]}'
if [[ "${selected}" == "all" ]]; then
@@ -246,8 +246,7 @@ jobs:
echo "Selected platforms: ${selected}"
build-rustfs:
# Acceptance workflows select artifacts by this stable platform name.
name: Build RustFS (${{ matrix.target_id }})
name: Build RustFS
needs: [ build-check, prepare-platform-matrix ]
if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
runs-on: ${{ matrix.os }}
@@ -274,7 +273,7 @@ jobs:
with:
rust-version: stable
target: ${{ matrix.target }}
cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}
cache-shared-key: build-${{ matrix.target }}
# main only. A cache saved on refs/tags/X is scoped to that tag: no
# other tag, no main run and no PR can restore it, so every release
# cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
@@ -411,64 +410,35 @@ jobs:
shell: bash
env:
BUILD_TARGET: ${{ matrix.target }}
BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }}
run: |
# Force rebuild by touching build.rs
touch rustfs/build.rs
BINARY_NAMES=(rustfs)
if [[ "${{ matrix.platform }}" == "linux" ]]; then
BINARY_NAMES+=(rustfs-cli)
fi
FEATURE_ARGS=()
if [[ -n "${{ matrix.features }}" ]]; then
FEATURE_ARGS=(--features "${{ matrix.features }}")
fi
# gnu targets pin an explicit glibc baseline (`<target>.<glibc>` for
# cargo-zigbuild) so release binaries link no symbols newer than it
# and run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
TARGET_ARG="$BUILD_TARGET"
if [[ -n "${{ matrix.glibc }}" ]]; then
TARGET_ARG="${TARGET_ARG}.${{ matrix.glibc }}"
fi
PROFILE_ARGS=()
if [[ "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
# The in-process profiler resolves names from the final executable.
# Retain function symbols without enabling DWARF source information.
PROFILE_ARGS=(--config 'profile.release.strip="none"')
fi
if [[ "${{ matrix.cross }}" == "true" ]]; then
cargo zigbuild --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
else
cargo build --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
fi
# All release targets use same-architecture runners, including GNU
# builds that use Zig only to select the glibc ABI floor.
bin="target/${{ matrix.target }}/release/rustfs"
if [[ -f "${bin}.exe" ]]; then
bin="${bin}.exe"
fi
version_output=$("$bin" --version)
if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
! grep -Fxq "build os : ${BUILD_OS}" <<< "$version_output" || \
! grep -Fxq "rust channel : stable-${BUILD_TARGET}" <<< "$version_output"; then
echo "release binary reports an unexpected compile target" >&2
exit 1
fi
if [[ -n "${{ matrix.glibc }}" ]]; then
python3 scripts/release/check_glibc.py "${{ matrix.glibc }}" \
"target/${{ matrix.target }}/release/rustfs"
fi
- name: Verify macOS binary has no dynamic liblzma dependency
if: matrix.platform == 'macos'
shell: bash
run: |
binary="target/${{ matrix.target }}/release/rustfs"
if otool -L "$binary" | grep -E 'liblzma.*\.dylib'; then
echo "macOS release binary must not depend on a dynamic liblzma library" >&2
exit 1
fi
"$binary" --version
# Build one binary at a time so release LTO links cannot exhaust a
# self-hosted runner by running concurrently.
for binary in "${BINARY_NAMES[@]}"; do
PROFILE_ARGS=()
if [[ "$binary" == "rustfs" && "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
# The in-process profiler resolves names from the final executable.
# Retain function symbols without enabling DWARF source information.
PROFILE_ARGS=(--config 'profile.release.strip="none"')
fi
if [[ "${{ matrix.cross }}" == "true" ]]; then
# All cross targets in the matrix are Linux; zigbuild handles them.
cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
else
cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
fi
done
- name: Generate the CPU symbol catalog
if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
@@ -573,6 +543,9 @@ jobs:
fi
BINARY_FILES=("$BINARY_NAME")
if [[ "${{ matrix.platform }}" == "linux" ]]; then
BINARY_FILES+=("rustfs-cli")
fi
if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256)
@@ -687,7 +660,7 @@ jobs:
echo "🔧 Build type: ${BUILD_TYPE}"
echo "📊 Version: ${VERSION}"
- name: Verify packaged Linux binary
- name: Verify packaged Linux binaries
if: matrix.platform == 'linux'
shell: bash
run: |
@@ -695,7 +668,7 @@ jobs:
package_file="${{ steps.package.outputs.package_file }}"
archive_members="$(unzip -Z1 "$package_file" | sort)"
expected_members='rustfs'
expected_members=$'rustfs\nrustfs-cli'
target="${{ matrix.target }}"
features="${{ matrix.features }}"
if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then
@@ -707,14 +680,13 @@ jobs:
exit 1
fi
# musl targets build natively; the gnu legs are same-arch
# cargo-zigbuild builds pinned to the release glibc baseline. Either
# way the binaries run on their runner, so executing them here
# Every Linux target builds natively, so executing the binaries here
# proves each artifact matches its runner architecture.
binary_dir="$(mktemp -d)"
trap 'rm -rf "$binary_dir"' EXIT
unzip -q "$package_file" rustfs -d "$binary_dir"
"$binary_dir/rustfs" --version
cli_dir="$(mktemp -d)"
trap 'rm -rf "$cli_dir"' EXIT
unzip -q "$package_file" rustfs rustfs-cli -d "$cli_dir"
"$cli_dir/rustfs" --version
"$cli_dir/rustfs-cli" --help >/dev/null
- name: Verify packaged console
if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
+1 -1
View File
@@ -99,7 +99,7 @@ env:
RUST_BACKTRACE: 1
jobs:
# Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary,
# Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
# e2e-tests, e2e-full.
warm-ci-dev:
name: Warm ci-dev
+86 -56
View File
@@ -36,9 +36,7 @@ on:
- "flake.lock"
pull_request:
types: [ opened, synchronize, reopened, closed ]
# gateway/integration carries main's ruleset (rustfs/backlog#2734), so its
# pull requests must produce the same required checks.
branches: [ main, gateway/integration ]
branches: [ main ]
merge_group:
types: [ checks_requested ]
schedule:
@@ -168,10 +166,13 @@ jobs:
echo "started_at=$(date --utc --iso-8601=seconds)"
} > artifacts/test-and-lint/run-metadata.txt
# Keep full Clippy validation on main, merge groups, and scheduled/manual
# runs. PRs retain the complete default test suite and compile-free checks.
# Clippy runs before the test pass: lint failures are the most common
# CI-only breakage and should surface in minutes, not after 20+ minutes
# of tests.
# Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
# experiment, since --all-targets is check-only for workspace members and
# never links the ~100 test binaries the limit exists to throttle.
- name: Run clippy lints
if: github.event_name != 'pull_request'
run: |
./scripts/ci/resource_sampler.sh start clippy
trap './scripts/ci/resource_sampler.sh stop' EXIT
@@ -280,49 +281,6 @@ jobs:
- name: Verify rebalance/decommission migration proofs
run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
# The workspace already compiled these ignored tests. Keep the same Cargo
# package selection to reuse their binaries, then run them serially in
# separate processes because they share fixed ports and global state.
# test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
# its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
# disk_index 0), not an EC metadata-distribution issue.
# restore_object_usecase_reports_ongoing_conflict_and_completion was
# re-enabled by backlog#1304 (restore accepts serialize on a short CAS
# guard; the copy-back no longer holds the #4877 whole-copy-back lock,
# so the mid-restore ongoing read and fast 409 rejection it asserts are
# the implemented contract).
- name: Run ignored ILM integration tests serially
if: needs.classify-changes.outputs.mode == 'full'
env:
# Match the preceding workspace build and reuse its feature graph.
CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
run: |
mkdir -p artifacts/ilm-integration
set +e
NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
cargo nextest run --profile default -j1 --run-ignored ignored-only \
--all --exclude e2e_test \
-E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
--status-level all --final-status-level all \
2>&1 | tee artifacts/ilm-integration/nextest.log
status=${PIPESTATUS[0]}
{
echo "exit_status=${status}"
echo "finished_at=$(date --utc --iso-8601=seconds)"
echo
echo "Remaining test-related processes:"
pgrep -af 'cargo|nextest|target/.*/deps/' || true
} > artifacts/ilm-integration/diagnostics.txt
exit "${status}"
- name: Upload ILM test diagnostics
if: always() && needs.classify-changes.outputs.mode == 'full'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
path: artifacts/ilm-integration
retention-days: 3
# This gate builds into a fresh target directory to isolate the E2E root.
# Give its cold build a separate budget from workspace tests and migration proofs.
offline-enrollment-e2e:
@@ -354,11 +312,83 @@ jobs:
- name: Check offline enrollment E2E root boundary
run: ./scripts/check_offline_enrollment_e2e.sh
# Dedicated serial lane for the ILM / lifecycle integration tests. These tests
# drive the object layer through process-global singletons (the GLOBAL_ENV
# ECStore, the global tier-config manager, background-expiry workers) and bind
# fixed ports (9002 for the scanner suite, 9003 for the app suite), so they are
# marked #[ignore] to keep them out of the default parallel `cargo nextest run
# --all` pass. Run them here explicitly with `--run-ignored ignored-only` and
# `-j1` so no two of them share global state or race for a port at once.
# serial_test's #[serial] does NOT serialize across nextest's process-per-test
# boundary (see .config/nextest.toml); `-j1` is what actually serializes them.
# See rustfs/backlog#1148 (ilm-1) and #1155.
test-ilm-integration-serial:
name: ILM Integration (serial)
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
# test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
# its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
# disk_index 0), not an EC metadata-distribution issue.
# restore_object_usecase_reports_ongoing_conflict_and_completion was
# re-enabled by backlog#1304 (restore accepts serialize on a short CAS
# guard; the copy-back no longer holds the #4877 whole-copy-back lock,
# so the mid-restore ongoing read and fast 409 rejection it asserts are
# the implemented contract).
- name: Run ignored ILM integration tests serially
env:
# Match the measured Test and Lint link budget. The default exposed
# all 14 pod CPUs and a cold cache spent the full 80m compiling
# without starting one ILM test (main run 32982910990).
CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
run: |
mkdir -p artifacts/ilm-integration
set +e
NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
cargo nextest run -j1 --run-ignored ignored-only \
-p rustfs-scanner -p rustfs \
-E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
--status-level all --final-status-level all \
2>&1 | tee artifacts/ilm-integration/nextest.log
status=${PIPESTATUS[0]}
{
echo "exit_status=${status}"
echo "finished_at=$(date --utc --iso-8601=seconds)"
echo
echo "Remaining test-related processes:"
pgrep -af 'cargo|nextest|target/.*/deps/' || true
} > artifacts/ilm-integration/diagnostics.txt
exit "${status}"
- name: Upload ILM test diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
path: |
artifacts/ilm-integration
target/nextest/ci/junit.xml
test-and-lint-rio-v2:
name: Test and Lint (rio-v2)
# Alternate feature configurations run on main, merge groups, and scheduled
# or manual validation; the default workspace remains the PR gate.
if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
@@ -474,12 +504,12 @@ jobs:
test-and-lint-protocols:
name: "Test and Lint (${{ matrix.features.name }})"
if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
strategy:
# Preserve independent results when one protocol fails, so diagnosis
# Preserve both independent results when one protocol fails, so diagnosis
# and a focused fix do not require rebuilding an interrupted sibling.
fail-fast: false
matrix:
@@ -488,8 +518,6 @@ jobs:
flags: "--features swift"
- name: sftp
flags: "--features sftp"
- name: http3
flags: "--features http3,swift"
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
@@ -1232,6 +1260,7 @@ jobs:
- quick-checks
- test-and-lint
- offline-enrollment-e2e
- test-ilm-integration-serial
- test-and-lint-rio-v2
- connect-short-credential-boundary
- test-and-lint-protocols
@@ -1265,6 +1294,7 @@ jobs:
- quick-checks
- test-and-lint
- offline-enrollment-e2e
- test-ilm-integration-serial
- test-and-lint-rio-v2
- test-and-lint-protocols
- build-rustfs-debug-binary
@@ -57,14 +57,6 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -102,7 +94,7 @@ jobs:
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
@@ -145,7 +137,7 @@ jobs:
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
assert {item.filename for item in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
assert {item.filename for item in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
assert all(regular(item) and 0 < item.file_size <= 1073741824 for item in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
@@ -170,7 +162,7 @@ jobs:
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/diagnostic-schedule-evidence.test.sh)
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_DIAGNOSTIC_SCHEDULE_EVIDENCE="$GITHUB_WORKSPACE/diagnostic-schedule-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=diagnostic-schedule
@@ -57,14 +57,6 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -135,7 +127,7 @@ jobs:
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -180,7 +172,7 @@ jobs:
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
assert {entry.filename for entry in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
@@ -207,7 +199,7 @@ jobs:
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/health-service-evidence.test.sh)
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_HEALTH_SERVICE_EVIDENCE="$GITHUB_WORKSPACE/health-service-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=health-service
@@ -73,14 +73,6 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -115,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PERFORMANCE_DRIVE_EVIDENCE="$GITHUB_WORKSPACE/performance-drive-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=performance-drive
@@ -73,14 +73,6 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -115,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PERFORMANCE_NETWORK_EVIDENCE="$GITHUB_WORKSPACE/performance-network-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=performance-network
@@ -77,14 +77,6 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -126,7 +118,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.jobs[] | select(.name == $name)] as $matches
@@ -181,7 +173,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_CPU_CATALOG="$GITHUB_WORKSPACE/binary/rustfs.cpu-symbol-catalog.json" \
CONNECT_E2E_CPU_CATALOG_SHA256="$CATALOG_SHA256" \
@@ -57,14 +57,6 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -135,7 +127,7 @@ jobs:
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -180,7 +172,7 @@ jobs:
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
@@ -206,7 +198,7 @@ jobs:
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/profile-memory-evidence.test.sh)
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PROFILE_MEMORY_EVIDENCE="$GITHUB_WORKSPACE/profile-memory-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=profile-memory
@@ -81,14 +81,6 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -156,7 +148,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -209,7 +201,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PROFILE_THREADS_EVIDENCE="$GITHUB_WORKSPACE/profile-threads-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=profile-threads
@@ -73,14 +73,6 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -115,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_API_EVIDENCE="$GITHUB_WORKSPACE/top-api-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-api
@@ -68,14 +68,6 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -146,7 +138,7 @@ jobs:
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -191,7 +183,7 @@ jobs:
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
@@ -221,7 +213,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_DISK_EVIDENCE="$GITHUB_WORKSPACE/top-disk-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-disk
@@ -73,14 +73,6 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -115,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_LOCKS_EVIDENCE="$GITHUB_WORKSPACE/top-locks-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-locks
@@ -73,14 +73,6 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -115,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_RPC_EVIDENCE="$GITHUB_WORKSPACE/top-rpc-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-rpc
+1 -4
View File
@@ -78,7 +78,6 @@ concurrency:
jobs:
distributed:
if: github.repository == 'rustfs/rustfs'
name: Distributed 4-node 4-disk e2e
# GitHub-hosted VM: loop and tmpfs mounts work here. sm-standard-4 is an
# ARC pod and rejects both (`mount -o loop` ENOENT, tmpfs "read-only").
@@ -168,13 +167,11 @@ jobs:
FILTER: ${{ inputs.filter }}
run: |
set -euo pipefail
rm -f target/nextest/e2e-distributed/junit.xml
if [ -n "${FILTER}" ]; then
python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test -E "${FILTER}"
else
python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test --no-tests=fail
fi
test -s target/nextest/e2e-distributed/junit.xml
- name: Upload distributed e2e diagnostics
if: always()
@@ -203,7 +200,7 @@ jobs:
alert-on-failure:
name: Alert on scheduled failure
needs: [distributed]
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
@@ -53,7 +53,6 @@ concurrency:
jobs:
repl-nightly:
if: github.repository == 'rustfs/rustfs'
name: Replication e2e (nightly)
# dual-node tests spawn two full rustfs servers each; the extra headroom
# over the per-PR sm-standard-2 e2e job keeps the parallel process fan-out
@@ -120,7 +119,6 @@ jobs:
if-no-files-found: ignore
cluster-nightly:
if: github.repository == 'rustfs/rustfs'
name: Cluster fault e2e (nightly)
runs-on: sm-standard-4
timeout-minutes: 90
@@ -170,7 +168,6 @@ jobs:
if-no-files-found: warn
protocols-nightly:
if: github.repository == 'rustfs/rustfs'
name: Protocol e2e (nightly)
runs-on: sm-standard-4
timeout-minutes: 90
@@ -232,7 +229,7 @@ jobs:
# Only scheduled runs open/append the tracking issue (backlog#1149 ci-8);
# manual workflow_dispatch runs stay quiet so a debugging run never files a
# spurious alert.
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
+41 -37
View File
@@ -96,8 +96,38 @@ jobs:
overwrite: true
upgrade:
name: Upgrade compatibility
name: ${{ matrix.name }}
needs: build
strategy:
fail-fast: false
matrix:
include:
# The two `_from_rc2_` tests keep their names: they assert
# release-independent object contracts and pass unchanged against the
# newer pinned source, so renaming them would only churn history and
# the CI required-check names. UPGRADE_SOURCE_VERSION above is the
# single source of truth for which release they actually run against.
- name: Direct upgrade from the previous release
test: direct_upgrade_from_rc2_preserves_object_contracts
artifact: direct-upgrade
- name: Mixed-version rolling upgrade from the previous release
test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
artifact: mixed-version-upgrade
- name: Bucket configuration survives the upgrade
test: direct_upgrade_from_previous_release_preserves_bucket_configuration
artifact: bucket-config-upgrade
- name: Rollback reads current bucket metadata
test: rollback_to_previous_release_reads_current_bucket_metadata
artifact: bucket-config-rollback
- name: ODM configuration recovery after rc.5 rollback
test: rc5_rollback_requires_restoring_odm_configuration
artifact: odm-config-rollback
- name: Multipart layouts survive the rc.5 upgrade
test: direct_upgrade_from_rc5_preserves_multipart_layouts
artifact: multipart-layout-upgrade
- name: rc.5 multipart replication baseline
test: rc5_baseline_replicates_multipart_layouts
artifact: multipart-layout-baseline
runs-on: sm-standard-2
timeout-minutes: 60
env:
@@ -112,7 +142,8 @@ jobs:
uses: ./.github/actions/setup
with:
cache-shared-key: e2e-upgrade-tests
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# One main-branch consumer warms the shared test-harness dependencies.
cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
@@ -139,49 +170,22 @@ jobs:
echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
# Reuse one compiled harness, while keeping each case in its own process.
# Continue after a failure to retain the former matrix's complete verdict.
- name: Run upgrade compatibility tests
shell: bash
- name: Run upgrade compatibility test
env:
RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
NO_PROXY: 127.0.0.1,localhost,::1
run: |
export no_proxy="$NO_PROXY"
cases=(
direct-upgrade:direct_upgrade_from_rc2_preserves_object_contracts
mixed-version-upgrade:rolling_upgrade_from_rc2_preserves_mixed_version_contracts
bucket-config-upgrade:direct_upgrade_from_previous_release_preserves_bucket_configuration
bucket-config-rollback:rollback_to_previous_release_reads_current_bucket_metadata
odm-config-rollback:rc5_rollback_requires_restoring_odm_configuration
multipart-layout-upgrade:direct_upgrade_from_rc5_preserves_multipart_layouts
multipart-layout-baseline:rc5_baseline_replicates_multipart_layouts
)
status=0
for entry in "${cases[@]}"; do
artifact="${entry%%:*}"
test="${entry#*:}"
export RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR="$RUNNER_TEMP/rustfs-upgrade-g09-evidence/$artifact"
export RUSTFS_E2E_LOG_DIR="$RUNNER_TEMP/rustfs-upgrade-logs/$artifact"
mkdir -p "$RUSTFS_E2E_LOG_DIR"
echo "::group::$test"
if python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
"upgrade_compatibility_test::$test" -- --ignored --exact --nocapture \
2>&1 | tee "$RUSTFS_E2E_LOG_DIR/test.log"; then
echo "PASS: $test"
else
echo "::error::Upgrade compatibility failed: $test"
status=1
fi
echo "::endgroup::"
done
exit "$status"
python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
"upgrade_compatibility_test::${{ matrix.test }}" \
-- --ignored --exact --nocapture
- name: Upload scanner/heal G09 evidence
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: upgrade-g09-evidence-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence
name: ${{ matrix.artifact }}-g09-evidence-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
if-no-files-found: ignore
retention-days: 14
@@ -189,7 +193,7 @@ jobs:
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: upgrade-server-logs-${{ github.run_number }}
name: ${{ matrix.artifact }}-server-logs-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-logs
if-no-files-found: warn
retention-days: 14
@@ -11,7 +11,6 @@ concurrency:
cancel-in-progress: false
jobs:
collect:
if: github.repository == 'rustfs/rustfs'
runs-on: smoke-testing
timeout-minutes: 15
steps:
+4 -6
View File
@@ -72,11 +72,9 @@ jobs:
fuzz-build:
name: Build Fuzz Harness
if: >
github.repository == 'rustfs/rustfs' && (
(github.event_name == 'pull_request' && github.event.action != 'closed') ||
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch'
)
(github.event_name == 'pull_request' && github.event.action != 'closed') ||
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch'
runs-on: sm-standard-4
timeout-minutes: 60
env:
@@ -241,7 +239,7 @@ jobs:
# `always()` is required: without it this job is skipped when a needed
# job fails. Alerts only for scheduled (nightly) runs (backlog#1149
# ci-8); PR and manual dispatch failures are already watched by a human.
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
@@ -1,197 +0,0 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Gateway Integration Sync
# Merges `main` into the long-lived `gateway/integration` branch every weekday
# so the s3s -> gateway migration (rustfs/backlog#2734, task T0.1 in #2736)
# never accumulates merge debt. A clean merge lands as a `--no-ff` merge commit
# pushed straight to the branch. A conflicting merge is aborted and surfaced as
# a pull request from `sync/<source>-<date>`, whose head is the un-merged
# source commit, so the conflict is resolved the same day, in review, with no
# other change riding along. The run itself stays green in that case: the pull
# request is the signal. Any other failure (push rejected, pull request not
# created) fails the run and, on the schedule, files the shared failure issue.
#
# Resolving a conflict pull request: merge `origin/gateway/integration` into
# the sync branch, resolve, push, get the review. Land it with a fast-forward
# push of the resolved merge commit to `gateway/integration`, not with the
# squash button: a squash drops the merge ancestry and the same hunks conflict
# again on the next run.
#
# Write scope: only `gateway/integration` and `sync/*` are ever pushed, and
# both push targets below are literal refs. The pushes use a one-hour
# installation token of the organisation's own GitHub App (contents: write)
# because that app is the only non-admin bypass actor of the
# `gateway/integration` ruleset; GITHUB_TOKEN cannot be a ruleset bypass actor,
# so it cannot push through the required reviews and checks. The conflict pull
# request is opened with FLAKE_UPDATE_TOKEN, the PAT that already opens the
# weekly flake.lock pull requests, because this repository keeps "Allow GitHub
# Actions to create and approve pull requests" switched off. GITHUB_TOKEN is
# therefore read-only here.
#
# `schedule` fires only from the default branch, so the cron is live once this
# file exists on `main`. `workflow_dispatch` runs from any ref that carries the
# file and always acts on `gateway/integration`.
on:
schedule:
- cron: "0 2 * * 1-5" # Weekdays 02:00 UTC
workflow_dispatch:
inputs:
source_ref:
description: >-
Branch to merge into gateway/integration. The schedule always merges
main; only the conflict acceptance test (rustfs/backlog#2736) sets
another branch.
required: false
default: main
type: string
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: false
env:
TARGET_BRANCH: gateway/integration
jobs:
sync:
name: Merge main into gateway/integration
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Create the branch push token
id: push-token
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3
with:
app-id: ${{ vars.CLA_BOT_APP_ID }}
private-key: ${{ secrets.CLA_BOT_APP_PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
repositories: rustfs
permission-contents: write
- name: Checkout gateway/integration
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push
# credential — the merge commit and the sync branch are pushed with it
# below. It is a one-hour installation token scoped to contents.
ref: gateway/integration
fetch-depth: 0
token: ${{ steps.push-token.outputs.token }}
- name: Configure the bot identity
run: |
git config user.name "rustfs-cla-assistant[bot]"
git config user.email "273121464+rustfs-cla-assistant[bot]@users.noreply.github.com"
- name: Merge the source branch
id: merge
env:
SOURCE_REF: ${{ inputs.source_ref || 'main' }}
run: |
set -euo pipefail
if ! [[ "$SOURCE_REF" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] \
|| [[ "$SOURCE_REF" == *..* ]] \
|| [[ "$SOURCE_REF" == "$TARGET_BRANCH" ]]; then
echo "::error::refusing source_ref '${SOURCE_REF}'"
exit 1
fi
git fetch origin "refs/heads/${SOURCE_REF}:refs/remotes/origin/${SOURCE_REF}"
source_sha=$(git rev-parse "origin/${SOURCE_REF}")
target_sha=$(git rev-parse HEAD)
{
echo "source_ref=${SOURCE_REF}"
echo "source_sha=${source_sha}"
echo "target_sha=${target_sha}"
} >> "$GITHUB_OUTPUT"
if git merge --no-ff --no-edit "origin/${SOURCE_REF}"; then
if [[ "$(git rev-parse HEAD)" == "$target_sha" ]]; then
echo "result=up-to-date" >> "$GITHUB_OUTPUT"
echo "\`${TARGET_BRANCH}\` already contains \`${SOURCE_REF}\` (${source_sha}); nothing to push." >> "$GITHUB_STEP_SUMMARY"
else
echo "result=merged" >> "$GITHUB_OUTPUT"
fi
exit 0
fi
git diff --name-only --diff-filter=U > "$RUNNER_TEMP/conflicts.txt"
git merge --abort
if [[ ! -s "$RUNNER_TEMP/conflicts.txt" ]]; then
echo "::error::git merge failed without reporting conflicting paths; this is not a conflict"
exit 1
fi
echo "result=conflict" >> "$GITHUB_OUTPUT"
- name: Push the merge commit
if: steps.merge.outputs.result == 'merged'
env:
SOURCE_REF: ${{ steps.merge.outputs.source_ref }}
SOURCE_SHA: ${{ steps.merge.outputs.source_sha }}
run: |
set -euo pipefail
git push origin HEAD:refs/heads/gateway/integration
echo "Merged \`${SOURCE_REF}\` (${SOURCE_SHA}) into \`${TARGET_BRANCH}\` as $(git rev-parse HEAD)." >> "$GITHUB_STEP_SUMMARY"
- name: Open the conflict pull request
if: steps.merge.outputs.result == 'conflict'
env:
GH_TOKEN: ${{ secrets.FLAKE_UPDATE_TOKEN }}
SOURCE_REF: ${{ steps.merge.outputs.source_ref }}
SOURCE_SHA: ${{ steps.merge.outputs.source_sha }}
TARGET_SHA: ${{ steps.merge.outputs.target_sha }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
set -euo pipefail
date_utc=$(date -u +%Y-%m-%d)
sync_branch="sync/${SOURCE_REF//\//-}-${date_utc}"
git push origin "refs/remotes/origin/${SOURCE_REF}:refs/heads/${sync_branch}"
{
echo "The scheduled merge of \`${SOURCE_REF}\` into \`${TARGET_BRANCH}\` stopped on conflicts and was aborted; \`${TARGET_BRANCH}\` is unchanged."
echo
echo "This branch is the un-merged \`${SOURCE_REF}\` head. Merge \`origin/${TARGET_BRANCH}\` into it, resolve the files below, push, and get the review. Land the resolved merge commit with a fast-forward push to \`${TARGET_BRANCH}\`; a squash merge drops the merge ancestry and the same hunks conflict again on the next run. Do not include any change that is not conflict resolution."
echo
echo "- Run: ${RUN_URL}"
echo "- Source: \`${SOURCE_REF}\` at ${SOURCE_SHA}"
echo "- Target: \`${TARGET_BRANCH}\` at ${TARGET_SHA}"
echo
echo "## Conflicting files"
echo
echo "Output of \`git diff --name-only --diff-filter=U\` captured before \`git merge --abort\`:"
echo
sed 's/^/- /' "$RUNNER_TEMP/conflicts.txt"
} > "$RUNNER_TEMP/pr-body.md"
title="sync: merge ${SOURCE_REF} ${date_utc} (conflicts)"
existing=$(gh pr list --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --state open --json number --jq '.[0].number // empty')
if [[ -n "$existing" ]]; then
gh pr edit "$existing" --repo "$GITHUB_REPOSITORY" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md"
pr_url=$(gh pr view "$existing" --repo "$GITHUB_REPOSITORY" --json url --jq .url)
else
pr_url=$(gh pr create --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md")
fi
echo "Conflicts in \`${SOURCE_REF}\` -> \`${TARGET_BRANCH}\`; pull request: ${pr_url}" >> "$GITHUB_STEP_SUMMARY"
alert-on-failure:
name: Alert on scheduled failure
needs: [sync]
# Same ci-8 mechanism audit.yml and coverage.yml use: a scheduled run that
# could not push or could not open the conflict pull request must not stay
# visible only in the Actions tab.
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}
+4 -18
View File
@@ -43,7 +43,6 @@ env:
jobs:
resolve-source:
if: github.repository == 'rustfs/rustfs'
name: Resolve nightly source
runs-on: sm-standard-2
timeout-minutes: 10
@@ -80,24 +79,13 @@ jobs:
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
cache-shared-key: build-x86_64-unknown-linux-gnu-glibc-2.31
cache-shared-key: build-x86_64-unknown-linux-gnu
cache-save-if: 'false'
install-cross-tools: 'true'
target: x86_64-unknown-linux-gnu
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Build RustFS
# Pin the glibc baseline (`<target>.<glibc>` for cargo-zigbuild) so
# nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins
- name: Verify glibc symbol baseline
run: |
python3 scripts/release/check_glibc.py 2.31 \
target/x86_64-unknown-linux-gnu/release/rustfs \
target/x86_64-unknown-linux-gnu/release/rustfs-cli
target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu'
run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
- name: Build DEB package
id: deb
@@ -508,10 +496,8 @@ jobs:
name: Alert on scheduled failure
needs: [build, kms-vault-lane, kms-vault-ha-failover]
if: >-
github.repository == 'rustfs/rustfs' && (
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
)
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
-1
View File
@@ -32,7 +32,6 @@ concurrency:
jobs:
update-flake:
if: github.repository == 'rustfs/rustfs'
name: Update flake.lock
runs-on: sm-standard-2
timeout-minutes: 90
+2 -4
View File
@@ -20,9 +20,8 @@ on:
- ".github/workflows/oidc-keycloak.yml"
- "crates/config/src/constants/oidc.rs"
- "crates/iam/src/federation/**"
- "crates/iam/src/oidc/**"
- "crates/iam/src/oidc.rs"
- "rustfs/src/admin/handlers/oidc.rs"
- "rustfs/src/admin/service/oidc_config.rs"
- "rustfs/src/admin/handlers/sts.rs"
- "scripts/test/oidc_keycloak_live.sh"
- "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"
@@ -32,9 +31,8 @@ on:
- ".github/workflows/oidc-keycloak.yml"
- "crates/config/src/constants/oidc.rs"
- "crates/iam/src/federation/**"
- "crates/iam/src/oidc/**"
- "crates/iam/src/oidc.rs"
- "rustfs/src/admin/handlers/oidc.rs"
- "rustfs/src/admin/service/oidc_config.rs"
- "rustfs/src/admin/handlers/sts.rs"
- "scripts/test/oidc_keycloak_live.sh"
- "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"
+2 -5
View File
@@ -49,7 +49,6 @@ env:
jobs:
warp-ab:
if: github.repository == 'rustfs/rustfs'
name: Warp A/B budget gate
runs-on: sm-standard-2
# A normal nightly restores the last successful binary and builds only the
@@ -365,10 +364,8 @@ jobs:
# composite action already reports cancelled/timed-out jobs in the issue
# body.
if: >-
github.repository == 'rustfs/rustfs' && (
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
)
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
+1 -2
View File
@@ -42,7 +42,6 @@ concurrency:
jobs:
check-ephemerality:
if: github.repository == 'rustfs/rustfs'
name: Check runner ephemerality
runs-on: sm-standard-2
timeout-minutes: 15
@@ -66,7 +65,7 @@ jobs:
# Same ci-8 mechanism as coverage.yml, audit.yml and the nightly lanes:
# scheduled runs file a tracking issue, manual dispatch stays quiet so
# debugging never produces a spurious alert.
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
+20 -4
View File
@@ -266,10 +266,7 @@ jobs:
# the previous 45m step cap killed four consecutive runs mid-rebalance
# with steps 1-7 green. 240m covers the observed pace (rebalance 1/4
# in ~21m) plus one decommission pass; override per dispatch.
# fromJSON coerces the string dispatch input to the number that
# timeout-minutes requires; the bare || form yielded the string '240'
# and the runner rejected the template (Unexpected value '240').
timeout-minutes: ${{ fromJSON(inputs.pool_timeout_minutes || '240') }}
timeout-minutes: ${{ inputs.pool_timeout_minutes || 240 }}
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
@@ -586,6 +583,25 @@ jobs:
echo "pool test script reported a failure (step FAIL verdict or die() abort)" >&2
failed=1
fi
# A runner-level kill (step timeout) truncates the suite without a
# FAIL verdict or an [ERROR] line, so require every expected
# [POOL-STEP] marker to be present with PASS - missing markers mean
# the run was truncated, not green. This also closes the gap for
# standalone dispatches, where the chain driver's evidence
# validation does not run.
last_step=6
if [ "${{ inputs.pools || '3' }}" = "3" ]; then
last_step=8
if [ "${{ inputs.run_decommission != 'false' }}" = "true" ]; then
last_step=9
fi
fi
for n in $(seq 1 "${last_step}"); do
grep -q "^\[POOL-STEP\] ${n} .* PASS" "${POOL_ARTIFACT_DIR}/pool-test.log" 2>/dev/null || {
echo "pool test log is missing [POOL-STEP] ${n} PASS (expected through step ${last_step}); the run was truncated" >&2
failed=1
}
done
[ "${failed}" -eq 0 ] || exit 1
- name: Upload functional report to dashboard
@@ -28,7 +28,6 @@ concurrency:
jobs:
check-freshness:
if: github.repository == 'rustfs/rustfs'
name: Check scheduled validation freshness
# Stay on GitHub-hosted runners: this monitor must keep running when the
# self-hosted sm-standard-* pool is down, which is exactly when the
@@ -39,7 +39,6 @@ jobs:
alert-on-incomplete-run:
name: Alert on incomplete scheduled run
if: >-
github.repository == 'rustfs/rustfs' &&
github.event.workflow_run.event == 'schedule' &&
github.event.workflow_run.conclusion != 'success' &&
github.event.workflow_run.conclusion != 'failure'
-1
View File
@@ -26,7 +26,6 @@ on:
jobs:
stale:
if: github.repository == 'rustfs/rustfs'
runs-on: sm-standard-2
timeout-minutes: 30
steps:
-1
View File
@@ -14,7 +14,6 @@ concurrency:
jobs:
update:
if: github.repository == 'rustfs/rustfs'
runs-on: sm-standard-2
timeout-minutes: 10
steps:
+2 -5
View File
@@ -40,7 +40,6 @@ env:
jobs:
targets-live:
if: github.repository == 'rustfs/rustfs'
name: PostgreSQL, MySQL, AMQP, and NATS
runs-on: sm-standard-2
timeout-minutes: 90
@@ -176,10 +175,8 @@ jobs:
name: Alert on scheduled failure
needs: [targets-live]
if: >-
github.repository == 'rustfs/rustfs' && (
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
)
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
-1
View File
@@ -90,4 +90,3 @@ worktrees/*
# insta scratch files; the accepted .snap files ARE the assertions and are committed
*.snap.new
*.pending-snap
.mimosa
+28 -22
View File
@@ -7,9 +7,7 @@
"name": "Debug(only) executable 'rustfs'",
"env": {
"RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info",
"RUSTFS_SKIP_BACKGROUND_TASK": "on",
// Local test volumes share one physical disk.
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true"
"RUSTFS_SKIP_BACKGROUND_TASK": "on"
//"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
// "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
// "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
@@ -46,7 +44,6 @@
"env": {
"RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
"RUSTFS_SKIP_BACKGROUND_TASK": "on",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
//"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
// "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
// "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
@@ -120,8 +117,7 @@
"env": {
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
//"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
// "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
@@ -139,31 +135,42 @@
{
"type": "lldb",
"request": "launch",
"name": "Debug test_transition_and_restore_flows",
"name": "Debug test_lifecycle_transition_basic",
"cargo": {
"args": [
"test",
"--no-run",
"-p",
"rustfs-scanner",
"--test",
"lifecycle_integration_test",
"serial_tests::test_lifecycle_transition_basic",
"-j",
"1"
],
"filter": {
"name": "lifecycle_integration_test",
"kind": "test"
}
]
},
"args": [
"serial_tests::test_transition_and_restore_flows",
"--exact",
"--ignored",
"--test-threads=1"
],
"args": [],
"cwd": "${workspaceFolder}"
},
{
"name": "Debug executable target/debug/test",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/deps/lifecycle_integration_test-5915cbfcab491b3b",
"args": [
"--skip",
"test_lifecycle_expiry_basic",
"--skip",
"test_lifecycle_expiry_deletemarker",
//"--skip",
//"test_lifecycle_transition_basic",
],
"cwd": "${workspaceFolder}",
//"stopAtEntry": false,
//"preLaunchTask": "cargo build",
"sourceLanguages": [
"rust"
],
},
{
"name": "Debug executable target/debug/rustfs with sse kms",
"type": "lldb",
@@ -278,7 +285,7 @@
"RUSTFS_COMPRESSION_ENABLED": "true",
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "local",
"RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/rio-v2/kms-key-dir",
"RUSTFS_KMS_KEY_DIR": "./target/rio-v2/kms-key-dir",
"RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
},
@@ -300,7 +307,6 @@
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
"RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
"RUSTFS_REGION": "us-east-1",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
@@ -309,7 +315,7 @@
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "local",
"RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/e2e-key-dir",
"RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir",
"RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
},
Generated
+462 -521
View File
File diff suppressed because it is too large Load Diff
+89 -97
View File
@@ -73,8 +73,8 @@ resolver = "3"
edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.99.0"
version = "1.0.2"
rust-version = "1.98.1"
version = "1.0.1"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -91,56 +91,56 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.2" }
rustfs-heal = { path = "crates/heal", version = "1.0.2" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.2" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.2" }
rustfs-audit = { path = "crates/audit", version = "1.0.2" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.2" }
rustfs-common = { path = "crates/common", version = "1.0.2" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.2" }
rustfs-config = { path = "./crates/config", version = "1.0.2" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.2" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.2" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.2" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.2" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.2" }
rustfs-iam = { path = "crates/iam", version = "1.0.2" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.2" }
rustfs-license = { path = "crates/license", version = "1.0.2" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.2" }
rustfs-kms = { path = "crates/kms", version = "1.0.2" }
rustfs-lock = { path = "crates/lock", version = "1.0.2" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.2" }
rustfs-notify = { path = "crates/notify", version = "1.0.2" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.2" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.2" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.2" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.2", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.2" }
rustfs-obs = { path = "crates/obs", version = "1.0.2" }
rustfs-policy = { path = "crates/policy", version = "1.0.2" }
rustfs-protos = { path = "crates/protos", version = "1.0.2" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.2" }
rustfs-replication = { path = "crates/replication", version = "1.0.2" }
rustfs-rio = { path = "crates/rio", version = "1.0.2" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.2" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.2" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.2" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.2" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.2" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.2" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.2" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.2" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.2" }
rustfs-signer = { path = "crates/signer", version = "1.0.2" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.2" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.2" }
rustfs-targets = { path = "crates/targets", version = "1.0.2" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.2" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" }
rustfs-utils = { path = "crates/utils", version = "1.0.2" }
rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
rustfs = { path = "./rustfs", version = "1.0.1" }
rustfs-heal = { path = "crates/heal", version = "1.0.1" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.1" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.1" }
rustfs-audit = { path = "crates/audit", version = "1.0.1" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.1" }
rustfs-common = { path = "crates/common", version = "1.0.1" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.1" }
rustfs-config = { path = "./crates/config", version = "1.0.1" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.1" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.1" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.1" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.1" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.1" }
rustfs-iam = { path = "crates/iam", version = "1.0.1" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.1" }
rustfs-license = { path = "crates/license", version = "1.0.1" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.1" }
rustfs-kms = { path = "crates/kms", version = "1.0.1" }
rustfs-lock = { path = "crates/lock", version = "1.0.1" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.1" }
rustfs-notify = { path = "crates/notify", version = "1.0.1" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.1" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.1" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.1" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.1", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.1" }
rustfs-obs = { path = "crates/obs", version = "1.0.1" }
rustfs-policy = { path = "crates/policy", version = "1.0.1" }
rustfs-protos = { path = "crates/protos", version = "1.0.1" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.1" }
rustfs-replication = { path = "crates/replication", version = "1.0.1" }
rustfs-rio = { path = "crates/rio", version = "1.0.1" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.1" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.1" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.1" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.1" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.1" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.1" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.1" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.1" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.1" }
rustfs-signer = { path = "crates/signer", version = "1.0.1" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.1" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.1" }
rustfs-targets = { path = "crates/targets", version = "1.0.1" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.1" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.1" }
rustfs-utils = { path = "crates/utils", version = "1.0.1" }
rustfs-zip = { path = "./crates/zip", version = "1.0.1" }
# Async Runtime and Networking
async-channel = "2.5.0"
@@ -156,9 +156,8 @@ futures-lite = "2.6.1"
futures-util = "0.3.34"
pollster = "1.0.1"
pulsar = { default-features = false, version = "6.9.0" }
lapin = { default-features = false, version = "4.12.2" }
hyper = { version = "1.12.0" }
h2 = { version = "0.4.20" }
lapin = { default-features = false, version = "4.12.0" }
hyper = { version = "1.11.1" }
hyper-rustls = { default-features = false, version = "0.27.10" }
hyper-util = { version = "0.1.21" }
http = "1.5.0"
@@ -167,25 +166,18 @@ http-body-util = "0.1.5"
memchr = "2.8.3"
minlz = "1.2.3"
reqwest = "0.13.5"
rustfs-kafka-async = { version = "1.4.0" }
rustfs-kafka-async = { version = "1.3.1" }
socket2 = { version = "0.6.5" }
tokio = { version = "1.53.3" }
tokio = { version = "1.53.1" }
tokio-rustls = { default-features = false, version = "0.26.6" }
tokio-stream = { version = "0.1.19" }
tokio-test = "0.4.6"
tokio-util = { version = "0.7.20" }
tokio-util = { version = "0.7.19" }
tonic = { version = "0.14.6" }
tonic-prost = { version = "0.14.6" }
tonic-prost-build = { version = "0.14.6" }
tower = { version = "0.5.3" }
tower-http = { version = "0.7.1" }
h3 = "0.0.8"
h3-quinn = "0.0.10"
quinn = { version = "0.11.12", default-features = false, features = [
"futures-io",
"runtime-tokio",
"rustls-aws-lc-rs",
] }
# Serialization and Data Formats
apache-avro = { version = "0.22.0", features = ["snappy", "zstandard"] }
@@ -201,7 +193,7 @@ rmp-serde = { version = "1.3.1" }
serde = { version = "1.0.229" }
serde_ignored = { version = "0.1" }
serde_with = { version = "3", default-features = false, features = ["macros", "std"] }
serde_json = { version = "1.0.154" }
serde_json = { version = "1.0.151" }
serde_urlencoded = "0.7.1"
# Cryptography and Security
@@ -225,15 +217,15 @@ rustls-native-certs = "0.8"
rustls-pki-types = "1.15.1"
x509-parser = "0.18.1"
sha1 = "0.11.0"
sha2 = "0.11.1"
sha2 = "0.11.0"
subtle = "2.6"
zeroize = { version = "1.9.1" }
zeroize = { version = "1.9.0" }
proptest = "1"
# Time and Date
chrono = { version = "0.4.45" }
humantime = "2.4.0"
jiff = { version = "0.2.38" }
jiff = { version = "0.2.37" }
time = { version = "0.3.55" }
# Database
@@ -252,9 +244,9 @@ atoi = "3.1.0"
atomic_enum = "0.3.0"
aws-config = { version = "1.12.0" }
aws-credential-types = { version = "1.3.0" }
aws-sdk-kms = { default-features = false, version = "1.123.0" }
aws-sdk-s3 = { default-features = false, version = "1.152.0" }
aws-sdk-sts = { default-features = false, version = "1.119.0" }
aws-sdk-kms = { default-features = false, version = "1.122.0" }
aws-sdk-s3 = { default-features = false, version = "1.151.0" }
aws-sdk-sts = { default-features = false, version = "1.118.0" }
aws-smithy-async = { version = "1.3.0" }
aws-smithy-http-client = { default-features = false, version = "1.5.0" }
aws-smithy-runtime-api = { version = "1.19.0" }
@@ -269,7 +261,7 @@ crossbeam-queue = "0.3.14"
crossbeam-channel = "0.5.17"
crossbeam-deque = "0.8.8"
crossbeam-utils = "0.8.23"
datafusion = { default-features = false, version = "55.2.0" }
datafusion = { default-features = false, version = "55.1.0" }
derive_builder = "0.20.2"
ed25519-dalek = "3.0.0"
enumset = "1.1.14"
@@ -287,7 +279,7 @@ highway = { version = "1.3.0" }
hostname = "0.4.2"
ipnetwork = { version = "0.21.1" }
lazy_static = "1.5.1"
libc = "0.2.190"
libc = "0.2.189"
libsystemd = "0.7.2"
local-ip-address = "0.6.13"
log = "0.4"
@@ -307,7 +299,7 @@ percent-encoding = "2.3.2"
# Server-side QR rendering for TOTP enrollment, so neither the console nor the
# CLI needs its own QR encoder. No default features: the image/render backends
# pull in an image stack this only needs SVG and text output from.
qrcode-rs = { version = "2.2.0", default-features = false, features = ["std", "svg"] }
qrcode-rs = { version = "2.1.1", default-features = false, features = ["std", "svg"] }
pin-project-lite = "0.2.17"
pretty_assertions = "1.4.1"
rand = { version = "0.10.3" }
@@ -320,17 +312,17 @@ rumqttc = { package = "rumqttc-next", version = "0.34.0" }
redis = { version = "1.7.1" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.5" }
rust-embed = { version = "8.13.0" }
rust-embed = { version = "8.12.0" }
rustc-hash = { version = "2.1.3" }
s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
s3s = { version = "0.17.0", features = ["minio"] }
s3s-sigv4 = "0.17.0"
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.4"
smallvec = { version = "1.16.3" }
smallvec = { version = "1.16.2" }
compact_str = "0.10.0"
snap = "1.1.2"
starshard = { version = "2.4.0" }
starshard = { version = "2.3.0" }
strum = { version = "0.28.0" }
sysinfo = "0.39.6"
temp-env = "0.3.6"
@@ -341,12 +333,12 @@ tracing = { version = "0.1.44" }
tracing-appender = "0.2.5"
tracing-core = "0.1.36"
tracing-error = "0.2.1"
tracing-opentelemetry = { version = "0.34.0" }
tracing-opentelemetry = { version = "0.34" }
tracing-subscriber = { version = "0.3.23" }
transform-stream = "0.3.1"
url = "2.5.8"
urlencoding = "2.1.3"
uuid = { version = "1.28.0" }
uuid = { version = "1.26.1" }
vaultrs = { version = "0.8.0" }
tar = "0.4.46"
walkdir = "2.5.0"
@@ -354,28 +346,28 @@ winapi-util = "0.1.11"
windows = { version = "0.62.2" }
windows-sys = "0.61.2"
xxhash-rust = { version = "0.8.19" }
zip = "9.0.3"
zstd = "0.14.2"
zip = "8.6.0"
zstd = "0.14.0"
# Observability and Metrics
metrics = "0.24.6"
metrics-util = "0.20"
dial9 = { version = "0.5.4", default-features = false }
opentelemetry = { version = "0.33.1" }
opentelemetry-appender-tracing = { version = "0.33.1" }
opentelemetry-otlp = { version = "0.33.1" }
opentelemetry-proto = { version = "0.33.1", default-features = false, features = ["metrics", "gen-tonic-messages"] }
opentelemetry_sdk = { version = "0.33.1" }
opentelemetry-semantic-conventions = { version = "0.33.1" }
opentelemetry-stdout = { version = "0.33.1" }
pyroscope = { version = "2.1.2" }
dial9 = { version = "0.5.2", default-features = false }
opentelemetry = { version = "0.33.0" }
opentelemetry-appender-tracing = { version = "0.33.0" }
opentelemetry-otlp = { version = "0.33.0" }
opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
opentelemetry_sdk = { version = "0.33.0" }
opentelemetry-semantic-conventions = { version = "0.33.0" }
opentelemetry-stdout = { version = "0.33.0" }
pyroscope = { version = "2.1.1" }
# FTP and SFTP
libunftp = { version = "0.23.1" }
unftp-core = "0.1.0"
suppaftp = { version = "12.2.0" }
suppaftp = { version = "12.1.1" }
rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.64.1" }
russh = { version = "0.63.3" }
russh-sftp = "3.0.1"
# WebDAV
@@ -384,9 +376,9 @@ dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
rustfs-mimalloc = { version = "0.5.6" }
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
hotpath = { version = "0.28.6", default-features = false }
hotpath = { version = "0.28.0", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.49" }
insta = { version = "1.48" }
[workspace.metadata.cargo-shear]
ignored = ["hotpath", "rustfs"]
-2
View File
@@ -51,8 +51,6 @@ unparseable = "unparseable"
# Disaster-recovery objectives: recovery time and recovery point.
RTO = "RTO"
rto = "rto"
# rustix OFlags variant (POSIX O_WRONLY equivalent).
WRONLY = "WRONLY"
[files]
extend-exclude = []
-1
View File
@@ -34,7 +34,6 @@ hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
arc-swap.workspace = true
hotpath.workspace = true
tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
tonic = { workspace = true, features = ["gzip", "deflate"] }
+5 -131
View File
@@ -12,10 +12,9 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use arc_swap::ArcSwap;
use std::collections::HashMap;
use std::sync::LazyLock;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, LazyLock};
use tokio::sync::RwLock;
use tonic::transport::Channel;
@@ -28,42 +27,6 @@ pub static GLOBAL_ROOT_CERT: LazyLock<RwLock<Option<Vec<u8>>>> = LazyLock::new(|
pub static GLOBAL_MTLS_IDENTITY: LazyLock<RwLock<Option<MtlsIdentityPem>>> = LazyLock::new(|| RwLock::new(None));
pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
#[derive(Clone, Default)]
pub struct OutboundTlsSnapshot {
pub generation: u64,
pub root_ca_pem: Option<Vec<u8>>,
pub mtls_identity: Option<MtlsIdentityPem>,
}
static GLOBAL_OUTBOUND_TLS_STATE: LazyLock<ArcSwap<OutboundTlsSnapshot>> =
LazyLock::new(|| ArcSwap::from_pointee(OutboundTlsSnapshot::default()));
pub fn get_global_outbound_tls_snapshot() -> Arc<OutboundTlsSnapshot> {
GLOBAL_OUTBOUND_TLS_STATE.load_full()
}
/// Publish identity, trust and generation together; the individual globals are legacy views.
pub async fn publish_global_outbound_tls_snapshot(snapshot: OutboundTlsSnapshot) -> u64 {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
if current.generation != 0
&& current.root_ca_pem == snapshot.root_ca_pem
&& current.mtls_identity == snapshot.mtls_identity
{
return Arc::clone(current);
}
Arc::new(OutboundTlsSnapshot {
generation: snapshot.generation.max(current.generation.saturating_add(1)),
root_ca_pem: snapshot.root_ca_pem.clone(),
mtls_identity: snapshot.mtls_identity.clone(),
})
});
let accepted = get_global_outbound_tls_snapshot();
*GLOBAL_ROOT_CERT.write().await = accepted.root_ca_pem.clone();
*GLOBAL_MTLS_IDENTITY.write().await = accepted.mtls_identity.clone();
GLOBAL_OUTBOUND_TLS_GENERATION.store(accepted.generation, Ordering::Relaxed);
accepted.generation
}
/// Log level to use when reporting cached gRPC connection eviction.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum ConnectionEvictionLogLevel {
@@ -122,27 +85,17 @@ pub async fn get_global_addr() -> String {
/// # Arguments
/// * `cert` - A vector of bytes representing the PEM-encoded root CA certificate.
pub async fn set_global_root_cert(cert: Vec<u8>) {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
let mut next = (**current).clone();
next.root_ca_pem = Some(cert.clone());
Arc::new(next)
});
*GLOBAL_ROOT_CERT.write().await = Some(cert);
}
/// Clear the global root CA certificate for outbound gRPC clients.
pub async fn clear_global_root_cert() {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
let mut next = (**current).clone();
next.root_ca_pem = None;
Arc::new(next)
});
*GLOBAL_ROOT_CERT.write().await = None;
}
/// Get the global root CA certificate for outbound gRPC clients.
pub async fn get_global_root_cert() -> Option<Vec<u8>> {
get_global_outbound_tls_snapshot().root_ca_pem.clone()
GLOBAL_ROOT_CERT.read().await.clone()
}
/// Set the global mTLS identity (cert+key PEM) for outbound gRPC clients.
@@ -152,32 +105,22 @@ pub async fn get_global_root_cert() -> Option<Vec<u8>> {
/// # Arguments
/// * `identity` - An optional MtlsIdentityPem struct containing the cert and key PEM.
pub async fn set_global_mtls_identity(identity: Option<MtlsIdentityPem>) {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
let mut next = (**current).clone();
next.mtls_identity = identity.clone();
Arc::new(next)
});
*GLOBAL_MTLS_IDENTITY.write().await = identity;
}
/// Get the global mTLS identity for outbound gRPC clients.
pub async fn get_global_mtls_identity() -> Option<MtlsIdentityPem> {
get_global_outbound_tls_snapshot().mtls_identity.clone()
GLOBAL_MTLS_IDENTITY.read().await.clone()
}
/// Set the global outbound TLS generation.
pub fn set_global_outbound_tls_generation(generation: u64) {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
let mut next = (**current).clone();
next.generation = generation;
Arc::new(next)
});
GLOBAL_OUTBOUND_TLS_GENERATION.store(generation, Ordering::Relaxed);
}
/// Get the global outbound TLS generation.
pub fn get_global_outbound_tls_generation() -> u64 {
get_global_outbound_tls_snapshot().generation
GLOBAL_OUTBOUND_TLS_GENERATION.load(Ordering::Relaxed)
}
/// Evict a stale/dead connection from the global connection cache.
@@ -251,77 +194,8 @@ pub async fn clear_all_connections() {
///
/// When present, gRPC clients will present this identity to servers requesting/requiring mTLS.
/// When absent, clients proceed with standard server-authenticated TLS.
#[derive(Clone, PartialEq, Eq)]
#[derive(Clone, Debug)]
pub struct MtlsIdentityPem {
pub cert_pem: Vec<u8>,
pub key_pem: Vec<u8>,
}
impl std::fmt::Debug for MtlsIdentityPem {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("MtlsIdentityPem")
.field("cert_bytes", &self.cert_pem.len())
.field("key_pem", &"[REDACTED]")
.finish()
}
}
#[cfg(test)]
mod tls_snapshot_tests {
use super::*;
#[tokio::test]
async fn outbound_snapshot_publication_is_coherent_and_skips_unchanged_material() {
let start = get_global_outbound_tls_generation().saturating_add(1);
let snapshot = OutboundTlsSnapshot {
generation: start,
root_ca_pem: Some(vec![1]),
mtls_identity: Some(MtlsIdentityPem {
cert_pem: vec![1],
key_pem: vec![1],
}),
};
let first = publish_global_outbound_tls_snapshot(snapshot.clone()).await;
assert_eq!(
publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
generation: first.saturating_add(1),
..snapshot
})
.await,
first
);
let writer = tokio::spawn(async move {
for tag in 2_u8..=255 {
publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
generation: start.saturating_add(u64::from(tag)),
root_ca_pem: Some(vec![tag]),
mtls_identity: Some(MtlsIdentityPem {
cert_pem: vec![tag],
key_pem: vec![tag],
}),
})
.await;
tokio::task::yield_now().await;
}
});
while !writer.is_finished() {
let state = get_global_outbound_tls_snapshot();
let identity = state.mtls_identity.as_ref().expect("identity in complete snapshot");
assert_eq!(state.root_ca_pem.as_ref().expect("roots in complete snapshot"), &identity.cert_pem);
assert_eq!(identity.cert_pem, identity.key_pem);
tokio::task::yield_now().await;
}
writer.await.expect("writer completes");
}
#[test]
fn mtls_identity_debug_redacts_private_key() {
let identity = MtlsIdentityPem {
cert_pem: b"public certificate".to_vec(),
key_pem: b"PRIVATE-KEY-MARKER".to_vec(),
};
let debug = format!("{identity:?}");
assert!(!debug.contains("PRIVATE-KEY-MARKER"));
assert!(debug.contains("[REDACTED]"));
}
}
+13 -3
View File
@@ -18,8 +18,10 @@ use const_str::concat;
/// Default value: RustFS
/// Environment variable: RUSTFS_APP_NAME
pub const APP_NAME: &str = "RustFS";
/// Cargo package version for this workspace member.
pub const VERSION: &str = env!("CARGO_PKG_VERSION");
/// Application version
/// Default value: 1.0.0
/// Environment variable: RUSTFS_VERSION
pub const VERSION: &str = "1.0.0";
/// Default configuration logger level
/// Default value: error
@@ -37,6 +39,12 @@ pub const SAMPLE_RATIO: f64 = 1.0;
/// Default value: 30
pub const METER_INTERVAL: u64 = 30;
/// Default configuration service version
/// Default value: 1.0.0
/// Environment variable: RUSTFS_OBS_SERVICE_VERSION
/// Uses the same value as VERSION constant
pub const SERVICE_VERSION: &str = "1.0.0";
/// Default configuration environment
/// Default value: production
pub const ENVIRONMENT: &str = "production";
@@ -402,6 +410,7 @@ mod tests {
APP_NAME,
VERSION,
DEFAULT_LOG_LEVEL,
SERVICE_VERSION,
ENVIRONMENT,
RUSTFS_TLS_KEY,
RUSTFS_TLS_CERT,
@@ -437,7 +446,8 @@ mod tests {
fn test_configuration_consistency() {
// Test configuration consistency
assert_eq!(VERSION, env!("CARGO_PKG_VERSION"));
// Version consistency
assert_eq!(VERSION, SERVICE_VERSION, "Application version should match service version");
// Port conflict check
let ports = [DEFAULT_PORT, DEFAULT_CONSOLE_PORT];
-6
View File
@@ -207,9 +207,3 @@ pub const ENV_TLS_RELOAD_INTERVAL: &str = "RUSTFS_TLS_RELOAD_INTERVAL";
/// Default interval for TLS certificate reload check
pub const DEFAULT_TLS_RELOAD_INTERVAL: u64 = 30;
/// Environment variable to enable experimental HTTP/3 support.
pub const ENV_HTTP3_ENABLE: &str = "RUSTFS_HTTP3_ENABLE";
/// HTTP/3 is disabled unless explicitly enabled.
pub const DEFAULT_HTTP3_ENABLE: bool = false;
-14
View File
@@ -51,11 +51,6 @@ pub const ENV_OBS_PROFILING_EXPORT_ENABLED: &str = "RUSTFS_OBS_PROFILING_EXPORT_
/// PUT records multiple timers and histograms when attribution is active.
pub const ENV_OBS_PUT_STAGE_METRICS_ENABLED: &str = "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED";
/// Enables sampled in-process GetObject stage summaries without OTLP histograms.
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED";
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE";
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS";
pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED";
pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
@@ -164,15 +159,6 @@ mod tests {
assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
assert_eq!(ENV_OBS_PROFILING_EXPORT_ENABLED, "RUSTFS_OBS_PROFILING_EXPORT_ENABLED");
assert_eq!(ENV_OBS_PUT_STAGE_METRICS_ENABLED, "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED");
assert_eq!(ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED, "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED");
assert_eq!(
ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE,
"RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE"
);
assert_eq!(
ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS,
"RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS"
);
// Test log cleanup related env keys
assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");
@@ -33,10 +33,6 @@
//! tracked separately.
use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
use crate::fault_proxy::FaultMode;
use aws_sdk_s3::Client;
use std::time::Duration;
use tokio::time::{Instant, timeout};
type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
@@ -148,99 +144,7 @@ async fn cluster_volume_fault_proxy_pass_smoke() -> TestResult {
cluster.start().await?;
cluster.create_test_bucket(BUCKET).await?;
let payload = vec![0x6Du8; 256 * 1024];
put_get_roundtrip(&cluster, "volume-proxy/object", &payload).await?;
// Node 1 owns only two disks; a successful three-vote write must
// authenticate at least one disk mutation through node 0's proxy.
let client = cluster.create_s3_client(1)?;
let client = Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
.build(),
);
let forwarded_before = proxy.forwarded_bytes();
client
.put_object()
.bucket(BUCKET)
.key("volume-proxy/authenticated")
.body(bytes::Bytes::copy_from_slice(&payload).into())
.send()
.await?;
assert!(
proxy.forwarded_bytes() > forwarded_before,
"successful peer-disk publication must traverse the proxy"
);
let dropped_before = proxy.dropped_bytes();
proxy.set_mode(FaultMode::Blackhole);
assert_eq!(proxy.mode(), FaultMode::Blackhole);
let error = timeout(
Duration::from_secs(30),
client
.put_object()
.bucket(BUCKET)
.key("volume-proxy/rejected")
.body(bytes::Bytes::copy_from_slice(&payload).into())
.send(),
)
.await?
.expect_err("two local disks cannot commit a write while the peer is blackholed");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("ServiceUnavailable"));
assert!(
proxy.dropped_bytes() > dropped_before,
"the injected blackhole must discard actual peer traffic"
);
proxy.set_mode(FaultMode::Pass);
assert_eq!(proxy.mode(), FaultMode::Pass);
let http = reqwest::Client::builder()
.no_proxy()
.timeout(Duration::from_secs(3))
.build()?;
let deadline = Instant::now() + Duration::from_secs(30);
loop {
let response = http.get(format!("{}/health/ready", cluster.nodes[1].url)).send().await?;
let status = response.status();
let body: serde_json::Value = response.json().await?;
// Reachable peers can still have Returning disk handles that reject
// writes. Wait for this node's live writer inventory to be Online.
if status.as_u16() == 200
&& body["details"]["storage"]["source"] == "local_runtime"
&& body["details"]["storage"]["writeQuorum"] == true
&& body["details"]["storage"]["unavailableDrives"]
.as_array()
.is_some_and(Vec::is_empty)
&& body["details"]["lock"]["ready"] == true
{
break;
}
assert!(Instant::now() < deadline, "peer write quorum did not recover: HTTP {status}, {body}");
tokio::time::sleep(Duration::from_millis(200)).await;
}
client
.put_object()
.bucket(BUCKET)
.key("volume-proxy/recovered")
.body(bytes::Bytes::copy_from_slice(&payload).into())
.send()
.await?;
for key in ["volume-proxy/object", "volume-proxy/authenticated", "volume-proxy/recovered"] {
let got = client.get_object().bucket(BUCKET).key(key).send().await?;
let body = got.body.collect().await?.into_bytes();
assert_eq!(body.as_ref(), payload, "proxy recovery must preserve the full object body for {key}");
}
let rejected = client
.head_object()
.bucket(BUCKET)
.key("volume-proxy/rejected")
.send()
.await
.expect_err("a rejected proxied write must never become visible after recovery");
assert_eq!(rejected.raw_response().map(|response| response.status().as_u16()), Some(404));
Ok(())
put_get_roundtrip(&cluster, "volume-proxy/object", &payload).await
}
.await;
+2 -29
View File
@@ -35,7 +35,7 @@ use s3s::Body;
use serde_json;
use std::fs as stdfs;
use std::io::ErrorKind;
use std::net::{Ipv6Addr, SocketAddr};
use std::net::SocketAddr;
use std::path::{Path, PathBuf};
use std::process::{Child, Command, Stdio};
use std::sync::Once;
@@ -1442,11 +1442,7 @@ impl RustFSTestClusterEnvironment {
return Err(format!("a volume proxy is already configured for node {node_idx}").into());
}
let target = self.nodes[node_idx].address.parse::<SocketAddr>()?;
// Endpoint locality and RPC audience both use the advertised authority.
// Keep the node's port while the second loopback IP separates the proxy
// listener from the direct IPv4 server and S3 client address.
let listen = SocketAddr::from((Ipv6Addr::LOCALHOST, target.port()));
let proxy = crate::fault_proxy::FaultProxy::start_on(listen, target).await?;
let proxy = crate::fault_proxy::FaultProxy::start(target).await?;
self.volume_proxy_addresses[node_idx] = Some(proxy.local_addr());
Ok(proxy)
}
@@ -1542,15 +1538,6 @@ impl RustFSTestClusterEnvironment {
self.start_node_from_binary(node_idx, &binary_path).await
}
/// Start a process and wait only for TCP, allowing tests to bring up a
/// quorum one node at a time before the S3 service can become ready.
pub async fn start_node_process(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let binary_path = rustfs_binary_path();
let volumes_arg = self.build_volumes_arg();
self.spawn_node(node_idx, &binary_path, &volumes_arg)?;
self.wait_for_node_ready(&self.nodes[node_idx].address, node_idx).await
}
/// Start one stopped cluster node with a specific RustFS binary while
/// preserving the cluster's volume layout and that node's data directory.
pub async fn start_node_from_binary(
@@ -2480,20 +2467,6 @@ mod tests {
assert!(volumes.contains(&proxied), "volumes must use the proxy address: {volumes}");
assert!(!volumes.contains(&direct), "volumes must not retain the direct address: {volumes}");
assert_eq!(proxy.local_addr().port(), proxy.target_addr().port());
assert_ne!(proxy.local_addr().ip(), proxy.target_addr().ip());
assert_eq!(env.nodes[0].address, direct, "S3 clients must retain the direct address");
let mut endpoint =
crate::storage_api::Endpoint::try_from(volumes.as_str()).expect("proxied volume should be a valid endpoint");
endpoint
.update_is_local(proxy.target_addr().port())
.expect("proxied endpoint locality should resolve");
assert!(endpoint.is_local, "the proxied node must keep its local disk and first-disk authority");
assert_eq!(
endpoint.host_port(),
proxied,
"the local RPC audience must be the advertised proxy authority"
);
proxy.shutdown().await;
}
+4 -6
View File
@@ -860,12 +860,10 @@ pub(crate) async fn start_rebalance(cluster: &RustFSTestClusterEnvironment) -> T
if status.is_success() {
break response;
}
// The init stage answers 503 + Retry-After while the cluster capability
// proof is still missing; older binaries surfaced the same condition as
// a 500 carrying the proof marker.
let waiting_for_readiness = status == StatusCode::SERVICE_UNAVAILABLE
|| (status == StatusCode::INTERNAL_SERVER_ERROR && response.contains("requires a live fleet capability proof"));
if waiting_for_readiness && Instant::now() < deadline {
if status == StatusCode::INTERNAL_SERVER_ERROR
&& response.contains("requires a live fleet capability proof")
&& Instant::now() < deadline
{
sleep(Duration::from_secs(1)).await;
continue;
}
@@ -31,7 +31,6 @@ use super::harness::{
use crate::common::{
AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, init_logging,
};
use crate::upgrade_compatibility_test::wait_for_upgrade_write_readiness;
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use std::path::{Path, PathBuf};
@@ -401,19 +400,6 @@ async fn four_node_direct_upgrade_preserves_history_and_iam_credentials() -> Tes
Ok(())
}
async fn replace_node_and_wait_for_write_readiness(dist: &mut DistCluster, seed: &UpgradeSeed, node_idx: usize) -> TestResult {
dist.replace_node_with_current_binary(node_idx).await?;
// Node readiness proves reads can succeed; a rolling writer must also
// recover before replacing the next peer or checking upgrade contracts.
wait_for_upgrade_write_readiness(
&dist.clients()?,
&seed.history_bucket,
&format!("replaced-node-{node_idx}"),
CREDENTIAL_TIMEOUT,
)
.await
}
#[tokio::test]
async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> TestResult {
init_logging();
@@ -424,15 +410,15 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
let seed = seed_history_and_iam(&dist).await?;
replace_node_and_wait_for_write_readiness(&mut dist, &seed, 0).await?;
dist.replace_node_with_current_binary(0).await?;
assert_history_and_iam(&dist, &seed, "one-current-node").await?;
for node_idx in [1, 2] {
replace_node_and_wait_for_write_readiness(&mut dist, &seed, node_idx).await?;
dist.replace_node_with_current_binary(node_idx).await?;
}
assert_history_and_iam(&dist, &seed, "one-previous-node").await?;
replace_node_and_wait_for_write_readiness(&mut dist, &seed, 3).await?;
dist.replace_node_with_current_binary(3).await?;
assert_history_and_iam(&dist, &seed, "homogeneous-current").await?;
Ok(())
}
+14 -379
View File
@@ -50,8 +50,8 @@ use std::collections::{BTreeMap, HashMap, VecDeque};
use std::error::Error;
use std::io;
use std::net::SocketAddr;
use std::sync::{Arc, Mutex, MutexGuard, OnceLock};
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use std::sync::{Arc, Mutex, MutexGuard};
use std::time::{Duration, SystemTime};
use tokio::net::TcpListener;
use tokio::sync::{OwnedSemaphorePermit, Semaphore, watch};
use tokio::task::JoinHandle;
@@ -363,14 +363,6 @@ pub struct RequestRecord {
pub upload_id: Option<String>,
pub part_number: Option<i32>,
pub content_length: Option<u64>,
pub source_version_id: Option<String>,
pub source_mtime: Option<String>,
pub source_etag: Option<String>,
pub source_replication_request: bool,
pub journaled_at_unix_millis: Option<u64>,
/// Computed by the target, before response stall/disconnect faults. This
/// does not prove that any response bytes reached the client.
pub prepared_response: Option<PreparedResponse>,
pub consumed_bytes: Option<usize>,
pub replication_timestamps: ReplicationTimestampHeaders,
pub proxy_headers: ProxyHeaderSnapshot,
@@ -389,20 +381,6 @@ pub struct RequestRecord {
pub fault: Option<FaultAction>,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PreparedResponse {
pub status: u16,
pub version_id: Option<String>,
pub etag: Option<String>,
pub last_modified: Option<String>,
pub content_length: Option<u64>,
pub prepared_at_unix_millis: Option<u64>,
pub elapsed: Duration,
}
#[derive(Clone, Default)]
struct RequestSequence(Arc<OnceLock<u64>>);
/// Integrity and framing headers of an upload. A remote target's acceptance
/// rules key off exactly these (rustfs#6853: `aws-chunked` framing stored
/// verbatim; rustfs#7082: Object Lock PUTs need `Content-MD5` or
@@ -452,10 +430,6 @@ impl TransportSnapshot {
#[derive(Debug, Clone, Default, PartialEq, Eq)]
struct JournaledHeaders {
content_length: Option<u64>,
source_version_id: Option<String>,
source_mtime: Option<String>,
source_etag: Option<String>,
source_replication_request: bool,
replication_timestamps: ReplicationTimestampHeaders,
proxy_headers: ProxyHeaderSnapshot,
transport: TransportSnapshot,
@@ -470,10 +444,6 @@ impl JournaledHeaders {
.get(CONTENT_LENGTH)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse().ok()),
source_version_id: header_value(headers, &SOURCE_VERSION_ID_HEADERS).map(bounded_journal_value),
source_mtime: header_value(headers, &SOURCE_MTIME_HEADERS).map(bounded_journal_value),
source_etag: header_value(headers, &SOURCE_ETAG_HEADERS).map(bounded_journal_value),
source_replication_request: header_value(headers, &SOURCE_REPLICATION_REQUEST_HEADERS).as_deref() == Some("true"),
replication_timestamps: ReplicationTimestampHeaders::from_headers(headers),
proxy_headers: ProxyHeaderSnapshot::from_headers(headers),
transport: TransportSnapshot::from_headers(headers),
@@ -541,10 +511,6 @@ struct StoreState {
/// version id the target never had answers 404 `NoSuchVersion` instead of
/// the idempotent 204 RustFS/MinIO give.
reject_unknown_version_deletes: bool,
/// Models MinIO and its derivatives (Storj): `X-Minio-Internal-` is a reserved
/// metadata prefix, so a PutObject or UploadPart carrying any such request
/// header is refused with `InvalidArgument` (rustfs#8362).
reject_reserved_internal_headers: bool,
limits: StoreLimits,
buckets: HashMap<String, BucketState>,
uploads: HashMap<String, MultipartState>,
@@ -772,7 +738,6 @@ impl FakeS3Target {
let connection_limit = Arc::new(Semaphore::new(MAX_CONNECTIONS));
let (active_connections_tx, active_connections) = watch::channel(0usize);
let task_connection_limit = Arc::clone(&connection_limit);
let task_control = Arc::clone(&control);
let task = tokio::spawn(async move {
let mut connections = tokio::task::JoinSet::new();
@@ -786,7 +751,6 @@ impl FakeS3Target {
};
active_connections_tx.send_modify(|count| *count += 1);
let service = service.clone();
let control = Arc::clone(&task_control);
let active_connections = active_connections_tx.clone();
let mut connection_gate = connection_gate.clone();
connections.spawn(async move {
@@ -800,7 +764,7 @@ impl FakeS3Target {
{
return;
}
let handler = service_fn(move |request| handle_request(request, service.clone(), Arc::clone(&control)));
let handler = service_fn(move |request| handle_request(request, service.clone()));
let mut connection = http1::Builder::new();
connection
.keep_alive(false)
@@ -1011,43 +975,6 @@ impl FakeS3Target {
lock(&self.backend.store).reject_unknown_version_deletes = enabled;
}
/// Refuse every PutObject / UploadPart that carries an `x-minio-internal-*`
/// request header, the way MinIO and Storj do.
pub fn reject_reserved_internal_headers(&self, enabled: bool) {
lock(&self.backend.store).reject_reserved_internal_headers = enabled;
}
/// Keys of every stored, non-delete object in the bucket, sorted.
pub fn stored_keys(&self, bucket: &str) -> Vec<String> {
let state = lock(&self.backend.store);
let mut keys: Vec<String> = state
.buckets
.get(bucket)
.map(|bucket| {
bucket
.objects
.iter()
.filter(|(_, versions)| versions.last().is_some_and(|version| !version.delete_marker))
.map(|(key, _)| key.clone())
.collect()
})
.unwrap_or_default();
keys.sort();
keys
}
/// Body and user metadata (names as stored, without the `x-amz-meta-` prefix)
/// of the latest non-delete version of an object.
pub fn stored_object(&self, bucket: &str, key: &str) -> Option<(Bytes, HashMap<String, String>)> {
lock(&self.backend.store)
.buckets
.get(bucket)
.and_then(|bucket| bucket.objects.get(key))
.and_then(|versions| versions.last())
.filter(|version| !version.delete_marker)
.map(|version| (version.body.clone(), version.metadata.clone().unwrap_or_default()))
}
pub fn require_checksum_for_object_lock(&self, enabled: bool) {
lock(&self.backend.store).require_checksum_for_object_lock = enabled;
}
@@ -1234,10 +1161,7 @@ impl S3Access for FaultAccess {
let operation = operation_from_s3_name(context.s3_op().name());
let journaled = JournaledHeaders::from_headers(context.headers());
let aws_chunked = journaled.transport.aws_chunked;
let (sequence, fault) = record_request(&self.control, operation, context.method().clone(), parsed, journaled);
if let Some(slot) = context.extensions_mut().get::<RequestSequence>() {
let _ = slot.0.set(sequence);
}
let fault = record_request(&self.control, operation, context.method().clone(), parsed, journaled);
if let Some(status) = fault.as_ref().and_then(|fault| scripted_status(&fault.action)) {
return Err(scripted_status_error(status));
}
@@ -1250,15 +1174,6 @@ impl S3Access for FaultAccess {
"this target does not decode aws-chunked request bodies; send a plain signed payload with an exact Content-Length"
));
}
if matches!(operation, Operation::PutObject | Operation::UploadPart)
&& lock(&self.store).reject_reserved_internal_headers
&& context
.headers()
.keys()
.any(|name| name.as_str().starts_with("x-minio-internal-"))
{
return Err(s3s::s3_error!(InvalidArgument, "Your metadata headers are not supported."));
}
let prebody_permit = if operation == Operation::CompleteMultipartUpload {
Some(
timeout(
@@ -1313,7 +1228,7 @@ fn record_request(
method: Method,
parsed: ParsedRequest,
headers: JournaledHeaders,
) -> (u64, Option<RequestFault>) {
) -> Option<RequestFault> {
let mut state = lock(control);
let action = parsed
.key
@@ -1336,12 +1251,6 @@ fn record_request(
upload_id: parsed.upload_id.map(bounded_journal_value),
part_number: parsed.part_number,
content_length: headers.content_length,
source_version_id: headers.source_version_id,
source_mtime: headers.source_mtime,
source_etag: headers.source_etag,
source_replication_request: headers.source_replication_request,
journaled_at_unix_millis: unix_millis(),
prepared_response: None,
consumed_bytes: None,
replication_timestamps: headers.replication_timestamps,
proxy_headers: headers.proxy_headers,
@@ -1352,14 +1261,7 @@ fn record_request(
continuation_token: parsed.continuation_token.map(bounded_journal_value),
fault: action.clone(),
});
(sequence, action.map(|action| RequestFault { sequence, action }))
}
fn unix_millis() -> Option<u64> {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.ok()
.and_then(|elapsed| u64::try_from(elapsed.as_millis()).ok())
action.map(|action| RequestFault { sequence, action })
}
fn bounded_journal_value(mut value: String) -> String {
@@ -1367,31 +1269,14 @@ fn bounded_journal_value(mut value: String) -> String {
while !value.is_char_boundary(end) {
end -= 1;
}
if value.capacity() > MAX_RETAINED_IDENTIFIER_BYTES {
value[..end].to_owned()
} else {
value.truncate(end);
value
}
value.truncate(end);
value
}
async fn handle_request(
mut request: Request<Incoming>,
service: S3Service,
control: Arc<Mutex<ControlState>>,
) -> Result<Response<Body>, BoxError> {
let started = Instant::now();
let sequence = RequestSequence::default();
request.extensions_mut().insert(sequence.clone());
let mut response = timeout(MAX_REQUEST_DURATION, async {
let result = call_s3(service, request.map(Body::from)).await;
if let (Ok(response), Some(sequence)) = (&result, sequence.0.get()) {
update_prepared_response(&control, *sequence, response, started.elapsed());
}
result
})
.await
.map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "fake target request exceeded 65 seconds"))??;
async fn handle_request(request: Request<Incoming>, service: S3Service) -> Result<Response<Body>, BoxError> {
let mut response = timeout(MAX_REQUEST_DURATION, call_s3(service, request.map(Body::from)))
.await
.map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "fake target request exceeded 65 seconds"))??;
if response.headers_mut().remove(DISCONNECT_HEADER).is_some() {
return Err(io::Error::new(io::ErrorKind::ConnectionAborted, "fake target scripted disconnect").into());
}
@@ -1646,25 +1531,6 @@ fn update_consumed(control: &Mutex<ControlState>, sequence: u64, consumed: usize
}
}
fn update_prepared_response(control: &Mutex<ControlState>, sequence: u64, response: &Response<Body>, elapsed: Duration) {
let prepared = PreparedResponse {
status: response.status().as_u16(),
version_id: header_value(response.headers(), &["x-amz-version-id"]).map(bounded_journal_value),
etag: header_value(response.headers(), &[ETAG.as_str()]).map(bounded_journal_value),
last_modified: header_value(response.headers(), &[LAST_MODIFIED.as_str()]).map(bounded_journal_value),
content_length: response
.headers()
.get(CONTENT_LENGTH)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse().ok()),
prepared_at_unix_millis: unix_millis(),
elapsed,
};
if let Some(record) = lock(control).requests.iter_mut().find(|record| record.sequence == sequence) {
record.prepared_response = Some(prepared);
}
}
async fn apply_non_body_fault(fault: Option<&RequestFault>, control: &Mutex<ControlState>) -> S3Result<()> {
match fault.map(|fault| &fault.action) {
Some(action @ (FaultAction::Status(_) | FaultAction::ResponseStatus(_))) => {
@@ -3364,38 +3230,10 @@ mod tests {
let retry = client.clone();
requests.spawn(async move { get_bytes(&retry, bucket, "held", None).await });
timeout(Duration::from_secs(2), gate.entered.wait_for(|count| *count == 2)).await??;
let held_retry = target
.requests()
.into_iter()
.rev()
.find(|record| record.operation == Operation::GetObject && record.key.as_deref() == Some("held"))
.expect("the held retry must be journaled");
assert_eq!(held_retry.prepared_response, None);
assert_eq!(
timeout(Duration::from_secs(2), get_bytes(&client, bucket, "unrelated", None)).await??,
Bytes::from_static(b"payload")
);
let unrelated = target
.requests()
.into_iter()
.find(|record| record.operation == Operation::GetObject && record.key.as_deref() == Some("unrelated"))
.expect("the unrelated GET must be journaled");
let unrelated_response = unrelated.prepared_response.expect("the unrelated GET prepared a response");
assert_eq!(unrelated_response.status, 200);
assert_eq!(
unrelated_response.version_id.as_deref(),
Some(target.stored_versions(bucket, "unrelated")[0].0.as_str())
);
assert!(held_retry.sequence < unrelated.sequence);
assert_eq!(
target
.requests()
.into_iter()
.find(|record| record.sequence == held_retry.sequence)
.expect("the held retry remains journaled")
.prepared_response,
None
);
assert!(requests.try_join_next().is_none(), "the retry must remain behind the gate");
drop(gate);
assert_eq!(
@@ -3404,18 +3242,6 @@ mod tests {
.expect("retried GET task")??,
Bytes::from_static(b"payload")
);
let held_response = target
.requests()
.into_iter()
.find(|record| record.sequence == held_retry.sequence)
.and_then(|record| record.prepared_response)
.expect("the released retry prepared its own response");
assert_eq!(held_response.status, 200);
assert_eq!(
held_response.version_id.as_deref(),
Some(target.stored_versions(bucket, "held")[0].0.as_str())
);
assert_ne!(held_response.version_id, unrelated_response.version_id);
assert_eq!(get_bytes(&client, bucket, "held", None).await?, Bytes::from_static(b"payload"));
assert_eq!(target.count_requests(Operation::GetObject, "held"), 3);
Ok(())
@@ -3760,23 +3586,6 @@ mod tests {
.await?;
assert_eq!(first_head.content_length(), Some(5));
assert_eq!(first_head.version_id(), first.version_id.as_deref());
let journaled_head = target
.requests()
.into_iter()
.rev()
.find(|record| record.operation == Operation::HeadObject)
.expect("the version HEAD must be journaled");
assert_eq!(journaled_head.version_id, first.version_id);
assert!(journaled_head.journaled_at_unix_millis.is_some());
let prepared_head = journaled_head
.prepared_response
.expect("the version HEAD prepared a response");
assert_eq!(prepared_head.status, 200);
assert_eq!(prepared_head.version_id.as_deref(), first_head.version_id());
assert_eq!(prepared_head.etag.as_deref(), first_head.e_tag());
assert_eq!(prepared_head.content_length, Some(5));
assert!(prepared_head.last_modified.is_some());
assert!(prepared_head.prepared_at_unix_millis.is_some());
assert_eq!(
get_bytes(&client, "target-bucket", "nested/key", None).await?,
Bytes::from_static(b"second")
@@ -3802,20 +3611,6 @@ mod tests {
.await
.expect_err("deleted version must not remain readable");
assert_sdk_error!(deleted, 404, "NoSuchVersion");
let deleted_record = target
.requests()
.into_iter()
.rev()
.find(|record| record.operation == Operation::GetObject)
.expect("the deleted version GET must be journaled");
assert_eq!(deleted_record.version_id, first.version_id);
assert_eq!(
deleted_record
.prepared_response
.expect("the error response was prepared")
.status,
404
);
let marker = client
.delete_object()
.bucket("target-bucket")
@@ -4247,21 +4042,7 @@ mod tests {
.await
.expect_err("wrong part ETag must fail without consuming the upload");
assert_sdk_error!(invalid_complete, 400, "InvalidPart");
let rejected_complete = target
.requests()
.into_iter()
.rev()
.find(|record| record.operation == Operation::CompleteMultipartUpload)
.expect("the rejected completion must be journaled");
assert_eq!(rejected_complete.upload_id.as_deref(), Some(upload_id.as_str()));
assert_eq!(
rejected_complete
.prepared_response
.expect("the rejection was prepared")
.status,
400
);
let successful_complete = client
client
.complete_multipart_upload()
.bucket("target-bucket")
.key("multipart")
@@ -4269,18 +4050,6 @@ mod tests {
.multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(parts)).build())
.send()
.await?;
let completed_record = target
.requests()
.into_iter()
.rev()
.find(|record| record.operation == Operation::CompleteMultipartUpload)
.expect("the successful completion must be journaled");
assert_eq!(completed_record.upload_id.as_deref(), Some(upload_id.as_str()));
let prepared_complete = completed_record
.prepared_response
.expect("the completion response was prepared");
assert_eq!(prepared_complete.status, 200);
assert_eq!(prepared_complete.version_id.as_deref(), successful_complete.version_id());
let completed = get_bytes(&client, "target-bucket", "multipart", None).await?;
assert_eq!(completed.len(), MIN_MULTIPART_PART_BYTES + 5);
assert_eq!(&completed[..MIN_MULTIPART_PART_BYTES], first_body.as_ref());
@@ -4661,32 +4430,6 @@ mod tests {
.send()
.await?;
assert_eq!(completed.version_id(), Some(completed_version.as_str()));
let receipt_records = target.requests();
let receipt_create = receipt_records
.iter()
.find(|record| {
record.operation == Operation::CreateMultipartUpload && record.key.as_deref() == Some("multipart-receipt-time")
})
.expect("the source version must be retained on MPU creation");
assert_eq!(receipt_create.source_version_id.as_deref(), Some(completed_version.as_str()));
let receipt_complete = receipt_records
.iter()
.find(|record| {
record.operation == Operation::CompleteMultipartUpload
&& record.upload_id.as_deref() == Some(receipt_upload.as_str())
})
.expect("the source mtime must be retained on completion");
assert_eq!(receipt_complete.source_mtime.as_deref(), Some("1970-01-01T00:00:00Z"));
assert!(receipt_complete.source_replication_request);
assert_eq!(
receipt_complete
.prepared_response
.as_ref()
.expect("the receipt-time response was prepared")
.version_id
.as_deref(),
completed.version_id()
);
assert_eq!(
get_bytes(&client, "target-bucket", "multipart-receipt-time", None).await?,
Bytes::from_static(b"completed"),
@@ -4905,23 +4648,6 @@ mod tests {
.expect_err("disconnected request must not store an object");
assert_sdk_error!(disconnected, 404, "NotFound");
target.inject_for_key(Operation::PutObject, "lost-response", FaultAction::DisconnectAfterResponse, 1);
client
.put_object()
.bucket("target-bucket")
.key("lost-response")
.body(ByteStream::from_static(b"stored"))
.send()
.await
.expect_err("a prepared response does not prove client delivery");
let lost_response_head = client
.head_object()
.bucket("target-bucket")
.key("lost-response")
.send()
.await?;
assert_eq!(lost_response_head.content_length(), Some(6));
assert_eq!(
parse_request(&Method::PUT, &"/target-bucket/key?uploadId=upload&partNumber=bad".parse::<Uri>()?).operation,
Operation::Unknown
@@ -4935,28 +4661,6 @@ mod tests {
.count(),
4
);
assert!(
requests
.iter()
.filter(|request| request.fault == Some(FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE)))
.all(|request| {
request
.prepared_response
.as_ref()
.is_some_and(|response| response.status == 503)
})
);
let lost_response = requests
.iter()
.find(|request| request.operation == Operation::PutObject && request.key.as_deref() == Some("lost-response"))
.expect("the disconnected response must be journaled");
assert_eq!(lost_response.fault, Some(FaultAction::DisconnectAfterResponse));
let prepared_lost_response = lost_response
.prepared_response
.as_ref()
.expect("the target prepared the successful response");
assert_eq!(prepared_lost_response.status, 200);
assert_eq!(prepared_lost_response.version_id.as_deref(), lost_response_head.version_id());
assert!(
requests
.iter()
@@ -5147,44 +4851,8 @@ mod tests {
assert_eq!(records.len(), MAX_REQUEST_RECORDS);
assert_eq!(records.front().map(|record| record.sequence), Some(2));
assert_eq!(records.back().map(|record| record.sequence), Some((MAX_REQUEST_RECORDS + 1) as u64));
let mut oversized_allocation = String::with_capacity(MAX_RETAINED_IDENTIFIER_BYTES * 16);
oversized_allocation.push_str("short");
let short_value = bounded_journal_value(oversized_allocation);
assert_eq!(short_value, "short");
assert!(short_value.capacity() <= MAX_RETAINED_IDENTIFIER_BYTES);
let mut response = Response::new(Body::empty());
for name in ["x-amz-version-id", "etag", "last-modified"] {
response.headers_mut().insert(
name,
HeaderValue::from_str(&"r".repeat(MAX_RETAINED_IDENTIFIER_BYTES + 1)).expect("header"),
);
}
response.headers_mut().insert(CONTENT_LENGTH, HeaderValue::from_static("17"));
response
.headers_mut()
.insert("authorization", HeaderValue::from_static("unrecorded-response-authorization"));
response.headers_mut().insert(
"x-amz-server-side-encryption-customer-key",
HeaderValue::from_static("unrecorded-response-customer-key"),
);
update_prepared_response(&control, 1, &response, Duration::ZERO);
update_prepared_response(&control, u64::MAX, &response, Duration::ZERO);
assert_eq!(lock(&control).requests, records, "misses and evicted records must not be resurrected");
let bounded_control = Mutex::new(ControlState::default());
let utf8_boundary = format!("{}é", "a".repeat(MAX_RETAINED_IDENTIFIER_BYTES - 1));
let mut source_headers = HeaderMap::new();
for name in ["x-rustfs-source-version-id", "x-minio-source-mtime", "x-rustfs-source-etag"] {
source_headers.insert(
name,
HeaderValue::from_str(&"s".repeat(MAX_RETAINED_IDENTIFIER_BYTES + 1)).expect("header"),
);
}
source_headers.insert("x-minio-source-replication-request", HeaderValue::from_static("true"));
source_headers.insert("authorization", HeaderValue::from_static("unrecorded-request-authorization"));
source_headers.insert(
"x-amz-server-side-encryption-customer-key",
HeaderValue::from_static("unrecorded-request-customer-key"),
);
record_request(
&bounded_control,
Operation::GetObject,
@@ -5197,9 +4865,8 @@ mod tests {
upload_id: Some("u".repeat(MAX_RETAINED_IDENTIFIER_BYTES + 1)),
..Default::default()
},
JournaledHeaders::from_headers(&source_headers),
JournaledHeaders::default(),
);
update_prepared_response(&bounded_control, 1, &response, Duration::from_millis(7));
{
let bounded_records = lock(&bounded_control);
let bounded = &bounded_records.requests[0];
@@ -5207,39 +4874,7 @@ mod tests {
assert_eq!(bounded.key.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES - 1));
assert_eq!(bounded.version_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
assert_eq!(bounded.upload_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
assert_eq!(bounded.source_version_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
assert_eq!(bounded.source_mtime.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
assert_eq!(bounded.source_etag.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
for value in [&bounded.source_version_id, &bounded.source_mtime, &bounded.source_etag]
.into_iter()
.flatten()
{
assert!(value.capacity() <= MAX_RETAINED_IDENTIFIER_BYTES);
}
assert!(bounded.source_replication_request);
let prepared = bounded
.prepared_response
.as_ref()
.expect("the retained record receives its response");
assert_eq!(prepared.version_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
assert_eq!(prepared.etag.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
assert_eq!(prepared.last_modified.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
for value in [&prepared.version_id, &prepared.etag, &prepared.last_modified]
.into_iter()
.flatten()
{
assert!(value.capacity() <= MAX_RETAINED_IDENTIFIER_BYTES);
}
assert_eq!(prepared.content_length, Some(17));
assert_eq!(prepared.elapsed, Duration::from_millis(7));
assert!(
!format!("{bounded:?}").contains("unrecorded-"),
"authorization and customer keys must not enter the journal"
);
}
lock(&bounded_control).requests.clear();
update_prepared_response(&bounded_control, 1, &response, Duration::ZERO);
assert!(lock(&bounded_control).requests.is_empty(), "a drained record must not be resurrected");
let target = FakeS3Target::start().await.expect("target");
target.create_bucket("b".repeat(63));
let empty_bucket = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
+6 -49
View File
@@ -61,8 +61,6 @@
use std::io;
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
@@ -110,33 +108,19 @@ pub struct FaultProxy {
mode_tx: watch::Sender<FaultMode>,
shutdown_tx: watch::Sender<bool>,
accept_task: JoinHandle<()>,
traffic: Arc<ProxyTraffic>,
}
#[derive(Default)]
struct ProxyTraffic {
forwarded_bytes: AtomicUsize,
dropped_bytes: AtomicUsize,
}
impl FaultProxy {
/// Bind a listener on `127.0.0.1:0` and start forwarding accepted
/// connections to `target`. Starts in [`FaultMode::Pass`].
pub async fn start(target: SocketAddr) -> io::Result<Self> {
Self::start_on(SocketAddr::from((Ipv4Addr::LOCALHOST, 0)), target).await
}
/// Bind a specific address so advertised volume endpoints can retain the
/// server's port and authenticated RPC identity on another loopback IP.
pub async fn start_on(listen: SocketAddr, target: SocketAddr) -> io::Result<Self> {
let listener = TcpListener::bind(listen).await?;
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let listen_addr = listener.local_addr()?;
let (mode_tx, mode_rx) = watch::channel(FaultMode::Pass);
let (shutdown_tx, shutdown_rx) = watch::channel(false);
let traffic = Arc::new(ProxyTraffic::default());
let accept_task = tokio::spawn(accept_loop(listener, target, mode_rx, shutdown_rx, traffic.clone()));
let accept_task = tokio::spawn(accept_loop(listener, target, mode_rx, shutdown_rx));
Ok(Self {
listen_addr,
@@ -144,20 +128,9 @@ impl FaultProxy {
mode_tx,
shutdown_tx,
accept_task,
traffic,
})
}
/// Bytes actually forwarded in either direction, excluding dropped traffic.
pub fn forwarded_bytes(&self) -> usize {
self.traffic.forwarded_bytes.load(Ordering::Relaxed)
}
/// Bytes actually discarded by a blackhole or one-way partition.
pub fn dropped_bytes(&self) -> usize {
self.traffic.dropped_bytes.load(Ordering::Relaxed)
}
/// Local address the proxy is listening on. Point clients here instead of
/// the real target to route their traffic through the proxy.
pub fn local_addr(&self) -> SocketAddr {
@@ -198,7 +171,6 @@ async fn accept_loop(
target: SocketAddr,
mode_rx: watch::Receiver<FaultMode>,
mut shutdown_rx: watch::Receiver<bool>,
traffic: Arc<ProxyTraffic>,
) {
loop {
tokio::select! {
@@ -216,7 +188,6 @@ async fn accept_loop(
target,
mode_rx.clone(),
shutdown_rx.clone(),
traffic.clone(),
));
}
// A transient accept error should not tear the proxy down.
@@ -234,7 +205,6 @@ async fn handle_connection(
target: SocketAddr,
mode_rx: watch::Receiver<FaultMode>,
shutdown_rx: watch::Receiver<bool>,
traffic: Arc<ProxyTraffic>,
) {
let server = match TcpStream::connect(target).await {
Ok(s) => s,
@@ -251,9 +221,8 @@ async fn handle_connection(
server_wr,
mode_rx.clone(),
shutdown_rx.clone(),
traffic.clone(),
));
let down = tokio::spawn(pump(Direction::ServerToClient, server_rd, client_wr, mode_rx, shutdown_rx, traffic));
let down = tokio::spawn(pump(Direction::ServerToClient, server_rd, client_wr, mode_rx, shutdown_rx));
let _ = up.await;
let _ = down.await;
@@ -268,7 +237,6 @@ async fn pump<R, W>(
mut to: W,
mut mode_rx: watch::Receiver<FaultMode>,
mut shutdown_rx: watch::Receiver<bool>,
traffic: Arc<ProxyTraffic>,
) where
R: AsyncReadExt + Unpin,
W: AsyncWriteExt + Unpin,
@@ -289,22 +257,15 @@ async fn pump<R, W>(
let mode = *mode_rx.borrow_and_update();
match mode {
// Whole connection is a black hole: drain and drop.
FaultMode::Blackhole => {
traffic.dropped_bytes.fetch_add(n, Ordering::Relaxed);
continue;
}
FaultMode::Blackhole => continue,
// This direction is partitioned off: drop its bytes; the peer keeps
// reading nothing while the other direction may still flow.
FaultMode::Partition(blocked) if blocked == dir => {
traffic.dropped_bytes.fetch_add(n, Ordering::Relaxed);
continue;
}
FaultMode::Partition(blocked) if blocked == dir => continue,
FaultMode::Latency(delay) => {
tokio::time::sleep(delay).await;
if to.write_all(&buf[..n]).await.is_err() {
break;
}
traffic.forwarded_bytes.fetch_add(n, Ordering::Relaxed);
if to.flush().await.is_err() {
break;
}
@@ -313,7 +274,6 @@ async fn pump<R, W>(
if to.write_all(&buf[..n]).await.is_err() {
break;
}
traffic.forwarded_bytes.fetch_add(n, Ordering::Relaxed);
if to.flush().await.is_err() {
break;
}
@@ -325,6 +285,7 @@ async fn pump<R, W>(
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
use std::time::Instant;
use tokio::sync::Mutex;
@@ -414,8 +375,6 @@ mod tests {
assert_eq!(got, payload, "pass mode must forward bytes unchanged");
assert_eq!(echo.received().await, payload, "server must have received the bytes");
assert_eq!(proxy.forwarded_bytes(), payload.len() * 2);
assert_eq!(proxy.dropped_bytes(), 0);
proxy.shutdown().await;
}
@@ -455,8 +414,6 @@ mod tests {
assert!(got.is_empty(), "blackhole mode must not return any bytes, got {got:?}");
assert!(echo.received().await.is_empty(), "blackhole must not forward to the server");
assert_eq!(proxy.forwarded_bytes(), 0);
assert_eq!(proxy.dropped_bytes(), b"into-the-void".len());
// Proxy is still alive and steerable after a blackholed connection.
proxy.set_mode(FaultMode::Pass);
@@ -1191,59 +1191,11 @@ mod tests {
.into())
}
fn scanner_cycle_completion_budget(
cycle_end: u64,
interval: Option<u64>,
now: u64,
active: bool,
) -> Result<Duration, &'static str> {
if cycle_end > now {
return Err("scanner cycle timestamp is in the future");
}
if active {
return Ok(Duration::from_secs(60));
}
let interval = interval
.filter(|interval| *interval != 0)
.ok_or("scanner schedule has no positive interval")?;
let next_cycle = cycle_end
.checked_add(interval)
.ok_or("scanner schedule timestamp overflows")?;
let budget = next_cycle
.saturating_sub(now)
.checked_add(60)
.ok_or("scanner completion budget overflows")?;
Ok(Duration::from_secs(budget))
}
#[test]
fn scanner_cycle_completion_budget_accounts_for_captured_retry_schedule() {
assert_eq!(
scanner_cycle_completion_budget(1_791_815_390, Some(256), 1_791_815_447, false),
Ok(Duration::from_secs(259)),
);
assert_eq!(scanner_cycle_completion_budget(100, Some(1), 110, false), Ok(Duration::from_secs(60)));
}
#[test]
fn scanner_cycle_completion_budget_does_not_grant_planned_sleep_to_active_cycle() {
assert_eq!(scanner_cycle_completion_budget(100, Some(256), 110, true), Ok(Duration::from_secs(60)));
assert_eq!(scanner_cycle_completion_budget(0, None, 110, true), Ok(Duration::from_secs(60)));
}
#[test]
fn scanner_cycle_completion_budget_rejects_invalid_schedule() {
for (end, interval, now) in [(101, 1, 100), (100, 0, 100), (u64::MAX, 1, u64::MAX), (1, u64::MAX - 1, 1)] {
assert!(scanner_cycle_completion_budget(end, Some(interval), now, false).is_err());
}
}
async fn wait_for_scanner_cycle_after(
cluster: &RustFSTestClusterEnvironment,
previous_cycle_end: u64,
) -> Result<u64, Box<dyn Error + Send + Sync>> {
let mut deadline = Instant::now() + Duration::from_secs(60);
let mut schedule_accounted = false;
let deadline = Instant::now() + Duration::from_secs(60);
loop {
let mut latest_cycle_end = 0;
let mut versions_observed = false;
@@ -1285,36 +1237,14 @@ mod tests {
let versions_scanned = metrics["versions_scanned"]
.as_u64()
.ok_or("scanner status is missing its version-coverage counter")?;
let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
if cycle_end > now {
return Err("scanner cycle timestamp is in the future".into());
}
let schedule = &status["cycle_schedule"];
if !schedule_accounted && schedule["execution_role"].as_str() == Some("leader") {
let active = metrics["current_cycle_active"]
.as_bool()
.ok_or("scanner status is missing its active-cycle flag")?;
if active || (cycle_end != 0 && schedule["effective_interval_available"].as_bool() == Some(true)) {
let interval = schedule["effective_interval_seconds"].as_u64();
let budget = scanner_cycle_completion_budget(cycle_end, interval, now, active)?;
deadline = Instant::now().checked_add(budget).ok_or("scanner deadline overflows")?;
schedule_accounted = true;
}
}
// Superseded walks complete all sets while withholding quota authority.
// Deferred and partial attempts do not certify completed scanning.
if matches!(metrics["last_cycle_result"].as_str(), Some("success" | "superseded")) {
latest_cycle_end = latest_cycle_end.max(cycle_end);
}
latest_cycle_end = latest_cycle_end.max(cycle_end);
versions_observed |= versions_scanned > 0;
observations.push(format!(
"node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}, partial_reason={}, schedule={}",
"node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}",
metrics["current_cycle"],
metrics["current_cycle_active"],
metrics["leader_lock_state"],
metrics["last_cycle_result"],
metrics["last_cycle_partial_reason"],
schedule,
));
}
// The coordinator records cycle completion, but remote workers
@@ -1625,8 +1555,7 @@ mod tests {
}
if background_enabled {
let baseline_cycle_floor = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
wait_for_scanner_cycle_after(&cluster, baseline_cycle_floor).await?;
wait_for_scanner_cycle_after(&cluster, 0).await?;
}
cluster.stop_node(1)?;
@@ -1969,16 +1898,11 @@ mod tests {
let status_body = signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key).await?;
let status: serde_json::Value = serde_json::from_str(&status_body)
.map_err(|err| format!("pre-interrupt background heal status is not JSON ({err}): {status_body}"))?;
let admin_active = background_rejoin_heal_evidence
|| (status["state"].as_str() == Some("active")
&& status["healOperations"]["activeBySource"]["admin"].as_u64() == Some(1));
if admin_active && (background_enabled || only_admin_heal_is_active(&status)) {
if background_enabled || only_admin_heal_is_active(&status) {
break status;
}
if Instant::now() >= partial_deadline {
return Err(
format!("heal tasks did not reach the required interruption state before the deadline: {status}").into(),
);
return Err(format!("non-admin heal did not drain before the interruption deadline: {status}").into());
}
sleep(Duration::from_millis(50)).await;
};
@@ -2638,157 +2562,4 @@ mod tests {
};
assert!(!outage_candidate_targets_replacement(&insufficient_peer_shards, 12, replacement_set_slot));
}
async fn two_node_heal_status(
cluster: &RustFSTestClusterEnvironment,
bucket: Option<&str>,
recreate: bool,
scan_mode: u8,
) -> Result<serde_json::Value, Box<dyn Error + Send + Sync>> {
let url = format!("{}/rustfs/admin/v3/heal/{}", cluster.nodes[0].url, bucket.unwrap_or_default());
let options = serde_json::json!({
"recursive": true, "dryRun": false, "remove": false,
"recreate": recreate, "scanMode": scan_mode, "updateParity": false, "nolock": false,
});
let started =
start_root_heal_when_control_ready(&url, &options.to_string(), &cluster.access_key, &cluster.secret_key).await?;
let started: serde_json::Value = serde_json::from_str(&started)?;
let token = started["clientToken"]
.as_str()
.ok_or_else(|| format!("missing heal token: {started}"))?;
let status_url = format!("{url}?clientToken={token}");
let deadline = Instant::now() + Duration::from_secs(240);
loop {
let body = timeout(
Duration::from_secs(20),
signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key),
)
.await??;
let status: serde_json::Value = serde_json::from_str(&body)?;
match status["summary"].as_str() {
Some("finished" | "stopped") => return Ok(status),
Some("running") if Instant::now() < deadline => sleep(Duration::from_millis(250)).await,
_ => return Err(format!("heal did not reach a terminal state: {status}").into()),
}
}
}
async fn verify_two_node_directory_recovery(
remove_bucket: bool,
scan_mode: u8,
bucket_scoped: bool,
recreate: bool,
) -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut cluster = RustFSTestClusterEnvironment::new(2).await?;
// Isolate explicit administrator recovery from scanner, MRF, and disk healing.
for key in [
"RUSTFS_HEAL_AUTO_HEAL_ENABLE",
"RUSTFS_HEAL_MRF_ENABLE",
"RUSTFS_SCANNER_ENABLED",
"RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE",
] {
cluster.set_env(key, "false");
}
cluster.start().await?;
let source = cluster.create_s3_client(0)?;
let target = cluster.create_s3_client(1)?;
let bucket = "two-node-directory-recovery";
source.create_bucket().bucket(bucket).send().await?;
let objects = [
("empty", Vec::new()),
("nested/inline", vec![0x3c; 1024]),
("nested/external", (0_u8..=255).cycle().take(1024 * 1024).collect::<Vec<_>>()),
];
for (key, bytes) in &objects {
source
.put_object()
.bucket(bucket)
.key(*key)
.body(ByteStream::from(bytes.clone()))
.send()
.await?;
}
let target_bucket = Path::new(&cluster.nodes[1].data_dir).join(bucket);
let original = objects
.iter()
.map(|(key, _)| census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None))
.collect::<Result<Vec<_>, _>>()?;
for census in &original {
assert!(census.has_xl_meta, "fixture must have a target metadata copy: {census:?}");
}
assert!(
!original[2].present_part_fingerprints.is_empty(),
"large fixture must exercise external parts"
);
let removed = Path::new(&cluster.temp_dir).join("removed");
tokio::fs::create_dir(&removed).await?;
if remove_bucket {
tokio::fs::rename(&target_bucket, removed.join(bucket)).await?;
} else {
for (index, (key, _)) in objects.iter().enumerate() {
tokio::fs::rename(target_bucket.join(key), removed.join(index.to_string())).await?;
}
}
let status = two_node_heal_status(&cluster, bucket_scoped.then_some(bucket), recreate, scan_mode).await?;
assert_eq!(status["summary"], "finished", "administrator recovery should complete: {status}");
assert_eq!(status["outcome"]["execution"]["state"], "completed");
assert_eq!(status["outcome"]["coverage"], "complete");
for ((key, _), before) in objects.iter().zip(&original) {
let after = census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None)?;
assert!(after.has_xl_meta, "metadata must be restored before GET: {key}");
assert_eq!(
after.present_part_fingerprints, before.present_part_fingerprints,
"external shard bytes must match: {key}"
);
assert_eq!(
after.inline_data_fingerprint, before.inline_data_fingerprint,
"inline shard bytes must match: {key}"
);
}
cluster.stop_node(0)?;
for (key, expected) in &objects {
let actual = timeout(Duration::from_secs(30), async {
target
.get_object()
.bucket(bucket)
.key(*key)
.send()
.await?
.body
.collect()
.await
.map(|body| body.into_bytes())
.map_err(Box::<dyn Error + Send + Sync>::from)
})
.await??;
assert_eq!(actual.len(), expected.len(), "target-only read length: {key}");
assert_eq!(sha256_hex(&actual), sha256_hex(expected), "target-only content: {key}");
}
Ok(())
}
#[tokio::test]
async fn test_two_node_bucket_discovery_and_explicit_recreate() -> Result<(), Box<dyn Error + Send + Sync>> {
verify_two_node_directory_recovery(true, 2, true, true).await
}
#[tokio::test]
async fn test_two_node_missing_bucket_default_cluster_recovery() -> Result<(), Box<dyn Error + Send + Sync>> {
verify_two_node_directory_recovery(true, 2, false, false).await
}
#[tokio::test]
async fn test_two_node_missing_bucket_default_scoped_recovery() -> Result<(), Box<dyn Error + Send + Sync>> {
verify_two_node_directory_recovery(true, 2, true, false).await
}
#[tokio::test]
async fn test_two_node_object_directory_recovery_normal() -> Result<(), Box<dyn Error + Send + Sync>> {
verify_two_node_directory_recovery(false, 1, false, false).await
}
#[tokio::test]
async fn test_two_node_object_directory_recovery_deep() -> Result<(), Box<dyn Error + Send + Sync>> {
verify_two_node_directory_recovery(false, 2, false, false).await
}
}
@@ -264,7 +264,7 @@ async fn call_make_volume_response(
request: MakeVolumeRequest,
headers: HeaderMap,
) -> Result<Response<MakeVolumeResponse>, Status> {
let mut client = node_service_time_out_client_no_auth(url)
let mut client = node_service_time_out_client_no_auth(&url.to_string())
.await
.map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
let mut rpc_request = Request::new(request);
@@ -273,7 +273,7 @@ async fn call_make_volume_response(
}
async fn call_ping_response(url: &str, headers: HeaderMap) -> Result<Response<PingResponse>, Status> {
let mut client = node_service_time_out_client_no_auth(url)
let mut client = node_service_time_out_client_no_auth(&url.to_string())
.await
.map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
let mut rpc_request = Request::new(PingRequest {
@@ -562,23 +562,17 @@ mod tests {
create_bucket(&client, bucket).await.expect("Failed to create bucket");
let object_count = 1002;
// Finish every durable PUT before listing, with the same bounded fanout as the delimiter fixtures.
stream::iter(0..object_count)
.for_each_concurrent(16, |i| {
let client = &client;
async move {
let key = format!("object{i:04}.txt");
client
.put_object()
.bucket(bucket)
.key(&key)
.body(ByteStream::from_static(b"test content"))
.send()
.await
.unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
}
})
.await;
for i in 0..object_count {
let key = format!("object{:04}.txt", i);
client
.put_object()
.bucket(bucket)
.key(&key)
.body(ByteStream::from_static(b"test content"))
.send()
.await
.expect("Failed to put object");
}
eprintln!("Seeded {object_count} objects in {bucket}; starting ListObjectsV2 pagination");
let deadline = Instant::now() + PAGINATION_TIMEOUT;
+24 -441
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
use crate::common::RustFSTestClusterEnvironment;
use aws_sdk_s3::Client;
use aws_sdk_s3::error::SdkError;
use bytes::Bytes;
@@ -185,113 +185,21 @@ async fn wait_for_quorum_read_admission(clients: &[Client], bucket: &str) -> Tes
return Ok(());
}
if tokio::time::Instant::now() >= deadline {
let diagnostics = clients[0].get_object().bucket(bucket).key("warm-small").send().await;
return Err(format!(
"read quorum did not become available after lease convergence for {bucket}: GET={diagnostics:?}"
)
.into());
return Err(format!("read quorum did not become available after lease convergence for {bucket}").into());
}
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
}
}
#[tokio::test]
async fn test_cold_degraded_object_mutations_require_only_storage_quorum() -> TestResult {
crate::common::init_logging();
let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
cluster.set_env("RUSTFS_GET_OBJECT_METADATA_CACHE_DISTRIBUTED_ENABLE", "false");
cluster.start().await?;
let clients = cluster
.create_all_clients()?
.into_iter()
.map(|client| {
Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
.build(),
)
})
.collect::<Vec<_>>();
let bucket = "cold-degraded-mutations";
clients[0].create_bucket().bucket(bucket).send().await?;
// No object mutation may warm a process-wide configuration probe before
// the missing peer is introduced. Three shards still satisfy EC:2 writes.
cluster.stop_node(3)?;
let admission_deadline = Instant::now() + Duration::from_secs(30);
for client in &clients[..3] {
loop {
match client
.get_object()
.bucket(bucket)
.key("never-created-readiness-probe")
.send()
.await
{
Err(error) if error.as_service_error().and_then(|error| error.meta().code()) == Some("NoSuchKey") => break,
Err(error)
if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
&& Instant::now() < admission_deadline =>
{
tokio::time::sleep(Duration::from_millis(100)).await;
}
Err(error) => return Err(error.into()),
Ok(_) => panic!("a cold readiness probe must not create an object"),
}
}
}
let body = b"first write while one peer is offline";
clients[0]
.put_object()
.bucket(bucket)
.key("object")
.body(Bytes::from_static(body).into())
.send()
.await?;
for client in &clients[..3] {
assert_quorum_object_body(client, bucket, "object", body).await?;
}
// This coordinator has not previously committed an object mutation either.
clients[1].delete_object().bucket(bucket).key("object").send().await?;
let error = clients[2]
.get_object()
.bucket(bucket)
.key("object")
.send()
.await
.expect_err("acknowledged deletion must be visible at another entrypoint");
assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("NoSuchKey"));
cluster.stop_node(2)?;
let error = clients[0]
.put_object()
.bucket(bucket)
.key("below-quorum")
.body(Bytes::from_static(body).into())
.send()
.await
.expect_err("removing the cache gate must not weaken storage write quorum");
assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("ServiceUnavailable"));
cluster.stop();
Ok(())
}
#[tokio::test]
async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResult {
crate::common::init_logging();
for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
for (node_count, parity) in [(4, 2), (6, 3), (6, 2)] {
let read_quorum = node_count - parity;
let write_quorum = read_quorum + usize::from(read_quorum == parity);
let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
if parity == 0 {
cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
}
// Wait for every seed fanout before removing any physical shard.
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
@@ -392,28 +300,17 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul
}
cluster.stop_node(read_quorum - 1)?;
let lock_read_quorum = node_count - node_count / 2;
// EC:2 on six nodes still has three shared lock votes after losing
// its fourth data shard. Preserve the distinct data-quorum error.
let expected_code = if read_quorum - 1 < lock_read_quorum {
"ServiceUnavailable"
} else {
"SlowDownRead"
};
for client in clients.iter().take(read_quorum - 1) {
match client.get_object().bucket(&bucket).key("below-quorum").send().await {
Ok(response) => assert!(
response.body.collect().await.is_err(),
"fewer than {read_quorum} valid fragments must not reconstruct an uncached object"
),
Err(error) => {
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
assert_eq!(
error.as_service_error().and_then(|error| error.meta().code()),
Some(expected_code),
"a quorum loss must not be mistaken for a missing object"
);
}
Err(error) => assert_eq!(
error.as_service_error().and_then(|error| error.meta().code()),
Some("ServiceUnavailable"),
"a quorum loss must not be mistaken for a missing object"
),
}
}
@@ -430,277 +327,6 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul
Ok(())
}
#[tokio::test]
async fn test_cold_start_read_quorum_follows_erasure_layout() -> TestResult {
crate::common::init_logging();
for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
let read_quorum = node_count - parity;
let write_quorum = read_quorum + usize::from(read_quorum == parity);
let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
if parity == 0 {
cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
}
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
cluster.set_env("RUST_LOG", "warn");
cluster.start().await?;
let clients = cluster
.create_all_clients()?
.into_iter()
.map(|client| {
Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
.build(),
)
})
.collect::<Vec<_>>();
let bucket = format!("cold-read-quorum-{node_count}-{parity}");
clients[0].create_bucket().bucket(&bucket).send().await?;
let small = b"cold startup follows the stored EC layout".to_vec();
let large = vec![73; 256 * 1024];
for (key, body) in [
("warm-small", &small),
("warm-large", &large),
("cold-large", &large),
("below-quorum", &large),
] {
clients[0]
.put_object()
.bucket(&bucket)
.key(key)
.body(Bytes::copy_from_slice(body).into())
.send()
.await?;
for node in &cluster.nodes {
let census =
crate::chaos::census_object_version_on_disk(std::path::Path::new(&node.data_dir), &bucket, key, None)?;
assert!(census.is_complete(), "seed shard must exist before cold startup: {census:?}");
assert_eq!((census.data_blocks, census.parity_blocks), (Some(read_quorum), Some(parity)));
}
}
for node in 0..node_count {
cluster.stop_node(node)?;
}
// Rebuild every process's in-memory state below read quorum, then
// cross the EC boundary. A format bootstrap must not admit S3 reads
// merely because half the peers are online (e.g. six disks, EC:2).
for node in 0..read_quorum - 1 {
cluster.start_node_process(node).await?;
}
let error = clients[0]
.get_object()
.bucket(&bucket)
.key("below-quorum")
.send()
.await
.expect_err("cold startup below the configured EC read quorum must stay unavailable");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
cluster.start_node_process(read_quorum - 1).await?;
wait_for_quorum_read_admission(&clients[..read_quorum], &bucket).await?;
for client in clients.iter().take(read_quorum) {
assert_quorum_object_body(client, &bucket, "warm-small", &small).await?;
assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
let write = client
.put_object()
.bucket(&bucket)
.key("cold-quorum-write")
.body(Bytes::copy_from_slice(&small).into())
.send()
.await;
if read_quorum >= write_quorum {
write?;
assert_quorum_object_body(client, &bucket, "cold-quorum-write", &small).await?;
} else {
let error = write.expect_err("cold startup at read quorum must preserve the EC write quorum");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
}
}
for node in read_quorum..node_count {
cluster.start_node(node).await?;
}
for client in &clients {
assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
}
}
Ok(())
}
#[tokio::test]
async fn test_four_by_six_cold_start_at_read_quorum() -> TestResult {
crate::common::init_logging();
let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 6)).await?;
// Explicit URLs are consumed in order. Place one drive from each node in
// every set, matching a four-host, six-drive-per-host deployment.
let volumes = (0..6)
.flat_map(|drive| {
cluster
.nodes
.iter()
.map(move |node| format!("http://{}{}", node.address, node.data_dirs[drive]))
})
.collect::<Vec<_>>()
.join(" ");
cluster.set_env("RUSTFS_VOLUMES", volumes);
cluster.set_env("RUSTFS_ERASURE_SET_DRIVE_COUNT", "4");
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
cluster.set_env("RUST_LOG", "error");
cluster.start().await?;
let clients = cluster
.create_all_clients()?
.into_iter()
.map(|client| {
Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
.build(),
)
})
.collect::<Vec<_>>();
let bucket = "cold-start-read-quorum";
clients[0].create_bucket().bucket(bucket).send().await?;
let mut fixtures = Vec::new();
let mut covered = [[false; 2]; 6];
for index in 0..128 {
let key = match index {
0 => "warm-small".to_string(),
1 => "warm-large".to_string(),
_ => format!("cold-{index}"),
};
let size_class = index % 2;
let size = if size_class == 0 { 71 } else { 256 * 1024 };
let body = (0..size)
.map(|offset| u8::try_from(offset % 251).expect("bounded fixture byte"))
.collect::<Vec<_>>();
clients[0]
.put_object()
.bucket(bucket)
.key(&key)
.body(Bytes::copy_from_slice(&body).into())
.send()
.await?;
for node in &cluster.nodes {
let mut complete_shards = 0;
for (drive, path) in node.data_dirs.iter().enumerate() {
let census = crate::chaos::census_object_version_on_disk(std::path::Path::new(path), bucket, &key, None)?;
if census.has_xl_meta {
assert!(census.is_complete(), "seed shard must be complete before shutdown: {census:?}");
assert_eq!((census.data_blocks, census.parity_blocks), (Some(2), Some(2)));
covered[drive][size_class] = true;
complete_shards += 1;
}
}
assert_eq!(complete_shards, 1, "each node must own one shard of every seed object");
}
fixtures.push((key, body));
if covered.iter().flatten().all(|covered| *covered) {
break;
}
}
assert!(
covered.iter().flatten().all(|covered| *covered),
"fixtures must cover all six sets with inline and large objects"
);
for first in 0..4 {
for second in first + 1..4 {
for node in 0..4 {
cluster.stop_node(node)?;
}
cluster.start_node_process(first).await?;
let error = clients[first]
.get_object()
.bucket(bucket)
.key("warm-large")
.send()
.await
.expect_err("one node cannot satisfy read quorum after a cold start");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
cluster.start_node_process(second).await?;
wait_for_quorum_read_admission(&[clients[first].clone(), clients[second].clone()], bucket).await?;
for node in [first, second] {
for (key, body) in &fixtures {
assert_quorum_object_body(&clients[node], bucket, key, body).await?;
}
let range = clients[node]
.get_object()
.bucket(bucket)
.key("warm-large")
.range("bytes=17-1023")
.send()
.await?;
assert_eq!(range.body.collect().await?.into_bytes().as_ref(), &fixtures[1].1[17..1024]);
let listing = clients[node].list_objects_v2().bucket(bucket).send().await?;
for (key, _) in &fixtures {
assert!(
listing.contents().iter().any(|object| object.key() == Some(key.as_str())),
"listing omitted {key}"
);
}
let error = clients[node]
.put_object()
.bucket(bucket)
.key("recovery-write")
.body(Bytes::from_static(b"after recovery").into())
.send()
.await
.expect_err("two nodes must not authorize a write");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
}
let error = clients[first]
.delete_object()
.bucket(bucket)
.key("warm-large")
.send()
.await
.expect_err("two nodes must not authorize a delete");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
assert_quorum_object_body(&clients[first], bucket, "warm-large", &fixtures[1].1).await?;
let remaining = (0..4).filter(|node| *node != first && *node != second).collect::<Vec<_>>();
cluster.start_node(remaining[0]).await?;
let deadline = tokio::time::Instant::now() + Duration::from_secs(90);
loop {
match clients[first]
.put_object()
.bucket(bucket)
.key("recovery-write")
.body(Bytes::from_static(b"after recovery").into())
.send()
.await
{
Ok(_) => break,
Err(error)
if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
&& tokio::time::Instant::now() < deadline =>
{
tokio::time::sleep(Duration::from_millis(200)).await
}
Err(error) => return Err(error.into()),
}
}
for node in [first, second, remaining[0]] {
assert_quorum_object_body(&clients[node], bucket, "recovery-write", b"after recovery").await?;
}
cluster.start_node(remaining[1]).await?;
for client in &clients {
for (key, body) in &fixtures {
assert_quorum_object_body(client, bucket, key, body).await?;
}
assert_quorum_object_body(client, bucket, "recovery-write", b"after recovery").await?;
}
}
}
Ok(())
}
async fn put_object(client: Client, payload: Vec<u8>, writer_id: usize) -> Result<(), String> {
client
.put_object()
@@ -799,60 +425,6 @@ async fn test_concurrent_cluster_overwrites_do_not_fail_namespace_lock_quorum()
Ok(())
}
fn unavailable_readiness_drive_indices(payload: &serde_json::Value) -> Vec<usize> {
let mut indices = payload["details"]["storage"]["unavailableDrives"]
.as_array()
.expect("node readiness must report unavailable drives")
.iter()
.map(|drive| {
usize::try_from(drive["diskIndex"].as_u64().expect("drive index must be unsigned"))
.expect("drive index must fit usize")
})
.collect::<Vec<_>>();
indices.sort_unstable();
indices
}
fn assert_node_readiness_contract(payload: &serde_json::Value, survivors: usize) {
let read_ready = survivors >= 2;
let write_ready = survivors >= 3;
let storage = &payload["details"]["storage"];
assert_eq!(payload["ready"], read_ready);
assert_eq!(storage["ready"], read_ready);
assert_eq!(storage["readQuorum"], read_ready);
assert_eq!(storage["writeQuorum"], write_ready);
assert_eq!(storage["readinessScope"], "read_quorum");
assert_eq!(storage["source"], "local_runtime");
assert_eq!(storage["status"], if read_ready { "connected" } else { "disconnected" });
assert_eq!(payload["details"]["poolMetadata"]["ready"], true);
assert_eq!(payload["details"]["iam"]["ready"], true);
assert_eq!(payload["details"]["lock"]["ready"], read_ready);
assert_eq!(unavailable_readiness_drive_indices(payload), (survivors..4).collect::<Vec<_>>());
for drive in storage["unavailableDrives"].as_array().expect("unavailable drives") {
assert_eq!(drive["poolIndex"], 0);
assert_eq!(drive["setIndex"], 0);
let state = drive["runtimeState"].as_str().expect("runtime state must be reported");
let host_online = drive["hostOnline"].as_bool().expect("host reachability must be reported");
assert!(
matches!(state, "online" | "suspect" | "offline" | "returning"),
"unknown runtime state: {state}"
);
assert!(
state != "online" || !host_online,
"a reachable online drive cannot be excluded from quorum"
);
}
if !read_ready {
assert!(
payload["degradedReasons"]
.as_array()
.expect("degraded reasons")
.iter()
.any(|reason| reason == "storage_and_lock_unavailable")
);
}
}
async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvironment) -> TestResult {
let clients: Vec<_> = cluster
.create_all_clients()?
@@ -911,14 +483,27 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
&& payload["details"]["poolMetadata"]["ready"] == true
&& payload["details"]["iam"]["ready"] == true
&& payload["details"]["lock"]["ready"] == read_quorum
&& unavailable_readiness_drive_indices(&payload) == (survivors..4).collect::<Vec<_>>()
{
break payload;
}
assert!(Instant::now() < deadline, "node {idx}, survivors={survivors}: HTTP {status}, {payload}");
tokio::time::sleep(Duration::from_millis(200)).await;
};
assert_node_readiness_contract(&payload, survivors);
assert_eq!(payload["details"]["storage"]["readinessScope"], "read_quorum");
assert_eq!(payload["details"]["storage"]["source"], "local_runtime");
assert_eq!(
payload["details"]["storage"]["status"],
if node_ready { "connected" } else { "disconnected" }
);
if !node_ready {
assert!(
payload["degradedReasons"]
.as_array()
.expect("degraded reasons")
.iter()
.any(|reason| reason == "storage_and_lock_unavailable")
);
}
for path in ["/health/ready", "/minio/health/ready"] {
let head = http.head(format!("{url}{path}")).send().await?;
@@ -927,9 +512,7 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
let response = http.get(format!("{url}{path}")).send().await?;
assert_eq!(response.status().as_u16(), node_status);
let body: serde_json::Value = response.json().await?;
// Each response samples live diagnostics; suspect -> offline
// must not change the quorum contract or excluded topology.
assert_node_readiness_contract(&body, survivors);
assert_eq!(body["details"]["storage"], payload["details"]["storage"]);
assert_eq!(body["details"]["poolMetadata"], payload["details"]["poolMetadata"]);
}
let live = http.get(format!("{url}/health/live")).send().await?;
@@ -465,13 +465,6 @@ impl NodeService for MinimalLockNodeService {
Err(Status::unimplemented("lock-only test server"))
}
async fn recover_delete_transaction(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionRequest>,
) -> Result<Response<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionResponse>, Status> {
Err(Status::unimplemented("delete recovery is not supported by the lock fixture"))
}
async fn settle_part_transaction(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::SettlePartTransactionRequest>,
@@ -3945,15 +3945,7 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
(sse_key, sse_put.version_id().ok_or("SSE PUT omitted version ID")?.to_string()),
(large_key, large_put.version_id().ok_or("large PUT omitted version ID")?.to_string()),
];
// Managed-SSE listings may omit ETags. Check the replicated version's body
// before deleting it rather than comparing keyless listing fingerprints.
for (key, version_id) in &purged {
let source_body = get_version_body(&source_client, &source_bucket, key, version_id).await?;
let expected_sha256: [u8; 32] = Sha256::digest(&source_body).into();
wait_for_replicated_sha256(&target_client, &target_bucket, key, expected_sha256).await?;
let target_body = get_version_body(&target_client, &target_bucket, key, version_id).await?;
assert_eq!(target_body, source_body, "replicated version body differs for {key}/{version_id}");
}
assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?;
for (key, version_id) in &purged {
source_client
@@ -3964,6 +3956,10 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
.send()
.await?;
}
assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?;
let target_state = list_replication_state(&target_client, &target_bucket).await?;
assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}");
// The purge state is applied on the source asynchronously after the target
// acknowledges the delete; only then does the retained version go away and
// the bucket become deletable. A listing that is empty while DeleteBucket
@@ -3988,9 +3984,6 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
}
}
let target_state = list_replication_state(&target_client, &target_bucket).await?;
assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}");
Ok(())
}
+1 -1
View File
@@ -15,7 +15,7 @@
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::bucket::bucket_target_sys::BucketTargetSys;
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::disk::{Endpoint, RUSTFS_META_BUCKET, VolumeInfo, WalkDirOptions};
pub(crate) use rustfs_ecstore::api::disk::{RUSTFS_META_BUCKET, VolumeInfo, WalkDirOptions};
pub(crate) use rustfs_ecstore::api::rpc::{AuthenticatedChannel, TonicInterceptor, node_service_time_out_client_no_auth};
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::rpc::{
+1 -1
View File
@@ -740,7 +740,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
.version_id(version)
.send()
.await?;
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
opa.expect_bucket_tags("s3:GetObject", source, Some("finance")).await?;
client.list_objects_v2().bucket(source).send().await?;
opa.expect_bucket_tags("s3:ListBucket", source, Some("finance")).await?;
client.list_object_versions().bucket(source).send().await?;
@@ -165,210 +165,4 @@ mod tests {
info!("RT-13c PASS: scanner/config persists across restart");
Ok(())
}
/// RT-14: a transition PUT must not carry reserved `x-minio-internal-*` headers.
///
/// Regression pattern: transition forwarded the object's internal metadata as raw
/// request headers, so MinIO-derived remotes (Storj) refused every transition with
/// `400 InvalidArgument` while `tier add` still succeeded (rustfs#8362). The fake
/// target refuses any PutObject carrying such a header, like MinIO and Storj do.
#[tokio::test]
async fn test_transition_put_omits_reserved_internal_headers() -> TestResult {
use crate::fake_s3_target::{BucketMode, FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, Operation};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
BucketLifecycleConfiguration, ExpirationStatus, LifecycleRule, LifecycleRuleFilter, Transition,
TransitionStorageClass,
};
use std::time::{Duration, Instant};
init_logging();
info!("RT-14: transition PUT omits reserved internal headers");
const TIER: &str = "RT14COLD";
const TARGET_BUCKET: &str = "rt14-cold";
const SOURCE_BUCKET: &str = "rt14-hot";
let target = FakeS3Target::start().await.expect("start fake target");
target.create_bucket_with_mode(TARGET_BUCKET, BucketMode::Unversioned);
target.reject_reserved_internal_headers(true);
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(
vec![],
&[
("RUSTFS_CONSOLE_ENABLE", "false"),
("RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT", "true"),
],
)
.await
.expect("start RustFS");
let tier_body = serde_json::json!({
"type": "rustfs",
"rustfs": {
"name": TIER,
"endpoint": target.endpoint(),
"accessKey": FAKE_ACCESS_KEY,
"secretKey": FAKE_SECRET_KEY,
"bucket": TARGET_BUCKET,
"prefix": "cold/",
"region": "us-east-1",
"storageClass": ""
}
})
.to_string();
let deadline = Instant::now() + Duration::from_secs(30);
loop {
match admin_ok(&env, http::Method::PUT, "/rustfs/admin/v3/tier", Some(tier_body.clone())).await {
Ok(_) => break,
Err(err) if Instant::now() < deadline => {
info!(" add tier not ready yet: {err}");
tokio::time::sleep(Duration::from_millis(500)).await;
}
Err(err) => panic!("RT-14 FAIL: add tier: {err}"),
}
}
let client = env.create_s3_client();
client
.create_bucket()
.bucket(SOURCE_BUCKET)
.send()
.await
.expect("create bucket");
let objects: Vec<(&str, Vec<u8>)> = vec![
("small.bin", (0..1024u32).map(|i| (i % 251) as u8).collect()),
("medium.bin", (0..200_000u32).map(|i| (i.wrapping_mul(31) % 253) as u8).collect()),
];
for (key, body) in &objects {
client
.put_object()
.bucket(SOURCE_BUCKET)
.key(*key)
.body(ByteStream::from(body.clone()))
.send()
.await
.expect("put source object");
}
let rule = LifecycleRule::builder()
.id("to-cold")
.status(ExpirationStatus::Enabled)
.filter(LifecycleRuleFilter::builder().prefix("").build())
.transitions(
Transition::builder()
.days(0)
.storage_class(TransitionStorageClass::from(TIER))
.build(),
)
.build()
.expect("build rule");
client
.put_bucket_lifecycle_configuration()
.bucket(SOURCE_BUCKET)
.lifecycle_configuration(
BucketLifecycleConfiguration::builder()
.rules(rule)
.build()
.expect("build config"),
)
.send()
.await
.expect("put lifecycle");
admin_ok(
&env,
http::Method::POST,
&format!("/rustfs/admin/v3/ilm/transition/run?bucket={SOURCE_BUCKET}&prefix=&tier={TIER}&dryRun=false"),
None,
)
.await
.expect("run transition");
let deadline = Instant::now() + Duration::from_secs(60);
while target.stored_keys(TARGET_BUCKET).len() < objects.len() {
assert!(
Instant::now() < deadline,
"RT-14 FAIL: transition did not reach the target; stored keys: {:?}",
target.stored_keys(TARGET_BUCKET)
);
tokio::time::sleep(Duration::from_millis(500)).await;
}
let keys = target.stored_keys(TARGET_BUCKET);
assert_eq!(keys.len(), objects.len(), "RT-14 FAIL: unexpected remote objects: {keys:?}");
let mut stored_bodies = Vec::new();
for key in &keys {
let (body, metadata) = target.stored_object(TARGET_BUCKET, key).expect("remote object");
for name in [
"x-rustfs-internal-transition-transaction-id",
"x-minio-internal-transition-transaction-id",
"x-rustfs-internal-transition-tier-destination-id",
"x-minio-internal-transition-tier-destination-id",
] {
assert!(
metadata.contains_key(name),
"RT-14 FAIL: remote object {key} lost transition identity {name}: {:?}",
metadata.keys().collect::<Vec<_>>()
);
}
stored_bodies.push(body.to_vec());
}
let mut expected_bodies: Vec<Vec<u8>> = objects.iter().map(|(_, body)| body.clone()).collect();
expected_bodies.sort();
stored_bodies.sort();
assert_eq!(stored_bodies, expected_bodies, "RT-14 FAIL: remote bytes differ from the source objects");
let transition_puts = target
.requests()
.iter()
.filter(|record| {
record.operation == Operation::PutObject
&& record
.key
.as_deref()
.is_some_and(|key| key.contains("transition-transactions"))
})
.count();
assert_eq!(
transition_puts,
objects.len(),
"RT-14 FAIL: transition PUTs were retried or refused by the target"
);
for (key, body) in &objects {
let head = client
.head_object()
.bucket(SOURCE_BUCKET)
.key(*key)
.send()
.await
.expect("head transitioned");
assert_eq!(
head.storage_class().map(|class| class.as_str()),
Some(TIER),
"RT-14 FAIL: {key} is not reported as transitioned"
);
let got = client
.get_object()
.bucket(SOURCE_BUCKET)
.key(*key)
.send()
.await
.expect("get transitioned")
.body
.collect()
.await
.expect("read body")
.into_bytes();
assert_eq!(got.as_ref(), body.as_slice(), "RT-14 FAIL: read-back of {key} differs");
}
target.shutdown().await;
info!("RT-14 PASS: transition PUT omits reserved internal headers");
Ok(())
}
}
+33 -371
View File
@@ -33,7 +33,6 @@ use aws_sdk_s3::types::{
};
use http::{Method, StatusCode};
use serde_json::Value;
use std::future::Future;
use std::io::Write;
use std::path::{Path, PathBuf};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
@@ -58,10 +57,6 @@ const UPGRADE_READINESS_BODY: &[u8] = b"upgrade write readiness";
const PREVIOUS_RELEASE_SEED_KEY: &str = ".upgrade-readiness/previous-seed/node-0";
const MULTIPART_WORKERS: usize = 16;
const MULTIPART_UPLOADS_PER_WORKER: usize = 16;
// Bound each 64 KiB upload, including completion acknowledgement. A peer
// that accepts the request but never responds must fail the compatibility
// check with the writer and object, rather than consume the whole CI job.
const MIXED_MULTIPART_UPLOAD_TIMEOUT: Duration = Duration::from_secs(30);
// Peers keep a restarted node's drive in Suspect/Returning for roughly
// probe_interval (2s) x success_threshold (3) after it comes back; 30s
// comfortably covers that window plus CI scheduling jitter.
@@ -340,21 +335,14 @@ fn configure_cluster_logs(cluster: &mut RustFSTestClusterEnvironment) -> TestRes
async fn write_multipart_load(clients: &[Client], phase: &str) -> Result<Vec<String>, Box<dyn std::error::Error + Send + Sync>> {
let mut tasks = JoinSet::new();
for worker in 0..MULTIPART_WORKERS {
let node = worker % clients.len();
let client = clients[node].clone();
let client = clients[worker % clients.len()].clone();
let phase = phase.to_string();
tasks.spawn(async move {
let mut keys = Vec::with_capacity(MULTIPART_UPLOADS_PER_WORKER);
for upload in 0..MULTIPART_UPLOADS_PER_WORKER {
let key = format!("{phase}/multipart/{worker:02}/{upload:02}");
let part = vec![u8::try_from(worker)?; 64 * 1024];
mixed_multipart_before(
Instant::now() + MIXED_MULTIPART_UPLOAD_TIMEOUT,
node,
&key,
write_multipart(&client, MIXED_BUCKET, &key, &[part]),
)
.await?;
write_multipart(&client, MIXED_BUCKET, &key, &[part]).await?;
keys.push(key);
}
Ok::<_, Box<dyn std::error::Error + Send + Sync>>(keys)
@@ -368,172 +356,6 @@ async fn write_multipart_load(clients: &[Client], phase: &str) -> Result<Vec<Str
Ok(keys)
}
async fn mixed_multipart_before(
deadline: Instant,
node: usize,
key: &str,
upload: impl Future<Output = TestResult>,
) -> TestResult {
tokio::time::timeout_at(deadline, upload)
.await
.map_err(|_| MultipartAcknowledgementDeadline {
node,
key: key.to_owned(),
})?
}
#[derive(Debug)]
struct MultipartAcknowledgementDeadline {
node: usize,
key: String,
}
impl std::fmt::Display for MultipartAcknowledgementDeadline {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
formatter,
"node {}: multipart upload {MIXED_BUCKET}/{} acknowledgement deadline exceeded",
self.node, self.key
)
}
}
impl std::error::Error for MultipartAcknowledgementDeadline {}
// Decode only the public lock snapshot fields. Unknown fields, including
// credential/configuration payloads, never enter the failure artifact.
#[derive(serde::Deserialize, serde::Serialize)]
struct MixedLockSnapshot {
total: Option<u64>,
truncated: Option<bool>,
capability_note: Option<String>,
locks: Vec<MixedLockEntry>,
}
#[derive(serde::Deserialize, serde::Serialize)]
struct MixedLockEntry {
resource: Option<String>,
bucket: Option<String>,
object: Option<String>,
version: Option<String>,
owner: Option<String>,
#[serde(rename = "type")]
lock_type: Option<String>,
priority: Option<String>,
since: Option<String>,
elapsed_secs: Option<u64>,
ttl_secs: Option<u64>,
}
async fn read_mixed_lock_snapshot(response: &mut reqwest::Response) -> Result<MixedLockSnapshot, &'static str> {
const MAX_BODY_BYTES: usize = 1024 * 1024;
let mut body = Vec::new();
while let Some(chunk) = response.chunk().await.map_err(|_| "body_read_error")? {
if chunk.len() > MAX_BODY_BYTES - body.len() {
return Err("body_limit_exceeded");
}
body.extend_from_slice(&chunk);
}
serde_json::from_slice(&body).map_err(|_| "invalid_lock_snapshot")
}
async fn probe_mixed_multipart_failure(
cluster: &RustFSTestClusterEnvironment,
node: usize,
path: &str,
head: bool,
deadline: Instant,
) -> Value {
let started = Instant::now();
let method = if head { Method::HEAD } else { Method::GET };
let mut observation = serde_json::json!({
"node": node, "method": method.as_str(), "path": path,
"status": null, "observation": "unknown", "timeout": false,
});
// The shared absolute deadline bounds both headers and a slow/infinite
// response body; a send-only timeout would leave the cluster alive forever.
let result = tokio::time::timeout_at(deadline, async {
let mut response = signed_request(
method,
&format!("{}{path}", cluster.nodes[node].url.trim_end_matches('/')),
&cluster.access_key,
&cluster.secret_key,
None,
None,
)
.await
.map_err(|_| "request_error")?;
let status = response.status();
observation["status"] = serde_json::json!(status.as_u16());
if head {
observation["observation"] = serde_json::json!(match status {
StatusCode::OK => "visible",
StatusCode::NOT_FOUND => "not_visible",
_ => "unknown",
});
for (field, header) in [
("content_length", "content-length"),
("etag", "etag"),
("version", "x-amz-version-id"),
] {
observation[field] = serde_json::json!(response.headers().get(header).and_then(|value| value.to_str().ok()));
}
} else if status == StatusCode::OK {
observation["snapshot"] =
serde_json::to_value(read_mixed_lock_snapshot(&mut response).await?).map_err(|_| "snapshot_encode_error")?;
observation["observation"] = serde_json::json!("captured");
}
Ok::<_, &'static str>(())
})
.await;
match result {
Ok(Ok(())) => {}
Ok(Err(reason)) => observation["error"] = serde_json::json!(reason),
Err(_) => observation["timeout"] = serde_json::json!(true),
}
observation["elapsed_secs"] = serde_json::json!(started.elapsed().as_secs_f64());
observation
}
async fn observe_mixed_multipart_failure(
cluster: &RustFSTestClusterEnvironment,
phase: &str,
error: BoxError,
log_dir: Option<&Path>,
) -> TestResult {
if let (Some(log_dir), Some(failure)) = (log_dir, error.downcast_ref::<MultipartAcknowledgementDeadline>()) {
let started = Instant::now();
let deadline = started + Duration::from_secs(5);
// Reserve the last second for serialization and bounded artifact I/O.
// All probes run directly in this future; no diagnostic task is spawned.
let network_deadline = started + Duration::from_secs(4);
let path = format!("/{MIXED_BUCKET}/{}", failure.key);
let _ = tokio::time::timeout_at(deadline, async {
let (heads, locks) = tokio::join!(
futures::future::join_all((0..cluster.nodes.len()).map(|node| probe_mixed_multipart_failure(
cluster,
node,
&path,
true,
network_deadline
))),
probe_mixed_multipart_failure(cluster, 0, "/rustfs/admin/v3/top/locks?count=1000", false, network_deadline),
);
let snapshot = serde_json::json!({
"phase": phase, "bucket": MIXED_BUCKET, "key": failure.key,
"writer_node": failure.node, "heads": heads, "top_locks": locks,
});
let body = serde_json::to_vec_pretty(&snapshot).map_err(|_| ())?;
tokio::fs::write(log_dir.join("mixed-multipart-failure.json"), body)
.await
.map_err(|_| ())
})
.await;
}
// Visibility and diagnostic failures cannot acknowledge the original MPU.
Err(error)
}
/// Assert that `client` eventually lists exactly `expected` objects under
/// `{phase}/`, polling until [`LISTING_CONVERGENCE_TIMEOUT`].
///
@@ -578,7 +400,7 @@ async fn exercise_mixed_cluster(
previous_node: usize,
) -> TestResult {
let clients = cluster.create_all_clients()?;
wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, phase, LISTING_CONVERGENCE_TIMEOUT).await?;
wait_for_upgrade_write_readiness(&clients, phase, LISTING_CONVERGENCE_TIMEOUT).await?;
let current_client = &clients[current_node];
let previous_client = &clients[previous_node];
@@ -604,13 +426,7 @@ async fn exercise_mixed_cluster(
.await?;
assert_eq!(read_object(current_client, MIXED_BUCKET, &previous_key, None).await?.1, previous_body);
let multipart_keys = match write_multipart_load(&clients, phase).await {
Ok(keys) => keys,
Err(error) => {
let log_dir = std::env::var_os("RUSTFS_E2E_LOG_DIR");
return observe_mixed_multipart_failure(cluster, phase, error, log_dir.as_deref().map(Path::new)).await;
}
};
let multipart_keys = write_multipart_load(&clients, phase).await?;
let expected_count = multipart_keys.len() + 2;
for (label, client) in [("current", current_client), ("previous", previous_client)] {
assert_eq!(
@@ -646,12 +462,7 @@ fn upgrade_probe_client(client: &Client) -> Client {
)
}
pub(crate) async fn wait_for_upgrade_write_readiness(
clients: &[Client],
bucket: &str,
phase: &str,
budget: Duration,
) -> TestResult {
async fn wait_for_upgrade_write_readiness(clients: &[Client], phase: &str, budget: Duration) -> TestResult {
// ListBuckets can succeed before peers recover a restarted disk. Cluster
// health also accepts Returning disks whose write health is still FAULTY.
// Probe every writer outside the asserted phase prefix; compatibility
@@ -671,7 +482,7 @@ pub(crate) async fn wait_for_upgrade_write_readiness(
deadline,
client
.put_object()
.bucket(bucket)
.bucket(MIXED_BUCKET)
.key(&key)
.body(ByteStream::from_static(UPGRADE_READINESS_BODY))
.send(),
@@ -706,20 +517,19 @@ async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnviro
// waits for pool metadata (#7473). First prove the elected writer can
// persist data, then restart each old process once with three peers still
// readable. This preparation ends before any current binary is started.
wait_for_upgrade_write_readiness(&clients[..1], MIXED_BUCKET, "previous-seed", LISTING_CONVERGENCE_TIMEOUT).await?;
wait_for_upgrade_write_readiness(&clients[..1], "previous-seed", LISTING_CONVERGENCE_TIMEOUT).await?;
for node in [1, 2, 3, 0] {
cluster.stop_node(node)?;
cluster.start_node_from_binary(node, previous_binary).await?;
wait_for_upgrade_write_readiness(
std::slice::from_ref(&clients[node]),
MIXED_BUCKET,
&format!("previous-restart-{node}"),
LISTING_CONVERGENCE_TIMEOUT,
)
.await?;
}
wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?;
wait_for_upgrade_write_readiness(&clients, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?;
for (reader, client) in clients.iter().enumerate() {
assert_eq!(
read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
@@ -746,10 +556,9 @@ mod upgrade_write_readiness_tests {
#[tokio::test]
async fn waits_for_each_writer_after_metadata_is_ready() -> TestResult {
let target = FakeS3Target::start().await?;
let bucket = "distributed-upgrade-history";
target.create_bucket(bucket);
target.create_bucket(MIXED_BUCKET);
let client = fake_source_client(&target);
client.head_bucket().bucket(bucket).send().await?;
client.head_bucket().bucket(MIXED_BUCKET).send().await?;
let phase = "one-previous-node";
let first_key = format!(".upgrade-readiness/{phase}/node-0");
@@ -769,7 +578,7 @@ mod upgrade_write_readiness_tests {
// Metadata readiness does not prove that a data write can succeed.
let premature = client
.put_object()
.bucket(bucket)
.bucket(MIXED_BUCKET)
.key(&first_key)
.body(ByteStream::from_static(b"upgrade write readiness"))
.send()
@@ -777,15 +586,15 @@ mod upgrade_write_readiness_tests {
.expect_err("metadata readiness does not prove write readiness");
assert_eq!(premature.raw_response().map(|response| response.status().as_u16()), Some(503));
wait_for_upgrade_write_readiness(&[client.clone(), client.clone()], bucket, phase, Duration::from_secs(5)).await?;
wait_for_upgrade_write_readiness(&[client.clone(), client.clone()], phase, Duration::from_secs(5)).await?;
assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &first_key), 3);
assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &second_key), 2);
assert!(target.has_object(bucket, &first_key));
assert!(target.has_object(bucket, &second_key));
assert!(target.has_object(MIXED_BUCKET, &first_key));
assert!(target.has_object(MIXED_BUCKET, &second_key));
assert!(
client
.list_objects_v2()
.bucket(bucket)
.bucket(MIXED_BUCKET)
.prefix(format!("{phase}/"))
.send()
.await?
@@ -815,10 +624,9 @@ mod upgrade_write_readiness_tests {
let phase = format!("permanent-{}", status.as_u16());
let key = format!(".upgrade-readiness/{phase}/node-0");
target.inject_for_key(FakeTargetOperation::PutObject, &key, FaultAction::Status(status), 1);
let error =
wait_for_upgrade_write_readiness(std::slice::from_ref(&client), MIXED_BUCKET, &phase, Duration::from_secs(5))
.await
.expect_err("a permanent error must not be retried into success");
let error = wait_for_upgrade_write_readiness(std::slice::from_ref(&client), &phase, Duration::from_secs(5))
.await
.expect_err("a permanent error must not be retried into success");
assert!(error.to_string().contains("node 0 write readiness failed"), "{error}");
assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &key), 1);
assert!(!target.has_object(MIXED_BUCKET, &key));
@@ -841,7 +649,7 @@ mod upgrade_write_readiness_tests {
FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE),
10,
);
let error = wait_for_upgrade_write_readiness(&[client], MIXED_BUCKET, phase, Duration::from_secs(1))
let error = wait_for_upgrade_write_readiness(&[client], phase, Duration::from_secs(1))
.await
.expect_err("persistent unavailability must exhaust the shared deadline");
// Transport scheduling consumes the same budget; do not require a
@@ -860,134 +668,10 @@ mod upgrade_write_readiness_tests {
let phase = "stalled";
let key = format!(".upgrade-readiness/{phase}/node-0");
target.inject_for_key(FakeTargetOperation::PutObject, &key, FaultAction::Stall(Duration::from_secs(30)), 1);
let error = wait_for_upgrade_write_readiness(std::slice::from_ref(&client), MIXED_BUCKET, phase, Duration::from_secs(1))
let error = wait_for_upgrade_write_readiness(&[client], phase, Duration::from_secs(1))
.await
.expect_err("a stalled request must not outlive the readiness deadline");
assert!(error.to_string().contains("deadline exceeded during PutObject"), "{error}");
let part = vec![2_u8; 64 * 1024];
let successful_key = "stalled/multipart/02/00";
mixed_multipart_before(
Instant::now() + MIXED_MULTIPART_UPLOAD_TIMEOUT,
2,
successful_key,
write_multipart(&client, MIXED_BUCKET, successful_key, std::slice::from_ref(&part)),
)
.await?;
assert_eq!(read_object(&client, MIXED_BUCKET, successful_key, None).await?.1, part);
let stalled_key = "stalled/multipart/02/01";
target.inject_for_key(
FakeTargetOperation::CompleteMultipartUpload,
stalled_key,
FaultAction::Stall(Duration::from_secs(30)),
1,
);
let error = {
let upload = write_multipart(&client, MIXED_BUCKET, stalled_key, std::slice::from_ref(&part));
tokio::pin!(upload);
tokio::select! {
result = &mut upload => panic!("stalled completion unexpectedly returned: {result:?}"),
observed = tokio::time::timeout(Duration::from_secs(10), async {
while target.count_requests(FakeTargetOperation::CompleteMultipartUpload, stalled_key) == 0
|| !target.has_object(MIXED_BUCKET, stalled_key)
{
sleep(Duration::from_millis(10)).await;
}
}) => observed.expect("multipart completion must reach and commit on the fake peer"),
}
// An already committed object is not an acknowledged client success.
// Expire only after observing the real completion request, so host
// scheduling cannot turn this into a timeout before admission.
tokio::time::timeout(Duration::from_secs(1), mixed_multipart_before(Instant::now(), 2, stalled_key, upload))
.await
.expect("expired deadline must stop the in-flight completion")
.expect_err("unacknowledged completion must fail even if the peer committed")
};
assert!(error.to_string().contains("node 2"), "{error}");
assert!(error.to_string().contains(stalled_key), "{error}");
assert!(error.to_string().contains("acknowledgement deadline exceeded"), "{error}");
assert_eq!(target.count_requests(FakeTargetOperation::CompleteMultipartUpload, stalled_key), 1);
let failure = error
.downcast_ref::<MultipartAcknowledgementDeadline>()
.expect("the timeout must retain typed writer/object context");
assert_eq!(failure.node, 2);
assert_eq!(failure.key, stalled_key);
let original_message = error.to_string();
// These four probe addresses share an in-process peer. This verifies
// bounded diagnostic fan-out, not a four-node RustFS cluster.
let mut probes = RustFSTestClusterEnvironment::new(MIXED_NODE_COUNT).await?;
probes.access_key = FAKE_ACCESS_KEY.to_owned();
probes.secret_key = FAKE_SECRET_KEY.to_owned();
for node in &mut probes.nodes {
node.url = target.endpoint().to_owned();
}
let log_dir = tempfile::tempdir()?;
target.inject_for_key(
FakeTargetOperation::HeadObject,
stalled_key,
FaultAction::Stall(Duration::from_secs(30)),
1,
);
let error = tokio::time::timeout(
Duration::from_secs(7),
observe_mixed_multipart_failure(&probes, phase, error, Some(log_dir.path())),
)
.await
.expect("the shared diagnostic budget must include artifact writing")
.expect_err("observing committed data cannot acknowledge the stalled upload");
assert_eq!(error.to_string(), original_message);
assert!(error.downcast_ref::<MultipartAcknowledgementDeadline>().is_some());
let snapshot: Value =
serde_json::from_slice(&tokio::fs::read(log_dir.path().join("mixed-multipart-failure.json")).await?)?;
assert_eq!(snapshot["key"], stalled_key);
assert_eq!(snapshot["writer_node"], 2);
let heads = snapshot["heads"]
.as_array()
.expect("failure artifact must retain every probe");
assert_eq!(heads.len(), MIXED_NODE_COUNT);
assert_eq!(
heads
.iter()
.filter(|head| head["timeout"] == true && head["observation"] == "unknown")
.count(),
1
);
assert_eq!(
heads
.iter()
.filter(|head| head["status"] == 200 && head["observation"] == "visible")
.count(),
3
);
assert_eq!(target.count_requests(FakeTargetOperation::HeadObject, stalled_key), MIXED_NODE_COUNT);
assert_eq!(target.count_requests(FakeTargetOperation::CompleteMultipartUpload, stalled_key), 1);
assert_eq!(snapshot["top_locks"]["observation"], "unknown", "the fake peer has no admin lock API");
let locks: MixedLockSnapshot = serde_json::from_value(serde_json::json!({
"total": 1, "truncated": true, "capability_note": "partial snapshot",
"credentials": { "secret_key": "must not persist" },
"locks": [{
"resource": format!("{MIXED_BUCKET}/{stalled_key}"), "owner": "writer", "type": "WRITE",
"elapsed_secs": 30, "ttl_secs": 1, "merged_config": { "secret_key": "must not persist" },
}],
}))?;
let safe_locks = serde_json::to_value(locks)?;
assert_eq!(safe_locks["total"], 1);
assert_eq!(safe_locks["truncated"], true);
assert_eq!(safe_locks["locks"][0]["owner"], "writer");
assert_eq!(safe_locks["locks"][0]["elapsed_secs"], 30);
assert!(safe_locks.get("credentials").is_none());
assert!(safe_locks["locks"][0].get("merged_config").is_none());
let mut body = br#"{"locks":[]}"#.to_vec();
body.resize(1024 * 1024, b' ');
let mut at_limit = reqwest::Response::from(http::Response::new(body.clone()));
assert!(read_mixed_lock_snapshot(&mut at_limit).await?.locks.is_empty());
body.push(b' ');
let mut oversized = reqwest::Response::from(http::Response::new(body));
assert_eq!(read_mixed_lock_snapshot(&mut oversized).await.err(), Some("body_limit_exceeded"));
target.shutdown().await;
Ok(())
}
@@ -2345,7 +2029,7 @@ fn layout_reported_parts(attributes: &aws_sdk_s3::operation::get_object_attribut
attributes.object_parts().map(|parts| parts.parts().len())
}
async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
let label = case.label();
let head = layout_head(client, case).await?;
assert_eq!(
@@ -2362,12 +2046,7 @@ async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &s
let full = layout_get(client, case, None, None).await?.body.collect().await?.into_bytes();
assert_eq!(full.len(), case.body.len(), "{context}: {label}: full GET length");
assert!(full == case.body, "{context}: {label}: full GET body must equal the rc.5 upload");
Ok(())
}
async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
assert_layout_full_body(client, case, context).await?;
let label = case.label();
if case.is_multipart_layout() {
let first = case.part_sizes[0];
let range = format!("bytes={}-{}", first - 32, first + 31);
@@ -2502,9 +2181,9 @@ struct LayoutTransport {
uploaded_parts: Vec<i32>,
single_puts: usize,
completes: usize,
/// Allowlisted request identity and prepared response metadata. A prepared
/// response does not prove that the client received it.
journal: Vec<Value>,
/// Raw per-key journal in target order: (sequence, operation, part number,
/// upload id), so duplicate drives can be told apart from retries.
journal: Vec<(u64, String, Option<i32>, Option<String>)>,
}
/// Configure every layout bucket to replicate its existing objects to a fresh
@@ -2566,28 +2245,12 @@ async fn replicate_layouts(
journal: key_requests
.iter()
.map(|record| {
serde_json::json!({
"sequence": record.sequence,
"operation": format!("{:?}", record.operation),
"part_number": record.part_number,
"upload_id": record.upload_id,
"requested_version_id": record.version_id,
"source_version_id": record.source_version_id,
"source_mtime": record.source_mtime,
"source_etag": record.source_etag,
"source_replication_request": record.source_replication_request,
"request_content_length": record.content_length,
"journaled_at_unix_millis": record.journaled_at_unix_millis,
"prepared_response": record.prepared_response.as_ref().map(|response| serde_json::json!({
"status": response.status,
"version_id": response.version_id,
"etag": response.etag,
"last_modified": response.last_modified,
"content_length": response.content_length,
"prepared_at_unix_millis": response.prepared_at_unix_millis,
"elapsed_secs": response.elapsed.as_secs_f64(),
})),
})
(
record.sequence,
format!("{:?}", record.operation),
record.part_number,
record.upload_id.as_ref().map(|id| id.chars().take(12).collect()),
)
})
.collect(),
};
@@ -2646,11 +2309,10 @@ async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult {
"rc.5 wrote a legacy layout"
);
}
// Prove the rc.5 writer persisted every original byte before upgrading.
// Its compressed cross-part range reader has a legacy limitation; the
// upgraded reader must still satisfy every range and partNumber assertion.
// The rc.5 writer must itself still read what it wrote, so a later
// failure is attributable to the upgrade rather than to the fixture.
for case in &cases {
assert_layout_full_body(&old_client, case, "rc.5").await?;
assert_layout_readable(&old_client, case, "rc.5").await?;
}
// Upgrade in place.
+3 -4
View File
@@ -176,14 +176,13 @@ regex = { workspace = true }
path-absolutize = { workspace = true }
rmp.workspace = true
rmp-serde.workspace = true
tokio-util = { workspace = true, features = ["io", "compat", "rt"] }
tokio-util = { workspace = true, features = ["io", "compat"] }
tokio-stream = { workspace = true, features = ["sync"] }
hmac = { workspace = true }
sha2 = { workspace = true }
hex-simd = { workspace = true }
tempfile.workspace = true
hyper = { workspace = true, features = ["http2", "http1", "server"] }
h2 = { workspace = true }
hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
hyper-rustls = { workspace = true, default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs"] }
hostname.workspace = true
@@ -201,8 +200,8 @@ md-5.workspace = true
memmap2 = { workspace = true }
libc.workspace = true
# "process" adds getrlimit for the io_uring fd-cache RLIMIT_NOFILE gate
# (backlog#1178); "thread" provides cleanup affinity; "fs" is a workspace default.
rustix = { workspace = true, features = ["process", "fs", "thread"] }
# (backlog#1178); "fs" comes from the workspace default.
rustix = { workspace = true, features = ["process", "fs"] }
rustfs-madmin.workspace = true
reqwest = { workspace = true }
aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+15 -24
View File
@@ -199,10 +199,10 @@ pub mod bucket {
get_object_lock_config, get_object_lock_config_state, get_on_demand_migration_config,
get_on_demand_migration_config_in, get_public_access_block_config, get_quota_config, get_replication_config,
get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
remove_bucket_metadata_if_absent, set_bucket_metadata, update, update_bucket_targets_under_transaction_lock,
update_config_with, update_if_incarnation, update_if_incarnation_at, update_quota_if_incarnation,
update_quota_if_incarnation_at, update_under_transaction_lock, update_under_transaction_lock_at,
init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, set_bucket_metadata,
update, update_bucket_targets_under_transaction_lock, update_config_with, update_if_incarnation,
update_if_incarnation_at, update_quota_if_incarnation, update_quota_if_incarnation_at, update_under_transaction_lock,
update_under_transaction_lock_at,
};
#[cfg(feature = "test-util")]
pub use crate::bucket::metadata_sys::{ConfigWriteLockProbe, test_support};
@@ -210,8 +210,7 @@ pub mod bucket {
pub mod migration {
pub use crate::bucket::migration::{
LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_bucket_metadata_for_buckets,
try_migrate_iam_config,
LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_iam_config,
};
}
@@ -241,10 +240,7 @@ pub mod bucket {
}
pub mod quota {
pub use crate::bucket::quota::{
BucketQuota, QUOTA_RESERVATION_PROTOCOL_V1, QUOTA_RESERVATION_PROTOCOL_V2, QuotaCheckResult, QuotaError,
QuotaOperation,
};
pub use crate::bucket::quota::{BucketQuota, QuotaCheckResult, QuotaError, QuotaOperation};
pub mod checker {
pub use crate::bucket::quota::checker::QuotaChecker;
@@ -366,11 +362,10 @@ pub mod config {
ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config, delete_config_no_lock,
is_server_config_corrupt_error, lookup_configs, read_config, read_config_no_lock, read_config_with_metadata,
read_config_without_migrate, read_config_without_migrate_no_lock, read_existing_server_config_no_lock,
read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_config_with_opts_quiet,
save_server_config, save_server_config_no_lock, save_server_config_snapshot,
save_server_config_snapshot_with_generation, server_config_path, try_migrate_server_config,
with_config_object_read_lock, with_config_object_write_lock, with_server_config_read_lock,
with_server_config_write_lock,
read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_server_config,
save_server_config_no_lock, save_server_config_snapshot, save_server_config_snapshot_with_generation,
server_config_path, try_migrate_server_config, with_config_object_read_lock, with_config_object_write_lock,
with_server_config_read_lock, with_server_config_write_lock,
};
}
@@ -423,9 +418,7 @@ pub mod disk {
pub use crate::disk::os::{LocalPublicationPause, LocalPublicationStage};
pub use crate::disk::{
BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction,
DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation,
DeleteRecoveryRpcReply, DeleteRecoveryState, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
@@ -475,7 +468,7 @@ pub mod event {
pub mod global {
pub use crate::runtime::global::{
set_global_endpoints, set_global_region, set_global_rustfs_port, set_object_store_resolver, shutdown_background_services,
update_erasure_type, wait_for_detached_mutations,
update_erasure_type,
};
}
@@ -522,14 +515,13 @@ pub mod integrity {
pub mod object {
pub use crate::object_api::{
BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReadCondition, GetObjectReadMetadata, GetObjectReader,
NamespaceLockFence, ObjectEncryptionResolver, ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions,
PutObjReader, QuotaAdmission, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission,
RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, ScannerPublicationCommitScope, ScannerPublicationCommitStartError,
ScannerPublicationCommitState, ShardIntegrityWriteMode, StreamConsumer, WriteCommitGuard, WriteCompletion,
get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
with_get_object_read_condition,
};
pub use crate::store::{
PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
@@ -594,7 +586,6 @@ pub mod set_disk {
#[cfg(feature = "test-util")]
pub mod test_util {
pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;
pub use crate::set_disk::ConditionalReadBenchmarkMetadataGuard;
pub use crate::set_disk::{MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause};
/// Keep a namespace commit pending until the returned owner is dropped.
@@ -87,7 +87,6 @@ use rustfs_filemeta::{
FileInfo, FileInfoOpts, NULL_VERSION_ID, RestoreStatus, RestoreStatusOps, TRANSITION_COMPLETE, get_file_info,
is_restored_object_on_disk,
};
use rustfs_lock::{NamespaceLockGuard, error::LockError};
use rustfs_scanner_metrics::metrics::{
IlmAction, Metrics, ScannerLifecycleExpiryStateUpdate, ScannerLifecycleTransitionStateUpdate, global_metrics,
};
@@ -191,8 +190,6 @@ const TIER_FREE_VERSION_RECOVERY_BASE_INTERVAL: StdDuration = StdDuration::from_
// before another full sweep can discover work persisted by another node.
const TIER_FREE_VERSION_RECOVERY_MAX_IDLE_INTERVAL: StdDuration = StdDuration::from_secs(10 * 60);
const TIER_FREE_VERSION_RECOVERY_JITTER_PERCENT: u64 = 10;
const TIER_FREE_VERSION_RECOVERY_LEADER_LOCK: &str = "tier-free-version-recovery/leader.lock";
const TIER_FREE_VERSION_RECOVERY_LOCK_TIMEOUT: StdDuration = StdDuration::from_secs(1);
const DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS: i64 = 5;
const EXPIRY_WORKER_QUEUE_CAPACITY: usize = 1000;
const DEFAULT_MANUAL_TRANSITION_JOB_RECOVERY_LIMIT: usize = 100;
@@ -1444,24 +1441,13 @@ pub(crate) async fn enqueue_committed_free_versions(api: &ECStore, free_versions
queued
}
async fn enqueue_recovered_free_version_with_state(
state: &Arc<RwLock<ExpiryState>>,
oi: ObjectInfo,
cancel: &CancellationToken,
) -> bool {
async fn enqueue_recovered_free_version_with_state(state: &Arc<RwLock<ExpiryState>>, oi: ObjectInfo) -> bool {
let task = FreeVersionTask(oi);
let hash = task.op_hash();
let (wrkr, stats) = {
let state = select! {
biased;
_ = cancel.cancelled() => return false,
state = state.read() => state,
};
let state = state.read().await;
(state.get_worker_ch(hash), Arc::clone(&state.stats))
};
if cancel.is_cancelled() {
return false;
}
let Some(wrkr) = wrkr else {
stats.increment_missed_freevers_tasks();
stats.record_scanner_expiry_state();
@@ -1511,13 +1497,9 @@ fn set_recovered_free_version_enqueue_observer(
RecoveredFreeVersionEnqueueObserverGuard
}
pub(super) async fn enqueue_recovered_free_version_with_cancel(
api: &ECStore,
oi: ObjectInfo,
cancel: &CancellationToken,
) -> bool {
pub async fn enqueue_recovered_free_version(api: &ECStore, oi: ObjectInfo) -> bool {
let expiry_state = api.ctx.expiry_state();
let queued = enqueue_recovered_free_version_with_state(&expiry_state, oi, cancel).await;
let queued = enqueue_recovered_free_version_with_state(&expiry_state, oi).await;
#[cfg(test)]
if let Some(observer) = RECOVERED_FREE_VERSION_ENQUEUE_OBSERVER
@@ -2417,16 +2399,6 @@ fn is_slow_down(err: &Error) -> bool {
matches!(err, Error::SlowDown)
}
/// Lifecycle expiry is idempotent when the exact object or version has already
/// disappeared between the scanner listing and the delete admission.
///
/// Keep this predicate narrower than bucket/volume "not found" helpers: an
/// unavailable bucket or disk is an operational failure and must not be
/// reported as a completed expiry.
fn is_lifecycle_expiry_already_gone(err: &Error) -> bool {
is_err_object_not_found(err) || is_err_version_not_found(err)
}
/// Resolves the expiry worker count from the single documented knob,
/// `RUSTFS_MAX_EXPIRY_WORKERS`: a set, parsable, non-zero value wins;
/// anything else falls back to `min(cpus, 16)`. The historical
@@ -2949,20 +2921,6 @@ fn resolve_tier_free_version_recovery_enabled(value: Result<String, env::VarErro
}
fn spawn_tier_free_version_recovery_once(api: Arc<ECStore>, started: &OnceLock<()>) -> Option<JoinHandle<()>> {
spawn_tier_free_version_recovery_once_with_options(
api,
started,
TIER_FREE_VERSION_RECOVERY_LEADER_LOCK.to_string(),
tier_free_version_recovery_leader_retry_delay,
)
}
fn spawn_tier_free_version_recovery_once_with_options(
api: Arc<ECStore>,
started: &OnceLock<()>,
leader_lock_name: String,
retry_delay: fn() -> StdDuration,
) -> Option<JoinHandle<()>> {
if !tier_free_version_recovery_enabled() {
warn!(
event = EVENT_LIFECYCLE_WORKER_STATE,
@@ -2979,43 +2937,20 @@ fn spawn_tier_free_version_recovery_once_with_options(
Some(tokio::spawn(async move {
let cancel_token = api.ctx.background_cancel_token().unwrap_or_default();
let acquire_api = Arc::clone(&api);
let acquire_lock_name = Arc::<str>::from(leader_lock_name);
run_tier_free_version_recovery_coordinator(
let expiry_state = api.ctx.expiry_state();
run_tier_free_version_recovery_loop(
cancel_token,
retry_delay,
move || {
let api = Arc::clone(&acquire_api);
let lock_name = Arc::clone(&acquire_lock_name);
async move {
let lock = api.new_ns_lock(RUSTFS_META_BUCKET, &lock_name).await?;
match lock.get_write_lock_quiet(TIER_FREE_VERSION_RECOVERY_LOCK_TIMEOUT).await {
Ok(guard) => Ok(Some(guard)),
Err(LockError::Timeout { .. }) => Ok(None),
Err(err) => Err(err.into()),
}
}
},
move |leader_cancel| {
expiry_state,
jitter_tier_free_version_recovery_delay,
move |bucket_marker, object_marker, recovery_cancel| {
let api = Arc::clone(&api);
async move {
run_tier_free_version_recovery_loop(
leader_cancel,
api.ctx.expiry_state(),
jitter_tier_free_version_recovery_delay,
move |bucket_marker, object_marker, recovery_cancel| {
let api = Arc::clone(&api);
async move {
recover_tier_free_versions_with_cancel(
api,
DEFAULT_FREE_VERSION_RECOVERY_LIMIT,
bucket_marker,
object_marker,
recovery_cancel,
)
.await
}
},
recover_tier_free_versions_with_cancel(
api,
DEFAULT_FREE_VERSION_RECOVERY_LIMIT,
bucket_marker,
object_marker,
recovery_cancel,
)
.await
}
@@ -3025,86 +2960,6 @@ fn spawn_tier_free_version_recovery_once_with_options(
}))
}
fn tier_free_version_recovery_leader_retry_delay() -> StdDuration {
StdDuration::from_secs(rand::rng().random_range(15..=30))
}
async fn run_tier_free_version_recovery_coordinator<A, AF, R, RF>(
cancel: CancellationToken,
retry_delay: fn() -> StdDuration,
mut acquire: A,
mut recover: R,
) where
A: FnMut() -> AF,
AF: Future<Output = crate::error::Result<Option<NamespaceLockGuard>>>,
R: FnMut(CancellationToken) -> RF,
RF: Future<Output = ()>,
{
loop {
let acquired = select! {
biased;
_ = cancel.cancelled() => return,
result = acquire() => result,
};
match acquired {
Ok(Some(mut guard)) => {
let leader_cancel = cancel.child_token();
// Hold ownership across every page, cooldown and idle wait. A
// follower taking over starts a fresh sweep of persisted markers.
let recovery = recover(leader_cancel.clone());
tokio::pin!(recovery);
debug!(
event = EVENT_LIFECYCLE_WORKER_STATE,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
state = "free_version_recovery_leader_acquired",
lock_name = TIER_FREE_VERSION_RECOVERY_LEADER_LOCK,
"Tier free-version recovery leader acquired"
);
select! {
biased;
_ = cancel.cancelled() => {
leader_cancel.cancel();
recovery.await;
}
_ = guard.lock_lost_notified() => {
debug!(
event = EVENT_LIFECYCLE_WORKER_STATE,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
state = "free_version_recovery_leader_lost",
lock_name = TIER_FREE_VERSION_RECOVERY_LEADER_LOCK,
"Tier free-version recovery leader lost"
);
// Dropping the recovery future would detach its active
// walk. Cancel it and poll the same future through cleanup.
leader_cancel.cancel();
recovery.await;
}
_ = &mut recovery => {}
}
leader_cancel.cancel();
guard.release();
}
Ok(None) => {}
Err(err) => warn!(
event = EVENT_LIFECYCLE_WORKER_STATE,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
state = "free_version_recovery_leader_acquire_failed",
lock_name = TIER_FREE_VERSION_RECOVERY_LEADER_LOCK,
error = %err,
"Tier free-version recovery leader acquisition failed"
),
}
select! {
biased;
_ = cancel.cancelled() => return,
_ = tokio::time::sleep(retry_delay()) => {}
}
}
}
async fn run_tier_free_version_recovery_loop<F, Fut>(
cancel_token: CancellationToken,
expiry_state: Arc<RwLock<ExpiryState>>,
@@ -3114,11 +2969,7 @@ async fn run_tier_free_version_recovery_loop<F, Fut>(
F: FnMut(Option<String>, Option<String>, CancellationToken) -> Fut,
Fut: Future<Output = crate::error::Result<FreeVersionRecoveryStats>>,
{
let recovery_notify = select! {
biased;
_ = cancel_token.cancelled() => return,
state = expiry_state.read() => Arc::clone(&state.recovery_notify),
};
let recovery_notify = Arc::clone(&expiry_state.read().await.recovery_notify);
let mut schedule = TierFreeVersionRecoverySchedule::default();
loop {
@@ -3141,11 +2992,7 @@ async fn run_tier_free_version_recovery_loop<F, Fut>(
schedule.record_success(&stats);
rustfs_io_metrics::record_stage_duration("lifecycle_free_version_recovery", elapsed.as_secs_f64() * 1000.0);
let (pending_tasks, active_tasks) = {
let state = select! {
biased;
_ = cancel_token.cancelled() => return,
state = expiry_state.read() => state,
};
let state = expiry_state.read().await;
(state.pending_tasks(), state.stats.active_tasks())
};
debug!(
@@ -4901,7 +4748,7 @@ async fn expire_transitioned_object_with_lock_lost_signal(
bucket_incarnation_id: Uuid,
lock_lost_signal: Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>>,
) -> Result<ObjectInfo, std::io::Error> {
let publication_guard = lifecycle_expiry_publication_guard(&api, &oi.bucket, &oi.name, bucket_incarnation_id)
let publication_guard = lifecycle_expiry_publication_guard(&api, oi, bucket_incarnation_id)
.await
.ok_or_else(|| std::io::Error::other("lifecycle expiry is not allowed for this bucket"))?;
let snapshot = lifecycle_delete_config_snapshot(&api, oi)
@@ -4930,20 +4777,6 @@ async fn expire_transitioned_object_with_lock_lost_signal(
//audit_log_lifecycle(*oi, ILMExpiry, tags, traceFn);
Ok(dobj)
}
Err(err) if is_lifecycle_expiry_already_gone(&err) => {
debug!(
event = EVENT_LIFECYCLE_DELETE_COMPLETED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
bucket = %oi.bucket,
object = %oi.name,
operation = "delete_restored_object",
reason = "already_gone",
"Lifecycle delete target was already absent"
);
crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
Ok(oi.clone())
}
Err(err) => Err(std::io::Error::other(err)),
};
}
@@ -4954,8 +4787,6 @@ async fn expire_transitioned_object_with_lock_lost_signal(
// pointing to a remote tier version that has already been removed. If this
// only creates a delete marker, remote cleanup must be driven by persisted
// free-version recovery rather than the visible delete result.
// Keep remote transitioned not-found errors visible: local metadata
// absence alone does not prove that the tier object was cleaned up.
let dobj = match api.delete_object(&oi.bucket, &oi.name, opts).await {
Ok(obj) => obj,
Err(e) => {
@@ -5007,14 +4838,10 @@ pub fn gen_transition_objname(bucket: &str) -> Result<String, Error> {
pub async fn transition_object(api: Arc<ECStore>, oi: &ObjectInfo, lae: LcAuditEvent) -> Result<(), Error> {
let time_ilm = Metrics::time_ilm(lae.event.action);
let (publication_guard, bucket_incarnation_id) = lifecycle_transition_publication_guard(&api, oi)
.await
.ok_or_else(|| Error::other("lifecycle transition is not allowed for this bucket"))?;
let etag = if let Some(etag) = &oi.etag { etag } else { "" };
let etag = etag.to_string();
let mut opts = ObjectOptions {
let opts = ObjectOptions {
transition: TransitionOptions {
status: lifecycle::TRANSITION_PENDING.to_string(),
tier: lae.event.storage_class,
@@ -5026,10 +4853,8 @@ pub async fn transition_object(api: Arc<ECStore>, oi: &ObjectInfo, lae: LcAuditE
versioned: BucketVersioningSys::prefix_enabled(&oi.bucket, &oi.name).await,
version_suspended: BucketVersioningSys::prefix_suspended(&oi.bucket, &oi.name).await,
mod_time: oi.mod_time,
expected_bucket_incarnation_id: Some(bucket_incarnation_id),
..Default::default()
};
opts.add_namespace_lock_guard(&publication_guard);
let result = api.transition_object(&oi.bucket, &oi.name, &opts).await;
time_ilm(1)();
result
@@ -5633,26 +5458,20 @@ pub async fn apply_transition_rule(event: &lifecycle::Event, src: &LcEventSrc, o
.await
}
async fn acquire_lifecycle_publication_guard(api: &ECStore, bucket: &str) -> Result<rustfs_lock::NamespaceLockGuard, Error> {
let lock = api
.new_ns_lock(bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
.await?;
let guard = lock.get_read_lock(get_lock_acquire_timeout()).await.map_err(Error::other)?;
if guard.is_lock_lost() {
return Err(Error::other("table-bucket publication lock was lost before lifecycle admission"));
}
Ok(guard)
}
pub(crate) async fn lifecycle_expiry_publication_guard(
async fn lifecycle_expiry_publication_guard(
api: &ECStore,
bucket: &str,
object: &str,
oi: &ObjectInfo,
bucket_incarnation_id: Uuid,
) -> Option<rustfs_lock::NamespaceLockGuard> {
let result: std::result::Result<Option<rustfs_lock::NamespaceLockGuard>, Error> = async {
let guard = acquire_lifecycle_publication_guard(api, bucket).await?;
if !metadata_boundary::lifecycle_expiry_allowed(api, bucket, bucket_incarnation_id).await? {
let result = async {
let lock = api
.new_ns_lock(&oi.bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
.await?;
let guard = lock.get_read_lock(get_lock_acquire_timeout()).await.map_err(Error::other)?;
if guard.is_lock_lost() {
return Err(Error::other("table-bucket publication lock was lost before lifecycle delete admission"));
}
if !metadata_boundary::lifecycle_expiry_allowed(api, &oi.bucket, bucket_incarnation_id).await? {
return Ok(None);
}
Ok(Some(guard))
@@ -5665,8 +5484,8 @@ pub(crate) async fn lifecycle_expiry_publication_guard(
event = EVENT_LIFECYCLE_DELETE_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
bucket,
object,
bucket = %oi.bucket,
object = %oi.name,
operation = "authorize_lifecycle_expiry",
error = %err,
"Lifecycle delete admission failed"
@@ -5676,34 +5495,6 @@ pub(crate) async fn lifecycle_expiry_publication_guard(
}
}
async fn lifecycle_transition_publication_guard(
api: &ECStore,
oi: &ObjectInfo,
) -> Option<(rustfs_lock::NamespaceLockGuard, Uuid)> {
let result: std::result::Result<(rustfs_lock::NamespaceLockGuard, Uuid), Error> = async {
let guard = acquire_lifecycle_publication_guard(api, &oi.bucket).await?;
let configs = metadata_boundary::get_expiry_configs(api, &oi.bucket).await?;
Ok((guard, configs.bucket_incarnation_id))
}
.await;
match result {
Ok(state) => Some(state),
Err(err) => {
warn!(
event = EVENT_LIFECYCLE_TIER_OPERATION_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
bucket = %oi.bucket,
object = %oi.name,
operation = "authorize_lifecycle_transition",
error = %err,
"Lifecycle transition admission failed"
);
None
}
}
}
pub async fn apply_expiry_on_transitioned_object(
api: Arc<ECStore>,
oi: &ObjectInfo,
@@ -5760,8 +5551,7 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
bucket_incarnation_id: Uuid,
lock_lost_signal: Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>>,
) -> bool {
let Some(publication_guard) = lifecycle_expiry_publication_guard(&api, &oi.bucket, &oi.name, bucket_incarnation_id).await
else {
let Some(publication_guard) = lifecycle_expiry_publication_guard(&api, oi, bucket_incarnation_id).await else {
return false;
};
let snapshot = match lifecycle_delete_config_snapshot(&api, oi).await {
@@ -5821,27 +5611,11 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
} else {
None
};
// The delete owns the publication guard from here on, so a cancelled
// expiry worker cannot let table-bucket publication overtake it.
let mut dobj = match api
.delete_object_with_tier_delete_journal_and_guards(&oi.bucket, &encode_dir_object(&oi.name), opts, [publication_guard])
.delete_object_with_tier_delete_journal(&oi.bucket, &encode_dir_object(&oi.name), opts)
.await
{
Ok(dobj) => dobj,
Err(err) if is_lifecycle_expiry_already_gone(&err) => {
debug!(
event = EVENT_LIFECYCLE_DELETE_COMPLETED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
bucket = %oi.bucket,
object = %oi.name,
operation = "delete_object",
reason = "already_gone",
"Lifecycle delete target was already absent"
);
crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
return true;
}
Err(e) => {
error!(
event = EVENT_LIFECYCLE_DELETE_FAILED,
@@ -6070,7 +5844,6 @@ pub async fn apply_lifecycle_action(event: &lifecycle::Event, src: &LcEventSrc,
#[cfg(test)]
mod tests {
use super::expiry_worker_count;
use super::lifecycle_transition_publication_guard;
use super::{
DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS, DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX,
DEFAULT_TRANSITION_WORKERS_CAP, EVENT_LIFECYCLE_EVALUATION_FAILED, EVENT_LIFECYCLE_EXPIRED_DETECTED,
@@ -6081,16 +5854,15 @@ mod tests {
cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at,
enqueue_recovered_free_version_with_state, enqueue_transition_for_existing_objects_scoped,
enqueue_transition_with_lifecycle, enqueue_transition_with_lifecycle_report, eval_action_from_lifecycle,
get_lock_acquire_timeout, is_lifecycle_expiry_already_gone, jitter_tier_free_version_recovery_delay,
lifecycle_action_blocked_by_replication, lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object,
lifecycle_replication_blocks_action, lifecycle_rule_has_date_expiration, lifecycle_version_delete_target,
manual_transition_duration_elapsed, manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink,
manual_transition_version_marker, manual_transition_worker_failure_reason,
mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate,
persist_manual_transition_job_progress_if_owned, persist_manual_transition_page_checkpoint,
recover_manual_transition_job, recover_manual_transition_jobs, resolve_tier_free_version_recovery_enabled,
resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count,
resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator, run_tier_free_version_recovery_loop,
get_lock_acquire_timeout, jitter_tier_free_version_recovery_delay, lifecycle_action_blocked_by_replication,
lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object, lifecycle_replication_blocks_action,
lifecycle_rule_has_date_expiration, lifecycle_version_delete_target, manual_transition_duration_elapsed,
manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink, manual_transition_version_marker,
manual_transition_worker_failure_reason, mark_delete_opts_skip_decommissioned_on_remote_success,
merge_stale_multipart_candidate, persist_manual_transition_job_progress_if_owned,
persist_manual_transition_page_checkpoint, recover_manual_transition_job, recover_manual_transition_jobs,
resolve_tier_free_version_recovery_enabled, resolve_transition_queue_capacity, resolve_transition_queue_send_timeout,
resolve_transition_worker_count, resolve_transition_workers_absolute_max, run_tier_free_version_recovery_loop,
select_restore_s3_location, set_lifecycle_observability_observer, set_recovered_free_version_enqueue_observer,
should_defer_date_expiry_for_recent_config_update, transitioned_cleanup_tuple, transitioned_object_delete_opts,
wait_for_tier_free_version_recovery,
@@ -6187,231 +5959,6 @@ mod tests {
use tokio_util::sync::CancellationToken;
use uuid::Uuid;
fn recovery_test_lock() -> (Arc<rustfs_lock::NamespaceLock>, Vec<Arc<dyn rustfs_lock::LockClient>>) {
let clients: Vec<Arc<dyn rustfs_lock::LockClient>> = (0..3)
.map(|_| {
Arc::new(rustfs_lock::LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::new())))
as Arc<dyn rustfs_lock::LockClient>
})
.collect();
let lock =
rustfs_lock::NamespaceLock::with_clients_and_quorum(format!("recovery-test-{}", Uuid::new_v4()), clients.clone(), 2);
(Arc::new(lock), clients)
}
async fn acquire_recovery_test_lock(
lock: Arc<rustfs_lock::NamespaceLock>,
owner: String,
) -> crate::error::Result<Option<rustfs_lock::NamespaceLockGuard>> {
// Keep this lease longer than the virtual-time cooldown/idle advances;
// production uses the namespace wrapper's renewable 30-second lease.
lock.lock_guard(
rustfs_lock::ObjectKey::new(crate::disk::RUSTFS_META_BUCKET, super::TIER_FREE_VERSION_RECOVERY_LEADER_LOCK),
&owner,
StdDuration::from_millis(5),
StdDuration::from_secs(3600),
)
.await
.map_err(Into::into)
}
fn recovery_test_retry_delay() -> StdDuration {
StdDuration::from_secs(15)
}
#[tokio::test(start_paused = true)]
#[serial]
async fn tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over() {
let (lock, _clients) = recovery_test_lock();
let (pages_tx, mut pages_rx) = tokio::sync::mpsc::unbounded_channel();
let mut tasks = Vec::new();
let mut cancels = Vec::new();
for node in 0..8 {
let cancel = CancellationToken::new();
cancels.push(cancel.clone());
let lock = Arc::clone(&lock);
let pages_tx = pages_tx.clone();
tasks.push(tokio::spawn(async move {
run_tier_free_version_recovery_coordinator(
cancel,
recovery_test_retry_delay,
move || acquire_recovery_test_lock(Arc::clone(&lock), format!("node-{node}")),
move |leader_cancel| {
let pages_tx = pages_tx.clone();
async move {
let mut pages = 0;
run_tier_free_version_recovery_loop(
leader_cancel,
ExpiryState::new(),
std::convert::identity,
move |bucket, object, _| {
pages += 1;
pages_tx.send((node, bucket, object)).expect("page observer should be open");
std::future::ready(Ok(free_version_recovery_stats(0, 0, pages == 1)))
},
)
.await;
}
},
)
.await;
}));
}
let (leader, bucket, object) = pages_rx.recv().await.expect("one coordinator should start recovery");
assert!(bucket.is_none() && object.is_none());
// The same leader must survive both a truncated page's cooldown and
// complete idle sweeps. All other coordinators keep probing the lock.
let mut observed_pages = 1;
for _ in 0..27 {
tokio::time::advance(StdDuration::from_secs(60)).await;
for _ in 0..100 {
tokio::task::yield_now().await;
}
while let Ok((node, bucket, object)) = pages_rx.try_recv() {
assert_eq!(node, leader, "followers must not repeat a namespace sweep");
if observed_pages == 1 {
assert_eq!(bucket.as_deref(), Some("bucket"));
assert_eq!(object.as_deref(), Some("object"));
} else {
assert!(bucket.is_none() && object.is_none(), "complete sweeps should reset their cursors");
}
observed_pages += 1;
}
}
assert!(observed_pages >= 7, "the leader should finish a full ten-minute idle wait");
cancels[leader].cancel();
let (successor, bucket, object) = tokio::time::timeout(StdDuration::from_secs(60), pages_rx.recv())
.await
.expect("a follower should take over after leader shutdown")
.expect("page observer should remain open");
assert_ne!(successor, leader);
assert!(bucket.is_none() && object.is_none(), "a new leader must restart discovery safely");
for cancel in cancels {
cancel.cancel();
}
for task in tasks {
task.await.expect("all coordinators should shut down cleanly");
}
}
#[tokio::test(start_paused = true)]
#[serial]
async fn tier_free_version_recovery_lock_loss_waits_for_active_page_cleanup() {
let (lock, _clients) = recovery_test_lock();
let cancel = CancellationToken::new();
let loop_cancel = cancel.clone();
let (page_tx, mut page_rx) = tokio::sync::mpsc::unbounded_channel();
let cleanup_started = Arc::new(tokio::sync::Notify::new());
let finish_cleanup = Arc::new(tokio::sync::Notify::new());
let observed_cleanup = Arc::clone(&cleanup_started);
let release_cleanup = Arc::clone(&finish_cleanup);
let task = tokio::spawn(async move {
run_tier_free_version_recovery_coordinator(
loop_cancel,
recovery_test_retry_delay,
move || acquire_recovery_test_lock(Arc::clone(&lock), "old-leader".to_string()),
move |leader_cancel| {
let page_tx = page_tx.clone();
let cleanup_started = Arc::clone(&cleanup_started);
let finish_cleanup = Arc::clone(&finish_cleanup);
async move {
run_tier_free_version_recovery_loop(
leader_cancel,
ExpiryState::new(),
std::convert::identity,
move |_, _, page_cancel| {
let cleanup_started = Arc::clone(&cleanup_started);
let finish_cleanup = Arc::clone(&finish_cleanup);
page_tx.send(()).expect("page observer should remain open");
async move {
page_cancel.cancelled().await;
cleanup_started.notify_one();
finish_cleanup.notified().await;
Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "cancelled page").into())
}
},
)
.await;
}
},
)
.await;
});
page_rx.recv().await.expect("leader should start an active page");
// Let authoritative backend leases expire before the next heartbeat
// can renew them. A failed refresh must cancel the active page.
tokio::time::advance(StdDuration::from_secs(3600)).await;
tokio::time::timeout(StdDuration::from_secs(1), observed_cleanup.notified())
.await
.expect("expired backend leases must trigger recovery cancellation");
assert!(!task.is_finished(), "lock loss must wait for the active page's cleanup");
assert!(page_rx.try_recv().is_err(), "the old leader must not start another page");
cancel.cancel();
release_cleanup.notify_one();
task.await.expect("coordinator should finish cleanup before exiting");
}
#[tokio::test(start_paused = true)]
async fn tier_free_version_recovery_coordinator_retries_without_scanning_after_lock_error() {
let cancel = CancellationToken::new();
let loop_cancel = cancel.clone();
let attempts = Arc::new(AtomicUsize::new(0));
let observed_attempts = Arc::clone(&attempts);
let task = tokio::spawn(async move {
run_tier_free_version_recovery_coordinator(
loop_cancel,
recovery_test_retry_delay,
move || {
observed_attempts.fetch_add(1, Ordering::SeqCst);
std::future::ready(Err(rustfs_lock::LockError::InsufficientNodes {
required: 2,
available: 1,
}
.into()))
},
|_| async { panic!("a node without ownership must never start discovery") },
)
.await;
});
tokio::task::yield_now().await;
assert_eq!(attempts.load(Ordering::SeqCst), 1);
tokio::time::advance(StdDuration::from_secs(14)).await;
assert_eq!(attempts.load(Ordering::SeqCst), 1, "lock errors must not cause a tight retry loop");
tokio::time::advance(StdDuration::from_secs(1)).await;
tokio::task::yield_now().await;
assert_eq!(attempts.load(Ordering::SeqCst), 2);
cancel.cancel();
task.await.expect("a follower should respond to shutdown during retry wait");
}
#[tokio::test]
async fn tier_free_version_recovery_enqueue_cancels_while_waiting_for_state_lock() {
let state = ExpiryState::new();
let (tx, mut rx) = tokio::sync::mpsc::channel(1);
let mut held = state.write().await;
held.tasks_tx.push(tx);
let cancel = CancellationToken::new();
let enqueue_cancel = cancel.clone();
let enqueue_state = Arc::clone(&state);
let enqueue = tokio::spawn(async move {
enqueue_recovered_free_version_with_state(&enqueue_state, ObjectInfo::default(), &enqueue_cancel).await
});
tokio::task::yield_now().await;
assert!(!enqueue.is_finished(), "enqueue should be blocked on the state lock");
cancel.cancel();
let queued = tokio::time::timeout(StdDuration::from_secs(1), enqueue)
.await
.expect("cancellation must not wait for the state write guard")
.expect("enqueue should return cleanly");
assert!(!queued);
assert_eq!(held.pending_tasks(), 0);
assert_eq!(held.stats.missed_free_vers_tasks(), 0);
assert!(rx.try_recv().is_err(), "a cancelled recovery must not enqueue a task");
drop(held);
assert!(!enqueue_recovered_free_version_with_state(&state, ObjectInfo::default(), &cancel).await);
assert!(rx.try_recv().is_err(), "cancellation must win even when the lock is available");
}
fn free_version_recovery_stats(enqueued: usize, failed: usize, truncated: bool) -> FreeVersionRecoveryStats {
FreeVersionRecoveryStats {
scanned: enqueued.saturating_add(failed),
@@ -8091,7 +7638,7 @@ mod tests {
assert_eq!(state.read().await.stats.pending_tasks(), 1);
// The single-slot queue is full for both enqueue paths.
assert!(!enqueue_recovered_free_version_with_state(&state, oi.clone(), &CancellationToken::new()).await);
assert!(!enqueue_recovered_free_version_with_state(&state, oi.clone()).await);
assert_eq!(state.read().await.stats.pending_tasks(), 1);
assert_eq!(state.read().await.stats.missed_free_vers_tasks(), 1);
@@ -8123,7 +7670,7 @@ mod tests {
..Default::default()
};
let queued = enqueue_recovered_free_version_with_state(&state, oi, &CancellationToken::new()).await;
let queued = enqueue_recovered_free_version_with_state(&state, oi).await;
let state = state.read().await;
assert!(!queued);
@@ -8252,8 +7799,8 @@ mod tests {
..Default::default()
};
let first = enqueue_recovered_free_version_with_state(&state, oi.clone(), &CancellationToken::new()).await;
let second = enqueue_recovered_free_version_with_state(&state, oi, &CancellationToken::new()).await;
let first = enqueue_recovered_free_version_with_state(&state, oi.clone()).await;
let second = enqueue_recovered_free_version_with_state(&state, oi).await;
let state = state.read().await;
assert!(first);
@@ -8354,7 +7901,7 @@ mod tests {
};
assert!(
super::enqueue_recovered_free_version_with_cancel(&ecstore, oi, &CancellationToken::new()).await,
super::enqueue_recovered_free_version(&ecstore, oi).await,
"the resized production worker queue should accept the task"
);
stop_tx.send(None).await.expect("worker stop signal should be delivered");
@@ -10552,26 +10099,6 @@ mod tests {
);
}
#[test]
fn lifecycle_expiry_already_gone_only_accepts_object_or_version_absence() {
assert!(is_lifecycle_expiry_already_gone(&Error::FileNotFound));
assert!(is_lifecycle_expiry_already_gone(&Error::ObjectNotFound(
"bucket".to_string(),
"object".to_string(),
)));
assert!(is_lifecycle_expiry_already_gone(&Error::FileVersionNotFound));
assert!(is_lifecycle_expiry_already_gone(&Error::VersionNotFound(
"bucket".to_string(),
"object".to_string(),
"version".to_string(),
)));
assert!(!is_lifecycle_expiry_already_gone(&Error::VolumeNotFound));
assert!(!is_lifecycle_expiry_already_gone(&Error::DiskNotFound));
assert!(!is_lifecycle_expiry_already_gone(&Error::ErasureReadQuorum));
assert!(!is_lifecycle_expiry_already_gone(&Error::MethodNotAllowed));
}
#[tokio::test]
async fn manual_transition_counts_already_transitioned_object() {
let lc = latest_transition_lifecycle();
@@ -13446,9 +12973,7 @@ mod tests {
.bucket_incarnation_id_from_disk(&bucket)
.await
.expect("bucket incarnation");
// Replaying the same scanner snapshot after the delete is an
// already-gone success and must not be reported as a failure.
for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, true)] {
for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, false)] {
assert_eq!(
super::apply_expiry_on_non_transitioned_objects(
ecstore.clone(),
@@ -13503,171 +13028,6 @@ mod tests {
}
}
#[tokio::test]
#[serial]
async fn queued_noncurrent_batch_expiry_does_not_delete_from_table_bucket() {
let (_disk_paths, ecstore) = setup_test_env().await;
let bucket = format!("table-noncurrent-lifecycle-{}", Uuid::new_v4().simple());
let object = "tables/table-id/data/part-00001.parquet";
create_test_bucket(&ecstore, &bucket).await;
metadata_sys::update_in(
&ecstore.ctx,
&bucket,
BUCKET_VERSIONING_CONFIG,
b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".to_vec(),
)
.await
.expect("bucket versioning should be enabled");
let mut old_reader = PutObjReader::from_vec(b"old table data".to_vec());
let old = ecstore
.put_object(
&bucket,
object,
&mut old_reader,
&ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
.expect("the historical table version should be created");
let old_version_id = old.version_id.expect("the historical version should have an identity");
let mut current_reader = PutObjReader::from_vec(b"current table data".to_vec());
ecstore
.put_object(
&bucket,
object,
&mut current_reader,
&ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
.expect("the current table version should be created");
let incarnation = ecstore
.bucket_incarnation_id_from_disk(&bucket)
.await
.expect("bucket incarnation should be available");
let publication_lock = ecstore
.new_ns_lock(&bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
.await
.expect("table-bucket publication lock should be created");
let enable_guard = publication_lock
.get_write_lock(get_lock_acquire_timeout())
.await
.expect("table-bucket enablement should acquire the publication lock");
let delete_store = ecstore.clone();
let delete_bucket = bucket.clone();
let delete_object = object.to_string();
let mut delete = tokio::spawn(async move {
let target = ObjectToDelete {
object_name: delete_object,
version_id: Some(old_version_id),
..Default::default()
};
crate::bucket::lifecycle::object_handlers_common::delete_object_versions(
&delete_store,
&delete_bucket,
std::slice::from_ref(&target),
lifecycle::Event::default(),
incarnation,
)
.await
});
tokio::task::yield_now().await;
assert!(
tokio::time::timeout(StdDuration::from_millis(100), &mut delete)
.await
.is_err(),
"noncurrent lifecycle cleanup must wait for the table-bucket publication fence"
);
let sys = metadata_sys::bucket_metadata_sys_of(&ecstore.ctx).expect("metadata system should be initialized");
let sys = sys.read().await.clone();
let mut metadata = (*sys.get(&bucket).await.expect("bucket metadata should exist")).clone();
metadata.table_bucket_config_json = br#"{"enabled":true}"#.to_vec();
sys.persist_and_set(metadata)
.await
.expect("table bucket marker should be persisted");
sys.reload_from_store(&bucket)
.await
.expect("table bucket marker should become authoritative");
drop(enable_guard);
let failed = tokio::time::timeout(StdDuration::from_secs(2), delete)
.await
.expect("noncurrent lifecycle cleanup should resume after the fence is released")
.expect("noncurrent lifecycle cleanup task should join");
assert_eq!(failed, 1, "table-bucket noncurrent cleanup must be rejected");
assert!(
ecstore
.get_object_info(
&bucket,
object,
&ObjectOptions {
version_id: Some(old_version_id.to_string()),
versioned: true,
..Default::default()
},
)
.await
.is_ok(),
"table data must remain after queued noncurrent cleanup is rejected"
);
}
#[tokio::test]
#[serial]
async fn lifecycle_transition_admission_holds_publication_reader() {
let (_disk_paths, ecstore) = setup_test_env().await;
let bucket = format!("lifecycle-transition-{}", Uuid::new_v4().simple());
let object = "tables/table-id/data/part-00001.parquet";
create_test_bucket(&ecstore, &bucket).await;
let object_info = ObjectInfo {
bucket: bucket.clone(),
name: object.to_string(),
..Default::default()
};
let publication_lock = ecstore
.new_ns_lock(&bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
.await
.expect("table-bucket publication lock should be created");
let enable_guard = publication_lock
.get_write_lock(get_lock_acquire_timeout())
.await
.expect("publication writer should be acquired");
let admission_store = ecstore.clone();
let admission_object = object_info.clone();
let mut admission =
tokio::spawn(async move { lifecycle_transition_publication_guard(&admission_store, &admission_object).await });
assert!(
tokio::time::timeout(StdDuration::from_millis(100), &mut admission)
.await
.is_err(),
"transition admission must wait for an in-progress table publication"
);
drop(enable_guard);
let (transition_guard, incarnation) = tokio::time::timeout(StdDuration::from_secs(2), admission)
.await
.expect("transition admission should resume after publication")
.expect("transition admission task should join")
.expect("transition admission should succeed");
assert_ne!(incarnation, Uuid::nil(), "transition admission must bind a bucket incarnation");
let blocked_writer = publication_lock.get_write_lock(StdDuration::from_millis(100)).await;
assert!(blocked_writer.is_err(), "transition admission must retain the publication read lock");
drop(transition_guard);
publication_lock
.get_write_lock(get_lock_acquire_timeout())
.await
.expect("publication writer should proceed after transition admission releases its guard");
}
#[tokio::test]
#[serial]
async fn queued_delete_all_rechecks_a_same_id_rule_moved_into_the_future() {
@@ -14270,10 +13630,6 @@ mod tests {
.await;
}
fn fast_tier_free_version_recovery_leader_retry_delay() -> StdDuration {
StdDuration::from_millis(10)
}
#[tokio::test]
#[serial]
async fn tier_free_version_recovery_production_entrypoint_enqueues_seeded_item() {
@@ -14289,13 +13645,8 @@ mod tests {
seed_recoverable_free_version(&disk_paths, &bucket, object, None, None).await;
let started = OnceLock::new();
let recovery = super::spawn_tier_free_version_recovery_once_with_options(
Arc::clone(&ecstore),
&started,
format!("tier-free-version-recovery/test/{}", Uuid::new_v4()),
fast_tier_free_version_recovery_leader_retry_delay,
)
.expect("production recovery entrypoint should start once");
let recovery = super::spawn_tier_free_version_recovery_once(Arc::clone(&ecstore), &started)
.expect("production recovery entrypoint should start once");
let task = tokio::time::timeout(StdDuration::from_secs(30), async { recovery_rx.lock().await.recv().await })
.await
.expect("production recovery entrypoint should enqueue the seeded item")
@@ -14338,14 +13689,13 @@ mod tests {
};
let mut recovery_rx = recovery_rx.lock().await;
assert!(
super::enqueue_recovered_free_version_with_cancel(
super::enqueue_recovered_free_version(
&ecstore,
ObjectInfo {
bucket: "prefill".to_string(),
name: "prefill".to_string(),
..Default::default()
},
&CancellationToken::new(),
)
.await,
"the production recovery queue should accept its first task"
@@ -36,10 +36,6 @@ pub async fn delete_object_versions(
_lc_event: lifecycle::Event,
bucket_incarnation_id: Uuid,
) -> usize {
if to_del.is_empty() {
return 0;
}
if to_del.iter().any(|target| {
target.version_id.is_none()
|| (target.version_id.is_some_and(|version_id| version_id.is_nil()) && target.expected_identity.is_none())
@@ -56,25 +52,6 @@ pub async fn delete_object_versions(
return to_del.len();
}
let Some(publication_guard) = super::bucket_lifecycle_ops::lifecycle_expiry_publication_guard(
api,
bucket,
to_del[0].object_name.as_str(),
bucket_incarnation_id,
)
.await
else {
debug!(
event = EVENT_LIFECYCLE_CLEANUP_SKIPPED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
bucket,
target_count = to_del.len(),
reason = "publication_admission_rejected",
"Skipped lifecycle noncurrent version cleanup"
);
return to_del.len();
};
let delete_config_snapshot = match ReplicationObjectBridge::delete_request_config(api, bucket).await {
Ok(snapshot) => Arc::new(snapshot),
Err(err) => {
@@ -101,13 +78,17 @@ pub async fn delete_object_versions(
remaining = &[];
}
let mut delete_options = ObjectOptions {
delete_replication_config_snapshot: Some(Arc::clone(&delete_config_snapshot)),
expected_bucket_incarnation_id: Some(bucket_incarnation_id),
..Default::default()
};
delete_options.add_namespace_lock_guard(&publication_guard);
let (mut deleted_objs, errors) = api.delete_objects(bucket, to_del.to_vec(), delete_options).await;
let (mut deleted_objs, errors) = api
.delete_objects(
bucket,
to_del.to_vec(),
ObjectOptions {
delete_replication_config_snapshot: Some(Arc::clone(&delete_config_snapshot)),
expected_bucket_incarnation_id: Some(bucket_incarnation_id),
..Default::default()
},
)
.await;
failed += errors.iter().filter(|err| err.is_some()).count();
for (i, deleted_obj) in deleted_objs.iter_mut().enumerate() {
@@ -20,7 +20,7 @@ use std::time::Duration;
use tokio::sync::mpsc;
use tokio_util::sync::CancellationToken;
use crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_recovered_free_version_with_cancel;
use crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_recovered_free_version;
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::Result;
use crate::object_api::ObjectInfo;
@@ -201,11 +201,7 @@ pub(super) async fn recover_tier_free_versions_with_cancel(
return Err(tier_free_version_recovery_cancelled());
}
retry_cursor.visit(&oi);
let queued = enqueue_recovered_free_version_with_cancel(&api, oi, &cancel_token).await;
if cancel_token.is_cancelled() {
return Err(tier_free_version_recovery_cancelled());
}
if !record_recovered_free_version_enqueue(&mut stats, queued) {
if !record_recovered_free_version_enqueue(&mut stats, enqueue_recovered_free_version(&api, oi).await) {
let (bucket_marker, object_marker) = retry_cursor.retry_markers();
stats.truncated = true;
stats.next_bucket_marker = bucket_marker;
-161
View File
@@ -1210,7 +1210,6 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.notification_config = None;
if !self.notification_config_xml.is_empty()
&& let Err(e) = deserialize::<NotificationConfiguration>(&self.notification_config_xml)
.map(|c| self.notification_config = Some(c))
@@ -1225,7 +1224,6 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.lifecycle_config = None;
if !self.lifecycle_config_xml.is_empty()
&& let Err(e) =
deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml).map(|c| self.lifecycle_config = Some(c))
@@ -1240,7 +1238,6 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.object_lock_config = None;
if !self.object_lock_config_xml.is_empty()
&& let Err(e) =
deserialize::<ObjectLockConfiguration>(&self.object_lock_config_xml).map(|c| self.object_lock_config = Some(c))
@@ -1270,7 +1267,6 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.sse_config = None;
if !self.encryption_config_xml.is_empty()
&& let Err(e) =
deserialize::<ServerSideEncryptionConfiguration>(&self.encryption_config_xml).map(|c| self.sse_config = Some(c))
@@ -1285,7 +1281,6 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.tagging_config = None;
if !self.tagging_config_xml.is_empty()
&& let Err(e) = deserialize::<Tagging>(&self.tagging_config_xml).map(|c| self.tagging_config = Some(c))
{
@@ -1351,7 +1346,6 @@ impl BucketMetadata {
} else {
self.bucket_target_config = Some(BucketTargets::default());
}
self.cors_config = None;
if !self.cors_config_xml.is_empty()
&& let Err(e) = deserialize::<CORSConfiguration>(&self.cors_config_xml).map(|c| self.cors_config = Some(c))
{
@@ -1365,7 +1359,6 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.logging_config = None;
if !self.logging_config_xml.is_empty()
&& let Err(e) = deserialize::<BucketLoggingStatus>(&self.logging_config_xml).map(|c| self.logging_config = Some(c))
{
@@ -1379,7 +1372,6 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.website_config = None;
if !self.website_config_xml.is_empty()
&& let Err(e) = deserialize::<WebsiteConfiguration>(&self.website_config_xml).map(|c| self.website_config = Some(c))
{
@@ -1393,7 +1385,6 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.accelerate_config = None;
if !self.accelerate_config_xml.is_empty()
&& let Err(e) =
deserialize::<AccelerateConfiguration>(&self.accelerate_config_xml).map(|c| self.accelerate_config = Some(c))
@@ -1408,7 +1399,6 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.request_payment_config = None;
if !self.request_payment_config_xml.is_empty()
&& let Err(e) = deserialize::<RequestPaymentConfiguration>(&self.request_payment_config_xml)
.map(|c| self.request_payment_config = Some(c))
@@ -1423,7 +1413,6 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.public_access_block_config = None;
if !self.public_access_block_config_xml.is_empty()
&& let Err(e) = deserialize::<PublicAccessBlockConfiguration>(&self.public_access_block_config_xml)
.map(|c| self.public_access_block_config = Some(c))
@@ -1984,156 +1973,6 @@ mod test {
assert_eq!(bm.lifecycle_config_xml, malformed_xml, "raw bytes are retained");
}
const REPARSE_XML_FIXTURES: [(&str, &[u8]); 13] = [
(BUCKET_NOTIFICATION_CONFIG, br#"<NotificationConfiguration/>"#),
(
BUCKET_LIFECYCLE_CONFIG,
br#"<LifecycleConfiguration><Rule><ID>keep</ID><Status>Enabled</Status><Expiration><Days>30</Days></Expiration></Rule></LifecycleConfiguration>"#,
),
(
OBJECT_LOCK_CONFIG,
br#"<ObjectLockConfiguration><ObjectLockEnabled>Enabled</ObjectLockEnabled><Rule><DefaultRetention><Mode>GOVERNANCE</Mode><Days>7</Days></DefaultRetention></Rule></ObjectLockConfiguration>"#,
),
(BUCKET_VERSIONING_CONFIG, br#"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>"#),
(
BUCKET_SSECONFIG,
br#"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"#,
),
(BUCKET_TAGGING_CONFIG, br#"<Tagging><TagSet><Tag><Key>team</Key><Value>blue</Value></Tag></TagSet></Tagging>"#),
(
BUCKET_REPLICATION_CONFIG,
br#"<ReplicationConfiguration><Role>arn:aws:iam::123456789012:role/replication-role</Role><Rule><ID>copy</ID><Status>Enabled</Status><Prefix>documents/</Prefix><Destination><Bucket>arn:aws:s3:::destination-bucket</Bucket></Destination></Rule></ReplicationConfiguration>"#,
),
(
BUCKET_CORS_CONFIG,
br#"<CORSConfiguration><CORSRule><AllowedOrigin>https://example.test</AllowedOrigin><AllowedMethod>GET</AllowedMethod></CORSRule></CORSConfiguration>"#,
),
(BUCKET_LOGGING_CONFIG, br#"<BucketLoggingStatus/>"#),
(BUCKET_WEBSITE_CONFIG, br#"<WebsiteConfiguration><IndexDocument><Suffix>index.html</Suffix></IndexDocument></WebsiteConfiguration>"#),
(BUCKET_ACCELERATE_CONFIG, br#"<AccelerateConfiguration><Status>Enabled</Status></AccelerateConfiguration>"#),
(BUCKET_REQUEST_PAYMENT_CONFIG, br#"<RequestPaymentConfiguration><Payer>BucketOwner</Payer></RequestPaymentConfiguration>"#),
(
BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG,
br#"<PublicAccessBlockConfiguration><BlockPublicAcls>true</BlockPublicAcls><IgnorePublicAcls>true</IgnorePublicAcls><BlockPublicPolicy>true</BlockPublicPolicy><RestrictPublicBuckets>true</RestrictPublicBuckets></PublicAccessBlockConfiguration>"#,
),
];
fn reparse_xml_state(bm: &BucketMetadata, config: &str) -> &'static str {
fn state<T>(raw: &[u8], parsed: &Option<T>) -> &'static str {
match ConfigState::of(raw, parsed) {
ConfigState::Absent => "absent",
ConfigState::Valid(_) => "valid",
ConfigState::Unreadable { .. } => "unreadable",
}
}
match config {
BUCKET_NOTIFICATION_CONFIG => state(&bm.notification_config_xml, &bm.notification_config),
BUCKET_LIFECYCLE_CONFIG => state(&bm.lifecycle_config_xml, &bm.lifecycle_config),
OBJECT_LOCK_CONFIG => state(&bm.object_lock_config_xml, &bm.object_lock_config),
BUCKET_VERSIONING_CONFIG => state(&bm.versioning_config_xml, &bm.versioning_config),
BUCKET_SSECONFIG => state(&bm.encryption_config_xml, &bm.sse_config),
BUCKET_TAGGING_CONFIG => state(&bm.tagging_config_xml, &bm.tagging_config),
BUCKET_REPLICATION_CONFIG => state(&bm.replication_config_xml, &bm.replication_config),
BUCKET_CORS_CONFIG => state(&bm.cors_config_xml, &bm.cors_config),
BUCKET_LOGGING_CONFIG => state(&bm.logging_config_xml, &bm.logging_config),
BUCKET_WEBSITE_CONFIG => state(&bm.website_config_xml, &bm.website_config),
BUCKET_ACCELERATE_CONFIG => state(&bm.accelerate_config_xml, &bm.accelerate_config),
BUCKET_REQUEST_PAYMENT_CONFIG => state(&bm.request_payment_config_xml, &bm.request_payment_config),
BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG => state(&bm.public_access_block_config_xml, &bm.public_access_block_config),
_ => panic!("unknown XML config: {config}"),
}
}
// A persisted payload can change without going through the request decoder.
fn replace_xml_raw(bm: &mut BucketMetadata, config: &str, raw: Vec<u8>) {
match config {
BUCKET_NOTIFICATION_CONFIG => bm.notification_config_xml = raw,
BUCKET_LIFECYCLE_CONFIG => bm.lifecycle_config_xml = raw,
OBJECT_LOCK_CONFIG => bm.object_lock_config_xml = raw,
BUCKET_VERSIONING_CONFIG => bm.versioning_config_xml = raw,
BUCKET_SSECONFIG => bm.encryption_config_xml = raw,
BUCKET_TAGGING_CONFIG => bm.tagging_config_xml = raw,
BUCKET_REPLICATION_CONFIG => bm.replication_config_xml = raw,
BUCKET_CORS_CONFIG => bm.cors_config_xml = raw,
BUCKET_LOGGING_CONFIG => bm.logging_config_xml = raw,
BUCKET_WEBSITE_CONFIG => bm.website_config_xml = raw,
BUCKET_ACCELERATE_CONFIG => bm.accelerate_config_xml = raw,
BUCKET_REQUEST_PAYMENT_CONFIG => bm.request_payment_config_xml = raw,
BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG => bm.public_access_block_config_xml = raw,
_ => panic!("unknown XML config: {config}"),
}
}
#[test]
fn reparse_xml_rejects_stale_views_after_corruption() {
let corrupt = b"<not-a-valid-document";
for (config, valid) in REPARSE_XML_FIXTURES {
let mut bm = BucketMetadata::new("reparse-corrupt");
bm.update_config(config, valid.to_vec()).expect("store valid XML");
bm.parse_all_configs().expect("parse valid XML");
assert_eq!(reparse_xml_state(&bm, config), "valid", "{config}: valid setup");
replace_xml_raw(&mut bm, config, corrupt.to_vec());
let before = bm.marshal_msg().expect("marshal corrupt raw metadata");
bm.parse_all_configs()
.expect("one corrupt config must not fail the metadata load");
assert_eq!(reparse_xml_state(&bm, config), "unreadable", "{config}: stale view accepted");
assert_eq!(bm.xml_config_unreadable_len(config), Some(corrupt.len()), "{config}: missing refusal");
assert_eq!(
bm.marshal_msg().expect("marshal reparsed metadata"),
before,
"{config}: persisted bytes changed"
);
}
}
#[test]
fn reparse_xml_clears_stale_views_after_raw_is_removed() {
for (config, valid) in REPARSE_XML_FIXTURES {
let mut bm = BucketMetadata::new("reparse-removed");
bm.update_config(config, valid.to_vec()).expect("store valid XML");
bm.parse_all_configs().expect("parse valid XML");
assert_eq!(reparse_xml_state(&bm, config), "valid", "{config}: valid setup");
replace_xml_raw(&mut bm, config, Vec::new());
let before = bm.marshal_msg().expect("marshal absent raw metadata");
bm.parse_all_configs().expect("parse removed XML");
assert_eq!(reparse_xml_state(&bm, config), "absent", "{config}: stale view survived removal");
assert!(!bm.xml_config_unreadable(config), "{config}: absence is not corruption");
assert_eq!(
bm.marshal_msg().expect("marshal reparsed metadata"),
before,
"{config}: persisted bytes changed"
);
}
}
#[test]
fn reparse_xml_recovers_after_corrupt_raw_is_repaired() {
for (config, valid) in REPARSE_XML_FIXTURES {
let mut bm = BucketMetadata::new("reparse-repaired");
bm.update_config(config, valid.to_vec()).expect("store valid XML");
bm.parse_all_configs().expect("parse valid XML");
replace_xml_raw(&mut bm, config, b"<not-a-valid-document".to_vec());
bm.parse_all_configs().expect("parse corrupt XML");
assert_eq!(reparse_xml_state(&bm, config), "unreadable", "{config}: corrupt setup");
replace_xml_raw(&mut bm, config, valid.to_vec());
let before = bm.marshal_msg().expect("marshal repaired raw metadata");
bm.parse_all_configs().expect("parse repaired XML");
assert_eq!(reparse_xml_state(&bm, config), "valid", "{config}: repaired config must become readable");
assert!(!bm.xml_config_unreadable(config), "{config}: repaired config is not corrupt");
assert_eq!(
bm.marshal_msg().expect("marshal reparsed metadata"),
before,
"{config}: persisted bytes changed"
);
}
}
#[test]
fn lifecycle_update_config_clears_parsed_config_on_delete() {
let mut bm = BucketMetadata::new("test-bucket");
+26 -690
View File
@@ -178,13 +178,6 @@ pub(crate) fn object_lock_config_state_from_authoritative_metadata(bm: &BucketMe
Ok(ObjectLockConfigState::ConfirmedAbsent)
}
#[derive(Debug, Clone, PartialEq)]
pub(crate) struct CachedQuotaMetadataSnapshot {
pub(crate) quota: Option<BucketQuota>,
pub(crate) bucket_incarnation: Uuid,
pub(crate) quota_revision: OffsetDateTime,
}
/// Convert the persisted serving-layer configuration into the storage-level
/// [`DefaultRetention`](crate::bucket::object_lock::types::DefaultRetention)
/// the WORM evaluation code consumes (rustfs/backlog#1842). A rule without a
@@ -321,33 +314,6 @@ pub async fn remove_bucket_metadata(bucket: &str) -> Result<bool> {
Ok(lock.remove(bucket).await)
}
/// Remove a peer's cached metadata only when the bucket is absent under the
/// namespace read fence. A delayed delete notification must not withdraw the
/// metadata of a same-name bucket that has already been recreated.
pub async fn remove_bucket_metadata_if_absent(api: Arc<ECStore>, bucket: &str) -> Result<bool> {
if is_meta_bucketname(bucket) {
return Ok(false);
}
// Lock order: bucket namespace read -> metadata system read (released)
// -> metadata publish, matching peer reloads.
let namespace_lock = api.new_ns_lock(bucket, bucket).await?;
let namespace_guard = namespace_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
let sys = bucket_metadata_sys_of(&api.ctx)?;
let sys = sys.read().await.clone();
if sys
.bucket_exists(bucket, &namespace_guard, "peer bucket metadata removal existence check")
.await?
{
return Ok(false);
}
await_bucket_namespace_operation(Some(&namespace_guard), bucket, "peer bucket metadata removal", async {
Ok(sys.remove(bucket).await)
})
.await
}
fn start_refresh_buckets_metadata_loop(sys: Arc<RwLock<BucketMetadataSys>>) {
let Some(cancel_token) = runtime_sources::background_services_cancel_token() else {
warn!("bucket metadata refresh loop skipped because background cancellation token is not initialized");
@@ -525,36 +491,6 @@ pub(crate) async fn get_bucket_incarnation_id_in(ctx: &crate::runtime::instance:
sys.get_bucket_incarnation_id_from_disk(bucket).await
}
/// Directory loss is recoverable only for a committed, non-retired generation.
/// The caller retains the bucket lifecycle fence through the ensuing reconstruction.
pub(crate) async fn validate_bucket_reconstruction_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
expected: Uuid,
) -> Result<()> {
let api = bucket_metadata_sys_of(ctx)?.read().await.object_store();
let lock = api
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
let (metadata, persisted) = load_bucket_metadata_parse_with_presence(api.clone(), bucket, true).await?;
if !persisted
|| !metadata.bucket_incarnation_sidecar
|| expected.is_nil()
|| metadata.bucket_incarnation_id != expected
|| !metadata.bucket_creation_committed
{
return Err(Error::PreconditionFailed);
}
if crate::bucket::retirement::is_retired(api, bucket, expected).await? {
return Err(Error::BucketNotFound(bucket.to_owned()));
}
if guard.is_lock_lost() {
return Err(Error::PreconditionFailed);
}
Ok(())
}
/// Validate against disk while retaining an already-held Object Lock fence.
pub(crate) async fn get_bucket_incarnation_id_for_options_in(
ctx: &crate::runtime::instance::InstanceContext,
@@ -640,27 +576,6 @@ pub(crate) async fn update_in(
Box::pin(update_with_sys(require_bucket_metadata_sys_in(ctx)?, bucket, config_file, data)).await
}
/// Validate a caller's fence after acquiring the lifecycle and metadata
/// transaction locks, before changing the configuration.
pub(crate) async fn update_validated_in<F>(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
config_file: &str,
data: Vec<u8>,
validate: F,
) -> Result<OffsetDateTime>
where
F: FnOnce() -> Result<()> + Send,
{
Box::pin(async {
let sys = require_bucket_metadata_sys_in(ctx)?;
let guard = acquire_config_write_guard(sys.clone(), bucket).await?;
validate()?;
update_under_config_write_guard(sys, &guard, config_file, data, None).await
})
.await
}
pub async fn delete(bucket: &str, config_file: &str) -> Result<OffsetDateTime> {
delete_with_sys(get_bucket_metadata_sys()?, bucket, config_file).await
}
@@ -1217,7 +1132,7 @@ async fn acquire_transaction_lock_with_sys(
/// agree on, so renaming it would leave a mixed-version cluster with two
/// disjoint keys — and old and new nodes would stop excluding each other on
/// the very writes that are serialized today.
pub(crate) fn bucket_metadata_transaction_lock_key(bucket: &str) -> String {
fn bucket_metadata_transaction_lock_key(bucket: &str) -> String {
format!("bucket-targets/{bucket}/transaction.lock")
}
@@ -1356,34 +1271,14 @@ pub(crate) async fn get_object_lock_config_state_in(
pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> {
get_object_lock_config_and_incarnation_from_disk_with_guard_in(ctx, bucket, None).await
}
/// Bucket deletion already owns the namespace/lifecycle fences. Reuse its
/// live guard rather than recursively acquiring the bucket namespace lock.
/// The caller must also hold the metadata transaction read lock.
pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_with_guard_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
guard: Option<&rustfs_lock::NamespaceLockGuard>,
) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> {
let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
let authority = match guard {
Some(guard) => {
bucket_meta_sys
.read_authoritative_metadata_from_disk_under_guard(bucket, guard)
.await?
}
None => {
bucket_meta_sys
.read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
.await?
}
};
match authority {
match bucket_meta_sys
.read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
.await?
{
BucketMetadataAuthority::Authoritative(metadata)
if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() =>
{
@@ -1436,60 +1331,13 @@ pub(crate) async fn get_quota_config_and_incarnation_from_disk_in(
let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
quota_config_and_incarnation_from_authority(
bucket,
bucket_meta_sys
.read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
.await?,
)
}
pub(crate) async fn get_quota_config_and_incarnation_from_disk_for_options_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
opts: &ObjectOptions,
transaction_guard: &Arc<rustfs_lock::NamespaceLockGuard>,
) -> Result<(Option<BucketQuota>, Uuid, OffsetDateTime)> {
let Some(snapshot) = opts.object_lock_config_snapshot.as_ref() else {
return get_quota_config_and_incarnation_from_disk_in(ctx, bucket).await;
};
let sys = bucket_metadata_sys_of(ctx)?.read().await.clone();
let store = sys.object_store();
let snapshot_guard = snapshot
.metadata_transaction_guard_for(store.id, bucket, opts.expected_bucket_incarnation_id)
.filter(|guard| Arc::ptr_eq(guard, transaction_guard))
.ok_or_else(|| Error::other("quota snapshot does not hold this bucket's metadata transaction fence"))?;
if snapshot_guard.is_released() {
return Err(Error::other("quota snapshot metadata transaction fence was released"));
}
// The snapshot carries the caller's lifecycle fence, excluding deletion.
// Its transaction guard excludes config writes, so a same-name object
// owner need not recursively acquire the bucket namespace here.
let authority = sys
.read_authoritative_metadata_from_disk_under_guard(bucket, transaction_guard)
.await?;
if snapshot_guard.is_lock_lost()
|| snapshot_guard.is_released()
|| matches!(&authority, BucketMetadataAuthority::Authoritative(metadata)
if !snapshot.is_valid_for_destructive_put(store.id, bucket, metadata.bucket_incarnation_id))
match bucket_meta_sys
.read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
.await?
{
return Err(Error::other("quota snapshot fence or bucket incarnation changed during the disk read"));
}
quota_config_and_incarnation_from_authority(bucket, authority)
}
fn quota_config_and_incarnation_from_authority(
bucket: &str,
authority: BucketMetadataAuthority,
) -> Result<(Option<BucketQuota>, Uuid, OffsetDateTime)> {
match authority {
BucketMetadataAuthority::Authoritative(metadata)
if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() =>
{
if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() {
return Err(Error::other("persisted bucket quota configuration is invalid"));
}
Ok((
metadata.quota_config.clone(),
metadata.bucket_incarnation_id,
@@ -1504,15 +1352,6 @@ fn quota_config_and_incarnation_from_authority(
}
}
pub(crate) async fn get_cached_quota_config_and_incarnation_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
) -> Result<Option<CachedQuotaMetadataSnapshot>> {
let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
bucket_meta_sys.cached_quota_config_and_incarnation(bucket).await
}
pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -2864,30 +2703,29 @@ impl BucketMetadataSys {
async fn read_authoritative_metadata_from_disk_under_transaction_lock(
&self,
bucket: &str,
) -> Result<BucketMetadataAuthority> {
let namespace_lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let namespace_guard = namespace_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
self.read_authoritative_metadata_from_disk_under_guard(bucket, &namespace_guard)
.await
}
async fn read_authoritative_metadata_from_disk_under_guard(
&self,
bucket: &str,
guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<BucketMetadataAuthority> {
#[cfg(test)]
if self.object_lock_disk_read_errors.write().await.remove(bucket) {
return Err(Error::other(format!("injected Object Lock metadata disk read failure: {bucket}")));
}
match await_bucket_namespace_operation(Some(guard), bucket, "bucket metadata snapshot existence check", async {
self.object_store()
.get_bucket_info_from_sets_at_read_quorum(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
.await
})
let namespace_lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let namespace_guard = namespace_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
match await_bucket_namespace_operation(
Some(&namespace_guard),
bucket,
"bucket metadata snapshot existence check",
async {
self.object_store()
.get_bucket_info_from_sets_at_read_quorum(
bucket,
&crate::storage_api_contracts::bucket::BucketOptions::default(),
)
.await
},
)
.await
{
Ok(_) => {}
@@ -2896,7 +2734,7 @@ impl BucketMetadataSys {
}
let (metadata, persisted) = await_bucket_namespace_operation(
Some(guard),
Some(&namespace_guard),
bucket,
"bucket metadata authoritative snapshot",
load_bucket_metadata_parse_with_presence(self.object_store(), bucket, true),
@@ -3028,27 +2866,6 @@ impl BucketMetadataSys {
}
}
async fn cached_quota_config_and_incarnation(&self, bucket: &str) -> Result<Option<CachedQuotaMetadataSnapshot>> {
let metadata = {
let metadata_map = self.metadata_map.read().await;
metadata_map.get(bucket).cloned()
};
let Some(metadata) = metadata else {
return Ok(None);
};
if !metadata.bucket_incarnation_sidecar || metadata.bucket_incarnation_id.is_nil() {
return Ok(None);
}
if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() {
return Err(Error::other("persisted bucket quota configuration is invalid"));
}
Ok(Some(CachedQuotaMetadataSnapshot {
quota: metadata.quota_config.clone(),
bucket_incarnation: metadata.bucket_incarnation_id,
quota_revision: metadata.quota_config_updated_at,
}))
}
pub async fn get_replication_config(&self, bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
let (bm, _) = self.get_config(bucket).await?;
@@ -4304,114 +4121,6 @@ mod tests {
sys.persist_new_and_set(bm).await.expect("initial metadata should persist");
}
#[tokio::test]
async fn reparse_xml_rmw_corruption_refuses_cached_and_reloaded_encryption() {
use crate::bucket::metadata::{BUCKET_SSECONFIG, is_unreadable_config_error};
use std::sync::atomic::{AtomicUsize, Ordering};
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = BucketMetadataSys::new(ecstore);
let bucket = "reparse-encryption-corrupt";
let valid = br#"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"#;
let corrupt = b"<ServerSideEncryptionConfiguration";
persist_bucket_with_raw_config(&sys, &dirs, bucket, BUCKET_SSECONFIG, valid).await;
sys.get_sse_config(bucket).await.expect("valid encryption must be readable");
let updated = sys
.update_config_with(bucket, BUCKET_SSECONFIG, |bm| {
assert!(bm.sse_config.is_some(), "rewrite must start with parsed encryption");
Ok(corrupt.to_vec())
})
.await
.expect("persist future or corrupt raw encryption without failing the whole bucket");
let err = sys
.get_sse_config(bucket)
.await
.expect_err("cached old encryption must not hide corrupt raw");
assert!(is_unreadable_config_error(&err), "cached refusal must retain its type: {err}");
let (cached, _) = sys.get_config(bucket).await.expect("bucket remains readable");
assert_eq!(cached.encryption_config_xml, corrupt);
assert_eq!(cached.encryption_config_updated_at, updated);
sys.metadata_map.write().await.clear();
let err = sys
.get_sse_config(bucket)
.await
.expect_err("reloaded encryption must still refuse corrupt raw");
assert!(is_unreadable_config_error(&err), "reloaded refusal must retain its type: {err}");
let calls = AtomicUsize::new(0);
let err = sys
.update_config_with(bucket, BUCKET_SSECONFIG, |_| {
calls.fetch_add(1, Ordering::SeqCst);
Ok(Vec::new())
})
.await
.expect_err("another rewrite must refuse before it consumes corrupt encryption");
assert!(is_unreadable_config_error(&err));
assert_eq!(calls.load(Ordering::SeqCst), 0, "refused rewrite must not erase the original raw");
let disk = sys
.get_config_from_disk(bucket)
.await
.expect("read unchanged metadata from disk");
assert_eq!(disk.encryption_config_xml, corrupt);
assert_eq!(disk.encryption_config_updated_at, updated);
sys.update(bucket, BUCKET_SSECONFIG, valid.to_vec())
.await
.expect("explicit repair must remain possible");
sys.get_sse_config(bucket)
.await
.expect("repaired cached encryption must be readable");
sys.metadata_map.write().await.clear();
sys.get_sse_config(bucket).await.expect("repair must survive a disk reload");
}
#[tokio::test]
async fn reparse_xml_rmw_removal_clears_cached_and_reloaded_encryption() {
use crate::bucket::metadata::BUCKET_SSECONFIG;
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = BucketMetadataSys::new(ecstore);
let bucket = "reparse-encryption-removed";
let valid = br#"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"#;
persist_bucket_with_raw_config(&sys, &dirs, bucket, BUCKET_SSECONFIG, valid).await;
sys.get_sse_config(bucket).await.expect("valid encryption must be readable");
let updated = sys
.update_config_with(bucket, BUCKET_SSECONFIG, |bm| {
assert!(bm.sse_config.is_some(), "rewrite must start with parsed encryption");
Ok(Vec::new())
})
.await
.expect("removing encryption must persist");
assert_eq!(
sys.get_sse_config(bucket)
.await
.expect_err("old cached encryption must be gone"),
Error::ConfigNotFound
);
let (cached, _) = sys.get_config(bucket).await.expect("bucket remains readable");
assert!(cached.encryption_config_xml.is_empty());
assert_eq!(cached.encryption_config_updated_at, updated);
sys.metadata_map.write().await.clear();
assert_eq!(
sys.get_sse_config(bucket)
.await
.expect_err("removed encryption must stay absent after reload"),
Error::ConfigNotFound
);
let disk = sys
.get_config_from_disk(bucket)
.await
.expect("read removed encryption from disk");
assert!(disk.encryption_config_xml.is_empty());
assert_eq!(disk.encryption_config_updated_at, updated);
}
/// rustfs/backlog#1734: a read-modify-write of a stored config that cannot
/// be parsed must be refused before `mutate` runs. Otherwise `mutate` sees
/// the unreadable config as absent, rebuilds it from nothing, and the
@@ -4953,154 +4662,6 @@ mod tests {
);
}
#[tokio::test]
#[serial]
async fn quota_admission_rejects_stale_cached_no_quota() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("scoped metadata system");
let bucket = "stale-no-quota-admission";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let (stale, _) = sys.read().await.get_config(bucket).await.expect("cache initial metadata");
store
.update_bucket_metadata_config(bucket, "quota.json", br#"{"quota":1}"#.to_vec())
.await
.expect("publish quota to authoritative storage");
// Model a peer that has not received the newer configuration snapshot.
sys.read().await.set(bucket.to_owned(), stale).await;
assert!(
get_cached_quota_config_and_incarnation_in(&store.ctx, bucket)
.await
.expect("read stale cache")
.expect("cached bucket identity")
.quota
.is_none()
);
temp_env::async_with_vars([("RUSTFS_QUOTA_BEGIN_SAFE_NO_QUOTA_CACHE_FAST_PATH", Some("true"))], async {
let error = crate::bucket::quota::reservation::begin(&store.ctx, bucket, "object", &ObjectOptions::default(), 0, 0)
.await
.err()
.expect("persisted quota must reject admission without its snapshot");
assert!(matches!(error, Error::PartMissingOrCorrupt), "{error:?}");
})
.await;
}
#[tokio::test]
#[serial]
async fn peer_reload_clears_negative_cache_after_remote_bucket_creation() {
let (dirs, node_a, node_b) = two_nodes_over_one_store().await;
let bucket = "cross-node-negative-cache-refresh";
let node_b_sys = node_b.read().await.clone();
assert_eq!(
node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap_err(),
Error::BucketNotFound(bucket.to_string()),
"the probe must populate node B's missing-bucket cache"
);
for dir in &dirs {
std::fs::create_dir_all(dir.path().join(bucket)).expect("physical bucket should exist before creation metadata");
}
let created = BucketMetadata::new(bucket);
let created_incarnation = created.bucket_incarnation_id;
node_a
.read()
.await
.persist_new_and_set(created)
.await
.expect("node A should publish the created bucket metadata");
node_b_sys
.reload_from_store(bucket)
.await
.expect("peer reload should clear the stale negative cache");
assert_eq!(
node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap(),
created_incarnation,
"node B should resolve the newly created bucket immediately"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn peer_delete_clears_metadata_for_absent_bucket() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
let bucket = "peer-delete-absent-bucket";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("initial bucket creation");
let cached = sys.read().await.get(bucket).await.expect("initial cached metadata");
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("bucket deletion");
// Model a peer that still caches the metadata after the physical delete.
sys.read().await.set(bucket.to_string(), cached).await;
assert!(
remove_bucket_metadata_if_absent(store.clone(), bucket)
.await
.expect("conditional peer delete"),
"the peer must evict the deleted bucket's cached metadata"
);
let sys = sys.read().await;
assert!(matches!(sys.get(bucket).await, Err(Error::ConfigNotFound)));
assert_eq!(
sys.get_bucket_incarnation_id(bucket)
.await
.expect_err("deleted bucket lookup"),
Error::BucketNotFound(bucket.to_string()),
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn delayed_peer_delete_preserves_recreated_bucket_metadata() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
let bucket = "delayed-peer-delete-recreated-bucket";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("initial bucket creation");
let old_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("old incarnation");
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("bucket deletion");
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket recreation");
let new_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("new incarnation");
assert_ne!(old_incarnation, new_incarnation);
let removed = remove_bucket_metadata_if_absent(store.clone(), bucket)
.await
.expect("conditional peer delete");
assert!(!removed, "a delayed delete must not remove metadata for a live recreated bucket");
assert_eq!(
sys.read()
.await
.get(bucket)
.await
.expect("recreated metadata should remain cached")
.bucket_incarnation_id,
new_incarnation,
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn stale_config_request_cannot_mutate_a_recreated_bucket() {
@@ -5175,231 +4736,6 @@ mod tests {
.expect("DeleteBucket should succeed");
}
#[tokio::test]
#[serial]
async fn quota_snapshot_reuses_same_name_owner_and_rereads_disk_authority() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "quota-same-name-owner";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("quota fixture bucket");
let snapshot = store
.object_lock_config_snapshot(bucket)
.await
.expect("guarded Object Lock snapshot");
let incarnation = store.bucket_incarnation_id(bucket).await.expect("fixture incarnation");
let transaction_guard = snapshot
.metadata_transaction_guard_for(store.id, bucket, Some(incarnation))
.expect("snapshot's actual transaction guard");
let expected = get_quota_config_and_incarnation_from_disk_for_options_in(
&store.ctx,
bucket,
&ObjectOptions::default(),
&transaction_guard,
)
.await
.expect("a reader without a snapshot must retain the ordinary namespace path");
let sys = bucket_metadata_sys_of(&store.ctx).expect("fixture metadata system");
let sys = sys.read().await.clone();
let mut stale = (*sys.get(bucket).await.expect("cached fixture metadata")).clone();
let metadata_path = stale.save_file_path();
stale.quota_config = Some(BucketQuota {
quota: Some(123),
..Default::default()
});
sys.set(bucket.to_string(), Arc::new(stale)).await;
let namespace = store.new_ns_lock(bucket, bucket).await.expect("same-name object namespace");
let _owner = namespace
.get_write_lock(crate::set_disk::get_lock_acquire_timeout())
.await
.expect("same-name object write owner");
let opts = ObjectOptions {
object_lock_config_snapshot: Some(snapshot),
expected_bucket_incarnation_id: Some(incarnation),
..Default::default()
};
assert_eq!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
.await
.expect("quota disk read must reuse the held lifecycle and transaction fences"),
expected,
"the cached quota must not replace disk authority"
);
crate::config::com::save_config_with_opts(
store.clone(),
&metadata_path,
b"corrupt bucket metadata".to_vec(),
&ObjectOptions {
write_completion: crate::object_api::WriteCompletion::TailDrained,
..Default::default()
},
)
.await
.expect("inject committed metadata corruption");
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
.await
.is_err(),
"a valid snapshot must not hide corrupt disk metadata"
);
crate::config::com::delete_config(store.clone(), &metadata_path)
.await
.expect("remove the persisted metadata while retaining its sidecar");
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
.await
.is_err(),
"a valid snapshot must not promote fabricated metadata to authority"
);
}
#[tokio::test]
#[serial]
async fn quota_snapshot_rejects_wrong_scope_incarnation_and_transaction_guard() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "quota-snapshot-scope";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("quota fixture bucket");
let incarnation = store.bucket_incarnation_id(bucket).await.expect("fixture incarnation");
let snapshot = store
.object_lock_config_snapshot(bucket)
.await
.expect("guarded Object Lock snapshot");
let transaction_guard = snapshot
.metadata_transaction_guard_for(store.id, bucket, Some(incarnation))
.expect("snapshot's actual transaction guard");
let mut opts = ObjectOptions {
object_lock_config_snapshot: Some(snapshot),
expected_bucket_incarnation_id: Some(incarnation),
..Default::default()
};
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, "another-bucket", &opts, &transaction_guard)
.await
.is_err()
);
opts.expected_bucket_incarnation_id = Some(Uuid::new_v4());
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
.await
.is_err()
);
opts.expected_bucket_incarnation_id = Some(incarnation);
let other_guard = Arc::new(
acquire_bucket_metadata_transaction_read_lock_in(&store.ctx, bucket)
.await
.expect("distinct real transaction read guard"),
);
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &other_guard)
.await
.is_err(),
"the same namespace key does not prove that this is the snapshot's actual guard"
);
let (_other_dirs, other_store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(other_store.clone(), Vec::new()).await;
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&other_store.ctx, bucket, &opts, &transaction_guard)
.await
.is_err()
);
opts.object_lock_config_snapshot = Some(Arc::new(crate::object_api::ObjectLockConfigSnapshot::new(
ObjectLockConfigState::ConfirmedAbsent,
)));
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
.await
.is_err(),
"an unguarded snapshot must not authorize namespace reuse"
);
}
#[tokio::test]
#[serial]
async fn quota_snapshot_rejects_lifecycle_loss_during_read_and_changed_disk_incarnation() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "quota-snapshot-loss";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("quota fixture bucket");
let sys = bucket_metadata_sys_of(&store.ctx).expect("fixture metadata system");
let sys = sys.read().await.clone();
let mut metadata = (*sys.get(bucket).await.expect("fixture metadata")).clone();
let incarnation = metadata.bucket_incarnation_id;
let _lifecycle_guard = store
.acquire_bucket_lifecycle_read_lock(bucket)
.await
.expect("real lifecycle owner");
let transaction_guard = Arc::new(
acquire_bucket_metadata_transaction_read_lock_in(&store.ctx, bucket)
.await
.expect("real metadata transaction owner"),
);
let (lifecycle_fence, lost) = crate::object_api::NamespaceLockFence::loss_handle_for_test();
let snapshot = crate::object_api::ObjectLockConfigSnapshot::for_store_bucket_under_lifecycle_fence(
store.id,
bucket,
incarnation,
metadata.object_lock_config_updated_at,
ObjectLockConfigState::ConfirmedAbsent,
lifecycle_fence,
Arc::clone(&transaction_guard),
);
let opts = ObjectOptions {
object_lock_config_snapshot: Some(Arc::new(snapshot)),
expected_bucket_incarnation_id: Some(incarnation),
..Default::default()
};
let mut read = Box::pin(get_quota_config_and_incarnation_from_disk_for_options_in(
&store.ctx,
bucket,
&opts,
&transaction_guard,
));
assert!(
futures::poll!(read.as_mut()).is_pending(),
"the authoritative disk read must be in flight"
);
lost.store(true, std::sync::atomic::Ordering::Release);
assert!(read.await.is_err(), "a lifecycle fence lost during the disk read must reject its result");
let snapshot = store
.object_lock_config_snapshot(bucket)
.await
.expect("fresh guarded snapshot");
let transaction_guard = snapshot
.metadata_transaction_guard_for(store.id, bucket, Some(incarnation))
.expect("snapshot's actual transaction guard");
let opts = ObjectOptions {
object_lock_config_snapshot: Some(snapshot),
expected_bucket_incarnation_id: Some(incarnation),
..Default::default()
};
metadata.bucket_incarnation_id = Uuid::new_v4();
save_bucket_incarnation(store.clone(), bucket, metadata.bucket_incarnation_id)
.await
.expect("inject a new disk incarnation");
metadata
.save_with_store_committed(store.clone())
.await
.expect("persist its matching metadata blob");
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
.await
.is_err(),
"a matching sidecar and blob from another incarnation must not reuse the old snapshot"
);
}
/// Writers on different nodes updating *different* config files of one
/// bucket must both survive. Each rewrites the whole metadata blob, so
/// without a lock spanning the read-modify-write the later save carries
-24
View File
@@ -273,30 +273,6 @@ where
let buckets: Vec<String> = buckets_list.into_iter().map(|b| b.name).collect();
try_migrate_bucket_metadata_for_buckets(store, &buckets).await
}
/// Migrate the already discovered startup candidates without repeating a
/// majority-only namespace listing. Every migration write retains its quorum.
pub async fn try_migrate_bucket_metadata_for_buckets<S>(store: Arc<S>, buckets: &[String]) -> Result<()>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
> + ObjectOperations<
Error = Error,
ObjectInfo = ObjectInfo,
ObjectOptions = ObjectOptions,
FileInfo = FileInfo,
ObjectToDelete = ObjectToDelete,
DeletedObject = DeletedObject,
>,
{
if buckets.is_empty() {
debug!("No migrating bucket metadata found");
return Ok(());
+6 -40
View File
@@ -32,10 +32,7 @@ pub enum QuotaType {
Hard,
}
pub const QUOTA_RESERVATION_PROTOCOL_V1: u32 = 1;
/// Sharded durable reservations. Older nodes reject this protocol instead of
/// silently falling back to the single-ledger protocol.
pub const QUOTA_RESERVATION_PROTOCOL_V2: u32 = 2;
pub(crate) const QUOTA_RESERVATION_PROTOCOL_V1: u32 = 1;
/// Bucket quota configuration. quota_type defaults to Hard when omitted.
#[derive(Debug, Default, Clone, PartialEq)]
@@ -92,10 +89,7 @@ impl<'de> Deserialize<'de> for BucketQuota {
D: Deserializer<'de>,
{
let wire = BucketQuotaWire::deserialize(deserializer)?;
let quota = if matches!(
wire.reservation_protocol,
Some(QUOTA_RESERVATION_PROTOCOL_V1) | Some(QUOTA_RESERVATION_PROTOCOL_V2)
) {
let quota = if wire.reservation_protocol == Some(QUOTA_RESERVATION_PROTOCOL_V1) {
Some(
wire.reservation_quota
.ok_or_else(|| D::Error::custom("reservation_quota is required for reservation protocol v1"))?,
@@ -135,33 +129,17 @@ impl BucketQuota {
}
}
/// Creates a quota using the sharded durable reservation protocol.
/// Deployment must complete the mixed-version rollout before enabling it;
/// older nodes fail closed when they see protocol v2.
pub fn new_sharded(quota: u64) -> Self {
let mut config = Self::new(Some(quota));
config.reservation_protocol = Some(QUOTA_RESERVATION_PROTOCOL_V2);
config
}
pub fn get_quota_limit(&self) -> Option<u64> {
self.quota
}
pub fn uses_durable_reservations(&self) -> bool {
matches!(
self.reservation_protocol,
Some(QUOTA_RESERVATION_PROTOCOL_V1) | Some(QUOTA_RESERVATION_PROTOCOL_V2)
)
}
pub fn uses_sharded_reservations(&self) -> bool {
self.reservation_protocol == Some(QUOTA_RESERVATION_PROTOCOL_V2)
self.reservation_protocol == Some(QUOTA_RESERVATION_PROTOCOL_V1)
}
pub fn has_unsupported_reservation_protocol(&self) -> bool {
self.reservation_protocol
.is_some_and(|version| !matches!(version, QUOTA_RESERVATION_PROTOCOL_V1 | QUOTA_RESERVATION_PROTOCOL_V2))
.is_some_and(|version| version != QUOTA_RESERVATION_PROTOCOL_V1)
}
pub fn check_operation_allowed(&self, current_usage: u64, operation_size: u64) -> bool {
@@ -344,21 +322,9 @@ mod tests {
}
#[test]
fn sharded_quota_roundtrips_with_authoritative_reservation_quota() {
let quota = BucketQuota::new_sharded(4096);
let json = serde_json::to_value(&quota).expect("sharded quota should serialize");
assert_eq!(json["quota"], 0);
assert_eq!(json["reservation_protocol"], QUOTA_RESERVATION_PROTOCOL_V2);
assert_eq!(json["reservation_quota"], 4096);
let restored: BucketQuota = serde_json::from_value(json).expect("sharded quota should parse");
assert_eq!(restored.quota, Some(4096));
assert!(restored.uses_sharded_reservations());
}
#[test]
fn unknown_reservation_protocol_does_not_activate_any_known_protocol() {
fn unknown_reservation_protocol_does_not_activate_v1() {
let quota: BucketQuota =
serde_json::from_str(r#"{"quota":0,"quota_type":"Hard","reservation_protocol":3,"reservation_quota":2048}"#)
serde_json::from_str(r#"{"quota":0,"quota_type":"Hard","reservation_protocol":2,"reservation_quota":2048}"#)
.expect("future protocol should remain parseable");
assert!(!quota.uses_durable_reservations());
File diff suppressed because it is too large Load Diff
@@ -65,9 +65,9 @@ use super::replication_target_boundary::{
SsecPassthroughGate, TargetClient, VersionIdentityCapability, is_replication_target_offline_error,
replication_action_for_target_head, replication_complete_multipart_options, replication_delete_marker_purge_remove_options,
replication_delete_remove_options, replication_force_delete_remove_options, replication_object_is_ssec_encrypted,
replication_put_object_header_size, replication_put_object_options, replication_source_etag_is_comparable,
replication_target_head_is_newer_null_version, resolve_read_api_version_id, ssec_passthrough_evidence_present,
ssec_passthrough_gate, version_identity_capability_from_put, version_identity_drifted,
replication_put_object_header_size, replication_put_object_options, replication_target_head_is_newer_null_version,
resolve_read_api_version_id, ssec_passthrough_evidence_present, ssec_passthrough_gate, version_identity_capability_from_put,
version_identity_drifted,
};
use super::replication_versioning_boundary::ReplicationVersioningStore;
use super::runtime_boundary as runtime_sources;
@@ -349,22 +349,15 @@ fn replica_etag_verification_enabled() -> bool {
/// target that stores a transformed payload (e.g. undecoded `aws-chunked`
/// frames, #6853) returns the ETag of what it actually wrote. Reporting
/// COMPLETED over such a replica is silent corruption, so a decidable
/// mismatch fails the replication instead. Encrypted sources are exempt: an
/// SSE-C ciphertext passthrough sends ciphertext while the source ETag is the
/// plaintext MD5 (that path has its own HEAD-back audit), and a managed-SSE
/// (SSE-S3 / SSE-KMS) source stores a keyed fingerprint, or a legacy plaintext
/// MD5, that the ETag of the target's re-encrypted replica can neither confirm
/// nor refute. The checksum header on the replication PUT stays the integrity
/// check for those objects.
/// mismatch fails the replication instead. An SSE-C ciphertext passthrough
/// transfer is exempt: the wire bytes are ciphertext while the source ETag is
/// the plaintext MD5, and that path has its own HEAD-back audit.
fn verify_single_part_replica(
object_info: &ObjectInfo,
response: &RemotePutObjectResponse,
ciphertext_passthrough: bool,
) -> std::result::Result<(), std::io::Error> {
if ciphertext_passthrough
|| !replication_source_etag_is_comparable(&object_info.user_defined)
|| !replica_etag_verification_enabled()
{
if ciphertext_passthrough || !replica_etag_verification_enabled() {
return Ok(());
}
if single_part_replica_etag_mismatch(object_info.etag.as_deref(), response.etag.as_deref()) {
@@ -455,26 +448,6 @@ fn is_version_id_format_mismatch(err: &SdkError<HeadObjectError>) -> bool {
is_version_id_mismatch(code, raw_status)
}
/// A version-addressed HEAD rejected because the target cannot address the
/// source version id (AWS S3: bodiless 400). Narrower than
/// [`is_version_id_format_mismatch`]: a bodiless 403 may be a genuine
/// `s3:GetObjectVersion` denial, so it is not accepted where a mutation or a
/// completion verdict follows instead of another non-destructive HEAD.
fn rejects_source_version_id(err: &SdkError<HeadObjectError>) -> bool {
is_version_id_format_mismatch(err) && !has_raw_status(err, 403)
}
/// Whether the target's latest version of `object` is a delete marker. For
/// targets that mint their own version ids, which cannot be asked about the
/// source marker id: S3 answers a versionless HEAD on such a key with 404
/// (405 on some targets) plus `x-amz-delete-marker: true`.
async fn latest_target_version_is_delete_marker(tgt_client: &TargetClient, object: &str) -> bool {
match head_object_for_worker(tgt_client, &tgt_client.bucket, object, None).await {
Ok(head) => head.delete_marker == Some(true),
Err(e) => raw_header_value(e.as_ref(), "x-amz-delete-marker").as_deref() == Some("true"),
}
}
async fn mark_replication_target_offline_if_needed(target_client: &Arc<TargetClient>, err: &(impl Display + ?Sized)) {
if is_replication_target_offline_error(err) {
ReplicationTargetStore::mark_target_offline(target_client).await;
@@ -1873,17 +1846,7 @@ async fn verify_resync_head_result(
// only absence proves completion. All other outcomes remain
// failed (backlog#862 / #799 B13).
let retryable = if roi.version_purge_status.is_empty() {
match classify_delete_marker_head_error(&err) {
// A target that mints its own version ids never holds the
// source marker id; verify the marker on the latest version.
DeleteMarkerHeadOutcome::Ambiguous => {
let mints_own = rejects_source_version_id(&err)
|| ReplicationTargetStore::version_identity_capability(&target_client.arn)
.version_addressing_unreliable();
!(mints_own && latest_target_version_is_delete_marker(target_client, &roi.name).await)
}
DeleteMarkerHeadOutcome::Exists | DeleteMarkerHeadOutcome::Missing => false,
}
matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Ambiguous)
} else {
!matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Missing)
};
@@ -3604,16 +3567,7 @@ async fn replicate_delete_to_target(
}
}
// A target that mints its own version ids never holds the source marker
// id, so a version-addressed pre-check cannot succeed there; the marker
// is created by the versionless DELETE below.
let version_addressable =
!ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable();
if version_addressable
&& !is_version_purge
&& dobj.delete_object.delete_marker
&& dobj.delete_object.delete_marker_version_id.is_some()
{
if !is_version_purge && dobj.delete_object.delete_marker && dobj.delete_object.delete_marker_version_id.is_some() {
match head_object_for_worker(
tgt_client.as_ref(),
&tgt_client.bucket,
@@ -3622,11 +3576,6 @@ async fn replicate_delete_to_target(
)
.await
{
// The verdict is not known yet (e.g. no PUT since restart) and the
// target rejected the source id: learn it as the PUT path does.
Err(e) if rejects_source_version_id(e.as_ref()) => {
ReplicationTargetStore::record_version_identity_capability(&tgt_client.arn, VersionIdentityCapability::MintsOwn);
}
Ok(head) if head.delete_marker == Some(true) => {
// Some targets expose an existing marker as a successful HEAD
// with the marker bit instead of the usual 405. Treat that as
@@ -5877,57 +5826,6 @@ mod tests {
use super::super::replication_filemeta_boundary::ReplicateTargetDecision;
use super::super::replication_object_decision_boundary::ReplicationMultipartPlanError;
#[test]
fn verify_single_part_replica_exempts_managed_sse_sources() {
// A new encrypted write stores a keyed fingerprint; a target that
// computes its own ETag answers with the MD5 of the plaintext it
// received. Both are 32 hex characters, so without the exemption the
// audit would fail every SSE-S3 / SSE-KMS replica (rustfs/backlog).
let fingerprint = "00112233445566778899aabbccddeeff";
let replica_md5 = "ffeeddccbbaa99887766554433221100";
let response = RemotePutObjectResponse {
version_id: None,
etag: Some(replica_md5.to_string()),
};
for sse in ["AES256", "aws:kms"] {
let object_info = ObjectInfo {
bucket: "source".to_string(),
name: "encrypted".to_string(),
etag: Some(fingerprint.to_string()),
user_defined: HashMap::from([("x-amz-server-side-encryption".to_string(), sse.to_string())]).into(),
..Default::default()
};
verify_single_part_replica(&object_info, &response, false)
.unwrap_or_else(|err| panic!("{sse}: a managed-SSE source must not be audited against the replica MD5: {err}"));
}
}
#[test]
fn verify_single_part_replica_still_fails_plaintext_mismatch() {
if !replica_etag_verification_enabled() {
return;
}
let object_info = ObjectInfo {
bucket: "source".to_string(),
name: "plain".to_string(),
etag: Some("00112233445566778899aabbccddeeff".to_string()),
..Default::default()
};
let mismatch = RemotePutObjectResponse {
version_id: None,
etag: Some("ffeeddccbbaa99887766554433221100".to_string()),
};
let err = verify_single_part_replica(&object_info, &mismatch, false)
.expect_err("a plaintext source whose replica MD5 differs must fail the replication");
assert_eq!(err.to_string(), REPLICA_ETAG_MISMATCH_ERROR);
let matching = RemotePutObjectResponse {
version_id: None,
etag: object_info.etag.clone(),
};
verify_single_part_replica(&object_info, &matching, false).expect("matching plaintext MD5s pass the audit");
}
#[test]
fn multipart_read_plan_preserves_legacy_plain_part_ranges() {
const MIB: usize = 1024 * 1024;
@@ -6777,93 +6675,6 @@ mod tests {
assert!(seen[0].starts_with("HEAD "), "{seen:?}");
}
/// AWS S3 answers a HEAD for a version id it did not mint with a bodiless
/// 400. On a target with no verdict yet, that teaches `MintsOwn` and the
/// marker is still created with one versionless DELETE.
#[tokio::test]
async fn live_delete_marker_worker_learns_mints_own_from_bodiless_400() {
let (endpoint, server) = spawn_scripted_target_server(2, |line| {
if line.starts_with("HEAD ") {
empty_response("400 Bad Request")
} else if line.starts_with("DELETE ") {
empty_response("204 No Content")
} else {
empty_response("500 Unexpected")
}
});
let target = test_target_client(endpoint);
register_test_target(&target).await;
let dobj = marker_creation_dobj(&target.arn);
let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
assert_eq!(
ReplicationTargetStore::version_identity_capability(&target.arn),
VersionIdentityCapability::MintsOwn
);
let seen = server.join().expect("test HTTP server should finish");
assert_eq!(seen.len(), 2, "{seen:?}");
assert!(seen[0].starts_with("HEAD "), "{seen:?}");
assert!(seen[1].starts_with("DELETE "), "{seen:?}");
assert!(!seen[1].contains("versionId="), "marker creation must remain versionless: {seen:?}");
}
/// A bodiless 403 on the version-addressed HEAD may be a genuine
/// GetObjectVersion denial on an id-adopting target. It must fail closed:
/// no DELETE, and no `MintsOwn` verdict cached for the ARN.
#[tokio::test]
async fn live_delete_marker_worker_keeps_bodiless_403_failure_closed() {
let (endpoint, server) = spawn_scripted_target_server(1, |line| {
if line.starts_with("HEAD ") {
empty_response("403 Forbidden")
} else if line.starts_with("DELETE ") {
empty_response("204 No Content")
} else {
empty_response("500 Unexpected")
}
});
let target = test_target_client(endpoint);
register_test_target(&target).await;
let dobj = marker_creation_dobj(&target.arn);
let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
assert_eq!(rinfo.replication_status, ReplicationStatusType::Failed, "{rinfo:?}");
assert_ne!(
ReplicationTargetStore::version_identity_capability(&target.arn),
VersionIdentityCapability::MintsOwn
);
let seen = server.join().expect("test HTTP server should finish");
assert_eq!(seen.len(), 1, "a denied HEAD must not trigger DELETE: {seen:?}");
assert!(seen[0].starts_with("HEAD "), "{seen:?}");
}
/// A target already known to mint its own version ids is never asked for
/// the source marker id: the marker is created by the versionless DELETE.
#[tokio::test]
async fn live_delete_marker_worker_skips_version_head_on_mints_own_target() {
let (endpoint, server) = spawn_scripted_target_server(1, |line| {
if line.starts_with("DELETE ") {
empty_response("204 No Content")
} else {
empty_response("500 Unexpected")
}
});
let target = test_target_client(endpoint);
register_test_target(&target).await;
ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn);
let dobj = marker_creation_dobj(&target.arn);
let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
let seen = server.join().expect("test HTTP server should finish");
assert_eq!(seen.len(), 1, "no version-addressed HEAD on a MintsOwn target: {seen:?}");
assert!(seen[0].starts_with("DELETE "), "{seen:?}");
assert!(!seen[0].contains("versionId="), "marker creation must remain versionless: {seen:?}");
}
/// A target answers `HEAD ?versionId=<delete marker>` with a bodiless
/// 405, so the SDK reports no error code. That is the marker having
/// propagated, not a failure (rustfs/backlog#2479).
@@ -6954,76 +6765,6 @@ mod tests {
server.join().expect("test HTTP server should finish");
}
/// AWS answers `HEAD ?versionId=<source marker id>` with a bodiless 400.
/// The marker is verified through the latest version instead: a
/// versionless 404 with `x-amz-delete-marker: true` proves it propagated.
#[tokio::test]
async fn resync_verification_accepts_latest_delete_marker_when_target_rejects_source_id() {
let (endpoint, server) = spawn_scripted_target_server(2, |line| {
if line.contains("versionId=") {
empty_response("400 Bad Request")
} else {
"HTTP/1.1 404 Not Found\r\nX-Amz-Delete-Marker: true\r\nX-Amz-Version-Id: target-marker\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
.to_string()
}
});
let target = test_target_client(endpoint);
let roi = delete_marker_roi();
let mut st = TargetReplicationResyncStatus::default();
let head_result =
head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
assert!(err.is_none(), "{err:?}");
assert_eq!((size, st.replicated_count, st.failed_count), (0, 1, 0));
let seen = server.join().expect("test HTTP server should finish");
assert!(!seen[1].contains("versionId="), "the re-check must address the latest version: {seen:?}");
}
/// A live latest version on a target that rejects the source id means the
/// marker has not propagated: the verification stays failed.
#[tokio::test]
async fn resync_verification_fails_delete_marker_when_latest_target_version_is_live() {
let (endpoint, server) = spawn_scripted_target_server(2, |line| {
if line.contains("versionId=") {
empty_response("400 Bad Request")
} else {
empty_response("200 OK")
}
});
let target = test_target_client(endpoint);
let roi = delete_marker_roi();
let mut st = TargetReplicationResyncStatus::default();
let head_result =
head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
assert!(err.is_some(), "a live latest version must not verify the marker");
assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1));
server.join().expect("test HTTP server should finish");
}
/// A bodiless 403 may be a genuine permission denial: it is not evidence
/// that the target mints its own ids, so no latest-version re-check runs.
#[tokio::test]
async fn resync_verification_keeps_delete_marker_failed_on_bodiless_403() {
let (endpoint, server) = spawn_scripted_target_server(1, |_| empty_response("403 Forbidden"));
let target = test_target_client(endpoint);
let roi = delete_marker_roi();
let mut st = TargetReplicationResyncStatus::default();
let head_result =
head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
assert!(err.is_some(), "a 403 must not count as a propagated delete marker");
assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1));
let seen = server.join().expect("test HTTP server should finish");
assert_eq!(seen.len(), 1, "{seen:?}");
}
/// Serves exactly `requests` connections, answering each from
/// `respond(request_line)`, and returns the request lines it saw. Reads
/// the whole request (headers plus `Content-Length` body, honoring
@@ -157,19 +157,6 @@ pub(crate) fn replication_object_is_ssec_encrypted(user_defined: &HashMap<String
rustfs_replication::is_ssec_encrypted(user_defined)
}
/// Whether the stored ETag of a replication source can be compared with the
/// ETag a target returns for the bytes it received. Only an unencrypted source
/// stores the MD5 of the bytes that go on the wire. A managed-SSE (SSE-S3 /
/// SSE-KMS) or SSE-C source stores a keyed fingerprint (new writes) or a legacy
/// plaintext MD5, and the target re-encrypts what it receives, so neither
/// equality nor inequality of the two ETags proves anything about the replica.
pub(crate) fn replication_source_etag_is_comparable(user_defined: &HashMap<String, String>) -> bool {
matches!(
classify_replication_source_encryption(user_defined),
ReplicationSourceEncryption::Plaintext
)
}
/// HeadObjectOutput adapter over the pure SSE-C passthrough evidence
/// judgment owned by `rustfs-replication`: extract the echoed
/// customer-algorithm header and let the crate-owned policy decide.
@@ -591,22 +578,6 @@ mod tests {
use time::Duration;
use uuid::Uuid;
#[test]
fn replication_source_etag_is_comparable_only_for_plaintext_sources() {
assert!(replication_source_etag_is_comparable(&HashMap::new()));
for (key, value) in [
(metadata_keys::SERVER_SIDE_ENCRYPTION, "AES256"),
(metadata_keys::SERVER_SIDE_ENCRYPTION, "aws:kms"),
(SSEC_ALGORITHM_HEADER, "AES256"),
] {
let metadata = HashMap::from([(key.to_string(), value.to_string())]);
assert!(
!replication_source_etag_is_comparable(&metadata),
"{key}={value} must not expose its stored ETag to the replica audit"
);
}
}
/// backlog#1735 A3b SSE key-case finding. Every RustFS writer persists the
/// SSE intent as lowercase `x-amz-server-side-encryption`
/// (`encryption_material_to_metadata`; header-derived keys come from a
@@ -12,11 +12,9 @@
// See the License for the specific language governing permissions and
// limitations under the License.
#[cfg(test)]
use crate::disk::DiskIoClass;
use crate::disk::disk_store::{get_drive_walkdir_peek_timeout, get_drive_walkdir_stall_timeout};
use crate::disk::error::DiskError;
use crate::disk::{self, DiskAPI, DiskStore, WalkDirOptions, current_disk_io_context, with_optional_disk_io_context};
use crate::disk::{self, DiskAPI, DiskStore, WalkDirOptions};
use futures::future::join_all;
use metrics::counter;
use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetacacheReader, is_io_eof};
@@ -247,8 +245,6 @@ pub struct ListPathRawOptions {
pub(crate) test_fallback_reader_behaviors: Vec<TestReaderBehavior>,
#[cfg(test)]
pub(crate) peek_timeout: Option<Duration>,
#[cfg(test)]
pub(crate) test_disk_io_context_observed: Option<Arc<AtomicBool>>,
// pub agreed: Option<Arc<dyn Fn(MetaCacheEntry) + Send + Sync>>,
// pub partial: Option<Arc<dyn Fn(MetaCacheEntries, &[Option<Error>]) + Send + Sync>>,
// pub finished: Option<Arc<dyn Fn(&[Option<Error>]) + Send + Sync>>,
@@ -280,8 +276,6 @@ impl Clone for ListPathRawOptions {
test_fallback_reader_behaviors: self.test_fallback_reader_behaviors.clone(),
#[cfg(test)]
peek_timeout: self.peek_timeout,
#[cfg(test)]
test_disk_io_context_observed: self.test_disk_io_context_observed.clone(),
..Default::default()
}
}
@@ -350,7 +344,6 @@ async fn list_path_raw_inner(
let log_path = opts.path.clone();
let mut jobs = JoinSet::new();
let disk_io_context = current_disk_io_context();
let mut readers = Vec::with_capacity(opts.disks.len());
let fds = Arc::new(TokioMutex::new(opts.fallback_disks.iter().flatten().cloned().collect::<VecDeque<_>>()));
#[cfg(test)]
@@ -371,19 +364,11 @@ async fn list_path_raw_inner(
let test_fallbacks_clone = test_fallbacks.clone();
let cancel_rx_clone = cancel_rx.clone();
let producer_errs_clone = producer_errs.clone();
let disk_io_context_clone = disk_io_context.clone();
let (rd, wr) = tokio::io::duplex(64);
readers.push(MetacacheReader::new(rd));
jobs.spawn(with_optional_disk_io_context(disk_io_context_clone, async move {
jobs.spawn(async move {
let mut wr = PublishedBytesWriter::new(wr);
#[cfg(test)]
if let Some(observer) = &opts_clone.test_disk_io_context_observed {
observer.store(
crate::disk::current_disk_io_context().is_some_and(|context| context.class == DiskIoClass::Background),
std::sync::atomic::Ordering::Release,
);
}
#[cfg(test)]
let test_primary_error = if let Some(behavior) = opts_clone.test_reader_behaviors.get(disk_idx).cloned() {
match behavior {
TestReaderBehavior::Eof => return Ok(()),
@@ -667,7 +652,7 @@ async fn list_path_raw_inner(
// warn!("list_path_raw: while need_fallback done");
Ok(())
}));
});
}
let revjob_rx = rx.clone();
@@ -1183,68 +1168,6 @@ mod tests {
assert_eq!(err, DiskError::ErasureReadQuorum);
}
#[tokio::test]
async fn list_path_raw_joinset_producers_inherit_scanner_disk_io_class() {
let observed = Arc::new(AtomicBool::new(false));
crate::disk::with_background_disk_io(
CancellationToken::new(),
list_path_raw(
CancellationToken::new(),
ListPathRawOptions {
disks: vec![None],
min_disks: 1,
test_reader_behaviors: vec![TestReaderBehavior::Eof],
test_disk_io_context_observed: Some(observed.clone()),
..Default::default()
},
),
)
.await
.expect("empty test producer should complete the listing");
assert!(
observed.load(std::sync::atomic::Ordering::Acquire),
"the per-disk JoinSet producer must retain the scanner background class"
);
}
#[tokio::test]
async fn scanner_joinset_walk_uses_the_local_disk_background_gate() {
let root = tempfile::tempdir().expect("local disk root should be created");
let endpoint = crate::disk::endpoint::Endpoint::try_from(root.path().to_str().expect("disk path should be UTF-8"))
.expect("endpoint should parse");
let local = Arc::new(
crate::disk::local::LocalDisk::new(&endpoint, false)
.await
.expect("local disk should open"),
);
let wrapper = crate::disk::disk_store::LocalDiskWrapper::new(local, false);
wrapper.make_volume("bucket").await.expect("bucket volume should be created");
let disk = Arc::new(crate::disk::Disk::Local(Box::new(wrapper.clone())));
let cancellation = CancellationToken::new();
crate::disk::with_background_disk_io(
cancellation.clone(),
list_path_raw(
cancellation,
ListPathRawOptions {
disks: vec![Some(disk)],
bucket: "bucket".to_string(),
min_disks: 1,
recursive: true,
..Default::default()
},
),
)
.await
.expect("scanner metacache walk should complete");
assert!(
wrapper.background_io_admissions_for_tests() > 0,
"the LocalDiskWrapper walk should acquire background admission inside its JoinSet producer"
);
}
#[tokio::test]
async fn list_path_raw_returns_typed_quorum_error_for_multiple_drive_failures() {
let err = list_path_raw(
+30 -184
View File
@@ -24,11 +24,10 @@ use crate::cluster::rpc::{gen_tonic_replay_scope_headers, gen_tonic_signature_he
#[cfg(test)]
use crate::cluster::rpc::{tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers};
use crate::disk::error::{DiskError, Error as DiskErrorType, RpcStatusError};
#[cfg(test)]
use crate::runtime::sources as runtime_sources;
use http::{Request as HttpRequest, Response as HttpResponse, Uri};
use rustfs_protos::{
ChannelClass, get_channel_for_class,
ChannelClass, create_new_channel, get_channel_for_class,
proto_gen::node_service::{
heal_control_service_client::HealControlServiceClient, node_service_client::NodeServiceClient,
scanner_control_service_client::ScannerControlServiceClient,
@@ -54,7 +53,7 @@ use super::context_propagation::{inject_request_id_into_metadata, inject_trace_c
/// 3. Subsequent calls will attempt fresh connections
/// 4. If node is still down, connection will fail fast (3s timeout)
pub async fn node_service_time_out_client(
addr: &str,
addr: &String,
interceptor: TonicInterceptor,
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
// Default to the latency-sensitive control channel; bulk `bytes` RPCs opt in via the
@@ -67,9 +66,12 @@ pub(crate) async fn scanner_control_time_out_client(
interceptor: TonicInterceptor,
) -> crate::error::Result<ScannerControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = get_channel_for_class(addr, ChannelClass::Control)
.await
.map_err(|err| crate::error::Error::other(err.to_string()))?;
let channel = match runtime_sources::cached_node_channel(addr).await {
Some(channel) => channel,
None => create_new_channel(addr)
.await
.map_err(|err| crate::error::Error::other(err.to_string()))?,
};
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
let limit = rustfs_protos::scoped_dirty_usage::SCOPED_DIRTY_USAGE_MAX_REQUEST_BYTES as usize;
Ok(ScannerControlServiceClient::with_interceptor(channel, interceptor)
@@ -82,7 +84,10 @@ pub async fn heal_control_time_out_client(
interceptor: TonicInterceptor,
) -> Result<HealControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
let channel = match runtime_sources::cached_node_channel(addr).await {
Some(channel) => channel,
None => create_new_channel(addr).await?,
};
let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
Ok(HealControlServiceClient::with_interceptor(channel, interceptor)
@@ -95,7 +100,10 @@ pub async fn tier_mutation_control_time_out_client(
interceptor: TonicInterceptor,
) -> Result<TierMutationControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
let channel = match runtime_sources::cached_node_channel(addr).await {
Some(channel) => channel,
None => create_new_channel(addr).await?,
};
let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
Ok(TierMutationControlServiceClient::with_interceptor(channel, interceptor)
@@ -110,12 +118,22 @@ pub async fn tier_mutation_control_time_out_client(
/// from lock/health RPCs; everything else uses [`ChannelClass::Control`]. When isolation is
/// disabled the two classes resolve to the same cached channel, i.e. legacy behavior.
pub async fn node_service_time_out_client_for_class(
addr: &str,
addr: &String,
interceptor: TonicInterceptor,
class: ChannelClass,
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = get_channel_for_class(addr, class).await?;
let channel = match class {
ChannelClass::Control => match runtime_sources::cached_node_channel(addr).await {
Some(channel) => {
debug!("Using cached gRPC channel for: {}", addr);
channel
}
// No cached connection, create new one.
None => create_new_channel(addr).await?,
},
ChannelClass::Bulk => get_channel_for_class(addr, ChannelClass::Bulk).await?,
};
let max_message_size = rustfs_protos::internode_rpc_max_message_size();
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
@@ -124,24 +142,8 @@ pub async fn node_service_time_out_client_for_class(
.max_encoding_message_size(max_message_size))
}
pub(crate) async fn node_service_delete_recovery_client(
addr: &str,
interceptor: TonicInterceptor,
timeout: std::time::Duration,
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, DiskError> {
let interceptor = interceptor.with_rpc_audience(addr).map_err(DiskError::from)?;
let channel = rustfs_protos::create_channel_with_timeout(addr, timeout)
.await
.map_err(|error| DiskError::RemoteClientUnavailable(error.to_string()))?;
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
let max_message_size = rustfs_protos::internode_rpc_max_message_size();
Ok(NodeServiceClient::with_interceptor(channel, interceptor)
.max_decoding_message_size(max_message_size)
.max_encoding_message_size(max_message_size))
}
pub async fn node_service_time_out_client_no_auth(
addr: &str,
addr: &String,
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
node_service_time_out_client(addr, TonicInterceptor::NoOp(NoOpInterceptor)).await
}
@@ -353,13 +355,6 @@ fn apply_peer_replay_response(
}
}
fn rename_data_grpc_stage(path: &str) -> bool {
matches!(
path,
"/node_service.NodeService/RenameData" | "/node_service.NodeService/RenameDataAtIncarnation"
)
}
impl<S, ReqBody, ResBody> Service<HttpRequest<ReqBody>> for ReplayScopeChannel<S>
where
S: Service<HttpRequest<ReqBody>, Response = HttpResponse<ResBody>>,
@@ -377,12 +372,6 @@ where
}
fn call(&mut self, mut request: HttpRequest<ReqBody>) -> Self::Future {
let observe_rename_data = rustfs_io_metrics::put_stage_metrics_enabled() && rename_data_grpc_stage(request.uri().path());
let request_scope_started = if observe_rename_data {
rustfs_io_metrics::put_stage_timer()
} else {
None
};
let authenticated = self.audience.as_ref().is_some_and(|_| {
request
.headers()
@@ -417,30 +406,11 @@ where
}
}
}
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REQUEST_SCOPE,
request_scope_started,
);
let audience = self.audience.clone();
let future = self.inner.call(request);
Box::pin(async move {
let response = if observe_rename_data {
rustfs_io_metrics::observe_put_stage_future(
future,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_POLL_PENDING_COUNT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_FIRST_PENDING_TO_READY,
)
.await?
} else {
future.await?
};
let replay_response_started = if observe_rename_data {
rustfs_io_metrics::put_stage_timer()
} else {
None
};
let response = future.await?;
if let (Some(audience), Some(challenge)) = (audience, challenge) {
let response_state = verify_tonic_peer_replay_capabilities_response(&audience, challenge, response.headers());
if let Err(error) = &response_state
@@ -458,10 +428,6 @@ where
}
apply_peer_replay_response(audience, sent_state, response_state);
}
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REPLAY_RESPONSE,
replay_response_started,
);
Ok(response)
})
}
@@ -580,126 +546,6 @@ mod tests {
use tracing_opentelemetry::OpenTelemetrySpanExt;
use tracing_subscriber::{Registry, layer::SubscriberExt};
#[tokio::test]
async fn delete_recovery_channel_uses_its_scoped_deadline() {
use bytes::Bytes;
use http_body_util::StreamBody;
use hyper::{body::Frame, service::service_fn};
use hyper_util::rt::{TokioExecutor, TokioIo};
use rustfs_protos::proto_gen::node_service::{RecoverDeleteTransactionRequest, RecoverDeleteTransactionResponse};
use tonic_prost::prost::Message;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
.await
.expect("bind recovery peer");
let addr = format!("http://{}", listener.local_addr().expect("peer address"));
let server = tokio::spawn(async move {
loop {
let (stream, _) = listener.accept().await.expect("accept recovery connection");
tokio::spawn(async move {
let service = service_fn(|_request: HttpRequest<hyper::body::Incoming>| async {
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
let payload = RecoverDeleteTransactionResponse {
success: true,
protocol_version: 1,
result: Bytes::new(),
error: None,
}
.encode_to_vec();
let mut data = vec![0];
data.extend_from_slice(&u32::try_from(payload.len()).expect("small response").to_be_bytes());
data.extend_from_slice(&payload);
let mut trailers = http::HeaderMap::new();
trailers.insert("grpc-status", "0".parse().expect("status"));
let frames: Vec<Result<Frame<Bytes>, std::convert::Infallible>> =
vec![Ok(Frame::data(data.into())), Ok(Frame::trailers(trailers))];
let mut response = HttpResponse::new(StreamBody::new(futures::stream::iter(frames)));
response
.headers_mut()
.insert("content-type", "application/grpc".parse().expect("content type"));
Ok::<_, std::convert::Infallible>(response)
});
let _ = hyper::server::conn::http2::Builder::new(TokioExecutor::new())
.serve_connection(TokioIo::new(stream), service)
.await;
});
}
});
let request = || RecoverDeleteTransactionRequest {
disk: "disk".into(),
volume: "bucket".into(),
protocol_version: 1,
operation: Bytes::new(),
};
// Seed an existing short-lived control channel to reproduce the inherited deadline.
let control = tonic::transport::Endpoint::from_shared(addr.clone())
.expect("test endpoint")
.timeout(std::time::Duration::from_millis(50))
.connect()
.await
.expect("control connect");
rustfs_common::cache_connection(addr.clone(), control).await;
let mut foreground = node_service_time_out_client(&addr, TonicInterceptor::NoOp(NoOpInterceptor))
.await
.expect("control client");
let error = foreground
.recover_delete_transaction(request())
.await
.expect_err("short control deadline");
assert!(matches!(error.code(), tonic::Code::Cancelled | tonic::Code::DeadlineExceeded));
// The 2-second test override bypasses the production range parser, which rejects 2.
let mut recovery = node_service_delete_recovery_client(
&addr,
TonicInterceptor::NoOp(NoOpInterceptor),
std::time::Duration::from_secs(2),
)
.await
.expect("scoped recovery client");
assert!(
recovery
.recover_delete_transaction(request())
.await
.expect("scoped deadline must survive control deadline")
.into_inner()
.success
);
server.abort();
rustfs_protos::evict_failed_connection(&addr).await;
}
#[tokio::test]
async fn rpc_client_factories_do_not_reuse_https_channels_without_tls_generation() {
let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("reserve local address");
let addr = format!("https://{}", listener.local_addr().expect("local address"));
drop(listener);
for factory in 0..4 {
let placeholder = tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy();
runtime_sources::cache_test_node_channel(addr.clone(), placeholder).await;
let interceptor = TonicInterceptor::NoOp(NoOpInterceptor);
let _result = match factory {
0 => scanner_control_time_out_client(&addr, interceptor)
.await
.map(|_| ())
.map_err(|e| e.to_string()),
1 => heal_control_time_out_client(&addr, interceptor)
.await
.map(|_| ())
.map_err(|e| e.to_string()),
2 => tier_mutation_control_time_out_client(&addr, interceptor)
.await
.map(|_| ())
.map_err(|e| e.to_string()),
_ => node_service_time_out_client(&addr, interceptor)
.await
.map(|_| ())
.map_err(|e| e.to_string()),
};
assert!(
!runtime_sources::test_node_channel_is_cached(&addr).await,
"factory {factory} must evict an unbound HTTPS cache entry before attempting a fresh dial"
);
}
}
#[derive(Clone)]
struct EpochProofService {
audience: String,
@@ -289,9 +289,13 @@ async fn client(
>,
NetworkPeerProbeError,
> {
node_service_time_out_client_for_class(address, TonicInterceptor::Signature(gen_tonic_signature_interceptor()), class)
.await
.map_err(|_| NetworkPeerProbeError::Unreachable)
node_service_time_out_client_for_class(
&address.to_owned(),
TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
class,
)
.await
.map_err(|_| NetworkPeerProbeError::Unreachable)
}
fn ping_request(payload: &[u8]) -> PingRequest {
@@ -325,24 +329,6 @@ fn map_status(status: tonic::Status) -> NetworkPeerProbeError {
match status.code() {
Code::Unavailable | Code::Unknown => NetworkPeerProbeError::Unreachable,
Code::DeadlineExceeded => NetworkPeerProbeError::TimedOut,
Code::Cancelled | Code::Internal => {
let mut source = std::error::Error::source(&status);
while let Some(error) = source {
if error
.downcast_ref::<h2::Error>()
.is_some_and(|error| matches!(error.reason(), Some(h2::Reason::CANCEL | h2::Reason::NO_ERROR)))
{
return NetworkPeerProbeError::Unreachable;
}
source = if let Some(error) = error.downcast_ref::<std::io::Error>() {
// io::Error::source skips its immediate custom error.
error.get_ref().map(|inner| inner as &(dyn std::error::Error + 'static))
} else {
error.source()
};
}
NetworkPeerProbeError::ProtocolFailure
}
_ => NetworkPeerProbeError::ProtocolFailure,
}
}
@@ -353,35 +339,6 @@ mod tests {
use crate::layout::endpoint::Endpoint;
use crate::layout::endpoints::{Endpoints, PoolEndpoints};
#[test]
fn peer_transport_resets_are_unreachable_without_reclassifying_application_errors() {
for reason in [h2::Reason::CANCEL, h2::Reason::NO_ERROR] {
let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
assert!(matches!(status.code(), Code::Cancelled | Code::Internal));
let code = status.code();
assert_eq!(map_status(status), NetworkPeerProbeError::Unreachable);
let mut wrapped = tonic::Status::new(code, "wrapped transport reset");
wrapped.set_source(Arc::new(std::io::Error::other(h2::Error::from(reason))));
assert_eq!(map_status(wrapped), NetworkPeerProbeError::Unreachable);
}
for reason in [h2::Reason::PROTOCOL_ERROR, h2::Reason::COMPRESSION_ERROR] {
let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
}
for status in [
tonic::Status::cancelled("peer cancelled the application request"),
tonic::Status::internal("peer rejected the probe"),
tonic::Status::unauthenticated("authentication rejected"),
tonic::Status::permission_denied("authorization rejected"),
] {
assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
}
assert_eq!(
map_status(tonic::Status::deadline_exceeded("probe deadline elapsed")),
NetworkPeerProbeError::TimedOut
);
}
fn pacing(duration: Duration) -> Arc<DiagnosticPacing> {
NetworkPeerProbeClient::from_endpoint_pools(&topology())
.with_diagnostic_pacing(Instant::now(), duration)
@@ -640,7 +640,6 @@ pub struct PeerLiveEventsBatch {
pub struct ScannerPublicationLease {
pub token: Uuid,
pub movement_generation: u64,
pub observational_only: bool,
/// Stable storage owner identity. This is distinct from the activity
/// session and is bound into both acquire and release proofs.
pub owner_id: String,
@@ -659,7 +658,6 @@ fn validate_scanner_publication_lease_response_fields(
response: &ScannerPublicationLeaseResponse,
expected_session_id: &str,
expected_generation: u64,
expected_observational_only: bool,
) -> Result<(Uuid, String)> {
if !response.success {
return Err(Error::other(
@@ -676,9 +674,6 @@ fn validate_scanner_publication_lease_response_fields(
if response.session_id != expected_session_id {
return Err(Error::other("peer returned a different scanner publication lease session"));
}
if response.observational_only != expected_observational_only {
return Err(Error::other("peer returned a different scanner publication lease purpose"));
}
let owner_id = Uuid::parse_str(&response.owner_id)
.ok()
.filter(|owner_id| !owner_id.is_nil())
@@ -2423,7 +2418,6 @@ impl PeerRestClient {
&self,
expected_session_id: &str,
expected_generation: u64,
observational_only: bool,
) -> Result<ScannerPublicationLease> {
let request_started = std::time::Instant::now();
self.finalize_result(
@@ -2440,7 +2434,6 @@ impl PeerRestClient {
ttl_ms: crate::store::SCANNER_PUBLICATION_LEASE_TTL_MS,
expected_session_id: expected_session_id.to_string(),
token: Bytes::new(),
observational_only,
});
let canonical = rustfs_protos::canonical_scanner_publication_lease_request_body(request.get_ref())
.map_err(|_| Error::other("scanner publication lease request is too large to authenticate"))?;
@@ -2451,16 +2444,11 @@ impl PeerRestClient {
.map_err(|_| Error::other("scanner publication lease response is too large to authenticate"))?;
verify_tonic_rpc_response_proof(&response_body, &response.response_proof)
.map_err(|_| Error::other("peer returned an invalid scanner publication lease proof"))?;
let (token, owner_id) = validate_scanner_publication_lease_response_fields(
&response,
expected_session_id,
expected_generation,
observational_only,
)?;
let (token, owner_id) =
validate_scanner_publication_lease_response_fields(&response, expected_session_id, expected_generation)?;
Ok(ScannerPublicationLease {
token,
movement_generation: response.movement_generation,
observational_only,
owner_id,
session_id: response.session_id,
expires_at: scanner_publication_lease_deadline(
@@ -2495,7 +2483,6 @@ impl PeerRestClient {
ttl_ms: crate::store::SCANNER_PUBLICATION_LEASE_TTL_MS,
expected_session_id: lease.session_id.clone(),
token: lease.token.as_bytes().to_vec().into(),
observational_only: lease.observational_only,
});
let canonical = rustfs_protos::canonical_scanner_publication_lease_request_body(request.get_ref())
.map_err(|_| Error::other("scanner publication lease validation request is too large to authenticate"))?;
@@ -2507,12 +2494,8 @@ impl PeerRestClient {
)?;
verify_tonic_rpc_response_proof(&response_body, &response.response_proof)
.map_err(|_| Error::other("peer returned an invalid scanner publication lease validation proof"))?;
let (token, owner_id) = validate_scanner_publication_lease_response_fields(
&response,
&lease.session_id,
lease.movement_generation,
lease.observational_only,
)?;
let (token, owner_id) =
validate_scanner_publication_lease_response_fields(&response, &lease.session_id, lease.movement_generation)?;
if token != lease.token {
return Err(Error::other("peer returned a different scanner publication lease token"));
}
@@ -3018,16 +3001,15 @@ mod tests {
response_proof: Bytes::new(),
owner_id: Uuid::new_v4().to_string(),
session_id: "session-a".to_string(),
observational_only: false,
};
assert!(validate_scanner_publication_lease_response_fields(&response, "session-a", 7, false).is_ok());
assert!(validate_scanner_publication_lease_response_fields(&response, "session-a", 7).is_ok());
let stale_generation = ScannerPublicationLeaseResponse {
movement_generation: 6,
..response.clone()
};
let error = validate_scanner_publication_lease_response_fields(&stale_generation, "session-a", 7, false)
let error = validate_scanner_publication_lease_response_fields(&stale_generation, "session-a", 7)
.expect_err("a response from an older movement generation must be rejected");
assert!(error.to_string().contains("different scanner publication lease generation"));
@@ -3035,21 +3017,9 @@ mod tests {
session_id: "session-b".to_string(),
..response
};
let error = validate_scanner_publication_lease_response_fields(&stale_session, "session-a", 7, false)
let error = validate_scanner_publication_lease_response_fields(&stale_session, "session-a", 7)
.expect_err("a response from an older scanner session must be rejected");
assert!(error.to_string().contains("different scanner publication lease session"));
let observational = ScannerPublicationLeaseResponse {
observational_only: true,
..stale_session
};
assert!(
validate_scanner_publication_lease_response_fields(&observational, "session-b", 7, true).is_ok(),
"the signed purpose echo should validate an observation lease"
);
let error = validate_scanner_publication_lease_response_fields(&observational, "session-b", 7, false)
.expect_err("a peer must not upgrade an observational lease to authoritative");
assert!(error.to_string().contains("different scanner publication lease purpose"));
}
#[test]
+39 -418
View File
@@ -14,8 +14,7 @@
use crate::cluster::rpc::client::{
AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
node_service_delete_recovery_client, node_service_time_out_client, node_service_time_out_client_for_class,
node_service_time_out_client_no_auth,
node_service_time_out_client, node_service_time_out_client_for_class, node_service_time_out_client_no_auth,
};
use crate::cluster::rpc::internode_data_transport::{
InternodeDataTransport, NsScannerCapabilityRequest, NsScannerStreamRequest, ReadStreamRequest, WalkDirStreamRequest,
@@ -23,11 +22,10 @@ use crate::cluster::rpc::internode_data_transport::{
};
use crate::disk::error::{Error, Result};
use crate::disk::{
BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION,
DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection,
DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation,
DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq, ReadMultipleResp, ReadOptions,
RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one,
BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DeleteOptions, DiskAPI, DiskInfo,
DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq,
ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
batch_read_version_one_by_one,
disk_store::{
DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE,
get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout,
@@ -61,10 +59,10 @@ use rustfs_protos::proto_gen::node_service::{
CompareAndUpdateFileRequest, DeletePathsRequest, DeleteRequest, DeleteVersionRequest, DeleteVersionsRequest,
DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, ListVolumesRequest, MakeVolumeRequest,
MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, ReadMetadataRequest, ReadMultipleRequest,
ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RecoverDeleteTransactionRequest,
RenameDataRequest, RenameFileRequest, SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest,
SnapshotLeaseRequest, SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest,
WriteMetadataRequest, node_service_client::NodeServiceClient,
ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest, RenameFileRequest,
SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest,
SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest,
node_service_client::NodeServiceClient,
};
use serde::{Serialize, de::DeserializeOwned};
use std::{
@@ -144,45 +142,6 @@ fn decode_delete_versions_errors(response: DeleteVersionsResponse, expected_len:
.collect()
}
fn decode_delete_recovery_batch_errors(codes: Vec<Option<u32>>, expected_len: usize) -> Result<Vec<Option<DiskError>>> {
if codes.len() != expected_len {
return Err(Error::other("malformed recoverable batch delete response length"));
}
codes
.into_iter()
.map(|code| match code {
None => Ok(None),
Some(code) => DiskError::from_u32(code)
.map(Some)
.ok_or_else(|| Error::other("invalid recoverable batch delete error code")),
})
.collect()
}
fn validate_delete_recovery_inspection(
volume: &str,
object: &str,
transaction: Uuid,
inspection: &DeleteRecoveryInspection,
) -> Result<()> {
if inspection.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
|| inspection.id.object != object
|| inspection.id.transaction != transaction
|| transaction.is_nil()
{
return Err(Error::other("delete recovery inspection does not match request"));
}
if let Some(journal) = &inspection.journal
&& (journal.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
|| journal.volume != volume
|| journal.object != object
|| journal.transaction != transaction)
{
return Err(Error::other("delete recovery journal does not match request"));
}
Ok(())
}
fn snapshot_lease_token_from_response(response: SnapshotLeaseResponse) -> Result<SnapshotLeaseToken> {
if !response.success {
return Err(response.error.unwrap_or_default().into());
@@ -899,75 +858,6 @@ fn spawn_control_channel_prewarm(addr: String) {
}
impl RemoteDisk {
async fn recover_delete_transaction(
&self,
volume: &str,
operation: DeleteRecoveryRpcOperation,
) -> Result<DeleteRecoveryRpcReply> {
let payload = rmp_serde::to_vec_named(&operation)?;
if payload.len() > operation.max_wire_payload_bytes() {
return Err(Error::other("delete recovery operation exceeds payload limit"));
}
let settle_timeout = if matches!(&operation, DeleteRecoveryRpcOperation::Settle { .. }) {
Some(crate::disk::get_delete_recovery_timeout()?)
} else {
None
};
self.execute_with_timeout(
|| async {
let mut client = if let Some(timeout) = settle_timeout {
if let Some(error) = self.offline_bypass_error() {
return Err(error);
}
node_service_delete_recovery_client(
&self.addr,
TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
timeout,
)
.await?
} else {
self.get_client().await?
};
let mut request = Request::new(RecoverDeleteTransactionRequest {
disk: self.endpoint.to_string(),
volume: volume.to_owned(),
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
operation: payload.into(),
});
let canonical_body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref())
.map_err(|_| Error::other("recover_delete_transaction request length cannot be represented"))?;
crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical_body).map_err(Error::other)?;
// UNIMPLEMENTED is deliberately propagated: older peers cannot settle transactions.
let response = client
.recover_delete_transaction(request)
.await
.map_err(|status| {
if status.code() == Code::Unimplemented {
DiskError::UnsupportedDisk
} else {
status.into()
}
})?
.into_inner();
if response.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION {
return Err(DiskError::UnsupportedDisk);
}
if !response.success {
return Err(response
.error
.ok_or_else(|| Error::other("missing delete recovery error"))?
.into());
}
if response.error.is_some() {
return Err(Error::other("malformed delete recovery success"));
}
rustfs_protos::decode_delete_recovery_payload(&response.result).map_err(Into::into)
},
settle_timeout.unwrap_or_else(get_max_timeout_duration),
)
.await
}
async fn rename_file_with_durability(
&self,
src_volume: &str,
@@ -2030,12 +1920,6 @@ fn encode_msgpack<T: Serialize>(value: &T) -> Result<Vec<u8>> {
encode_msgpack_with_capacity(value, MSGPACK_ENCODE_CAPACITY_HINT)
}
// Optional fields are omitted by serde; named maps keep rollback flags and
// transaction UUIDs independent of field order across peer versions.
fn encode_delete_options_msgpack(value: &DeleteOptions) -> Result<Vec<u8>> {
Ok(rmp_serde::to_vec_named(value)?)
}
fn encode_file_info_msgpack(value: &FileInfo) -> Result<Vec<u8>> {
encode_msgpack_with_capacity(value, FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT)
}
@@ -2279,7 +2163,6 @@ impl RemoteDisk {
self.execute_with_timeout_for_op(
"rename_data",
|| async {
let prepare_started = rustfs_io_metrics::put_stage_timer();
let file_info = compat_json(fi)?;
let file_info_bin = encode_file_info_msgpack(fi)?;
let mut client = self.get_client().await?;
@@ -2307,67 +2190,24 @@ impl RemoteDisk {
} else {
attach_mutation_body_digest(&mut request, canonical_body, "rename_data")?;
}
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE,
prepare_started,
);
let rpc_started = rustfs_io_metrics::put_stage_timer();
let response = rustfs_io_metrics::observe_put_stage_future(
async {
if incarnation_bound {
// Older peers return Unimplemented before mutation; never downgrade.
client.rename_data_at_incarnation(request).await
} else {
client.rename_data(request).await
}
},
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY,
)
.await?;
let response_process_started = rustfs_io_metrics::put_stage_timer();
let response_headers_started = rustfs_io_metrics::put_stage_timer();
let _ = response.metadata().len();
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_HEADERS,
response_headers_started,
);
let into_inner_started = rustfs_io_metrics::put_stage_timer();
let response = response.into_inner();
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_INTO_INNER,
into_inner_started,
);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC,
rpc_started,
);
let response = if incarnation_bound {
// Older peers return Unimplemented before mutation; never downgrade.
client.rename_data_at_incarnation(request).await?
} else {
client.rename_data(request).await?
}
.into_inner();
let success_check_started = rustfs_io_metrics::put_stage_timer();
if !response.success {
return Err(response.error.unwrap_or_default().into());
}
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_SUCCESS_CHECK,
success_check_started,
);
let decode_started = rustfs_io_metrics::put_stage_timer();
let rename_data_resp = decode_msgpack_or_json::<RenameDataResp>(
&response.rename_data_resp_bin,
&response.rename_data_resp,
"RenameDataResp",
)?;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_DECODE,
decode_started,
);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_PROCESS,
response_process_started,
);
Ok(rename_data_resp)
},
@@ -2726,7 +2566,7 @@ impl DiskAPI for RemoteDisk {
// `_bin` support for DeleteVersion is new (grpc-optimization P2); always dual-write
// JSON + msgpack until its fallback counter has read zero across a release window.
let file_info_bin = encode_file_info_msgpack(&fi)?;
let opts_bin = encode_delete_options_msgpack(&opts)?;
let opts_bin = encode_msgpack(&opts)?;
let conditional_marker = opts.expected_delete_marker.is_some();
let file_info = serde_json::to_string(&fi)?;
let opts = serde_json::to_string(&opts)?;
@@ -2798,7 +2638,7 @@ impl DiskAPI for RemoteDisk {
// `_bin` support for DeleteVersions is new (grpc-optimization P2); always dual-write JSON +
// msgpack until its fallback counter has read zero across a release window.
let opts_bin = match encode_delete_options_msgpack(&opts) {
let opts_bin = match encode_msgpack(&opts) {
Ok(opts_bin) => opts_bin,
Err(err) => {
let mut errors = Vec::with_capacity(versions.len());
@@ -3777,138 +3617,6 @@ impl DiskAPI for RemoteDisk {
}
#[tracing::instrument(level = "trace", skip_all)]
async fn delete_version_recoverable(
&self,
volume: &str,
object: &str,
file_info: FileInfo,
force_del_marker: bool,
opts: DeleteOptions,
) -> Result<()> {
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::ApplyVersion {
object: object.to_owned(),
file_info: Box::new(file_info),
force_del_marker,
opts,
},
)
.await?
{
DeleteRecoveryRpcReply::AppliedVersion => Ok(()),
_ => Err(Error::other("unexpected recoverable delete reply")),
}
}
async fn delete_versions_recoverable(
&self,
volume: &str,
versions: Vec<FileInfoVersions>,
opts: DeleteOptions,
) -> Vec<Option<DiskError>> {
let expected_len = versions.len();
let result = match self
.recover_delete_transaction(volume, DeleteRecoveryRpcOperation::ApplyVersions { versions, opts })
.await
{
Ok(DeleteRecoveryRpcReply::AppliedVersions(codes)) => decode_delete_recovery_batch_errors(codes, expected_len),
Ok(_) => Err(Error::other("unexpected recoverable batch delete reply")),
Err(error) => Err(error),
};
match result {
Ok(errors) => errors,
Err(error) => vec![Some(error); expected_len],
}
}
async fn discover_delete_transactions(
&self,
volume: &str,
object: Option<&str>,
cursor: Option<DeleteRecoveryCursor>,
limit: u32,
) -> Result<DeleteRecoveryPage> {
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::Discover {
object: object.map(str::to_owned),
cursor,
limit,
},
)
.await?
{
DeleteRecoveryRpcReply::Discovered(page) => {
if object.is_some_and(|object| page.transactions.iter().any(|id| id.object != object)) {
return Err(Error::other("delete recovery discovery escaped requested object"));
}
if page.transactions.len() > usize::try_from(limit).map_err(|_| Error::other("invalid discovery limit"))?
|| page
.transactions
.iter()
.any(|id| id.transaction.is_nil() || id.object.is_empty())
{
return Err(Error::other("malformed delete discovery page"));
}
Ok(page)
}
_ => Err(Error::other("unexpected delete discovery reply")),
}
}
async fn inspect_delete_transaction(
&self,
volume: &str,
object: &str,
transaction: Uuid,
) -> Result<DeleteRecoveryInspection> {
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::Inspect {
object: object.to_owned(),
transaction,
},
)
.await?
{
DeleteRecoveryRpcReply::Inspected(inspection) => {
validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
Ok(inspection)
}
_ => Err(Error::other("unexpected delete inspection reply")),
}
}
async fn settle_delete_transaction(
&self,
volume: &str,
object: &str,
transaction: Uuid,
expected_fingerprint: DeleteRecoveryDigest,
action: DeleteRecoveryAction,
) -> Result<DeleteRecoveryInspection> {
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::Settle {
object: object.to_owned(),
transaction,
expected_fingerprint,
action,
},
)
.await?
{
DeleteRecoveryRpcReply::Settled(inspection) => {
validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
Ok(inspection)
}
_ => Err(Error::other("unexpected delete settlement reply")),
}
}
async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
self.execute_with_timeout(
|| async {
@@ -4322,45 +4030,6 @@ mod tests {
static INIT: Once = Once::new();
#[test]
fn recoverable_batch_delete_rejects_invalid_error_codes_and_length() {
let errors = decode_delete_recovery_batch_errors(vec![None, Some(DiskError::FileNotFound.to_u32())], 2)
.expect("valid per-item errors");
assert!(errors[0].is_none());
assert_eq!(errors[1], Some(DiskError::FileNotFound));
assert!(decode_delete_recovery_batch_errors(vec![None], 2).is_err());
assert!(decode_delete_recovery_batch_errors(vec![Some(0)], 1).is_err());
assert!(decode_delete_recovery_batch_errors(vec![Some(u32::MAX)], 1).is_err());
}
#[test]
fn delete_recovery_inspection_rejects_wrong_transaction_object_and_protocol() {
let transaction = Uuid::new_v4();
let inspection = DeleteRecoveryInspection {
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
id: crate::disk::DeleteTransactionId {
object: "object".into(),
transaction,
},
state: crate::disk::DeleteRecoveryState::Absent,
journal: None,
current_digest: None,
current_versions: Vec::new(),
backup_digest: None,
rollback_data_complete: false,
commit_data_safe: false,
explicit_delete_safe: false,
fingerprint: [0; 32],
};
validate_delete_recovery_inspection("bucket", "object", transaction, &inspection).expect("matching inspection");
assert!(validate_delete_recovery_inspection("bucket", "other", transaction, &inspection).is_err());
assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::new_v4(), &inspection).is_err());
assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::nil(), &inspection).is_err());
let mut incompatible = inspection;
incompatible.protocol_version += 1;
assert!(validate_delete_recovery_inspection("bucket", "object", transaction, &incompatible).is_err());
}
#[test]
fn compare_and_update_wire_outcome_fails_closed() {
assert_eq!(
@@ -4394,19 +4063,32 @@ mod tests {
}
#[test]
fn delete_options_named_wire_preserves_rollback_and_legacy_marker_fields() {
fn retired_marker_options_preserve_legacy_positional_wire_shape() {
#[derive(Debug, serde::Serialize, serde::Deserialize, PartialEq)]
struct LegacyDeleteOptions {
recursive: bool,
immediate: bool,
undo_write: bool,
#[serde(default)]
undo_delete: bool,
old_data_dir: Option<Uuid>,
#[serde(default, skip_serializing_if = "Option::is_none")]
expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
}
let transaction = Uuid::new_v4();
let legacy = LegacyDeleteOptions {
recursive: false,
immediate: false,
undo_write: false,
undo_delete: false,
old_data_dir: None,
};
let original = encode_msgpack(&legacy).unwrap();
let current = encode_msgpack(&DeleteOptions::default()).unwrap();
assert_eq!(current, original, "ordinary deletes must retain the older peer's positional payload");
assert_eq!(rmp_serde::from_slice::<LegacyDeleteOptions>(&current).unwrap(), legacy);
assert!(
rmp_serde::from_slice::<DeleteOptions>(&original)
.unwrap()
.expected_delete_marker
.is_none()
);
let mut marker = FileInfo {
deleted: true,
version_id: Some(Uuid::new_v4()),
@@ -4415,69 +4097,12 @@ mod tests {
};
marker.set_delete_marker_incarnation(Uuid::new_v4());
let marker = rustfs_filemeta::MetaDeleteMarker::from(marker);
for expected_delete_marker in [None, Some(marker)] {
for (undo_write, undo_delete) in [(true, false), (false, true)] {
let legacy = LegacyDeleteOptions {
recursive: false,
immediate: false,
undo_write,
undo_delete,
old_data_dir: Some(transaction),
expected_delete_marker: expected_delete_marker.clone(),
};
// Older senders use positional arrays. Their unchanged field prefix and
// missing optional fields must also decode on the named-map sender's peer.
for (encoding, payload) in [
("positional", rmp_serde::to_vec(&legacy).expect("encode legacy positional options")),
("named", rmp_serde::to_vec_named(&legacy).expect("encode legacy named options")),
] {
let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode legacy options on current peer");
assert_eq!(decoded.undo_write, undo_write, "{encoding} undo-write flag");
assert_eq!(decoded.undo_delete, undo_delete, "{encoding} undo-delete flag");
assert_eq!(decoded.old_data_dir, Some(transaction), "{encoding} transaction UUID");
assert_eq!(decoded.expected_delete_marker, expected_delete_marker, "{encoding} marker precondition");
assert!(decoded.delete_recovery_bucket_incarnation.is_none(), "{encoding} omitted incarnation");
assert!(!decoded.recoverable_delete, "{encoding} omitted recovery flag");
}
for incarnation in [None, Some(Uuid::new_v4())] {
let current = DeleteOptions {
undo_write,
undo_delete,
old_data_dir: Some(transaction),
expected_delete_marker: expected_delete_marker.clone(),
delete_recovery_bucket_incarnation: incarnation,
recoverable_delete: true,
..Default::default()
};
let payload = encode_delete_options_msgpack(&current).expect("encode current named options");
let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode current options on current peer");
assert_eq!(decoded.undo_write, undo_write);
assert_eq!(decoded.undo_delete, undo_delete);
assert_eq!(decoded.old_data_dir, Some(transaction));
assert_eq!(decoded.expected_delete_marker, expected_delete_marker);
assert_eq!(decoded.delete_recovery_bucket_incarnation, incarnation);
assert!(decoded.recoverable_delete);
let decoded: LegacyDeleteOptions =
rmp_serde::from_slice(&payload).expect("decode current options on legacy peer");
assert_eq!(decoded, legacy, "unknown recovery fields cannot shift rollback or marker values");
}
}
}
}
#[test]
fn malformed_nonempty_delete_options_do_not_fall_back_to_json() {
let options = DeleteOptions {
undo_write: true,
old_data_dir: Some(Uuid::new_v4()),
expected_delete_marker: Some(marker.clone()),
..Default::default()
};
let json = serde_json::to_string(&options).expect("valid compatibility JSON");
assert!(decode_msgpack_or_json::<DeleteOptions>(&[0xc1], &json, "DeleteOptions").is_err());
let decoded: DeleteOptions =
decode_msgpack_or_json(&[], &json, "DeleteOptions").expect("absent MessagePack permits JSON compatibility");
assert!(decoded.undo_write);
assert_eq!(decoded.old_data_dir, options.old_data_dir);
let decoded: DeleteOptions = rmp_serde::from_slice(&encode_msgpack(&options).unwrap()).unwrap();
assert_eq!(decoded.expected_delete_marker, Some(marker));
}
#[test]
@@ -5160,8 +4785,6 @@ mod tests {
cleanup_data_dir: Some(Uuid::new_v4()),
sign: Some(vec![0x14, 0x35]),
old_current_size: Some(crate::disk::OldCurrentSize::Present(64 * 1024)),
old_current_source_checked: false,
old_current_source: None,
};
let json = serde_json::to_string(&response).expect("legacy rename_data JSON response should encode");
let decode_before = rustfs_io_metrics::internode_metrics::global_internode_metrics().msgpack_json_decode_total_for_test();
@@ -5564,8 +5187,6 @@ mod tests {
cleanup_data_dir: Some(Uuid::new_v4()),
sign: Some(vec![1_u8; 32]),
old_current_size: Some(crate::disk::OldCurrentSize::Present(4096)),
old_current_source_checked: false,
old_current_source: None,
};
let json = serde_json::to_vec(&response).expect("rename data response json should encode");
let named_msgpack = rmp_serde::encode::to_vec_named(&response).expect("rename data response named msgpack should encode");
-90
View File
@@ -3691,55 +3691,6 @@ fn activation_pool_meta_lock_error(err: rustfs_lock::LockError) -> Error {
}
}
fn rebalance_worker_read_lock_error(err: rustfs_lock::LockError, object: &str) -> Error {
match err {
rustfs_lock::LockError::QuorumNotReached { required, achieved } => Error::NamespaceLockQuorumUnavailable {
mode: "read",
bucket: RUSTFS_META_BUCKET.to_string(),
object: object.to_string(),
required,
achieved,
},
other => Error::Lock(other),
}
}
/// Read-only admission to an already durable rebalance run. Keeping this
/// separate from the activation writer prevents admission from authorizing a
/// pool or rebalance metadata write under shared namespace locks.
pub(crate) struct PoolRebalanceWorkerFence {
pool_meta_guard: rustfs_lock::NamespaceLockGuard,
rebalance_meta_guard: rustfs_lock::NamespaceLockGuard,
fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
}
impl PoolRebalanceWorkerFence {
pub(crate) fn set_fleet_proof(
&mut self,
fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
) {
self.fleet_proof = fleet_proof;
}
pub(crate) fn ensure_held(&self) -> Result<()> {
if self.pool_meta_guard.is_lock_lost()
|| self.pool_meta_guard.is_released()
|| self.rebalance_meta_guard.is_lock_lost()
|| self.rebalance_meta_guard.is_released()
{
return Err(Error::other("worker admission lock lost before rebalance activation"));
}
if self
.fleet_proof
.as_ref()
.is_some_and(|proof| !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof))
{
return Err(Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED));
}
Ok(())
}
}
pub(crate) struct PoolRebalanceActivationFence {
pool_meta_guard: Arc<rustfs_lock::NamespaceLockGuard>,
rebalance_meta_guard: Arc<rustfs_lock::NamespaceLockGuard>,
@@ -3811,35 +3762,6 @@ tokio::task_local! {
pub(crate) static REBALANCE_ACTIVATION_LOCK_ATTEMPT: Arc<tokio::sync::Notify>;
}
pub(crate) async fn acquire_pool_rebalance_worker_locks<S>(pool: Arc<S>) -> Result<PoolRebalanceWorkerFence>
where
S: crate::storage_api_contracts::namespace::NamespaceLocking<
Error = Error,
NamespaceLock = rustfs_lock::NamespaceLockWrapper,
>,
{
// Match activation and entry admission: rebalance.bin -> pool.bin. Joining
// the same durable run must coexist with entries retaining their run fence.
let rebalance_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, REBAL_META_NAME).await?;
#[cfg(test)]
let _ = REBALANCE_ACTIVATION_LOCK_ATTEMPT.try_with(|attempted| attempted.notify_one());
let rebalance_meta_guard = rebalance_meta_lock
.get_read_lock(get_lock_acquire_timeout())
.await
.map_err(|err| rebalance_worker_read_lock_error(err, REBAL_META_NAME))?;
let pool_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, POOL_META_NAME).await?;
let pool_meta_guard = pool_meta_lock
.get_read_lock(get_lock_acquire_timeout())
.await
.map_err(|err| rebalance_worker_read_lock_error(err, POOL_META_NAME))?;
Ok(PoolRebalanceWorkerFence {
pool_meta_guard,
rebalance_meta_guard,
fleet_proof: None,
})
}
pub(crate) async fn acquire_pool_rebalance_activation_locks<S>(
pool: Arc<S>,
fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
@@ -12508,18 +12430,6 @@ impl ECStore {
Ok(pool_meta)
}
pub(crate) async fn load_runtime_pool_meta_under_worker_fence(
&self,
write_state: &mut PoolMetaWriteState,
worker_fence: &PoolRebalanceWorkerFence,
operation: &str,
) -> Result<PoolMeta> {
worker_fence.ensure_held()?;
let pool_meta = self.load_runtime_pool_meta_observing(write_state, operation).await?;
worker_fence.ensure_held()?;
Ok(pool_meta)
}
pub(crate) async fn load_runtime_pool_meta_under_activation_fence(
&self,
write_state: &mut PoolMetaWriteState,
+1 -75
View File
@@ -738,12 +738,7 @@ where
.filter(|etag| !etag.is_empty())
.map(str::to_owned)
.ok_or_else(|| Error::other("data usage snapshot has no ETag"))?;
let mut data_usage_info = parse_usage_snapshot(
&reader
.read_all()
.await
.map_err(|err| map_data_usage_metadata_read_error(err, object))?,
)?;
let mut data_usage_info = parse_usage_snapshot(&reader.read_all().await?)?;
populate_backward_compatible_usage_maps(&mut data_usage_info);
validate_complete_usage_snapshot(&mut data_usage_info);
Ok(Some((data_usage_info, revision)))
@@ -3002,61 +2997,6 @@ mod tests {
}
}
#[derive(Debug)]
struct UsageSnapshotReadFailureStore;
struct UsageSnapshotFailureReader;
impl tokio::io::AsyncRead for UsageSnapshotFailureReader {
fn poll_read(
self: std::pin::Pin<&mut Self>,
_cx: &mut std::task::Context<'_>,
_buf: &mut tokio::io::ReadBuf<'_>,
) -> std::task::Poll<std::io::Result<()>> {
std::task::Poll::Ready(Err(std::io::Error::other(Error::VolumeNotFound)))
}
}
#[async_trait::async_trait]
impl crate::storage_api_contracts::object::ObjectIO for UsageSnapshotReadFailureStore {
type Error = Error;
type RangeSpec = crate::storage_api_contracts::range::HTTPRangeSpec;
type HeaderMap = http::HeaderMap;
type ObjectOptions = ObjectOptions;
type ObjectInfo = ObjectInfo;
type GetObjectReader = crate::object_api::GetObjectReader;
type PutObjectReader = PutObjReader;
async fn get_object_reader(
&self,
_bucket: &str,
_object: &str,
_range: Option<Self::RangeSpec>,
_headers: Self::HeaderMap,
_opts: &Self::ObjectOptions,
) -> Result<Self::GetObjectReader, Self::Error> {
Ok(crate::object_api::GetObjectReader {
stream: Box::new(UsageSnapshotFailureReader),
object_info: ObjectInfo {
etag: Some("usage-revision".to_string()),
..Default::default()
},
buffered_body: None,
body_source: Default::default(),
})
}
async fn put_object(
&self,
_bucket: &str,
_object: &str,
_data: &mut Self::PutObjectReader,
_opts: &Self::ObjectOptions,
) -> Result<Self::ObjectInfo, Self::Error> {
unimplemented!("the read-failure fixture does not write")
}
}
/// Minimal ObjectIO backing `load_data_usage_cache` tests: records the keys
/// read and fails the first N reads with a transient (non-absence) error.
#[derive(Debug, Default)]
@@ -3221,20 +3161,6 @@ mod tests {
}
}
#[tokio::test]
async fn data_usage_removal_maps_missing_system_volume_during_snapshot_stream_read() {
let store = UsageSnapshotReadFailureStore;
let error = load_data_usage_for_bucket_removal(&store, "bucket-metadata/.usage.json")
.await
.expect_err("a system volume disappearing during body read must remain a retryable storage error");
assert_eq!(
error,
Error::InsufficientReadQuorum(RUSTFS_META_BUCKET.to_string(), "bucket-metadata/.usage.json".to_string())
);
}
#[tokio::test]
async fn load_data_usage_cache_retries_a_transient_failure() {
let name = "usage-cache";
@@ -15,9 +15,7 @@
use crate::cluster::rpc::{
ScannerBucketListing, TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client,
};
use crate::data_usage::{
DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_admin_data_usage_from_backend_cached as load_server_info_usage_snapshot,
};
use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend_cached};
use crate::error::{Error, Result};
use crate::{
disk::endpoint::{Endpoint, EndpointType},
@@ -44,11 +42,7 @@ use std::{
use time::OffsetDateTime;
use tokio::time::timeout;
use tonic::Request;
use tracing::{error, trace, warn};
const EVENT_SERVER_PING: &str = "server_ping";
const LOG_COMPONENT_ECSTORE: &str = "ecstore";
const LOG_SUBSYSTEM_DIAGNOSTICS: &str = "diagnostics";
use tracing::warn;
use shadow_rs::shadow;
@@ -163,22 +157,11 @@ async fn is_server_resolvable(endpoint: &Endpoint) -> Result<()> {
let response: PingResponse = client.ping(request).await?.into_inner();
if flatbuffers::root::<PingBody>(&response.body).is_err() {
error!(
event = EVENT_SERVER_PING,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DIAGNOSTICS,
result = "invalid_response",
"Server ping response decoding failed"
);
let ping_response_body = flatbuffers::root::<PingBody>(&response.body);
if let Err(e) = ping_response_body {
eprintln!("{e}");
} else {
trace!(
event = EVENT_SERVER_PING,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DIAGNOSTICS,
result = "success",
"Server ping completed"
);
println!("ping_resp:body(flatbuffer): {ping_response_body:?}");
}
Ok(())
@@ -311,7 +294,7 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
if let Some(store) = runtime_sources::object_store_handle() {
mode = ITEM_ONLINE;
apply_data_usage_result(
load_server_info_usage_snapshot(store.clone()).await,
load_data_usage_from_backend_cached(store.clone()).await,
&mut buckets,
&mut objects,
&mut versions,
@@ -984,93 +967,6 @@ mod tests {
assert_eq!(usage.size, 6);
}
#[tokio::test]
#[serial]
async fn server_info_reads_complete_observation_without_promoting_authority() {
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
use crate::config::com::{read_config, save_config};
use crate::data_usage::{
DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache,
load_data_usage_from_backend_cached,
};
use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
use std::time::{Duration, SystemTime};
let (_dirs, store) = isolated_store_over_temp_disks().await;
let primary = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
scanner_epoch: Some(1),
usage_snapshot_bootstrap_pending: true,
usage_snapshot_converged: Some(false),
..Default::default()
};
let observed = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(2)),
scanner_epoch: Some(1),
scanner_cycle: Some(1),
buckets_count: 1,
objects_total_count: 3,
versions_total_count: 4,
delete_markers_total_count: 1,
objects_total_size: 6,
buckets_usage: [(
"bucket".to_string(),
BucketUsageInfo {
objects_count: 3,
versions_count: 4,
delete_markers_count: 1,
size: 6,
..Default::default()
},
)]
.into_iter()
.collect(),
usage_snapshot_complete: true,
usage_snapshot_converged: Some(false),
usage_snapshot_authoritative_baseline: Some(primary.snapshot_identity()),
..Default::default()
};
let primary_bytes = serde_json::to_vec(&primary).expect("bootstrap should encode");
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
.await
.expect("bootstrap should persist");
save_config(
store.clone(),
DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&observed).expect("observation should encode"),
)
.await
.expect("observation should persist");
invalidate_data_usage_snapshot_cache().await;
let displayed = super::load_server_info_usage_snapshot(store.clone())
.await
.expect("server info should load the complete observation");
assert_eq!(displayed.usage_snapshot_converged, Some(false));
let mut buckets = rustfs_madmin::Buckets::default();
let mut objects = rustfs_madmin::Objects::default();
let mut versions = rustfs_madmin::Versions::default();
let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
let mut usage = rustfs_madmin::Usage::default();
apply_data_usage_result(Ok(displayed), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
assert!(objects.error.is_none());
assert_eq!(
(buckets.count, objects.count, versions.count, delete_markers.count, usage.size),
(1, 3, 4, 1, 6)
);
let quota = load_data_usage_from_backend_cached(store.clone())
.await
.expect("quota should retain its bootstrap state");
assert!(!quota.is_complete_bucket_usage_snapshot());
assert_eq!(
read_config(store, DATA_USAGE_OBJ_NAME_PATH.as_str())
.await
.expect("primary should remain readable"),
primary_bytes
);
invalidate_data_usage_snapshot_cache().await;
}
#[test]
fn incomplete_data_usage_is_unavailable_in_server_info() {
let mut buckets = rustfs_madmin::Buckets::default();
-1
View File
@@ -182,7 +182,6 @@ pub(crate) const GET_METADATA_CACHE_REASON_DELETE_MARKER: &str = "delete_marker"
pub(crate) const GET_METADATA_CACHE_REASON_DIST_ERASURE: &str = "dist_erasure";
pub(crate) const GET_METADATA_CACHE_REASON_INCL_FREE_VERSIONS: &str = "incl_free_versions";
pub(crate) const GET_METADATA_CACHE_REASON_INSUFFICIENT_CACHED_QUORUM: &str = "insufficient_cached_quorum";
pub(crate) const GET_METADATA_CACHE_REASON_BODY_LAYOUT_REQUIRED: &str = "body_layout_required";
pub(crate) const GET_METADATA_CACHE_REASON_META_BUCKET: &str = "meta_bucket";
pub(crate) const GET_METADATA_CACHE_REASON_NO_LOCK: &str = "no_lock";
pub(crate) const GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED: &str = "not_found_or_expired";
@@ -12,7 +12,6 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use rustfs_io_metrics::record_object_lock_acquire_failure;
use rustfs_lock::{LockError, NamespaceLockGuard, NamespaceLockWrapper};
use std::time::{Duration, Instant};
use tracing::Span;
@@ -22,26 +21,6 @@ const EVENT_OBJECT_NAMESPACE_LOCK: &str = "object_namespace_lock";
const LOG_COMPONENT_ECSTORE: &str = "ecstore";
const LOG_SUBSYSTEM_OBJECT_LOCK: &str = "object_lock";
pub(crate) fn record_lock_acquire_failure(op: &'static str, bucket: &str, object: &str, mode: &'static str, error: &LockError) {
let resource_class = if bucket == crate::disk::RUSTFS_META_MULTIPART_BUCKET {
"multipart_metadata"
} else if bucket == crate::disk::RUSTFS_META_BUCKET || bucket == ".minio.sys" {
"system_metadata"
} else if object.is_empty() || object == bucket {
"bucket_namespace"
} else {
"object_namespace"
};
let reason = match error {
LockError::Timeout { .. } => "timeout",
LockError::AlreadyLocked { .. } => "contention",
LockError::QuorumNotReached { .. } | LockError::InsufficientNodes { .. } => "quorum",
LockError::Network { .. } | LockError::Internal { .. } => "transport",
_ => "other",
};
record_object_lock_acquire_failure(op, mode, resource_class, reason);
}
/// Diagnostic lifetime only; it never owns or changes a namespace lock.
/// Keep it alongside the real guard when ownership moves to a commit tail.
#[derive(Default)]
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+4 -27
View File
@@ -131,15 +131,6 @@ pub fn build_write_quorum_failure_summary(
fn dominant_error_label(errors: &[Option<Error>], ignored_errs: &[Error], dominant_error: Option<&Error>) -> &'static str {
let Some(dominant_error) = dominant_error else {
// The reducer also returns None when every failure is ignored for
// quorum voting. That case contains no successful nil entry.
if !errors.is_empty() && errors.iter().all(Option::is_some) {
return if errors.iter().all(|error| error.as_ref() == Some(&Error::DiskNotFound)) {
"disk_not_found"
} else {
"all_ignored"
};
}
return "nil_dominated";
};
@@ -396,9 +387,9 @@ mod tests {
}
#[test]
fn test_build_write_quorum_failure_summary_all_missing_is_disk_not_found() {
// Ignoring unavailable disks for quorum reduction must not imply a
// successful nil entry in the diagnostic label.
fn test_build_write_quorum_failure_summary_all_ignored_is_nil_dominated() {
// All disks offline (all DiskNotFound). The dominant error must be None and
// the label must be "nil_dominated" rather than falling back to "other_error".
let errors = vec![
Some(Error::DiskNotFound),
Some(Error::DiskNotFound),
@@ -406,21 +397,7 @@ mod tests {
];
let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
assert_eq!(summary.dominant_error, None);
assert_eq!(summary.achieved, 0);
assert_eq!(summary.dominant_error_label, "disk_not_found");
}
#[test]
fn test_build_write_quorum_failure_summary_all_ignored_has_no_success() {
let errors = [Some(Error::FaultyDisk), Some(Error::DiskNotFound)];
let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
assert_eq!(summary.dominant_error, None);
assert_eq!(summary.achieved, 0);
assert_eq!(summary.dominant_error_label, "all_ignored");
assert_eq!(
reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 2),
Some(Error::ErasureWriteQuorum)
);
assert_eq!(summary.dominant_error_label, "nil_dominated");
}
#[test]
+28
View File
@@ -223,6 +223,12 @@ pub fn rename_std(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()
std::fs::rename(from, to)
}
#[cfg(any(not(windows), test))]
#[tracing::instrument(level = "debug", skip_all)]
pub async fn read_file(path: impl AsRef<Path>) -> io::Result<Vec<u8>> {
fs::read(path.as_ref()).await
}
// Bucket existence cache - reduces statx syscalls for repeated bucket checks
/// Cache for bucket directory existence checks.
@@ -608,6 +614,28 @@ mod tests {
assert_eq!(content, "test content");
}
#[tokio::test]
async fn test_read_file() {
let temp_dir = TempDir::new().unwrap();
let file_path = temp_dir.path().join("test_read.txt");
let test_content = b"This is test content for reading";
tokio::fs::write(&file_path, test_content).await.unwrap();
// Read file
let read_content = read_file(&file_path).await.unwrap();
assert_eq!(read_content, test_content);
}
#[tokio::test]
async fn test_read_file_nonexistent() {
let temp_dir = TempDir::new().unwrap();
let file_path = temp_dir.path().join("nonexistent.txt");
// Should fail for non-existent file
assert!(read_file(&file_path).await.is_err());
}
#[tokio::test]
async fn test_same_file() {
let temp_dir = TempDir::new().unwrap();
-706
View File
@@ -1,706 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Per-local-disk foreground-priority admission for scanner I/O.
use std::collections::VecDeque;
use std::future::Future;
use std::io;
use std::pin::Pin;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use std::task::{Context, Poll};
use std::time::Duration;
use metrics::{counter, gauge, histogram};
use rustfs_rio::ChunkReader;
use tokio::io::{AsyncRead, ReadBuf};
use tokio::sync::Notify;
use tokio::time::Instant;
use tokio_util::sync::CancellationToken;
const BACKGROUND_FOREGROUND_BURST: u32 = 8;
const BACKGROUND_MAX_WAIT: Duration = Duration::from_millis(100);
tokio::task_local! {
static DISK_IO_CONTEXT: DiskIoContext;
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) enum DiskIoClass {
Foreground,
Background,
}
#[derive(Clone, Debug)]
pub(crate) struct DiskIoContext {
pub(crate) class: DiskIoClass,
pub(crate) cancellation: CancellationToken,
pub(crate) admission: Option<Arc<DiskIoAdmission>>,
pub(crate) permit: Option<DiskIoPermit>,
}
/// Mark disk I/O issued by `future` as background work for local-disk admission.
#[doc(hidden)]
pub async fn with_background_disk_io<F: Future>(cancellation: CancellationToken, future: F) -> F::Output {
with_disk_io_context(
DiskIoContext {
class: DiskIoClass::Background,
cancellation,
admission: None,
permit: None,
},
future,
)
.await
}
pub(crate) async fn with_disk_io_context<F: Future>(context: DiskIoContext, future: F) -> F::Output {
DISK_IO_CONTEXT.scope(context, future).await
}
pub(crate) async fn with_disk_io_permit<F: Future>(permit: Option<DiskIoPermit>, future: F) -> F::Output {
let Some(permit) = permit else {
return future.await;
};
let mut context = current_disk_io_context().unwrap_or_else(|| DiskIoContext {
class: DiskIoClass::Foreground,
cancellation: CancellationToken::new(),
admission: None,
permit: None,
});
context.admission = Some(permit.admission());
with_disk_io_context(
DiskIoContext {
permit: Some(permit),
..context
},
future,
)
.await
}
pub(crate) async fn with_optional_disk_io_context<F: Future>(context: Option<DiskIoContext>, future: F) -> F::Output {
match context {
Some(context) => with_disk_io_context(context, future).await,
None => future.await,
}
}
pub(crate) fn current_disk_io_context() -> Option<DiskIoContext> {
DISK_IO_CONTEXT.try_with(Clone::clone).ok()
}
/// Acquire one low-level permit only when the current walker explicitly owns a gate.
/// Ordinary wrapper calls use their outer operation permit instead.
pub(crate) async fn acquire_context_disk_io_permit() -> io::Result<Option<DiskIoPermit>> {
let Some(context) = current_disk_io_context() else {
return Ok(None);
};
let Some(admission) = context.admission else {
return Ok(None);
};
if let Some(permit) = context.permit
&& permit.is_for(&admission)
{
return Ok(Some(permit));
}
match context.class {
DiskIoClass::Foreground => Ok(Some(admission.foreground())),
DiskIoClass::Background => Ok(Some(admission.background(&context.cancellation).await?)),
}
}
#[derive(Debug, Default)]
struct AdmissionState {
foreground_completions: u32,
next_background_ticket: u64,
background_queue: VecDeque<u64>,
background_front_since: Option<Instant>,
}
/// One gate is shared by all handles to a local disk, including reconnects.
/// Foreground I/O never waits on this gate. Background I/O runs exclusively
/// while the disk is idle, or after a bounded foreground burst/wait.
#[derive(Debug)]
pub(crate) struct DiskIoAdmission {
state: Mutex<AdmissionState>,
changed: Notify,
disk: String,
foreground_active: AtomicUsize,
background_active: AtomicBool,
background_waiters: AtomicUsize,
#[cfg(test)]
background_admissions: AtomicUsize,
}
impl DiskIoAdmission {
pub(crate) fn new(disk: impl Into<String>) -> Arc<Self> {
Arc::new(Self {
state: Mutex::new(AdmissionState::default()),
changed: Notify::new(),
disk: disk.into(),
foreground_active: AtomicUsize::new(0),
background_active: AtomicBool::new(false),
background_waiters: AtomicUsize::new(0),
#[cfg(test)]
background_admissions: AtomicUsize::new(0),
})
}
#[cfg(test)]
pub(crate) fn background_admissions_for_tests(&self) -> usize {
self.background_admissions.load(Ordering::Relaxed)
}
#[cfg(test)]
pub(crate) fn background_active_for_tests(&self) -> bool {
self.background_active.load(Ordering::Acquire)
}
#[cfg(test)]
fn foreground_active_for_tests(&self) -> usize {
self.foreground_active.load(Ordering::Acquire)
}
pub(crate) fn foreground(self: &Arc<Self>) -> DiskIoPermit {
self.foreground_active.fetch_add(1, Ordering::AcqRel);
if self.background_waiters.load(Ordering::Acquire) > 0 {
self.record_state();
}
DiskIoPermit {
lease: Arc::new(DiskIoPermitLease {
admission: self.clone(),
class: DiskIoClass::Foreground,
}),
}
}
pub(crate) async fn background(self: &Arc<Self>, cancellation: &CancellationToken) -> std::io::Result<DiskIoPermit> {
let started_at = Instant::now();
let ticket = {
let mut state = self.state.lock().unwrap_or_else(|error| error.into_inner());
let ticket = state.next_background_ticket;
state.next_background_ticket = state.next_background_ticket.wrapping_add(1);
if state.background_queue.is_empty() && !self.background_active.load(Ordering::Acquire) {
state.background_front_since = Some(Instant::now());
}
state.background_queue.push_back(ticket);
self.background_waiters.fetch_add(1, Ordering::Release);
ticket
};
self.record_state();
let mut waiter = BackgroundWaiter {
admission: self.clone(),
ticket,
waiting: true,
};
loop {
let notified = self.changed.notified();
tokio::pin!(notified);
notified.as_mut().enable();
let granted = {
let mut state = self.state.lock().unwrap_or_else(|error| error.into_inner());
let is_front = state.background_queue.front() == Some(&ticket);
let bounded_wait_elapsed = is_front
&& state
.background_front_since
.is_some_and(|front_since| front_since.elapsed() >= BACKGROUND_MAX_WAIT);
if is_front
&& !self.background_active.load(Ordering::Acquire)
&& (self.foreground_active.load(Ordering::Acquire) == 0
|| state.foreground_completions >= BACKGROUND_FOREGROUND_BURST
|| bounded_wait_elapsed)
{
self.background_active.store(true, Ordering::Release);
state.background_queue.pop_front();
state.background_front_since = None;
self.background_waiters.fetch_sub(1, Ordering::AcqRel);
state.foreground_completions = 0;
true
} else {
false
}
};
if granted {
waiter.waiting = false;
#[cfg(test)]
self.background_admissions.fetch_add(1, Ordering::Relaxed);
histogram!("rustfs_disk_io_admission_wait_seconds", "disk" => self.disk.clone(), "class" => "background")
.record(started_at.elapsed().as_secs_f64());
counter!("rustfs_disk_io_admission_total", "disk" => self.disk.clone(), "class" => "background").increment(1);
self.record_state();
return Ok(DiskIoPermit {
lease: Arc::new(DiskIoPermitLease {
admission: self.clone(),
class: DiskIoClass::Background,
}),
});
}
let remaining = {
let state = self.state.lock().unwrap_or_else(|error| error.into_inner());
if state.background_queue.front() == Some(&ticket) {
state
.background_front_since
.map(|front_since| BACKGROUND_MAX_WAIT.saturating_sub(front_since.elapsed()))
} else {
None
}
};
if let Some(remaining) = remaining.filter(|remaining| !remaining.is_zero()) {
tokio::select! {
_ = cancellation.cancelled() => {
return Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "background disk I/O admission cancelled"));
}
_ = tokio::time::sleep(remaining) => {
self.changed.notify_waiters();
}
_ = &mut notified => {}
}
} else {
tokio::select! {
_ = cancellation.cancelled() => {
return Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "background disk I/O admission cancelled"));
}
_ = &mut notified => {}
}
}
}
}
fn record_state(&self) {
let (foreground_active, background_waiters) = {
let state = self.state.lock().unwrap_or_else(|error| error.into_inner());
(
if state.background_queue.is_empty() {
0
} else {
self.foreground_active.load(Ordering::Acquire)
},
state.background_queue.len(),
)
};
let foreground_active = u32::try_from(foreground_active).unwrap_or(u32::MAX);
let background_waiters = u32::try_from(background_waiters).unwrap_or(u32::MAX);
gauge!("rustfs_disk_io_admission_active", "disk" => self.disk.clone(), "class" => "foreground")
.set(f64::from(foreground_active));
gauge!("rustfs_disk_io_admission_active", "disk" => self.disk.clone(), "class" => "background").set(
if self.background_active.load(Ordering::Acquire) {
1.0
} else {
0.0
},
);
gauge!("rustfs_disk_io_admission_waiters", "disk" => self.disk.clone(), "class" => "background")
.set(f64::from(background_waiters));
}
}
struct BackgroundWaiter {
admission: Arc<DiskIoAdmission>,
ticket: u64,
waiting: bool,
}
impl Drop for BackgroundWaiter {
fn drop(&mut self) {
if !self.waiting {
return;
}
let mut state = self.admission.state.lock().unwrap_or_else(|error| error.into_inner());
if let Some(index) = state.background_queue.iter().position(|ticket| *ticket == self.ticket) {
state.background_queue.remove(index);
if index == 0 {
state.background_front_since =
if state.background_queue.is_empty() || self.admission.background_active.load(Ordering::Acquire) {
None
} else {
Some(Instant::now())
};
}
self.admission.background_waiters.fetch_sub(1, Ordering::AcqRel);
}
drop(state);
self.admission.record_state();
self.admission.changed.notify_waiters();
}
}
#[derive(Debug, Clone)]
pub(crate) struct DiskIoPermit {
lease: Arc<DiskIoPermitLease>,
}
impl DiskIoPermit {
pub(crate) fn is_for(&self, admission: &Arc<DiskIoAdmission>) -> bool {
Arc::ptr_eq(&self.lease.admission, admission)
}
fn admission(&self) -> Arc<DiskIoAdmission> {
self.lease.admission.clone()
}
}
#[derive(Debug)]
struct DiskIoPermitLease {
admission: Arc<DiskIoAdmission>,
class: DiskIoClass,
}
pub(crate) fn spawn_blocking_with_disk_io_permit<T, F>(
io_permit: Option<DiskIoPermit>,
operation: F,
) -> tokio::task::JoinHandle<T>
where
T: Send + 'static,
F: FnOnce() -> T + Send + 'static,
{
tokio::task::spawn_blocking(move || {
let _io_permit = io_permit;
operation()
})
}
type AdmissionWait = Pin<Box<dyn Future<Output = io::Result<DiskIoPermit>> + Send>>;
impl Drop for DiskIoPermitLease {
fn drop(&mut self) {
match self.class {
DiskIoClass::Foreground => {
self.admission.foreground_active.fetch_sub(1, Ordering::AcqRel);
let record_state = self.admission.background_waiters.load(Ordering::Acquire) > 0;
if record_state {
let mut state = self.admission.state.lock().unwrap_or_else(|error| error.into_inner());
if !state.background_queue.is_empty() {
state.foreground_completions = state.foreground_completions.saturating_add(1);
}
}
if record_state {
self.admission.record_state();
self.admission.changed.notify_waiters();
}
}
DiskIoClass::Background => {
let mut state = self.admission.state.lock().unwrap_or_else(|error| error.into_inner());
self.admission.background_active.store(false, Ordering::Release);
state.background_front_since = if state.background_queue.is_empty() {
None
} else {
Some(Instant::now())
};
drop(state);
self.admission.record_state();
self.admission.changed.notify_waiters();
}
}
}
}
pub(crate) struct AdmissionReader<R> {
inner: R,
admission: Arc<DiskIoAdmission>,
context: DiskIoContext,
wait: Mutex<Option<AdmissionWait>>,
permit: Option<DiskIoPermit>,
}
impl<R> AdmissionReader<R> {
pub(crate) fn new(inner: R, admission: Arc<DiskIoAdmission>, context: DiskIoContext) -> Self {
Self {
inner,
admission,
context,
wait: Mutex::new(None),
permit: None,
}
}
fn poll_permit(&mut self, cx: &mut Context<'_>) -> Poll<io::Result<()>> {
if self.permit.is_some() {
return Poll::Ready(Ok(()));
}
if self.context.class == DiskIoClass::Foreground {
self.permit = Some(self.admission.foreground());
return Poll::Ready(Ok(()));
}
let mut wait = self.wait.lock().unwrap_or_else(|error| error.into_inner());
if wait.is_none() {
let admission = self.admission.clone();
let cancellation = self.context.cancellation.clone();
*wait = Some(Box::pin(async move { admission.background(&cancellation).await }));
}
let Some(wait_future) = wait.as_mut() else {
return Poll::Ready(Err(io::Error::other("background disk I/O admission waiter is missing")));
};
let result = wait_future.as_mut().poll(cx);
match result {
Poll::Ready(Ok(permit)) => {
wait.take();
drop(wait);
self.permit = Some(permit);
Poll::Ready(Ok(()))
}
Poll::Ready(Err(error)) => {
wait.take();
Poll::Ready(Err(error))
}
Poll::Pending => Poll::Pending,
}
}
fn finish_poll<T>(&mut self, result: Poll<io::Result<T>>) -> Poll<io::Result<T>> {
if result.is_ready() {
self.permit = None;
}
result
}
}
impl<R: AsyncRead + Unpin> AsyncRead for AdmissionReader<R> {
fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
match self.poll_permit(cx) {
Poll::Ready(Ok(())) => {}
Poll::Ready(Err(error)) => return Poll::Ready(Err(error)),
Poll::Pending => return Poll::Pending,
}
let result = Pin::new(&mut self.inner).poll_read(cx, buf);
self.finish_poll(result)
}
}
impl<R: ChunkReader + Unpin> ChunkReader for AdmissionReader<R> {
fn poll_read_chunk(mut self: Pin<&mut Self>, cx: &mut Context<'_>, max: usize) -> Poll<io::Result<Option<bytes::Bytes>>> {
match self.poll_permit(cx) {
Poll::Ready(Ok(())) => {}
Poll::Ready(Err(error)) => return Poll::Ready(Err(error)),
Poll::Pending => return Poll::Pending,
}
let result = Pin::new(&mut self.inner).poll_read_chunk(cx, max);
self.finish_poll(result)
}
}
pub(crate) struct BoxedChunkReader(pub(crate) rustfs_rio::ChunkReaderBox);
impl AsyncRead for BoxedChunkReader {
fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
Pin::new(&mut self.0).poll_read(cx, buf)
}
}
impl ChunkReader for BoxedChunkReader {
fn poll_read_chunk(self: Pin<&mut Self>, cx: &mut Context<'_>, max: usize) -> Poll<io::Result<Option<bytes::Bytes>>> {
Pin::new(self.get_mut().0.as_mut()).poll_read_chunk(cx, max)
}
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
struct BlockingWorker(tokio::task::JoinHandle<()>);
async fn wait_for_background_waiters(admission: &DiskIoAdmission, expected: usize) {
tokio::time::timeout(Duration::from_secs(1), async {
loop {
let waiters = admission
.state
.lock()
.unwrap_or_else(|error| error.into_inner())
.background_queue
.len();
if waiters == expected {
return;
}
tokio::task::yield_now().await;
}
})
.await
.expect("background admission waiter should reach expected count");
}
async fn wait_for_foreground_active(admission: &DiskIoAdmission, expected: usize) {
tokio::time::timeout(Duration::from_secs(1), async {
loop {
let active = admission.foreground_active.load(Ordering::Acquire);
if active == expected {
return;
}
tokio::task::yield_now().await;
}
})
.await
.expect("foreground disk read should retain its permit while pending");
}
#[tokio::test]
async fn foreground_bypasses_background_and_background_progress_is_bounded() {
let admission = DiskIoAdmission::new("disk-a");
let foreground = admission.foreground();
let background_admission = admission.clone();
let cancellation = CancellationToken::new();
let background = tokio::spawn(async move { background_admission.background(&cancellation).await });
wait_for_background_waiters(&admission, 1).await;
for _ in 0..BACKGROUND_FOREGROUND_BURST {
drop(admission.foreground());
}
let background_permit = tokio::time::timeout(Duration::from_secs(1), background)
.await
.expect("background admission should progress after the foreground burst")
.expect("background waiter task should complete")
.expect("background admission should not be cancelled");
assert_eq!(
admission.foreground_active.load(Ordering::Acquire),
1,
"foreground operations remain admitted while one background operation runs"
);
drop(background_permit);
drop(foreground);
}
#[tokio::test]
async fn cancellation_removes_a_queued_background_waiter() {
let admission = DiskIoAdmission::new("disk-a");
let background = admission
.background(&CancellationToken::new())
.await
.expect("first background operation should be admitted");
let cancellation = CancellationToken::new();
let waiter_admission = admission.clone();
let waiter_cancellation = cancellation.clone();
let waiter = tokio::spawn(async move { waiter_admission.background(&waiter_cancellation).await });
wait_for_background_waiters(&admission, 1).await;
cancellation.cancel();
let error = tokio::time::timeout(Duration::from_secs(1), waiter)
.await
.expect("cancelled background waiter should stop")
.expect("background waiter task should complete")
.expect_err("cancelled admission must not start disk I/O");
assert_eq!(error.kind(), io::ErrorKind::Interrupted);
assert_eq!(
admission
.state
.lock()
.unwrap_or_else(|error| error.into_inner())
.background_queue
.len(),
0
);
drop(background);
}
#[tokio::test]
async fn background_admission_is_isolated_per_disk() {
let disk_a = DiskIoAdmission::new("disk-a");
let disk_b = DiskIoAdmission::new("disk-b");
let _disk_a_permit = disk_a
.background(&CancellationToken::new())
.await
.expect("disk A background operation should be admitted");
let _disk_b_permit = tokio::time::timeout(Duration::from_secs(1), disk_b.background(&CancellationToken::new()))
.await
.expect("disk A must not block disk B")
.expect("disk B background operation should be admitted");
}
#[tokio::test]
async fn streamed_read_holds_admission_until_disk_read_resolves() {
let admission = DiskIoAdmission::new("disk-a");
let (mut writer, reader) = tokio::io::duplex(8);
let mut reader = AdmissionReader::new(
reader,
admission.clone(),
DiskIoContext {
class: DiskIoClass::Foreground,
cancellation: CancellationToken::new(),
admission: None,
permit: None,
},
);
let read = tokio::spawn(async move {
let mut bytes = [0; 4];
reader
.read_exact(&mut bytes)
.await
.expect("streamed disk read should complete");
bytes
});
wait_for_foreground_active(&admission, 1).await;
writer.write_all(b"rust").await.expect("test bytes should be written");
assert_eq!(read.await.expect("reader task should finish"), *b"rust");
wait_for_foreground_active(&admission, 0).await;
}
#[tokio::test]
async fn blocking_worker_keeps_permit_after_caller_timeout() {
let admission = DiskIoAdmission::new("disk-a");
let (started_tx, started_rx) = tokio::sync::oneshot::channel();
let worker = spawn_blocking_with_disk_io_permit(Some(admission.foreground()), move || {
started_tx.send(()).expect("worker start should be observed");
std::thread::sleep(Duration::from_millis(100));
});
started_rx.await.expect("blocking worker should start");
assert!(tokio::time::timeout(Duration::from_millis(10), worker).await.is_err());
assert_eq!(admission.foreground_active_for_tests(), 1);
wait_for_foreground_active(&admission, 0).await;
}
#[tokio::test]
async fn cloned_permit_remains_active_until_all_workers_finish() {
let admission = DiskIoAdmission::new("disk-a");
let permit = admission.foreground();
let worker_permit = permit.clone();
drop(permit);
assert_eq!(admission.foreground_active_for_tests(), 1);
drop(worker_permit);
assert_eq!(admission.foreground_active_for_tests(), 0);
}
#[tokio::test]
async fn scoped_foreground_permit_is_inherited_by_blocking_worker() {
let admission = DiskIoAdmission::new("disk-a");
let permit = admission.foreground();
let worker = with_disk_io_permit(Some(permit), async {
let worker_permit = acquire_context_disk_io_permit()
.await
.expect("context permit should be available")
.expect("scoped permit should be inherited");
let (started_tx, started_rx) = tokio::sync::oneshot::channel();
let worker = spawn_blocking_with_disk_io_permit(Some(worker_permit), move || {
started_tx.send(()).expect("worker start should be observed");
std::thread::sleep(Duration::from_millis(100));
});
started_rx.await.expect("blocking worker should start");
BlockingWorker(worker)
})
.await;
assert_eq!(admission.foreground_active_for_tests(), 1);
assert!(tokio::time::timeout(Duration::from_millis(10), worker.0).await.is_err());
wait_for_foreground_active(&admission, 0).await;
}
}
File diff suppressed because it is too large Load Diff
+17 -75
View File
@@ -47,33 +47,6 @@ use tokio::fs;
use tracing::{info, warn};
use uuid::Uuid;
fn transitioned_old_current_source(
dst_meta_existed: bool,
dst_meta_unparsable: bool,
xlmeta: &FileMeta,
volume: &str,
path: &str,
version_id: Uuid,
) -> (bool, Option<FileInfo>) {
if dst_meta_unparsable {
return (false, None);
}
if !dst_meta_existed {
return (true, None);
}
let Ok(old_current) = xlmeta.into_fileinfo(volume, path, "", false, false, true) else {
return (false, None);
};
if old_current.version_id.unwrap_or_default() != version_id {
return (true, None);
}
if old_current.transition_status == rustfs_filemeta::TRANSITION_COMPLETE {
(true, Some(old_current))
} else {
(true, None)
}
}
/// Hold later repair publications after admitting one baseline object. The
/// fixture arms this on one replacement disk before rejoining the cluster.
#[cfg(feature = "e2e-test-hooks")]
@@ -241,7 +214,7 @@ pub(super) async fn lock_rename_commit_directories(
publication_root: &os::PublicationRoot,
mutation_lease: Arc<os::NamespaceMutationLease>,
) -> Result<os::RenameCommitGuard> {
#[cfg(any(windows, target_os = "linux"))]
#[cfg(windows)]
let result = {
let source_parent = source_parent.to_path_buf();
let destination_parent = destination_parent.to_path_buf();
@@ -249,7 +222,7 @@ pub(super) async fn lock_rename_commit_directories(
let publication_root = publication_root.clone();
os::run_blocking_namespace_operation(mutation_lease, move || {
let result = os::prepare_rename_commit_guard(&source_parent, &destination_parent, &base_dir, &publication_root);
#[cfg(all(test, windows))]
#[cfg(test)]
if result.is_ok() {
run_destination_commit_directory_preparation(&destination_parent);
}
@@ -257,7 +230,7 @@ pub(super) async fn lock_rename_commit_directories(
})
.await
};
#[cfg(not(any(windows, target_os = "linux")))]
#[cfg(not(windows))]
let result = {
let _ = mutation_lease;
os::prepare_rename_commit_guard(source_parent, destination_parent, base_dir, publication_root)
@@ -271,12 +244,12 @@ pub(super) async fn lock_rename_commit_directories(
result.map_err(to_file_error).map_err(DiskError::from)
}
async fn prepare_rename_destination_metadata(
async fn read_rename_destination_metadata(
file_path: &Path,
rename_commit_guard: &os::RenameCommitGuard,
mutation_lease: Arc<os::NamespaceMutationLease>,
) -> Result<Option<Bytes>> {
#[cfg(any(windows, target_os = "linux"))]
#[cfg(windows)]
let result = {
let file_path = file_path.to_path_buf();
let rename_commit_guard = rename_commit_guard.clone();
@@ -285,10 +258,10 @@ async fn prepare_rename_destination_metadata(
})
.await
};
#[cfg(not(any(windows, target_os = "linux")))]
#[cfg(not(windows))]
let _ = (rename_commit_guard, mutation_lease);
#[cfg(not(any(windows, target_os = "linux")))]
let result = match fs::read(file_path).await {
#[cfg(not(windows))]
let result = match super::super::fs::read_file(file_path).await {
Ok(data) => Ok(Some(data)),
Err(err) if err.kind() == ErrorKind::NotFound => Ok(None),
Err(err) => Err(err),
@@ -425,7 +398,7 @@ impl LocalDisk {
// xl.meta path
let src_file_path = self.io_get_object_path(src_volume, format!("{}/{}", src_path, STORAGE_FORMAT_FILE).as_str())?;
let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
let dst_file_path = self.io_get_object_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
// data_dir path
let has_data_dir_path = {
@@ -480,7 +453,8 @@ impl LocalDisk {
fs::create_dir_all(src_file_parent).await.map_err(to_file_error)?;
}
// Acquire the common trees before reading destination metadata. On
// Windows this pins the object directory identity across publication.
// Windows this pins the object directory identity across metadata
// preparation, data publication, rollback backup, and final commit.
let rename_commit_guard = lock_rename_commit_directories(
src_file_parent,
dst_file_parent,
@@ -498,12 +472,9 @@ impl LocalDisk {
source_volume_dir: &src_volume_dir,
data_paths: has_data_dir_path.as_ref(),
};
let has_dst_buf = prepare_rename_destination_metadata(
commit.destination_metadata,
&commit.directory_guard,
commit.mutation_lease.clone(),
)
.await?;
let has_dst_buf =
read_rename_destination_metadata(commit.destination_metadata, &commit.directory_guard, commit.mutation_lease.clone())
.await?;
if no_inline {
// Non-inline: read xl.meta, parse, write, rename data dir, rename xl.meta
@@ -537,14 +508,6 @@ impl LocalDisk {
}
let version_id = fi.version_id.unwrap_or_default();
let (old_current_source_checked, old_current_source) = transitioned_old_current_source(
has_dst_buf.is_some(),
dst_meta_unparsable,
&xlmeta,
dst_volume,
dst_path,
version_id,
);
let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok();
let rollback_data_dir = has_old_data_dir.or_else(|| {
@@ -835,9 +798,7 @@ impl LocalDisk {
&& let Some(parent) = dst_file_path.parent()
{
let fsync_started = rustfs_io_metrics::put_stage_timer();
if let Err(err) =
os::fsync_commit_directory(parent, &commit.directory_guard, commit.mutation_lease.clone(), None).await
{
if let Err(err) = os::fsync_dst_dir_group_commit(parent, Some(commit.mutation_lease.clone())).await {
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC,
fsync_started,
@@ -936,8 +897,6 @@ impl LocalDisk {
cleanup_data_dir: has_old_data_dir,
sign: version_signature,
old_current_size,
old_current_source_checked,
old_current_source,
})
} else {
// Inline metadata preparation is blocking. The transaction lease is
@@ -951,8 +910,6 @@ impl LocalDisk {
None
};
let dst_path_for_failpoint = dst_path.to_string();
let dst_volume_for_source = dst_volume.to_string();
let dst_path_for_source = dst_path.to_string();
#[cfg(windows)]
let source_parent = src_file_parent.to_path_buf();
let rename_commit_guard_for_preparation = commit.directory_guard.clone();
@@ -997,14 +954,6 @@ impl LocalDisk {
};
let version_id = fi.version_id.unwrap_or_default();
let (old_current_source_checked, old_current_source) = transitioned_old_current_source(
has_dst_buf.is_some(),
dst_meta_unparsable,
&xlmeta,
&dst_volume_for_source,
&dst_path_for_source,
version_id,
);
let old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok();
let rollback_data_dir = old_data_dir.or_else(|| {
@@ -1054,8 +1003,6 @@ impl LocalDisk {
old_data_dir,
version_signature,
old_current_size,
old_current_source_checked,
old_current_source,
staged_rollback_path,
has_dst_buf.is_none(),
prepared_metadata_source,
@@ -1075,8 +1022,6 @@ impl LocalDisk {
cleanup_data_dir,
version_signature,
old_current_size,
old_current_source_checked,
old_current_source,
mut local_rollback_path,
destination_was_absent,
prepared_metadata_source,
@@ -1202,11 +1147,10 @@ impl LocalDisk {
&& let Some(dst_parent) = dst_file_path.parent()
{
let fsync_started = rustfs_io_metrics::put_stage_timer();
if let Err(err) = os::fsync_commit_directory(
if let Err(err) = os::fsync_dst_dir_group_commit_or_namespace_file_sync_limit(
dst_parent,
&commit.directory_guard,
commit.mutation_lease.clone(),
Some(admission),
admission,
)
.await
{
@@ -1318,8 +1262,6 @@ impl LocalDisk {
cleanup_data_dir,
sign: version_signature,
old_current_size,
old_current_source_checked,
old_current_source,
})
}
}

Some files were not shown because too many files have changed in this diff Show More