Compare commits

..

34 Commits

Author SHA1 Message Date
overtrue 5983594da9 feat(capture): add explicit schema-bound checkpoint capabilities 2026-10-11 21:28:05 +08:00
overtrue dfa5d50565 feat(capture): persist first-decision provenance 2026-10-11 18:58:12 +08:00
overtrue a902886737 feat(capture): atomically migrate legacy Created history 2026-10-11 16:39:10 +08:00
overtrue 8e53b87421 test(capture): specify atomic populated native migration 2026-10-11 16:39:10 +08:00
overtrue f7631e91b1 feat(capture): stream an owned coherent snapshot view 2026-10-11 16:30:12 +08:00
overtrue f9bc4e921f test(capture): specify coherent snapshot view ownership 2026-10-11 16:30:12 +08:00
overtrue 781584307a feat(capture): apply committed native entries on the ordered worker 2026-10-11 15:35:02 +08:00
overtrue 59d641a5fe docs(capture): define recoverable checkpoint and purge contracts 2026-10-11 15:00:38 +08:00
overtrue c618b57870 feat(capture): stream snapshot rows with a permanent failure fence 2026-10-11 14:44:05 +08:00
overtrue bbe1d54098 feat(capture): add bounded streaming snapshot codec 2026-10-11 14:21:30 +08:00
overtrue 9e8ea4dcdb feat(capture-authority): serialize native async writes 2026-10-11 13:35:15 +08:00
overtrue 63cf8e2f2f feat(capture-authority): add native OpenRaft log reader 2026-10-11 13:01:43 +08:00
overtrue b7fe40f4b3 feat(capture): support complete native log range bounds 2026-10-11 12:28:23 +08:00
overtrue 5b872b8e56 feat(capture): apply native entries atomically with durable replay 2026-10-11 11:50:32 +08:00
overtrue ca5be1ef95 feat(capture): persist native Raft entries in explicitly enabled stores 2026-10-11 05:49:17 +08:00
overtrue 9373e8fa3e Bind capture commands and responses to native Raft entries 2026-10-11 03:45:17 +08:00
overtrue cb32965522 feat(capture): preserve exact raft values and membership codec 2026-10-11 02:04:47 +08:00
overtrue fb6cf49a67 feat(capture): validate peer metadata and CNOD codec 2026-10-11 01:25:21 +08:00
overtrue 8c97049e68 build(capture): pin OpenRaft and guard resolved features 2026-10-11 00:56:30 +08:00
overtrue 17d4fe27cc feat(capture): apply committed Created decisions atomically 2026-10-10 23:35:52 +08:00
overtrue fb0c7972b2 feat(capture): persist committed log boundary 2026-10-10 23:10:28 +08:00
overtrue eba7967755 feat(capture): persist atomic log suffix truncation 2026-10-10 22:59:26 +08:00
overtrue 451c1c31ad feat(capture): add durable local authority storage 2026-10-10 22:33:44 +08:00
overtrue 8780dd670e test(ecstore): verify publication across process kills 2026-10-10 20:19:46 +08:00
overtrue f29bf2c902 fix(iam): preserve empty identity read failures 2026-10-10 11:37:07 +08:00
overtrue fd03436090 feat(ecstore): expose unbounded preserve-empty config reads 2026-10-10 11:10:27 +08:00
overtrue 1f1e7b1c97 fix(iam): preserve identity decryption errors 2026-10-10 10:37:15 +08:00
overtrue 45a952609e test(iam): add real identity storage fixtures 2026-10-10 10:01:03 +08:00
overtrue 323190a95a test(test-utils): support caller-owned instance contexts 2026-10-10 09:43:01 +08:00
overtrue 38cdda9292 fix(ecstore): expose config reader without private bounds 2026-10-10 09:11:47 +08:00
overtrue 85c84fa73d feat(policy): add dormant principal incarnation records 2026-10-10 08:12:14 +08:00
overtrue 51402087c5 feat(ecstore): expose bounded config snapshot reader 2026-10-10 08:03:27 +08:00
overtrue a994984fe2 build(policy): prepare identity incarnation UUID dependency 2026-10-10 06:57:51 +08:00
overtrue 48eb0e402c test(iam): use identity constructor in existing fixtures 2026-10-10 06:52:04 +08:00
404 changed files with 22549 additions and 51982 deletions
@@ -45,13 +45,11 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
- `GHSA-9wgh-5r68-49p7`: substituted policy-variable values could recreate `${...}` and loop inside a single resolution pass, never reaching the outer iteration limit. Lesson: bound substitutions within each pass and test cycles formed across otherwise literal resolver values; confirm whether attacker-controlled claims can reach those values before assigning remote impact.
### S3 object actions, copy, multipart, and upload policy validation
- `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
- `GHSA-cg33-pxfp-qcj3` and `GHSA-65wq-gjw8-xr9w`: `HeadObject` and object-tagging paths authorized current-object actions while operating on an explicit historical `versionId`. Lesson: select the corresponding version action for metadata and tag reads and writes too; audit every operation that accepts a version ID.
- `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
@@ -112,7 +110,6 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
- `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
- `GHSA-q23w-fp7p-472v`: `PutObjectRetention` policy conditions read `x-amz-object-lock-*` headers while the handler applied mode and date from the XML body. Lesson: authorize the parsed retention that will be stored, or reject mismatched header/body values; compare retention dates as instants so offsets cannot change the applied bound.
### Serde deserialization and input validation
@@ -146,13 +143,13 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
- Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, multipart-copy source, `HeadObject`, and tag read/write cases.
- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
- Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, cyclic variable substitutions, plugin mode, and built-in policy mode.
- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. For `PutObjectRetention`, test conflicting header/body values and offset dates against policy bounds and read-back retention.
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions.
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=ad81c65dcca3a3994c7d62562f477777d7d07fa5aa9daa799741365ac9e5e886
sha256-linux=a5f013b21fed4162e6bc33077841f0f07b3ffbbbb665ddbce4e77a2513c34d76
sha256-darwin=98dff0961c6eb21d6128bfd2d4a6100a624ddf1fb7db05dba360a5d3d0a310c2
sha256-linux=890f5925b87e1bf3710c356253f3b2276d901e156df9ccc04ec9315374de6d22
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=ce7b419faf1ff79840dfc6325d31ee8d1ca5eab85125b94d0bcd58cdf07f0f76
sha256-linux=5e9e3d701ee873c13fb9094575f6889779c6078b1107624bb57663bef9b9d21f
sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1
+1 -1
View File
@@ -1 +1 @@
sha256=bf77250d94604db7ba094c5d63bf8a284beb872983871bcfd0dc02e46ff53326
sha256=ba8f392b617cf9f2338142233a417e43927b28096635e628233ce8a60cf49d10
-5
View File
@@ -70,11 +70,6 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
@echo "📦 Checking s3s footprint ratchet..."
./scripts/check_s3s_footprint.sh
.PHONY: test-count-ratchet-check
test-count-ratchet-check: ## Check no workspace crate lost tests or gained #[ignore] against the committed baseline
@echo "🧪 Checking per-crate test-count ratchet..."
./scripts/check_test_count_ratchet.sh
.PHONY: fips-wording-check
fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
@echo "📣 Checking cryptographic capability wording guard..."
+2 -2
View File
@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
./scripts/check_no_planning_docs.sh
.PHONY: pre-commit
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY: pre-pr
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!"
.PHONY: dev-check
-3
View File
@@ -50,11 +50,8 @@ script-tests: ## Run shell script tests
./scripts/test_fuzz_runner.sh
./scripts/test_python_bin.sh
./scripts/check_embedded_secrets.sh --self-test
./scripts/check_s3s_footprint.sh --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
-9
View File
@@ -589,14 +589,6 @@ test-group = 'ecstore-serial-flaky'
# the PR-lane e2e coverage for scanner usage snapshots consumed by quota and
# admin surfaces. It uses the same single-node, random-port, isolated-temp-dir
# fixture as the existing smoke modules.
#
# Dual-stack differential (rustfs/backlog#2740): `dual_stack_differential::`
# starts two single-node servers from the same binary, one per
# RUSTFS_S3_STACK value, on random ports with isolated temp dirs, and runs one
# SDK script covering every operation of `impl S3 for FS` against both. Its
# comparator unit tests need no server. It meets the admission criteria
# unchanged; the module clause admits a new differential test without another
# filter edit.
[profile.e2e-smoke]
default-filter = """
package(e2e_test) & (
@@ -604,7 +596,6 @@ default-filter = """
| test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
| test(/^reliant::lifecycle::/)
| test(/^reliant::tiering::/)
| test(/^dual_stack_differential::/)
| test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
| test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
)
-37
View File
@@ -1,37 +0,0 @@
# Files that may still reference s3s (`s3s::` or `s3s_sigv4`) when Phase 1 of
# the gateway migration ends (rustfs/backlog#2734 task T0.6, rustfs/backlog#2741).
# Consumed only by `scripts/check_s3s_footprint.sh --mode allowlist`.
#
# One entry per line: a repository-relative file path or a git `:(glob)`
# pathspec. `**` matches across directories, `*` stays within one path
# component, and a directory entry matches everything under it. Blank lines
# and lines starting with `#` are ignored; an entry may not contain whitespace,
# so a note belongs on its own comment line.
#
# Every Rust file listed by `git grep -l '\bs3s\(::\|_sigv4\)' -- '*.rs'` that
# no entry matches fails the allowlist mode. Task T1.9 switches CI to that
# mode; until then `--dry-run` prints the files still outside this list, which
# is the Phase 1 tracking list.
#
# Never add crates/** other than crates/e2e_test: the engine crates leave s3s
# entirely, and the script rejects such an entry. Only the s3s-facing edge of
# the rustfs binary and the e2e oracle keep it.
# rustfs binary: the s3s-facing edge that survives Phase 1
rustfs/src/storage/ecfs.rs
rustfs/src/storage/s3_compat/**
rustfs/src/storage/access.rs
rustfs/src/auth.rs
rustfs/src/error.rs
rustfs/src/storage_api.rs
rustfs/src/app/storage_api.rs
rustfs/src/admin/router.rs
rustfs/src/app/metadata_route.rs
rustfs/src/server/http.rs
rustfs/src/server/layer.rs
rustfs/src/server/hybrid.rs
rustfs/src/protocols/client.rs
# e2e oracle: the test target that verifies S3 behaviour against s3s
crates/e2e_test/src/fake_s3_target/**
crates/e2e_test/src/**
-17
View File
@@ -1,17 +0,0 @@
# Exact baseline for the direct s3s footprint, consumed only by
# scripts/check_s3s_footprint.sh (rustfs/backlog#1677; rustfs/backlog#2734 T0.4).
#
# Each value is the exact measurement of the tree, not a ceiling. CI fails when a
# measured counter differs from its line here in either direction:
# - a larger measurement means new code widened the s3s surface: route it
# through the gateway abstractions instead; never raise a value here;
# - a smaller measurement means the PR shrank the surface: lower the matching
# value to the measured number in the same PR.
#
# files Rust files referencing s3s paths, repo-wide minus crates/e2e_test
# s3_error_lines lines holding s3_error!, same scope, minus those the script
# header's rule clears (rustfs_s3_types's macro); fails closed
# ecstore_files Rust files referencing s3s paths under crates/ecstore/src
files=194
s3_error_lines=1566
ecstore_files=36
-63
View File
@@ -1,63 +0,0 @@
# Per-crate test-count baseline consumed by scripts/check_test_count_ratchet.sh
# (rustfs/backlog#2734 section 1 rule 3; T0.3 in rustfs/backlog#2738).
#
# One row per workspace member: <package>, tests, ignored, tab-separated.
# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]
# attribute lines under the package directory; ignored counts #[ignore] and
# #[ignore = "..."]. Both columns are exact: CI fails when a crate's tests fall
# below or rise above its row, or when its ignored count changes. Raise a row in
# the PR that adds tests; lowering one is a reviewed statement that tests were
# intentionally removed. Regenerate with:
# scripts/check_test_count_ratchet.sh --update
e2e_test 898 18
rustfs 5585 63
rustfs-audit 70 0
rustfs-checksums 26 0
rustfs-common 25 0
rustfs-concurrency 27 0
rustfs-config 43 0
rustfs-connect-symbol-catalog 6 0
rustfs-credentials 24 0
rustfs-crypto 67 0
rustfs-data-usage 54 0
rustfs-ecstore 6443 1
rustfs-extension-schema 23 0
rustfs-filemeta 306 0
rustfs-heal 839 3
rustfs-heal-contracts 6 0
rustfs-iam 388 0
rustfs-io-core 49 0
rustfs-io-metrics 125 0
rustfs-keystone 29 0
rustfs-kms 715 25
rustfs-license 2 0
rustfs-lifecycle 152 0
rustfs-lock 135 0
rustfs-log-analyzer 89 0
rustfs-madmin 209 0
rustfs-notify 136 0
rustfs-object-capacity 92 0
rustfs-object-data-cache 140 0
rustfs-obs 366 0
rustfs-policy 681 0
rustfs-protocols 736 1
rustfs-protos 63 0
rustfs-replication 200 0
rustfs-rio 200 0
rustfs-rio-v2 34 6
rustfs-s3-client 65 0
rustfs-s3-ops 8 0
rustfs-s3-types 97 0
rustfs-s3select-api 206 0
rustfs-s3select-query 188 2
rustfs-scanner 997 20
rustfs-scanner-metrics 81 0
rustfs-security-governance 20 0
rustfs-signer 31 0
rustfs-storage-api 31 0
rustfs-targets 603 26
rustfs-test-utils 1 0
rustfs-tls-runtime 43 0
rustfs-trusted-proxies 68 0
rustfs-utils 217 0
rustfs-zip 30 0
-4
View File
@@ -86,10 +86,6 @@ runs:
shell: bash
run: ./scripts/check_s3s_footprint.sh
- name: Check per-crate test-count ratchet
shell: bash
run: ./scripts/check_test_count_ratchet.sh
- name: Check cryptographic capability wording
shell: bash
run: ./scripts/check_fips_wording.sh
+1 -1
View File
@@ -99,7 +99,7 @@ env:
RUST_BACKTRACE: 1
jobs:
# Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary,
# Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
# e2e-tests, e2e-full.
warm-ci-dev:
name: Warm ci-dev
+89 -146
View File
@@ -16,9 +16,7 @@ name: Continuous Integration
on:
push:
# gateway/integration receives the daily merge of main as a direct push
# (rustfs/backlog#2734), so its pushes must prove the result like main's.
branches: [ main, gateway/integration ]
branches: [ main ]
paths-ignore:
- "**.md"
- "docs/**"
@@ -170,10 +168,13 @@ jobs:
echo "started_at=$(date --utc --iso-8601=seconds)"
} > artifacts/test-and-lint/run-metadata.txt
# Keep full Clippy validation on main, merge groups, and scheduled/manual
# runs. PRs retain the complete default test suite and compile-free checks.
# Clippy runs before the test pass: lint failures are the most common
# CI-only breakage and should surface in minutes, not after 20+ minutes
# of tests.
# Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
# experiment, since --all-targets is check-only for workspace members and
# never links the ~100 test binaries the limit exists to throttle.
- name: Run clippy lints
if: github.event_name != 'pull_request'
run: |
./scripts/ci/resource_sampler.sh start clippy
trap './scripts/ci/resource_sampler.sh stop' EXIT
@@ -282,49 +283,6 @@ jobs:
- name: Verify rebalance/decommission migration proofs
run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
# The workspace already compiled these ignored tests. Keep the same Cargo
# package selection to reuse their binaries, then run them serially in
# separate processes because they share fixed ports and global state.
# test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
# its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
# disk_index 0), not an EC metadata-distribution issue.
# restore_object_usecase_reports_ongoing_conflict_and_completion was
# re-enabled by backlog#1304 (restore accepts serialize on a short CAS
# guard; the copy-back no longer holds the #4877 whole-copy-back lock,
# so the mid-restore ongoing read and fast 409 rejection it asserts are
# the implemented contract).
- name: Run ignored ILM integration tests serially
if: needs.classify-changes.outputs.mode == 'full'
env:
# Match the preceding workspace build and reuse its feature graph.
CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
run: |
mkdir -p artifacts/ilm-integration
set +e
NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
cargo nextest run --profile default -j1 --run-ignored ignored-only \
--all --exclude e2e_test \
-E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
--status-level all --final-status-level all \
2>&1 | tee artifacts/ilm-integration/nextest.log
status=${PIPESTATUS[0]}
{
echo "exit_status=${status}"
echo "finished_at=$(date --utc --iso-8601=seconds)"
echo
echo "Remaining test-related processes:"
pgrep -af 'cargo|nextest|target/.*/deps/' || true
} > artifacts/ilm-integration/diagnostics.txt
exit "${status}"
- name: Upload ILM test diagnostics
if: always() && needs.classify-changes.outputs.mode == 'full'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
path: artifacts/ilm-integration
retention-days: 3
# This gate builds into a fresh target directory to isolate the E2E root.
# Give its cold build a separate budget from workspace tests and migration proofs.
offline-enrollment-e2e:
@@ -356,11 +314,83 @@ jobs:
- name: Check offline enrollment E2E root boundary
run: ./scripts/check_offline_enrollment_e2e.sh
# Dedicated serial lane for the ILM / lifecycle integration tests. These tests
# drive the object layer through process-global singletons (the GLOBAL_ENV
# ECStore, the global tier-config manager, background-expiry workers) and bind
# fixed ports (9002 for the scanner suite, 9003 for the app suite), so they are
# marked #[ignore] to keep them out of the default parallel `cargo nextest run
# --all` pass. Run them here explicitly with `--run-ignored ignored-only` and
# `-j1` so no two of them share global state or race for a port at once.
# serial_test's #[serial] does NOT serialize across nextest's process-per-test
# boundary (see .config/nextest.toml); `-j1` is what actually serializes them.
# See rustfs/backlog#1148 (ilm-1) and #1155.
test-ilm-integration-serial:
name: ILM Integration (serial)
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
# test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
# its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
# disk_index 0), not an EC metadata-distribution issue.
# restore_object_usecase_reports_ongoing_conflict_and_completion was
# re-enabled by backlog#1304 (restore accepts serialize on a short CAS
# guard; the copy-back no longer holds the #4877 whole-copy-back lock,
# so the mid-restore ongoing read and fast 409 rejection it asserts are
# the implemented contract).
- name: Run ignored ILM integration tests serially
env:
# Match the measured Test and Lint link budget. The default exposed
# all 14 pod CPUs and a cold cache spent the full 80m compiling
# without starting one ILM test (main run 32982910990).
CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
run: |
mkdir -p artifacts/ilm-integration
set +e
NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
cargo nextest run -j1 --run-ignored ignored-only \
-p rustfs-scanner -p rustfs \
-E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
--status-level all --final-status-level all \
2>&1 | tee artifacts/ilm-integration/nextest.log
status=${PIPESTATUS[0]}
{
echo "exit_status=${status}"
echo "finished_at=$(date --utc --iso-8601=seconds)"
echo
echo "Remaining test-related processes:"
pgrep -af 'cargo|nextest|target/.*/deps/' || true
} > artifacts/ilm-integration/diagnostics.txt
exit "${status}"
- name: Upload ILM test diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
path: |
artifacts/ilm-integration
target/nextest/ci/junit.xml
test-and-lint-rio-v2:
name: Test and Lint (rio-v2)
# Alternate feature configurations run on main, merge groups, and scheduled
# or manual validation; the default workspace remains the PR gate.
if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
@@ -476,7 +506,7 @@ jobs:
test-and-lint-protocols:
name: "Test and Lint (${{ matrix.features.name }})"
if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
@@ -750,8 +780,6 @@ jobs:
needs: [ build-rustfs-debug-binary ]
runs-on: sm-standard-2
timeout-minutes: 30
env:
RUSTFS_S3_STACK: legacy
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -870,105 +898,18 @@ jobs:
path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
retention-days: 3
# The same e2e-smoke suite against the same debug binary with the gateway
# S3 stack selected (rustfs/backlog#2734). Report-only: it is outside the
# Test and Lint aggregate and becomes required at T2.10. The archive and run
# steps must stay identical to e2e-tests; scripts/ci_gate.py enforces that.
e2e-tests-gateway:
name: e2e-smoke (gateway, report-only until T2.10)
needs: [ build-rustfs-debug-binary ]
runs-on: sm-standard-2
timeout-minutes: 30
continue-on-error: true
env:
RUSTFS_S3_STACK: gateway
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install awscurl
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
- name: Verify awscurl
run: test -x "$AWSCURL_PATH"
- name: Download debug binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
- name: Archive e2e smoke test binaries
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
run: |
cargo nextest archive --profile e2e-smoke -p e2e_test --archive-file "${NEXTEST_ARCHIVE}"
cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --message-format json > "${NEXTEST_LISTING}"
python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
- name: Run e2e smoke suite
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
SAMPLER_INTERVAL_SECS: "15"
run: |
./scripts/ci/resource_sampler.sh start e2e-smoke
trap './scripts/ci/resource_sampler.sh stop' EXIT
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
--status-level all --final-status-level all --failure-output final
- name: Upload e2e smoke diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-gateway-diagnostics-${{ github.run_number }}
path: |
${{ runner.temp }}/rustfs-e2e-smoke-logs/
${{ runner.temp }}/rustfs-e2e-smoke-list.json
if-no-files-found: warn
- name: Upload e2e smoke JUnit report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-gateway-junit-${{ github.run_number }}
path: target/nextest/e2e-smoke/junit.xml
if-no-files-found: warn
e2e-full:
name: End-to-End Tests (full merge gate)
# Merge gate only (backlog#1149 ci-5): the never-automated user-visible
# suites — KMS, object_lock, multipart_auth, quota, checksum, encryption,
# security-boundary, ... — via the e2e-full nextest profile. Too heavy for
# every PR, so it is gated to main and gateway/integration pushes, the merge
# queue, and manual dispatch. protocols / the 7 cluster suites / replication
# / #[ignore] are owned by other lanes (see .config/nextest.toml profile.e2e-full).
# every PR, so it is gated to main pushes, the merge queue, and manual
# dispatch. protocols / the 7 cluster suites / replication / #[ignore] are
# owned by other lanes (see .config/nextest.toml profile.e2e-full).
if: >-
github.event_name == 'workflow_dispatch' ||
github.event_name == 'merge_group' ||
(github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/gateway/integration'))
(github.event_name == 'push' && github.ref == 'refs/heads/main')
needs: [ build-rustfs-debug-binary ]
runs-on: sm-standard-2
timeout-minutes: 55
@@ -1323,6 +1264,7 @@ jobs:
- quick-checks
- test-and-lint
- offline-enrollment-e2e
- test-ilm-integration-serial
- test-and-lint-rio-v2
- connect-short-credential-boundary
- test-and-lint-protocols
@@ -1356,6 +1298,7 @@ jobs:
- quick-checks
- test-and-lint
- offline-enrollment-e2e
- test-ilm-integration-serial
- test-and-lint-rio-v2
- test-and-lint-protocols
- build-rustfs-debug-binary
+41 -37
View File
@@ -96,8 +96,38 @@ jobs:
overwrite: true
upgrade:
name: Upgrade compatibility
name: ${{ matrix.name }}
needs: build
strategy:
fail-fast: false
matrix:
include:
# The two `_from_rc2_` tests keep their names: they assert
# release-independent object contracts and pass unchanged against the
# newer pinned source, so renaming them would only churn history and
# the CI required-check names. UPGRADE_SOURCE_VERSION above is the
# single source of truth for which release they actually run against.
- name: Direct upgrade from the previous release
test: direct_upgrade_from_rc2_preserves_object_contracts
artifact: direct-upgrade
- name: Mixed-version rolling upgrade from the previous release
test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
artifact: mixed-version-upgrade
- name: Bucket configuration survives the upgrade
test: direct_upgrade_from_previous_release_preserves_bucket_configuration
artifact: bucket-config-upgrade
- name: Rollback reads current bucket metadata
test: rollback_to_previous_release_reads_current_bucket_metadata
artifact: bucket-config-rollback
- name: ODM configuration recovery after rc.5 rollback
test: rc5_rollback_requires_restoring_odm_configuration
artifact: odm-config-rollback
- name: Multipart layouts survive the rc.5 upgrade
test: direct_upgrade_from_rc5_preserves_multipart_layouts
artifact: multipart-layout-upgrade
- name: rc.5 multipart replication baseline
test: rc5_baseline_replicates_multipart_layouts
artifact: multipart-layout-baseline
runs-on: sm-standard-2
timeout-minutes: 60
env:
@@ -112,7 +142,8 @@ jobs:
uses: ./.github/actions/setup
with:
cache-shared-key: e2e-upgrade-tests
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# One main-branch consumer warms the shared test-harness dependencies.
cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
@@ -139,49 +170,22 @@ jobs:
echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
# Reuse one compiled harness, while keeping each case in its own process.
# Continue after a failure to retain the former matrix's complete verdict.
- name: Run upgrade compatibility tests
shell: bash
- name: Run upgrade compatibility test
env:
RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
NO_PROXY: 127.0.0.1,localhost,::1
run: |
export no_proxy="$NO_PROXY"
cases=(
direct-upgrade:direct_upgrade_from_rc2_preserves_object_contracts
mixed-version-upgrade:rolling_upgrade_from_rc2_preserves_mixed_version_contracts
bucket-config-upgrade:direct_upgrade_from_previous_release_preserves_bucket_configuration
bucket-config-rollback:rollback_to_previous_release_reads_current_bucket_metadata
odm-config-rollback:rc5_rollback_requires_restoring_odm_configuration
multipart-layout-upgrade:direct_upgrade_from_rc5_preserves_multipart_layouts
multipart-layout-baseline:rc5_baseline_replicates_multipart_layouts
)
status=0
for entry in "${cases[@]}"; do
artifact="${entry%%:*}"
test="${entry#*:}"
export RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR="$RUNNER_TEMP/rustfs-upgrade-g09-evidence/$artifact"
export RUSTFS_E2E_LOG_DIR="$RUNNER_TEMP/rustfs-upgrade-logs/$artifact"
mkdir -p "$RUSTFS_E2E_LOG_DIR"
echo "::group::$test"
if python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
"upgrade_compatibility_test::$test" -- --ignored --exact --nocapture \
2>&1 | tee "$RUSTFS_E2E_LOG_DIR/test.log"; then
echo "PASS: $test"
else
echo "::error::Upgrade compatibility failed: $test"
status=1
fi
echo "::endgroup::"
done
exit "$status"
python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
"upgrade_compatibility_test::${{ matrix.test }}" \
-- --ignored --exact --nocapture
- name: Upload scanner/heal G09 evidence
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: upgrade-g09-evidence-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence
name: ${{ matrix.artifact }}-g09-evidence-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
if-no-files-found: ignore
retention-days: 14
@@ -189,7 +193,7 @@ jobs:
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: upgrade-server-logs-${{ github.run_number }}
name: ${{ matrix.artifact }}-server-logs-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-logs
if-no-files-found: warn
retention-days: 14
+2 -4
View File
@@ -20,9 +20,8 @@ on:
- ".github/workflows/oidc-keycloak.yml"
- "crates/config/src/constants/oidc.rs"
- "crates/iam/src/federation/**"
- "crates/iam/src/oidc/**"
- "crates/iam/src/oidc.rs"
- "rustfs/src/admin/handlers/oidc.rs"
- "rustfs/src/admin/service/oidc_config.rs"
- "rustfs/src/admin/handlers/sts.rs"
- "scripts/test/oidc_keycloak_live.sh"
- "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"
@@ -32,9 +31,8 @@ on:
- ".github/workflows/oidc-keycloak.yml"
- "crates/config/src/constants/oidc.rs"
- "crates/iam/src/federation/**"
- "crates/iam/src/oidc/**"
- "crates/iam/src/oidc.rs"
- "rustfs/src/admin/handlers/oidc.rs"
- "rustfs/src/admin/service/oidc_config.rs"
- "rustfs/src/admin/handlers/sts.rs"
- "scripts/test/oidc_keycloak_live.sh"
- "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"
+28 -22
View File
@@ -7,9 +7,7 @@
"name": "Debug(only) executable 'rustfs'",
"env": {
"RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info",
"RUSTFS_SKIP_BACKGROUND_TASK": "on",
// Local test volumes share one physical disk.
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true"
"RUSTFS_SKIP_BACKGROUND_TASK": "on"
//"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
// "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
// "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
@@ -46,7 +44,6 @@
"env": {
"RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
"RUSTFS_SKIP_BACKGROUND_TASK": "on",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
//"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
// "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
// "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
@@ -120,8 +117,7 @@
"env": {
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
//"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
// "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
@@ -139,31 +135,42 @@
{
"type": "lldb",
"request": "launch",
"name": "Debug test_transition_and_restore_flows",
"name": "Debug test_lifecycle_transition_basic",
"cargo": {
"args": [
"test",
"--no-run",
"-p",
"rustfs-scanner",
"--test",
"lifecycle_integration_test",
"serial_tests::test_lifecycle_transition_basic",
"-j",
"1"
],
"filter": {
"name": "lifecycle_integration_test",
"kind": "test"
}
]
},
"args": [
"serial_tests::test_transition_and_restore_flows",
"--exact",
"--ignored",
"--test-threads=1"
],
"args": [],
"cwd": "${workspaceFolder}"
},
{
"name": "Debug executable target/debug/test",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/deps/lifecycle_integration_test-5915cbfcab491b3b",
"args": [
"--skip",
"test_lifecycle_expiry_basic",
"--skip",
"test_lifecycle_expiry_deletemarker",
//"--skip",
//"test_lifecycle_transition_basic",
],
"cwd": "${workspaceFolder}",
//"stopAtEntry": false,
//"preLaunchTask": "cargo build",
"sourceLanguages": [
"rust"
],
},
{
"name": "Debug executable target/debug/rustfs with sse kms",
"type": "lldb",
@@ -278,7 +285,7 @@
"RUSTFS_COMPRESSION_ENABLED": "true",
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "local",
"RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/rio-v2/kms-key-dir",
"RUSTFS_KMS_KEY_DIR": "./target/rio-v2/kms-key-dir",
"RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
},
@@ -300,7 +307,6 @@
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
"RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
"RUSTFS_REGION": "us-east-1",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
@@ -309,7 +315,7 @@
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "local",
"RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/e2e-key-dir",
"RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir",
"RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
},
+1 -1
View File
@@ -155,7 +155,7 @@ default build (lifecycle:
- ⚠️ PARTIALLY VIOLATED: serving-side `s3s` references remain in ecstore
(bucket metadata/replication/lifecycle DTOs and error mapping). The
count is ratcheted shrink-only by `scripts/check_s3s_footprint.sh`
(the `ecstore_files` line of `.config/s3s-footprint-baseline.txt`; the `object_lock` module was converted to
(`S3S_ECSTORE_FILES_BASELINE`; the `object_lock` module was converted to
storage-level types as the first ratchet step). Target state: the
baseline reaches zero and ecstore's `Cargo.toml` drops `s3s`.
Generated
+269 -112
View File
@@ -275,6 +275,15 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "anyerror"
version = "0.1.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "71add24cc141a1e8326f249b74c41cfd217aeb2a67c9c6cf9134d175469afd49"
dependencies = [
"serde",
]
[[package]]
name = "anyhow"
version = "1.0.104"
@@ -1706,6 +1715,30 @@ dependencies = [
"syn 3.0.6",
]
[[package]]
name = "borsh"
version = "1.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "553c5d846a6ba5150c65e3b1b8ec073bcf1abc20f9b7220de384a4443ea4e20a"
dependencies = [
"borsh-derive",
"bytes",
"cfg_aliases",
]
[[package]]
name = "borsh-derive"
version = "1.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "12cdfe656708a01f89b451a7d36466e6fe6c414de0aa18fc54f864f6f9ca9f56"
dependencies = [
"once_cell",
"proc-macro-crate",
"proc-macro2",
"quote",
"syn 3.0.6",
]
[[package]]
name = "brotli"
version = "8.0.4"
@@ -1772,6 +1805,18 @@ version = "3.20.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649"
[[package]]
name = "byte-unit"
version = "5.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c719d56f7e96194cfc53460976d3ba51c85719747c9c62ed99981847b551152b"
dependencies = [
"rust_decimal",
"schemars 1.2.2",
"serde",
"utf8-width",
]
[[package]]
name = "bytemuck"
version = "1.25.2"
@@ -2839,9 +2884,9 @@ checksum = "4583a4551df46e2792f82ceeac45e850d2e2d5debba0b91f102385cda5b11f06"
[[package]]
name = "datafusion"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4cd5317db77ba6bca25c761a8091b9d294a9f4188e4f9dcd0243e6a57de862ac"
checksum = "14313430439ef858ed5f52ca9c840e9eea9844da34d3716d05a476449634da5d"
dependencies = [
"arrow",
"arrow-schema",
@@ -2888,9 +2933,9 @@ dependencies = [
[[package]]
name = "datafusion-catalog"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f35d36794509b7bfcbf90e11a1b5e4d047123c34610105e2e80a2048351cdb2f"
checksum = "2c0756ebc770bcebae366dfd273dad50935f6103b4d212253cb17e2e53c5427f"
dependencies = [
"arrow",
"async-trait",
@@ -2913,9 +2958,9 @@ dependencies = [
[[package]]
name = "datafusion-catalog-listing"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a6454bfb7f43b83942e2ca2576fd09342eedbd0d323e1eef72c3c8e4d57d7203"
checksum = "bbde2961698705d8402f0f382f08876fc6f1126ffde790ef17833cf0ab87cd59"
dependencies = [
"arrow",
"async-trait",
@@ -2937,9 +2982,9 @@ dependencies = [
[[package]]
name = "datafusion-common"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d26626e12b224b83351239465dcd00f72d47b02fb5e5b65a643e2dc8ed37f687"
checksum = "a6e06fe711ff815c8cf3143158547aea599702fd43f76f5866c70d5a5b3693a0"
dependencies = [
"arrow",
"arrow-ipc",
@@ -2964,9 +3009,9 @@ dependencies = [
[[package]]
name = "datafusion-common-runtime"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a106420cbfa980f242c22243ae9a6cce661d4d86ce96cff87dacabf7754a4470"
checksum = "f7a5107acff4e3ed50a7732acec06bba40e74ab49df021397ed250d60f04e42a"
dependencies = [
"futures",
"log",
@@ -2975,9 +3020,9 @@ dependencies = [
[[package]]
name = "datafusion-datasource"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "210aa1e9525e9454f2399dbb8b0b95510145bd8c1fdc35de74e9ccbf81bcb699"
checksum = "f69f45d2d700785be8df9d4b44410e1c8e4a856f10cc728b418a1066c0b594fb"
dependencies = [
"arrow",
"async-trait",
@@ -3005,9 +3050,9 @@ dependencies = [
[[package]]
name = "datafusion-datasource-arrow"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "65b1b041688cad707fa103c3e7f4c6d89b3134ebf67da0129070dad3381851eb"
checksum = "81650af64894ef34b46d2ef818ef244b40a81d90818f3bc642bf8fbf6758f8e0"
dependencies = [
"arrow",
"arrow-ipc",
@@ -3029,9 +3074,9 @@ dependencies = [
[[package]]
name = "datafusion-datasource-csv"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a7b34e253c4fa5251f174977af910a2f2ec831ec39dbbb56b29ce505cff278b"
checksum = "b5e2519bed59a6907c79298b9c21fe807bbc34e4ff8414577d699b0741083593"
dependencies = [
"arrow",
"async-trait",
@@ -3052,9 +3097,9 @@ dependencies = [
[[package]]
name = "datafusion-datasource-json"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1663338f43be6a5746473008e2d0fbde282577ddf0bae90d6c7386e988d71b29"
checksum = "635e61290e171597f6e50427e0cb486eda468ab47e880fa24d15bbb0fa24ac7e"
dependencies = [
"arrow",
"async-trait",
@@ -3075,9 +3120,9 @@ dependencies = [
[[package]]
name = "datafusion-datasource-parquet"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7c521ae91b6d899d7190812881f45fe9463c222c11d90149f5332a2e56555904"
checksum = "07cd518f85b3a027d913047b4ab0811111a2b734fdf84a51b78234455f10d8fa"
dependencies = [
"arrow",
"arrow-schema",
@@ -3107,15 +3152,15 @@ dependencies = [
[[package]]
name = "datafusion-doc"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c2569c2e74a34fcae3a369ab3912d61e41a062592510f11e31b1dae2725ecdc"
checksum = "d44c5919b560862dd1cb20a5c75b3e48c3d9e49719c91d78da5c38eaad5495c4"
[[package]]
name = "datafusion-execution"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cfdd5ac0769d1e74a885450e49fde8f63ca7ab41eb479a068564aaea80b837"
checksum = "7620ad535af8730186bdec05d720e40d9a4584b40c7eb75f72c43aa7978e5484"
dependencies = [
"arrow",
"arrow-buffer",
@@ -3139,9 +3184,9 @@ dependencies = [
[[package]]
name = "datafusion-expr"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "86560fd7df4dda924f5474b588c56e7ae8e96dcf158c7327c9cfe9daff448ec5"
checksum = "ea229af8b73e566fc6ebb1c8dd42ab250f8855c5304bd173f4b50ead78b64304"
dependencies = [
"arrow",
"arrow-schema",
@@ -3156,16 +3201,15 @@ dependencies = [
"indexmap 2.14.2",
"itertools 0.15.0",
"recursive",
"serde",
"serde_json",
"sqlparser",
]
[[package]]
name = "datafusion-expr-common"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e82e313cf655cf6bc2dce6cbd23bcb0d20236a54a1b09c436615e4bac4d8c49f"
checksum = "6a18baa840f77760dcc9eaad281329bc57536c5c39e65e8649596f2a51216be7"
dependencies = [
"arrow",
"datafusion-common",
@@ -3175,9 +3219,9 @@ dependencies = [
[[package]]
name = "datafusion-functions"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "939ae2771da8a3fdc23ab4de610ae8fef9515476626b49d96d0d5c782b105771"
checksum = "763bbfeec40b55a8a961d7349c0fcfcec4f3f66cc234d6295250b95371b85e34"
dependencies = [
"arrow",
"arrow-buffer",
@@ -3203,9 +3247,9 @@ dependencies = [
[[package]]
name = "datafusion-functions-aggregate"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fd6c3aee9f2bc65716787d651e023743a90b4efbc57a2f679f036632e16fb110"
checksum = "46c0c2b2e856883ecf6a4fa8b23e7c6041183af8a68d0583dc95d1d21e4eaee0"
dependencies = [
"arrow",
"datafusion-common",
@@ -3224,9 +3268,9 @@ dependencies = [
[[package]]
name = "datafusion-functions-aggregate-common"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "988b5d3dfb4939d81c2f9fab267063610fe52a843620f2fa6cb2a1598e43369c"
checksum = "e35e42c87dfbff06934ea0e99cfed3440ac614c3f3bfb8c33be35ca0587d4140"
dependencies = [
"arrow",
"datafusion-common",
@@ -3236,9 +3280,9 @@ dependencies = [
[[package]]
name = "datafusion-functions-nested"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08149d433a97c9ea3cdf80de06480ab5c7f5453485f1e452d5b3583c81d5c0dd"
checksum = "08079b0604cdcc43c7d6cf006cfd5d3ed9c235ade0c1ebe91085a62f875603df"
dependencies = [
"arrow",
"arrow-ord",
@@ -3261,9 +3305,9 @@ dependencies = [
[[package]]
name = "datafusion-functions-table"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2c767567cce2782d4d0bdfa55fab010aaba0bdbe533509e9e0b9a6a3ee29ed59"
checksum = "fe2070ecf5638a93c81c8babc7858339ef52165a9f54da1411afd69adf7d85ce"
dependencies = [
"arrow",
"async-trait",
@@ -3277,9 +3321,9 @@ dependencies = [
[[package]]
name = "datafusion-functions-window"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0a311c1f558d1f792f1f631c3670da12ee72a20fbf5d6ca151325d66191c03c1"
checksum = "0e9a164ebe1998c5c3f9e15c9a0c4e6b599a0c8739b5471bfb089752441963f1"
dependencies = [
"arrow",
"datafusion-common",
@@ -3294,9 +3338,9 @@ dependencies = [
[[package]]
name = "datafusion-functions-window-common"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fffd54b8acf6ecef284cc0677e01e1b8c71be64fec50612ba00a0ff0d9b41a52"
checksum = "98965b183f92201c16798994820ae032649c4821e089c4c80e7d0f4485d41df5"
dependencies = [
"datafusion-common",
"datafusion-physical-expr-common",
@@ -3304,9 +3348,9 @@ dependencies = [
[[package]]
name = "datafusion-macros"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "14d6df9763c8fb968b2417d797aef4c55ac2e4a895aaef38c72c16a287aa4a40"
checksum = "3a711bd5a06ec85d9683e142d28a98ad0679a7220669f5c1b8589e57e9b29e46"
dependencies = [
"datafusion-doc",
"quote",
@@ -3315,9 +3359,9 @@ dependencies = [
[[package]]
name = "datafusion-optimizer"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "acfb8cea6e05e2e75bae183340cf7d53cccb3db0449e2f1fbc07654d980fd19f"
checksum = "ff3256b62ff44616ec83c7f04741d21124970833bb377ed56d9abc5ec6f707ec"
dependencies = [
"arrow",
"chrono",
@@ -3335,9 +3379,9 @@ dependencies = [
[[package]]
name = "datafusion-physical-expr"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f002cef849e10c0597a217ec7891c2796c0f8b42d739f5d24abc24496373406"
checksum = "c1e37a25b5aa1f3a1db114f0ee3605bf01639b83a831dcde35119ee61f0b11d6"
dependencies = [
"arrow",
"arrow-schema",
@@ -3358,9 +3402,9 @@ dependencies = [
[[package]]
name = "datafusion-physical-expr-adapter"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "82761522c4536e7059d3b185cd99f10525c2bf48f1bd26a0a57e73e19eadc0fb"
checksum = "3a4d12787d42e88b26b0d3ec006192fa23ceb832ef2a9e87439ce820a8e038cc"
dependencies = [
"arrow",
"datafusion-common",
@@ -3373,9 +3417,9 @@ dependencies = [
[[package]]
name = "datafusion-physical-expr-common"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8c5c67aa252886f4d6b331c8114797f5b140464e774c4da36889269a1bf1bd8"
checksum = "d13bf7f50fd902f5afc46d50759f5185b5620c40b24054f80f7c648897d3e5a6"
dependencies = [
"arrow",
"chrono",
@@ -3390,9 +3434,9 @@ dependencies = [
[[package]]
name = "datafusion-physical-optimizer"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1a49dd3a76e4879012682c59dc3735282e60640c97483546a4113564a60bf8f"
checksum = "11fbe7e6665072957d44ed9587676e17a1f9ab3acfadfc287dac23750585f18a"
dependencies = [
"arrow",
"datafusion-common",
@@ -3410,9 +3454,9 @@ dependencies = [
[[package]]
name = "datafusion-physical-plan"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1451d6817e5ac0dd21819b19949d1c420ab51d748f55dd9a96ecb5418a24c97"
checksum = "1265d58e5bce07d154e642a51ff43033b576a6ae40d50a29ac2b9f311004eb52"
dependencies = [
"arrow",
"arrow-data",
@@ -3439,16 +3483,15 @@ dependencies = [
"num-traits",
"parking_lot",
"pin-project-lite",
"serde",
"serde_json",
"tokio",
]
[[package]]
name = "datafusion-pruning"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f11f0dbe2b5a642f6106e44de8bbd6654fa7220ca14f32313d4efd5514a86968"
checksum = "74b06b333405c05015836ed36cea02cc29d1e2ee8fb414a77532d8346fb56492"
dependencies = [
"arrow",
"datafusion-common",
@@ -3462,9 +3505,9 @@ dependencies = [
[[package]]
name = "datafusion-session"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "62dc5ce37a5ce1ff5362949776829aded2529a11dbaa13d24b0ffa6a059e106a"
checksum = "e7616b17b1d2fee3e9b13c651eac91b768d153265e6ba88e06e0b4af7883f69c"
dependencies = [
"arrow-schema",
"async-trait",
@@ -3477,9 +3520,9 @@ dependencies = [
[[package]]
name = "datafusion-sql"
version = "55.2.0"
version = "55.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "816c224a0ef5469a32a562b5d4830544f714f97c5f50142109286d40a288c7db"
checksum = "5b4848be66dd008d55adbe1dbc8cbd6a2502588ad0c0b8483bab0f43fd4495d0"
dependencies = [
"arrow",
"bigdecimal",
@@ -3726,13 +3769,34 @@ dependencies = [
"syn 2.0.119",
]
[[package]]
name = "derive_more"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4a9b99b9cbbe49445b21764dc0625032a89b145a2642e67603e1c936f5458d05"
dependencies = [
"derive_more-impl 1.0.0",
]
[[package]]
name = "derive_more"
version = "2.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d751e9e49156b02b44f9c1815bcb94b984cdcc4396ecc32521c739452808b134"
dependencies = [
"derive_more-impl",
"derive_more-impl 2.1.1",
]
[[package]]
name = "derive_more-impl"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb7330aeadfbe296029522e6c40f315320aba36fc43a5b3632f3795348f3bd22"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"unicode-xid",
]
[[package]]
@@ -3949,7 +4013,7 @@ dependencies = [
"libc",
"option-ext",
"redox_users",
"windows-sys 0.59.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -4299,7 +4363,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
"windows-sys 0.59.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -5299,9 +5363,9 @@ dependencies = [
[[package]]
name = "hotpath"
version = "0.28.6"
version = "0.28.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e2ee4f8d17ebe2400fa845f079acf9ede235be6472512876779af45f7890f934"
checksum = "34aabdeba3c81cef2075a2961a370ebfe1d9cc3f8d2242fd400ce6ef6f3bbfcb"
dependencies = [
"arc-swap",
"async-channel",
@@ -5334,15 +5398,15 @@ dependencies = [
[[package]]
name = "hotpath-drain"
version = "0.28.6"
version = "0.28.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fd9e65572a429f44a48c360559a3f59028579b53e1df07c5f31dc147d2b94a50"
checksum = "b4b61a4a77e6563bdc37f831cd6aa5e2089429b62ea377dd759bbabafc207d7a"
[[package]]
name = "hotpath-macros"
version = "0.28.6"
version = "0.28.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2b90aafbaf5f6af01ee9db2888644b15995f5306e5c280aec8261feb3e5e214a"
checksum = "8c6275c8d1931d280bd505debb7ef91dbeae2c23477aabbfe10487b7430bdeb3"
dependencies = [
"proc-macro2",
"quote",
@@ -5704,7 +5768,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "20fd6de4ccfcc187e38bc21cfa543cb5a302cb86a8b114eb7f0bf0dc9f8ac00f"
dependencies = [
"io-lifetimes 3.0.1",
"windows-sys 0.59.0",
"windows-sys 0.60.2",
]
[[package]]
@@ -6236,7 +6300,7 @@ dependencies = [
"bytes",
"chrono",
"dashmap",
"derive_more",
"derive_more 2.1.1",
"futures-util",
"getrandom 0.4.3",
"lazy_static",
@@ -6390,6 +6454,12 @@ dependencies = [
"quote",
]
[[package]]
name = "maplit"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3e2e65a1a2e43cfcb47a895c4c8b10d1f4a61097f9f254f183aee60cad9c651d"
[[package]]
name = "matchers"
version = "0.2.0"
@@ -6963,7 +7033,7 @@ version = "0.50.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
dependencies = [
"windows-sys 0.59.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -7331,6 +7401,42 @@ dependencies = [
"url",
]
[[package]]
name = "openraft"
version = "0.9.25"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a97014fb78acb77be3a40ac2da305f6dd3a6b243f3a908ace87d29b3972eaafd"
dependencies = [
"anyerror",
"byte-unit",
"chrono",
"clap",
"derive_more 1.0.0",
"futures",
"maplit",
"openraft-macros",
"rand 0.8.8",
"serde",
"thiserror 1.0.69",
"tokio",
"tracing",
"tracing-futures",
"validit",
]
[[package]]
name = "openraft-macros"
version = "0.9.25"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "55417d79f46a787dd88c043d831d8b7a0713f7c69d1925f9de51cc42c437250f"
dependencies = [
"chrono",
"proc-macro2",
"quote",
"semver",
"syn 2.0.119",
]
[[package]]
name = "openssl-probe"
version = "0.2.1"
@@ -8345,7 +8451,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
dependencies = [
"heck 0.5.0",
"itertools 0.10.5",
"itertools 0.14.0",
"log",
"multimap",
"once_cell",
@@ -8365,7 +8471,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
dependencies = [
"heck 0.5.0",
"itertools 0.10.5",
"itertools 0.14.0",
"log",
"multimap",
"petgraph 0.8.3",
@@ -8386,7 +8492,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
dependencies = [
"anyhow",
"itertools 0.10.5",
"itertools 0.14.0",
"proc-macro2",
"quote",
"syn 2.0.119",
@@ -8399,7 +8505,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf"
dependencies = [
"anyhow",
"itertools 0.10.5",
"itertools 0.14.0",
"proc-macro2",
"quote",
"syn 2.0.119",
@@ -8726,7 +8832,7 @@ dependencies = [
"once_cell",
"socket2",
"tracing",
"windows-sys 0.59.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -8960,6 +9066,15 @@ dependencies = [
"syn 2.0.119",
]
[[package]]
name = "redb"
version = "4.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb338a6c67830a61bed824b78c2bb034ab1ff401a20616bd7957524f4fdc2f22"
dependencies = [
"libc",
]
[[package]]
name = "redis"
version = "1.7.1"
@@ -9432,9 +9547,9 @@ dependencies = [
[[package]]
name = "rust-embed"
version = "8.13.0"
version = "8.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19afa5b4b6a611de00bd1bdae6ae6f39084c9399f0679c3f52d8469cf335cc23"
checksum = "e9e7760e252aaba7b09f4be00e36476cf585bdb68a53552ac954cdf504ab4bc9"
dependencies = [
"rust-embed-impl",
"rust-embed-utils",
@@ -9443,9 +9558,9 @@ dependencies = [
[[package]]
name = "rust-embed-impl"
version = "8.13.0"
version = "8.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e0d8afda6374eac59e066abee06d265247ebbaf3006cf878e2879e8356e34053"
checksum = "3bcfc4d6f53af43755f7a723e4b6b8794fcce052a178dd8c6c1dadc5f5343097"
dependencies = [
"mime_guess",
"proc-macro2",
@@ -9458,14 +9573,31 @@ dependencies = [
[[package]]
name = "rust-embed-utils"
version = "8.13.0"
version = "8.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d84e8ba78bd384263e5922f084cbe1b081c3b7e69add59c8fb097b879ba968a"
checksum = "42ffa149f6aa81b58a5b3011d01a857c4ed12c7a732d2c51947a4c7c692185f0"
dependencies = [
"sha2 0.11.0",
"walkdir",
]
[[package]]
name = "rust_decimal"
version = "1.43.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7653272e75dcac41dc199fbea6f5797633994fafd339943c06c9af16bf29cd3a"
dependencies = [
"arrayvec",
"borsh",
"bytes",
"num-traits",
"rand 0.8.8",
"rand 0.9.5",
"serde",
"serde_json",
"wasm-bindgen",
]
[[package]]
name = "rustc-demangle"
version = "0.1.28"
@@ -9668,6 +9800,21 @@ dependencies = [
"url",
]
[[package]]
name = "rustfs-capture-authority"
version = "1.0.1"
dependencies = [
"openraft",
"redb",
"serde",
"serde_json",
"sha2 0.11.0",
"tempfile",
"thiserror 2.0.21",
"tokio",
"url",
]
[[package]]
name = "rustfs-checksums"
version = "1.0.1"
@@ -9688,7 +9835,6 @@ dependencies = [
name = "rustfs-common"
version = "1.0.1"
dependencies = [
"arc-swap",
"hotpath",
"metrics",
"smallvec",
@@ -10048,7 +10194,6 @@ dependencies = [
"tokio",
"tokio-util",
"tracing",
"tracing-subscriber",
"url",
]
@@ -10288,6 +10433,7 @@ dependencies = [
"reqwest",
"rmp-serde",
"rustfs-signer",
"s3s",
"serde",
"serde_json",
"sysinfo",
@@ -10476,6 +10622,7 @@ dependencies = [
"tokio",
"tracing",
"tracing-subscriber",
"uuid",
]
[[package]]
@@ -10512,7 +10659,6 @@ dependencies = [
"rustfs-keystone",
"rustfs-policy",
"rustfs-rio",
"rustfs-s3-types",
"rustfs-storage-api",
"rustfs-test-utils",
"rustfs-tls-runtime",
@@ -10527,7 +10673,6 @@ dependencies = [
"sha2 0.11.0",
"socket2",
"subtle",
"sync_wrapper",
"tempfile",
"thiserror 2.0.21",
"time",
@@ -10675,7 +10820,6 @@ dependencies = [
"rustfs-checksums",
"rustfs-config",
"rustfs-rio",
"rustfs-s3-types",
"rustfs-signer",
"rustfs-storage-api",
"rustfs-tls-runtime",
@@ -10709,19 +10853,9 @@ dependencies = [
name = "rustfs-s3-types"
version = "1.0.1"
dependencies = [
"bytes",
"futures-core",
"hotpath",
"http 1.5.0",
"http-body",
"http-body-util",
"hyper",
"hyper-util",
"pin-project-lite",
"s3s",
"serde",
"serde_json",
"tokio",
]
[[package]]
@@ -10745,7 +10879,6 @@ dependencies = [
"parking_lot",
"rustfs-common",
"rustfs-ecstore",
"rustfs-s3-types",
"rustfs-storage-api",
"rustfs-test-utils",
"s3s",
@@ -10859,10 +10992,9 @@ dependencies = [
"bytes",
"hotpath",
"http 1.5.0",
"http-body",
"hyper",
"rustfs-s3-types",
"rustfs-utils",
"s3s",
"serde_urlencoded",
"thiserror 2.0.21",
"time",
@@ -10920,6 +11052,7 @@ dependencies = [
"rustfs-utils",
"rustls",
"rustls-native-certs",
"s3s",
"serde",
"serde_json",
"sha2 0.11.0",
@@ -10973,7 +11106,6 @@ dependencies = [
"thiserror 2.0.21",
"tokio",
"tracing",
"x509-parser",
]
[[package]]
@@ -11144,7 +11276,7 @@ dependencies = [
"errno",
"libc",
"linux-raw-sys",
"windows-sys 0.59.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -11227,7 +11359,7 @@ dependencies = [
"security-framework",
"security-framework-sys",
"webpki-root-certs",
"windows-sys 0.59.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -12498,10 +12630,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom 0.3.4",
"getrandom 0.4.3",
"once_cell",
"rustix",
"windows-sys 0.59.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -13064,6 +13196,16 @@ dependencies = [
"tracing-subscriber",
]
[[package]]
name = "tracing-futures"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "97d095ae15e245a057c8e8451bab9b3ee1e1f68e9ba2b4fbc18d0ac5237835f2"
dependencies = [
"pin-project",
"tracing",
]
[[package]]
name = "tracing-log"
version = "0.2.0"
@@ -13214,7 +13356,7 @@ checksum = "79acf4b257aa56dd7f00466a81693d6c79c362c55092b3137909d6e76249ad80"
dependencies = [
"async-trait",
"chrono",
"derive_more",
"derive_more 2.1.1",
"libc",
"md-5 0.10.6",
"thiserror 2.0.21",
@@ -13314,6 +13456,12 @@ version = "2.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da"
[[package]]
name = "utf8-width"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "159a7cadce548703edd50d24069bc294c5415ecab0a480e0cd1ca06d112dc94a"
[[package]]
name = "utf8_iter"
version = "1.0.4"
@@ -13340,6 +13488,15 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "validit"
version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4417835826b78aa6265724624d424df2b1e6e71a33329227d8b04a88950780dc"
dependencies = [
"anyerror",
]
[[package]]
name = "valuable"
version = "0.1.1"
@@ -13595,7 +13752,7 @@ version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [
"windows-sys 0.59.0",
"windows-sys 0.61.2",
]
[[package]]
+4 -6
View File
@@ -18,6 +18,7 @@ members = [
"crates/audit", # Audit target management system with multi-target fan-out
"crates/checksums", # client checksums
"crates/common", # Shared utilities and data structures
"crates/capture-authority", # Durable capture authority storage
"crates/data-usage", # Shared data usage models and algorithms
"crates/config", # Configuration management
"crates/connect-symbol-catalog", # Release-only CPU profile symbol catalog generator
@@ -48,7 +49,7 @@ members = [
"crates/replication", # Replication contracts and wire formats
"crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
"crates/s3-client", # S3 client for engine-side consumption of remote S3 endpoints (tiering, transition targets)
"crates/s3-types", # S3 event and error type definitions
"crates/s3-types", # S3 event type definitions
"crates/s3-ops", # S3 operation definitions and mapping
"crates/s3select-api", # S3 Select API interface
"crates/s3select-query", # S3 Select query engine
@@ -314,6 +315,8 @@ ratelimit = "2.0.0"
rayon = "1.12.0"
rustfs-erasure-codec = { version = "9.0.0" }
reed-solomon-simd = "3.1.0"
openraft = { version = "=0.9.25", default-features = false, features = ["serde"] }
redb = "=4.3.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.34.0" }
redis = { version = "1.7.1" }
@@ -323,10 +326,6 @@ rust-embed = { version = "8.12.0" }
rustc-hash = { version = "2.1.3" }
s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
# The same s3s pin under the name the legacy edge of the gateway migration uses
# (crates/s3-types, feature `compat-s3s`). Keep `rev` identical to `s3s` above.
# DELETE BY T4.2 (rustfs/backlog#2784).
legacy-s3s = { package = "s3s", git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.4"
@@ -335,7 +334,6 @@ compact_str = "0.10.0"
snap = "1.1.2"
starshard = { version = "2.4.0" }
strum = { version = "0.28.0" }
sync_wrapper = "1.0.2"
sysinfo = "0.39.6"
temp-env = "0.3.6"
tempfile = "3.27.0"
+24
View File
@@ -0,0 +1,24 @@
[package]
name = "rustfs-capture-authority"
version.workspace = true
edition.workspace = true
license.workspace = true
rust-version.workspace = true
repository.workspace = true
description = "Durable local storage for capture authority"
[dependencies]
openraft = { workspace = true }
redb = { workspace = true }
sha2 = { workspace = true }
serde = { workspace = true, features = ["derive"] }
thiserror = { workspace = true }
tokio = { workspace = true }
url = { workspace = true }
[dev-dependencies]
serde_json = { workspace = true }
tempfile = { workspace = true }
[lints]
workspace = true
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,531 @@
use super::*;
use crate::application::{APPLY_RESULTS, EVENTS, OPERATIONS, VERSIONS};
use crate::tests::CrashBackend;
use redb::ReadableTableMetadata;
use std::{
io::{BufRead, BufReader, Write},
process::{Command, Stdio},
sync::mpsc,
time::Duration,
};
fn identity() -> NodeIdentity {
NodeIdentity {
cluster_id: [7; 16],
node_id: 0,
}
}
fn binding() -> CaptureBinding {
CaptureBinding {
revision: 9,
binding_id: [8; 16],
target_id: [9; 16],
}
}
fn command() -> DecideCreated {
DecideCreated {
operation_id: [1; 16],
object: ObjectIdentity {
bucket_incarnation: [2; 16],
key: vec![0, 255, 3],
},
expected_head: None,
prepared: PreparedIdentity {
version_id: [3; 16],
preparation_id: [4; 16],
content_digest: [5; 32],
content_length: 123,
},
binding_revision: 9,
}
}
fn id(index: u64) -> LogId {
LogId {
term: 8,
leader_node: 0,
index,
}
}
fn append(store: &mut Store, index: u64, c: &DecideCreated) {
store
.append(&[LogEntry {
id: id(index),
payload: c.encode().expect("encode command"),
}])
.expect("append command");
}
fn apply(store: &mut Store, index: u64, c: &DecideCreated) -> CreatedResult {
append(store, index, c);
store.save_committed(id(index)).expect("commit log");
store.apply_next_created(id(index)).expect("apply command")
}
fn success(c: &DecideCreated, index: u64) -> CreatedResult {
CreatedResult::Created {
version_id: c.prepared.version_id,
event_id: c.operation_id,
decision_log_id: id(index),
}
}
fn complete(store: &Store) {
let c = command();
assert_eq!(store.capture_binding().expect("binding"), Some(binding()));
assert_eq!(
store.created_head(&c.object).expect("head"),
Some(c.prepared.version_id),
"U11 acknowledged head lost"
);
assert_eq!(
store.created_version(&c.object, c.prepared.version_id).expect("version"),
Some(CreatedVersion {
prepared: c.prepared.clone(),
decision_log_id: id(0),
binding_revision: 9
})
);
assert_eq!(
store.created_event(c.operation_id).expect("event"),
Some(CreatedEvent {
object: c.object.clone(),
prepared: c.prepared.clone(),
decision_log_id: id(0),
binding: binding()
}),
"U11 complete event missing"
);
assert_eq!(
store.created_operation(c.operation_id).expect("operation"),
Some(CreatedOperation {
canonical_command: c.encode().expect("encode"),
result: success(&c, 0)
})
);
assert_eq!(
store.created_apply_result(0).expect("result"),
Some(CreatedApplyResult {
log_id: id(0),
result: success(&c, 0)
})
);
assert_eq!(store.last_applied().expect("last applied"), Some(id(0)));
}
fn fixture() -> (tempfile::TempDir, Store) {
let dir = tempfile::tempdir().expect("tempdir");
let mut store = Store::initialize(dir.path().join("db"), identity()).expect("initialize");
store.pin_capture_binding(binding()).expect("pin");
(dir, store)
}
fn reopen(path: &std::path::Path) -> Store {
let result = Store::open(path, identity());
assert!(result.is_ok(), "U11 complete reopen: {:?}", result.as_ref().err());
result.expect("asserted successful reopen")
}
#[test]
fn created_codec_is_canonical_and_strict() {
let c = command();
let b = c.encode().expect("encode");
let mut golden = b"CRTD\x01".to_vec();
golden.extend([1; 16]);
golden.extend([2; 16]);
golden.extend(3u32.to_be_bytes());
golden.extend([0, 255, 3]);
golden.push(0);
golden.extend([3; 16]);
golden.extend([4; 16]);
golden.extend([5; 32]);
golden.extend(123u64.to_be_bytes());
golden.extend(9u64.to_be_bytes());
assert_eq!(b, golden);
assert_eq!(DecideCreated::decode(&b).expect("decode"), c);
let mut variants = Vec::new();
let mut v = c.clone();
v.operation_id = [6; 16];
variants.push(v);
let mut v = c.clone();
v.object.bucket_incarnation = [6; 16];
variants.push(v);
let mut v = c.clone();
v.object.key.push(4);
variants.push(v);
let mut v = c.clone();
v.expected_head = Some([6; 16]);
variants.push(v);
let mut v = c.clone();
v.prepared.version_id = [6; 16];
variants.push(v);
let mut v = c.clone();
v.prepared.preparation_id = [6; 16];
variants.push(v);
let mut v = c.clone();
v.prepared.content_digest = [6; 32];
variants.push(v);
let mut v = c.clone();
v.prepared.content_length += 1;
variants.push(v);
let mut v = c.clone();
v.binding_revision += 1;
variants.push(v);
for v in variants {
assert_ne!(v.encode().expect("variant"), b)
}
for n in 0..b.len() {
assert!(DecideCreated::decode(&b[..n]).is_err())
}
let mut bad = b.clone();
bad[4] = 2;
assert!(DecideCreated::decode(&bad).is_err());
let mut bad = b.clone();
bad[44] = 2;
assert!(DecideCreated::decode(&bad).is_err());
let mut bad = b.clone();
bad.push(0);
assert!(DecideCreated::decode(&bad).is_err());
let mut bad = b;
bad[37..41].copy_from_slice(&4097u32.to_be_bytes());
assert!(DecideCreated::decode(&bad).is_err());
let mut bad = c.clone();
bad.operation_id = [0; 16];
assert!(bad.encode().is_err());
bad = c;
bad.object.key = vec![0; MAX_OBJECT_KEY_LENGTH + 1];
assert!(bad.encode().is_err());
}
#[test]
fn created_committed_order_and_identity_are_enforced() {
let (_dir, mut s) = fixture();
let c = command();
append(&mut s, 0, &c);
append(&mut s, 1, &c);
assert!(s.apply_next_created(id(0)).is_err());
assert_eq!(s.last_applied().expect("no apply"), None);
s.save_committed(id(1)).expect("committed");
assert!(matches!(s.apply_next_created(id(1)), Err(StoreError::InvalidApplySequence)));
let mut wrong = id(0);
wrong.term += 1;
assert!(s.apply_next_created(wrong).is_err());
wrong = id(0);
wrong.leader_node += 1;
assert!(s.apply_next_created(wrong).is_err());
assert_eq!(s.apply_next_created(id(0)).expect("first"), success(&c, 0));
complete(&s);
assert_eq!(s.apply_next_created(id(0)).expect("retry"), success(&c, 0));
complete(&s);
assert!(s.apply_next_created(wrong).is_err());
assert_eq!(s.apply_next_created(id(1)).expect("duplicate log"), success(&c, 0));
assert_eq!(s.last_applied().expect("last"), Some(id(1)));
assert_eq!(
s.created_apply_result(1).expect("result"),
Some(CreatedApplyResult {
log_id: id(1),
result: success(&c, 0)
})
);
let tx = s.database.begin_read().expect("read");
assert_eq!(tx.open_table(EVENTS).expect("events").len().expect("len"), 1);
}
#[test]
fn created_head_version_event_and_dedup_reopen_together() {
let (dir, mut s) = fixture();
let c = command();
assert_eq!(apply(&mut s, 0, &c), success(&c, 0));
drop(s);
let mut s = reopen(&dir.path().join("db"));
complete(&s);
assert_eq!(apply(&mut s, 1, &c), success(&c, 0));
let mut changed = c.clone();
changed.prepared.content_digest = [6; 32];
assert_eq!(
apply(&mut s, 2, &changed),
CreatedResult::OperationConflict,
"U11 full canonical operation conflict"
);
assert_eq!(
s.created_operation(c.operation_id)
.expect("original")
.expect("operation")
.result,
success(&c, 0)
);
let mut stale = c.clone();
stale.operation_id = [6; 16];
stale.prepared.version_id = [6; 16];
assert_eq!(
apply(&mut s, 3, &stale),
CreatedResult::HeadMismatch {
actual_head: Some(c.prepared.version_id)
}
);
let mut reuse = c.clone();
reuse.operation_id = [7; 16];
reuse.expected_head = Some(c.prepared.version_id);
assert_eq!(apply(&mut s, 4, &reuse), CreatedResult::VersionConflict);
let mut replacement = stale.clone();
replacement.operation_id = [10; 16];
replacement.expected_head = Some(c.prepared.version_id);
assert_eq!(apply(&mut s, 5, &replacement), success(&replacement, 5));
assert_eq!(s.created_head(&c.object).expect("head"), Some(replacement.prepared.version_id));
assert_eq!(
s.created_version(&c.object, c.prepared.version_id)
.expect("old version")
.expect("version")
.decision_log_id,
id(0)
);
assert!(s.created_event(c.operation_id).expect("old event").is_some());
assert!(s.created_event(replacement.operation_id).expect("new event").is_some());
assert!(s.created_event(stale.operation_id).expect("rejection event").is_none());
assert_eq!(
apply(&mut s, 6, &stale),
CreatedResult::HeadMismatch {
actual_head: Some(c.prepared.version_id)
}
);
drop(s);
let s = reopen(&dir.path().join("db"));
assert_eq!(s.last_applied().expect("last"), Some(id(6)));
let tx = s.database.begin_read().expect("read");
assert_eq!(tx.open_table(EVENTS).expect("events").len().expect("len"), 2);
}
#[test]
fn created_binding_is_pinned_and_not_caller_selected() {
let (dir, mut s) = fixture();
let mut bad = command();
bad.binding_revision = 10;
assert_eq!(apply(&mut s, 0, &bad), CreatedResult::BindingMismatch);
assert!(s.created_head(&bad.object).expect("head").is_none());
assert!(s.created_event(bad.operation_id).expect("event").is_none());
assert_eq!(
s.created_operation(bad.operation_id)
.expect("operation")
.expect("saved")
.result,
CreatedResult::BindingMismatch
);
s.pin_capture_binding(binding()).expect("idempotent pin");
let mut changed = binding();
changed.target_id = [10; 16];
assert!(matches!(s.pin_capture_binding(changed), Err(StoreError::BindingConflict)));
drop(s);
let s = reopen(&dir.path().join("db"));
assert_eq!(s.capture_binding().expect("binding"), Some(binding()));
}
#[test]
fn created_schema_compatibility_and_corruption_are_checked() {
let dir = tempfile::tempdir().expect("dir");
let path = dir.path().join("db");
let s = Store::initialize(&path, identity()).expect("schema1");
assert_eq!(s.capture_binding().expect("unmigrated"), None);
drop(s);
let mut s = reopen(&path);
assert_eq!(s.last_applied().expect("no apply"), None);
append(&mut s, 0, &command());
s.save_committed(id(0)).expect("schema2");
drop(s);
let mut s = reopen(&path);
assert_eq!(s.capture_binding().expect("schema2 unchanged"), None);
s.pin_capture_binding(binding()).expect("explicit upgrade");
drop(s);
let mut s = reopen(&path);
assert_eq!(s.capture_binding().expect("schema3"), Some(binding()));
s.apply_next_created(id(0)).expect("apply");
drop(s);
let db = Database::open(&path).expect("raw db");
let tx = db.begin_write().expect("write corruption");
tx.open_table(EVENTS)
.expect("events")
.remove(command().operation_id.as_slice())
.expect("remove reference");
tx.commit().expect("commit corruption");
drop(db);
let result = Store::open(&path, identity());
assert!(
matches!(result, Err(StoreError::CorruptRecord)),
"U11 missing event rejected before expect"
);
let (dir, mut s) = fixture();
assert_eq!(s.read_committed().expect("empty committed"), None);
append(&mut s, 0, &command());
s.save_committed(id(0)).expect("committed schema3");
assert_eq!(s.capture_binding().expect("no downgrade"), Some(binding()));
drop(s);
let path = dir.path().join("db");
let db = Database::open(&path).expect("raw db");
let tx = db.begin_write().expect("write");
let mut marker = vec![1];
marker.extend(encode_id(id(0)));
tx.open_table(META)
.expect("meta")
.insert("last_applied", marker.as_slice())
.expect("corrupt applied");
tx.commit().expect("commit");
drop(db);
assert!(
matches!(Store::open(&path, identity()), Err(StoreError::CorruptRecord)),
"U11 last applied missing result rejected"
);
let (dir, s) = fixture();
drop(s);
let path = dir.path().join("db");
let db = Database::open(&path).expect("raw pinned db");
let tx = db.begin_write().expect("missing binding write");
tx.open_table(META)
.expect("meta")
.remove("capture_binding")
.expect("remove binding");
tx.commit().expect("binding corruption commit");
drop(db);
assert!(
matches!(Store::open(&path, identity()), Err(StoreError::CorruptRecord)),
"U11 missing pinned binding rejected"
);
let (dir, mut s) = fixture();
let c = command();
assert_eq!(apply(&mut s, 0, &c), success(&c, 0));
assert_eq!(apply(&mut s, 1, &c), success(&c, 0));
drop(s);
let path = dir.path().join("db");
let db = Database::open(&path).expect("raw duplicate db");
let tx = db.begin_write().expect("retarget decision write");
let mut key = created::object_bytes(&c.object).expect("object key");
key.extend(c.prepared.version_id);
let mut version = Vec::new();
created::put_prepared(&mut version, &c.prepared);
version.extend(encode_id(id(1)));
version.extend(binding().revision.to_be_bytes());
tx.open_table(VERSIONS)
.expect("versions")
.insert(key.as_slice(), version.as_slice())
.expect("retarget version");
let mut event = created::object_bytes(&c.object).expect("event object");
created::put_prepared(&mut event, &c.prepared);
event.extend(encode_id(id(1)));
event.extend(created::binding_bytes(&binding()).expect("binding bytes"));
tx.open_table(EVENTS)
.expect("events")
.insert(c.operation_id.as_slice(), event.as_slice())
.expect("retarget event");
let canonical = c.encode().expect("canonical");
let mut operation = u64::try_from(canonical.len()).expect("length").to_be_bytes().to_vec();
operation.extend(canonical);
operation.extend(created::result_bytes(&success(&c, 1)));
tx.open_table(OPERATIONS)
.expect("operations")
.insert(c.operation_id.as_slice(), operation.as_slice())
.expect("retarget original response");
for index in 0..=1 {
let mut result = encode_id(id(index));
result.extend(created::result_bytes(&success(&c, 1)));
tx.open_table(APPLY_RESULTS)
.expect("results")
.insert(index, result.as_slice())
.expect("retarget duplicate response");
}
tx.commit().expect("coherent retarget commit");
drop(db);
assert!(
matches!(Store::open(&path, identity()), Err(StoreError::CorruptRecord)),
"U11 original decision cannot retarget duplicate log"
);
}
#[test]
fn created_process_child() {
if let Some(path) = std::env::var_os("U11_CHILD_DB") {
let mut s = Store::initialize(path, identity()).expect("child init");
s.pin_capture_binding(binding()).expect("child pin");
assert_eq!(apply(&mut s, 0, &command()), success(&command(), 0));
complete(&s);
println!("U11_ACK");
std::io::stdout().flush().expect("flush ACK");
loop {
std::thread::park()
}
}
let (_dir, mut s) = fixture();
assert_eq!(apply(&mut s, 0, &command()), success(&command(), 0));
complete(&s);
}
#[test]
fn created_acknowledged_apply_survives_child_kill() {
let dir = tempfile::tempdir().expect("dir");
let path = dir.path().join("db");
let mut child = Command::new(std::env::current_exe().expect("test executable"))
.args(["--exact", "application_tests::created_process_child", "--nocapture"])
.env("U11_CHILD_DB", &path)
.stdout(Stdio::piped())
.stderr(Stdio::inherit())
.spawn()
.expect("spawn");
let stdout = child.stdout.take().expect("stdout");
let (tx, rx) = mpsc::channel();
let reader = std::thread::spawn(move || {
for line in BufReader::new(stdout).lines() {
let line = line.expect("child line");
if line.ends_with("U11_ACK") {
let _ = tx.send(());
break;
}
}
});
let acknowledged = rx.recv_timeout(Duration::from_secs(20));
child.kill().expect("kill child");
child.wait().expect("wait child");
reader.join().expect("reader cleanup");
assert!(acknowledged.is_ok(), "U11 bounded child ACK missing");
complete(&reopen(&path));
}
#[test]
fn created_power_cut_before_and_after_apply_flush() {
let (dir, mut s) = fixture();
append(&mut s, 0, &command());
s.save_committed(id(0)).expect("committed");
drop(s);
let path = dir.path().join("db");
let backend = CrashBackend::new(&path);
let mut s = Store::from_database(backend.database(), identity()).expect("backend");
assert_eq!(s.apply_next_created(id(0)).expect("apply ACK"), success(&command(), 0));
backend.cut();
drop(s);
complete(&reopen(&path));
for sync in 1..=2 {
for after in [false, true] {
let (dir, mut s) = fixture();
append(&mut s, 0, &command());
s.save_committed(id(0)).expect("committed");
drop(s);
let path = dir.path().join("db");
let backend = CrashBackend::new(&path);
let mut s = Store::from_database(backend.database(), identity()).expect("backend");
let (entered, resume) = backend.pause_sync(sync, after);
let writer = std::thread::spawn(move || {
let result = s.apply_next_created(id(0));
(s, result)
});
entered.wait();
backend.cut();
resume.wait();
let (s, result) = writer.join().expect("writer");
drop(s);
let reopened = Store::open(&path, identity());
assert!(
reopened.is_ok(),
"U11 apply cut must reopen whole old/new state sync={sync} after={after}: {:?}",
reopened.as_ref().err()
);
let s = reopened.expect("asserted reopen");
if s.last_applied().expect("last").is_some() {
complete(&s)
} else {
assert!(result.is_err());
assert_eq!(s.created_head(&command().object).expect("old head"), None);
assert_eq!(
s.created_version(&command().object, command().prepared.version_id)
.expect("old version"),
None
);
assert_eq!(s.created_event(command().operation_id).expect("old event"), None);
assert_eq!(s.created_operation(command().operation_id).expect("old op"), None);
assert_eq!(s.created_apply_result(0).expect("old result"), None)
}
eprintln!("U11 cut boundary sync={sync} after={after}");
}
}
}
+63
View File
@@ -0,0 +1,63 @@
use crate::{LOGS, LogId, META, Store, StoreError, decode_id, decode_log, encode_id, number};
use redb::{ReadableDatabase, ReadableTable};
pub(crate) fn validate_committed(
meta: &impl ReadableTable<&'static str, &'static [u8]>,
logs: &impl ReadableTable<u64, &'static [u8]>,
) -> Result<Option<LogId>, StoreError> {
let schema = number(meta.get("schema")?.ok_or(StoreError::CorruptRecord)?.value())?;
let marker = meta.get("committed")?;
match (schema, marker) {
(1 | 3 | 4 | 5 | 6, None) => {
if matches!(schema, 3..=6) && meta.get("last_applied")?.is_none_or(|v| v.value() != [0]) {
return Err(StoreError::CorruptRecord);
}
Ok(None)
}
(2 | 3 | 4 | 5 | 6, Some(marker)) => {
let id = decode_id(marker.value())?;
let record = logs.get(id.index)?.ok_or(StoreError::CorruptRecord)?;
if decode_log(record.value())?.id != id {
return Err(StoreError::CorruptRecord);
}
Ok(Some(id))
}
(1 | 2, _) => Err(StoreError::CorruptRecord),
_ => Err(StoreError::UnsupportedSchema),
}
}
impl Store {
/// Read the exact committed boundary, validating it against the stored log.
pub fn read_committed(&self) -> Result<Option<LogId>, StoreError> {
let tx = self.database.begin_read()?;
validate_committed(&tx.open_table(META)?, &tx.open_table(LOGS)?)
}
/// Persist a monotonic committed boundary supplied by the authority owner.
pub fn save_committed(&mut self, id: LogId) -> Result<(), StoreError> {
let tx = self.write_transaction()?;
{
let mut meta = tx.open_table(META)?;
let logs = tx.open_table(LOGS)?;
let previous = validate_committed(&meta, &logs)?;
let record = logs.get(id.index)?.ok_or(StoreError::InvalidCommittedLog)?;
if decode_log(record.value())?.id != id {
return Err(StoreError::InvalidCommittedLog);
}
if let Some(previous) = previous {
if id.index < previous.index {
return Err(StoreError::CommittedRegression);
}
if id == previous {
return Ok(());
}
}
meta.insert("committed", encode_id(id).as_slice())?;
if previous.is_none() && !matches!(number(meta.get("schema")?.ok_or(StoreError::CorruptRecord)?.value())?, 3..=6) {
meta.insert("schema", 2u64.to_be_bytes().as_slice())?;
}
}
self.commit(tx)
}
}
+247
View File
@@ -0,0 +1,247 @@
use crate::{LogId, StoreError, decode_id, encode_id};
pub const MAX_OBJECT_KEY_LENGTH: usize = 4096;
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ObjectIdentity {
pub bucket_incarnation: [u8; 16],
pub key: Vec<u8>,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PreparedIdentity {
pub version_id: [u8; 16],
pub preparation_id: [u8; 16],
pub content_digest: [u8; 32],
pub content_length: u64,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DecideCreated {
pub operation_id: [u8; 16],
pub object: ObjectIdentity,
pub expected_head: Option<[u8; 16]>,
pub prepared: PreparedIdentity,
pub binding_revision: u64,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CaptureBinding {
pub revision: u64,
pub binding_id: [u8; 16],
pub target_id: [u8; 16],
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum CreatedResult {
Created {
version_id: [u8; 16],
event_id: [u8; 16],
decision_log_id: LogId,
},
HeadMismatch {
actual_head: Option<[u8; 16]>,
},
OperationConflict,
BindingMismatch,
VersionConflict,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CreatedVersion {
pub prepared: PreparedIdentity,
pub decision_log_id: LogId,
pub binding_revision: u64,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CreatedEvent {
pub object: ObjectIdentity,
pub prepared: PreparedIdentity,
pub decision_log_id: LogId,
pub binding: CaptureBinding,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CreatedOperation {
pub canonical_command: Vec<u8>,
pub result: CreatedResult,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CreatedOperationProvenance {
pub operation: CreatedOperation,
pub first_decision_log_id: LogId,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CreatedApplyResult {
pub log_id: LogId,
pub result: CreatedResult,
}
pub(crate) struct Decoder<'a> {
bytes: &'a [u8],
}
impl<'a> Decoder<'a> {
pub(crate) fn new(bytes: &'a [u8]) -> Self {
Self { bytes }
}
pub(crate) fn take<const N: usize>(&mut self) -> Result<[u8; N], StoreError> {
let a = self
.bytes
.get(..N)
.ok_or(StoreError::CorruptRecord)?
.try_into()
.map_err(|_| StoreError::CorruptRecord)?;
self.bytes = &self.bytes[N..];
Ok(a)
}
pub(crate) fn u64(&mut self) -> Result<u64, StoreError> {
Ok(u64::from_be_bytes(self.take()?))
}
pub(crate) fn uuid(&mut self) -> Result<[u8; 16], StoreError> {
let id = self.take()?;
if id == [0; 16] {
return Err(StoreError::CorruptRecord);
}
Ok(id)
}
pub(crate) fn option(&mut self) -> Result<Option<[u8; 16]>, StoreError> {
match self.take::<1>()?[0] {
0 => Ok(None),
1 => Ok(Some(self.uuid()?)),
_ => Err(StoreError::CorruptRecord),
}
}
pub(crate) fn object(&mut self) -> Result<ObjectIdentity, StoreError> {
let bucket_incarnation = self.uuid()?;
let len = usize::try_from(u32::from_be_bytes(self.take()?)).map_err(|_| StoreError::CorruptRecord)?;
if len > MAX_OBJECT_KEY_LENGTH {
return Err(StoreError::CorruptRecord);
}
let key = self.bytes.get(..len).ok_or(StoreError::CorruptRecord)?.to_vec();
self.bytes = &self.bytes[len..];
Ok(ObjectIdentity { bucket_incarnation, key })
}
pub(crate) fn prepared(&mut self) -> Result<PreparedIdentity, StoreError> {
Ok(PreparedIdentity {
version_id: self.uuid()?,
preparation_id: self.uuid()?,
content_digest: self.take()?,
content_length: self.u64()?,
})
}
pub(crate) fn id(&mut self) -> Result<LogId, StoreError> {
decode_id(&self.take::<24>()?)
}
pub(crate) fn binding(&mut self) -> Result<CaptureBinding, StoreError> {
Ok(CaptureBinding {
revision: self.u64()?,
binding_id: self.uuid()?,
target_id: self.uuid()?,
})
}
pub(crate) fn finish(self) -> Result<(), StoreError> {
if self.bytes.is_empty() {
Ok(())
} else {
Err(StoreError::CorruptRecord)
}
}
pub(crate) fn result(&mut self) -> Result<CreatedResult, StoreError> {
match self.take::<1>()?[0] {
0 => Ok(CreatedResult::Created {
version_id: self.uuid()?,
event_id: self.uuid()?,
decision_log_id: self.id()?,
}),
1 => Ok(CreatedResult::HeadMismatch {
actual_head: self.option()?,
}),
2 => Ok(CreatedResult::OperationConflict),
3 => Ok(CreatedResult::BindingMismatch),
4 => Ok(CreatedResult::VersionConflict),
_ => Err(StoreError::CorruptRecord),
}
}
}
pub(crate) fn put_option(b: &mut Vec<u8>, id: Option<[u8; 16]>) {
b.push(u8::from(id.is_some()));
if let Some(id) = id {
b.extend(id)
}
}
pub(crate) fn object_bytes(o: &ObjectIdentity) -> Result<Vec<u8>, StoreError> {
if o.bucket_incarnation == [0; 16] || o.key.len() > MAX_OBJECT_KEY_LENGTH {
return Err(StoreError::CorruptRecord);
}
let capacity = 20usize.checked_add(o.key.len()).ok_or(StoreError::CorruptRecord)?;
let mut b = Vec::with_capacity(capacity);
b.extend(o.bucket_incarnation);
b.extend(
u32::try_from(o.key.len())
.map_err(|_| StoreError::CorruptRecord)?
.to_be_bytes(),
);
b.extend(&o.key);
Ok(b)
}
pub(crate) fn put_prepared(b: &mut Vec<u8>, p: &PreparedIdentity) {
b.extend(p.version_id);
b.extend(p.preparation_id);
b.extend(p.content_digest);
b.extend(p.content_length.to_be_bytes())
}
pub(crate) fn binding_bytes(v: &CaptureBinding) -> Result<Vec<u8>, StoreError> {
let mut b = v.revision.to_be_bytes().to_vec();
b.extend(v.binding_id);
b.extend(v.target_id);
let mut d = Decoder::new(&b);
d.binding()?;
d.finish()?;
Ok(b)
}
pub(crate) fn result_bytes(r: &CreatedResult) -> Vec<u8> {
let mut b = Vec::new();
match r {
CreatedResult::Created {
version_id,
event_id,
decision_log_id,
} => {
b.push(0);
b.extend(version_id);
b.extend(event_id);
b.extend(encode_id(*decision_log_id))
}
CreatedResult::HeadMismatch { actual_head } => {
b.push(1);
put_option(&mut b, *actual_head)
}
CreatedResult::OperationConflict => b.push(2),
CreatedResult::BindingMismatch => b.push(3),
CreatedResult::VersionConflict => b.push(4),
}
b
}
impl DecideCreated {
pub fn encode(&self) -> Result<Vec<u8>, StoreError> {
let object = object_bytes(&self.object)?;
let capacity = object.len().checked_add(134).ok_or(StoreError::CorruptRecord)?;
let mut b = Vec::with_capacity(capacity);
b.extend(b"CRTD\x01");
b.extend(self.operation_id);
b.extend(object);
put_option(&mut b, self.expected_head);
put_prepared(&mut b, &self.prepared);
b.extend(self.binding_revision.to_be_bytes());
Self::decode(&b)?;
Ok(b)
}
pub fn decode(bytes: &[u8]) -> Result<Self, StoreError> {
let mut d = Decoder::new(bytes);
if d.take::<5>()? != *b"CRTD\x01" {
return Err(StoreError::CorruptRecord);
}
let c = Self {
operation_id: d.uuid()?,
object: d.object()?,
expected_head: d.option()?,
prepared: d.prepared()?,
binding_revision: d.u64()?,
};
d.finish()?;
Ok(c)
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,118 @@
//! Owned consistent native application snapshot view.
use crate::{Store, StoreError, application, snapshot};
use redb::{ReadTransaction, ReadableDatabase, ReadableTable, ReadableTableMetadata};
use std::{
io::Write,
sync::{Arc, Mutex},
};
#[derive(Debug, thiserror::Error)]
pub enum NativeSnapshotError {
#[error(transparent)]
Store(#[from] StoreError),
#[error(transparent)]
Snapshot(#[from] snapshot::SnapshotError),
#[error("native snapshot Store lock poisoned")]
Poisoned,
#[error("native snapshot worker failed")]
Worker(#[source] tokio::task::JoinError),
}
pub struct NativeSnapshotView {
// Drop the transaction before the owner so the Database stays open throughout the view.
transaction: ReadTransaction,
header: snapshot::Header,
counts: [u64; 4],
schema: u64,
_owner: Arc<Mutex<Store>>,
}
impl NativeSnapshotView {
pub(crate) fn acquire(owner: Arc<Mutex<Store>>) -> Result<Self, NativeSnapshotError> {
let transaction = {
let store = owner.lock().map_err(|_| NativeSnapshotError::Poisoned)?;
if !store.writable {
return Err(StoreError::WriteDisabled.into());
}
store.database.begin_read().map_err(StoreError::from)?
};
let (applied, membership) = application::native_application_state(&transaction)?;
let (header, schema) = {
let meta = transaction.open_table(crate::META).map_err(StoreError::from)?;
let identity = meta
.get("identity")
.map_err(StoreError::from)?
.ok_or(StoreError::CorruptRecord)?;
let schema = crate::number(
meta.get("schema")
.map_err(StoreError::from)?
.ok_or(StoreError::CorruptRecord)?
.value(),
)?;
(
snapshot::Header {
cluster_id: crate::decode_identity(identity.value())?.cluster_id,
applied,
membership,
binding: application::read_binding(&meta)?,
},
schema,
)
};
let mut counts = [0; 4];
for (index, definition) in [
application::HEADS,
application::VERSIONS,
application::EVENTS,
application::OPERATIONS,
]
.into_iter()
.enumerate()
{
counts[index] = transaction
.open_table(definition)
.map_err(StoreError::from)?
.len()
.map_err(StoreError::from)?;
}
Ok(Self {
transaction,
header,
counts,
schema,
_owner: owner,
})
}
pub fn stream_capability_artifact<W: Write>(
self,
writer: W,
limits: snapshot::Limits,
) -> Result<snapshot::CapabilityArtifact<W>, NativeSnapshotError> {
if self.schema != 6 {
return Err(StoreError::UnsupportedSchema.into());
}
let output = snapshot::CapabilityOutput::new(writer, limits.total_bytes);
Ok(self.stream(output, limits)?.finish())
}
pub fn stream<W: Write>(self, writer: W, limits: snapshot::Limits) -> Result<W, NativeSnapshotError> {
let mut encoder = snapshot::Encoder::new(writer, &self.header, self.counts, limits)?;
for (definition, section) in [
(application::HEADS, snapshot::Section::Heads),
(application::VERSIONS, snapshot::Section::Versions),
(application::EVENTS, snapshot::Section::Events),
(application::OPERATIONS, snapshot::Section::Operations),
] {
let table = self.transaction.open_table(definition).map_err(StoreError::from)?;
for row in table.iter().map_err(StoreError::from)? {
let (key, value) = row.map_err(StoreError::from)?;
encoder.row(snapshot::Row {
section,
key: key.value(),
value: value.value(),
})?;
}
}
Ok(encoder.finish()?)
}
}
+371
View File
@@ -0,0 +1,371 @@
//! Validated capture peer metadata and its independent CNOD v1 codec.
use serde::{Deserialize, Deserializer, Serialize, Serializer, ser::SerializeStruct};
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct CaptureNode {
rpc_uri: String,
}
#[derive(Debug, thiserror::Error)]
pub enum CaptureNodeError {
#[error("invalid RPC URI")]
InvalidRpcUri,
#[error("invalid RPC URI")]
InvalidRpcUriParse(#[source] url::ParseError),
#[error("RPC URI too long")]
RpcUriTooLong,
#[error("invalid node magic")]
InvalidNodeMagic,
#[error("unsupported node version")]
UnsupportedNodeVersion,
#[error("invalid node length")]
InvalidNodeLength,
#[error("truncated node record")]
TruncatedNodeRecord,
#[error("invalid node UTF-8")]
InvalidNodeUtf8(#[source] std::str::Utf8Error),
#[error("trailing node bytes")]
TrailingNodeBytes,
}
impl CaptureNodeError {
pub fn code(&self) -> &'static str {
match self {
Self::InvalidRpcUri | Self::InvalidRpcUriParse(_) => "invalid_rpc_uri",
Self::RpcUriTooLong => "rpc_uri_too_long",
Self::InvalidNodeMagic => "invalid_node_magic",
Self::UnsupportedNodeVersion => "unsupported_node_version",
Self::InvalidNodeLength => "invalid_node_length",
Self::TruncatedNodeRecord => "truncated_node_record",
Self::InvalidNodeUtf8(_) => "invalid_node_utf8",
Self::TrailingNodeBytes => "trailing_node_bytes",
}
}
}
impl CaptureNode {
pub fn new(rpc_uri: String) -> Result<Self, CaptureNodeError> {
let node = Self { rpc_uri };
node.validate()?;
Ok(node)
}
pub fn rpc_uri(&self) -> &str {
&self.rpc_uri
}
pub fn validate(&self) -> Result<(), CaptureNodeError> {
let uri = &self.rpc_uri;
if uri.len() > 4096 {
return Err(CaptureNodeError::RpcUriTooLong);
}
if uri.is_empty() || uri.chars().any(|c| c.is_whitespace() || c.is_control() || c == '\\') {
return Err(CaptureNodeError::InvalidRpcUri);
}
let rest = uri
.strip_prefix("http://")
.or_else(|| uri.strip_prefix("https://"))
.ok_or(CaptureNodeError::InvalidRpcUri)?;
if uri.contains('?') || uri.contains('#') {
return Err(CaptureNodeError::InvalidRpcUri);
}
let (authority, path) = rest.split_once('/').map_or((rest, ""), |(a, p)| (a, p));
// An empty suffix after the first slash is the only allowed path.
if !path.is_empty() || authority.contains('@') {
return Err(CaptureNodeError::InvalidRpcUri);
}
let (host, port) = if authority.starts_with('[') {
let end = authority.find(']').ok_or(CaptureNodeError::InvalidRpcUri)?;
let port = authority
.get(end + 1..)
.and_then(|suffix| suffix.strip_prefix(':'))
.ok_or(CaptureNodeError::InvalidRpcUri)?;
(&authority[..=end], port)
} else {
authority.split_once(':').ok_or(CaptureNodeError::InvalidRpcUri)?
};
if host.is_empty() || port.is_empty() || !port.bytes().all(|b| b.is_ascii_digit()) {
return Err(CaptureNodeError::InvalidRpcUri);
}
let port = port.parse::<u16>().map_err(|_| CaptureNodeError::InvalidRpcUri)?;
if port == 0 {
return Err(CaptureNodeError::InvalidRpcUri);
}
let parsed_url = url::Url::parse(uri).map_err(CaptureNodeError::InvalidRpcUriParse)?;
if parsed_url.host().is_none() {
return Err(CaptureNodeError::InvalidRpcUri);
}
Ok(())
}
pub fn encode(&self) -> Result<Vec<u8>, CaptureNodeError> {
self.validate()?;
let length = u32::try_from(self.rpc_uri.len()).map_err(|_| CaptureNodeError::InvalidNodeLength)?;
let mut bytes = b"CNOD\x01".to_vec();
bytes.extend_from_slice(&length.to_be_bytes());
bytes.extend_from_slice(self.rpc_uri.as_bytes());
Ok(bytes)
}
pub fn decode(bytes: &[u8]) -> Result<Self, CaptureNodeError> {
if bytes.len() < 9 {
return Err(CaptureNodeError::TruncatedNodeRecord);
}
if &bytes[..4] != b"CNOD" {
return Err(CaptureNodeError::InvalidNodeMagic);
}
if bytes[4] != 1 {
return Err(CaptureNodeError::UnsupportedNodeVersion);
}
let length = u32::from_be_bytes(bytes[5..9].try_into().map_err(|_| CaptureNodeError::InvalidNodeLength)?);
if !(1..=4096).contains(&length) {
return Err(CaptureNodeError::InvalidNodeLength);
}
let length = usize::try_from(length).map_err(|_| CaptureNodeError::InvalidNodeLength)?;
let end = 9_usize.checked_add(length).ok_or(CaptureNodeError::InvalidNodeLength)?;
if bytes.len() < end {
return Err(CaptureNodeError::TruncatedNodeRecord);
}
if bytes.len() != end {
return Err(CaptureNodeError::TrailingNodeBytes);
}
let uri = std::str::from_utf8(&bytes[9..end]).map_err(CaptureNodeError::InvalidNodeUtf8)?;
Self::new(uri.to_owned())
}
}
impl Serialize for CaptureNode {
fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {
self.validate().map_err(serde::ser::Error::custom)?;
let mut state = serializer.serialize_struct("CaptureNode", 1)?;
state.serialize_field("rpc_uri", &self.rpc_uri)?;
state.end()
}
}
impl<'de> Deserialize<'de> for CaptureNode {
fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {
#[derive(Deserialize)]
#[serde(deny_unknown_fields)]
struct Fields {
rpc_uri: String,
}
let fields = Fields::deserialize(deserializer).map_err(|_| serde::de::Error::custom("invalid capture node fields"))?;
Self::new(fields.rpc_uri).map_err(serde::de::Error::custom)
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::error::Error;
fn record(payload: &[u8]) -> Vec<u8> {
let mut bytes = b"CNOD\x01".to_vec();
bytes.extend_from_slice(&u32::try_from(payload.len()).expect("test payload length").to_be_bytes());
bytes.extend_from_slice(payload);
bytes
}
fn reject_uri(uri: &str) {
let result = CaptureNode::new(uri.to_owned());
assert!(result.is_err(), "URI should reject: {uri:?}");
assert_eq!(result.expect_err("rejected URI").code(), "invalid_rpc_uri");
}
fn reject_record(bytes: &[u8], code: &str) {
let result = CaptureNode::decode(bytes);
assert!(result.is_err(), "record should reject with {code}");
assert_eq!(result.expect_err("rejected record").code(), code);
}
#[test]
fn actual_openraft_node_bound() {
fn require_node<T: openraft::Node>() {}
require_node::<CaptureNode>();
assert_eq!(CaptureNode::default().rpc_uri(), "");
}
#[test]
fn default_and_empty_rejected_at_every_boundary() {
let node = CaptureNode::default();
assert_eq!(node.rpc_uri(), "");
reject_uri("");
assert_eq!(node.validate().expect_err("empty validation").code(), "invalid_rpc_uri");
assert!(serde_json::to_string(&node).is_err());
assert!(serde_json::from_str::<CaptureNode>(r#"{"rpc_uri":""}"#).is_err());
assert_eq!(node.encode().expect_err("empty encoding").code(), "invalid_rpc_uri");
reject_record(&record(b""), "invalid_node_length");
}
#[test]
fn accepted_uris_preserve_exact_bytes() {
for uri in [
"http://node:80",
"https://node:443",
"http://node:1",
"https://node:65535/",
"http://127.0.0.1:9000",
"https://[::1]:443/",
"http://[2001:db8::1]:1",
"http://NoDe:00080/",
] {
let result = CaptureNode::new(uri.to_owned());
assert!(result.is_ok(), "valid URI construction: {uri}");
let node = result.expect("valid URI");
assert_eq!(node.rpc_uri(), uri, "preserve explicit default port and raw URI");
assert_eq!(node.encode().expect("encode URI"), record(uri.as_bytes()));
assert_eq!(CaptureNode::decode(&record(uri.as_bytes())).expect("decode URI"), node);
}
}
#[test]
fn golden_cnod_bytes() {
let node = CaptureNode {
rpc_uri: "http://node:80".to_owned(),
};
let golden = b"CNOD\x01\x00\x00\x00\x0ehttp://node:80";
assert_eq!(node.encode().ok().as_deref(), Some(golden.as_slice()), "golden CNOD bytes");
}
#[test]
fn valid_cnod_decode() {
let golden = b"CNOD\x01\x00\x00\x00\x0ehttp://node:80";
let decoded = CaptureNode::decode(golden);
assert!(decoded.is_ok(), "golden decode succeeds");
assert_eq!(decoded.expect("golden decoded node").rpc_uri(), "http://node:80");
}
#[test]
fn serde_shape_roundtrip_and_strict_fields() {
let node = CaptureNode::new("https://node:443/".to_owned()).expect("serde node");
let json = r#"{"rpc_uri":"https://node:443/"}"#;
assert_eq!(serde_json::to_string(&node).expect("serialize node"), json);
assert_eq!(serde_json::from_str::<CaptureNode>(json).expect("deserialize node"), node);
for json in [
r#"{}"#,
r#"{"rpc_uri":1}"#,
r#"{"rpc_uri":null}"#,
r#"{"rpc_uri":"http://node:1","extra":0}"#,
r#"{"rpc_uri":"http://node:1","rpc_uri":"http://node:2"}"#,
] {
assert!(serde_json::from_str::<CaptureNode>(json).is_err(), "strict fields: {json}");
}
}
#[test]
fn credentials_and_raw_uri_shapes_reject() {
for uri in [
"http://user@node:1",
"http://user:password@node:1",
"http://@node:1",
"HTTP://node:1",
"Https://node:1",
"http://node",
"http://node:",
"http://node:0",
"http://node:-1",
"http://node:+1",
"http://node:65536",
"http://node:999999999999999999999",
"http://[::1:1",
"http://[::1]extra:1",
"http://[invalid]:1",
"http://::1:1",
"http://:1",
"/disk/path",
"file:///disk/path",
"http://node:1/./",
"http://node:1/../",
"http://node:1/%2e/",
"http://node:1/%2E%2e/",
"http://node:1/path",
"http://node:1\\",
" http://node:1",
"http://node:1\n",
"http://no\tde:1",
"http://node:1\u{00a0}",
"http://node:1\u{0085}",
"http://node:1\0",
] {
reject_uri(uri);
}
}
#[test]
fn query_and_fragment_markers_reject() {
for uri in [
"http://node:1?",
"http://node:1?x=y",
"http://node:1/?",
"http://node:1#",
"http://node:1#fragment",
"http://node:1/#",
] {
reject_uri(uri);
}
}
#[test]
fn utf8_byte_length_boundaries() {
for length in [4095, 4096] {
let prefix = "http://node:";
let uri = format!("{prefix}{}1", "0".repeat(length - prefix.len() - 1));
assert_eq!(uri.len(), length);
assert!(url::Url::parse(&uri).is_ok(), "boundary fixture accepted by actual URL parser");
let node = CaptureNode::new(uri.clone()).expect("boundary node");
assert_eq!(node.rpc_uri(), uri);
assert_eq!(
CaptureNode::decode(&node.encode().expect("boundary encoding")).expect("boundary decoding"),
node
);
}
let uri = format!("http://node:{}1", "0".repeat(4097 - "http://node:".len() - 1));
assert_eq!(CaptureNode::new(uri).expect_err("oversize URI").code(), "rpc_uri_too_long");
let unicode = format!("http://{}:1", "é".repeat(2044));
assert!(unicode.chars().count() < 4096);
assert!(unicode.len() > 4096);
assert_eq!(CaptureNode::new(unicode).expect_err("UTF-8 byte limit").code(), "rpc_uri_too_long");
}
#[test]
fn cnod_headers_and_lengths_reject_exact_codes() {
let valid = record(b"http://node:80");
for end in 0..9 {
reject_record(&valid[..end], "truncated_node_record");
}
let mut bad = valid.clone();
bad[0] = b'X';
reject_record(&bad, "invalid_node_magic");
bad = valid.clone();
bad[4] = 2;
reject_record(&bad, "unsupported_node_version");
for length in [0_u32, 4097, u32::MAX] {
bad = b"CNOD\x01".to_vec();
bad.extend_from_slice(&length.to_be_bytes());
reject_record(&bad, "invalid_node_length");
}
for end in 9..valid.len() {
reject_record(&valid[..end], "truncated_node_record");
}
reject_record(&record(&[0xff]), "invalid_node_utf8");
reject_record(&record(&[0xc3, 0x28]), "invalid_node_utf8");
reject_record(&record(b"invalid URI"), "invalid_rpc_uri");
}
#[test]
fn trailing_bytes_reject() {
for tail in [&b"x"[..], &b"xyz"[..]] {
let mut bytes = record(b"http://node:80");
bytes.extend_from_slice(tail);
reject_record(&bytes, "trailing_node_bytes");
}
}
#[test]
fn errors_and_sources_redact_input() {
let sentinel = "http://SECRET_USER:SECRET_PASSWORD@node:1";
let error = CaptureNode::new(sentinel.to_owned()).expect_err("credentials reject");
let mut current: &dyn Error = &error;
loop {
let rendered = format!("{current} {current:?}");
assert!(!rendered.contains("SECRET"));
match current.source() {
Some(source) => current = source,
None => break,
}
}
let json = format!(r#"{{"rpc_uri":"{sentinel}"}}"#);
let error = serde_json::from_str::<CaptureNode>(&json).expect_err("serde credentials reject");
assert!(!format!("{error} {error:?}").contains("SECRET"));
let json = format!(r#"{{"{sentinel}":0}}"#);
let error = serde_json::from_str::<CaptureNode>(&json).expect_err("unknown fields reject");
assert!(!format!("{error} {error:?}").contains("SECRET"));
let error = CaptureNode::new("http://[bad]:1".to_owned()).expect_err("parser rejection");
assert!(error.source().is_some(), "retain URL parser source");
let error = CaptureNode::decode(&record(&[0xff])).expect_err("UTF-8 rejection");
assert!(error.source().is_some(), "retain UTF-8 source");
}
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+43
View File
@@ -0,0 +1,43 @@
use crate::{LOGS, META, Store, StoreError, decode_id, decode_log, encode_id};
use redb::ReadableTable;
impl Store {
/// Atomically remove every log entry at or after `from`.
pub fn truncate_suffix(&mut self, from: u64) -> Result<(), StoreError> {
let tx = self.write_transaction()?;
{
let mut meta = tx.open_table(META)?;
let logs = tx.open_table(LOGS)?;
if crate::committed::validate_committed(&meta, &logs)?.is_some_and(|id| from <= id.index) {
return Err(StoreError::CommittedLogProtected);
}
drop(logs);
let tail = meta.get("tail")?.map(|v| decode_id(v.value())).transpose()?;
if tail.is_none_or(|id| from > id.index) {
return Ok(());
}
let mut logs = tx.open_table(LOGS)?;
let new_tail = if from == 0 {
None
} else {
let index = from - 1;
let record = logs.get(index)?.ok_or(StoreError::InvalidLogSequence)?;
let id = decode_log(record.value())?.id;
if id.index != index {
return Err(StoreError::CorruptRecord);
}
Some(id)
};
logs.retain(|index, _| index < from)?;
match new_tail {
Some(id) => {
meta.insert("tail", encode_id(id).as_slice())?;
}
None => {
meta.remove("tail")?;
}
}
}
self.commit(tx)
}
}
-1
View File
@@ -34,7 +34,6 @@ hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
arc-swap.workspace = true
hotpath.workspace = true
tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
tonic = { workspace = true, features = ["gzip", "deflate"] }
+5 -131
View File
@@ -12,10 +12,9 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use arc_swap::ArcSwap;
use std::collections::HashMap;
use std::sync::LazyLock;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, LazyLock};
use tokio::sync::RwLock;
use tonic::transport::Channel;
@@ -28,42 +27,6 @@ pub static GLOBAL_ROOT_CERT: LazyLock<RwLock<Option<Vec<u8>>>> = LazyLock::new(|
pub static GLOBAL_MTLS_IDENTITY: LazyLock<RwLock<Option<MtlsIdentityPem>>> = LazyLock::new(|| RwLock::new(None));
pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
#[derive(Clone, Default)]
pub struct OutboundTlsSnapshot {
pub generation: u64,
pub root_ca_pem: Option<Vec<u8>>,
pub mtls_identity: Option<MtlsIdentityPem>,
}
static GLOBAL_OUTBOUND_TLS_STATE: LazyLock<ArcSwap<OutboundTlsSnapshot>> =
LazyLock::new(|| ArcSwap::from_pointee(OutboundTlsSnapshot::default()));
pub fn get_global_outbound_tls_snapshot() -> Arc<OutboundTlsSnapshot> {
GLOBAL_OUTBOUND_TLS_STATE.load_full()
}
/// Publish identity, trust and generation together; the individual globals are legacy views.
pub async fn publish_global_outbound_tls_snapshot(snapshot: OutboundTlsSnapshot) -> u64 {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
if current.generation != 0
&& current.root_ca_pem == snapshot.root_ca_pem
&& current.mtls_identity == snapshot.mtls_identity
{
return Arc::clone(current);
}
Arc::new(OutboundTlsSnapshot {
generation: snapshot.generation.max(current.generation.saturating_add(1)),
root_ca_pem: snapshot.root_ca_pem.clone(),
mtls_identity: snapshot.mtls_identity.clone(),
})
});
let accepted = get_global_outbound_tls_snapshot();
*GLOBAL_ROOT_CERT.write().await = accepted.root_ca_pem.clone();
*GLOBAL_MTLS_IDENTITY.write().await = accepted.mtls_identity.clone();
GLOBAL_OUTBOUND_TLS_GENERATION.store(accepted.generation, Ordering::Relaxed);
accepted.generation
}
/// Log level to use when reporting cached gRPC connection eviction.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum ConnectionEvictionLogLevel {
@@ -122,27 +85,17 @@ pub async fn get_global_addr() -> String {
/// # Arguments
/// * `cert` - A vector of bytes representing the PEM-encoded root CA certificate.
pub async fn set_global_root_cert(cert: Vec<u8>) {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
let mut next = (**current).clone();
next.root_ca_pem = Some(cert.clone());
Arc::new(next)
});
*GLOBAL_ROOT_CERT.write().await = Some(cert);
}
/// Clear the global root CA certificate for outbound gRPC clients.
pub async fn clear_global_root_cert() {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
let mut next = (**current).clone();
next.root_ca_pem = None;
Arc::new(next)
});
*GLOBAL_ROOT_CERT.write().await = None;
}
/// Get the global root CA certificate for outbound gRPC clients.
pub async fn get_global_root_cert() -> Option<Vec<u8>> {
get_global_outbound_tls_snapshot().root_ca_pem.clone()
GLOBAL_ROOT_CERT.read().await.clone()
}
/// Set the global mTLS identity (cert+key PEM) for outbound gRPC clients.
@@ -152,32 +105,22 @@ pub async fn get_global_root_cert() -> Option<Vec<u8>> {
/// # Arguments
/// * `identity` - An optional MtlsIdentityPem struct containing the cert and key PEM.
pub async fn set_global_mtls_identity(identity: Option<MtlsIdentityPem>) {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
let mut next = (**current).clone();
next.mtls_identity = identity.clone();
Arc::new(next)
});
*GLOBAL_MTLS_IDENTITY.write().await = identity;
}
/// Get the global mTLS identity for outbound gRPC clients.
pub async fn get_global_mtls_identity() -> Option<MtlsIdentityPem> {
get_global_outbound_tls_snapshot().mtls_identity.clone()
GLOBAL_MTLS_IDENTITY.read().await.clone()
}
/// Set the global outbound TLS generation.
pub fn set_global_outbound_tls_generation(generation: u64) {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
let mut next = (**current).clone();
next.generation = generation;
Arc::new(next)
});
GLOBAL_OUTBOUND_TLS_GENERATION.store(generation, Ordering::Relaxed);
}
/// Get the global outbound TLS generation.
pub fn get_global_outbound_tls_generation() -> u64 {
get_global_outbound_tls_snapshot().generation
GLOBAL_OUTBOUND_TLS_GENERATION.load(Ordering::Relaxed)
}
/// Evict a stale/dead connection from the global connection cache.
@@ -251,77 +194,8 @@ pub async fn clear_all_connections() {
///
/// When present, gRPC clients will present this identity to servers requesting/requiring mTLS.
/// When absent, clients proceed with standard server-authenticated TLS.
#[derive(Clone, PartialEq, Eq)]
#[derive(Clone, Debug)]
pub struct MtlsIdentityPem {
pub cert_pem: Vec<u8>,
pub key_pem: Vec<u8>,
}
impl std::fmt::Debug for MtlsIdentityPem {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("MtlsIdentityPem")
.field("cert_bytes", &self.cert_pem.len())
.field("key_pem", &"[REDACTED]")
.finish()
}
}
#[cfg(test)]
mod tls_snapshot_tests {
use super::*;
#[tokio::test]
async fn outbound_snapshot_publication_is_coherent_and_skips_unchanged_material() {
let start = get_global_outbound_tls_generation().saturating_add(1);
let snapshot = OutboundTlsSnapshot {
generation: start,
root_ca_pem: Some(vec![1]),
mtls_identity: Some(MtlsIdentityPem {
cert_pem: vec![1],
key_pem: vec![1],
}),
};
let first = publish_global_outbound_tls_snapshot(snapshot.clone()).await;
assert_eq!(
publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
generation: first.saturating_add(1),
..snapshot
})
.await,
first
);
let writer = tokio::spawn(async move {
for tag in 2_u8..=255 {
publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
generation: start.saturating_add(u64::from(tag)),
root_ca_pem: Some(vec![tag]),
mtls_identity: Some(MtlsIdentityPem {
cert_pem: vec![tag],
key_pem: vec![tag],
}),
})
.await;
tokio::task::yield_now().await;
}
});
while !writer.is_finished() {
let state = get_global_outbound_tls_snapshot();
let identity = state.mtls_identity.as_ref().expect("identity in complete snapshot");
assert_eq!(state.root_ca_pem.as_ref().expect("roots in complete snapshot"), &identity.cert_pem);
assert_eq!(identity.cert_pem, identity.key_pem);
tokio::task::yield_now().await;
}
writer.await.expect("writer completes");
}
#[test]
fn mtls_identity_debug_redacts_private_key() {
let identity = MtlsIdentityPem {
cert_pem: b"public certificate".to_vec(),
key_pem: b"PRIVATE-KEY-MARKER".to_vec(),
};
let debug = format!("{identity:?}");
assert!(!debug.contains("PRIVATE-KEY-MARKER"));
assert!(debug.contains("[REDACTED]"));
}
}
-9
View File
@@ -1542,15 +1542,6 @@ impl RustFSTestClusterEnvironment {
self.start_node_from_binary(node_idx, &binary_path).await
}
/// Start a process and wait only for TCP, allowing tests to bring up a
/// quorum one node at a time before the S3 service can become ready.
pub async fn start_node_process(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let binary_path = rustfs_binary_path();
let volumes_arg = self.build_volumes_arg();
self.spawn_node(node_idx, &binary_path, &volumes_arg)?;
self.wait_for_node_ready(&self.nodes[node_idx].address, node_idx).await
}
/// Start one stopped cluster node with a specific RustFS binary while
/// preserving the cluster's volume layout and that node's data directory.
pub async fn start_node_from_binary(
@@ -1,512 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! What one stack answered, and how two answers are compared.
//!
//! An [`Observation`] keeps only what a client acts on: the HTTP status, the S3 error code, the
//! allowlisted response headers and a few typed fields decoded from the response. The error
//! message is kept for diagnostics and never compared: message wording is judged by the wire
//! corpus, not here. The comparator is pure so it can be tested without a server.
use serde::Serialize;
use std::collections::{BTreeMap, BTreeSet};
use std::path::{Path, PathBuf};
/// Response headers compared exactly (after [`normalized_header_value`]).
const HEADER_ALLOWLIST: [&str; 7] = [
"content-type",
"x-amz-version-id",
"etag",
"x-amz-delete-marker",
"content-range",
"accept-ranges",
"x-amz-request-charged",
];
/// Every response header with this prefix is compared as well.
const HEADER_ALLOWLIST_PREFIX: &str = "x-amz-server-side-encryption";
/// Placeholder for a field one side did not send.
const ABSENT: &str = "<absent>";
/// The lowercase header name if the comparator looks at this header.
pub(crate) fn compared_header(name: &str) -> Option<String> {
let name = name.to_ascii_lowercase();
(HEADER_ALLOWLIST.contains(&name.as_str()) || name.starts_with(HEADER_ALLOWLIST_PREFIX)).then_some(name)
}
/// A version ID is minted per process, so its value never matches across two servers; only
/// whether it is the literal `null` (an unversioned write) carries meaning.
fn normalized_header_value(name: &str, value: &str) -> String {
if name == "x-amz-version-id" && value != "null" {
"<version-id>".to_owned()
} else {
value.to_owned()
}
}
/// One request's outcome as a client sees it.
#[derive(Clone, Debug, Serialize)]
pub(crate) struct Observation {
/// The S3 operation, named as in the `s3s::S3` trait (`GetObject`).
pub op: &'static str,
/// Which request of the script this was (`normal`, `negative`, or a setup label).
pub case: &'static str,
/// HTTP status; 0 when no response arrived.
pub status: u16,
/// The S3 error code, or the SDK's error kind when there was no service error.
pub error_code: Option<String>,
/// Allowlisted headers only, lowercase names, normalized values.
pub headers: BTreeMap<String, String>,
/// Typed values decoded from the response body, never free text.
pub fields: BTreeMap<String, String>,
/// Diagnostic only; never compared.
#[serde(skip)]
pub message: Option<String>,
}
impl Observation {
/// Builds an observation, keeping only the allowlisted headers.
pub(crate) fn new<'h>(
op: &'static str,
case: &'static str,
status: u16,
error_code: Option<String>,
headers: impl IntoIterator<Item = (&'h str, &'h str)>,
) -> Self {
let headers = headers
.into_iter()
.filter_map(|(name, value)| compared_header(name).map(|name| (name, value)))
.map(|(name, value)| {
let value = normalized_header_value(&name, value);
(name, value)
})
.collect();
Self {
op,
case,
status,
error_code,
headers,
fields: BTreeMap::new(),
message: None,
}
}
pub(crate) fn with_field(mut self, name: &str, value: impl Into<String>) -> Self {
self.fields.insert(name.to_owned(), value.into());
self
}
pub(crate) fn is_success(&self) -> bool {
(200..300).contains(&self.status)
}
}
/// One field on which the two stacks answered differently.
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub(crate) struct Difference {
pub op: String,
pub case: String,
pub field: String,
pub a: String,
pub b: String,
}
/// Every difference between two transcripts of the same script, matched by `(op, case)`.
pub(crate) fn compare(a: &[Observation], b: &[Observation]) -> Vec<Difference> {
let mut differences = Vec::new();
let (index_a, duplicates_a) = index(a);
let (index_b, duplicates_b) = index(b);
for (op, case) in duplicates_a.union(&duplicates_b) {
let count = |side: &[Observation]| {
side.iter()
.filter(|observation| (observation.op, observation.case) == (*op, *case))
.count()
.to_string()
};
differences.push(difference(op, case, "duplicate_case", count(a), count(b)));
}
let keys: BTreeSet<_> = index_a.keys().chain(index_b.keys()).copied().collect();
for key @ (op, case) in keys {
match (index_a.get(&key), index_b.get(&key)) {
(Some(a), Some(b)) => compare_one(a, b, &mut differences),
(a, b) => {
let presence = |side: Option<&&Observation>| if side.is_some() { "present" } else { ABSENT }.to_owned();
differences.push(difference(op, case, "observation", presence(a), presence(b)));
}
}
}
differences
}
type Key = (&'static str, &'static str);
fn index(side: &[Observation]) -> (BTreeMap<Key, &Observation>, BTreeSet<Key>) {
let mut index = BTreeMap::new();
let mut duplicates = BTreeSet::new();
for observation in side {
let key = (observation.op, observation.case);
if index.insert(key, observation).is_some() {
duplicates.insert(key);
}
}
(index, duplicates)
}
fn difference(op: &str, case: &str, field: &str, a: String, b: String) -> Difference {
Difference {
op: op.to_owned(),
case: case.to_owned(),
field: field.to_owned(),
a,
b,
}
}
fn compare_one(a: &Observation, b: &Observation, differences: &mut Vec<Difference>) {
let mut field = |name: String, value_a: Option<&str>, value_b: Option<&str>| {
if value_a != value_b {
let show = |value: Option<&str>| value.unwrap_or(ABSENT).to_owned();
differences.push(difference(a.op, a.case, &name, show(value_a), show(value_b)));
}
};
field(
"status".to_owned(),
Some(a.status.to_string().as_str()),
Some(b.status.to_string().as_str()),
);
field("error_code".to_owned(), a.error_code.as_deref(), b.error_code.as_deref());
let headers: BTreeSet<_> = a.headers.keys().chain(b.headers.keys()).collect();
for name in headers {
field(
format!("header.{name}"),
a.headers.get(name).map(String::as_str),
b.headers.get(name).map(String::as_str),
);
}
let fields: BTreeSet<_> = a.fields.keys().chain(b.fields.keys()).collect();
for name in fields {
field(
format!("body.{name}"),
a.fields.get(name).map(String::as_str),
b.fields.get(name).map(String::as_str),
);
}
}
/// Operations observed on both sides.
pub(crate) fn compared_ops(a: &[Observation], b: &[Observation]) -> BTreeSet<&'static str> {
let ops_b: BTreeSet<_> = b.iter().map(|observation| observation.op).collect();
a.iter()
.map(|observation| observation.op)
.filter(|op| ops_b.contains(op))
.collect()
}
/// `(op, case)` pairs observed on both sides.
pub(crate) fn compared_cases(a: &[Observation], b: &[Observation]) -> usize {
let cases_b: BTreeSet<_> = b.iter().map(|observation| (observation.op, observation.case)).collect();
a.iter()
.map(|observation| (observation.op, observation.case))
.collect::<BTreeSet<_>>()
.intersection(&cases_b)
.count()
}
/// The JSON report written for every run.
#[derive(Debug, Serialize)]
pub(crate) struct Report {
pub schema: u32,
pub run: String,
pub stack_a: String,
pub stack_b: String,
pub region_a: Option<String>,
pub region_b: Option<String>,
pub compared_ops: usize,
pub compared_cases: usize,
pub differences: Vec<Difference>,
}
impl Report {
/// Writes the report as `<directory>/<run>.json` and returns its path.
pub(crate) fn write_to(&self, directory: &Path) -> std::io::Result<PathBuf> {
std::fs::create_dir_all(directory)?;
let path = directory.join(format!("{}.json", self.run));
let json = serde_json::to_vec_pretty(self).map_err(std::io::Error::other)?;
std::fs::write(&path, json)?;
Ok(path)
}
}
/// One line per difference, for a failing assertion.
pub(crate) fn render(differences: &[Difference]) -> String {
differences
.iter()
.map(|d| format!("({}, {}, {}, a={:?}, b={:?})", d.op, d.case, d.field, d.a, d.b))
.collect::<Vec<_>>()
.join("\n")
}
#[cfg(test)]
mod tests {
use super::*;
fn ok(op: &'static str) -> Observation {
Observation::new(op, "normal", 200, None, [("content-type", "application/xml")])
}
fn fields(differences: &[Difference]) -> Vec<&str> {
differences.iter().map(|d| d.field.as_str()).collect()
}
#[test]
fn identical_transcripts_have_no_difference() {
let a = vec![
ok("HeadBucket"),
ok("GetBucketLocation").with_field("location_constraint", "us-east-1"),
];
let b = a.clone();
assert_eq!(compare(&a, &b), Vec::new());
assert_eq!(compared_ops(&a, &b).len(), 2);
}
#[test]
fn a_status_difference_is_reported() {
let a = vec![ok("PutBucketAcl")];
let b = vec![Observation::new(
"PutBucketAcl",
"normal",
403,
None,
[("content-type", "application/xml")],
)];
assert_eq!(
compare(&a, &b),
vec![Difference {
op: "PutBucketAcl".into(),
case: "normal".into(),
field: "status".into(),
a: "200".into(),
b: "403".into(),
}]
);
}
#[test]
fn an_error_code_difference_is_reported() {
let a = vec![Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), [])];
let b = vec![Observation::new(
"GetObject",
"negative",
404,
Some("NoSuchBucket".into()),
[],
)];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["error_code"]);
assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("NoSuchKey", "NoSuchBucket"));
}
#[test]
fn an_error_code_on_one_side_only_is_reported() {
let a = vec![Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), [])];
let b = vec![Observation::new("HeadBucket", "negative", 404, None, [])];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["error_code"]);
assert_eq!(differences[0].b, ABSENT);
}
#[test]
fn an_allowlisted_header_difference_is_reported() {
let a = vec![Observation::new("HeadObject", "normal", 200, None, [("ETag", "\"aa\"")])];
let b = vec![Observation::new("HeadObject", "normal", 200, None, [("etag", "\"bb\"")])];
assert_eq!(fields(&compare(&a, &b)), ["header.etag"]);
}
#[test]
fn a_header_sent_by_one_side_only_is_reported() {
let a = vec![Observation::new(
"DeleteObject",
"normal",
204,
None,
[("x-amz-delete-marker", "true")],
)];
let b = vec![Observation::new("DeleteObject", "normal", 204, None, [])];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["header.x-amz-delete-marker"]);
assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("true", ABSENT));
}
#[test]
fn every_server_side_encryption_header_is_compared() {
let header = "x-amz-server-side-encryption-customer-key-md5";
let a = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "one")])];
let b = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "two")])];
assert_eq!(fields(&compare(&a, &b)), [format!("header.{header}")]);
}
#[test]
fn headers_outside_the_allowlist_are_not_compared() {
let a = vec![Observation::new(
"ListBuckets",
"normal",
200,
None,
[
("date", "Mon"),
("x-amz-request-id", "1"),
("server", "a"),
("content-length", "10"),
],
)];
let b = vec![Observation::new(
"ListBuckets",
"normal",
200,
None,
[
("date", "Tue"),
("x-amz-request-id", "2"),
("server", "b"),
("content-length", "11"),
],
)];
assert_eq!(compare(&a, &b), Vec::new());
assert!(a[0].headers.is_empty(), "non-allowlisted headers were kept: {:?}", a[0].headers);
}
#[test]
fn distinct_version_ids_are_not_a_difference() {
let a = vec![Observation::new(
"PutObject",
"normal",
200,
None,
[("x-amz-version-id", "0b1c")],
)];
let b = vec![Observation::new(
"PutObject",
"normal",
200,
None,
[("x-amz-version-id", "9f8e")],
)];
assert_eq!(compare(&a, &b), Vec::new());
}
#[test]
fn a_null_version_id_against_a_real_one_is_reported() {
let a = vec![Observation::new(
"PutObject",
"normal",
200,
None,
[("x-amz-version-id", "null")],
)];
let b = vec![Observation::new(
"PutObject",
"normal",
200,
None,
[("x-amz-version-id", "9f8e")],
)];
assert_eq!(fields(&compare(&a, &b)), ["header.x-amz-version-id"]);
}
#[test]
fn a_body_field_difference_is_reported() {
let a = vec![ok("GetBucketLocation").with_field("location_constraint", "us-east-1")];
let b = vec![ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["body.location_constraint"]);
assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("us-east-1", "eu-west-1"));
}
#[test]
fn a_body_field_on_one_side_only_is_reported() {
let a = vec![ok("ListObjectsV2").with_field("keys", "[]")];
let b = vec![ok("ListObjectsV2")];
assert_eq!(fields(&compare(&a, &b)), ["body.keys"]);
}
#[test]
fn error_messages_are_never_compared() {
let mut a = Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), []);
let mut b = a.clone();
a.message = Some("The specified key does not exist.".into());
b.message = Some("Object not found".into());
assert_eq!(compare(&[a], &[b]), Vec::new());
}
#[test]
fn a_case_observed_on_one_side_only_is_reported() {
let a = vec![ok("HeadBucket"), ok("ListBuckets")];
let b = vec![ok("HeadBucket")];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["observation"]);
assert_eq!(differences[0].op, "ListBuckets");
assert_eq!(compared_ops(&a, &b).into_iter().collect::<Vec<_>>(), ["HeadBucket"]);
}
#[test]
fn a_case_recorded_twice_is_reported() {
let a = vec![ok("HeadBucket"), ok("HeadBucket")];
let b = vec![ok("HeadBucket")];
assert_eq!(fields(&compare(&a, &b)), ["duplicate_case"]);
}
#[test]
fn compared_ops_counts_operations_not_cases() {
let negative = Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), []);
let a = vec![ok("HeadBucket"), negative, ok("ListBuckets")];
let b = a.clone();
assert_eq!(compared_ops(&a, &b).len(), 2);
}
#[test]
fn the_report_lists_each_difference_as_op_field_a_b() {
let directory = std::env::temp_dir().join(format!("dual-stack-report-{}", uuid::Uuid::new_v4()));
let report = Report {
schema: 1,
run: "unit".into(),
stack_a: "legacy".into(),
stack_b: "legacy".into(),
region_a: None,
region_b: Some("eu-west-1".into()),
compared_ops: 1,
compared_cases: 1,
differences: compare(
&[ok("GetBucketLocation").with_field("location_constraint", "us-east-1")],
&[ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")],
),
};
let path = report.write_to(&directory).expect("write the report");
let json: serde_json::Value = serde_json::from_slice(&std::fs::read(&path).expect("read the report")).expect("parse");
let _ = std::fs::remove_dir_all(&directory);
assert_eq!(path.file_name().and_then(|name| name.to_str()), Some("unit.json"));
assert_eq!(json["compared_ops"], 1);
assert_eq!(
json["differences"],
serde_json::json!([{
"op": "GetBucketLocation",
"case": "normal",
"field": "body.location_constraint",
"a": "us-east-1",
"b": "eu-west-1",
}])
);
}
}
@@ -1,197 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Dual-stack differential: one `rustfs` binary, two processes, one SDK script (rustfs/backlog#2740).
//!
//! Stack A always runs `RUSTFS_S3_STACK=legacy`. Stack B runs the value of `RUSTFS_E2E_STACK_B`
//! (default `legacy` until the gateway stack is assembled), and `RUSTFS_E2E_STACK_B_REGION`, when
//! set, starts B in another region so that a difference can be injected by hand. Both servers
//! answer the script in `scenarios`, which covers every operation of `impl S3 for FS`; `compare`
//! reports each `(op, field, a, b)` on which the answers differ, and every run writes its report
//! to `<target>/dual-stack/<run>.json`. Two processes, not two embedded servers: bucket metadata
//! is process-global, so two stacks in one process would read each other's buckets.
mod compare;
mod scenarios;
use crate::common::{RustFSTestEnvironment, init_logging, workspace_root};
use compare::{Difference, Observation, Report, compare, compared_cases, compared_ops, render};
use std::path::PathBuf;
use std::time::{SystemTime, UNIX_EPOCH};
type BoxError = Box<dyn std::error::Error + Send + Sync>;
const STACK_A: &str = "legacy";
const ENV_STACK_B: &str = "RUSTFS_E2E_STACK_B";
const ENV_STACK_B_REGION: &str = "RUSTFS_E2E_STACK_B_REGION";
/// Becomes `gateway` once rustfs/backlog#2734 T2.5 assembles the gateway stack.
const DEFAULT_STACK_B: &str = "legacy";
/// The region the negative control gives stack A explicitly, so an inherited `RUSTFS_REGION`
/// cannot erase the injected difference.
const CONTROL_REGION_A: &str = "us-east-1";
const CONTROL_REGION_B: &str = "eu-west-1";
/// `normal` cases the legacy stack must answer with a 2xx. The comparison alone cannot tell a
/// script that exercises real behavior from one in which both stacks fail the same way. Five of
/// the 73 fail on legacy by design of this script: PutBucketReplication (no registered remote
/// target), GetBucketReplication (so no configuration), PutBucketNotificationConfiguration (no
/// notification target configured), RestoreObject (the object is not archived) and
/// GetObjectTorrent (legacy answers `NoSuchKey`).
const MIN_NORMAL_SUCCESSES: usize = 68;
struct PairConfig {
stack_b: String,
region_a: Option<String>,
region_b: Option<String>,
}
impl PairConfig {
fn from_env() -> Self {
Self {
stack_b: std::env::var(ENV_STACK_B).unwrap_or_else(|_| DEFAULT_STACK_B.to_owned()),
region_a: None,
region_b: std::env::var(ENV_STACK_B_REGION).ok(),
}
}
}
/// Starts one server on `stack`, overriding its region only when asked.
async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
let mut env = RustFSTestEnvironment::new().await?;
let mut vars = vec![("RUSTFS_S3_STACK", stack)];
if let Some(region) = region {
vars.push(("RUSTFS_REGION", region));
}
env.start_rustfs_server_without_cleanup_with_env(&vars).await?;
Ok(env)
}
/// Starts stack A and stack B side by side, each with its own port and data directory.
async fn spawn_pair(config: &PairConfig) -> Result<(RustFSTestEnvironment, RustFSTestEnvironment), BoxError> {
tokio::try_join!(
start_stack(STACK_A, config.region_a.as_deref()),
start_stack(&config.stack_b, config.region_b.as_deref())
)
}
fn report_directory() -> PathBuf {
let target = std::env::var_os("CARGO_TARGET_DIR").map_or_else(|| PathBuf::from("target"), PathBuf::from);
let target = if target.is_absolute() {
target
} else {
workspace_root().join(target)
};
target.join("dual-stack")
}
struct Run {
report: Report,
path: PathBuf,
a: Vec<Observation>,
}
/// Runs the script against both stacks concurrently and writes the report.
async fn run_differential(label: &str, config: &PairConfig) -> Run {
let (server_a, server_b) = spawn_pair(config).await.expect("start stack A and stack B");
let (a, b) = tokio::join!(Box::pin(scenarios::run(&server_a)), Box::pin(scenarios::run(&server_b)));
let millis = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map_or(0, |elapsed| elapsed.as_millis());
let report = Report {
schema: 1,
run: format!("{label}-{millis}-{}", std::process::id()),
stack_a: STACK_A.to_owned(),
stack_b: config.stack_b.clone(),
region_a: config.region_a.clone(),
region_b: config.region_b.clone(),
compared_ops: compared_ops(&a, &b).len(),
compared_cases: compared_cases(&a, &b),
differences: compare(&a, &b),
};
let path = report.write_to(&report_directory()).expect("write the dual-stack report");
Run { report, path, a }
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn both_stacks_answer_every_operation_identically() {
init_logging();
let config = PairConfig::from_env();
let run = run_differential("self-comparison", &config).await;
let report = &run.report;
let mut observed: Vec<_> = run.a.iter().map(|observation| observation.op).collect();
observed.sort_unstable();
observed.dedup();
assert_eq!(
observed,
scenarios::OPERATIONS,
"the script must drive exactly the operations of impl S3 for FS"
);
assert_eq!(report.compared_ops, scenarios::OPERATIONS.len(), "report: {}", run.path.display());
assert!(
report.differences.is_empty(),
"stack {} and stack {} differ on {} of {} cases (report: {}):\n{}",
report.stack_a,
report.stack_b,
report.differences.len(),
report.compared_cases,
run.path.display(),
render(&report.differences)
);
let not_4xx: Vec<_> = run
.a
.iter()
.filter(|observation| observation.case == "negative" && !(400..500).contains(&observation.status))
.collect();
assert!(not_4xx.is_empty(), "negative cases must fail with a 4xx on {STACK_A}: {not_4xx:#?}");
let successes = run
.a
.iter()
.filter(|observation| observation.case == "normal" && observation.is_success())
.count();
assert!(
successes >= MIN_NORMAL_SUCCESSES,
"only {successes} normal cases succeeded on {STACK_A}: {:#?}",
run.a
);
}
#[tokio::test]
async fn differential_detects_an_injected_difference() {
init_logging();
let config = PairConfig {
stack_b: STACK_A.to_owned(),
region_a: Some(CONTROL_REGION_A.to_owned()),
region_b: Some(CONTROL_REGION_B.to_owned()),
};
let run = run_differential("injected-region", &config).await;
let expected = Difference {
op: "GetBucketLocation".to_owned(),
case: "normal".to_owned(),
field: "body.location_constraint".to_owned(),
a: CONTROL_REGION_A.to_owned(),
b: CONTROL_REGION_B.to_owned(),
};
assert!(
run.report.differences.contains(&expected),
"B started in {CONTROL_REGION_B} but GetBucketLocation was not reported (report: {}):\n{}",
run.path.display(),
render(&run.report.differences)
);
}
}
File diff suppressed because it is too large Load Diff
@@ -1191,59 +1191,11 @@ mod tests {
.into())
}
fn scanner_cycle_completion_budget(
cycle_end: u64,
interval: Option<u64>,
now: u64,
active: bool,
) -> Result<Duration, &'static str> {
if cycle_end > now {
return Err("scanner cycle timestamp is in the future");
}
if active {
return Ok(Duration::from_secs(60));
}
let interval = interval
.filter(|interval| *interval != 0)
.ok_or("scanner schedule has no positive interval")?;
let next_cycle = cycle_end
.checked_add(interval)
.ok_or("scanner schedule timestamp overflows")?;
let budget = next_cycle
.saturating_sub(now)
.checked_add(60)
.ok_or("scanner completion budget overflows")?;
Ok(Duration::from_secs(budget))
}
#[test]
fn scanner_cycle_completion_budget_accounts_for_captured_retry_schedule() {
assert_eq!(
scanner_cycle_completion_budget(1_791_815_390, Some(256), 1_791_815_447, false),
Ok(Duration::from_secs(259)),
);
assert_eq!(scanner_cycle_completion_budget(100, Some(1), 110, false), Ok(Duration::from_secs(60)));
}
#[test]
fn scanner_cycle_completion_budget_does_not_grant_planned_sleep_to_active_cycle() {
assert_eq!(scanner_cycle_completion_budget(100, Some(256), 110, true), Ok(Duration::from_secs(60)));
assert_eq!(scanner_cycle_completion_budget(0, None, 110, true), Ok(Duration::from_secs(60)));
}
#[test]
fn scanner_cycle_completion_budget_rejects_invalid_schedule() {
for (end, interval, now) in [(101, 1, 100), (100, 0, 100), (u64::MAX, 1, u64::MAX), (1, u64::MAX - 1, 1)] {
assert!(scanner_cycle_completion_budget(end, Some(interval), now, false).is_err());
}
}
async fn wait_for_scanner_cycle_after(
cluster: &RustFSTestClusterEnvironment,
previous_cycle_end: u64,
) -> Result<u64, Box<dyn Error + Send + Sync>> {
let mut deadline = Instant::now() + Duration::from_secs(60);
let mut schedule_accounted = false;
let deadline = Instant::now() + Duration::from_secs(60);
loop {
let mut latest_cycle_end = 0;
let mut versions_observed = false;
@@ -1285,36 +1237,14 @@ mod tests {
let versions_scanned = metrics["versions_scanned"]
.as_u64()
.ok_or("scanner status is missing its version-coverage counter")?;
let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
if cycle_end > now {
return Err("scanner cycle timestamp is in the future".into());
}
let schedule = &status["cycle_schedule"];
if !schedule_accounted && schedule["execution_role"].as_str() == Some("leader") {
let active = metrics["current_cycle_active"]
.as_bool()
.ok_or("scanner status is missing its active-cycle flag")?;
if active || (cycle_end != 0 && schedule["effective_interval_available"].as_bool() == Some(true)) {
let interval = schedule["effective_interval_seconds"].as_u64();
let budget = scanner_cycle_completion_budget(cycle_end, interval, now, active)?;
deadline = Instant::now().checked_add(budget).ok_or("scanner deadline overflows")?;
schedule_accounted = true;
}
}
// Superseded walks complete all sets while withholding quota authority.
// Deferred and partial attempts do not certify completed scanning.
if matches!(metrics["last_cycle_result"].as_str(), Some("success" | "superseded")) {
latest_cycle_end = latest_cycle_end.max(cycle_end);
}
latest_cycle_end = latest_cycle_end.max(cycle_end);
versions_observed |= versions_scanned > 0;
observations.push(format!(
"node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}, partial_reason={}, schedule={}",
"node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}",
metrics["current_cycle"],
metrics["current_cycle_active"],
metrics["leader_lock_state"],
metrics["last_cycle_result"],
metrics["last_cycle_partial_reason"],
schedule,
));
}
// The coordinator records cycle completion, but remote workers
@@ -1625,8 +1555,7 @@ mod tests {
}
if background_enabled {
let baseline_cycle_floor = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
wait_for_scanner_cycle_after(&cluster, baseline_cycle_floor).await?;
wait_for_scanner_cycle_after(&cluster, 0).await?;
}
cluster.stop_node(1)?;
@@ -264,7 +264,7 @@ async fn call_make_volume_response(
request: MakeVolumeRequest,
headers: HeaderMap,
) -> Result<Response<MakeVolumeResponse>, Status> {
let mut client = node_service_time_out_client_no_auth(url)
let mut client = node_service_time_out_client_no_auth(&url.to_string())
.await
.map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
let mut rpc_request = Request::new(request);
@@ -273,7 +273,7 @@ async fn call_make_volume_response(
}
async fn call_ping_response(url: &str, headers: HeaderMap) -> Result<Response<PingResponse>, Status> {
let mut client = node_service_time_out_client_no_auth(url)
let mut client = node_service_time_out_client_no_auth(&url.to_string())
.await
.map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
let mut rpc_request = Request::new(PingRequest {
-6
View File
@@ -31,12 +31,6 @@ pub mod chaos;
#[cfg(test)]
pub mod fake_s3_target;
// Dual-stack differential (rustfs/backlog#2740): one binary, two processes on
// different RUSTFS_S3_STACK values, the same SDK script for every S3 operation,
// compared on status, error code, allowlisted headers and typed body fields.
#[cfg(test)]
mod dual_stack_differential;
// On-demand migration (backlog#2147): shared two-server environment, admin
// wrappers, and the harness self-test (backlog#2151). Behavior scenarios are
// added by later ODM tasks.
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
use crate::common::RustFSTestClusterEnvironment;
use aws_sdk_s3::Client;
use aws_sdk_s3::error::SdkError;
use bytes::Bytes;
@@ -185,11 +185,7 @@ async fn wait_for_quorum_read_admission(clients: &[Client], bucket: &str) -> Tes
return Ok(());
}
if tokio::time::Instant::now() >= deadline {
let diagnostics = clients[0].get_object().bucket(bucket).key("warm-small").send().await;
return Err(format!(
"read quorum did not become available after lease convergence for {bucket}: GET={diagnostics:?}"
)
.into());
return Err(format!("read quorum did not become available after lease convergence for {bucket}").into());
}
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
}
@@ -199,14 +195,11 @@ async fn wait_for_quorum_read_admission(clients: &[Client], bucket: &str) -> Tes
async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResult {
crate::common::init_logging();
for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
for (node_count, parity) in [(4, 2), (6, 3), (6, 2)] {
let read_quorum = node_count - parity;
let write_quorum = read_quorum + usize::from(read_quorum == parity);
let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
if parity == 0 {
cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
}
// Wait for every seed fanout before removing any physical shard.
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
@@ -345,277 +338,6 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul
Ok(())
}
#[tokio::test]
async fn test_cold_start_read_quorum_follows_erasure_layout() -> TestResult {
crate::common::init_logging();
for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
let read_quorum = node_count - parity;
let write_quorum = read_quorum + usize::from(read_quorum == parity);
let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
if parity == 0 {
cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
}
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
cluster.set_env("RUST_LOG", "warn");
cluster.start().await?;
let clients = cluster
.create_all_clients()?
.into_iter()
.map(|client| {
Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
.build(),
)
})
.collect::<Vec<_>>();
let bucket = format!("cold-read-quorum-{node_count}-{parity}");
clients[0].create_bucket().bucket(&bucket).send().await?;
let small = b"cold startup follows the stored EC layout".to_vec();
let large = vec![73; 256 * 1024];
for (key, body) in [
("warm-small", &small),
("warm-large", &large),
("cold-large", &large),
("below-quorum", &large),
] {
clients[0]
.put_object()
.bucket(&bucket)
.key(key)
.body(Bytes::copy_from_slice(body).into())
.send()
.await?;
for node in &cluster.nodes {
let census =
crate::chaos::census_object_version_on_disk(std::path::Path::new(&node.data_dir), &bucket, key, None)?;
assert!(census.is_complete(), "seed shard must exist before cold startup: {census:?}");
assert_eq!((census.data_blocks, census.parity_blocks), (Some(read_quorum), Some(parity)));
}
}
for node in 0..node_count {
cluster.stop_node(node)?;
}
// Rebuild every process's in-memory state below read quorum, then
// cross the EC boundary. A format bootstrap must not admit S3 reads
// merely because half the peers are online (e.g. six disks, EC:2).
for node in 0..read_quorum - 1 {
cluster.start_node_process(node).await?;
}
let error = clients[0]
.get_object()
.bucket(&bucket)
.key("below-quorum")
.send()
.await
.expect_err("cold startup below the configured EC read quorum must stay unavailable");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
cluster.start_node_process(read_quorum - 1).await?;
wait_for_quorum_read_admission(&clients[..read_quorum], &bucket).await?;
for client in clients.iter().take(read_quorum) {
assert_quorum_object_body(client, &bucket, "warm-small", &small).await?;
assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
let write = client
.put_object()
.bucket(&bucket)
.key("cold-quorum-write")
.body(Bytes::copy_from_slice(&small).into())
.send()
.await;
if read_quorum >= write_quorum {
write?;
assert_quorum_object_body(client, &bucket, "cold-quorum-write", &small).await?;
} else {
let error = write.expect_err("cold startup at read quorum must preserve the EC write quorum");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
}
}
for node in read_quorum..node_count {
cluster.start_node(node).await?;
}
for client in &clients {
assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
}
}
Ok(())
}
#[tokio::test]
async fn test_four_by_six_cold_start_at_read_quorum() -> TestResult {
crate::common::init_logging();
let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 6)).await?;
// Explicit URLs are consumed in order. Place one drive from each node in
// every set, matching a four-host, six-drive-per-host deployment.
let volumes = (0..6)
.flat_map(|drive| {
cluster
.nodes
.iter()
.map(move |node| format!("http://{}{}", node.address, node.data_dirs[drive]))
})
.collect::<Vec<_>>()
.join(" ");
cluster.set_env("RUSTFS_VOLUMES", volumes);
cluster.set_env("RUSTFS_ERASURE_SET_DRIVE_COUNT", "4");
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
cluster.set_env("RUST_LOG", "error");
cluster.start().await?;
let clients = cluster
.create_all_clients()?
.into_iter()
.map(|client| {
Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
.build(),
)
})
.collect::<Vec<_>>();
let bucket = "cold-start-read-quorum";
clients[0].create_bucket().bucket(bucket).send().await?;
let mut fixtures = Vec::new();
let mut covered = [[false; 2]; 6];
for index in 0..128 {
let key = match index {
0 => "warm-small".to_string(),
1 => "warm-large".to_string(),
_ => format!("cold-{index}"),
};
let size_class = index % 2;
let size = if size_class == 0 { 71 } else { 256 * 1024 };
let body = (0..size)
.map(|offset| u8::try_from(offset % 251).expect("bounded fixture byte"))
.collect::<Vec<_>>();
clients[0]
.put_object()
.bucket(bucket)
.key(&key)
.body(Bytes::copy_from_slice(&body).into())
.send()
.await?;
for node in &cluster.nodes {
let mut complete_shards = 0;
for (drive, path) in node.data_dirs.iter().enumerate() {
let census = crate::chaos::census_object_version_on_disk(std::path::Path::new(path), bucket, &key, None)?;
if census.has_xl_meta {
assert!(census.is_complete(), "seed shard must be complete before shutdown: {census:?}");
assert_eq!((census.data_blocks, census.parity_blocks), (Some(2), Some(2)));
covered[drive][size_class] = true;
complete_shards += 1;
}
}
assert_eq!(complete_shards, 1, "each node must own one shard of every seed object");
}
fixtures.push((key, body));
if covered.iter().flatten().all(|covered| *covered) {
break;
}
}
assert!(
covered.iter().flatten().all(|covered| *covered),
"fixtures must cover all six sets with inline and large objects"
);
for first in 0..4 {
for second in first + 1..4 {
for node in 0..4 {
cluster.stop_node(node)?;
}
cluster.start_node_process(first).await?;
let error = clients[first]
.get_object()
.bucket(bucket)
.key("warm-large")
.send()
.await
.expect_err("one node cannot satisfy read quorum after a cold start");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
cluster.start_node_process(second).await?;
wait_for_quorum_read_admission(&[clients[first].clone(), clients[second].clone()], bucket).await?;
for node in [first, second] {
for (key, body) in &fixtures {
assert_quorum_object_body(&clients[node], bucket, key, body).await?;
}
let range = clients[node]
.get_object()
.bucket(bucket)
.key("warm-large")
.range("bytes=17-1023")
.send()
.await?;
assert_eq!(range.body.collect().await?.into_bytes().as_ref(), &fixtures[1].1[17..1024]);
let listing = clients[node].list_objects_v2().bucket(bucket).send().await?;
for (key, _) in &fixtures {
assert!(
listing.contents().iter().any(|object| object.key() == Some(key.as_str())),
"listing omitted {key}"
);
}
let error = clients[node]
.put_object()
.bucket(bucket)
.key("recovery-write")
.body(Bytes::from_static(b"after recovery").into())
.send()
.await
.expect_err("two nodes must not authorize a write");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
}
let error = clients[first]
.delete_object()
.bucket(bucket)
.key("warm-large")
.send()
.await
.expect_err("two nodes must not authorize a delete");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
assert_quorum_object_body(&clients[first], bucket, "warm-large", &fixtures[1].1).await?;
let remaining = (0..4).filter(|node| *node != first && *node != second).collect::<Vec<_>>();
cluster.start_node(remaining[0]).await?;
let deadline = tokio::time::Instant::now() + Duration::from_secs(90);
loop {
match clients[first]
.put_object()
.bucket(bucket)
.key("recovery-write")
.body(Bytes::from_static(b"after recovery").into())
.send()
.await
{
Ok(_) => break,
Err(error)
if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
&& tokio::time::Instant::now() < deadline =>
{
tokio::time::sleep(Duration::from_millis(200)).await
}
Err(error) => return Err(error.into()),
}
}
for node in [first, second, remaining[0]] {
assert_quorum_object_body(&clients[node], bucket, "recovery-write", b"after recovery").await?;
}
cluster.start_node(remaining[1]).await?;
for client in &clients {
for (key, body) in &fixtures {
assert_quorum_object_body(client, bucket, key, body).await?;
}
assert_quorum_object_body(client, bucket, "recovery-write", b"after recovery").await?;
}
}
}
Ok(())
}
async fn put_object(client: Client, payload: Vec<u8>, writer_id: usize) -> Result<(), String> {
client
.put_object()
@@ -465,13 +465,6 @@ impl NodeService for MinimalLockNodeService {
Err(Status::unimplemented("lock-only test server"))
}
async fn recover_delete_transaction(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionRequest>,
) -> Result<Response<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionResponse>, Status> {
Err(Status::unimplemented("delete recovery is not supported by the lock fixture"))
}
async fn settle_part_transaction(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::SettlePartTransactionRequest>,
+1 -1
View File
@@ -740,7 +740,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
.version_id(version)
.send()
.await?;
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
opa.expect_bucket_tags("s3:GetObject", source, Some("finance")).await?;
client.list_objects_v2().bucket(source).send().await?;
opa.expect_bucket_tags("s3:ListBucket", source, Some("finance")).await?;
client.list_object_versions().bucket(source).send().await?;
@@ -2345,7 +2345,7 @@ fn layout_reported_parts(attributes: &aws_sdk_s3::operation::get_object_attribut
attributes.object_parts().map(|parts| parts.parts().len())
}
async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
let label = case.label();
let head = layout_head(client, case).await?;
assert_eq!(
@@ -2362,12 +2362,7 @@ async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &s
let full = layout_get(client, case, None, None).await?.body.collect().await?.into_bytes();
assert_eq!(full.len(), case.body.len(), "{context}: {label}: full GET length");
assert!(full == case.body, "{context}: {label}: full GET body must equal the rc.5 upload");
Ok(())
}
async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
assert_layout_full_body(client, case, context).await?;
let label = case.label();
if case.is_multipart_layout() {
let first = case.part_sizes[0];
let range = format!("bytes={}-{}", first - 32, first + 31);
@@ -2646,11 +2641,10 @@ async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult {
"rc.5 wrote a legacy layout"
);
}
// Prove the rc.5 writer persisted every original byte before upgrading.
// Its compressed cross-part range reader has a legacy limitation; the
// upgraded reader must still satisfy every range and partNumber assertion.
// The rc.5 writer must itself still read what it wrote, so a later
// failure is attributable to the upgrade rather than to the fixture.
for case in &cases {
assert_layout_full_body(&old_client, case, "rc.5").await?;
assert_layout_readable(&old_client, case, "rc.5").await?;
}
// Upgrade in place.
+31 -17
View File
@@ -199,10 +199,10 @@ pub mod bucket {
get_object_lock_config, get_object_lock_config_state, get_on_demand_migration_config,
get_on_demand_migration_config_in, get_public_access_block_config, get_quota_config, get_replication_config,
get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
remove_bucket_metadata_if_absent, set_bucket_metadata, update, update_bucket_targets_under_transaction_lock,
update_config_with, update_if_incarnation, update_if_incarnation_at, update_quota_if_incarnation,
update_quota_if_incarnation_at, update_under_transaction_lock, update_under_transaction_lock_at,
init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, set_bucket_metadata,
update, update_bucket_targets_under_transaction_lock, update_config_with, update_if_incarnation,
update_if_incarnation_at, update_quota_if_incarnation, update_quota_if_incarnation_at, update_under_transaction_lock,
update_under_transaction_lock_at,
};
#[cfg(feature = "test-util")]
pub use crate::bucket::metadata_sys::{ConfigWriteLockProbe, test_support};
@@ -210,8 +210,7 @@ pub mod bucket {
pub mod migration {
pub use crate::bucket::migration::{
LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_bucket_metadata_for_buckets,
try_migrate_iam_config,
LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_iam_config,
};
}
@@ -372,6 +371,24 @@ pub mod config {
with_config_object_read_lock, with_config_object_write_lock, with_server_config_read_lock,
with_server_config_write_lock,
};
/// Read config bytes and metadata without treating an empty payload as absent.
pub async fn read_config_preserve_empty_with_metadata_opts(
api: std::sync::Arc<crate::store::ECStore>,
file: &str,
opts: &crate::object_api::ObjectOptions,
) -> crate::error::Result<(Vec<u8>, crate::object_api::ObjectInfo)> {
crate::config::com::read_config_preserve_empty_with_metadata_opts(api, file, opts).await
}
pub async fn read_config_limited_preserve_empty_with_metadata_opts(
api: std::sync::Arc<crate::store::ECStore>,
file: &str,
opts: &crate::object_api::ObjectOptions,
max_bytes: usize,
) -> crate::error::Result<(Vec<u8>, crate::object_api::ObjectInfo)> {
crate::config::com::read_config_limited_preserve_empty_with_metadata_opts(api, file, opts, max_bytes).await
}
}
pub mod storageclass {
@@ -423,12 +440,11 @@ pub mod disk {
pub use crate::disk::os::{LocalPublicationPause, LocalPublicationStage};
pub use crate::disk::{
BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction,
DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation,
DeleteRecoveryRpcReply, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore, FileInfoVersions,
FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize, PartTransactionAction,
RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE,
SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk, validate_batch_read_version_item_count,
CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
validate_batch_read_version_item_count,
};
pub use bytes::Bytes;
pub use endpoint::Endpoint;
@@ -521,14 +537,13 @@ pub mod integrity {
pub mod object {
pub use crate::object_api::{
BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReadCondition, GetObjectReadMetadata, GetObjectReader,
NamespaceLockFence, ObjectEncryptionResolver, ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions,
PutObjReader, QuotaAdmission, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission,
RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, ScannerPublicationCommitScope, ScannerPublicationCommitStartError,
ScannerPublicationCommitState, ShardIntegrityWriteMode, StreamConsumer, WriteCommitGuard, WriteCompletion,
get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
with_get_object_read_condition,
};
pub use crate::store::{
PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
@@ -593,7 +608,6 @@ pub mod set_disk {
#[cfg(feature = "test-util")]
pub mod test_util {
pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;
pub use crate::set_disk::ConditionalReadBenchmarkMetadataGuard;
pub use crate::set_disk::{MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause};
/// Keep a namespace commit pending until the returned owner is dropped.
@@ -14214,11 +14214,7 @@ mod tests {
#[tokio::test]
#[serial]
async fn tier_free_version_recovery_production_entrypoint_enqueues_seeded_item() {
let (disk_paths, ecstore) =
temp_env::async_with_vars([(super::ENV_TIER_FREE_VERSION_RECOVERY_ENABLED, Some("false"))], async {
setup_test_env().await
})
.await;
let (disk_paths, ecstore) = setup_test_env().await;
let runtime_state = install_unconsumed_runtime_expiry_worker(&ecstore, 1).await;
let recovery_rx = {
let state = runtime_state.read().await;
+5 -23
View File
@@ -439,8 +439,6 @@ pub struct BucketMetadata {
pub bucket_incarnation_id: Uuid,
pub(crate) bucket_incarnation_sidecar: bool,
pub(crate) bucket_creation_committed: bool,
// A legacy default is not proof of an interrupted creation.
pub(crate) bucket_creation_commit_record_present: bool,
pub policy_config_json: Vec<u8>,
pub notification_config_xml: Vec<u8>,
pub lifecycle_config_xml: Vec<u8>,
@@ -516,7 +514,6 @@ impl Default for BucketMetadata {
bucket_incarnation_id: Uuid::nil(),
bucket_incarnation_sidecar: false,
bucket_creation_committed: false,
bucket_creation_commit_record_present: false,
policy_config_json: Default::default(),
notification_config_xml: Default::default(),
lifecycle_config_xml: Default::default(),
@@ -755,10 +752,7 @@ impl BucketMetadata {
"TableBucketConfigUpdatedAt" => self.table_bucket_config_updated_at = read_msgp_time_value(rd)?,
"DurabilityConfigUpdatedAt" => self.durability_config_updated_at = read_msgp_time_value(rd)?,
"OnDemandMigrationConfigUpdatedAt" => self.on_demand_migration_config_updated_at = read_msgp_time_value(rd)?,
"BucketCreationCommitted" => {
self.bucket_creation_committed = read_msgp_bool(rd)?;
self.bucket_creation_commit_record_present = true;
}
"BucketCreationCommitted" => self.bucket_creation_committed = read_msgp_bool(rd)?,
other => {
tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
skip_msgp_value(rd)?;
@@ -771,10 +765,8 @@ impl BucketMetadata {
/// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
// Preserve the absence of a legacy commit record across config rewrites.
let commit_record_present = self.bucket_creation_commit_record_present || self.bucket_creation_committed;
// Map size: MinIO fields (25) + RustFS extensions (21) + optional commit record.
let map_len: u32 = 46 + u32::from(commit_record_present);
// Map size: MinIO fields (25) + RustFS extensions (22)
let map_len: u32 = 47;
rmp::encode::write_map_len(wr, map_len)?;
// MinIO field order (same as Go struct)
@@ -855,10 +847,8 @@ impl BucketMetadata {
write_msgp_time(wr, self.durability_config_updated_at)?;
rmp::encode::write_str(wr, "OnDemandMigrationConfigUpdatedAt")?;
write_msgp_time(wr, self.on_demand_migration_config_updated_at)?;
if commit_record_present {
rmp::encode::write_str(wr, "BucketCreationCommitted")?;
rmp::encode::write_bool(wr, self.bucket_creation_committed)?;
}
rmp::encode::write_str(wr, "BucketCreationCommitted")?;
rmp::encode::write_bool(wr, self.bucket_creation_committed)?;
Ok(())
}
@@ -1701,13 +1691,6 @@ mod test {
let mut bm = BucketMetadata::unmarshal(&body[4..]).expect("unmarshal recovered blob");
assert_eq!(bm.name, "interop");
assert!(!bm.bucket_creation_committed, "legacy metadata must default to uncommitted");
assert!(!bm.bucket_creation_commit_record_present, "legacy metadata has no creation-commit record");
let rewritten = BucketMetadata::unmarshal(&bm.marshal_msg().expect("rewrite legacy metadata"))
.expect("read rewritten legacy metadata");
assert!(
!rewritten.bucket_creation_commit_record_present,
"rewriting legacy metadata must not fabricate a creation intent"
);
bm.parse_all_configs().expect("parse recovered configs");
assert!(bm.lifecycle_config.is_some());
}
@@ -1725,7 +1708,6 @@ mod test {
assert_eq!(bm.name, new.name);
assert!(new.bucket_creation_committed);
assert!(new.bucket_creation_commit_record_present);
assert!(!bm.bucket_incarnation_id.is_nil());
assert_eq!(bm.bucket_incarnation_id, new.bucket_incarnation_id);
}
-421
View File
@@ -314,33 +314,6 @@ pub async fn remove_bucket_metadata(bucket: &str) -> Result<bool> {
Ok(lock.remove(bucket).await)
}
/// Remove a peer's cached metadata only when the bucket is absent under the
/// namespace read fence. A delayed delete notification must not withdraw the
/// metadata of a same-name bucket that has already been recreated.
pub async fn remove_bucket_metadata_if_absent(api: Arc<ECStore>, bucket: &str) -> Result<bool> {
if is_meta_bucketname(bucket) {
return Ok(false);
}
// Lock order: bucket namespace read -> metadata system read (released)
// -> metadata publish, matching peer reloads.
let namespace_lock = api.new_ns_lock(bucket, bucket).await?;
let namespace_guard = namespace_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
let sys = bucket_metadata_sys_of(&api.ctx)?;
let sys = sys.read().await.clone();
if sys
.bucket_exists(bucket, &namespace_guard, "peer bucket metadata removal existence check")
.await?
{
return Ok(false);
}
await_bucket_namespace_operation(Some(&namespace_guard), bucket, "peer bucket metadata removal", async {
Ok(sys.remove(bucket).await)
})
.await
}
fn start_refresh_buckets_metadata_loop(sys: Arc<RwLock<BucketMetadataSys>>) {
let Some(cancel_token) = runtime_sources::background_services_cancel_token() else {
warn!("bucket metadata refresh loop skipped because background cancellation token is not initialized");
@@ -1600,27 +1573,9 @@ struct MetadataPublishGuard {
_guard: tokio::sync::OwnedMutexGuard<MetadataPublishLockState>,
}
#[derive(Debug, Clone)]
pub(crate) struct PendingBucketCreationProof {
pub(crate) generation: Uuid,
pub(crate) intent: Vec<u8>,
pub(crate) versioning_requested: bool,
pub(crate) witnesses: Vec<crate::disk::os::NativeBucketCreationWitness>,
}
impl PendingBucketCreationProof {
pub(crate) fn matches_metadata(&self, metadata: &BucketMetadata) -> Result<bool> {
Ok(!metadata.bucket_creation_committed
&& metadata.bucket_incarnation_sidecar
&& metadata.bucket_incarnation_id == self.generation
&& metadata.marshal_msg()? == self.intent)
}
}
#[derive(Debug, Clone)]
pub struct BucketMetadataSys {
metadata_map: Arc<RwLock<HashMap<String, Arc<BucketMetadata>>>>,
pending_bucket_creations: Arc<RwLock<HashMap<String, PendingBucketCreationProof>>>,
/// Serializes metadata-map commits and their derived cache updates for one
/// bucket. Namespace locks, when present, are acquired before this lock.
metadata_publish_locks: Arc<MetadataPublishLockRegistry>,
@@ -1664,7 +1619,6 @@ impl BucketMetadataSys {
pub fn new(api: Arc<ECStore>) -> Self {
Self {
metadata_map: Arc::new(RwLock::new(HashMap::new())),
pending_bucket_creations: Arc::new(RwLock::new(HashMap::new())),
metadata_publish_locks: Arc::new(MetadataPublishLockRegistry {
locks: StdMutex::new(HashMap::new()),
}),
@@ -1695,81 +1649,6 @@ impl BucketMetadataSys {
.expect("bucket metadata object store should still be live")
}
pub(crate) async fn pending_bucket_creation(&self, bucket: &str) -> Option<PendingBucketCreationProof> {
self.pending_bucket_creations.read().await.get(bucket).cloned()
}
pub(crate) async fn grant_pending_bucket_creation(&self, bucket: &str, proof: PendingBucketCreationProof) {
self.pending_bucket_creations.write().await.insert(bucket.to_owned(), proof);
}
pub(crate) async fn revoke_pending_bucket_creation(&self, bucket: &str) {
self.pending_bucket_creations.write().await.remove(bucket);
}
pub(crate) async fn consume_pending_bucket_creation(
&self,
bucket: &str,
metadata: &BucketMetadata,
witnesses: &[crate::disk::os::NativeBucketCreationWitness],
) -> Result<bool> {
let mut pending = self.pending_bucket_creations.write().await;
let Some(proof) = pending.get(bucket) else {
return Ok(false);
};
if !proof.matches_metadata(metadata)?
|| proof.witnesses.len() != witnesses.len()
|| !proof
.witnesses
.iter()
.zip(witnesses)
.all(|(old, current)| old.matches(current))
|| !crate::disk::os::consume_native_creation_witnesses(witnesses)
{
return Ok(false);
}
pending.remove(bucket);
Ok(true)
}
#[cfg(test)]
pub(crate) async fn refresh_bucket_for_test(&self, bucket: &str) -> Result<()> {
let expected = self.metadata_map.read().await.get(bucket).cloned();
let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
self.load_bucket_under_namespace(bucket, MetadataLoadMode::Refresh, expected.as_ref(), &guard)
.await
}
#[cfg(test)]
pub(crate) async fn publish_persisted_metadata_for_test(&self, bucket: &str) -> Result<()> {
let expected = self.metadata_map.read().await.get(bucket).cloned();
let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
let (metadata, persisted) = await_bucket_namespace_operation(
Some(&guard),
bucket,
"test cached metadata load",
load_bucket_metadata_parse_with_presence(self.object_store(), bucket, true),
)
.await?;
self.publish_if_unchanged(bucket, expected.as_ref(), metadata, persisted, &guard)
.await
}
#[cfg(test)]
pub(crate) async fn migrate_bucket_creation_commit_for_test(&self, bucket: &str) -> Result<()> {
self.migrate_bucket_creation_commit(bucket).await.map(|_| ())
}
#[cfg(test)]
pub(crate) async fn legacy_migration_reports_missing_bucket_for_test(&self, bucket: &str) -> Result<bool> {
Ok(matches!(
self.migrate_legacy_metadata(bucket).await?,
BucketMetadataAuthority::MissingBucket
))
}
fn object_store_if_live(&self) -> Option<Arc<ECStore>> {
self.api.upgrade()
}
@@ -1960,7 +1839,6 @@ impl BucketMetadataSys {
expected: Option<&Arc<BucketMetadata>>,
namespace_guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<()> {
self.revoke_pending_bucket_creation(bucket).await;
if !await_bucket_namespace_operation(
Some(namespace_guard),
bucket,
@@ -2044,7 +1922,6 @@ impl BucketMetadataSys {
persisted: bool,
namespace_guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<()> {
self.revoke_pending_bucket_creation(bucket).await;
if !persisted {
let _publish_guard = self
.lock_metadata_publish(bucket, namespace_guard, "refreshed bucket metadata absence publish")
@@ -2106,7 +1983,6 @@ impl BucketMetadataSys {
}
pub async fn set(&self, bucket: String, bm: Arc<BucketMetadata>) {
self.revoke_pending_bucket_creation(&bucket).await;
if !is_meta_bucketname(&bucket) {
let publish_lock = self.metadata_publish_lock(&bucket);
let _publish_guard = publish_lock.lock().await;
@@ -2128,7 +2004,6 @@ impl BucketMetadataSys {
if is_meta_bucketname(bucket) {
return false;
}
self.revoke_pending_bucket_creation(bucket).await;
let publish_lock = self.metadata_publish_lock(bucket);
let _publish_guard = publish_lock.lock().await;
let mut map = self.metadata_map.write().await;
@@ -2146,7 +2021,6 @@ impl BucketMetadataSys {
}
async fn _reset(&mut self) {
self.pending_bucket_creations.write().await.clear();
let mut map = self.metadata_map.write().await;
map.clear();
drop(map);
@@ -2285,7 +2159,6 @@ impl BucketMetadataSys {
/// server's metadata never leaks into the ambient (first) instance.
pub(crate) async fn persist_and_set(&self, bm: BucketMetadata) -> Result<()> {
let mut bm = bm;
self.revoke_pending_bucket_creation(&bm.name).await;
bm.save_with_store(self.object_store()).await?;
self.set(bm.name.clone(), Arc::new(bm)).await;
@@ -2294,7 +2167,6 @@ impl BucketMetadataSys {
}
async fn persist_new_and_set(&self, mut bm: BucketMetadata) -> Result<()> {
self.revoke_pending_bucket_creation(&bm.name).await;
bm.bucket_creation_committed = true;
bm.save_with_store_committed(self.object_store()).await?;
save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
@@ -2304,9 +2176,7 @@ impl BucketMetadataSys {
}
async fn persist_new_bucket_metadata_intent(&self, mut bm: BucketMetadata) -> Result<()> {
self.revoke_pending_bucket_creation(&bm.name).await;
bm.bucket_creation_committed = false;
bm.bucket_creation_commit_record_present = true;
bm.save_with_store(self.object_store()).await?;
save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
bm.bucket_incarnation_sidecar = true;
@@ -2316,7 +2186,6 @@ impl BucketMetadataSys {
}
async fn commit_bucket_metadata(&self, mut bm: BucketMetadata) -> Result<()> {
self.revoke_pending_bucket_creation(&bm.name).await;
bm.parse_all_configs()?;
bm.bucket_incarnation_sidecar = true;
// The caller holds the full bucket creation fence (lifecycle, metadata
@@ -2365,7 +2234,6 @@ impl BucketMetadataSys {
bucket: &str,
namespace_guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<()> {
self.revoke_pending_bucket_creation(bucket).await;
let expected = self.metadata_map.read().await.get(bucket).cloned();
if !self
.bucket_exists(bucket, namespace_guard, "peer bucket metadata existence check")
@@ -2422,7 +2290,6 @@ impl BucketMetadataSys {
#[cfg(test)]
self.lazy_disk_loads.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
self.revoke_pending_bucket_creation(bucket).await;
let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
#[cfg(test)]
@@ -2731,7 +2598,6 @@ impl BucketMetadataSys {
/// revalidated at write quorum, so a stale snapshot can neither revert an
/// acknowledged configuration update nor outlive a delete/recreate.
async fn migrate_bucket_creation_commit(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
self.revoke_pending_bucket_creation(bucket).await;
let transaction_lock = self
.object_store()
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
@@ -2809,7 +2675,6 @@ impl BucketMetadataSys {
&self,
bucket: &str,
) -> Result<BucketMetadataAuthority> {
self.revoke_pending_bucket_creation(bucket).await;
#[cfg(test)]
if self.object_lock_disk_read_errors.write().await.remove(bucket) {
return Err(Error::other(format!("injected Object Lock metadata disk read failure: {bucket}")));
@@ -3181,7 +3046,6 @@ mod tests {
};
use crate::bucket::target::{BucketTarget, BucketTargetType, Credentials};
use crate::config::com::read_config;
use crate::disk::{DiskAPI as _, RUSTFS_META_TMP_BUCKET};
use crate::storage_api_contracts::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions};
use byteorder::{ByteOrder as _, LittleEndian};
use serial_test::serial;
@@ -3236,180 +3100,6 @@ mod tests {
),
];
#[tokio::test]
#[serial(storage_class_env)]
async fn persistence_entry_failures_revoke_pending_creation_before_cache_publish() {
let (dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("instance metadata system");
let sys = sys.read().await.clone();
let options = MakeBucketOptions {
lock_enabled: true,
..Default::default()
};
for entry in ["save", "persist", "persist-new", "commit"] {
let bucket = format!("revoke-{entry}-{}", Uuid::new_v4());
store.fail_next_bucket_creation_after_intent_for_test(&bucket);
store
.make_bucket(&bucket, &options)
.await
.expect_err("stop the real producer before physical publication");
sys.pending_bucket_creation(&bucket)
.await
.expect("the producer must leave a real live creation permission");
let (intent, persisted) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the original producer intent");
assert!(persisted, "{entry}: the complete original intent must be persisted");
let original_intent = intent.marshal_msg().expect("encode the original intent");
// Keep the creation fence order used by the real producer.
let lifecycle_guard = store
.acquire_bucket_lifecycle_write_lock(&bucket)
.await
.expect("hold the bucket lifecycle fence");
let transaction_guard = acquire_bucket_metadata_transaction_lock_in(&store.ctx, &bucket)
.await
.expect("hold the metadata transaction fence");
let namespace_lock = store
.new_ns_lock(&bucket, &bucket)
.await
.expect("create the exact bucket namespace lock");
let namespace_guard = namespace_lock
.get_write_lock(crate::set_disk::get_lock_acquire_timeout())
.await
.expect("hold the exact bucket namespace fence");
// A file at each native staging root causes actual IO to fail before
// metadata can reach the cache-publishing set call. Preserve the
// original staging directories so no fixture data is discarded.
for dir in &dirs {
let staging = dir.path().join(RUSTFS_META_TMP_BUCKET);
let saved = dir.path().join(".rustfs.sys/revocation-saved-tmp");
std::fs::rename(&staging, &saved).expect("preserve the native staging directory");
std::fs::write(&staging, b"block metadata staging with a non-directory")
.expect("install an actual non-directory IO obstruction");
}
let result = match entry {
"save" => sys.save(intent).await,
"persist" => sys.persist_and_set(intent).await,
"persist-new" => sys.persist_new_and_set(intent).await,
"commit" => sys.commit_bucket_metadata(intent).await,
_ => unreachable!(),
};
for dir in &dirs {
let staging = dir.path().join(RUSTFS_META_TMP_BUCKET);
let saved = dir.path().join(".rustfs.sys/revocation-saved-tmp");
std::fs::remove_file(&staging).expect("remove only the fixture's staging obstruction");
std::fs::rename(&saved, &staging).expect("restore the complete original staging directory");
}
drop((namespace_guard, transaction_guard, lifecycle_guard));
result.expect_err("native persistence must fail before its downstream cache set");
assert!(
sys.pending_bucket_creation(&bucket).await.is_none(),
"{entry}: the entry revoker must retire live permission even when persistence fails"
);
let (after_failure, present) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the intent after the failed persistence entry");
assert!(present, "{entry}: a failed persistence entry must retain the original metadata");
assert_eq!(
after_failure
.marshal_msg()
.expect("encode the original intent after persistence failure"),
original_intent,
"{entry}: native IO failure must not change the original generation or complete intent"
);
let error = store
.make_bucket(&bucket, &options)
.await
.expect_err("a failed ordinary persistence entry cannot preserve creation permission");
assert!(error.to_string().contains("administrator recovery"), "{entry}: {error}");
let (after_retry, _) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the original intent after the rejected retry");
assert_eq!(
after_retry.marshal_msg().expect("encode the intent after the rejected retry"),
original_intent,
"{entry}: the rejected retry must preserve the complete original intent"
);
for disk in store.pools[0].disk_set[0].disks.read().await.iter() {
assert!(
matches!(
disk.as_ref().expect("native disk").stat_volume(&bucket).await,
Err(crate::disk::error::DiskError::VolumeNotFound)
),
"{entry}: failed persistence and the rejected retry must not publish a physical volume"
);
}
}
}
#[tokio::test]
#[serial(storage_class_env)]
async fn metadata_reset_revokes_real_pending_creation_without_changing_persisted_intent() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = format!("revoke-reset-{}", Uuid::new_v4());
let options = MakeBucketOptions {
lock_enabled: true,
..Default::default()
};
store.fail_next_bucket_creation_after_intent_for_test(&bucket);
store
.make_bucket(&bucket, &options)
.await
.expect_err("stop the real producer before physical publication");
let sys = bucket_metadata_sys_of(&store.ctx).expect("instance metadata system");
let mut sys = sys.read().await.clone();
sys.pending_bucket_creation(&bucket)
.await
.expect("the real producer must leave live creation permission before reset");
let (intent, persisted) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the original intent before cache reset");
assert!(persisted, "cache reset starts from a complete real persisted intent");
let original_intent = intent.marshal_msg().expect("encode the original reset intent");
sys._reset().await;
assert!(
sys.pending_bucket_creation(&bucket).await.is_none(),
"reset must revoke the producer-issued permission shared with the instance metadata system"
);
let (after_reset, present) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the original intent after cache reset");
assert!(present, "cache reset must retain the persisted intent");
assert_eq!(
after_reset.marshal_msg().expect("encode the intent after cache reset"),
original_intent,
"cache reset must leave the original generation and complete persisted intent unchanged"
);
let error = store
.make_bucket(&bucket, &options)
.await
.expect_err("reset cannot regrant the producer's old creation permission");
assert!(error.to_string().contains("administrator recovery"), "{error}");
let (after_retry, _) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the intent after the reset-induced retry rejection");
assert_eq!(
after_retry
.marshal_msg()
.expect("encode the intent after the rejected reset retry"),
original_intent,
"the reset-induced rejection must preserve the original complete intent"
);
for disk in store.pools[0].disk_set[0].disks.read().await.iter() {
assert!(
matches!(
disk.as_ref().expect("native disk").stat_volume(&bucket).await,
Err(crate::disk::error::DiskError::VolumeNotFound)
),
"cache reset and the rejected retry must not publish a physical volume"
);
}
}
#[tokio::test]
async fn g_d3_003_new_writer_replication_loads_without_fail_closed_state() {
let (dirs, store) = isolated_store_over_temp_disks().await;
@@ -5166,117 +4856,6 @@ mod tests {
);
}
#[tokio::test]
#[serial]
async fn peer_reload_clears_negative_cache_after_remote_bucket_creation() {
let (dirs, node_a, node_b) = two_nodes_over_one_store().await;
let bucket = "cross-node-negative-cache-refresh";
let node_b_sys = node_b.read().await.clone();
assert_eq!(
node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap_err(),
Error::BucketNotFound(bucket.to_string()),
"the probe must populate node B's missing-bucket cache"
);
for dir in &dirs {
std::fs::create_dir_all(dir.path().join(bucket)).expect("physical bucket should exist before creation metadata");
}
let created = BucketMetadata::new(bucket);
let created_incarnation = created.bucket_incarnation_id;
node_a
.read()
.await
.persist_new_and_set(created)
.await
.expect("node A should publish the created bucket metadata");
node_b_sys
.reload_from_store(bucket)
.await
.expect("peer reload should clear the stale negative cache");
assert_eq!(
node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap(),
created_incarnation,
"node B should resolve the newly created bucket immediately"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn peer_delete_clears_metadata_for_absent_bucket() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
let bucket = "peer-delete-absent-bucket";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("initial bucket creation");
let cached = sys.read().await.get(bucket).await.expect("initial cached metadata");
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("bucket deletion");
// Model a peer that still caches the metadata after the physical delete.
sys.read().await.set(bucket.to_string(), cached).await;
assert!(
remove_bucket_metadata_if_absent(store.clone(), bucket)
.await
.expect("conditional peer delete"),
"the peer must evict the deleted bucket's cached metadata"
);
let sys = sys.read().await;
assert!(matches!(sys.get(bucket).await, Err(Error::ConfigNotFound)));
assert_eq!(
sys.get_bucket_incarnation_id(bucket)
.await
.expect_err("deleted bucket lookup"),
Error::BucketNotFound(bucket.to_string()),
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn delayed_peer_delete_preserves_recreated_bucket_metadata() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
let bucket = "delayed-peer-delete-recreated-bucket";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("initial bucket creation");
let old_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("old incarnation");
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("bucket deletion");
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket recreation");
let new_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("new incarnation");
assert_ne!(old_incarnation, new_incarnation);
let removed = remove_bucket_metadata_if_absent(store.clone(), bucket)
.await
.expect("conditional peer delete");
assert!(!removed, "a delayed delete must not remove metadata for a live recreated bucket");
assert_eq!(
sys.read()
.await
.get(bucket)
.await
.expect("recreated metadata should remain cached")
.bucket_incarnation_id,
new_incarnation,
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn stale_config_request_cannot_mutate_a_recreated_bucket() {
-24
View File
@@ -273,30 +273,6 @@ where
let buckets: Vec<String> = buckets_list.into_iter().map(|b| b.name).collect();
try_migrate_bucket_metadata_for_buckets(store, &buckets).await
}
/// Migrate the already discovered startup candidates without repeating a
/// majority-only namespace listing. Every migration write retains its quorum.
pub async fn try_migrate_bucket_metadata_for_buckets<S>(store: Arc<S>, buckets: &[String]) -> Result<()>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
> + ObjectOperations<
Error = Error,
ObjectInfo = ObjectInfo,
ObjectOptions = ObjectOptions,
FileInfo = FileInfo,
ObjectToDelete = ObjectToDelete,
DeletedObject = DeletedObject,
>,
{
if buckets.is_empty() {
debug!("No migrating bucket metadata found");
return Ok(());
+29 -147
View File
@@ -24,11 +24,10 @@ use crate::cluster::rpc::{gen_tonic_replay_scope_headers, gen_tonic_signature_he
#[cfg(test)]
use crate::cluster::rpc::{tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers};
use crate::disk::error::{DiskError, Error as DiskErrorType, RpcStatusError};
#[cfg(test)]
use crate::runtime::sources as runtime_sources;
use http::{Request as HttpRequest, Response as HttpResponse, Uri};
use rustfs_protos::{
ChannelClass, get_channel_for_class,
ChannelClass, create_new_channel, get_channel_for_class,
proto_gen::node_service::{
heal_control_service_client::HealControlServiceClient, node_service_client::NodeServiceClient,
scanner_control_service_client::ScannerControlServiceClient,
@@ -54,7 +53,7 @@ use super::context_propagation::{inject_request_id_into_metadata, inject_trace_c
/// 3. Subsequent calls will attempt fresh connections
/// 4. If node is still down, connection will fail fast (3s timeout)
pub async fn node_service_time_out_client(
addr: &str,
addr: &String,
interceptor: TonicInterceptor,
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
// Default to the latency-sensitive control channel; bulk `bytes` RPCs opt in via the
@@ -67,9 +66,12 @@ pub(crate) async fn scanner_control_time_out_client(
interceptor: TonicInterceptor,
) -> crate::error::Result<ScannerControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = get_channel_for_class(addr, ChannelClass::Control)
.await
.map_err(|err| crate::error::Error::other(err.to_string()))?;
let channel = match runtime_sources::cached_node_channel(addr).await {
Some(channel) => channel,
None => create_new_channel(addr)
.await
.map_err(|err| crate::error::Error::other(err.to_string()))?,
};
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
let limit = rustfs_protos::scoped_dirty_usage::SCOPED_DIRTY_USAGE_MAX_REQUEST_BYTES as usize;
Ok(ScannerControlServiceClient::with_interceptor(channel, interceptor)
@@ -82,7 +84,10 @@ pub async fn heal_control_time_out_client(
interceptor: TonicInterceptor,
) -> Result<HealControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
let channel = match runtime_sources::cached_node_channel(addr).await {
Some(channel) => channel,
None => create_new_channel(addr).await?,
};
let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
Ok(HealControlServiceClient::with_interceptor(channel, interceptor)
@@ -95,7 +100,10 @@ pub async fn tier_mutation_control_time_out_client(
interceptor: TonicInterceptor,
) -> Result<TierMutationControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
let channel = match runtime_sources::cached_node_channel(addr).await {
Some(channel) => channel,
None => create_new_channel(addr).await?,
};
let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
Ok(TierMutationControlServiceClient::with_interceptor(channel, interceptor)
@@ -110,12 +118,22 @@ pub async fn tier_mutation_control_time_out_client(
/// from lock/health RPCs; everything else uses [`ChannelClass::Control`]. When isolation is
/// disabled the two classes resolve to the same cached channel, i.e. legacy behavior.
pub async fn node_service_time_out_client_for_class(
addr: &str,
addr: &String,
interceptor: TonicInterceptor,
class: ChannelClass,
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = get_channel_for_class(addr, class).await?;
let channel = match class {
ChannelClass::Control => match runtime_sources::cached_node_channel(addr).await {
Some(channel) => {
debug!("Using cached gRPC channel for: {}", addr);
channel
}
// No cached connection, create new one.
None => create_new_channel(addr).await?,
},
ChannelClass::Bulk => get_channel_for_class(addr, ChannelClass::Bulk).await?,
};
let max_message_size = rustfs_protos::internode_rpc_max_message_size();
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
@@ -124,24 +142,8 @@ pub async fn node_service_time_out_client_for_class(
.max_encoding_message_size(max_message_size))
}
pub(crate) async fn node_service_delete_recovery_client(
addr: &str,
interceptor: TonicInterceptor,
timeout: std::time::Duration,
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, DiskError> {
let interceptor = interceptor.with_rpc_audience(addr).map_err(DiskError::from)?;
let channel = rustfs_protos::create_channel_with_timeout(addr, timeout)
.await
.map_err(|error| DiskError::RemoteClientUnavailable(error.to_string()))?;
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
let max_message_size = rustfs_protos::internode_rpc_max_message_size();
Ok(NodeServiceClient::with_interceptor(channel, interceptor)
.max_decoding_message_size(max_message_size)
.max_encoding_message_size(max_message_size))
}
pub async fn node_service_time_out_client_no_auth(
addr: &str,
addr: &String,
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
node_service_time_out_client(addr, TonicInterceptor::NoOp(NoOpInterceptor)).await
}
@@ -544,126 +546,6 @@ mod tests {
use tracing_opentelemetry::OpenTelemetrySpanExt;
use tracing_subscriber::{Registry, layer::SubscriberExt};
#[tokio::test]
async fn delete_recovery_channel_uses_its_scoped_deadline() {
use bytes::Bytes;
use http_body_util::StreamBody;
use hyper::{body::Frame, service::service_fn};
use hyper_util::rt::{TokioExecutor, TokioIo};
use rustfs_protos::proto_gen::node_service::{RecoverDeleteTransactionRequest, RecoverDeleteTransactionResponse};
use tonic_prost::prost::Message;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
.await
.expect("bind recovery peer");
let addr = format!("http://{}", listener.local_addr().expect("peer address"));
let server = tokio::spawn(async move {
loop {
let (stream, _) = listener.accept().await.expect("accept recovery connection");
tokio::spawn(async move {
let service = service_fn(|_request: HttpRequest<hyper::body::Incoming>| async {
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
let payload = RecoverDeleteTransactionResponse {
success: true,
protocol_version: 1,
result: Bytes::new(),
error: None,
}
.encode_to_vec();
let mut data = vec![0];
data.extend_from_slice(&u32::try_from(payload.len()).expect("small response").to_be_bytes());
data.extend_from_slice(&payload);
let mut trailers = http::HeaderMap::new();
trailers.insert("grpc-status", "0".parse().expect("status"));
let frames: Vec<Result<Frame<Bytes>, std::convert::Infallible>> =
vec![Ok(Frame::data(data.into())), Ok(Frame::trailers(trailers))];
let mut response = HttpResponse::new(StreamBody::new(futures::stream::iter(frames)));
response
.headers_mut()
.insert("content-type", "application/grpc".parse().expect("content type"));
Ok::<_, std::convert::Infallible>(response)
});
let _ = hyper::server::conn::http2::Builder::new(TokioExecutor::new())
.serve_connection(TokioIo::new(stream), service)
.await;
});
}
});
let request = || RecoverDeleteTransactionRequest {
disk: "disk".into(),
volume: "bucket".into(),
protocol_version: 1,
operation: Bytes::new(),
};
// Seed an existing short-lived control channel to reproduce the inherited deadline.
let control = tonic::transport::Endpoint::from_shared(addr.clone())
.expect("test endpoint")
.timeout(std::time::Duration::from_millis(50))
.connect()
.await
.expect("control connect");
rustfs_common::cache_connection(addr.clone(), control).await;
let mut foreground = node_service_time_out_client(&addr, TonicInterceptor::NoOp(NoOpInterceptor))
.await
.expect("control client");
let error = foreground
.recover_delete_transaction(request())
.await
.expect_err("short control deadline");
assert!(matches!(error.code(), tonic::Code::Cancelled | tonic::Code::DeadlineExceeded));
// The 2-second test override bypasses the production range parser, which rejects 2.
let mut recovery = node_service_delete_recovery_client(
&addr,
TonicInterceptor::NoOp(NoOpInterceptor),
std::time::Duration::from_secs(2),
)
.await
.expect("scoped recovery client");
assert!(
recovery
.recover_delete_transaction(request())
.await
.expect("scoped deadline must survive control deadline")
.into_inner()
.success
);
server.abort();
rustfs_protos::evict_failed_connection(&addr).await;
}
#[tokio::test]
async fn rpc_client_factories_do_not_reuse_https_channels_without_tls_generation() {
let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("reserve local address");
let addr = format!("https://{}", listener.local_addr().expect("local address"));
drop(listener);
for factory in 0..4 {
let placeholder = tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy();
runtime_sources::cache_test_node_channel(addr.clone(), placeholder).await;
let interceptor = TonicInterceptor::NoOp(NoOpInterceptor);
let _result = match factory {
0 => scanner_control_time_out_client(&addr, interceptor)
.await
.map(|_| ())
.map_err(|e| e.to_string()),
1 => heal_control_time_out_client(&addr, interceptor)
.await
.map(|_| ())
.map_err(|e| e.to_string()),
2 => tier_mutation_control_time_out_client(&addr, interceptor)
.await
.map(|_| ())
.map_err(|e| e.to_string()),
_ => node_service_time_out_client(&addr, interceptor)
.await
.map(|_| ())
.map_err(|e| e.to_string()),
};
assert!(
!runtime_sources::test_node_channel_is_cached(&addr).await,
"factory {factory} must evict an unbound HTTPS cache entry before attempting a fresh dial"
);
}
}
#[derive(Clone)]
struct EpochProofService {
audience: String,
@@ -289,9 +289,13 @@ async fn client(
>,
NetworkPeerProbeError,
> {
node_service_time_out_client_for_class(address, TonicInterceptor::Signature(gen_tonic_signature_interceptor()), class)
.await
.map_err(|_| NetworkPeerProbeError::Unreachable)
node_service_time_out_client_for_class(
&address.to_owned(),
TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
class,
)
.await
.map_err(|_| NetworkPeerProbeError::Unreachable)
}
fn ping_request(payload: &[u8]) -> PingRequest {
+32 -363
View File
@@ -14,8 +14,7 @@
use crate::cluster::rpc::client::{
AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
node_service_delete_recovery_client, node_service_time_out_client, node_service_time_out_client_for_class,
node_service_time_out_client_no_auth,
node_service_time_out_client, node_service_time_out_client_for_class, node_service_time_out_client_no_auth,
};
use crate::cluster::rpc::internode_data_transport::{
InternodeDataTransport, NsScannerCapabilityRequest, NsScannerStreamRequest, ReadStreamRequest, WalkDirStreamRequest,
@@ -23,11 +22,10 @@ use crate::cluster::rpc::internode_data_transport::{
};
use crate::disk::error::{Error, Result};
use crate::disk::{
BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION,
DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection,
DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation,
DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq, ReadMultipleResp, ReadOptions,
RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one,
BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DeleteOptions, DiskAPI, DiskInfo,
DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq,
ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
batch_read_version_one_by_one,
disk_store::{
DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE,
get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout,
@@ -61,10 +59,10 @@ use rustfs_protos::proto_gen::node_service::{
CompareAndUpdateFileRequest, DeletePathsRequest, DeleteRequest, DeleteVersionRequest, DeleteVersionsRequest,
DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, ListVolumesRequest, MakeVolumeRequest,
MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, ReadMetadataRequest, ReadMultipleRequest,
ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RecoverDeleteTransactionRequest,
RenameDataRequest, RenameFileRequest, SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest,
SnapshotLeaseRequest, SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest,
WriteMetadataRequest, node_service_client::NodeServiceClient,
ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest, RenameFileRequest,
SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest,
SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest,
node_service_client::NodeServiceClient,
};
use serde::{Serialize, de::DeserializeOwned};
use std::{
@@ -144,45 +142,6 @@ fn decode_delete_versions_errors(response: DeleteVersionsResponse, expected_len:
.collect()
}
fn decode_delete_recovery_batch_errors(codes: Vec<Option<u32>>, expected_len: usize) -> Result<Vec<Option<DiskError>>> {
if codes.len() != expected_len {
return Err(Error::other("malformed recoverable batch delete response length"));
}
codes
.into_iter()
.map(|code| match code {
None => Ok(None),
Some(code) => DiskError::from_u32(code)
.map(Some)
.ok_or_else(|| Error::other("invalid recoverable batch delete error code")),
})
.collect()
}
fn validate_delete_recovery_inspection(
volume: &str,
object: &str,
transaction: Uuid,
inspection: &DeleteRecoveryInspection,
) -> Result<()> {
if inspection.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
|| inspection.id.object != object
|| inspection.id.transaction != transaction
|| transaction.is_nil()
{
return Err(Error::other("delete recovery inspection does not match request"));
}
if let Some(journal) = &inspection.journal
&& (journal.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
|| journal.volume != volume
|| journal.object != object
|| journal.transaction != transaction)
{
return Err(Error::other("delete recovery journal does not match request"));
}
Ok(())
}
fn snapshot_lease_token_from_response(response: SnapshotLeaseResponse) -> Result<SnapshotLeaseToken> {
if !response.success {
return Err(response.error.unwrap_or_default().into());
@@ -899,75 +858,6 @@ fn spawn_control_channel_prewarm(addr: String) {
}
impl RemoteDisk {
async fn recover_delete_transaction(
&self,
volume: &str,
operation: DeleteRecoveryRpcOperation,
) -> Result<DeleteRecoveryRpcReply> {
let payload = rmp_serde::to_vec_named(&operation)?;
if payload.len() > operation.max_wire_payload_bytes() {
return Err(Error::other("delete recovery operation exceeds payload limit"));
}
let settle_timeout = if matches!(&operation, DeleteRecoveryRpcOperation::Settle { .. }) {
Some(crate::disk::get_delete_recovery_timeout()?)
} else {
None
};
self.execute_with_timeout(
|| async {
let mut client = if let Some(timeout) = settle_timeout {
if let Some(error) = self.offline_bypass_error() {
return Err(error);
}
node_service_delete_recovery_client(
&self.addr,
TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
timeout,
)
.await?
} else {
self.get_client().await?
};
let mut request = Request::new(RecoverDeleteTransactionRequest {
disk: self.endpoint.to_string(),
volume: volume.to_owned(),
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
operation: payload.into(),
});
let canonical_body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref())
.map_err(|_| Error::other("recover_delete_transaction request length cannot be represented"))?;
crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical_body).map_err(Error::other)?;
// UNIMPLEMENTED is deliberately propagated: older peers cannot settle transactions.
let response = client
.recover_delete_transaction(request)
.await
.map_err(|status| {
if status.code() == Code::Unimplemented {
DiskError::UnsupportedDisk
} else {
status.into()
}
})?
.into_inner();
if response.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION {
return Err(DiskError::UnsupportedDisk);
}
if !response.success {
return Err(response
.error
.ok_or_else(|| Error::other("missing delete recovery error"))?
.into());
}
if response.error.is_some() {
return Err(Error::other("malformed delete recovery success"));
}
rustfs_protos::decode_delete_recovery_payload(&response.result).map_err(Into::into)
},
settle_timeout.unwrap_or_else(get_max_timeout_duration),
)
.await
}
async fn rename_file_with_durability(
&self,
src_volume: &str,
@@ -2030,12 +1920,6 @@ fn encode_msgpack<T: Serialize>(value: &T) -> Result<Vec<u8>> {
encode_msgpack_with_capacity(value, MSGPACK_ENCODE_CAPACITY_HINT)
}
// Optional fields are omitted by serde; named maps keep rollback flags and
// transaction UUIDs independent of field order across peer versions.
fn encode_delete_options_msgpack(value: &DeleteOptions) -> Result<Vec<u8>> {
Ok(rmp_serde::to_vec_named(value)?)
}
fn encode_file_info_msgpack(value: &FileInfo) -> Result<Vec<u8>> {
encode_msgpack_with_capacity(value, FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT)
}
@@ -2682,7 +2566,7 @@ impl DiskAPI for RemoteDisk {
// `_bin` support for DeleteVersion is new (grpc-optimization P2); always dual-write
// JSON + msgpack until its fallback counter has read zero across a release window.
let file_info_bin = encode_file_info_msgpack(&fi)?;
let opts_bin = encode_delete_options_msgpack(&opts)?;
let opts_bin = encode_msgpack(&opts)?;
let conditional_marker = opts.expected_delete_marker.is_some();
let file_info = serde_json::to_string(&fi)?;
let opts = serde_json::to_string(&opts)?;
@@ -2754,7 +2638,7 @@ impl DiskAPI for RemoteDisk {
// `_bin` support for DeleteVersions is new (grpc-optimization P2); always dual-write JSON +
// msgpack until its fallback counter has read zero across a release window.
let opts_bin = match encode_delete_options_msgpack(&opts) {
let opts_bin = match encode_msgpack(&opts) {
Ok(opts_bin) => opts_bin,
Err(err) => {
let mut errors = Vec::with_capacity(versions.len());
@@ -3733,138 +3617,6 @@ impl DiskAPI for RemoteDisk {
}
#[tracing::instrument(level = "trace", skip_all)]
async fn delete_version_recoverable(
&self,
volume: &str,
object: &str,
file_info: FileInfo,
force_del_marker: bool,
opts: DeleteOptions,
) -> Result<()> {
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::ApplyVersion {
object: object.to_owned(),
file_info: Box::new(file_info),
force_del_marker,
opts,
},
)
.await?
{
DeleteRecoveryRpcReply::AppliedVersion => Ok(()),
_ => Err(Error::other("unexpected recoverable delete reply")),
}
}
async fn delete_versions_recoverable(
&self,
volume: &str,
versions: Vec<FileInfoVersions>,
opts: DeleteOptions,
) -> Vec<Option<DiskError>> {
let expected_len = versions.len();
let result = match self
.recover_delete_transaction(volume, DeleteRecoveryRpcOperation::ApplyVersions { versions, opts })
.await
{
Ok(DeleteRecoveryRpcReply::AppliedVersions(codes)) => decode_delete_recovery_batch_errors(codes, expected_len),
Ok(_) => Err(Error::other("unexpected recoverable batch delete reply")),
Err(error) => Err(error),
};
match result {
Ok(errors) => errors,
Err(error) => vec![Some(error); expected_len],
}
}
async fn discover_delete_transactions(
&self,
volume: &str,
object: Option<&str>,
cursor: Option<DeleteRecoveryCursor>,
limit: u32,
) -> Result<DeleteRecoveryPage> {
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::Discover {
object: object.map(str::to_owned),
cursor,
limit,
},
)
.await?
{
DeleteRecoveryRpcReply::Discovered(page) => {
if object.is_some_and(|object| page.transactions.iter().any(|id| id.object != object)) {
return Err(Error::other("delete recovery discovery escaped requested object"));
}
if page.transactions.len() > usize::try_from(limit).map_err(|_| Error::other("invalid discovery limit"))?
|| page
.transactions
.iter()
.any(|id| id.transaction.is_nil() || id.object.is_empty())
{
return Err(Error::other("malformed delete discovery page"));
}
Ok(page)
}
_ => Err(Error::other("unexpected delete discovery reply")),
}
}
async fn inspect_delete_transaction(
&self,
volume: &str,
object: &str,
transaction: Uuid,
) -> Result<DeleteRecoveryInspection> {
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::Inspect {
object: object.to_owned(),
transaction,
},
)
.await?
{
DeleteRecoveryRpcReply::Inspected(inspection) => {
validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
Ok(inspection)
}
_ => Err(Error::other("unexpected delete inspection reply")),
}
}
async fn settle_delete_transaction(
&self,
volume: &str,
object: &str,
transaction: Uuid,
expected_fingerprint: DeleteRecoveryDigest,
action: DeleteRecoveryAction,
) -> Result<DeleteRecoveryInspection> {
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::Settle {
object: object.to_owned(),
transaction,
expected_fingerprint,
action,
},
)
.await?
{
DeleteRecoveryRpcReply::Settled(inspection) => {
validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
Ok(inspection)
}
_ => Err(Error::other("unexpected delete settlement reply")),
}
}
async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
self.execute_with_timeout(
|| async {
@@ -4278,45 +4030,6 @@ mod tests {
static INIT: Once = Once::new();
#[test]
fn recoverable_batch_delete_rejects_invalid_error_codes_and_length() {
let errors = decode_delete_recovery_batch_errors(vec![None, Some(DiskError::FileNotFound.to_u32())], 2)
.expect("valid per-item errors");
assert!(errors[0].is_none());
assert_eq!(errors[1], Some(DiskError::FileNotFound));
assert!(decode_delete_recovery_batch_errors(vec![None], 2).is_err());
assert!(decode_delete_recovery_batch_errors(vec![Some(0)], 1).is_err());
assert!(decode_delete_recovery_batch_errors(vec![Some(u32::MAX)], 1).is_err());
}
#[test]
fn delete_recovery_inspection_rejects_wrong_transaction_object_and_protocol() {
let transaction = Uuid::new_v4();
let inspection = DeleteRecoveryInspection {
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
id: crate::disk::DeleteTransactionId {
object: "object".into(),
transaction,
},
state: crate::disk::DeleteRecoveryState::Absent,
journal: None,
current_digest: None,
current_versions: Vec::new(),
backup_digest: None,
rollback_data_complete: false,
commit_data_safe: false,
explicit_delete_safe: false,
fingerprint: [0; 32],
};
validate_delete_recovery_inspection("bucket", "object", transaction, &inspection).expect("matching inspection");
assert!(validate_delete_recovery_inspection("bucket", "other", transaction, &inspection).is_err());
assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::new_v4(), &inspection).is_err());
assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::nil(), &inspection).is_err());
let mut incompatible = inspection;
incompatible.protocol_version += 1;
assert!(validate_delete_recovery_inspection("bucket", "object", transaction, &incompatible).is_err());
}
#[test]
fn compare_and_update_wire_outcome_fails_closed() {
assert_eq!(
@@ -4350,19 +4063,32 @@ mod tests {
}
#[test]
fn delete_options_named_wire_preserves_rollback_and_legacy_marker_fields() {
fn retired_marker_options_preserve_legacy_positional_wire_shape() {
#[derive(Debug, serde::Serialize, serde::Deserialize, PartialEq)]
struct LegacyDeleteOptions {
recursive: bool,
immediate: bool,
undo_write: bool,
#[serde(default)]
undo_delete: bool,
old_data_dir: Option<Uuid>,
#[serde(default, skip_serializing_if = "Option::is_none")]
expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
}
let transaction = Uuid::new_v4();
let legacy = LegacyDeleteOptions {
recursive: false,
immediate: false,
undo_write: false,
undo_delete: false,
old_data_dir: None,
};
let original = encode_msgpack(&legacy).unwrap();
let current = encode_msgpack(&DeleteOptions::default()).unwrap();
assert_eq!(current, original, "ordinary deletes must retain the older peer's positional payload");
assert_eq!(rmp_serde::from_slice::<LegacyDeleteOptions>(&current).unwrap(), legacy);
assert!(
rmp_serde::from_slice::<DeleteOptions>(&original)
.unwrap()
.expected_delete_marker
.is_none()
);
let mut marker = FileInfo {
deleted: true,
version_id: Some(Uuid::new_v4()),
@@ -4371,69 +4097,12 @@ mod tests {
};
marker.set_delete_marker_incarnation(Uuid::new_v4());
let marker = rustfs_filemeta::MetaDeleteMarker::from(marker);
for expected_delete_marker in [None, Some(marker)] {
for (undo_write, undo_delete) in [(true, false), (false, true)] {
let legacy = LegacyDeleteOptions {
recursive: false,
immediate: false,
undo_write,
undo_delete,
old_data_dir: Some(transaction),
expected_delete_marker: expected_delete_marker.clone(),
};
// Older senders use positional arrays. Their unchanged field prefix and
// missing optional fields must also decode on the named-map sender's peer.
for (encoding, payload) in [
("positional", rmp_serde::to_vec(&legacy).expect("encode legacy positional options")),
("named", rmp_serde::to_vec_named(&legacy).expect("encode legacy named options")),
] {
let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode legacy options on current peer");
assert_eq!(decoded.undo_write, undo_write, "{encoding} undo-write flag");
assert_eq!(decoded.undo_delete, undo_delete, "{encoding} undo-delete flag");
assert_eq!(decoded.old_data_dir, Some(transaction), "{encoding} transaction UUID");
assert_eq!(decoded.expected_delete_marker, expected_delete_marker, "{encoding} marker precondition");
assert!(decoded.delete_recovery_bucket_incarnation.is_none(), "{encoding} omitted incarnation");
assert!(!decoded.recoverable_delete, "{encoding} omitted recovery flag");
}
for incarnation in [None, Some(Uuid::new_v4())] {
let current = DeleteOptions {
undo_write,
undo_delete,
old_data_dir: Some(transaction),
expected_delete_marker: expected_delete_marker.clone(),
delete_recovery_bucket_incarnation: incarnation,
recoverable_delete: true,
..Default::default()
};
let payload = encode_delete_options_msgpack(&current).expect("encode current named options");
let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode current options on current peer");
assert_eq!(decoded.undo_write, undo_write);
assert_eq!(decoded.undo_delete, undo_delete);
assert_eq!(decoded.old_data_dir, Some(transaction));
assert_eq!(decoded.expected_delete_marker, expected_delete_marker);
assert_eq!(decoded.delete_recovery_bucket_incarnation, incarnation);
assert!(decoded.recoverable_delete);
let decoded: LegacyDeleteOptions =
rmp_serde::from_slice(&payload).expect("decode current options on legacy peer");
assert_eq!(decoded, legacy, "unknown recovery fields cannot shift rollback or marker values");
}
}
}
}
#[test]
fn malformed_nonempty_delete_options_do_not_fall_back_to_json() {
let options = DeleteOptions {
undo_write: true,
old_data_dir: Some(Uuid::new_v4()),
expected_delete_marker: Some(marker.clone()),
..Default::default()
};
let json = serde_json::to_string(&options).expect("valid compatibility JSON");
assert!(decode_msgpack_or_json::<DeleteOptions>(&[0xc1], &json, "DeleteOptions").is_err());
let decoded: DeleteOptions =
decode_msgpack_or_json(&[], &json, "DeleteOptions").expect("absent MessagePack permits JSON compatibility");
assert!(decoded.undo_write);
assert_eq!(decoded.old_data_dir, options.old_data_dir);
let decoded: DeleteOptions = rmp_serde::from_slice(&encode_msgpack(&options).unwrap()).unwrap();
assert_eq!(decoded.expected_delete_marker, Some(marker));
}
#[test]
+76 -8
View File
@@ -495,10 +495,22 @@ pub(crate) async fn read_config_preserve_empty<S>(api: Arc<S>, file: &str) -> Re
where
S: EcstoreObjectIO,
{
let (data, _obj) = read_config_with_metadata_inner(api, file, &ObjectOptions::default(), true, None).await?;
let (data, _obj) = read_config_preserve_empty_with_metadata_opts(api, file, &ObjectOptions::default()).await?;
Ok(data)
}
/// Preserve an empty config payload, its metadata, and caller-supplied read options.
pub(crate) async fn read_config_preserve_empty_with_metadata_opts<S>(
api: Arc<S>,
file: &str,
opts: &ObjectOptions,
) -> Result<(Vec<u8>, ObjectInfo)>
where
S: EcstoreObjectIO,
{
read_config_with_metadata_inner(api, file, opts, true, None).await
}
pub async fn read_config_no_lock<S>(api: Arc<S>, file: &str) -> Result<Vec<u8>>
where
S: ObjectIO<
@@ -527,15 +539,13 @@ pub(crate) async fn read_config_no_lock_preserve_empty_with_metadata<S>(api: Arc
where
S: EcstoreObjectIO,
{
read_config_with_metadata_inner(
read_config_preserve_empty_with_metadata_opts(
api,
file,
&ObjectOptions {
no_lock: true,
..Default::default()
},
true,
None,
)
.await
}
@@ -2815,9 +2825,10 @@ mod tests {
config_task_join_error, configs_semantically_equal, decode_server_config_blob, encode_server_config_blob,
heal_config_descriptor, is_standard_object_server_config, lookup_configs, map_system_metadata_write_error,
new_and_save_server_config, read_config, read_config_no_lock_preserve_empty_with_metadata, read_config_preserve_empty,
read_config_with_metadata, read_config_without_migrate, read_server_config_snapshot, save_config_with_opts_inner,
save_server_config, save_server_config_snapshot, save_server_config_snapshot_with_generation,
server_config_transaction_lock_path, should_warn_ignored_scalar_section, storage_class_kvs_mut,
read_config_preserve_empty_with_metadata_opts, read_config_with_metadata, read_config_without_migrate,
read_server_config_snapshot, save_config_with_opts_inner, save_server_config, save_server_config_snapshot,
save_server_config_snapshot_with_generation, server_config_transaction_lock_path, should_warn_ignored_scalar_section,
storage_class_kvs_mut,
};
use crate::config::{audit, heal, notify, oidc, scanner};
use crate::disk::{RUSTFS_META_BUCKET, endpoint::Endpoint};
@@ -5065,6 +5076,7 @@ mod tests {
heal_replacement: Option<Vec<u8>>,
heal_calls: AtomicUsize,
write_calls: AtomicUsize,
last_get_no_lock: AtomicBool,
last_put_no_lock: AtomicBool,
last_put_preconditions: Mutex<Option<HTTPPreconditions>>,
revision: AtomicUsize,
@@ -5080,6 +5092,7 @@ mod tests {
heal_replacement,
heal_calls: AtomicUsize::new(0),
write_calls: AtomicUsize::new(0),
last_get_no_lock: AtomicBool::new(false),
last_put_no_lock: AtomicBool::new(false),
last_put_preconditions: Mutex::new(None),
revision: AtomicUsize::new(1),
@@ -5135,8 +5148,9 @@ mod tests {
_object: &str,
_range: Option<HTTPRangeSpec>,
_h: HeaderMap,
_opts: &ObjectOptions,
opts: &ObjectOptions,
) -> Result<GetObjectReader> {
self.last_get_no_lock.store(opts.no_lock, Ordering::SeqCst);
let data = match &*self.state.lock().expect("state lock poisoned") {
RecoveryReadState::Missing => return Err(Error::ConfigNotFound),
RecoveryReadState::Blob(data) => data.clone(),
@@ -5431,6 +5445,60 @@ mod tests {
assert!(data.is_empty());
}
#[tokio::test]
async fn read_config_preserve_empty_metadata_keeps_empty_and_options() {
let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(Vec::new()), None));
for no_lock in [false, true] {
let opts = ObjectOptions {
no_lock,
..Default::default()
};
let (data, info) = read_config_preserve_empty_with_metadata_opts(store.clone(), "config/empty.json", &opts)
.await
.expect("an existing empty config must retain its payload and metadata");
assert!(data.is_empty());
assert_eq!(info.size, 0);
assert_eq!(info.actual_size, 0);
assert_eq!(info.etag.as_deref(), Some("config-1"));
assert_eq!(info.data_dir, Some(uuid::Uuid::from_u128(1)));
assert_eq!(store.last_get_no_lock.load(Ordering::SeqCst), no_lock);
}
let err = read_config(store, "config/empty.json")
.await
.expect_err("the existing reader must still treat empty configs as missing");
assert!(matches!(err, Error::ConfigNotFound));
}
#[tokio::test]
async fn read_config_preserve_empty_metadata_keeps_payload_and_metadata() {
let payload = b"original config bytes".to_vec();
let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(payload.clone()), None));
let (data, info) =
read_config_preserve_empty_with_metadata_opts(store, "config/nonempty.json", &ObjectOptions::default())
.await
.expect("a nonempty config must retain its payload and metadata");
assert_eq!(data, payload);
let size = i64::try_from(payload.len()).expect("test payload size fits i64");
assert_eq!(info.size, size);
assert_eq!(info.actual_size, size);
assert_eq!(info.etag.as_deref(), Some("config-1"));
assert_eq!(info.data_dir, Some(uuid::Uuid::from_u128(1)));
}
#[tokio::test]
async fn read_config_preserve_empty_metadata_preserves_read_errors() {
let missing = Arc::new(RecoveryMockStore::new(RecoveryReadState::Missing, None));
let err = read_config_preserve_empty_with_metadata_opts(missing, "config/missing.json", &ObjectOptions::default())
.await
.expect_err("physical absence must remain ConfigNotFound");
assert!(matches!(err, Error::ConfigNotFound));
let quorum = Arc::new(RecoveryMockStore::new(RecoveryReadState::QuorumError, None));
let err = read_config_preserve_empty_with_metadata_opts(quorum, "config/quorum.json", &ObjectOptions::default())
.await
.expect_err("a quorum failure must retain its typed read error");
assert!(matches!(err, Error::ErasureReadQuorum));
}
#[async_trait::async_trait]
impl StorageAdminApi for RecoveryMockStore {
type BackendInfo = rustfs_madmin::BackendInfo;
-40
View File
@@ -72,38 +72,6 @@ type WalkOptions = StorageWalkOptions<fn(&FileInfo) -> bool>;
const LIST_MULTIPART_SETS_CONCURRENCY: usize = 4;
#[cfg(test)]
type TestSetLockers = HashMap<usize, Vec<Arc<dyn rustfs_lock::LockClient>>>;
#[cfg(test)]
static TEST_SET_LOCKERS: std::sync::LazyLock<std::sync::Mutex<TestSetLockers>> =
std::sync::LazyLock::new(|| std::sync::Mutex::new(HashMap::new()));
/// Keep a fixture's real locker domain scoped to its owning instance.
#[cfg(test)]
pub(crate) struct ScopedSetLockers(Arc<InstanceContext>);
#[cfg(test)]
impl ScopedSetLockers {
pub(crate) fn install(ctx: Arc<InstanceContext>, lockers: Vec<Arc<dyn rustfs_lock::LockClient>>) -> Self {
let key = Arc::as_ptr(&ctx) as usize;
let mut domains = TEST_SET_LOCKERS.lock().expect("test set lockers mutex");
assert!(!domains.contains_key(&key), "the instance already owns a test locker domain");
domains.insert(key, lockers);
Self(ctx)
}
}
#[cfg(test)]
impl Drop for ScopedSetLockers {
fn drop(&mut self) {
TEST_SET_LOCKERS
.lock()
.expect("test set lockers mutex")
.remove(&(Arc::as_ptr(&self.0) as usize));
}
}
fn is_idempotent_delete_prefix_error(err: &Error) -> bool {
is_err_object_not_found(err) || is_err_strict_volume_not_found(err)
}
@@ -305,14 +273,6 @@ impl Sets {
}
}
#[cfg(test)]
let pool_lockers = TEST_SET_LOCKERS
.lock()
.expect("test set lockers mutex")
.get(&(Arc::as_ptr(&instance_ctx) as usize))
.cloned()
.unwrap_or_else(|| pool_lockers.clone());
let set_disks = SetDisks::new_with_instance_ctx(
runtime_sources::local_node_name().await,
Arc::new(RwLock::new(set_drive)),
@@ -15,9 +15,7 @@
use crate::cluster::rpc::{
ScannerBucketListing, TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client,
};
use crate::data_usage::{
DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_admin_data_usage_from_backend_cached as load_server_info_usage_snapshot,
};
use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend_cached};
use crate::error::{Error, Result};
use crate::{
disk::endpoint::{Endpoint, EndpointType},
@@ -296,7 +294,7 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
if let Some(store) = runtime_sources::object_store_handle() {
mode = ITEM_ONLINE;
apply_data_usage_result(
load_server_info_usage_snapshot(store.clone()).await,
load_data_usage_from_backend_cached(store.clone()).await,
&mut buckets,
&mut objects,
&mut versions,
@@ -969,93 +967,6 @@ mod tests {
assert_eq!(usage.size, 6);
}
#[tokio::test]
#[serial]
async fn server_info_reads_complete_observation_without_promoting_authority() {
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
use crate::config::com::{read_config, save_config};
use crate::data_usage::{
DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache,
load_data_usage_from_backend_cached,
};
use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
use std::time::{Duration, SystemTime};
let (_dirs, store) = isolated_store_over_temp_disks().await;
let primary = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
scanner_epoch: Some(1),
usage_snapshot_bootstrap_pending: true,
usage_snapshot_converged: Some(false),
..Default::default()
};
let observed = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(2)),
scanner_epoch: Some(1),
scanner_cycle: Some(1),
buckets_count: 1,
objects_total_count: 3,
versions_total_count: 4,
delete_markers_total_count: 1,
objects_total_size: 6,
buckets_usage: [(
"bucket".to_string(),
BucketUsageInfo {
objects_count: 3,
versions_count: 4,
delete_markers_count: 1,
size: 6,
..Default::default()
},
)]
.into_iter()
.collect(),
usage_snapshot_complete: true,
usage_snapshot_converged: Some(false),
usage_snapshot_authoritative_baseline: Some(primary.snapshot_identity()),
..Default::default()
};
let primary_bytes = serde_json::to_vec(&primary).expect("bootstrap should encode");
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
.await
.expect("bootstrap should persist");
save_config(
store.clone(),
DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&observed).expect("observation should encode"),
)
.await
.expect("observation should persist");
invalidate_data_usage_snapshot_cache().await;
let displayed = super::load_server_info_usage_snapshot(store.clone())
.await
.expect("server info should load the complete observation");
assert_eq!(displayed.usage_snapshot_converged, Some(false));
let mut buckets = rustfs_madmin::Buckets::default();
let mut objects = rustfs_madmin::Objects::default();
let mut versions = rustfs_madmin::Versions::default();
let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
let mut usage = rustfs_madmin::Usage::default();
apply_data_usage_result(Ok(displayed), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
assert!(objects.error.is_none());
assert_eq!(
(buckets.count, objects.count, versions.count, delete_markers.count, usage.size),
(1, 3, 4, 1, 6)
);
let quota = load_data_usage_from_backend_cached(store.clone())
.await
.expect("quota should retain its bootstrap state");
assert!(!quota.is_complete_bucket_usage_snapshot());
assert_eq!(
read_config(store, DATA_USAGE_OBJ_NAME_PATH.as_str())
.await
.expect("primary should remain readable"),
primary_bytes
);
invalidate_data_usage_snapshot_cache().await;
}
#[test]
fn incomplete_data_usage_is_unavailable_in_server_info() {
let mut buckets = rustfs_madmin::Buckets::default();
-1
View File
@@ -182,7 +182,6 @@ pub(crate) const GET_METADATA_CACHE_REASON_DELETE_MARKER: &str = "delete_marker"
pub(crate) const GET_METADATA_CACHE_REASON_DIST_ERASURE: &str = "dist_erasure";
pub(crate) const GET_METADATA_CACHE_REASON_INCL_FREE_VERSIONS: &str = "incl_free_versions";
pub(crate) const GET_METADATA_CACHE_REASON_INSUFFICIENT_CACHED_QUORUM: &str = "insufficient_cached_quorum";
pub(crate) const GET_METADATA_CACHE_REASON_BODY_LAYOUT_REQUIRED: &str = "body_layout_required";
pub(crate) const GET_METADATA_CACHE_REASON_META_BUCKET: &str = "meta_bucket";
pub(crate) const GET_METADATA_CACHE_REASON_NO_LOCK: &str = "no_lock";
pub(crate) const GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED: &str = "not_found_or_expired";
+42 -170
View File
@@ -466,11 +466,12 @@ impl LocalDiskWrapper {
self.track_disk_health_mutation(
"delete_version",
DiskMetricMutation::Delete,
|| {
|| async {
Box::pin(
self.disk
.delete_version_with_namespace_owner(volume, path, fi, force_del_marker, opts, namespace_owner),
)
.await
},
get_max_timeout_duration(),
)
@@ -1815,8 +1816,6 @@ impl LocalDiskWrapper {
undo_delete: false,
old_data_dir: None,
expected_delete_marker: None,
recoverable_delete: false,
delete_recovery_bucket_incarnation: None,
},
)
.await?;
@@ -2002,86 +2001,10 @@ impl LocalDiskWrapper {
}
/// Get the current disk ID
pub(crate) async fn delete_versions_with_namespace_owner(
&self,
volume: &str,
versions: Vec<FileInfoVersions>,
opts: DeleteOptions,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Vec<Option<Error>> {
let version_count = versions.len();
self.metrics.record_operation_call("delete_versions");
// Check if disk is faulty before proceeding
if self.health.is_faulty() {
self.metrics.record_availability_error();
return vec![Some(DiskError::FaultyDisk); versions.len()];
}
// Check if disk is stale
if let Err(e) = self.check_disk_stale().await {
self.metrics.record_availability_error();
return vec![Some(e); versions.len()];
}
let _io_permit = match self.acquire_disk_io_permit().await {
Ok(permit) => permit,
Err(error) => return vec![Some(error); version_count],
};
// Record operation start
self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
let waiting_guard = self.health.waiting_guard();
let metric_waiting_guard = self.metrics.waiting_guard();
let started = Instant::now();
// Execute the operation
let result = self
.disk
.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
.await;
self.metrics.record_operation_latency("delete_versions", started.elapsed());
self.record_batch_delete_error_metrics(&result);
drop(waiting_guard);
drop(metric_waiting_guard);
let has_err = result.iter().any(|e| e.is_some());
if !has_err {
// Log success and decrement waiting counter
self.health.record_operation_success(&self.endpoint(), "operation_success");
self.metrics.record_mutation_success(DiskMetricMutation::Delete);
}
result
}
pub async fn get_current_disk_id(&self) -> Option<Uuid> {
*self.disk_id.read().await
}
pub(crate) async fn settle_delete_transaction_with_namespace_owner(
&self,
volume: &str,
object: &str,
transaction: uuid::Uuid,
expected_fingerprint: crate::disk::DeleteRecoveryDigest,
action: crate::disk::DeleteRecoveryAction,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<crate::disk::DeleteRecoveryInspection> {
self.track_disk_health_mutation(
"settle_delete_transaction",
DiskMetricMutation::Delete,
|| {
Box::pin(async {
self.disk
.delete_recovery_settle(volume, object, transaction, expected_fingerprint, action, namespace_owner)
.await
})
},
crate::disk::get_delete_recovery_timeout()?,
)
.await
}
/// Track disk health for an operation.
/// This method should wrap disk operations to ensure health checking.
pub async fn track_disk_health<T, F, Fut>(&self, operation: F, timeout_duration: Duration) -> Result<T>
@@ -2272,87 +2195,6 @@ impl LocalDiskWrapper {
#[async_trait::async_trait]
impl DiskAPI for LocalDiskWrapper {
async fn delete_version_recoverable(
&self,
volume: &str,
object: &str,
file_info: FileInfo,
force_del_marker: bool,
mut opts: DeleteOptions,
) -> Result<()> {
crate::disk::validate_delete_recovery_options(&opts)?;
if object.starts_with('/') {
return Err(DiskError::InvalidPath);
}
opts.recoverable_delete = true;
self.delete_version(volume, object, file_info, force_del_marker, opts).await
}
async fn delete_versions_recoverable(
&self,
volume: &str,
versions: Vec<FileInfoVersions>,
mut opts: DeleteOptions,
) -> Vec<Option<DiskError>> {
if let Err(err) = crate::disk::validate_delete_recovery_options(&opts) {
return vec![Some(err); versions.len()];
}
if versions.iter().any(|version| version.name.starts_with('/')) {
return vec![Some(DiskError::InvalidPath); versions.len()];
}
opts.recoverable_delete = true;
self.delete_versions(volume, versions, opts).await
}
async fn discover_delete_transactions(
&self,
volume: &str,
object: Option<&str>,
cursor: Option<crate::disk::DeleteRecoveryCursor>,
limit: u32,
) -> Result<crate::disk::DeleteRecoveryPage> {
self.track_disk_health_with_op(
"discover_delete_transactions",
|| async {
self.disk
.discover_delete_transactions(volume, object, cursor.clone(), limit)
.await
},
get_max_timeout_duration(),
)
.await
}
async fn inspect_delete_transaction(
&self,
volume: &str,
object: &str,
transaction: uuid::Uuid,
) -> Result<crate::disk::DeleteRecoveryInspection> {
self.track_disk_health_with_op(
"inspect_delete_transaction",
|| async { self.disk.inspect_delete_transaction(volume, object, transaction).await },
get_max_timeout_duration(),
)
.await
}
async fn settle_delete_transaction(
&self,
volume: &str,
object: &str,
transaction: uuid::Uuid,
expected_fingerprint: crate::disk::DeleteRecoveryDigest,
action: crate::disk::DeleteRecoveryAction,
) -> Result<crate::disk::DeleteRecoveryInspection> {
self.track_disk_health_mutation(
"settle_delete_transaction",
DiskMetricMutation::Delete,
|| async {
self.disk
.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
.await
},
crate::disk::get_delete_recovery_timeout()?,
)
.await
}
fn has_replacement_mount_lease(&self) -> bool {
self.disk.has_replacement_mount_lease()
}
@@ -2478,15 +2320,6 @@ impl DiskAPI for LocalDiskWrapper {
})
}
async fn bucket_creation_witness(&self, volume: &str) -> Result<Option<super::os::NativeBucketCreationWitness>> {
self.track_disk_health_with_op(
"bucket_creation_witness",
|| async { self.disk.bucket_creation_witness(volume).await },
Duration::ZERO,
)
.await
}
async fn make_volume(&self, volume: &str) -> Result<()> {
// Scoped heal must drain directory creation before releasing its lifecycle owner.
let timeout = if crate::store::bucket_heal_scope(volume).is_some() {
@@ -2575,7 +2408,46 @@ impl DiskAPI for LocalDiskWrapper {
}
async fn delete_versions(&self, volume: &str, versions: Vec<FileInfoVersions>, opts: DeleteOptions) -> Vec<Option<Error>> {
self.delete_versions_with_namespace_owner(volume, versions, opts, None).await
let version_count = versions.len();
self.metrics.record_operation_call("delete_versions");
// Check if disk is faulty before proceeding
if self.health.is_faulty() {
self.metrics.record_availability_error();
return vec![Some(DiskError::FaultyDisk); versions.len()];
}
// Check if disk is stale
if let Err(e) = self.check_disk_stale().await {
self.metrics.record_availability_error();
return vec![Some(e); versions.len()];
}
let _io_permit = match self.acquire_disk_io_permit().await {
Ok(permit) => permit,
Err(error) => return vec![Some(error); version_count],
};
// Record operation start
self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
let waiting_guard = self.health.waiting_guard();
let metric_waiting_guard = self.metrics.waiting_guard();
let started = Instant::now();
// Execute the operation
let result = self.disk.delete_versions(volume, versions, opts).await;
self.metrics.record_operation_latency("delete_versions", started.elapsed());
self.record_batch_delete_error_metrics(&result);
drop(waiting_guard);
drop(metric_waiting_guard);
let has_err = result.iter().any(|e| e.is_some());
if !has_err {
// Log success and decrement waiting counter
self.health.record_operation_success(&self.endpoint(), "operation_success");
self.metrics.record_mutation_success(DiskMetricMutation::Delete);
}
result
}
async fn delete_paths(&self, volume: &str, paths: &[String]) -> Result<()> {
+4 -27
View File
@@ -131,15 +131,6 @@ pub fn build_write_quorum_failure_summary(
fn dominant_error_label(errors: &[Option<Error>], ignored_errs: &[Error], dominant_error: Option<&Error>) -> &'static str {
let Some(dominant_error) = dominant_error else {
// The reducer also returns None when every failure is ignored for
// quorum voting. That case contains no successful nil entry.
if !errors.is_empty() && errors.iter().all(Option::is_some) {
return if errors.iter().all(|error| error.as_ref() == Some(&Error::DiskNotFound)) {
"disk_not_found"
} else {
"all_ignored"
};
}
return "nil_dominated";
};
@@ -396,9 +387,9 @@ mod tests {
}
#[test]
fn test_build_write_quorum_failure_summary_all_missing_is_disk_not_found() {
// Ignoring unavailable disks for quorum reduction must not imply a
// successful nil entry in the diagnostic label.
fn test_build_write_quorum_failure_summary_all_ignored_is_nil_dominated() {
// All disks offline (all DiskNotFound). The dominant error must be None and
// the label must be "nil_dominated" rather than falling back to "other_error".
let errors = vec![
Some(Error::DiskNotFound),
Some(Error::DiskNotFound),
@@ -406,21 +397,7 @@ mod tests {
];
let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
assert_eq!(summary.dominant_error, None);
assert_eq!(summary.achieved, 0);
assert_eq!(summary.dominant_error_label, "disk_not_found");
}
#[test]
fn test_build_write_quorum_failure_summary_all_ignored_has_no_success() {
let errors = [Some(Error::FaultyDisk), Some(Error::DiskNotFound)];
let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
assert_eq!(summary.dominant_error, None);
assert_eq!(summary.achieved, 0);
assert_eq!(summary.dominant_error_label, "all_ignored");
assert_eq!(
reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 2),
Some(Error::ErasureWriteQuorum)
);
assert_eq!(summary.dominant_error_label, "nil_dominated");
}
#[test]
-7
View File
@@ -117,9 +117,6 @@ pub const O_APPEND: FileMode = 0x00400;
// create_new: bool,
pub async fn open_file(path: impl AsRef<Path>, mode: FileMode) -> io::Result<File> {
if mode & O_CREATE != 0 {
super::os::record_native_publication(path.as_ref());
}
let base_opts = match mode & (O_RDONLY | O_WRONLY | O_RDWR) {
O_RDONLY => get_readonly_options(),
O_WRONLY => get_writeonly_options(),
@@ -163,7 +160,6 @@ pub fn lstat_std(path: impl AsRef<Path>) -> io::Result<Metadata> {
}
pub async fn make_dir_all(path: impl AsRef<Path>) -> io::Result<()> {
super::os::record_native_publication(path.as_ref());
fs::create_dir_all(path.as_ref()).await
}
@@ -215,18 +211,15 @@ pub fn remove_all_std(path: impl AsRef<Path>) -> io::Result<()> {
}
pub async fn mkdir(path: impl AsRef<Path>) -> io::Result<()> {
super::os::record_native_publication(path.as_ref());
fs::create_dir(path.as_ref()).await
}
pub async fn rename(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()> {
super::os::record_native_publication(to.as_ref());
fs::rename(from, to).await
}
#[cfg(any(not(windows), test))]
pub fn rename_std(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()> {
super::os::record_native_publication(to.as_ref());
std::fs::rename(from, to)
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
-708
View File
@@ -230,105 +230,6 @@ pub enum Disk {
}
impl Disk {
pub(crate) async fn delete_versions_with_namespace_owner(
&self,
volume: &str,
versions: Vec<FileInfoVersions>,
opts: DeleteOptions,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Vec<Option<Error>> {
match self {
Self::Local(disk) => {
disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
.await
}
Self::Remote(disk) => {
let _owner = namespace_owner;
disk.delete_versions(volume, versions, opts).await
}
}
}
pub(crate) async fn delete_versions_recoverable_with_namespace_owner(
&self,
volume: &str,
versions: Vec<FileInfoVersions>,
mut opts: DeleteOptions,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Vec<Option<Error>> {
if let Err(error) = validate_delete_recovery_options(&opts) {
return vec![Some(error); versions.len()];
}
if versions.iter().any(|version| version.name.starts_with('/')) {
return vec![Some(DiskError::InvalidPath); versions.len()];
}
opts.recoverable_delete = true;
match self {
Self::Local(disk) => {
disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
.await
}
Self::Remote(disk) => {
let _owner = namespace_owner;
disk.delete_versions_recoverable(volume, versions, opts).await
}
}
}
pub(crate) async fn settle_delete_transaction_with_namespace_owner(
&self,
volume: &str,
object: &str,
transaction: Uuid,
expected_fingerprint: DeleteRecoveryDigest,
action: DeleteRecoveryAction,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<DeleteRecoveryInspection> {
match self {
Self::Local(disk) => {
disk.settle_delete_transaction_with_namespace_owner(
volume,
object,
transaction,
expected_fingerprint,
action,
namespace_owner,
)
.await
}
Self::Remote(disk) => {
let _owner = namespace_owner;
disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
.await
}
}
}
pub(crate) async fn delete_version_recoverable_with_namespace_owner(
&self,
volume: &str,
path: &str,
fi: FileInfo,
force_del_marker: bool,
mut opts: DeleteOptions,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<()> {
validate_delete_recovery_options(&opts)?;
if path.starts_with('/') {
return Err(DiskError::InvalidPath);
}
opts.recoverable_delete = true;
match self {
Self::Local(disk) => {
disk.delete_version_with_namespace_owner(volume, path, fi, force_del_marker, opts, namespace_owner)
.await
}
Self::Remote(disk) => {
disk.delete_version_recoverable(volume, path, fi, force_del_marker, opts)
.await
}
}
}
pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) -> Result<()> {
match self {
Disk::Local(local_disk) => {
@@ -356,70 +257,6 @@ impl Disk {
#[async_trait::async_trait]
impl DiskAPI for Disk {
async fn delete_version_recoverable(
&self,
volume: &str,
object: &str,
file_info: FileInfo,
force_del_marker: bool,
opts: DeleteOptions,
) -> Result<()> {
self.delete_version_recoverable_with_namespace_owner(volume, object, file_info, force_del_marker, opts, None)
.await
}
async fn delete_versions_recoverable(
&self,
volume: &str,
versions: Vec<FileInfoVersions>,
opts: DeleteOptions,
) -> Vec<Option<DiskError>> {
match self {
Self::Local(disk) => disk.delete_versions_recoverable(volume, versions, opts).await,
Self::Remote(disk) => disk.delete_versions_recoverable(volume, versions, opts).await,
}
}
async fn discover_delete_transactions(
&self,
volume: &str,
object: Option<&str>,
cursor: Option<DeleteRecoveryCursor>,
limit: u32,
) -> Result<DeleteRecoveryPage> {
match self {
Self::Local(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await,
Self::Remote(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await,
}
}
async fn inspect_delete_transaction(
&self,
volume: &str,
object: &str,
transaction: Uuid,
) -> Result<DeleteRecoveryInspection> {
match self {
Self::Local(disk) => disk.inspect_delete_transaction(volume, object, transaction).await,
Self::Remote(disk) => disk.inspect_delete_transaction(volume, object, transaction).await,
}
}
async fn settle_delete_transaction(
&self,
volume: &str,
object: &str,
transaction: Uuid,
expected_fingerprint: DeleteRecoveryDigest,
action: DeleteRecoveryAction,
) -> Result<DeleteRecoveryInspection> {
match self {
Self::Local(disk) => {
disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
.await
}
Self::Remote(disk) => {
disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
.await
}
}
}
fn to_string(&self) -> String {
match self {
Disk::Local(local_disk) => local_disk.to_string(),
@@ -490,13 +327,6 @@ impl DiskAPI for Disk {
}
}
async fn bucket_creation_witness(&self, volume: &str) -> Result<Option<os::NativeBucketCreationWitness>> {
match self {
Disk::Local(local_disk) => local_disk.bucket_creation_witness(volume).await,
Disk::Remote(_) => Ok(None),
}
}
#[tracing::instrument(level = "trace", skip_all)]
async fn make_volume(&self, volume: &str) -> Result<()> {
match self {
@@ -1291,11 +1121,6 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
// NSScanner
// Volume operations.
/// Unknown or remote backends cannot attest process-local publication history.
async fn bucket_creation_witness(&self, _volume: &str) -> Result<Option<os::NativeBucketCreationWitness>> {
Ok(None)
}
async fn make_volume(&self, volume: &str) -> Result<()>;
async fn make_volumes(&self, volume: Vec<&str>) -> Result<()>;
async fn list_volumes(&self) -> Result<Vec<VolumeInfo>>;
@@ -1421,51 +1246,6 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
Err(DiskError::MethodNotAllowed)
}
async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()>;
async fn discover_delete_transactions(
&self,
_volume: &str,
_object: Option<&str>,
_cursor: Option<DeleteRecoveryCursor>,
_limit: u32,
) -> Result<DeleteRecoveryPage> {
Err(DiskError::MethodNotAllowed)
}
async fn delete_version_recoverable(
&self,
_volume: &str,
_object: &str,
_file_info: FileInfo,
_force_del_marker: bool,
_opts: DeleteOptions,
) -> Result<()> {
Err(DiskError::MethodNotAllowed)
}
async fn delete_versions_recoverable(
&self,
_volume: &str,
versions: Vec<FileInfoVersions>,
_opts: DeleteOptions,
) -> Vec<Option<DiskError>> {
vec![Some(DiskError::MethodNotAllowed); versions.len()]
}
async fn inspect_delete_transaction(
&self,
_volume: &str,
_object: &str,
_transaction: Uuid,
) -> Result<DeleteRecoveryInspection> {
Err(DiskError::MethodNotAllowed)
}
async fn settle_delete_transaction(
&self,
_volume: &str,
_object: &str,
_transaction: Uuid,
_expected_fingerprint: DeleteRecoveryDigest,
_action: DeleteRecoveryAction,
) -> Result<DeleteRecoveryInspection> {
Err(DiskError::MethodNotAllowed)
}
// VerifyFile
async fn verify_file(&self, volume: &str, path: &str, fi: &FileInfo) -> Result<CheckPartsResp>;
// CheckParts
@@ -1802,272 +1582,6 @@ pub struct DeleteOptions {
/// Remote calls carrying it must use DeleteRetiredMarker, never DeleteVersion.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub delete_recovery_bucket_incarnation: Option<Uuid>,
#[serde(default)]
pub recoverable_delete: bool,
}
pub const DELETE_RECOVERY_PROTOCOL_VERSION: u32 = 1;
pub(crate) fn get_delete_recovery_timeout() -> Result<std::time::Duration> {
let value = std::env::var("RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS");
match value {
Ok(value) => parse_delete_recovery_timeout(Some(&value)),
Err(std::env::VarError::NotPresent) => parse_delete_recovery_timeout(None),
Err(std::env::VarError::NotUnicode(_)) => Err(DiskError::other("invalid delete recovery timeout configuration")),
}
}
fn parse_delete_recovery_timeout(value: Option<&str>) -> Result<std::time::Duration> {
let seconds = match value {
None => 900,
Some(value) => value
.parse::<u64>()
.ok()
.filter(|seconds| (30..=86400).contains(seconds))
.ok_or_else(|| DiskError::other("delete recovery timeout must be an integer between 30 and 86400 seconds"))?,
};
Ok(std::time::Duration::from_secs(seconds))
}
pub(crate) fn validate_delete_recovery_options(opts: &DeleteOptions) -> Result<()> {
if opts.old_data_dir.is_none_or(|id| id.is_nil())
|| opts.undo_write
|| opts.undo_delete
|| opts.recursive
|| opts.immediate
|| opts.expected_delete_marker.is_some()
|| opts.delete_recovery_bucket_incarnation.is_some_and(|id| id.is_nil())
{
return Err(DiskError::FileCorrupt);
}
Ok(())
}
pub type DeleteRecoveryDigest = [u8; 32];
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DeleteTransactionId {
pub object: String,
pub transaction: Uuid,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DeleteRecoveryCursor {
pub token: Uuid,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DeleteRecoveryPage {
pub transactions: Vec<DeleteTransactionId>,
pub next: Option<DeleteRecoveryCursor>,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DeleteRecoveryJournal {
pub protocol_version: u32,
pub volume: String,
pub object: String,
pub transaction: Uuid,
pub bucket_incarnation: Option<Uuid>,
pub original_digest: Option<DeleteRecoveryDigest>,
pub target_digest: Option<DeleteRecoveryDigest>,
pub data_dirs: Vec<Uuid>,
pub deleted_versions: Vec<Uuid>,
pub requests: Vec<FileInfo>,
pub original_versions: Vec<DeleteRecoveryVersionIdentity>,
#[serde(default)]
pub predecessors: Vec<Uuid>,
}
impl DeleteRecoveryJournal {
/// Checks source generations only; the coordinator must supply a certified
/// decision and the receiver must retain its namespace/incarnation fences.
pub(crate) fn matches_adoption_source(&self, current: &[DeleteRecoveryVersionIdentity]) -> bool {
if self.requests.is_empty() {
return false;
}
let mut originals = std::collections::HashMap::with_capacity(self.original_versions.len());
for version in &self.original_versions {
originals.entry(version.version_id.unwrap_or_default()).or_insert(version);
}
let mut locals = std::collections::HashMap::with_capacity(current.len());
for version in current {
locals.entry(version.version_id.unwrap_or_default()).or_insert(version);
}
self.requests.iter().all(|request| {
if request.deleted && request.mark_deleted {
return current.first() == self.original_versions.first();
}
let id = request.version_id.unwrap_or_default();
let Some(original) = originals.get(&id) else {
return !id.is_nil() && !locals.contains_key(&id);
};
match locals.get(&id) {
Some(local) => local == original,
None => !id.is_nil(),
}
})
}
/// Matches immutable local preparation intent against an independently
/// inspected same-transaction commit receipt. This match is not authority
/// by itself; the coordinator must supply the certified receipt.
pub(crate) fn matches_prepared_authority(&self, authority: &Self) -> bool {
if self.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
|| authority.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
|| self.transaction.is_nil()
|| self.transaction != authority.transaction
|| self.volume != authority.volume
|| self.object != authority.object
|| self.bucket_incarnation != authority.bucket_incarnation
|| self.bucket_incarnation.is_some_and(|id| id.is_nil())
|| self.requests.is_empty()
|| self.requests != authority.requests
|| self.deleted_versions.iter().collect::<std::collections::HashSet<_>>()
!= authority.deleted_versions.iter().collect::<std::collections::HashSet<_>>()
{
return false;
}
let sources: std::collections::HashMap<_, _> = self
.original_versions
.iter()
.map(|version| (version.version_id.unwrap_or_default(), version))
.collect();
let certified: std::collections::HashMap<_, _> = authority
.original_versions
.iter()
.map(|version| (version.version_id.unwrap_or_default(), version))
.collect();
if sources.len() != self.original_versions.len() || certified.len() != authority.original_versions.len() {
return false;
}
self.requests.iter().all(|request| {
let (local, remote) = if request.deleted && request.mark_deleted {
(self.original_versions.first(), authority.original_versions.first())
} else {
let id = request.version_id.unwrap_or_default();
(sources.get(&id).copied(), certified.get(&id).copied())
};
match (local, remote) {
(Some(local), Some(remote)) => local == remote,
(None, None) => {
(!(request.deleted && request.mark_deleted) && request.version_id.is_some_and(|id| !id.is_nil()))
|| (self.original_digest.is_none() && authority.original_digest.is_none())
}
_ => false,
}
})
}
}
#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DeleteRecoveryVersionIdentity {
pub version_id: Option<Uuid>,
pub mod_time: Option<OffsetDateTime>,
pub data_dir: Option<Uuid>,
pub deleted: bool,
pub metadata_digest: DeleteRecoveryDigest,
}
impl DeleteRecoveryVersionIdentity {
pub(crate) fn from_file_info(info: FileInfo) -> error::Result<Self> {
local::delete_recovery::generation(info)
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum DeleteRecoveryState {
Absent,
NewPrepared,
QuorumCommitted,
CommitFinished,
LegacyAmbiguous,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DeleteRecoveryInspection {
pub protocol_version: u32,
pub id: DeleteTransactionId,
pub state: DeleteRecoveryState,
pub journal: Option<DeleteRecoveryJournal>,
pub current_digest: Option<DeleteRecoveryDigest>,
pub current_versions: Vec<DeleteRecoveryVersionIdentity>,
pub backup_digest: Option<DeleteRecoveryDigest>,
pub rollback_data_complete: bool,
pub commit_data_safe: bool,
pub explicit_delete_safe: bool,
pub fingerprint: DeleteRecoveryDigest,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub enum DeleteRecoveryAction {
PublishCommit,
FinishCommit,
FinalizeCommit,
Rollback,
DiscardLegacy,
AdoptCommit { authority: Box<DeleteRecoveryJournal> },
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub enum DeleteRecoveryRpcOperation {
Discover {
object: Option<String>,
cursor: Option<DeleteRecoveryCursor>,
limit: u32,
},
Inspect {
object: String,
transaction: Uuid,
},
Settle {
object: String,
transaction: Uuid,
expected_fingerprint: DeleteRecoveryDigest,
action: DeleteRecoveryAction,
},
ApplyVersion {
object: String,
file_info: Box<FileInfo>,
force_del_marker: bool,
opts: DeleteOptions,
},
ApplyVersions {
versions: Vec<FileInfoVersions>,
opts: DeleteOptions,
},
}
impl DeleteRecoveryRpcOperation {
/// Full metadata application and adoption share the existing mutation envelope.
pub fn max_wire_payload_bytes(&self) -> usize {
match self {
Self::ApplyVersion { .. }
| Self::ApplyVersions { .. }
| Self::Settle {
action: DeleteRecoveryAction::AdoptCommit { .. },
..
} => rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES,
_ => rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES,
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub enum DeleteRecoveryRpcReply {
Discovered(DeleteRecoveryPage),
Inspected(DeleteRecoveryInspection),
Settled(DeleteRecoveryInspection),
AppliedVersion,
AppliedVersions(Vec<Option<u32>>),
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -2358,8 +1872,6 @@ mod tests {
undo_delete: false,
old_data_dir: Some(Uuid::new_v4()),
expected_delete_marker: None,
delete_recovery_bucket_incarnation: None,
recoverable_delete: false,
};
assert!(opts.recursive);
@@ -2784,223 +2296,3 @@ mod tests {
}
}
}
#[cfg(test)]
mod delete_recovery_timeout_tests {
use super::*;
#[test]
fn recovery_timeout_defaults_and_validates_range() {
assert_eq!(
parse_delete_recovery_timeout(None).expect("default timeout"),
std::time::Duration::from_secs(900)
);
for seconds in ["30", "900", "86400"] {
assert!(parse_delete_recovery_timeout(Some(seconds)).is_ok());
}
for value in ["", "invalid", "0", "2", "29", "86401", "-1", "18446744073709551616"] {
assert!(parse_delete_recovery_timeout(Some(value)).is_err());
}
}
}
#[cfg(test)]
mod delete_recovery_wire_limit_tests {
use super::*;
#[test]
fn boxed_apply_version_preserves_existing_wire_payload() {
#[derive(Serialize)]
enum PriorOperation<'a> {
ApplyVersion {
object: &'a str,
file_info: &'a FileInfo,
force_del_marker: bool,
opts: &'a DeleteOptions,
},
}
let file_info = FileInfo {
name: "object".into(),
version_id: Some(Uuid::new_v4()),
metadata: std::collections::HashMap::from([("metadata".into(), "value".into())]),
..Default::default()
};
let opts = DeleteOptions {
old_data_dir: Some(Uuid::new_v4()),
recoverable_delete: true,
delete_recovery_bucket_incarnation: Some(Uuid::new_v4()),
..Default::default()
};
let prior = PriorOperation::ApplyVersion {
object: "object",
file_info: &file_info,
force_del_marker: true,
opts: &opts,
};
let encoded = rmp_serde::to_vec_named(&prior).expect("encode the previous unboxed request shape");
let operation = DeleteRecoveryRpcOperation::ApplyVersion {
object: "object".into(),
file_info: Box::new(file_info.clone()),
force_del_marker: true,
opts: opts.clone(),
};
assert_eq!(rmp_serde::to_vec_named(&operation).expect("encode boxed request"), encoded);
assert_eq!(
serde_json::to_value(&operation).expect("boxed request JSON"),
serde_json::to_value(prior).expect("previous request JSON")
);
let decoded: DeleteRecoveryRpcOperation =
rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode the previous payload");
assert_eq!(rmp_serde::to_vec_named(&decoded).expect("re-encode decoded prior request"), encoded);
}
#[test]
fn adoption_and_inspection_reply_support_large_version_journals() {
let transaction = Uuid::new_v4();
let request = FileInfo {
metadata: std::collections::HashMap::from([("metadata".into(), "x".repeat(2048))]),
..Default::default()
};
let journal = DeleteRecoveryJournal {
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
volume: "bucket".into(),
object: "object".into(),
transaction,
bucket_incarnation: None,
original_digest: None,
target_digest: None,
data_dirs: Vec::new(),
deleted_versions: Vec::new(),
requests: vec![request; 1000],
original_versions: Vec::new(),
predecessors: Vec::new(),
};
let operation = DeleteRecoveryRpcOperation::Settle {
object: "object".into(),
transaction,
expected_fingerprint: [0; 32],
action: DeleteRecoveryAction::AdoptCommit {
authority: Box::new(journal.clone()),
},
};
let encoded = rmp_serde::to_vec_named(&operation).expect("encode large adoption");
assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
assert!(encoded.len() < operation.max_wire_payload_bytes());
let decoded: DeleteRecoveryRpcOperation =
rustfs_protos::decode_delete_recovery_payload(&encoded).expect("large adoption can decode");
assert_eq!(decoded.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES);
let reply = DeleteRecoveryRpcReply::Inspected(DeleteRecoveryInspection {
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
id: DeleteTransactionId {
object: "object".into(),
transaction,
},
state: DeleteRecoveryState::QuorumCommitted,
journal: Some(journal),
current_digest: None,
current_versions: Vec::new(),
backup_digest: None,
rollback_data_complete: false,
commit_data_safe: true,
explicit_delete_safe: false,
fingerprint: [0; 32],
});
let encoded = rmp_serde::to_vec_named(&reply).expect("encode large inspection");
assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
assert!(rustfs_protos::decode_delete_recovery_payload::<DeleteRecoveryRpcReply>(&encoded).is_ok());
let inspect = DeleteRecoveryRpcOperation::Inspect {
object: "object".into(),
transaction,
};
assert_eq!(inspect.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
}
}
#[cfg(test)]
mod delete_recovery_adoption_tests {
use super::*;
#[test]
fn source_matching_preserves_non_null_absence_and_rejects_foreign_or_missing_null() {
let id = Uuid::new_v4();
let source = DeleteRecoveryVersionIdentity {
version_id: Some(id),
mod_time: None,
data_dir: None,
deleted: false,
metadata_digest: [1; 32],
};
let mut journal = DeleteRecoveryJournal {
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
volume: "bucket".into(),
object: "object".into(),
transaction: Uuid::new_v4(),
bucket_incarnation: None,
original_digest: None,
target_digest: None,
data_dirs: Vec::new(),
deleted_versions: vec![id],
requests: vec![FileInfo {
version_id: Some(id),
..Default::default()
}],
original_versions: vec![source.clone()],
predecessors: Vec::new(),
};
assert!(journal.matches_adoption_source(&[]), "certified non-null removal can already be absent");
assert!(journal.matches_adoption_source(std::slice::from_ref(&source)));
let missing = Uuid::new_v4();
journal.requests.push(FileInfo {
version_id: Some(missing),
..Default::default()
});
journal.original_digest = Some([3; 32]);
assert!(
journal.matches_prepared_authority(&journal),
"an explicit UUID absent from both prepared sources remains a no-op with original metadata"
);
assert!(journal.matches_adoption_source(&[]), "both non-null versions can remain absent");
assert!(
journal.matches_adoption_source(std::slice::from_ref(&source)),
"a missing non-null version in the batch remains a no-op"
);
let mut unexpected = source.clone();
unexpected.version_id = Some(missing);
let mut foreign_preparation = journal.clone();
foreign_preparation.original_versions.push(unexpected.clone());
assert!(
!journal.matches_prepared_authority(&foreign_preparation)
&& !foreign_preparation.matches_prepared_authority(&journal),
"absence from only one prepared source cannot establish a generation"
);
assert!(
!journal.matches_adoption_source(&[source.clone(), unexpected]),
"a version absent from the authority cannot be deleted locally"
);
for version_id in [None, Some(Uuid::nil())] {
journal.requests.last_mut().expect("missing-version request").version_id = version_id;
assert!(
!journal.matches_prepared_authority(&journal),
"a missing null version with original metadata cannot establish its generation"
);
assert!(
!journal.matches_adoption_source(std::slice::from_ref(&source)),
"a missing null version cannot establish its generation"
);
}
journal.requests.pop();
let mut foreign = source.clone();
foreign.metadata_digest = [2; 32];
assert!(!journal.matches_adoption_source(std::slice::from_ref(&foreign)));
journal.original_versions.push(foreign.clone());
assert!(
journal.matches_adoption_source(&[source.clone(), foreign.clone()]),
"retain original first-match semantics"
);
assert!(!journal.matches_adoption_source(&[foreign, source]));
journal.requests[0].version_id = None;
journal.original_versions[0].version_id = None;
assert!(!journal.matches_adoption_source(&[]), "null absence does not establish its generation");
assert!(journal.matches_adoption_source(std::slice::from_ref(&journal.original_versions[0])));
journal.requests.clear();
assert!(!journal.matches_adoption_source(&journal.original_versions));
}
}
+2 -587
View File
@@ -21,7 +21,7 @@ use futures::TryStreamExt;
use parking_lot::Mutex;
use rustfs_utils::path::SLASH_SEPARATOR;
use std::{
collections::{HashMap, HashSet, VecDeque},
collections::{HashMap, VecDeque},
io,
path::{Component, Path, PathBuf},
sync::{Arc, LazyLock, Weak},
@@ -276,14 +276,6 @@ pub(crate) mod prepared_publication_test_hooks {
Rollback,
#[cfg(test)]
DirFsync,
#[cfg(test)]
CreateDirectory,
#[cfg(test)]
NativeRootIdentity,
#[cfg(all(test, target_os = "linux"))]
DirectWriteOpen,
#[cfg(all(test, target_os = "linux"))]
BeforeDataUsageLayout,
}
type Hook = Box<dyn FnOnce() + Send>;
@@ -1389,226 +1381,6 @@ fn fsync_spawn_blocking<T: Send + 'static>(f: impl FnOnce() -> T + Send + 'stati
}
static DISK_VOLUME_MUTATION_LOCKS: LazyLock<Mutex<HashMap<PathBuf, Weak<RwLock<()>>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
#[cfg(windows)]
type NativeRootIdentity = (u64, [u8; 16]);
#[cfg(not(windows))]
type NativeRootIdentity = (u64, u64);
// Publication history outlives disk handles, cleanup, and metadata snapshots.
// Reusing a pathname or descriptor for another physical root is not new proof.
#[derive(Debug, Default)]
struct NativePublicationHistory {
roots: HashMap<NativeRootIdentity, NativeRootHistory>,
aliases: HashMap<PathBuf, HashSet<NativeRootIdentity>>,
}
#[derive(Debug, Default)]
struct NativeRootHistory {
unknown: bool,
started: HashSet<String>,
}
static NATIVE_PUBLICATION_HISTORY: LazyLock<Mutex<NativePublicationHistory>> =
LazyLock::new(|| Mutex::new(NativePublicationHistory::default()));
#[derive(Clone, Debug)]
pub(crate) struct NativePublicationRegistration {
identity: NativeRootIdentity,
root: PathBuf,
io_root: PathBuf,
}
/// Opaque, process-local observation. Unsupported disks cannot construct it.
#[derive(Clone, Debug)]
pub struct NativeBucketCreationWitness {
registration: NativePublicationRegistration,
backend: Arc<dyn super::local::LocalIoBackend>,
bucket: String,
disk_id: uuid::Uuid,
}
fn native_root_identity(path: &Path) -> Option<NativeRootIdentity> {
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
#[cfg(test)]
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::NativeRootIdentity, path);
let metadata = std::fs::metadata(path).ok()?;
metadata.is_dir().then(|| (metadata.dev(), metadata.ino()))
}
#[cfg(windows)]
{
// Resolve and validate the same directory anchor used for publication;
// keep its volume and complete 128-bit file ID in the history key.
let (_, directory) = open_windows_publication_root(path).ok()?;
windows_file_identity(directory.handle.as_ref()).ok()
}
#[cfg(all(not(unix), not(windows)))]
{
let _ = path;
None
}
}
pub(crate) fn register_native_publication_root(root: &Path, io_root: &Path) -> Option<NativePublicationRegistration> {
let identity = native_root_identity(root)?;
if native_root_identity(io_root) != Some(identity) {
return None;
}
let mut history = NATIVE_PUBLICATION_HISTORY.lock();
history.roots.entry(identity).or_default();
for alias in [root, io_root] {
let alias: PathBuf = alias.components().collect();
let previous = history.aliases.entry(alias).or_default();
let collisions: Vec<_> = previous.iter().copied().filter(|old| *old != identity).collect();
previous.insert(identity);
if !collisions.is_empty() {
history.roots.get_mut(&identity)?.unknown = true;
for old in collisions {
history.roots.get_mut(&old)?.unknown = true;
}
}
}
Some(NativePublicationRegistration {
identity,
root: root.to_path_buf(),
io_root: io_root.to_path_buf(),
})
}
/// Record before queueing filesystem work. Cancellation never removes Started.
pub(crate) fn record_native_publication(path: &Path) {
let aliases: Vec<_> = {
let history = NATIVE_PUBLICATION_HISTORY.lock();
path.ancestors()
.filter(|alias| history.aliases.contains_key(*alias))
.map(Path::to_path_buf)
.collect()
};
// Physical lookups must never hold the process-wide publication mutex.
let observations: Vec<_> = aliases
.into_iter()
.filter_map(|alias| {
let relative = path.strip_prefix(&alias).ok()?;
let mut components = relative.components().filter(|part| !matches!(part, Component::CurDir));
let first = components.next()?;
let bucket = match first {
Component::Normal(bucket) if components.all(|part| matches!(part, Component::Normal(_))) => {
bucket.to_str().map(str::to_owned)
}
_ => None,
};
let physical = native_root_identity(&alias);
Some((alias, physical, bucket))
})
.collect();
let mut history = NATIVE_PUBLICATION_HISTORY.lock();
for (alias, physical, bucket) in observations {
// Aliases only accumulate identities. Include registration concurrent
// with the stat, rather than using a stale captured identity list.
let identities = history.aliases.get(&alias).cloned().unwrap_or_default();
if physical.is_none() || identities.iter().any(|identity| Some(*identity) != physical) {
for identity in identities {
history.roots.entry(identity).or_default().unknown = true;
}
if let Some(identity) = physical {
history.roots.entry(identity).or_default().unknown = true;
}
} else if let Some(identity) = physical {
let root = history.roots.entry(identity).or_default();
match bucket {
Some(bucket) => {
root.started.insert(bucket);
}
None => root.unknown = true,
}
}
}
}
impl NativePublicationRegistration {
pub(crate) fn invalidate(&self) {
if let Some(root) = NATIVE_PUBLICATION_HISTORY.lock().roots.get_mut(&self.identity) {
root.unknown = true;
}
}
pub(crate) fn witness(
&self,
bucket: &str,
disk_id: uuid::Uuid,
backend: Arc<dyn super::local::LocalIoBackend>,
) -> Option<NativeBucketCreationWitness> {
if disk_id.is_nil()
|| !matches!(Path::new(bucket).components().next(), Some(Component::Normal(_)))
|| Path::new(bucket).components().count() != 1
{
return None;
}
let witness = NativeBucketCreationWitness {
registration: self.clone(),
backend,
bucket: bucket.to_owned(),
disk_id,
};
validate_native_creation_witnesses(std::slice::from_ref(&witness)).then_some(witness)
}
}
impl NativeBucketCreationWitness {
pub(crate) fn matches(&self, other: &Self) -> bool {
self.registration.identity == other.registration.identity
&& Arc::ptr_eq(&self.backend, &other.backend)
&& self.disk_id == other.disk_id
&& self.bucket == other.bucket
}
fn physical_root_is_current(&self) -> bool {
let root = native_root_identity(&self.registration.root);
let io_root = native_root_identity(&self.registration.io_root);
if root == Some(self.registration.identity) && io_root == Some(self.registration.identity) {
return true;
}
let mut history = NATIVE_PUBLICATION_HISTORY.lock();
history.roots.entry(self.registration.identity).or_default().unknown = true;
for identity in [root, io_root].into_iter().flatten() {
history.roots.entry(identity).or_default().unknown = true;
}
false
}
fn valid_in(&self, history: &NativePublicationHistory) -> bool {
history
.roots
.get(&self.registration.identity)
.is_some_and(|root| !root.unknown && !root.started.contains(&self.bucket))
}
}
pub(crate) fn validate_native_creation_witnesses(witnesses: &[NativeBucketCreationWitness]) -> bool {
if witnesses.is_empty() || !witnesses.iter().all(NativeBucketCreationWitness::physical_root_is_current) {
return false;
}
let history = NATIVE_PUBLICATION_HISTORY.lock();
witnesses.iter().all(|witness| witness.valid_in(&history))
}
pub(crate) fn consume_native_creation_witnesses(witnesses: &[NativeBucketCreationWitness]) -> bool {
if witnesses.is_empty() || !witnesses.iter().all(NativeBucketCreationWitness::physical_root_is_current) {
return false;
}
let mut history = NATIVE_PUBLICATION_HISTORY.lock();
if !witnesses.iter().all(|witness| witness.valid_in(&history)) {
return false;
}
for witness in witnesses {
if let Some(root) = history.roots.get_mut(&witness.registration.identity) {
root.started.insert(witness.bucket.clone());
}
}
true
}
type NamespaceMutationLock = AsyncMutex<()>;
type NamespaceMutationLockRegistry = HashMap<PathBuf, Weak<NamespaceMutationLock>>;
static DISK_NAMESPACE_MUTATION_LOCKS: LazyLock<Mutex<NamespaceMutationLockRegistry>> =
@@ -2311,7 +2083,6 @@ pub(in crate::disk) async fn rename_with_namespace_owner(
dst: &Path,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> io::Result<()> {
record_native_publication(dst);
if namespace_owner.is_none() {
return tokio::fs::rename(src, dst).await;
}
@@ -2333,19 +2104,12 @@ pub(in crate::disk) async fn create_dir_all_with_namespace_owner(
path: &Path,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> io::Result<()> {
record_native_publication(path);
if namespace_owner.is_none() {
return tokio::fs::create_dir_all(path).await;
}
let path = path.to_path_buf();
let lease = acquire_namespace_mutation_lease_with_owner(&path, namespace_owner).await;
run_blocking_namespace_operation(lease, move || {
#[cfg(all(test, not(windows)))]
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::CreateDirectory, &path);
record_native_publication(&path);
std::fs::create_dir_all(path)
})
.await
run_blocking_namespace_operation(lease, move || std::fs::create_dir_all(path)).await
}
#[tracing::instrument(name = "rename_all", level = "debug", skip_all)]
@@ -2356,7 +2120,6 @@ pub(crate) async fn rename_all_with_owner(
publication_root: &PublicationRoot,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<()> {
record_native_publication(dst_file_path.as_ref());
let lease = acquire_namespace_mutation_lease_with_owner(dst_file_path.as_ref(), namespace_owner).await;
rename_all_with_lease(src_file_path, dst_file_path, base_dir, publication_root, lease).await
}
@@ -2368,7 +2131,6 @@ pub(crate) async fn rename_all_with_lease(
publication_root: &PublicationRoot,
lease: Arc<NamespaceMutationLease>,
) -> Result<()> {
record_native_publication(dst_file_path.as_ref());
reliable_rename_inner_with_lease(
src_file_path.as_ref().to_path_buf(),
dst_file_path.as_ref().to_path_buf(),
@@ -2393,7 +2155,6 @@ pub(crate) async fn rename_reconciled_metadata(
owner: Option<Arc<dyn Send + Sync>>,
authority: Arc<crate::bucket::lifecycle::legacy_transition_state_reconcile::TransitionStateReconcileAuthority>,
) -> Result<()> {
record_native_publication(&destination);
let lease = acquire_namespace_mutation_lease_with_owner(&destination, owner).await;
run_blocking_namespace_operation(lease, move || {
let preparation = prepare_rename_with_retry(&source, &destination, &base_dir, &publication_root)?;
@@ -2638,7 +2399,6 @@ pub(crate) async fn rename_all_ignore_missing_source_with_owner(
publication_root: &PublicationRoot,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<()> {
record_native_publication(dst_file_path.as_ref());
let src_file_path = src_file_path.as_ref();
let lease = acquire_namespace_mutation_lease_with_owner(dst_file_path.as_ref(), namespace_owner).await;
match reliable_rename_inner_with_lease(
@@ -2693,7 +2453,6 @@ async fn reliable_rename_inner(
publication_root: &PublicationRoot,
warn_on_missing_source: bool,
) -> io::Result<()> {
record_native_publication(dst_file_path.as_ref());
let src_file_path = src_file_path.as_ref().to_path_buf();
let dst_file_path = dst_file_path.as_ref().to_path_buf();
let base_dir = base_dir.as_ref().to_path_buf();
@@ -2717,7 +2476,6 @@ async fn reliable_rename_inner_with_lease(
warn_on_missing_source: bool,
lease: Arc<NamespaceMutationLease>,
) -> io::Result<()> {
record_native_publication(&dst_file_path);
let operation = {
let src_file_path = src_file_path.clone();
let dst_file_path = dst_file_path.clone();
@@ -3120,7 +2878,6 @@ fn rename_into_existing_parent(
dst_file_path: &Path,
parent_guard: Option<&ExistingBaseDirectoryGuard>,
) -> io::Result<()> {
record_native_publication(dst_file_path);
use rustix::fs::{Mode, OFlags, open, renameat};
let Some(parent_guard) = parent_guard else {
@@ -3169,7 +2926,6 @@ fn rename_into_existing_parent(
parent_guard: Option<&ExistingBaseDirectoryGuard>,
source: &winapi_util::Handle,
) -> io::Result<()> {
record_native_publication(dst_file_path);
use std::{
mem::size_of,
os::windows::{ffi::OsStrExt, io::AsRawHandle},
@@ -4458,7 +4214,6 @@ pub(crate) fn mkdir_all_below_existing_base_std(
base_dir: &Path,
publication_root: &PublicationRoot,
) -> io::Result<ExistingBaseDirectoryGuard> {
record_native_publication(dir_path);
let relative = dir_path
.strip_prefix(base_dir)
.map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "rename destination must remain below its base directory"))?;
@@ -5963,77 +5718,6 @@ mod tests {
assert_eq!(err.raw_os_error(), Some(disk_full));
}
#[cfg(windows)]
#[test]
fn windows_native_root_identity_matches_directory_handles() {
let temp = tempdir().expect("create native identity fixture");
let first = temp.path().join("first");
let second = temp.path().join("second");
std::fs::create_dir(&first).expect("create first root");
std::fs::create_dir(&second).expect("create second root");
let (_, first_anchor) = open_windows_publication_root(&first).expect("open first directory anchor");
let (_, second_anchor) = open_windows_publication_root(&second).expect("open second directory anchor");
let first_identity = windows_file_identity(first_anchor.handle.as_ref()).expect("query full first identity");
let second_identity = windows_file_identity(second_anchor.handle.as_ref()).expect("query full second identity");
assert_eq!(native_root_identity(&first), Some(first_identity));
assert_eq!(native_root_identity(&first.join(".")), Some(first_identity));
assert_eq!(native_root_identity(&second), Some(second_identity));
assert_ne!(first_identity, second_identity, "distinct physical roots must have distinct identities");
}
#[cfg(windows)]
#[tokio::test]
async fn windows_native_creation_witness_retains_started_history() {
let temp = tempdir().expect("create native witness fixture");
let root = std::fs::canonicalize(temp.path()).expect("canonical native root");
let _publication_root = PublicationRoot::new(&root).expect("retain native directory anchor");
let registration = register_native_publication_root(&root, &root.join(".")).expect("register same physical root");
let backend: Arc<dyn super::super::local::LocalIoBackend> = Arc::new(super::super::local::StdBackend::new(root.clone()));
let disk_id = uuid::Uuid::new_v4();
let witness = registration
.witness("published", disk_id, Arc::clone(&backend))
.expect("fresh native witness");
let bucket = root.join("published");
super::super::fs::mkdir(&bucket)
.await
.expect("publish through the actual native entry");
assert!(bucket.is_dir(), "the production entry must create the directory");
assert!(!validate_native_creation_witnesses(std::slice::from_ref(&witness)));
std::fs::remove_dir(&bucket).expect("remove published directory");
assert!(registration.witness("published", disk_id, Arc::clone(&backend)).is_none());
let reconnected = register_native_publication_root(&root, &root).expect("reconnect same physical root");
assert!(reconnected.witness("published", disk_id, Arc::clone(&backend)).is_none());
let unstarted = reconnected
.witness("unstarted", disk_id, Arc::clone(&backend))
.expect("untouched bucket keeps its witness");
assert!(consume_native_creation_witnesses(std::slice::from_ref(&unstarted)));
assert!(!validate_native_creation_witnesses(std::slice::from_ref(&unstarted)));
assert!(reconnected.witness("unstarted", disk_id, backend).is_none());
}
#[cfg(windows)]
#[test]
fn windows_native_creation_witness_rejects_unknown_roots() {
let temp = tempdir().expect("create unknown identity fixture");
let root = temp.path().join("root");
let other = temp.path().join("other");
let file = temp.path().join("file");
std::fs::create_dir(&root).expect("create native root");
std::fs::create_dir(&other).expect("create distinct native root");
std::fs::write(&file, b"not a directory").expect("create regular file");
assert!(native_root_identity(&temp.path().join("missing")).is_none());
assert!(native_root_identity(&file).is_none());
assert!(register_native_publication_root(&root, &other).is_none());
let registration = register_native_publication_root(&root, &root).expect("register actual native root");
let backend: Arc<dyn super::super::local::LocalIoBackend> = Arc::new(super::super::local::StdBackend::new(root));
registration.invalidate();
assert!(registration.witness("unstarted", uuid::Uuid::new_v4(), backend).is_none());
}
#[cfg(windows)]
#[test]
fn windows_rename_all_legacy_file_identity_distinguishes_sources() {
@@ -8053,272 +7737,3 @@ mod tests {
assert_eq!(err.kind(), io::ErrorKind::NotFound);
}
}
#[cfg(all(test, unix))]
mod native_creation_history_tests {
use super::*;
struct NativeFixture {
_temp: tempfile::TempDir,
root: PathBuf,
registration: NativePublicationRegistration,
backend: Arc<dyn super::super::local::LocalIoBackend>,
disk_id: uuid::Uuid,
}
impl NativeFixture {
fn new() -> Self {
let temp = tempfile::tempdir().expect("native root");
let root = std::fs::canonicalize(temp.path()).expect("canonical root");
let registration = register_native_publication_root(&root, &root).expect("native registration");
let backend = Arc::new(super::super::local::StdBackend::new(root.clone()));
Self {
_temp: temp,
root,
registration,
backend,
disk_id: uuid::Uuid::new_v4(),
}
}
fn witness(&self, bucket: &str) -> Option<NativeBucketCreationWitness> {
self.registration.witness(bucket, self.disk_id, Arc::clone(&self.backend))
}
}
#[tokio::test]
async fn every_native_directory_publication_entry_invalidates_retry() {
let fixture = NativeFixture::new();
for operation in [
"recursive",
"single",
"open",
"rename",
"rename_std",
"fallback",
"owned_mkdir",
"owned_rename",
"handle_relative",
] {
let bucket = format!("native-{operation}-{}", uuid::Uuid::new_v4());
let path = fixture.root.join(&bucket);
let witness = fixture.witness(&bucket).expect("fresh native witness");
match operation {
"recursive" => super::super::fs::make_dir_all(path.join("object/child"))
.await
.expect("recursive mkdir"),
"single" => super::super::fs::mkdir(&path).await.expect("single mkdir"),
"open" => {
let file = super::super::fs::open_file(&path, super::super::fs::O_CREATE | super::super::fs::O_WRONLY)
.await
.expect("create file");
drop(file);
}
"rename" | "rename_std" | "owned_rename" => {
let source = fixture.root.join(format!("source-{}", uuid::Uuid::new_v4()));
std::fs::create_dir(&source).expect("rename source");
match operation {
"rename" => super::super::fs::rename(&source, &path).await.expect("async rename"),
"rename_std" => super::super::fs::rename_std(&source, &path).expect("sync rename"),
_ => rename_with_namespace_owner(&source, &path, Some(Arc::new(())))
.await
.expect("owned rename"),
}
}
"fallback" => make_dir_all(path.join("object/child"), &path)
.await
.expect("implicit volume fallback"),
"owned_mkdir" => create_dir_all_with_namespace_owner(&path.join("child"), Some(Arc::new(())))
.await
.expect("owned recursive mkdir"),
"handle_relative" => {
let publication_root = PublicationRoot::new(&fixture.root).expect("publication root");
let guard = mkdir_all_below_existing_base_std(&path.join("child"), &fixture.root, &publication_root)
.expect("first-level handle-relative mkdir");
drop(guard);
}
_ => unreachable!(),
}
assert!(
!validate_native_creation_witnesses(std::slice::from_ref(&witness)),
"{operation}: old proof remains live"
);
if path.is_dir() {
std::fs::remove_dir_all(&path).expect("remove published directory");
} else {
std::fs::remove_file(&path).expect("remove published file");
}
assert!(fixture.witness(&bucket).is_none(), "{operation}: removal must not regrant proof");
}
}
#[tokio::test]
async fn cancelled_namespace_waiter_cannot_restore_native_proof() {
let fixture = NativeFixture::new();
let bucket = format!("native-cancel-{}", uuid::Uuid::new_v4());
let path = fixture.root.join(&bucket);
let witness = fixture.witness(&bucket).expect("fresh witness");
let lease = acquire_namespace_mutation_lease(&path).await;
let mut attempt = Box::pin(create_dir_all_with_namespace_owner(&path, Some(Arc::new(()))));
assert!(matches!(futures::poll!(&mut attempt), std::task::Poll::Pending));
assert!(
!validate_native_creation_witnesses(std::slice::from_ref(&witness)),
"Started must precede admission wait"
);
assert!(!path.exists(), "the namespace wait must still precede the syscall");
drop(attempt);
drop(lease);
assert!(fixture.witness(&bucket).is_none(), "cancellation cannot undo an admitted attempt");
}
#[tokio::test]
async fn queued_mkdir_cancellation_retains_lease_and_consumed_history() {
use futures::FutureExt;
use std::sync::mpsc;
let fixture = NativeFixture::new();
let bucket = format!("native-queued-cancel-{}", uuid::Uuid::new_v4());
let path = fixture.root.join(&bucket);
let witness = fixture.witness(&bucket).expect("fresh witness");
let (entered_tx, entered_rx) = mpsc::channel();
let (release_tx, release_rx) = mpsc::channel();
let _hook = prepared_publication_test_hooks::install_at(
prepared_publication_test_hooks::Stage::CreateDirectory,
&path,
move || {
entered_tx.send(()).expect("signal real executor entry");
let _ = release_rx.recv();
},
);
let operation_path = path.clone();
let operation =
tokio::spawn(async move { create_dir_all_with_namespace_owner(&operation_path, Some(Arc::new(()))).await });
tokio::task::spawn_blocking(move || entered_rx.recv_timeout(std::time::Duration::from_secs(30)))
.await
.expect("entry waiter")
.expect("the real blocking executor must enter");
operation.abort();
assert!(operation.await.expect_err("cancel the async waiter").is_cancelled());
assert!(
!validate_native_creation_witnesses(std::slice::from_ref(&witness)),
"queued publication is permanently Started"
);
assert!(
acquire_namespace_mutation_lease(&path).now_or_never().is_none(),
"the actual blocked executor must still own the namespace lease"
);
release_tx.send(()).expect("release the syscall");
let drained = tokio::time::timeout(std::time::Duration::from_secs(30), acquire_namespace_mutation_lease(&path))
.await
.expect("the lease must drain with the syscall");
drop(drained);
assert!(path.is_dir(), "the cancelled waiter cannot discard the already queued syscall");
std::fs::remove_dir_all(&path).expect("remove the published directory");
assert!(fixture.witness(&bucket).is_none(), "drain and removal cannot restore proof");
}
#[tokio::test]
async fn publication_detects_retargeted_physical_alias_without_registration() {
let source = NativeFixture::new();
let destination = NativeFixture::new();
let bucket = format!("native-retarget-{}", uuid::Uuid::new_v4());
let witness = destination.witness(&bucket).expect("destination witness");
let retired = source.root.with_extension("retired");
std::fs::rename(&source.root, &retired).expect("move the original physical root");
std::os::unix::fs::symlink(&destination.root, &source.root).expect("retarget the registered alias");
super::super::fs::mkdir(source.root.join(&bucket))
.await
.expect("publish through the changed alias");
assert!(
!validate_native_creation_witnesses(std::slice::from_ref(&witness)),
"publication-time alias drift must invalidate the actual destination root"
);
std::fs::remove_dir(destination.root.join(&bucket)).expect("remove the published bucket");
assert!(destination.witness(&bucket).is_none(), "removal must not undo alias uncertainty");
std::fs::remove_file(&source.root).expect("remove the symlink");
std::fs::rename(retired, &source.root).expect("restore the test root");
}
#[test]
fn blocked_physical_identity_lookup_does_not_block_other_publication() {
use std::sync::mpsc;
let stalled = NativeFixture::new();
let healthy = NativeFixture::new();
let witness = stalled.witness("unpublished").expect("stalled root witness");
let (entered_tx, entered_rx) = mpsc::channel();
let (release_tx, release_rx) = mpsc::channel();
let _hook = prepared_publication_test_hooks::install_at(
prepared_publication_test_hooks::Stage::NativeRootIdentity,
&stalled.root,
move || {
entered_tx.send(()).expect("signal physical lookup");
let _ = release_rx.recv();
},
);
let validator = std::thread::spawn(move || validate_native_creation_witnesses(&[witness]));
entered_rx
.recv_timeout(std::time::Duration::from_secs(30))
.expect("enter the actual identity lookup");
let (published_tx, published_rx) = mpsc::channel();
let path = healthy.root.join("published");
let publisher = std::thread::spawn(move || {
record_native_publication(&path);
published_tx.send(()).expect("signal independent publication");
});
let published = published_rx.recv_timeout(std::time::Duration::from_secs(5));
release_tx.send(()).expect("release the identity lookup");
validator.join().expect("identity lookup thread");
publisher.join().expect("publication thread");
assert!(
published.is_ok(),
"a slow physical lookup must not retain the process-wide publication mutex"
);
}
#[test]
fn native_consume_is_atomic_and_history_survives_reconnect() {
let fixture = NativeFixture::new();
let first = format!("native-first-{}", uuid::Uuid::new_v4());
let second = format!("native-second-{}", uuid::Uuid::new_v4());
let first_witness = fixture.witness(&first).expect("first witness");
let second_witness = fixture.witness(&second).expect("second witness");
record_native_publication(&fixture.root.join(&second));
assert!(!consume_native_creation_witnesses(&[first_witness.clone(), second_witness]));
assert!(
validate_native_creation_witnesses(std::slice::from_ref(&first_witness)),
"failed cohort consume must be atomic"
);
assert!(consume_native_creation_witnesses(std::slice::from_ref(&first_witness)));
assert!(!validate_native_creation_witnesses(std::slice::from_ref(&first_witness)));
let reconnected = register_native_publication_root(&fixture.root, &fixture.root).expect("reconnected registration");
let backend: Arc<dyn super::super::local::LocalIoBackend> =
Arc::new(super::super::local::StdBackend::new(fixture.root.clone()));
assert!(
reconnected.witness(&first, fixture.disk_id, backend).is_none(),
"reconnect must retain Started"
);
}
#[test]
fn physical_alias_reuse_is_unknown_and_backend_identity_is_real() {
let temp = tempfile::tempdir().expect("alias parent");
let root = temp.path().join("root");
std::fs::create_dir(&root).expect("first physical root");
let old = register_native_publication_root(&root, &root).expect("first registration");
let backend: Arc<dyn super::super::local::LocalIoBackend> = Arc::new(super::super::local::StdBackend::new(root.clone()));
let id = uuid::Uuid::new_v4();
let witness = old.witness("same-name", id, Arc::clone(&backend)).expect("old witness");
let other_backend: Arc<dyn super::super::local::LocalIoBackend> =
Arc::new(super::super::local::StdBackend::new(root.clone()));
let other = old.witness("same-name", id, other_backend).expect("new backend witness");
assert!(!witness.matches(&other), "a different actual backend cannot reuse a pending observation");
std::fs::rename(&root, temp.path().join("retired-root")).expect("retire physical root");
std::fs::create_dir(&root).expect("replacement root at the same alias");
let replacement = register_native_publication_root(&root, &root).expect("replacement registration");
assert!(replacement.witness("same-name", id, backend).is_none(), "alias reuse must remain Unknown");
assert!(
!validate_native_creation_witnesses(&[witness]),
"old physical identity must not validate at a new root"
);
}
}
+1 -1
View File
@@ -17,8 +17,8 @@
use crate::bucket::error::BucketMetadataError;
use crate::disk::error::DiskError;
use crate::storage_api_contracts::{error::StorageErrorCode, range::HTTPRangeError};
use rustfs_s3_types::S3ErrorCode;
use rustfs_utils::path::decode_dir_object;
use s3s::S3ErrorCode;
pub type Error = StorageError;
pub type Result<T> = core::result::Result<T, Error>;
-4
View File
@@ -174,14 +174,10 @@ async fn open_reader_source(
metrics_path: Option<&'static str>,
) -> disk::error::Result<Option<BoxedObjectReader>> {
if let Some(data) = inline_data {
#[cfg(test)]
crate::set_disk::disk_call_counters::record_shard_open(path, true);
let mut reader = Cursor::new(data);
reader.set_position(u64::try_from(offset).map_err(|_| DiskError::FileCorrupt)?);
Ok(Some(ShardReader::InMemory(reader)))
} else if let Some(disk) = disk {
#[cfg(test)]
crate::set_disk::disk_call_counters::record_shard_open(path, false);
open_disk_reader(disk, bucket, path, offset, length, use_mmap_read, metrics_path)
.await
.map(Some)
-4
View File
@@ -102,14 +102,10 @@ mod body_cache_hook;
mod encryption;
mod hook_slot;
mod object_mutation_hook;
mod read_condition;
mod readers;
mod types;
pub(crate) mod write_commit_context;
pub use read_condition::{GetObjectReadCondition, GetObjectReadMetadata, with_get_object_read_condition};
pub(crate) use read_condition::{get_object_read_condition_is_active, get_object_read_condition_is_terminal};
#[cfg(test)]
mod persisted_metadata_keys_tests;
@@ -29,7 +29,7 @@ use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions};
use crate::storage_api_contracts::range::HTTPRangeSpec;
use rustfs_filemeta::ObjectPartInfo;
use rustfs_rio::HashReader;
use rustfs_s3_types::S3ErrorCode;
use s3s::S3ErrorCode;
//#[derive(Clone)]
pub struct PutObjReader {
@@ -231,45 +231,6 @@ mod tests {
assert_eq!(result.2, 1024 * 1024);
}
#[test]
fn test_new_getobjectreader_rejects_a_range_past_the_object_as_invalid_range() {
let oi = multipart_object_info();
let past_the_end = Some(HTTPRangeSpec {
is_suffix_length: false,
start: oi.size,
end: oi.size + 10,
});
let err = new_getobjectreader(&past_the_end, &oi, &ObjectOptions::default(), &HeaderMap::new())
.err()
.expect("a range starting at the object size must be rejected");
assert_eq!(err.code, S3ErrorCode::InvalidRange);
assert_eq!(err.code.as_str(), "InvalidRange");
assert_eq!(err.key, None);
assert_eq!(err.bucket_name, None);
}
#[test]
fn test_new_getobjectreader_rejects_an_unknown_size_without_a_range_as_invalid_range() {
let oi = ObjectInfo {
bucket: "bucket".to_string(),
name: "object".to_string(),
size: -1,
..Default::default()
};
let err = new_getobjectreader(&None, &oi, &ObjectOptions::default(), &HeaderMap::new())
.err()
.expect("an object of unknown size cannot be read without a range");
assert_eq!(err.code, S3ErrorCode::InvalidRange);
assert_eq!(err.code.as_str(), "InvalidRange");
assert_eq!(err.message, "Invalid range");
assert_eq!(err.key.as_deref(), Some("object"));
assert_eq!(err.bucket_name.as_deref(), Some("bucket"));
}
#[test]
fn test_to_s3s_etag() {
// Test unquoted ETag - should become strong etag
@@ -1,113 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use rustfs_filemeta::{FileInfo, ObjectPartInfo};
use std::future::Future;
use std::sync::Arc;
use time::OffsetDateTime;
/// Borrowed fields needed to decide whether an object read needs a body.
pub struct GetObjectReadMetadata<'a> {
pub bucket: &'a str,
pub object: &'a str,
pub etag: Option<&'a str>,
pub mod_time: Option<OffsetDateTime>,
pub parts: &'a [ObjectPartInfo],
pub delete_marker: bool,
}
impl<'a> GetObjectReadMetadata<'a> {
pub(crate) fn from_file_info(bucket: &'a str, object: &'a str, info: &'a FileInfo) -> Self {
Self {
bucket,
object,
etag: info.metadata.get("etag").map(String::as_str),
mod_time: info.mod_time,
parts: &info.parts,
delete_marker: info.deleted,
}
}
}
/// A pure decision on quorum-validated metadata. Authorization belongs to the caller.
pub trait GetObjectReadCondition: Send + Sync {
fn is_terminal(&self, metadata: GetObjectReadMetadata<'_>) -> bool;
}
tokio::task_local! {
static READ_CONDITION: Arc<dyn GetObjectReadCondition>;
}
/// Applies a condition to this reader call, without inheriting it into shared producers.
pub async fn with_get_object_read_condition<F: Future>(condition: Arc<dyn GetObjectReadCondition>, read: F) -> F::Output {
READ_CONDITION.scope(condition, read).await
}
pub(crate) fn get_object_read_condition_is_terminal(metadata: GetObjectReadMetadata<'_>) -> bool {
READ_CONDITION
.try_with(|condition| condition.is_terminal(metadata))
.unwrap_or(false)
}
pub(crate) fn get_object_read_condition_is_active() -> bool {
READ_CONDITION.try_with(|_| ()).is_ok()
}
#[cfg(test)]
mod tests {
use super::*;
struct Stop;
impl GetObjectReadCondition for Stop {
fn is_terminal(&self, _: GetObjectReadMetadata<'_>) -> bool {
true
}
}
fn metadata() -> GetObjectReadMetadata<'static> {
GetObjectReadMetadata {
bucket: "bucket",
object: "object",
etag: Some("etag"),
mod_time: None,
parts: &[],
delete_marker: false,
}
}
#[tokio::test]
async fn read_condition_scope_does_not_escape_or_enter_shared_producers() {
assert!(!get_object_read_condition_is_terminal(metadata()));
with_get_object_read_condition(Arc::new(Stop), async {
assert!(get_object_read_condition_is_terminal(metadata()));
assert!(
!tokio::spawn(async { get_object_read_condition_is_terminal(metadata()) })
.await
.expect("shared producer")
);
})
.await;
assert!(!get_object_read_condition_is_terminal(metadata()));
}
#[tokio::test]
async fn dropping_a_pending_read_removes_its_condition_scope() {
let mut read = Box::pin(with_get_object_read_condition(Arc::new(Stop), async {
assert!(get_object_read_condition_is_terminal(metadata()));
std::future::pending::<()>().await;
}));
assert!(futures::poll!(&mut read).is_pending());
drop(read);
assert!(!get_object_read_condition_is_terminal(metadata()));
}
}
-19
View File
@@ -528,14 +528,6 @@ pub struct GetObjectReader {
}
impl GetObjectReader {
pub(crate) fn condition_stopped(object_info: ObjectInfo) -> Self {
Self {
stream: Box::new(ConditionStoppedBody),
object_info,
buffered_body: None,
body_source: GetObjectBodySource::Unprobed,
}
}
/// Builds a fully materialized reader from a cache-coordinated body.
pub fn from_cache_body(mut object_info: ObjectInfo, body: Bytes) -> Result<Self> {
object_info.size = i64::try_from(body.len()).map_err(|_| Error::other("cached GET body length exceeds i64::MAX"))?;
@@ -560,17 +552,6 @@ impl GetObjectReader {
}
}
struct ConditionStoppedBody;
impl AsyncRead for ConditionStoppedBody {
fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
if buf.remaining() == 0 {
return Poll::Ready(Ok(()));
}
Poll::Ready(Err(std::io::Error::other("object read condition must be handled before body access")))
}
}
enum ReadTransform {
// Written but never read by production code: the enclosing struct already
// carries the same pair as `storage_offset`/`storage_length`. They survive
-5
View File
@@ -938,10 +938,6 @@ pub struct ObjectOptions {
/// A genuine target write guard; loss fences alone do not authorize a borrowed write.
#[doc(hidden)]
pub write_lock_context: Option<WriteLockContext>,
/// In-process delete ownership for late filesystem completion. Never
/// serialized or copied into object metadata.
#[doc(hidden)]
pub delete_mutation_owner: Option<Arc<dyn Send + Sync>>,
/// Internal read-only inspection must not enqueue metadata or payload repairs.
pub suppress_read_repair: bool,
/// Control-plane writers that immediately read or CAS the same namespace
@@ -1162,7 +1158,6 @@ impl std::fmt::Debug for ObjectOptions {
.field("http_preconditions", &self.http_preconditions.is_some())
.field("delete_replication", &self.delete_replication.is_some())
.field("delete_replication_config_snapshot", &self.delete_replication_config_snapshot)
.field("delete_mutation_owner", &self.delete_mutation_owner.is_some())
.field("namespace_lock_fence", &self.namespace_lock_fence.is_some())
.field("bucket_lifecycle_lock_fence", &self.bucket_lifecycle_lock_fence.is_some())
.field("replication_request", &self.replication_request)
+4 -1
View File
@@ -50,7 +50,6 @@ use s3s::dto::BucketLifecycleConfiguration;
use s3s::region::Region;
use tokio::sync::{OwnedRwLockReadGuard, RwLock};
use tokio_util::sync::CancellationToken;
#[cfg(test)]
use tonic::transport::Channel;
use uuid::Uuid;
@@ -252,6 +251,10 @@ pub(crate) async fn root_disk_threshold_for_erasure_disk() -> Option<u64> {
}
}
pub(crate) async fn cached_node_channel(addr: &str) -> Option<Channel> {
rustfs_common::cached_connection(addr).await
}
#[cfg(test)]
pub(crate) async fn cache_test_node_channel(addr: String, channel: Channel) {
rustfs_common::cache_connection(addr, channel).await;
+23 -31
View File
@@ -50,7 +50,7 @@ fn ensure_rebalance_entry_active(cancel: &CancellationToken) -> Result<()> {
Ok(())
}
#[cfg(any(test, feature = "test-util"))]
#[cfg(test)]
tokio::task_local! {
static REBALANCE_ENTRY_RUN_FENCE_BARRIER: (Arc<tokio::sync::Notify>, Arc<tokio::sync::Notify>);
}
@@ -268,7 +268,7 @@ impl ECStore {
self.rebalance_run_guard(rebalance_id.as_ref(), "rebalance entry")
})
.await?;
#[cfg(any(test, feature = "test-util"))]
#[cfg(test)]
if let Ok((arrived, release)) = REBALANCE_ENTRY_RUN_FENCE_BARRIER.try_with(Clone::clone) {
arrived.notify_one();
release.notified().await;
@@ -836,8 +836,7 @@ pub mod test_util {
_temp_dirs: Vec<tempfile::TempDir>,
store: Arc<ECStore>,
cancel: CancellationToken,
entry_arrived: Arc<tokio::sync::Notify>,
entry_release: Arc<tokio::sync::Notify>,
barrier: crate::data_movement::SourceCleanupDeleteBarrier,
stop_probe: super::super::control::RebalanceStopWaitProbe,
entry_task: Option<tokio::task::JoinHandle<Result<RebalanceEntryOutcome>>>,
}
@@ -898,36 +897,31 @@ pub mod test_util {
cached: Some(file_meta),
reusable: false,
};
let entry_arrived = Arc::new(tokio::sync::Notify::new());
let entry_release = Arc::new(tokio::sync::Notify::new());
let barrier = crate::data_movement::SourceCleanupDeleteBarrier::install(bucket, object);
let entry_store = Arc::clone(&store);
let entry_set = Arc::clone(&set);
let entry_cancel = cancel.clone();
let entry_task = tokio::spawn(REBALANCE_ENTRY_RUN_FENCE_BARRIER.scope(
(Arc::clone(&entry_arrived), Arc::clone(&entry_release)),
async move {
entry_store
.rebalance_entry(
RebalanceEntryTarget {
bucket: bucket.to_string(),
pool_index: 0,
},
entry,
entry_set,
Arc::new(RebalanceBucketConfigs::default()),
Arc::from(rebalance_id),
entry_cancel,
)
.await
},
));
let entry_task = tokio::spawn(async move {
entry_store
.rebalance_entry(
RebalanceEntryTarget {
bucket: bucket.to_string(),
pool_index: 0,
},
entry,
entry_set,
Arc::new(RebalanceBucketConfigs::default()),
Arc::from(rebalance_id),
entry_cancel,
)
.await
});
Self {
_temp_dirs,
store,
cancel,
entry_arrived,
entry_release,
barrier,
stop_probe: super::super::control::RebalanceStopWaitProbe::install(rebalance_id),
entry_task: Some(entry_task),
}
@@ -938,9 +932,7 @@ pub mod test_util {
}
pub async fn wait_until_entry_paused(&self) {
tokio::time::timeout(std::time::Duration::from_secs(30), self.entry_arrived.notified())
.await
.expect("the entry should acquire its run fence before pausing");
self.barrier.wait_until_paused().await;
}
pub async fn wait_until_admission_cancelled(&self) {
@@ -954,7 +946,7 @@ pub mod test_util {
}
pub fn release_entry(&self) {
self.entry_release.notify_one();
self.barrier.release();
}
pub async fn assert_entry_cancelled(&mut self) {
@@ -972,7 +964,7 @@ pub mod test_util {
impl Drop for PausedRebalanceEntryTestFixture {
fn drop(&mut self) {
self.entry_release.notify_one();
self.barrier.release();
if let Some(task) = self.entry_task.take() {
task.abort();
}
+1 -2
View File
@@ -82,9 +82,8 @@ use crate::{
use rustfs_filemeta::FileInfo;
use rustfs_rio::HashReader;
use rustfs_s3_client::{admin_handler_utils::AdminError, provider_versions::ProviderVersionCapabilities};
use rustfs_s3_types::S3ErrorCode;
use rustfs_utils::path::{SLASH_SEPARATOR, path_join};
use s3s::dto::BucketLifecycleConfiguration;
use s3s::{S3ErrorCode, dto::BucketLifecycleConfiguration};
use super::{
tier_handlers::{ERR_TIER_BUCKET_NOT_FOUND, ERR_TIER_CONNECT_ERR, ERR_TIER_INVALID_CREDENTIALS, ERR_TIER_PERM_ERR},
@@ -1,63 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Test-util-only controls for the paired read benchmark's dedicated bucket.
use std::sync::atomic::{AtomicBool, Ordering};
static ACTIVE: AtomicBool = AtomicBool::new(false);
/// Keeps metadata off the cache in the benchmark's isolated bucket so GET
/// metadata reads exercise the configured slow disk. Body-cache policy
/// remains intact.
#[must_use]
pub struct ConditionalReadBenchmarkMetadataGuard {
_private: (),
}
impl ConditionalReadBenchmarkMetadataGuard {
pub fn acquire() -> Self {
assert!(
ACTIVE
.compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire)
.is_ok()
);
Self { _private: () }
}
}
impl Drop for ConditionalReadBenchmarkMetadataGuard {
fn drop(&mut self) {
ACTIVE.store(false, Ordering::Release);
}
}
pub(super) fn cache_bypass_applies(bucket: &str) -> bool {
bucket == "conditional-bench" && ACTIVE.load(Ordering::Acquire)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn benchmark_metadata_profile_is_scoped_to_its_bucket_and_owner() {
assert!(!cache_bypass_applies("conditional-bench"));
let owner = ConditionalReadBenchmarkMetadataGuard::acquire();
assert!(cache_bypass_applies("conditional-bench"));
assert!(!cache_bypass_applies("other-bucket"));
drop(owner);
assert!(!cache_bypass_applies("conditional-bench"));
}
}
@@ -1260,10 +1260,6 @@ async fn try_create_bitrot_readers_via_batch_pread(
}
let requests: Vec<_> = batch_items.iter().map(|(_, p, off, len)| (p.clone(), *off, *len)).collect();
#[cfg(test)]
for (disk_index, _, _, _) in &batch_items {
disk_call_counters::record(object, disk_call_counters::KIND_DISK_SHARD_OPEN, *disk_index);
}
let batch_results = batch_shard_pread(requests).await;
let mut setup = BitrotReaderSetup::new(disks.len());
@@ -6082,10 +6078,6 @@ pub(crate) mod disk_call_counters {
pub const KIND_READ_VERSION: &str = "read_version";
pub const KIND_BATCH_READ_VERSION: &str = "batch_read_version";
pub const KIND_METADATA_SLOWTAIL_FAULT: &str = "metadata_slowtail_fault";
pub const KIND_INLINE_SHARD_OPEN: &str = "inline_shard_open";
pub const KIND_DISK_SHARD_OPEN: &str = "disk_shard_open";
/// Body verification, direct shard assembly, and erasure decoder entry points.
pub const KIND_BODY_DECODE: &str = "body_decode";
/// Registry key: (object, kind, disk_index).
type CountKey = (String, String, usize);
@@ -6116,28 +6108,6 @@ pub(crate) mod disk_call_counters {
.or_insert(0) += 1;
}
/// Matches shard paths to the observed object's namespace path. This also
/// records deferred opens running in spawned decoder tasks.
pub(crate) fn record_shard_open(path: &str, inline: bool) {
let mut reg = registry().lock().expect("disk call-counter registry poisoned");
let objects: Vec<_> = reg
.observed
.iter()
.filter(|object| {
path == object.as_str() || path.strip_prefix(object.as_str()).is_some_and(|rest| rest.starts_with('/'))
})
.cloned()
.collect();
let kind = if inline {
KIND_INLINE_SHARD_OPEN
} else {
KIND_DISK_SHARD_OPEN
};
for object in objects {
*reg.counts.entry((object, kind.to_owned(), 0)).or_insert(0) += 1;
}
}
/// RAII scope that observes call counts for a single `object`. Counts recorded
/// while the scope is alive — including those recorded inside spawned tasks —
/// are queryable; the scope clears its own counts on drop.
@@ -66,7 +66,6 @@ use tokio::task::JoinSet;
pub(in crate::set_disk) struct MetadataReadResult {
pub(in crate::set_disk) slots: Vec<MetadataDiskObservation>,
pub(in crate::set_disk) diagnostics: MetadataFanoutDiagnostics,
pub(in crate::set_disk) condition_stopped: bool,
}
impl MetadataReadResult {
@@ -656,8 +655,6 @@ async fn inline_body_miss_reason_from_observations<'a>(
return Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY);
};
#[cfg(test)]
crate::set_disk::disk_call_counters::record(object, crate::set_disk::disk_call_counters::KIND_BODY_DECODE, 0);
match try_read_inline_data_shards_direct(&mut readers, erasure.data_shards, read_length, object_size).await {
Some(body) if body.len() == object_size => None,
_ => Some(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_BODY_VERIFY),
@@ -1081,11 +1078,7 @@ impl SetDisks {
0,
);
}
Ok(MetadataReadResult {
slots,
diagnostics,
condition_stopped: false,
})
Ok(MetadataReadResult { slots, diagnostics })
}
#[allow(clippy::too_many_arguments)]
@@ -1128,7 +1121,6 @@ impl SetDisks {
let mut next_fanout_index = 0usize;
let mut scheduled_count = 0usize;
let mut force_full_wait = false;
let read_condition_active = read_data && crate::object_api::get_object_read_condition_is_active();
let mut final_miss_reason_override = None;
let mut non_inline_candidate_eligible = None;
let mut single_pending_hedge_deadline = None;
@@ -1266,28 +1258,12 @@ impl SetDisks {
}
}
if (!force_full_wait || read_condition_active)
if !force_full_wait
&& let Some(decision) = accumulator
.early_stop_decision()
.or_else(|| accumulator.version_early_stop_decision())
{
let condition_stopped = read_condition_active
&& decision.reason == crate::diagnostics::get::GET_METADATA_EARLY_STOP_REASON_VALID_QUORUM
&& accumulator.candidate.as_ref().is_some_and(|candidate| {
!candidate.deleted
&& crate::object_api::get_object_read_condition_is_terminal(
crate::object_api::GetObjectReadMetadata::from_file_info(
bucket.as_ref(),
object.as_ref(),
candidate,
),
)
});
let should_return_early = if condition_stopped {
true
} else if force_full_wait {
false
} else if read_data {
let should_return_early = if read_data {
match accumulator.candidate.as_ref() {
Some(_candidate) if non_inline_candidate_eligible == Some(true) => {
accumulator.candidate_has_read_reserve()
@@ -1356,11 +1332,7 @@ impl SetDisks {
cancelled_count,
);
let diagnostics = MetadataFanoutDiagnostics::new(fanout_start.elapsed(), observations);
return Ok(MetadataReadResult {
slots,
diagnostics,
condition_stopped,
});
return Ok(MetadataReadResult { slots, diagnostics });
}
}
@@ -1438,11 +1410,7 @@ impl SetDisks {
rustfs_io_metrics::record_get_object_metadata_early_stop_saved_responses(metrics_path, 0);
rustfs_io_metrics::record_get_object_metadata_fanout_lifecycle(metrics_path, scheduled_count, scheduled_count, 0);
let diagnostics = MetadataFanoutDiagnostics::new(fanout_start.elapsed(), observations);
Ok(MetadataReadResult {
slots,
diagnostics,
condition_stopped: false,
})
Ok(MetadataReadResult { slots, diagnostics })
}
/// Test-only seam that records one per-disk `read_version` metadata RPC for
-262
View File
@@ -793,11 +793,6 @@ const DEFAULT_RUSTFS_GET_METADATA_TWO_PHASE_READ_PLAN_ENABLE: bool = false;
const ENV_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT: &str = "RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT";
const DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT: bool = true;
#[cfg(feature = "test-util")]
mod conditional_read_bench;
#[cfg(feature = "test-util")]
pub use conditional_read_bench::ConditionalReadBenchmarkMetadataGuard;
const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS";
const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DISKS: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DISKS";
const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_BUCKET: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_BUCKET";
@@ -879,7 +874,6 @@ mod ctx;
mod metadata;
mod ops;
pub(crate) use ops::bucket::BucketInfoQuorum;
pub(crate) use ops::delete_recovery::{DeleteRecoveryContext, DeleteRecoverySummary};
pub(crate) use ops::heal::AbsenceProofRequest;
#[cfg(test)]
pub(crate) use ops::heal::DanglingDeleteFailure;
@@ -924,7 +918,6 @@ pub use ops::heal_walk::HealWalkVersion;
pub(in crate::set_disk) struct GetObjectFileInfo {
owned: Option<OwnedGetObjectFileInfo>,
shared: Option<Arc<GetObjectMetadataCacheEntry>>,
condition_stopped: bool,
}
struct OwnedGetObjectFileInfo {
@@ -946,7 +939,6 @@ impl GetObjectFileInfo {
late_metadata_fanout_disks: None,
}),
shared: None,
condition_stopped: false,
}
}
@@ -964,16 +956,13 @@ impl GetObjectFileInfo {
late_metadata_fanout_disks: Some(late_metadata_fanout_disks),
}),
shared: None,
condition_stopped: false,
}
}
fn shared(entry: Arc<GetObjectMetadataCacheEntry>) -> Self {
let condition_stopped = entry.capability == GetObjectMetadataCacheCapability::ConditionOnly;
Self {
owned: None,
shared: Some(entry),
condition_stopped,
}
}
@@ -1154,248 +1143,6 @@ mod prepared_get_object_metadata_tests {
assert!(take_prepared_get_object_metadata().is_none());
}
struct StopReadForEtag(String);
impl crate::object_api::GetObjectReadCondition for StopReadForEtag {
fn is_terminal(&self, metadata: crate::object_api::GetObjectReadMetadata<'_>) -> bool {
!metadata.delete_marker && metadata.etag == Some(self.0.as_str())
}
}
#[tokio::test]
#[serial_test::serial(body_cache_hook)]
async fn read_condition_stops_on_quorum_before_slow_disk_and_body_verification() {
use crate::object_api::with_get_object_read_condition;
for size in [4096, 1_300_000] {
let (dirs, set) = make_local_set_disks(4, 2).await;
let bucket = "read-condition-quorum";
let object = object_with_initial_data_shards(bucket, &format!("read-condition-{size}"));
let opts = ObjectOptions {
no_lock: true,
..Default::default()
};
set.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let mut put = PutObjReader::from_vec(vec![b'v'; size]);
let info = set.put_object(bucket, &object, &mut put, &opts).await.expect("write object");
for dir in &dirs {
let object_path = dir.path().join(bucket).join(&object);
if size == 4096 {
let path = object_path.join("xl.meta");
let mut bytes = tokio::fs::read(&path).await.expect("read inline metadata");
*bytes.last_mut().expect("inline payload") ^= 1;
tokio::fs::write(path, bytes).await.expect("damage inline data");
} else {
tokio::fs::remove_dir_all(object_path.join(info.data_dir.expect("data directory").to_string()))
.await
.expect("remove shards");
}
}
let slow = bounded_initial_parity_disk_index(bucket, &object);
let barrier = rename_fanout_barrier::arm(&object, slow, rename_fanout_barrier::PHASE_READ_VERSION);
let calls = disk_call_counters::observe(&object);
let read_opts = ObjectOptions {
metadata_cache_safe: true,
..opts
};
let followup_opts = read_opts.clone();
let condition = Arc::new(StopReadForEtag(info.etag.clone().expect("committed ETag")));
let set_for_read = Arc::clone(&set);
let object_for_read = object.clone();
let mut read = tokio::spawn(async move {
with_get_object_read_condition(
condition,
set_for_read.get_object_reader(bucket, &object_for_read, None, HeaderMap::new(), &read_opts),
)
.await
});
tokio::time::timeout(READ_VERSION_BARRIER_GUARD, barrier.wait_until_paused())
.await
.expect("slow metadata read is pending");
let mut reader = tokio::time::timeout(READ_VERSION_BARRIER_GUARD, &mut read)
.await
.expect("condition must not wait for the slow disk or decode damaged data")
.expect("reader task")
.expect("metadata-only result");
assert_eq!(reader.object_info.etag, info.etag);
assert_eq!(
calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN),
0,
"damaged inline data was never opened"
);
assert_eq!(
calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN),
0,
"missing shards were never opened"
);
assert_eq!(
calls.total(disk_call_counters::KIND_BODY_DECODE),
0,
"the body verifier and decoder were never entered"
);
let mut body = Vec::new();
assert!(
reader.stream.read_to_end(&mut body).await.is_err(),
"terminal metadata cannot be served as a successful body"
);
assert!(body.is_empty());
let first_calls = calls.total(disk_call_counters::KIND_READ_VERSION);
drop(reader);
barrier.release();
let snapshot = set
.get_object_fileinfo_for_get_object_reader(bucket, &object, &followup_opts, true, true)
.await
.expect("fresh metadata after terminal scope");
assert!(!snapshot.condition_stopped);
assert!(
calls.total(disk_call_counters::KIND_READ_VERSION) > first_calls,
"an incomplete terminal snapshot must not satisfy the next metadata read from cache"
);
}
}
#[tokio::test]
#[serial_test::serial(body_cache_hook)]
async fn read_condition_skips_shard_open_and_decode_with_live_positive_controls() {
for size in [4096, 1_300_000] {
let (_dirs, set) = make_local_set_disks(4, 2).await;
let bucket = "read-condition-body-counters";
let object = object_with_initial_data_shards(bucket, &format!("body-counters-{size}"));
let opts = ObjectOptions {
no_lock: true,
..Default::default()
};
let payload: Vec<_> = (0..size)
.map(|index| u8::try_from(index % 251).expect("pattern byte"))
.collect();
set.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let mut put = PutObjReader::from_vec(payload.clone());
let info = set.put_object(bucket, &object, &mut put, &opts).await.expect("write object");
let calls = disk_call_counters::observe(&object);
let reader = crate::object_api::with_get_object_read_condition(
Arc::new(StopReadForEtag(info.etag.expect("committed ETag"))),
set.get_object_reader(bucket, &object, None, HeaderMap::new(), &opts),
)
.await
.expect("terminal metadata reader");
drop(reader);
assert!(calls.total(disk_call_counters::KIND_READ_VERSION) > 0, "metadata really was read");
assert_eq!(
calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN),
0,
"no inline shard source opened"
);
assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 0, "no disk shard source opened");
assert_eq!(
calls.total(disk_call_counters::KIND_BODY_DECODE),
0,
"no body verifier or decoder entered"
);
let mut reader = crate::object_api::with_get_object_read_condition(
Arc::new(StopReadForEtag("different-etag".to_owned())),
set.get_object_reader(bucket, &object, None, HeaderMap::new(), &opts),
)
.await
.expect("unmatched condition opens the intact object");
let mut body = Vec::new();
reader.stream.read_to_end(&mut body).await.expect("positive-control body");
assert_eq!(body, payload);
let source_kind = if size == 4096 {
disk_call_counters::KIND_INLINE_SHARD_OPEN
} else {
disk_call_counters::KIND_DISK_SHARD_OPEN
};
assert!(
calls.total(source_kind) > 0,
"the same source counter observes the positive control: size={size}, inline={}, disk={}, decode={}",
calls.total(disk_call_counters::KIND_INLINE_SHARD_OPEN),
calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN),
calls.total(disk_call_counters::KIND_BODY_DECODE),
);
assert!(
calls.total(disk_call_counters::KIND_BODY_DECODE) > 0,
"the same decode counter observes the positive control"
);
}
}
#[tokio::test]
#[serial_test::serial(body_cache_hook)]
async fn read_condition_rechecks_selected_metadata_before_body_access() {
struct Changing(std::sync::atomic::AtomicUsize);
impl crate::object_api::GetObjectReadCondition for Changing {
fn is_terminal(&self, _: crate::object_api::GetObjectReadMetadata<'_>) -> bool {
self.0.fetch_add(1, std::sync::atomic::Ordering::Relaxed) == 0
}
}
let (_dirs, set) = make_local_set_disks(4, 2).await;
let bucket = "read-condition-consistency";
let object = object_with_initial_data_shards(bucket, "read-condition-consistency");
let opts = ObjectOptions {
no_lock: true,
..Default::default()
};
set.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let mut put = PutObjReader::from_vec(vec![b'x'; 4096]);
set.put_object(bucket, &object, &mut put, &opts).await.expect("write object");
let result = crate::object_api::with_get_object_read_condition(
Arc::new(Changing(std::sync::atomic::AtomicUsize::new(0))),
set.get_object_reader(bucket, &object, None, HeaderMap::new(), &opts),
)
.await;
assert!(result.is_err(), "an unverified terminal snapshot must never become a 200 reader");
}
#[tokio::test]
#[serial_test::serial(body_cache_hook)]
async fn read_condition_miss_preserves_inline_early_stop_and_exact_body() {
let (_dirs, set) = make_local_set_disks(4, 2).await;
let bucket = "read-condition-miss";
let object = object_with_initial_data_shards(bucket, "read-condition-miss");
let opts = ObjectOptions {
no_lock: true,
..Default::default()
};
let payload: Vec<u8> = (0..8192)
.map(|index| u8::try_from(index % 251).expect("pattern byte"))
.collect();
set.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let mut put = PutObjReader::from_vec(payload.clone());
set.put_object(bucket, &object, &mut put, &opts)
.await
.expect("write inline object");
let slow = bounded_initial_parity_disk_index(bucket, &object);
let barrier = rename_fanout_barrier::arm(&object, slow, rename_fanout_barrier::PHASE_READ_VERSION);
let set_for_read = Arc::clone(&set);
let object_for_read = object.clone();
let mut read = tokio::spawn(async move {
crate::object_api::with_get_object_read_condition(
Arc::new(StopReadForEtag("other".to_owned())),
set_for_read.get_object_reader(bucket, &object_for_read, None, HeaderMap::new(), &opts),
)
.await
});
tokio::time::timeout(READ_VERSION_BARRIER_GUARD, barrier.wait_until_paused())
.await
.expect("slow metadata read is pending");
let mut reader = tokio::time::timeout(READ_VERSION_BARRIER_GUARD, &mut read)
.await
.expect("condition miss preserves the normal early stop")
.expect("reader task")
.expect("reader");
let mut bytes = Vec::new();
reader.stream.read_to_end(&mut bytes).await.expect("normal body");
assert_eq!(bytes, payload);
barrier.release();
}
#[test]
fn cache_hit_consumers_release_snapshot_at_legacy_ownership() {
let fi = FileInfo {
@@ -1404,7 +1151,6 @@ mod prepared_get_object_metadata_tests {
};
let cached = Arc::new(GetObjectMetadataCacheEntry {
created_at: Instant::now(),
capability: GetObjectMetadataCacheCapability::Full,
parts_metadata: vec![fi.clone()],
fi,
online_disks: vec![None],
@@ -4687,13 +4433,6 @@ impl Hash for GetObjectMetadataCacheKey {
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum GetObjectMetadataCacheCapability {
Full,
// Quorum-validated metadata without a complete body or write-target layout.
ConditionOnly,
}
#[derive(Debug)]
struct GetObjectMetadataCacheEntry {
#[allow(dead_code)] // Kept for debugging; moka handles TTL internally
@@ -4702,7 +4441,6 @@ struct GetObjectMetadataCacheEntry {
parts_metadata: Vec<FileInfo>,
online_disks: Vec<Option<DiskStore>>,
read_quorum: usize,
capability: GetObjectMetadataCacheCapability,
}
#[derive(Clone, Debug)]
+2 -16
View File
@@ -123,18 +123,8 @@ impl SetDisks {
}
pub(crate) async fn list_bucket_for_scanner(&self, _opts: &BucketOptions) -> Result<(Vec<BucketInfo>, bool)> {
self.list_bucket_with_quorum(BucketInfoQuorum::Write).await
}
pub(crate) async fn list_bucket_with_quorum(&self, quorum: BucketInfoQuorum) -> Result<(Vec<BucketInfo>, bool)> {
let disks = self.disk_inventory().await;
let write_quorum = (disks.len() / 2) + 1;
// Discovery uses the same namespace read boundary as stat_bucket.
// It grants no authority to mutate buckets or accept object reads.
let required = match quorum {
BucketInfoQuorum::Read => disks.len().div_ceil(2).max(1),
BucketInfoQuorum::Write => write_quorum,
};
let mut futures = Vec::with_capacity(disks.len());
for disk in disks {
@@ -163,11 +153,7 @@ impl SetDisks {
}
}
let error = match quorum {
BucketInfoQuorum::Read => reduce_read_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, required),
BucketInfoQuorum::Write => reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, required),
};
if let Some(err) = error {
if let Some(err) = reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, write_quorum) {
return Err(err.into());
}
@@ -193,7 +179,7 @@ impl SetDisks {
topology_complete &= counts.values().all(|(count, _)| *count >= write_quorum);
let mut buckets = counts
.into_values()
.filter_map(|(count, bucket)| (count >= required).then_some(bucket))
.filter_map(|(count, bucket)| (count >= write_quorum).then_some(bucket))
.collect::<Vec<_>>();
buckets.sort_by(|left, right| left.name.cmp(&right.name));
Ok((buckets, topology_complete))
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
-1
View File
@@ -19,7 +19,6 @@
pub(crate) mod bitrot_self_verify;
pub(crate) mod bucket;
pub(crate) mod delete_recovery;
pub(crate) mod heal;
pub(crate) mod heal_walk;
pub(crate) mod list;
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
-128
View File
@@ -270,134 +270,6 @@ impl ECStore {
.map_err(|error| Error::other_with_context("bucket heal owner task failed", error))?
}
/// Recover hidden delete journals through the lifecycle-owned heal boundary.
pub async fn recover_delete_transactions(self: &Arc<Self>, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
Ok(self
.recover_delete_transactions_at_incarnation(bucket, None, None, opts)
.await?
.unwrap_or_else(|| HealResultItem {
heal_item_type: HealItemType::Bucket.to_string(),
bucket: bucket.to_owned(),
detail: "delete recovery: discovered=0".to_owned(),
..Default::default()
}))
}
pub async fn recover_delete_transactions_at_incarnation(
self: &Arc<Self>,
bucket: &str,
object: Option<&str>,
expected: Option<Uuid>,
opts: &HealOpts,
) -> Result<Option<HealResultItem>> {
self.recover_delete_transactions_in_scope(bucket, object, None, expected, opts)
.await
}
pub async fn recover_delete_prefix_at_incarnation(
self: &Arc<Self>,
bucket: &str,
prefix: &str,
expected: Option<Uuid>,
opts: &HealOpts,
) -> Result<Option<HealResultItem>> {
self.recover_delete_transactions_in_scope(bucket, None, Some(prefix), expected, opts)
.await
}
async fn recover_delete_transactions_in_scope(
self: &Arc<Self>,
bucket: &str,
object: Option<&str>,
prefix: Option<&str>,
expected: Option<Uuid>,
opts: &HealOpts,
) -> Result<Option<HealResultItem>> {
if opts.no_lock && !opts.dry_run {
return Err(StorageError::InvalidArgument(
"heal".to_string(),
"no_lock".to_string(),
"delete recovery requires namespace locking".to_string(),
));
}
let expected = match expected {
Some(expected) => expected,
None => self.bucket_incarnation_id_from_disk(bucket).await?,
};
let object = object.map(str::to_owned);
let prefix = prefix.map(str::to_owned);
self.run_bucket_heal_at_incarnation(bucket, expected, opts, |store, bucket, opts| async move {
let scope = bucket_heal_scope(&bucket).ok_or(StorageError::PreconditionFailed)?;
let explicit_delete = opts.remove
&& metadata_sys::has_authoritative_never_versioned_state_in(&store.ctx, &bucket).await?
&& matches!(
metadata_sys::get_object_lock_config_state_in(&store.ctx, &bucket).await?,
metadata_sys::ObjectLockConfigState::ConfirmedAbsent
);
let movement_gate = store.ctx.data_movement_operation_gate();
let _movement_guard = movement_gate.read().await;
let can_commit = || scope.check().is_ok();
let recovery_context = crate::set_disk::DeleteRecoveryContext {
dry_run: opts.dry_run,
explicit_delete,
namespace_owner: (!opts.dry_run)
.then(|| Arc::new((scope.clone(), store.ctx.begin_namespace_commit())) as Arc<dyn Send + Sync>),
..crate::set_disk::DeleteRecoveryContext::new(Some(scope.incarnation), &can_commit)
};
let mut summary = crate::set_disk::DeleteRecoverySummary::default();
for pool in store.get_pools_for_heal_object(&opts)? {
if store.is_suspended(pool.pool_idx).await || store.is_pool_rebalancing(pool.pool_idx).await {
return Err(StorageError::SlowDown);
}
for (set_index, set) in pool.disk_set.iter().enumerate() {
if opts.set.is_some_and(|requested| requested != set_index) {
continue;
}
if let Some(object) = &object {
if !Arc::ptr_eq(set, &pool.get_disks_by_key(object)) {
continue;
}
summary.add(set.recover_delete_object(&bucket, object, recovery_context.clone()).await?);
} else if let Some(prefix) = &prefix {
summary.add(set.recover_delete_prefix(&bucket, prefix, recovery_context.clone()).await?);
} else {
summary.add(set.recover_delete_bucket(&bucket, recovery_context.clone()).await?);
}
}
}
if summary.pending > 0 {
return Err(StorageError::ErasureReadQuorum);
}
if object.is_some() && !opts.dry_run && summary.blocked > 0 {
return Err(StorageError::SlowDown);
}
if summary.discovered == 0 {
return Ok(None);
}
Ok(Some(HealResultItem {
heal_item_type: if object.is_some() {
HealItemType::Object
} else {
HealItemType::Bucket
}
.to_string(),
object: object.or(prefix).unwrap_or_default(),
bucket,
detail: format!(
"delete recovery{}: discovered={}, committed={}, rolled_back={}, reclaimed={}, blocked={}",
if opts.dry_run { " (dry run)" } else { "" },
summary.discovered,
summary.committed,
summary.rolled_back,
summary.reclaimed,
summary.blocked
),
..Default::default()
}))
})
.await
}
pub async fn heal_bucket_at_incarnation(
self: &Arc<Self>,
bucket: &str,
File diff suppressed because it is too large Load Diff
-233
View File
@@ -134,18 +134,6 @@ pub(crate) async fn connect_load_init_formats_with_instance_ctx(
{
return Err(Error::CorruptedFormat);
}
if matches!(format_quorum, Some(Err(Error::ErasureReadQuorum)))
&& let Some((format, members)) = select_existing_format_read_quorum(&formats, &errs, set_drive_count)
{
retain_format_quorum_members(instance_ctx, disks, &format, &members, set_drive_count).await?;
// Loading an existing format grants no authority to initialize pool
// metadata or migrate formats. The persisted storage classes and each
// object's EC metadata still determine service and operation quorums.
return Ok(LoadedFormat {
format,
pool_meta_bootstrap_authority: PoolMetaBootstrapAuthority::None,
});
}
let resumable_partial_migration = formats_present > 0
&& errs.iter().any(|error| matches!(error, Some(DiskError::UnformattedDisk)))
&& format_quorum.as_ref().is_some_and(Result::is_err)
@@ -590,43 +578,6 @@ pub fn get_format_erasure_in_quorum(formats: &[Option<FormatV3>], slot_offset: u
select_format_erasure_in_quorum(formats, slot_offset).map(|(format, _)| format)
}
/// Load a consistent existing layout with enough members per set to read the
/// max-parity system metadata containing the persisted storage classes. This
/// is not object-read admission: STANDARD, RRS and historical objects can have
/// different EC layouts, which are validated after this bootstrap read.
/// Format migration and healing still use strict-majority election.
fn select_existing_format_read_quorum(
formats: &[Option<FormatV3>],
errors: &[Option<DiskError>],
set_drive_count: usize,
) -> Option<(FormatV3, Vec<bool>)> {
if set_drive_count == 0 || formats.is_empty() || formats.len() != errors.len() {
return None;
}
let metadata_parity = set_drive_count / 2;
let metadata_read_quorum = set_drive_count - metadata_parity;
let reference = formats.iter().flatten().next()?;
if !formats_match_reference_slots(formats, reference, 0)
|| check_format_erasure_value_for_topology(reference, formats.len(), set_drive_count).is_err()
|| formats.iter().zip(errors).any(|(format, error)| match (format, error) {
(Some(_), None) => false,
(None, Some(DiskError::DiskNotFound)) => false,
(None, Some(error)) => !is_network_like_disk_error(error),
_ => true,
})
|| formats
.chunks(set_drive_count)
.any(|set| set.iter().flatten().count() < metadata_read_quorum)
{
return None;
}
let mut format = reference.clone();
format.erasure.this = Uuid::nil();
format.disk_info = None;
Some((format, formats.iter().map(Option::is_some).collect()))
}
pub(crate) fn select_format_erasure_in_quorum(formats: &[Option<FormatV3>], slot_offset: usize) -> Result<(FormatV3, Vec<bool>)> {
let mut candidates = HashMap::new();
let formats_present = formats.iter().flatten().count();
@@ -1169,190 +1120,6 @@ mod tests {
assert!(!formats_match_reference_slots(&formats, &reference, 0));
}
fn half_pool_formats() -> (Vec<Option<FormatV3>>, Vec<Option<DiskError>>) {
let reference = FormatV3::new(6, 4);
let formats = (0..24)
.map(|slot| {
(slot % 4 < 2).then(|| {
let mut format = reference.clone();
format.erasure.this = reference.erasure.sets[slot / 4][slot % 4];
format
})
})
.collect::<Vec<_>>();
let errors = formats
.iter()
.map(|format| format.is_none().then_some(DiskError::DiskNotFound))
.collect();
(formats, errors)
}
#[test]
fn existing_format_read_quorum_follows_each_sets_metadata_layout() {
// These are system-metadata EC boundaries, independent of user-object
// storage classes and of how many drives are hosted by each node.
for (set_width, read_quorum) in [(1, 1), (2, 1), (3, 2), (4, 2), (6, 3), (8, 4), (12, 6), (16, 8)] {
for set_count in [1, 2, 5] {
let reference = FormatV3::new(set_count, set_width);
let mut formats = (0..set_count * set_width)
.map(|slot| {
let set = slot / set_width;
let drive = slot % set_width;
// Rotate surviving slots between sets so admission
// cannot depend on the first drive or node being up.
((drive + set) % set_width < read_quorum).then(|| {
let mut format = reference.clone();
format.erasure.this = reference.erasure.sets[set][drive];
format
})
})
.collect::<Vec<_>>();
let mut errors = formats
.iter()
.map(|format| format.is_none().then_some(DiskError::DiskNotFound))
.collect::<Vec<_>>();
let (loaded, members) = select_existing_format_read_quorum(&formats, &errors, set_width)
.unwrap_or_else(|| panic!("metadata read quorum must load {set_count} sets of {set_width} drives"));
assert_eq!(loaded.shared_identity(), reference.shared_identity());
assert!(loaded.erasure.this.is_nil());
assert_eq!(members.iter().filter(|member| **member).count(), set_count * read_quorum);
// More members in other sets cannot compensate for one set
// falling below its own metadata reconstruction boundary.
for slot in 0..(set_count - 1) * set_width {
let mut format = reference.clone();
format.erasure.this = reference.erasure.sets[slot / set_width][slot % set_width];
formats[slot] = Some(format);
errors[slot] = None;
}
let lost_slot = formats
.iter()
.rposition(Option::is_some)
.expect("at least one surviving drive");
formats[lost_slot] = None;
errors[lost_slot] = Some(DiskError::DiskNotFound);
assert!(
select_existing_format_read_quorum(&formats, &errors, set_width).is_none(),
"one set below read quorum must block {set_count} sets of {set_width} drives"
);
}
}
}
#[test]
fn existing_format_read_quorum_preserves_election_and_set_boundaries() {
let (mut formats, mut errors) = half_pool_formats();
assert!(select_existing_format_read_quorum(&[], &[], 0).is_none());
assert!(select_existing_format_read_quorum(&formats, &errors[..23], 4).is_none());
assert!(select_existing_format_read_quorum(&formats, &errors, 0).is_none());
assert!(select_existing_format_read_quorum(&formats, &errors, 6).is_none());
assert!(select_existing_format_read_quorum(&formats, &errors, 4).is_some());
assert!(matches!(get_format_erasure_in_quorum(&formats, 0), Err(Error::ErasureReadQuorum)));
let mut extra = formats[0].as_ref().expect("seed format").clone();
extra.erasure.this = extra.erasure.sets[0][2];
formats[2] = Some(extra);
errors[2] = None;
assert!(get_format_erasure_in_quorum(&formats, 0).is_ok(), "13 votes must establish a majority");
formats[5] = None;
errors[5] = Some(DiskError::DiskNotFound);
assert!(
select_existing_format_read_quorum(&formats, &errors, 4).is_none(),
"12 votes cannot compensate for a set with only one readable drive"
);
formats[2] = None;
errors[2] = Some(DiskError::DiskNotFound);
assert!(select_existing_format_read_quorum(&formats, &errors, 4).is_none(), "11 votes must wait");
}
#[test]
fn existing_format_read_quorum_rejects_conflicts_invalid_slots_and_blank_disks() {
for failure in [
"conflict",
"wrong_slot",
"nil",
"duplicate",
"blank",
"corrupt",
"access_denied",
"tie",
] {
let (mut formats, mut errors) = half_pool_formats();
match failure {
"conflict" => formats[1].as_mut().expect("second format").id = Uuid::new_v4(),
"wrong_slot" => {
let first_id = formats[0].as_ref().expect("first format").erasure.this;
formats[1].as_mut().expect("second format").erasure.this = first_id;
}
"nil" => {
for format in formats.iter_mut().flatten() {
format.id = Uuid::nil();
}
}
"duplicate" => {
for format in formats.iter_mut().flatten() {
format.erasure.sets[5][3] = format.erasure.sets[0][0];
}
}
"blank" => errors[2] = Some(DiskError::UnformattedDisk),
"corrupt" => errors[2] = Some(DiskError::CorruptedFormat),
"access_denied" => errors[2] = Some(DiskError::FileAccessDenied),
"tie" => {
let other = FormatV3::new(6, 4);
for slot in 0..24 {
if formats[slot].is_none() {
let mut format = other.clone();
format.erasure.this = other.erasure.sets[slot / 4][slot % 4];
formats[slot] = Some(format);
errors[slot] = None;
}
}
}
_ => unreachable!(),
}
assert!(
select_existing_format_read_quorum(&formats, &errors, 4).is_none(),
"{failure} must not authorize degraded startup"
);
}
}
#[tokio::test]
async fn existing_format_load_accepts_each_half_of_four_by_six_without_format_writes() {
let (_root, all_disks) = local_disks_with_set_width(24, 4).await;
let reference = FormatV3::new(6, 4);
for (slot, disk) in all_disks.iter().enumerate() {
let mut format = reference.clone();
format.erasure.this = reference.erasure.sets[slot / 4][slot % 4];
save_format_file(disk, &Some(format)).await.expect("persist existing format");
}
let before = format_bytes(&all_disks).await;
for first in 0..4 {
for second in first + 1..4 {
let mut disks = all_disks.clone();
for (slot, disk) in disks.iter_mut().enumerate() {
if slot % 4 != first && slot % 4 != second {
*disk = None;
}
}
let loaded = connect_load_init_formats_with_instance_ctx(
&Arc::new(InstanceContext::new()),
first == 0,
&mut disks,
6,
4,
None,
)
.await
.expect("two nodes must load the existing format");
assert_eq!(loaded.format.shared_identity(), reference.shared_identity());
assert!(matches!(loaded.pool_meta_bootstrap_authority, PoolMetaBootstrapAuthority::None));
assert_eq!(disks.iter().flatten().count(), 12);
assert_eq!(format_bytes(&all_disks).await, before, "read startup must never rewrite formats");
}
}
}
#[tokio::test]
async fn existing_format_load_succeeds_with_a_strict_majority() {
let (_temp_dir, mut disks) = two_local_disks_with_missing_third().await;
+1 -1
View File
@@ -9221,7 +9221,7 @@ mod test {
};
let snapshot = PersistentListMetadataObject::from_object_info(&object);
assert!(snapshot.to_object_info("bucket").etag.is_none());
let mut old_snapshot = snapshot;
let mut old_snapshot = snapshot.clone();
old_snapshot.etag = object.etag.clone();
let old_contents = format!(
"# rustfs-listobjects-key-only-v2\n{}\n",
+10 -272
View File
@@ -4849,9 +4849,7 @@ impl ECStore {
opts: ObjectOptions,
guards: impl IntoIterator<Item = rustfs_lock::NamespaceLockGuard>,
) -> Result<ObjectInfo> {
let guards = Arc::new(guards.into_iter().collect::<Vec<_>>());
let mut opts = opts;
opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guards.clone())));
let guards: Vec<_> = guards.into_iter().collect();
let store = Arc::clone(self);
let bucket = bucket.to_owned();
let object = object.to_owned();
@@ -4863,7 +4861,7 @@ impl ECStore {
if result.is_ok() {
list_objects::observe_list_objects_mutation(&store, &bucket).await;
}
// Transient options also retain these guards through late disk I/O.
// Keep: the only use of `guards` here, it moves them into the task so the locks outlive a dropped caller.
drop(guards);
result
})
@@ -4932,7 +4930,7 @@ impl ECStore {
let mut opts = opts;
let receipt_sink = install_tier_free_version_receipt_sink(&mut opts);
let result = self
.handle_delete_object_with_journal_inner_boxed(bucket, object, opts, tier_journal_api)
.handle_delete_object_with_journal_inner(bucket, object, opts, tier_journal_api)
.await;
enqueue_recorded_tier_free_versions(self, receipt_sink).await;
result
@@ -4940,16 +4938,6 @@ impl ECStore {
.await
}
fn handle_delete_object_with_journal_inner_boxed<'a>(
&'a self,
bucket: &'a str,
object: &'a str,
opts: ObjectOptions,
tier_journal_api: Option<Arc<ECStore>>,
) -> impl std::future::Future<Output = Result<ObjectInfo>> + Send + 'a {
Box::pin(self.handle_delete_object_with_journal_inner(bucket, object, opts, tier_journal_api))
}
async fn handle_delete_object_with_journal_inner(
&self,
bucket: &str,
@@ -4967,9 +4955,9 @@ impl ECStore {
if is_meta_bucketname(bucket) || (opts.delete_prefix && opts.bucket_lifecycle_lock_fence.is_some()) {
None
} else if opts.delete_prefix {
Some(Arc::new(self.acquire_bucket_lifecycle_write_lock(bucket).await?))
Some(self.acquire_bucket_lifecycle_write_lock(bucket).await?)
} else {
Some(Arc::new(self.acquire_bucket_lifecycle_read_lock(bucket).await?))
Some(self.acquire_bucket_lifecycle_read_lock(bucket).await?)
};
let object = if opts.delete_prefix && !opts.delete_prefix_object {
object.to_owned()
@@ -4987,7 +4975,6 @@ impl ECStore {
opts.tier_delete_journal_api = tier_journal_api.clone();
if let Some(guard) = _bucket_lifecycle_guard.as_ref() {
opts.add_bucket_lifecycle_lock_guard(guard);
opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guard.clone())));
}
if !is_meta_bucketname(bucket) {
@@ -5240,40 +5227,6 @@ impl ECStore {
return Ok(obj);
}
Err(err) => {
if (is_err_object_not_found(&err) || is_err_version_not_found(&err))
&& (opts.version_id.is_some() || (!opts.versioned && !opts.version_suspended))
&& opts.http_preconditions.is_none()
&& !opts.replication_request
{
let mut reclaimed = None;
for pool in &self.pools {
if self.is_suspended(pool.pool_idx).await || self.is_pool_rebalancing(pool.pool_idx).await {
return Err(StorageError::SlowDown);
}
let pool_idx = pool.pool_idx;
let pool = pool.clone();
match self
.run_external_decommission_capacity_object_delete(
pool_idx,
bucket,
object,
object,
opts.clone(),
|opts| async move { pool.delete_object(bucket, object, opts).await },
)
.await
{
Ok(result) => reclaimed = Some(result),
Err(missing) if is_err_object_not_found(&missing) || is_err_version_not_found(&missing) => {}
Err(error) => return Err(error),
}
}
if let Some(mut result) = reclaimed {
result.bucket = bucket.to_owned();
result.name = decode_dir_object(object);
return Ok(result);
}
}
// A folder key (`prefix/`) with no object metadata may still exist on
// disk as an orphan empty-directory tree (issue #4189): listings show
// it as a common prefix, but no regular delete path can remove it.
@@ -5469,13 +5422,12 @@ impl ECStore {
None
} else {
match self.acquire_bucket_lifecycle_read_lock(bucket).await {
Ok(guard) => Some(Arc::new(guard)),
Ok(guard) => Some(guard),
Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err),
}
};
if let Some(guard) = _bucket_lifecycle_guard.as_ref() {
opts.add_bucket_lifecycle_lock_guard(guard);
opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guard.clone())));
}
if opts.delete_replication_config_snapshot.is_none() {
match ReplicationObjectBridge::delete_request_config_in(&self.ctx, bucket).await {
@@ -5541,11 +5493,10 @@ impl ECStore {
if current_bucket_incarnation_id.is_some() {
pause_delete_after_object_lock_snapshot(bucket).await;
}
let _object_lock_guards = Arc::new(match self.acquire_delete_objects_write_locks(bucket, &objects, &mut opts).await {
let _object_lock_guards = match self.acquire_delete_objects_write_locks(bucket, &objects, &mut opts).await {
Ok(guards) => guards,
Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err),
});
opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), _object_lock_guards.clone())));
};
#[cfg(any(test, feature = "test-util"))]
if !_object_lock_guards.is_empty() {
notify_delete_namespace_acquired(bucket);
@@ -9290,31 +9241,8 @@ mod tests {
.expect("read original marker");
assert_eq!(info.delete_marker_incarnation(), Some(old_incarnation));
}
// Save each stale member's inline versions before converging the old
// bucket's delete receipts, then reintroduce them after recreation.
let mut stale_metadata = Vec::new();
for disk in before.iter().skip(12).flatten() {
for version in &historical {
let info = disk
.read_version(
"",
bucket,
"history.bin",
&version.to_string(),
&crate::disk::ReadOptions {
read_data: true,
..Default::default()
},
)
.await
.expect("read stale inline data version");
assert!(info.inline_data() && info.data.is_some(), "fixture payload must be preserved in xl.meta");
}
stale_metadata.push((
disk.read_all(bucket, "history.bin/xl.meta").await.unwrap(),
disk.read_all(bucket, &format!("{object}/xl.meta")).await.unwrap(),
));
}
// Taking four actual members offline preserves their existing xl.meta.
// All deletion/recreation work below goes through the normal store API.
for slot in 12..16 {
set.disks.write().await[slot] = None;
}
@@ -9344,29 +9272,6 @@ mod tests {
)
.await
.expect("delete exact old marker with four drives offline");
let pending = store
.handle_delete_bucket(bucket, &crate::storage_api_contracts::bucket::DeleteBucketOptions::default())
.await
.expect_err("offline participants require retained delete receipts");
assert!(matches!(
pending,
StorageError::BucketNotEmpty(_) | StorageError::BucketNotEmptyWithDetails { .. }
));
assert!(
!crate::bucket::retirement::is_retired(store.clone(), bucket, old_incarnation)
.await
.unwrap()
);
*set.disks.write().await = before.clone();
store
.recover_delete_transactions_at_incarnation(
bucket,
None,
Some(old_incarnation),
&rustfs_heal_contracts::heal_channel::HealOpts::default(),
)
.await
.expect("converge old bucket deletes on all participants");
store
.handle_delete_bucket(bucket, &crate::storage_api_contracts::bucket::DeleteBucketOptions::default())
.await
@@ -9376,9 +9281,6 @@ mod tests {
.await
.expect("durable retirement")
);
for slot in 12..16 {
set.disks.write().await[slot] = None;
}
store
.handle_make_bucket(bucket, &create)
.await
@@ -9404,13 +9306,6 @@ mod tests {
.expect("write new generation");
new_versions.push((key, info.version_id.unwrap(), payload));
}
for (slot, (history, marker)) in (12..16).zip(stale_metadata) {
for (key, bytes) in [("history.bin", history), (object, marker)] {
let root = dirs[slot].path().join(bucket).join(key);
tokio::fs::create_dir_all(&root).await.unwrap();
tokio::fs::write(root.join("xl.meta"), bytes).await.unwrap();
}
}
*set.disks.write().await = before.clone();
let heal = rustfs_heal_contracts::heal_channel::HealOpts {
remove: true,
@@ -11582,160 +11477,3 @@ mod tests {
.await;
}
}
#[cfg(test)]
mod missing_key_delete_quorum_tests {
use super::*;
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::bucket::{BucketOperations, BucketOptions, MakeBucketOptions};
async fn degraded_store(bucket: &str) -> (Vec<tempfile::TempDir>, Arc<ECStore>) {
let (dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create authoritative never-versioned bucket");
store.pools[0].disk_set[0].disks.write().await[0] = None;
let info = store
.get_bucket_info(bucket, &BucketOptions::default())
.await
.expect("existing bucket must remain readable at quorum");
assert_eq!(info.name, bucket);
let lookup = store
.get_object_info_for_delete(bucket, "never-existed", &ObjectOptions::default())
.await;
assert!(
lookup.as_ref().is_err_and(is_err_object_not_found),
"remaining disks must establish absence through production lookup: {lookup:?}"
);
(dirs, store)
}
#[tokio::test]
async fn single_delete_missing_key_preserves_quorum_noop() {
let (_dirs, store) = degraded_store("missing-key-quorum-single").await;
let result = store
.delete_object_with_tier_delete_journal("missing-key-quorum-single", "never-existed", ObjectOptions::default())
.await;
assert!(
result.as_ref().is_err_and(is_err_object_not_found),
"single missing-key delete must preserve its idempotent not-found outcome at quorum: {result:?}"
);
}
#[tokio::test]
async fn batch_delete_missing_key_preserves_quorum_noop() {
let (_dirs, store) = degraded_store("missing-key-quorum-batch").await;
let (_, errors, _) = store
.delete_objects_with_tier_delete_journal_and_accounting(
"missing-key-quorum-batch",
vec![ObjectToDelete {
object_name: "never-existed".to_owned(),
..Default::default()
}],
ObjectOptions::default(),
)
.await;
assert_eq!(errors.len(), 1, "batch DELETE must return one result per key");
assert!(
errors.iter().all(Option::is_none),
"batch missing-key delete must preserve its idempotent success at quorum: {errors:?}"
);
}
#[cfg(not(windows))]
#[tokio::test]
async fn batch_delete_timeout_retains_physical_key_guard() {
use crate::disk::os::prepared_publication_test_hooks as hooks;
use crate::disk::{DiskAPI, ReadOptions};
use crate::storage_api_contracts::object::ObjectIO;
temp_env::async_with_vars([("RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS", Some("30"))], async {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "review-batch-late-owner";
let key = "object";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("fixture bucket");
let mut reader = PutObjReader::from_vec(vec![0x51; 1024 * 1024]);
store
.put_object(
bucket,
key,
&mut reader,
&ObjectOptions {
write_completion: crate::object_api::WriteCompletion::TailDrained,
..Default::default()
},
)
.await
.expect("seed real external object");
let disks = store.pools[0].disk_set[0].disk_inventory().await;
let disk = disks[0].as_ref().expect("online drive");
let info = disk
.read_version("", bucket, key, "", &ReadOptions::default())
.await
.expect("seed metadata");
assert!(!info.inline_data());
let crate::disk::Disk::Local(local) = disk.as_ref() else {
panic!("local disk");
};
let part = local
.get_disk()
.get_object_path_for_io(bucket, key)
.expect("leased key")
.join(info.data_dir.expect("external data directory").to_string())
.join("part.1");
let (entered_tx, entered_rx) = tokio::sync::oneshot::channel();
let (release, release_rx) = std::sync::mpsc::channel::<()>();
let hook = hooks::install_at(hooks::Stage::Remove, &part, move || {
let _ = entered_tx.send(());
let _ = release_rx.recv();
});
let deleting = store.clone();
let delete = tokio::spawn(async move {
deleting
.delete_objects_with_tier_delete_journal_and_accounting(
bucket,
vec![ObjectToDelete {
object_name: key.to_owned(),
..Default::default()
}],
ObjectOptions::default(),
)
.await
});
tokio::time::timeout(Duration::from_secs(10), entered_rx)
.await
.expect("physical cleanup entry")
.expect("entry hook");
assert!(part.exists(), "payload unlink has not run");
let (_, errors, _) = tokio::time::timeout(Duration::from_secs(40), delete)
.await
.expect("real settlement timeout must return")
.expect("delete task");
assert!(errors.iter().all(Option::is_none), "certified batch response: {errors:?}");
assert!(
store.ctx.namespace_commits_pending(),
"background physical operation must retain its owner"
);
let ns = store.handle_new_ns_lock(bucket, key).await.expect("same production key lock");
let probe = ns.get_write_lock(Duration::from_millis(300)).await;
let acquired_while_pending = probe.is_ok();
drop(probe);
drop(release);
drop(hook);
assert!(
!acquired_while_pending,
"batch released the exact-key guard while physical cleanup was still paused after its real recovery timeout"
);
let _released = ns
.get_write_lock(Duration::from_secs(10))
.await
.expect("physical cleanup must eventually release the exact-key guard");
})
.await;
}
}
@@ -1,497 +0,0 @@
// Copyright 2026 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Byte-level goldens for the 13 persisted bucket-configuration XML families
//! (rustfs/backlog#2744).
//!
//! Responsible for: pinning the exact bytes that `deserialize` followed by
//! `serialize` produces for every family in its three source shapes (AWS
//! documentation shape, MinIO-written shape, RustFS-written shape), and the
//! exact bytes RustFS writes for a typed value. Only bytes are compared; a
//! structural comparison would let a codec replacement change the persisted
//! encoding without failing here.
//!
//! Not responsible for: validating configurations, touching the bucket
//! metadata store, or choosing a canonical shape.
//!
//! Upstream: the bucket-config `deserialize`/`serialize` pair, reached through the
//! integration-test facade `tests/storage_api.rs`, and the s3s DTOs they are generic over. Downstream: the DTO/codec
//! replacement (T1.4) and the persisted-bytes comparison (T3.7), which must
//! keep every golden green without regenerating it. Regeneration is reserved
//! to `scripts/gen_bucket_config_goldens.sh`, which sets
//! `BUCKET_CONFIG_GOLDENS_WRITE`: in that mode every test writes its files
//! and then fails on purpose, so a write run never prints a green line.
mod storage_api;
use s3s::dto::{
AbortIncompleteMultipartUpload, AccelerateConfiguration, BucketAccelerateStatus, BucketLifecycleConfiguration,
BucketLoggingStatus, BucketVersioningStatus, CORSConfiguration, CORSRule, Condition, DefaultRetention, DelMarkerExpiration,
DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication, DeleteReplicationStatus, Destination,
ErrorDocument, Event, ExcludedPrefix, ExistingObjectReplication, ExistingObjectReplicationStatus, ExpirationStatus,
FilterRule, FilterRuleName, IndexDocument, LifecycleExpiration, LifecycleRule, LifecycleRuleAndOperator, LifecycleRuleFilter,
LoggingEnabled, NoncurrentVersionExpiration, NotificationConfiguration, NotificationConfigurationFilter,
ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRetentionMode, ObjectLockRule, Payer, PublicAccessBlockConfiguration,
QueueConfiguration, Redirect, ReplicaModifications, ReplicaModificationsStatus, ReplicationConfiguration, ReplicationRule,
ReplicationRuleAndOperator, ReplicationRuleFilter, ReplicationRuleStatus, RequestPaymentConfiguration, RoutingRule,
S3KeyFilter, ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration,
ServerSideEncryptionRule, SourceSelectionCriteria, StorageClass, Tag, Tagging, Transition, TransitionStorageClass,
VersioningConfiguration, WebsiteConfiguration,
};
use s3s::xml;
use std::path::{Path, PathBuf};
use storage_api::bucket_config_codec::{deserialize, serialize};
/// Set only by `scripts/gen_bucket_config_goldens.sh`. Every pair test then
/// writes its `.out.xml` (and, for the `rustfs` shape, its `.in.xml`) and
/// fails on purpose, so a write run can never be mistaken for a verification.
const WRITE_ENV: &str = "BUCKET_CONFIG_GOLDENS_WRITE";
fn goldens_dir() -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR"))
.join("tests")
.join("fixtures")
.join("bucket-config-goldens")
}
fn pair_path(family: &str, shape: &str, side: &str) -> PathBuf {
goldens_dir().join(family).join(format!("{shape}.{side}.xml"))
}
fn read(path: &Path) -> Vec<u8> {
std::fs::read(path).unwrap_or_else(|err| panic!("read {}: {err}", path.display()))
}
fn write(path: &Path, bytes: &[u8]) {
std::fs::write(path, bytes).unwrap_or_else(|err| panic!("write {}: {err}", path.display()));
}
fn contains(haystack: &[u8], needle: &[u8]) -> bool {
haystack.windows(needle.len()).any(|window| window == needle)
}
fn round_trip<T>(input: &[u8], label: &str) -> Vec<u8>
where
T: for<'xml> xml::Deserialize<'xml> + xml::Serialize,
{
let parsed: T = deserialize(input).unwrap_or_else(|err| panic!("{label}: deserialize failed: {err}"));
serialize(&parsed).unwrap_or_else(|err| panic!("{label}: serialize failed: {err}"))
}
/// Byte comparison with a readable report. Parsed values are never compared.
fn assert_bytes_eq(actual: &[u8], expected: &[u8], context: &str) {
if actual != expected {
panic!(
"{context}\n--- expected ({} bytes) ---\n{}\n--- actual ({} bytes) ---\n{}\n",
expected.len(),
String::from_utf8_lossy(expected),
actual.len(),
String::from_utf8_lossy(actual),
);
}
}
/// Checks one `<family>/<shape>` pair. `typed` is the value RustFS writes for
/// the `rustfs` shape; the `aws` and `minio` shapes pass `None`.
fn check_pair<T>(family: &str, shape: &str, typed: Option<&T>)
where
T: for<'xml> xml::Deserialize<'xml> + xml::Serialize,
{
let in_path = pair_path(family, shape, "in");
let out_path = pair_path(family, shape, "out");
let label = format!("{family}/{shape}");
if std::env::var_os(WRITE_ENV).is_some() {
if let Some(value) = typed {
let written = serialize(value).unwrap_or_else(|err| panic!("{label}: typed value failed to serialize: {err}"));
write(&in_path, &written);
}
let input = read(&in_path);
write(&out_path, &round_trip::<T>(&input, &label));
panic!(
"{WRITE_ENV} is set: wrote {} without verifying anything; rerun without {WRITE_ENV}",
out_path.display()
);
}
let input = read(&in_path);
let expected = read(&out_path);
if let Some(value) = typed {
let written = serialize(value).unwrap_or_else(|err| panic!("{label}: typed value failed to serialize: {err}"));
assert_bytes_eq(
&written,
&input,
&format!(
"{label}: RustFS no longer writes the pinned bytes for the typed value ({})",
in_path.display()
),
);
}
let actual = round_trip::<T>(&input, &label);
assert_bytes_eq(
&actual,
&expected,
&format!("{label}: deserialize->serialize bytes drifted from {}", out_path.display()),
);
let again = round_trip::<T>(&expected, &label);
assert_bytes_eq(
&again,
&expected,
&format!("{label}: the golden is not a fixed point of deserialize->serialize"),
);
}
/// The typed values behind every `rustfs.in.xml`: what RustFS writes today
/// through `serialize` when it persists a configuration it built itself.
/// MinIO-only members (`DelMarkerExpiration`, `ExcludeFolders`,
/// `ExcludedPrefixes`, `DeleteReplication`) are included where RustFS accepts
/// them, so the `rustfs` shape is the one that pins their encoding.
mod rustfs_values {
use super::*;
fn tag(key: &str, value: &str) -> Tag {
Tag {
key: Some(key.to_owned()),
value: Some(value.to_owned()),
}
}
pub(super) fn notification() -> NotificationConfiguration {
NotificationConfiguration {
queue_configurations: Some(vec![QueueConfiguration {
events: vec![
Event::from("s3:ObjectCreated:*".to_owned()),
Event::from("s3:ObjectRemoved:*".to_owned()),
],
filter: Some(NotificationConfigurationFilter {
key: Some(S3KeyFilter {
filter_rules: Some(vec![FilterRule {
name: Some(FilterRuleName::from_static(FilterRuleName::PREFIX)),
value: Some("uploads/".to_owned()),
}]),
}),
}),
id: Some("rustfs-webhook".to_owned()),
queue_arn: "arn:rustfs:sqs::primary:webhook".to_owned(),
}]),
..Default::default()
}
}
pub(super) fn lifecycle() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
rules: vec![
LifecycleRule {
abort_incomplete_multipart_upload: Some(AbortIncompleteMultipartUpload {
days_after_initiation: Some(3),
}),
del_marker_expiration: Some(DelMarkerExpiration { days: Some(7) }),
expiration: Some(LifecycleExpiration {
days: Some(30),
..Default::default()
}),
filter: Some(LifecycleRuleFilter {
prefix: Some("tmp/".to_owned()),
..Default::default()
}),
id: Some("expire-tmp".to_owned()),
noncurrent_version_expiration: Some(NoncurrentVersionExpiration {
newer_noncurrent_versions: Some(2),
noncurrent_days: Some(7),
}),
noncurrent_version_transitions: None,
prefix: None,
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
transitions: None,
},
LifecycleRule {
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
expiration: None,
filter: Some(LifecycleRuleFilter {
and: Some(LifecycleRuleAndOperator {
prefix: Some("archive/".to_owned()),
tags: Some(vec![tag("tier", "cold")]),
..Default::default()
}),
..Default::default()
}),
id: Some("tier-cold".to_owned()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
transitions: Some(vec![Transition {
days: Some(90),
storage_class: Some(TransitionStorageClass::from_static("COLDTIER")),
..Default::default()
}]),
},
],
..Default::default()
}
}
pub(super) fn object_lock() -> ObjectLockConfiguration {
ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
rule: Some(ObjectLockRule {
default_retention: Some(DefaultRetention {
days: Some(30),
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::GOVERNANCE)),
years: None,
}),
}),
}
}
pub(super) fn versioning() -> VersioningConfiguration {
VersioningConfiguration {
exclude_folders: Some(true),
excluded_prefixes: Some(vec![
ExcludedPrefix {
prefix: Some("tmp/".to_owned()),
},
ExcludedPrefix {
prefix: Some("cache/".to_owned()),
},
]),
mfa_delete: None,
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
}
}
pub(super) fn encryption() -> ServerSideEncryptionConfiguration {
ServerSideEncryptionConfiguration {
rules: vec![ServerSideEncryptionRule {
apply_server_side_encryption_by_default: Some(ServerSideEncryptionByDefault {
kms_master_key_id: Some("rustfs-bucket-key".to_owned()),
sse_algorithm: ServerSideEncryption::from_static(ServerSideEncryption::AWS_KMS),
}),
blocked_encryption_types: None,
bucket_key_enabled: Some(false),
}],
}
}
pub(super) fn tagging() -> Tagging {
Tagging {
tag_set: vec![tag("team", "storage"), tag("env", "staging")],
}
}
pub(super) fn replication() -> ReplicationConfiguration {
ReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
delete_marker_replication: Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
}),
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
}),
destination: Destination {
bucket: "arn:rustfs:replication::0b6f3c2a-5e6a-4d2b-9c1e-7a8f9d0e1b2c:interop-dr".to_owned(),
storage_class: Some(StorageClass::from_static(StorageClass::STANDARD)),
..Default::default()
},
existing_object_replication: Some(ExistingObjectReplication {
status: ExistingObjectReplicationStatus::from_static(ExistingObjectReplicationStatus::ENABLED),
}),
filter: Some(ReplicationRuleFilter {
and: Some(ReplicationRuleAndOperator {
prefix: Some("docs/".to_owned()),
tags: Some(vec![tag("replicate", "yes")]),
}),
..Default::default()
}),
id: Some("rustfs-dr".to_owned()),
prefix: None,
priority: Some(1),
source_selection_criteria: Some(SourceSelectionCriteria {
replica_modifications: Some(ReplicaModifications {
status: ReplicaModificationsStatus::from_static(ReplicaModificationsStatus::ENABLED),
}),
sse_kms_encrypted_objects: None,
}),
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
}],
}
}
pub(super) fn cors() -> CORSConfiguration {
CORSConfiguration {
cors_rules: vec![CORSRule {
allowed_headers: Some(vec!["Authorization".to_owned(), "Content-Type".to_owned()]),
allowed_methods: vec!["GET".to_owned(), "PUT".to_owned()],
allowed_origins: vec!["https://app.example.internal".to_owned()],
expose_headers: Some(vec!["ETag".to_owned(), "x-amz-request-id".to_owned()]),
id: Some("rustfs-app".to_owned()),
max_age_seconds: Some(3600),
}],
}
}
pub(super) fn logging() -> BucketLoggingStatus {
BucketLoggingStatus {
logging_enabled: Some(LoggingEnabled {
target_bucket: "rustfs-access-logs".to_owned(),
target_grants: None,
target_object_key_format: None,
target_prefix: "buckets/interop/".to_owned(),
}),
}
}
pub(super) fn website() -> WebsiteConfiguration {
WebsiteConfiguration {
error_document: Some(ErrorDocument {
key: "404.html".to_owned(),
}),
index_document: Some(IndexDocument {
suffix: "index.html".to_owned(),
}),
redirect_all_requests_to: None,
routing_rules: Some(vec![RoutingRule {
condition: Some(Condition {
http_error_code_returned_equals: None,
key_prefix_equals: Some("legacy/".to_owned()),
}),
redirect: Redirect {
host_name: None,
http_redirect_code: Some("301".to_owned()),
protocol: None,
replace_key_prefix_with: Some("current/".to_owned()),
replace_key_with: None,
},
}]),
}
}
pub(super) fn accelerate() -> AccelerateConfiguration {
AccelerateConfiguration {
status: Some(BucketAccelerateStatus::from_static(BucketAccelerateStatus::SUSPENDED)),
}
}
pub(super) fn request_payment() -> RequestPaymentConfiguration {
RequestPaymentConfiguration {
payer: Payer::from_static(Payer::BUCKET_OWNER),
}
}
pub(super) fn public_access_block() -> PublicAccessBlockConfiguration {
PublicAccessBlockConfiguration {
block_public_acls: Some(true),
block_public_policy: Some(false),
ignore_public_acls: Some(true),
restrict_public_buckets: Some(false),
}
}
}
/// One test per `<family>/<shape>` pair: 13 families x 3 shapes = 39.
mod bucket_config_goldens {
use super::*;
macro_rules! golden {
($name:ident, $family:literal, $shape:literal, $ty:ty) => {
#[test]
fn $name() {
check_pair::<$ty>($family, $shape, None);
}
};
($name:ident, $family:literal, rustfs = $value:expr) => {
#[test]
fn $name() {
check_pair($family, "rustfs", Some(&$value));
}
};
}
golden!(notification_aws, "notification", "aws", NotificationConfiguration);
golden!(notification_minio, "notification", "minio", NotificationConfiguration);
golden!(notification_rustfs, "notification", rustfs = rustfs_values::notification());
golden!(lifecycle_aws, "lifecycle", "aws", BucketLifecycleConfiguration);
golden!(lifecycle_minio, "lifecycle", "minio", BucketLifecycleConfiguration);
golden!(lifecycle_rustfs, "lifecycle", rustfs = rustfs_values::lifecycle());
golden!(object_lock_aws, "object_lock", "aws", ObjectLockConfiguration);
golden!(object_lock_minio, "object_lock", "minio", ObjectLockConfiguration);
golden!(object_lock_rustfs, "object_lock", rustfs = rustfs_values::object_lock());
golden!(versioning_aws, "versioning", "aws", VersioningConfiguration);
golden!(versioning_minio, "versioning", "minio", VersioningConfiguration);
golden!(versioning_rustfs, "versioning", rustfs = rustfs_values::versioning());
golden!(encryption_aws, "encryption", "aws", ServerSideEncryptionConfiguration);
golden!(encryption_minio, "encryption", "minio", ServerSideEncryptionConfiguration);
golden!(encryption_rustfs, "encryption", rustfs = rustfs_values::encryption());
golden!(tagging_aws, "tagging", "aws", Tagging);
golden!(tagging_minio, "tagging", "minio", Tagging);
golden!(tagging_rustfs, "tagging", rustfs = rustfs_values::tagging());
/// `replication/aws.in.xml` is the unknown-element sample from
/// `crates/replication/src/config.rs`
/// (`s3_xml_parser_discards_unknown_replication_elements_before_validation`).
/// Beyond the byte golden, the pair proves the root-level `FutureTopLevel`
/// element is skipped on read and never re-emitted on write.
#[test]
fn replication_aws() {
check_pair::<ReplicationConfiguration>("replication", "aws", None);
let input = read(&pair_path("replication", "aws", "in"));
let output = read(&pair_path("replication", "aws", "out"));
assert!(
contains(&input, b"<FutureTopLevel>future</FutureTopLevel>"),
"the sample must still carry the unknown element"
);
assert!(
!contains(&output, b"FutureTopLevel"),
"an unknown element must not be re-emitted: {}",
String::from_utf8_lossy(&output)
);
}
golden!(replication_minio, "replication", "minio", ReplicationConfiguration);
golden!(replication_rustfs, "replication", rustfs = rustfs_values::replication());
golden!(cors_aws, "cors", "aws", CORSConfiguration);
golden!(cors_minio, "cors", "minio", CORSConfiguration);
golden!(cors_rustfs, "cors", rustfs = rustfs_values::cors());
golden!(logging_aws, "logging", "aws", BucketLoggingStatus);
golden!(logging_minio, "logging", "minio", BucketLoggingStatus);
golden!(logging_rustfs, "logging", rustfs = rustfs_values::logging());
golden!(website_aws, "website", "aws", WebsiteConfiguration);
golden!(website_minio, "website", "minio", WebsiteConfiguration);
golden!(website_rustfs, "website", rustfs = rustfs_values::website());
golden!(accelerate_aws, "accelerate", "aws", AccelerateConfiguration);
golden!(accelerate_minio, "accelerate", "minio", AccelerateConfiguration);
golden!(accelerate_rustfs, "accelerate", rustfs = rustfs_values::accelerate());
golden!(request_payment_aws, "request_payment", "aws", RequestPaymentConfiguration);
golden!(request_payment_minio, "request_payment", "minio", RequestPaymentConfiguration);
golden!(request_payment_rustfs, "request_payment", rustfs = rustfs_values::request_payment());
golden!(public_access_block_aws, "public_access_block", "aws", PublicAccessBlockConfiguration);
golden!(public_access_block_minio, "public_access_block", "minio", PublicAccessBlockConfiguration);
golden!(
public_access_block_rustfs,
"public_access_block",
rustfs = rustfs_values::public_access_block()
);
}
@@ -1,58 +0,0 @@
# Persisted bucket-configuration XML goldens (rustfs/backlog#2744)
Byte-level goldens for the 13 XML families that `BucketMetadata` persists as
`*_config_xml`. They pin what `rustfs_ecstore::api::bucket::utils::deserialize`
followed by `serialize` emits today, so that the DTO/codec replacement (T1.4)
and the persisted-bytes comparison (T3.7) can be proven byte-identical: an
older binary must keep reading what a newer one writes, and the reverse.
Generated at commit: `6b1554003ebf8f2037ffb7da9c9b906527e758da` (2026-10-07T17:40:56Z)
Checked by `crates/ecstore/tests/bucket_config_goldens.rs`
(`cargo nextest run -p rustfs-ecstore bucket_config_goldens`, 39 tests).
Generated once by `scripts/gen_bucket_config_goldens.sh`, which records the
commit above and refuses to run again unless `--force` is passed. Do not
regenerate after a DTO or codec replacement: at that point a diff here is the
finding, not noise.
## Layout
`<family>/<shape>.in.xml` is an input; `<family>/<shape>.out.xml` is the exact
bytes of `serialize(deserialize(in))` on the generating commit. The test
asserts `bytes == out` (never structural equality), that `out` is a fixed point
of the same round trip, and for the `rustfs` shape that `serialize(typed
value)` still equals `rustfs.in.xml`.
| Shape | Source |
| --- | --- |
| `aws` | Written for this corpus from the public S3 XML schema in the shape of the AWS documentation examples: XML declaration, `xmlns` on the root, indented. No AWS prose is copied. |
| `minio` | The bytes a MinIO server persists. Seven families are carved verbatim out of `../minio/bucket_metadata.blob.hex` (MinIO `RELEASE.2025-07-23T15-54-02Z`); the rest are hand-shaped as compact Go `encoding/xml` output with the S3 namespace on the root, because MinIO does not persist those families. |
| `rustfs` | The bytes RustFS writes today through `serialize` for a typed value (`rustfs_values` in the test). This shape carries the MinIO-only members RustFS accepts (`DelMarkerExpiration`, `ExcludeFolders`, `ExcludedPrefixes`, `DeleteReplication`). |
| Family | Root element | `minio.in.xml` provenance |
| --- | --- | --- |
| `notification` | `NotificationConfiguration` | MinIO fixture |
| `lifecycle` | `LifecycleConfiguration` | MinIO fixture (carries the MinIO `ExpiryUpdatedAt` extension) |
| `object_lock` | `ObjectLockConfiguration` | MinIO fixture |
| `versioning` | `VersioningConfiguration` | MinIO fixture |
| `encryption` | `ServerSideEncryptionConfiguration` | MinIO fixture |
| `tagging` | `Tagging` | MinIO fixture |
| `replication` | `ReplicationConfiguration` | MinIO fixture |
| `cors` | `CORSConfiguration` | hand-shaped |
| `logging` | `BucketLoggingStatus` | hand-shaped |
| `website` | `WebsiteConfiguration` | hand-shaped |
| `accelerate` | `AccelerateConfiguration` | hand-shaped |
| `request_payment` | `RequestPaymentConfiguration` | hand-shaped |
| `public_access_block` | `PublicAccessBlockConfiguration` | hand-shaped |
## Unknown elements
`replication/aws.in.xml` is the unknown-element sample from
`crates/replication/src/config.rs`
(`s3_xml_parser_discards_unknown_replication_elements_before_validation`),
re-indented to two spaces. Its golden proves the root-level `FutureTopLevel`
element is skipped on read and never re-emitted on write, which is the
persisted-configuration exception recorded in the root `AGENTS.md`. On the
generating commit every one of the 13 root elements skips unknown children;
nested members (`Rule`, `Filter`, `Destination`, ...) reject them. A
replacement codec must keep that boundary where it is.
@@ -1,4 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<AccelerateConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Status>Enabled</Status>
</AccelerateConfiguration>
@@ -1 +0,0 @@
<AccelerateConfiguration><Status>Enabled</Status></AccelerateConfiguration>
@@ -1 +0,0 @@
<AccelerateConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Status>Suspended</Status></AccelerateConfiguration>
@@ -1 +0,0 @@
<AccelerateConfiguration><Status>Suspended</Status></AccelerateConfiguration>
@@ -1 +0,0 @@
<AccelerateConfiguration><Status>Suspended</Status></AccelerateConfiguration>
@@ -1 +0,0 @@
<AccelerateConfiguration><Status>Suspended</Status></AccelerateConfiguration>
@@ -1,17 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<ID>AllowWebApp</ID>
<AllowedOrigin>https://www.example.com</AllowedOrigin>
<AllowedMethod>PUT</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>DELETE</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
<ExposeHeader>x-amz-server-side-encryption</ExposeHeader>
<MaxAgeSeconds>3000</MaxAgeSeconds>
</CORSRule>
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
</CORSRule>
</CORSConfiguration>
@@ -1 +0,0 @@
<CORSConfiguration><CORSRule><AllowedHeader>*</AllowedHeader><AllowedMethod>PUT</AllowedMethod><AllowedMethod>POST</AllowedMethod><AllowedMethod>DELETE</AllowedMethod><AllowedOrigin>https://www.example.com</AllowedOrigin><ExposeHeader>x-amz-server-side-encryption</ExposeHeader><ID>AllowWebApp</ID><MaxAgeSeconds>3000</MaxAgeSeconds></CORSRule><CORSRule><AllowedMethod>GET</AllowedMethod><AllowedOrigin>*</AllowedOrigin></CORSRule></CORSConfiguration>
@@ -1 +0,0 @@
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><CORSRule><ID>console</ID><AllowedOrigin>https://console.example.internal</AllowedOrigin><AllowedMethod>GET</AllowedMethod><AllowedMethod>HEAD</AllowedMethod><AllowedHeader>Authorization</AllowedHeader><AllowedHeader>Content-Type</AllowedHeader><ExposeHeader>ETag</ExposeHeader><MaxAgeSeconds>600</MaxAgeSeconds></CORSRule></CORSConfiguration>

Some files were not shown because too many files have changed in this diff Show More