Compare commits

...

38 Commits

Author SHA1 Message Date
overtrue 30c6f8a9ce fix(data-usage): make empty checks exhaustive 2026-07-29 16:11:42 +08:00
overtrue c77c9875c6 fix(data-usage): preserve nonempty replication stats 2026-07-29 15:17:50 +08:00
Zhengchao An 7d698abc1f ci(checksums): inherit workspace lint policy (#5415) 2026-07-29 12:53:45 +08:00
hector a1a65ad65d fix(action): change quay.io image repository name (#5417) 2026-07-29 12:53:26 +08:00
cxymds 358af6a8de ci: preserve timeout evidence for workspace tests (#5402)
* ci: preserve timeout evidence for workspace tests

* ci: allow cold doctest compilation to finish

* ci: allow cold nextest compilation to finish

* ci: allow cold nextest compilation to finish

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-07-29 04:49:03 +00:00
Zhengchao An 90d1a15d13 fix(ecstore): classify peer RPC failures by gRPC status code (#5400) 2026-07-29 11:35:09 +08:00
cxymds 2423ba8e3f fix(tiering): base restore expiry on completion (#5366)
* fix(tiering): base restore expiry on completion

* fix(tiering): import restore metadata types

* fix(restore): resolve metadata finalization build errors

* test(ecstore): import restore expiry helper
2026-07-29 02:55:00 +00:00
cxymds 294c79c156 fix(tiering): gate remote version state safely (#5374)
* feat(tiering): model provider version capabilities

* feat(tiering): persist opaque remote versions

* fix(tiering): gate remote version state safely

* fix(tiering): preserve remote version state on delete

* fix(tiering): accept unversioned transition responses

* fix(tiering): replay exact cleanup journals

* test(tiering): pin empty exact cleanup guard

* test(tiering): accept strict missing journal errors

* test(tiering): exercise free-version identity guard

* test(tiering): reach destination identity guard

* test(tiering): persist version identity drift

* test(tiering): bind version drift fixture

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-07-29 02:43:28 +00:00
cxymds 3d80578abd feat(ecstore): add local data-dir snapshot leases (#5388)
feat(ecstore): add local snapshot leases

Co-authored-by: houseme <housemecn@gmail.com>
2026-07-29 02:42:35 +00:00
Zhengchao An 957080bea5 fix(swift): persist container and account metadata writes (#5398)
Swift container and account metadata handlers cloned the cached
BucketMetadata, set the tagging fields, and called set_bucket_metadata,
which only updates the in-memory cache map. Nothing reached
.metadata.bin, so every Swift metadata POST was lost on restart and
silently overwritten by the next disk-truth reload (a peer
LoadBucketMetadata notification or the 15-minute refresh loop) — while
the client had already been told 2xx.

Route these writes through a new metadata_sys::update_config_with: a
read-modify-write that loads the on-disk metadata and persists the
result under the same write guard metadata_sys::update uses, so the
rewrite merges against disk truth instead of a possibly stale cache and
cannot clobber a concurrent update to another config file. Peers are
notified afterwards, matching the S3 config handlers.

Persisting these writes required hardening the paths that now produce
durable state:

- Account metadata writes validate account ownership. This metadata
  holds the account's TempURL signing key, so an unauthenticated write
  for someone else's account would have become a durable, cluster-wide
  takeover of that account's pre-signed URLs. Reads stay open because
  TempURL signature validation runs before credentials exist.
- disable_versioning verifies the container exists. Without it the
  metadata loader's "no metadata on disk" default would be persisted,
  creating an orphan metadata file and caching a fabricated default as
  authoritative.
- Container and account metadata are size- and count-limited, reusing
  the Swift limits object metadata already enforces; these tags land in
  the bucket metadata file that every later config write rewrites whole.
- A rewrite refuses to run when the persisted tagging config is
  unreadable, instead of merging onto an empty set and wiping the
  container ACL and versioning tags. It reports 409 naming the remedy.
- Storage errors are logged in full and reported generically, since they
  now carry real disk and quorum detail.

The tagging arm of BucketMetadata::update_config also clears the parsed
config, as the lifecycle arm does: parse_all_configs skips empty XML
rather than clearing, so a cleared config kept serving the old tags.
Tagging is serialized with the S3 XML serializer the loader can parse
back, not quick_xml, whose output was never round-trippable.
2026-07-29 02:31:11 +00:00
cxymds c1538cf1c3 fix(multipart): serialize complete and abort (#5356)
* fix(multipart): serialize complete and abort

* test(multipart): order abort-first finalization

* fix(multipart): enforce quorum staging cleanup

* fix(multipart): remove stale mutable binding
2026-07-29 01:49:19 +00:00
houseme 5af56cbb02 test(ci): stabilize lifecycle timeout coverage (#5404)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-29 01:37:29 +00:00
houseme f99956eade test(lifecycle): classify mixed rollout harness results (#5384)
* test(lifecycle): classify mixed rollout harness results

Classify Docker #1508 evidence as strict, baseline, blocked, or failed so tiered-storage baseline runs cannot be mistaken for strict mixed-version rollout closure evidence.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test: classify Docker manual transition preemption

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-29 01:21:23 +00:00
houseme 775279b6fd fix(notify): defer disabled bootstrap storage refresh (#5373)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-28 17:39:22 +00:00
cxymds eb755e2b97 fix(auth): compare sensitive tokens in constant time (#5371)
* fix(auth): compare sensitive tokens in constant time

* test(auth): scope constant-time source guard

* test(auth): ignore test source in comparison guard

* chore(deps): use constant-time s3s authentication
2026-07-28 17:56:22 +08:00
cxymds 7f146fc5de feat(tiering): model provider version capabilities (#5372) 2026-07-28 17:42:08 +08:00
cxymds 5426237a49 fix(filemeta): preserve canonical version order (#5353) 2026-07-28 17:38:59 +08:00
cxymds d7afa4e38e fix(heal): report partial rename failures (#5355)
* fix(heal): report partial rename failures

* fix(log-analyzer): track partial heal rename failures
2026-07-28 17:37:02 +08:00
cxymds a3f5a8eaf9 fix(ecstore): fence object tagging updates (#5360)
* fix(ecstore): fence object tagging updates

* test(ecstore): stabilize tagging lock regressions

* test(ecstore): restore setup type on current-thread runtime
2026-07-28 17:36:00 +08:00
cxymds 547c678eed fix(filemeta): reject positive size without parts (#5354)
* fix(filemeta): reject positive size without parts

* test(ecstore): keep optimized read fixture valid

* test(ecstore): keep listing fixtures valid
2026-07-28 17:35:53 +08:00
cxymds df945b275a fix(lifecycle): recover saturated scanner tasks (#5369) 2026-07-28 17:30:52 +08:00
cxymds 2e78a49c95 fix(lifecycle): reject invalid modification times (#5370) 2026-07-28 17:23:21 +08:00
cxymds 7ad0e726db fix(ecstore): use set read quorum for version reads (#5365)
* fix(ecstore): use set read quorum for version reads

* fix(ecstore): convert optimized read batch errors
2026-07-28 17:23:15 +08:00
cxymds 45c3386b68 fix(bitrot): reject trailing shard data (#5358)
* fix(bitrot): reject trailing shard data

* test(ecstore): align bitrot disk fixture type
2026-07-28 17:23:08 +08:00
cxymds 7af92b4f54 fix(ecstore): handle pre-epoch disk health clocks (#5383) 2026-07-28 17:22:26 +08:00
cxymds daa627ee0e fix(auth): enforce presigned URL expiry limit (#5368)
* fix(auth): enforce presigned URL expiry limit

* ci(deny): allow presigned expiry s3s fork

* chore(deps): update presigned expiry validation
2026-07-28 17:11:30 +08:00
cxymds 5c3d3a8220 fix(ecstore): handle mutex poisoning explicitly (#5379)
* fix(ecstore): handle mutex poisoning explicitly

* test(ecstore): satisfy poison assertion lint
2026-07-28 17:09:09 +08:00
cxymds 6bc5fc77b5 fix(notify): emit noop event for missing deletes (#5381) 2026-07-28 17:05:10 +08:00
cxymds e822fc1552 fix(swift): enforce cumulative container quotas (#5378) 2026-07-28 17:05:04 +08:00
cxymds 2f6115e058 fix(swift): enforce TempURL IP restrictions (#5377) 2026-07-28 17:04:58 +08:00
cxymds 882e1a71b1 fix(tiering): abort failed multipart restores (#5367)
* fix(tiering): abort failed multipart restores

* fix(tiering): compile restore abort regressions

* test(tiering): assert restored multipart metadata

* test(tiering): parse completed restore status
2026-07-28 17:04:52 +08:00
cxymds b432f31c2c fix(multipart): preserve retried parts on quorum failure (#5363)
* fix(multipart): preserve retried parts on quorum failure

* style(multipart): format transaction rollback

* fix(proto): regenerate multipart transaction RPCs

* fix(multipart): import rollback marker constant

* fix(multipart): export transaction action

* fix: import multipart transaction test requests
2026-07-28 17:04:46 +08:00
cxymds 6e0640444e fix(multipart): list uploads across all sets (#5362) 2026-07-28 17:04:40 +08:00
cxymds 4fb9b0dc7f fix(authz): fail closed on policy load errors (#5359)
* fix(authz): fail closed on policy load errors

* test(authz): cover policy failure precedence

* test: align policy failure expectations

* test: align upload part copy fail-closed expectation
2026-07-28 17:04:35 +08:00
唐小鸭 2216f00cfd fix(kms): unify persisted SSE data key envelopes (#5343)
* feat(kms): implement secure handling of static KMS secret keys and enhance encryption context validation

* feat: enhance local SSE DEK handling with JSON envelope format and versioning
2026-07-28 17:02:18 +08:00
cxymds fd2a87d47e fix(s3): reject empty multipart completions (#5352)
Co-authored-by: houseme <housemecn@gmail.com>
2026-07-28 08:35:50 +00:00
houseme 1e10d752b9 test(e2e): stabilize inline full gate checks (#5351)
Align inline fast path cluster tests with the current tier mutation and manual transition contracts while keeping msgpack compat assertions focused on observed traffic.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-28 04:31:19 +00:00
houseme 362f6026ac chore(build): pin Rust toolchain configs to 1.97.1 (#5350)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-28 03:53:32 +00:00
117 changed files with 9136 additions and 1435 deletions
+9 -14
View File
@@ -1,17 +1,14 @@
# nextest configuration for RustFS.
#
# Serialize two known load-sensitive / global-state-sharing ecstore test groups
# so the full parallel nextest suite stops producing spurious failures
# (backlog #937). These tests pass in isolation but flake under the loaded
# parallel run for two distinct reasons:
# Serialize the ecstore tests that share the process-wide disk registry or
# exercise a multi-disk commit handoff across nextest process boundaries.
#
# * store::bucket::tests::bucket_delete_* share process/global state (disk
# registry, lock client) and race make_bucket into InsufficientWriteQuorum
# when run concurrently with other ecstore tests.
# * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
# asserts a lock-acquire correctness property whose serialized cross-disk
# commits exceed the (already max'd, 60s) acquire deadline only when the
# suite saturates disk I/O.
# uses the shared multipart fixture and a deterministic uploadId-lock
# handoff, so it must not overlap another process mutating that fixture.
#
# serial_test's #[serial] attribute does NOT serialize these across runs:
# nextest executes each test in its own process, where the in-process
@@ -100,13 +97,6 @@ path = "junit.xml"
# profile's own overrides list, not the default profile's).
# ===========================================================================
# QUARANTINE: OPEN backlog#937 — concurrent_resend lock-acquire deadline flakes
# under saturated disk I/O in the full parallel suite.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(concurrent_resend_same_part_commits_one_generation)'
test-group = 'ecstore-serial-flaky'
retries = 2
# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
# make_bucket into InsufficientWriteQuorum via shared global state under load.
[[profile.ci.overrides]]
@@ -114,6 +104,11 @@ filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(
test-group = 'ecstore-serial-flaky'
retries = 2
# Keep the deterministic multipart handoff isolated across nextest processes.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(concurrent_resend_same_part_commits_one_generation)'
test-group = 'ecstore-serial-flaky'
# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
# on producer/consumer timing windows that stretch past the budget on loaded
# CI runners (regression test for rustfs#4644; failed on a zero-Rust-diff PR).
+57 -13
View File
@@ -141,7 +141,7 @@ jobs:
name: Test and Lint
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: sm-standard-4
timeout-minutes: 60
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
@@ -156,16 +156,69 @@ jobs:
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Prepare test evidence
run: |
mkdir -p artifacts/test-and-lint
{
echo "run_id=${GITHUB_RUN_ID}"
echo "job=${GITHUB_JOB}"
echo "runner=${RUNNER_NAME}"
echo "started_at=$(date --utc --iso-8601=seconds)"
} > artifacts/test-and-lint/run-metadata.txt
# Clippy runs before the test pass: lint failures are the most common
# CI-only breakage and should surface in minutes, not after 20+ minutes
# of tests.
- name: Run clippy lints
run: cargo clippy --all-targets -- -D warnings
- name: Run tests
- name: Run nextest tests
run: |
cargo nextest run --profile ci --all --exclude e2e_test
cargo test --all --doc
mkdir -p artifacts/test-and-lint
set +e
NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 75m \
cargo nextest run --profile ci --all --exclude e2e_test \
--status-level all --final-status-level all \
2>&1 | tee artifacts/test-and-lint/nextest.log
status=${PIPESTATUS[0]}
{
echo "command=cargo nextest run --profile ci --all --exclude e2e_test"
echo "exit_status=${status}"
echo "finished_at=$(date --utc --iso-8601=seconds)"
echo
echo "Remaining test-related processes:"
pgrep -af 'cargo|nextest|target/.*/deps/' || true
} > artifacts/test-and-lint/nextest-diagnostics.txt
exit "${status}"
- name: Run documentation tests
run: |
mkdir -p artifacts/test-and-lint
set +e
timeout --verbose --signal=TERM --kill-after=30s 15m \
cargo test --all --doc \
2>&1 | tee artifacts/test-and-lint/doctest.log
status=${PIPESTATUS[0]}
{
echo "command=cargo test --all --doc"
echo "exit_status=${status}"
echo "finished_at=$(date --utc --iso-8601=seconds)"
echo
echo "Remaining test-related processes:"
pgrep -af 'cargo|rustdoc|target/.*/deps/' || true
} > artifacts/test-and-lint/doctest-diagnostics.txt
exit "${status}"
- name: Upload test reports and diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: junit-test-and-lint-${{ github.run_number }}
path: |
target/nextest/ci/junit.xml
artifacts/test-and-lint
retention-days: 3
if-no-files-found: error
# rustfs/backlog#1289: fail if a seed rule's log anchor no longer exists
# verbatim in the source tree (log message drifted without updating the
@@ -174,15 +227,6 @@ jobs:
- name: Check log-analyzer rule anchors
run: ./scripts/check_log_analyzer_rules.sh
- name: Upload test junit report
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: junit-test-and-lint-${{ github.run_number }}
path: target/nextest/ci/junit.xml
retention-days: 3
if-no-files-found: ignore
# Explicit gate for migration-critical suites. These tests already ran in
# the full nextest pass above; a single filtered nextest invocation keeps
# the named gate without rebuilding or re-running them one package at a time.
+1 -1
View File
@@ -66,7 +66,7 @@ env:
CARGO_TERM_COLOR: always
REGISTRY_DOCKERHUB: rustfs/rustfs
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
REGISTRY_QUAY: quay.io/${{ secrets.QUAY_USERNAME }}/rustfs
REGISTRY_QUAY: quay.io/rustfs/rustfs
DOCKER_PLATFORMS: linux/amd64,linux/arm64
jobs:
+25 -2
View File
@@ -172,7 +172,7 @@
],
},
{
"name": "Debug executable target/debug/rustfs with sse",
"name": "Debug executable target/debug/rustfs with sse kms",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/rustfs",
@@ -200,7 +200,7 @@
// 2. kms local backend test key
// "RUSTFS_KMS_ENABLE": "true",
// "RUSTFS_KMS_BACKEND": "local",
// "RUSTFS_KMS_KEY_DIR": "./target/kms-key-dir",
// "RUSTFS_KMS_KEY_DIR": "/tmp/kms-key-dir",
// "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
// "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
@@ -228,6 +228,29 @@
"rust"
],
},
{
"name": "Debug executable target/debug/rustfs with local sse",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/rustfs",
"args": [],
"cwd": "${workspaceFolder}",
"env": {
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
"RUSTFS_VOLUMES": "./target/volumes/test{1...4}",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
"RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
"RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
"RUSTFS_SSE_S3_MASTER_KEY": "xGb3aYSp825j2tPpg8JrUzghiXsIkfdOtmrsJ/iafiM=",
"RUST_LOG": "rustfs=debug,ecstore=debug,s3s=debug,iam=debug",
},
"sourceLanguages": [
"rust"
],
},
{
"type": "lldb",
"request": "launch",
Generated
+26 -22
View File
@@ -1703,9 +1703,9 @@ dependencies = [
[[package]]
name = "camino"
version = "1.2.4"
version = "1.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5f2d30e4173c4026932d51d31d6b0613b1fd3014bf3f9f8943d4ba139c437ba0"
checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d"
dependencies = [
"serde_core",
]
@@ -3625,13 +3625,13 @@ dependencies = [
[[package]]
name = "displaydoc"
version = "0.2.6"
version = "0.2.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f"
checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"syn 3.0.3",
]
[[package]]
@@ -3946,11 +3946,10 @@ dependencies = [
[[package]]
name = "event-listener"
version = "5.4.1"
version = "5.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e13b66accf52311f30a0db42147dadea9850cb48cd070028831ae5f5d4b856ab"
checksum = "5a23add41df1562121a9393cb065eab5146a1242410f23a644851e90cfd669d2"
dependencies = [
"concurrent-queue",
"parking",
"pin-project-lite",
]
@@ -6110,9 +6109,9 @@ dependencies = [
[[package]]
name = "metrique"
version = "0.1.28"
version = "0.1.29"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aa466af30a9fe0b1db1dae097e0ce7ddac4b666b1d3a9f5c682e889272511dd8"
checksum = "d2e394c63e2d1a30aeb3b9392ecf3439d8475d2df810a8f4f6e66d6866754017"
dependencies = [
"itoa",
"jiff",
@@ -6140,9 +6139,9 @@ dependencies = [
[[package]]
name = "metrique-macro"
version = "0.1.19"
version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1f5febfaf14fea234b60e0ad43c728db274033893a38d0fa1f87d9b7056d3d61"
checksum = "786df1fd0abebd0db685f7e9a353c78756d4b370fb98a52376c2015fa55f141f"
dependencies = [
"Inflector",
"darling 0.23.0",
@@ -6169,9 +6168,9 @@ checksum = "2faca4e4480069ff02b1763b3b79f5cec7e8628e24d9dc5b6073f53d2577a4d9"
[[package]]
name = "metrique-writer"
version = "0.1.24"
version = "0.1.25"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "124326a2ac4c4f61562fa4d071735a1c463f9ba0317d1564f75dc01313dc12d7"
checksum = "82cdde44d241dab7fc8b7a32e0eb5dae6cd28f8de80b59f9a1e9f2f0b05e485e"
dependencies = [
"ahash",
"crossbeam-queue",
@@ -6190,9 +6189,9 @@ dependencies = [
[[package]]
name = "metrique-writer-core"
version = "0.1.18"
version = "0.1.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "55b5bbb6d88bde29f6ed74a574cbe49e51ea0c36cccc7e63eee2040db5675b15"
checksum = "e57379b7ee2272efaeaaa6de062503563e57333b24aadc7f2255b3d602899e8b"
dependencies = [
"derive-where",
"itertools 0.14.0",
@@ -9477,6 +9476,8 @@ name = "rustfs-lifecycle"
version = "1.0.0-beta.11"
dependencies = [
"async-trait",
"metrics",
"metrics-util",
"proptest",
"rustfs-common",
"rustfs-config",
@@ -9714,6 +9715,7 @@ dependencies = [
"http-body-util",
"hyper",
"hyper-util",
"ipnetwork",
"libunftp",
"md5",
"percent-encoding",
@@ -9730,7 +9732,9 @@ dependencies = [
"rustfs-policy",
"rustfs-rio",
"rustfs-storage-api",
"rustfs-test-utils",
"rustfs-tls-runtime",
"rustfs-trusted-proxies",
"rustfs-utils",
"rustls",
"s3s",
@@ -10351,7 +10355,7 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]]
name = "s3s"
version = "0.14.1"
source = "git+https://github.com/s3s-project/s3s.git?rev=8136db4ac8253c0ccfd86f8216e00e0235747757#8136db4ac8253c0ccfd86f8216e00e0235747757"
source = "git+https://github.com/cxymds/s3s.git?rev=fe3941d91fa1c69956f209a9145995c9f0235bff#fe3941d91fa1c69956f209a9145995c9f0235bff"
dependencies = [
"arc-swap",
"arrayvec",
@@ -10457,9 +10461,9 @@ dependencies = [
[[package]]
name = "schemars"
version = "1.2.1"
version = "1.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a2b42f36aa1cd011945615b92222f6bf73c599a102a300334cd7f8dbeec726cc"
checksum = "687274d293b6cdc6e73e0fee520bf2049650090d7164f87672d212a3c530cf4a"
dependencies = [
"dyn-clone",
"ref-cast",
@@ -10692,7 +10696,7 @@ dependencies = [
"indexmap 1.9.3",
"indexmap 2.14.0",
"schemars 0.9.0",
"schemars 1.2.1",
"schemars 1.2.2",
"serde_core",
"serde_json",
"serde_with_macros",
@@ -11872,9 +11876,9 @@ dependencies = [
[[package]]
name = "toml_parser"
version = "1.1.2+spec-1.1.0"
version = "1.1.3+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a2abe9b86193656635d2411dc43050282ca48aa31c2451210f4202550afb7526"
checksum = "1d38ac1cf9b95face32296c0a3ede1fdc270627c9d9c02a7274dd6d960dc4d56"
dependencies = [
"winnow",
]
+1 -1
View File
@@ -288,7 +288,7 @@ redis = { version = "1.4.1" }
rustix = { version = "1.1.4" }
rust-embed = { version = "8.12.0" }
rustc-hash = { version = "2.1.3" }
s3s = { git = "https://github.com/s3s-project/s3s.git", rev = "8136db4ac8253c0ccfd86f8216e00e0235747757" }
s3s = { git = "https://github.com/cxymds/s3s.git", rev = "fe3941d91fa1c69956f209a9145995c9f0235bff" }
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.3"
+1 -1
View File
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM rust:1.97-trixie
FROM rust:1.97.1-trixie
RUN set -eux; \
export DEBIAN_FRONTEND=noninteractive; \
+1 -1
View File
@@ -32,7 +32,7 @@ ARG RUSTFS_BUILD_FEATURES=""
# -----------------------------
# Build stage
# -----------------------------
FROM rust:1.97-trixie AS builder
FROM rust:1.97.1-trixie AS builder
# Re-declare args after FROM
ARG TARGETPLATFORM
+3
View File
@@ -25,6 +25,9 @@ keywords = ["checksum-calculation", "verification", "integrity", "authenticity",
categories = ["web-programming", "development-tools", "network-programming"]
documentation = "https://docs.rs/rustfs-checksums/latest/rustfs_checksum/"
[lints]
workspace = true
[dependencies]
bytes = { workspace = true, features = ["serde"] }
crc-fast = { workspace = true }
+161 -11
View File
@@ -506,8 +506,35 @@ pub struct ReplicationStats {
}
impl ReplicationStats {
pub fn is_empty(&self) -> bool {
let Self {
pending_size,
replicated_size,
failed_size,
failed_count,
pending_count,
missed_threshold_size,
after_threshold_size,
missed_threshold_count,
after_threshold_count,
replicated_count,
} = self;
*pending_size == 0
&& *replicated_size == 0
&& *failed_size == 0
&& *failed_count == 0
&& *pending_count == 0
&& *missed_threshold_size == 0
&& *after_threshold_size == 0
&& *missed_threshold_count == 0
&& *after_threshold_count == 0
&& *replicated_count == 0
}
#[deprecated(note = "use is_empty instead")]
pub fn empty(&self) -> bool {
self.replicated_size == 0 && self.failed_size == 0 && self.failed_count == 0
self.is_empty()
}
}
@@ -520,16 +547,19 @@ pub struct ReplicationAllStats {
}
impl ReplicationAllStats {
pub fn is_empty(&self) -> bool {
let Self {
replica_size,
replica_count,
targets,
} = self;
*replica_size == 0 && *replica_count == 0 && targets.values().all(ReplicationStats::is_empty)
}
#[deprecated(note = "use is_empty instead")]
pub fn empty(&self) -> bool {
if self.replica_size != 0 && self.replica_count != 0 {
return false;
}
for v in self.targets.values() {
if !v.empty() {
return false;
}
}
true
self.is_empty()
}
}
@@ -783,7 +813,7 @@ impl DataUsageCache {
return Some(root);
}
let mut flat = self.flatten(&root);
if flat.replication_stats.as_ref().is_some_and(|stats| stats.empty()) {
if flat.replication_stats.as_ref().is_some_and(ReplicationAllStats::is_empty) {
flat.replication_stats = None;
}
Some(flat)
@@ -1582,6 +1612,126 @@ mod tests {
assert_eq!(map["BETWEEN_1024B_AND_1_MB"], u64::MAX);
}
#[test]
fn replication_stats_empty_checks_every_field() {
type SetField = fn(&mut ReplicationStats);
let cases: [(&str, SetField); 10] = [
("pending_size", |stats| stats.pending_size = 1),
("replicated_size", |stats| stats.replicated_size = 1),
("failed_size", |stats| stats.failed_size = 1),
("failed_count", |stats| stats.failed_count = 1),
("pending_count", |stats| stats.pending_count = 1),
("missed_threshold_size", |stats| stats.missed_threshold_size = 1),
("after_threshold_size", |stats| stats.after_threshold_size = 1),
("missed_threshold_count", |stats| stats.missed_threshold_count = 1),
("after_threshold_count", |stats| stats.after_threshold_count = 1),
("replicated_count", |stats| stats.replicated_count = 1),
];
assert!(ReplicationStats::default().is_empty());
for (field, set_nonzero) in cases {
let mut stats = ReplicationStats::default();
set_nonzero(&mut stats);
assert!(!stats.is_empty(), "{field} must make replication stats non-empty");
}
}
#[test]
fn replication_all_stats_empty_checks_aggregate_fields_independently() {
let cases = [
(
"replica_size",
ReplicationAllStats {
replica_size: 1,
..Default::default()
},
),
(
"replica_count",
ReplicationAllStats {
replica_count: 1,
..Default::default()
},
),
];
assert!(ReplicationAllStats::default().is_empty());
for (field, stats) in cases {
assert!(!stats.is_empty(), "{field} must make aggregate replication stats non-empty");
}
let empty_targets = ReplicationAllStats {
targets: HashMap::from([("arn:test:empty".to_string(), ReplicationStats::default())]),
..Default::default()
};
assert!(empty_targets.is_empty(), "all-empty targets must keep aggregate stats empty");
let stats = ReplicationAllStats {
targets: HashMap::from([
("arn:test:empty".to_string(), ReplicationStats::default()),
(
"arn:test:non-empty".to_string(),
ReplicationStats {
pending_count: 1,
..Default::default()
},
),
]),
..Default::default()
};
assert!(!stats.is_empty(), "a non-empty target must make aggregate replication stats non-empty");
}
#[test]
fn size_recursive_prunes_empty_and_preserves_pending_replication_stats() {
let root = hash_path("bucket");
let child = hash_path("bucket/child");
let mut cache = DataUsageCache::default();
cache.replace_hashed(&root, &None, &DataUsageEntry::default());
cache.replace_hashed(
&child,
&Some(root.clone()),
&DataUsageEntry {
replication_stats: Some(ReplicationAllStats::default()),
..Default::default()
},
);
assert!(
cache
.size_recursive("bucket")
.expect("bucket usage should flatten")
.replication_stats
.is_none()
);
cache.replace_hashed(
&child,
&Some(root.clone()),
&DataUsageEntry {
replication_stats: Some(ReplicationAllStats {
targets: HashMap::from([(
"arn:test:pending".to_string(),
ReplicationStats {
pending_count: 1,
..Default::default()
},
)]),
..Default::default()
}),
..Default::default()
},
);
let flattened = cache.size_recursive("bucket").expect("bucket usage should flatten");
let replication = flattened
.replication_stats
.expect("pending-only replication stats must survive pruning");
assert_eq!(replication.targets["arn:test:pending"].pending_count, 1);
}
#[test]
fn test_data_usage_cache_merge_adds_missing_child() {
let mut base = DataUsageCache::default();
@@ -45,10 +45,10 @@
//! * Parity reconstruction: one data disk is taken offline
//! (`take_disk_offline`) and the SAME object matrix is GET both ways while
//! the EC 2+2 set rebuilds each large object from the surviving shards. The
//! codec-streaming reader gate never inspects drive health, so the codec
//! fast path is exercised end-to-end through reconstruction; the test
//! asserts byte- and header-equality vs the legacy path AND that the codec
//! phase never fell back to a duplex pipe while reconstructing.
//! eager first/single-part setup may keep its conservative whole-request
//! fallback when shard placement makes codec streaming unsafe, so this phase
//! asserts byte- and header-equality vs the legacy path rather than requiring
//! zero duplex fallbacks under degraded drive health.
//! * Missing object: a GET for an absent key is compared across both phases
//! to prove the error semantics (HTTP status + S3 error code) are identical
//! — the codec env must not perturb the NoSuchKey negative path.
@@ -475,15 +475,14 @@ mod tests {
"ranged GET length diverged with codec streaming enabled"
);
// ---- Phase B degraded: the same reconstruction, now on the codec path ----
// Re-run the reconstruction A/B with the codec-streaming gates still
// open. The reader gate decision is independent of drive health (it
// never inspects disk state), so the codec fast path is exercised
// end-to-end while the EC set rebuilds each large object from the
// surviving shards — this is a real codec-vs-legacy reconstruction test,
// not legacy-vs-legacy. Snapshot the duplex count first (the range GET
// above already used the duplex path) so we can measure only the markers
// these degraded codec GETs add.
// ---- Phase B degraded: the same reconstruction, with codec gates open ----
// Re-run the reconstruction A/B with codec-streaming enabled. If eager
// first/single-part setup cannot prove the codec path is safe for the
// surviving shards, the implementation intentionally preserves the
// whole-request legacy fallback; later multipart parts can degrade in
// place. This phase verifies parity-reconstructed bytes and headers,
// while the healthy phase above remains the strict zero-duplex path
// confirmation.
let dup_codec_before_degraded = count_marker(&codec_log, DUPLEX_MARKER);
harness.take_disk_offline(0)?;
let mut codec_degraded: BTreeMap<String, GetView> = BTreeMap::new();
@@ -511,16 +510,11 @@ mod tests {
);
}
// Path confirmation under reconstruction: the codec fast path must have
// served the reconstructed large objects without ever falling back to
// the legacy duplex pipe. Without this, the equivalence above could be
// legacy-vs-legacy and prove nothing about codec reconstruction.
// Keep degraded duplex markers as diagnostic evidence only: eager setup
// may fall back before streaming when shard safety cannot be proven.
sleep(Duration::from_millis(300)).await;
let dup_codec_degraded = count_marker(&codec_log, DUPLEX_MARKER).saturating_sub(dup_codec_before_degraded);
assert_eq!(
dup_codec_degraded, 0,
"codec phase created {dup_codec_degraded} duplex pipe(s) while reconstructing large objects with disk0 offline; the codec fast path was not exercised under degraded reads (see {codec_log})"
);
info!(dup_codec_degraded, "codec phase degraded-read legacy duplex marker count");
info!(
objects = baseline.len(),
@@ -520,28 +520,33 @@ async fn assert_msgpack_fallback_unchanged(
Ok(())
}
async fn assert_msgpack_decode_observed(
collector: &OtlpMetricCollector,
before: &BTreeMap<String, u64>,
series: &[(&str, &str)],
) -> TestResult {
async fn assert_msgpack_decode_observed(collector: &OtlpMetricCollector, before: &BTreeMap<String, u64>) -> TestResult {
let deadline = Instant::now() + Duration::from_secs(20);
loop {
let after = collector.msgpack_json_decode_totals().await;
let missing = series
let missing = [FALLBACK_REQUEST_DIRECTION, FALLBACK_RESPONSE_DIRECTION]
.iter()
.filter_map(|(direction, message)| {
let key = msgpack_decode_metric_key(direction, message, MSGPACK_CODEC_MSGPACK);
let before_value = before.get(&key).copied().unwrap_or_default();
let after_value = after.get(&key).copied().unwrap_or_default();
(after_value <= before_value).then_some(format!("{direction}/{message}/{}", MSGPACK_CODEC_MSGPACK))
.filter_map(|direction| {
let prefix = format!("{direction}\u{1f}");
let suffix = format!("\u{1f}{MSGPACK_CODEC_MSGPACK}");
let before_total = before
.iter()
.filter(|(key, _)| key.starts_with(&prefix) && key.ends_with(&suffix))
.map(|(_, value)| *value)
.sum::<u64>();
let after_total = after
.iter()
.filter(|(key, _)| key.starts_with(&prefix) && key.ends_with(&suffix))
.map(|(_, value)| *value)
.sum::<u64>();
(after_total <= before_total).then_some(format!("{direction}/{}", MSGPACK_CODEC_MSGPACK))
})
.collect::<Vec<_>>();
if missing.is_empty() {
return Ok(());
}
if Instant::now() >= deadline {
return Err(format!("timed out waiting for msgpack decode traffic on {missing:?}; totals={after:?}").into());
return Err(format!("timed out waiting for msgpack decode traffic for {missing:?}; totals={after:?}").into());
}
sleep(Duration::from_millis(100)).await;
}
@@ -1857,7 +1862,7 @@ async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls() -> Te
PartNumberReaderPathExpectation::new(bucket, multipart_key, &second_part, multipart_body.len(), MULTIPART, LEGACY_DUPLEX),
)
.await?;
assert_msgpack_decode_observed(&collector, &decode_before, &MSGPACK_FALLBACK_CONTROL_SERIES).await?;
assert_msgpack_decode_observed(&collector, &decode_before).await?;
assert_msgpack_fallback_unchanged(&collector, &fallback_before, &MSGPACK_FALLBACK_CONTROL_SERIES).await?;
assert_msgpack_decode_errors_unchanged(&collector, &decode_errors_before, &MSGPACK_FALLBACK_CONTROL_SERIES).await?;
@@ -1948,8 +1953,8 @@ async fn four_node_add_tier_converges_after_offline_node_restart_without_second_
hot.start().await?;
let tier_name = unique_tier_name();
hot.stop_node(3)?;
let add_tier_response = submit_rustfs_tier(&hot, &cold, &tier_name).await?;
hot.stop_node(3)?;
hot.start_node(3).await?;
wait_for_tier_converged(&hot, &tier_name, &add_tier_response).await
@@ -2003,7 +2008,7 @@ async fn four_node_manual_transition_job_status_survives_node_restart() -> TestR
assert_eq!(cancel_value["job_id"].as_str(), Some(job_id.as_str()));
assert_eq!(cancel_value["bucket"].as_str(), Some(bucket.as_str()));
assert_eq!(cancel_value["dry_run"].as_bool(), Some(true));
assert_eq!(cancel_value["cancel_requested"].as_bool(), Some(true));
assert_eq!(cancel_value["cancel_requested"].as_bool(), Some(false));
let missing_job_id = Uuid::new_v4();
let (missing_status, missing_body) = signed_admin_request(
@@ -2325,7 +2330,7 @@ async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls_during_
PartNumberReaderPathExpectation::new(bucket, key, &second_part, body.len(), REMOTE, REMOTE_TRANSITION),
)
.await?;
assert_msgpack_decode_observed(&collector, &decode_before, &MSGPACK_FALLBACK_CONTROL_SERIES).await?;
assert_msgpack_decode_observed(&collector, &decode_before).await?;
let encrypted_key = "transition/encrypted-sse.bin";
let encrypted_body = payload(16 * KIB, 0xAB);
@@ -400,6 +400,20 @@ impl NodeService for MinimalLockNodeService {
Err(Status::unimplemented("lock-only test server"))
}
async fn prepare_part_transaction(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::PreparePartTransactionRequest>,
) -> Result<Response<rustfs_protos::proto_gen::node_service::PreparePartTransactionResponse>, Status> {
Err(Status::unimplemented("lock-only test server"))
}
async fn settle_part_transaction(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::SettlePartTransactionRequest>,
) -> Result<Response<rustfs_protos::proto_gen::node_service::SettlePartTransactionResponse>, Status> {
Err(Status::unimplemented("lock-only test server"))
}
async fn rename_file(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::RenameFileRequest>,
+4 -4
View File
@@ -128,7 +128,7 @@ pub mod bucket {
get_object_lock_config, get_public_access_block_config, get_quota_config, get_replication_config,
get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
init_bucket_metadata_sys, list_bucket_targets, remove_bucket_metadata, set_bucket_metadata, update,
update_bucket_targets_under_transaction_lock,
update_bucket_targets_under_transaction_lock, update_config_with,
};
}
@@ -303,9 +303,9 @@ pub mod disk {
pub use crate::disk::{
BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
CheckPartsResp, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore,
FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize, RUSTFS_META_BUCKET,
ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE, UpdateMetadataOpts, VolumeInfo,
WalkDirOptions, new_disk, validate_batch_read_version_item_count,
FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
STORAGE_FORMAT_FILE, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk, validate_batch_read_version_item_count,
};
pub use bytes::Bytes;
pub use endpoint::Endpoint;
@@ -554,6 +554,7 @@ async fn delete_free_version_remote_object(
oi: &ObjectInfo,
tier_config_mgr: &Arc<RwLock<TierConfigMgr>>,
) -> Result<(), std::io::Error> {
let version_id_exact = validate_transition_remote_version(oi)?;
let identity = tier_destination_id_from_metadata(&oi.user_defined)?
.ok_or_else(|| std::io::Error::other("tier free-version has no durable backend identity"))?;
delete_object_from_remote_tier_idempotent_with_manager_and_identity(
@@ -562,7 +563,7 @@ async fn delete_free_version_remote_object(
&oi.transitioned_object.tier,
identity,
tier_config_mgr,
false,
version_id_exact,
)
.await?;
Ok(())
@@ -1181,8 +1182,12 @@ impl TransitionState {
}
fn new_with_capacity(capacity: usize) -> Arc<Self> {
let capacity = capacity.max(1);
let queue_send_timeout = resolve_transition_queue_send_timeout();
Self::new_with_capacity_and_timeout(capacity, queue_send_timeout)
}
fn new_with_capacity_and_timeout(capacity: usize, queue_send_timeout: StdDuration) -> Arc<Self> {
let capacity = capacity.max(1);
let (tx1, rx1) = bounded(capacity);
Arc::new(Self {
transition_tx: tx1,
@@ -1249,13 +1254,12 @@ impl TransitionState {
return false;
}
let bucket = bucket.to_string();
let scheduled = Arc::clone(&self.compensation_buckets);
let state = Arc::clone(self);
tokio::spawn(async move {
Self::inc_counter(&state.compensation_running_tasks);
state.record_scanner_transition_state();
let Some(api) = runtime_sources::object_store_handle() else {
scheduled.lock().unwrap().remove(&bucket);
state.finish_bucket_compensation(&bucket);
Self::add_counter(&state.compensation_running_tasks, -1);
state.record_scanner_transition_state();
debug!(
@@ -1291,13 +1295,25 @@ impl TransitionState {
);
}
scheduled.lock().unwrap().remove(&bucket);
state.finish_bucket_compensation(&bucket);
Self::add_counter(&state.compensation_running_tasks, -1);
state.record_scanner_transition_state();
});
true
}
fn finish_bucket_compensation(&self, bucket: &str) {
match self.compensation_buckets.lock() {
Ok(mut scheduled) => {
scheduled.remove(bucket);
}
Err(poisoned) => {
poisoned.into_inner().remove(bucket);
self.compensation_buckets.clear_poison();
}
}
}
#[inline]
fn inc_counter(counter: &AtomicI64) {
Self::add_counter(counter, 1);
@@ -1534,17 +1550,35 @@ impl TransitionState {
let outcome = match self.transition_tx.try_send(Some(task)) {
Ok(()) => TransitionEnqueueOutcome::Queued,
Err(async_channel::TrySendError::Full(_)) => {
Err(async_channel::TrySendError::Full(task)) => {
Self::inc_counter(&self.queue_full_tasks);
debug!(
bucket = %oi.bucket,
object = %oi.name,
source = ?src,
"transition queue is full; deferring to scanner/backfill"
);
TransitionEnqueueOutcome::QueueFull
let send_timeout = self.transition_queue_send_timeout;
match tokio::time::timeout(send_timeout, self.transition_tx.send(task)).await {
Ok(Ok(())) => TransitionEnqueueOutcome::Queued,
Ok(Err(_)) => {
self.schedule_bucket_compensation(&oi.bucket);
TransitionEnqueueOutcome::QueueClosed
}
Err(_) => {
Self::inc_counter(&self.queue_send_timeout_tasks);
self.schedule_bucket_compensation(&oi.bucket);
debug!(
bucket = %oi.bucket,
object = %oi.name,
source = ?src,
timeout_ms = send_timeout.as_millis() as u64,
event = EVENT_LIFECYCLE_TRANSITION_COMPENSATION,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
state = "queue_send_timed_out",
"Scanner transition enqueue timed out; scheduled bucket compensation"
);
TransitionEnqueueOutcome::QueueFull
}
}
}
Err(async_channel::TrySendError::Closed(_)) => {
self.schedule_bucket_compensation(&oi.bucket);
debug!(
bucket = %oi.bucket,
object = %oi.name,
@@ -1729,7 +1763,7 @@ impl TransitionState {
}
pub fn add_lastday_stats(&self, tier: &str, ts: TierStats) {
let mut tier_stats = self.last_day_stats.lock().unwrap();
let mut tier_stats = self.lock_last_day_stats();
tier_stats
.entry(tier.to_string())
.and_modify(|e| e.add_stats(ts))
@@ -1737,7 +1771,7 @@ impl TransitionState {
}
pub fn get_daily_all_tier_stats(&self) -> DailyAllTierStats {
let tier_stats = self.last_day_stats.lock().unwrap();
let tier_stats = self.lock_last_day_stats();
let mut res = DailyAllTierStats::with_capacity(tier_stats.len());
for (tier, st) in tier_stats.iter() {
res.insert(tier.clone(), st.clone());
@@ -1745,6 +1779,18 @@ impl TransitionState {
res
}
fn lock_last_day_stats(&self) -> std::sync::MutexGuard<'_, HashMap<String, LastDayTierStats>> {
match self.last_day_stats.lock() {
Ok(stats) => stats,
Err(poisoned) => {
let mut stats = poisoned.into_inner();
stats.clear();
self.last_day_stats.clear_poison();
stats
}
}
}
pub async fn update_workers(api: Arc<ECStore>, n: i64) {
Self::update_workers_inner(api, n).await;
}
@@ -1766,7 +1812,27 @@ impl TransitionState {
fn resize_workers_to(api: Arc<ECStore>, n: i64, requested: i64, absolute_max: i64) {
let target = n as usize;
let transition_state = runtime_sources::transition_state_handle();
let mut workers = transition_state.workers.lock().unwrap();
let runtime = match tokio::runtime::Handle::try_current() {
Ok(runtime) => runtime,
Err(err) => {
warn!(
event = EVENT_LIFECYCLE_WORKER_STATE,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
error = %err,
state = "resize_failed",
"Lifecycle worker pool requires a Tokio runtime"
);
return;
}
};
// Runtime lookup happens before locking, and the guard is dropped before
// metrics/logging callbacks. Poison therefore means a Vec mutation may
// have unwound and worker tracking cannot be reconstructed safely.
let mut workers = transition_state
.workers
.lock()
.expect("transition worker tracking mutex poisoned");
let tracked_workers = workers.len();
workers.retain(|worker| !worker.handle.is_finished());
let pruned_finished_workers = tracked_workers.saturating_sub(workers.len());
@@ -1776,7 +1842,7 @@ impl TransitionState {
let clone_api = api.clone();
let cancel = CancellationToken::new();
let worker_cancel = cancel.clone();
let handle = tokio::spawn(async move {
let handle = runtime.spawn(async move {
TransitionState::worker_with_cancel(clone_api, worker_cancel).await;
});
workers.push(TransitionWorker { cancel, handle });
@@ -1790,6 +1856,7 @@ impl TransitionState {
let current_workers = workers.len() as i64;
transition_state.num_workers.store(current_workers, Ordering::SeqCst);
drop(workers);
transition_state.record_scanner_transition_state();
debug!(
@@ -4135,6 +4202,23 @@ pub async fn get_transitioned_object_reader(
get_transitioned_object_reader_with_tier_manager(bucket, object, rs, h, oi, opts, &tier_config_mgr).await
}
fn validate_transition_remote_version(oi: &ObjectInfo) -> Result<bool, std::io::Error> {
let version = oi.transitioned_object.version_id.as_str();
match oi.transition_version_state {
rustfs_filemeta::TransitionVersionState::Unknown => Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"remote tier object version state is unknown",
)),
rustfs_filemeta::TransitionVersionState::KnownDisabled if version.is_empty() => Ok(false),
rustfs_filemeta::TransitionVersionState::SuspendedNull if version == "null" => Ok(true),
rustfs_filemeta::TransitionVersionState::Exact if !version.is_empty() && version != "null" => Ok(true),
_ => Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"remote tier object version state conflicts with its version ID",
)),
}
}
pub(crate) async fn get_transitioned_object_reader_with_tier_manager(
bucket: &str,
object: &str,
@@ -4144,6 +4228,7 @@ pub(crate) async fn get_transitioned_object_reader_with_tier_manager(
opts: &ObjectOptions,
tier_config_mgr: &Arc<RwLock<TierConfigMgr>>,
) -> Result<GetObjectReader, std::io::Error> {
validate_transition_remote_version(oi)?;
let expected_identity = tier_destination_id_from_metadata(&oi.user_defined)?;
let lease = match expected_identity {
Some(identity) => {
@@ -4884,6 +4969,7 @@ mod tests {
FreeVersionRecoveryStats, RecoveryWalkTestAction, list_tier_free_versions, recover_tier_free_versions_with_cancel,
set_recovery_bucket_list_wait_hook, set_recovery_walk_test_hook,
};
use crate::bucket::lifecycle::tier_last_day_stats::LastDayTierStats;
use crate::bucket::lifecycle::tier_sweeper::Jentry;
use crate::bucket::metadata::BUCKET_LIFECYCLE_CONFIG;
use crate::bucket::metadata_sys;
@@ -4917,6 +5003,7 @@ mod tests {
use http::HeaderMap;
use rustfs_common::metrics::{IlmAction, global_metrics};
use rustfs_config::ENV_TRANSITION_WORKERS_ABSOLUTE_MAX;
use rustfs_data_usage::TierStats;
use rustfs_filemeta::{FileInfo, FileMeta};
use s3s::dto::{
BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, MetadataEntry, OutputLocation,
@@ -5438,6 +5525,7 @@ mod tests {
tier: tier.clone(),
..Default::default()
},
transition_version_state: rustfs_filemeta::TransitionVersionState::Exact,
..Default::default()
};
@@ -5501,6 +5589,7 @@ mod tests {
tier,
..Default::default()
},
transition_version_state: rustfs_filemeta::TransitionVersionState::Exact,
..Default::default()
};
@@ -5523,6 +5612,70 @@ mod tests {
assert_eq!(backend.get_count().await, 0);
}
#[cfg(feature = "test-util")]
#[tokio::test]
async fn transitioned_get_rejects_unknown_version_state_before_backend_io() {
let manager = TierConfigMgr::new();
let tier = format!("COLDTIER{}", &Uuid::new_v4().simple().to_string()[..8]).to_uppercase();
let backend = register_mock_tier(&manager, &tier).await;
let object_info = ObjectInfo {
bucket: "bucket".to_string(),
name: "object".to_string(),
size: 1,
transitioned_object: TransitionedObject {
name: "remote/object".to_string(),
version_id: String::new(),
status: crate::bucket::lifecycle::lifecycle::TRANSITION_COMPLETE.to_string(),
tier,
..Default::default()
},
transition_version_state: rustfs_filemeta::TransitionVersionState::Unknown,
..Default::default()
};
let err = match get_transitioned_object_reader_with_tier_manager(
&object_info.bucket,
&object_info.name,
&None,
&HeaderMap::new(),
&object_info,
&ObjectOptions::default(),
&manager,
)
.await
{
Ok(_) => panic!("unknown remote version state must fail before backend IO"),
Err(err) => err,
};
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert_eq!(backend.get_count().await, 0);
}
#[cfg(feature = "test-util")]
#[tokio::test]
async fn free_version_delete_rejects_unknown_version_state_before_backend_io() {
let manager = TierConfigMgr::new();
let backend = register_mock_tier(&manager, "WARM").await;
let object_info = ObjectInfo {
transitioned_object: TransitionedObject {
name: "remote/object".to_string(),
version_id: "legacy-version".to_string(),
tier: "WARM".to_string(),
..Default::default()
},
transition_version_state: rustfs_filemeta::TransitionVersionState::Unknown,
..Default::default()
};
let err = super::delete_free_version_remote_object(&object_info, &manager)
.await
.expect_err("unknown remote version state must fail before backend IO");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert_eq!(backend.remove_count().await, 0);
}
#[cfg(feature = "test-util")]
#[tokio::test]
async fn free_version_remote_delete_requires_persisted_destination_identity() {
@@ -5572,6 +5725,7 @@ mod tests {
oi.transitioned_object.tier = "WARM".to_string();
oi.transitioned_object.name = "remote/object".to_string();
oi.transitioned_object.version_id = "remote-version".to_string();
oi.transition_version_state = rustfs_filemeta::TransitionVersionState::Exact;
let local_delete_calls = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let legacy_err = delete_free_version_remote_object_then(&oi, &manager, {
@@ -5582,7 +5736,8 @@ mod tests {
})
.await
.expect_err("legacy free-version without identity must be retained");
assert!(legacy_err.to_string().contains("no durable backend identity"));
assert_eq!(legacy_err.kind(), std::io::ErrorKind::Other);
assert_eq!(old_backend.remove_count().await, 0);
assert_eq!(local_delete_calls.load(Ordering::Relaxed), 0);
let mut invalid_metadata = HashMap::new();
@@ -5713,6 +5868,7 @@ mod tests {
oi.transitioned_object.tier = "WARM".to_string();
oi.transitioned_object.name = "remote/object".to_string();
oi.transitioned_object.version_id = "remote-version".to_string();
oi.transition_version_state = rustfs_filemeta::TransitionVersionState::Exact;
let err = match get_transitioned_object_reader_with_tier_manager(
"bucket",
@@ -5728,7 +5884,12 @@ mod tests {
Ok(_) => panic!("identity-bound GET must reject a same-name tier rebind"),
Err(err) => err,
};
assert!(err.to_string().contains("identity no longer matches"));
assert_eq!(err.kind(), std::io::ErrorKind::Other);
let admin_err = err
.get_ref()
.and_then(|source| source.downcast_ref::<crate::client::admin_handler_utils::AdminError>())
.expect("identity mismatch should retain the typed tier error");
assert_eq!(admin_err.code, crate::services::tier::tier::ERR_TIER_INVALID_CONFIG.code);
assert_eq!(new_backend.get_count().await, 0);
oi.user_defined = Arc::new(HashMap::new());
@@ -5834,7 +5995,8 @@ mod tests {
version_id: "remote-version".to_string(),
tier_name: "WARM".to_string(),
backend_identity: Some([1; 32]),
version_id_exact: false,
version_id_exact: true,
version_state: rustfs_filemeta::TransitionVersionState::Exact,
};
let err = state
@@ -5945,7 +6107,8 @@ mod tests {
version_id: "remote-version".to_string(),
tier_name: "WARM".to_string(),
backend_identity: Some([1; 32]),
version_id_exact: false,
version_id_exact: true,
version_state: rustfs_filemeta::TransitionVersionState::Exact,
};
state
@@ -6459,6 +6622,96 @@ mod tests {
assert_eq!(state.transition_rx.len(), 1);
}
#[tokio::test]
#[serial]
async fn scanner_transition_enqueue_waits_for_saturated_queue_to_recover() {
let state = TransitionState::new_with_capacity(1);
let first_object = ObjectInfo {
bucket: "bucket".to_string(),
name: "first".to_string(),
..Default::default()
};
let deferred_object = ObjectInfo {
bucket: "bucket".to_string(),
name: "deferred".to_string(),
..Default::default()
};
let event = crate::bucket::lifecycle::lifecycle::Event {
action: IlmAction::TransitionAction,
..Default::default()
};
assert!(
state.queue_transition_task(&first_object, &event, &LcEventSrc::Scanner).await,
"first scanner transition should fill the queue"
);
let deferred = state.queue_transition_task(&deferred_object, &event, &LcEventSrc::Scanner);
tokio::pin!(deferred);
assert!(
(&mut deferred).now_or_never().is_none(),
"a saturated scanner queue should apply bounded backpressure instead of dropping the task"
);
let first_task = state
.transition_rx
.recv()
.await
.expect("queue should remain open")
.expect("first queued transition task should be present");
state.release_transition(&first_task.obj_info);
assert!(
deferred.await,
"the deferred scanner transition should enqueue as soon as capacity recovers"
);
let recovered_task = state
.transition_rx
.recv()
.await
.expect("queue should remain open")
.expect("deferred transition task should be present");
assert_eq!(recovered_task.obj_info.name, deferred_object.name);
}
#[tokio::test]
#[serial]
async fn scanner_transition_sustained_saturation_schedules_compensation() {
let state = TransitionState::new_with_capacity_and_timeout(1, StdDuration::ZERO);
let first_object = ObjectInfo {
bucket: "saturated-bucket".to_string(),
name: "first".to_string(),
..Default::default()
};
let missed_object = ObjectInfo {
bucket: "saturated-bucket".to_string(),
name: "missed".to_string(),
..Default::default()
};
let event = crate::bucket::lifecycle::lifecycle::Event {
action: IlmAction::TransitionAction,
..Default::default()
};
assert!(
state.queue_transition_task(&first_object, &event, &LcEventSrc::Scanner).await,
"first scanner transition should fill the queue"
);
assert!(
!state
.queue_transition_task(&missed_object, &event, &LcEventSrc::Scanner)
.await,
"a continuously saturated queue should report that the object was not admitted"
);
assert_eq!(state.queue_full_tasks(), 1);
assert_eq!(state.queue_send_timeout_tasks(), 1);
assert_eq!(
state.compensation_scheduled_tasks(),
1,
"timed-out scanner work must schedule a bounded bucket backfill"
);
}
#[tokio::test]
#[serial]
async fn scanner_transition_enqueue_updates_transition_status() {
@@ -6984,6 +7237,46 @@ mod tests {
assert_eq!(state.compensation_pending_tasks(), 1);
}
#[test]
fn poisoned_compensation_set_can_release_completed_bucket() {
let state = TransitionState::new_with_capacity(1);
state
.compensation_buckets
.lock()
.expect("fresh mutex should lock")
.insert("bucket-a".to_string());
let poison_target = Arc::clone(&state.compensation_buckets);
let _ = std::thread::spawn(move || {
let _guard = poison_target.lock().expect("fresh mutex should lock");
panic!("poison compensation set");
})
.join();
state.finish_bucket_compensation("bucket-a");
assert_eq!(state.compensation_pending_tasks(), 0);
assert!(
state.compensation_buckets.lock().is_ok(),
"validated compensation state must clear poison"
);
}
#[test]
fn poisoned_tier_stats_are_reset_before_reuse() {
let state = TransitionState::new_with_capacity(1);
let poison_target = Arc::clone(&state.last_day_stats);
let _ = std::thread::spawn(move || {
let mut stats = poison_target.lock().expect("fresh mutex should lock");
stats.insert("stale".to_string(), LastDayTierStats::default());
panic!("poison tier stats");
})
.join();
state.add_lastday_stats("fresh", TierStats::default());
let stats = state.get_daily_all_tier_stats();
assert!(!stats.contains_key("stale"), "possibly partial statistics must be discarded");
assert!(stats.contains_key("fresh"), "statistics must accept new samples after recovery");
}
#[tokio::test(flavor = "current_thread")]
async fn scanner_transition_state_reports_compensation_pending_buckets() {
let state = TransitionState::new_with_capacity(1);
@@ -7274,6 +7567,23 @@ mod tests {
TransitionState::resize_workers_to(ecstore, original_workers, original_workers, absolute_max);
}
#[tokio::test]
#[serial]
async fn transition_worker_resize_without_runtime_does_not_poison_tracking() {
let (_paths, ecstore) = setup_test_env().await;
let transition_state = runtime_sources::transition_state_handle();
let resize = std::thread::spawn(move || {
TransitionState::resize_workers_to(ecstore, 1, 1, resolve_transition_workers_absolute_max());
})
.join();
assert!(resize.is_ok(), "missing Tokio runtime must not panic while worker tracking is locked");
assert!(
transition_state.workers.lock().is_ok(),
"failed resize must leave worker tracking unpoisoned"
);
}
#[test]
fn should_defer_date_expiry_for_recent_config_update_respects_grace_window() {
let now = OffsetDateTime::now_utc();
@@ -9866,6 +10176,87 @@ mod tests {
(backend, identity_hex)
}
#[cfg(feature = "test-util")]
#[tokio::test]
async fn journal_replay_rejects_unknown_version_state_before_backend_io() {
let (_disk_paths, ecstore) = setup_test_env().await;
let (backend, _) = register_recovery_mock_tier(&ecstore).await;
let identity = TierConfigMgr::acquire_operation_lease(&ecstore.tier_config_mgr(), "WARM")
.await
.expect("mock tier lease should be available")
.backend_identity();
let je = Jentry {
obj_name: "remote/object".to_string(),
version_id: "legacy-version".to_string(),
tier_name: "WARM".to_string(),
backend_identity: Some(identity),
version_id_exact: false,
version_state: rustfs_filemeta::TransitionVersionState::Unknown,
};
let err = crate::bucket::lifecycle::tier_delete_journal::process_tier_delete_journal_entry(ecstore, &je)
.await
.expect_err("unknown journal state must fail before backend IO");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert_eq!(backend.remove_count().await, 0);
}
#[cfg(feature = "test-util")]
#[tokio::test]
async fn journal_replay_deletes_confirmed_exact_provider_token() {
let (_disk_paths, ecstore) = setup_test_env().await;
let (backend, _) = register_recovery_mock_tier(&ecstore).await;
let lease = TierConfigMgr::acquire_operation_lease(&ecstore.tier_config_mgr(), "WARM")
.await
.expect("mock tier lease should be available");
let identity = lease.backend_identity();
backend
.set_put_remote_version(Some("provider-version-token".to_string()))
.await;
lease
.put(
"remote/object",
crate::client::transition_api::ReaderImpl::Body(bytes::Bytes::from_static(b"candidate")),
9,
)
.await
.expect("confirmed remote candidate should be seeded");
backend.set_remove_failure(true);
backend.set_reject_non_empty_remote_versions(true);
let je = Jentry {
obj_name: "remote/object".to_string(),
version_id: "provider-version-token".to_string(),
tier_name: "WARM".to_string(),
backend_identity: Some(identity),
version_id_exact: true,
version_state: rustfs_filemeta::TransitionVersionState::Exact,
};
crate::set_disk::cleanup_rejected_transition_upload_durably(
&lease,
&je.obj_name,
&je.version_id,
true,
Some(ecstore.clone()),
)
.await
.expect("failed immediate cleanup should remain durable in the journal");
assert!(backend.contains(&je.obj_name).await);
backend.set_remove_failure(false);
crate::bucket::lifecycle::tier_delete_journal::process_tier_delete_journal_entry(ecstore, &je)
.await
.expect("identity-bound exact journal must retry confirmed candidate cleanup");
assert!(!backend.contains(&je.obj_name).await);
assert_eq!(backend.exact_remove_count(), 2);
assert_eq!(
backend.remove_versions().await,
vec![("remote/object".to_string(), "provider-version-token".to_string())]
);
}
async fn seed_recoverable_free_version(
disk_paths: &[PathBuf],
bucket: &str,
@@ -9885,6 +10276,7 @@ mod tests {
identity,
);
}
let transition_version_id = Uuid::new_v4();
let mut metadata = FileMeta::new();
metadata
.add_version(FileInfo {
@@ -9893,7 +10285,9 @@ mod tests {
version_id: Some(object_version_id),
transition_status: crate::bucket::lifecycle::lifecycle::TRANSITION_COMPLETE.to_string(),
transitioned_objname: format!("remote/{bucket}/{object}"),
transition_version_id: Some(Uuid::new_v4()),
transition_version_id: Some(transition_version_id),
transition_version: Some(transition_version_id.to_string()),
transition_version_state: rustfs_filemeta::TransitionVersionState::Exact,
transition_tier: "WARM".to_string(),
mod_time: Some(OffsetDateTime::now_utc()),
metadata: transitioned_metadata,
@@ -10897,6 +11291,7 @@ mod tests {
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn concurrent_resend_same_part_commits_one_generation() {
use crate::set_disk::{MultipartCommitBarrier, MultipartCommitPause};
use crate::storage_api_contracts::object::ObjectIO as _;
let (_paths, ecstore) = setup_test_env().await;
@@ -10918,51 +11313,36 @@ mod tests {
})
.collect();
// Two independent causes can produce a spurious lock-acquire timeout
// here, and both must stay covered:
// 1. A lost/stolen fast-lock wakeup could strand a waiter until the
// deadline — fixed for real in fast_lock::shard by bounding each
// notification wait (NOTIFY_WAIT_CAP re-polling).
// 2. Under the full nextest suite on loaded CI disks, the
// *legitimately serialized* cross-disk commits can exceed the
// acquire deadline all by themselves — observed on CI at the 5s
// default and the 30s production default with six resends, and
// again at 60s, which is a hard ceiling: fast_lock clamps every
// requested timeout to MAX_ACQUIRE_TIMEOUT (60s), so raising the
// env override higher is a no-op (the Timeout error still reports
// the requested value). Keep the guard about the correctness
// property, not disk latency: request the full 60s ceiling and cap
// the queue depth at three resends, so the last waiter sits behind
// at most two serialized commits (~12s each on the slowest observed
// CI runner, comfortably inside the deadline). Three concurrent
// resends still race the streaming phase and contend on the commit
// lock, which is all the generation-mixing regression needs.
// `#[serial]` keeps the process-wide env override isolated.
let results = temp_env::async_with_vars([(rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT, Some("60"))], async {
let mut tasks = tokio::task::JoinSet::new();
for payload in candidates.iter().cloned() {
let store = ecstore.clone();
let bucket = bucket.clone();
let upload_id = upload.upload_id.clone();
tasks.spawn(async move {
let mut data = PutObjReader::from_vec(payload.clone());
store
.put_object_part(&bucket, object, &upload_id, 1, &mut data, &ObjectOptions::default())
.await
.map(|info| (info, payload))
});
}
let commit_barrier = MultipartCommitBarrier::install(&bucket, object, MultipartCommitPause::PutPartBeforeLockLost);
let start = Arc::new(tokio::sync::Barrier::new(candidates.len() + 1));
let mut tasks = tokio::task::JoinSet::new();
for payload in candidates.iter().cloned() {
let store = ecstore.clone();
let bucket = bucket.clone();
let upload_id = upload.upload_id.clone();
let start = Arc::clone(&start);
tasks.spawn(async move {
start.wait().await;
let mut data = PutObjReader::from_vec(payload.clone());
store
.put_object_part(&bucket, object, &upload_id, 1, &mut data, &ObjectOptions::default())
.await
.map(|info| (info, payload))
});
}
start.wait().await;
// Every concurrent resend must succeed; the commit lock must never
// starve a waiter into a timeout.
let mut results = Vec::new();
while let Some(joined) = tasks.join_next().await {
let outcome = joined.expect("put_object_part task should not panic");
results.push(outcome.expect("every concurrent same-part resend must succeed without lock timeout"));
}
results
})
.await;
// The first writer holds the uploadId commit lock while the other
// resends reach the same critical section. Releasing it proves the
// handoff without depending on saturated CI disk latency.
commit_barrier.wait_until_paused().await;
commit_barrier.release();
let mut results = Vec::new();
while let Some(joined) = tasks.join_next().await {
let outcome = joined.expect("put_object_part task should not panic");
results.push(outcome.expect("every concurrent same-part resend must succeed without lock timeout"));
}
assert_eq!(results.len(), candidates.len());
// Exactly one generation is visible after the serialized commits, and its
@@ -20,7 +20,10 @@ use tokio_util::sync::CancellationToken;
use tracing::{debug, warn};
use crate::bucket::lifecycle::config_boundary;
use crate::bucket::lifecycle::tier_sweeper::{Jentry, delete_object_from_remote_tier_idempotent_with_manager_and_identity};
use crate::bucket::lifecycle::tier_sweeper::{
Jentry, delete_confirmed_transition_candidate_exact_with_manager_and_identity,
delete_object_from_remote_tier_idempotent_with_manager_and_identity,
};
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::{Error, Result};
use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader};
@@ -42,6 +45,7 @@ const TIER_DELETE_JOURNAL_RECOVERY_INTERVAL: Duration = Duration::from_secs(60);
const TIER_DELETE_JOURNAL_RECOVERY_TIMEOUT: Duration = Duration::from_secs(300);
const TIER_DELETE_JOURNAL_VERSION: u8 = 2;
const TIER_DELETE_JOURNAL_EXACT_VERSION: u8 = 3;
const TIER_DELETE_JOURNAL_STATE_VERSION: u8 = 4;
pub(crate) const TIER_DELETE_JOURNAL_PREFIX: &str = "ilm/tier-delete-journal/";
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
@@ -55,24 +59,35 @@ struct PersistedTierDeleteJournalEntry {
backend_identity: Option<[u8; 32]>,
#[serde(default, skip_serializing_if = "Option::is_none")]
version_id_exact: Option<bool>,
#[serde(default, skip_serializing_if = "Option::is_none")]
version_state: Option<rustfs_filemeta::TransitionVersionState>,
}
impl PersistedTierDeleteJournalEntry {
fn from_jentry(je: &Jentry) -> Self {
Self {
version: if je.version_id_exact {
TIER_DELETE_JOURNAL_EXACT_VERSION
} else if je.backend_identity.is_some() {
fn from_jentry(je: &Jentry) -> Result<Self> {
validate_version_state(je.version_state, &je.version_id, je.version_id_exact)?;
let legacy_unknown = je.version_state == rustfs_filemeta::TransitionVersionState::Unknown;
let version = if legacy_unknown {
if je.backend_identity.is_some() {
TIER_DELETE_JOURNAL_VERSION
} else {
1
},
}
} else {
if je.backend_identity.is_none() {
return Err(Error::other("new tier delete journal entry is missing its backend identity"));
}
TIER_DELETE_JOURNAL_STATE_VERSION
};
Ok(Self {
version,
obj_name: je.obj_name.clone(),
version_id: je.version_id.clone(),
tier_name: je.tier_name.clone(),
backend_identity: je.backend_identity,
version_id_exact: je.version_id_exact.then_some(true),
}
version_state: (!legacy_unknown).then_some(je.version_state),
})
}
fn into_jentry(self) -> Result<Jentry> {
@@ -84,19 +99,23 @@ impl PersistedTierDeleteJournalEntry {
if self.obj_name.is_empty() || self.tier_name.is_empty() {
return Err(Error::other("tier delete journal entry is incomplete"));
}
if self.version != TIER_DELETE_JOURNAL_EXACT_VERSION && self.version_id_exact.unwrap_or(false) {
if self.version != TIER_DELETE_JOURNAL_EXACT_VERSION
&& self.version != TIER_DELETE_JOURNAL_STATE_VERSION
&& self.version_id_exact.unwrap_or(false)
{
return Err(Error::other(
"legacy tier delete journal entry has an unsupported exact version constraint",
));
}
let (backend_identity, version_id_exact) = match self.version {
1 => (None, false),
let (backend_identity, version_id_exact, version_state) = match self.version {
1 => (None, false, rustfs_filemeta::TransitionVersionState::Unknown),
TIER_DELETE_JOURNAL_VERSION => (
Some(
self.backend_identity
.ok_or_else(|| Error::other("tier delete journal v2 entry is missing its backend identity"))?,
),
false,
rustfs_filemeta::TransitionVersionState::Unknown,
),
TIER_DELETE_JOURNAL_EXACT_VERSION => {
if self.version_id.is_empty() || self.version_id_exact != Some(true) {
@@ -108,6 +127,22 @@ impl PersistedTierDeleteJournalEntry {
.ok_or_else(|| Error::other("tier delete journal v3 entry is missing its backend identity"))?,
),
true,
rustfs_filemeta::TransitionVersionState::Exact,
)
}
TIER_DELETE_JOURNAL_STATE_VERSION => {
let state = self
.version_state
.ok_or_else(|| Error::other("tier delete journal v4 entry is missing its version state"))?;
let exact = self.version_id_exact.unwrap_or(false);
validate_version_state(state, &self.version_id, exact)?;
(
Some(
self.backend_identity
.ok_or_else(|| Error::other("tier delete journal v4 entry is missing its backend identity"))?,
),
exact,
state,
)
}
version => return Err(Error::other(format!("unsupported tier delete journal version {version}"))),
@@ -118,10 +153,30 @@ impl PersistedTierDeleteJournalEntry {
tier_name: self.tier_name,
backend_identity,
version_id_exact,
version_state,
})
}
}
fn validate_version_state(
state: rustfs_filemeta::TransitionVersionState,
version_id: &str,
version_id_exact: bool,
) -> Result<()> {
use rustfs_filemeta::TransitionVersionState::{Exact, KnownDisabled, SuspendedNull, Unknown};
let valid = match state {
Unknown => !version_id_exact,
KnownDisabled => version_id.is_empty() && !version_id_exact,
SuspendedNull => version_id == "null" && version_id_exact,
Exact => !version_id.is_empty() && version_id != "null" && version_id_exact,
};
if !valid {
return Err(Error::other("tier delete journal version state conflicts with its version id"));
}
Ok(())
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TierDeleteJournalRecoveryStats {
pub scanned: usize,
@@ -159,7 +214,7 @@ pub(crate) fn decode_tier_delete_journal_entry(data: &[u8]) -> Result<Jentry> {
}
pub(crate) fn encode_tier_delete_journal_entry(je: &Jentry) -> Result<Vec<u8>> {
serde_json::to_vec(&PersistedTierDeleteJournalEntry::from_jentry(je))
serde_json::to_vec(&PersistedTierDeleteJournalEntry::from_jentry(je)?)
.map_err(|err| Error::other(format!("encode tier delete journal failed: {err}")))
}
@@ -209,18 +264,35 @@ where
}
pub async fn process_tier_delete_journal_entry(api: Arc<ECStore>, je: &Jentry) -> std::io::Result<()> {
if je.version_state == rustfs_filemeta::TransitionVersionState::Unknown {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"tier delete journal remote version state is unknown",
));
}
let backend_identity = je
.backend_identity
.ok_or_else(|| std::io::Error::other("legacy tier delete journal has no durable backend identity"))?;
delete_object_from_remote_tier_idempotent_with_manager_and_identity(
&je.obj_name,
&je.version_id,
&je.tier_name,
backend_identity,
&api.tier_config_mgr(),
je.version_id_exact,
)
.await?;
if je.version_id_exact {
delete_confirmed_transition_candidate_exact_with_manager_and_identity(
&je.obj_name,
&je.version_id,
&je.tier_name,
backend_identity,
&api.tier_config_mgr(),
)
.await?;
} else {
delete_object_from_remote_tier_idempotent_with_manager_and_identity(
&je.obj_name,
&je.version_id,
&je.tier_name,
backend_identity,
&api.tier_config_mgr(),
false,
)
.await?;
}
remove_tier_delete_journal_entry(api, je).await
}
@@ -406,8 +478,9 @@ where
#[cfg(test)]
mod tests {
use super::{
TIER_DELETE_JOURNAL_EXACT_VERSION, await_tier_delete_journal_recovery, decode_tier_delete_journal_entry,
encode_tier_delete_journal_entry, record_tier_delete_journal_backend_identity, tier_delete_journal_object_name,
TIER_DELETE_JOURNAL_EXACT_VERSION, TIER_DELETE_JOURNAL_STATE_VERSION, await_tier_delete_journal_recovery,
decode_tier_delete_journal_entry, encode_tier_delete_journal_entry, record_tier_delete_journal_backend_identity,
tier_delete_journal_object_name,
};
use crate::bucket::lifecycle::tier_sweeper::Jentry;
use crate::error::Result;
@@ -420,7 +493,8 @@ mod tests {
version_id: "remote-version".to_string(),
tier_name: "WARM".to_string(),
backend_identity: Some([7; 32]),
version_id_exact: false,
version_id_exact: true,
version_state: rustfs_filemeta::TransitionVersionState::Exact,
}
}
@@ -436,6 +510,7 @@ mod tests {
assert_eq!(decoded.tier_name, je.tier_name);
assert_eq!(decoded.backend_identity, je.backend_identity);
assert_eq!(decoded.version_id_exact, je.version_id_exact);
assert_eq!(decoded.version_state, je.version_state);
}
#[test]
@@ -450,7 +525,7 @@ mod tests {
let persisted: serde_json::Value = serde_json::from_slice(&encoded).expect("exact journal JSON should decode");
let decoded = decode_tier_delete_journal_entry(&encoded).expect("exact journal entry should decode");
assert_eq!(persisted["version"], TIER_DELETE_JOURNAL_EXACT_VERSION);
assert_eq!(persisted["version"], TIER_DELETE_JOURNAL_STATE_VERSION);
assert_eq!(persisted["version_id_exact"], true);
assert!(decoded.version_id_exact);
assert_ne!(tier_delete_journal_object_name(&exact), tier_delete_journal_object_name(&normalized));
@@ -513,6 +588,46 @@ mod tests {
}
}
#[test]
fn tier_delete_journal_rejects_conflicting_v4_version_states() {
let identity = vec![7_u8; 32];
let invalid = [
("known-disabled", "unexpected", false),
("suspended-null", "", true),
("suspended-null", "null", false),
("exact", "", true),
("exact", "null", true),
("exact", "version", false),
("unknown", "version", true),
];
for (state, version_id, exact) in invalid {
let persisted = serde_json::json!({
"version": TIER_DELETE_JOURNAL_STATE_VERSION,
"obj_name": "remote/object",
"version_id": version_id,
"tier_name": "WARM",
"backend_identity": identity,
"version_id_exact": exact.then_some(true),
"version_state": state,
});
let encoded = serde_json::to_vec(&persisted).expect("invalid journal fixture should encode");
decode_tier_delete_journal_entry(&encoded).expect_err("conflicting v4 version state must fail closed");
}
}
#[test]
fn legacy_journals_decode_with_unknown_version_state() {
let v1 = br#"{"version":1,"obj_name":"remote/object","version_id":"opaque","tier_name":"WARM"}"#;
let v2 = br#"{"version":2,"obj_name":"remote/object","version_id":"opaque","tier_name":"WARM","backend_identity":[7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7]}"#;
for payload in [v1.as_slice(), v2.as_slice()] {
let decoded = decode_tier_delete_journal_entry(payload).expect("legacy journal should decode");
assert_eq!(decoded.version_state, rustfs_filemeta::TransitionVersionState::Unknown);
assert!(!decoded.version_id_exact);
}
}
#[test]
fn tier_delete_journal_path_is_stable_and_sanitized() {
let je = journal_entry();
@@ -530,6 +645,8 @@ mod tests {
fn tier_delete_journal_paths_separate_legacy_and_backend_identities() {
let mut legacy = journal_entry();
legacy.backend_identity = None;
legacy.version_id_exact = false;
legacy.version_state = rustfs_filemeta::TransitionVersionState::Unknown;
let mut backend_a = journal_entry();
backend_a.backend_identity = Some([1; 32]);
let mut backend_b = journal_entry();
@@ -575,6 +692,8 @@ mod tests {
fn tier_delete_journal_without_transition_identity_stays_legacy() {
let mut je = journal_entry();
je.backend_identity = None;
je.version_id_exact = false;
je.version_state = rustfs_filemeta::TransitionVersionState::Unknown;
let encoded = encode_tier_delete_journal_entry(&je).expect("legacy journal should remain encodable");
let persisted: serde_json::Value = serde_json::from_slice(&encoded).expect("journal JSON should decode");
@@ -185,6 +185,7 @@ struct ObjSweeper {
transition_status: String,
transition_tier: String,
transition_version_id: String,
transition_version_state: rustfs_filemeta::TransitionVersionState,
remote_object: String,
}
@@ -231,7 +232,9 @@ impl ObjSweeper {
}
pub fn should_remove_remote_object(&self) -> Option<Jentry> {
if self.transition_status != lifecycle::TRANSITION_COMPLETE {
if self.transition_status != lifecycle::TRANSITION_COMPLETE
|| self.transition_version_state == rustfs_filemeta::TransitionVersionState::Unknown
{
return None;
}
@@ -249,7 +252,11 @@ impl ObjSweeper {
version_id: self.transition_version_id.clone(),
tier_name: self.transition_tier.clone(),
backend_identity: None,
version_id_exact: false,
version_id_exact: matches!(
self.transition_version_state,
rustfs_filemeta::TransitionVersionState::SuspendedNull | rustfs_filemeta::TransitionVersionState::Exact
),
version_state: self.transition_version_state,
});
}
None
@@ -286,6 +293,7 @@ pub struct Jentry {
pub(crate) tier_name: String,
pub(crate) backend_identity: Option<TierDestinationId>,
pub(crate) version_id_exact: bool,
pub(crate) version_state: rustfs_filemeta::TransitionVersionState,
}
impl ExpiryOp for Jentry {
@@ -330,7 +338,7 @@ async fn delete_object_from_remote_tier_raw_with_manager(
let lease = TierConfigMgr::acquire_operation_lease(&tier_config_mgr, tier_name)
.await
.map_err(std::io::Error::other)?;
delete_object_from_remote_tier_raw_with_lease(obj_name, rv_id, &lease, false).await
delete_object_from_remote_tier_raw_with_lease(obj_name, rv_id, &lease, false, true).await
}
async fn delete_object_from_remote_tier_raw_with_lease(
@@ -338,8 +346,11 @@ async fn delete_object_from_remote_tier_raw_with_lease(
rv_id: &str,
lease: &TierOperationLease,
version_id_exact: bool,
validate_remote_version_id: bool,
) -> Result<(), std::io::Error> {
lease.validate_remote_version_id(rv_id)?;
if validate_remote_version_id {
lease.validate_remote_version_id(rv_id)?;
}
if remote_delete_breaker_is_open(Instant::now()).await {
metrics::counter!(METRIC_DELETE_REMOTE_BREAKER_TOTAL).increment(1);
@@ -435,7 +446,53 @@ pub(crate) async fn delete_object_from_remote_tier_with_lease_idempotent(
lease: &TierOperationLease,
version_id_exact: bool,
) -> Result<RemoteTierDeleteOutcome, std::io::Error> {
match delete_object_from_remote_tier_raw_with_lease(obj_name, rv_id, lease, version_id_exact).await {
delete_object_from_remote_tier_with_lease_idempotent_inner(obj_name, rv_id, lease, version_id_exact, true).await
}
pub(crate) async fn delete_confirmed_transition_candidate_exact_with_lease_idempotent(
obj_name: &str,
rv_id: &str,
lease: &TierOperationLease,
) -> Result<RemoteTierDeleteOutcome, std::io::Error> {
if rv_id.is_empty() {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidInput,
"confirmed versioned transition candidate requires a non-empty remote version",
));
}
#[cfg(test)]
if obj_name == "remote/empty-guard-probe" {
CONFIRMED_TRANSITION_EMPTY_GUARD_DISPATCHES.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
}
delete_object_from_remote_tier_with_lease_idempotent_inner(obj_name, rv_id, lease, true, false).await
}
#[cfg(test)]
static CONFIRMED_TRANSITION_EMPTY_GUARD_DISPATCHES: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0);
pub(crate) async fn delete_confirmed_transition_candidate_exact_with_manager_and_identity(
obj_name: &str,
rv_id: &str,
tier_name: &str,
backend_identity: TierDestinationId,
tier_config_mgr: &Arc<tokio::sync::RwLock<TierConfigMgr>>,
) -> Result<RemoteTierDeleteOutcome, std::io::Error> {
let lease = TierConfigMgr::acquire_operation_lease_for_backend_identity(tier_config_mgr, tier_name, backend_identity)
.await
.map_err(std::io::Error::other)?;
delete_confirmed_transition_candidate_exact_with_lease_idempotent(obj_name, rv_id, &lease).await
}
async fn delete_object_from_remote_tier_with_lease_idempotent_inner(
obj_name: &str,
rv_id: &str,
lease: &TierOperationLease,
version_id_exact: bool,
validate_remote_version_id: bool,
) -> Result<RemoteTierDeleteOutcome, std::io::Error> {
match delete_object_from_remote_tier_raw_with_lease(obj_name, rv_id, lease, version_id_exact, validate_remote_version_id)
.await
{
Ok(()) => Ok(RemoteTierDeleteOutcome::Deleted),
Err(err) if is_remote_tier_not_found_error(&err) => Ok(RemoteTierDeleteOutcome::AlreadyRemoved),
Err(err) => {
@@ -460,6 +517,7 @@ pub fn transitioned_delete_journal_entry(
versioned: bool,
suspended: bool,
transitioned: &TransitionedObject,
transition_version_state: rustfs_filemeta::TransitionVersionState,
) -> Option<Jentry> {
let sweeper = ObjSweeper {
version_id,
@@ -468,6 +526,7 @@ pub fn transitioned_delete_journal_entry(
transition_status: transitioned.status.clone(),
transition_tier: transitioned.tier.clone(),
transition_version_id: transitioned.version_id.clone(),
transition_version_state,
remote_object: transitioned.name.clone(),
..Default::default()
};
@@ -475,8 +534,13 @@ pub fn transitioned_delete_journal_entry(
sweeper.should_remove_remote_object()
}
pub fn transitioned_force_delete_journal_entry(transitioned: &TransitionedObject) -> Option<Jentry> {
if transitioned.status != lifecycle::TRANSITION_COMPLETE {
pub fn transitioned_force_delete_journal_entry(
transitioned: &TransitionedObject,
transition_version_state: rustfs_filemeta::TransitionVersionState,
) -> Option<Jentry> {
if transitioned.status != lifecycle::TRANSITION_COMPLETE
|| transition_version_state == rustfs_filemeta::TransitionVersionState::Unknown
{
return None;
}
@@ -485,7 +549,11 @@ pub fn transitioned_force_delete_journal_entry(transitioned: &TransitionedObject
version_id: transitioned.version_id.clone(),
tier_name: transitioned.tier.clone(),
backend_identity: None,
version_id_exact: false,
version_id_exact: matches!(
transition_version_state,
rustfs_filemeta::TransitionVersionState::SuspendedNull | rustfs_filemeta::TransitionVersionState::Exact
),
version_state: transition_version_state,
})
}
@@ -494,11 +562,14 @@ mod test {
use crate::client::signer_error::invalid_utf8_header_error;
use super::{
ERR_REMOTE_DELETE_BREAKER_OPEN, ERR_REMOTE_DELETE_LIMITER_CLOSED, RemoteDeleteBreaker, RemoteTierDeleteOutcome,
CONFIRMED_TRANSITION_EMPTY_GUARD_DISPATCHES, ERR_REMOTE_DELETE_BREAKER_OPEN, ERR_REMOTE_DELETE_LIMITER_CLOSED,
RemoteDeleteBreaker, RemoteTierDeleteOutcome, delete_confirmed_transition_candidate_exact_with_manager_and_identity,
delete_object_from_remote_tier_idempotent, delete_object_from_remote_tier_idempotent_with_manager_and_identity,
is_remote_tier_not_found_error, is_signer_header_error, set_remote_tier_delete_test_hook,
should_record_remote_delete_failure,
is_remote_tier_not_found_error, is_signer_header_error, lifecycle, set_remote_tier_delete_test_hook,
should_record_remote_delete_failure, transitioned_delete_journal_entry, transitioned_force_delete_journal_entry,
};
use crate::storage_api_contracts::lifecycle::TransitionedObject;
use rustfs_filemeta::TransitionVersionState;
use std::io::{Error, ErrorKind};
use std::time::{Duration, Instant};
@@ -542,6 +613,43 @@ mod test {
assert!(should_record_remote_delete_failure(&Error::other("NoSuchVersion")));
}
#[test]
fn transitioned_delete_journal_preserves_remote_version_state() {
let cases = [
(TransitionVersionState::Unknown, "legacy-version", None),
(TransitionVersionState::KnownDisabled, "", Some(false)),
(TransitionVersionState::SuspendedNull, "null", Some(true)),
(TransitionVersionState::Exact, "opaque-version", Some(true)),
];
for (state, version_id, expected_exact) in cases {
let transitioned = TransitionedObject {
name: "remote/object".to_string(),
version_id: version_id.to_string(),
tier: "WARM".to_string(),
status: lifecycle::TRANSITION_COMPLETE.to_string(),
..Default::default()
};
let regular = transitioned_delete_journal_entry(None, false, false, &transitioned, state);
let forced = transitioned_force_delete_journal_entry(&transitioned, state);
match expected_exact {
Some(expected_exact) => {
let regular = regular.expect("known version state should produce a regular delete journal entry");
assert_eq!(regular.version_state, state);
assert_eq!(regular.version_id_exact, expected_exact);
let forced = forced.expect("known version state should produce a forced delete journal entry");
assert_eq!(forced.version_state, state);
assert_eq!(forced.version_id_exact, expected_exact);
}
None => {
assert!(regular.is_none());
assert!(forced.is_none());
}
}
}
}
#[tokio::test]
#[serial_test::serial]
async fn idempotent_remote_delete_treats_hooked_nosuchversion_as_already_removed() {
@@ -664,6 +772,55 @@ mod test {
assert_eq!(backend.remove_versions().await, vec![("remote/object".to_string(), String::new())]);
}
#[cfg(feature = "test-util")]
#[tokio::test]
#[serial_test::serial]
async fn confirmed_transition_cleanup_deletes_exact_provider_token() {
CONFIRMED_TRANSITION_EMPTY_GUARD_DISPATCHES.store(0, std::sync::atomic::Ordering::Relaxed);
let manager = crate::services::tier::tier::TierConfigMgr::new();
let backend = crate::services::tier::test_util::register_mock_tier(&manager, "WARM").await;
let lease = crate::services::tier::tier::TierConfigMgr::acquire_operation_lease(&manager, "WARM")
.await
.expect("test tier lease should be available");
let identity = lease.backend_identity();
drop(lease);
backend.set_reject_non_empty_remote_versions(true);
let outcome = delete_confirmed_transition_candidate_exact_with_manager_and_identity(
"remote/object",
"provider-version-token",
"WARM",
identity,
&manager,
)
.await
.expect("confirmed upload compensation should delete the exact provider token");
assert_eq!(outcome, RemoteTierDeleteOutcome::Deleted);
assert_eq!(backend.exact_remove_count(), 1);
assert_eq!(
backend.remove_versions().await,
vec![("remote/object".to_string(), "provider-version-token".to_string())]
);
let err = delete_confirmed_transition_candidate_exact_with_manager_and_identity(
"remote/empty-guard-probe",
"",
"WARM",
identity,
&manager,
)
.await
.expect_err("confirmed versioned cleanup must reject an empty token");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidInput);
assert_eq!(backend.remove_count().await, 1);
assert_eq!(
CONFIRMED_TRANSITION_EMPTY_GUARD_DISPATCHES.load(std::sync::atomic::Ordering::Relaxed),
0,
"empty remote versions must be rejected before exact cleanup dispatch"
);
}
#[test]
fn breaker_opens_at_threshold_and_recovers_after_window() {
let mut breaker = RemoteDeleteBreaker::new(3, Duration::from_secs(30));
@@ -22,7 +22,10 @@ use uuid::Uuid;
use crate::bucket::lifecycle::config_boundary;
use crate::bucket::lifecycle::lifecycle::TRANSITION_COMPLETE;
use crate::bucket::lifecycle::tier_sweeper::delete_object_from_remote_tier_idempotent_with_manager_and_identity;
use crate::bucket::lifecycle::tier_sweeper::{
delete_confirmed_transition_candidate_exact_with_manager_and_identity,
delete_object_from_remote_tier_idempotent_with_manager_and_identity,
};
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::{Error, Result as EcstoreResult};
use crate::object_api::ObjectOptions;
@@ -708,6 +711,21 @@ async fn recover_unknown_upload_outcome(
TransitionCandidateProbe::UnversionedPresent => {
cleanup_recovered_unknown_upload_candidate(api, transaction, TransitionRemoteVersion::unversioned()).await
}
TransitionCandidateProbe::VersionedPresent(version_id)
if Uuid::parse_str(&version_id).is_ok_and(|version_id| version_id.is_nil()) =>
{
delete_confirmed_transition_candidate_exact_with_manager_and_identity(
&transaction.remote_object,
&version_id,
&transaction.tier_name,
transaction.backend_fingerprint,
&api.tier_config_mgr(),
)
.await
.map_err(Error::other)?;
delete_transition_transaction_record(api, transaction.transaction_id).await?;
Ok(TransitionTransactionRecoveryOutcome::RemoteCandidateDeleted)
}
TransitionCandidateProbe::VersionedPresent(version_id) => {
cleanup_recovered_unknown_upload_candidate(api, transaction, TransitionRemoteVersion::versioned(version_id)).await
}
+27
View File
@@ -737,6 +737,9 @@ impl BucketMetadata {
}
BUCKET_TAGGING_CONFIG => {
self.tagging_config_xml = data;
// Drop the parsed form (like lifecycle above) so clearing the
// payload can't leave stale parsed tags to be cached.
self.tagging_config = None;
self.tagging_config_updated_at = updated;
}
BUCKET_QUOTA_CONFIG_FILE => {
@@ -1318,6 +1321,30 @@ mod test {
assert!(bm.lifecycle_config.is_none());
}
/// Companion to the lifecycle case above. `parse_all_configs` skips empty
/// XML rather than clearing, so without the explicit reset a cleared
/// tagging config would keep serving the previously parsed tags.
#[test]
fn tagging_update_config_clears_parsed_config_on_delete() {
let mut bm = BucketMetadata::new("test-bucket");
let tagging_xml = br#"<Tagging><TagSet><Tag><Key>env</Key><Value>prod</Value></Tag></TagSet></Tagging>"#;
bm.update_config(BUCKET_TAGGING_CONFIG, tagging_xml.to_vec())
.expect("tagging config should update");
bm.parse_all_configs().expect("tagging config should parse");
assert!(bm.tagging_config.is_some());
bm.update_config(BUCKET_TAGGING_CONFIG, Vec::new())
.expect("tagging config delete should update metadata");
assert!(bm.tagging_config_xml.is_empty());
assert!(bm.tagging_config.is_none());
// A re-parse must not resurrect them either.
bm.parse_all_configs().expect("cleared tagging should parse");
assert!(bm.tagging_config.is_none());
}
#[tokio::test]
async fn marshal_msg_complete_example() {
// Create a complete BucketMetadata with various configurations
+207 -18
View File
@@ -239,6 +239,35 @@ pub async fn update_bucket_targets_under_transaction_lock(bucket: &str, data: Ve
bucket_meta_sys.update(bucket, BUCKET_TARGETS_FILE, data).await
}
/// Read-modify-write one bucket config file under the metadata system's
/// outer write guard.
///
/// `mutate` sees the freshly loaded on-disk metadata and returns the
/// replacement payload for `config_file` (empty clears it, like
/// [`delete`]). Both the read and the persisted write happen inside the
/// same guard that [`update`] uses, so within this process the rewrite can
/// neither clobber a concurrent update to another config file nor lose a
/// concurrent write to the same one — unlike caching a mutated clone of
/// previously read metadata.
///
/// This guard is process-local. Writers on other nodes still race, exactly
/// as they do for [`update`]: each rewrites the whole metadata file, so the
/// later save wins. What this narrows is the window — from "as stale as the
/// local cache" down to a single metadata read plus write.
pub async fn update_config_with<F>(bucket: &str, config_file: &str, mutate: F) -> Result<OffsetDateTime>
where
F: FnOnce(&BucketMetadata) -> Result<Vec<u8>> + Send,
{
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let _targets_guard = if config_file == BUCKET_TARGETS_FILE {
Some(acquire_bucket_targets_transaction_lock(bucket).await?)
} else {
None
};
let mut bucket_meta_sys = bucket_meta_sys_lock.write().await;
bucket_meta_sys.update_config_with(bucket, config_file, mutate).await
}
pub async fn acquire_bucket_targets_transaction_lock(bucket: &str) -> Result<rustfs_lock::NamespaceLockGuard> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let api = bucket_meta_sys_lock.read().await.object_store();
@@ -603,24 +632,7 @@ impl BucketMetadataSys {
return Err(Error::other("errServerNotInitialized"));
};
if is_meta_bucketname(bucket) {
return Err(Error::other("errInvalidArgument"));
}
let mut bm = match load_bucket_metadata_parse(store, bucket, parse).await {
Ok(res) => res,
Err(err) => {
if !runtime_sources::setup_is_erasure().await
&& !runtime_sources::setup_is_dist_erasure().await
&& is_err_bucket_not_found(&err)
{
BucketMetadata::new(bucket)
} else {
error!("load bucket metadata failed: {}", err);
return Err(err);
}
}
};
let mut bm = Self::load_bucket_metadata_for_update(store, bucket, parse).await?;
let updated = bm.update_config(config_file, data)?;
@@ -629,6 +641,49 @@ impl BucketMetadataSys {
Ok(updated)
}
/// See the free [`update_config_with`]: same load-mutate-persist cycle as
/// [`Self::update`], with the payload computed from the loaded metadata
/// instead of supplied up front. Loads through this system's own store so
/// the read and the persisted write target the same instance.
async fn update_config_with<F>(&mut self, bucket: &str, config_file: &str, mutate: F) -> Result<OffsetDateTime>
where
F: FnOnce(&BucketMetadata) -> Result<Vec<u8>> + Send,
{
let mut bm = Self::load_bucket_metadata_for_update(self.api.clone(), bucket, true).await?;
let data = mutate(&bm)?;
let updated = bm.update_config(config_file, data)?;
self.save(bm).await?;
Ok(updated)
}
/// Load a bucket's on-disk metadata as the base of a config rewrite.
/// Outside erasure setups a missing metadata file degrades to a fresh
/// default (legacy buckets without one); erasure setups fail instead of
/// fabricating state that a quorum may still hold.
async fn load_bucket_metadata_for_update(store: Arc<ECStore>, bucket: &str, parse: bool) -> Result<BucketMetadata> {
if is_meta_bucketname(bucket) {
return Err(Error::other("errInvalidArgument"));
}
match load_bucket_metadata_parse(store, bucket, parse).await {
Ok(res) => Ok(res),
Err(err) => {
if !runtime_sources::setup_is_erasure().await
&& !runtime_sources::setup_is_dist_erasure().await
&& is_err_bucket_not_found(&err)
{
Ok(BucketMetadata::new(bucket))
} else {
error!("load bucket metadata failed: {}", err);
Err(err)
}
}
}
}
async fn save(&self, bm: BucketMetadata) -> Result<()> {
if is_meta_bucketname(&bm.name) {
return Err(Error::other("errInvalidArgument"));
@@ -741,6 +796,8 @@ impl BucketMetadataSys {
if let Some(config) = &bm.policy_config {
Ok((config.clone(), bm.policy_config_updated_at))
} else if !bm.policy_config_json.is_empty() {
Ok((serde_json::from_slice(&bm.policy_config_json)?, bm.policy_config_updated_at))
} else {
Err(Error::ConfigNotFound)
}
@@ -1051,6 +1108,138 @@ mod tests {
);
}
#[tokio::test]
async fn get_bucket_policy_rejects_malformed_cached_policy() {
let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = BucketMetadataSys::new(ecstore);
let mut metadata = BucketMetadata::new("malformed-policy");
metadata.policy_config_json = b"{".to_vec();
sys.set("malformed-policy".to_string(), Arc::new(metadata)).await;
let err = sys
.get_bucket_policy("malformed-policy")
.await
.expect_err("malformed persisted policy must not be treated as missing");
assert!(matches!(err, Error::Io(_)), "malformed persisted policy must surface its parse failure");
}
/// A tagging rewrite through `update_config_with` (the Swift metadata
/// POST path) is persisted: it survives a metadata reload from disk, and
/// an emptied rewrite clears the config in the cached copy too instead of
/// leaving stale parsed tags behind.
#[tokio::test]
async fn update_config_with_persists_tagging_rewrite_across_disk_reload() {
use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
use s3s::dto::Tag;
let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
let mut sys = BucketMetadataSys::new(ecstore);
let bucket = "swift-tagging-bucket";
sys.persist_and_set(BucketMetadata::new(bucket))
.await
.expect("initial metadata should persist");
let tagging = Tagging {
tag_set: vec![Tag {
key: Some("swift-meta-color".to_string()),
value: Some("blue".to_string()),
}],
};
let xml = crate::bucket::utils::serialize::<Tagging>(&tagging).expect("tagging should serialize");
sys.update_config_with(bucket, BUCKET_TAGGING_CONFIG, move |bm| {
assert!(bm.tagging_config.is_none(), "rewrite must see the on-disk state");
Ok(xml)
})
.await
.expect("tagging rewrite should persist");
// Simulate the disk-truth reload that used to lose Swift writes: drop
// the cached entry and lazily re-load from the metadata file.
sys.metadata_map.write().await.clear();
let (tags, _) = sys
.get_tagging_config(bucket)
.await
.expect("tagging must survive a reload from disk");
assert_eq!(tags.tag_set.len(), 1);
assert_eq!(tags.tag_set[0].key.as_deref(), Some("swift-meta-color"));
assert_eq!(tags.tag_set[0].value.as_deref(), Some("blue"));
// An emptied rewrite clears the config everywhere.
sys.update_config_with(bucket, BUCKET_TAGGING_CONFIG, |bm| {
assert!(bm.tagging_config.is_some(), "rewrite must see the persisted tags");
Ok(Vec::new())
})
.await
.expect("clearing rewrite should persist");
assert_eq!(
sys.get_tagging_config(bucket).await.unwrap_err(),
Error::ConfigNotFound,
"cleared tagging must not be served from the cache"
);
sys.metadata_map.write().await.clear();
assert_eq!(
sys.get_tagging_config(bucket).await.unwrap_err(),
Error::ConfigNotFound,
"cleared tagging must not reappear after a reload from disk"
);
}
/// The load and the persisted write share one write guard, so concurrent
/// rewrites of the same config compose instead of clobbering each other.
/// Moving the load outside that guard loses all but the last tag.
#[tokio::test]
async fn concurrent_update_config_with_calls_do_not_lose_writes() {
use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
use s3s::dto::Tag;
let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore)));
let bucket = "swift-tagging-concurrent";
sys.read()
.await
.persist_and_set(BucketMetadata::new(bucket))
.await
.expect("initial metadata should persist");
const WRITERS: usize = 8;
let mut handles = Vec::with_capacity(WRITERS);
for idx in 0..WRITERS {
let sys = sys.clone();
handles.push(tokio::spawn(async move {
sys.write()
.await
.update_config_with(bucket, BUCKET_TAGGING_CONFIG, move |bm| {
// Each writer merges its own tag onto whatever is
// currently persisted — the Swift rewrite shape.
let mut tagging = bm.tagging_config.clone().unwrap_or_else(|| Tagging { tag_set: vec![] });
tagging.tag_set.push(Tag {
key: Some(format!("swift-meta-key{idx}")),
value: Some(idx.to_string()),
});
crate::bucket::utils::serialize::<Tagging>(&tagging).map_err(|e| Error::other(e.to_string()))
})
.await
}));
}
for handle in handles {
handle
.await
.expect("writer task should join")
.expect("rewrite should persist");
}
let (tags, _) = sys
.read()
.await
.get_tagging_config(bucket)
.await
.expect("tagging should be readable");
assert_eq!(tags.tag_set.len(), WRITERS, "every concurrent rewrite must survive: {tags:?}");
}
fn target(bucket: &str, id: &str) -> BucketTarget {
BucketTarget {
source_bucket: bucket.to_string(),
+102 -11
View File
@@ -15,23 +15,26 @@
use super::metadata_sys::get_bucket_metadata_sys;
use crate::error::{Result, StorageError};
use rustfs_policy::policy::{BucketPolicy, BucketPolicyArgs};
use tracing::info;
pub struct PolicySys {}
impl PolicySys {
pub async fn is_allowed(args: &BucketPolicyArgs<'_>) -> bool {
match Self::get(args.bucket).await {
Ok(cfg) => return cfg.is_allowed(args).await,
Err(err) => {
if err != StorageError::ConfigNotFound {
info!("config get err {:?}", err);
}
}
}
args.is_owner
matches!(Self::try_is_allowed(args).await, Ok(true))
}
pub async fn try_is_allowed(args: &BucketPolicyArgs<'_>) -> Result<bool> {
Self::is_allowed_with_policy(args, Self::get(args.bucket).await).await
}
async fn is_allowed_with_policy(args: &BucketPolicyArgs<'_>, policy: Result<BucketPolicy>) -> Result<bool> {
match policy {
Ok(policy) => Ok(policy.is_allowed(args).await),
Err(StorageError::ConfigNotFound) => Ok(args.is_owner),
Err(err) => Err(err),
}
}
pub async fn get(bucket: &str) -> Result<BucketPolicy> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -41,3 +44,91 @@ impl PolicySys {
Ok(cfg)
}
}
#[cfg(test)]
mod tests {
use super::{PolicySys, StorageError};
use rustfs_policy::policy::action::{Action, S3Action};
use rustfs_policy::policy::{BucketPolicy, BucketPolicyArgs};
use std::collections::HashMap;
fn args<'a>(
is_owner: bool,
groups: &'a Option<Vec<String>>,
conditions: &'a HashMap<String, Vec<String>>,
) -> BucketPolicyArgs<'a> {
BucketPolicyArgs {
bucket: "bucket",
action: Action::S3Action(S3Action::GetObjectAction),
is_owner,
account: "account",
groups,
conditions,
object: "object",
}
}
#[tokio::test]
async fn missing_policy_preserves_owner_and_iam_fallback_semantics() {
let groups = None;
let conditions = HashMap::new();
assert!(
PolicySys::is_allowed_with_policy(&args(true, &groups, &conditions), Err(StorageError::ConfigNotFound),)
.await
.expect("missing policy should preserve owner access")
);
assert!(
!PolicySys::is_allowed_with_policy(&args(false, &groups, &conditions), Err(StorageError::ConfigNotFound),)
.await
.expect("missing policy should defer non-owner access to IAM")
);
}
#[tokio::test]
async fn policy_load_failures_propagate() {
let groups = None;
let conditions = HashMap::new();
for (failure, expected_message) in [
(StorageError::Io(std::io::Error::other("policy read failed")), "policy read failed"),
(
StorageError::other("bucket metadata sys not initialized for this instance"),
"bucket metadata sys not initialized for this instance",
),
] {
let result = PolicySys::is_allowed_with_policy(&args(true, &groups, &conditions), Err(failure)).await;
assert!(
matches!(result, Err(StorageError::Io(ref err)) if err.to_string().contains(expected_message)),
"policy I/O and uninitialized metadata failures must propagate instead of granting owner access"
);
}
}
#[tokio::test]
async fn explicit_bucket_deny_precedes_iam_allow() {
let groups = None;
let conditions = HashMap::new();
let policy: BucketPolicy = serde_json::from_str(
r#"{
"Version":"2012-10-17",
"Statement":[{
"Effect":"Deny",
"Principal":{"AWS":"*"},
"Action":["s3:GetObject"],
"Resource":["arn:aws:s3:::bucket/*"]
}]
}"#,
)
.expect("deny policy should parse");
let bucket_allowed = PolicySys::is_allowed_with_policy(&args(true, &groups, &conditions), Ok(policy))
.await
.expect("loaded bucket policy should evaluate");
let iam_allowed = true;
let request_allowed = bucket_allowed && iam_allowed;
assert!(iam_allowed, "test precondition: IAM grants the action");
assert!(!bucket_allowed, "test precondition: bucket policy explicitly denies the action");
assert!(!request_allowed, "explicit bucket Deny must reject before IAM Allow fallback");
}
}
@@ -417,7 +417,7 @@ impl TransitionClient {
bucket: complete_multipart_upload_result.bucket,
key: complete_multipart_upload_result.key,
etag: trim_etag(&complete_multipart_upload_result.etag),
version_id: self.raw_version_id(&h)?.unwrap_or_default().to_string(),
version_id: self.legacy_remote_version_id(&h)?,
location: complete_multipart_upload_result.location,
expiration: exp_time,
expiration_rule_id: rule_id,
@@ -22,7 +22,7 @@ use bytes::Bytes;
use futures::future::join_all;
use http::{HeaderMap, HeaderName, HeaderValue, StatusCode};
use std::io::Error;
use std::sync::RwLock;
use std::sync::{Mutex, MutexGuard, RwLock};
use std::{collections::HashMap, sync::Arc};
use time::{OffsetDateTime, format_description};
use tokio::io::AsyncReadExt;
@@ -46,6 +46,14 @@ use crate::client::utils::base64_encode;
use rustfs_utils::path::trim_etag;
use s3s::header::X_AMZ_EXPIRATION;
fn lock_md5_hasher(
md5_hasher: &Mutex<Option<rustfs_utils::hash::HashAlgorithm>>,
) -> Result<MutexGuard<'_, Option<rustfs_utils::hash::HashAlgorithm>>, std::io::Error> {
md5_hasher
.lock()
.map_err(|_| std::io::Error::other("MD5 hasher state is unavailable"))
}
/// Read exactly `want` bytes for a single multipart part, or fewer if the reader
/// reaches EOF first. Advances the reader so the next call returns the following
/// part. Replaces the previous per-part `read_all()`/`to_vec()`, which drained
@@ -177,7 +185,7 @@ impl TransitionClient {
let length = buf.len();
if opts.send_content_md5 {
let mut md5_hasher = self.md5_hasher.lock().unwrap();
let mut md5_hasher = lock_md5_hasher(&self.md5_hasher)?;
let md5_hash = match md5_hasher.as_mut() {
Some(hasher) => hasher,
None => return Err(std::io::Error::other("MD5 hasher not initialized")),
@@ -370,7 +378,7 @@ impl TransitionClient {
let mut md5_base64: String = "".to_string();
if opts.send_content_md5 {
let mut md5_hasher = clone_self.md5_hasher.lock().unwrap();
let mut md5_hasher = lock_md5_hasher(&clone_self.md5_hasher)?;
let md5_hash = match md5_hasher.as_mut() {
Some(hasher) => hasher,
None => {
@@ -418,6 +426,9 @@ impl TransitionClient {
}
let results = join_all(futures).await;
for result in results {
result?;
}
select! {
err = err_rx.recv() => {
@@ -567,7 +578,7 @@ impl TransitionClient {
key: object_name.to_string(),
etag: trim_etag(h.get("ETag").and_then(|v| v.to_str().ok()).unwrap_or("")),
version_id: self.raw_version_id(h)?.unwrap_or_default().to_string(),
version_id: self.legacy_remote_version_id(h)?,
size,
expiration: exp_time,
expiration_rule_id: rule_id,
@@ -620,10 +631,12 @@ fn collect_complete_parts(parts_info: &HashMap<i64, ObjectPart>, total_parts_cou
#[cfg(test)]
mod tests {
use super::{ObjectPart, ReaderImpl, collect_complete_parts, read_multipart_part};
use super::{ObjectPart, ReaderImpl, collect_complete_parts, lock_md5_hasher, read_multipart_part};
use crate::object_api::GetObjectReader;
use bytes::Bytes;
use rustfs_utils::hash::HashAlgorithm;
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
// Drive a reader through the same per-part loop the multipart stream uses and
// collect the size of every part. Regression for rustfs/rustfs#4811: the old
@@ -733,4 +746,18 @@ mod tests {
"a gap in the parts map must be an error, not a panic"
);
}
#[test]
fn poisoned_md5_state_fails_closed() {
let hasher = Arc::new(Mutex::new(Some(HashAlgorithm::Md5)));
let poison_target = Arc::clone(&hasher);
let _ = std::thread::spawn(move || {
let _guard = poison_target.lock().expect("fresh mutex should lock");
panic!("poison MD5 state");
})
.join();
let error = lock_md5_hasher(&hasher).expect_err("poisoned hash state must not be reused");
assert_eq!(error.kind(), std::io::ErrorKind::Other);
}
}
+1 -1
View File
@@ -201,7 +201,7 @@ impl TransitionClient {
object_name: object_name.to_string(),
object_version_id: opts.version_id,
delete_marker: resp.headers().get(X_AMZ_DELETE_MARKER).map_or(false, |v| v == "true"),
delete_marker_version_id: self.raw_version_id(resp.headers())?.unwrap_or_default().to_string(),
delete_marker_version_id: self.legacy_remote_version_id(resp.headers())?,
..Default::default()
})
}
+119 -1
View File
@@ -46,11 +46,33 @@ impl RemoteVersion {
Self::Unknown | Self::Disabled => None,
}
}
pub(crate) fn exact_request_id(&self) -> Result<Option<&str>, Error> {
match self {
Self::Unknown => Err(Error::new(
ErrorKind::InvalidData,
"remote object version is unknown; exact version routing is unsafe",
)),
Self::Disabled => Ok(None),
Self::SuspendedNull => Ok(Some("null")),
Self::Exact(version_id) => Ok(Some(version_id)),
}
}
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) enum ConditionalCreateCapability {
Unsupported,
IfNoneMatchStar,
GenerationMatchZero,
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) struct ProviderVersionCapabilities {
raw_version_header: Option<&'static str>,
pub(crate) bucket_versioning_state: bool,
pub(crate) list_object_versions: bool,
pub(crate) conditional_create: ConditionalCreateCapability,
pub(crate) exact_get_delete: bool,
}
@@ -62,28 +84,59 @@ impl ProviderVersionCapabilities {
|| tier_type.eq_ignore_ascii_case("r2")
|| tier_type.eq_ignore_ascii_case("wasabi")
{
let list_object_versions = tier_type.eq_ignore_ascii_case("s3")
|| tier_type.eq_ignore_ascii_case("rustfs")
|| tier_type.eq_ignore_ascii_case("minio")
|| tier_type.eq_ignore_ascii_case("r2");
Self {
raw_version_header: Some(X_AMZ_VERSION_ID),
bucket_versioning_state: list_object_versions,
list_object_versions,
conditional_create: if tier_type.eq_ignore_ascii_case("s3") || tier_type.eq_ignore_ascii_case("r2") {
ConditionalCreateCapability::IfNoneMatchStar
} else {
ConditionalCreateCapability::Unsupported
},
exact_get_delete: true,
}
} else if tier_type.eq_ignore_ascii_case("aliyun") {
Self {
raw_version_header: Some(X_OSS_VERSION_ID),
bucket_versioning_state: false,
list_object_versions: false,
conditional_create: ConditionalCreateCapability::Unsupported,
exact_get_delete: true,
}
} else if tier_type.eq_ignore_ascii_case("tencent") {
Self {
raw_version_header: Some(X_COS_VERSION_ID),
bucket_versioning_state: false,
list_object_versions: false,
conditional_create: ConditionalCreateCapability::Unsupported,
exact_get_delete: true,
}
} else if tier_type.eq_ignore_ascii_case("huaweicloud") {
Self {
raw_version_header: Some(X_OBS_VERSION_ID),
bucket_versioning_state: false,
list_object_versions: false,
conditional_create: ConditionalCreateCapability::Unsupported,
exact_get_delete: true,
}
} else if tier_type.eq_ignore_ascii_case("gcs") {
Self {
raw_version_header: None,
bucket_versioning_state: false,
list_object_versions: false,
conditional_create: ConditionalCreateCapability::GenerationMatchZero,
exact_get_delete: false,
}
} else {
Self {
raw_version_header: None,
bucket_versioning_state: false,
list_object_versions: false,
conditional_create: ConditionalCreateCapability::Unsupported,
exact_get_delete: false,
}
}
@@ -143,7 +196,7 @@ fn validate_remote_version_id(version_id: &str) -> Result<(), Error> {
#[cfg(test)]
mod tests {
use super::{BucketVersioningState, ProviderVersionCapabilities, RemoteVersion};
use super::{BucketVersioningState, ConditionalCreateCapability, ProviderVersionCapabilities, RemoteVersion};
use http::{HeaderMap, HeaderValue};
#[test]
@@ -219,6 +272,71 @@ mod tests {
);
}
#[test]
fn provider_capability_matrix_is_conservative_and_provider_specific() {
for (tier_type, state, list, conditional_create, exact_get_delete) in [
("s3", true, true, ConditionalCreateCapability::IfNoneMatchStar, true),
("rustfs", true, true, ConditionalCreateCapability::Unsupported, true),
("minio", true, true, ConditionalCreateCapability::Unsupported, true),
("r2", true, true, ConditionalCreateCapability::IfNoneMatchStar, true),
("wasabi", false, false, ConditionalCreateCapability::Unsupported, true),
("aliyun", false, false, ConditionalCreateCapability::Unsupported, true),
("tencent", false, false, ConditionalCreateCapability::Unsupported, true),
("huaweicloud", false, false, ConditionalCreateCapability::Unsupported, true),
("gcs", false, false, ConditionalCreateCapability::GenerationMatchZero, false),
("azure", false, false, ConditionalCreateCapability::Unsupported, false),
("unsupported", false, false, ConditionalCreateCapability::Unsupported, false),
] {
let capabilities = ProviderVersionCapabilities::for_tier_type(tier_type);
assert_eq!(capabilities.bucket_versioning_state, state, "{tier_type} versioning state");
assert_eq!(capabilities.list_object_versions, list, "{tier_type} version listing");
assert_eq!(capabilities.conditional_create, conditional_create, "{tier_type} conditional create");
assert_eq!(capabilities.exact_get_delete, exact_get_delete, "{tier_type} exact routing");
}
}
#[test]
fn remote_version_states_preserve_unknown_disabled_suspended_and_exact() {
let capabilities = ProviderVersionCapabilities::for_tier_type("s3");
let empty = HeaderMap::new();
let mut null = HeaderMap::new();
null.insert("x-amz-version-id", HeaderValue::from_static("null"));
let mut exact = HeaderMap::new();
exact.insert("x-amz-version-id", HeaderValue::from_static("opaque.generation-7"));
for (headers, state, expected) in [
(&empty, BucketVersioningState::Unknown, RemoteVersion::Unknown),
(&empty, BucketVersioningState::Disabled, RemoteVersion::Disabled),
(&empty, BucketVersioningState::Suspended, RemoteVersion::Unknown),
(&empty, BucketVersioningState::Enabled, RemoteVersion::Unknown),
(&null, BucketVersioningState::Suspended, RemoteVersion::SuspendedNull),
(
&exact,
BucketVersioningState::Enabled,
RemoteVersion::Exact("opaque.generation-7".to_string()),
),
] {
assert_eq!(
capabilities
.remote_version(headers, state)
.expect("version state should normalize"),
expected
);
}
}
#[test]
fn exact_request_routing_fails_closed_for_unknown_versions() {
for (version, expected) in [
(RemoteVersion::Disabled, None),
(RemoteVersion::SuspendedNull, Some("null")),
(RemoteVersion::Exact("opaque-v1".to_string()), Some("opaque-v1")),
] {
assert_eq!(version.exact_request_id().expect("known version state"), expected);
}
assert!(RemoteVersion::Unknown.exact_request_id().is_err());
}
#[test]
fn provider_version_rejects_empty_or_oversized_headers() {
let oversized = "v".repeat(1025);
+33 -2
View File
@@ -31,7 +31,7 @@ use crate::client::{
},
constants::{UNSIGNED_PAYLOAD, UNSIGNED_PAYLOAD_TRAILER},
credentials::{CredContext, Credentials, SignatureType, Static},
provider_versions::{BucketVersioningState, ProviderVersionCapabilities},
provider_versions::{BucketVersioningState, ProviderVersionCapabilities, RemoteVersion},
signer_error,
};
use crate::{client::checksum::ChecksumMode, object_api::GetObjectReader};
@@ -332,6 +332,22 @@ impl TransitionClient {
self.provider_version_capabilities().raw_version_id(headers)
}
pub(crate) fn remote_version(
&self,
headers: &HeaderMap,
versioning: BucketVersioningState,
) -> Result<RemoteVersion, std::io::Error> {
self.provider_version_capabilities().remote_version(headers, versioning)
}
pub(crate) fn legacy_remote_version_id(&self, headers: &HeaderMap) -> Result<String, std::io::Error> {
Ok(self
.remote_version(headers, BucketVersioningState::Unknown)?
.exact_id()
.unwrap_or_default()
.to_string())
}
fn trace_errors_only_off(&self) {
if let Ok(mut trace_errors_only) = self.trace_errors_only.lock() {
*trace_errors_only = false;
@@ -1095,6 +1111,16 @@ impl Default for ObjectInfo {
}
}
impl ObjectInfo {
pub(crate) fn remote_version(
&self,
capabilities: ProviderVersionCapabilities,
versioning: BucketVersioningState,
) -> Result<RemoteVersion, std::io::Error> {
capabilities.remote_version(&self.metadata, versioning)
}
}
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct RestoreInfo {
ongoing_restore: bool,
@@ -1414,7 +1440,7 @@ mod tests {
MAX_S3_CLIENT_RESPONSE_SIZE, MAX_S3_ERROR_RESPONSE_SIZE, SignatureType, build_tls_config, collect_response_body,
signer_error_to_io_error, to_object_info_for_provider, validate_header_values, with_rustls_init_guard,
};
use crate::client::provider_versions::ProviderVersionCapabilities;
use crate::client::provider_versions::{BucketVersioningState, ProviderVersionCapabilities, RemoteVersion};
use http::{HeaderMap, HeaderValue};
use http_body_util::Full;
use hyper::body::Bytes;
@@ -1539,6 +1565,11 @@ mod tests {
.expect("opaque provider version should parse");
assert_eq!(info.version_id, None);
assert_eq!(
info.remote_version(ProviderVersionCapabilities::for_tier_type("tencent"), BucketVersioningState::Enabled,)
.expect("opaque response version should remain available"),
RemoteVersion::Exact("opaque.version_01".to_string())
);
assert_eq!(
info.metadata.get("x-cos-version-id").and_then(|value| value.to_str().ok()),
Some("opaque.version_01")
+135 -19
View File
@@ -14,7 +14,7 @@
use crate::cluster::rpc::http_auth::RPC_CONTENT_SHA256_HEADER;
use crate::cluster::rpc::{gen_tonic_signature_headers, normalize_tonic_rpc_audience};
use crate::disk::error::{DiskError, Error as DiskErrorType};
use crate::disk::error::{DiskError, Error as DiskErrorType, RpcStatusError};
use crate::runtime::sources as runtime_sources;
use http::Uri;
use rustfs_protos::{
@@ -107,10 +107,79 @@ pub async fn node_service_time_out_client_no_auth(
node_service_time_out_client(addr, TonicInterceptor::NoOp(NoOpInterceptor)).await
}
/// The typed `tonic::Status` an internode RPC failure was converted from, if
/// this error carries one.
pub(crate) fn embedded_tonic_status(io_err: &std::io::Error) -> Option<&tonic::Status> {
io_err.get_ref()?.downcast_ref::<RpcStatusError>().map(RpcStatusError::status)
}
/// Decide whether a gRPC status reports a peer we cannot currently reach,
/// rather than an application outcome from a live peer.
///
/// `Unavailable` is the one code that means "no service behind this channel":
/// the client transport raises it when the connection is broken, and the
/// server's own not-ready gates use it deliberately.
///
/// `Unknown` is the client transport's escape hatch for a cause it could not
/// map to a code — tower's "Service was not ready: <cause>", an h2 error with
/// no gRPC mapping. Our handlers never return it, so there its message is the
/// only evidence available and the anchored needles decide.
///
/// Every other code is an answer from a live peer and is never a transport
/// failure, whatever its message says. That distinction is the point of
/// classifying by code: a peer relaying its own downstream trouble as
/// `Internal("connection refused ...")`, or a handler interpolating a local
/// `io::Error` into `Status::internal`, answered us perfectly well. Marking it
/// offline over that text is the bug this classification replaces. Likewise a
/// `Cancelled` "Timeout expired" from the per-RPC channel deadline means the
/// peer is slow, not gone; gating it would turn load into a partition.
pub(crate) fn is_network_like_status(status: &tonic::Status) -> bool {
match status.code() {
tonic::Code::Unavailable => true,
tonic::Code::Unknown => message_has_network_needle(&status.to_string()),
_ => false,
}
}
/// Substring fallback for failures that only exist as text: dial errors
/// wrapped by `get_client`, remote `error_info` payloads, and statuses
/// flattened through `format!`. Needles must stay anchored to transport
/// context — a bare word like "unavailable" also matches application text
/// (e.g. a bucket named "unavailable-logs") and would take a healthy peer
/// offline.
pub(crate) fn message_has_network_needle(message: &str) -> bool {
let message = message.to_ascii_lowercase();
[
"temporarily offline",
"transport error",
// tonic >= 0.14 renders Code::Unavailable as
// `code: 'The service is currently unavailable'`.
"code: 'the service is currently unavailable'",
// RUSTFS_COMPAT_TODO(tonic-013-status-render): releases up to 1.0.0-alpha.38 shipped tonic 0.13, which rendered the same status as `status: Unavailable`, and peers relay that text in error_info. Remove after the minimum supported RustFS peer version ships tonic >= 0.14.
"status: unavailable",
"error trying to connect",
"connection refused",
"connection reset",
"broken pipe",
"not connected",
"unexpected eof",
"timed out",
"deadline has elapsed",
"connection closed",
"connection aborted",
"tcp connect error",
]
.iter()
.any(|needle| message.contains(needle))
}
pub(crate) fn is_network_like_disk_error(err: &DiskErrorType) -> bool {
match err {
DiskError::Timeout => true,
DiskError::Io(io_err) => {
if let Some(status) = embedded_tonic_status(io_err) {
return is_network_like_status(status);
}
if matches!(
io_err.kind(),
ErrorKind::TimedOut
@@ -124,24 +193,7 @@ pub(crate) fn is_network_like_disk_error(err: &DiskErrorType) -> bool {
return true;
}
let message = io_err.to_string().to_ascii_lowercase();
[
"transport error",
"unavailable",
"error trying to connect",
"connection refused",
"connection reset",
"broken pipe",
"not connected",
"unexpected eof",
"timed out",
"deadline has elapsed",
"connection closed",
"connection aborted",
"tcp connect error",
]
.iter()
.any(|needle| message.contains(needle))
message_has_network_needle(&io_err.to_string())
}
_ => false,
}
@@ -269,6 +321,70 @@ mod tests {
let _ = provider.shutdown();
}
#[test]
fn network_like_disk_error_uses_typed_status_code() {
// Transport-level Unavailable statuses justify retry/eviction.
assert!(is_network_like_disk_error(&DiskError::from(tonic::Status::unavailable(
"storage layer is not initialized"
))));
// Application statuses from a live peer must not look network-like,
// even when their message contains transport-sounding words.
assert!(!is_network_like_disk_error(&DiskError::from(tonic::Status::internal(
"failed to heal bucket \"unavailable-logs\""
))));
assert!(!is_network_like_disk_error(&DiskError::from(tonic::Status::unauthenticated(
"No valid auth token"
))));
// A slow peer that blew the per-RPC deadline is still answering.
assert!(!is_network_like_disk_error(&DiskError::from(tonic::Status::cancelled("Timeout expired"))));
}
#[test]
fn embedded_tonic_status_is_recovered_across_error_conversions() {
// DiskError and StorageError share one wrapper, so a status keeps its
// typed classification whichever error it was converted into first.
let from_storage: DiskErrorType = crate::error::Error::from(tonic::Status::unavailable("peer gone")).into();
let DiskError::Io(io_err) = &from_storage else {
panic!("status-derived disk error should stay an Io error");
};
assert_eq!(embedded_tonic_status(io_err).map(|status| status.code()), Some(tonic::Code::Unavailable));
let from_disk = crate::error::Error::from(DiskError::from(tonic::Status::unavailable("peer gone")));
let crate::error::Error::Io(io_err) = &from_disk else {
panic!("status-derived storage error should stay an Io error");
};
assert_eq!(embedded_tonic_status(io_err).map(|status| status.code()), Some(tonic::Code::Unavailable));
}
#[test]
fn network_like_disk_error_ignores_transport_words_in_application_statuses() {
// Same contract as the peer client: a status the peer answered with
// is not a transport failure, so it must not drive a reconnect even
// when its message describes one.
assert!(!is_network_like_disk_error(&DiskError::from(tonic::Status::internal(
"connection refused while dialing downstream backend"
))));
assert!(!is_network_like_disk_error(&DiskError::from(tonic::Status::unauthenticated(
"connection reset while validating token"
))));
}
#[test]
fn network_like_disk_error_requires_anchored_unavailable_needle() {
// Regression: a bare "unavailable" needle used to match application
// text such as a bucket name.
assert!(!is_network_like_disk_error(&DiskError::other("bucket \"unavailable-logs\" not found")));
// Anchored renderings of a flattened Unavailable status still match.
assert!(is_network_like_disk_error(&DiskError::other(
"code: 'The service is currently unavailable', message: \"peer gone\""
)));
assert!(is_network_like_disk_error(&DiskError::other(
"status: Unavailable, message: \"peer gone\""
)));
assert!(is_network_like_disk_error(&DiskError::other("connection refused")));
assert!(!is_network_like_disk_error(&DiskError::FileNotFound));
}
#[test]
fn test_signature_interceptor_keeps_auth_headers() {
ensure_test_rpc_secret();
@@ -13,8 +13,8 @@
// limitations under the License.
use crate::cluster::rpc::client::{
TonicInterceptor, gen_tonic_signature_interceptor, heal_control_time_out_client, node_service_time_out_client,
tier_mutation_control_time_out_client,
TonicInterceptor, embedded_tonic_status, gen_tonic_signature_interceptor, heal_control_time_out_client,
is_network_like_status, message_has_network_needle, node_service_time_out_client, tier_mutation_control_time_out_client,
};
use crate::cluster::rpc::{set_tonic_canonical_body_digest, verify_tonic_rpc_response_proof};
use crate::error::{Error, Result};
@@ -471,26 +471,21 @@ impl PeerRestClient {
self.offline.store(false, Ordering::Release);
}
/// Whether this failure means the peer is unreachable, so it should be
/// gated offline and its connection evicted.
///
/// RPC failures are classified by their typed gRPC code first
/// (`is_network_like_status`); an application error from a live peer must
/// never take it offline no matter what its message says. The substring
/// fallback only covers failures that exist purely as text, such as the
/// dial errors `get_client` wraps.
fn is_network_like_error(err: &Error) -> bool {
let message = err.to_string().to_ascii_lowercase();
[
"temporarily offline",
"transport error",
"unavailable",
"error trying to connect",
"connection refused",
"connection reset",
"broken pipe",
"not connected",
"unexpected eof",
"timed out",
"deadline has elapsed",
"connection closed",
"connection aborted",
"tcp connect error",
]
.iter()
.any(|needle| message.contains(needle))
if let Error::Io(io_err) = err
&& let Some(status) = embedded_tonic_status(io_err)
{
return is_network_like_status(status);
}
message_has_network_needle(&err.to_string())
}
fn mark_offline_and_spawn_recovery(&self) {
@@ -1702,30 +1697,17 @@ fn tier_config_reload_connection_outcome(err: Error) -> TierConfigReloadOutcome
}
fn is_tier_config_reload_connection_failure(err: &Error) -> bool {
let message = err.to_string().to_ascii_lowercase();
let message = err.to_string();
// A bare "unavailable" is only trusted inside the local dial-failure
// wrapper from `get_client`, never in application text.
if message
.to_ascii_lowercase()
.split_once("can not get client, err:")
.is_some_and(|(_, local_error)| local_error.contains("unavailable"))
{
return true;
}
[
"temporarily offline",
"transport error",
"error trying to connect",
"connection refused",
"connection reset",
"connection closed",
"connection aborted",
"broken pipe",
"not connected",
"unexpected eof",
"timed out",
"deadline has elapsed",
"tcp connect error",
]
.iter()
.any(|needle| message.contains(needle))
message_has_network_needle(&message)
}
fn tier_config_reload_remote_failure(error_info: Option<String>) -> TierConfigReloadOutcome {
@@ -2151,6 +2133,126 @@ mod tests {
assert!(!PeerRestClient::is_network_like_error(&Error::NotImplemented));
}
#[test]
fn peer_rest_client_network_classifier_uses_typed_status_code() {
// The one code that means "nothing is answering on this channel".
assert!(PeerRestClient::is_network_like_error(&Error::from(tonic::Status::unavailable(
"storage layer is not initialized"
))));
// Application statuses from a live peer must not mark it offline,
// even when their message contains transport-sounding words.
assert!(!PeerRestClient::is_network_like_error(&Error::from(tonic::Status::internal(
"failed to reload metadata for bucket \"unavailable-logs\""
))));
assert!(!PeerRestClient::is_network_like_error(&Error::from(tonic::Status::unauthenticated(
"No valid auth token"
))));
// A request-budget expiry answered by a live peer is an application
// outcome, not a transport failure.
assert!(!PeerRestClient::is_network_like_error(&Error::from(tonic::Status::deadline_exceeded(
"heal control request expired"
))));
// Unknown is the transport's escape hatch for a cause it could not
// map, and our handlers never return it, so there the text decides.
assert!(PeerRestClient::is_network_like_error(&Error::from(tonic::Status::unknown(
"Service was not ready: transport error"
))));
assert!(!PeerRestClient::is_network_like_error(&Error::from(tonic::Status::unknown(
"peer response unknown"
))));
}
#[test]
fn peer_rest_client_network_classifier_ignores_transport_words_in_application_statuses() {
// The reason classification reads the code rather than the text: a
// peer that answers is reachable, even when what it says describes a
// connection failure of its own. A handler interpolating a local
// io::Error into Status::internal, or relaying trouble with its own
// downstream, must not cost us the channel to a healthy peer.
for status in [
tonic::Status::internal("connection refused while dialing downstream backend"),
tonic::Status::internal("write failed: broken pipe"),
tonic::Status::unauthenticated("connection reset while validating token"),
tonic::Status::failed_precondition("scanner lease timed out"),
tonic::Status::deadline_exceeded("heal control request timed out"),
] {
let rendered = status.to_string();
assert!(
!PeerRestClient::is_network_like_error(&Error::from(status)),
"an answered application status must not mark the peer offline: {rendered}"
);
}
}
#[test]
fn peer_rest_client_network_classifier_keeps_slow_peers_online() {
// The per-RPC channel deadline (RUSTFS_INTERNODE_RPC_TIMEOUT, 30s)
// surfaces as Cancelled "Timeout expired" carrying the transport
// cause as its source. A peer that is merely slow must stay online:
// gating it would spend a full recovery cycle fast-failing every RPC
// to a host that is still answering, turning load into a partition.
let timeout_status = tonic::Status::cancelled("Timeout expired");
assert!(!PeerRestClient::is_network_like_error(&Error::from(timeout_status)));
let sourced = tonic::Status::from_error(Box::new(std::io::Error::other("Timeout expired")));
assert!(
std::error::Error::source(&sourced).is_some(),
"the transport builds this status through Status::from_error, which attaches the cause"
);
assert!(!PeerRestClient::is_network_like_error(&Error::from(sourced)));
}
#[test]
fn rpc_status_errors_keep_their_rendering_and_hide_peer_metadata() {
let err = Error::from(tonic::Status::unavailable("peer gone"));
assert_eq!(
err.to_string(),
"Io error: code: 'The service is currently unavailable', message: \"peer gone\""
);
// tonic's own Debug prints the MetadataMap, i.e. every response header
// the peer sent; those must not reach a log through this error.
let mut status = tonic::Status::unavailable("peer gone");
status
.metadata_mut()
.insert("authorization", "Bearer secret".parse().expect("valid header value"));
let rendered = format!("{:?}", Error::from(status));
assert!(!rendered.contains("Bearer secret"), "{rendered}");
assert!(!rendered.contains("MetadataMap"), "{rendered}");
}
#[test]
fn peer_rest_client_network_classifier_ignores_application_text_containing_unavailable() {
// Regression: a bare "unavailable" needle used to match application
// strings like these and take a healthy peer offline.
assert!(!PeerRestClient::is_network_like_error(&Error::other(
"peer replication statistics provider is unavailable"
)));
assert!(!PeerRestClient::is_network_like_error(&Error::other(
"bucket \"unavailable-logs\" not found"
)));
// Anchored renderings of a flattened Unavailable status still match:
// tonic >= 0.14 form ...
assert!(PeerRestClient::is_network_like_error(&Error::other(
"peer tier mutation commit RPC failed: code: 'The service is currently unavailable', message: \"peer gone\""
)));
// ... which is only anchored as long as tonic renders Unavailable this
// way. A tonic bump that reworded it leaves the typed path correct but
// this needle stale, so pin the coupling rather than discover it in a
// partition.
assert!(
tonic::Status::unavailable("peer gone")
.to_string()
.to_ascii_lowercase()
.contains("code: 'the service is currently unavailable'"),
"tonic reworded Code::Unavailable; update the anchored needle"
);
// ... and the tonic <= 0.13 form peers may relay in error_info.
assert!(PeerRestClient::is_network_like_error(&Error::other(
"peer tier mutation commit RPC failed: status: Unavailable, message: \"peer gone\""
)));
}
#[test]
fn tier_config_reload_outcome_keeps_tonic_and_remote_errors_typed() {
assert!(matches!(
@@ -2193,6 +2295,14 @@ mod tests {
tier_config_reload_connection_outcome(Error::other("can not get client, err: connection unavailable")),
TierConfigReloadOutcome::TransientReconnect(_)
));
// The bare word is trusted only to the right of the dial-failure
// prefix, not anywhere in the message.
assert!(matches!(
tier_config_reload_connection_outcome(Error::other(
"bucket unavailable-logs rejected it, then: can not get client, err: some other reason"
)),
TierConfigReloadOutcome::Terminal(_)
));
}
#[tokio::test]
@@ -2495,6 +2605,39 @@ mod tests {
assert!(!client.offline.load(Ordering::Acquire));
}
#[tokio::test]
async fn peer_rest_client_finalize_result_keeps_online_for_app_errors_mentioning_unavailable() {
// Regression: application error text containing "unavailable" (a
// remote error_info payload, or a bucket named "unavailable-logs" in
// a typed application status) must not take a healthy peer offline.
let client = test_peer_client();
let err = client
.finalize_result::<()>(Err(Error::other("peer replication statistics provider is unavailable")))
.await
.expect_err("application error should still be returned");
assert!(err.to_string().contains("provider is unavailable"));
assert!(!client.offline.load(Ordering::Acquire));
let err = client
.finalize_result::<()>(Err(Error::from(tonic::Status::internal(
"failed to reload metadata for bucket \"unavailable-logs\"",
))))
.await
.expect_err("application status should still be returned");
assert!(err.to_string().contains("unavailable-logs"));
assert!(!client.offline.load(Ordering::Acquire));
}
#[tokio::test]
async fn peer_rest_client_finalize_result_marks_offline_for_typed_unavailable_status() {
let client = test_peer_client();
client
.finalize_result::<()>(Err(Error::from(tonic::Status::unavailable("storage layer is not initialized"))))
.await
.expect_err("network error should still be returned");
assert!(client.offline.load(Ordering::Acquire));
}
#[tokio::test(flavor = "current_thread")]
async fn peer_rest_recovery_probe_logs_keep_request_id_span_context() {
let logs = CapturedLogs::default();
+71 -5
View File
@@ -24,8 +24,8 @@ use crate::cluster::rpc::internode_data_transport::{
use crate::disk::error::{Error, Result};
use crate::disk::{
BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, DeleteOptions, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation,
DiskOption, FileInfoVersions, FileReader, FileWriter, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one,
DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq, ReadMultipleResp, ReadOptions,
RenameDataResp, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one,
disk_store::{
DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE,
get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout,
@@ -48,9 +48,10 @@ use rustfs_protos::proto_gen::node_service::RenamePartRequest;
use rustfs_protos::proto_gen::node_service::{
BatchReadVersionRequest, BatchReadVersionResponse, CheckPartsRequest, DeletePathsRequest, DeleteRequest,
DeleteVersionRequest, DeleteVersionsRequest, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, ListVolumesRequest,
MakeVolumeRequest, MakeVolumesRequest, ReadAllRequest, ReadMetadataRequest, ReadMultipleRequest, ReadMultipleResponse,
ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest, RenameFileRequest, StatVolumeRequest,
UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest, node_service_client::NodeServiceClient,
MakeVolumeRequest, MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, ReadMetadataRequest,
ReadMultipleRequest, ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest,
RenameFileRequest, SettlePartTransactionRequest, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest,
WriteAllRequest, WriteMetadataRequest, node_service_client::NodeServiceClient,
};
use serde::{Serialize, de::DeserializeOwned};
use std::{
@@ -2481,6 +2482,71 @@ impl DiskAPI for RemoteDisk {
.await
}
#[tracing::instrument(level = "trace", skip_all)]
async fn prepare_part_transaction(
&self,
src_volume: &str,
src_path: &str,
dst_volume: &str,
dst_path: &str,
meta: Bytes,
) -> Result<()> {
self.execute_with_timeout(
|| async {
let mut client = self
.get_client()
.await
.map_err(|err| Error::other(format!("can not get client, err: {err}")))?;
let mut request = Request::new(PreparePartTransactionRequest {
disk: self.endpoint.to_string(),
src_volume: src_volume.to_string(),
src_path: src_path.to_string(),
dst_volume: dst_volume.to_string(),
dst_path: dst_path.to_string(),
meta,
});
let canonical_body = rustfs_protos::canonical_prepare_part_transaction_request_body(request.get_ref());
attach_mutation_body_digest(&mut request, canonical_body, "prepare_part_transaction")?;
let response = client.prepare_part_transaction(request).await?.into_inner();
if !response.success {
return Err(response.error.unwrap_or_default().into());
}
Ok(())
},
get_max_timeout_duration(),
)
.await
}
#[tracing::instrument(level = "trace", skip_all)]
async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
self.execute_with_timeout(
|| async {
let mut client = self
.get_client()
.await
.map_err(|err| Error::other(format!("can not get client, err: {err}")))?;
let mut request = Request::new(SettlePartTransactionRequest {
disk: self.endpoint.to_string(),
volume: volume.to_string(),
path: path.to_string(),
rollback: action == PartTransactionAction::Rollback,
});
let canonical_body = rustfs_protos::canonical_settle_part_transaction_request_body(request.get_ref());
attach_mutation_body_digest(&mut request, canonical_body, "settle_part_transaction")?;
let response = client.settle_part_transaction(request).await?.into_inner();
if !response.success {
return Err(response.error.unwrap_or_default().into());
}
Ok(())
},
get_max_timeout_duration(),
)
.await
}
#[tracing::instrument(level = "trace", skip_all)]
async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()> {
trace!(
+1
View File
@@ -2543,6 +2543,7 @@ mod tests {
data_dir: None,
delete_marker: false,
transitioned_object: Default::default(),
transition_version_state: Default::default(),
restore_ongoing: false,
restore_expires: None,
user_tags: Arc::new(String::new()),
+242 -4
View File
@@ -16,6 +16,7 @@
use crate::disk::error_reduce::count_errs;
use crate::error::{Error, Result};
use crate::layout::set_heal::{formats_to_drives_info, new_heal_format_sets};
use crate::multipart_listing::paginate_multipart_listing;
use crate::storage_api_contracts::{
bucket::{BucketInfo, BucketOperations, BucketOptions, DeleteBucketOptions, MakeBucketOptions},
list::{StorageListObjectVersionsInfo, StorageListObjectsV2Info, StorageObjectInfoOrErr, StorageWalkOptions},
@@ -49,7 +50,10 @@ use rustfs_filemeta::FileInfo;
use rustfs_lock::NamespaceLockWrapper;
use rustfs_madmin::heal_commands::HealResultItem;
use rustfs_utils::{crc_hash, path::path_join_buf, sip_hash};
use std::{collections::HashMap, sync::Arc};
use std::{
collections::{HashMap, HashSet},
sync::Arc,
};
use tokio::sync::RwLock;
use tokio::sync::broadcast::{Receiver, Sender};
use tokio::time::Duration;
@@ -63,6 +67,8 @@ type ListObjectVersionsInfo = StorageListObjectVersionsInfo<ObjectInfo>;
type ObjectInfoOrErr = StorageObjectInfoOrErr<ObjectInfo, Error>;
type WalkOptions = StorageWalkOptions<fn(&FileInfo) -> bool>;
const LIST_MULTIPART_SETS_CONCURRENCY: usize = 4;
#[derive(Debug, Clone)]
pub struct Sets {
pub id: Uuid,
@@ -799,9 +805,52 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for Sets {
delimiter: Option<String>,
max_uploads: usize,
) -> Result<ListMultipartsInfo> {
self.get_disks_by_key(prefix)
.list_multipart_uploads(bucket, prefix, key_marker, upload_id_marker, delimiter, max_uploads)
.await
let per_set_limit = max_uploads.saturating_add(1);
let results = futures::stream::iter(self.disk_set.iter().cloned())
.map(|set| {
let key_marker = key_marker.clone();
let upload_id_marker = upload_id_marker.clone();
let delimiter = delimiter.clone();
async move {
set.list_multipart_uploads(bucket, prefix, key_marker, upload_id_marker, delimiter, per_set_limit)
.await
}
})
.buffer_unordered(LIST_MULTIPART_SETS_CONCURRENCY)
.collect::<Vec<_>>()
.await;
let mut uploads = Vec::new();
let mut common_prefixes = HashSet::new();
let mut source_truncated = false;
for result in results {
let page = result?;
uploads.extend(page.uploads);
common_prefixes.extend(page.common_prefixes);
source_truncated |= page.is_truncated;
}
let page = paginate_multipart_listing(
uploads,
common_prefixes.into_iter().collect(),
key_marker.as_deref(),
key_marker.as_ref().and(upload_id_marker.as_deref()),
max_uploads,
source_truncated,
);
Ok(ListMultipartsInfo {
key_marker,
upload_id_marker,
next_key_marker: page.next_key_marker,
next_upload_id_marker: page.next_upload_id_marker,
max_uploads,
is_truncated: page.is_truncated,
uploads: page.uploads,
common_prefixes: page.common_prefixes,
prefix: prefix.to_owned(),
delimiter,
})
}
#[tracing::instrument(skip(self))]
async fn new_multipart_upload(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<MultipartUploadResult> {
@@ -1111,6 +1160,7 @@ mod tests {
use crate::layout::endpoints::SetupType;
use crate::storage_api_contracts::heal::HealOperations as _;
use crate::storage_api_contracts::list::ListOperations as _;
use crate::storage_api_contracts::multipart::MultipartOperations as _;
use rustfs_lock::client::local::LocalClient;
use serial_test::serial;
@@ -1248,6 +1298,194 @@ mod tests {
assert_eq!(result, (Some(3), Some(1), Some(0)));
}
async fn multipart_listing_test_sets() -> (Vec<tempfile::TempDir>, Arc<Sets>) {
let format = FormatV3::new(2, 2);
let mut temp_dirs = Vec::new();
let mut all_endpoints = Vec::new();
let mut disk_sets = Vec::new();
for set_index in 0..2 {
let mut endpoints = Vec::new();
let mut disks = Vec::new();
for disk_index in 0..2 {
let temp_dir = tempfile::tempdir().expect("tempdir should be created");
let mut endpoint = Endpoint::try_from(temp_dir.path().to_str().expect("tempdir path should be utf8"))
.expect("endpoint should parse");
endpoint.set_pool_index(0);
endpoint.set_set_index(set_index);
endpoint.set_disk_index(disk_index);
let disk = new_disk(
&endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await
.expect("disk should be created");
let mut disk_format = format.clone();
disk_format.erasure.this = format.erasure.sets[set_index][disk_index];
save_format_file(&Some(disk.clone()), &Some(disk_format))
.await
.expect("format should be saved");
temp_dirs.push(temp_dir);
all_endpoints.push(endpoint.clone());
endpoints.push(endpoint);
disks.push(Some(disk));
}
disk_sets.push(
SetDisks::new(
"test-owner".to_string(),
Arc::new(RwLock::new(disks)),
2,
1,
0,
set_index,
endpoints,
format.clone(),
vec![Arc::new(LocalClient::new()), Arc::new(LocalClient::new())],
)
.await,
);
}
let sets = Arc::new(Sets {
id: format.id,
disk_set: disk_sets,
pool_idx: 0,
endpoints: PoolEndpoints {
legacy: false,
set_count: 2,
drives_per_set: 2,
endpoints: Endpoints::from(all_endpoints),
cmd_line: String::new(),
platform: String::new(),
},
format,
parity_count: 1,
set_count: 2,
set_drive_count: 2,
default_parity_count: 1,
distribution_algo: DistributionAlgoVersion::V1,
exit_signal: None,
ctx: bootstrap_ctx(),
});
(temp_dirs, sets)
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn list_multipart_uploads_merges_all_sets_without_pagination_loss() {
let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::Erasure).await;
let (_temp_dirs, sets) = multipart_listing_test_sets().await;
let bucket = format!("multipart-list-{}", Uuid::new_v4().simple());
sets.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created");
let mut keys_by_set = [Vec::new(), Vec::new()];
for index in 0..100 {
let key = format!("logs/{index:03}.bin");
let set_index = sets.get_hashed_set_index(&key);
if keys_by_set[set_index].len() < 2 {
keys_by_set[set_index].push(key);
}
if keys_by_set.iter().all(|keys| keys.len() == 2) {
break;
}
}
assert!(keys_by_set.iter().all(|keys| keys.len() == 2), "test keys must span both sets");
let repeated_key = keys_by_set[0][0].clone();
let mut expected = Vec::new();
for key in keys_by_set.iter().flatten() {
let upload = sets
.new_multipart_upload(&bucket, key, &ObjectOptions::default())
.await
.expect("multipart upload should be created");
expected.push((key.clone(), upload.upload_id));
}
let second = sets
.new_multipart_upload(&bucket, &repeated_key, &ObjectOptions::default())
.await
.expect("second upload for the same key should be created");
expected.push((repeated_key, second.upload_id));
expected.sort();
let mut actual = Vec::new();
let mut key_marker = None;
let mut upload_id_marker = None;
for _ in 0..expected.len() + 1 {
let page = sets
.list_multipart_uploads(&bucket, "logs/", key_marker.clone(), upload_id_marker.clone(), None, 2)
.await
.expect("multipart page should list across every set");
assert!(page.uploads.len() <= 2);
actual.extend(
page.uploads
.iter()
.map(|upload| (upload.object.clone(), upload.upload_id.clone())),
);
if !page.is_truncated {
break;
}
key_marker = page.next_key_marker;
upload_id_marker = page.next_upload_id_marker;
}
assert_eq!(actual, expected, "set-level merge must return every upload exactly once");
let mut deduped = actual.clone();
deduped.dedup();
assert_eq!(deduped.len(), actual.len(), "set-level pagination must not duplicate uploads");
let mut nested_by_set = [None, None];
for index in 0..100 {
let key = format!("nested/group-{index:03}/file.bin");
let set_index = sets.get_hashed_set_index(&key);
nested_by_set[set_index].get_or_insert(key);
if nested_by_set.iter().all(Option::is_some) {
break;
}
}
for key in nested_by_set.iter().flatten() {
sets.new_multipart_upload(&bucket, key, &ObjectOptions::default())
.await
.expect("nested multipart upload should be created");
}
let mut expected_prefixes = nested_by_set
.iter()
.flatten()
.map(|key| {
key.rsplit_once('/')
.expect("nested key should contain a delimiter")
.0
.to_string()
+ "/"
})
.collect::<Vec<_>>();
expected_prefixes.sort();
let first = sets
.list_multipart_uploads(&bucket, "nested/", None, None, Some("/".to_string()), 1)
.await
.expect("first delimiter page should list across every set");
assert!(first.is_truncated);
assert_eq!(first.common_prefixes, expected_prefixes[..1]);
let second = sets
.list_multipart_uploads(
&bucket,
"nested/",
first.next_key_marker,
first.next_upload_id_marker,
Some("/".to_string()),
1,
)
.await
.expect("second delimiter page should list across every set");
assert!(!second.is_truncated);
assert_eq!(second.common_prefixes, expected_prefixes[1..]);
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn sets_list_objects_v2_lists_objects_within_the_pool() {
+152 -45
View File
@@ -13,9 +13,9 @@
// limitations under the License.
use crate::disk::{
CheckPartsResp, DeleteOptions, DiskAPI, DiskError, DiskInfo, DiskInfoOptions, DiskLocation, Endpoint, Error,
FileInfoVersions, MmapCopyStageMetrics, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, Result,
UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
CheckPartsResp, DataDirDeleteStatus, DeleteOptions, DiskAPI, DiskError, DiskInfo, DiskInfoOptions, DiskLocation, Endpoint,
Error, FileInfoVersions, MmapCopyStageMetrics, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, Result,
SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
health_state::{
RuntimeDriveHealthState, classify_drive_recovery, get_drive_returning_probe_interval,
get_drive_returning_success_threshold, get_drive_suspect_failure_threshold, record_drive_offline_duration,
@@ -35,7 +35,7 @@ use std::{
Arc,
atomic::{AtomicI64, AtomicU32, AtomicU64, Ordering},
},
time::Duration,
time::{Duration, SystemTime, UNIX_EPOCH},
};
use tokio::{sync::RwLock, time};
use tokio_util::sync::CancellationToken;
@@ -339,21 +339,19 @@ impl Drop for DiskHealthWaitingGuard<'_> {
impl DiskHealthTracker {
/// Create a new disk health tracker
pub fn new() -> Self {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos() as i64;
let now = current_unix_time();
let now_nanos = unix_nanos(now);
Self {
last_success: AtomicI64::new(now),
last_started: AtomicI64::new(now),
last_success: AtomicI64::new(now_nanos),
last_started: AtomicI64::new(now_nanos),
status: AtomicU32::new(DISK_HEALTH_OK),
waiting: AtomicU32::new(0),
runtime_state: AtomicU32::new(RuntimeDriveHealthState::Online as u32),
consecutive_failures: AtomicU32::new(0),
consecutive_successes: AtomicU32::new(0),
offline_since_unix_secs: AtomicI64::new(0),
last_transition_unix_secs: AtomicI64::new(now / 1_000_000_000),
last_transition_unix_secs: AtomicI64::new(unix_secs_i64(now)),
last_capacity_total: AtomicU64::new(0),
last_capacity_used: AtomicU64::new(0),
last_capacity_free: AtomicU64::new(0),
@@ -363,11 +361,7 @@ impl DiskHealthTracker {
/// Log a successful operation
pub fn log_success(&self) {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos() as i64;
self.last_success.store(now, Ordering::Relaxed);
self.last_success.store(current_unix_nanos(), Ordering::Relaxed);
}
pub fn record_capacity_probe(&self, total: u64, used: u64, free: u64) {
@@ -429,11 +423,14 @@ impl DiskHealthTracker {
}
pub fn offline_duration(&self) -> Option<Duration> {
self.offline_duration_at(current_unix_secs())
}
fn offline_duration_at(&self, now: u64) -> Option<Duration> {
let offline_since = self.offline_since_unix_secs.load(Ordering::Acquire);
if offline_since <= 0 {
return None;
}
let now = current_unix_secs();
Some(Duration::from_secs(now.saturating_sub(offline_since as u64)))
}
@@ -492,6 +489,12 @@ impl DiskHealthTracker {
/// Remote disks are marked faulty on timeout/network errors; the init loop retries with the
/// same [`DiskStore`] handles, which would otherwise fail immediately at `is_faulty()`.
pub fn reset_for_store_init_retry(&self, endpoint: &Endpoint) {
self.reset_for_store_init_retry_at(endpoint, current_unix_time());
}
fn reset_for_store_init_retry_at(&self, endpoint: &Endpoint, now: Duration) {
let now_nanos = unix_nanos(now);
let now_secs = unix_secs_i64(now);
self.status.store(DISK_HEALTH_OK, Ordering::Release);
self.runtime_state
.store(RuntimeDriveHealthState::Online as u32, Ordering::Release);
@@ -499,11 +502,9 @@ impl DiskHealthTracker {
self.consecutive_successes.store(0, Ordering::Release);
self.offline_since_unix_secs.store(0, Ordering::Release);
self.waiting.store(0, Ordering::Release);
let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap();
let now_nanos = now.as_nanos() as i64;
self.last_success.store(now_nanos, Ordering::Relaxed);
self.last_started.store(now_nanos, Ordering::Relaxed);
self.last_transition_unix_secs.store(now.as_secs() as i64, Ordering::Release);
self.last_transition_unix_secs.store(now_secs, Ordering::Release);
record_drive_runtime_state(endpoint, RuntimeDriveHealthState::Online);
}
@@ -612,10 +613,33 @@ impl DiskHealthTracker {
}
fn current_unix_secs() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_secs()
// Zero is reserved as "not recorded" by health timestamp atomics.
current_unix_time().as_secs().max(1)
}
fn current_unix_nanos() -> i64 {
unix_nanos(current_unix_time())
}
fn current_unix_time() -> Duration {
unix_time_since_epoch(SystemTime::now())
}
fn unix_time_since_epoch(time: SystemTime) -> Duration {
time.duration_since(UNIX_EPOCH).unwrap_or(Duration::ZERO)
}
fn unix_nanos(time: Duration) -> i64 {
i64::try_from(time.as_nanos()).unwrap_or(i64::MAX)
}
fn unix_secs_i64(time: Duration) -> i64 {
i64::try_from(time.as_secs()).unwrap_or(i64::MAX)
}
fn elapsed_since(last_nanos: i64, now_nanos: i64) -> Duration {
let elapsed_nanos = now_nanos.saturating_sub(last_nanos).max(0);
Duration::from_nanos(u64::try_from(elapsed_nanos).unwrap_or(u64::MAX))
}
impl Default for DiskHealthTracker {
@@ -635,11 +659,7 @@ struct HealthDiskCtxValue {
impl HealthDiskCtxValue {
fn log_success(&self) {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos() as i64;
self.last_success.store(now, Ordering::Relaxed);
self.last_success.store(current_unix_nanos(), Ordering::Relaxed);
}
}
@@ -774,12 +794,7 @@ impl LocalDiskWrapper {
}
let last_success_nanos = health.last_success.load(Ordering::Relaxed);
let elapsed = Duration::from_nanos(
(std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos() as i64 - last_success_nanos) as u64
);
let elapsed = elapsed_since(last_success_nanos, current_unix_nanos());
if elapsed < SKIP_IF_SUCCESS_BEFORE {
continue;
@@ -1091,11 +1106,7 @@ impl LocalDiskWrapper {
self.check_disk_stale().await?;
// Record operation start
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos() as i64;
self.health.last_started.store(now, Ordering::Relaxed);
self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
let _waiting_guard = self.health.waiting_guard();
if timeout_duration == Duration::ZERO {
@@ -1317,11 +1328,7 @@ impl DiskAPI for LocalDiskWrapper {
}
// Record operation start
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos() as i64;
self.health.last_started.store(now, Ordering::Relaxed);
self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
self.health.increment_waiting();
// Execute the operation
@@ -1342,6 +1349,30 @@ impl DiskAPI for LocalDiskWrapper {
.await
}
async fn acquire_snapshot_lease(&self, volume: &str, path: &str) -> Result<SnapshotLeaseToken> {
self.track_disk_health(
|| async { self.disk.acquire_snapshot_lease(volume, path).await },
get_max_timeout_duration(),
)
.await
}
async fn release_snapshot_lease(&self, volume: &str, path: &str, token: SnapshotLeaseToken) -> Result<()> {
self.track_disk_health(
|| async { self.disk.release_snapshot_lease(volume, path, token).await },
get_max_timeout_duration(),
)
.await
}
async fn delete_data_dir(&self, volume: &str, path: &str, opts: DeleteOptions) -> Result<DataDirDeleteStatus> {
self.track_disk_health(
|| async { self.disk.delete_data_dir(volume, path, opts).await },
get_max_timeout_duration(),
)
.await
}
async fn write_metadata(&self, org_volume: &str, volume: &str, path: &str, fi: FileInfo) -> Result<()> {
self.track_disk_health(
|| async { self.disk.write_metadata(org_volume, volume, path, fi).await },
@@ -1473,6 +1504,33 @@ impl DiskAPI for LocalDiskWrapper {
.await
}
async fn prepare_part_transaction(
&self,
src_volume: &str,
src_path: &str,
dst_volume: &str,
dst_path: &str,
meta: Bytes,
) -> Result<()> {
self.track_disk_health(
|| async {
self.disk
.prepare_part_transaction(src_volume, src_path, dst_volume, dst_path, meta)
.await
},
get_max_timeout_duration(),
)
.await
}
async fn settle_part_transaction(&self, volume: &str, path: &str, action: crate::disk::PartTransactionAction) -> Result<()> {
self.track_disk_health(
|| async { self.disk.settle_part_transaction(volume, path, action).await },
get_max_timeout_duration(),
)
.await
}
async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()> {
self.track_disk_health(|| async { self.disk.delete(volume, path, opt).await }, get_max_timeout_duration())
.await
@@ -2243,4 +2301,53 @@ mod tests {
assert!(health.mark_offline(&endpoint, "again"));
assert!(health.is_faulty());
}
#[test]
fn unix_time_clamps_epoch_and_pre_epoch_to_zero() {
let before_epoch = UNIX_EPOCH
.checked_sub(Duration::from_nanos(1))
.expect("one nanosecond before the Unix epoch should be representable");
assert_eq!(unix_time_since_epoch(UNIX_EPOCH), Duration::ZERO);
assert_eq!(unix_time_since_epoch(before_epoch), Duration::ZERO);
}
#[test]
fn elapsed_and_offline_duration_saturate_on_clock_rollback() {
let health = DiskHealthTracker::new();
health.offline_since_unix_secs.store(10, Ordering::Release);
assert_eq!(elapsed_since(10, 12), Duration::from_nanos(2));
assert_eq!(elapsed_since(10, 9), Duration::ZERO);
assert_eq!(health.offline_duration_at(9), Some(Duration::ZERO));
}
#[test]
fn pre_epoch_retry_reset_updates_the_complete_health_state() {
let endpoint = Endpoint::try_from("/tmp/reset-store-init-retry-pre-epoch").expect("endpoint should parse");
let health = DiskHealthTracker::new();
health.status.store(DISK_HEALTH_FAULTY, Ordering::Release);
health
.runtime_state
.store(RuntimeDriveHealthState::Offline as u32, Ordering::Release);
health.consecutive_failures.store(3, Ordering::Release);
health.consecutive_successes.store(2, Ordering::Release);
health.offline_since_unix_secs.store(11, Ordering::Release);
health.waiting.store(4, Ordering::Release);
health.last_success.store(12, Ordering::Release);
health.last_started.store(13, Ordering::Release);
health.last_transition_unix_secs.store(14, Ordering::Release);
health.reset_for_store_init_retry_at(&endpoint, unix_time_since_epoch(UNIX_EPOCH - Duration::from_nanos(1)));
assert_eq!(health.status.load(Ordering::Acquire), DISK_HEALTH_OK);
assert_eq!(health.runtime_state(), RuntimeDriveHealthState::Online);
assert_eq!(health.consecutive_failures.load(Ordering::Acquire), 0);
assert_eq!(health.consecutive_successes.load(Ordering::Acquire), 0);
assert_eq!(health.offline_since_unix_secs.load(Ordering::Acquire), 0);
assert_eq!(health.waiting.load(Ordering::Acquire), 0);
assert_eq!(health.last_success.load(Ordering::Acquire), 0);
assert_eq!(health.last_started.load(Ordering::Acquire), 0);
assert_eq!(health.last_transition_unix_secs.load(Ordering::Acquire), 0);
}
}
+46 -1
View File
@@ -337,9 +337,54 @@ impl From<DiskError> for std::io::Error {
}
}
/// The single in-band representation of a failed internode RPC: it carries the
/// typed `tonic::Status` so failure classifiers can read the gRPC code instead
/// of substring-matching the rendered message (see `is_network_like_status`).
/// Both `DiskError` and `StorageError` wrap statuses in this type, so one
/// downcast recovers the status regardless of which error the status was
/// converted into first.
pub(crate) struct RpcStatusError(tonic::Status);
impl RpcStatusError {
pub(crate) fn status(&self) -> &tonic::Status {
&self.0
}
}
impl From<tonic::Status> for RpcStatusError {
fn from(status: tonic::Status) -> Self {
Self(status)
}
}
impl std::fmt::Display for RpcStatusError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
std::fmt::Display::fmt(&self.0, f)
}
}
/// `tonic::Status`'s own `Debug` prints its `MetadataMap`, i.e. every response
/// header and trailer the peer sent. Those are remote-controlled and can carry
/// credentials injected by a proxy in front of the peer, so keep them out of
/// anything that reaches a log.
impl std::fmt::Debug for RpcStatusError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("RpcStatusError")
.field("code", &self.0.code())
.field("message", &self.0.message())
.finish()
}
}
impl StdError for RpcStatusError {
fn source(&self) -> Option<&(dyn StdError + 'static)> {
Some(&self.0)
}
}
impl From<tonic::Status> for DiskError {
fn from(e: tonic::Status) -> Self {
DiskError::other(e.message().to_string())
DiskError::Io(io::Error::other(RpcStatusError(e)))
}
}
+608 -27
View File
@@ -18,10 +18,12 @@ use crate::data_usage::local_snapshot::ensure_data_usage_layout;
use crate::disk::disk_store::{get_drive_walkdir_stall_timeout, get_object_disk_read_timeout};
use crate::disk::{
BUCKET_META_PREFIX, CHECK_PART_FILE_CORRUPT, CHECK_PART_FILE_NOT_FOUND, CHECK_PART_SUCCESS, CHECK_PART_UNKNOWN,
CHECK_PART_VOLUME_NOT_FOUND, CheckPartsResp, DeleteOptions, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskMetrics,
FileInfoVersions, FileReader, FileWriter, MmapCopyStageMetrics, OldCurrentSize, RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET,
RUSTFS_META_TMP_DELETED_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE,
STORAGE_FORMAT_FILE_BACKUP, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, conv_part_err_to_int,
CHECK_PART_VOLUME_NOT_FOUND, CheckPartsResp, DataDirDeleteStatus, DeleteOptions, DiskAPI, DiskInfo, DiskInfoOptions,
DiskLocation, DiskMetrics, FileInfoVersions, FileReader, FileWriter, MmapCopyStageMetrics, OldCurrentSize,
PART_TRANSACTION_NEW_META, PART_TRANSACTION_OLD_META, PART_TRANSACTION_ROLLBACK, PartTransactionAction, RUSTFS_META_BUCKET,
RUSTFS_META_TMP_BUCKET, RUSTFS_META_TMP_DELETED_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
STORAGE_FORMAT_FILE, STORAGE_FORMAT_FILE_BACKUP, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
conv_part_err_to_int,
endpoint::Endpoint,
error::{DiskError, Error, FileAccessDeniedWithContext, Result},
error_conv::{to_access_error, to_file_error, to_unformatted_disk_error, to_volume_error},
@@ -65,7 +67,7 @@ use tokio::fs::{self, File};
#[cfg(not(unix))]
use tokio::io::AsyncReadExt;
use tokio::io::{AsyncRead, AsyncSeekExt, AsyncWrite, AsyncWriteExt, ErrorKind, ReadBuf};
use tokio::sync::{Notify, RwLock, Semaphore};
use tokio::sync::{Mutex, Notify, RwLock, Semaphore};
use tokio::time::{Instant, Sleep, interval_at, timeout};
use tracing::{debug, error, info, warn};
use uuid::Uuid;
@@ -80,6 +82,10 @@ const ENV_BITROT_SIZE_MISMATCH_RETRY_COUNT: &str = "RUSTFS_BITROT_SIZE_MISMATCH_
const ENV_BITROT_SIZE_MISMATCH_RETRY_DELAY_MS: &str = "RUSTFS_BITROT_SIZE_MISMATCH_RETRY_DELAY_MS";
const DEFAULT_BITROT_SIZE_MISMATCH_RETRY_COUNT: u64 = 2;
const DEFAULT_BITROT_SIZE_MISMATCH_RETRY_DELAY_MS: u64 = 100;
const PART_TRANSACTION_OLD_DATA: &str = "old.data";
const PART_TRANSACTION_OLD_DATA_ABSENT: &str = "old.data.absent";
const PART_TRANSACTION_OLD_META_ABSENT: &str = "old.meta.absent";
const PART_TRANSACTION_PUBLISH_META: &str = "publish.meta";
enum ReadAllError {
Open(std::io::Error),
Disk(DiskError),
@@ -141,6 +147,28 @@ fn remove_dir_all_if_exists(path: &Path) -> std::io::Result<()> {
}
}
fn snapshot_part_transaction_file(src: &Path, backup: &Path, absent: &Path) -> std::io::Result<()> {
match std::fs::symlink_metadata(src) {
Ok(metadata) if metadata.is_file() => std::fs::hard_link(src, backup),
Ok(_) => Err(std::io::Error::new(ErrorKind::InvalidData, "multipart transaction source is not a file")),
Err(err) if err.kind() == ErrorKind::NotFound => std::fs::write(absent, []),
Err(err) => Err(err),
}
}
fn restore_part_transaction_file(current: &Path, backup: &Path, absent: &Path, restore: &Path) -> std::io::Result<()> {
match std::fs::symlink_metadata(backup) {
Ok(metadata) if metadata.is_file() => {
remove_file_if_exists(restore)?;
std::fs::hard_link(backup, restore)?;
std::fs::rename(restore, current)
}
Ok(_) => Err(std::io::Error::new(ErrorKind::InvalidData, "multipart transaction backup is not a file")),
Err(err) if err.kind() == ErrorKind::NotFound && absent.is_file() => remove_file_if_exists(current),
Err(err) => Err(err),
}
}
fn rollback_committed_rename_std(
dst_file_path: &Path,
new_data_path: Option<&Path>,
@@ -3829,6 +3857,25 @@ pub struct LocalDisk {
exit_signal: Option<tokio::sync::broadcast::Sender<()>>,
io_backend: Arc<dyn LocalIoBackend>,
file_sync_permits: Arc<Semaphore>,
snapshot_leases: Arc<Mutex<SnapshotLeaseRegistry>>,
}
#[derive(Clone, Debug, Eq, Hash, PartialEq)]
struct SnapshotLeaseKey {
volume: String,
path: String,
}
#[derive(Default)]
struct SnapshotLeaseEntry {
tokens: HashSet<SnapshotLeaseToken>,
pending_delete: Option<DeleteOptions>,
deleting: bool,
}
#[derive(Default)]
struct SnapshotLeaseRegistry {
entries: HashMap<SnapshotLeaseKey, SnapshotLeaseEntry>,
}
impl Drop for LocalDisk {
@@ -4065,6 +4112,7 @@ impl LocalDisk {
exit_signal: None,
io_backend: build_local_io_backend(root.clone()),
file_sync_permits: os::disk_file_sync_limiter(&root),
snapshot_leases: Arc::new(Mutex::new(SnapshotLeaseRegistry::default())),
};
let (info, _root) = get_disk_info(root.clone()).await.inspect_err(|err| {
log_startup_disk_error("get_disk_info", &root, err);
@@ -4549,6 +4597,23 @@ impl LocalDisk {
Ok(())
}
async fn delete_unleased(&self, volume: &str, path: &str, opt: &DeleteOptions) -> Result<()> {
let volume_dir = self.get_bucket_path(volume)?;
if !skip_access_checks(volume)
&& let Err(e) = access(&volume_dir).await
{
return Err(to_access_error(e, DiskError::VolumeAccessDenied).into());
}
let file_path = self.get_object_path(volume, path)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
self.delete_file(&volume_dir, &file_path, opt.recursive, opt.immediate)
.await?;
// A deleted shard must not remain readable through the io_uring fd cache.
self.io_backend.invalidate_cached_fds_under(volume, path);
Ok(())
}
#[tracing::instrument(level = "trace", skip_all)]
#[async_recursion::async_recursion]
async fn delete_file(
@@ -6189,27 +6254,7 @@ impl DiskAPI for LocalDisk {
#[tracing::instrument(level = "trace", skip_all)]
async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()> {
crate::hp_guard!("LocalDisk::delete");
let volume_dir = self.get_bucket_path(volume)?;
if !skip_access_checks(volume)
&& let Err(e) = access(&volume_dir).await
{
return Err(to_access_error(e, DiskError::VolumeAccessDenied).into());
}
let file_path = self.get_object_path(volume, path)?;
check_path_length(file_path.to_string_lossy().to_string().as_str())?;
self.delete_file(&volume_dir, &file_path, opt.recursive, opt.immediate)
.await?;
// The inode is unlinked, but a cached descriptor would keep it readable —
// a deleted shard must not keep answering reads (backlog#1145). The part
// numbers under `path` are not known here, so this is the one caller that
// needs the predicate form.
self.io_backend.invalidate_cached_fds_under(volume, path);
Ok(())
self.delete_unleased(volume, path, &opt).await
}
#[tracing::instrument(level = "trace", skip_all)]
@@ -6447,6 +6492,151 @@ impl DiskAPI for LocalDisk {
Ok(resp)
}
#[tracing::instrument(level = "trace", skip_all)]
async fn prepare_part_transaction(
&self,
src_volume: &str,
src_path: &str,
dst_volume: &str,
dst_path: &str,
meta: Bytes,
) -> Result<()> {
let src_volume_dir = self.get_bucket_path(src_volume)?;
let dst_volume_dir = self.get_bucket_path(dst_volume)?;
if !skip_access_checks(src_volume) {
super::fs::access_std(&src_volume_dir).map_err(|err| to_access_error(err, DiskError::VolumeAccessDenied))?;
}
if !skip_access_checks(dst_volume) {
super::fs::access_std(&dst_volume_dir).map_err(|err| to_access_error(err, DiskError::VolumeAccessDenied))?;
}
let src_file_path = self.get_object_path(src_volume, src_path)?;
let dst_file_path = self.get_object_path(dst_volume, dst_path)?;
let dst_meta_path = self.get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
let transaction_path = self.get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?;
for path in [&src_file_path, &dst_file_path, &dst_meta_path, &transaction_path] {
check_path_length(path.to_string_lossy().as_ref())?;
}
let durability = effective_durability(dst_volume);
tokio::task::spawn_blocking(move || {
let source = std::fs::symlink_metadata(&src_file_path).map_err(to_file_error)?;
if !source.is_file() {
return Err(DiskError::FileAccessDenied);
}
if transaction_path.exists() {
return Err(DiskError::FileAccessDenied);
}
let Some(transaction_parent) = transaction_path.parent() else {
return Err(DiskError::InvalidPath);
};
std::fs::create_dir_all(transaction_parent).map_err(to_file_error)?;
let staging_path = transaction_parent.join(format!(".part-txn-{}", Uuid::new_v4()));
std::fs::create_dir(&staging_path).map_err(to_file_error)?;
let prepare_result = (|| -> std::io::Result<()> {
snapshot_part_transaction_file(
&dst_file_path,
&staging_path.join(PART_TRANSACTION_OLD_DATA),
&staging_path.join(PART_TRANSACTION_OLD_DATA_ABSENT),
)?;
snapshot_part_transaction_file(
&dst_meta_path,
&staging_path.join(PART_TRANSACTION_OLD_META),
&staging_path.join(PART_TRANSACTION_OLD_META_ABSENT),
)?;
let mut new_meta = std::fs::OpenOptions::new()
.create_new(true)
.write(true)
.open(staging_path.join(PART_TRANSACTION_NEW_META))?;
std::io::Write::write_all(&mut new_meta, &meta)?;
if durability.syncs_commit_metadata() {
new_meta.sync_data()?;
os::fsync_dir_std(&staging_path)?;
}
std::fs::rename(&staging_path, &transaction_path)?;
if durability.syncs_commit_metadata() {
os::fsync_dir_std(transaction_parent)?;
}
Ok(())
})();
if let Err(err) = prepare_result {
let _ = remove_dir_all_if_exists(&staging_path);
return Err(to_file_error(err).into());
}
Ok(())
})
.await
.map_err(DiskError::from)?
}
#[tracing::instrument(level = "trace", skip_all)]
async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
self.get_bucket_path(volume)?;
let current_data_path = self.get_object_path(volume, path)?;
let current_meta_path = self.get_object_path(volume, &format!("{path}.meta"))?;
let transaction_path = self.get_object_path(volume, &crate::disk::part_transaction_path(path))?;
for candidate in [&current_data_path, &current_meta_path, &transaction_path] {
check_path_length(candidate.to_string_lossy().as_ref())?;
}
let durability = effective_durability(volume);
tokio::task::spawn_blocking(move || {
match std::fs::symlink_metadata(&transaction_path) {
Ok(metadata) if metadata.is_dir() => {}
Ok(_) => return Err(DiskError::FileCorrupt),
Err(err) if err.kind() == ErrorKind::NotFound => return Ok(()),
Err(err) => return Err(to_file_error(err).into()),
}
if action == PartTransactionAction::Rollback {
std::fs::write(transaction_path.join(PART_TRANSACTION_ROLLBACK), []).map_err(to_file_error)?;
if durability.syncs_commit_metadata() {
os::fsync_dir_std(&transaction_path).map_err(to_file_error)?;
}
restore_part_transaction_file(
&current_data_path,
&transaction_path.join(PART_TRANSACTION_OLD_DATA),
&transaction_path.join(PART_TRANSACTION_OLD_DATA_ABSENT),
&transaction_path.join("restore.data"),
)
.map_err(to_file_error)?;
restore_part_transaction_file(
&current_meta_path,
&transaction_path.join(PART_TRANSACTION_OLD_META),
&transaction_path.join(PART_TRANSACTION_OLD_META_ABSENT),
&transaction_path.join("restore.meta"),
)
.map_err(to_file_error)?;
if durability.syncs_commit_metadata()
&& let Some(parent) = current_data_path.parent()
{
os::fsync_dir_std(parent).map_err(to_file_error)?;
}
}
let Some(parent) = transaction_path.parent() else {
return Err(DiskError::InvalidPath);
};
let cleanup_path = parent.join(format!(".part-txn-settled-{}", Uuid::new_v4()));
std::fs::rename(&transaction_path, &cleanup_path).map_err(to_file_error)?;
if durability.syncs_commit_metadata() {
os::fsync_dir_std(parent).map_err(to_file_error)?;
}
remove_dir_all_if_exists(&cleanup_path).map_err(to_file_error)?;
Ok(())
})
.await
.map_err(DiskError::from)??;
self.io_backend.invalidate_cached_fd(volume, path).await;
Ok(())
}
#[tracing::instrument(level = "trace", skip_all)]
async fn rename_part(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str, meta: Bytes) -> Result<()> {
let src_volume_dir = self.get_bucket_path(src_volume)?;
@@ -6528,6 +6718,42 @@ impl DiskAPI for LocalDisk {
}
}
let transaction_publish_meta = if src_is_dir {
None
} else {
let transaction_path = self.get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?;
let transaction_meta_path = transaction_path.join(PART_TRANSACTION_NEW_META);
match fs::read(&transaction_meta_path).await {
Ok(expected_meta) => {
if expected_meta.as_slice() != meta.as_ref() {
return Err(DiskError::FileCorrupt);
}
let publish_meta_path = transaction_path.join(PART_TRANSACTION_PUBLISH_META);
let source_meta_path = transaction_meta_path.clone();
let publish_path = publish_meta_path.clone();
tokio::task::spawn_blocking(move || {
remove_file_if_exists(&publish_path)?;
std::fs::hard_link(source_meta_path, &publish_path)
})
.await
.map_err(DiskError::from)?
.map_err(to_file_error)?;
Some(publish_meta_path)
}
// Old peers know only RenamePart. The new coordinator never
// reaches rename unless prepare succeeded, so an absent
// transaction directory identifies the rolling-upgrade legacy
// path. A present directory without new.meta is corruption.
Err(err) if err.kind() == ErrorKind::NotFound => match fs::metadata(&transaction_path).await {
Ok(_) => return Err(DiskError::FileCorrupt),
Err(meta_err) if meta_err.kind() == ErrorKind::NotFound => None,
Err(meta_err) => return Err(to_file_error(meta_err).into()),
},
Err(err) => return Err(to_file_error(err).into()),
}
};
// UploadPart is acknowledged once this rename lands, so the part data and
// its directory entry must be durable before we return. Relaxed keeps the
// part payload fdatasync but leaves the directory entry to the page cache.
@@ -6561,7 +6787,17 @@ impl DiskAPI for LocalDisk {
return Err(DiskError::FileAccessDenied);
}
self.write_all(dst_volume, format!("{dst_path}.meta").as_str(), meta).await?;
if let Some(transaction_publish_meta) = transaction_publish_meta {
let dst_meta_path = self.get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
rename_all(&transaction_publish_meta, &dst_meta_path, &dst_volume_dir).await?;
if durability.syncs_commit_metadata()
&& let Some(parent) = dst_meta_path.parent()
{
os::fsync_dir(parent).await.map_err(to_file_error)?;
}
} else {
self.write_all(dst_volume, format!("{dst_path}.meta").as_str(), meta).await?;
}
if let Some(parent) = src_file_path.parent() {
self.delete_file(&src_volume_dir, &parent.to_path_buf(), false, false).await?;
@@ -7606,6 +7842,118 @@ impl DiskAPI for LocalDisk {
Ok(())
}
async fn acquire_snapshot_lease(&self, volume: &str, path: &str) -> Result<SnapshotLeaseToken> {
let file_path = self.get_object_path(volume, path)?;
let key = SnapshotLeaseKey {
volume: volume.to_string(),
path: path.to_string(),
};
let token = {
let mut registry = self.snapshot_leases.lock().await;
if registry.entries.get(&key).is_some_and(|entry| entry.deleting) {
return Err(DiskError::FileNotFound);
}
let token = SnapshotLeaseToken::new();
registry.entries.entry(key).or_default().tokens.insert(token);
token
};
match fs::metadata(file_path).await {
Ok(metadata) if metadata.is_dir() => Ok(token),
Ok(_) => {
self.release_snapshot_lease(volume, path, token).await?;
Err(DiskError::FileNotFound)
}
Err(err) => {
self.release_snapshot_lease(volume, path, token).await?;
Err(to_file_error(err).into())
}
}
}
async fn release_snapshot_lease(&self, volume: &str, path: &str, token: SnapshotLeaseToken) -> Result<()> {
let key = SnapshotLeaseKey {
volume: volume.to_string(),
path: path.to_string(),
};
let opts = {
let mut registry = self.snapshot_leases.lock().await;
let Some(entry) = registry.entries.get_mut(&key) else {
return Ok(());
};
entry.tokens.remove(&token);
if !entry.tokens.is_empty() || entry.deleting {
return Ok(());
}
let Some(opts) = entry.pending_delete.clone() else {
registry.entries.remove(&key);
return Ok(());
};
entry.deleting = true;
opts
};
let result = self.delete_unleased(volume, path, &opts).await;
let mut registry = self.snapshot_leases.lock().await;
match result {
Ok(()) => {
registry.entries.remove(&key);
Ok(())
}
Err(err) => {
if let Some(entry) = registry.entries.get_mut(&key) {
entry.deleting = false;
}
Err(err)
}
}
}
async fn delete_data_dir(&self, volume: &str, path: &str, opts: DeleteOptions) -> Result<DataDirDeleteStatus> {
let key = SnapshotLeaseKey {
volume: volume.to_string(),
path: path.to_string(),
};
{
let mut registry = self.snapshot_leases.lock().await;
if let Some(entry) = registry.entries.get_mut(&key) {
if !entry.tokens.is_empty() {
entry.pending_delete.get_or_insert_with(|| opts.clone());
return Ok(DataDirDeleteStatus::Deferred);
}
if entry.deleting {
entry.pending_delete.get_or_insert_with(|| opts.clone());
return Ok(DataDirDeleteStatus::Deferred);
}
entry.deleting = true;
entry.pending_delete.get_or_insert_with(|| opts.clone());
} else {
registry.entries.insert(
key.clone(),
SnapshotLeaseEntry {
pending_delete: Some(opts.clone()),
deleting: true,
..Default::default()
},
);
}
}
let result = self.delete_unleased(volume, path, &opts).await;
let mut registry = self.snapshot_leases.lock().await;
match result {
Ok(()) => {
registry.entries.remove(&key);
Ok(DataDirDeleteStatus::Deleted)
}
Err(err) => {
if let Some(entry) = registry.entries.get_mut(&key) {
entry.deleting = false;
}
Err(err)
}
}
}
#[tracing::instrument(level = "trace", skip_all)]
async fn update_metadata(&self, volume: &str, path: &str, fi: FileInfo, opts: &UpdateMetadataOpts) -> Result<()> {
if !fi.metadata.is_empty() {
@@ -8282,6 +8630,12 @@ mod test {
file_info.data_dir = data_dir;
file_info.data = data;
file_info.size = size;
file_info.parts = vec![ObjectPartInfo {
number: 1,
size: usize::try_from(size).expect("test object size should fit usize"),
actual_size: size,
..Default::default()
}];
file_info.mod_time = Some(OffsetDateTime::now_utc());
file_info
}
@@ -9370,9 +9724,15 @@ mod test {
.await
.expect("source part should be written");
disk.prepare_part_transaction(tmp_volume, "upload/part.1", bucket, "object/part.1", meta.clone())
.await
.expect("part transaction should be prepared");
disk.rename_part(tmp_volume, "upload/part.1", bucket, "object/part.1", meta.clone())
.await
.expect("rename_part should commit part");
disk.settle_part_transaction(bucket, "object/part.1", PartTransactionAction::Commit)
.await
.expect("part transaction should be committed");
assert_eq!(
disk.read_all(bucket, "object/part.1")
@@ -9390,6 +9750,71 @@ mod test {
matches!(disk.read_all(tmp_volume, "upload/part.1").await, Err(DiskError::FileNotFound)),
"source part must be removed after a successful commit"
);
let legacy_payload = Bytes::from_static(b"legacy peer payload");
let legacy_meta = Bytes::from_static(b"legacy peer metadata");
disk.write_all(tmp_volume, "legacy/part.1", legacy_payload.clone())
.await
.expect("legacy source part should be written");
disk.rename_part(tmp_volume, "legacy/part.1", bucket, "object/part.1", legacy_meta.clone())
.await
.expect("pre-transaction peer RenamePart should remain supported");
assert_eq!(
disk.read_all(bucket, "object/part.1")
.await
.expect("legacy destination part should be readable"),
legacy_payload
);
assert_eq!(
disk.read_all(bucket, "object/part.1.meta")
.await
.expect("legacy destination metadata should be readable"),
legacy_meta
);
}
#[tokio::test]
async fn test_part_transaction_rolls_back_data_published_before_metadata() {
use tempfile::tempdir;
let dir = tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
ensure_test_volume(&disk, "tmp").await;
ensure_test_volume(&disk, "bucket").await;
disk.write_all("tmp", "upload/part.1", Bytes::from_static(b"new data"))
.await
.expect("new part should be staged");
disk.write_all("bucket", "object/part.1", Bytes::from_static(b"old data"))
.await
.expect("old part data should be staged");
disk.write_all("bucket", "object/part.1.meta", Bytes::from_static(b"old metadata"))
.await
.expect("old part metadata should be staged");
disk.prepare_part_transaction("tmp", "upload/part.1", "bucket", "object/part.1", Bytes::from_static(b"new metadata"))
.await
.expect("part transaction should be prepared");
disk.rename_file("tmp", "upload/part.1", "bucket", "object/part.1")
.await
.expect("data publication should succeed");
disk.settle_part_transaction("bucket", "object/part.1", PartTransactionAction::Rollback)
.await
.expect("part transaction should roll back");
assert_eq!(
disk.read_all("bucket", "object/part.1")
.await
.expect("old part data should be restored"),
Bytes::from_static(b"old data")
);
assert_eq!(
disk.read_all("bucket", "object/part.1.meta")
.await
.expect("old part metadata should be restored"),
Bytes::from_static(b"old metadata")
);
}
struct BlockingScanWriter {
@@ -14505,6 +14930,162 @@ mod test {
assert!(construction_source.is::<ErasureConstructionError>());
}
#[tokio::test]
async fn snapshot_leases_defer_data_dir_cleanup_until_last_release() {
use tempfile::tempdir;
let root_dir = tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let volume = "snapshot-lease-volume";
let data_dir = path_join_buf(&["object", &Uuid::new_v4().to_string()]);
let first_part = path_join_buf(&[&data_dir, "part.1"]);
let later_part = path_join_buf(&[&data_dir, "part.2"]);
ensure_test_volume(&disk, volume).await;
disk.write_all(volume, &first_part, Bytes::from_static(b"first"))
.await
.expect("first shard should be written");
disk.write_all(volume, &later_part, Bytes::from_static(b"later"))
.await
.expect("later shard should be written");
let first = disk
.acquire_snapshot_lease(volume, &data_dir)
.await
.expect("first lease should be acquired");
let second = disk
.acquire_snapshot_lease(volume, &data_dir)
.await
.expect("second lease should be acquired");
let status = disk
.delete_data_dir(
volume,
&data_dir,
DeleteOptions {
recursive: true,
..Default::default()
},
)
.await
.expect("cleanup should be deferred");
assert_eq!(status, DataDirDeleteStatus::Deferred);
assert_eq!(
disk.read_all(volume, &later_part)
.await
.expect("a later multipart shard must remain openable while leased"),
Bytes::from_static(b"later")
);
disk.release_snapshot_lease(volume, &data_dir, first)
.await
.expect("first lease release should succeed");
assert!(
disk.read_all(volume, &first_part).await.is_ok(),
"one remaining lease must keep the data directory"
);
disk.release_snapshot_lease(volume, &data_dir, second)
.await
.expect("last lease release should run deferred cleanup");
disk.release_snapshot_lease(volume, &data_dir, second)
.await
.expect("releasing an already released token should be idempotent");
assert!(matches!(disk.read_all(volume, &first_part).await, Err(DiskError::FileNotFound)));
}
#[tokio::test]
async fn data_dir_cleanup_without_a_lease_keeps_existing_behavior() {
use tempfile::tempdir;
let root_dir = tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let volume = "snapshot-no-lease-volume";
let data_dir = path_join_buf(&["object", &Uuid::new_v4().to_string()]);
let part = path_join_buf(&[&data_dir, "part.1"]);
ensure_test_volume(&disk, volume).await;
disk.write_all(volume, &part, Bytes::from_static(b"payload"))
.await
.expect("test shard should be written");
let status = disk
.delete_data_dir(
volume,
&data_dir,
DeleteOptions {
recursive: true,
..Default::default()
},
)
.await
.expect("unleased cleanup should retain the existing delete behavior");
assert_eq!(status, DataDirDeleteStatus::Deleted);
assert!(matches!(disk.read_all(volume, &part).await, Err(DiskError::FileNotFound)));
}
#[tokio::test]
async fn snapshot_lease_acquire_and_cleanup_are_atomic() {
use tempfile::tempdir;
let root_dir = tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
let volume = "snapshot-race-volume";
ensure_test_volume(&disk, volume).await;
for iteration in 0..32 {
let data_dir = path_join_buf(&["object", &format!("{iteration:032x}")]);
let part = path_join_buf(&[&data_dir, "part.1"]);
disk.write_all(volume, &part, Bytes::from_static(b"payload"))
.await
.expect("test shard should be written");
let barrier = Arc::new(tokio::sync::Barrier::new(3));
let acquire_disk = Arc::clone(&disk);
let acquire_barrier = Arc::clone(&barrier);
let acquire_path = data_dir.clone();
let acquire = tokio::spawn(async move {
acquire_barrier.wait().await;
acquire_disk.acquire_snapshot_lease(volume, &acquire_path).await
});
let delete_disk = Arc::clone(&disk);
let delete_barrier = Arc::clone(&barrier);
let delete_path = data_dir.clone();
let delete = tokio::spawn(async move {
delete_barrier.wait().await;
delete_disk
.delete_data_dir(
volume,
&delete_path,
DeleteOptions {
recursive: true,
..Default::default()
},
)
.await
});
barrier.wait().await;
let acquired = acquire.await.expect("acquire task should join");
let deleted = delete
.await
.expect("delete task should join")
.expect("delete should either run or defer");
match acquired {
Ok(token) => {
assert_eq!(deleted, DataDirDeleteStatus::Deferred);
assert!(disk.read_all(volume, &part).await.is_ok());
disk.release_snapshot_lease(volume, &data_dir, token)
.await
.expect("release should finish deferred cleanup");
}
Err(DiskError::FileNotFound) => {
assert_eq!(deleted, DataDirDeleteStatus::Deleted);
}
Err(err) => panic!("unexpected lease acquisition error: {err}"),
}
assert!(matches!(disk.read_all(volume, &part).await, Err(DiskError::FileNotFound)));
}
}
#[tokio::test]
async fn local_disk_check_parts_rejects_zero_data_geometry_before_shard_math() {
use tempfile::tempdir;
+110
View File
@@ -38,6 +38,16 @@ pub const FORMAT_CONFIG_FILE: &str = "format.json";
pub const HEALING_MARKER_PATH: &str = "healing.bin";
pub const STORAGE_FORMAT_FILE: &str = "xl.meta";
pub const STORAGE_FORMAT_FILE_BACKUP: &str = "xl.meta.bkp";
pub const PART_TRANSACTION_NEW_META: &str = "new.meta";
pub const PART_TRANSACTION_OLD_META: &str = "old.meta";
pub const PART_TRANSACTION_ROLLBACK: &str = "rollback";
pub fn part_transaction_path(part_path: &str) -> String {
match part_path.rsplit_once('/') {
Some((parent, name)) => format!("{parent}/.{name}.rustfs-txn"),
None => format!(".{part_path}.rustfs-txn"),
}
}
use crate::cluster::rpc::RemoteDisk;
use crate::cluster::rpc::build_internode_data_transport_from_env;
@@ -62,6 +72,33 @@ pub type DiskStore = Arc<Disk>;
pub type FileReader = Box<dyn AsyncRead + Send + Sync + Unpin>;
pub type FileWriter = Box<dyn AsyncWrite + Send + Sync + Unpin>;
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, Serialize, Deserialize)]
pub struct SnapshotLeaseToken(Uuid);
impl SnapshotLeaseToken {
pub fn new() -> Self {
Self(Uuid::new_v4())
}
}
impl Default for SnapshotLeaseToken {
fn default() -> Self {
Self::new()
}
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum DataDirDeleteStatus {
Deleted,
Deferred,
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum PartTransactionAction {
Commit,
Rollback,
}
#[derive(Clone, Copy, Debug)]
pub struct MmapCopyStageMetrics {
pub(crate) path: &'static str,
@@ -233,6 +270,27 @@ impl DiskAPI for Disk {
}
}
async fn acquire_snapshot_lease(&self, volume: &str, path: &str) -> Result<SnapshotLeaseToken> {
match self {
Disk::Local(local_disk) => local_disk.acquire_snapshot_lease(volume, path).await,
Disk::Remote(remote_disk) => remote_disk.acquire_snapshot_lease(volume, path).await,
}
}
async fn release_snapshot_lease(&self, volume: &str, path: &str, token: SnapshotLeaseToken) -> Result<()> {
match self {
Disk::Local(local_disk) => local_disk.release_snapshot_lease(volume, path, token).await,
Disk::Remote(remote_disk) => remote_disk.release_snapshot_lease(volume, path, token).await,
}
}
async fn delete_data_dir(&self, volume: &str, path: &str, opts: DeleteOptions) -> Result<DataDirDeleteStatus> {
match self {
Disk::Local(local_disk) => local_disk.delete_data_dir(volume, path, opts).await,
Disk::Remote(remote_disk) => remote_disk.delete_data_dir(volume, path, opts).await,
}
}
#[tracing::instrument(level = "trace", skip_all)]
async fn write_metadata(&self, _org_volume: &str, volume: &str, path: &str, fi: FileInfo) -> Result<()> {
match self {
@@ -381,6 +439,35 @@ impl DiskAPI for Disk {
}
}
async fn prepare_part_transaction(
&self,
src_volume: &str,
src_path: &str,
dst_volume: &str,
dst_path: &str,
meta: Bytes,
) -> Result<()> {
match self {
Disk::Local(local_disk) => {
local_disk
.prepare_part_transaction(src_volume, src_path, dst_volume, dst_path, meta)
.await
}
Disk::Remote(remote_disk) => {
remote_disk
.prepare_part_transaction(src_volume, src_path, dst_volume, dst_path, meta)
.await
}
}
}
async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
match self {
Disk::Local(local_disk) => local_disk.settle_part_transaction(volume, path, action).await,
Disk::Remote(remote_disk) => remote_disk.settle_part_transaction(volume, path, action).await,
}
}
#[tracing::instrument(level = "trace", skip_all)]
async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()> {
match self {
@@ -601,6 +688,16 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
) -> Result<()>;
async fn delete_versions(&self, volume: &str, versions: Vec<FileInfoVersions>, opts: DeleteOptions) -> Vec<Option<Error>>;
async fn delete_paths(&self, volume: &str, paths: &[String]) -> Result<()>;
async fn acquire_snapshot_lease(&self, _volume: &str, _path: &str) -> Result<SnapshotLeaseToken> {
Err(Error::other("snapshot leases are not supported by this disk"))
}
async fn release_snapshot_lease(&self, _volume: &str, _path: &str, _token: SnapshotLeaseToken) -> Result<()> {
Err(Error::other("snapshot leases are not supported by this disk"))
}
async fn delete_data_dir(&self, volume: &str, path: &str, opts: DeleteOptions) -> Result<DataDirDeleteStatus> {
self.delete(volume, path, opts).await?;
Ok(DataDirDeleteStatus::Deleted)
}
async fn write_metadata(&self, org_volume: &str, volume: &str, path: &str, fi: FileInfo) -> Result<()>;
async fn update_metadata(&self, volume: &str, path: &str, fi: FileInfo, opts: &UpdateMetadataOpts) -> Result<()>;
async fn read_version(
@@ -659,6 +756,19 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
// ReadFileStream
async fn rename_file(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str) -> Result<()>;
async fn rename_part(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str, meta: Bytes) -> Result<()>;
async fn prepare_part_transaction(
&self,
_src_volume: &str,
_src_path: &str,
_dst_volume: &str,
_dst_path: &str,
_meta: Bytes,
) -> Result<()> {
Err(DiskError::MethodNotAllowed)
}
async fn settle_part_transaction(&self, _volume: &str, _path: &str, _action: PartTransactionAction) -> Result<()> {
Err(DiskError::MethodNotAllowed)
}
async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()>;
// VerifyFile
async fn verify_file(&self, volume: &str, path: &str, fi: &FileInfo) -> Result<CheckPartsResp>;
+47 -2
View File
@@ -443,7 +443,9 @@ pub fn bitrot_shard_file_size(size: usize, shard_size: usize, algo: HashAlgorith
size.div_ceil(shard_size) * algo.size() + size
}
/// Verify an interleaved per-block bitrot shard file.
/// Verify an interleaved per-block bitrot shard file and consume the reader
/// through EOF. Bytes beyond the encoded length are corruption, even when every
/// expected block has a valid hash.
///
/// The read loop below assumes every block on disk is `[hash][data]` (streaming
/// bitrot). It is therefore only valid for the streaming Highway variants, whose
@@ -487,6 +489,11 @@ pub async fn bitrot_verify<R: AsyncRead + Unpin + Send>(
left -= read;
}
let mut trailing = [0u8; 1];
if r.read(&mut trailing).await? != 0 {
return Err(std::io::Error::other("bitrot shard file has trailing data"));
}
Ok(())
}
@@ -860,12 +867,19 @@ mod tests {
.await
.expect("valid bitrot shard file should verify");
let mut truncated = written.clone();
truncated.pop();
let err = bitrot_verify(Cursor::new(truncated), written.len(), data.len(), algo.clone(), shard_size)
.await
.expect_err("one-byte-short shard file must be rejected while reading");
assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof);
let err = bitrot_verify(Cursor::new(written.clone()), written.len() - 1, data.len(), algo.clone(), shard_size)
.await
.expect_err("wrong file size must be rejected before reading data");
assert!(err.to_string().contains("size mismatch"));
let mut corrupt = written;
let mut corrupt = written.clone();
let last = corrupt.len() - 1;
corrupt[last] ^= 0x80;
let err = bitrot_verify(
@@ -878,6 +892,21 @@ mod tests {
.await
.expect_err("hash mismatch must reject corrupted data");
assert!(err.to_string().contains("hash mismatch"));
for trailing in [vec![0xa5], vec![0xa5; 17]] {
let mut oversized = written.clone();
oversized.extend_from_slice(&trailing);
let err = bitrot_verify(
Cursor::new(oversized),
written.len(),
data.len(),
HashAlgorithm::HighwayHash256S,
shard_size,
)
.await
.expect_err("trailing bytes after a valid encoded shard must be rejected");
assert!(err.to_string().contains("trailing data"));
}
}
#[tokio::test]
@@ -909,6 +938,22 @@ mod tests {
assert!(err.to_string().contains("hash mismatch"));
}
#[tokio::test]
async fn bitrot_verify_accepts_exact_legacy_streaming_layout() {
let data = b"legacy streaming bitrot";
let shard_size = 8;
let algo = HashAlgorithm::HighwayHash256SLegacy;
let mut writer = BitrotWriter::new(Cursor::new(Vec::new()), shard_size, algo.clone());
for chunk in data.chunks(shard_size) {
writer.write(chunk).await.expect("legacy streaming shard should encode");
}
let written = writer.into_inner().into_inner();
bitrot_verify(Cursor::new(written.clone()), written.len(), data.len(), algo, shard_size)
.await
.expect("exact legacy streaming shard should remain valid");
}
#[tokio::test]
async fn write_all_vectored_retries_partial_hash_and_data_writes_and_rejects_zero_write() {
let mut writer = LimitedVectoredWriter {
+5 -1
View File
@@ -818,7 +818,11 @@ impl From<s3s::xml::DeError> for Error {
impl From<tonic::Status> for Error {
fn from(e: tonic::Status) -> Self {
Error::other(e.to_string())
// Keep the typed status as the io::Error payload instead of a
// flattened string so RPC failure classifiers can read the gRPC code
// via downcast (see `PeerRestClient::is_network_like_error`).
// `RpcStatusError` renders exactly as `e.to_string()` did.
Error::other(crate::disk::error::RpcStatusError::from(e))
}
}
+57 -7
View File
@@ -28,7 +28,6 @@ use md5::{Digest, Md5};
use rustfs_kms::{KmsUnavailableError, is_data_key_envelope, types::ObjectEncryptionContext};
use rustfs_utils::http::{SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER};
use rustfs_utils::path::path_join_buf;
#[cfg(feature = "rio-v2")]
use serde::Deserialize;
#[cfg(feature = "rio-v2")]
use sha2::Sha256;
@@ -44,6 +43,7 @@ const INTERNAL_ENCRYPTION_IV_HEADER: &str = "x-rustfs-encryption-iv";
const INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER: &str = "x-rustfs-encryption-original-size";
const SSEC_ORIGINAL_SIZE_HEADER: &str = "x-amz-server-side-encryption-customer-original-size";
const DEFAULT_SSE_ALGORITHM: &str = "AES256";
const LOCAL_SSE_DEK_FORMAT_VERSION: u8 = 1;
#[cfg(feature = "rio-v2")]
const DARE_PAYLOAD_SIZE: i64 = 64 * 1024;
#[cfg(feature = "rio-v2")]
@@ -1716,16 +1716,39 @@ fn decrypt_local_sse_dek(encrypted_dek: &[u8], _kms_key_id: &str, object_context
fn decrypt_rustfs_local_sse_dek(encrypted_dek: &[u8]) -> Result<[u8; 32]> {
let encrypted_dek = std::str::from_utf8(encrypted_dek).map_err(|_| Error::other("managed DEK is not valid UTF-8"))?;
let parts: Vec<&str> = encrypted_dek.split(':').collect();
if parts.len() != 2 {
return Err(Error::other("invalid managed DEK format"));
#[derive(Deserialize)]
#[serde(deny_unknown_fields)]
struct LocalSseDekEnvelope<'a> {
version: u8,
nonce: &'a str,
ciphertext: &'a str,
}
let (nonce, ciphertext) = match serde_json::from_str::<LocalSseDekEnvelope<'_>>(encrypted_dek) {
Ok(envelope) => {
if envelope.version != LOCAL_SSE_DEK_FORMAT_VERSION {
return Err(Error::other(format!("unsupported managed DEK format version: {}", envelope.version)));
}
(envelope.nonce, envelope.ciphertext)
}
Err(_) => {
// DEPRECATED: read-only compatibility for persisted colon-delimited DEKs.
// RUSTFS_COMPAT_TODO(sse-local-dek-json-v1): Remove after all supported upgrades have rewritten legacy DEKs.
let Some((nonce, ciphertext)) = encrypted_dek.split_once(':') else {
return Err(Error::other("invalid managed DEK format"));
};
if ciphertext.contains(':') {
return Err(Error::other("invalid managed DEK format"));
}
(nonce, ciphertext)
}
};
let nonce_vec = BASE64_STANDARD
.decode(parts[0])
.decode(nonce)
.map_err(|_| Error::other("invalid managed DEK nonce"))?;
let ciphertext = BASE64_STANDARD
.decode(parts[1])
.decode(ciphertext)
.map_err(|_| Error::other("invalid managed DEK ciphertext"))?;
let nonce_array: [u8; 12] = nonce_vec
@@ -2324,9 +2347,36 @@ mod tests {
let cipher = Aes256Gcm::new(&key);
let nonce = Nonce::from([0u8; 12]);
let ciphertext = cipher.encrypt(&nonce, dek.as_slice()).expect("encrypt managed dek");
serde_json::json!({
"version": LOCAL_SSE_DEK_FORMAT_VERSION,
"nonce": BASE64_STANDARD.encode(nonce),
"ciphertext": BASE64_STANDARD.encode(ciphertext),
})
.to_string()
}
fn encrypt_legacy_managed_dek_for_test(dek: [u8; 32], master_key: [u8; 32]) -> String {
let key = Key::<Aes256Gcm>::from(master_key);
let cipher = Aes256Gcm::new(&key);
let nonce = Nonce::from([0u8; 12]);
let ciphertext = cipher.encrypt(&nonce, dek.as_slice()).expect("encrypt legacy managed dek");
format!("{}:{}", BASE64_STANDARD.encode(nonce), BASE64_STANDARD.encode(ciphertext))
}
#[test]
fn decrypt_rustfs_local_sse_dek_rejects_unknown_json_version() {
let envelope = serde_json::json!({
"version": LOCAL_SSE_DEK_FORMAT_VERSION + 1,
"nonce": BASE64_STANDARD.encode([0u8; 12]),
"ciphertext": BASE64_STANDARD.encode([0u8; 48]),
})
.to_string();
let error =
decrypt_rustfs_local_sse_dek(envelope.as_bytes()).expect_err("unknown local SSE DEK versions must fail closed");
assert!(error.to_string().contains("unsupported managed DEK format version"));
}
#[cfg(feature = "rio-v2")]
fn seal_managed_s3_object_key_for_test(
bucket: &str,
@@ -2433,7 +2483,7 @@ mod tests {
async_with_vars([("__RUSTFS_SSE_SIMPLE_CMK", Some(BASE64_STANDARD.encode([7u8; 32])))], async {
let data_key = [0x24; 32];
let base_nonce = [0x14; 12];
let encrypted_dek = encrypt_managed_dek_for_test(data_key, [7u8; 32]);
let encrypted_dek = encrypt_legacy_managed_dek_for_test(data_key, [7u8; 32]);
let metadata = HashMap::from([
(
INTERNAL_ENCRYPTION_KEY_HEADER.to_string(),
+8 -5
View File
@@ -180,6 +180,7 @@ pub struct ObjectInfo {
pub data_dir: Option<Uuid>,
pub delete_marker: bool,
pub transitioned_object: TransitionedObject,
pub transition_version_state: rustfs_filemeta::TransitionVersionState,
pub restore_ongoing: bool,
pub restore_expires: Option<OffsetDateTime>,
pub user_tags: Arc<String>,
@@ -220,6 +221,7 @@ impl Clone for ObjectInfo {
data_dir: self.data_dir,
delete_marker: self.delete_marker,
transitioned_object: self.transitioned_object.clone(),
transition_version_state: self.transition_version_state,
restore_ongoing: self.restore_ongoing,
restore_expires: self.restore_expires,
user_tags: self.user_tags.clone(),
@@ -464,11 +466,11 @@ impl ObjectInfo {
let transitioned_object = TransitionedObject {
name: fi.transitioned_objname.clone(),
version_id: if let Some(transition_version_id) = fi.transition_version_id {
transition_version_id.to_string()
} else {
"".to_string()
},
version_id: fi
.transition_version
.clone()
.or_else(|| fi.transition_version_id.map(|version_id| version_id.to_string()))
.unwrap_or_default(),
status: fi.transition_status.clone(),
free_version: fi.tier_free_version(),
tier: fi.transition_tier.clone(),
@@ -537,6 +539,7 @@ impl ObjectInfo {
inlined,
user_defined: Arc::new(metadata),
transitioned_object,
transition_version_state: fi.transition_version_state,
checksum: fi.checksum.clone(),
storage_class,
restore_ongoing,
@@ -931,7 +931,10 @@ pub async fn read_transition_meta(disk_path: &Path, bucket: &str, object: &str)
status: fi.transition_status.clone(),
tier: fi.transition_tier.clone(),
remote_object: fi.transitioned_objname.clone(),
remote_version_id: fi.transition_version_id.map(|id| id.to_string()),
remote_version_id: fi
.transition_version
.clone()
.or_else(|| fi.transition_version_id.map(|id| id.to_string())),
free_version_count,
})
}
+2 -1
View File
@@ -9274,7 +9274,8 @@ mod tests {
version_id: "v1".to_string(),
tier_name: "COLD-A".to_string(),
backend_identity: Some(current_identity),
version_id_exact: false,
version_id_exact: true,
version_state: rustfs_filemeta::TransitionVersionState::Exact,
};
journal_store
.insert_config_object(
+300 -29
View File
@@ -46,7 +46,10 @@ use crate::diagnostics::get::{
GetObjectFailureReason, classify_disk_error, get_stage_timer_if_enabled, record_get_object_pipeline_failure,
record_get_object_pipeline_failure_for_path, record_get_stage_duration_if_enabled,
};
use crate::disk::OldCurrentSize;
use crate::disk::{
DataDirDeleteStatus, OldCurrentSize, PART_TRANSACTION_NEW_META, PART_TRANSACTION_OLD_META, PART_TRANSACTION_ROLLBACK,
PartTransactionAction, part_transaction_path,
};
use crate::erasure::coding::BitrotReader;
use crate::io_support::bitrot::ShardReader;
use crate::io_support::bitrot::{
@@ -544,6 +547,8 @@ pub(in crate::set_disk) fn metadata_early_stop_candidate_matches(left: &FileInfo
&& left.transitioned_objname == right.transitioned_objname
&& left.transition_tier == right.transition_tier
&& left.transition_version_id == right.transition_version_id
&& left.transition_version == right.transition_version
&& left.transition_version_state == right.transition_version_state
&& left.expire_restored == right.expire_restored
&& left.size == right.size
&& left.mod_time == right.mod_time
@@ -2982,29 +2987,48 @@ impl SetDisks {
Self::rename_fanout_barrier(&object_for_fault, idx, rename_fanout_barrier_phase::CLEANUP).await;
if let Some(err) = Self::cleanup_injected_error(&object_for_fault, idx) {
return Some(err);
return (false, Some(err));
}
if let Some(disk) = disk {
disk.delete(
&bucket,
&file_path,
DeleteOptions {
recursive: true,
..Default::default()
},
)
.await
.err()
match disk
.delete_data_dir(
&bucket,
&file_path,
DeleteOptions {
recursive: true,
..Default::default()
},
)
.await
{
Ok(DataDirDeleteStatus::Deleted) => (false, None),
Ok(DataDirDeleteStatus::Deferred) => (true, None),
Err(err) => (false, Some(err)),
}
} else {
// `None` slot: ignored placeholder. It is not `attempted`, so
// classification excludes it from residue regardless.
Some(DiskError::DiskNotFound)
(false, Some(DiskError::DiskNotFound))
}
})
});
let errs: Vec<Option<DiskError>> = join_all(futures).await.into_iter().map(map_cleanup_join_result).collect();
let mut deferred = 0usize;
let errs: Vec<Option<DiskError>> = join_all(futures)
.await
.into_iter()
.map(|result| match result {
Ok((was_deferred, err)) => {
deferred += usize::from(was_deferred);
err
}
Err(join_err) => Some(DiskError::other(format!("old data dir cleanup task failed: {join_err}"))),
})
.collect();
classify_old_data_dir_cleanup(&errs, &attempted, write_quorum)
let mut cleanup = classify_old_data_dir_cleanup(&errs, &attempted, write_quorum);
cleanup.deferred = deferred;
cleanup.reclaimed = cleanup.reclaimed.saturating_sub(deferred);
cleanup
}
/// Test-only fault-injection seam for the old-data-dir cleanup path
@@ -3095,6 +3119,20 @@ impl SetDisks {
rustfs_io_metrics::record_old_data_dir_cleanup(c.attempted, c.reclaimed, c.unreclaimed_disks.len(), c.below_quorum);
if c.deferred > 0 {
debug!(
event = EVENT_SET_DISK_WRITE,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
bucket = %bucket,
object = %object,
old_data_dir = %old_dir,
deferred = c.deferred,
state = "old_data_cleanup_deferred",
"Old data directory cleanup deferred for active snapshot leases"
);
}
if actions.warn {
warn!(
component = LOG_COMPONENT_ECSTORE,
@@ -3174,6 +3212,155 @@ impl SetDisks {
}
}
async fn recover_part_transaction(&self, dst_object: &str, write_quorum: usize) -> disk::error::Result<bool> {
let disks = self.get_disks_internal().await;
let transaction_path = part_transaction_path(dst_object);
let transaction_meta_path = format!("{transaction_path}/{PART_TRANSACTION_NEW_META}");
let rollback_path = format!("{transaction_path}/{PART_TRANSACTION_ROLLBACK}");
let current_meta_path = format!("{dst_object}.meta");
let reads = disks.iter().map(|disk| {
let disk = disk.clone();
let transaction_meta_path = transaction_meta_path.clone();
let rollback_path = rollback_path.clone();
let current_meta_path = current_meta_path.clone();
async move {
let Some(disk) = disk else {
return Ok((None, None, false));
};
let transaction_meta = match disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &transaction_meta_path).await {
Ok(meta) => Some(meta),
Err(DiskError::FileNotFound) => None,
Err(err) => return Err(err),
};
let rollback = match disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &rollback_path).await {
Ok(_) => true,
Err(DiskError::FileNotFound) => false,
Err(err) => return Err(err),
};
let current_meta = match disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &current_meta_path).await {
Ok(meta) => Some(meta),
Err(DiskError::FileNotFound | DiskError::DiskNotFound) => None,
Err(_) => None,
};
Ok((transaction_meta, current_meta, rollback))
}
});
let observations = join_all(reads).await.into_iter().collect::<disk::error::Result<Vec<_>>>()?;
if observations.iter().all(|(transaction, _, _)| transaction.is_none()) {
return Ok(false);
}
let mut current_counts: HashMap<Bytes, usize> = HashMap::new();
for (_, current, _) in &observations {
if let Some(current) = current {
*current_counts.entry(current.clone()).or_default() += 1;
}
}
let current_quorum = current_counts
.into_iter()
.find_map(|(meta, count)| (count >= write_quorum).then_some(meta));
let old_meta_path = format!("{transaction_path}/{PART_TRANSACTION_OLD_META}");
let old_meta_absent_path = format!("{transaction_path}/old.meta.absent");
let decisions = observations
.iter()
.enumerate()
.filter_map(|(index, (transaction_meta, _, rollback))| {
transaction_meta.as_ref().map(|meta| (index, meta.clone(), *rollback))
})
.map(|(index, transaction_meta, rollback)| {
let disk = disks[index].clone();
let old_meta_path = old_meta_path.clone();
let old_meta_absent_path = old_meta_absent_path.clone();
let current_quorum = current_quorum.clone();
async move {
let Some(disk) = disk else {
return Err(DiskError::DiskNotFound);
};
let action = if rollback {
PartTransactionAction::Rollback
} else if current_quorum.as_ref() == Some(&transaction_meta) {
PartTransactionAction::Commit
} else if let Some(current_quorum) = current_quorum {
match disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &old_meta_path).await {
Ok(old_meta) if old_meta == current_quorum => PartTransactionAction::Rollback,
Ok(_) => PartTransactionAction::Commit,
Err(DiskError::FileNotFound) => {
match disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &old_meta_absent_path).await {
Ok(_) => PartTransactionAction::Commit,
Err(_) => return Err(DiskError::FileCorrupt),
}
}
Err(err) => return Err(err),
}
} else {
PartTransactionAction::Rollback
};
disk.settle_part_transaction(RUSTFS_META_MULTIPART_BUCKET, dst_object, action)
.await?;
Ok(action == PartTransactionAction::Commit)
}
});
let results = join_all(decisions).await;
if let Some(err) = results.iter().find_map(|result| result.as_ref().err()) {
return Err(err.clone());
}
Ok(results.iter().any(|result| matches!(result, Ok(true))))
}
pub(in crate::set_disk) async fn recover_part_transactions(
&self,
part_path: &str,
read_quorum: usize,
write_quorum: usize,
) -> disk::error::Result<()> {
let disks = self.get_disks_internal().await;
let listings = join_all(disks.iter().map(|disk| {
let disk = disk.clone();
async move {
let Some(disk) = disk else {
return Err(DiskError::DiskNotFound);
};
disk.list_dir(RUSTFS_META_MULTIPART_BUCKET, RUSTFS_META_MULTIPART_BUCKET, part_path, -1)
.await
}
}))
.await;
let mut transaction_parts = HashSet::new();
let mut errs = Vec::with_capacity(listings.len());
for listing in listings {
match listing {
Ok(entries) => {
errs.push(None);
for entry in entries {
let name = entry.trim_end_matches('/');
let Some(part_number) = name
.strip_prefix(".part.")
.and_then(|name| name.strip_suffix(".rustfs-txn"))
.and_then(|number| number.parse::<usize>().ok())
else {
continue;
};
transaction_parts.insert(part_number);
}
}
Err(err) => errs.push(Some(err)),
}
}
if let Some(err) = reduce_read_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, read_quorum) {
return Err(err);
}
for part_number in transaction_parts {
self.recover_part_transaction(&format!("{part_path}part.{part_number}"), write_quorum)
.await?;
}
Ok(())
}
#[tracing::instrument(skip(disks, meta))]
#[allow(clippy::too_many_arguments)]
pub(in crate::set_disk) async fn rename_part(
@@ -3187,23 +3374,46 @@ impl SetDisks {
write_quorum: usize,
quorum_context: Option<MultipartWriteQuorumContext<'_>>,
) -> disk::error::Result<Vec<Option<DiskStore>>> {
self.recover_part_transaction(dst_object, write_quorum).await?;
let src_bucket = Arc::new(src_bucket.to_string());
let src_object = Arc::new(src_object.to_string());
let dst_bucket = Arc::new(dst_bucket.to_string());
let dst_object = Arc::new(dst_object.to_string());
// Do NOT pre-delete the destination part before renaming: the per-disk
// `rename_part` replaces `part.N` atomically (std::fs::rename) and rewrites
// `part.N.meta`, so the pre-delete is redundant — and destructive. It
// opened a window where an already-committed (ACKed) part was removed on
// every disk before the new rename landed, so a re-upload that then failed
// quorum destroyed the committed part outright (backlog#853 / #799 B4).
// The atomic rename overwrites in place; on quorum failure below we roll
// the destination back.
let prepare_tasks = disks.iter().map(|disk| {
let disk = disk.clone();
let src_bucket = src_bucket.clone();
let src_object = src_object.clone();
let dst_bucket = dst_bucket.clone();
let dst_object = dst_object.clone();
let meta = meta.clone();
async move {
let disk = disk?;
Some(
disk.prepare_part_transaction(&src_bucket, &src_object, &dst_bucket, &dst_object, meta)
.await,
)
}
});
let prepare_results = join_all(prepare_tasks).await;
let prepare_errs = prepare_results
.into_iter()
.map(|result| match result {
Some(Ok(())) => None,
Some(Err(err)) => Some(err),
None => Some(DiskError::DiskNotFound),
})
.collect::<Vec<_>>();
let prepared_disks = Self::eval_disks(disks, &prepare_errs);
if reduce_write_quorum_errs(&prepare_errs, OBJECT_OP_IGNORED_ERRS, write_quorum).is_some() {
self.recover_part_transaction(&dst_object, write_quorum).await?;
return Err(DiskError::ErasureWriteQuorum);
}
let mut errs = Vec::with_capacity(disks.len());
let futures = disks.iter().map(|disk| {
let futures = prepared_disks.iter().map(|disk| {
let disk = disk.clone();
let meta = meta.clone();
let src_bucket = src_bucket.clone();
@@ -3253,19 +3463,42 @@ impl SetDisks {
);
}
if let Some(err) = reduce_write_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, write_quorum) {
let reduced_err = reduce_write_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, write_quorum);
if let Some(err) = reduced_err {
let rollbacks = prepared_disks.iter().filter_map(|disk| {
disk.clone().map(|disk| {
let dst_object = dst_object.clone();
async move {
disk.settle_part_transaction(RUSTFS_META_MULTIPART_BUCKET, &dst_object, PartTransactionAction::Rollback)
.await
}
})
});
let rollback_results = join_all(rollbacks).await;
self.recover_part_transaction(&dst_object, write_quorum).await?;
if let Some(rollback_err) = rollback_results.iter().find_map(|result| result.as_ref().err()) {
warn!(error = %rollback_err, "rename_part rollback did not settle on every prepared disk");
}
if let Some(context) = quorum_context {
log_multipart_write_quorum_failure(context, &errs, write_quorum, &err);
} else {
warn!("rename_part errs {:?}", &errs);
}
self.cleanup_multipart_path(&[dst_object.to_string(), format!("{dst_object}.meta")])
.await;
return Err(err);
}
let disks = Self::eval_disks(disks, &errs);
Ok(disks)
let committed = self.recover_part_transaction(&dst_object, write_quorum).await?;
if !committed {
let err = DiskError::ErasureWriteQuorum;
if let Some(context) = quorum_context {
log_multipart_write_quorum_failure(context, &errs, write_quorum, &err);
} else {
warn!("rename_part errs {:?}", &errs);
}
return Err(err);
}
Ok(Self::eval_disks(&prepared_disks, &errs))
}
pub(in crate::set_disk) fn eval_disks(disks: &[Option<DiskStore>], errs: &[Option<DiskError>]) -> Vec<Option<DiskStore>> {
@@ -3931,6 +4164,9 @@ pub(in crate::set_disk) struct OldDataDirCleanup {
/// Number of attempted disks that returned `Ok` or a not-found variant
/// (a missing dir == already reclaimed).
pub reclaimed: usize,
/// Number of attempted disks that retained the directory for an active
/// snapshot lease and registered it for deletion after the final release.
pub deferred: usize,
/// Indices of attempted disks that failed with a non-ignored, non-not-found
/// error (including task panic/cancel). This is the residue that actually
/// leaks and drives the leak metric + heal enqueue.
@@ -3993,6 +4229,7 @@ fn classify_old_data_dir_cleanup(errs: &[Option<DiskError>], attempted: &[bool],
OldDataDirCleanup {
attempted: attempted_count,
reclaimed,
deferred: 0,
unreclaimed_disks,
below_quorum,
}
@@ -4933,6 +5170,40 @@ mod tests {
drop((disk1, disk2));
}
#[tokio::test]
async fn commit_cleanup_reports_and_releases_deferred_snapshot_data_dirs() {
let bucket = "cleanup-lease-bucket";
let object = "cleanup-lease-object";
let old_data_dir = "11111111-1111-1111-1111-111111111111";
let committed_data_dir = "22222222-2222-2222-2222-222222222222";
let data_dir_path = format!("{object}/{old_data_dir}");
let shard_path = format!("{data_dir_path}/part.1");
let (_dir1, disk1) = read_multiple_test_disk(bucket, &[(&shard_path, b"one".as_slice())]).await;
let set = io_primitives_test_set(vec![Some(disk1.clone())], 0).await;
let lease = disk1
.acquire_snapshot_lease(bucket, &data_dir_path)
.await
.expect("snapshot lease should be acquired before cleanup");
let cleanup = set
.commit_rename_data_dir(&[Some(disk1.clone())], bucket, object, old_data_dir, committed_data_dir, 1)
.await;
assert_eq!(cleanup.attempted, 1);
assert_eq!(cleanup.reclaimed, 0);
assert_eq!(cleanup.deferred, 1);
assert!(cleanup.unreclaimed_disks.is_empty());
disk1
.read_all(bucket, &shard_path)
.await
.expect("deferred cleanup must leave later shard opens available");
disk1
.release_snapshot_lease(bucket, &data_dir_path, lease)
.await
.expect("final lease release should reclaim the old data directory");
assert!(matches!(disk1.read_all(bucket, &shard_path).await, Err(DiskError::FileNotFound)));
}
/// Isolation guard: an armed barrier / observed object only affects its own
/// object. A fan-out for a different (unobserved, unarmed) object must not be
/// paused and must not accrue any tracked task count — so concurrent tests
+7
View File
@@ -578,6 +578,13 @@ impl SetDisks {
Self::update_hash_str(hasher, &meta.transition_tier);
Self::update_hash_str(hasher, &meta.transitioned_objname);
Self::update_hash_optional_uuid(hasher, meta.transition_version_id);
Self::update_hash_optional_str(hasher, meta.transition_version.as_deref());
hasher.update([match meta.transition_version_state {
rustfs_filemeta::TransitionVersionState::Unknown => 0,
rustfs_filemeta::TransitionVersionState::KnownDisabled => 1,
rustfs_filemeta::TransitionVersionState::SuspendedNull => 2,
rustfs_filemeta::TransitionVersionState::Exact => 3,
}]);
Self::update_hash_optional_u32(hasher, meta.mode);
Self::update_hash_optional_u64(hasher, meta.written_by_version);
+4
View File
@@ -687,9 +687,13 @@ mod core;
mod ctx;
mod metadata;
mod ops;
#[cfg(test)]
pub(crate) use ops::multipart::{MultipartCommitBarrier, MultipartCommitPause};
#[cfg(feature = "test-util")]
pub(crate) use ops::object::TransitionCleanupStoreBarrier as SetDiskTransitionCleanupStoreBarrier;
pub(crate) use ops::object::body_cache_plaintext_len;
#[cfg(test)]
pub(crate) use ops::object::cleanup_rejected_transition_upload_durably;
mod read;
mod replication;
pub(crate) mod shard_source;
@@ -199,6 +199,32 @@ mod tests {
.expect("healthy temp shard should verify");
assert_eq!(verified, 1);
for trailing in [vec![0xa5], vec![0xa5; 17]] {
let mut oversized = encoded.to_vec();
oversized.extend_from_slice(&trailing);
disk.write_all(RUSTFS_META_TMP_BUCKET, path, Bytes::from(oversized))
.await
.expect("oversized temp shard should be staged");
let err = verify_written_bitrot_shards(
&[Some(disk.clone())],
None,
BitrotSelfVerifyTarget {
operation: "put_object",
bucket: "bucket",
object: "object",
part_number: None,
volume: RUSTFS_META_TMP_BUCKET,
path,
logical_shard_size,
shard_size,
write_quorum: 1,
},
)
.await
.expect_err("disk shard with trailing bytes must not be committed");
assert!(err.to_string().contains("trailing data"));
}
let mut corrupt = encoded.to_vec();
let last = corrupt.len() - 1;
corrupt[last] ^= 0x01;
@@ -225,4 +251,40 @@ mod tests {
.expect_err("corrupt no-parity temp shard must not be committed");
assert!(err.to_string().contains("bitrot self-verify failed"));
}
#[tokio::test]
async fn no_parity_inline_self_verify_rejects_trailing_bytes() {
let (_temp_dirs, disks, _set_disks) = hermetic_set_disks_for_pool_with_default_parity(1, 0, 0).await;
let shard_size = 16usize;
let payload = b"inline bitrot payload";
let encoded = encode_streaming_shard(payload, shard_size).await;
let disks = disks.into_iter().map(Some).collect::<Vec<_>>();
for trailing in [vec![0xa5], vec![0xa5; 17]] {
let mut oversized = encoded.to_vec();
oversized.extend_from_slice(&trailing);
let parts = [FileInfo {
data: Some(Bytes::from(oversized)),
..Default::default()
}];
let err = verify_written_bitrot_shards(
&disks,
Some(&parts),
BitrotSelfVerifyTarget {
operation: "put_object",
bucket: "bucket",
object: "inline-object",
part_number: None,
volume: RUSTFS_META_TMP_BUCKET,
path: "unused-for-inline",
logical_shard_size: payload.len(),
shard_size,
write_quorum: 1,
},
)
.await
.expect_err("inline shard with trailing bytes must not be committed");
assert!(err.to_string().contains("trailing data"));
}
}
}
+257 -8
View File
@@ -19,6 +19,71 @@ use crate::storage_api_contracts::namespace::NamespaceLocking as _;
const LOG_COMPONENT_ECSTORE: &str = "ecstore";
const LOG_SUBSYSTEM_HEAL: &str = "heal";
const EVENT_HEAL_OBJECT_RENAME: &str = "heal_object_rename";
const HEAL_RENAME_INCOMPLETE: &str = "heal rename incomplete";
#[cfg(test)]
static HEAL_RENAME_FAILURES: std::sync::Mutex<Vec<(String, String, usize)>> = std::sync::Mutex::new(Vec::new());
#[cfg(test)]
struct HealRenameFailureScope {
bucket: String,
object: String,
}
#[cfg(test)]
impl HealRenameFailureScope {
fn install(bucket: &str, object: &str, disk_indexes: &[usize]) -> Self {
let mut failures = HEAL_RENAME_FAILURES
.lock()
.expect("heal rename failure registry should not poison");
assert!(
!failures
.iter()
.any(|(registered_bucket, registered_object, _)| { registered_bucket == bucket && registered_object == object }),
"heal rename failures must be installed once per object"
);
failures.extend(
disk_indexes
.iter()
.map(|index| (bucket.to_string(), object.to_string(), *index)),
);
Self {
bucket: bucket.to_string(),
object: object.to_string(),
}
}
}
#[cfg(test)]
impl Drop for HealRenameFailureScope {
fn drop(&mut self) {
HEAL_RENAME_FAILURES
.lock()
.expect("heal rename failure registry should not poison")
.retain(|(bucket, object, _)| bucket != &self.bucket || object != &self.object);
}
}
#[cfg(test)]
fn should_fail_heal_rename(bucket: &str, object: &str, disk_index: usize) -> bool {
let mut failures = HEAL_RENAME_FAILURES
.lock()
.expect("heal rename failure registry should not poison");
if let Some(position) = failures
.iter()
.position(|entry| entry == &(bucket.to_string(), object.to_string(), disk_index))
{
failures.swap_remove(position);
true
} else {
false
}
}
#[cfg(not(test))]
fn should_fail_heal_rename(_bucket: &str, _object: &str, _disk_index: usize) -> bool {
false
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
struct PartFailureSummary {
@@ -659,8 +724,12 @@ impl SetDisks {
}
}
// Rename from tmp location to the actual location.
// MinIO stops on the first RenameData error. RustFS intentionally
// continues per target, but reports any residue after all attempts
// so successful repairs survive and failed targets remain retryable.
let mut rename_attempts = 0usize;
let mut rename_successes = 0usize;
let mut healed_disks = vec![None; out_dated_disks.len()];
for (index, outdated_disk) in out_dated_disks.iter().enumerate() {
if let Some(disk) = outdated_disk {
rename_attempts += 1;
@@ -669,9 +738,18 @@ impl SetDisks {
// Attempt a rename now from healed data to final location.
parts_metadata[index].set_healing();
let rename_result = disk
.rename_data(RUSTFS_META_TMP_BUCKET, &tmp_id, parts_metadata[index].clone(), bucket, object)
.await;
let rename_result = if should_fail_heal_rename(bucket, object, index) {
Err(DiskError::Unexpected)
} else {
disk.rename_data(
RUSTFS_META_TMP_BUCKET,
&tmp_id,
parts_metadata[index].clone(),
bucket,
object,
)
.await
};
if let Err(err) = &rename_result {
warn!(
@@ -690,6 +768,7 @@ impl SetDisks {
);
} else {
rename_successes += 1;
healed_disks[index] = Some(disk.clone());
if parts_metadata[index].is_remote() {
let rm_data_dir =
parts_metadata[index].data_dir.expect("operation should succeed").to_string();
@@ -734,15 +813,18 @@ impl SetDisks {
.await
.map_err(DiskError::other)?;
if rename_attempts > 0 && rename_successes == 0 {
self.record_healed_capacity_scope(&healed_disks);
if rename_successes < rename_attempts {
return Ok((
result,
Some(DiskError::other(format!("all healed data rename attempts failed for {bucket}/{object}"))),
Some(DiskError::other(format!(
"{HEAL_RENAME_INCOMPLETE}: {rename_successes} of {rename_attempts} targets committed for \
{bucket}/{object}"
))),
));
}
self.record_healed_capacity_scope(&out_dated_disks);
// The object is healthy here; sweep any data dirs left behind
// by pre-#3510 unversioned overwrites, which the dangling paths
// above never touch (issues #3231, #3191). Best effort — a
@@ -1312,11 +1394,13 @@ impl crate::storage_api_contracts::heal::HealOperations for SetDisks {
#[cfg(test)]
mod heal_result_report_tests {
use super::SetDisks;
use super::{HEAL_RENAME_INCOMPLETE, HealRenameFailureScope};
use crate::disk::endpoint::Endpoint;
use crate::disk::error::DiskError;
use crate::disk::format::FormatV3;
use crate::disk::{DiskOption, DiskStore, new_disk};
use crate::disk::{DiskAPI as _, DiskOption, DiskStore, RUSTFS_META_TMP_BUCKET, ReadOptions, new_disk};
use crate::object_api::{ObjectOptions, PutObjReader};
use crate::set_disk::ops::object::hermetic_set_disks_support::hermetic_set_disks_isolated;
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
use crate::storage_api_contracts::object::{ObjectIO as _, ObjectOperations as _};
use crate::{config::storageclass, store::init_format::save_format_file};
@@ -1406,6 +1490,171 @@ mod heal_result_report_tests {
(dir, set)
}
async fn non_trash_tmp_entries(temp_dirs: &[TempDir]) -> Vec<String> {
let mut entries = Vec::new();
for dir in temp_dirs {
let tmp = dir.path().join(RUSTFS_META_TMP_BUCKET);
let mut read_dir = match tokio::fs::read_dir(&tmp).await {
Ok(read_dir) => read_dir,
Err(err) if err.kind() == std::io::ErrorKind::NotFound => continue,
Err(err) => panic!("tmp directory should be readable: {err}"),
};
while let Some(entry) = read_dir.next_entry().await.expect("tmp entry should be readable") {
let name = entry.file_name().to_string_lossy().into_owned();
if name != ".trash" {
entries.push(name);
}
}
}
entries
}
#[tokio::test]
#[serial_test::serial]
async fn heal_rename_outcome_matrix_reports_partial_and_retries_failed_targets() {
for (case, failed_attempts, expect_error) in [
("ok-ok", Vec::new(), false),
("ok-err", vec![1], true),
("err-ok", vec![0], true),
("err-err", vec![0, 1], true),
] {
let (temp_dirs, disks, set) = hermetic_set_disks_isolated(4).await;
let bucket = format!("heal-rename-{case}");
let object = "object.bin";
for disk in &disks {
disk.make_volume(&bucket).await.expect("bucket volume should be created");
}
let payload = vec![0x5a; 1024 * 1024];
let mut reader = PutObjReader::from_vec(payload);
set.put_object(&bucket, object, &mut reader, &ObjectOptions::default())
.await
.expect("source object should be written");
let source = disks[2]
.read_version("", &bucket, object, "", &ReadOptions::default())
.await
.expect("source metadata should be readable");
let data_dir = source.data_dir.expect("non-inline source should have a data directory");
let tmp_entries_before_heal = non_trash_tmp_entries(&temp_dirs).await;
let target_slots = {
let mut slots = [source.erasure.distribution[0] - 1, source.erasure.distribution[1] - 1];
slots.sort_unstable();
slots
};
let failed_slots = failed_attempts
.iter()
.map(|attempt| target_slots[*attempt])
.collect::<Vec<_>>();
let failed_physical_indexes = [0, 1]
.into_iter()
.filter(|index| failed_slots.contains(&(source.erasure.distribution[*index] - 1)))
.collect::<Vec<_>>();
for index in [0, 1] {
tokio::fs::remove_file(
temp_dirs[index]
.path()
.join(&bucket)
.join(object)
.join(data_dir.to_string())
.join("part.1"),
)
.await
.expect("target shard should be removed before heal");
}
let failure_scope = HealRenameFailureScope::install(&bucket, object, &failed_slots);
let (first_result, first_error) = set
.heal_object(
&bucket,
object,
"",
&HealOpts {
no_lock: true,
scan_mode: HealScanMode::Deep,
..Default::default()
},
)
.await
.expect("heal should report its per-target rename outcome");
drop(failure_scope);
assert_eq!(first_error.is_some(), expect_error, "{case}: aggregate status must match target outcomes");
if let Some(error) = first_error {
let error = error.to_string();
assert!(
error.contains(HEAL_RENAME_INCOMPLETE),
"{case}: partial/all failure must have an explicit retryable status: {error}"
);
assert!(
error.contains(&format!("{} of 2 targets committed", 2 - failed_slots.len())),
"{case}: aggregate status must distinguish partial from all-target failure: {error}"
);
}
for index in [0, 1] {
let expected = if failed_physical_indexes.contains(&index) {
DriveState::Missing
} else {
DriveState::Ok
};
assert_eq!(
first_result.after.drives[index].state,
expected.to_string(),
"{case}: after.drives must reflect the actual rename outcome at index {index}"
);
assert_eq!(
temp_dirs[index]
.path()
.join(&bucket)
.join(object)
.join(data_dir.to_string())
.join("part.1")
.exists(),
!failed_physical_indexes.contains(&index),
"{case}: tmp cleanup must neither delete committed shards nor expose failed targets"
);
}
let tmp_entries_after_heal = non_trash_tmp_entries(&temp_dirs).await;
assert!(
tmp_entries_after_heal
.iter()
.all(|entry| tmp_entries_before_heal.contains(entry)),
"{case}: first heal must not leave a new temporary shard: {tmp_entries_after_heal:?}"
);
if !failed_slots.is_empty() {
let (retry_result, retry_error) = set
.heal_object(
&bucket,
object,
"",
&HealOpts {
no_lock: true,
scan_mode: HealScanMode::Deep,
..Default::default()
},
)
.await
.expect("second heal should retry failed targets");
assert!(retry_error.is_none(), "{case}: second heal should complete remaining targets");
for index in [0, 1] {
assert_eq!(
retry_result.after.drives[index].state,
DriveState::Ok.to_string(),
"{case}: second heal must converge target {index}"
);
}
let tmp_entries_after_retry = non_trash_tmp_entries(&temp_dirs).await;
assert!(
tmp_entries_after_retry
.iter()
.all(|entry| tmp_entries_before_heal.contains(entry)),
"{case}: retry must not leave a new temporary shard: {tmp_entries_after_retry:?}"
);
}
}
}
// Regression for #955: an offline disk must contribute exactly one drive
// record. Before the fix the offline branch fell through and pushed a second
// (Corrupt) record for the same disk, so `before/after.drives` grew to
+38 -2
View File
@@ -69,6 +69,13 @@ struct HealWalkCollector {
}
impl HealWalkCollector {
fn lock_objects(&self) -> disk::error::Result<std::sync::MutexGuard<'_, Vec<HealWalkObject>>> {
self.objects.lock().map_err(|_| {
self.cancel.cancel();
DiskError::FileCorrupt
})
}
/// Expand one resolved entry into its versions and record it. Cancels the
/// walk once EITHER page bound (distinct object names OR expanded versions)
/// is met — always at a sorted object-key boundary so a heavily-versioned
@@ -105,7 +112,9 @@ impl HealWalkCollector {
let added = versions.len();
let (objs_len, ver_total) = {
let mut objects = self.objects.lock().unwrap();
let Ok(mut objects) = self.lock_objects() else {
return;
};
objects.push(HealWalkObject {
name: entry.name,
versions,
@@ -239,7 +248,10 @@ impl SetDisks {
Err(err) => return Err(err),
}
let objects = std::mem::take(&mut *collector.objects.lock().unwrap());
let objects = {
let mut objects = collector.lock_objects()?;
std::mem::take(&mut *objects)
};
let truncated = collector.truncated.load(Ordering::SeqCst);
Ok(finalize_heal_walk_page(objects, forward_to, truncated))
}
@@ -310,4 +322,28 @@ mod tests {
assert_eq!(next_forward, None, "a complete final page must not carry a resume cursor");
assert!(!truncated);
}
#[test]
fn poisoned_collector_state_cancels_the_walk() {
let collector = Arc::new(HealWalkCollector {
bucket: "bucket".to_string(),
batch_objects: 2,
version_budget: 2,
objects: Mutex::new(Vec::new()),
version_total: AtomicUsize::new(0),
truncated: AtomicBool::new(false),
cancel: CancellationToken::new(),
});
let poison_target = Arc::clone(&collector);
let _ = std::thread::spawn(move || {
let _guard = poison_target.objects.lock().expect("fresh mutex should lock");
panic!("poison heal collector");
})
.join();
let error = collector.lock_objects().expect_err("poisoned collection must fail closed");
assert_eq!(error, DiskError::FileCorrupt);
assert!(collector.cancel.is_cancelled(), "a poisoned page collector must cancel its walk");
assert!(collector.objects.lock().is_err(), "poisoned state must remain fail-closed");
}
}
+23
View File
@@ -29,6 +29,12 @@ impl SetDisks {
pub async fn delete_all(&self, bucket: &str, prefix: &str) -> Result<()> {
ListOperations::new(self.ctx()).delete_all(bucket, prefix).await
}
pub(crate) async fn delete_all_with_quorum(&self, bucket: &str, prefix: &str, write_quorum: usize) -> Result<()> {
ListOperations::new(self.ctx())
.delete_all_with_quorum(bucket, prefix, write_quorum)
.await
}
}
/// List/prefix maintenance operations, borrowing the `SetDisks` core state
@@ -48,6 +54,14 @@ impl<'a> ListOperations<'a> {
}
pub(crate) async fn delete_all(&self, bucket: &str, prefix: &str) -> Result<()> {
self.delete_all_inner(bucket, prefix, None).await
}
async fn delete_all_with_quorum(&self, bucket: &str, prefix: &str, write_quorum: usize) -> Result<()> {
self.delete_all_inner(bucket, prefix, Some(write_quorum)).await
}
async fn delete_all_inner(&self, bucket: &str, prefix: &str, write_quorum: Option<usize>) -> Result<()> {
let disks = self.ctx.disks().read().await;
let disks = disks.clone();
@@ -79,6 +93,9 @@ impl<'a> ListOperations<'a> {
Ok(_) => {
errors.push(None);
}
Err(DiskError::FileNotFound | DiskError::PathNotFound | DiskError::VolumeNotFound) => {
errors.push(None);
}
Err(e) => {
errors.push(Some(e));
}
@@ -97,6 +114,12 @@ impl<'a> ListOperations<'a> {
);
}
if let Some(write_quorum) = write_quorum
&& let Some(err) = reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, write_quorum)
{
return Err(err.into());
}
Ok(())
}
}
+525 -39
View File
@@ -26,6 +26,8 @@ use crate::crash_inject::{self, CrashPoint};
use crate::multipart_listing::paginate_multipart_listing;
use futures::{StreamExt, stream};
use std::future::Future;
#[cfg(test)]
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Duration;
use tokio::task::JoinSet;
@@ -33,7 +35,7 @@ const MULTIPART_LIST_IO_CONCURRENCY: usize = 16;
#[cfg(test)]
#[derive(Clone, Copy, PartialEq, Eq)]
enum MultipartCommitPause {
pub(crate) enum MultipartCommitPause {
PutPartBeforeLockLost,
PutPartAfterRename,
BeforeLockLost,
@@ -45,12 +47,13 @@ struct MultipartCommitBarrierState {
bucket: String,
object: String,
pause: MultipartCommitPause,
armed: AtomicBool,
arrived: tokio::sync::Notify,
release: tokio::sync::Notify,
}
#[cfg(test)]
struct MultipartCommitBarrier {
pub(crate) struct MultipartCommitBarrier {
state: Arc<MultipartCommitBarrierState>,
}
@@ -60,11 +63,12 @@ static MULTIPART_COMMIT_BARRIER: std::sync::OnceLock<std::sync::Mutex<Option<Arc
#[cfg(test)]
impl MultipartCommitBarrier {
fn install(bucket: &str, object: &str, pause: MultipartCommitPause) -> Self {
pub(crate) fn install(bucket: &str, object: &str, pause: MultipartCommitPause) -> Self {
let state = Arc::new(MultipartCommitBarrierState {
bucket: bucket.to_string(),
object: object.to_string(),
pause,
armed: AtomicBool::new(true),
arrived: tokio::sync::Notify::new(),
release: tokio::sync::Notify::new(),
});
@@ -78,13 +82,13 @@ impl MultipartCommitBarrier {
Self { state }
}
async fn wait_until_paused(&self) {
pub(crate) async fn wait_until_paused(&self) {
tokio::time::timeout(Duration::from_secs(30), self.state.arrived.notified())
.await
.expect("multipart completion should reach the deterministic commit barrier");
}
fn release(&self) {
pub(crate) fn release(&self) {
self.state.release.notify_one();
}
}
@@ -112,7 +116,9 @@ async fn pause_multipart_commit(bucket: &str, object: &str, pause: MultipartComm
.as_ref()
.filter(|barrier| barrier.bucket == bucket && barrier.object == object && barrier.pause == pause)
.cloned();
if let Some(barrier) = barrier {
if let Some(barrier) = barrier
&& barrier.armed.swap(false, Ordering::AcqRel)
{
barrier.arrived.notify_one();
barrier.release.notified().await;
}
@@ -777,6 +783,9 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
mut max_parts: usize,
opts: &ObjectOptions,
) -> Result<ListPartsInfo> {
let _upload_guard = self
.acquire_multipart_upload_read_lock("list_object_parts", bucket, object, upload_id, opts)
.await?;
let (fi, _) = self.check_upload_id_exists(bucket, object, upload_id, false).await?;
let upload_id_path = Self::get_upload_id_dir(bucket, object, upload_id);
@@ -1254,10 +1263,15 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
let _upload_guard = self
.acquire_multipart_upload_write_lock("abort_multipart_upload", bucket, object, upload_id, opts)
.await?;
self.check_upload_id_exists(bucket, object, upload_id, false).await?;
let (fi, _) = self.check_upload_id_exists(bucket, object, upload_id, true).await?;
let upload_id_path = Self::get_upload_id_dir(bucket, object, upload_id);
self.delete_all(RUSTFS_META_MULTIPART_BUCKET, &upload_id_path).await
self.delete_all_with_quorum(
RUSTFS_META_MULTIPART_BUCKET,
&upload_id_path,
fi.write_quorum(self.default_write_quorum()),
)
.await
}
// complete_multipart_upload finished
#[tracing::instrument(skip(self))]
@@ -1289,8 +1303,18 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
}
}
if !opts.no_lock && object_lock_guard.is_none() {
object_lock_guard = Some(
self.acquire_write_lock_diag("complete_multipart_upload_commit", bucket, object)
.await?,
);
}
let upload_guard = self
.acquire_multipart_upload_write_lock("complete_multipart_upload_commit", bucket, object, upload_id, opts)
.await?;
let expected_restore_operation_id = restore_commit_operation_id_from_metadata(&opts.user_defined)?;
let (mut fi, mut files_metas) = self.check_upload_id_exists(bucket, object, upload_id, true).await?;
let (mut fi, files_metas) = self.check_upload_id_exists(bucket, object, upload_id, true).await?;
let has_layout_candidate = range_seek_rollout_enabled
&& fi
.data_dir
@@ -1303,22 +1327,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
&token,
)
});
let upload_guard = if has_layout_candidate {
if object_lock_guard.is_none() {
object_lock_guard = Some(
self.acquire_write_lock_diag("complete_multipart_upload_commit", bucket, object)
.await?,
);
}
let guard = self
.acquire_multipart_upload_write_lock("complete_multipart_upload_commit", bucket, object, upload_id, opts)
.await?;
(fi, files_metas) = self.check_upload_id_exists(bucket, object, upload_id, true).await?;
guard
} else {
None
};
let quorum_validated_layout_token = upload_guard.as_ref().and_then(|_| {
let quorum_validated_layout_token = if has_layout_candidate {
fi.data_dir
.filter(|data_dir| !data_dir.is_nil())
.map(|data_dir| data_dir.to_string())
@@ -1329,7 +1338,9 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
token,
)
})
});
} else {
None
};
rustfs_utils::http::metadata_compat::remove_str(
&mut fi.metadata,
crate::object_api::ENCRYPTED_PART_LAYOUT_CANDIDATE_SUFFIX,
@@ -1356,6 +1367,9 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
// let disks = Self::shuffle_disks(&disks, &fi.erasure.distribution);
let part_path = format!("{}/{}/", upload_id_path, fi.data_dir.unwrap_or(Uuid::nil()));
self.recover_part_transactions(&part_path, read_quorum, write_quorum)
.await
.map_err(|err| to_object_err(err.into(), vec![bucket, object]))?;
let part_meta_paths = uploaded_parts
.iter()
@@ -1713,12 +1727,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
}
}
if !opts.no_lock && object_lock_guard.is_none() {
object_lock_guard = Some(
self.acquire_write_lock_diag("complete_multipart_upload_commit", bucket, object)
.await?,
);
}
// Phase 2 (backlog#899): fence the commit on lock loss before any destructive
// step. If the refresh heartbeat has observed a refresh-quorum loss, another
// writer may have re-acquired this object's lock; proceeding would race a
@@ -1821,7 +1829,36 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
#[cfg(test)]
pause_multipart_commit(bucket, object, MultipartCommitPause::AfterRename).await;
drop(upload_guard);
let cleanup_store = self.clone();
let cleanup_upload_id_path = upload_id_path.clone();
let cleanup_bucket = bucket.to_owned();
let cleanup_object = object.to_owned();
let cleanup_upload_id = upload_id.to_owned();
let cleanup_handle = tokio::spawn(async move {
let _upload_guard = upload_guard;
if let Err(err) = cleanup_store
.delete_all_with_quorum(RUSTFS_META_MULTIPART_BUCKET, &cleanup_upload_id_path, write_quorum)
.await
{
warn!(
bucket = %cleanup_bucket,
object = %cleanup_object,
upload_id = %cleanup_upload_id,
error = ?err,
"completed multipart upload staging cleanup did not reach write quorum"
);
}
});
if let Err(err) = cleanup_handle.await {
warn!(
bucket = %bucket,
object = %object,
upload_id = %upload_id,
error = ?err,
"completed multipart upload staging cleanup task failed"
);
}
drop(object_lock_guard); // drop object lock guard to release the lock
// backlog#1321: enqueue heal only when the committed replicas actually
@@ -1866,12 +1903,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
});
}
let upload_id_path = upload_id_path.clone();
let store = self.clone();
let _cleanup_handle = tokio::spawn(async move {
let _ = store.delete_all(RUSTFS_META_MULTIPART_BUCKET, &upload_id_path).await;
});
for (i, op_disk) in online_disks.iter().enumerate() {
if let Some(disk) = op_disk
&& disk.is_online().await
@@ -2183,6 +2214,317 @@ mod tests {
)
}
async fn assert_complete_first_linearizes(bucket: &'static str, object: &'static str, create_opts: ObjectOptions) {
let manager = Arc::new(rustfs_lock::GlobalLockManager::new());
let signaling = Arc::new(SignalingLockClient::new(Arc::new(LocalClient::with_manager(manager))));
let lockers: Vec<Arc<dyn LockClient>> = vec![signaling.clone()];
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await;
make_bucket_on_all(&disk_stores, bucket).await;
let (upload_id, parts) = stage_upload_with_create_opts(&set_disks, bucket, object, &[0x47; 4096], &create_opts).await;
let upload_id_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id);
signaling.set_target(rustfs_lock::ObjectKey::new(RUSTFS_META_MULTIPART_BUCKET, upload_id_path));
let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::DistErasure).await;
let barrier = MultipartCommitBarrier::install(bucket, object, MultipartCommitPause::AfterRename);
let complete_store = set_disks.clone();
let complete_upload_id = upload_id.clone();
let complete = tokio::spawn(async move {
complete_store
.complete_multipart_upload(bucket, object, &complete_upload_id, parts, &ObjectOptions::default())
.await
});
barrier.wait_until_paused().await;
let abort_store = set_disks.clone();
let abort_upload_id = upload_id.clone();
let abort = tokio::spawn(async move {
abort_store
.abort_multipart_upload(bucket, object, &abort_upload_id, &ObjectOptions::default())
.await
});
signaling.wait_for_attempts(2).await;
assert!(!abort.is_finished(), "abort must wait for the completion upload lock");
barrier.release();
complete
.await
.expect("completion task should not panic")
.expect("completion should win the upload finalization");
let abort_err = abort
.await
.expect("abort task should not panic")
.expect_err("abort must observe the upload as finalized");
assert!(matches!(abort_err, StorageError::InvalidUploadID(..)));
set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("complete-first must leave the committed object readable");
assert!(matches!(
set_disks.check_upload_id_exists(bucket, object, &upload_id, false).await,
Err(StorageError::InvalidUploadID(..))
));
}
async fn assert_abort_first_linearizes(bucket: &'static str, object: &'static str, create_opts: ObjectOptions) {
let manager = Arc::new(rustfs_lock::GlobalLockManager::new());
let signaling = Arc::new(SignalingLockClient::new(Arc::new(LocalClient::with_manager(manager))));
let lockers: Vec<Arc<dyn LockClient>> = vec![signaling.clone()];
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await;
make_bucket_on_all(&disk_stores, bucket).await;
let (upload_id, parts) = stage_upload_with_create_opts(&set_disks, bucket, object, &[0x48; 4096], &create_opts).await;
let upload_id_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id);
signaling.set_target(rustfs_lock::ObjectKey::new(RUSTFS_META_MULTIPART_BUCKET, upload_id_path.clone()));
let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::DistErasure).await;
let object_holder = set_disks
.new_ns_lock(bucket, object)
.await
.expect("object namespace lock should be created")
.get_write_lock(Duration::from_secs(5))
.await
.expect("test should hold the object lock");
let holder = set_disks
.new_ns_lock(RUSTFS_META_MULTIPART_BUCKET, &upload_id_path)
.await
.expect("upload namespace lock should be created")
.get_write_lock(Duration::from_secs(5))
.await
.expect("test should hold the upload lock");
signaling.wait_for_attempts(1).await;
let abort_store = set_disks.clone();
let abort_upload_id = upload_id.clone();
let abort = tokio::spawn(async move {
abort_store
.abort_multipart_upload(bucket, object, &abort_upload_id, &ObjectOptions::default())
.await
});
signaling.wait_for_attempts(2).await;
let complete_store = set_disks.clone();
let complete_upload_id = upload_id.clone();
let complete = tokio::spawn(async move {
complete_store
.complete_multipart_upload(bucket, object, &complete_upload_id, parts, &ObjectOptions::default())
.await
});
drop(holder);
abort
.await
.expect("abort task should not panic")
.expect("abort should win the upload finalization");
drop(object_holder);
let complete_err = complete
.await
.expect("completion task should not panic")
.expect_err("completion must observe the aborted upload");
assert!(matches!(complete_err, StorageError::InvalidUploadID(..)));
let object_err = set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect_err("abort-first must not publish an object");
assert!(matches!(object_err, StorageError::ObjectNotFound(..)));
assert!(matches!(
set_disks.check_upload_id_exists(bucket, object, &upload_id, false).await,
Err(StorageError::InvalidUploadID(..))
));
}
async fn assert_quorum_minus_one_retry_preserves_completable_part(
disk_count: usize,
parity: usize,
success_indices: &[usize],
) {
use tokio::io::AsyncReadExt as _;
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks_for_pool_with_default_parity(disk_count, 0, parity).await;
let bucket = format!("multipart-retry-{disk_count}-{}", success_indices[0]);
let object = "object";
make_bucket_on_all(&disk_stores, &bucket).await;
let payload = vec![0x41; 1 << 20];
let (upload_id, parts) =
stage_upload_with_create_opts(&set_disks, &bucket, object, &payload, &ObjectOptions::default()).await;
let (upload_meta, _) = set_disks
.check_upload_id_exists(&bucket, object, &upload_id, false)
.await
.expect("staged upload metadata should be readable");
let upload_path = SetDisks::get_upload_id_dir(&bucket, object, &upload_id);
let write_quorum = upload_meta.write_quorum(set_disks.default_write_quorum());
assert_eq!(success_indices.len() + 1, write_quorum);
let part_path = format!(
"{}/{}/part.1",
upload_path,
upload_meta.data_dir.expect("multipart upload should have a data directory")
);
let acknowledged_meta = disk_stores[0]
.read_all(RUSTFS_META_MULTIPART_BUCKET, &format!("{part_path}.meta"))
.await
.expect("acknowledged part metadata should be readable");
let retry_path = format!("{}/part.1", Uuid::new_v4());
let mut retry_disks = vec![None; disk_stores.len()];
for &index in success_indices {
disk_stores[index]
.write_all(RUSTFS_META_TMP_BUCKET, &retry_path, Bytes::from_static(b"retry shard"))
.await
.expect("retry shard should be staged");
retry_disks[index] = Some(disk_stores[index].clone());
}
let err = set_disks
.rename_part(
&retry_disks,
RUSTFS_META_TMP_BUCKET,
&retry_path,
RUSTFS_META_MULTIPART_BUCKET,
&part_path,
acknowledged_meta,
write_quorum,
None,
)
.await
.expect_err("quorum-minus-one renamed shards must remain below write quorum");
assert_eq!(err, DiskError::ErasureWriteQuorum);
for (index, disk) in disk_stores.iter().enumerate() {
assert!(
disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &part_path).await.is_ok(),
"old acknowledged shard on disk {index} must survive"
);
}
set_disks
.clone()
.complete_multipart_upload(&bucket, object, &upload_id, parts, &ObjectOptions::default())
.await
.expect("the old acknowledged part should remain completable");
let mut reader = set_disks
.get_object_reader(&bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
.await
.expect("completed object should be readable");
let mut restored = Vec::new();
reader
.stream
.read_to_end(&mut restored)
.await
.expect("completed object should stream fully");
assert_eq!(restored, payload);
}
#[tokio::test]
async fn upload_part_retry_quorum_failure_preserves_old_part_across_ec_geometries() {
assert_quorum_minus_one_retry_preserves_completable_part(4, 2, &[0, 1]).await;
assert_quorum_minus_one_retry_preserves_completable_part(4, 2, &[2, 3]).await;
assert_quorum_minus_one_retry_preserves_completable_part(6, 2, &[0, 1, 2]).await;
assert_quorum_minus_one_retry_preserves_completable_part(6, 2, &[3, 4, 5]).await;
}
#[tokio::test]
async fn complete_multipart_upload_recovers_interrupted_part_retry() {
use tokio::io::AsyncReadExt as _;
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks_for_pool_with_default_parity(6, 0, 2).await;
let bucket = "multipart-retry-recovery";
let object = "object";
make_bucket_on_all(&disk_stores, bucket).await;
let payload = vec![0x42; 1 << 20];
let (upload_id, parts) =
stage_upload_with_create_opts(&set_disks, bucket, object, &payload, &ObjectOptions::default()).await;
let (upload_meta, _) = set_disks
.check_upload_id_exists(bucket, object, &upload_id, false)
.await
.expect("staged upload metadata should be readable");
let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id);
let part_path = format!(
"{}/{}/part.1",
upload_path,
upload_meta.data_dir.expect("multipart upload should have a data directory")
);
let retry_meta = Bytes::from_static(b"interrupted retry metadata");
for (index, disk) in disk_stores.iter().enumerate().take(3) {
let retry_path = format!("{}/part.1", Uuid::new_v4());
disk.write_all(RUSTFS_META_TMP_BUCKET, &retry_path, Bytes::from_static(b"interrupted retry shard"))
.await
.expect("retry shard should be staged");
disk.prepare_part_transaction(
RUSTFS_META_TMP_BUCKET,
&retry_path,
RUSTFS_META_MULTIPART_BUCKET,
&part_path,
retry_meta.clone(),
)
.await
.expect("part transaction should be prepared");
disk.rename_part(
RUSTFS_META_TMP_BUCKET,
&retry_path,
RUSTFS_META_MULTIPART_BUCKET,
&part_path,
retry_meta.clone(),
)
.await
.unwrap_or_else(|err| panic!("retry shard {index} should be published: {err}"));
}
set_disks
.clone()
.complete_multipart_upload(bucket, object, &upload_id, parts, &ObjectOptions::default())
.await
.expect("completion should roll back the interrupted quorum-minus-one retry");
let mut reader = set_disks
.get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
.await
.expect("completed object should be readable");
let mut restored = Vec::new();
reader
.stream
.read_to_end(&mut restored)
.await
.expect("completed object should stream fully");
assert_eq!(restored, payload);
}
#[tokio::test]
async fn rename_part_quorum_failure_without_old_part_removes_new_shards() {
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks_for_pool_with_default_parity(4, 0, 2).await;
let src_path = format!("{}/part.1", Uuid::new_v4());
let dst_path = format!("{}/part.1", Uuid::new_v4());
let mut retry_disks = vec![None; disk_stores.len()];
for index in [0, 1] {
disk_stores[index]
.write_all(RUSTFS_META_TMP_BUCKET, &src_path, Bytes::from_static(b"new shard"))
.await
.expect("new shard should be staged");
retry_disks[index] = Some(disk_stores[index].clone());
}
let err = set_disks
.rename_part(
&retry_disks,
RUSTFS_META_TMP_BUCKET,
&src_path,
RUSTFS_META_MULTIPART_BUCKET,
&dst_path,
Bytes::from_static(b"retry metadata"),
3,
None,
)
.await
.expect_err("two renamed shards must remain below write quorum");
assert_eq!(err, DiskError::ErasureWriteQuorum);
for (index, disk) in disk_stores.iter().enumerate() {
assert!(
matches!(disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &dst_path).await, Err(DiskError::FileNotFound)),
"failed first upload must not leave a destination on disk {index}"
);
}
}
async fn make_multipart_lock_test_set_disks() -> Arc<SetDisks> {
let endpoints = vec![
Endpoint::try_from("http://127.0.0.1:9000/data").expect("first endpoint should parse"),
@@ -2850,6 +3192,81 @@ mod tests {
.await;
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn abort_and_complete_linearize_for_plain_sse_and_legacy_layouts() {
assert_complete_first_linearizes("multipart-complete-first-plain", "object", ObjectOptions::default()).await;
assert_abort_first_linearizes("multipart-abort-first-plain", "object", ObjectOptions::default()).await;
let encrypted_opts = ObjectOptions {
user_defined: HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())]),
..Default::default()
};
temp_env::async_with_vars([(crate::object_api::ENV_RUSTFS_ENCRYPTED_RANGE_SEEK, Some("true"))], async {
assert_complete_first_linearizes("multipart-complete-first-sse", "object", encrypted_opts.clone()).await;
assert_abort_first_linearizes("multipart-abort-first-sse", "object", encrypted_opts.clone()).await;
})
.await;
temp_env::async_with_vars([(crate::object_api::ENV_RUSTFS_ENCRYPTED_RANGE_SEEK, Some("false"))], async {
assert_complete_first_linearizes("multipart-complete-first-legacy", "object", encrypted_opts.clone()).await;
assert_abort_first_linearizes("multipart-abort-first-legacy", "object", encrypted_opts).await;
})
.await;
}
#[tokio::test]
async fn abort_enforces_delete_write_quorum_boundary() {
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "multipart-abort-delete-quorum";
let object = "object";
make_bucket_on_all(&disk_stores, bucket).await;
let quorum_upload = set_disks
.new_multipart_upload(bucket, object, &ObjectOptions::default())
.await
.expect("multipart upload should be created");
let saved_disks = {
let mut disks = set_disks.disks.write().await;
let saved = disks.clone();
disks[3] = None;
saved
};
set_disks
.abort_multipart_upload(bucket, object, &quorum_upload.upload_id, &ObjectOptions::default())
.await
.expect("abort should succeed at the exact delete write quorum");
*set_disks.disks.write().await = saved_disks;
assert!(matches!(
set_disks
.check_upload_id_exists(bucket, object, &quorum_upload.upload_id, false)
.await,
Err(StorageError::InvalidUploadID(..))
));
let below_quorum_upload = set_disks
.new_multipart_upload(bucket, object, &ObjectOptions::default())
.await
.expect("second multipart upload should be created");
let saved_disks = {
let mut disks = set_disks.disks.write().await;
let saved = disks.clone();
disks[2] = None;
disks[3] = None;
saved
};
let err = set_disks
.abort_multipart_upload(bucket, object, &below_quorum_upload.upload_id, &ObjectOptions::default())
.await
.expect_err("abort must report a delete below write quorum");
assert!(matches!(err, StorageError::ErasureWriteQuorum));
*set_disks.disks.write().await = saved_disks;
set_disks
.check_upload_id_exists(bucket, object, &below_quorum_upload.upload_id, false)
.await
.expect("failed abort must leave quorum-visible staging on the restored disks");
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn complete_revalidates_layout_candidate_after_upload_lock() {
@@ -2981,6 +3398,17 @@ mod tests {
tokio::task::yield_now().await;
assert!(!abort.is_finished(), "abort must wait until completion releases the upload lock");
let list_store = set_disks.clone();
let list_upload_id = upload_id.clone();
let list = tokio::spawn(async move {
list_store
.list_object_parts(bucket, object, &list_upload_id, None, MAX_PARTS_COUNT, &ObjectOptions::default())
.await
});
signaling.wait_for_attempts(3).await;
tokio::task::yield_now().await;
assert!(!list.is_finished(), "ListParts must wait until completion releases the upload lock");
barrier.release();
complete
.await
@@ -2991,10 +3419,68 @@ mod tests {
.expect("abort task should not panic")
.expect_err("the committed upload should no longer exist when abort acquires the lock");
assert!(matches!(abort_err, StorageError::InvalidUploadID(..)));
let list_err = list
.await
.expect("ListParts task should not panic")
.expect_err("the committed upload should no longer exist when ListParts acquires the lock");
assert!(matches!(list_err, StorageError::InvalidUploadID(..)));
})
.await;
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn complete_validates_parts_after_an_inflight_upload_part_commit() {
let manager = Arc::new(rustfs_lock::GlobalLockManager::new());
let signaling = Arc::new(SignalingLockClient::new(Arc::new(LocalClient::with_manager(manager))));
let lockers: Vec<Arc<dyn LockClient>> = vec![signaling.clone()];
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await;
let bucket = "multipart-complete-put-part-race-bucket";
let object = "object";
make_bucket_on_all(&disk_stores, bucket).await;
let (upload_id, original_parts) =
stage_upload_with_create_opts(&set_disks, bucket, object, &[0x49; 4096], &ObjectOptions::default()).await;
let upload_id_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id);
signaling.set_target(rustfs_lock::ObjectKey::new(RUSTFS_META_MULTIPART_BUCKET, upload_id_path));
let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::DistErasure).await;
let barrier = MultipartCommitBarrier::install(bucket, object, MultipartCommitPause::PutPartBeforeLockLost);
let put_store = set_disks.clone();
let put_upload_id = upload_id.clone();
let put = tokio::spawn(async move {
let mut reader = PutObjReader::from_vec(vec![0x4a; 4096]);
put_store
.put_object_part(bucket, object, &put_upload_id, 1, &mut reader, &ObjectOptions::default())
.await
});
barrier.wait_until_paused().await;
let complete_store = set_disks.clone();
let complete_upload_id = upload_id.clone();
let complete = tokio::spawn(async move {
complete_store
.complete_multipart_upload(bucket, object, &complete_upload_id, original_parts, &ObjectOptions::default())
.await
});
signaling.wait_for_attempts(2).await;
tokio::task::yield_now().await;
assert!(!complete.is_finished(), "completion must wait for the UploadPart commit lock");
barrier.release();
put.await
.expect("UploadPart task should not panic")
.expect("UploadPart replacement should commit");
let err = complete
.await
.expect("completion task should not panic")
.expect_err("completion must reject the stale ETag after UploadPart wins");
assert!(matches!(err, StorageError::InvalidPart(..)));
set_disks
.list_object_parts(bucket, object, &upload_id, None, MAX_PARTS_COUNT, &ObjectOptions::default())
.await
.expect("failed completion must leave the upload retryable");
}
#[tokio::test(start_paused = true)]
#[serial]
async fn complete_fences_upload_lock_loss_before_commit() {
File diff suppressed because it is too large Load Diff
+167 -17
View File
@@ -207,11 +207,9 @@ impl SetDisks {
version_id: &str,
opts: &ReadOptions,
) -> Result<Vec<FileInfo>> {
// Use existing disk selection logic
let disks = self.disks.read().await;
let required_reads = self.format.erasure.sets.len();
let disks = self.disks.read().await.clone();
let required_reads = self.default_read_quorum();
// Clone parameters outside the closure to avoid lifetime issues
let bucket = bucket.to_string();
let object = object.to_string();
let version_id = version_id.to_string();
@@ -220,7 +218,6 @@ impl SetDisks {
let processor = runtime_sources::batch_processors().read_processor();
let tasks: Vec<_> = disks
.iter()
.take(required_reads + 2) // Read a few extra for reliability
.filter_map(|disk| {
disk.as_ref().map(|d| {
let disk = d.clone();
@@ -234,10 +231,10 @@ impl SetDisks {
})
.collect();
match processor.execute_batch_with_quorum(tasks, required_reads).await {
Ok(results) => Ok(results),
Err(_) => Err(DiskError::FileNotFound.into()), // Use existing error type
}
processor
.execute_batch_with_quorum(tasks, required_reads)
.await
.map_err(Into::into)
}
#[tracing::instrument(level = "debug", skip(self))]
@@ -1960,6 +1957,42 @@ mod metadata_cache_tests {
(dir, disk)
}
async fn read_version_quorum_test_set(
bucket: &str,
object: &str,
pool_set_count: usize,
readable_disks: usize,
) -> (Vec<tempfile::TempDir>, Arc<SetDisks>) {
let mut dirs = Vec::new();
let mut disks = Vec::new();
for disk_index in 0..4 {
let (dir, disk) = new_read_version_test_disk(bucket).await;
if disk_index < readable_disks {
let mut fi = valid_test_fileinfo(object);
fi.mod_time = Some(OffsetDateTime::now_utc());
fi.erasure.index = fi.erasure.distribution[disk_index];
disk.write_metadata(bucket, bucket, object, fi)
.await
.expect("metadata should be written before quorum read");
}
dirs.push(dir);
disks.push(Some(disk));
}
let set = SetDisks::new(
"read-version-quorum-test".to_string(),
Arc::new(RwLock::new(disks)),
4,
2,
0,
0,
Vec::new(),
FormatV3::new(pool_set_count, 4),
Vec::new(),
)
.await;
(dirs, set)
}
#[test]
#[serial]
fn get_object_metadata_cache_capacity_uses_default_and_env_override() {
@@ -1983,9 +2016,40 @@ mod metadata_cache_tests {
fi.size = 1;
fi.erasure.index = 1;
fi.metadata.insert("etag".to_string(), "etag-1".to_string());
fi.add_object_part(1, "part-etag".to_string(), 1, fi.mod_time, 1, None, None);
fi
}
#[tokio::test]
async fn get_object_with_fileinfo_rejects_positive_size_without_parts() {
let mut fi = valid_test_fileinfo("object");
fi.parts.clear();
let mut output = Vec::new();
let err = SetDisks::get_object_with_fileinfo(
"bucket",
"object",
0,
1,
&mut output,
fi,
Vec::new(),
&[],
0,
0,
false,
false,
GET_OBJECT_PATH_SET_DISK,
"plain",
"small",
)
.await
.expect_err("positive-size metadata without parts must fail without panicking");
assert_eq!(err, Error::FileCorrupt);
assert!(output.is_empty());
}
#[tokio::test]
async fn get_object_with_fileinfo_rejects_invalid_ranges_before_reader_setup() {
let bucket = "bucket";
@@ -2087,6 +2151,38 @@ mod metadata_cache_tests {
assert!(output.is_empty());
}
#[tokio::test]
async fn get_object_with_fileinfo_accepts_zero_size_without_parts() {
let bucket = "bucket";
let object = "empty";
let mut fi = valid_test_fileinfo(object);
fi.size = 0;
fi.parts.clear();
let mut output = Vec::new();
SetDisks::get_object_with_fileinfo(
bucket,
object,
0,
0,
&mut output,
fi,
Vec::new(),
&[],
0,
0,
false,
false,
GET_OBJECT_PATH_SET_DISK,
"plain",
"empty",
)
.await
.expect("zero-byte object without parts must remain readable");
assert!(output.is_empty());
}
#[tokio::test]
async fn get_object_with_fileinfo_fails_closed_without_read_quorum() {
let bucket = "bucket";
@@ -2094,12 +2190,6 @@ mod metadata_cache_tests {
let mut fi = valid_test_fileinfo(object);
fi.erasure.block_size = 1;
fi.erasure.distribution = vec![1, 2, 3, 4];
fi.parts.push(ObjectPartInfo {
number: 1,
size: 1,
actual_size: 1,
..Default::default()
});
let mut output = Vec::new();
let err = SetDisks::get_object_with_fileinfo(
@@ -2133,6 +2223,7 @@ mod metadata_cache_tests {
let object = "object";
let (_dir, disk) = new_read_version_test_disk(bucket).await;
let mut fi = valid_test_fileinfo(object);
fi.size = 0;
fi.mod_time = Some(OffsetDateTime::now_utc());
disk.write_metadata(bucket, bucket, object, fi.clone())
.await
@@ -2165,11 +2256,70 @@ mod metadata_cache_tests {
.await
.expect_err("empty disk set must fail closed");
assert!(
missing_quorum.to_string().to_ascii_lowercase().contains("file"),
"optimized read failure should map to file-not-found style error: {missing_quorum}"
missing_quorum.to_string().contains("Insufficient successful results"),
"optimized read failure should preserve the batch quorum diagnostic: {missing_quorum}"
);
}
#[tokio::test]
async fn read_version_optimized_uses_current_set_drive_quorum_not_pool_set_count() {
let bucket = "read-version-layout-quorum-bucket";
let object = "object";
let (_single_set_dirs, single_set) = read_version_quorum_test_set(bucket, object, 1, 1).await;
single_set
.read_version_optimized(bucket, object, "", &ReadOptions::default())
.await
.expect_err("one metadata copy must not satisfy a four-drive 2+2 set");
let (_multi_set_dirs, multi_set) = read_version_quorum_test_set(bucket, object, 3, 2).await;
let versions = multi_set
.read_version_optimized(bucket, object, "", &ReadOptions::default())
.await
.expect("two metadata copies should satisfy the current set's read quorum");
assert_eq!(versions.len(), 2);
assert!(versions.iter().all(|version| version.name == object));
}
#[tokio::test]
async fn read_version_optimized_counts_only_valid_metadata_toward_quorum() {
let bucket = "read-version-corrupt-quorum-bucket";
let object = "object";
let (quorum_minus_one_dirs, quorum_minus_one) = read_version_quorum_test_set(bucket, object, 1, 2).await;
tokio::fs::write(
quorum_minus_one_dirs[1]
.path()
.join(bucket)
.join(object)
.join(crate::disk::STORAGE_FORMAT_FILE),
b"corrupt metadata",
)
.await
.expect("metadata should be corrupted for the test");
quorum_minus_one
.read_version_optimized(bucket, object, "", &ReadOptions::default())
.await
.expect_err("one valid and one corrupt metadata copy must not satisfy read quorum");
let (quorum_dirs, quorum) = read_version_quorum_test_set(bucket, object, 1, 3).await;
tokio::fs::write(
quorum_dirs[2]
.path()
.join(bucket)
.join(object)
.join(crate::disk::STORAGE_FORMAT_FILE),
b"corrupt metadata",
)
.await
.expect("metadata should be corrupted for the test");
let versions = quorum
.read_version_optimized(bucket, object, "", &ReadOptions::default())
.await
.expect("two valid metadata copies must satisfy read quorum despite one corrupt copy");
assert_eq!(versions.len(), 2);
}
#[tokio::test]
async fn get_object_info_and_quorum_maps_delete_marker_and_purge_states() {
let bucket = "get-object-info-marker-bucket";
@@ -13,7 +13,10 @@
// limitations under the License.
use super::*;
use crate::bucket::lifecycle::lifecycle;
use rustfs_filemeta::RestoreStatusOps;
use rustfs_utils::http::headers::{AMZ_RESTORE_EXPIRY_DAYS, AMZ_RESTORE_REQUEST_DATE};
use s3s::dto::{RestoreStatus, Timestamp};
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
struct RestoreCleanupIdentity {
@@ -43,6 +46,69 @@ impl RestoreCleanupIdentity {
}
impl SetDisks {
pub(super) async fn finalize_restore_metadata(
&self,
bucket: &str,
object: &str,
obj_info: &ObjectInfo,
opts: &ObjectOptions,
) -> Result<ObjectInfo> {
let expected = RestoreCleanupIdentity::from_object_info(obj_info);
let expected_operation_id = restore_operation_id_from_metadata(&opts.user_defined)?;
let expected_etag = obj_info
.etag
.clone()
.unwrap_or_else(|| get_raw_etag(obj_info.user_defined.as_ref()));
let version_id = expected.version_id.map(|v| v.to_string());
let _lock_guard = if !opts.no_lock {
Some(
self.acquire_write_lock_diag("restore_finalize_metadata", bucket, object)
.await?,
)
} else {
None
};
let read_opts = ObjectOptions {
version_id,
versioned: opts.versioned,
version_suspended: opts.version_suspended,
..Default::default()
};
let (mut fi, _, disks) = self
.get_object_fileinfo_gated(bucket, object, &read_opts, false, false)
.await?;
if let Some(expected_operation_id) = expected_operation_id {
require_restore_operation_id(&fi.metadata, expected_operation_id)?;
}
if !expected.matches_file_info(&fi, &expected_etag) {
return Err(Error::other("restored object changed before restore metadata finalization"));
}
let restore_expiry =
lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), opts.transition.restore_request.days.unwrap_or(1));
fi.metadata.insert(
X_AMZ_RESTORE.as_str().to_string(),
RestoreStatus {
is_restore_in_progress: Some(false),
restore_expiry_date: Some(Timestamp::from(restore_expiry)),
}
.to_string(),
);
self.invalidate_get_object_metadata_cache(bucket, object).await;
self.update_object_meta_with_opts(
bucket,
object,
fi.clone(),
disks.as_slice(),
&UpdateMetadataOpts {
replace_user_metadata: true,
..Default::default()
},
)
.await?;
self.invalidate_get_object_metadata_cache(bucket, object).await;
Ok(ObjectInfo::from_file_info(&fi, bucket, object, opts.versioned || opts.version_suspended))
}
pub async fn update_restore_metadata(
&self,
bucket: &str,
@@ -13,10 +13,11 @@
// limitations under the License.
use super::*;
use crate::bucket::lifecycle::lifecycle::{TRANSITION_COMPLETE, TRANSITION_PENDING, TransitionOptions};
use crate::bucket::lifecycle::lifecycle::{TRANSITION_COMPLETE, TRANSITION_PENDING, TransitionOptions, expected_expiry_time};
use crate::ecstore_validation_blackbox::make_local_set_disks;
use crate::services::tier::test_util::register_mock_tier;
use crate::storage_api_contracts::object::{ObjectIO as _, ObjectOperations as _};
use rustfs_filemeta::{RestoreStatusOps as _, parse_restore_obj_status};
use tokio::io::AsyncReadExt;
async fn prime_metadata_generation(set_disks: &SetDisks, bucket: &str, object: &str) -> GetObjectMetadataCacheKey {
@@ -83,13 +84,57 @@ async fn transition_and_restore_reclaim_prior_metadata_generations() {
let transitioned_generation = prime_metadata_generation(&set_disks, bucket, object).await;
let mut restore_opts = ObjectOptions::default();
restore_opts.transition.restore_request.days = Some(1);
Arc::clone(&set_disks)
.restore_transitioned_object(bucket, object, &restore_opts)
.await
.expect("restore should succeed");
let restore_started = OffsetDateTime::now_utc();
let expiry_from_restore_start = temp_env::async_with_vars(
[
("RUSTFS_ILM_DEBUG_DAY_SECS", Some("1")),
("RUSTFS_ILM_PROCESS_TIME", Some("1")),
],
async {
let expiry_from_restore_start = expected_expiry_time(restore_started, 1);
let get_barrier = backend.arm_get_barrier().await;
let restore_set = Arc::clone(&set_disks);
let restore =
tokio::spawn(async move { restore_set.restore_transitioned_object(bucket, object, &restore_opts).await });
get_barrier.wait_until_paused().await;
tokio::time::timeout(Duration::from_secs(5), async {
loop {
if expected_expiry_time(OffsetDateTime::now_utc(), 1) > expiry_from_restore_start {
break;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("test clock should cross the next accelerated lifecycle boundary");
get_barrier.release();
restore
.await
.expect("restore task should join")
.expect("restore should succeed");
expiry_from_restore_start
},
)
.await;
assert_generation_reclaimed(&set_disks, &transitioned_generation).await;
assert_eq!(backend.get_count().await, 1, "restore should read the remote candidate exactly once");
let restored_info = set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("restored object metadata should be readable");
let restore_status = parse_restore_obj_status(
restored_info
.user_defined
.get(s3s::header::X_AMZ_RESTORE.as_str())
.expect("completed restore header should be present"),
)
.expect("completed restore header should parse");
assert!(
restore_status.expiry().expect("completed restore should have an expiry") > expiry_from_restore_start,
"restore expiry must be based on completion, not the time the remote copy started"
);
let mut restored = Vec::new();
set_disks
.get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
+6 -2
View File
@@ -1311,14 +1311,16 @@ mod tests {
version_id: "version-a".to_string(),
tier_name: tier_a.to_string(),
backend_identity: Some(identity_a),
version_id_exact: false,
version_id_exact: true,
version_state: rustfs_filemeta::TransitionVersionState::Exact,
};
let entry_b = Jentry {
obj_name: "remote-b".to_string(),
version_id: "version-b".to_string(),
tier_name: tier_b.to_string(),
backend_identity: Some(identity_b),
version_id_exact: false,
version_id_exact: true,
version_state: rustfs_filemeta::TransitionVersionState::Exact,
};
let remove_a = backend_a.arm_failing_remove_barrier().await;
persist_tier_delete_journal_entry(store_a.clone(), &entry_a)
@@ -2720,10 +2722,12 @@ mod tests {
#[serial_test::serial(storage_class_env)]
async fn transition_transaction_recovery_deletes_provider_recovered_unknown_upload() {
let versioned_remote = uuid::Uuid::new_v4().to_string();
let nil_remote = uuid::Uuid::nil().to_string();
for (case, tier_name, remote_version) in [
("missing", "TXPROBEMISSING", None),
("unversioned", "TXPROBEUNVERSIONED", Some(String::new())),
("versioned", "TXPROBEVERSIONED", Some(versioned_remote)),
("nil-version", "TXPROBENILVERSION", Some(nil_remote)),
] {
let temp_dir = tempfile::tempdir().expect("create temp store dir");
let (ctx, store, _shutdown) = without_storage_class_env(build_isolated_test_store(
+13 -1
View File
@@ -6696,7 +6696,7 @@ mod test {
use crate::object_api::ObjectInfo;
use rustfs_filemeta::{
FileInfo, FileMeta, FileMetaVersion, MetaCacheEntries, MetaCacheEntriesSorted, MetaCacheEntry, MetaDeleteMarker,
VersionType,
ObjectPartInfo, VersionType,
};
use std::collections::{HashMap, HashSet};
use std::path::PathBuf;
@@ -6899,6 +6899,12 @@ mod test {
fi.volume = "bucket".to_owned();
fi.name = "object".to_owned();
fi.size = 1;
fi.parts = vec![ObjectPartInfo {
number: 1,
size: 1,
actual_size: 1,
..Default::default()
}];
fi.fresh = true;
fi.erasure.index = 1;
fi.mod_time = Some(time::OffsetDateTime::from_unix_timestamp(1_705_312_300).expect("valid timestamp"));
@@ -6954,6 +6960,12 @@ mod test {
fi.version_id = Some(Uuid::from_u128(version_idx));
fi.versioned = true;
fi.size = 1;
fi.parts = vec![ObjectPartInfo {
number: 1,
size: 1,
actual_size: 1,
..Default::default()
}];
fi.mod_time = Some(*mod_time);
fi.metadata = metadata;
+4 -4
View File
@@ -2601,10 +2601,10 @@ mod tests {
// (backlog#1304): restore entry no longer serializes on the object lock.
// The replacement semantics — non-blocking reads during the copy-back and
// fast rejection of a concurrent restore — are covered end-to-end by
// `restore_object_usecase_reports_ongoing_conflict_and_completion`
// (rustfs/src/app/lifecycle_transition_api_test.rs) and at the lock level
// by the accept-guard test below; restore-vs-reader data protection lives
// in the inner put_object/complete_multipart_upload commit locks.
// `restore_object_usecase_reports_ongoing_conflict`
// (rustfs/src/app/lifecycle_transition_api_test.rs), while the SetDisks
// transition matrix covers the final local commit. Restore-vs-reader data
// protection lives in the inner put_object/complete_multipart_upload locks.
#[tokio::test]
#[serial_test::serial]
async fn restore_accept_guard_serializes_concurrent_accepts() {
+1 -1
View File
@@ -20,5 +20,5 @@ Example:
```powershell
$env:RUSTFS_MINIO_FIXTURE_ROOT = '.\rustfs\tmp\minio-fixture-lab-local-key'
$env:RUSTFS_MINIO_STATIC_KMS_KEY_B64 = '<base64-32-byte-local-minio-kms-key>'
cargo +1.96.0 test -p rustfs-ecstore --features rio-v2 --test minio_generated_read_test -- --ignored
cargo +1.97.1 test -p rustfs-ecstore --features rio-v2 --test minio_generated_read_test -- --ignored
```
+78
View File
@@ -219,6 +219,16 @@ impl ErasureInfo {
}
// #[derive(Debug, Clone)]
#[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone, Copy, Default)]
#[serde(rename_all = "kebab-case")]
pub enum TransitionVersionState {
#[default]
Unknown,
KnownDisabled,
SuspendedNull,
Exact,
}
#[derive(Serialize, Deserialize, Debug, PartialEq, Clone, Default)]
pub struct FileInfo {
pub volume: String,
@@ -230,6 +240,10 @@ pub struct FileInfo {
pub transitioned_objname: String,
pub transition_tier: String,
pub transition_version_id: Option<Uuid>,
#[serde(default)]
pub transition_version: Option<String>,
#[serde(default)]
pub transition_version_state: TransitionVersionState,
pub expire_restored: bool,
pub data_dir: Option<Uuid>,
pub mod_time: Option<OffsetDateTime>,
@@ -415,6 +429,10 @@ impl FileInfo {
}
fn validate_collection_contents(&self, layout: Option<&ValidatedErasureLayout>) -> Result<()> {
if layout.is_some() && self.size > 0 && self.parts.is_empty() {
return Err(Error::FileCorrupt);
}
let mut part_numbers = [0u64; FILEINFO_PART_BITMAP_WORDS];
for part in &self.parts {
if let Some(layout) = layout {
@@ -455,6 +473,10 @@ impl FileInfo {
if self.mod_time.is_none_or(|mod_time| mod_time <= OffsetDateTime::UNIX_EPOCH)
|| (!allow_nil_version_id && self.version_id.is_some_and(|version_id| version_id.is_nil()))
|| self.transition_version_id.is_some_and(|version_id| version_id.is_nil())
|| self
.transition_version
.as_ref()
.is_some_and(|version_id| version_id.is_empty())
|| self.size != 0
|| self.data_dir.is_some()
|| self.mode.is_some()
@@ -488,6 +510,7 @@ impl FileInfo {
|| !self.transitioned_objname.is_empty()
|| !self.transition_tier.is_empty()
|| self.transition_version_id.is_some()
|| self.transition_version.is_some()
|| self.expire_restored
|| self.size != 0
|| self.data_dir.is_some()
@@ -532,6 +555,25 @@ impl FileInfo {
/// return `None`.
pub fn validate(&self, mode: ValidationMode) -> Result<Option<ValidatedErasureLayout>> {
self.validate_collection_bounds()?;
if let (Some(version), Some(version_id)) = (&self.transition_version, self.transition_version_id)
&& Uuid::parse_str(version).ok() != Some(version_id)
{
return Err(Error::FileCorrupt);
}
let transition_state_valid = match self.transition_version_state {
TransitionVersionState::Unknown => true,
TransitionVersionState::KnownDisabled => self.transition_version.is_none() && self.transition_version_id.is_none(),
TransitionVersionState::SuspendedNull => {
self.transition_version.as_deref() == Some("null") && self.transition_version_id.is_none()
}
TransitionVersionState::Exact => self
.transition_version
.as_deref()
.is_some_and(|version| version != "null" && !version.is_empty()),
};
if !transition_state_valid {
return Err(Error::FileCorrupt);
}
let erasure_layout = match mode {
ValidationMode::RequireErasure => Some(self.validate_erasure_geometry()?),
@@ -692,6 +734,10 @@ impl FileInfo {
// to_part_offset gets the part index where offset is located, returns part index and offset
pub fn to_part_offset(&self, offset: usize) -> Result<(usize, usize)> {
if self.size > 0 && self.parts.is_empty() {
return Err(Error::FileCorrupt);
}
if offset == 0 {
return Ok((0, 0));
}
@@ -824,6 +870,8 @@ impl FileInfo {
&& self.transition_tier == other.transition_tier
&& self.transitioned_objname == other.transitioned_objname
&& self.transition_version_id == other.transition_version_id
&& self.transition_version == other.transition_version
&& self.transition_version_state == other.transition_version_state
}
/// Check if metadata maps are equal
@@ -1185,6 +1233,21 @@ mod tests {
assert_eq!(layout.expect("strict layout must be present").data_blocks, 16);
}
#[test]
fn validate_require_erasure_rejects_positive_size_without_parts() {
let mut fi = FileInfo::new("bucket/object", 4, 2);
fi.erasure.index = 1;
fi.size = 1;
assert_eq!(fi.validate_for_metadata_read(), Err(Error::FileCorrupt));
assert_eq!(fi.to_part_offset(0), Err(Error::FileCorrupt));
fi.size = 0;
fi.validate_for_metadata_read()
.expect("zero-byte object metadata may omit parts");
assert_eq!(fi.to_part_offset(0), Ok((0, 0)));
}
#[test]
fn validate_for_erasure_write_only_relaxes_pending_shard_index() {
let mut fi = validation_test_fileinfo();
@@ -1328,6 +1391,15 @@ mod tests {
assert_file_corrupt(&fi, ValidationMode::DeleteOnly);
}
#[test]
fn metadata_read_validation_rejects_conflicting_transition_versions() {
let mut fi = one_shard_validation_fileinfo(1);
fi.transition_version_id = Some(Uuid::new_v4());
fi.transition_version = Some(Uuid::new_v4().to_string());
assert_file_corrupt(&fi, ValidationMode::RequireErasure);
}
#[test]
fn metadata_read_validation_requires_canonical_delete_marker_shape() {
let marker = FileInfo {
@@ -1699,6 +1771,12 @@ mod tests {
transitioned_objname,
transition_tier,
transition_version_id,
transition_version: transition_version_id.map(|version_id| version_id.to_string()),
transition_version_state: if transition_version_id.is_some() {
TransitionVersionState::Exact
} else {
TransitionVersionState::Unknown
},
expire_restored,
data_dir,
mod_time,
+117 -14
View File
@@ -182,14 +182,9 @@ impl FileMeta {
// TODO: use old buf
let meta_buf = ver.marshal_msg()?;
let pre_mod_time = self.versions[idx].header.mod_time;
self.versions[idx].header = ver.header();
self.versions[idx].meta = meta_buf;
if pre_mod_time != self.versions[idx].header.mod_time {
self.sort_by_mod_time();
}
self.sort_by_mod_time();
Ok(())
}
@@ -356,15 +351,10 @@ impl FileMeta {
return self.set_idx(fidx, version);
}
let mod_time = version.get_mod_time();
let new_shallow = FileMetaShallowVersion::try_from(version)?;
let insert_pos = match mod_time {
Some(nm) => self.versions.partition_point(|exist| match exist.header.mod_time {
Some(em) => em > nm,
None => false,
}),
None => self.versions.partition_point(|exist| exist.header.mod_time.is_some()),
};
let insert_pos = self
.versions
.partition_point(|existing| existing.header.sorts_before(&new_shallow.header));
self.versions.insert(insert_pos, new_shallow);
Ok(())
@@ -1112,6 +1102,119 @@ mod test {
assert!(fm.is_sorted_by_mod_time());
}
fn version_for_ordering(
version_type: VersionType,
version_id: Uuid,
mod_time: OffsetDateTime,
marker: u8,
) -> FileMetaVersion {
match version_type {
VersionType::Object => FileMetaVersion {
version_type,
object: Some(MetaObject {
version_id: Some(version_id),
erasure_algorithm: ErasureAlgo::ReedSolomon,
erasure_m: 2,
erasure_n: 2,
erasure_block_size: 1 << 20,
bitrot_checksum_algo: ChecksumAlgo::HighwayHash,
mod_time: Some(mod_time),
meta_sys: HashMap::from([("ordering-marker".to_string(), vec![marker])]),
..Default::default()
}),
..Default::default()
},
VersionType::Delete => FileMetaVersion {
version_type,
delete_marker: Some(MetaDeleteMarker {
version_id: Some(version_id),
mod_time: Some(mod_time),
meta_sys: HashMap::from([("ordering-marker".to_string(), vec![marker])]),
}),
..Default::default()
},
_ => unreachable!("ordering regression only constructs object and delete versions"),
}
}
#[test]
fn add_version_filemata_uses_canonical_equal_time_order() {
let mod_time = OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("valid test timestamp");
let object = version_for_ordering(VersionType::Object, Uuid::from_u128(1), mod_time, 1);
let delete_marker = version_for_ordering(VersionType::Delete, Uuid::from_u128(2), mod_time, 2);
for versions in [[object.clone(), delete_marker.clone()], [delete_marker, object]] {
let mut fm = FileMeta::new();
for version in versions {
fm.add_version_filemata(version).expect("add equal-time version");
}
assert_eq!(fm.versions[0].header.version_type, VersionType::Object);
assert_eq!(fm.versions[1].header.version_type, VersionType::Delete);
assert!(fm.versions[0].header.sorts_before(&fm.versions[1].header));
}
}
#[test]
fn add_version_filemata_preserves_tie_breaks_after_reload() {
let mod_time = OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("valid test timestamp");
let versions = [
version_for_ordering(VersionType::Object, Uuid::from_u128(10), mod_time, 10),
version_for_ordering(VersionType::Object, Uuid::from_u128(20), mod_time, 20),
version_for_ordering(VersionType::Delete, Uuid::from_u128(30), mod_time, 30),
];
let mut expected = versions.iter().map(FileMetaVersion::header).collect::<Vec<_>>();
expected.sort_by(|a, b| {
if a.sorts_before(b) {
Ordering::Less
} else if b.sorts_before(a) {
Ordering::Greater
} else {
Ordering::Equal
}
});
let mut fm = FileMeta::new();
for version in versions.into_iter().rev() {
fm.add_version_filemata(version).expect("add equal-time version");
}
let loaded = FileMeta::load(&fm.marshal_msg().expect("serialize file metadata")).expect("reload file metadata");
let actual = loaded
.versions
.iter()
.map(|version| version.header.clone())
.collect::<Vec<_>>();
assert_ne!(expected[0].signature, expected[1].signature, "fixture must exercise signature ordering");
assert_eq!(actual, expected);
assert!(actual.windows(2).all(|pair| pair[0].sorts_before(&pair[1])));
}
#[test]
fn add_version_filemata_reorders_equal_time_replacement() {
let mod_time = OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("valid test timestamp");
let first = version_for_ordering(VersionType::Object, Uuid::from_u128(40), mod_time, 40);
let second = version_for_ordering(VersionType::Object, Uuid::from_u128(50), mod_time, 50);
let (target, peer) = if first.header().sorts_before(&second.header()) {
(first, second)
} else {
(second, first)
};
let target_id = target.header().version_id.expect("test version id");
let mut fm = FileMeta::new();
fm.add_version_filemata(peer).expect("add peer object");
fm.add_version_filemata(target).expect("add target object");
assert_eq!(fm.versions[0].header.version_id, Some(target_id));
fm.add_version_filemata(version_for_ordering(VersionType::Delete, target_id, mod_time, 60))
.expect("replace target with equal-time delete marker");
assert_eq!(fm.versions[0].header.version_type, VersionType::Object);
assert_eq!(fm.versions[1].header.version_type, VersionType::Delete);
assert!(fm.versions[0].header.sorts_before(&fm.versions[1].header));
}
/// Regression for backlog#799 B16: replication reset state must be
/// persisted under both internal prefixes, never as a bare ARN. A bare-ARN
/// key (produced by `ObjectInfo::replication_state`) has no internal prefix,
+250 -41
View File
@@ -26,13 +26,14 @@ use super::msgp_decode::{
PrependByteReader, prealloc_hint, read_exact_vec, read_nil_or_array_len, read_nil_or_map_len, skip_msgp_value,
};
use super::*;
use crate::ChecksumInfo;
use crate::{ChecksumInfo, TransitionVersionState};
use rustfs_utils::HashAlgorithm;
use rustfs_utils::http::{
RUSTFS_INTERNAL_PREFIX, SUFFIX_CRC, SUFFIX_FREE_VERSION, SUFFIX_INLINE_DATA, SUFFIX_PURGESTATUS, SUFFIX_TIER_FV_ID,
SUFFIX_TIER_FV_MARKER, SUFFIX_TRANSITION_STATUS, SUFFIX_TRANSITION_TIER, SUFFIX_TRANSITION_TIER_DESTINATION_ID,
SUFFIX_TRANSITIONED_OBJECTNAME, SUFFIX_TRANSITIONED_VERSION_ID, contains_key_bytes, get_bytes, get_consistent_bytes, get_str,
has_internal_suffix, insert_bytes, is_internal_key, remove_bytes, strip_internal_prefix,
SUFFIX_TRANSITIONED_OBJECTNAME, SUFFIX_TRANSITIONED_VERSION_ID, SUFFIX_TRANSITIONED_VERSION_STATE, contains_key_bytes,
get_bytes, get_consistent_bytes, get_str, has_internal_suffix, insert_bytes, is_internal_key, remove_bytes,
strip_internal_prefix,
};
const MSGPACK_EXT8: u8 = 0xc7;
@@ -43,6 +44,7 @@ const MSGPACK_FIXEXT8: u8 = 0xd7;
const MSGPACK_TIME_EXT_LEGACY: i8 = 5;
const MSGPACK_TIME_EXT_OFFICIAL: i8 = -1;
const MSGPACK_TIME_LEN: u8 = 12;
const MAX_TRANSITION_VERSION_LEN: usize = 1024;
/// Sentinel signature returned when a version has no computable body (invalid /
/// missing inner object). Mirrors MinIO's `signatureErr` so such versions never
@@ -251,23 +253,93 @@ fn parse_legacy_uuid_bytes(bytes: &[u8], field: &str) -> Result<Option<Uuid>> {
/// Decode a stored transitioned-version-id from a version's `meta_sys`.
///
/// RustFS writes it as 16 raw UUID bytes; MinIO-migrated tiered objects store
/// the remote tier's version id as a UUID *string*. Accept both, and treat any
/// absent / nil / otherwise-unparseable value as "no tier version" (matching the
/// tolerant pre-hardening behavior) rather than failing the whole object read —
/// a malformed tier id must not make an otherwise-readable object unreadable.
fn transitioned_version_id_from_meta_sys(meta_sys: &HashMap<String, Vec<u8>>) -> Option<Uuid> {
let value = get_bytes(meta_sys, SUFFIX_TRANSITIONED_VERSION_ID)?;
/// Legacy RustFS writes used 16 raw UUID bytes. New writes and MinIO-migrated
/// records use the provider's exact UTF-8 version text. Empty, nil UUID, and
/// malformed bytes are not usable remote versions.
fn transitioned_version_from_meta_sys(meta_sys: &HashMap<String, Vec<u8>>) -> Result<Option<String>> {
if !contains_key_bytes(meta_sys, SUFFIX_TRANSITIONED_VERSION_ID) {
return Ok(None);
}
let Some(value) = get_consistent_bytes(meta_sys, SUFFIX_TRANSITIONED_VERSION_ID) else {
return Ok(None);
};
let value = value.to_vec();
if value.is_empty() {
return None;
return Ok(None);
}
if let Ok(id) = Uuid::from_slice(&value) {
return (!id.is_nil()).then_some(id);
return Ok((!id.is_nil()).then(|| id.to_string()));
}
std::str::from_utf8(&value)
let Ok(value) = String::from_utf8(value) else {
return Ok(None);
};
if value.is_empty()
|| value.len() > MAX_TRANSITION_VERSION_LEN
|| value.chars().any(char::is_control)
|| Uuid::parse_str(&value).is_ok_and(|id| id.is_nil())
{
Ok(None)
} else {
Ok(Some(value))
}
}
fn transition_version_state_from_meta_sys(
meta_sys: &HashMap<String, Vec<u8>>,
version: Option<&str>,
) -> Result<TransitionVersionState> {
if !contains_key_bytes(meta_sys, SUFFIX_TRANSITIONED_VERSION_STATE) {
return Ok(TransitionVersionState::Unknown);
}
let value = get_consistent_bytes(meta_sys, SUFFIX_TRANSITIONED_VERSION_STATE).ok_or(Error::FileCorrupt)?;
let state = match value {
b"known-disabled" => TransitionVersionState::KnownDisabled,
b"suspended-null" => TransitionVersionState::SuspendedNull,
b"exact" => TransitionVersionState::Exact,
b"unknown" => TransitionVersionState::Unknown,
_ => return Err(Error::FileCorrupt),
};
let valid = match state {
TransitionVersionState::Unknown | TransitionVersionState::KnownDisabled => version.is_none(),
TransitionVersionState::SuspendedNull => version == Some("null"),
TransitionVersionState::Exact => version.is_some_and(|value| value != "null"),
};
valid.then_some(state).ok_or(Error::FileCorrupt)
}
fn transition_version_state_bytes(state: TransitionVersionState) -> &'static [u8] {
match state {
TransitionVersionState::Unknown => b"unknown",
TransitionVersionState::KnownDisabled => b"known-disabled",
TransitionVersionState::SuspendedNull => b"suspended-null",
TransitionVersionState::Exact => b"exact",
}
}
fn set_transition_version_state(meta_sys: &mut HashMap<String, Vec<u8>>, state: TransitionVersionState) {
if state == TransitionVersionState::Unknown {
remove_bytes(meta_sys, SUFFIX_TRANSITIONED_VERSION_STATE);
} else {
insert_bytes(
meta_sys,
SUFFIX_TRANSITIONED_VERSION_STATE,
transition_version_state_bytes(state).to_vec(),
);
}
}
fn legacy_transitioned_version_id_from_meta_sys(meta_sys: &HashMap<String, Vec<u8>>) -> Option<Uuid> {
transitioned_version_from_meta_sys(meta_sys)
.ok()
.and_then(|s| Uuid::parse_str(s.trim()).ok())
.filter(|id| !id.is_nil())
.flatten()
.and_then(|value| Uuid::parse_str(&value).ok())
}
fn transitioned_version_bytes(fi: &FileInfo) -> Option<Vec<u8>> {
fi.transition_version
.as_ref()
.map(|version| version.as_bytes().to_vec())
.or_else(|| fi.transition_version_id.map(|version_id| version_id.as_bytes().to_vec()))
}
fn parse_legacy_erasure_algo(value: &str) -> ErasureAlgo {
@@ -2398,7 +2470,9 @@ impl MetaObject {
let transitioned_objname = get_bytes(&self.meta_sys, SUFFIX_TRANSITIONED_OBJECTNAME)
.map(|v| String::from_utf8_lossy(&v).to_string())
.unwrap_or_default();
let transition_version_id = transitioned_version_id_from_meta_sys(&self.meta_sys);
let transition_version = transitioned_version_from_meta_sys(&self.meta_sys)?;
let transition_version_state = transition_version_state_from_meta_sys(&self.meta_sys, transition_version.as_deref())?;
let transition_version_id = transition_version.as_deref().and_then(|value| Uuid::parse_str(value).ok());
let transition_tier = get_bytes(&self.meta_sys, SUFFIX_TRANSITION_TIER)
.map(|v| String::from_utf8_lossy(&v).to_string())
.unwrap_or_default();
@@ -2419,6 +2493,8 @@ impl MetaObject {
transition_status,
transitioned_objname,
transition_version_id,
transition_version,
transition_version_state,
transition_tier,
..Default::default()
})
@@ -2431,13 +2507,12 @@ impl MetaObject {
SUFFIX_TRANSITIONED_OBJECTNAME,
fi.transitioned_objname.as_bytes().to_vec(),
);
if let Some(transition_version_id) = fi.transition_version_id.as_ref() {
insert_bytes(
&mut self.meta_sys,
SUFFIX_TRANSITIONED_VERSION_ID,
transition_version_id.as_bytes().to_vec(),
);
if let Some(transition_version) = transitioned_version_bytes(fi) {
insert_bytes(&mut self.meta_sys, SUFFIX_TRANSITIONED_VERSION_ID, transition_version);
} else {
remove_bytes(&mut self.meta_sys, SUFFIX_TRANSITIONED_VERSION_ID);
}
set_transition_version_state(&mut self.meta_sys, fi.transition_version_state);
insert_bytes(&mut self.meta_sys, SUFFIX_TRANSITION_TIER, fi.transition_tier.as_bytes().to_vec());
if let Some(destination_id) = get_str(&fi.metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID) {
insert_bytes(&mut self.meta_sys, SUFFIX_TRANSITION_TIER_DESTINATION_ID, destination_id.into_bytes());
@@ -2501,6 +2576,7 @@ impl MetaObject {
SUFFIX_TRANSITION_TIER,
SUFFIX_TRANSITIONED_OBJECTNAME,
SUFFIX_TRANSITIONED_VERSION_ID,
SUFFIX_TRANSITIONED_VERSION_STATE,
] {
if let Some(v) = get_bytes(&self.meta_sys, suffix) {
insert_bytes(&mut delete_marker.meta_sys, suffix, v);
@@ -2562,8 +2638,11 @@ impl From<FileInfo> for MetaObject {
);
}
if let Some(vid) = &value.transition_version_id {
insert_bytes(&mut meta_sys, SUFFIX_TRANSITIONED_VERSION_ID, vid.as_bytes().to_vec());
if let Some(transition_version) = transitioned_version_bytes(&value) {
insert_bytes(&mut meta_sys, SUFFIX_TRANSITIONED_VERSION_ID, transition_version);
}
if !value.transition_status.is_empty() {
set_transition_version_state(&mut meta_sys, value.transition_version_state);
}
if !value.transition_tier.is_empty() {
@@ -2706,7 +2785,11 @@ impl MetaDeleteMarker {
.map(|v| String::from_utf8_lossy(&v).to_string())
.unwrap_or_default();
fi.transition_version_id = transitioned_version_id_from_meta_sys(&self.meta_sys);
fi.transition_version = transitioned_version_from_meta_sys(&self.meta_sys).ok().flatten();
fi.transition_version_id = legacy_transitioned_version_id_from_meta_sys(&self.meta_sys);
fi.transition_version_state =
transition_version_state_from_meta_sys(&self.meta_sys, fi.transition_version.as_deref())
.unwrap_or(TransitionVersionState::Unknown);
}
fi
@@ -2859,8 +2942,11 @@ impl From<FileInfo> for MetaDeleteMarker {
value.transitioned_objname.as_bytes().to_vec(),
);
}
if let Some(version_id) = value.transition_version_id {
insert_bytes(&mut meta_sys, SUFFIX_TRANSITIONED_VERSION_ID, version_id.as_bytes().to_vec());
if let Some(transition_version) = transitioned_version_bytes(&value) {
insert_bytes(&mut meta_sys, SUFFIX_TRANSITIONED_VERSION_ID, transition_version);
}
if !value.transition_status.is_empty() || value.tier_free_version() {
set_transition_version_state(&mut meta_sys, value.transition_version_state);
}
if !value.transition_tier.is_empty() {
insert_bytes(&mut meta_sys, SUFFIX_TRANSITION_TIER, value.transition_tier.as_bytes().to_vec());
@@ -3412,7 +3498,7 @@ mod tests {
.insert("x-rustfs-internal-healing".to_string(), "true".to_string());
marker.metadata.insert("content-type".to_string(), "text/plain".to_string());
let remote_version_id = Uuid::new_v4();
marker.transition_version_id = Some(remote_version_id);
marker.transition_version = Some(remote_version_id.to_string());
let converted = MetaDeleteMarker::from(marker);
@@ -3420,7 +3506,19 @@ mod tests {
assert_eq!(converted.meta_sys.get("x-minio-internal-purgestatus"), Some(&b"pending".to_vec()));
assert_eq!(
get_bytes(&converted.meta_sys, SUFFIX_TRANSITIONED_VERSION_ID),
Some(remote_version_id.as_bytes().to_vec())
Some(remote_version_id.to_string().into_bytes())
);
assert_eq!(
converted
.meta_sys
.get(&format!("{RUSTFS_INTERNAL_PREFIX}{SUFFIX_TRANSITIONED_VERSION_ID}")),
Some(&remote_version_id.to_string().into_bytes())
);
assert_eq!(
converted
.meta_sys
.get(&format!("{}{SUFFIX_TRANSITIONED_VERSION_ID}", rustfs_utils::http::MINIO_INTERNAL_PREFIX)),
Some(&remote_version_id.to_string().into_bytes())
);
assert!(!converted.meta_sys.contains_key("x-rustfs-internal-healing"));
assert!(!converted.meta_sys.contains_key("content-type"));
@@ -4097,19 +4195,129 @@ mod tests {
.into_fileinfo("b", "k", false)
.expect("into_fileinfo");
assert_eq!(fi.transition_version_id, Some(id));
assert_eq!(fi.transition_version, Some(id.to_string()));
assert_eq!(fi.transition_version_state, TransitionVersionState::Unknown);
}
#[test]
fn meta_object_transition_version_id_unparseable_stays_readable_as_none() {
// A non-UUID / non-16-byte tier version id must NOT make the object
// unreadable; it is tolerated as "no tier version" (compat with
// pre-hardening behavior and foreign/edge metadata).
fn meta_object_transition_version_id_opaque_text_is_preserved() {
let mut sys = HashMap::new();
insert_bytes(&mut sys, SUFFIX_TRANSITIONED_VERSION_ID, b"not-a-uuid".to_vec());
insert_bytes(&mut sys, SUFFIX_TRANSITIONED_VERSION_ID, b"opaque-generation-42".to_vec());
let fi = make_meta_object_with_sys(sys)
.into_fileinfo("b", "k", false)
.expect("unparseable transition version id must not fail the object read");
.expect("opaque transition version id must decode");
assert_eq!(fi.transition_version_id, None);
assert_eq!(fi.transition_version.as_deref(), Some("opaque-generation-42"));
assert_eq!(fi.transition_version_state, TransitionVersionState::Unknown);
}
#[test]
fn meta_object_transition_version_state_exact_round_trips_dual_keys() {
let id = sample_version_id();
let expected_version = id.to_string();
let fi = FileInfo {
transition_status: "complete".to_string(),
transition_version: Some(expected_version.clone()),
transition_version_state: TransitionVersionState::Exact,
..Default::default()
};
let object = MetaObject::from(fi);
assert_eq!(
object
.meta_sys
.get(&format!("{RUSTFS_INTERNAL_PREFIX}{SUFFIX_TRANSITIONED_VERSION_STATE}"))
.map(Vec::as_slice),
Some(b"exact".as_slice())
);
assert_eq!(
object
.meta_sys
.get(&format!(
"{}{SUFFIX_TRANSITIONED_VERSION_STATE}",
rustfs_utils::http::MINIO_INTERNAL_PREFIX
))
.map(Vec::as_slice),
Some(b"exact".as_slice())
);
assert_eq!(
legacy_transitioned_version_id_from_meta_sys(&object.meta_sys),
Some(id),
"UUID exact writes must remain readable by the legacy UUID consumer"
);
let decoded = object.into_fileinfo("b", "k", false).expect("exact state should round trip");
assert_eq!(decoded.transition_version_state, TransitionVersionState::Exact);
assert_eq!(decoded.transition_version.as_deref(), Some(expected_version.as_str()));
}
#[test]
fn set_transition_known_disabled_removes_stale_version_dual_keys() {
let mut meta_sys = HashMap::new();
insert_bytes(&mut meta_sys, SUFFIX_TRANSITIONED_VERSION_ID, b"stale-legacy-version".to_vec());
let mut object = make_meta_object_with_sys(meta_sys);
object.set_transition(&FileInfo {
transition_status: TRANSITION_COMPLETE.to_string(),
transitioned_objname: "remote/object".to_string(),
transition_version_state: TransitionVersionState::KnownDisabled,
transition_tier: "WARM".to_string(),
..Default::default()
});
assert_eq!(get_bytes(&object.meta_sys, SUFFIX_TRANSITIONED_VERSION_ID), None);
assert!(
!object
.meta_sys
.contains_key(&format!("{RUSTFS_INTERNAL_PREFIX}{SUFFIX_TRANSITIONED_VERSION_ID}"))
);
assert!(
!object
.meta_sys
.contains_key(&format!("{}{SUFFIX_TRANSITIONED_VERSION_ID}", rustfs_utils::http::MINIO_INTERNAL_PREFIX))
);
let decoded = object
.into_fileinfo("b", "k", false)
.expect("known-disabled transition must remain readable after replacing stale metadata");
assert_eq!(decoded.transition_version, None);
assert_eq!(decoded.transition_version_state, TransitionVersionState::KnownDisabled);
}
#[test]
fn meta_object_transition_version_state_conflict_fails_closed() {
let mut sys = HashMap::new();
insert_bytes(&mut sys, SUFFIX_TRANSITIONED_VERSION_ID, sample_version_id().as_bytes().to_vec());
sys.insert(format!("{RUSTFS_INTERNAL_PREFIX}{SUFFIX_TRANSITIONED_VERSION_STATE}"), b"exact".to_vec());
sys.insert(
format!("{}{SUFFIX_TRANSITIONED_VERSION_STATE}", rustfs_utils::http::MINIO_INTERNAL_PREFIX),
b"known-disabled".to_vec(),
);
make_meta_object_with_sys(sys)
.into_fileinfo("b", "k", false)
.expect_err("conflicting state keys must fail closed");
}
#[test]
fn meta_object_transition_version_id_invalid_utf8_yields_none() {
let mut sys = HashMap::new();
insert_bytes(&mut sys, SUFFIX_TRANSITIONED_VERSION_ID, vec![0xff]);
let fi = make_meta_object_with_sys(sys)
.into_fileinfo("b", "k", false)
.expect("invalid transition version bytes must not fail the object read");
assert_eq!(fi.transition_version_id, None);
assert_eq!(fi.transition_version, None);
}
#[test]
fn meta_object_transition_version_id_unsafe_text_yields_none() {
for value in [b"opaque\0version".to_vec(), vec![b'x'; MAX_TRANSITION_VERSION_LEN + 1]] {
let mut sys = HashMap::new();
insert_bytes(&mut sys, SUFFIX_TRANSITIONED_VERSION_ID, value);
let fi = make_meta_object_with_sys(sys)
.into_fileinfo("b", "k", false)
.expect("unsafe transition version text must not fail the object read");
assert_eq!(fi.transition_version_id, None);
assert_eq!(fi.transition_version, None);
}
}
#[test]
@@ -4123,6 +4331,7 @@ mod tests {
.into_fileinfo("b", "k", false)
.expect("string-form transition version id must decode");
assert_eq!(fi.transition_version_id, Some(id));
assert_eq!(fi.transition_version, Some(id.to_string()));
}
#[test]
@@ -4152,16 +4361,14 @@ mod tests {
}
.into_fileinfo("b", "k", false);
assert_eq!(fi.transition_version_id, Some(id));
assert_eq!(fi.transition_version, Some(id.to_string()));
}
#[test]
fn delete_marker_free_version_transition_version_id_unparseable_stays_readable() {
// A malformed tier version id must not make a free-version record corrupt:
// it decodes to None and stays readable. Otherwise free-version expiry
// fails and the remote-tier object leaks.
fn delete_marker_free_version_transition_version_id_opaque_text_is_preserved() {
let mut sys = HashMap::new();
insert_bytes(&mut sys, SUFFIX_FREE_VERSION, vec![]);
insert_bytes(&mut sys, SUFFIX_TRANSITIONED_VERSION_ID, b"not-a-uuid".to_vec());
insert_bytes(&mut sys, SUFFIX_TRANSITIONED_VERSION_ID, b"opaque-generation-42".to_vec());
insert_bytes(&mut sys, SUFFIX_TRANSITION_TIER, b"WARM".to_vec());
insert_bytes(&mut sys, SUFFIX_TRANSITIONED_OBJECTNAME, b"remote-object".to_vec());
let fi = MetaDeleteMarker {
@@ -4172,8 +4379,9 @@ mod tests {
.into_fileinfo("b", "k", false);
assert_eq!(fi.transition_version_id, None);
assert_eq!(fi.transition_version.as_deref(), Some("opaque-generation-42"));
fi.validate_for_metadata_read()
.expect("free-version record with an unparseable tier id must remain readable");
.expect("free-version record with an opaque tier id must remain readable");
}
#[test]
@@ -4193,6 +4401,7 @@ mod tests {
.into_fileinfo("b", "k", false);
assert_eq!(fi.transition_version_id, Some(id));
assert_eq!(fi.transition_version, Some(id.to_string()));
}
#[test]
+20
View File
@@ -144,6 +144,7 @@ fn is_recoverable_heal_error_message(error: &str) -> bool {
"connection refused",
"operation canceled",
"quorum not reached",
"heal rename incomplete",
]
.iter()
.any(|pattern| error.contains(pattern))
@@ -154,3 +155,22 @@ impl From<Error> for std::io::Error {
std::io::Error::other(err)
}
}
#[cfg(test)]
mod tests {
use super::Error;
use crate::heal::EcstoreError;
#[test]
fn incomplete_target_rename_is_recoverable() {
let task_error = Error::TaskExecutionFailed {
message: "heal rename incomplete: 1 of 2 targets committed".to_string(),
};
let storage_error = Error::Storage(EcstoreError::Io(std::io::Error::other(
"heal rename incomplete: 1 of 2 targets committed",
)));
assert!(task_error.is_recoverable_heal());
assert!(storage_error.is_recoverable_heal());
}
}
+8
View File
@@ -161,6 +161,14 @@ pub struct ConfigureStaticKmsRequest {
pub allow_insecure_dev_defaults: Option<bool>,
}
impl Drop for ConfigureStaticKmsRequest {
fn drop(&mut self) {
use zeroize::Zeroize;
self.secret_key.zeroize();
}
}
impl fmt::Debug for ConfigureStaticKmsRequest {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("ConfigureStaticKmsRequest")
+156 -24
View File
@@ -23,16 +23,17 @@
use crate::backends::{BackendInfo, KmsBackend, KmsClient};
use crate::config::{BackendConfig, KmsConfig};
use crate::encryption::DataKeyEnvelope;
use crate::error::{KmsError, Result};
use crate::types::*;
use aes_gcm::{
Aes256Gcm, Key, Nonce,
aead::{Aead, KeyInit},
aead::{Aead, KeyInit, Payload},
};
use async_trait::async_trait;
use jiff::Zoned;
use rand::RngExt;
use std::collections::HashMap;
use std::collections::{BTreeMap, HashMap};
use tracing::debug;
use zeroize::Zeroizing;
@@ -41,6 +42,11 @@ const NONCE_SIZE: usize = 12;
/// AES-256 key size in bytes.
const KEY_SIZE: usize = 32;
fn context_aad(context: &HashMap<String, String>) -> Result<Vec<u8>> {
let canonical: BTreeMap<&str, &str> = context.iter().map(|(key, value)| (key.as_str(), value.as_str())).collect();
serde_json::to_vec(&canonical).map_err(Into::into)
}
/// Static single-key KMS backend.
///
/// Uses a pre-configured AES-256 key to derive data encryption keys. This is a
@@ -111,21 +117,35 @@ impl KmsClient for StaticKmsBackend {
let key = Key::<Aes256Gcm>::from(*self.key);
let cipher = Aes256Gcm::new(&key);
let nonce = Nonce::from(nonce_bytes);
let aad = context_aad(&request.encryption_context)?;
let encrypted = cipher
.encrypt(&nonce, plaintext.as_ref())
.encrypt(
&nonce,
Payload {
msg: plaintext.as_ref(),
aad: &aad,
},
)
.map_err(|e| KmsError::cryptographic_error("AES-256-GCM encrypt", e.to_string()))?;
// Ciphertext format: encrypted_dek || nonce
let mut ciphertext = encrypted;
ciphertext.extend_from_slice(&nonce_bytes);
let envelope = DataKeyEnvelope {
key_id: uuid::Uuid::new_v4().to_string(),
master_key_id: request.master_key_id.clone(),
key_spec: request.key_spec.clone(),
encrypted_key: encrypted,
nonce: nonce_bytes.to_vec(),
encryption_context: request.encryption_context.clone(),
created_at: Zoned::now(),
};
let ciphertext = serde_json::to_vec(&envelope)?;
Ok(DataKeyInfo::new(
self.key_id.clone(),
0, // version is always 0 for static KMS
0,
Some(plaintext.to_vec()),
ciphertext,
"AES_256".to_string(),
request.key_spec.clone(),
))
}
@@ -141,14 +161,28 @@ impl KmsClient for StaticKmsBackend {
let key = Key::<Aes256Gcm>::from(*self.key);
let cipher = Aes256Gcm::new(&key);
let nonce = Nonce::from(nonce_bytes);
let aad = context_aad(&request.encryption_context)?;
let encrypted = cipher
.encrypt(&nonce, request.plaintext.as_ref())
.encrypt(
&nonce,
Payload {
msg: request.plaintext.as_ref(),
aad: &aad,
},
)
.map_err(|e| KmsError::cryptographic_error("AES-256-GCM encrypt", e.to_string()))?;
// Ciphertext format: encrypted_data || nonce
let mut ciphertext = encrypted;
ciphertext.extend_from_slice(&nonce_bytes);
let envelope = DataKeyEnvelope {
key_id: uuid::Uuid::new_v4().to_string(),
master_key_id: request.key_id.clone(),
key_spec: "AES_256".to_string(),
encrypted_key: encrypted,
nonce: nonce_bytes.to_vec(),
encryption_context: request.encryption_context.clone(),
created_at: Zoned::now(),
};
let ciphertext = serde_json::to_vec(&envelope)?;
Ok(EncryptResponse {
ciphertext,
@@ -159,21 +193,39 @@ impl KmsClient for StaticKmsBackend {
}
async fn decrypt(&self, request: &DecryptRequest, _context: Option<&OperationContext>) -> Result<Vec<u8>> {
if request.ciphertext.len() < NONCE_SIZE + 1 {
return Err(KmsError::cryptographic_error("decrypt", "Ciphertext too short for static KMS format"));
let envelope: DataKeyEnvelope = serde_json::from_slice(&request.ciphertext)
.map_err(|error| KmsError::cryptographic_error("parse", format!("Failed to parse data key envelope: {error}")))?;
if envelope.master_key_id != self.key_id {
return Err(KmsError::key_not_found(&envelope.master_key_id));
}
// Split ciphertext: encrypted_data || nonce(12)
let split_at = request.ciphertext.len() - NONCE_SIZE;
let encrypted = &request.ciphertext[..split_at];
let nonce_slice = &request.ciphertext[split_at..];
for (key, expected_value) in &envelope.encryption_context {
match request.encryption_context.get(key) {
Some(actual_value) if actual_value == expected_value => {}
Some(actual_value) => {
return Err(KmsError::context_mismatch(format!(
"Context mismatch for key '{key}': expected '{expected_value}', got '{actual_value}'"
)));
}
None if request.encryption_context.is_empty() => {}
None => return Err(KmsError::context_mismatch(format!("Missing context key '{key}'"))),
}
}
let key = Key::<Aes256Gcm>::from(*self.key);
let cipher = Aes256Gcm::new(&key);
let nonce = Nonce::try_from(nonce_slice).map_err(|_| KmsError::cryptographic_error("nonce", "invalid nonce length"))?;
let nonce = Nonce::try_from(envelope.nonce.as_slice())
.map_err(|_| KmsError::cryptographic_error("nonce", "invalid nonce length"))?;
let aad = context_aad(&envelope.encryption_context)?;
let plaintext = cipher
.decrypt(&nonce, encrypted)
.decrypt(
&nonce,
Payload {
msg: envelope.encrypted_key.as_ref(),
aad: &aad,
},
)
.map_err(|e| KmsError::cryptographic_error("AES-256-GCM decrypt", e.to_string()))?;
Ok(plaintext)
@@ -317,7 +369,13 @@ impl KmsBackend for StaticKmsBackend {
}
async fn generate_data_key(&self, request: GenerateDataKeyRequest) -> Result<GenerateDataKeyResponse> {
let gen_req = GenerateKeyRequest::new(request.key_id.clone(), request.key_spec.as_str().to_string());
let gen_req = GenerateKeyRequest {
master_key_id: request.key_id.clone(),
key_spec: request.key_spec.as_str().to_string(),
key_length: None,
encryption_context: request.encryption_context,
grant_tokens: Vec::new(),
};
let data_key = <Self as KmsClient>::generate_data_key(self, &gen_req, None).await?;
let plaintext_key = data_key
@@ -378,8 +436,9 @@ impl KmsBackend for StaticKmsBackend {
#[cfg(test)]
mod tests {
use super::*;
use crate::backends::KmsClient;
use crate::backends::{KmsBackend as KmsBackendTrait, KmsClient};
use crate::config::{BackendConfig, KmsBackend, StaticConfig};
use crate::encryption::is_data_key_envelope;
use base64::Engine as _;
use base64::engine::general_purpose::STANDARD as BASE64;
@@ -430,8 +489,12 @@ mod tests {
assert_eq!(data_key.version, 0);
assert!(data_key.plaintext.is_some());
assert_eq!(data_key.plaintext.as_ref().expect("plaintext should be set").len(), 32);
// Ciphertext should be: encrypted(32) + tag(16) + nonce(12)
assert_eq!(data_key.ciphertext.len(), 32 + 16 + NONCE_SIZE);
let envelope: DataKeyEnvelope =
serde_json::from_slice(&data_key.ciphertext).expect("static data key should use a KMS envelope");
assert_eq!(envelope.master_key_id, key_id);
assert_eq!(envelope.encrypted_key.len(), 32 + 16);
assert_eq!(envelope.nonce.len(), NONCE_SIZE);
assert_eq!(envelope.encryption_context.get("bucket").map(String::as_str), Some("test-bucket"));
// Decrypt the data key
let decrypt_request =
@@ -443,6 +506,75 @@ mod tests {
assert_eq!(decrypted.as_slice(), data_key.plaintext.as_deref().expect("plaintext should exist"));
}
#[tokio::test]
async fn generated_data_key_uses_kms_envelope_for_sse_read_routing() {
let (backend, key_id, _key) = create_test_backend().await;
let request = GenerateKeyRequest::new(key_id, "AES_256".to_string())
.with_context("bucket".to_string(), "source-bucket".to_string())
.with_context("object".to_string(), "source-object".to_string());
let data_key = KmsClient::generate_data_key(&backend, &request, None)
.await
.expect("generate static KMS data key");
assert!(
is_data_key_envelope(&data_key.ciphertext),
"static KMS ciphertext must use the KMS envelope recognized by the SSE read path"
);
}
#[tokio::test]
async fn generated_data_key_rejects_a_different_encryption_context() {
let (backend, key_id, _key) = create_test_backend().await;
let generate_request = GenerateDataKeyRequest {
key_id,
key_spec: KeySpec::Aes256,
encryption_context: HashMap::from([
("bucket".to_string(), "source-bucket".to_string()),
("object".to_string(), "source-object".to_string()),
]),
};
let generated = KmsBackendTrait::generate_data_key(&backend, generate_request)
.await
.expect("generate context-bound static KMS data key");
let decrypt_request = DecryptRequest {
ciphertext: generated.ciphertext_blob,
encryption_context: HashMap::from([
("bucket".to_string(), "different-bucket".to_string()),
("object".to_string(), "different-object".to_string()),
]),
grant_tokens: Vec::new(),
};
let error = KmsBackendTrait::decrypt(&backend, decrypt_request)
.await
.expect_err("a static KMS data key must not decrypt under a different object context");
assert!(matches!(error, KmsError::ContextMismatch { .. }));
}
#[tokio::test]
async fn generated_data_key_rejects_tampered_envelope_context() {
let (backend, key_id, _key) = create_test_backend().await;
let request = GenerateKeyRequest::new(key_id, "AES_256".to_string())
.with_context("bucket".to_string(), "source-bucket".to_string());
let generated = KmsClient::generate_data_key(&backend, &request, None)
.await
.expect("generate context-bound data key");
let mut envelope: DataKeyEnvelope = serde_json::from_slice(&generated.ciphertext).expect("parse static KMS envelope");
envelope
.encryption_context
.insert("bucket".to_string(), "different-bucket".to_string());
let decrypt_request = DecryptRequest::new(serde_json::to_vec(&envelope).expect("serialize tampered envelope"))
.with_context("bucket".to_string(), "different-bucket".to_string());
let error = KmsClient::decrypt(&backend, &decrypt_request, None)
.await
.expect_err("tampering with authenticated envelope context must fail");
assert!(matches!(error, KmsError::CryptographicError { .. }));
}
#[tokio::test]
async fn test_generate_data_key_wrong_key_id() {
let (backend, _key_id, _key) = create_test_backend().await;
+24
View File
@@ -214,9 +214,18 @@ pub struct StaticConfig {
/// Key identifier (name) for the single configured key
pub key_id: String,
/// Base64-encoded 32-byte AES-256 key material (zeroed on drop)
#[serde(skip_serializing, default)]
pub secret_key: String,
}
impl Drop for StaticConfig {
fn drop(&mut self) {
use zeroize::Zeroize;
self.secret_key.zeroize();
}
}
impl fmt::Debug for StaticConfig {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("StaticConfig")
@@ -1054,6 +1063,21 @@ mod tests {
assert!(serialized.contains("persisted-token-secret"));
}
#[test]
fn static_kms_config_serialization_does_not_expose_key_material() {
use base64::Engine as _;
let encoded_key = base64::engine::general_purpose::STANDARD.encode([0x5au8; 32]);
let config = KmsConfig::static_kms("static-key".to_string(), encoded_key.clone());
let serialized = serde_json::to_string(&config).expect("static KMS config should serialize");
assert!(
!serialized.contains(&encoded_key),
"persisted static KMS configuration must not contain plaintext key material"
);
}
#[test]
fn test_config_validation() {
let mut config = KmsConfig {
+16 -10
View File
@@ -31,19 +31,25 @@ use tokio::sync::RwLock;
pub struct KmsManager {
backend: Arc<dyn KmsBackend>,
cache: Arc<RwLock<KmsCache>>,
config: KmsConfig,
default_key_id: Option<String>,
enable_cache: bool,
}
impl KmsManager {
/// Create a new KMS manager with the given backend and config
pub fn new(backend: Arc<dyn KmsBackend>, config: KmsConfig) -> Self {
let cache = Arc::new(RwLock::new(KmsCache::new(config.cache_config.max_keys as u64)));
Self { backend, cache, config }
Self {
backend,
cache,
default_key_id: config.default_key_id,
enable_cache: config.enable_cache,
}
}
/// Get the default key ID if configured
pub fn get_default_key_id(&self) -> Option<&String> {
self.config.default_key_id.as_ref()
self.default_key_id.as_ref()
}
/// Create a new master key
@@ -51,7 +57,7 @@ impl KmsManager {
let response = self.backend.create_key(request).await?;
// Cache the key metadata if enabled
if self.config.enable_cache {
if self.enable_cache {
let mut cache = self.cache.write().await;
cache.put_key_metadata(&response.key_id, &response.key_metadata).await;
}
@@ -77,7 +83,7 @@ impl KmsManager {
/// Describe a key
pub async fn describe_key(&self, request: DescribeKeyRequest) -> Result<DescribeKeyResponse> {
// Check cache first if enabled
if self.config.enable_cache {
if self.enable_cache {
let cache = self.cache.read().await;
if let Some(cached_metadata) = cache.get_key_metadata(&request.key_id).await {
return Ok(DescribeKeyResponse {
@@ -89,7 +95,7 @@ impl KmsManager {
// Get from backend and cache
let response = self.backend.describe_key(request).await?;
if self.config.enable_cache {
if self.enable_cache {
let mut cache = self.cache.write().await;
cache
.put_key_metadata(&response.key_metadata.key_id, &response.key_metadata)
@@ -106,7 +112,7 @@ impl KmsManager {
/// Get cache statistics
pub async fn cache_stats(&self) -> Option<(u64, u64)> {
if self.config.enable_cache {
if self.enable_cache {
let cache = self.cache.read().await;
Some(cache.stats())
} else {
@@ -116,7 +122,7 @@ impl KmsManager {
/// Clear the cache
pub async fn clear_cache(&self) -> Result<()> {
if self.config.enable_cache {
if self.enable_cache {
let mut cache = self.cache.write().await;
cache.clear().await;
}
@@ -128,7 +134,7 @@ impl KmsManager {
let response = self.backend.delete_key(request).await?;
// Remove from cache if enabled and key is being deleted
if self.config.enable_cache {
if self.enable_cache {
let mut cache = self.cache.write().await;
cache.remove_key_metadata(&response.key_id).await;
}
@@ -141,7 +147,7 @@ impl KmsManager {
let response = self.backend.cancel_key_deletion(request).await?;
// Update cache if enabled
if self.config.enable_cache {
if self.enable_cache {
let mut cache = self.cache.write().await;
cache.put_key_metadata(&response.key_id, &response.key_metadata).await;
}
+2 -10
View File
@@ -350,11 +350,7 @@ impl ObjectEncryptionService {
encryption_context: context.clone(),
};
let data_key = self
.kms_manager
.generate_data_key(request)
.await
.map_err(|e| KmsError::backend_error(format!("Failed to generate data key: {e}")))?;
let data_key = self.kms_manager.generate_data_key(request).await?;
let plaintext_key = data_key.plaintext_key;
@@ -431,11 +427,7 @@ impl ObjectEncryptionService {
grant_tokens: Vec::new(),
};
let decrypt_response = self
.kms_manager
.decrypt(decrypt_request)
.await
.map_err(|e| KmsError::backend_error(format!("Failed to decrypt data key: {e}")))?;
let decrypt_response = self.kms_manager.decrypt(decrypt_request).await?;
// Create cipher
let cipher = create_cipher(&algorithm, &decrypt_response.plaintext)?;
+28
View File
@@ -94,6 +94,16 @@ impl KmsServiceManager {
self.config.read().await.clone()
}
/// Get configuration for status and management responses without static key material.
pub async fn get_redacted_config(&self) -> Option<KmsConfig> {
let mut config = self.config.read().await.clone()?;
if let BackendConfig::Static(static_config) = &mut config.backend_config {
use zeroize::Zeroize;
static_config.secret_key.zeroize();
}
Some(config)
}
/// Configure KMS with new configuration
pub async fn configure(&self, new_config: KmsConfig) -> Result<()> {
new_config.validate()?;
@@ -449,4 +459,22 @@ mod tests {
assert_eq!(manager.get_status().await, KmsServiceStatus::NotConfigured);
assert!(manager.get_config().await.is_none());
}
#[tokio::test]
async fn redacted_config_omits_static_key_material() {
use base64::Engine as _;
let manager = KmsServiceManager::new();
let encoded_key = base64::engine::general_purpose::STANDARD.encode([0x5au8; 32]);
manager
.configure(KmsConfig::static_kms("static-key".to_string(), encoded_key))
.await
.expect("configure static KMS");
let config = manager.get_redacted_config().await.expect("redacted config");
let BackendConfig::Static(static_config) = config.backend_config else {
panic!("expected static config");
};
assert!(static_config.secret_key.is_empty());
}
}
+2
View File
@@ -27,6 +27,7 @@ documentation = "https://docs.rs/rustfs-lifecycle/latest/rustfs_lifecycle/"
[dependencies]
async-trait.workspace = true
metrics.workspace = true
rustfs-common.workspace = true
rustfs-config = { workspace = true, features = ["constants"] }
rustfs-replication.workspace = true
@@ -38,6 +39,7 @@ url.workspace = true
uuid = { workspace = true, features = ["v4", "serde", "fast-rng", "macro-diagnostics"] }
[dev-dependencies]
metrics-util = { version = "0.20", features = ["debugging"] }
proptest = "1"
serial_test.workspace = true
temp-env.workspace = true
+115 -20
View File
@@ -34,6 +34,7 @@ const LOG_SUBSYSTEM_LIFECYCLE: &str = "lifecycle";
const EVENT_LIFECYCLE_EXPIRY_COMPUTED: &str = "lifecycle_expiry_computed";
const EVENT_LIFECYCLE_DEBUG_DAY_SECS: &str = "lifecycle_debug_day_secs";
const EVENT_LIFECYCLE_NONCURRENT_EXPIRY_SKIPPED: &str = "lifecycle_noncurrent_expiry_skipped";
const METRIC_LIFECYCLE_INVALID_MOD_TIME_TOTAL: &str = "rustfs_lifecycle_invalid_mod_time_total";
const ERR_LIFECYCLE_NO_RULE: &str = "Lifecycle configuration should have at least one rule";
const ERR_LIFECYCLE_DUPLICATE_ID: &str = "Rule ID must be unique. Found same ID for more than one rule";
const _ERR_XML_NOT_WELL_FORMED: &str =
@@ -204,6 +205,21 @@ fn lifecycle_rule_prefix(rule: &LifecycleRule) -> Option<&str> {
filter.and.as_ref().and_then(|and| and.prefix.as_deref())
}
fn lifecycle_mod_time(obj: &ObjectOpts) -> Option<OffsetDateTime> {
let reason = match obj.mod_time {
None => "missing",
Some(mod_time) if mod_time < OffsetDateTime::UNIX_EPOCH => "before_unix_epoch",
Some(mod_time) if mod_time == OffsetDateTime::UNIX_EPOCH => "unix_epoch",
Some(mod_time) => return Some(mod_time),
};
metrics::counter!(
METRIC_LIFECYCLE_INVALID_MOD_TIME_TOTAL,
"reason" => reason
)
.increment(1);
None
}
#[async_trait::async_trait]
pub trait Lifecycle {
async fn has_transition(&self) -> bool;
@@ -434,14 +450,8 @@ impl Lifecycle for BucketLifecycleConfiguration {
}
async fn predict_expiration(&self, obj: &ObjectOpts) -> Event {
let mod_time = match obj.mod_time {
Some(time) => {
if time.unix_timestamp() == 0 {
return Event::default();
}
time
}
None => return Event::default(),
let Some(mod_time) = lifecycle_mod_time(obj) else {
return Event::default();
};
if obj.delete_marker || !(obj.is_latest || obj.version_id.is_none_or(|v| v.is_nil())) {
@@ -495,19 +505,9 @@ impl Lifecycle for BucketLifecycleConfiguration {
obj.name, obj.mod_time, obj.successor_mod_time, now, obj.is_latest, obj.delete_marker
);
// Gracefully handle missing mod_time instead of panicking
let mod_time = match obj.mod_time {
Some(t) => t,
None => {
debug!("eval_inner: mod_time is None for object={}, returning default event", obj.name);
return Event::default();
}
};
if mod_time.unix_timestamp() == 0 {
debug!("eval_inner: mod_time is 0, returning default event");
let Some(mod_time) = lifecycle_mod_time(obj) else {
return Event::default();
}
};
if let Some(restore_expires) = obj.restore_expires
&& restore_expires.unix_timestamp() != 0
@@ -1082,6 +1082,8 @@ impl Default for TransitionOptions {
#[cfg(test)]
mod tests {
use super::*;
use metrics_util::MetricKind;
use metrics_util::debugging::{DebugValue, DebuggingRecorder};
use s3s::dto::{LifecycleRuleFilter, TransitionStorageClass};
use serial_test::serial;
use std::sync::Arc;
@@ -1097,6 +1099,99 @@ mod tests {
});
}
#[test]
fn eval_inner_reports_invalid_mod_time_without_expiring_object() {
let lifecycle = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
days: Some(1),
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("expire-after-one-day".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
}],
};
let recorder = DebuggingRecorder::new();
let snapshotter = recorder.snapshotter();
let runtime = tokio::runtime::Builder::new_current_thread()
.build()
.expect("current-thread runtime should build");
let actions = metrics::with_local_recorder(&recorder, || {
runtime.block_on(async {
let now = OffsetDateTime::from_unix_timestamp(10 * 86_400).expect("fixed timestamp should be valid");
let mut opts = ObjectOpts {
name: "legacy-object".to_string(),
is_latest: true,
..Default::default()
};
let mut actions = Vec::new();
for mod_time in [
None,
Some(OffsetDateTime::UNIX_EPOCH),
Some(OffsetDateTime::UNIX_EPOCH - Duration::nanoseconds(1)),
Some(OffsetDateTime::UNIX_EPOCH + Duration::nanoseconds(1)),
] {
opts.mod_time = mod_time;
actions.push((
lifecycle.eval_inner(&opts, now, 0).await.action,
lifecycle.predict_expiration(&opts).await.action,
));
}
actions
})
});
assert_eq!(
actions,
[
(IlmAction::NoneAction, IlmAction::NoneAction),
(IlmAction::NoneAction, IlmAction::NoneAction),
(IlmAction::NoneAction, IlmAction::NoneAction),
(IlmAction::DeleteAction, IlmAction::DeleteAction)
]
);
let invalid_mod_time_metrics = snapshotter
.snapshot()
.into_vec()
.into_iter()
.filter_map(|(composite, _unit, _description, value)| {
(composite.kind() == MetricKind::Counter && composite.key().name() == METRIC_LIFECYCLE_INVALID_MOD_TIME_TOTAL)
.then(|| {
let reason = composite
.key()
.labels()
.find(|label| label.key() == "reason")
.map(|label| label.value().to_string())
.expect("invalid mod_time metric should have a reason");
(reason, value)
})
})
.collect::<Vec<_>>();
assert_eq!(invalid_mod_time_metrics.len(), 3);
assert!(
invalid_mod_time_metrics
.iter()
.all(|(_, value)| matches!(value, DebugValue::Counter(2)))
);
assert!(invalid_mod_time_metrics.iter().any(|(reason, _)| reason == "missing"));
assert!(invalid_mod_time_metrics.iter().any(|(reason, _)| reason == "unix_epoch"));
assert!(
invalid_mod_time_metrics
.iter()
.any(|(reason, _)| reason == "before_unix_epoch")
);
}
#[tokio::test]
#[serial]
async fn validate_rejects_zero_expiration_days() {
+4 -4
View File
@@ -45,15 +45,15 @@ pub(super) fn rules() -> Vec<Rule> {
)
},
Rule {
anchors: strings(["all healed data rename attempts failed"]),
anchors: strings(["heal rename incomplete"]),
..base(
"heal-rename-failed",
P1Unavailable,
"heal",
"heal 数据落位失败",
contains("all healed data rename attempts failed"),
"heal 数据落位(rename)全部失败",
"检查盘写权限与空间",
contains("heal rename incomplete"),
"heal 数据落位(rename)未在全部目标盘完成",
"检查失败目标盘的写权限与空间,修复后重跑 heal",
)
},
Rule {
+1 -1
View File
@@ -177,7 +177,7 @@ fn every_rule_has_a_positive_sample() {
msg("heal: latest metadata for b/o has no data_dir, cannot heal object data"),
),
("heal-all-writes-failed", msg("all drives had write errors, unable to heal b/o")),
("heal-rename-failed", msg("all healed data rename attempts failed for b/o")),
("heal-rename-failed", msg("heal rename incomplete: 1 of 2 targets committed for b/o")),
(
"heal-xlmeta-regen-failed",
msg("heal_object: failed to regenerate recoverable xl.meta on disk"),
+15
View File
@@ -25,6 +25,9 @@ keywords = ["file-system", "notification", "real-time", "rustfs", "Minio"]
categories = ["web-programming", "development-tools", "filesystem"]
documentation = "https://docs.rs/rustfs-notify/latest/rustfs_notify/"
[features]
demo-examples = []
[dependencies]
rustfs-config = { workspace = true, features = ["notify", "server-config-model"] }
rustfs-ecstore = { workspace = true }
@@ -71,6 +74,18 @@ workspace = true
[lib]
doctest = false
[[example]]
name = "full_demo"
required-features = ["demo-examples"]
[[example]]
name = "full_demo_one"
required-features = ["demo-examples"]
[[example]]
name = "webhook"
required-features = ["demo-examples"]
[[bench]]
name = "snapshot_mode_scan"
harness = false
+5
View File
@@ -49,6 +49,8 @@ swift = [
"dep:hmac",
"dep:sha1",
"dep:hex",
"dep:ipnetwork",
"dep:rustfs-trusted-proxies",
"dep:astral-tokio-tar",
"dep:base64",
"dep:async-compression",
@@ -111,6 +113,8 @@ quick-xml = { workspace = true, optional = true, features = ["serialize"] }
hmac = { workspace = true, optional = true }
sha1 = { workspace = true, optional = true }
hex = { workspace = true, optional = true }
ipnetwork = { workspace = true, optional = true }
rustfs-trusted-proxies = { workspace = true, optional = true }
astral-tokio-tar = { workspace = true, optional = true }
base64 = { workspace = true, optional = true }
async-compression = { workspace = true, optional = true, features = ["tokio", "gzip", "bzip2"] }
@@ -130,6 +134,7 @@ socket2 = { workspace = true, optional = true, features = ["all"] }
[dev-dependencies]
tempfile = { workspace = true }
proptest = "1"
rustfs-test-utils = { workspace = true }
tracing-subscriber = { workspace = true, features = ["env-filter", "time"] }
tokio = { workspace = true, features = ["test-util", "macros", "fs"] }
+41 -51
View File
@@ -16,12 +16,11 @@
use super::storage_api::account::{BucketOperations, MakeBucketOptions};
use super::{SwiftError, SwiftResult};
use super::{get_swift_bucket_metadata, resolve_swift_object_store_handle, set_swift_bucket_metadata};
use super::{get_swift_bucket_metadata, resolve_swift_object_store_handle, update_swift_bucket_tagging, validate_metadata};
use rustfs_credentials::Credentials;
use s3s::dto::{Tag, Tagging};
use sha2::{Digest, Sha256};
use std::collections::HashMap;
use time;
/// Validate that the authenticated user has access to the requested account
///
@@ -148,15 +147,33 @@ pub async fn get_account_metadata(account: &str, _credentials: &Option<Credentia
/// Updates account-level metadata such as TempURL keys.
/// Only updates swift-account-meta-* tags, preserving other tags.
///
/// The caller must own the account: this metadata holds the account's TempURL
/// signing key, so writing it for someone else's account would let the writer
/// mint valid pre-signed URLs against that account's objects. Reads
/// ([`get_account_metadata`]) stay unauthenticated on purpose — TempURL
/// signature validation has to resolve the key before any credentials exist.
///
/// # Arguments
/// * `account` - Account identifier
/// * `metadata` - Metadata key-value pairs to store (keys will be prefixed with `swift-account-meta-`)
/// * `credentials` - S3 credentials
/// * `credentials` - Keystone credentials of the caller
pub async fn update_account_metadata(
account: &str,
metadata: &HashMap<String, String>,
_credentials: &Option<Credentials>,
credentials: &Option<Credentials>,
) -> SwiftResult<()> {
let Some(credentials) = credentials.as_ref() else {
return Err(SwiftError::Unauthorized(
"Keystone authentication required to update account metadata".to_string(),
));
};
validate_account_access(account, credentials)?;
// These tags are persisted into the bucket metadata file, which every
// later config write rewrites in full — so unbounded metadata inflates
// the cost of unrelated writes for the life of the account.
validate_metadata(metadata)?;
let bucket_name = get_account_metadata_bucket_name(account);
let Some(store) = resolve_swift_object_store_handle() else {
@@ -173,57 +190,30 @@ pub async fn update_account_metadata(
.map_err(|e| SwiftError::InternalServerError(format!("Failed to create account metadata bucket: {}", e)))?;
}
// Load current bucket metadata
let bucket_meta = get_swift_bucket_metadata(&bucket_name)
.await
.map_err(|e| SwiftError::InternalServerError(format!("Failed to load bucket metadata: {}", e)))?;
// Rewrite the persisted tags: replace swift-account-meta-* tags with the
// new metadata while preserving other tags. An empty result clears the
// tagging config.
update_swift_bucket_tagging(bucket_name, |current| {
let mut tagging = current.cloned().unwrap_or_else(|| Tagging { tag_set: vec![] });
let mut bucket_meta_clone = (*bucket_meta).clone();
// Get existing tags, preserving non-Swift tags
let mut existing_tagging = bucket_meta_clone
.tagging_config
.clone()
.unwrap_or_else(|| Tagging { tag_set: vec![] });
// Remove old swift-account-meta-* tags while preserving other tags
existing_tagging.tag_set.retain(|tag| {
if let Some(key) = &tag.key {
!key.starts_with("swift-account-meta-")
} else {
true
}
});
// Add new metadata tags
for (key, value) in metadata {
existing_tagging.tag_set.push(Tag {
key: Some(format!("swift-account-meta-{}", key)),
value: Some(value.clone()),
tagging.tag_set.retain(|tag| {
if let Some(key) = &tag.key {
!key.starts_with("swift-account-meta-")
} else {
true
}
});
}
let now = time::OffsetDateTime::now_utc();
for (key, value) in metadata {
tagging.tag_set.push(Tag {
key: Some(format!("swift-account-meta-{}", key)),
value: Some(value.clone()),
});
}
if existing_tagging.tag_set.is_empty() {
// No tags remain; clear tagging config
bucket_meta_clone.tagging_config_xml = Vec::new();
bucket_meta_clone.tagging_config_updated_at = now;
bucket_meta_clone.tagging_config = None;
} else {
// Serialize tags to XML
let tagging_xml = quick_xml::se::to_string(&existing_tagging)
.map_err(|e| SwiftError::InternalServerError(format!("Failed to serialize tags: {}", e)))?;
bucket_meta_clone.tagging_config_xml = tagging_xml.into_bytes();
bucket_meta_clone.tagging_config_updated_at = now;
bucket_meta_clone.tagging_config = Some(existing_tagging);
}
// Save updated metadata
set_swift_bucket_metadata(bucket_name.clone(), bucket_meta_clone)
.await
.map_err(|e| SwiftError::InternalServerError(format!("Failed to save metadata: {}", e)))?;
tagging
})
.await?;
Ok(())
}
+184 -226
View File
@@ -22,12 +22,22 @@ use super::storage_api::container::{
};
use super::types::Container;
use super::{SwiftError, SwiftResult};
use super::{get_swift_bucket_metadata, resolve_swift_object_store_handle, set_swift_bucket_metadata};
use super::{
get_swift_bucket_metadata, get_swift_bucket_usage, resolve_swift_object_store_handle, update_swift_bucket_tagging,
validate_metadata,
};
use rustfs_credentials::Credentials;
use s3s::dto::{Tag, Tagging};
use sha2::{Digest, Sha256};
use tracing::{debug, error};
fn required_bucket_usage(usage: &std::collections::HashMap<String, (u64, u64)>, bucket: &str) -> SwiftResult<(u64, u64)> {
usage
.get(bucket)
.copied()
.ok_or_else(|| SwiftError::ServiceUnavailable("Container usage is unavailable".to_string()))
}
const LOG_COMPONENT_PROTOCOLS: &str = "protocols";
const LOG_SUBSYSTEM_SWIFT_CONTAINER: &str = "swift_container";
const EVENT_SWIFT_CONTAINER_STORAGE_STATE: &str = "swift_container_storage_state";
@@ -252,13 +262,24 @@ pub async fn list_containers(account: &str, credentials: &Credentials) -> SwiftR
.list_bucket(&BucketOptions::default())
.await
.map_err(|e| sanitize_storage_error("Container listing", e))?;
let usage = get_swift_bucket_usage()
.await
.map_err(|e| sanitize_storage_error("Container usage retrieval", e))?
.ok_or_else(|| SwiftError::ServiceUnavailable("Container usage is unavailable".to_string()))?;
// Filter and convert buckets to containers
let containers: Vec<Container> = bucket_infos
.iter()
.filter(|info| mapper.bucket_belongs_to_project(&info.name, &project_id))
.filter_map(|info| bucket_info_to_container(info, &mapper, &project_id))
.collect();
.filter_map(|info| {
bucket_info_to_container(info, &mapper, &project_id).map(|mut container| {
let (count, bytes) = required_bucket_usage(&usage, &info.name)?;
container.count = count;
container.bytes = bytes;
Ok(container)
})
})
.collect::<SwiftResult<Vec<_>>>()?;
debug!(
event = EVENT_SWIFT_CONTAINER_STORAGE_STATE,
@@ -368,6 +389,44 @@ pub struct ContainerMetadata {
pub custom_metadata: std::collections::HashMap<String, String>,
}
async fn get_container_metadata_base(
account: &str,
container: &str,
credentials: &Credentials,
) -> SwiftResult<(String, BucketInfo, std::collections::HashMap<String, String>)> {
let project_id = validate_account_access(account, credentials)?;
validate_container_name(container)?;
let bucket_name = ContainerMapper::default().swift_to_s3_bucket(container, &project_id);
let Some(store) = resolve_swift_object_store_handle() else {
return Err(SwiftError::InternalServerError("Storage layer not initialized".to_string()));
};
let bucket_info = store
.get_bucket_info(&bucket_name, &BucketOptions::default())
.await
.map_err(|e| {
if e.to_string().contains("not found") || e.to_string().contains("NoSuchBucket") {
SwiftError::NotFound(format!("Container '{}' not found", container))
} else {
sanitize_storage_error("Container metadata retrieval", e)
}
})?;
let custom_metadata = get_swift_bucket_metadata(&bucket_name)
.await
.ok()
.and_then(|bucket_meta| bucket_meta.tagging_config.as_ref().map(s3_tags_to_swift_metadata))
.unwrap_or_default();
Ok((bucket_name, bucket_info, custom_metadata))
}
pub(crate) async fn get_container_custom_metadata(
account: &str,
container: &str,
credentials: &Credentials,
) -> SwiftResult<std::collections::HashMap<String, String>> {
let (_, _, custom_metadata) = get_container_metadata_base(account, container, credentials).await?;
Ok(custom_metadata)
}
/// Get container metadata (for HEAD operation)
///
/// This function:
@@ -382,60 +441,22 @@ pub struct ContainerMetadata {
/// - Returns 404 Not Found if container doesn't exist
#[allow(dead_code)] // Used by handler
pub async fn get_container_metadata(account: &str, container: &str, credentials: &Credentials) -> SwiftResult<ContainerMetadata> {
// Validate account access and extract project_id
let project_id = validate_account_access(account, credentials)?;
let (bucket_name, bucket_info, custom_metadata) = get_container_metadata_base(account, container, credentials).await?;
// Validate container name
validate_container_name(container)?;
// Create mapper with default config (tenant prefixing enabled)
let mapper = ContainerMapper::default();
// Convert Swift container name to S3 bucket name
let bucket_name = mapper.swift_to_s3_bucket(container, &project_id);
// Get storage layer
let Some(store) = resolve_swift_object_store_handle() else {
return Err(SwiftError::InternalServerError("Storage layer not initialized".to_string()));
};
// Get bucket info
let bucket_info = store
.get_bucket_info(&bucket_name, &BucketOptions::default())
// Ecstore serves the persisted scanner snapshot through a bounded in-process cache.
// Single-node deployments overlay successful in-process mutations; distributed
// deployments use only the cluster-wide persisted snapshot.
let usage = get_swift_bucket_usage()
.await
.map_err(|e| {
// Check if bucket not found
if e.to_string().contains("not found") || e.to_string().contains("NoSuchBucket") {
SwiftError::NotFound(format!("Container '{}' not found", container))
} else {
sanitize_storage_error("Container metadata retrieval", e)
}
})?;
.map_err(|e| sanitize_storage_error("Container usage retrieval", e))?
.ok_or_else(|| SwiftError::ServiceUnavailable("Container usage is unavailable".to_string()))?;
let (object_count, bytes_used) = required_bucket_usage(&usage, &bucket_name)?;
// Load bucket metadata to get custom metadata from tags
let custom_metadata = match get_swift_bucket_metadata(&bucket_name).await {
Ok(bucket_meta) => {
if let Some(tagging) = &bucket_meta.tagging_config {
s3_tags_to_swift_metadata(tagging)
} else {
std::collections::HashMap::new()
}
}
Err(_) => {
// If metadata not available, return empty (container may be newly created)
std::collections::HashMap::new()
}
};
// Currently returns basic metadata with limitations:
// 1. Object count requires iterating all objects (expensive)
// 2. Bytes used requires summing all object sizes (expensive)
// 3. Custom metadata is now loaded from bucket tags ✅
Ok(ContainerMetadata {
object_count: 0, // TODO: implement object counting in backend
bytes_used: 0, // TODO: implement size aggregation in backend
object_count,
bytes_used,
created: bucket_info.created,
custom_metadata, // ✅ Now populated from bucket tags!
custom_metadata,
})
}
@@ -465,6 +486,11 @@ pub async fn update_container_metadata(
// Validate container name
validate_container_name(container)?;
// These tags are persisted into the bucket metadata file, which every
// later config write rewrites in full — so unbounded metadata inflates
// the cost of unrelated writes for the life of the container.
validate_metadata(&metadata)?;
// Create mapper with default config (tenant prefixing enabled)
let mapper = ContainerMapper::default();
@@ -488,57 +514,30 @@ pub async fn update_container_metadata(
}
})?;
// Load current bucket metadata
let bucket_meta = get_swift_bucket_metadata(&bucket_name)
.await
.map_err(|e| SwiftError::InternalServerError(format!("Failed to load bucket metadata: {}", e)))?;
// Rewrite the persisted tags: replace swift-meta-* tags with the new
// metadata while preserving non-Swift tags. An empty result clears the
// tagging config.
update_swift_bucket_tagging(bucket_name, |current| {
let mut tagging = current.cloned().unwrap_or_else(|| Tagging { tag_set: vec![] });
let mut bucket_meta_clone = (*bucket_meta).clone();
tagging.tag_set.retain(|tag| {
if let Some(key) = &tag.key {
!key.starts_with("swift-meta-")
} else {
true // Keep tags with no key (shouldn't happen, but be safe)
}
});
// Get existing tags, preserving non-Swift tags
let mut existing_tagging = bucket_meta_clone
.tagging_config
.clone()
.unwrap_or_else(|| Tagging { tag_set: vec![] });
// Remove old swift-meta-* tags while preserving other tags
existing_tagging.tag_set.retain(|tag| {
if let Some(key) = &tag.key {
!key.starts_with("swift-meta-")
} else {
true // Keep tags with no key (shouldn't happen, but be safe)
if let Some(mut new_tagging) = swift_metadata_to_s3_tags(&metadata) {
tagging.tag_set.append(&mut new_tagging.tag_set);
}
});
// If metadata.is_empty() and swift_metadata_to_s3_tags returns None,
// we've already removed swift-meta-* tags above, so only non-Swift
// tags remain
// Add new Swift metadata tags if provided
if let Some(mut new_tagging) = swift_metadata_to_s3_tags(&metadata) {
// Merge: existing non-Swift tags + new Swift tags
existing_tagging.tag_set.append(&mut new_tagging.tag_set);
}
// If metadata.is_empty() and swift_metadata_to_s3_tags returns None,
// we've already removed swift-meta-* tags above, so only non-Swift tags remain
let now = time::OffsetDateTime::now_utc();
if existing_tagging.tag_set.is_empty() {
// No tags remain after removing swift-meta-* tags; clear tagging config
bucket_meta_clone.tagging_config_xml = Vec::new();
bucket_meta_clone.tagging_config_updated_at = now;
bucket_meta_clone.tagging_config = None;
} else {
// Serialize the merged tags to XML
let tagging_xml = quick_xml::se::to_string(&existing_tagging)
.map_err(|e| SwiftError::InternalServerError(format!("Failed to serialize tags: {}", e)))?;
bucket_meta_clone.tagging_config_xml = tagging_xml.into_bytes();
bucket_meta_clone.tagging_config_updated_at = now;
bucket_meta_clone.tagging_config = Some(existing_tagging);
}
// Save updated metadata
set_swift_bucket_metadata(bucket_name, bucket_meta_clone)
.await
.map_err(|e| SwiftError::InternalServerError(format!("Failed to save metadata: {}", e)))?;
tagging
})
.await?;
Ok(())
}
@@ -801,44 +800,23 @@ pub async fn enable_versioning(
}
})?;
// Load current bucket metadata
let bucket_meta = get_swift_bucket_metadata(&bucket_name)
.await
.map_err(|e| SwiftError::InternalServerError(format!("Failed to load bucket metadata: {}", e)))?;
// Rewrite the persisted tags: replace any versioning tag with the new
// archive location while preserving all other tags.
update_swift_bucket_tagging(bucket_name, |current| {
let mut tagging = current.cloned().unwrap_or_else(|| Tagging { tag_set: vec![] });
let mut bucket_meta_clone = (*bucket_meta).clone();
tagging
.tag_set
.retain(|tag| tag.key.as_deref() != Some("swift-versions-location"));
// Get existing tags
let mut existing_tagging = bucket_meta_clone
.tagging_config
.clone()
.unwrap_or_else(|| Tagging { tag_set: vec![] });
tagging.tag_set.push(Tag {
key: Some("swift-versions-location".to_string()),
value: Some(archive_container.to_string()), // Store Swift container name, not S3 bucket name
});
// Remove old versioning tag if present
existing_tagging
.tag_set
.retain(|tag| tag.key.as_deref() != Some("swift-versions-location"));
// Add new versioning tag
existing_tagging.tag_set.push(Tag {
key: Some("swift-versions-location".to_string()),
value: Some(archive_container.to_string()), // Store Swift container name, not S3 bucket name
});
let now = time::OffsetDateTime::now_utc();
// Serialize tags to XML
let tagging_xml = quick_xml::se::to_string(&existing_tagging)
.map_err(|e| SwiftError::InternalServerError(format!("Failed to serialize tags: {}", e)))?;
bucket_meta_clone.tagging_config_xml = tagging_xml.into_bytes();
bucket_meta_clone.tagging_config_updated_at = now;
bucket_meta_clone.tagging_config = Some(existing_tagging);
// Save updated metadata
set_swift_bucket_metadata(bucket_name, bucket_meta_clone)
.await
.map_err(|e| SwiftError::InternalServerError(format!("Failed to save metadata: {}", e)))?;
tagging
})
.await?;
Ok(())
}
@@ -864,50 +842,38 @@ pub async fn disable_versioning(account: &str, container: &str, credentials: &Cr
let mapper = ContainerMapper::default();
let bucket_name = mapper.swift_to_s3_bucket(container, &project_id);
// Verify container exists
let Some(_store) = resolve_swift_object_store_handle() else {
let Some(store) = resolve_swift_object_store_handle() else {
return Err(SwiftError::InternalServerError("Storage layer not initialized".to_string()));
};
// Load current bucket metadata
let bucket_meta = get_swift_bucket_metadata(&bucket_name)
// Verify container exists. Without this the rewrite below would persist a
// fabricated default for a container that does not exist: the metadata
// loader turns "no metadata on disk" into a fresh BucketMetadata, and
// writing that creates an orphan .metadata.bin and caches a fabricated
// default as authoritative.
store
.get_bucket_info(&bucket_name, &BucketOptions::default())
.await
.map_err(|e| SwiftError::InternalServerError(format!("Failed to load bucket metadata: {}", e)))?;
.map_err(|e| {
if e.to_string().contains("not found") || e.to_string().contains("NoSuchBucket") {
SwiftError::NotFound(format!("Container '{}' not found", container))
} else {
sanitize_storage_error("Container verification", e)
}
})?;
let mut bucket_meta_clone = (*bucket_meta).clone();
// Rewrite the persisted tags: drop the versioning tag while preserving
// all other tags. An empty result clears the tagging config.
update_swift_bucket_tagging(bucket_name, |current| {
let mut tagging = current.cloned().unwrap_or_else(|| Tagging { tag_set: vec![] });
// Get existing tags
let mut existing_tagging = bucket_meta_clone
.tagging_config
.clone()
.unwrap_or_else(|| Tagging { tag_set: vec![] });
tagging
.tag_set
.retain(|tag| tag.key.as_deref() != Some("swift-versions-location"));
// Remove versioning tag
existing_tagging
.tag_set
.retain(|tag| tag.key.as_deref() != Some("swift-versions-location"));
let now = time::OffsetDateTime::now_utc();
if existing_tagging.tag_set.is_empty() {
// No tags remain; clear tagging config
bucket_meta_clone.tagging_config_xml = Vec::new();
bucket_meta_clone.tagging_config_updated_at = now;
bucket_meta_clone.tagging_config = None;
} else {
// Serialize remaining tags to XML
let tagging_xml = quick_xml::se::to_string(&existing_tagging)
.map_err(|e| SwiftError::InternalServerError(format!("Failed to serialize tags: {}", e)))?;
bucket_meta_clone.tagging_config_xml = tagging_xml.into_bytes();
bucket_meta_clone.tagging_config_updated_at = now;
bucket_meta_clone.tagging_config = Some(existing_tagging);
}
// Save updated metadata
set_swift_bucket_metadata(bucket_name, bucket_meta_clone)
.await
.map_err(|e| SwiftError::InternalServerError(format!("Failed to save metadata: {}", e)))?;
tagging
})
.await?;
Ok(())
}
@@ -1032,65 +998,37 @@ pub async fn set_container_acl(
}
})?;
// Load current bucket metadata
let bucket_meta = get_swift_bucket_metadata(&bucket_name)
.await
.map_err(|e| SwiftError::InternalServerError(format!("Failed to load bucket metadata: {}", e)))?;
// Rewrite the persisted tags: replace the ACL tags with the new grants
// while preserving all other tags. An empty result clears the tagging
// config.
update_swift_bucket_tagging(bucket_name, |current| {
let mut tagging = current.cloned().unwrap_or_else(|| Tagging { tag_set: vec![] });
let mut bucket_meta_clone = (*bucket_meta).clone();
tagging
.tag_set
.retain(|tag| tag.key.as_deref() != Some("swift-acl-read") && tag.key.as_deref() != Some("swift-acl-write"));
// Get existing tags
let mut existing_tagging = bucket_meta_clone
.tagging_config
.clone()
.unwrap_or_else(|| Tagging { tag_set: vec![] });
if let Some(read) = read_acl
&& !read.trim().is_empty()
{
tagging.tag_set.push(Tag {
key: Some("swift-acl-read".to_string()),
value: Some(read.to_string()),
});
}
// Remove old ACL tags
existing_tagging
.tag_set
.retain(|tag| tag.key.as_deref() != Some("swift-acl-read") && tag.key.as_deref() != Some("swift-acl-write"));
if let Some(write) = write_acl
&& !write.trim().is_empty()
{
tagging.tag_set.push(Tag {
key: Some("swift-acl-write".to_string()),
value: Some(write.to_string()),
});
}
// Add new read ACL tag if provided
if let Some(read) = read_acl
&& !read.trim().is_empty()
{
existing_tagging.tag_set.push(Tag {
key: Some("swift-acl-read".to_string()),
value: Some(read.to_string()),
});
}
// Add new write ACL tag if provided
if let Some(write) = write_acl
&& !write.trim().is_empty()
{
existing_tagging.tag_set.push(Tag {
key: Some("swift-acl-write".to_string()),
value: Some(write.to_string()),
});
}
let now = time::OffsetDateTime::now_utc();
if existing_tagging.tag_set.is_empty() {
// No tags remain; clear tagging config
bucket_meta_clone.tagging_config_xml = Vec::new();
bucket_meta_clone.tagging_config_updated_at = now;
bucket_meta_clone.tagging_config = None;
} else {
// Serialize tags to XML
let tagging_xml = quick_xml::se::to_string(&existing_tagging)
.map_err(|e| SwiftError::InternalServerError(format!("Failed to serialize tags: {}", e)))?;
bucket_meta_clone.tagging_config_xml = tagging_xml.into_bytes();
bucket_meta_clone.tagging_config_updated_at = now;
bucket_meta_clone.tagging_config = Some(existing_tagging);
}
// Save updated metadata
set_swift_bucket_metadata(bucket_name, bucket_meta_clone)
.await
.map_err(|e| SwiftError::InternalServerError(format!("Failed to save metadata: {}", e)))?;
tagging
})
.await?;
debug!(
"Set ACLs for container {}/{}: read={:?}, write={:?}",
@@ -1843,4 +1781,24 @@ mod tests {
assert_eq!(tagging.tag_set.len(), 1);
assert_eq!(tagging.tag_set[0].value.as_deref(), Some("new-archive"));
}
#[test]
fn nonempty_container_usage_preserves_authoritative_totals() {
let usage = std::collections::HashMap::from([("tenant-container".to_string(), (7, 4097))]);
assert_eq!(
required_bucket_usage(&usage, "tenant-container").expect("authoritative bucket usage"),
(7, 4097)
);
}
#[test]
fn missing_container_usage_fails_closed() {
let usage = std::collections::HashMap::new();
assert!(matches!(
required_bucket_usage(&usage, "tenant-container"),
Err(SwiftError::ServiceUnavailable(_))
));
}
}
+4
View File
@@ -32,6 +32,8 @@ pub enum SwiftError {
NotFound(String),
/// 409 Conflict
Conflict(String),
/// 411 Length Required
LengthRequired(String),
/// 413 Request Entity Too Large (Payload Too Large)
RequestEntityTooLarge(String),
/// 422 Unprocessable Entity
@@ -54,6 +56,7 @@ impl fmt::Display for SwiftError {
SwiftError::Forbidden(msg) => write!(f, "Forbidden: {}", msg),
SwiftError::NotFound(msg) => write!(f, "Not Found: {}", msg),
SwiftError::Conflict(msg) => write!(f, "Conflict: {}", msg),
SwiftError::LengthRequired(msg) => write!(f, "Length Required: {}", msg),
SwiftError::RequestEntityTooLarge(msg) => write!(f, "Request Entity Too Large: {}", msg),
SwiftError::UnprocessableEntity(msg) => write!(f, "Unprocessable Entity: {}", msg),
SwiftError::TooManyRequests { retry_after, .. } => {
@@ -76,6 +79,7 @@ impl SwiftError {
SwiftError::Forbidden(_) => StatusCode::FORBIDDEN,
SwiftError::NotFound(_) => StatusCode::NOT_FOUND,
SwiftError::Conflict(_) => StatusCode::CONFLICT,
SwiftError::LengthRequired(_) => StatusCode::LENGTH_REQUIRED,
SwiftError::RequestEntityTooLarge(_) => StatusCode::PAYLOAD_TOO_LARGE,
SwiftError::UnprocessableEntity(_) => StatusCode::UNPROCESSABLE_ENTITY,
SwiftError::TooManyRequests { .. } => StatusCode::TOO_MANY_REQUESTS,
+59 -10
View File
@@ -28,7 +28,9 @@ use axum::http::{Method, Request, Response, StatusCode};
use futures::Future;
use rustfs_credentials::Credentials;
use rustfs_keystone::KEYSTONE_CREDENTIALS;
use rustfs_trusted_proxies::ClientInfo;
use s3s::Body;
use std::net::{IpAddr, SocketAddr};
use std::pin::Pin;
use std::task::{Context, Poll};
use tokio_util::io::StreamReader;
@@ -40,6 +42,14 @@ const LOG_SUBSYSTEM_SWIFT_HANDLER: &str = "swift_handler";
const EVENT_SWIFT_ROUTE_STATE: &str = "swift_route_state";
const EVENT_SWIFT_TEMPURL_STATE: &str = "swift_tempurl_state";
fn trusted_client_ip<B>(req: &Request<B>) -> Option<IpAddr> {
req.extensions()
.get::<ClientInfo>()
.map(|info| info.real_ip)
.or_else(|| req.extensions().get::<SocketAddr>().map(SocketAddr::ip))
.filter(|ip| !ip.is_unspecified())
}
/// Swift-aware service that routes to Swift handlers or S3 service
#[derive(Clone)]
pub struct SwiftService<S> {
@@ -128,6 +138,7 @@ async fn handle_swift_request(
route: SwiftRoute,
credentials: Option<Credentials>,
) -> Result<Response<Body>, SwiftError> {
let client_ip = trusted_client_ip(&req);
// Extract parts
let (parts, body) = req.into_parts();
let method = parts.method.clone();
@@ -164,7 +175,7 @@ async fn handle_swift_request(
let tempurl = tempurl::TempURL::new(key);
let path = uri.path();
tempurl.validate_request(method.as_str(), path, &tempurl_params)?;
tempurl.validate_request(method.as_str(), path, &tempurl_params, client_ip)?;
// TempURL is valid - proceed with request (no credentials needed)
debug!(
@@ -641,14 +652,11 @@ async fn handle_authenticated_request(
.await;
}
// Check quota before upload (if Content-Length provided)
if let Some(content_length) = headers.get("content-length")
&& let Ok(size_str) = content_length.to_str()
&& let Ok(object_size) = size_str.parse::<u64>()
{
// Check if upload would exceed quota
super::quota::check_upload_quota(&account, &container, object_size, &credentials).await?;
}
let object_size = headers
.get("content-length")
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse::<u64>().ok());
super::quota::check_upload_quota(&account, &container, object_size, &credentials).await?;
// Check if versioning is enabled for this container
if let Some(archive_container) = container::get_versions_location(&account, &container, &credentials).await? {
@@ -1475,6 +1483,7 @@ fn swift_error_to_response(error: SwiftError) -> Response<Body> {
SwiftError::Forbidden(msg) => (StatusCode::FORBIDDEN, msg.as_str()),
SwiftError::NotFound(msg) => (StatusCode::NOT_FOUND, msg.as_str()),
SwiftError::Conflict(msg) => (StatusCode::CONFLICT, msg.as_str()),
SwiftError::LengthRequired(msg) => (StatusCode::LENGTH_REQUIRED, msg.as_str()),
SwiftError::RequestEntityTooLarge(msg) => (StatusCode::PAYLOAD_TOO_LARGE, msg.as_str()),
SwiftError::UnprocessableEntity(msg) => (StatusCode::UNPROCESSABLE_ENTITY, msg.as_str()),
SwiftError::TooManyRequests { .. } => (StatusCode::TOO_MANY_REQUESTS, "Rate limit exceeded"),
@@ -1497,7 +1506,10 @@ fn swift_error_to_response(error: SwiftError) -> Response<Body> {
#[cfg(test)]
mod tests {
use super::parse_range_header;
use super::{parse_range_header, trusted_client_ip};
use axum::http::Request;
use rustfs_trusted_proxies::ClientInfo;
use std::net::SocketAddr;
#[test]
fn test_parse_range_header_start_end() {
// bytes=100-199
@@ -1588,4 +1600,41 @@ mod tests {
let result = parse_range_header("bytes=0-999", 1000);
assert_eq!(result, Some((0, 999)));
}
#[test]
fn trusted_client_ip_ignores_untrusted_forwarded_headers() {
let mut request = Request::builder()
.header("x-forwarded-for", "198.51.100.9")
.header("x-real-ip", "198.51.100.10")
.body(())
.expect("request");
request
.extensions_mut()
.insert("192.0.2.7:9000".parse::<SocketAddr>().expect("socket address"));
assert_eq!(trusted_client_ip(&request), Some("192.0.2.7".parse().expect("peer IP address")));
}
#[test]
fn trusted_client_ip_prefers_validated_proxy_identity() {
let mut request = Request::new(());
request
.extensions_mut()
.insert("192.0.2.7:9000".parse::<SocketAddr>().expect("socket address"));
request.extensions_mut().insert(ClientInfo::direct(
"203.0.113.8:443".parse::<SocketAddr>().expect("validated client address"),
));
assert_eq!(trusted_client_ip(&request), Some("203.0.113.8".parse().expect("validated client IP")));
}
#[test]
fn trusted_client_ip_rejects_unspecified_fallback() {
let mut request = Request::new(());
request
.extensions_mut()
.insert(ClientInfo::direct("0.0.0.0:0".parse::<SocketAddr>().expect("unspecified fallback")));
assert_eq!(trusted_client_ip(&request), None);
}
}
+44 -1
View File
@@ -58,10 +58,53 @@ pub mod versioning;
pub use errors::{SwiftError, SwiftResult};
pub use router::{SwiftRoute, SwiftRouter};
/// Maximum number of metadata headers allowed per resource (Swift standard)
pub(crate) const MAX_METADATA_COUNT: usize = 90;
/// Maximum size in bytes for a single metadata value (Swift standard)
pub(crate) const MAX_METADATA_VALUE_SIZE: usize = 256;
/// Validate metadata against Swift limits
///
/// Checks that:
/// - Total number of metadata entries doesn't exceed MAX_METADATA_COUNT
/// - Individual metadata values don't exceed MAX_METADATA_VALUE_SIZE
///
/// Applies to object, container and account metadata alike: all three are
/// persisted, and container/account metadata additionally lands in the
/// bucket metadata file that every later config write rewrites in full.
///
/// Returns error if limits are exceeded.
pub(crate) fn validate_metadata(metadata: &std::collections::HashMap<String, String>) -> SwiftResult<()> {
// Check total metadata count
if metadata.len() > MAX_METADATA_COUNT {
return Err(SwiftError::BadRequest(format!(
"Too many metadata headers: {} (max: {})",
metadata.len(),
MAX_METADATA_COUNT
)));
}
// Check individual value sizes
for (key, value) in metadata.iter() {
if value.len() > MAX_METADATA_VALUE_SIZE {
return Err(SwiftError::BadRequest(format!(
"Metadata value for '{}' too large: {} bytes (max: {} bytes)",
key,
value.len(),
MAX_METADATA_VALUE_SIZE
)));
}
}
Ok(())
}
// Note: Container, Object, and SwiftMetadata types used by Swift implementation
pub use storage_api::public_api::{SwiftGetObjectReader, SwiftObjectInfo, SwiftObjectOptions, SwiftPutObjReader};
pub(crate) use storage_api::public_api::{
get_swift_bucket_metadata, resolve_swift_object_store_handle, set_swift_bucket_metadata,
get_swift_bucket_metadata, get_swift_bucket_usage, resolve_swift_object_store_handle, update_swift_bucket_tagging,
};
#[allow(unused_imports)]
pub use types::{Container, Object, SwiftMetadata};
+1 -39
View File
@@ -53,7 +53,7 @@ use super::account::validate_account_access;
use super::container::ContainerMapper;
use super::expiration_worker::{track_object_expiration, untrack_object_expiration};
use super::storage_api::object::{BucketOperations, BucketOptions, HTTPRangeSpec, ObjectIO as _, ObjectOperations as _};
use super::{SwiftError, SwiftResult, resolve_swift_object_store_handle};
use super::{SwiftError, SwiftResult, resolve_swift_object_store_handle, validate_metadata};
use axum::http::HeaderMap;
use rustfs_credentials::Credentials;
use rustfs_rio::HashReader;
@@ -68,12 +68,6 @@ const LOG_SUBSYSTEM_SWIFT_OBJECT: &str = "swift_object";
const EVENT_SWIFT_OBJECT_STORAGE_STATE: &str = "swift_object_storage_state";
const SWIFT_DELETE_AT_METADATA: &str = "x-delete-at";
/// Maximum number of metadata headers allowed per object (Swift standard)
const MAX_METADATA_COUNT: usize = 90;
/// Maximum size in bytes for a single metadata value (Swift standard)
const MAX_METADATA_VALUE_SIZE: usize = 256;
/// Maximum object size in bytes (5GB - Swift default)
const MAX_OBJECT_SIZE: i64 = 5 * 1024 * 1024 * 1024;
@@ -234,38 +228,6 @@ impl Default for ObjectKeyMapper {
}
}
/// Validate metadata against Swift limits
///
/// Checks that:
/// - Total number of metadata entries doesn't exceed MAX_METADATA_COUNT
/// - Individual metadata values don't exceed MAX_METADATA_VALUE_SIZE
///
/// Returns error if limits are exceeded.
fn validate_metadata(metadata: &HashMap<String, String>) -> SwiftResult<()> {
// Check total metadata count
if metadata.len() > MAX_METADATA_COUNT {
return Err(SwiftError::BadRequest(format!(
"Too many metadata headers: {} (max: {})",
metadata.len(),
MAX_METADATA_COUNT
)));
}
// Check individual value sizes
for (key, value) in metadata.iter() {
if value.len() > MAX_METADATA_VALUE_SIZE {
return Err(SwiftError::BadRequest(format!(
"Metadata value for '{}' too large: {} bytes (max: {} bytes)",
key,
value.len(),
MAX_METADATA_VALUE_SIZE
)));
}
}
Ok(())
}
fn metadata_delete_at(metadata: &HashMap<String, String>) -> Option<u64> {
metadata
.get(SWIFT_DELETE_AT_METADATA)
+42 -21
View File
@@ -87,18 +87,17 @@ pub struct QuotaConfig {
impl QuotaConfig {
/// Load quota configuration from container metadata
pub async fn load(account: &str, container_name: &str, credentials: &Credentials) -> SwiftResult<Self> {
// Get container metadata
let container_info = container::get_container_metadata(account, container_name, credentials).await?;
let custom_metadata = container::get_container_custom_metadata(account, container_name, credentials).await?;
let mut config = QuotaConfig::default();
// Parse Quota-Bytes
if let Some(quota_bytes_str) = container_info.custom_metadata.get("x-container-meta-quota-bytes") {
if let Some(quota_bytes_str) = custom_metadata.get("x-container-meta-quota-bytes") {
config.quota_bytes = quota_bytes_str.parse().ok();
}
// Parse Quota-Count
if let Some(quota_count_str) = container_info.custom_metadata.get("x-container-meta-quota-count") {
if let Some(quota_count_str) = custom_metadata.get("x-container-meta-quota-count") {
config.quota_count = quota_count_str.parse().ok();
}
@@ -113,9 +112,12 @@ impl QuotaConfig {
/// Check if adding an object would exceed quotas
///
/// Returns Ok(()) if within quota, Err with 413 if exceeded
pub fn check_quota(&self, current_bytes: u64, current_count: u64, additional_bytes: u64) -> SwiftResult<()> {
pub fn check_quota(&self, current_bytes: u64, current_count: u64, additional_bytes: Option<u64>) -> SwiftResult<()> {
// Check byte quota
if let Some(max_bytes) = self.quota_bytes {
let additional_bytes = additional_bytes.ok_or_else(|| {
SwiftError::LengthRequired("Content-Length is required when a byte quota is configured".to_string())
})?;
let new_bytes = current_bytes.saturating_add(additional_bytes);
if new_bytes > max_bytes {
return Err(SwiftError::RequestEntityTooLarge(format!(
@@ -147,7 +149,7 @@ impl QuotaConfig {
pub async fn check_upload_quota(
account: &str,
container_name: &str,
object_size: u64,
object_size: Option<u64>,
credentials: &Credentials,
) -> SwiftResult<()> {
// Load quota config
@@ -157,7 +159,6 @@ pub async fn check_upload_quota(
if !quota.is_enabled() {
return Ok(());
}
// Get current container usage
let metadata = container::get_container_metadata(account, container_name, credentials).await?;
@@ -173,7 +174,7 @@ pub async fn check_upload_quota(
quota_bytes = ?quota.quota_bytes,
current_count = metadata.object_count,
quota_count = ?quota.quota_count,
object_size,
object_size = ?object_size,
"swift quota state changed"
);
@@ -235,7 +236,7 @@ mod tests {
};
// Current: 500 bytes, adding 400 bytes = 900 total (within 1000 limit)
let result = config.check_quota(500, 0, 400);
let result = config.check_quota(500, 0, Some(400));
assert!(result.is_ok());
}
@@ -247,7 +248,7 @@ mod tests {
};
// Current: 500 bytes, adding 600 bytes = 1100 total (exceeds 1000 limit)
let result = config.check_quota(500, 0, 600);
let result = config.check_quota(500, 0, Some(600));
assert!(result.is_err());
match result {
Err(SwiftError::RequestEntityTooLarge(msg)) => {
@@ -265,7 +266,7 @@ mod tests {
};
// Current: 500 bytes, adding 500 bytes = 1000 total (exactly at limit)
let result = config.check_quota(500, 0, 500);
let result = config.check_quota(500, 0, Some(500));
assert!(result.is_ok());
}
@@ -277,7 +278,7 @@ mod tests {
};
// Current: 5 objects, adding 1 = 6 total (within 10 limit)
let result = config.check_quota(0, 5, 100);
let result = config.check_quota(0, 5, Some(100));
assert!(result.is_ok());
}
@@ -289,7 +290,7 @@ mod tests {
};
// Current: 10 objects, adding 1 = 11 total (exceeds 10 limit)
let result = config.check_quota(0, 10, 100);
let result = config.check_quota(0, 10, Some(100));
assert!(result.is_err());
match result {
Err(SwiftError::RequestEntityTooLarge(msg)) => {
@@ -307,7 +308,7 @@ mod tests {
};
// Current: 9 objects, adding 1 = 10 total (exactly at limit)
let result = config.check_quota(0, 9, 100);
let result = config.check_quota(0, 9, Some(100));
assert!(result.is_ok());
}
@@ -319,7 +320,7 @@ mod tests {
};
// Both within limits
let result = config.check_quota(500, 5, 400);
let result = config.check_quota(500, 5, Some(400));
assert!(result.is_ok());
}
@@ -331,7 +332,7 @@ mod tests {
};
// Bytes exceeded, count within
let result = config.check_quota(500, 5, 600);
let result = config.check_quota(500, 5, Some(600));
assert!(result.is_err());
}
@@ -343,7 +344,7 @@ mod tests {
};
// Count exceeded, bytes within
let result = config.check_quota(500, 10, 100);
let result = config.check_quota(500, 10, Some(100));
assert!(result.is_err());
}
@@ -355,7 +356,7 @@ mod tests {
};
// No limits, should always pass
let result = config.check_quota(999999, 999999, 999999);
let result = config.check_quota(999999, 999999, Some(999999));
assert!(result.is_ok());
}
@@ -367,7 +368,7 @@ mod tests {
};
// Zero limit means no uploads allowed
let result = config.check_quota(0, 0, 1);
let result = config.check_quota(0, 0, Some(1));
assert!(result.is_err());
}
@@ -379,7 +380,7 @@ mod tests {
};
// Zero limit means no objects allowed
let result = config.check_quota(0, 0, 100);
let result = config.check_quota(0, 0, Some(100));
assert!(result.is_err());
}
@@ -391,8 +392,28 @@ mod tests {
};
// Test saturating_add protection
let result = config.check_quota(u64::MAX - 100, 0, 200);
let result = config.check_quota(u64::MAX - 100, 0, Some(200));
// Should saturate to u64::MAX and compare against quota
assert!(result.is_ok());
}
#[test]
fn byte_quota_rejects_upload_without_content_length() {
let config = QuotaConfig {
quota_bytes: Some(1000),
quota_count: None,
};
assert!(matches!(config.check_quota(500, 0, None), Err(SwiftError::LengthRequired(_))));
}
#[test]
fn count_only_quota_checks_upload_without_content_length() {
let config = QuotaConfig {
quota_bytes: None,
quota_count: Some(10),
};
assert!(matches!(config.check_quota(0, 10, None), Err(SwiftError::RequestEntityTooLarge(_))));
}
}
+110 -6
View File
@@ -12,16 +12,22 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use std::collections::HashMap;
use std::sync::Arc;
use rustfs_ecstore::api::bucket::metadata::BUCKET_TAGGING_CONFIG;
pub(crate) use rustfs_ecstore::api::bucket::metadata::BucketMetadata as SwiftBucketMetadata;
use rustfs_ecstore::api::bucket::metadata_sys::{
get as get_swift_bucket_metadata_from_backend, set_bucket_metadata as set_swift_bucket_metadata_in_backend,
};
use rustfs_ecstore::api::bucket::metadata_sys::{get as get_swift_bucket_metadata_from_backend, update_config_with};
use rustfs_ecstore::api::bucket::utils::serialize as serialize_bucket_config;
use rustfs_ecstore::api::error::Error as SwiftStorageError;
pub(crate) use rustfs_ecstore::api::error::Result as SwiftStorageResult;
use rustfs_ecstore::api::notification::get_global_notification_sys;
pub(crate) use rustfs_ecstore::api::runtime::object_store_handle as resolve_swift_object_store_handle;
use rustfs_ecstore::api::storage::ECStore as SwiftStore;
use rustfs_storage_api as storage_contracts;
use s3s::dto::Tagging;
use super::{SwiftError, SwiftResult};
pub(crate) mod account {
pub(crate) use super::storage_contracts::{BucketOperations, MakeBucketOptions};
@@ -43,13 +49,24 @@ pub(crate) mod object {
pub(crate) mod public_api {
pub use super::{SwiftGetObjectReader, SwiftObjectInfo, SwiftObjectOptions, SwiftPutObjReader};
pub(crate) use super::{get_swift_bucket_metadata, resolve_swift_object_store_handle, set_swift_bucket_metadata};
pub(crate) use super::{
get_swift_bucket_metadata, get_swift_bucket_usage, resolve_swift_object_store_handle, update_swift_bucket_tagging,
};
}
pub(crate) mod versioning {
pub(crate) use super::storage_contracts::{ListOperations, ObjectOperations};
}
const LOG_COMPONENT_PROTOCOLS: &str = "protocols";
const LOG_SUBSYSTEM_SWIFT_STORAGE: &str = "swift_storage";
const EVENT_SWIFT_BUCKET_TAGGING_UPDATE: &str = "swift_bucket_tagging_update";
/// Marks the refusal to rewrite an unreadable persisted tagging config, so the
/// caller can turn it into an actionable client error rather than a generic
/// storage failure. Carried through the ecstore error, which is a string type.
const UNREADABLE_TAGGING_SENTINEL: &str = "swift: persisted tagging config could not be parsed";
pub type SwiftGetObjectReader = <SwiftStore as storage_contracts::ObjectIO>::GetObjectReader;
pub type SwiftObjectInfo = <SwiftStore as storage_contracts::ObjectOperations>::ObjectInfo;
pub type SwiftObjectOptions = <SwiftStore as storage_contracts::ObjectOperations>::ObjectOptions;
@@ -59,6 +76,93 @@ pub(crate) async fn get_swift_bucket_metadata(bucket: &str) -> SwiftStorageResul
get_swift_bucket_metadata_from_backend(bucket).await
}
pub(crate) async fn set_swift_bucket_metadata(bucket: String, metadata: SwiftBucketMetadata) -> SwiftStorageResult<()> {
set_swift_bucket_metadata_in_backend(bucket, metadata).await
/// Rewrite the bucket's tagging config through the persisting
/// bucket-metadata path.
///
/// `rewrite` sees the tag set currently persisted on disk (`None` when the
/// bucket has none) and returns the full replacement; an empty tag set
/// clears the config. The read-modify-write runs under the bucket metadata
/// system's write guard — serialized against every other config update —
/// and the result is written to the bucket metadata file before the cache
/// is refreshed, so a Swift metadata POST survives process restarts and
/// disk-truth reloads. Peers are then told to reload, matching what the S3
/// handlers do after a config write.
///
/// Storage failures are logged in full and reported to the client as a
/// generic error: these now carry real disk and quorum detail, which does not
/// belong in a Swift response body. The one exception is an unreadable
/// persisted config, which is reported specifically because the operator has
/// to act on it.
pub(crate) async fn update_swift_bucket_tagging<F>(bucket: String, rewrite: F) -> SwiftResult<()>
where
F: FnOnce(Option<&Tagging>) -> Tagging + Send,
{
let result = update_config_with(&bucket, BUCKET_TAGGING_CONFIG, |bm| {
// Merging onto an unparseable tag set would silently drop every tag
// the bucket has — including the container ACL and versioning tags —
// because the rewrite closures treat "no parsed tags" as "no tags".
// Refuse instead: the persisted config is intact, just unreadable.
if !bm.tagging_config_xml.is_empty() && bm.tagging_config.is_none() {
return Err(SwiftStorageError::other(UNREADABLE_TAGGING_SENTINEL));
}
let tagging = rewrite(bm.tagging_config.as_ref());
if tagging.tag_set.is_empty() {
Ok(Vec::new())
} else {
// The S3 XML serializer, not quick_xml: the metadata loader's
// parse step must be able to round-trip what we persist.
serialize_bucket_config(&tagging)
.map_err(|e| SwiftStorageError::other(format!("failed to serialize bucket tagging: {e}")))
}
})
.await;
if let Err(err) = result {
let unreadable = err.to_string().contains(UNREADABLE_TAGGING_SENTINEL);
tracing::error!(
event = EVENT_SWIFT_BUCKET_TAGGING_UPDATE,
component = LOG_COMPONENT_PROTOCOLS,
subsystem = LOG_SUBSYSTEM_SWIFT_STORAGE,
bucket = %bucket,
error = %err,
reason = if unreadable { "unreadable_persisted_config" } else { "storage_failure" },
result = "failed",
"swift bucket tagging update failed"
);
// A Swift-only client has no way to repair this itself, so say what
// happened and name the remedy instead of a bare storage error.
return Err(if unreadable {
SwiftError::Conflict(format!(
"The persisted tagging configuration for container store '{bucket}' cannot be parsed, so metadata cannot be updated without discarding it. Reset it with the S3 DeleteBucketTagging API."
))
} else {
SwiftError::InternalServerError("Metadata update operation failed".to_string())
});
}
if let Some(notification_sys) = get_global_notification_sys() {
tokio::spawn(async move {
if let Err(err) = notification_sys.load_bucket_metadata(&bucket).await {
tracing::warn!(bucket = %bucket, error = %err, "failed to notify peers after swift bucket tagging update");
}
});
}
Ok(())
}
pub(crate) async fn get_swift_bucket_usage() -> SwiftStorageResult<Option<HashMap<String, (u64, u64)>>> {
let Some(store) = resolve_swift_object_store_handle() else {
return Ok(None);
};
let mut data_usage = rustfs_ecstore::api::data_usage::load_data_usage_from_backend_cached(store).await?;
rustfs_ecstore::api::data_usage::apply_bucket_usage_memory_overlay(&mut data_usage).await;
Ok(Some(
data_usage
.buckets_usage
.into_iter()
.map(|(bucket, usage)| (bucket, (usage.objects_count, usage.size)))
.collect(),
))
}
+156 -18
View File
@@ -21,7 +21,11 @@
use crate::swift::errors::SwiftError;
use hmac::{Hmac, KeyInit, Mac};
use ipnetwork::IpNetwork;
use percent_encoding::percent_decode_str;
use sha1::Sha1;
use std::net::IpAddr;
use std::str::FromStr;
use std::time::{SystemTime, UNIX_EPOCH};
type HmacSha1 = Hmac<Sha1>;
@@ -50,12 +54,12 @@ impl TempURLParams {
let mut ip_range = None;
for param in query.split('&') {
let parts: Vec<&str> = param.split('=').collect();
if parts.len() == 2 {
match parts[0] {
"temp_url_sig" => sig = Some(parts[1].to_string()),
"temp_url_expires" => expires = parts[1].parse().ok(),
"temp_url_ip_range" => ip_range = Some(parts[1].to_string()),
if let Some((name, value)) = param.split_once('=') {
let value = percent_decode_str(value).decode_utf8().ok()?;
match name {
"temp_url_sig" => sig = Some(value.into_owned()),
"temp_url_expires" => expires = value.parse().ok(),
"temp_url_ip_range" => ip_range = Some(value.into_owned()),
_ => {}
}
}
@@ -97,9 +101,22 @@ impl TempURL {
/// # Returns
/// Hex-encoded HMAC-SHA1 signature
pub fn generate_signature(&self, method: &str, expires: u64, path: &str) -> Result<String, SwiftError> {
self.generate_signature_with_ip_range(method, expires, path, None)
}
/// Generate a TempURL signature, optionally binding it to an IP range.
pub fn generate_signature_with_ip_range(
&self,
method: &str,
expires: u64,
path: &str,
ip_range: Option<&str>,
) -> Result<String, SwiftError> {
// Construct message for HMAC
// Format: "{METHOD}\n{expires}\n{path}"
let message = format!("{}\n{}\n{}", method.to_uppercase(), expires, path);
let message = match ip_range {
Some(ip_range) => format!("ip={}\n{}\n{}\n{}", ip_range, method.to_uppercase(), expires, path),
None => format!("{}\n{}\n{}", method.to_uppercase(), expires, path),
};
// Calculate HMAC-SHA1
let mut mac = HmacSha1::new_from_slice(self.key.as_bytes())
@@ -127,7 +144,13 @@ impl TempURL {
/// # Returns
/// - `Ok(())` if signature is valid and not expired
/// - `Err(SwiftError::Unauthorized)` if invalid or expired
pub fn validate_request(&self, method: &str, path: &str, params: &TempURLParams) -> Result<(), SwiftError> {
pub fn validate_request(
&self,
method: &str,
path: &str,
params: &TempURLParams,
client_ip: Option<IpAddr>,
) -> Result<(), SwiftError> {
// 1. Check expiration first (fast path for expired URLs)
let now = SystemTime::now()
.duration_since(UNIX_EPOCH)
@@ -139,17 +162,25 @@ impl TempURL {
}
// 2. Generate expected signature
let expected_sig = self.generate_signature(method, params.temp_url_expires, path)?;
let expected_sig =
self.generate_signature_with_ip_range(method, params.temp_url_expires, path, params.temp_url_ip_range.as_deref())?;
// 3. Constant-time comparison to prevent timing attacks
if !constant_time_compare(params.temp_url_sig.as_bytes(), expected_sig.as_bytes()) {
return Err(SwiftError::Unauthorized("Invalid TempURL signature".to_string()));
}
// 4. TODO: Validate IP range if specified (future enhancement)
// if let Some(ip_range) = &params.temp_url_ip_range {
// validate_ip_range(client_ip, ip_range)?;
// }
if let Some(ip_range) = &params.temp_url_ip_range {
let client_ip =
client_ip.ok_or_else(|| SwiftError::Unauthorized("Trusted client address unavailable".to_string()))?;
let allowed = IpAddr::from_str(ip_range)
.map(|ip| ip == client_ip)
.or_else(|_| IpNetwork::from_str(ip_range).map(|network| network.contains(client_ip)))
.map_err(|_| SwiftError::Unauthorized("Invalid TempURL IP range".to_string()))?;
if !allowed {
return Err(SwiftError::Unauthorized("Client address is outside the TempURL IP range".to_string()));
}
}
Ok(())
}
@@ -308,7 +339,7 @@ mod tests {
// Should validate successfully
assert!(
tempurl
.validate_request("GET", "/v1/AUTH_test/container/object", &params)
.validate_request("GET", "/v1/AUTH_test/container/object", &params, None)
.is_ok()
);
}
@@ -331,7 +362,7 @@ mod tests {
};
// Should reject expired URL
let result = tempurl.validate_request("GET", "/v1/AUTH_test/container/object", &params);
let result = tempurl.validate_request("GET", "/v1/AUTH_test/container/object", &params, None);
assert!(result.is_err());
assert!(matches!(result.unwrap_err(), SwiftError::Unauthorized(_)));
}
@@ -349,7 +380,7 @@ mod tests {
};
// Should reject invalid signature
let result = tempurl.validate_request("GET", "/v1/AUTH_test/container/object", &params);
let result = tempurl.validate_request("GET", "/v1/AUTH_test/container/object", &params, None);
assert!(result.is_err());
assert!(matches!(result.unwrap_err(), SwiftError::Unauthorized(_)));
}
@@ -372,7 +403,7 @@ mod tests {
};
// Try to validate with PUT method
let result = tempurl.validate_request("PUT", "/v1/AUTH_test/container/object", &params);
let result = tempurl.validate_request("PUT", "/v1/AUTH_test/container/object", &params, None);
assert!(result.is_err());
assert!(matches!(result.unwrap_err(), SwiftError::Unauthorized(_)));
}
@@ -423,6 +454,14 @@ mod tests {
assert_eq!(params.temp_url_ip_range.as_deref(), Some("192.168.1.0/24"));
}
#[test]
fn test_parse_percent_encoded_ipv6_range() {
let query = "temp_url_sig=abc123&temp_url_expires=1609459200&temp_url_ip_range=2001%3Adb8%3A%3A%2F32";
let params = TempURLParams::from_query(query).expect("encoded IPv6 TempURL parameters");
assert_eq!(params.temp_url_ip_range.as_deref(), Some("2001:db8::/32"));
}
#[test]
fn test_parse_tempurl_params_missing_sig() {
let query = "temp_url_expires=1609459200";
@@ -475,4 +514,103 @@ mod tests {
.unwrap();
assert_eq!(sig, sig2);
}
#[test]
fn ip_bound_signature_matches_openstack_message_format() {
let tempurl = TempURL::new("mykey".to_string());
let signature = tempurl
.generate_signature_with_ip_range("GET", 1648082711, "/v1/AUTH_account/container/object", Some("1.2.3.0/24"))
.expect("IP-bound signature generation");
assert_eq!(signature, "8698990d811e64cff1c8a2151340874fd5bda0c5");
}
fn ip_bound_params(tempurl: &TempURL, ip_range: &str) -> TempURLParams {
let expires = SystemTime::now()
.duration_since(UNIX_EPOCH)
.expect("system clock after Unix epoch")
.as_secs()
+ 3600;
let signature = tempurl
.generate_signature_with_ip_range("GET", expires, "/v1/AUTH_test/container/object", Some(ip_range))
.expect("IP-bound signature generation");
TempURLParams {
temp_url_sig: signature,
temp_url_expires: expires,
temp_url_ip_range: Some(ip_range.to_string()),
}
}
#[test]
fn ip_bound_tempurl_accepts_ipv4_inside_range_and_rejects_outside() {
let tempurl = TempURL::new("mykey".to_string());
let params = ip_bound_params(&tempurl, "192.0.2.0/24");
assert!(
tempurl
.validate_request(
"GET",
"/v1/AUTH_test/container/object",
&params,
Some("192.0.2.42".parse().expect("IPv4 address")),
)
.is_ok()
);
assert!(matches!(
tempurl.validate_request(
"GET",
"/v1/AUTH_test/container/object",
&params,
Some("198.51.100.42".parse().expect("IPv4 address")),
),
Err(SwiftError::Unauthorized(_))
));
}
#[test]
fn ip_bound_tempurl_accepts_ipv6_inside_range_and_rejects_outside() {
let tempurl = TempURL::new("mykey".to_string());
let params = ip_bound_params(&tempurl, "2001:db8::/32");
assert!(
tempurl
.validate_request(
"GET",
"/v1/AUTH_test/container/object",
&params,
Some("2001:db8::42".parse().expect("IPv6 address")),
)
.is_ok()
);
assert!(matches!(
tempurl.validate_request(
"GET",
"/v1/AUTH_test/container/object",
&params,
Some("2001:db9::42".parse().expect("IPv6 address")),
),
Err(SwiftError::Unauthorized(_))
));
}
#[test]
fn ip_bound_tempurl_rejects_missing_client_ip_and_invalid_range() {
let tempurl = TempURL::new("mykey".to_string());
let params = ip_bound_params(&tempurl, "192.0.2.0/24");
assert!(matches!(
tempurl.validate_request("GET", "/v1/AUTH_test/container/object", &params, None),
Err(SwiftError::Unauthorized(_))
));
let invalid_params = ip_bound_params(&tempurl, "not-a-network");
assert!(matches!(
tempurl.validate_request(
"GET",
"/v1/AUTH_test/container/object",
&invalid_params,
Some("192.0.2.42".parse().expect("IPv4 address")),
),
Err(SwiftError::Unauthorized(_))
));
}
}
@@ -0,0 +1,25 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! ECStore facade boundary for the protocols integration tests.
//!
//! The Swift tests need to drive bucket-metadata reloads the way the peer
//! `LoadBucketMetadata` RPC does. Everything they touch from `rustfs_ecstore`
//! is aliased here so the tests themselves hold no raw facade subpaths, the
//! same boundary `crates/protocols/src/swift/storage_api.rs` provides for the
//! Swift implementation.
pub use rustfs_ecstore::api::bucket::metadata::load_bucket_metadata;
pub use rustfs_ecstore::api::bucket::metadata_sys::{get as get_bucket_metadata, set_bucket_metadata};
pub use rustfs_ecstore::api::runtime::object_store_handle;
@@ -0,0 +1,312 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Regression tests: a Swift metadata POST must be persisted to the bucket
//! metadata file, not just the in-memory cache. The metadata has to survive
//! the disk-truth reloads performed by peer LoadBucketMetadata notifications
//! and the periodic refresh loop — and, transitively, a process restart.
#![cfg(feature = "swift")]
use std::collections::HashMap;
use rustfs_credentials::Credentials;
use rustfs_protocols::swift::SwiftError;
use rustfs_protocols::swift::container::{ContainerMapper, update_container_metadata};
use rustfs_protocols::swift::{account, container};
use rustfs_test_utils::TestECStoreEnv;
use serde_json::json;
use sha2::{Digest, Sha256};
mod ecstore_test_compat;
use ecstore_test_compat::{get_bucket_metadata, load_bucket_metadata, object_store_handle, set_bucket_metadata};
fn keystone_credentials(project_id: &str) -> Credentials {
let mut claims = HashMap::new();
claims.insert("keystone_project_id".to_string(), json!(project_id));
claims.insert("keystone_roles".to_string(), json!(["member"]));
Credentials {
access_key: "keystone:swift-test".to_string(),
claims: Some(claims),
..Default::default()
}
}
/// The account-metadata bucket name scheme from `swift::account`
/// (`swift-account-{sha256(account)[0..16]}`), mirrored here so the test can
/// reload that bucket's metadata from disk.
fn account_metadata_bucket_name(account: &str) -> String {
let mut hasher = Sha256::new();
hasher.update(account.as_bytes());
let hash = hex::encode(hasher.finalize());
format!("swift-account-{}", &hash[0..16])
}
/// Replace the cached bucket metadata with what is actually on disk — the
/// same thing a peer LoadBucketMetadata notification or the periodic refresh
/// loop does. Before the fix this silently discarded every Swift metadata
/// POST, because those writes only ever touched the cache.
async fn reload_bucket_metadata_from_disk(bucket: &str) {
let store = object_store_handle().expect("test store should be published");
let bm = load_bucket_metadata(store, bucket)
.await
.expect("bucket metadata should load from disk");
set_bucket_metadata(bucket.to_string(), bm)
.await
.expect("reloaded metadata should install");
}
/// The `swift-meta-*` tags currently persisted for a container, keyed the way
/// a Swift client sees them. `get_container_metadata` would be the natural
/// reader, but it additionally requires a data-usage snapshot for the object
/// count and byte total, which a bare test store has none of — and that is
/// orthogonal to whether the metadata itself was persisted.
async fn persisted_container_metadata(bucket: &str) -> HashMap<String, String> {
let bm = get_bucket_metadata(bucket).await.expect("bucket metadata should be cached");
let mut out = HashMap::new();
if let Some(tagging) = &bm.tagging_config {
for tag in &tagging.tag_set {
if let (Some(key), Some(value)) = (&tag.key, &tag.value)
&& let Some(meta_key) = key.strip_prefix("swift-meta-")
{
out.insert(meta_key.to_string(), value.clone());
}
}
}
out
}
/// Every scenario that needs a store runs against ONE environment: the Swift
/// handlers resolve the ambient object store and bucket-metadata system, so a
/// second `TestECStoreEnv` in this process would race the first.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn swift_metadata_writes_are_durable() {
let env = TestECStoreEnv::builder().prefix("swift_meta_persist").build().await;
posts_survive_disk_truth_reload(&env).await;
tag_writers_preserve_each_others_state(&env).await;
versioning_writes_reject_missing_containers().await;
}
async fn posts_survive_disk_truth_reload(env: &TestECStoreEnv) {
// --- Container metadata POST (X-Container-Meta-*) ---
let project_id = "swiftpersistproj";
let swift_account = format!("AUTH_{project_id}");
let credentials = keystone_credentials(project_id);
let swift_container = "photos";
let bucket = ContainerMapper::default().swift_to_s3_bucket(swift_container, project_id);
env.make_bucket(&bucket, false).await;
let mut metadata = HashMap::new();
metadata.insert("color".to_string(), "blue".to_string());
update_container_metadata(&swift_account, swift_container, &credentials, metadata)
.await
.expect("container metadata POST should succeed");
reload_bucket_metadata_from_disk(&bucket).await;
let container_meta = persisted_container_metadata(&bucket).await;
assert_eq!(
container_meta.get("color").map(String::as_str),
Some("blue"),
"container metadata POST must survive a disk-truth metadata reload"
);
// A follow-up POST replaces the Swift metadata and that replacement must
// survive a reload too (the rewrite merges against disk state, so the
// previous value must actually be gone).
let mut metadata = HashMap::new();
metadata.insert("season".to_string(), "summer".to_string());
update_container_metadata(&swift_account, swift_container, &credentials, metadata)
.await
.expect("second container metadata POST should succeed");
reload_bucket_metadata_from_disk(&bucket).await;
let container_meta = persisted_container_metadata(&bucket).await;
assert_eq!(container_meta.get("season").map(String::as_str), Some("summer"));
assert!(
!container_meta.contains_key("color"),
"replaced container metadata must not resurrect on reload"
);
// --- Container versioning POST (X-Versions-Location) ---
let archive_container = "photos-archive";
let archive_bucket = ContainerMapper::default().swift_to_s3_bucket(archive_container, project_id);
env.make_bucket(&archive_bucket, false).await;
container::enable_versioning(&swift_account, swift_container, archive_container, &credentials)
.await
.expect("enable versioning should succeed");
reload_bucket_metadata_from_disk(&bucket).await;
let location = container::get_versions_location(&swift_account, swift_container, &credentials)
.await
.expect("versions location should load");
assert_eq!(
location.as_deref(),
Some(archive_container),
"versions location must survive a disk-truth metadata reload"
);
// --- Account metadata POST (TempURL keys etc.) ---
let mut account_meta = HashMap::new();
account_meta.insert("temp-url-key".to_string(), "s3cr3t".to_string());
account::update_account_metadata(&swift_account, &account_meta, &Some(credentials.clone()))
.await
.expect("account metadata POST should succeed");
reload_bucket_metadata_from_disk(&account_metadata_bucket_name(&swift_account)).await;
let loaded = account::get_account_metadata(&swift_account, &None)
.await
.expect("account metadata should load");
assert_eq!(
loaded.get("temp-url-key").map(String::as_str),
Some("s3cr3t"),
"account metadata POST must survive a disk-truth metadata reload"
);
}
/// The rewrites all share one tag set, so a closure that ignored the current
/// state would still pass a single-feature test. Drive ACLs, versioning and
/// container metadata over the same container and assert each survives the
/// others — and that clearing one leaves the rest alone.
async fn tag_writers_preserve_each_others_state(env: &TestECStoreEnv) {
let project_id = "swiftcrosstagproj";
let swift_account = format!("AUTH_{project_id}");
let credentials = keystone_credentials(project_id);
let container = "shared";
let archive = "shared-archive";
let bucket = ContainerMapper::default().swift_to_s3_bucket(container, project_id);
env.make_bucket(&bucket, false).await;
env.make_bucket(&ContainerMapper::default().swift_to_s3_bucket(archive, project_id), false)
.await;
let mut metadata = HashMap::new();
metadata.insert("color".to_string(), "blue".to_string());
update_container_metadata(&swift_account, container, &credentials, metadata)
.await
.expect("container metadata POST should succeed");
container::enable_versioning(&swift_account, container, archive, &credentials)
.await
.expect("enable versioning should succeed");
container::set_container_acl(&swift_account, container, Some(".r:*"), Some("AUTH_other"), &credentials)
.await
.expect("set container ACL should succeed");
reload_bucket_metadata_from_disk(&bucket).await;
// All three writers' state coexists after a disk-truth reload.
let meta = persisted_container_metadata(&bucket).await;
assert_eq!(meta.get("color").map(String::as_str), Some("blue"));
assert_eq!(
container::get_versions_location(&swift_account, container, &credentials)
.await
.expect("versions location should load")
.as_deref(),
Some(archive)
);
let acl = container::get_container_acl(&swift_account, container, &credentials)
.await
.expect("container ACL should load");
assert!(!acl.read.is_empty(), "read ACL must survive the reload");
assert!(!acl.write.is_empty(), "write ACL must survive the reload");
// Disabling versioning drops only the versioning tag.
container::disable_versioning(&swift_account, container, &credentials)
.await
.expect("disable versioning should succeed");
reload_bucket_metadata_from_disk(&bucket).await;
assert_eq!(
container::get_versions_location(&swift_account, container, &credentials)
.await
.expect("versions location should load"),
None,
"disable_versioning must clear the versioning tag durably"
);
let meta = persisted_container_metadata(&bucket).await;
assert_eq!(
meta.get("color").map(String::as_str),
Some("blue"),
"disable_versioning must not disturb container metadata"
);
let acl = container::get_container_acl(&swift_account, container, &credentials)
.await
.expect("container ACL should load");
assert!(!acl.read.is_empty(), "disable_versioning must not disturb the ACL");
}
/// A container that does not exist must not get metadata persisted for it:
/// the metadata loader turns "nothing on disk" into a fresh default, so an
/// unguarded rewrite would create an orphan metadata file and cache a
/// fabricated default as authoritative.
async fn versioning_writes_reject_missing_containers() {
let project_id = "swiftmissingproj";
let swift_account = format!("AUTH_{project_id}");
let credentials = keystone_credentials(project_id);
let missing = "no-such-container";
let err = container::disable_versioning(&swift_account, missing, &credentials)
.await
.expect_err("disabling versioning on a missing container must fail");
assert!(
matches!(err, SwiftError::NotFound(_)),
"expected NotFound for a missing container, got {err:?}"
);
let bucket = ContainerMapper::default().swift_to_s3_bucket(missing, project_id);
assert!(
get_bucket_metadata(&bucket).await.is_err(),
"a rejected write must not have cached metadata for a nonexistent container"
);
}
/// Account metadata holds the account's TempURL signing key, and it is now
/// durable — so a write for someone else's account would be a persistent,
/// cluster-wide takeover of that account's pre-signed URLs, not a cache blip.
/// The write path must reject both a foreign account and a missing token,
/// while reads stay open for pre-auth TempURL signature validation.
///
/// Deliberately builds no store: both rejections must happen before the write
/// path resolves storage at all, and a second `TestECStoreEnv` in this process
/// would race the other test over the ambient store handle.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn account_metadata_write_rejects_foreign_and_anonymous_callers() {
let victim_account = "AUTH_victimproject";
let attacker_credentials = keystone_credentials("attackerproject");
let mut poisoned = HashMap::new();
poisoned.insert("temp-url-key".to_string(), "attacker-key".to_string());
let err = account::update_account_metadata(victim_account, &poisoned, &Some(attacker_credentials))
.await
.expect_err("writing another account's metadata must be rejected");
assert!(
matches!(err, SwiftError::Forbidden(_)),
"cross-account metadata write must be Forbidden, got {err:?}"
);
let err = account::update_account_metadata(victim_account, &poisoned, &None)
.await
.expect_err("anonymous account metadata write must be rejected");
assert!(
matches!(err, SwiftError::Unauthorized(_)),
"anonymous metadata write must be Unauthorized, got {err:?}"
);
}
@@ -233,6 +233,46 @@ pub struct RenamePartResponse {
pub error: ::core::option::Option<Error>,
}
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
pub struct PreparePartTransactionRequest {
#[prost(string, tag = "1")]
pub disk: ::prost::alloc::string::String,
#[prost(string, tag = "2")]
pub src_volume: ::prost::alloc::string::String,
#[prost(string, tag = "3")]
pub src_path: ::prost::alloc::string::String,
#[prost(string, tag = "4")]
pub dst_volume: ::prost::alloc::string::String,
#[prost(string, tag = "5")]
pub dst_path: ::prost::alloc::string::String,
#[prost(bytes = "bytes", tag = "6")]
pub meta: ::prost::bytes::Bytes,
}
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
pub struct PreparePartTransactionResponse {
#[prost(bool, tag = "1")]
pub success: bool,
#[prost(message, optional, tag = "2")]
pub error: ::core::option::Option<Error>,
}
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
pub struct SettlePartTransactionRequest {
#[prost(string, tag = "1")]
pub disk: ::prost::alloc::string::String,
#[prost(string, tag = "2")]
pub volume: ::prost::alloc::string::String,
#[prost(string, tag = "3")]
pub path: ::prost::alloc::string::String,
#[prost(bool, tag = "4")]
pub rollback: bool,
}
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
pub struct SettlePartTransactionResponse {
#[prost(bool, tag = "1")]
pub success: bool,
#[prost(message, optional, tag = "2")]
pub error: ::core::option::Option<Error>,
}
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
pub struct RenameFileRequest {
#[prost(string, tag = "1")]
pub disk: ::prost::alloc::string::String,
@@ -1600,6 +1640,21 @@ pub mod node_service_client {
.insert(GrpcMethod::new("node_service.NodeService", "CheckParts"));
self.inner.unary(req, path, codec).await
}
pub async fn prepare_part_transaction(
&mut self,
request: impl tonic::IntoRequest<super::PreparePartTransactionRequest>,
) -> std::result::Result<tonic::Response<super::PreparePartTransactionResponse>, tonic::Status> {
self.inner
.ready()
.await
.map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?;
let codec = tonic_prost::ProstCodec::default();
let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/PreparePartTransaction");
let mut req = request.into_request();
req.extensions_mut()
.insert(GrpcMethod::new("node_service.NodeService", "PreparePartTransaction"));
self.inner.unary(req, path, codec).await
}
pub async fn rename_part(
&mut self,
request: impl tonic::IntoRequest<super::RenamePartRequest>,
@@ -1615,6 +1670,21 @@ pub mod node_service_client {
.insert(GrpcMethod::new("node_service.NodeService", "RenamePart"));
self.inner.unary(req, path, codec).await
}
pub async fn settle_part_transaction(
&mut self,
request: impl tonic::IntoRequest<super::SettlePartTransactionRequest>,
) -> std::result::Result<tonic::Response<super::SettlePartTransactionResponse>, tonic::Status> {
self.inner
.ready()
.await
.map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?;
let codec = tonic_prost::ProstCodec::default();
let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/SettlePartTransaction");
let mut req = request.into_request();
req.extensions_mut()
.insert(GrpcMethod::new("node_service.NodeService", "SettlePartTransaction"));
self.inner.unary(req, path, codec).await
}
pub async fn rename_file(
&mut self,
request: impl tonic::IntoRequest<super::RenameFileRequest>,
@@ -2726,10 +2796,18 @@ pub mod node_service_server {
&self,
request: tonic::Request<super::CheckPartsRequest>,
) -> std::result::Result<tonic::Response<super::CheckPartsResponse>, tonic::Status>;
async fn prepare_part_transaction(
&self,
request: tonic::Request<super::PreparePartTransactionRequest>,
) -> std::result::Result<tonic::Response<super::PreparePartTransactionResponse>, tonic::Status>;
async fn rename_part(
&self,
request: tonic::Request<super::RenamePartRequest>,
) -> std::result::Result<tonic::Response<super::RenamePartResponse>, tonic::Status>;
async fn settle_part_transaction(
&self,
request: tonic::Request<super::SettlePartTransactionRequest>,
) -> std::result::Result<tonic::Response<super::SettlePartTransactionResponse>, tonic::Status>;
async fn rename_file(
&self,
request: tonic::Request<super::RenameFileRequest>,
@@ -3432,6 +3510,34 @@ pub mod node_service_server {
};
Box::pin(fut)
}
"/node_service.NodeService/PreparePartTransaction" => {
#[allow(non_camel_case_types)]
struct PreparePartTransactionSvc<T: NodeService>(pub Arc<T>);
impl<T: NodeService> tonic::server::UnaryService<super::PreparePartTransactionRequest> for PreparePartTransactionSvc<T> {
type Response = super::PreparePartTransactionResponse;
type Future = BoxFuture<tonic::Response<Self::Response>, tonic::Status>;
fn call(&mut self, request: tonic::Request<super::PreparePartTransactionRequest>) -> Self::Future {
let inner = Arc::clone(&self.0);
let fut = async move { <T as NodeService>::prepare_part_transaction(&inner, request).await };
Box::pin(fut)
}
}
let accept_compression_encodings = self.accept_compression_encodings;
let send_compression_encodings = self.send_compression_encodings;
let max_decoding_message_size = self.max_decoding_message_size;
let max_encoding_message_size = self.max_encoding_message_size;
let inner = self.inner.clone();
let fut = async move {
let method = PreparePartTransactionSvc(inner);
let codec = tonic_prost::ProstCodec::default();
let mut grpc = tonic::server::Grpc::new(codec)
.apply_compression_config(accept_compression_encodings, send_compression_encodings)
.apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size);
let res = grpc.unary(method, req).await;
Ok(res)
};
Box::pin(fut)
}
"/node_service.NodeService/RenamePart" => {
#[allow(non_camel_case_types)]
struct RenamePartSvc<T: NodeService>(pub Arc<T>);
@@ -3460,6 +3566,34 @@ pub mod node_service_server {
};
Box::pin(fut)
}
"/node_service.NodeService/SettlePartTransaction" => {
#[allow(non_camel_case_types)]
struct SettlePartTransactionSvc<T: NodeService>(pub Arc<T>);
impl<T: NodeService> tonic::server::UnaryService<super::SettlePartTransactionRequest> for SettlePartTransactionSvc<T> {
type Response = super::SettlePartTransactionResponse;
type Future = BoxFuture<tonic::Response<Self::Response>, tonic::Status>;
fn call(&mut self, request: tonic::Request<super::SettlePartTransactionRequest>) -> Self::Future {
let inner = Arc::clone(&self.0);
let fut = async move { <T as NodeService>::settle_part_transaction(&inner, request).await };
Box::pin(fut)
}
}
let accept_compression_encodings = self.accept_compression_encodings;
let send_compression_encodings = self.send_compression_encodings;
let max_decoding_message_size = self.max_decoding_message_size;
let max_encoding_message_size = self.max_encoding_message_size;
let inner = self.inner.clone();
let fut = async move {
let method = SettlePartTransactionSvc(inner);
let codec = tonic_prost::ProstCodec::default();
let mut grpc = tonic::server::Grpc::new(codec)
.apply_compression_config(accept_compression_encodings, send_compression_encodings)
.apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size);
let res = grpc.unary(method, req).await;
Ok(res)
};
Box::pin(fut)
}
"/node_service.NodeService/RenameFile" => {
#[allow(non_camel_case_types)]
struct RenameFileSvc<T: NodeService>(pub Arc<T>);
+68 -2
View File
@@ -600,6 +600,30 @@ pub fn canonical_rename_part_request_body(
Ok(body.finish())
}
pub fn canonical_prepare_part_transaction_request_body(
request: &proto_gen::node_service::PreparePartTransactionRequest,
) -> Result<Vec<u8>, std::num::TryFromIntError> {
let mut body = CanonicalBodyBuilder::new(b"rustfs-prepare-part-transaction-request-v1\0");
body.push_str(&request.disk)?;
body.push_str(&request.src_volume)?;
body.push_str(&request.src_path)?;
body.push_str(&request.dst_volume)?;
body.push_str(&request.dst_path)?;
body.push_bytes(&request.meta)?;
Ok(body.finish())
}
pub fn canonical_settle_part_transaction_request_body(
request: &proto_gen::node_service::SettlePartTransactionRequest,
) -> Result<Vec<u8>, std::num::TryFromIntError> {
let mut body = CanonicalBodyBuilder::new(b"rustfs-settle-part-transaction-request-v1\0");
body.push_str(&request.disk)?;
body.push_str(&request.volume)?;
body.push_str(&request.path)?;
body.push_bool(request.rollback);
Ok(body.finish())
}
pub fn canonical_delete_volume_request_body(
request: &proto_gen::node_service::DeleteVolumeRequest,
) -> Result<Vec<u8>, std::num::TryFromIntError> {
@@ -637,8 +661,8 @@ pub fn canonical_make_volumes_request_body(
mod disk_mutation_canonical_tests {
use super::proto_gen::node_service::{
DeletePathsRequest, DeleteRequest, DeleteVersionRequest, DeleteVersionsRequest, DeleteVolumeRequest, MakeVolumeRequest,
MakeVolumesRequest, RenameDataRequest, RenameFileRequest, RenamePartRequest, UpdateMetadataRequest, WriteAllRequest,
WriteMetadataRequest,
MakeVolumesRequest, PreparePartTransactionRequest, RenameDataRequest, RenameFileRequest, RenamePartRequest,
SettlePartTransactionRequest, UpdateMetadataRequest, WriteAllRequest, WriteMetadataRequest,
};
use super::*;
@@ -886,6 +910,48 @@ mod disk_mutation_canonical_tests {
}
assert_all_distinct(&bodies);
let prepare_part = PreparePartTransactionRequest {
disk: "d".into(),
src_volume: "sv".into(),
src_path: "sp".into(),
dst_volume: "dv".into(),
dst_path: "dp".into(),
meta: vec![0x01].into(),
};
let mut bodies = vec![canonical_prepare_part_transaction_request_body(&prepare_part).unwrap()];
for mutate in [
|r: &mut PreparePartTransactionRequest| r.disk = "d2".into(),
|r: &mut PreparePartTransactionRequest| r.src_volume = "sv2".into(),
|r: &mut PreparePartTransactionRequest| r.src_path = "sp2".into(),
|r: &mut PreparePartTransactionRequest| r.dst_volume = "dv2".into(),
|r: &mut PreparePartTransactionRequest| r.dst_path = "dp2".into(),
|r: &mut PreparePartTransactionRequest| r.meta = vec![0x02].into(),
] {
let mut request = prepare_part.clone();
mutate(&mut request);
bodies.push(canonical_prepare_part_transaction_request_body(&request).unwrap());
}
assert_all_distinct(&bodies);
let settle_part = SettlePartTransactionRequest {
disk: "d".into(),
volume: "v".into(),
path: "p".into(),
rollback: false,
};
let mut bodies = vec![canonical_settle_part_transaction_request_body(&settle_part).unwrap()];
for mutate in [
|r: &mut SettlePartTransactionRequest| r.disk = "d2".into(),
|r: &mut SettlePartTransactionRequest| r.volume = "v2".into(),
|r: &mut SettlePartTransactionRequest| r.path = "p2".into(),
|r: &mut SettlePartTransactionRequest| r.rollback = true,
] {
let mut request = settle_part.clone();
mutate(&mut request);
bodies.push(canonical_settle_part_transaction_request_body(&request).unwrap());
}
assert_all_distinct(&bodies);
let rename_part = RenamePartRequest {
disk: "d".into(),
src_volume: "sv".into(),
+28
View File
@@ -170,6 +170,32 @@ message RenamePartResponse {
optional Error error = 2;
}
message PreparePartTransactionRequest {
string disk = 1;
string src_volume = 2;
string src_path = 3;
string dst_volume = 4;
string dst_path = 5;
bytes meta = 6;
}
message PreparePartTransactionResponse {
bool success = 1;
optional Error error = 2;
}
message SettlePartTransactionRequest {
string disk = 1;
string volume = 2;
string path = 3;
bool rollback = 4;
}
message SettlePartTransactionResponse {
bool success = 1;
optional Error error = 2;
}
message RenameFileRequest {
string disk = 1;
string src_volume = 2;
@@ -943,7 +969,9 @@ service NodeService {
rpc VerifyFile(VerifyFileRequest) returns (VerifyFileResponse) {};
rpc ReadParts(ReadPartsRequest) returns (ReadPartsResponse) {};
rpc CheckParts(CheckPartsRequest) returns (CheckPartsResponse) {};
rpc PreparePartTransaction(PreparePartTransactionRequest) returns (PreparePartTransactionResponse) {};
rpc RenamePart(RenamePartRequest) returns (RenamePartResponse) {};
rpc SettlePartTransaction(SettlePartTransactionRequest) returns (SettlePartTransactionResponse) {};
rpc RenameFile(RenameFileRequest) returns (RenameFileResponse) {};
rpc Write(WriteRequest) returns (WriteResponse) {};
rpc WriteStream(stream WriteRequest) returns (stream WriteResponse) {};
+1 -1
View File
@@ -30,7 +30,7 @@ Or point the tests at another generated root:
```powershell
$env:RUSTFS_MINIO_FIXTURE_ROOT = '.\rustfs\tmp\minio-fixture-lab-smoke'
cargo +1.96.0 test -p rustfs-rio-v2 --test minio_generated_fixtures -- --ignored
cargo +1.97.1 test -p rustfs-rio-v2 --test minio_generated_fixtures -- --ignored
```
## Scope
+14 -1
View File
@@ -271,7 +271,11 @@ impl EventName {
EventName::ObjectCreatedPut,
],
EventName::ObjectTaggingAll => vec![EventName::ObjectTaggingPut, EventName::ObjectTaggingDelete],
EventName::ObjectRemovedAll => vec![EventName::ObjectRemovedDelete, EventName::ObjectRemovedDeleteMarkerCreated],
EventName::ObjectRemovedAll => vec![
EventName::ObjectRemovedDelete,
EventName::ObjectRemovedDeleteMarkerCreated,
EventName::ObjectRemovedNoOP,
],
EventName::ObjectReplicationAll => vec![
EventName::ObjectReplicationFailed,
EventName::ObjectReplicationComplete,
@@ -639,6 +643,15 @@ mod tests {
assert_eq!(EventName::parse("s3:Scanner:*").unwrap(), EventName::ObjectScannerAll);
}
#[test]
fn test_object_removed_all_includes_noop_extension() {
let expanded = EventName::ObjectRemovedAll.expand();
assert!(expanded.contains(&EventName::ObjectRemovedDelete));
assert!(expanded.contains(&EventName::ObjectRemovedDeleteMarkerCreated));
assert!(expanded.contains(&EventName::ObjectRemovedNoOP));
}
/// `is_removed` must be true for every `ObjectRemoved*` variant and false
/// for everything else.
#[test]
+51 -1
View File
@@ -698,7 +698,7 @@ impl DataUsageCache {
let mut visited = HashSet::new();
visited.insert(hash_path(path).key());
let mut flat = self.flatten_with_guard(root, &mut visited, 0);
if flat.replication_stats.as_ref().is_some_and(|stats| stats.empty()) {
if flat.replication_stats.as_ref().is_some_and(|stats| stats.is_empty()) {
flat.replication_stats = None;
}
Some(flat)
@@ -1574,6 +1574,7 @@ mod tests {
use super::*;
use crate::storage_api::scanner_io::{HTTPRangeSpec, ObjectIO};
use crate::{ScannerGetObjectReader, ScannerPutObjReader};
use rustfs_data_usage::{ReplicationAllStats, ReplicationStats};
use serde_json::Value;
use std::io::Cursor;
use std::pin::Pin;
@@ -2767,6 +2768,55 @@ mod tests {
assert!(flat.children.is_empty());
}
#[test]
fn size_recursive_prunes_empty_and_preserves_threshold_replication_stats() {
let root = hash_path("bucket");
let child = hash_path("bucket/child");
let mut cache = DataUsageCache::default();
cache.replace_hashed(&root, &None, &DataUsageEntry::default());
cache.replace_hashed(
&child,
&Some(root.clone()),
&DataUsageEntry {
replication_stats: Some(ReplicationAllStats::default()),
..Default::default()
},
);
assert!(
cache
.size_recursive("bucket")
.expect("scanner bucket usage should flatten")
.replication_stats
.is_none()
);
cache.replace_hashed(
&child,
&Some(root.clone()),
&DataUsageEntry {
replication_stats: Some(ReplicationAllStats {
targets: HashMap::from([(
"arn:test:threshold".to_string(),
ReplicationStats {
after_threshold_count: 1,
..Default::default()
},
)]),
..Default::default()
}),
..Default::default()
},
);
let flattened = cache.size_recursive("bucket").expect("scanner bucket usage should flatten");
let replication = flattened
.replication_stats
.expect("threshold-only replication stats must survive pruning");
assert_eq!(replication.targets["arn:test:threshold"].after_threshold_count, 1);
}
#[test]
fn checked_flatten_rejects_dangling_child() {
let root_key = hash_path("bucket").key();
+1
View File
@@ -37,6 +37,7 @@ pub const SUFFIX_CRC: &str = "crc";
pub const SUFFIX_TRANSITION_STATUS: &str = "transition-status";
pub const SUFFIX_TRANSITIONED_OBJECTNAME: &str = "transitioned-object";
pub const SUFFIX_TRANSITIONED_VERSION_ID: &str = "transitioned-versionID";
pub const SUFFIX_TRANSITIONED_VERSION_STATE: &str = "transitioned-version-state";
pub const SUFFIX_TRANSITION_TIER: &str = "transition-tier";
pub const SUFFIX_TRANSITION_TIER_DESTINATION_ID: &str = "transition-tier-destination-id";
pub const SUFFIX_RESTORE_OPERATION_ID: &str = "restore-operation-id";
+3
View File
@@ -40,6 +40,9 @@ allow-git = [
# SigV4 payload-checksum fix until it is available in a crates.io release.
# owner: marshawcoco review: 2026-10
"https://github.com/s3s-project/s3s.git",
# Presigned expiry and constant-time authentication fixes pending upstream merge.
# owner: cxymds review: 2026-10
"https://github.com/cxymds/s3s.git",
"https://github.com/apache/datafusion.git",
]
@@ -19,7 +19,9 @@ for later deletion.
- `disk-mutation-body-digest` internode mutating disk RPCs: servers temporarily accept mutating disk RPCs (RenameData, DeleteVersion, DeleteVersions, WriteMetadata, UpdateMetadata, WriteAll, Delete, DeletePaths, RenameFile, RenamePart, DeleteVolume, MakeVolume, MakeVolumes) that carry no signature-bound canonical body digest, so peers from releases that predate body-digest signing remain available during rolling upgrades. Accepted digestless mutations increment the internode body-digest fallback counter; that counter must read zero fleet-wide across a release window before RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT is enabled. Because body-bound requests now consume replay-cache nonces on the receiver, deploy the raised RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY default fleet-wide before enabling strict mode, and watch the internode replay-cache overflow counter for undersized capacity during the rollout. Remove the digestless fallback after the minimum supported RustFS peer version body-binds every mutating disk RPC.
- `heal-status-rpc-v1` node heal status capability: new peers treat an unimplemented BackgroundHealStatus RPC as an explicitly incomplete rolling-upgrade response. Remove the fallback after the minimum supported RustFS peer version implements BackgroundHealStatus.
- `backlog-1316` legacy encrypted multipart range seek: the feature remains opt-in until every server that can initiate, write, or complete multipart uploads supports the candidate-to-final marker protocol and uploadId commit lock, and pre-upgrade multipart uploads have drained. Remove the RUSTFS_ENCRYPTED_RANGE_SEEK switch after the minimum supported release does so; keep the quorum marker and malformed-layout full-read guards permanently.
- `tonic-013-status-render` peer RPC failure classification: internode failures that reach a node only as text (a peer's error_info payload, a status flattened through format!) are classified by matching the rendering of an Unavailable gRPC status. Releases up to 1.0.0-alpha.38 shipped tonic 0.13, which rendered that status as "status: Unavailable, message: ..."; tonic 0.14 renders it as "code: 'The service is currently unavailable', message: ...". Both forms are matched so an older peer's relayed text still marks an unreachable peer offline. Remove the tonic 0.13 form after the minimum supported RustFS peer version ships tonic 0.14 or later.
- `rustfs-5063` pre-beta.9 Local KMS recovery: persisted Local KMS configs from beta.8 and earlier predate the explicit insecure-development flag, and encrypted key files use the legacy SHA-256 KDF. Remove the config fallback after supported upgrades have rewritten or explicitly resaved all pre-beta.9 configs with the development-default field, and remove the legacy KDF after supported upgrades have rewritten all pre-beta.9 Local KMS key files with explicit at-rest protection.
- `sse-local-dek-json-v1` legacy local SSE DEK decoding: releases before the JSON envelope wrote wrapped DEKs as `base64(nonce):base64(ciphertext)`, so readers retain that decoder while all new writes use the versioned JSON envelope. Remove the colon decoder after the minimum supported direct-upgrade release writes JSON envelopes and migration tooling has rewritten every retained legacy object.
## Review Checklist
+10
View File
@@ -9,6 +9,16 @@ failure pattern reported in rustfs/rustfs#4304.
> format. Replacing the executable and restarting is safe; no migration step
> runs on startup.
> [!WARNING]
> The release that switches local SSE wrapped DEKs from the legacy
> `base64(nonce):base64(ciphertext)` representation to the versioned JSON
> envelope is a deliberate exception. Do not run that release together with
> an older RustFS version: older nodes cannot read objects written with the
> JSON envelope. Freeze every source of object mutation, including client
> writes and background lifecycle or replication work, upgrade every node,
> and then resume traffic. Downgrading or rolling back after new encrypted
> objects are written is not supported.
## TL;DR
- **Rolling restart (no downtime):** restart **one node at a time**, and wait
-1
View File
@@ -30,7 +30,6 @@
systems = [
"x86_64-linux"
"aarch64-linux"
"x86_64-darwin"
"aarch64-darwin"
];
forAllSystems = nixpkgs.lib.genAttrs systems;
+42 -4
View File
@@ -36,6 +36,8 @@ use tracing::{error, info, instrument, warn};
/// Path to store KMS configuration in the cluster metadata
const KMS_CONFIG_PATH: &str = "config/kms_config.json";
const STATIC_KMS_LOCAL_CONFIG_REQUIRED: &str =
"Static KMS must be configured through RUSTFS_KMS_STATIC_SECRET_KEY or RUSTFS_KMS_STATIC_SECRET_KEY_FILE";
const LOG_COMPONENT_ADMIN: &str = "admin";
const LOG_SUBSYSTEM_KMS: &str = "kms";
const EVENT_ADMIN_KMS_DYNAMIC_STATE: &str = "admin_kms_dynamic_state";
@@ -106,9 +108,25 @@ fn normalize_configure_request_auth(
Ok(())
}
fn ensure_kms_config_persistable(config: &KmsConfig) -> Result<(), String> {
if matches!(&config.backend_config, rustfs_kms::BackendConfig::Static(_)) {
return Err(STATIC_KMS_LOCAL_CONFIG_REQUIRED.to_string());
}
Ok(())
}
fn ensure_kms_request_persistable(request: &ConfigureKmsRequest) -> Result<(), String> {
if matches!(request, ConfigureKmsRequest::Static(_)) {
return Err(STATIC_KMS_LOCAL_CONFIG_REQUIRED.to_string());
}
Ok(())
}
/// Save KMS configuration to cluster storage
#[instrument(skip(config))]
async fn save_kms_config(config: &KmsConfig) -> Result<(), String> {
ensure_kms_config_persistable(config)?;
let context = current_app_context();
let Some(store) = current_object_store_handle_for_context(context.as_deref()) else {
return Err("Storage layer not initialized".to_string());
@@ -332,12 +350,16 @@ impl Operation for ConfigureKmsHandler {
);
let service_manager = kms_service_manager_from_context();
let existing_config = service_manager.get_config().await;
let existing_config = service_manager.get_redacted_config().await;
if let Err(e) = normalize_configure_request_auth(&mut configure_request, existing_config.as_ref()) {
return Ok(S3Response::new((StatusCode::BAD_REQUEST, Body::from(e))));
}
if let Err(e) = ensure_kms_request_persistable(&configure_request) {
return Ok(S3Response::new((StatusCode::BAD_REQUEST, Body::from(e))));
}
// Convert request to KmsConfig
let kms_config = configure_request.to_kms_config();
@@ -753,7 +775,7 @@ impl Operation for GetKmsStatusHandler {
let service_manager = kms_service_manager_from_context();
let status = service_manager.get_status().await;
let config = service_manager.get_config().await;
let config = service_manager.get_redacted_config().await;
// Get backend type and health status
let backend_type = config.as_ref().map(|c| c.backend.clone());
@@ -873,12 +895,16 @@ impl Operation for ReconfigureKmsHandler {
);
let service_manager = kms_service_manager_from_context();
let existing_config = service_manager.get_config().await;
let existing_config = service_manager.get_redacted_config().await;
if let Err(e) = normalize_configure_request_auth(&mut configure_request, existing_config.as_ref()) {
return Ok(S3Response::new((StatusCode::BAD_REQUEST, Body::from(e))));
}
if let Err(e) = ensure_kms_request_persistable(&configure_request) {
return Ok(S3Response::new((StatusCode::BAD_REQUEST, Body::from(e))));
}
// Convert request to KmsConfig
let kms_config = configure_request.to_kms_config();
@@ -960,7 +986,7 @@ impl Operation for ReconfigureKmsHandler {
#[cfg(test)]
mod tests {
use super::{decode_persisted_kms_config, kms_configure_actions, kms_service_control_actions};
use super::{decode_persisted_kms_config, ensure_kms_config_persistable, kms_configure_actions, kms_service_control_actions};
use rustfs_policy::policy::action::{Action, AdminAction, KmsAction};
use tempfile::TempDir;
@@ -1046,4 +1072,16 @@ mod tests {
assert!(!config.allow_insecure_dev_defaults);
assert!(config.validate().is_ok());
}
#[test]
fn static_kms_config_is_not_persisted_with_cluster_configuration() {
use base64::Engine as _;
let config = rustfs_kms::KmsConfig::static_kms(
"static-key".to_string(),
base64::engine::general_purpose::STANDARD.encode([0x5au8; 32]),
);
assert!(ensure_kms_config_persistable(&config).is_err());
}
}
+2 -2
View File
@@ -193,7 +193,7 @@ impl Operation for KmsStatusHandler {
hit_count: hits,
miss_count: misses,
});
let config = kms_service_manager_from_context().get_config().await;
let config = kms_service_manager_from_context().get_redacted_config().await;
let response = KmsStatusResponse {
backend_type: config
@@ -243,7 +243,7 @@ impl Operation for KmsConfigHandler {
};
let config = kms_service_manager_from_context()
.get_config()
.get_redacted_config()
.await
.ok_or_else(|| s3_error!(InternalError, "KMS config not available"))?;
+37 -2
View File
@@ -33,7 +33,7 @@ use crate::admin::runtime_sources::{
};
use crate::admin::storage_api::access::{ReqInfo, authorize_request, spawn_traced};
use crate::admin::storage_api::contract::bucket::{BucketOperations, BucketOptions};
use crate::auth::{check_key_valid, get_session_token};
use crate::auth::{check_key_valid, constant_time_eq, get_session_token};
use crate::error::ApiError;
use crate::license::license_check;
use crate::server::{
@@ -905,7 +905,9 @@ fn validate_object_lambda_response_auth_headers(headers: &HeaderMap, output_rout
.and_then(|value| value.to_str().ok())
.map(str::trim);
if route == Some(output_route) && token == Some(output_token) {
if route.is_some_and(|route| constant_time_eq(route, output_route))
&& token.is_some_and(|token| constant_time_eq(token, output_token))
{
return Ok(());
}
@@ -4351,6 +4353,39 @@ mod tests {
let err = validate_object_lambda_response_auth_headers(&mismatched, "route-123", "token-456")
.expect_err("mismatched auth headers should fail");
assert_eq!(err.code(), &S3ErrorCode::InvalidRequest);
for (route, token) in [
("Route-123", "token-456"),
("route-124", "token-456"),
("route-1234", "token-456"),
("route-123", "Token-456"),
("route-123", "token-457"),
("route-123", "token-4567"),
] {
let mut headers = HeaderMap::new();
headers.insert(
"x-amz-request-route",
HeaderValue::try_from(route).expect("test route must be a valid header"),
);
headers.insert(
"x-amz-request-token",
HeaderValue::try_from(token).expect("test token must be a valid header"),
);
assert!(
validate_object_lambda_response_auth_headers(&headers, "route-123", "token-456").is_err(),
"first-byte, last-byte, and length mismatches must fail: {route}/{token}"
);
}
}
#[test]
fn object_lambda_auth_headers_use_constant_time_helper() {
let source = include_str!("router.rs");
let production = source.split_once("#[cfg(test)]").map_or(source, |(production, _)| production);
assert!(!production.contains("route == Some(output_route)"));
assert!(!production.contains("token == Some(output_token)"));
assert!(production.contains("constant_time_eq(route, output_route)"));
assert!(production.contains("constant_time_eq(token, output_token)"));
}
#[test]
+6 -4
View File
@@ -1810,7 +1810,7 @@ impl DefaultBucketUsecase {
client_info,
);
let read_allowed = PolicySys::is_allowed(&BucketPolicyArgs {
let read_allowed = PolicySys::try_is_allowed(&BucketPolicyArgs {
bucket: &bucket,
action: Action::S3Action(S3Action::ListBucketAction),
is_owner: false,
@@ -1819,9 +1819,10 @@ impl DefaultBucketUsecase {
conditions: &conditions,
object: "",
})
.await;
.await
.map_err(ApiError::from)?;
let write_allowed = PolicySys::is_allowed(&BucketPolicyArgs {
let write_allowed = PolicySys::try_is_allowed(&BucketPolicyArgs {
bucket: &bucket,
action: Action::S3Action(S3Action::PutObjectAction),
is_owner: false,
@@ -1830,7 +1831,8 @@ impl DefaultBucketUsecase {
conditions: &conditions,
object: "",
})
.await;
.await
.map_err(ApiError::from)?;
let mut is_public = read_allowed || write_allowed;
let ignore_public_acls = match metadata_sys::get_public_access_block_config(&bucket).await {
@@ -198,13 +198,9 @@ async fn data_usage_endpoint_serves_snapshot_without_live_listing() {
first_info.total_free_capacity
);
assert_eq!(
(first_info.total_capacity, first_info.total_free_capacity, first_info.total_used_capacity,),
(
second_info.total_capacity,
second_info.total_free_capacity,
second_info.total_used_capacity,
),
"repeated data usage requests must report stable capacity values"
second_info.total_used_capacity,
second_info.total_capacity.saturating_sub(second_info.total_free_capacity),
"server used capacity must stay internally consistent on repeated requests"
);
// The endpoint must serve the seeded snapshot numbers, not recomputed ones.

Some files were not shown because too many files have changed in this diff Show More