Compare commits

...

37 Commits

Author SHA1 Message Date
Chris An 509869cfc5 fix(tier): recover orphaned prepared mutations (#9246) 2026-11-09 22:18:14 +08:00
Ramakrishna Chilaka 395876ccd0 fix(ecstore): remove hardlinked backup during delete rollback (#9239)
When a fused recoverable delete is interrupted after publishing the transaction directory and before updating xl.meta, xl.meta and <transaction>/xl.meta.bkp remain hardlinks to the same inode. Under POSIX rename semantics, renaming xl.meta.bkp over xl.meta during rollback succeeds as a no-op without unlinking xl.meta.bkp, which leaves xl.meta.bkp in the transaction directory after journal removal and strands the transaction in DeleteRecoveryState::LegacyAmbiguous.

Unlink xl.meta.bkp after syncing the restored object directory and before syncing the rollback directory and removing journal markers, and extend delete_recovery_fused_prepare_checkpoint_reopens_before_head_publication to preserve the hardlink and verify rollback completion and retry.
2026-11-09 06:41:04 +08:00
Chris An 6083fbbeea fix(catalog): enumerate table owners without skipping namespaces (#9242) 2026-11-09 06:38:58 +08:00
Chris An 42d3267d88 fix(ecstore): decouple pool metadata health from the authority gate (#9241)
* fix(ecstore): share immutable pool metadata authority reads

* fix(ecstore): read pool metadata authority without the save gate

Health inferred pool metadata write authority from acquiring the local
authority gate within 100 ms. Sharing the gate among read-only
admissions still left held writers, and writers queued behind a reader
parked on the pool.bin fence, turning into cluster-write 503s.

Each writer now publishes the authority it leaves before releasing the
gate, and health reads that published state without acquiring it.
Transaction latches stay shared with the published copy; recovery
clears the latch and its failure under one lock so inspection never
observes a half-cleared state. The inspection-timeout path that can no
longer occur (check_timed_out, the check_timeout admin state, the
pool_metadata_check_timeout reason, the node timeout cache and its
metric) is removed.
2026-11-09 06:38:30 +08:00
Chris An 1d35e381b7 fix(rpc): bound replay memory and atomically admit signed requests (#9238)
* fix(rpc): index nonce expiry by signed validity

* fix(rpc): size replay cache from allocated storage layout

* fix(rpc): make replay admission atomic and release expired tables

* fix(rpc): preserve replay admission refusal at the S3 boundary
2026-11-09 06:36:55 +08:00
Chris An 0b109b4885 fix(select): bound LZ4 scratch and Arrow schema expansion (#9237)
* fix(select): admit legacy and linked LZ4 working memory

* fix(select): bound embedded Arrow schema expansion
2026-11-09 06:36:44 +08:00
cxymds 1dea00ffa7 ci: repair e2e selection and verify tool downloads via API (#9240)
* ci: align full e2e selection with retired catalog test

* ci: verify pinned tools through GitHub release asset API

* ci: fetch pinned typos through the release asset API

* ci: download pinned actionlint through the release asset API

* ci: ensure GitHub CLI exists before pinned mc download

* ci: budget full quick checks after tool installation
2026-11-08 14:09:26 +00:00
cxymds addf7a6219 fix: reduce scanner backoff and classify downstream closures (#9235) 2026-11-08 02:18:17 -05:00
Ramakrishna Chilaka 420cf32b2a perf(ecstore): optimize PUT, DELETE, and multipart commit hotpaths (#9227) 2026-11-08 02:12:40 -05:00
Chris An 907410b2f5 fix(select): bound Parquet allocations before decoding (#9231) 2026-11-08 01:36:12 -05:00
Chris An f10079f1a3 fix(rpc): size and reserve replay cache for compact entries (#9233) 2026-11-08 01:35:58 -05:00
Chris An a6c3016493 fix(cache): respect process cgroup memory limits (#9232) 2026-11-08 01:35:49 -05:00
Hauser Zhi 68fb52b752 chore(deps): update flake.lock (#9236) 2026-11-08 01:34:01 -05:00
cxymds 16f618d41b fix: correct scanner pacing and unverified MRF outcomes (#9225)
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-08 06:48:25 +08:00
Henry Guo 565d262ede refactor(table-catalog): retire durable-strong backing (#9130)
* refactor(table-catalog): retire durable-strong backing

* fix(table-catalog): avoid backup locks during S3 authorization

* fix(table-catalog): preserve directory and maintenance behavior

* fix(table-catalog): back up retained v1 snapshot history

* test(table-catalog): cover retained v1 backup recovery

Exercise backup inventory sealing and restore after a v1-to-v2 upgrade, including missing or corrupt references, nonzero inline snapshot sequences, v2 manifests in retained v1 history, and unchanged commit admission rules.

Clarify when direct maintenance can replace reports and discard retry budgets, and require quiescing scheduler admission and workers before invocation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris An <anzhengchao@gmail.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-08 05:03:55 +08:00
Chris An 7b010ddd70 fix(ci,ecstore): restore validation and defer capacity exhaustion (#9223) 2026-11-07 15:17:25 -05:00
Chris An 272b6587e0 fix(ci): bound workspace archive builds to two workers (#9222) 2026-11-07 06:32:41 -05:00
Chris An e3ed2cf2f4 fix(decommission): reserve owned source usage during activation (#9220) 2026-11-07 01:49:42 -05:00
Chris An bf03b1117a fix(swift): preserve bucket metadata with Swift tag values (#9218) 2026-11-07 01:49:12 -05:00
Boyi (Elliot) Li c3fe0f9649 test(rio,ecstore): add negative-branch durability unit tests (#9216)
Expand positive and negative branch unit test coverage across durability-critical modules:
- crates/rio/src/checksum.rs: Checksum::matches (unset hasher, multipart want_parts mismatch, digest mismatch), new_with_type, as_map, obj_type, from_header, add_part (CRC32/CRC32C/CRC64_NVME combine + error branches), and get_content_checksum header validation.
- crates/rio/src/hash_reader.rs: HashReader::new wrapping invariants (already-read inner reader, conflicting MD5 checksum, conflicting positive size, unbounded size propagation) and checksum helper error paths.
- crates/ecstore/src/set_disk/ops/bitrot_self_verify.rs: verify_written_bitrot_shards missing inline data, missing disk shard, and under-quorum rejection.
- crates/ecstore/src/erasure/coding/encode.rs: read_full_buf_or_eof immediate read error, InvalidData preservation after partial fill, and UnexpectedEof normalization.
- crates/ecstore/src/io_support/shard_integrity.rs: ShardVerifier out-of-bounds coding_index, empty proof sources fail-closed, and spilled (>1 MiB) index disk write + verify_index_file + restore_proof_replicas round-trip and all-corrupt fail-closed boundary.
- crates/ecstore/src/services/integrity/tests.rs: JobRequest::validate boundary/format checks with exact error messages and model::protection / model::unsupported classifications.
2026-11-07 02:38:43 +00:00
唐小鸭 6e31f6a183 fix(ecstore): complete and abort native multipart uploads on Windows (#9191) 2026-11-06 22:36:16 -05:00
Hauser Zhi d66d144309 feat(observability): record unary read RPC durations (#9202)
* feat(observability): record unary read RPC durations

Record complete server-side durations for ReadMultiple and BatchReadVersion, including error and cancellation paths.\n\nCo-Authored-By: heihutu <heihutu@gmail.com>\nCo-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(observability): cover unary duration handlers

Exercise ReadMultiple and BatchReadVersion missing-disk paths and assert operation duration histograms.\n\nCo-Authored-By: heihutu <heihutu@gmail.com>\nCo-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(observability): cache unary read RPC duration handles

Reuse cached duration histograms for ReadMultiple and BatchReadVersion once
the stable server label is available. Preserve the existing ReadVersion
handle and the fallback for other operations and backends.

Add an integration regression for handle reuse, startup fallback, labels,
and recorded values using the production cache path.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(observability): separate server duration from client RTT

Record unary ReadMultiple and BatchReadVersion handler lifetimes in a
separate server duration histogram while preserving existing client RTT
series and allocation-free cached handles for both measurements.

Cover client/server sample separation, cache reuse, startup and backend
fallbacks, and missing-disk handlers. Document the new server metric.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-07 09:16:18 +08:00
cxymds 150836d707 feat(tier): support native Azure Blob endpoints (#9200)
* feat(tier): support native Azure Blob endpoints

* fix(tier): guard Azure version cleanup and prefix limits

* fix(tier): preserve Azure quoted headers and document version gate

* fix(tier): handle Azure forbidden current-version deletion

* test(tier): control transition commit lock failure timing
2026-11-07 09:16:06 +08:00
Henry Guo e4057b6746 fix(table-catalog): reject storage credentials for catalog access (#9185)
* fix(table-catalog): reject storage credentials for catalog access

* fix(table-catalog): avoid growing the s3s error macro footprint

Use the existing S3Error constructor for the storage-credential rejection
while preserving its AccessDenied code and message. Keep the merged PR
within the current s3s footprint baseline without relaxing the guard.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-07 08:07:58 +08:00
Hauser Zhi cfdd705b62 fix(cors): preserve readiness and complete cache selectors (#9207)
Use a direct response future for originless requests and forward through
the polled-ready service. Nominate every retained CORS header input and
transfer evaluated headers without rebuilding the map.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-07 08:07:04 +08:00
Hauser Zhi a1d0fa41d0 fix: preserve usage generations and unify request tracing (#9204)
* fix(ecstore): prevent cached usage snapshot rollback

Order refreshed snapshots by their authoritative lineage before replacing cached usage. Share the rule with early bucket refreshes while retaining quota baselines, invalidation fencing, legitimate deletions, and clock recovery.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(server): skip trace propagation for disabled RPC spans

Avoid extracting and assigning a remote parent when RPC tracing is disabled after authentication and admission. Cover disabled logging and the production HTTP span propagation path.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve usage ordering across refresh recovery

Retain the last good snapshot independently of negative-cache status. Order observations within the same authoritative baseline by their own coordinated epoch and cycle before wall time, including clock rollback and recovery. Capture persisted baseline identity without cloning its bucket maps.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(server): centralize request trace propagation

Let the HTTP span factory own parent extraction and assignment. Use actual span filtering, preserve span-only subscribers, and remove repeated tracing work from RPC authentication.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-07 08:06:47 +08:00
Chris An 7d4982f206 fix(storage): fence bounded deletes during rolling upgrades (#9214) 2026-11-07 08:06:28 +08:00
Chris An 1a7fb2461d fix(webdav): join connection and reload tasks on shutdown (#9213) 2026-11-07 08:06:14 +08:00
Chris An e1b9d1cb94 fix(ecstore): preserve ownership across multi-pool recovery (#9212) 2026-11-07 08:06:04 +08:00
Chris An 85313fc06f fix(runtime): release retained replay and allocator memory (#9211) 2026-11-07 08:05:46 +08:00
cxymds b5f3207378 fix(lock): preserve acquisition progress during multipart uploads (#9209)
* fix(lock): retain slow responses within the quorum deadline

* fix(lock): retain partial quorum timeout diagnostics

* fix(lock): avoid guard registry scheduling convoys

* chore(ecstore): trace metadata transaction lock purposes

* fix(scanner): keep capability probes on read fences

* fix(lock): preserve renewal and contention progress
2026-11-07 08:05:26 +08:00
Hauser Zhi 5490e493ab fix(s3): retain protected multipart ETags in object listings (#9208)
* fix(s3): retain protected multipart ETags in object listings

Record encrypted part ETag provenance, aggregate it at read quorum, and attest completed multipart ETags for keyless listings. Preserve legacy fingerprint protection and test versioned SSE-KMS and SSE-C upload, listing, and replay paths.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(s3): stream multipart ETag aggregation

Restore the preserved ETag fast path when selected parts lack protected provenance. Stream native part digests through a fixed stack buffer, preserve legacy hex-or-raw semantics, and strengthen exact-output and override regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): preserve encrypted ETag proof during data movement

Capture quorum-backed source proof before multipart completion metadata updates, require exact preserved ETag and version identity, and retain proof only when publication keeps the source version. Cover real two-pool rebalance, null versions, legacy sources, and injected or rewritten identities.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): retain encrypted multipart ETag proof on restore

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-11-07 08:04:54 +08:00
Hauser Zhi 134820c1e0 test(storage): reproduce HTTPS rejoin readiness in Docker (#9206)
* test(storage): reproduce HTTPS rejoin readiness in Docker

Compare cluster-only and node-ready/write gates on immutable native binaries. Preserve first-failure and latest-acknowledged-content evidence across recovery, capture lock diagnostics before restart, and limit cleanup to verified owned resources.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): unify Docker repro ownership and controls

Use typed resource absence and one verified container inventory for staging, partial setup and retained nodes. Reject inherited experiment overrides before creating resources and remove duplicated state.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): guard staging volumes and interrupted capture

Verify volume ownership before privileged staging and disable implicit image-content copying. Preserve failed evidence and continue owned finalization when capture or cleanup is interrupted.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): capture live rejoin logs and verify trace coverage

Read live tmpfs logs inside the verified container and use one explicit stdout sink for diagnostics. Record bounded capture provenance and fail diagnostic coverage when acquisition, guard-drop or physical-release records are missing, without masking HTTP failures.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): reject inherited aliases and finalize on SIGTERM

Reject inherited MINIO compatibility settings before Docker resource creation. Route SIGTERM through interrupt-aware finalization and restore the previous handler after each run. Cover real process signals and failure preservation.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): normalize legacy controls and termination

Reject inherited MINIO compatibility aliases in neutral runtime images. Bridge run-mode SIGTERM into the existing interruption finalization and restore the caller signal handler on return or exception.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-07 08:04:28 +08:00
Chris An 2b9217b6f9 test(e2e): fix quota and conditional write fixtures (#9210) 2026-11-07 08:03:00 +08:00
Hauser Zhi 70a46d0e8c fix(webdav): support configured browser CORS access (#9205)
* fix(webdav): support configured browser CORS access

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser Zhi <housemecn@gmail.com>

* perf(webdav): streamline CORS policy and preflight parsing

Borrow connection-owned policy state and clone the backend only for actual requests. Validate borrowed header names and normalize HTTP lists when required.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser Zhi <housemecn@gmail.com>

---------

Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-06 14:01:06 -05:00
Hauser Zhi a910ef0dd0 fix(cors): isolate cache variants and streamline policy evaluation (#9203)
Apply cache variation at the shared response boundary, preserve existing
Vary selectors, and reuse parsed global policy and compact CORS inputs.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-06 12:40:11 -05:00
Chris An 430ade191f fix(scanner): bound wide bucket-root fanout memory (#9193)
* fix(scanner): bound wide bucket-root fanout memory (#8961)

Root prefixes were excluded from ordinary compaction and force_compact only
flattened the root above a hardcoded 250k child ceiling, so buckets with
tens of thousands of top-level prefixes kept one cache entry each after
#8971. Compact wide roots outside checkpointed forward sweeps, and honor
force_compact's limit for root fanout.

* fix(scanner): keep root fanout on interrupted force_compact

Interrupted sweeps seal a resume frontier after force_compact. Applying
the ordinary 10k root limit there flattened frontier-named children and
made the next cycle replay completed prefixes. Use the emergency FORCE
ceiling on the interrupted path; successful completions still honor the
ordinary limit. Drop clippy assertions_on_constants in the new tests.

Signed-off-by: Chris An <anzhengchao@gmail.com>

* test(scanner): seal reloaded frontier after interrupted compaction

Const assertions keep the wide-root preconditions without tripping
clippy::assertions_on_constants. The interrupted and finished sweep
paths share one limit helper, and a cache wider than the ordinary
10k limit still seals its frontier child after reload.

Co-authored-by: Chris An <overtrue@users.noreply.github.com>

* fix(scanner): isolate unreadable children during root compaction

* fix(scanner): isolate compacted namespace child errors

* fix(scanner): preserve compacted scan accounting and coverage

Commit isolated namespace child usage only after successful traversal, while retaining completed accounting on cancellation and resource deferral. Synchronize reduced root totals before the cancellation-prone flush boundary.

Treat skipped namespaces as incomplete coverage and invalidate legacy sampling baselines until a fresh verification pass succeeds. Keep provisional usage out of progress updates and retain the upstream symlink regression.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): preserve legacy retry coverage

* test(scanner): regress cancel overwrite after wide-root reduce

houseme's 714f590 already refreshes the root accumulator from the
compacted cache entry before flush_pacing. Add a unit regression that
simulates the interrupt replace_hashed path so a future regression
cannot reintroduce dangling child links over the aggregate.

Signed-off-by: overtrue <anzhengchao@gmail.com>

---------

Signed-off-by: Chris An <anzhengchao@gmail.com>
Signed-off-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Chris An <overtrue@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-06 12:36:25 -05:00
248 changed files with 37914 additions and 26636 deletions
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=219ac11e720b3f54dfee4d5d5e95171785ba159b2fe01db3037a8f5061c1ad32
sha256-linux=2d34698cc8ec52d5b6c37ac612919a1d21f36cd56a1af539a7522a4eea518e26
sha256-darwin=9901a4950099b94d3d2618524d540f1ed4bf6023044903886f98b6c4bf5f30fb
sha256-linux=09fd5f48f5556c9fc576357222dccd7ebb0b90b790798f1834db2ec5d67b4952
+2
View File
@@ -54,6 +54,7 @@ script-tests: ## Run shell script tests
./scripts/check_embedded_secrets.sh --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_download_release_asset.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
@@ -71,6 +72,7 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_nightly_candidate.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_repro_rejoin_readiness_docker.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_validation_registry.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_validation_runner.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_retain_validation_evidence.py
+2 -2
View File
@@ -110,7 +110,7 @@ threads-required = "num-test-threads"
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_object_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.default.overrides]]
@@ -410,7 +410,7 @@ threads-required = "num-test-threads"
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_object_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.ci.overrides]]
+40
View File
@@ -0,0 +1,40 @@
# syntax=docker/dockerfile:1.6
# Copyright 2026 RustFS Team
# Licensed under the Apache License, Version 2.0.
# A compatible native runtime with Python 3.11+, getent and stat is required by
# the isolated fixture's staging/provenance checks. This is a test image.
ARG RUNTIME_BASE=python:3.13-slim-trixie
FROM ${RUNTIME_BASE}
ARG SOURCE_REVISION
ARG BINARY_SHA256
ARG BUILD_PROFILE=dev
COPY rustfs /usr/bin/rustfs
RUN python3 - "$SOURCE_REVISION" "$BINARY_SHA256" "$BUILD_PROFILE" <<'PY'
import hashlib
import os
import re
import shutil
import sys
source, digest, profile = sys.argv[1:]
if not re.fullmatch(r"[0-9a-f]{40}", source):
raise SystemExit("invalid source revision")
if not re.fullmatch(r"[0-9a-f]{64}", digest):
raise SystemExit("invalid binary digest")
if profile not in ("dev", "release"):
raise SystemExit("invalid build profile")
with open("/usr/bin/rustfs", "rb") as binary:
if hashlib.file_digest(binary, "sha256").hexdigest() != digest:
raise SystemExit("binary digest mismatch")
for tool in ("getent", "stat"):
if shutil.which(tool) is None:
raise SystemExit(f"missing runtime tool: {tool}")
os.chmod("/usr/bin/rustfs", 0o755)
PY
LABEL io.rustfs.repro.source_revision="${SOURCE_REVISION}" \
io.rustfs.repro.binary_sha256="${BINARY_SHA256}" \
io.rustfs.repro.build_profile="${BUILD_PROFILE}"
USER 10001:10001
ENTRYPOINT ["/usr/bin/rustfs"]
+4 -3
View File
@@ -28,11 +28,12 @@ runs:
- name: Install actionlint
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/../setup/download_release_asset.py
run: |
actionlint_dir="$(mktemp -d "${RUNNER_TEMP}/actionlint.XXXXXX")"
curl --fail --location --silent --show-error \
--output "$actionlint_dir/actionlint.tar.gz" \
https://github.com/rhysd/actionlint/releases/download/v1.7.12/actionlint_1.7.12_linux_amd64.tar.gz
python3 "$ASSET_DOWNLOADER" --repository rhysd/actionlint --tag v1.7.12 --asset actionlint_1.7.12_linux_amd64.tar.gz --output "$actionlint_dir/actionlint.tar.gz" --sha256 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8
echo "8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8 $actionlint_dir/actionlint.tar.gz" | sha256sum --check --status
tar -xzf "$actionlint_dir/actionlint.tar.gz" -C "$actionlint_dir" actionlint
rm "$actionlint_dir/actionlint.tar.gz"
+23 -5
View File
@@ -66,10 +66,7 @@ inputs:
runs:
using: "composite"
steps:
# protobuf-compiler is deliberately absent: the setup-protoc step below
# installs 35.1 into the tool cache and prepends it to PATH, so the apt
# build (older, and never version-matched) was shadowed on every run and
# simply never used.
# Use the pinned code generators rather than Ubuntu's older packages.
- name: Install system dependencies (Ubuntu)
if: runner.os == 'Linux'
shell: bash
@@ -79,7 +76,9 @@ runs:
build-essential \
pkg-config \
libssl-dev \
ripgrep
ripgrep \
python3 \
gh
# musl-gcc is needed by the native musl release leg, and zip/unzip by the
# release packaging steps. No CI test lane touches any of them.
@@ -88,13 +87,32 @@ runs:
shell: bash
run: sudo apt-get install -y musl-tools zip unzip
- name: Install pinned code generators through GitHub API (Linux x64)
if: runner.os == 'Linux' && runner.arch == 'X64'
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/download_release_asset.py
run: |
python3 "$ASSET_DOWNLOADER" --repository protocolbuffers/protobuf --tag v35.1 --asset protoc-35.1-linux-x86_64.zip --output "$RUNNER_TEMP/protoc.zip"
python3 "$ASSET_DOWNLOADER" --repository google/flatbuffers --tag v25.12.19 --asset 'Linux.flatc.binary.g++-13.zip' --output "$RUNNER_TEMP/flatc.zip"
python3 -m zipfile -e "$RUNNER_TEMP/protoc.zip" "$RUNNER_TEMP/protoc"
python3 -m zipfile -e "$RUNNER_TEMP/flatc.zip" "$RUNNER_TEMP/flatc"
chmod +x "$RUNNER_TEMP/protoc/bin/protoc" "$RUNNER_TEMP/flatc/flatc"
"$RUNNER_TEMP/protoc/bin/protoc" --version
"$RUNNER_TEMP/flatc/flatc" --version
echo "$RUNNER_TEMP/protoc/bin" >> "$GITHUB_PATH"
echo "$RUNNER_TEMP/flatc" >> "$GITHUB_PATH"
- name: Install protoc
if: runner.os != 'Linux' || runner.arch != 'X64'
uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
with:
version: "35.1"
repo-token: ${{ github.token }}
- name: Install flatc
if: runner.os != 'Linux' || runner.arch != 'X64'
uses: Nugine/setup-flatc@698800de72a96bfb22cf60431dc21a2ff9a7e07b # v1
with:
version: "25.12.19"
@@ -0,0 +1,84 @@
#!/usr/bin/env python3
"""Download a named GitHub release asset through its API and verify its digest."""
import argparse
import hashlib
import json
from pathlib import Path
import re
import subprocess
import sys
import tempfile
from urllib.parse import quote
def download(repository, tag, name, output, expected_sha256=None):
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*/[A-Za-z0-9][A-Za-z0-9_.-]*", repository):
raise ValueError("invalid release repository")
if not tag or not name or "/" in name or "\\" in name:
raise ValueError("invalid release tag or asset name")
if expected_sha256 is not None and not re.fullmatch(r"[0-9a-f]{64}", expected_sha256):
raise ValueError("invalid pinned asset digest")
result = subprocess.run(
["gh", "api", "--hostname", "github.com", f"repos/{repository}/releases/tags/{quote(tag, safe='')}"],
check=True, capture_output=True, text=True, timeout=60,
)
release = json.loads(result.stdout)
if not isinstance(release, dict) or release.get("tag_name") != tag:
raise ValueError("release metadata does not match the requested tag")
assets = release.get("assets")
if not isinstance(assets, list) or any(not isinstance(asset, dict) for asset in assets):
raise ValueError("invalid release asset inventory")
matches = [asset for asset in assets if asset.get("name") == name]
if len(matches) != 1:
raise ValueError("release must contain exactly one named asset")
asset = matches[0]
asset_id, size, digest = asset.get("id"), asset.get("size"), asset.get("digest")
if type(asset_id) is not int or asset_id <= 0 or type(size) is not int or not 0 < size <= 512 * 1024 * 1024:
raise ValueError("invalid release asset identity or size")
if not isinstance(digest, str) or not re.fullmatch(r"sha256:[0-9a-f]{64}", digest):
raise ValueError("release asset has no valid SHA-256 digest")
sha256 = digest.removeprefix("sha256:")
if expected_sha256 is not None and expected_sha256 != sha256:
raise ValueError("release digest differs from the pinned digest")
output = Path(output)
output.parent.mkdir(parents=True, exist_ok=True)
temporary = None
try:
with tempfile.NamedTemporaryFile(dir=output.parent, prefix=".release-asset-", delete=False) as stream:
temporary = Path(stream.name)
# gh authenticates to the API; its HTTP client strips credentials on
# cross-host redirects. No browser URL or token is supplied in argv.
subprocess.run(
["gh", "api", "--hostname", "github.com", "--header", "Accept: application/octet-stream", f"repos/{repository}/releases/assets/{asset_id}"],
check=True, stdout=stream, timeout=300,
)
with temporary.open("rb") as stream:
actual = hashlib.file_digest(stream, "sha256").hexdigest()
if temporary.stat().st_size != size or actual != sha256:
raise ValueError("downloaded release asset failed size or SHA-256 verification")
temporary.replace(output)
print(f"Verified release asset: {repository} {tag} {name} sha256={actual}")
finally:
if temporary is not None:
temporary.unlink(missing_ok=True)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--repository", required=True)
parser.add_argument("--tag", required=True)
parser.add_argument("--asset", required=True)
parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--sha256")
args = parser.parse_args()
try:
download(args.repository, args.tag, args.asset, args.output, args.sha256)
except (ValueError, KeyError, TypeError, json.JSONDecodeError, subprocess.SubprocessError, OSError) as error:
print(f"Release asset download failed: {error}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
+18 -4
View File
@@ -142,15 +142,23 @@ jobs:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install pinned typos through GitHub API
env:
GH_TOKEN: ${{ github.token }}
TYPOS_VERSION: 1.50.2
run: |
python3 .github/actions/setup/download_release_asset.py --repository crate-ci/typos --tag "v${TYPOS_VERSION}" --asset "typos-v${TYPOS_VERSION}-x86_64-unknown-linux-musl.tar.gz" --output "$RUNNER_TEMP/typos.tar.gz"
mkdir -p "$RUNNER_TEMP/typos"
tar -xzf "$RUNNER_TEMP/typos.tar.gz" -C "$RUNNER_TEMP/typos"
- name: Typos check with custom config file
uses: crate-ci/typos@512fc24f32f44ab01972217aaaf3dc86ec234d53 # v1.50.2
run: '"$RUNNER_TEMP/typos/typos" .'
# Fail early with compile-free checks for every pull request.
quick-checks:
name: Quick Checks
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
timeout-minutes: 20
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -215,7 +223,8 @@ jobs:
- name: Archive workspace test binaries
env:
CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
# Workspace test compilation has high per-process memory use.
CARGO_BUILD_JOBS: "2"
run: |
./scripts/ci/resource_sampler.sh start workspace-archive
trap './scripts/ci/resource_sampler.sh stop' EXIT
@@ -1136,11 +1145,16 @@ jobs:
- name: Install mc
env:
GH_TOKEN: ${{ github.token }}
MC_VERSION: RELEASE.2025-08-13T08-35-41Z
MC_SHA256: 01f866e9c5f9b87c2b09116fa5d7c06695b106242d829a8bb32990c00312e891
run: |
if ! command -v gh >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y gh
fi
MC_BINARY="mc.linux-amd64.${MC_VERSION}"
curl -fsSLo "$RUNNER_TEMP/mc" "https://github.com/minio/mc/releases/download/${MC_VERSION}/${MC_BINARY}"
python3 .github/actions/setup/download_release_asset.py --repository minio/mc --tag "$MC_VERSION" --asset "$MC_BINARY" --output "$RUNNER_TEMP/mc" --sha256 "$MC_SHA256"
echo "${MC_SHA256} $RUNNER_TEMP/mc" | sha256sum --check --status
chmod +x "$RUNNER_TEMP/mc"
echo "$RUNNER_TEMP" >> "$GITHUB_PATH"
+26
View File
@@ -107,6 +107,32 @@ jobs:
shell: pwsh
run: cargo test -p rustfs-ecstore --lib multipart_owned_record_parent_barriers_ -- --nocapture
# Native multipart completion publishes its payload from a different
# parent than its metadata, and abort renames under a precondition; both
# need their own Windows handle discipline.
- name: Test native multipart complete and abort publication
shell: pwsh
run: >-
cargo test -p rustfs-ecstore --lib --
native_abort_missing_state_with_actual_owned_records_never_becomes_legacy_or_missing
native_abort_pending_intent_is_not_a_delete_certificate_and_raw_cleanup_preserves_exact_evidence
--nocapture
# Owned-undo reclaim removes a directory it pinned, copy heal persists
# fault bytes through a handle, and metadata reads meet a directory.
- name: Test native multipart reclaim, copy heal, and directory reads
shell: pwsh
run: >-
cargo test -p rustfs-ecstore --lib --
native_multipart_owned_undo
native_multipart_copy_heal
orphan_multipart_cleanup_rejects_metadata_directory_on_one_disk
--nocapture
- name: Test directory removal fallback
shell: pwsh
run: "cargo test -p rustfs-ecstore --lib disk::fs::tests::test_remove -- --nocapture"
- name: Test Windows path conversion
shell: pwsh
run: "cargo test -p rustfs-utils --no-default-features --features path --lib dunce:: -- --nocapture"
Generated
+11
View File
@@ -9612,6 +9612,7 @@ dependencies = [
"rustfs-log-analyzer",
"rustfs-madmin",
"rustfs-mimalloc",
"rustfs-mimalloc-sys",
"rustfs-notify",
"rustfs-object-capacity",
"rustfs-object-data-cache",
@@ -9836,6 +9837,7 @@ dependencies = [
"google-cloud-auth",
"google-cloud-storage",
"h2",
"hashbrown 0.17.1",
"hex-simd",
"hmac 0.13.0",
"hostname",
@@ -9862,6 +9864,7 @@ dependencies = [
"path-absolutize",
"pin-project-lite",
"proptest",
"quick-xml",
"rand 0.10.3",
"ratelimit",
"rcgen",
@@ -9894,7 +9897,9 @@ dependencies = [
"rustfs-s3-ops",
"rustfs-s3-types",
"rustfs-scanner-metrics",
"rustfs-signer",
"rustfs-storage-api",
"rustfs-tls-runtime",
"rustfs-uring",
"rustfs-utils",
"rustix",
@@ -10755,15 +10760,20 @@ dependencies = [
"arc-swap",
"async-compression",
"async-trait",
"base64-simd",
"brotli 9.1.1",
"bytes",
"chrono",
"crc-fast",
"datafusion",
"flatbuffers",
"flate2",
"futures",
"futures-core",
"hotpath",
"http 1.5.0",
"lz4",
"lz4_flex",
"memchr",
"metrics",
"parking_lot 0.13.0",
@@ -10896,6 +10906,7 @@ dependencies = [
"time",
"tracing",
"tracing-subscriber",
"url",
]
[[package]]
+1
View File
@@ -296,6 +296,7 @@ local-ip-address = "0.6.13"
log = "0.4"
memmap2 = "0.9.12"
lz4 = "1.28.1"
lz4_flex = { version = "=0.14.0", default-features = false, features = ["std", "frame"] }
matchit = "0.9.2"
md-5 = "0.11.0"
mime_guess = "2.0.5"
+21 -9
View File
@@ -184,15 +184,27 @@ const _: () = assert!(!DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT);
/// Capacity (distinct nonces) of the process-local internode RPC replay cache that enforces
/// one-time consumption of authenticated RPC signatures.
///
/// The cache retains each nonce for the ~10-minute signature freshness envelope. Once peers use
/// replay-scoped v3 authentication, every authenticated RPC consumes one entry, so the steady
/// state holds roughly `authenticated RPC RPS x 601s` entries. This default is the minimum floor:
/// explicit operator values and resource-aware auto sizing both clamp upward to at least this
/// value. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
/// cache (replays are rejected before insertion, and an attacker cannot mint valid nonces without
/// the shared secret) — and increments
/// `rustfs_system_network_internode_replay_cache_overflow_total`, so a sustained non-zero overflow
/// counter means this capacity is undersized for the node's peak authenticated RPC rate.
/// Nonces remain protected until their signed timestamp is outside the observed wall-clock
/// validity window, including after clock rollback. Current-time signatures live about five
/// minutes; the maximum accepted forward clock skew can extend that to ten minutes. A body-bound
/// v3 request consumes both its v3 and compatibility-v2 nonces, so size for receiver nonce intake,
/// not S3 request rate or CPU count.
///
/// Automatic admission budgets 13% of effective process memory for signed-time
/// hash tables, allocator overhead and simultaneous old/new tables during growth.
/// The reported count is an absolute UUID payload ceiling, not guaranteed throughput:
/// actual byte admission can reject earlier depending on timestamp distribution.
/// Tables grow with demand and are released together when their signed time expires.
/// CPU count, a historical minimum, and a fixed 32M ceiling do not constrain this budget.
/// If memory detection fails, the historical default remains a bounded count fallback.
///
/// Explicit values remain count ceilings with the historical minimum of 1,048,576
/// for configuration compatibility, and bypass automatic byte admission. That
/// compatibility floor does not apply to the automatically detected memory budget.
/// Zero never disables replay protection.
/// Overflow rejects the request without evicting a still-valid identity, increments
/// `rustfs_system_network_internode_replay_cache_overflow_total`, and returns resource
/// exhaustion rather than invalid credentials. Measure headroom before overriding.
pub const ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: &str = "RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY";
pub const DEFAULT_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: usize = 1_048_576;
+3 -2
View File
@@ -32,8 +32,9 @@ pub const ENV_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD: &str = "RUSTFS_OBJECT_MEDIUM_
/// Environment variable name for maximum concurrent disk reads for object operations.
/// - Purpose: Limit the number of concurrent disk read operations for object reads to prevent I/O saturation.
/// - Unit: request count (usize).
/// - Semantics: Throttling disk reads helps maintain overall system responsiveness under load.
/// - Unit: request count (usize) for GET; estimated drive fanout for native LIST.
/// - Semantics: GET bodies and native LIST walks use separate per-instance budgets of this size.
/// A LIST consumes up to one budget, including its disk-producer cleanup. `0` disables both caps.
/// - Example: `export RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=16`
/// - Note: This setting may interact with OS-level I/O scheduling and should be tuned based on hardware capabilities.
pub const ENV_OBJECT_MAX_CONCURRENT_DISK_READS: &str = "RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS";
+55 -2
View File
@@ -675,6 +675,10 @@ impl DataUsageInfo {
/// cycle. The timestamp fallback preserves ordering for legacy snapshots that
/// predate those fields.
pub fn data_usage_snapshot_is_newer(candidate: &DataUsageInfo, baseline: &DataUsageInfo) -> bool {
data_usage_identity_is_newer(candidate.snapshot_identity(), baseline.snapshot_identity())
}
fn data_usage_identity_is_newer(candidate: DataUsageSnapshotIdentity, baseline: DataUsageSnapshotIdentity) -> bool {
match (
candidate.scanner_epoch.zip(candidate.scanner_cycle),
baseline.scanner_epoch.zip(baseline.scanner_cycle),
@@ -697,10 +701,18 @@ pub fn data_usage_snapshot_is_newer(candidate: &DataUsageInfo, baseline: &DataUs
/// already advance the authoritative timestamp when deleting a bucket, so a
/// rollback delete/recreate fences the previous bucket incarnation too.
pub fn observed_data_usage_is_newer(observed: &DataUsageInfo, authoritative: &DataUsageInfo) -> bool {
observed_data_usage_is_newer_than_identity(observed, authoritative.snapshot_identity())
}
/// Return whether a nonconverged observation may supersede `authoritative`.
///
/// This applies the structural, exact-baseline, and generation checks of
/// [`observed_data_usage_is_newer`] without retaining the baseline's bucket maps.
pub fn observed_data_usage_is_newer_than_identity(observed: &DataUsageInfo, authoritative: DataUsageSnapshotIdentity) -> bool {
observed.usage_snapshot_converged == Some(false)
&& (observed.is_complete_bucket_usage_snapshot() || observed.is_valid_partial_snapshot())
&& observed.usage_snapshot_authoritative_baseline.as_ref() == Some(&authoritative.snapshot_identity())
&& data_usage_snapshot_is_newer(observed, authoritative)
&& observed.usage_snapshot_authoritative_baseline == Some(authoritative)
&& data_usage_identity_is_newer(observed.snapshot_identity(), authoritative)
}
/// Metadata describing the status of a disk-level data usage snapshot.
@@ -3140,12 +3152,17 @@ mod tests {
};
assert!(observed_data_usage_is_newer(&observed, &authoritative));
assert!(observed_data_usage_is_newer_than_identity(&observed, authoritative.snapshot_identity()));
authoritative.last_update = Some(SystemTime::UNIX_EPOCH + Duration::from_secs(601));
assert!(
!observed_data_usage_is_newer(&observed, &authoritative),
"an old-binary namespace mutation must fence the prior bucket incarnation regardless of clock skew"
);
assert!(
!observed_data_usage_is_newer_than_identity(&observed, authoritative.snapshot_identity()),
"a captured identity must preserve the exact namespace fence"
);
}
#[test]
@@ -3172,6 +3189,18 @@ mod tests {
assert!(!observed_data_usage_is_newer(&candidate(2, 9, Some(false), true), &authoritative));
assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(true), true), &authoritative));
assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(false), false), &authoritative));
for (observed, expected) in [
(candidate(2, 11, Some(false), true), true),
(candidate(2, 9, Some(false), true), false),
(candidate(2, 11, Some(true), true), false),
(candidate(2, 11, Some(false), false), false),
] {
assert_eq!(
observed_data_usage_is_newer_than_identity(&observed, authoritative.snapshot_identity()),
expected,
"observation identity selection must preserve convergence, completeness, and generation gates"
);
}
let mut partial = candidate(2, 11, Some(false), false);
partial.usage_snapshot_partial = true;
@@ -3186,6 +3215,30 @@ mod tests {
segment_invalidation_proof: None,
}];
assert!(observed_data_usage_is_newer(&partial, &authoritative));
assert!(observed_data_usage_is_newer_than_identity(&partial, authoritative.snapshot_identity()));
}
#[test]
fn observation_identity_selection_keeps_legacy_timestamp_ordering() {
let baseline = DataUsageSnapshotIdentity {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(10)),
// An epoch without a cycle is not a coordinated position.
scanner_epoch: Some(2),
scanner_cycle: None,
};
let mut observed = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(11)),
scanner_epoch: Some(1),
usage_snapshot_complete: true,
usage_snapshot_converged: Some(false),
usage_snapshot_authoritative_baseline: Some(baseline),
..Default::default()
};
assert!(observed_data_usage_is_newer_than_identity(&observed, baseline));
observed.last_update = baseline.last_update;
assert!(!observed_data_usage_is_newer_than_identity(&observed, baseline));
observed.last_update = None;
assert!(!observed_data_usage_is_newer_than_identity(&observed, baseline));
}
#[test]
@@ -260,7 +260,7 @@ async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Clien
let key = "guarded.txt";
admin_create_user(env, &user, user_secret).await?;
let policy = serde_json::json!({
let mut policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
@@ -268,9 +268,8 @@ async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Clien
"Resource": [format!("arn:aws:s3:::{bucket}/*")],
"Condition": { "Null": { "s3:if-match": "false" } }
}]
})
.to_string();
admin_add_and_attach_policy(env, &policy_name, &policy, &user).await?;
});
admin_add_and_attach_policy(env, &policy_name, &policy.to_string(), &user).await?;
admin.create_bucket().bucket(&bucket).send().await?;
let original = admin
@@ -293,6 +292,46 @@ async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Clien
.expect_err("PutObject without If-Match must be denied");
assert_error(&denied, "AccessDenied", 403, "unconditional PutObject");
let denied = client
.put_object()
.bucket(&bucket)
.key(key)
.if_match(&etag)
.body(ByteStream::from_static(b"missing-read-permission"))
.send()
.await
.expect_err("If-Match without GetObject must be denied");
assert_error(&denied, "AccessDenied", 403, "conditional PutObject without GetObject");
// S3 requires GetObject as well as PutObject for an ETag-based write.
policy["Statement"]
.as_array_mut()
.expect("policy statements")
.push(serde_json::json!({
"Effect": "Allow",
"Action": ["s3:GetObject"],
"Resource": [format!("arn:aws:s3:::{bucket}/{key}")]
}));
admin_add_canned_policy_via(
AdminTransport::Signed,
&env.url,
&env.access_key,
&env.secret_key,
&policy_name,
&policy.to_string(),
)
.await?;
let denied = client
.put_object()
.bucket(&bucket)
.key(key)
.body(ByteStream::from_static(b"blind-overwrite"))
.send()
.await
.expect_err("GetObject permission must not allow an unconditional PutObject");
assert_error(&denied, "AccessDenied", 403, "unconditional PutObject with GetObject");
client
.put_object()
.bucket(&bucket)
@@ -301,6 +340,8 @@ async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Clien
.body(ByteStream::from_static(b"v2"))
.send()
.await?;
let written = client.get_object().bucket(&bucket).key(key).send().await?;
assert_eq!(written.body.collect().await?.into_bytes().as_ref(), b"v2");
let _ = admin.delete_object().bucket(&bucket).key(key).send().await;
let _ = admin.delete_bucket().bucket(&bucket).send().await;
+34 -4
View File
@@ -412,6 +412,38 @@ mod integration_tests {
);
}
async fn upload_after_quota_reconciliation(
env: &QuotaTestEnv,
key: &str,
size_bytes: usize,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let upload = async {
loop {
let result = env
.client
.put_object()
.bucket(&env.bucket_name)
.key(key)
.body(aws_sdk_s3::primitives::ByteStream::from(vec![0u8; size_bytes]))
.send()
.await;
match result {
Ok(_) => break,
Err(error) => assert_quota_rejection(
error.raw_response().map(|response| response.status().as_u16()),
error.as_service_error(),
&error,
),
}
sleep(Duration::from_secs(1)).await;
}
};
// The durable V1 ledger can retain the old charge for its 30-second reconciliation cooldown.
timeout(Duration::from_secs(45), upload)
.await
.map_err(|_| format!("PUT {key} did not succeed after quota reconciliation within 45 seconds").into())
}
async fn assert_put_rejected_by_quota(env: &QuotaTestEnv, key: &str, size_bytes: usize) {
let error = env
.client
@@ -573,8 +605,7 @@ mod integration_tests {
assert!(!env.object_exists("file1.txt").await?);
// Now we should be able to upload again (quota freed up)
env.upload_object("file3.txt", 256 * 1024).await?;
upload_after_quota_reconciliation(&env, "file3.txt", 256 * 1024).await?;
assert!(env.object_exists("file3.txt").await?);
env.cleanup_bucket().await?;
@@ -981,8 +1012,7 @@ mod integration_tests {
);
assert_eq!(delete_result.deleted().len(), 2);
// Now should be able to upload again (quota freed up)
env.upload_object("file3.txt", 256 * 1024).await?;
upload_after_quota_reconciliation(&env, "file3.txt", 256 * 1024).await?;
assert!(env.object_exists("file3.txt").await?);
env.cleanup_bucket().await?;
@@ -114,10 +114,10 @@ fn staged_commit(metadata: &Value, location: &str) -> Value {
})
}
async fn location_claim_preserves_other_users_objects_after_drop_and_restart(backing: &str) -> TestResult {
async fn location_claim_preserves_other_users_objects_after_drop_and_restart() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
let server_env = [("RUSTFS_TABLE_CATALOG_BACKING", backing)];
let server_env = [("RUSTFS_TABLE_CATALOG_BACKING", "object")];
env.start_rustfs_server_without_cleanup_with_env(&[("RUSTFS_TABLE_CATALOG_BACKING", "object")])
.await?;
env.wait_for_server_ready().await?;
@@ -147,39 +147,6 @@ async fn location_claim_preserves_other_users_objects_after_drop_and_restart(bac
StatusCode::OK,
)
.await?;
if backing == "durable-strong" {
catalog_request(
&env,
&root,
Method::POST,
&format!("/iceberg/v1/{bucket}/catalog/warehouse-index/backfill"),
None,
StatusCode::NO_CONTENT,
)
.await?;
let preflight = catalog_request(
&env,
&root,
Method::GET,
&format!("/iceberg/v1/{bucket}/catalog/migration"),
None,
StatusCode::OK,
)
.await?;
assert_eq!(preflight["blockers"], json!([]), "{preflight}");
let migration = catalog_request(
&env,
&root,
Method::POST,
&format!("/iceberg/v1/{bucket}/catalog/migration"),
None,
StatusCode::OK,
)
.await?;
assert_eq!(migration["ready_to_enable_durable_strong"], json!(true), "{migration}");
env.restart_server_preserving_data(vec![], &server_env).await?;
env.wait_for_server_ready().await?;
}
admin_create_user(&env, &creator.0, &creator.1).await?;
admin_create_user(&env, &reader.0, &reader.1).await?;
let create_statement = json!({
@@ -341,7 +308,7 @@ async fn location_claim_preserves_other_users_objects_after_drop_and_restart(bac
StatusCode::FORBIDDEN,
)
.await?;
if backing == "object" {
{
catalog_request(
&env,
&creator,
@@ -408,10 +375,5 @@ async fn location_claim_preserves_other_users_objects_after_drop_and_restart(bac
#[tokio::test]
async fn object_catalog_location_claim_preserves_objects_after_drop_and_restart() -> TestResult {
location_claim_preserves_other_users_objects_after_drop_and_restart("object").await
}
#[tokio::test]
async fn durable_strong_catalog_location_claim_preserves_objects_after_drop_and_restart() -> TestResult {
location_claim_preserves_other_users_objects_after_drop_and_restart("durable-strong").await
location_claim_preserves_other_users_objects_after_drop_and_restart().await
}
+5 -1
View File
@@ -31,7 +31,7 @@ workspace = true
[features]
default = []
fips = ["rustfs-s3-client/fips", "rustfs-utils/fips"]
fips = ["rustfs-s3-client/fips", "rustfs-utils/fips", "rustfs-tls-runtime/fips"]
diagnostic-read-version-caller = ["rustfs-io-metrics/diagnostic-read-version-caller"]
# Sampled wall/CPU diagnostics only; does not change the read backend.
diagnostic-local-batch = ["rustix/time"]
@@ -121,6 +121,7 @@ test-util = []
e2e-test-hooks = []
[dependencies]
hashbrown.workspace = true
hotpath.workspace = true
rustfs-filemeta.workspace = true
rustfs-utils = { workspace = true, features = ["full"] }
@@ -203,6 +204,9 @@ libc.workspace = true
# (backlog#1178); "thread" provides cleanup affinity; "fs" is a workspace default.
rustix = { workspace = true, features = ["process", "fs", "thread"] }
rustfs-madmin.workspace = true
rustfs-signer.workspace = true
rustfs-tls-runtime.workspace = true
quick-xml.workspace = true
reqwest = { workspace = true }
aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
urlencoding = { workspace = true }
+25 -20
View File
@@ -191,8 +191,11 @@ pub mod bucket {
OBJECT_LOCK_CONFIG, UnreadableBucketConfig, is_unreadable_config_error, load_bucket_metadata,
table_catalog_path_hash, unreadable_config_refusal,
};
pub use crate::bucket::metadata::{BUCKET_DURABILITY_CONFIG, BUCKET_ON_DEMAND_MIGRATION_CONFIG};
pub use crate::bucket::metadata::{MAX_BUCKET_TAGS, validate_tag_set_entries};
pub use crate::bucket::metadata::{
BUCKET_DURABILITY_CONFIG, BUCKET_ON_DEMAND_MIGRATION_CONFIG, MAX_SWIFT_METADATA_COUNT, MAX_SWIFT_METADATA_VALUE_SIZE,
SWIFT_ACCOUNT_META_TAG_PREFIX, SWIFT_CONTAINER_META_TAG_PREFIX, SWIFT_TAGGING_MARKER,
};
pub use crate::bucket::metadata::{MAX_BUCKET_TAGS, is_swift_bucket_metadata_tag_key, validate_tag_set_entries};
}
pub mod config_parse_mode {
@@ -210,19 +213,20 @@ pub mod bucket {
pub mod metadata_sys {
pub use crate::bucket::metadata_sys::{
BUCKET_CONFIG_PUBLISH_HOOK, BucketConfigPublishHook, BucketMetadataMutationGuard, BucketMetadataSys,
ObjectLockConfigState, acquire_bucket_metadata_transaction_lock,
ObjectLockConfigState, ScannerBucketIncarnationReadGuard, acquire_bucket_metadata_transaction_lock,
acquire_bucket_metadata_transaction_lock_for_incarnation, acquire_scanner_bucket_incarnation_fence,
capture_bucket_metadata_incarnation, delete, delete_if_incarnation, delete_if_incarnation_at,
delete_under_transaction_lock, get, get_accelerate_config, get_bucket_policy, get_bucket_policy_raw,
get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
acquire_scanner_bucket_incarnation_read_fence, capture_bucket_metadata_incarnation, delete, delete_if_incarnation,
delete_if_incarnation_at, delete_under_transaction_lock, get, get_accelerate_config, get_bucket_policy,
get_bucket_policy_raw, get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
get_global_bucket_metadata_sys, get_lifecycle_config, get_logging_config, get_notification_config,
get_object_lock_config, get_object_lock_config_state, get_on_demand_migration_config,
get_on_demand_migration_config_in, get_public_access_block_config, get_quota_config, get_replication_config,
get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
get_request_payment_config, get_s3_tagging_config, get_sse_config, get_tagging_config, get_versioning_config,
get_website_config, init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
remove_bucket_metadata_if_absent, set_bucket_metadata, update, update_bucket_targets_under_transaction_lock,
update_config_with, update_if_incarnation, update_if_incarnation_at, update_quota_if_incarnation,
update_quota_if_incarnation_at, update_under_transaction_lock, update_under_transaction_lock_at,
update_quota_if_incarnation_at, update_s3_tagging_config, update_under_transaction_lock,
update_under_transaction_lock_at,
};
#[cfg(feature = "test-util")]
pub use crate::bucket::metadata_sys::{ConfigWriteLockProbe, test_support};
@@ -450,6 +454,7 @@ pub mod data_usage {
pub mod disk {
pub use crate::cluster::rpc::background_pipe::{BackgroundPipeReader, BackgroundPipeWriter, background_pipe};
pub use crate::cluster::rpc::background_reader_stream::BackgroundReaderStream;
pub use crate::disk::BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION;
pub use crate::disk::background_memory::BackgroundEncodeBuffer;
pub use crate::disk::disk_store::get_object_disk_read_timeout;
#[cfg(feature = "test-util")]
@@ -635,17 +640,17 @@ pub mod rpc {
pub use crate::cluster::rpc::{
AuthenticatedChannel, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, MAX_NETWORK_PROBE_BYTES, MAX_NETWORK_PROBE_DURATION,
NetworkPeerProbeClient, NetworkPeerProbeError, NetworkPeerProbeMeasurement, NetworkPeerTarget, PEER_RESTDRY_RUN,
PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG,
SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing, ScannerDirtyUsageAcknowledgement, ScannerPeerActivity,
ScannerPeerDirtyUsageBucket, ScannerPeerDirtyUsageSnapshot, ScannerPublicationLease, ScannerScopedDirtyUsageAckEntry,
TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer, check_and_record_signed_rpc_nonce,
decode_heal_bucket_rpc_options, encode_heal_bucket_rpc_options, gen_signature_headers, gen_tonic_replay_scope_headers,
gen_tonic_signature_headers, gen_tonic_signature_interceptor, node_service_time_out_client,
node_service_time_out_client_no_auth, normalize_tonic_rpc_audience, rpc_background_admission,
set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_ns_scanner_capability_with_checkpoint_coverage,
sign_ns_scanner_capability_with_tier_registry_generation, sign_put_file_capability, sign_tonic_rpc_response_proof,
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason,
verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client, RpcReplayCacheAdmissionError, S3PeerSys,
SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing, ScannerDirtyUsageAcknowledgement,
ScannerPeerActivity, ScannerPeerDirtyUsageBucket, ScannerPeerDirtyUsageSnapshot, ScannerPublicationLease,
ScannerScopedDirtyUsageAckEntry, TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer,
check_and_record_signed_rpc_nonce, decode_heal_bucket_rpc_options, encode_heal_bucket_rpc_options, gen_signature_headers,
gen_tonic_replay_scope_headers, gen_tonic_signature_headers, gen_tonic_signature_interceptor,
node_service_time_out_client, node_service_time_out_client_no_auth, normalize_tonic_rpc_audience,
rpc_auth_is_resource_exhausted, rpc_background_admission, set_tonic_canonical_body_digest, sign_ns_scanner_capability,
sign_ns_scanner_capability_with_checkpoint_coverage, sign_ns_scanner_capability_with_tier_registry_generation,
sign_put_file_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
tonic_rpc_auth_failure_reason, verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
verify_ns_scanner_capability_with_tier_registry_generation, verify_put_file_auth_trailer, verify_put_file_capability,
verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned, verify_tonic_rpc_response_proof,
+139 -22
View File
@@ -256,15 +256,15 @@ pub const BUCKET_POLICY_CONFIG: &str = "policy.json";
pub const BUCKET_NOTIFICATION_CONFIG: &str = "notification.xml";
pub const BUCKET_LIFECYCLE_CONFIG: &str = "lifecycle.xml";
pub const BUCKET_SSECONFIG: &str = "bucket-encryption.xml";
/// Shared tag-entry validation for object requests and persisted bucket tags.
/// Validate tag entries supplied to the S3 API.
pub fn validate_tag_set_entries(tag_set: &[s3s::dto::Tag]) -> std::result::Result<(), &'static str> {
fn is_valid_tag_text(value: &str) -> bool {
static VALID_TAG_TEXT: std::sync::OnceLock<std::result::Result<regex::Regex, regex::Error>> = std::sync::OnceLock::new();
VALID_TAG_TEXT
.get_or_init(|| regex::Regex::new(r"^[\p{L}\p{N}\p{Z}+\-=._:/@]*$"))
.as_ref()
.is_ok_and(|pattern| pattern.is_match(value))
}
validate_tag_entries(tag_set, validate_s3_tag_entry)
}
fn validate_tag_entries(
tag_set: &[s3s::dto::Tag],
mut validate_entry: impl FnMut(&str, &str) -> std::result::Result<(), &'static str>,
) -> std::result::Result<(), &'static str> {
let mut keys = std::collections::HashSet::with_capacity(tag_set.len());
for tag in tag_set {
let key = tag
@@ -272,13 +272,8 @@ pub fn validate_tag_set_entries(tag_set: &[s3s::dto::Tag]) -> std::result::Resul
.as_deref()
.filter(|key| !key.is_empty())
.ok_or("Tag key cannot be empty")?;
if key.encode_utf16().count() > 128 || !is_valid_tag_text(key) {
return Err("The TagKey you have provided is invalid");
}
let value = tag.value.as_deref().ok_or("Tag value cannot be null")?;
if value.encode_utf16().count() > 256 || !is_valid_tag_text(value) {
return Err("The TagValue you have provided is invalid");
}
validate_entry(key, value)?;
if !keys.insert(key) {
return Err("Cannot provide multiple Tags with the same key");
}
@@ -286,8 +281,103 @@ pub fn validate_tag_set_entries(tag_set: &[s3s::dto::Tag]) -> std::result::Resul
Ok(())
}
fn validate_s3_tag_entry(key: &str, value: &str) -> std::result::Result<(), &'static str> {
fn is_valid_tag_text(value: &str) -> bool {
static VALID_TAG_TEXT: std::sync::OnceLock<std::result::Result<regex::Regex, regex::Error>> = std::sync::OnceLock::new();
VALID_TAG_TEXT
.get_or_init(|| regex::Regex::new(r"^[\p{L}\p{N}\p{Z}+\-=._:/@]*$"))
.as_ref()
.is_ok_and(|pattern| pattern.is_match(value))
}
if key.encode_utf16().count() > 128 || !is_valid_tag_text(key) {
return Err("The TagKey you have provided is invalid");
}
if value.encode_utf16().count() > 256 || !is_valid_tag_text(value) {
return Err("The TagValue you have provided is invalid");
}
Ok(())
}
pub const MAX_BUCKET_TAGS: usize = 50;
/// Maximum number of metadata headers on a Swift resource.
pub const MAX_SWIFT_METADATA_COUNT: usize = 90;
/// Maximum byte length of a Swift metadata value.
pub const MAX_SWIFT_METADATA_VALUE_SIZE: usize = 256;
/// Prefix for Swift account metadata persisted in tagging.xml.
pub const SWIFT_ACCOUNT_META_TAG_PREFIX: &str = "swift-account-meta-";
/// Prefix for Swift container metadata persisted in tagging.xml.
pub const SWIFT_CONTAINER_META_TAG_PREFIX: &str = "swift-meta-";
/// Marks a bucket whose shared tag config contains Swift-owned metadata.
/// The key cannot be supplied through S3's tag alphabet.
pub const SWIFT_TAGGING_MARKER: &str = "!rustfs-swift-metadata";
/// Whether a persisted tag belongs to Swift's internal metadata namespace.
pub fn is_swift_bucket_metadata_tag_key(key: &str) -> bool {
matches!(
key,
"swift-acl-read" | "swift-acl-write" | "swift-versions-location" | SWIFT_TAGGING_MARKER
) || key.starts_with(SWIFT_ACCOUNT_META_TAG_PREFIX)
|| key.starts_with(SWIFT_CONTAINER_META_TAG_PREFIX)
}
fn validate_persisted_bucket_tags(tags: &[s3s::dto::Tag]) -> std::result::Result<(), &'static str> {
const SWIFT_ACL_MAX_BYTES: usize = 8192;
const SWIFT_META_NAME_MAX_BYTES: usize = 128;
const MAX_PERSISTED_TAGS: usize = MAX_BUCKET_TAGS + 2 * MAX_SWIFT_METADATA_COUNT + 4;
if tags.len() > MAX_PERSISTED_TAGS {
return Err("Too many persisted bucket tags");
}
// Swift stores ACLs, versioning and metadata in the same XML as S3 tags.
// Only those known keys may carry values outside the S3 tag alphabet.
let mut account_metadata = 0;
let mut container_metadata = 0;
let mut s3_tags = 0;
validate_tag_entries(tags, |key, value| {
let swift_metadata = key
.strip_prefix(SWIFT_ACCOUNT_META_TAG_PREFIX)
.map(|name| (name, &mut account_metadata))
.or_else(|| {
key.strip_prefix(SWIFT_CONTAINER_META_TAG_PREFIX)
.map(|name| (name, &mut container_metadata))
});
if let Some((name, count)) = swift_metadata
&& name.len() <= SWIFT_META_NAME_MAX_BYTES
&& http::HeaderName::from_bytes(name.as_bytes()).is_ok()
&& value.len() <= MAX_SWIFT_METADATA_VALUE_SIZE
&& http::HeaderValue::from_str(value).is_ok()
{
*count += 1;
if *count > MAX_SWIFT_METADATA_COUNT {
return Err("Too many Swift metadata tags");
}
return Ok(());
}
if key == SWIFT_TAGGING_MARKER && value == "1" {
return Ok(());
}
let valid_swift_header = match key {
"swift-acl-read" | "swift-acl-write" => value.len() <= SWIFT_ACL_MAX_BYTES,
"swift-versions-location" => value.len() <= 256 && !value.is_empty() && !value.contains('/'),
_ => false,
};
if valid_swift_header && http::HeaderValue::from_str(value).is_ok() {
return Ok(());
}
s3_tags += 1;
if s3_tags > MAX_BUCKET_TAGS {
return Err("Cannot have more than 50 tags per bucket");
}
validate_s3_tag_entry(key, value)
})
}
pub const BUCKET_TAGGING_CONFIG: &str = "tagging.xml";
pub const BUCKET_QUOTA_CONFIG_FILE: &str = "quota.json";
pub const OBJECT_LOCK_CONFIG: &str = "object-lock.xml";
@@ -1026,14 +1116,7 @@ impl BucketMetadata {
let tagging = deserialize::<Tagging>(&data).map_err(|_| {
Error::InvalidArgument(self.name.clone(), String::new(), "Invalid bucket tagging XML".to_owned())
})?;
if tagging.tag_set.len() > MAX_BUCKET_TAGS {
return Err(Error::InvalidArgument(
self.name.clone(),
String::new(),
"Cannot have more than 50 tags per bucket".to_owned(),
));
}
validate_tag_set_entries(&tagging.tag_set)
validate_persisted_bucket_tags(&tagging.tag_set)
.map_err(|message| Error::InvalidArgument(self.name.clone(), String::new(), message.to_owned()))?;
}
self.tagging_config_xml = data;
@@ -1827,6 +1910,40 @@ mod test {
assert_eq!(metadata.bucket_incarnation_id, incarnation);
}
#[test]
fn persisted_bucket_tags_keep_separate_s3_and_swift_limits() {
use s3s::dto::Tag;
let tag = |key: String, value: &str| Tag {
key: Some(key),
value: Some(value.to_owned()),
};
let acl = tag("swift-acl-read".to_owned(), ".r:*,AUTH_other");
assert_eq!(
validate_tag_set_entries(std::slice::from_ref(&acl)),
Err("The TagValue you have provided is invalid")
);
let mut tags = vec![acl.clone()];
tags.extend((0..MAX_BUCKET_TAGS).map(|i| tag(format!("team-{i}"), "blue")));
tags.extend((0..MAX_SWIFT_METADATA_COUNT).map(|i| tag(format!("{SWIFT_CONTAINER_META_TAG_PREFIX}item!{i}"), "a,b")));
assert_eq!(validate_persisted_bucket_tags(&tags), Ok(()));
tags.push(tag(format!("{SWIFT_CONTAINER_META_TAG_PREFIX}overflow"), "a,b"));
assert_eq!(validate_persisted_bucket_tags(&tags), Err("Too many Swift metadata tags"));
tags.pop();
tags.push(acl);
assert_eq!(
validate_persisted_bucket_tags(&tags),
Err("Cannot provide multiple Tags with the same key")
);
tags.pop();
tags[0].value = Some(".r:*,".repeat(1700));
assert_eq!(validate_persisted_bucket_tags(&tags), Err("The TagValue you have provided is invalid"));
}
/// backlog#2292: a replicated config is stamped with the source
/// `updated_at` it was given, not the local clock, while the plain
/// `update_config` entry keeps stamping the local clock.
+417 -44
View File
@@ -22,6 +22,7 @@ use super::target::BucketTargets;
use crate::bucket::bucket_target_sys::BucketTargetSys;
use crate::bucket::metadata::{load_bucket_metadata_parse, load_bucket_metadata_parse_with_presence};
use crate::bucket::utils::is_meta_bucketname;
use crate::diagnostics::object_lock::ObjectLockAttempt;
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::{Error, Result, is_err_bucket_not_found, is_err_strict_volume_not_found};
use crate::object_api::ObjectOptions;
@@ -599,7 +600,9 @@ pub(crate) async fn get_bucket_incarnation_id_for_options_in(
}
let sys = bucket_metadata_sys_of(ctx)?;
let sys = sys.read().await.clone();
let guard = acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts).await?;
let guard =
acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts, "bucket_incarnation_options_read")
.await?;
sys.get_bucket_incarnation_id_under_transaction_lock(bucket, &guard).await
}
@@ -866,6 +869,30 @@ impl BucketMetadataMutationGuard {
}
}
/// Read-only scanner authority for one bucket incarnation.
///
/// Lock order is bucket lifecycle (read), then metadata transaction (read).
/// This guard cannot authorize a configuration mutation or scanner ACK write.
pub struct ScannerBucketIncarnationReadGuard {
bucket: String,
incarnation_id: Uuid,
lifecycle_guard: rustfs_lock::NamespaceLockGuard,
transaction_guard: rustfs_lock::NamespaceLockGuard,
}
impl ScannerBucketIncarnationReadGuard {
/// Returns the verified identity while both read fences remain valid.
///
/// # Errors
/// Returns an error if either lock has lost its lease.
pub fn checked_bucket_incarnation(&self) -> Result<(&str, Uuid)> {
if self.lifecycle_guard.is_lock_lost() || self.transaction_guard.is_lock_lost() {
return Err(Error::other("scanner bucket incarnation read lock was lost"));
}
Ok((&self.bucket, self.incarnation_id))
}
}
async fn acquire_config_write_guard(sys: Arc<RwLock<BucketMetadataSys>>, bucket: &str) -> Result<BucketMetadataMutationGuard> {
acquire_config_write_guard_for_incarnation(sys, bucket, None).await
}
@@ -892,6 +919,68 @@ pub async fn acquire_scanner_bucket_incarnation_fence(
acquire_config_write_guard_with_migration(sys, bucket, Some(expected_incarnation_id), false).await
}
/// Validate scanner read authority without serializing compatible metadata readers.
///
/// The returned guard retains both read locks. It never migrates metadata or
/// creates an incarnation; mutation callers must use the exclusive scanner fence.
///
/// # Errors
/// Returns an error for invalid bucket/owner identities, unavailable locks or
/// storage, absent incarnation metadata, or a deleted/recreated bucket.
pub async fn acquire_scanner_bucket_incarnation_read_fence(
bucket: &str,
expected_incarnation_id: Uuid,
expected_owner_id: Uuid,
) -> Result<ScannerBucketIncarnationReadGuard> {
acquire_scanner_bucket_incarnation_read_fence_with_sys(
get_bucket_metadata_sys()?,
bucket,
expected_incarnation_id,
expected_owner_id,
)
.await
}
async fn acquire_scanner_bucket_incarnation_read_fence_with_sys(
sys: Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
expected_incarnation_id: Uuid,
expected_owner_id: Uuid,
) -> Result<ScannerBucketIncarnationReadGuard> {
super::utils::check_valid_bucket_name(bucket)?;
let metadata_sys = sys.read().await.clone();
if expected_owner_id.is_nil() || metadata_sys.object_store().id != expected_owner_id || expected_incarnation_id.is_nil() {
return Err(Error::other("scanner bucket incarnation owner does not match"));
}
let lifecycle_guard = metadata_sys.object_store().acquire_bucket_lifecycle_read_lock(bucket).await?;
let transaction_guard = await_bucket_namespace_operation(
Some(&lifecycle_guard),
bucket,
"scanner incarnation read transaction lock acquisition",
acquire_metadata_transaction_guard(
metadata_sys.object_store(),
bucket,
"scanner_incarnation_probe",
rustfs_lock::LockType::Shared,
),
)
.await?;
let incarnation_id = validate_bucket_metadata_guard_incarnation(
&metadata_sys,
bucket,
Some(expected_incarnation_id),
&lifecycle_guard,
&transaction_guard,
)
.await?;
Ok(ScannerBucketIncarnationReadGuard {
bucket: bucket.to_owned(),
incarnation_id,
lifecycle_guard,
transaction_guard,
})
}
async fn acquire_config_write_guard_with_migration(
sys: Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
@@ -917,15 +1006,47 @@ async fn acquire_config_write_guard_with_migration(
Some(&lifecycle_guard),
bucket,
"bucket config transaction lock acquisition",
acquire_transaction_lock_with_sys(&sys, bucket),
acquire_transaction_lock_with_sys_for_purpose(
&sys,
bucket,
if migrate {
"bucket_config_update"
} else {
"scanner_incarnation_validation"
},
),
)
.await?;
let current_incarnation_id = validate_bucket_metadata_guard_incarnation(
&metadata_sys,
bucket,
expected_incarnation_id,
&lifecycle_guard,
&transaction_guard,
)
.await?;
Ok(BucketMetadataMutationGuard {
bucket: bucket.to_string(),
incarnation_id: current_incarnation_id,
lifecycle_guard,
transaction_guard,
})
}
async fn validate_bucket_metadata_guard_incarnation(
metadata_sys: &BucketMetadataSys,
bucket: &str,
expected_incarnation_id: Option<Uuid>,
lifecycle_guard: &rustfs_lock::NamespaceLockGuard,
transaction_guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<Uuid> {
await_bucket_namespace_operation(
Some(&lifecycle_guard),
Some(lifecycle_guard),
bucket,
"bucket config existence validation",
await_bucket_namespace_operation(
Some(&transaction_guard),
Some(transaction_guard),
bucket,
"bucket config existence transaction validation",
async {
@@ -943,11 +1064,11 @@ async fn acquire_config_write_guard_with_migration(
)
.await?;
let current_incarnation_id = await_bucket_namespace_operation(
Some(&lifecycle_guard),
Some(lifecycle_guard),
bucket,
"bucket config incarnation validation",
await_bucket_namespace_operation(
Some(&transaction_guard),
Some(transaction_guard),
bucket,
"bucket config incarnation transaction validation",
load_bucket_incarnation(metadata_sys.object_store(), bucket),
@@ -960,12 +1081,7 @@ async fn acquire_config_write_guard_with_migration(
return Err(Error::BucketNotFound(bucket.to_string()));
}
Ok(BucketMetadataMutationGuard {
bucket: bucket.to_string(),
incarnation_id: current_incarnation_id,
lifecycle_guard,
transaction_guard,
})
Ok(current_incarnation_id)
}
/// Rewrite one config file while the caller already holds this bucket's
@@ -1174,11 +1290,23 @@ async fn delete_under_config_write_guard(
/// targets one, so a writer on another node cannot land a whole-file save in
/// the middle of this read-modify-write.
pub async fn update_config_with<F>(bucket: &str, config_file: &str, mutate: F) -> Result<OffsetDateTime>
where
F: FnOnce(&BucketMetadata) -> Result<Vec<u8>> + Send,
{
update_config_with_expected(bucket, config_file, None, mutate).await
}
async fn update_config_with_expected<F>(
bucket: &str,
config_file: &str,
expected_incarnation_id: Option<Uuid>,
mutate: F,
) -> Result<OffsetDateTime>
where
F: FnOnce(&BucketMetadata) -> Result<Vec<u8>> + Send,
{
let sys = get_bucket_metadata_sys()?;
let guard = Box::pin(acquire_config_write_guard(sys.clone(), bucket)).await?;
let guard = Box::pin(acquire_config_write_guard_for_incarnation(sys.clone(), bucket, expected_incarnation_id)).await?;
guard.ensure_valid(bucket)?;
let metadata_sys = sys.read().await.clone();
let updated = await_bucket_namespace_operation(
@@ -1197,6 +1325,70 @@ where
Ok(updated)
}
fn swift_manages_bucket_tags(tagging: &Tagging) -> bool {
tagging
.tag_set
.iter()
.any(|tag| tag.key.as_deref() == Some(super::metadata::SWIFT_TAGGING_MARKER) && tag.value.as_deref() == Some("1"))
}
/// Read the S3 tag set while keeping Swift's internal tags private.
pub async fn get_s3_tagging_config(bucket: &str) -> Result<(Tagging, OffsetDateTime)> {
let (mut tagging, updated) = get_tagging_config(bucket).await?;
if swift_manages_bucket_tags(&tagging) {
tagging.tag_set.retain(|tag| {
!tag.key
.as_deref()
.is_some_and(super::metadata::is_swift_bucket_metadata_tag_key)
});
}
Ok((tagging, updated))
}
/// Replace or clear S3 tags atomically while retaining Swift metadata.
///
/// The expected incarnation, when present, is checked under the bucket
/// lifecycle and metadata guards. A failed write leaves the previous config
/// intact. Repeated writes replace the same S3 set.
pub async fn update_s3_tagging_config(
bucket: &str,
tagging: Option<Tagging>,
expected_incarnation_id: Option<Uuid>,
) -> Result<OffsetDateTime> {
let tags = tagging.map(|tagging| tagging.tag_set).unwrap_or_default();
let invalid = |message: &str| Error::InvalidArgument(bucket.to_owned(), String::new(), message.to_owned());
if tags.len() > super::metadata::MAX_BUCKET_TAGS {
return Err(invalid("Cannot have more than 50 tags per bucket"));
}
super::metadata::validate_tag_set_entries(&tags).map_err(invalid)?;
update_config_with_expected(bucket, BUCKET_TAGGING_CONFIG, expected_incarnation_id, |metadata| {
let mut merged = metadata.tagging_config.clone().unwrap_or_default();
if swift_manages_bucket_tags(&merged) {
if tags.iter().any(|tag| {
tag.key
.as_deref()
.is_some_and(super::metadata::is_swift_bucket_metadata_tag_key)
}) {
return Err(invalid("Tag key is reserved for Swift metadata on this bucket"));
}
merged.tag_set.retain(|tag| {
tag.key
.as_deref()
.is_some_and(super::metadata::is_swift_bucket_metadata_tag_key)
});
} else {
merged.tag_set.clear();
}
merged.tag_set.extend(tags);
if merged.tag_set.is_empty() {
Ok(Vec::new())
} else {
super::utils::serialize(&merged).map_err(Error::other)
}
})
.await
}
/// Acquire a bucket's metadata transaction lock, held across a whole
/// read-modify-write of its metadata file.
///
@@ -1235,13 +1427,18 @@ pub(crate) async fn acquire_bucket_metadata_transaction_lock_in(
pub(crate) async fn acquire_bucket_metadata_transaction_read_lock_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
) -> Result<rustfs_lock::NamespaceLockGuard> {
acquire_bucket_metadata_transaction_read_lock_in_for_purpose(ctx, bucket, "bucket_metadata_read").await
}
pub(crate) async fn acquire_bucket_metadata_transaction_read_lock_in_for_purpose(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
op: &'static str,
) -> Result<rustfs_lock::NamespaceLockGuard> {
let sys = bucket_metadata_sys_of(ctx)?;
let api = sys.read().await.object_store();
let lock = api
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
Ok(lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?)
acquire_metadata_transaction_guard(api, bucket, op, rustfs_lock::LockType::Shared).await
}
/// Readers already holding an Object Lock snapshot must share its guard:
@@ -1250,6 +1447,7 @@ pub(crate) async fn acquire_bucket_metadata_transaction_read_lock_for_options_in
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
opts: &ObjectOptions,
op: &'static str,
) -> Result<Arc<rustfs_lock::NamespaceLockGuard>> {
if let Some(snapshot) = opts.object_lock_config_snapshot.as_ref() {
let store = object_store_in(ctx).await?;
@@ -1259,12 +1457,22 @@ pub(crate) async fn acquire_bucket_metadata_transaction_read_lock_for_options_in
Error::other("Object Lock snapshot does not hold a valid metadata transaction fence for this bucket")
});
}
Ok(Arc::new(acquire_bucket_metadata_transaction_read_lock_in(ctx, bucket).await?))
Ok(Arc::new(
acquire_bucket_metadata_transaction_read_lock_in_for_purpose(ctx, bucket, op).await?,
))
}
async fn acquire_transaction_lock_with_sys(
sys: &Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
) -> Result<rustfs_lock::NamespaceLockGuard> {
acquire_transaction_lock_with_sys_for_purpose(sys, bucket, "bucket_metadata_transaction_write").await
}
async fn acquire_transaction_lock_with_sys_for_purpose(
sys: &Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
op: &'static str,
) -> Result<rustfs_lock::NamespaceLockGuard> {
// Resolve the store under a short-lived read guard: this runs before the
// write guard in `acquire_config_write_guards`, and must not still hold a
@@ -1273,12 +1481,22 @@ async fn acquire_transaction_lock_with_sys(
let lock = api
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let acquire = lock.get_write_lock(crate::set_disk::get_lock_acquire_timeout());
let timeout = crate::set_disk::get_lock_acquire_timeout();
let mut attempt = ObjectLockAttempt::start(
op,
RUSTFS_META_BUCKET,
&bucket_metadata_transaction_lock_key(bucket),
None,
&lock,
"write",
timeout,
);
let acquire = lock.get_write_lock(timeout);
#[cfg(any(test, feature = "test-util"))]
{
tokio::pin!(acquire);
let mut notified = false;
let guard = futures::future::poll_fn(|cx| match std::future::Future::poll(acquire.as_mut(), cx) {
let result = futures::future::poll_fn(|cx| match std::future::Future::poll(acquire.as_mut(), cx) {
std::task::Poll::Pending => {
if !notified {
notify_config_write_lock_attempt(bucket);
@@ -1288,11 +1506,38 @@ async fn acquire_transaction_lock_with_sys(
}
std::task::Poll::Ready(result) => std::task::Poll::Ready(result),
})
.await?;
Ok(guard)
.await;
attempt.observe_acquisition(&result);
Ok(result?)
}
#[cfg(not(any(test, feature = "test-util")))]
Ok(acquire.await?)
{
let result = acquire.await;
attempt.observe_acquisition(&result);
Ok(result?)
}
}
pub(crate) async fn acquire_metadata_transaction_guard(
api: Arc<ECStore>,
bucket: &str,
op: &'static str,
lock_type: rustfs_lock::LockType,
) -> Result<rustfs_lock::NamespaceLockGuard> {
let object = bucket_metadata_transaction_lock_key(bucket);
let lock = api.new_ns_lock(RUSTFS_META_BUCKET, &object).await?;
let timeout = crate::set_disk::get_lock_acquire_timeout();
let mode = match lock_type {
rustfs_lock::LockType::Shared => "read",
rustfs_lock::LockType::Exclusive => "write",
};
let mut attempt = ObjectLockAttempt::start(op, RUSTFS_META_BUCKET, &object, None, &lock, mode, timeout);
let result = match lock_type {
rustfs_lock::LockType::Shared => lock.get_read_lock(timeout).await,
rustfs_lock::LockType::Exclusive => lock.get_write_lock(timeout).await,
};
attempt.observe_acquisition(&result);
Ok(result?)
}
/// The lock resource name is deliberately still the `bucket-targets` one it
@@ -2794,13 +3039,13 @@ impl BucketMetadataSys {
}
async fn find_bucket_incarnation_id_from_disk(&self, bucket: &str) -> Result<Option<Uuid>> {
let transaction_lock = self
.object_store()
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let _transaction_guard = transaction_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
let _transaction_guard = acquire_metadata_transaction_guard(
self.object_store(),
bucket,
"bucket_incarnation_disk_read",
rustfs_lock::LockType::Shared,
)
.await?;
self.find_bucket_incarnation_id_under_transaction_lock(bucket, &_transaction_guard)
.await
}
@@ -2876,13 +3121,13 @@ impl BucketMetadataSys {
/// revalidated at write quorum, so a stale snapshot can neither revert an
/// acknowledged configuration update nor outlive a delete/recreate.
async fn migrate_bucket_creation_commit(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
let transaction_lock = self
.object_store()
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let transaction_guard = transaction_lock
.get_write_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
let transaction_guard = acquire_metadata_transaction_guard(
self.object_store(),
bucket,
"bucket_creation_commit_migration",
rustfs_lock::LockType::Exclusive,
)
.await?;
let namespace_lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let namespace_guard = namespace_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
@@ -2932,13 +3177,13 @@ impl BucketMetadataSys {
}
async fn migrate_legacy_metadata(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
let transaction_lock = self
.object_store()
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let _transaction_guard = transaction_lock
.get_write_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
let _transaction_guard = acquire_metadata_transaction_guard(
self.object_store(),
bucket,
"bucket_metadata_legacy_migration",
rustfs_lock::LockType::Exclusive,
)
.await?;
let authority = self
.load_authoritative_metadata_from_disk_under_transaction_lock(bucket)
@@ -3977,6 +4222,12 @@ mod tests {
.await
.is_err()
);
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, Uuid::new_v4(), store.id)
.await
.is_err(),
"read-only scanner validation must not migrate an absent incarnation"
);
assert!(load_bucket_incarnation(store, bucket).await.expect("read sidecar").is_none());
assert!(
sys.read()
@@ -4033,6 +4284,128 @@ mod tests {
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn scanner_incarnation_read_fence_shares_reads_and_blocks_mutations() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
let bucket = "scanner-read-fence";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let incarnation = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("bucket incarnation");
let guard = acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, incarnation, store.id)
.await
.expect("read-only scanner fence");
assert_eq!(guard.checked_bucket_incarnation().expect("valid read fences"), (bucket, incarnation));
let transaction_lock = store
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await
.expect("metadata transaction namespace");
assert!(
transaction_lock.get_write_lock(Duration::from_millis(20)).await.is_err(),
"scanner read authority must exclude metadata writers"
);
let lifecycle_lock = store
.new_ns_lock(bucket, crate::storage_api_contracts::bucket::BUCKET_LIFECYCLE_LOCK_OBJECT)
.await
.expect("bucket lifecycle namespace");
assert!(
lifecycle_lock.get_write_lock(Duration::from_millis(20)).await.is_err(),
"scanner read authority must exclude delete and recreation"
);
let another = timeout(
Duration::from_secs(1),
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, incarnation, store.id),
)
.await
.expect("scanner readers must not serialize")
.expect("second read-only scanner fence");
let compatible = timeout(
Duration::from_secs(1),
acquire_bucket_metadata_transaction_read_lock_in(&store.ctx, bucket),
)
.await
.expect("metadata readers must remain compatible")
.expect("metadata transaction reader");
drop(compatible);
drop(another);
let mut mutation = Box::pin(acquire_config_write_guard_with_migration(sys.clone(), bucket, Some(incarnation), false));
assert!(matches!(futures::poll!(&mut mutation), std::task::Poll::Pending));
let mut deletion = Box::pin(store.acquire_bucket_lifecycle_write_lock(bucket));
assert!(matches!(futures::poll!(&mut deletion), std::task::Poll::Pending));
drop(deletion);
drop(guard);
let mutation = timeout(Duration::from_secs(1), mutation)
.await
.expect("metadata mutation must progress after scanner read release")
.expect("exclusive mutation fence");
drop(mutation);
let lifecycle = timeout(Duration::from_secs(1), store.acquire_bucket_lifecycle_write_lock(bucket))
.await
.expect("bucket delete fence must progress after readers release")
.expect("exclusive lifecycle fence");
drop(lifecycle);
for (owner, expected) in [
(Uuid::new_v4(), incarnation),
(Uuid::nil(), incarnation),
(store.id, Uuid::nil()),
] {
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, expected, owner)
.await
.is_err(),
"invalid owner or nil incarnation must not receive authority"
);
}
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, Uuid::new_v4(), store.id)
.await
.is_err(),
"a stale incarnation must not receive authority"
);
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("delete bucket");
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, incarnation, store.id)
.await
.is_err(),
"deleted bucket must not receive authority"
);
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("recreate bucket");
let replacement = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("replacement incarnation");
assert_ne!(replacement, incarnation);
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, incarnation, store.id)
.await
.is_err(),
"recreation must reject the previous scanner identity"
);
let guard = acquire_scanner_bucket_incarnation_read_fence_with_sys(sys, bucket, replacement, store.id)
.await
.expect("replacement read-only scanner fence");
assert_eq!(
guard.checked_bucket_incarnation().expect("replacement remains fenced"),
(bucket, replacement)
);
}
#[tokio::test]
async fn old_node_metadata_rewrite_cannot_replace_bucket_incarnation_sidecar() {
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
@@ -886,7 +886,13 @@ pub(crate) async fn begin(
}
let metadata_lock_started = rustfs_io_metrics::put_stage_timer();
let metadata_guard = metadata_sys::acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts).await?;
let metadata_guard = metadata_sys::acquire_bucket_metadata_transaction_read_lock_for_options_in(
ctx,
bucket,
opts,
"quota_context_metadata_read",
)
.await?;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_METADATA_LOCK,
metadata_lock_started,
@@ -217,6 +217,8 @@ impl Drop for NotifyOnDrop {
#[derive(Default)]
pub struct ListPathRawOptions {
/// Retain native listing admission through producer cancellation/draining.
pub read_budget: Option<Arc<tokio::sync::OwnedSemaphorePermit>>,
pub disks: Vec<Option<DiskStore>>,
pub fallback_disks: Vec<Option<DiskStore>>,
pub bucket: String,
@@ -227,6 +229,7 @@ pub struct ListPathRawOptions {
pub hidden_entry_budget: usize,
pub pending_prefix: Option<String>,
pub filter_prefix: Option<String>,
pub identifier_tree_leaf: Option<String>,
pub forward_to: Option<String>,
pub min_disks: usize,
/// Deliver every replica to the partial callback, including matching headers.
@@ -255,6 +258,7 @@ pub struct ListPathRawOptions {
impl Clone for ListPathRawOptions {
fn clone(&self) -> Self {
Self {
read_budget: self.read_budget.clone(),
disks: self.disks.clone(),
fallback_disks: self.fallback_disks.clone(),
bucket: self.bucket.clone(),
@@ -265,6 +269,7 @@ impl Clone for ListPathRawOptions {
hidden_entry_budget: self.hidden_entry_budget,
pending_prefix: self.pending_prefix.clone(),
filter_prefix: self.filter_prefix.clone(),
identifier_tree_leaf: self.identifier_tree_leaf.clone(),
forward_to: self.forward_to.clone(),
min_disks: self.min_disks,
preserve_replica_metadata: self.preserve_replica_metadata,
@@ -297,6 +302,7 @@ fn walk_dir_options(opts: &ListPathRawOptions) -> WalkDirOptions {
pending_prefix: opts.pending_prefix.clone(),
report_notfound: opts.report_not_found,
filter_prefix: opts.filter_prefix.clone(),
identifier_tree_leaf: opts.identifier_tree_leaf.clone(),
forward_to: opts.forward_to.clone(),
limit: opts.per_disk_limit,
skip_total_timeout: opts.skip_walkdir_total_timeout,
@@ -389,7 +395,9 @@ async fn list_path_raw_inner(
let disk_io_context_clone = disk_io_context.clone();
let (rd, wr) = tokio::io::duplex(64 * 1024);
readers.push(MetacacheReader::new(rd));
let read_budget = opts.read_budget.clone();
let producer = with_optional_disk_io_context(disk_io_context_clone, async move {
let _read_budget = read_budget;
let mut wr = PublishedBytesWriter::new(wr);
#[cfg(test)]
if let Some(observer) = &test_disk_io_context_observed {
@@ -1228,6 +1236,21 @@ mod tests {
assert!(!options.skip_total_timeout);
}
#[test]
fn identifier_tree_selection_survives_job_clone_and_disk_request() {
let request = ListPathRawOptions {
path: "catalog/namespaces/".to_string(),
identifier_tree_leaf: Some("table-entry.json".to_string()),
recursive: true,
..Default::default()
};
let cloned = request.clone();
let walk = walk_dir_options(&cloned);
assert_eq!(walk.identifier_tree_leaf, request.identifier_tree_leaf);
assert_eq!(walk.base_dir, request.path);
assert!(walk.recursive);
}
#[test]
fn cloned_reader_options_preserve_hidden_scan_budget_and_pending_prefix() {
let request = ListPathRawOptions {
@@ -1897,6 +1920,42 @@ mod tests {
assert_eq!(seen.lock().expect("seen mutex poisoned").as_slice(), &["bucket/object".to_string()]);
}
#[tokio::test]
async fn list_read_budget_is_retained_by_raw_producer_until_cancellation() {
let semaphore = Arc::new(tokio::sync::Semaphore::new(4));
let grant = Arc::new(semaphore.clone().acquire_many_owned(4).await.expect("listing grant"));
let weak_grant = Arc::downgrade(&grant);
let entered = Arc::new(tokio::sync::Notify::new());
let stopped = Arc::new(tokio::sync::Notify::new());
let worker = tokio::spawn(list_path_raw(
CancellationToken::new(),
ListPathRawOptions {
read_budget: Some(grant),
disks: vec![None],
min_disks: 1,
test_reader_behaviors: vec![TestReaderBehavior::StallWithSignals {
entered: entered.clone(),
stopped: stopped.clone(),
}],
..Default::default()
},
));
timeout(Duration::from_secs(5), entered.notified())
.await
.expect("producer entered");
assert!(weak_grant.strong_count() >= 2, "producer must retain a grant independently of the parent");
assert_eq!(semaphore.available_permits(), 0);
worker.abort();
assert!(worker.await.expect_err("parent cancelled").is_cancelled());
timeout(Duration::from_secs(5), stopped.notified())
.await
.expect("producer stopped");
let _next = timeout(Duration::from_secs(5), semaphore.acquire_many(4))
.await
.expect("cancelled listing releases all permits")
.expect("budget stays open");
}
#[tokio::test]
async fn list_path_raw_aborts_unresponsive_producer_after_quorum_eof() {
let result = timeout(
File diff suppressed because it is too large Load Diff
+11 -11
View File
@@ -35,17 +35,17 @@ pub use client::{
// Re-exported through `api::rpc`; not every item is consumed inside this crate.
#[allow(unused_imports)]
pub use http_auth::{
RPC_BACKGROUND_ADMISSION_HEADER, TONIC_RPC_PREFIX, build_auth_headers, build_put_file_auth_trailer,
check_and_record_signed_rpc_nonce, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
normalize_tonic_rpc_audience, rpc_background_admission, set_tonic_canonical_body_digest, set_tonic_mutation_body_digest,
set_tonic_rolling_canonical_body_digest, set_tonic_rolling_mutation_body_digest, sign_ns_scanner_capability,
sign_ns_scanner_capability_with_checkpoint_coverage, sign_ns_scanner_capability_with_tier_registry_generation,
sign_put_file_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
tonic_rpc_auth_failure_reason, verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
verify_ns_scanner_capability_with_tier_registry_generation, verify_put_file_auth_trailer, verify_put_file_capability,
verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned, verify_tonic_rpc_response_proof,
verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
RPC_BACKGROUND_ADMISSION_HEADER, RpcReplayCacheAdmissionError, TONIC_RPC_PREFIX, build_auth_headers,
build_put_file_auth_trailer, check_and_record_signed_rpc_nonce, gen_signature_headers, gen_tonic_replay_scope_headers,
gen_tonic_signature_headers, normalize_tonic_rpc_audience, rpc_auth_is_resource_exhausted, rpc_background_admission,
set_tonic_canonical_body_digest, set_tonic_mutation_body_digest, set_tonic_rolling_canonical_body_digest,
set_tonic_rolling_mutation_body_digest, sign_ns_scanner_capability, sign_ns_scanner_capability_with_checkpoint_coverage,
sign_ns_scanner_capability_with_tier_registry_generation, sign_put_file_capability, sign_tonic_rpc_response_proof,
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason, verify_ns_scanner_capability,
verify_ns_scanner_capability_with_checkpoint_coverage, verify_ns_scanner_capability_with_tier_registry_generation,
verify_put_file_auth_trailer, verify_put_file_capability, verify_rpc_signature, verify_tonic_boot_epoch_response,
verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned,
verify_tonic_rpc_response_proof, verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
};
#[cfg(test)]
pub(crate) use internode_data_transport::TcpHttpInternodeDataTransport;
@@ -18,7 +18,7 @@ use crate::cluster::rpc::client::{
node_service_time_out_client,
};
use crate::cluster::rpc::set_tonic_mutation_body_digest;
use crate::core::pools::{PoolMeta, PoolMetaWriteState};
use crate::core::pools::{PoolMeta, PoolMetaSaveGate, PoolMetaSaveGateWriteGuard};
use crate::disk::error::DiskError;
use crate::disk::error::{Error, Result};
use crate::disk::error_reduce::{BUCKET_OP_IGNORED_ERRS, is_all_buckets_not_found, reduce_write_quorum_errs};
@@ -798,12 +798,12 @@ fn try_acquire_bucket_heal_movement_guard<'a>(
}
async fn acquire_bucket_heal_write_guard<'a>(
gate: Option<&'a tokio::sync::Mutex<PoolMetaWriteState>>,
) -> Result<Option<tokio::sync::MutexGuard<'a, PoolMetaWriteState>>> {
gate: Option<&'a PoolMetaSaveGate>,
) -> Result<Option<PoolMetaSaveGateWriteGuard<'a>>> {
let Some(gate) = gate else {
return Ok(None);
};
let guard = gate.lock().await;
let guard = gate.write().await;
guard
.ensure_write_safe("bucket heal cannot run while pool metadata requires recovery")
.map_err(|e| e.narrow_to_disk().unwrap_or_else(Error::other))?;
@@ -2388,8 +2388,8 @@ mod tests {
#[tokio::test]
async fn local_bucket_heal_refuses_receiver_with_unsafe_pool_metadata() {
let gate = tokio::sync::Mutex::new(PoolMetaWriteState::default());
gate.lock().await.observe_replicas(PoolMetaReplicaState {
let gate = PoolMetaSaveGate::default();
gate.write().await.observe_replicas(PoolMetaReplicaState {
needs_repair: true,
repair_write_safe: false,
});
+85 -13
View File
@@ -27,13 +27,13 @@ use crate::disk::multipart_source::{
MULTIPART_SOURCE_PROTOCOL_VERSION, MultipartSourceOperation, MultipartSourceRequest, MultipartSourceResponse,
};
use crate::disk::{
BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION,
DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection,
DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation,
DiskOption, FileInfoVersions, FileReader, FileWriter, GuardedDeleteVersionsPayload, GuardedFileInfoVersions,
PartTransactionAction, PoolMetaNativeRecoveryRequest, PoolMetaNativeRecoveryResponse, ReadMultipleReq, ReadMultipleResp,
ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
batch_read_version_one_by_one,
BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION, BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate,
DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest,
DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo,
DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions, FileReader, FileWriter, GuardedDeleteVersionsPayload,
GuardedFileInfoVersions, PartTransactionAction, PoolMetaNativeRecoveryRequest, PoolMetaNativeRecoveryResponse,
ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo,
WalkDirOptions, batch_read_version_one_by_one,
disk_store::{
DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE,
get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout,
@@ -376,6 +376,21 @@ struct WalkCopyError {
source: io::Error,
}
impl WalkCopyError {
fn into_disk_error(self) -> DiskError {
// Only a local output write/flush proves the metacache consumer closed.
// An identical I/O kind from the remote input remains a transport failure.
if matches!(self.reason, RemoteWalkFailureReason::Write | RemoteWalkFailureReason::Flush)
&& self.source.kind() == io::ErrorKind::BrokenPipe
&& self.source.get_ref().is_none()
{
DiskError::metacache_output_stream_closed()
} else {
DiskError::Io(self.source)
}
}
}
async fn copy_stream_with_buffer<R, W>(
reader: &mut R,
writer: &mut W,
@@ -1032,6 +1047,7 @@ impl RemoteDisk {
volume: &str,
payload: Vec<u8>,
probe_only: bool,
bounded_prepare: bool,
) -> Result<DeleteVersionsResponse> {
if payload.len() > rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES {
return Err(DiskError::FileCorrupt);
@@ -1056,6 +1072,8 @@ impl RemoteDisk {
.map_err(|status| {
if status.code() == Code::Unimplemented {
DiskError::GuardedDeleteUnsupported
} else if bounded_prepare && status.code() == Code::InvalidArgument {
DiskError::UnsupportedDisk
} else {
status.into()
}
@@ -1089,6 +1107,14 @@ impl RemoteDisk {
volume: &str,
operation: DeleteRecoveryRpcOperation,
) -> Result<DeleteRecoveryRpcReply> {
let protocol_version = match &operation {
DeleteRecoveryRpcOperation::ApplyVersion { opts, .. } | DeleteRecoveryRpcOperation::ApplyVersions { opts, .. }
if opts.requires_bounded_delete_recovery =>
{
BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION
}
_ => DELETE_RECOVERY_PROTOCOL_VERSION,
};
let payload = rmp_serde::to_vec_named(&operation)?;
if payload.len() > operation.max_wire_payload_bytes() {
return Err(Error::other("delete recovery operation exceeds payload limit"));
@@ -1119,7 +1145,7 @@ impl RemoteDisk {
let mut request = Request::new(RecoverDeleteTransactionRequest {
disk: self.endpoint.to_string(),
volume: volume.to_owned(),
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
protocol_version,
operation: payload.into(),
});
let canonical_body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref())
@@ -1137,7 +1163,7 @@ impl RemoteDisk {
}
})?
.into_inner();
if response.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION {
if response.protocol_version != protocol_version {
return Err(DiskError::UnsupportedDisk);
}
if !response.success {
@@ -2085,6 +2111,7 @@ impl RemoteDisk {
failure_health_action: FailureHealthAction,
) {
if let Err(err) = operation_result
&& !err.is_metacache_output_stream_closed()
&& is_network_like_disk_error(err)
{
counter!(
@@ -2779,7 +2806,7 @@ impl DiskAPI for RemoteDisk {
}
async fn supports_guarded_delete_versions(&self) -> Result<()> {
let response = self.guarded_delete_versions_rpc("", Vec::new(), true).await?;
let response = self.guarded_delete_versions_rpc("", Vec::new(), true, false).await?;
if !response.errors.is_empty() || !response.item_errors.is_empty() {
return Err(DiskError::FileCorrupt);
}
@@ -2792,7 +2819,12 @@ impl DiskAPI for RemoteDisk {
opts: DeleteOptions,
) -> Vec<Option<DiskError>> {
let count = versions.len();
let payload = GuardedDeleteVersionsPayload { versions, opts };
let bounded_prepare = opts.requires_bounded_delete_recovery;
let payload = GuardedDeleteVersionsPayload {
versions,
opts,
bounded_recovery_protocol: bounded_prepare.then_some(BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION),
};
if let Err(error) = payload.validate() {
return vec![Some(error); count];
}
@@ -2800,7 +2832,10 @@ impl DiskAPI for RemoteDisk {
Ok(encoded) => encoded,
Err(error) => return vec![Some(error.into()); count],
};
match self.guarded_delete_versions_rpc(volume, encoded, false).await {
match self
.guarded_delete_versions_rpc(volume, encoded, false, bounded_prepare)
.await
{
Ok(response) => {
// New guarded peers must return typed errors, never lossy strings.
if response.item_errors.len() != count {
@@ -4037,7 +4072,7 @@ impl DiskAPI for RemoteDisk {
Ok(_) => return Ok(()),
Err(error) => {
failure_reason = Some(error.reason);
return Err(DiskError::Io(error.source));
return Err(error.into_disk_error());
}
}
}
@@ -9668,6 +9703,43 @@ mod tests {
}
}
#[tokio::test]
async fn remote_walk_preserves_local_metacache_consumer_close() {
let transport =
RetryingWalkDirInternodeDataTransport::with_steps(vec![WalkDirTestStep::Data(b"metacache payload".to_vec())]);
let disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
let (mut writer, reader) = tokio::io::duplex(1);
drop(reader);
let error = disk
.walk_dir(WalkDirOptions::default(), &mut writer)
.await
.expect_err("a dropped metacache consumer must stop the producer");
assert!(
error.is_metacache_output_stream_closed(),
"preserve the benign completion marker: {error:?}"
);
assert_eq!(transport.calls().len(), 1, "local consumer closure is not a remote open retry");
assert!(!disk.health.is_faulty());
}
#[tokio::test]
async fn remote_walk_keeps_remote_read_broken_pipe_as_transport_failure() {
let transport = RetryingWalkDirInternodeDataTransport::with_steps(vec![WalkDirTestStep::PartialDataThenError {
data: b"partial metacache".to_vec(),
error: io::ErrorKind::BrokenPipe.into(),
}]);
let disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
let mut writer = Vec::new();
let error = disk
.walk_dir(WalkDirOptions::default(), &mut writer)
.await
.expect_err("an upstream close after partial output is a failed walk");
assert!(!error.is_metacache_output_stream_closed());
assert!(error.contains_io_error_kind(io::ErrorKind::BrokenPipe));
assert_eq!(writer, b"partial metacache");
assert_eq!(transport.calls().len(), 1, "partial output must not be replayed");
}
#[tokio::test]
async fn walk_stream_backpressure_limits_read_ahead_and_is_cancellable() {
let mut reader = Cursor::new(vec![5; 4 * DEFAULT_READ_BUFFER_SIZE]);
@@ -423,11 +423,15 @@ impl RemoteClient {
/// peer toward the offline threshold. See issue #4567.
fn is_transport_failure(status: &tonic::Status) -> bool {
use tonic::Code;
// A valid stale-epoch request is rejected before execution. Its signed
// response refreshes the peer epoch on the existing healthy channel.
let epoch_rejection = status.code() == Code::Unavailable && status.message() == "RPC boot epoch changed";
std::error::Error::source(status).is_some()
|| matches!(
status.code(),
Code::Unavailable | Code::DeadlineExceeded | Code::Unknown | Code::Cancelled
)
|| (!epoch_rejection
&& matches!(
status.code(),
Code::Unavailable | Code::DeadlineExceeded | Code::Unknown | Code::Cancelled
))
}
async fn evict_connection(&self, op: &'static str, reason: &str, resource_summary: &str) {
@@ -2827,6 +2831,37 @@ mod tests {
}
}
#[tokio::test]
async fn stale_boot_epoch_rejections_do_not_open_lock_request_breaker() {
let addr = "http://stale-boot-epoch-lock-breaker:9000";
let client = RemoteClient::new(addr.to_string());
reset_lock_peer_health_for_test(addr);
for _ in 0..LOCK_RPC_REQUEST_BACKOFF_THRESHOLD {
let error = client
.execute_rpc::<(), _>(
"lock",
"bucket/new-object",
Duration::from_secs(1),
async { Err(tonic::Status::unavailable("RPC boot epoch changed")) },
RpcCleanup::new(None, None, None),
)
.await
.expect_err("a stale epoch still rejects the operation before execution");
assert!(error.to_string().contains("RPC boot epoch changed"));
}
let health = lock_peer_health_for_test(addr);
assert_eq!(health.request_consecutive_timeouts, 0);
assert!(health.request_backoff_until.is_none());
assert!(health.last_eviction.is_none());
let admission = client
.admit_lock_request("lock")
.expect("authenticated epoch refresh may proceed immediately");
client.release_lock_request("lock", Some(admission));
reset_lock_peer_health_for_test(addr);
}
#[test]
#[serial_test::serial]
fn test_remote_client_rpc_timeout_honors_configured_deadline() {
@@ -628,7 +628,7 @@ async fn pool_meta_native_physical_runtime_fork_after_apply_latches_original_gat
let fork = identity_epoch_fork(&fixture).await;
let since = OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("fixed original failure timestamp should be valid");
{
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
assert_eq!(state.active_transactions.load(Ordering::SeqCst), 0);
state.write_blocked = false;
state.block_context = None;
@@ -685,7 +685,7 @@ async fn pool_meta_native_physical_runtime_fork_after_apply_latches_original_gat
);
assert_committed(&fixture).await;
{
let state = fixture.store.pool_meta_save_gate.lock().await;
let state = fixture.store.pool_meta_save_gate.write().await;
assert!(
state.write_blocked,
"candidate validation must propagate its permanent latch to the original state"
@@ -706,7 +706,7 @@ async fn pool_meta_native_physical_runtime_fork_after_apply_latches_original_gat
.expect("permanently blocked recovery should return without mutation"),
"ordinary-readable repaired heads must not clear the original known-fork latch"
);
let state = fixture.store.pool_meta_save_gate.lock().await;
let state = fixture.store.pool_meta_save_gate.write().await;
assert!(state.write_blocked);
assert_eq!(
state
@@ -979,7 +979,7 @@ async fn pool_meta_native_heal_v1_with_all_upgrade_gates_disabled_keeps_exact_ra
async {
let (fixture, raw) = legacy_native_owner_fixture(POOL_META_V1_VERSION, false).await;
let guard = owner(&fixture.store).await;
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
let result = tokio::time::timeout(
std::time::Duration::from_secs(30),
fixture.store.heal_pool_meta_replica_under_fence(0, guard, &mut state),
@@ -1036,7 +1036,7 @@ async fn pool_meta_native_heal_rejects_legacy_source_with_trailing_bytes_without
let before = snapshots(&fixture.store).await;
let identities = owner_repair_identity_heads(&fixture.store).await;
let guard = owner(&fixture.store).await;
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
let result = tokio::time::timeout(
std::time::Duration::from_secs(30),
fixture.store.heal_pool_meta_replica_under_fence(1, guard, &mut state),
@@ -1110,7 +1110,7 @@ async fn pool_meta_native_heal_mixed_v1_v2_preserves_observed_format_floor() {
assert_eq!(selection.revision.version, POOL_META_VERSION, "the observed V2 format floor must be retained");
assert_eq!(selection.canonical.as_deref(), Some(semantic_v2.as_slice()));
let guard = owner(&fixture.store).await;
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
tokio::time::timeout(std::time::Duration::from_secs(30),
fixture.store.heal_pool_meta_replica_under_fence(target_pool, guard, &mut state))
.await.expect("mixed legacy heal must terminate")
@@ -1199,7 +1199,7 @@ async fn pool_meta_native_heal_rejects_v1_and_v3_prepare_in_either_pool_order()
let before = snapshots(&fixture.store).await;
let identities = owner_repair_identity_heads(&fixture.store).await;
let guard = owner(&fixture.store).await;
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
let result = tokio::time::timeout(
std::time::Duration::from_secs(30),
fixture
@@ -1301,7 +1301,7 @@ async fn pool_meta_native_heal_rejects_v1_legacy_dont_save_true_without_writes()
let before = snapshots(&fixture.store).await;
let identities = owner_repair_identity_heads(&fixture.store).await;
let guard = owner(&fixture.store).await;
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
let result = tokio::time::timeout(
std::time::Duration::from_secs(30),
fixture.store.heal_pool_meta_replica_under_fence(1, guard, &mut state),
@@ -1410,7 +1410,7 @@ async fn pool_meta_native_heal_rejects_target_v1_dont_save_true_without_writes()
let before = snapshots(&fixture.store).await;
let identities = owner_repair_identity_heads(&fixture.store).await;
let guard = owner(&fixture.store).await;
let mut state = fixture.store.pool_meta_save_gate.lock().await;
let mut state = fixture.store.pool_meta_save_gate.write().await;
let result = tokio::time::timeout(
std::time::Duration::from_secs(30),
fixture.store.heal_pool_meta_replica_under_fence(1, guard, &mut state),
File diff suppressed because it is too large Load Diff
+37 -6
View File
@@ -527,8 +527,8 @@ mod decommission_lock_order_tests {
rebalance_meta: tokio::sync::RwLock::new(other_store.rebalance_meta.read().await.clone()),
decommission_cancelers: tokio::sync::RwLock::new(vec![None; other_store.pools.len()]),
start_gate: tokio::sync::Mutex::new(()),
pool_meta_save_gate: tokio::sync::Mutex::new(
other_store.pool_meta_save_gate.lock().await.independent_clone_for_test(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::new(
other_store.pool_meta_save_gate.write().await.independent_clone_for_test(),
),
ctx,
bucket_fence_registry: Arc::default(),
@@ -1616,7 +1616,7 @@ mod decommission_lock_order_tests {
run_routine_to_a_deferral_pause(
&store,
1,
"target pool 1 no longer fits the selected mutation: source pool 0 reservation estimate is exhausted",
"target pool 1 no longer fits the selected mutation: target pool 1 reservation requires",
)
.await;
});
@@ -1647,7 +1647,7 @@ mod decommission_lock_order_tests {
run_routine_to_a_deferral_pause(
&store,
1,
"target pool 1 no longer fits the selected mutation: source pool 0 reservation estimate is exhausted",
"target pool 1 no longer fits the selected mutation: target pool 1 reservation requires",
)
.await;
});
@@ -6062,6 +6062,12 @@ mod decommission_lock_order_tests {
)
.await
.expect("read the next source object for the multipart capacity oracle");
// Reconciled bytes remain charged. With no physical room for a new
// reservation, a distinct mutation must fail instead of reusing them.
let mut exhausted_capacity = capacity_snapshot();
exhausted_capacity[2] = DecommissionPoolCapacityInfo::for_test(2, layout, 0, target_total, target_total);
set_decommission_capacity_info_overrides_for_test(other_store.id, vec![exhausted_capacity]);
let next_result = tokio::time::timeout(
Duration::from_secs(30),
data_movement::migrate_decommission_object(
@@ -7827,6 +7833,12 @@ mod decommission_lock_order_tests {
.expect("same-mutation retry should publish a readable target");
assert_eq!(retry_info.size, i64::try_from(body.len()).expect("test body size fits i64"));
// Reconciled bytes remain charged. With no physical room for a new
// reservation, a distinct mutation must fail instead of reusing them.
let mut exhausted_capacity = capacity_snapshot();
exhausted_capacity[2] = DecommissionPoolCapacityInfo::for_test(2, layout, 0, target_total, target_total);
set_decommission_capacity_info_overrides_for_test(other_store.id, vec![exhausted_capacity]);
let next_object = "data-movement-put-after-retry.bin";
let next_version_id = uuid::Uuid::new_v4().to_string();
let mut next_opts = ObjectOptions {
@@ -7841,7 +7853,11 @@ mod decommission_lock_order_tests {
let next_result = other_store
.put_object_for_data_movement(RUSTFS_META_BUCKET, next_object, &mut next_data, &next_opts, None)
.await;
assert!(next_result.is_err(), "a different mutation must not reuse A's recovered capacity");
let next_error = match next_result {
Err(error) | Ok((_, Err(error))) => error,
Ok((_, Ok(_))) => panic!("a different mutation must not reuse A's recovered capacity"),
};
assert!(next_error.to_string().contains("decommission_capacity_blocked"), "{next_error}");
let next_target_err = other_store.pools[2]
.get_object_info(
RUSTFS_META_BUCKET,
@@ -9929,6 +9945,12 @@ mod decommission_lock_order_tests {
)
.await
.expect("read the next source object for the capacity oracle");
// Reconciled bytes remain charged. With no physical room for a new
// reservation, a distinct mutation must fail instead of reusing them.
let mut exhausted_capacity = capacity_snapshot();
exhausted_capacity[2] = DecommissionPoolCapacityInfo::for_test(2, layout, 0, target_total, target_total);
set_decommission_capacity_info_overrides_for_test(other_store.id, vec![exhausted_capacity]);
let next_result = tokio::time::timeout(
Duration::from_secs(30),
data_movement::migrate_decommission_object(
@@ -10190,11 +10212,20 @@ mod decommission_lock_order_tests {
assert_eq!(reconciled_reservation.pending_target_physical_bytes, 0);
assert_eq!(reconciled_reservation.consumed_target_physical_bytes, 1);
// Reconciled bytes remain charged. With no physical room for a new
// reservation, a distinct mutation must fail instead of reusing them.
let mut exhausted_capacity = capacity_snapshot();
exhausted_capacity[2] = DecommissionPoolCapacityInfo::for_test(2, layout, 1, 2, 1);
set_decommission_capacity_info_overrides_for_test(other_store.id, vec![exhausted_capacity]);
let next_after_reconcile = other_store
.delete_object(&bucket, next_object, next_opts)
.await
.expect_err("the consumed exact-fit capacity must reject the next marker");
assert!(next_after_reconcile.to_string().contains("decommission_capacity_blocked"));
assert!(
next_after_reconcile.to_string().contains("decommission_capacity_blocked"),
"{next_after_reconcile}"
);
let next_target_err = other_store.pools[2]
.get_object_info(
&bucket,
+415 -23
View File
@@ -34,7 +34,7 @@ pub use local_snapshot::{LocalUsageSnapshot, read_snapshot as read_local_snapsho
use rustfs_data_usage::{
BucketTargetUsageInfo, BucketUsageInfo, CompressionTotalInfo, DATA_USAGE_OBJECT_NAME, DATA_USAGE_OBSERVED_OBJECT_NAME,
DataUsageCache, DataUsageInfo, DiskUsageStatus, LEGACY_DATA_USAGE_OBJECT_NAME, SizeHistogram, VersionsHistogram,
observed_data_usage_is_newer,
observed_data_usage_is_newer, observed_data_usage_is_newer_than_identity,
};
use rustfs_io_metrics::record_system_path_failure;
use rustfs_utils::path::SLASH_SEPARATOR;
@@ -227,7 +227,10 @@ pub async fn lookup_degraded_bucket_usage_baseline(store: Arc<ECStore>, bucket:
/// endpoints for up to `DATA_USAGE_CACHE_TTL_SECS` between backend reads.
#[derive(Debug, Clone)]
struct CachedDataUsageSnapshot {
/// Last successful load, retained through failures for lineage comparison.
info: Option<DataUsageInfo>,
/// Controls serving independently from the retained ordering evidence.
last_load_failed: bool,
loaded_at: tokio::time::Instant,
/// Pre-discard per-bucket sizes from the same load, retained even when the
/// snapshot is incomplete and its bucket data is discarded. Consumed only
@@ -240,6 +243,7 @@ impl CachedDataUsageSnapshot {
fn result<T>(&self, project: impl FnOnce(&DataUsageInfo) -> T) -> Result<T, Error> {
self.info
.as_ref()
.filter(|_| !self.last_load_failed)
.map(project)
.ok_or_else(|| Error::other("data usage snapshot load recently failed"))
}
@@ -275,6 +279,17 @@ fn cache_data_usage_snapshot_result<T>(
mut degraded_baseline,
observed_unavailable,
}) => {
// A primary read can fall back to an older backup. Refresh the
// retry age without replacing a newer snapshot this process has
// already read; snapshot size itself is not an ordering signal.
if let Some(previous) = cache.as_mut()
&& let Some(cached_info) = previous.info.as_ref()
&& snapshot_predates(&info, cached_info)
{
previous.loaded_at = loaded_at;
previous.last_load_failed = false;
return Some(Ok(project(cached_info)));
}
// A flaky observed read must not shrink quota coverage for a TTL
// window: carry the previous refresh's baseline entries forward,
// letting the fresh (authoritative) values win where they exist.
@@ -286,21 +301,23 @@ fn cache_data_usage_snapshot_result<T>(
let projected = project(&info);
*cache = Some(CachedDataUsageSnapshot {
info: Some(info),
last_load_failed: false,
loaded_at,
degraded_baseline,
});
Ok(projected)
}
Err(e) => {
// Keep the previous baseline through a failed refresh: quota
// admission must not lose its last grounded values because one
// backend read errored.
let degraded_baseline = cache.take().map(|cached| cached.degraded_baseline).unwrap_or_default();
*cache = Some(CachedDataUsageSnapshot {
// A failed load is negatively cached, but cannot erase the last
// good snapshot's ordering proof or grounded quota baseline.
let cached = cache.get_or_insert_with(|| CachedDataUsageSnapshot {
info: None,
last_load_failed: true,
loaded_at,
degraded_baseline,
degraded_baseline: HashMap::new(),
});
cached.loaded_at = loaded_at;
cached.last_load_failed = true;
Err(e)
}
})
@@ -1245,7 +1262,7 @@ async fn load_data_usage_from_backend_with_baseline(store: Arc<ECStore>) -> Resu
let (loaded_snapshot, source) = load_data_usage_snapshot(store.clone()).await?;
// The observed-newness gate below compares against the snapshot as
// persisted, before normalization demotes or discards anything.
let authoritative_as_persisted = loaded_snapshot.clone();
let authoritative_as_persisted = loaded_snapshot.snapshot_identity();
let (info, mut degraded_baseline) = normalize_loaded_data_usage(loaded_snapshot, source.is_authoritative()).await;
// A bucket without a converged scanner cycle behind it — a freshly joined
@@ -1263,7 +1280,7 @@ async fn load_data_usage_from_backend_with_baseline(store: Arc<ECStore>) -> Resu
// read per snapshot refresh.
let mut observed_unavailable = false;
match load_observed_data_usage_snapshot(store).await {
Ok(Some(observed)) if observed_data_usage_is_newer(&observed, &authoritative_as_persisted) => {
Ok(Some(observed)) if observed_data_usage_is_newer_than_identity(&observed, authoritative_as_persisted) => {
backfill_degraded_baseline_from_observed(&mut degraded_baseline, &observed);
}
Ok(_) => {}
@@ -2367,6 +2384,9 @@ fn unknown_bucket_reload_due(cache: &Option<CachedDataUsageSnapshot>, bucket: &s
let Some(cached) = cache.as_ref() else {
return false;
};
if cached.last_load_failed {
return false;
}
let Some(info) = cached.info.as_ref() else {
return false;
};
@@ -2382,11 +2402,35 @@ fn unknown_bucket_reload_due(cache: &Option<CachedDataUsageSnapshot>, bucket: &s
.is_none_or(|attempted_at| now.duration_since(attempted_at) >= refresh_age)
}
/// Whether a freshly loaded snapshot is older than the cached one, as when a
/// primary read fails and the loader falls back to the previous `.bkp` copy.
fn snapshot_predates(loaded: Option<SystemTime>, cached: Option<SystemTime>) -> bool {
/// Compare authoritative lineage by the writer's epoch, cycle, then timestamp
/// order. Within one lineage, compare the observation's coordinated position
/// before its timestamp; a later observation cannot revive an older lineage.
fn snapshot_predates(loaded: &DataUsageInfo, cached: &DataUsageInfo) -> bool {
let loaded_baseline = loaded
.usage_snapshot_authoritative_baseline
.unwrap_or_else(|| loaded.snapshot_identity());
let cached_baseline = cached
.usage_snapshot_authoritative_baseline
.unwrap_or_else(|| cached.snapshot_identity());
let loaded_position = (loaded_baseline.scanner_epoch, loaded_baseline.scanner_cycle);
let cached_position = (cached_baseline.scanner_epoch, cached_baseline.scanner_cycle);
if loaded_position != cached_position {
return loaded_position < cached_position;
}
let (loaded, cached) = if loaded_baseline == cached_baseline {
let loaded_observation_position = loaded.scanner_epoch.zip(loaded.scanner_cycle);
let cached_observation_position = cached.scanner_epoch.zip(cached.scanner_cycle);
if loaded_observation_position != cached_observation_position {
return loaded_observation_position < cached_observation_position;
}
(loaded.last_update, cached.last_update)
} else {
(loaded_baseline.last_update, cached_baseline.last_update)
};
match (loaded, cached) {
(Some(loaded), Some(cached)) => loaded < cached,
(Some(loaded), Some(cached)) => {
loaded < cached && !rustfs_data_usage::usage_last_update_is_untrusted_future(cached, SystemTime::now())
}
(None, Some(_)) => true,
_ => false,
}
@@ -2427,22 +2471,15 @@ where
return false;
};
let mut cache = data_usage_snapshot_cache().write().await;
let cached_last_update = cache
.as_ref()
.and_then(|cached| cached.info.as_ref())
.and_then(|info| info.last_update);
if snapshot_predates(loaded.info.last_update, cached_last_update) {
return false;
}
cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded),
tokio::time::Instant::now(),
refresh_generation,
data_usage_snapshot_generation(),
|_| (),
|info| info.buckets_usage.contains_key(bucket),
)
.is_some()
.is_some_and(|result| matches!(result, Ok(true)))
}
// Calls of `reload_unknown_bucket_usage` per bucket. Unit tests do not run in
@@ -3399,7 +3436,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: TokioMutex::new(()),
pool_meta_save_gate: TokioMutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx,
bucket_fence_registry: Arc::default(),
})
@@ -4224,6 +4261,7 @@ mod tests {
let degraded_baseline = info.bucket_sizes.clone();
*data_usage_snapshot_cache().write().await = Some(CachedDataUsageSnapshot {
info: Some(info),
last_load_failed: false,
loaded_at: tokio::time::Instant::now()
.checked_sub(age)
.expect("the monotonic clock must reach back by the requested age"),
@@ -4700,6 +4738,355 @@ mod tests {
assert_snapshot_bucket(&cached, "bucket");
}
#[tokio::test]
#[serial]
async fn expired_usage_snapshot_refresh_keeps_newer_counts_over_an_older_backup() {
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
use crate::config::com::save_config;
let (_dirs, store) = isolated_store_over_temp_disks().await;
let mut current = usage_snapshot_with_new_bucket(SystemTime::UNIX_EPOCH + Duration::from_secs(2), 100);
current.scanner_epoch = Some(4);
current.scanner_cycle = Some(11);
current.usage_snapshot_converged = Some(true);
let mut backup = data_usage_info_for_test("existing", 1, 5, SystemTime::UNIX_EPOCH + Duration::from_secs(1));
backup.scanner_epoch = Some(4);
backup.scanner_cycle = Some(10);
backup.usage_snapshot_converged = Some(true);
for (path, info) in [
(DATA_USAGE_OBJ_NAME_PATH.as_str(), &current),
(DATA_USAGE_OBJ_BACKUP_PATH.as_str(), &backup),
] {
save_config(store.clone(), path, serde_json::to_vec(info).expect("encode usage snapshot"))
.await
.expect("persist initial usage snapshot");
}
invalidate_data_usage_snapshot_cache().await;
for loaded in [
load_data_usage_from_backend_cached(store.clone()).await,
load_admin_data_usage_from_backend_cached(store.clone()).await,
] {
assert_eq!(loaded.expect("prime usage cache"), current);
}
// Bucket cleanup can reduce usage without advancing the scanner
// position. Only its publication timestamp advances.
let mut after_delete = data_usage_info_for_test("existing", 1, 3, SystemTime::UNIX_EPOCH + Duration::from_secs(3));
after_delete.scanner_epoch = current.scanner_epoch;
after_delete.scanner_cycle = current.scanner_cycle;
after_delete.usage_snapshot_converged = Some(true);
let mut stale_observation =
data_usage_info_for_test("retired-incarnation", 99, 999, SystemTime::UNIX_EPOCH + Duration::from_secs(4));
stale_observation.scanner_epoch = Some(4);
stale_observation.scanner_cycle = Some(12);
stale_observation.usage_snapshot_converged = Some(false);
stale_observation.usage_snapshot_authoritative_baseline = Some(backup.snapshot_identity());
assert!(observed_data_usage_is_newer(&stale_observation, &backup));
assert!(!observed_data_usage_is_newer(&stale_observation, &current));
save_config(
store.clone(),
DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&stale_observation).expect("encode observation of an older namespace"),
)
.await
.expect("persist an observation anchored to the old backup");
for (payload, expected) in [
(b"not a usage snapshot".to_vec(), &current),
(serde_json::to_vec(&after_delete).expect("encode post-delete usage"), &after_delete),
] {
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), payload)
.await
.expect("replace primary usage snapshot");
for cache in [data_usage_snapshot_cache(), admin_data_usage_snapshot_cache()] {
cache
.write()
.await
.as_mut()
.expect("usage cache should be populated")
.loaded_at = tokio::time::Instant::now() - Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
}
for loaded in [
load_data_usage_from_backend_cached(store.clone()).await,
load_admin_data_usage_from_backend_cached(store.clone()).await,
] {
assert_eq!(loaded.expect("refresh an expired usage cache"), *expected);
}
assert!(
!data_usage_snapshot_cache()
.read()
.await
.as_ref()
.expect("quota cache should remain populated")
.degraded_baseline
.contains_key("retired-incarnation"),
"a newer observation anchored to an older namespace must not backfill ghost usage"
);
}
invalidate_data_usage_snapshot_cache().await;
}
#[tokio::test]
#[serial]
async fn expired_admin_usage_refresh_orders_observations_by_generation_before_time() {
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
use crate::config::com::save_config;
let (_dirs, store) = isolated_store_over_temp_disks().await;
let mut authoritative = data_usage_info_for_test("bucket", 1, 10, SystemTime::UNIX_EPOCH + Duration::from_secs(100));
authoritative.scanner_epoch = Some(7);
authoritative.scanner_cycle = Some(10);
authoritative.usage_snapshot_converged = Some(true);
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&authoritative).expect("encode authoritative usage"),
)
.await
.expect("persist the common authoritative baseline");
let observation = |epoch, cycle, seconds, objects, size| {
let mut info =
data_usage_info_for_test("bucket", objects, size, SystemTime::UNIX_EPOCH + Duration::from_secs(seconds));
info.scanner_epoch = Some(epoch);
info.scanner_cycle = Some(cycle);
info.usage_snapshot_converged = Some(false);
info.usage_snapshot_authoritative_baseline = Some(authoritative.snapshot_identity());
info
};
let first = observation(7, 11, 200, 11, 1_100);
let newer_cycle = observation(7, 12, 150, 12, 1_200);
let older_cycle = observation(7, 11, 300, 99, 9_900);
let newer_epoch = observation(8, 1, 50, 1, 100);
invalidate_data_usage_snapshot_cache().await;
for (observed, expected) in [
(&first, &first),
(&newer_cycle, &newer_cycle),
(&older_cycle, &newer_cycle),
(&newer_epoch, &newer_epoch),
] {
assert!(
observed_data_usage_is_newer(observed, &authoritative),
"the persisted observation must pass the real loader's selection gate"
);
save_config(
store.clone(),
DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(observed).expect("encode scanner observation"),
)
.await
.expect("persist scanner observation");
if let Some(cached) = admin_data_usage_snapshot_cache().write().await.as_mut() {
cached.loaded_at = tokio::time::Instant::now() - Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
}
let loaded = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("refresh admin usage through the real backend loader");
assert_eq!(
&loaded,
expected,
"same-baseline observation ordering must follow epoch/cycle before wall clock: incoming={:?}",
observed.snapshot_identity()
);
}
invalidate_data_usage_snapshot_cache().await;
}
#[test]
fn expired_usage_snapshot_refresh_accepts_new_authority_over_a_later_old_observation() {
let loaded_at = tokio::time::Instant::now();
let baseline = data_usage_info_for_test("bucket", 3, 42, SystemTime::UNIX_EPOCH + Duration::from_secs(1));
let mut observed = data_usage_info_for_test("bucket", 5, 60, SystemTime::UNIX_EPOCH + Duration::from_secs(4));
observed.usage_snapshot_converged = Some(false);
observed.usage_snapshot_authoritative_baseline = Some(baseline.snapshot_identity());
let mut cache = Some(CachedDataUsageSnapshot {
info: Some(observed),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::new(),
});
let after_delete = data_usage_info_for_test("bucket", 1, 7, SystemTime::UNIX_EPOCH + Duration::from_secs(3));
let refreshed = cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded_usage_for_test(after_delete.clone())),
loaded_at,
1,
1,
DataUsageInfo::clone,
)
.expect("matching refresh generation")
.expect("a newer authority must retire observations of its predecessor");
assert_eq!(refreshed, after_delete);
}
#[test]
fn expired_usage_snapshot_refresh_orders_lineage_by_epoch_cycle_then_time() {
let loaded_at = tokio::time::Instant::now();
for (cached_position, loaded_position, loaded_time, accept) in [
((Some(2), Some(10)), (Some(1), Some(99)), 110, false),
((Some(2), Some(10)), (Some(2), Some(9)), 110, false),
((Some(2), Some(10)), (Some(3), Some(1)), 90, true),
((Some(2), Some(10)), (Some(2), Some(11)), 90, true),
((Some(2), Some(10)), (Some(2), Some(10)), 110, true),
((Some(3), None), (Some(2), Some(10)), 110, false),
((Some(2), Some(10)), (Some(3), None), 90, true),
((None, None), (None, None), 110, true),
((None, None), (None, None), 90, false),
] {
let mut previous = data_usage_info_for_test("bucket", 3, 42, SystemTime::UNIX_EPOCH + Duration::from_secs(100));
(previous.scanner_epoch, previous.scanner_cycle) = cached_position;
let mut loaded = data_usage_info_for_test("bucket", 1, 7, SystemTime::UNIX_EPOCH + Duration::from_secs(loaded_time));
(loaded.scanner_epoch, loaded.scanner_cycle) = loaded_position;
let expected = if accept { &loaded } else { &previous };
let mut cache = Some(CachedDataUsageSnapshot {
info: Some(previous.clone()),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::new(),
});
let refreshed = cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded_usage_for_test(loaded.clone())),
loaded_at,
1,
1,
DataUsageInfo::clone,
)
.expect("matching refresh generation")
.expect("a valid usage snapshot remains available");
assert_eq!(
&refreshed, expected,
"lineage ordering must match scanner publication: cached={cached_position:?}, loaded={loaded_position:?}, loaded_time={loaded_time}"
);
}
}
#[test]
fn expired_usage_snapshot_refresh_preserves_rejection_backoff_and_baseline() {
let loaded_at = tokio::time::Instant::now();
let expected = data_usage_info_for_test("bucket", 3, 42, SystemTime::UNIX_EPOCH + Duration::from_secs(2));
let mut cache = Some(CachedDataUsageSnapshot {
info: Some(expected.clone()),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::from([("bucket".to_string(), 42)]),
});
let refreshed_at = loaded_at + Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
let refreshed = cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded_usage_for_test(data_usage_info_for_test("bucket", 1, 7, SystemTime::UNIX_EPOCH))),
refreshed_at,
1,
1,
DataUsageInfo::clone,
)
.expect("matching refresh generation")
.expect("the newer cached snapshot remains available");
assert_eq!(refreshed, expected);
let retained = cache.as_ref().expect("cache should remain populated");
assert_eq!(retained.degraded_baseline.get("bucket"), Some(&42));
assert_eq!(retained.loaded_at, refreshed_at, "an older fallback must retain the TTL backoff");
let result =
cache_data_usage_snapshot_result(&mut cache, Err(Error::ErasureReadQuorum), refreshed_at, 1, 1, DataUsageInfo::clone)
.expect("matching refresh generation");
assert!(matches!(result, Err(Error::ErasureReadQuorum)));
let failed = cache.as_ref().expect("negative cache should be populated");
assert!(failed.last_load_failed);
assert_eq!(failed.info.as_ref(), Some(&expected), "failure must retain the last good ordering proof");
}
#[test]
fn expired_usage_snapshot_refresh_accepts_clock_recovery() {
let loaded_at = tokio::time::Instant::now();
let now = SystemTime::now();
let future = now + rustfs_data_usage::USAGE_LAST_UPDATE_FUTURE_TOLERANCE + Duration::from_secs(60);
let mut cache = Some(CachedDataUsageSnapshot {
info: Some(data_usage_info_for_test("bucket", 3, 42, future)),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::new(),
});
let recovered = data_usage_info_for_test("bucket", 1, 7, now);
let refreshed = cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded_usage_for_test(recovered.clone())),
loaded_at,
1,
1,
DataUsageInfo::clone,
)
.expect("matching refresh generation")
.expect("a clock recovery snapshot must not be frozen behind an untrusted timestamp");
assert_eq!(refreshed, recovered);
}
#[test]
fn expired_usage_snapshot_refresh_keeps_last_good_lineage_across_a_failed_load() {
let loaded_at = tokio::time::Instant::now();
let current = data_usage_info_for_test("bucket", 3, 42, SystemTime::UNIX_EPOCH + Duration::from_secs(2));
let mut cache = None;
let first = cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded_usage_for_test(current.clone())),
loaded_at,
1,
1,
DataUsageInfo::clone,
)
.expect("matching refresh generation")
.expect("prime the last successful snapshot");
assert_eq!(first, current);
let failed_at = loaded_at + Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
let failure =
cache_data_usage_snapshot_result(&mut cache, Err(Error::ErasureReadQuorum), failed_at, 1, 1, DataUsageInfo::clone)
.expect("matching refresh generation");
assert!(matches!(failure, Err(Error::ErasureReadQuorum)));
assert!(
fresh_cached_data_usage_snapshot(
&cache,
failed_at + Duration::from_secs(DATA_USAGE_UNKNOWN_BUCKET_REFRESH_SECS),
Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS),
|_| panic!("a failed load must not serve its retained snapshot as a success"),
)
.expect("the failed load remains inside its negative TTL")
.is_err()
);
assert!(
!unknown_bucket_reload_due(
&cache,
"unknown",
failed_at + Duration::from_secs(DATA_USAGE_UNKNOWN_BUCKET_REFRESH_SECS),
),
"an unknown bucket must not bypass the failed load's TTL"
);
let recovered_at = failed_at + Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
let recovered = cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded_usage_for_test(data_usage_info_for_test("bucket", 1, 7, SystemTime::UNIX_EPOCH))),
recovered_at,
1,
1,
DataUsageInfo::clone,
)
.expect("matching refresh generation")
.expect("a readable older backup must preserve the last good lineage");
assert_eq!(recovered, current);
assert_eq!(
cache.as_ref().expect("recovered cache").degraded_baseline.get("bucket"),
Some(&42),
"a failed load must not erase the baseline's ordering proof"
);
assert_eq!(
fresh_cached_data_usage_snapshot(&cache, recovered_at, Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS), |info| info
.objects_total_size,)
.expect("the recovered load starts a fresh TTL")
.expect("the last successful snapshot is available again"),
42
);
}
#[test]
#[serial]
fn unavailable_observed_read_keeps_previous_baseline_coverage() {
@@ -4707,6 +5094,7 @@ mod tests {
let refresh_generation = data_usage_snapshot_generation();
let mut cache = Some(CachedDataUsageSnapshot {
info: Some(data_usage_info_for_test("bucket", 1, 42, SystemTime::UNIX_EPOCH)),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::from([("observed-only".to_string(), 7_u64), ("covered".to_string(), 1)]),
});
@@ -4740,6 +5128,7 @@ mod tests {
let refresh_generation = data_usage_snapshot_generation();
let mut cache = Some(CachedDataUsageSnapshot {
info: Some(data_usage_info_for_test("stale", 1, 42, SystemTime::UNIX_EPOCH)),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::new(),
});
@@ -5928,6 +6317,7 @@ mod tests {
let mut snapshot_cache = data_usage_snapshot_cache().write().await;
*snapshot_cache = Some(CachedDataUsageSnapshot {
info: Some(successor),
last_load_failed: false,
loaded_at: tokio::time::Instant::now(),
degraded_baseline: HashMap::new(),
});
@@ -5999,6 +6389,7 @@ mod tests {
let mut snapshot_cache = data_usage_snapshot_cache().write().await;
*snapshot_cache = Some(CachedDataUsageSnapshot {
info: Some(successor),
last_load_failed: false,
loaded_at: tokio::time::Instant::now(),
degraded_baseline: HashMap::new(),
});
@@ -6051,6 +6442,7 @@ mod tests {
let stale = data_usage_info_for_test(BUCKET, 7, 294, SystemTime::now());
*data_usage_snapshot_cache().write().await = Some(CachedDataUsageSnapshot {
info: Some(stale),
last_load_failed: false,
loaded_at: tokio::time::Instant::now(),
degraded_baseline: HashMap::new(),
});
@@ -21,6 +21,7 @@ fn scalar_projection_borrows_snapshot_and_preserves_unknown_and_zero() {
info.buckets_usage.insert("empty".to_string(), BucketUsageInfo::default());
let cache = Some(CachedDataUsageSnapshot {
info: Some(info),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::new(),
});
@@ -66,6 +67,7 @@ fn projection_is_skipped_for_missing_failed_expired_or_invalidated_snapshots() {
cache = Some(CachedDataUsageSnapshot {
info: Some(data_usage_info_for_test("known", 1, 42, SystemTime::UNIX_EPOCH)),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::new(),
});
@@ -127,6 +129,33 @@ fn projected_publication_moves_bucket_maps_into_cache() {
cached.buckets_usage.get_key_value("known").expect("cached bucket").0.as_ptr(),
original_key
);
let failure = cache_data_usage_snapshot_result(
&mut cache,
Err(Error::ErasureReadQuorum),
tokio::time::Instant::now(),
1,
1,
|_: &DataUsageInfo| -> u64 { panic!("a failed load must not project its retained snapshot") },
)
.expect("matching refresh generation");
assert!(matches!(failure, Err(Error::ErasureReadQuorum)));
let retained = cache
.as_ref()
.expect("negative cache")
.info
.as_ref()
.expect("last good ordering proof");
assert_eq!(
retained
.buckets_usage
.get_key_value("known")
.expect("retained bucket")
.0
.as_ptr(),
original_key,
"a failed load must retain the snapshot without copying its bucket maps"
);
}
#[test]
@@ -682,15 +682,11 @@ async fn get_pools_info(all_disks: &[Disk]) -> Result<HashMap<i32, HashMap<i32,
return Err(Error::other("ServerNotInitialized"));
};
let mut pools_info: HashMap<i32, HashMap<i32, ErasureSetInfo>> = HashMap::new();
for d in all_disks {
let pool_info = pools_info.entry(d.pool_index).or_default();
let erasure_set = pool_info.entry(d.set_index).or_default();
if erasure_set.id == 0 {
erasure_set.id = d.set_index;
collect_pools_info(all_disks, |pool_index, set_index| {
let store = store.clone();
async move {
let load = load_data_usage_cache(
store.pools[d.pool_index as usize].disk_set[d.set_index as usize].as_ref(),
store.pools[pool_index as usize].disk_set[set_index as usize].as_ref(),
DATA_USAGE_CACHE_NAME,
);
#[cfg(feature = "diagnostic-read-version-caller")]
@@ -698,11 +694,34 @@ async fn get_pools_info(all_disks: &[Disk]) -> Result<HashMap<i32, HashMap<i32,
rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ServerInfoPools,
load,
);
match load.await {
Ok(cache) => apply_erasure_set_usage(&cache, erasure_set),
Err(_) => erasure_set.usage_error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string()),
}
load.await
}
})
.await
}
async fn collect_pools_info<F, Fut>(all_disks: &[Disk], mut load_usage: F) -> Result<HashMap<i32, HashMap<i32, ErasureSetInfo>>>
where
F: FnMut(i32, i32) -> Fut,
Fut: std::future::Future<Output = Result<rustfs_data_usage::DataUsageCache>>,
{
let mut pools_info: HashMap<i32, HashMap<i32, ErasureSetInfo>> = HashMap::new();
for d in all_disks {
let pool_info = pools_info.entry(d.pool_index).or_default();
let erasure_set = match pool_info.entry(d.set_index) {
std::collections::hash_map::Entry::Occupied(entry) => entry.into_mut(),
std::collections::hash_map::Entry::Vacant(entry) => {
let erasure_set = entry.insert(ErasureSetInfo {
id: d.set_index,
..Default::default()
});
match load_usage(d.pool_index, d.set_index).await {
Ok(cache) => apply_erasure_set_usage(&cache, erasure_set),
Err(_) => erasure_set.usage_error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string()),
}
erasure_set
}
};
erasure_set.raw_capacity += d.total_space;
erasure_set.raw_usage += d.used_space;
@@ -1241,6 +1260,59 @@ mod tests {
assert_eq!(info.deployment_id, expected_deployment_id);
}
#[tokio::test]
async fn pool_usage_is_loaded_once_per_set_including_set_zero() {
let mut disks = Vec::new();
for (pool, set, count) in [(0, 0, 16), (1, 0, 16), (1, 1, 4)] {
disks.extend((0..count).map(|_| Disk {
pool_index: pool,
set_index: set,
used_space: 10,
total_space: 100,
..Default::default()
}));
}
let mut loads = Vec::new();
let info = super::collect_pools_info(&disks, |pool, set| {
loads.push((pool, set));
async move {
if pool == 1 && set == 0 {
return Err(crate::error::Error::other("usage unavailable"));
}
let mut cache = rustfs_data_usage::DataUsageCache::default();
cache.info.name = DATA_USAGE_ROOT.to_owned();
cache.info.last_update = Some(std::time::SystemTime::UNIX_EPOCH);
cache.info.snapshot_complete = true;
cache.replace(
DATA_USAGE_ROOT,
"",
rustfs_data_usage::DataUsageEntry {
size: 25,
objects: 2,
..Default::default()
},
);
Ok(cache)
}
})
.await
.expect("collect mixed successful and unavailable usage");
assert_eq!(loads, [(0, 0), (1, 0), (1, 1)]);
for (pool, set, count) in [(0, 0, 16), (1, 0, 16), (1, 1, 4)] {
let set_info = &info[&pool][&set];
assert_eq!(set_info.id, set);
assert_eq!(set_info.raw_usage, count * 10);
assert_eq!(set_info.raw_capacity, count * 100);
if pool == 1 && set == 0 {
assert_eq!(set_info.usage_error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
} else {
assert_eq!(set_info.usage, 25, "logical usage must not multiply by drive count");
assert_eq!(set_info.objects_count, 2);
assert!(set_info.usage_error.is_none());
}
}
}
#[serial]
#[tokio::test]
async fn local_server_property_includes_runtime_stats_without_endpoint_pools() {
@@ -167,6 +167,16 @@ impl ObjectLockAttempt {
}
}
}
/// Correlate a raw guard's acquisition with its peer identities without
/// claiming ownership of the guard's lifetime. Peer release events retain
/// the identities needed to reconstruct when the actual owner releases it.
pub(crate) fn observe_acquisition(&mut self, result: &Result<NamespaceLockGuard, LockError>) {
self.observe(result);
if let Some(context) = self.0.as_mut() {
context.finished = true;
}
}
}
fn duration_ms(duration: Duration) -> u64 {
@@ -500,6 +510,45 @@ pub(crate) mod tests {
)
}
#[test]
#[serial_test::serial]
fn raw_guard_acquisition_diagnostic_does_not_claim_guard_lifetime() {
temp_env::with_vars([(rustfs_config::ENV_OBJECT_LOCK_DIAG_ENABLE, Some("true"))], || {
let capture = CapturedLockEvents::default();
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("raw guard diagnostic runtime");
tracing::subscriber::with_default(capture.subscriber(), || {
runtime.block_on(async {
let lock = namespace();
let mut attempt = ObjectLockAttempt::start(
"bucket_incarnation_disk_read",
"bucket",
"transaction.lock",
None,
&lock,
"read",
Duration::from_secs(1),
);
let result = lock.get_read_lock(Duration::from_secs(1)).await;
attempt.observe_acquisition(&result);
drop(attempt);
let guard = result.expect("retain the raw guard after diagnostic completion");
assert!(!guard.is_lock_lost());
let states = capture
.rows()
.iter()
.map(|row| row["fields"]["state"].as_str().expect("diagnostic state string").to_owned())
.collect::<Vec<_>>();
assert_eq!(states, ["acquiring", "acquired"]);
drop(guard);
assert_eq!(capture.rows().len(), 2, "the acquisition observer must not invent a guard release");
});
});
});
}
#[test]
#[serial_test::serial]
fn object_lock_diagnostics_disabled_does_not_allocate_context_or_emit() {
+31 -20
View File
@@ -148,47 +148,58 @@ pub async fn make_dir_all(path: impl AsRef<Path>) -> io::Result<()> {
native_io(move || create_dir_all_std(&path)).await
}
fn is_dir_error(e: &io::Error) -> bool {
fn is_dir_error(path: &Path, e: &io::Error) -> bool {
e.raw_os_error() == Some(libc::EISDIR)
|| e.kind() == io::ErrorKind::IsADirectory
// macOS: remove_file on a directory returns EPERM
|| (cfg!(target_os = "macos") && e.raw_os_error() == Some(libc::EPERM))
|| is_windows_dir_access_denied(path, e)
}
/// Windows reports `remove_file` on a directory (or directory symlink) as
/// ERROR_ACCESS_DENIED, which is indistinguishable from a real permission
/// failure without inspecting the entry itself.
#[cfg(windows)]
fn is_windows_dir_access_denied(path: &Path, e: &io::Error) -> bool {
use std::os::windows::fs::FileTypeExt;
e.kind() == io::ErrorKind::PermissionDenied
&& std::fs::symlink_metadata(path).is_ok_and(|metadata| {
let file_type = metadata.file_type();
file_type.is_dir() || file_type.is_symlink_dir()
})
}
#[cfg(not(windows))]
fn is_windows_dir_access_denied(_path: &Path, _e: &io::Error) -> bool {
false
}
#[tracing::instrument(level = "debug", skip_all)]
pub async fn remove(path: impl AsRef<Path>) -> io::Result<()> {
// Try remove_file first; fall back to remove_dir if it's a directory
let file_path = path.as_ref().to_path_buf();
match native_io(move || std::fs::remove_file(file_path)).await {
Ok(()) => Ok(()),
Err(e) if is_dir_error(&e) => {
let path = path.as_ref().to_path_buf();
native_io(move || std::fs::remove_dir(path)).await
}
Err(e) => Err(e),
}
let path = path.as_ref().to_path_buf();
native_io(move || remove_std(path)).await
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub async fn remove_all(path: impl AsRef<Path>) -> io::Result<()> {
// Try remove_file first; fall back to remove_dir_all if it's a directory
let file_path = path.as_ref().to_path_buf();
match native_io(move || std::fs::remove_file(file_path)).await {
let path = path.as_ref().to_path_buf();
native_io(move || match std::fs::remove_file(&path) {
Ok(()) => Ok(()),
Err(e) if is_dir_error(&e) => {
let path = path.as_ref().to_path_buf();
native_io(move || std::fs::remove_dir_all(path)).await
}
Err(e) if is_dir_error(&path, &e) => std::fs::remove_dir_all(&path),
Err(e) => Err(e),
}
})
.await
}
#[tracing::instrument(level = "debug", skip_all)]
pub fn remove_std(path: impl AsRef<Path>) -> io::Result<()> {
// Try remove_file first; fall back to remove_dir if it's a directory
match std::fs::remove_file(path.as_ref()) {
let path = path.as_ref();
match std::fs::remove_file(path) {
Ok(()) => Ok(()),
Err(e) if is_dir_error(&e) => std::fs::remove_dir(path.as_ref()),
Err(e) if is_dir_error(path, &e) => std::fs::remove_dir(path),
Err(e) => Err(e),
}
}
+370 -89
View File
@@ -562,12 +562,57 @@ fn remove_dir_all_if_exists(path: &Path) -> std::io::Result<()> {
fn snapshot_part_transaction_file(src: &Path, backup: &Path, absent: &Path) -> std::io::Result<bool> {
match std::fs::symlink_metadata(src) {
Ok(metadata) if metadata.is_file() => std::fs::hard_link(src, backup).map(|()| true),
Ok(metadata) if metadata.file_type().is_symlink() => {
Err(std::io::Error::new(ErrorKind::InvalidInput, "multipart transaction source is a symlink"))
}
Ok(_) => Err(std::io::Error::new(ErrorKind::InvalidData, "multipart transaction source is not a file")),
Err(err) if err.kind() == ErrorKind::NotFound => std::fs::write(absent, []).map(|()| false),
Err(err) => Err(err),
}
}
struct PreparedPartTxnReceipt {
src_volume_dir: PathBuf,
dst_volume_dir: PathBuf,
src_file_path: PathBuf,
dst_file_path: PathBuf,
dst_meta_path: PathBuf,
transaction_path: PathBuf,
meta: Bytes,
had_old_meta: bool,
}
struct PublishedPartTxnReceipt {
current_data_path: PathBuf,
current_meta_path: PathBuf,
transaction_path: PathBuf,
had_old_meta: bool,
}
const PART_TXN_RECEIPTS_PER_SHARD: usize = 64;
static PREPARED_PART_TXN_RECEIPTS: std::sync::LazyLock<os::ShardedPathCache<PreparedPartTxnReceipt>> =
std::sync::LazyLock::new(|| os::ShardedPathCache::new(PART_TXN_RECEIPTS_PER_SHARD));
static PUBLISHED_PART_TXN_RECEIPTS: std::sync::LazyLock<os::ShardedPathCache<PublishedPartTxnReceipt>> =
std::sync::LazyLock::new(|| os::ShardedPathCache::new(PART_TXN_RECEIPTS_PER_SHARD));
fn put_prepared_part_txn_receipt(transaction_path: &Path, entry: PreparedPartTxnReceipt) {
PREPARED_PART_TXN_RECEIPTS.insert(transaction_path, entry);
}
fn take_prepared_part_txn_receipt(transaction_path: &Path) -> Option<PreparedPartTxnReceipt> {
PREPARED_PART_TXN_RECEIPTS.remove(transaction_path)
}
fn put_published_part_txn_receipt(transaction_path: &Path, entry: PublishedPartTxnReceipt) {
PUBLISHED_PART_TXN_RECEIPTS.insert(transaction_path, entry);
}
fn take_published_part_txn_receipt(transaction_path: &Path) -> Option<PublishedPartTxnReceipt> {
PUBLISHED_PART_TXN_RECEIPTS.remove(transaction_path)
}
fn restore_part_transaction_file(current: &Path, backup: &Path, absent: &Path, restore: &Path) -> std::io::Result<()> {
match std::fs::symlink_metadata(backup) {
Ok(metadata) if metadata.is_file() => {
@@ -932,6 +977,7 @@ async fn restore_delete_rollback_with_namespace_owner(
.insert(backup_path.clone());
os::rename_all_with_owner(&backup_path, xl_path, object_dir, publication_root, namespace_owner.clone()).await
};
let restored_backup = restored.is_ok();
match restored {
Ok(()) => {}
Err(DiskError::FileNotFound) if remove_new_metadata => {
@@ -961,6 +1007,16 @@ async fn restore_delete_rollback_with_namespace_owner(
os::fsync_dir_with_owner(object_dir, namespace_owner.clone())
.await
.map_err(to_file_error)?;
}
if restored_backup {
// Renaming two hardlinks to the same inode can leave the backup in place.
match os::remove_file_with_owner(&backup_path, namespace_owner.clone()).await {
Ok(()) => {}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(to_file_error(err).into()),
}
}
if effective_durability(volume).syncs_commit_metadata() {
os::fsync_dir_with_owner(&rollback_path, namespace_owner.clone())
.await
.map_err(to_file_error)?;
@@ -3160,6 +3216,10 @@ pub(super) fn has_rename_data_test_failpoints(dst_path: &str) -> bool {
|| os::has_dir_fsync_test_failures()
|| os::file_sync_probe::is_active()
|| os::prepared_publication_test_hooks::has_any_hooks()
|| !OWNED_FILE_BEFORE_OPEN
.lock()
.expect("test file open hook lock should not be poisoned")
.is_empty()
}
#[cfg(all(not(test), feature = "test-util", target_os = "linux"))]
@@ -3745,6 +3805,25 @@ async fn read_dir_entries_with_walk_stall(path: &Path, count: i32, stall: Option
const WALK_DIR_BATCH_SIZE: usize = 32;
#[cfg(not(test))]
const WALK_DIR_BATCH_SIZE: usize = 256;
async fn read_identifier_tree_entries(path: &Path, leaf: &str, stall: Option<Duration>) -> Result<Vec<String>> {
let mut entries = read_dir_with_walk_stall(path.to_path_buf(), stall).await?;
let mut names = Vec::new();
loop {
let (next, batch, eof) = read_dir_entry_batch_with_walk_stall(entries, READ_DIR_BATCH_SIZE, stall).await?;
entries = next;
// Filter before retaining or sorting the inventory. A report flood
// costs directory iteration, not one retained name/metadata read per report.
names.extend(batch.into_iter().filter(|name| {
name.strip_suffix(SLASH_SEPARATOR)
.is_some_and(|component| component == leaf || super::is_identifier_tree_component(component))
}));
if eof {
return Ok(names);
}
}
}
const READ_DIR_BATCH_SIZE: usize = 512;
async fn read_dir_with_walk_stall(path: PathBuf, stall: Option<Duration>) -> Result<std::fs::ReadDir> {
@@ -9142,11 +9221,15 @@ impl LocalDisk {
opts: DeleteOptions,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Vec<Option<DiskError>> {
let payload = super::GuardedDeleteVersionsPayload { versions, opts };
let payload = super::GuardedDeleteVersionsPayload {
versions,
opts,
bounded_recovery_protocol: None,
};
if let Err(error) = payload.validate() {
return vec![Some(error); payload.versions.len()];
}
let super::GuardedDeleteVersionsPayload { versions, mut opts } = payload;
let super::GuardedDeleteVersionsPayload { versions, mut opts, .. } = payload;
opts.recoverable_delete = true;
let mut errors = Vec::with_capacity(versions.len());
for version in &versions {
@@ -9216,15 +9299,14 @@ impl LocalDisk {
.await;
}
let volume_dir = self.io_get_bucket_path(volume)?;
let file_path = self.io_get_object_path(volume, path)?;
let (volume_dir, file_path) = self.io_get_bucket_and_object_path(volume, path)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
let xl_path = path_join(&[file_path.as_path(), Path::new(STORAGE_FORMAT_FILE)]);
let object_path = self.get_object_path(volume, path)?;
let mut namespace_owner: Option<Arc<dyn Send + Sync>> =
Some(os::acquire_metadata_mutation_lease(&self.get_object_path(volume, path)?, namespace_owner).await);
Some(os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await);
if opts.recoverable_delete || (opts.undo_write && opts.undo_delete) {
let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await;
namespace_owner = Some(Arc::new((namespace_owner, volume_guard)));
@@ -9249,7 +9331,15 @@ impl LocalDisk {
let rollback_dir = opts.old_data_dir;
if let Some(rollback_dir) = rollback_dir
&& let Some(()) = self
.try_fused_recoverable_delete_version(volume, path, &fi, &opts, rollback_dir, namespace_owner.clone())
.try_fused_recoverable_delete_version_with_paths(
volume,
path,
&fi,
&opts,
rollback_dir,
namespace_owner.clone(),
Some((object_path, volume_dir.clone(), file_path.clone())),
)
.await?
{
return Ok(());
@@ -9989,7 +10079,7 @@ impl LocalDisk {
if !std::fs::symlink_metadata(&receipt_path)?.is_file() {
return Err(std::io::Error::new(ErrorKind::InvalidData, "delete receipt is not a regular file"));
}
std::fs::File::open(receipt_path)?
os::open_file_for_sync_std(&receipt_path)?
}
Err(err) => return Err(err),
};
@@ -11686,6 +11776,14 @@ impl LocalDisk {
W: AsyncWrite + Unpin + Send + 'a,
{
Box::pin(async move {
if opts.identifier_tree_leaf.is_some() {
let relative = current
.strip_prefix(&opts.base_dir)
.ok_or_else(|| DiskError::other("identifier-tree walk escaped its root"))?;
if relative.split('/').filter(|component| !component.is_empty()).count() > 512 {
return Err(DiskError::other("identifier-tree walk exceeds its directory depth limit"));
}
}
// The part of forward_to below this directory, taken before `current`
// loses its trailing slash.
let forward_rest = opts
@@ -11718,7 +11816,12 @@ impl LocalDisk {
// as a single logical read in API semantics.
let read_dir_started = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now);
let dir_path_abs = self.io_get_object_path(&opts.bucket, current.trim_start_matches(SLASH_SEPARATOR))?;
let read_dir_result = match read_dir_entries_with_walk_stall(&dir_path_abs, -1, stall).await {
let read_dir_result = if let Some(leaf) = opts.identifier_tree_leaf.as_deref() {
read_identifier_tree_entries(&dir_path_abs, leaf, stall).await
} else {
read_dir_entries_with_walk_stall(&dir_path_abs, -1, stall).await
};
let read_dir_result = match read_dir_result {
Err(err) if err == Error::FileNotFound && !skip_access_checks(&opts.bucket) => {
let volume_dir = self.io_get_bucket_path(&opts.bucket)?;
if let Err(access_err) = cached_access(&volume_dir).await {
@@ -12015,6 +12118,16 @@ impl LocalDisk {
return Ok(true);
}
if let Some(leaf) = opts.identifier_tree_leaf.as_deref()
&& entry != leaf
{
// Identifier directories can be namespaces at any depth,
// including components named tables or maintenance. Their
// own object metadata is outside this selected leaf scan.
schedule_dir(&mut dir_stack, format!("{name}/"), true, None, true, false);
continue;
}
let mut meta = MetaCacheEntry {
name,
..Default::default()
@@ -12064,7 +12177,8 @@ impl LocalDisk {
} else {
adaptive_batch_cap
};
let batch_names = if batch_limit > 1
let batch_names = if opts.identifier_tree_leaf.is_none()
&& batch_limit > 1
&& dir_objes.is_empty()
&& dir_stack.iter().all(|(_, _, _, scan_required, _)| !*scan_required)
{
@@ -12117,6 +12231,11 @@ impl LocalDisk {
write_metacache_obj(out, &meta).await?;
}
if opts.identifier_tree_leaf.is_some() {
// A dotted leaf cannot contain identifier-tree descendants.
continue;
}
let has_namespace_child_candidate = read.has_namespace_child_candidate;
let dir_to_skip: HashSet<String> = if has_namespace_child_candidate || !dir_objes.is_empty() {
read.data_dirs.into_iter().map(|dir| dir.to_string()).collect()
@@ -12163,6 +12282,9 @@ impl LocalDisk {
}
Err(err) => {
if err == Error::FileNotFound || err == Error::IsNotRegular {
if opts.identifier_tree_leaf.is_some() {
continue;
}
adaptive_batch_cap = 1;
// NOT an object, append to stack (with slash)
// If dirObject, but no metadata (which is unexpected) we skip it.
@@ -14314,7 +14436,8 @@ impl DiskAPI for LocalDisk {
all_cached = false;
break;
};
let Ok(source) = self.io_get_object_path(bucket, multipart_staging::upload_scope_root(path_str, true)) else {
let Ok(source) = self.io_get_object_open_path(bucket, multipart_staging::upload_scope_root(path_str, true))
else {
all_cached = false;
break;
};
@@ -14441,8 +14564,8 @@ impl DiskAPI for LocalDisk {
multipart_staging::reject_noncanonical_scope(&self.root, self.io_root(), src_volume, src_path)?;
let source_owner = self.multipart_source_owner(dst_volume, dst_path, true, false, None).await?;
let admission = self.multipart_source_admission(dst_volume, dst_path, true, false)?;
let src_volume_dir = self.io_get_bucket_path(src_volume)?;
let dst_volume_dir = self.io_get_bucket_path(dst_volume)?;
let (src_volume_dir, src_file_path) = self.io_get_bucket_and_object_path(src_volume, src_path)?;
let (dst_volume_dir, dst_file_path) = self.io_get_bucket_and_object_path(dst_volume, dst_path)?;
if !skip_access_checks(src_volume) {
super::fs::access_std(&src_volume_dir).map_err(|err| to_access_error(err, DiskError::VolumeAccessDenied))?;
}
@@ -14450,10 +14573,8 @@ impl DiskAPI for LocalDisk {
super::fs::access_std(&dst_volume_dir).map_err(|err| to_access_error(err, DiskError::VolumeAccessDenied))?;
}
let src_file_path = self.io_get_object_path(src_volume, src_path)?;
let dst_file_path = self.io_get_object_path(dst_volume, dst_path)?;
let dst_meta_path = self.io_get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
let transaction_path = self.io_get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?;
let dst_meta_path = self.io_get_object_open_path(dst_volume, &format!("{dst_path}.meta"))?;
let transaction_path = self.io_get_object_open_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?;
for path in [&src_file_path, &dst_file_path, &dst_meta_path, &transaction_path] {
check_path_length(path.to_string_lossy().as_ref())?;
}
@@ -14498,8 +14619,11 @@ impl DiskAPI for LocalDisk {
if !source.is_file() {
return Ok(Err(DiskError::FileAccessDenied));
}
if transaction_path.exists() {
return Ok(Err(DiskError::FileAccessDenied));
match std::fs::symlink_metadata(&transaction_path) {
Err(err) if err.kind() == ErrorKind::NotFound => {}
Ok(m) if m.file_type().is_symlink() => return Ok(Err(DiskError::InvalidPath)),
Ok(_) => return Ok(Err(DiskError::FileAccessDenied)),
Err(err) => return Ok(Err(to_file_error(err).into())),
}
if let Some((integrity, source, destination, expected_len)) = integrity_prep {
let stat = match std::fs::symlink_metadata(&source) {
@@ -14510,7 +14634,7 @@ impl DiskAPI for LocalDisk {
return Ok(Err(DiskError::FileCorrupt));
}
let verify_and_sync = (|| -> std::io::Result<bool> {
let mut file = std::fs::File::open(&source)?;
let mut file = os::open_file_for_sync_std(&source)?;
let mut header = [0u8; rustfs_filemeta::shard_integrity::INDEX_HEADER_SIZE];
std::io::Read::read_exact(&mut file, &mut header)?;
if header != integrity.index_header() {
@@ -14556,7 +14680,7 @@ impl DiskAPI for LocalDisk {
return Ok(Err(to_file_error(err).into()));
}
let prepare_result = (|| -> std::io::Result<()> {
let prepare_result = (|| -> std::io::Result<bool> {
let had_old_data = snapshot_part_transaction_file(
&dst_file_path,
&staging_path.join(PART_TRANSACTION_OLD_DATA),
@@ -14568,10 +14692,11 @@ impl DiskAPI for LocalDisk {
&staging_path.join(PART_TRANSACTION_OLD_META_ABSENT),
)?;
let new_meta_path = staging_path.join(PART_TRANSACTION_NEW_META);
let mut new_meta = std::fs::OpenOptions::new()
.create_new(true)
.write(true)
.open(staging_path.join(PART_TRANSACTION_NEW_META))?;
.open(&new_meta_path)?;
std::io::Write::write_all(&mut new_meta, &meta)?;
if durability.syncs_commit_metadata() {
new_meta.sync_data()?;
@@ -14579,6 +14704,7 @@ impl DiskAPI for LocalDisk {
// Windows rejects renaming a directory while one of its children is
// still open, even when the child handle shares delete access.
drop(new_meta);
let _ = std::fs::hard_link(&new_meta_path, staging_path.join(PART_TRANSACTION_PUBLISH_META));
let sync_overwrite_backup = durability.syncs_commit_metadata() && (had_old_data || had_old_meta);
if sync_overwrite_backup {
os::fsync_dir_std(&staging_path)?;
@@ -14587,13 +14713,29 @@ impl DiskAPI for LocalDisk {
if sync_overwrite_backup {
os::fsync_dir_std(transaction_parent)?;
}
Ok(())
Ok(had_old_meta)
})();
if let Err(err) = prepare_result {
let _ = remove_dir_all_if_exists(&staging_path);
return Ok(Err(to_file_error(err).into()));
}
let had_old_meta = match prepare_result {
Ok(had_old_meta) => had_old_meta,
Err(err) => {
let _ = remove_dir_all_if_exists(&staging_path);
return Ok(Err(to_file_error(err).into()));
}
};
put_prepared_part_txn_receipt(
&transaction_path,
PreparedPartTxnReceipt {
src_volume_dir,
dst_volume_dir,
src_file_path,
dst_file_path,
dst_meta_path,
transaction_path: transaction_path.clone(),
meta,
had_old_meta,
},
);
Ok(Ok(()))
})
@@ -14612,13 +14754,28 @@ impl DiskAPI for LocalDisk {
let source_owner = self.multipart_source_owner(volume, path, true, false, None).await?;
let admission = self.multipart_source_admission(volume, path, true, false)?;
let admission_source = admission.as_ref().map(|a| a.source().to_path_buf());
self.io_get_bucket_path(volume)?;
let current_data_path = self.io_get_object_path(volume, path)?;
let current_meta_path = self.io_get_object_path(volume, &format!("{path}.meta"))?;
let transaction_path = self.io_get_object_path(volume, &crate::disk::part_transaction_path(path))?;
for candidate in [&current_data_path, &current_meta_path, &transaction_path] {
check_path_length(candidate.to_string_lossy().as_ref())?;
}
let txn_rel_path = crate::disk::part_transaction_path(path);
let cached_published = self
.io_get_object_open_path(volume, &txn_rel_path)
.ok()
.and_then(|txn_open| take_published_part_txn_receipt(&txn_open));
let (current_data_path, current_meta_path, transaction_path, skip_obsolete_read_on_commit) = match cached_published {
Some(entry) => (
entry.current_data_path,
entry.current_meta_path,
entry.transaction_path,
!entry.had_old_meta,
),
None => {
let (_, current_data_path) = self.io_get_bucket_and_object_path(volume, path)?;
let current_meta_path = self.io_get_object_path(volume, &format!("{path}.meta"))?;
let transaction_path = self.io_get_object_path(volume, &txn_rel_path)?;
for candidate in [&current_data_path, &current_meta_path, &transaction_path] {
check_path_length(candidate.to_string_lossy().as_ref())?;
}
(current_data_path, current_meta_path, transaction_path, false)
}
};
let durability = effective_durability(volume);
let sync = durability.syncs_commit_metadata();
let part_number = current_data_path
@@ -14684,7 +14841,8 @@ impl DiskAPI for LocalDisk {
PartTransactionAction::Commit => (PART_TRANSACTION_NEW_META, PART_TRANSACTION_OLD_META),
PartTransactionAction::Rollback => (PART_TRANSACTION_OLD_META, PART_TRANSACTION_NEW_META),
};
if let Ok(obsolete) = std::fs::read(transaction_path.join(obsolete_name))
if !(action == PartTransactionAction::Commit && skip_obsolete_read_on_commit && !cfg!(test))
&& let Ok(obsolete) = std::fs::read(transaction_path.join(obsolete_name))
&& let Ok(obsolete) = ObjectPartInfo::unmarshal(&obsolete)
&& let Some(integrity) = obsolete.integrity
&& integrity.validate().is_ok()
@@ -14750,15 +14908,6 @@ impl DiskAPI for LocalDisk {
let admission_source = admission.as_ref().map(|a| a.source().to_path_buf());
pool_meta_native::reject_raw_path(&self.root, src_volume, src_path)?;
pool_meta_native::reject_raw_path(&self.root, dst_volume, dst_path)?;
let src_volume_dir = self.io_get_bucket_path(src_volume)?;
let dst_volume_dir = self.io_get_bucket_path(dst_volume)?;
if !skip_access_checks(src_volume) {
super::fs::access_std(&src_volume_dir).map_err(|e| to_access_error(e, DiskError::VolumeAccessDenied))?
}
if !skip_access_checks(dst_volume) {
super::fs::access_std(&dst_volume_dir).map_err(|e| to_access_error(e, DiskError::VolumeAccessDenied))?
}
let src_is_dir = has_suffix(src_path, SLASH_SEPARATOR);
let dst_is_dir = has_suffix(dst_path, SLASH_SEPARATOR);
@@ -14775,25 +14924,74 @@ impl DiskAPI for LocalDisk {
return Err(DiskError::FileAccessDenied);
}
let src_file_path = self.io_get_object_path(src_volume, src_path)?;
let dst_file_path = self.io_get_object_path(dst_volume, dst_path)?;
// warn!("rename_part src_file_path:{:?}, dst_file_path:{:?}", &src_file_path, &dst_file_path);
check_path_length(src_file_path.to_string_lossy().as_ref())?;
check_path_length(dst_file_path.to_string_lossy().as_ref())?;
let dst_meta_path = if src_is_dir {
None
let cached_prepared = if !src_is_dir {
let txn_rel = crate::disk::part_transaction_path(dst_path);
self.io_get_object_open_path(dst_volume, &txn_rel)
.ok()
.and_then(|txn_open| take_prepared_part_txn_receipt(&txn_open))
.filter(|entry| {
entry.meta == meta
&& self
.io_get_object_open_path(src_volume, src_path)
.is_ok_and(|src_open| src_open == entry.src_file_path)
})
} else {
let path = self.io_get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
check_path_length(path.to_string_lossy().as_ref())?;
Some(path)
None
};
let transaction_path = if src_is_dir {
None
} else {
Some(self.io_get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?)
let (
src_volume_dir,
dst_volume_dir,
src_file_path,
dst_file_path,
dst_meta_path,
transaction_path,
preverified_meta_and_old_flag,
) = match cached_prepared {
Some(entry) => (
entry.src_volume_dir,
entry.dst_volume_dir,
entry.src_file_path,
entry.dst_file_path,
Some(entry.dst_meta_path),
Some(entry.transaction_path),
Some(entry.had_old_meta),
),
None => {
let (src_volume_dir, src_file_path) = self.io_get_bucket_and_object_path(src_volume, src_path)?;
let (dst_volume_dir, dst_file_path) = self.io_get_bucket_and_object_path(dst_volume, dst_path)?;
if !skip_access_checks(src_volume) {
super::fs::access_std(&src_volume_dir).map_err(|e| to_access_error(e, DiskError::VolumeAccessDenied))?
}
if !skip_access_checks(dst_volume) {
super::fs::access_std(&dst_volume_dir).map_err(|e| to_access_error(e, DiskError::VolumeAccessDenied))?
}
check_path_length(src_file_path.to_string_lossy().as_ref())?;
check_path_length(dst_file_path.to_string_lossy().as_ref())?;
let dst_meta_path = if src_is_dir {
None
} else {
let path = self.io_get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
check_path_length(path.to_string_lossy().as_ref())?;
Some(path)
};
let transaction_path = if src_is_dir {
None
} else {
Some(self.io_get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?)
};
(
src_volume_dir,
dst_volume_dir,
src_file_path,
dst_file_path,
dst_meta_path,
transaction_path,
None,
)
}
};
// UploadPart is acknowledged once this rename lands, so the part data and
@@ -14852,35 +15050,43 @@ impl DiskAPI for LocalDisk {
let transaction_publish_meta = if let Some(transaction_path) = transaction_path.as_ref() {
let transaction_meta_path = transaction_path.join(PART_TRANSACTION_NEW_META);
match std::fs::File::open(&transaction_meta_path) {
Ok(mut file) => {
let file_len = file.metadata().map_err(to_file_error)?.len();
if file_len != meta.len() as u64 {
return Err(DiskError::FileCorrupt.into());
}
let mut expected_meta = Vec::with_capacity(meta.len());
std::io::Read::read_to_end(&mut std::io::Read::take(&mut file, file_len), &mut expected_meta)
.map_err(to_file_error)?;
drop(file);
if expected_meta.as_slice() != meta.as_ref() {
return Err(DiskError::FileCorrupt.into());
}
let publish_meta_path = transaction_path.join(PART_TRANSACTION_PUBLISH_META);
if !cfg!(test)
&& preverified_meta_and_old_flag.is_some()
&& publish_meta_path.is_file()
&& std::fs::metadata(&transaction_meta_path).is_ok_and(|m| m.is_file() && m.len() == meta.len() as u64)
{
Some(publish_meta_path)
} else {
match std::fs::File::open(&transaction_meta_path) {
Ok(mut file) => {
let file_len = file.metadata().map_err(to_file_error)?.len();
if file_len != meta.len() as u64 {
return Err(DiskError::FileCorrupt.into());
}
let mut expected_meta = Vec::with_capacity(meta.len());
std::io::Read::read_to_end(&mut std::io::Read::take(&mut file, file_len), &mut expected_meta)
.map_err(to_file_error)?;
drop(file);
if expected_meta.as_slice() != meta.as_ref() {
return Err(DiskError::FileCorrupt.into());
}
let publish_meta_path = transaction_path.join(PART_TRANSACTION_PUBLISH_META);
remove_file_if_exists(&publish_meta_path).map_err(to_file_error)?;
std::fs::hard_link(&transaction_meta_path, &publish_meta_path).map_err(to_file_error)?;
Some(publish_meta_path)
remove_file_if_exists(&publish_meta_path).map_err(to_file_error)?;
std::fs::hard_link(&transaction_meta_path, &publish_meta_path).map_err(to_file_error)?;
Some(publish_meta_path)
}
// Old peers know only RenamePart. The new coordinator never
// reaches rename unless prepare succeeded, so an absent
// transaction directory identifies the rolling-upgrade legacy
// path. A present directory without new.meta is corruption.
Err(err) if err.kind() == ErrorKind::NotFound => match std::fs::metadata(transaction_path) {
Ok(_) => return Err(DiskError::FileCorrupt.into()),
Err(meta_err) if meta_err.kind() == ErrorKind::NotFound => None,
Err(meta_err) => return Err(to_file_error(meta_err)),
},
Err(err) => return Err(to_file_error(err)),
}
// Old peers know only RenamePart. The new coordinator never
// reaches rename unless prepare succeeded, so an absent
// transaction directory identifies the rolling-upgrade legacy
// path. A present directory without new.meta is corruption.
Err(err) if err.kind() == ErrorKind::NotFound => match std::fs::metadata(transaction_path) {
Ok(_) => return Err(DiskError::FileCorrupt.into()),
Err(meta_err) if meta_err.kind() == ErrorKind::NotFound => None,
Err(meta_err) => return Err(to_file_error(meta_err)),
},
Err(err) => return Err(to_file_error(err)),
}
} else {
None
@@ -14943,6 +15149,19 @@ impl DiskAPI for LocalDisk {
}
}
}
if let (Some(had_old_meta), Some(txn_path), Some(meta_path)) =
(preverified_meta_and_old_flag, transaction_path.as_ref(), dst_meta_path.as_path().into())
{
put_published_part_txn_receipt(
txn_path,
PublishedPartTxnReceipt {
current_data_path: dst_file_path,
current_meta_path: meta_path.to_path_buf(),
transaction_path: txn_path.clone(),
had_old_meta,
},
);
}
Ok(true)
} else {
Ok(false)
@@ -15274,6 +15493,12 @@ impl DiskAPI for LocalDisk {
super::lifecycle_test_io::observe(&opts.bucket, None);
self.wait_for_startup_cleanup().await;
if let Some(leaf) = opts.identifier_tree_leaf.as_deref()
&& (!opts.recursive || !opts.base_dir.ends_with('/') || !super::is_identifier_tree_leaf(leaf))
{
return Err(DiskError::other("invalid identifier-tree walk options"));
}
// Callers that do not pin a stall budget inherit the configured default, so
// every local walk keeps a liveness bound even when the wrapper-level total
// timeout is skipped.
@@ -15300,7 +15525,7 @@ impl DiskAPI for LocalDisk {
let mut skip_current_dir_object = false;
let mut multipart_dir_to_skip: HashSet<String> = HashSet::new();
if opts.base_dir.ends_with(SLASH_SEPARATOR) {
if opts.identifier_tree_leaf.is_none() && opts.base_dir.ends_with(SLASH_SEPARATOR) {
if let Ok(data) = with_walk_stall_timeout(
stall,
self.read_listing_metadata(
@@ -19250,6 +19475,55 @@ mod test {
}
}
#[tokio::test]
async fn identifier_tree_walk_filters_report_batches_and_bounds_depth() {
let dir = tempfile::tempdir().expect("create disk fixture");
let root = dir.path().join("catalog/prefix");
fs::create_dir_all(root.join("sales"))
.await
.expect("create identifier directory");
fs::create_dir_all(root.join("table-entry.json"))
.await
.expect("create selected leaf");
for index in 0..READ_DIR_BATCH_SIZE * 3 + 1 {
fs::create_dir(root.join(format!("report-{index}.json")))
.await
.expect("create report directory");
}
let mut names = read_identifier_tree_entries(&root, "table-entry.json", Some(Duration::from_secs(5)))
.await
.expect("all directory batches should be filtered");
names.sort();
assert_eq!(names, ["sales/", "table-entry.json/"]);
let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 fixture path")).expect("valid endpoint");
let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk");
let options = WalkDirOptions {
bucket: "catalog".to_string(),
base_dir: "prefix/".to_string(),
recursive: true,
identifier_tree_leaf: Some("table-entry.json".to_string()),
..Default::default()
};
let mut output = Vec::new();
let mut writer = MetacacheWriter::new(&mut output);
let mut count = 0;
let error = disk
.scan_dir(
format!("prefix/{}", "a/".repeat(513)),
String::new(),
&options,
&mut writer,
&mut count,
true,
None,
)
.await
.expect_err("excessive depth must fail before filesystem I/O");
assert!(error.to_string().contains("directory depth limit"));
assert_eq!(count, 0);
}
// #4644: the stall budget bounds a single drive read, never the walk as a
// whole. A walk that keeps making progress must survive even when the total
// time spent blocked on a slow consumer dwarfs the stall timeout.
@@ -19751,6 +20025,9 @@ mod test {
);
}
// Only the Unix grouped dst dir fsync worker consults the injected
// failure; the non-Unix worker syncs by path without the hook.
#[cfg(unix)]
#[tokio::test]
#[serial_test::serial(dst_dir_fsync_group_commit)]
async fn rename_data_non_inline_dst_dir_fsync_group_commit_failure_rolls_back_fresh_put() {
@@ -19839,6 +20116,9 @@ mod test {
);
}
// Only the Unix grouped dst dir fsync worker consults the injected
// failure; the non-Unix worker syncs by path without the hook.
#[cfg(unix)]
#[tokio::test]
#[serial_test::serial(dst_dir_fsync_group_commit)]
async fn rename_data_inline_dst_dir_fsync_group_commit_failure_rolls_back_fresh_put() {
@@ -30444,6 +30724,7 @@ mod test {
expected_delete_marker: None,
delete_recovery_bucket_incarnation: None,
recoverable_delete: false,
requires_bounded_delete_recovery: false,
orphan_data_dir: false,
};
disk.delete("test-volume", "test-file.txt", delete_opts)
+149 -51
View File
@@ -15,8 +15,6 @@
//! Single-disk object rename publication and rollback. The shared execution core
//! retains instrumentation, mutation leases, and commit guards through the syscall.
#[cfg(target_os = "linux")]
use super::has_rename_data_test_failpoints;
#[cfg(all(test, windows))]
use super::run_destination_commit_directory_preparation;
#[cfg(any(not(windows), test))]
@@ -24,8 +22,8 @@ use super::should_fail_local_inline_rollback_hardlink;
use super::{
EVENT_DISK_LOCAL_ACCESS_FAILED, EVENT_DISK_LOCAL_HEAL_PURGE_FAILED, EVENT_DISK_LOCAL_RENAME_REJECTED, LOG_COMPONENT_ECSTORE,
LOG_SUBSYSTEM_DISK_LOCAL, LocalDisk, SyncMode, effective_durability, has_inline_commit_test_failpoints,
inline_metadata_rollback_dir, observe_old_current, remove_dir_all_if_exists, remove_dst_base_before_commit,
remove_file_if_exists, rename_data_versions_signature, run_inline_preparation_before_backup,
has_rename_data_test_failpoints, inline_metadata_rollback_dir, observe_old_current, remove_dir_all_if_exists,
remove_dst_base_before_commit, remove_file_if_exists, rename_data_versions_signature, run_inline_preparation_before_backup,
should_fail_after_metadata_commit, should_fail_before_old_metadata_backup, should_fail_commit_rename,
should_remove_staged_meta_before_commit, skip_access_checks, tmp_staging_mutation_path,
};
@@ -172,6 +170,44 @@ impl RenameCommitContext<'_> {
}
}
#[cfg(any(not(windows), test))]
fn inline_rollback_backup_path(staging_file_path: &Path) -> std::io::Result<PathBuf> {
let Some(staging_parent) = staging_file_path.parent() else {
return Err(std::io::Error::new(ErrorKind::InvalidInput, "missing staging metadata parent"));
};
let backup_path = if staging_file_path.file_name().and_then(|n| n.to_str()) == Some(STORAGE_FORMAT_FILE) {
staging_parent.join(STORAGE_FORMAT_FILE_BACKUP)
} else {
staging_file_path.with_extension("meta.bkp")
};
remove_file_if_exists(&backup_path)?;
Ok(backup_path)
}
#[cfg(any(not(windows), test))]
fn write_local_inline_rollback_backup_from_bytes(
staging_file_path: &Path,
old_metadata: &[u8],
sync: bool,
) -> std::io::Result<PathBuf> {
let backup_path = inline_rollback_backup_path(staging_file_path)?;
let write_res = (|| -> std::io::Result<()> {
#[cfg(test)]
run_owned_file_before_open(&backup_path);
let mut file = std::fs::File::create(&backup_path)?;
std::io::Write::write_all(&mut file, old_metadata)?;
if sync && !old_metadata.is_empty() {
file.sync_data()?;
}
Ok(())
})();
if let Err(err) = write_res {
let _ = remove_file_if_exists(&backup_path);
return Err(err);
}
Ok(backup_path)
}
#[cfg(any(not(windows), test))]
pub(super) fn create_local_inline_rollback_backup(
dst_file_path: &Path,
@@ -179,28 +215,13 @@ pub(super) fn create_local_inline_rollback_backup(
old_metadata: &[u8],
sync: bool,
) -> std::io::Result<PathBuf> {
let Some(staging_parent) = staging_file_path.parent() else {
return Err(std::io::Error::new(ErrorKind::InvalidInput, "missing staging metadata parent"));
};
let backup_path = staging_parent.join(STORAGE_FORMAT_FILE_BACKUP);
remove_file_if_exists(&backup_path)?;
let backup_path = inline_rollback_backup_path(staging_file_path)?;
if should_fail_local_inline_rollback_hardlink(dst_file_path) || std::fs::hard_link(dst_file_path, &backup_path).is_err() {
let write_res = (|| -> std::io::Result<()> {
#[cfg(test)]
run_owned_file_before_open(&backup_path);
let mut file = std::fs::File::create(&backup_path)?;
std::io::Write::write_all(&mut file, old_metadata)?;
if sync && !old_metadata.is_empty() {
file.sync_data()?;
}
Ok(())
})();
if let Err(err) = write_res {
let _ = remove_file_if_exists(&backup_path);
return Err(err);
}
} else if sync && !old_metadata.is_empty() {
std::fs::File::open(&backup_path)?.sync_data()?;
return write_local_inline_rollback_backup_from_bytes(staging_file_path, old_metadata, sync);
} else if cfg!(test) && sync && !old_metadata.is_empty() {
// Skip redundant fdatasync on hardlinked rollback backups in production;
// keep a Windows-safe sync open in tests.
os::open_file_for_sync_std(&backup_path)?.sync_data()?;
}
Ok(backup_path)
}
@@ -658,7 +679,14 @@ impl LocalDisk {
})
.collect()
};
let src_volume_dir = self.io_get_bucket_path(src_volume)?;
let src_volume_dir = if cfg!(target_os = "linux")
&& src_volume == super::super::RUSTFS_META_TMP_BUCKET
&& skip_access_checks(src_volume)
{
self.io_root().join(src_volume)
} else {
self.io_get_bucket_path(src_volume)?
};
if !skip_access_checks(src_volume)
&& let Err(e) = super::super::fs::access_std(&src_volume_dir)
{
@@ -702,8 +730,24 @@ impl LocalDisk {
} else {
STORAGE_FORMAT_FILE
};
let flat_inline_tmp = cfg!(target_os = "linux")
&& !no_inline
&& src_volume == super::super::RUSTFS_META_TMP_BUCKET
&& !preserve_upload_metadata
&& multipart_publication.is_none()
&& pool_meta_condition.is_none()
&& !fi_healing
&& !os::dst_dir_fsync_group_commit_enabled()
&& !os::file_fdatasync_group_commit_enabled()
&& !has_rename_data_test_failpoints(dst_path);
let src_file_path = match multipart_publication.as_ref() {
Some(publication) => publication.canonical_head(),
None if flat_inline_tmp => {
self.io_get_object_open_path(src_volume, format!("{src_path}.{source_metadata_name}").as_str())?
}
None if cfg!(target_os = "linux") => {
self.io_get_object_open_path(src_volume, format!("{src_path}/{source_metadata_name}").as_str())?
}
None => self.io_get_object_path(src_volume, format!("{src_path}/{source_metadata_name}").as_str())?,
};
let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
@@ -714,14 +758,13 @@ impl LocalDisk {
.data_dir
.map(|dir| rustfs_utils::path::retain_slash(dir.to_string().as_str()))
{
let src_data_path = self.io_get_object_path(
src_volume,
rustfs_utils::path::retain_slash(format!("{}/{}", src_path, data_dir).as_str()).as_str(),
)?;
let dst_data_path = self.io_get_object_path(
dst_volume,
rustfs_utils::path::retain_slash(format!("{}/{}", dst_path, data_dir).as_str()).as_str(),
)?;
let src_rel = rustfs_utils::path::retain_slash(format!("{}/{}", src_path, data_dir).as_str());
let src_data_path = if cfg!(target_os = "linux") && multipart_publication.is_none() && !fi_healing {
self.io_get_object_open_path(src_volume, src_rel.as_str())?
} else {
self.io_get_object_path(src_volume, src_rel.as_str())?
};
let dst_data_path = destination_object_path.join(data_dir.trim_matches('/'));
Some((src_data_path, dst_data_path))
} else {
@@ -804,7 +847,7 @@ impl LocalDisk {
let fused_attempt = move |is_cancelled: &dyn Fn() -> bool| -> std::io::Result<FusedCommitAttempt> {
let (guard, has_dst_buf) = prepare_rename_commit_and_destination_metadata_std(
!no_inline,
!no_inline && !flat_inline_tmp,
&src_parent_owned,
&dst_parent_owned,
&dst_file_path_owned,
@@ -876,7 +919,16 @@ impl LocalDisk {
run_owned_file_before_open(&src_file_path_owned);
let mut prepared_metadata_source =
os::create_prepared_rename_source_with_commit_guard(&src_file_path_owned, &dst_file_path_owned, &guard)?;
prepared_metadata_source.write_all(&new_dst_buf, false)?;
let cleanup_flat_tmp = || {
if flat_inline_tmp {
let _ = remove_file_if_exists(&src_file_path_owned);
}
};
if let Err(err) = prepared_metadata_source.write_all(&new_dst_buf, false) {
drop(prepared_metadata_source);
cleanup_flat_tmp();
return Err(err);
}
if !no_inline {
run_inline_preparation_before_backup(&dst_path_str);
}
@@ -884,14 +936,21 @@ impl LocalDisk {
let mut staged_rollback_path = None;
if (!no_inline || rollback_data_dir.is_none())
&& let Some(ref old_metadata) = has_dst_buf
&& (rollback_data_dir.is_some() || sync_commit_metadata || cfg!(test))
&& (rollback_data_dir.is_some() || cfg!(test))
{
let backup_path = create_local_inline_rollback_backup(
let backup_path = match create_local_inline_rollback_backup(
&dst_file_path_owned,
&src_file_path_owned,
old_metadata,
sync_commit_metadata,
)?;
) {
Ok(path) => path,
Err(err) => {
drop(prepared_metadata_source);
cleanup_flat_tmp();
return Err(err);
}
};
staged_rollback_path = Some(StagedRollbackBackup(backup_path));
}
@@ -920,6 +979,8 @@ impl LocalDisk {
return Err(err);
}
} else if sync_commit_metadata && let Err(err) = prepared_metadata_source.sync_data() {
drop(prepared_metadata_source);
cleanup_flat_tmp();
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
@@ -928,6 +989,7 @@ impl LocalDisk {
if is_cancelled() {
drop(prepared_metadata_source);
cleanup_flat_tmp();
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
@@ -944,6 +1006,8 @@ impl LocalDisk {
) {
Ok(guard) => guard,
Err(err) => {
drop(prepared_metadata_source);
cleanup_flat_tmp();
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
@@ -1027,6 +1091,8 @@ impl LocalDisk {
None,
sync_commit_metadata,
) {
drop(prepared_metadata_source);
cleanup_flat_tmp();
let _ = remove_file_if_exists(staged_backup);
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
}
@@ -1047,6 +1113,7 @@ impl LocalDisk {
)
};
if let Err(err) = rename_res {
cleanup_flat_tmp();
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
@@ -1075,8 +1142,38 @@ impl LocalDisk {
return Ok(FusedCommitAttempt::CommitErr(DiskError::Unexpected));
}
if sync_commit_metadata
&& let Err(err) = fsync_inline_commit_std(
if sync_commit_metadata {
if staged_rollback_path.is_none()
&& rollback_data_dir.is_none()
&& let Some(ref old_metadata) = has_dst_buf
&& (!no_inline || rollback_data_dir.is_none())
{
let fsync_res = (|| -> std::io::Result<()> {
if let Some(dst_parent) = dst_file_path_owned.parent() {
os::fsync_commit_directory_limited_std(dst_parent, &guard)?;
}
Ok(())
})();
if let Err(err) = fsync_res {
let rollback_ok =
write_local_inline_rollback_backup_from_bytes(&src_file_path_owned, old_metadata, true)
.and_then(|backup_path| {
let backup = StagedRollbackBackup(backup_path);
let res =
rollback_inline_metadata_std(&dst_file_path_owned, rollback_data_dir, Some(&backup));
let _ = remove_file_if_exists(backup.path());
res
})
.is_ok();
if no_inline {
if rollback_ok && let Some((_, new_data_path)) = has_data_dir_path_owned.as_ref() {
let _ = remove_dir_all_if_exists(new_data_path);
}
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
}
return Ok(FusedCommitAttempt::CommitErr(DiskError::from(err)));
}
} else if let Err(err) = fsync_inline_commit_std(
&dst_file_path_owned,
&dst_volume_dir_owned,
destination_was_absent,
@@ -1087,19 +1184,19 @@ impl LocalDisk {
.filter(|_| no_inline)
.map(|(_, new_data_path)| new_data_path.as_path()),
&guard,
)
{
if no_inline {
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
) {
if no_inline {
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
}
return Ok(FusedCommitAttempt::CommitErr(DiskError::from(err)));
}
return Ok(FusedCommitAttempt::CommitErr(DiskError::from(err)));
}
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
if let Some(parent) = src_file_path_owned.parent() {
if !flat_inline_tmp && let Some(parent) = src_file_path_owned.parent() {
let _ = std::fs::remove_dir(parent);
}
@@ -1364,7 +1461,7 @@ impl LocalDisk {
}
}
}
match publication_owned.finish_on_thread(sync_commit_metadata, staging_cleanup) {
match publication_owned.finish_on_thread(sync_commit_metadata, !cfg!(test), staging_cleanup) {
Ok(result) => Ok(MultipartFusedCommitOutcome::Committed(Box::new(RenameDataResp {
multipart_source: Some(Box::new(result)),
old_data_dir: has_old_data_dir,
@@ -1658,12 +1755,13 @@ impl LocalDisk {
);
}
if let Some((src_data_path, dst_data_path)) = has_data_dir_path.as_ref() {
#[cfg(target_os = "linux")]
#[cfg(any(target_os = "linux", windows))]
let payload_guard = if multipart_publication.is_some() {
Some(
os::multipart_payload_commit_guard(
&commit.directory_guard,
src_data_path.parent().ok_or(DiskError::InvalidPath)?,
commit.publication_root,
commit.mutation_lease.clone(),
)
.await
@@ -1672,9 +1770,9 @@ impl LocalDisk {
} else {
None
};
#[cfg(target_os = "linux")]
#[cfg(any(target_os = "linux", windows))]
let directory_guard = payload_guard.as_ref().unwrap_or(&commit.directory_guard);
#[cfg(not(target_os = "linux"))]
#[cfg(not(any(target_os = "linux", windows)))]
let directory_guard = &commit.directory_guard;
if let Err(err) = os::rename_all_with_commit_guard(
src_data_path,
+390 -58
View File
@@ -856,6 +856,12 @@ pub(super) fn inspection_with_markers_bounded(
let prepared_digest = prepared.as_deref().map(digest);
let target_digest = target.as_deref().map(digest);
let receipt_digest = receipt_bytes.as_deref().map(digest);
let finished = finished
|| (committed
&& prepared.is_none()
&& before.is_none()
&& target.is_none()
&& current_digest == journal.as_ref().and_then(|j| j.target_digest));
if let Some(journal) = &journal {
if (!finished
&& journal.target_digest != target_digest
@@ -1113,13 +1119,34 @@ pub(super) fn inspection_with_markers_bounded(
Ok((inspection, has_physical_data_dirs_or_markers))
}
type PreparedDeleteReceiptEntry = (DeleteRecoveryJournal, bool);
#[derive(Clone)]
struct PreparedDeleteReceiptEntry {
object_path: PathBuf,
bucket_path: PathBuf,
file_path: PathBuf,
staging_root: PathBuf,
journal_raw: Vec<u8>,
journal: DeleteRecoveryJournal,
sole_transaction: bool,
}
#[derive(Clone)]
struct PublishedDeleteReceiptEntry {
object_path: PathBuf,
bucket_path: PathBuf,
file_path: PathBuf,
staging_root: PathBuf,
journal_raw: Vec<u8>,
}
const PREPARED_DELETE_RECEIPTS_PER_SHARD: usize = 64;
static PREPARED_DELETE_RECEIPTS: std::sync::LazyLock<os::ShardedPathCache<PreparedDeleteReceiptEntry>> =
std::sync::LazyLock::new(|| os::ShardedPathCache::new(PREPARED_DELETE_RECEIPTS_PER_SHARD));
static PUBLISHED_DELETE_RECEIPTS: std::sync::LazyLock<os::ShardedPathCache<PublishedDeleteReceiptEntry>> =
std::sync::LazyLock::new(|| os::ShardedPathCache::new(PREPARED_DELETE_RECEIPTS_PER_SHARD));
fn put_prepared_delete_receipt(rollback: impl AsRef<Path>, entry: PreparedDeleteReceiptEntry) {
PREPARED_DELETE_RECEIPTS.insert(rollback.as_ref(), entry);
}
@@ -1128,6 +1155,27 @@ fn take_prepared_delete_receipt(rollback: &Path) -> Option<PreparedDeleteReceipt
PREPARED_DELETE_RECEIPTS.remove(rollback)
}
fn put_published_delete_receipt(rollback: impl AsRef<Path>, entry: PublishedDeleteReceiptEntry) {
PUBLISHED_DELETE_RECEIPTS.insert(rollback.as_ref(), entry);
}
fn take_published_delete_receipt(rollback: &Path) -> Option<PublishedDeleteReceiptEntry> {
PUBLISHED_DELETE_RECEIPTS.remove(rollback)
}
fn format_commit_receipt_bytes(journal_raw: &[u8], finished: bool) -> Vec<u8> {
let suffix: &[u8] = if finished {
b",\"finished\":true}"
} else {
b",\"finished\":false}"
};
let mut out = Vec::with_capacity(11 + journal_raw.len() + suffix.len());
out.extend_from_slice(b"{\"journal\":");
out.extend_from_slice(journal_raw);
out.extend_from_slice(suffix);
out
}
impl LocalDisk {
// Native MPU source closure has its own conditional protocol. Generic
// object recovery must never restore its head or acquire SourceRW under
@@ -2376,6 +2424,7 @@ impl LocalDisk {
barrier(volume, object, transaction, stage, &self.root).await
}
#[cfg(test)]
pub(super) async fn try_fused_recoverable_delete_version(
&self,
volume: &str,
@@ -2384,6 +2433,21 @@ impl LocalDisk {
opts: &DeleteOptions,
transaction: Uuid,
owner: Option<Arc<dyn Send + Sync>>,
) -> Result<Option<()>> {
self.try_fused_recoverable_delete_version_with_paths(volume, object, fi, opts, transaction, owner, None)
.await
}
#[expect(clippy::too_many_arguments)]
pub(super) async fn try_fused_recoverable_delete_version_with_paths(
&self,
volume: &str,
object: &str,
fi: &FileInfo,
opts: &DeleteOptions,
transaction: Uuid,
owner: Option<Arc<dyn Send + Sync>>,
prevalidated_paths: Option<(PathBuf, PathBuf, PathBuf)>,
) -> Result<Option<()>> {
if !opts.recoverable_delete
|| opts.undo_write
@@ -2396,11 +2460,22 @@ impl LocalDisk {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
self.wait_for_startup_cleanup().await;
let (_, file_path) = self.io_get_bucket_and_object_path(volume, object)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
let (object_path, bucket_path, file_path) = match prevalidated_paths {
Some(paths) => paths,
None => {
let obj = self.get_object_path(volume, object)?;
let (b, f) = self.io_get_bucket_and_object_path(volume, object)?;
check_path_length(f.to_string_lossy().as_ref())?;
(obj, b, f)
}
};
let xl_path = path_join(&[file_path.as_path(), Path::new(STORAGE_FORMAT_FILE)]);
let destination = file_path.join(transaction.to_string());
let staging_root = self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?;
let staging_root = if cfg!(target_os = "linux") {
self.io_root().join(RUSTFS_META_TMP_BUCKET)
} else {
self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?
};
let sync = effective_durability(volume).syncs_commit_metadata();
let incarnation = opts.delete_recovery_bucket_incarnation;
let lease = os::acquire_namespace_mutation_lease_with_owner(&destination, owner).await;
@@ -2587,6 +2662,8 @@ impl LocalDisk {
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
#[cfg(all(test, unix))]
fused_prepare_checkpoint::run(&file_path);
let mut reserved_data_path = None;
if let Some(old_path) = old_data_path {
let reserved_marker = old_path.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"));
@@ -2606,6 +2683,7 @@ impl LocalDisk {
reserved_data_path = Some((old_path, reserved_marker));
}
if let Some(ref target_bytes) = target {
// A provisional head must stay outside the published recovery transaction.
let target_in_rollback = destination.join(TARGET);
let linked_xl = staging_root.join(format!("delete-xl-{transaction}"));
let _ = std::fs::remove_file(&linked_xl);
@@ -2653,7 +2731,18 @@ impl LocalDisk {
Err(err) => return Err(err),
}
}
put_prepared_delete_receipt(destination, (journal, sole_transaction));
put_prepared_delete_receipt(
destination,
PreparedDeleteReceiptEntry {
object_path,
bucket_path,
file_path,
staging_root,
journal_raw,
journal,
sole_transaction,
},
);
Ok(Ok(Some(old_dir)))
},
)
@@ -2690,19 +2779,45 @@ impl LocalDisk {
namespace_owner: Option<Arc<dyn Send + Sync>>,
work_budget: Option<Arc<crate::object_api::DeleteRecoveryWorkBudget>>,
) -> Result<Option<(DeleteRecoveryJournal, bool, bool)>> {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) || transaction.is_nil() {
if transaction.is_nil() {
return Err(DiskError::InvalidPath);
}
let object_path = self.get_object_path(volume, object)?;
let rollback_open_path = if volume != RUSTFS_META_MULTIPART_BUCKET {
self.io_get_object_open_path(volume, object)
.ok()
.map(|p| p.join(transaction.to_string()))
} else {
None
};
let cached_prepared = rollback_open_path.as_deref().and_then(take_prepared_delete_receipt);
let (object_path, bucket_path, root, staging_root, cached_receipt) = match cached_prepared {
Some(entry) => (
entry.object_path,
Some(entry.bucket_path),
entry.file_path,
entry.staging_root,
Some((entry.journal_raw, entry.journal, entry.sole_transaction)),
),
None => {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) {
return Err(DiskError::InvalidPath);
}
let obj_path = self.get_object_path(volume, object)?;
(
obj_path,
None,
self.io_get_object_path(volume, object)?,
self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?,
None,
)
}
};
let metadata_owner = os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await;
let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await;
let owner: Option<Arc<dyn Send + Sync>> = Some(Arc::new((metadata_owner, volume_guard)));
let root = self.io_get_object_path(volume, object)?;
let rollback = root.join(transaction.to_string());
let receipt_path = rollback.join(DELETE_QUORUM_COMMITTED_FILE);
let staging_root = self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?;
let sync = effective_durability(volume).syncs_commit_metadata();
let lease = os::acquire_namespace_mutation_lease_with_owner(&receipt_path, owner).await;
let volume_str = volume.to_owned();
@@ -2712,23 +2827,38 @@ impl LocalDisk {
lease,
sync,
move || -> std::io::Result<Result<Option<(DeleteRecoveryJournal, bool, bool)>>> {
if let Some((journal, sole_transaction)) = take_prepared_delete_receipt(&rollback)
if let Some((journal_raw, journal, sole_transaction)) = cached_receipt
&& rollback.join(PREPARED).is_file()
{
let receipt = CommitReceipt {
journal: journal.clone(),
finished: false,
};
let receipt_bytes = serde_json::to_vec(&receipt).map_err(std::io::Error::other)?;
std::fs::create_dir_all(&staging_root)?;
let mut staged = tempfile::NamedTempFile::new_in(&staging_root)?;
std::io::Write::write_all(staged.as_file_mut(), &receipt_bytes)?;
if sync && !receipt_bytes.is_empty() {
staged.as_file().sync_data()?;
let tmp_receipt = staging_root.join(format!("receipt-{transaction}.tmp"));
let receipt_bytes = format_commit_receipt_bytes(&journal_raw, false);
let write_res = (|| -> std::io::Result<()> {
let mut file = std::fs::File::create(&tmp_receipt)?;
std::io::Write::write_all(&mut file, &receipt_bytes)?;
if sync && !receipt_bytes.is_empty() {
file.sync_data()?;
}
std::fs::rename(&tmp_receipt, &receipt_path)?;
if sync {
os::fsync_dir_std(&rollback)?;
}
Ok(())
})();
if let Err(err) = write_res {
let _ = std::fs::remove_file(&tmp_receipt);
return Err(err);
}
std::fs::rename(staged.path(), &receipt_path)?;
if sync {
os::fsync_dir_std(&rollback)?;
if let Some(bucket_path) = bucket_path {
put_published_delete_receipt(
rollback,
PublishedDeleteReceiptEntry {
object_path,
bucket_path,
file_path: root,
staging_root,
journal_raw,
},
);
}
return Ok(Ok(Some((journal, sole_transaction, true))));
}
@@ -2851,14 +2981,46 @@ impl LocalDisk {
Err(err) => Err(err),
}
}
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) || transaction.is_nil() {
if transaction.is_nil() {
return Err(DiskError::InvalidPath);
}
let object_path = self.get_object_path(volume, object)?;
let rollback_open_path = if volume != RUSTFS_META_MULTIPART_BUCKET {
self.io_get_object_open_path(volume, object)
.ok()
.map(|p| p.join(transaction.to_string()))
} else {
None
};
let cached_published = rollback_open_path.as_deref().and_then(take_published_delete_receipt);
let (object_path, bucket_path, staging_root, root, cached_journal_raw) = match cached_published {
Some(entry) => (
entry.object_path,
entry.bucket_path,
entry.staging_root,
entry.file_path,
Some(entry.journal_raw),
),
None => {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) {
return Err(DiskError::InvalidPath);
}
let obj_path = self.get_object_path(volume, object)?;
(
obj_path,
self.io_get_bucket_path(volume)?,
self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?,
self.io_get_object_path(volume, object)?,
None,
)
}
};
let metadata_owner = os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await;
let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await;
let owner: Option<Arc<dyn Send + Sync>> = Some(Arc::new((metadata_owner, volume_guard)));
let rollback = root.join(transaction.to_string());
let receipt_path = rollback.join(DELETE_QUORUM_COMMITTED_FILE);
let sync = effective_durability(volume).syncs_commit_metadata();
let mut claims: Vec<SnapshotLeaseDeleteClaim> = Vec::with_capacity(journal.data_dirs.len());
for data_dir in &journal.data_dirs {
@@ -2880,15 +3042,9 @@ impl LocalDisk {
claims.push(claim);
}
let bucket_path = self.io_get_bucket_path(volume)?;
let staging_root = self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?;
let root = self.io_get_object_path(volume, object)?;
let rollback = root.join(transaction.to_string());
let receipt_path = rollback.join(DELETE_QUORUM_COMMITTED_FILE);
let sync = effective_durability(volume).syncs_commit_metadata();
let data_dirs = journal.data_dirs.clone();
let has_target = journal.target_digest.is_some();
let journal_for_receipt = journal.clone();
let journal_for_receipt = cached_journal_raw.is_none().then(|| journal.clone());
let lease = os::acquire_namespace_mutation_lease_with_owner(&receipt_path, owner).await;
let prune_paths: Vec<_> = rollback
.ancestors()
@@ -2897,6 +3053,31 @@ impl LocalDisk {
.collect();
let res = os::run_blocking_namespace_operation_with_sync(lease, sync, move || -> std::io::Result<()> {
let persist_receipt = |finished: bool| -> std::io::Result<()> {
let receipt_bytes = match &cached_journal_raw {
Some(raw) => format_commit_receipt_bytes(raw, finished),
None => {
let receipt = CommitReceipt {
journal: journal_for_receipt
.clone()
.ok_or_else(|| std::io::Error::other("missing journal for receipt"))?,
finished,
};
serde_json::to_vec(&receipt).map_err(std::io::Error::other)?
}
};
std::fs::create_dir_all(&staging_root)?;
let mut staged_receipt = tempfile::NamedTempFile::new_in(&staging_root)?;
std::io::Write::write_all(staged_receipt.as_file_mut(), &receipt_bytes)?;
if sync && !receipt_bytes.is_empty() {
staged_receipt.as_file().sync_data()?;
}
std::fs::rename(staged_receipt.path(), &receipt_path)?;
if sync {
let _ = os::fsync_dir_std(&rollback);
}
Ok(())
};
let committed = format!("{DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
let recovery_committed = format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
let reserved = format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
@@ -2951,38 +3132,38 @@ impl LocalDisk {
os::fsync_dir_std(&root)?;
}
for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] {
ignore_not_found(std::fs::remove_file(rollback.join(name)))?;
}
if has_target || had_physical_data_dirs {
let finished_receipt = CommitReceipt {
journal: journal_for_receipt,
finished: true,
};
let receipt_bytes = serde_json::to_vec(&finished_receipt).map_err(std::io::Error::other)?;
std::fs::create_dir_all(&staging_root)?;
let mut staged_receipt = tempfile::NamedTempFile::new_in(&staging_root)?;
std::io::Write::write_all(staged_receipt.as_file_mut(), &receipt_bytes)?;
if sync && !receipt_bytes.is_empty() {
staged_receipt.as_file().sync_data()?;
}
std::fs::rename(staged_receipt.path(), &receipt_path)?;
let _ = std::fs::File::create(rollback.join(FINISHED))?;
if sync {
os::fsync_dir_std(&rollback)?;
if let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name))) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
}
for name in [TARGET, PREPARED, FINISHED] {
for name in [PREPARED, TARGET, FINISHED] {
let path = rollback.join(name);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
ignore_not_found(std::fs::remove_file(path))?;
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
}
if sync {
os::fsync_dir_std(&rollback)?;
if sync && let Err(err) = os::fsync_dir_std(&rollback) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &receipt_path);
ignore_not_found(std::fs::remove_file(&receipt_path))?;
if let Err(err) = ignore_not_found(std::fs::remove_file(&receipt_path)) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
for dir_path in &prune_paths {
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, dir_path);
@@ -3105,6 +3286,42 @@ pub(super) async fn restore_barrier(volume: &str, object_dir: &Path, transaction
Ok(())
}
#[cfg(all(test, unix))]
mod fused_prepare_checkpoint {
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::{LazyLock, Mutex};
type Hook = Box<dyn FnOnce() + Send>;
static HOOKS: LazyLock<Mutex<HashMap<PathBuf, Hook>>> = LazyLock::new(|| Mutex::new(HashMap::new()));
pub(super) struct Guard(PathBuf);
impl Drop for Guard {
fn drop(&mut self) {
HOOKS.lock().expect("checkpoint hooks").remove(&self.0);
}
}
pub(super) fn install(root: &Path, hook: impl FnOnce() + Send + 'static) -> Guard {
assert!(
HOOKS
.lock()
.expect("checkpoint hooks")
.insert(root.to_path_buf(), Box::new(hook))
.is_none()
);
Guard(root.to_path_buf())
}
pub(super) fn run(root: &Path) {
let hook = HOOKS.lock().expect("checkpoint hooks").remove(root);
if let Some(hook) = hook {
hook();
}
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -7020,6 +7237,121 @@ mod tests {
marker.deleted = false;
}
#[cfg(unix)]
#[tokio::test]
async fn delete_recovery_fused_prepare_checkpoint_reopens_before_head_publication() {
use std::os::unix::fs::MetadataExt;
let _mode = durability_mode_override::set(DurabilityMode::Strict);
let (_temp, disk, mut first, _) = fixture().await;
first.version_id = Some(Uuid::new_v4());
let mut second = first.clone();
second.version_id = Some(Uuid::new_v4());
let mut meta = FileMeta::new();
meta.add_version(first).expect("add surviving version");
meta.add_version(second.clone()).expect("add deleted version");
let original = meta.marshal_msg().expect("encode original versions");
let (_, root) = disk.io_get_bucket_and_object_path("recovery", "object").expect("object path");
fs::write(root.join(STORAGE_FORMAT_FILE), &original)
.await
.expect("write versions");
let (checkpoint_temp, checkpoint_disk, _, _) = fixture().await;
let checkpoint_root = checkpoint_temp.path().join("recovery/object");
drop(checkpoint_disk);
let transaction = Uuid::new_v4();
let source_root = root.clone();
let copy_root = checkpoint_root.clone();
let _capture = fused_prepare_checkpoint::install(&root, move || {
let source = source_root.join(transaction.to_string());
let destination = copy_root.join(transaction.to_string());
std::fs::create_dir(&destination).expect("create checkpoint transaction");
let source_head = source_root.join(STORAGE_FORMAT_FILE);
let source_head_meta = std::fs::metadata(&source_head).expect("stat source head");
let copy_head = copy_root.join(STORAGE_FORMAT_FILE);
std::fs::copy(&source_head, &copy_head).expect("capture head before publication");
for entry in std::fs::read_dir(source).expect("read published preparation") {
let entry = entry.expect("prepared file");
assert!(entry.file_type().expect("prepared file type").is_file());
let dst = destination.join(entry.file_name());
if entry.file_name() == STORAGE_FORMAT_FILE_BACKUP {
let backup_meta = entry.metadata().expect("stat prepared backup");
assert_eq!(
(backup_meta.dev(), backup_meta.ino()),
(source_head_meta.dev(), source_head_meta.ino()),
"fused preparation must hardlink backup to live head"
);
std::fs::hard_link(&copy_head, &dst).expect("preserve hardlinked backup");
} else {
std::fs::copy(entry.path(), dst).expect("capture prepared file");
}
}
});
let opts = DeleteOptions {
recoverable_delete: true,
..Default::default()
};
assert_eq!(
disk.try_fused_recoverable_delete_version("recovery", "object", &second, &opts, transaction, None)
.await
.expect("finish original preparation"),
Some(())
);
assert!(checkpoint_root.join(transaction.to_string()).join(PREPARED).is_file());
let endpoint =
Endpoint::try_from(checkpoint_temp.path().to_str().expect("checkpoint path")).expect("checkpoint endpoint");
let reopened = LocalDisk::new(&endpoint, false)
.await
.expect("reopen checkpoint as a fresh disk");
let inspection = reopened
.delete_recovery_inspect("recovery", "object", transaction)
.await
.expect("actual preparation checkpoint must remain recoverable");
assert_eq!(inspection.state, DeleteRecoveryState::NewPrepared);
let reopened_tx = reopened
.io_get_object_path("recovery", &format!("object/{transaction}"))
.expect("reopened transaction path");
os::fsync_dir_recorder::set_failure(&reopened_tx, ErrorKind::Other);
assert!(
reopened
.settle_delete_transaction(
"recovery",
"object",
transaction,
inspection.fingerprint,
DeleteRecoveryAction::Rollback
)
.await
.is_err(),
"failed transaction directory sync after backup removal must retain journal"
);
assert!(reopened_tx.join(PREPARED).is_file());
assert!(!reopened_tx.join(STORAGE_FORMAT_FILE_BACKUP).exists());
let retry_inspection = reopened
.delete_recovery_inspect("recovery", "object", transaction)
.await
.expect("inspect interrupted rollback");
assert_eq!(retry_inspection.state, DeleteRecoveryState::NewPrepared);
let settled = reopened
.settle_delete_transaction(
"recovery",
"object",
transaction,
retry_inspection.fingerprint,
DeleteRecoveryAction::Rollback,
)
.await
.expect("roll back interrupted preparation");
assert_eq!(settled.state, DeleteRecoveryState::Absent, "same-inode backup must not survive rollback");
assert!(!checkpoint_root.join(transaction.to_string()).exists());
assert_eq!(
fs::read(checkpoint_root.join(STORAGE_FORMAT_FILE))
.await
.expect("read rolled-back versions"),
original
);
}
#[tokio::test]
async fn delete_recovery_fused_versioned_delete_with_surviving_version_and_interrupted_target_cleanup() {
let (_temp, disk, mut v1, _) = fixture().await;
@@ -372,29 +372,32 @@ impl IndexOwner {
#[cfg(unix)]
{
use rustix::fs::{Mode, OFlags, mkdirat, openat};
let mut directories = vec![self._identity_pin.try_clone()?];
for component in [".rustfs.sys", "multipart-index-v1", &sha(bucket)] {
let parent = directories.last().ok_or_else(|| io::Error::other("index parent missing"))?;
match mkdirat(parent, component, Mode::RWXU | Mode::RWXG | Mode::RWXO) {
Ok(()) | Err(rustix::io::Errno::EXIST) => {}
Err(error) => return Err(error.into()),
let open_dir = |parent: rustix::fd::BorrowedFd<'_>, component: &str| -> io::Result<rustix::fd::OwnedFd> {
let flags = OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC;
match openat(parent, component, flags, Mode::empty()) {
Ok(dir) => Ok(dir),
Err(rustix::io::Errno::NOENT) => {
match mkdirat(parent, component, Mode::RWXU | Mode::RWXG | Mode::RWXO) {
Ok(()) | Err(rustix::io::Errno::EXIST) => {}
Err(error) => return Err(error.into()),
}
openat(parent, component, flags, Mode::empty()).map_err(Into::into)
}
Err(error) => Err(error.into()),
}
let directory = openat(
parent,
component,
OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC,
Mode::empty(),
)?;
directories.push(std::fs::File::from(directory));
}
let parent = directories.last().ok_or_else(|| io::Error::other("index parent missing"))?;
};
use std::os::fd::AsFd;
let bucket_sha = sha(bucket);
let sys_dir = open_dir(self._identity_pin.as_fd(), ".rustfs.sys")?;
let index_dir = open_dir(sys_dir.as_fd(), "multipart-index-v1")?;
let parent = open_dir(index_dir.as_fd(), &bucket_sha)?;
// The index is boot-local: every start rescans the staging area
// before serving it, so a marker lost to a crash is recreated before
// it can be consulted. Creation is enough; no fsync of the marker or
// its ancestors. An existing marker is touched so pruning sees the
// new upload's age, not the previous one's.
match openat(
parent,
&parent,
source_root,
OFlags::CREATE | OFlags::EXCL | OFlags::WRONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC,
Mode::RUSR | Mode::WUSR,
@@ -409,7 +412,7 @@ impl IndexOwner {
// the previous upload's age, which only shortens the prune
// grace for this key; index completeness does not depend on it.
if let Ok(marker) =
openat(parent, source_root, OFlags::WRONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC, Mode::empty())
openat(&parent, source_root, OFlags::WRONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC, Mode::empty())
{
let _ = std::fs::File::from(marker).set_modified(std::time::SystemTime::now());
}
@@ -430,6 +433,18 @@ impl IndexOwner {
}
fn ensure_current_root(&self) -> io::Result<()> {
#[cfg(unix)]
let current = {
use std::os::unix::fs::MetadataExt;
let metadata = std::fs::symlink_metadata(&*self.root_path)?;
if !metadata.is_dir() {
return Err(io::Error::other("multipart index physical root is not a directory"));
}
let mut inode = [0; 16];
inode[..8].copy_from_slice(&metadata.ino().to_le_bytes());
(metadata.dev(), inode)
};
#[cfg(not(unix))]
let current = os::PublicationRoot::new(&self.root_path)?.physical_identity()?;
if current != self.physical {
self.disabled.store(true, Ordering::Release);
@@ -1517,7 +1517,7 @@ impl LocalDisk {
}
Err(error) => return Err(error),
};
let _directory = os::pin_multipart_source_directory(&directory, base, root)?;
let directory_pin = os::pin_multipart_source_directory(&directory, base, root)?;
if !parts.is_empty() {
let anchor_path = directory.join(RETIREMENT_ANCHOR);
let bytes = os::read_multipart_source_metadata(&anchor_path, base, root)?;
@@ -1550,7 +1550,13 @@ impl LocalDisk {
}
// An anchor-less empty DD is only rmdir'ed. A new child makes
// this fail closed; no recursive removal can consume its bytes.
// A Windows pin refuses DELETE sharing, so the leaf pin is
// released first; the object pin still holds the parent.
#[cfg(windows)]
drop(directory_pin);
std::fs::remove_dir(&directory)?;
#[cfg(not(windows))]
drop(directory_pin);
if sync {
os::fsync_dir_std(&object_path)?;
}
@@ -1973,25 +1979,28 @@ impl NativeMultipartPublication {
let canonical = self.canonical_head();
let prepared = (|| -> std::io::Result<os::PreparedRenameSource> {
if let Some((ref original_bytes, _, _)) = deferred_initial {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let snapshot = current
.snapshot
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_head = current
.head
.as_ref()
.and_then(SourceHead::raw)
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
if !current.head_matches_state
|| snapshot.phase != MultipartSourcePhase::Active
|| snapshot.capability_sha256.is_some()
|| snapshot.lineage != expected_lineage
|| snapshot.generation != expected_generation
|| snapshot.revision.checked_add(1) != Some(expected_revision)
|| current_head != original_bytes.as_slice()
{
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
if cfg!(test) || !can_mark_presynced {
let current =
load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let snapshot = current
.snapshot
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_head = current
.head
.as_ref()
.and_then(SourceHead::raw)
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
if !current.head_matches_state
|| snapshot.phase != MultipartSourcePhase::Active
|| snapshot.capability_sha256.is_some()
|| snapshot.lineage != expected_lineage
|| snapshot.generation != expected_generation
|| snapshot.revision.checked_add(1) != Some(expected_revision)
|| current_head != original_bytes.as_slice()
{
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
}
} else {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
@@ -2044,7 +2053,7 @@ impl NativeMultipartPublication {
let _ = std::fs::remove_file(prepared_path);
return Err(err);
}
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, base, root)
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, base, root, Some(directory_guard))
})()
.map_err(DiskError::from)?;
*self.state.lock() = state;
@@ -2154,7 +2163,7 @@ impl NativeMultipartPublication {
let mut prepared_metadata_source =
os::create_prepared_rename_source_with_commit_guard(&prepared_path, &dst_file_path, &owned_directory_guard)?;
prepared_metadata_source.write_all(&new_bytes, true)?;
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, &base, &root)
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, &base, &root, Some(&owned_directory_guard))
})
.await
.map_err(DiskError::from)?;
@@ -2201,7 +2210,7 @@ impl NativeMultipartPublication {
let base = self.store.base.clone();
let root = self.store.publication_root.clone();
os::run_blocking_namespace_operation(self.store.lease.clone(), move || {
os::install_multipart_prepared_head(prepared, &canonical, &base, &root)
os::install_multipart_prepared_head(prepared, &canonical, &base, &root, None)
})
.await
.map_err(DiskError::from)
@@ -2258,6 +2267,7 @@ impl NativeMultipartPublication {
pub(super) fn finish_on_thread(
&self,
target_already_synced: bool,
fused_in_place: bool,
staging_cleanup: Option<(PathBuf, Arc<std::sync::atomic::AtomicBool>)>,
) -> Result<MultipartPublicationResult> {
let mut state = self.state.lock().clone();
@@ -2287,62 +2297,64 @@ impl NativeMultipartPublication {
let disk_ids = &self.store.disk_ids;
let terminal = terminal_record_path(source, &state);
(|| -> std::io::Result<()> {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let current_state = current
.state
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_raw = encode(current_state).map_err(IoError::other)?;
if current.head.is_some() || current_raw != expected_state_raw {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
if target_already_synced {
let pinned = os::pin_multipart_source_directory(&target_dir, base, root).map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
let target = os::read_multipart_source_metadata_in_pinned(
&pinned,
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
if target.len() as u64 != expected_length || raw_sha(&target) != expected_sha256 {
if !fused_in_place {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let current_state = current
.state
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_raw = encode(current_state).map_err(IoError::other)?;
if current.head.is_some() || current_raw != expected_state_raw {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
} else {
os::read_and_sync_multipart_target_metadata(
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
|target| {
if target.len() as u64 != expected_length || raw_sha(target) != expected_sha256 {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
if target_already_synced {
let pinned = os::pin_multipart_source_directory(&target_dir, base, root).map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
Ok(())
},
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
})?;
let target = os::read_multipart_source_metadata_in_pinned(
&pinned,
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
if target.len() as u64 != expected_length || raw_sha(&target) != expected_sha256 {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
})?;
} else {
os::read_and_sync_multipart_target_metadata(
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
|target| {
if target.len() as u64 != expected_length || raw_sha(target) != expected_sha256 {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
Ok(())
},
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
}
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), true)?;
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), !fused_in_place)?;
if let Some((io_path, cleaned)) = staging_cleanup {
match std::fs::remove_dir(&io_path) {
Ok(()) => cleaned.store(true, std::sync::atomic::Ordering::Release),
@@ -2375,7 +2387,7 @@ impl NativeMultipartPublication {
) -> Result<MultipartPublicationResult> {
let this = Arc::clone(self);
os::run_blocking_namespace_operation_with_sync(target_lease, true, move || {
this.finish_on_thread(target_already_synced, staging_cleanup)
this.finish_on_thread(target_already_synced, false, staging_cleanup)
.map_err(IoError::other)
})
.await
+135
View File
@@ -2125,10 +2125,19 @@ pub struct GuardedFileInfoVersions {
pub struct GuardedDeleteVersionsPayload {
pub versions: Vec<GuardedFileInfoVersions>,
pub opts: DeleteOptions,
// The strict legacy decoder rejects this field before applying any change.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub bounded_recovery_protocol: Option<u32>,
}
impl GuardedDeleteVersionsPayload {
pub fn validate(&self) -> Result<()> {
if self
.bounded_recovery_protocol
.is_some_and(|version| version != BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION)
{
return Err(DiskError::UnsupportedDisk);
}
validate_delete_recovery_options(&self.opts)?;
if self.versions.is_empty()
|| self.versions.len() > 1000
@@ -2213,6 +2222,36 @@ pub struct WalkDirOptions {
// Override the remote stream stall timeout for long background walks.
#[serde(default)]
pub stall_timeout_ms: Option<u64>,
/// Select this leaf below identifier-like directory components. The root is
/// `base_dir`; unrelated object metadata must not be read or returned.
/// Older peers ignore this additive field, so consumers must also filter.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub identifier_tree_leaf: Option<String>,
}
pub(crate) fn is_identifier_tree_component(value: &str) -> bool {
!value.is_empty()
&& value
.bytes()
.all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-'))
}
pub(crate) fn is_identifier_tree_leaf(value: &str) -> bool {
value.len() <= 255
&& value.contains('.')
&& !matches!(value, "." | "..")
&& value
.bytes()
.all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-' | b'.'))
}
pub(crate) fn identifier_tree_leaf_matches(prefix: &str, leaf: &str, object: &str) -> bool {
let Some(relative) = object.strip_prefix(prefix) else {
return false;
};
let mut components = relative.split('/');
components.next_back() == Some(leaf) && components.all(is_identifier_tree_component)
}
impl WalkDirOptions {
@@ -2306,6 +2345,9 @@ pub struct DeleteOptions {
pub delete_recovery_bucket_incarnation: Option<Uuid>,
#[serde(default)]
pub recoverable_delete: bool,
/// Transport requirement for preparing a delete that will use bounded settlement.
#[serde(skip)]
pub requires_bounded_delete_recovery: bool,
/// Reclaim only plain erasure-part leaves, preserving every other entry.
/// Keep `recursive` false so an older peer cannot widen this operation.
#[serde(default)]
@@ -2313,6 +2355,8 @@ pub struct DeleteOptions {
}
pub const DELETE_RECOVERY_PROTOCOL_VERSION: u32 = 1;
/// Accepted only for prepare RPCs. Legacy peers must refuse before staging a delete.
pub const BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION: u32 = 2;
pub(crate) fn get_delete_recovery_timeout() -> Result<std::time::Duration> {
let value = std::env::var("RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS");
match value {
@@ -2909,6 +2953,7 @@ mod tests {
insert_str(&mut fi.metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, "ab".repeat(32));
let generation = CurrentDeleteGeneration::from_file_info(&fi).expect("capture complete remote identity");
let payload = GuardedDeleteVersionsPayload {
bounded_recovery_protocol: None,
versions: vec![GuardedFileInfoVersions {
group: FileInfoVersions {
name: "object".into(),
@@ -2963,6 +3008,7 @@ mod tests {
expected_current: CurrentDeleteGeneration::Absent,
};
let mut payload = GuardedDeleteVersionsPayload {
bounded_recovery_protocol: None,
versions: vec![group],
opts: DeleteOptions {
old_data_dir: Some(Uuid::new_v4()),
@@ -2974,6 +3020,25 @@ mod tests {
let decoded: GuardedDeleteVersionsPayload =
rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode guarded batch");
decoded.validate().expect("round-trip guarded batch remains valid");
#[derive(Debug, Deserialize)]
#[serde(deny_unknown_fields)]
struct LegacyGuardedDeletePayload {
#[serde(rename = "versions")]
_versions: Vec<GuardedFileInfoVersions>,
#[serde(rename = "opts")]
_opts: DeleteOptions,
}
rustfs_protos::decode_delete_recovery_payload::<LegacyGuardedDeletePayload>(&encoded)
.expect("legacy reader accepts legacy prepare");
payload.bounded_recovery_protocol = Some(BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION);
let bounded = rmp_serde::to_vec_named(&payload).expect("serialize bounded guarded prepare");
assert!(rustfs_protos::decode_delete_recovery_payload::<LegacyGuardedDeletePayload>(&bounded).is_err());
let bounded: GuardedDeleteVersionsPayload =
rustfs_protos::decode_delete_recovery_payload(&bounded).expect("current reader accepts bounded prepare");
bounded.validate().expect("valid bounded guarded prepare");
payload.bounded_recovery_protocol = Some(BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION + 1);
assert_eq!(payload.validate(), Err(DiskError::UnsupportedDisk));
payload.bounded_recovery_protocol = None;
let mut trailing = encoded;
trailing.push(0xc0);
assert!(rustfs_protos::decode_delete_recovery_payload::<GuardedDeleteVersionsPayload>(&trailing).is_err());
@@ -3122,6 +3187,7 @@ mod tests {
skip_total_timeout: false,
timeout_ms: Some(10_000),
stall_timeout_ms: Some(20_000),
identifier_tree_leaf: None,
};
assert_eq!(opts.bucket, "test-bucket");
@@ -3156,6 +3222,69 @@ mod tests {
assert!(!decoded.skip_hidden_prefix_check);
}
#[test]
fn identifier_tree_leaf_selection_preserves_nested_identifier_directories() {
let prefix = "catalog/namespaces/";
let leaf = "table-entry.json";
for relative in [
"sales/tables/orders/table-entry.json",
"sales/tables/orders/maintenance/tables/owner/table-entry.json",
"maintenance/tables/tables/table-entry.json",
] {
assert!(identifier_tree_leaf_matches(prefix, leaf, &format!("{prefix}{relative}")));
}
for relative in [
"sales/tables/orders/not-table-entry.json",
"sales/tables/orders/table-entry.json/child",
"sales/report.json/tables/owner/table-entry.json",
"sales//tables/owner/table-entry.json",
"sales/../tables/owner/table-entry.json",
] {
assert!(!identifier_tree_leaf_matches(prefix, leaf, &format!("{prefix}{relative}")));
}
assert!(!identifier_tree_leaf_matches(prefix, leaf, "other/sales/tables/orders/table-entry.json"));
assert!(is_identifier_tree_leaf(leaf));
for invalid in ["", ".", "..", "table-entry", "table/entry.json", "table\\entry.json"] {
assert!(!is_identifier_tree_leaf(invalid), "invalid selector {invalid}");
}
}
#[test]
fn identifier_tree_walk_options_are_additive_on_json_and_messagepack_maps() {
#[derive(serde::Deserialize)]
struct LegacyView {
bucket: String,
base_dir: String,
recursive: bool,
}
let options = WalkDirOptions {
bucket: "catalog".to_string(),
base_dir: "namespaces/".to_string(),
recursive: true,
identifier_tree_leaf: Some("table-entry.json".to_string()),
..Default::default()
};
let json = serde_json::to_vec(&options).expect("encode current HTTP options");
let messagepack = rmp_serde::to_vec_named(&options).expect("encode legacy RPC map");
for old in [
serde_json::from_slice::<LegacyView>(&json).expect("old JSON readers ignore the additive field"),
rmp_serde::from_slice::<LegacyView>(&messagepack).expect("old MessagePack readers ignore the additive field"),
] {
assert_eq!(old.bucket, options.bucket);
assert_eq!(old.base_dir, options.base_dir);
assert!(old.recursive);
}
let mut old = serde_json::to_value(&options).expect("encode legacy options");
old.as_object_mut().expect("options are a map").remove("identifier_tree_leaf");
let from_json: WalkDirOptions = serde_json::from_value(old.clone()).expect("decode old HTTP options");
let from_messagepack: WalkDirOptions =
rmp_serde::from_slice(&rmp_serde::to_vec_named(&old).expect("encode old RPC map")).expect("decode old RPC map");
assert!(from_json.identifier_tree_leaf.is_none());
assert!(from_messagepack.identifier_tree_leaf.is_none());
assert_eq!(from_json.base_dir, options.base_dir);
assert_eq!(from_messagepack.base_dir, options.base_dir);
}
/// Test DeleteOptions structure
#[test]
fn test_delete_options() {
@@ -3168,9 +3297,15 @@ mod tests {
expected_delete_marker: None,
delete_recovery_bucket_incarnation: None,
recoverable_delete: false,
requires_bounded_delete_recovery: true,
orphan_data_dir: false,
};
let wire = serde_json::to_value(&opts).expect("encode legacy wire options");
assert!(wire.get("requires_bounded_delete_recovery").is_none());
let decoded: DeleteOptions = serde_json::from_value(wire).expect("decode legacy wire options");
assert!(!decoded.requires_bounded_delete_recovery);
assert!(opts.recursive);
assert!(!opts.immediate);
assert!(opts.undo_write);
@@ -1149,6 +1149,8 @@ fn normalized_info(fi: &FileInfo, completion: bool) -> FileInfo {
part.checksums = None;
part.index = None;
part.integrity = None;
// The completed object carries the aggregate proof in metadata.
part.encrypted_etag = false;
}
}
fi
@@ -1739,6 +1741,12 @@ mod multipart_proof_authority_tests {
completed.mod_time = Some(original.mod_time.unwrap() + time::Duration::seconds(1));
completed.size = 15;
completed.add_object_part(1, "etag".into(), 15, completed.mod_time, 15, None, None);
completed.parts[0].encrypted_etag = true;
rustfs_utils::http::metadata_compat::insert_str(
&mut completed.metadata,
rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG,
"v1".to_owned(),
);
let publication = condition.attach(&mut completed).unwrap();
assert_eq!(publication.version_id, None);
assert_eq!(
@@ -1748,6 +1756,11 @@ mod multipart_proof_authority_tests {
);
let mut read_completed = persist_null_fixture(&completed);
assert_eq!(read_completed.version_id, None);
assert!(!read_completed.parts[0].encrypted_etag, "xl.meta stores the aggregate proof");
assert_eq!(
rustfs_utils::http::get_consistent_str(&read_completed.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG),
Some("v1")
);
let reread_publication = reread_condition.attach(&mut read_completed).unwrap();
assert_eq!(
publication, reread_publication,
+504 -138
View File
@@ -610,6 +610,26 @@ pub(crate) mod prepare_rename_test_hooks {
hook();
}
}
/// Hooks keyed by a directory component name, run before a Windows
/// directory pin retries a transient sharing conflict on it.
#[cfg(windows)]
static BEFORE_COMPONENT_RETRY: LazyLock<Mutex<HashMap<std::ffi::OsString, Hook>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
#[cfg(windows)]
pub(crate) fn install_before_component_retry(component: &std::ffi::OsStr, hook: impl FnOnce() + Send + 'static) {
BEFORE_COMPONENT_RETRY.lock().insert(component.to_os_string(), Box::new(hook));
}
#[cfg(windows)]
pub(crate) fn run_before_component_retry(component: &std::ffi::OsStr) {
let hook = BEFORE_COMPONENT_RETRY.lock().remove(component);
if let Some(hook) = hook {
let _pause = super::enter_blocking_test_hook();
hook();
}
}
}
/// Fsync a directory so recently created or renamed entries survive power loss.
@@ -644,6 +664,16 @@ pub(crate) fn fsync_directory_handle_std(dir: &Path, opened: Option<&std::fs::Fi
Ok(())
}
/// Open an existing file to read it and flush it. Windows flushes only a
/// handle with write access, so the file is opened read-write there; other
/// platforms keep the read-only open that `sync_all` already accepts.
pub(crate) fn open_file_for_sync_std(path: &Path) -> io::Result<std::fs::File> {
#[cfg(windows)]
return std::fs::OpenOptions::new().read(true).write(true).open(path);
#[cfg(not(windows))]
std::fs::File::open(path)
}
/// Async wrapper around [`fsync_dir_std`]; runs the blocking Unix fsync off the runtime.
pub async fn fsync_dir(dir: impl AsRef<Path>) -> io::Result<()> {
fsync_dir_with_owner(dir.as_ref(), None).await
@@ -2170,6 +2200,7 @@ pub(crate) const MAX_FSYNC_WORKERS_PER_DISK: usize = 8;
// Return busy workers to the shared pool so other disks can acquire a slot.
const MAX_BLOCKING_JOBS_PER_WORKER: usize = 64;
const WORKER_IDLE_SPIN_ITERS: usize = 256;
const FSYNC_WORKER_IDLE_SPIN_ITERS: usize = 1024;
const FALLBACK_NAMESPACE_QUEUE_SHARDS: usize = 8;
#[derive(Clone, Copy, PartialEq, Eq)]
@@ -2378,7 +2409,7 @@ impl DiskBlockingQueue {
.ready
.store(!state.jobs.is_empty(), std::sync::atomic::Ordering::Relaxed);
if job.is_none() {
if state.idle_workers > 0 {
if lane != DiskBlockingLane::Fsync && state.idle_workers > 0 {
state.active_workers = state.active_workers.saturating_sub(1);
guard.state = DiskBlockingWorkerState::Retired;
return;
@@ -2396,7 +2427,12 @@ impl DiskBlockingQueue {
}
idle_polled = false;
} else {
for _ in 0..WORKER_IDLE_SPIN_ITERS {
let spin_iters = if lane == DiskBlockingLane::Fsync {
FSYNC_WORKER_IDLE_SPIN_ITERS
} else {
WORKER_IDLE_SPIN_ITERS
};
for _ in 0..spin_iters {
if queue.lane(lane).ready.load(std::sync::atomic::Ordering::Relaxed) {
break;
}
@@ -4584,7 +4620,7 @@ pub(crate) fn rename_source_is_missing(src_file_path: &Path, publication_root: &
let Some(source_parent) = src_file_path.parent() else {
return false;
};
let source_parent_guard = match lock_windows_directory_tree(source_parent, None, publication_root) {
let source_parent_guard = match lock_windows_directory_tree(source_parent, publication_root) {
Ok(guard) => guard,
Err(err) => return err.kind() == io::ErrorKind::NotFound,
};
@@ -5023,7 +5059,7 @@ fn prepare_rename_with_retry(
open_windows_rename_source_identity(src_file_path, &source_parent_guard)?;
(source_parent_guard, parent_guard, source_identity_anchor, expected_source_identity)
} else {
let source_parent_guard = lock_windows_directory_tree(source_parent, destination_parent, publication_root)?;
let source_parent_guard = lock_windows_directory_tree(source_parent, publication_root)?;
let (source_identity_anchor, expected_source_identity) =
open_windows_rename_source_identity(src_file_path, &source_parent_guard)?;
let parent_guard = prepare_destination_parent()?;
@@ -5444,7 +5480,9 @@ impl PublicationRoot {
pub(crate) fn open_readonly(&self, path: &Path) -> io::Result<std::fs::File> {
#[cfg(target_os = "linux")]
return self.open_relative(path, rustix::fs::OFlags::RDONLY, rustix::fs::Mode::empty());
#[cfg(not(target_os = "linux"))]
#[cfg(windows)]
return open_windows_readonly(path);
#[cfg(not(any(target_os = "linux", windows)))]
std::fs::File::open(path)
}
@@ -5908,14 +5946,11 @@ impl RenameCommitGuard {
}
/// Reopen a destination tree for handle-relative child publication.
/// Ancestors remain write-exclusive while the final parent shares writes
/// required by the kernel's relative rename. Delete sharing stays omitted
/// throughout, so every retained directory identity remains pinned.
/// Every component is pinned by `open_windows_directory_component`.
fn destination_directory_guard(&self, directory: &Path, create_missing: bool) -> io::Result<RenameDestinationPathGuard> {
#[cfg(windows)]
{
use windows_sys::Wdk::Storage::FileSystem::{FILE_OPEN, FILE_OPEN_IF};
use windows_sys::Win32::Storage::FileSystem::{FILE_SHARE_READ, FILE_SHARE_WRITE};
let relative = directory.strip_prefix(&self.destination_parent).map_err(|_| {
io::Error::new(
@@ -5948,35 +5983,17 @@ impl RenameCommitGuard {
let parent = handles
.last()
.ok_or_else(|| io::Error::other("Windows destination guard lost its parent handle"))?;
let mut relative_components = relative
.components()
.filter_map(|component| match component {
Component::Normal(component) => Some(component),
_ => None,
})
.peekable();
let destination_parent_share = if relative_components.peek().is_none() {
FILE_SHARE_READ | FILE_SHARE_WRITE
} else {
FILE_SHARE_READ
};
handles.push(open_windows_relative_directory_component(
parent,
component,
FILE_OPEN,
destination_parent_share,
)?);
while let Some(component) = relative_components.next() {
handles.push(open_windows_directory_component(parent, component, FILE_OPEN)?);
let relative_components = relative.components().filter_map(|component| match component {
Component::Normal(component) => Some(component),
_ => None,
});
for component in relative_components {
let parent = handles
.last()
.ok_or_else(|| io::Error::other("Windows destination path guard lost its parent handle"))?;
let disposition = if create_missing { FILE_OPEN_IF } else { FILE_OPEN };
let share_access = if relative_components.peek().is_none() {
FILE_SHARE_READ | FILE_SHARE_WRITE
} else {
FILE_SHARE_READ
};
handles.push(open_windows_relative_directory_component(parent, component, disposition, share_access)?);
handles.push(open_windows_directory_component(parent, component, disposition)?);
}
Ok(RenameDestinationPathGuard {
directory: directory.to_path_buf(),
@@ -6020,18 +6037,47 @@ pub(crate) fn multipart_payload_commit_guard_std(
})
}
#[cfg(windows)]
fn multipart_payload_commit_guard_std(
metadata_guard: &RenameCommitGuard,
source_parent: &Path,
publication_root: &PublicationRoot,
) -> io::Result<RenameCommitGuard> {
// Pin the payload staging tree like a distinct-tree metadata source; the
// target stays the exact parent handle the metadata transaction holds.
let source_parent_guard = lock_windows_directory_tree(source_parent, publication_root)?;
Ok(RenameCommitGuard {
source_parent: source_parent.to_path_buf(),
destination_parent: metadata_guard.destination_parent.clone(),
source_parent_guard,
destination_parent_guard: metadata_guard.destination_parent_guard.clone(),
})
}
/// Native MPU metadata and payload have distinct source parents. Reuse the
/// exact target identity already held by the metadata transaction, while
/// independently pinning the private payload source under the same root.
#[cfg(target_os = "linux")]
#[cfg(any(target_os = "linux", windows))]
pub(crate) async fn multipart_payload_commit_guard(
metadata_guard: &RenameCommitGuard,
source_parent: &Path,
publication_root: &PublicationRoot,
lease: Arc<NamespaceMutationLease>,
) -> io::Result<RenameCommitGuard> {
let metadata_guard = metadata_guard.clone();
let source_parent = source_parent.to_path_buf();
run_blocking_namespace_operation(lease, move || multipart_payload_commit_guard_std(&metadata_guard, &source_parent)).await
#[cfg(target_os = "linux")]
let operation = {
// The Linux guard already carries its publication root.
let _ = publication_root;
move || multipart_payload_commit_guard_std(&metadata_guard, &source_parent)
};
#[cfg(windows)]
let operation = {
let publication_root = publication_root.clone();
move || multipart_payload_commit_guard_std(&metadata_guard, &source_parent, &publication_root)
};
run_blocking_namespace_operation(lease, operation).await
}
pub(crate) fn prepare_rename_commit_guard(
@@ -6073,20 +6119,16 @@ pub(crate) fn prepare_rename_commit_guard(
}
#[cfg(windows)]
{
// A same-directory rename must use one shared-write parent handle:
// retaining a second read-only-share handle would block the kernel's
// relative target open. Delete sharing stays excluded, so identity is
// still pinned. Distinct source trees remain strict.
// A same-directory rename reuses one guard for both sides. Every pin
// shares writes and excludes delete; see
// `open_windows_directory_component`.
let same_parent = publication_root.relative_path(source_parent)? == publication_root.relative_path(destination_parent)?;
let (source_parent_guard, destination_parent_guard) = if same_parent {
let destination_parent_guard =
mkdir_all_below_existing_base_std(destination_parent, destination_base, publication_root)?;
(destination_parent_guard.clone(), destination_parent_guard)
} else {
// The source parent also hosts private rollback staging files.
// Their handle-relative publication needs write sharing on this
// final directory while delete sharing remains excluded.
let source_parent_guard = lock_windows_directory_tree(source_parent, Some(source_parent), publication_root)?;
let source_parent_guard = lock_windows_directory_tree(source_parent, publication_root)?;
let destination_parent_guard =
mkdir_all_below_existing_base_std(destination_parent, destination_base, publication_root)?;
(source_parent_guard, destination_parent_guard)
@@ -6106,6 +6148,28 @@ pub(crate) fn prepare_rename_commit_guard(
}
}
/// Unix opens a directory read-only and lets the reader classify it, while
/// Windows refuses the plain open with ACCESS_DENIED. Reopen only an actual
/// directory with backup semantics so the same `is_dir` checks apply.
#[cfg(windows)]
fn open_windows_readonly(path: &Path) -> io::Result<std::fs::File> {
use std::os::windows::fs::OpenOptionsExt;
use windows_sys::Win32::Storage::FileSystem::FILE_FLAG_BACKUP_SEMANTICS;
match std::fs::File::open(path) {
Err(error)
if error.kind() == io::ErrorKind::PermissionDenied
&& std::fs::symlink_metadata(path).is_ok_and(|metadata| metadata.is_dir()) =>
{
std::fs::OpenOptions::new()
.read(true)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS)
.open(path)
}
result => result,
}
}
#[cfg(windows)]
fn open_windows_publication_root(path: &Path) -> io::Result<(PathBuf, WindowsDirectoryHandle)> {
use std::os::windows::fs::OpenOptionsExt;
@@ -6292,22 +6356,12 @@ fn windows_file_attribute_tag(
}
#[cfg(windows)]
fn lock_windows_directory_tree(
path: &Path,
shared_write_ancestor: Option<&Path>,
publication_root: &PublicationRoot,
) -> io::Result<ExistingBaseDirectoryGuard> {
fn lock_windows_directory_tree(path: &Path, publication_root: &PublicationRoot) -> io::Result<ExistingBaseDirectoryGuard> {
use windows_sys::Wdk::Storage::FileSystem::FILE_OPEN;
use windows_sys::Win32::Storage::FileSystem::{FILE_SHARE_READ, FILE_SHARE_WRITE};
let relative = publication_root.relative_path(path)?;
let shared_write_relative = shared_write_ancestor
.map(|ancestor| publication_root.relative_path(ancestor))
.transpose()?
.filter(|ancestor| relative.starts_with(*ancestor));
let mut handles = Vec::with_capacity(relative.components().count().saturating_add(1));
handles.push(publication_root.directory.clone());
let mut opened_relative = PathBuf::new();
for component in relative.components() {
let Component::Normal(component) = component else {
@@ -6322,26 +6376,13 @@ fn lock_windows_directory_tree(
let parent = handles
.last()
.ok_or_else(|| io::Error::other("Windows directory guard lost its root handle"))?;
opened_relative.push(component);
let child = if shared_write_relative.is_some_and(|ancestor| opened_relative.as_path() == ancestor) {
// A handle-relative rename opens the target parent for write. When
// that parent is also a source ancestor, this source-side handle
// must share write access or the transaction blocks itself. Delete
// sharing remains omitted, so the directory identity stays pinned.
open_windows_relative_directory_component(parent, component, FILE_OPEN, FILE_SHARE_READ | FILE_SHARE_WRITE)?
} else {
open_windows_directory_component(parent, component, FILE_OPEN)?
};
handles.push(child);
handles.push(open_windows_directory_component(parent, component, FILE_OPEN)?);
}
Ok(ExistingBaseDirectoryGuard::new(handles))
}
#[cfg(windows)]
// SAFETY: the object attributes borrow a checked UTF-16 component and live
// parent handle for the duration of the synchronous NtCreateFile call.
#[allow(unsafe_code)]
fn open_windows_relative(
parent: &winapi_util::Handle,
component: &std::ffi::OsStr,
@@ -6352,6 +6393,64 @@ fn open_windows_relative(
file_attributes: u32,
dont_reparse: bool,
) -> io::Result<winapi_util::Handle> {
open_windows_relative_with_status(
parent,
component,
&WindowsRelativeOpen {
desired_access,
share_access,
create_disposition,
create_options,
file_attributes,
dont_reparse,
},
)
.map_err(|failure| failure.error)
}
/// `NtCreateFile` arguments for one handle-relative open.
#[cfg(windows)]
struct WindowsRelativeOpen {
desired_access: u32,
share_access: u32,
create_disposition: u32,
create_options: u32,
file_attributes: u32,
dont_reparse: bool,
}
/// A failed relative open that keeps the NTSTATUS, which tells transient
/// states such as a pending delete apart from errors that share its Win32 code.
#[cfg(windows)]
struct WindowsRelativeOpenError {
status: Option<windows_sys::Win32::Foundation::NTSTATUS>,
error: io::Error,
}
#[cfg(windows)]
impl From<io::Error> for WindowsRelativeOpenError {
fn from(error: io::Error) -> Self {
Self { status: None, error }
}
}
#[cfg(windows)]
// SAFETY: the object attributes borrow a checked UTF-16 component and live
// parent handle for the duration of the synchronous NtCreateFile call.
#[allow(unsafe_code)]
fn open_windows_relative_with_status(
parent: &winapi_util::Handle,
component: &std::ffi::OsStr,
request: &WindowsRelativeOpen,
) -> std::result::Result<winapi_util::Handle, WindowsRelativeOpenError> {
let &WindowsRelativeOpen {
desired_access,
share_access,
create_disposition,
create_options,
file_attributes,
dont_reparse,
} = request;
use std::{
mem::size_of,
os::windows::{ffi::OsStrExt, io::AsRawHandle, io::FromRawHandle},
@@ -6366,10 +6465,7 @@ fn open_windows_relative(
let mut name = component.encode_wide().collect::<Vec<_>>();
if name.is_empty() || name.contains(&0) {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
"relative Windows file name is empty or contains a NUL",
));
return Err(io::Error::new(io::ErrorKind::InvalidInput, "relative Windows file name is empty or contains a NUL").into());
}
let name_bytes = name
.len()
@@ -6409,13 +6505,17 @@ fn open_windows_relative(
)
};
if status < 0 {
return match i32::try_from(unsafe { RtlNtStatusToDosError(status) }) {
Ok(code) => Err(io::Error::from_raw_os_error(code)),
Err(_) => Err(io::Error::other(format!("Windows relative open failed with NTSTATUS {status:#x}"))),
let error = match i32::try_from(unsafe { RtlNtStatusToDosError(status) }) {
Ok(code) => io::Error::from_raw_os_error(code),
Err(_) => io::Error::other(format!("Windows relative open failed with NTSTATUS {status:#x}")),
};
return Err(WindowsRelativeOpenError {
status: Some(status),
error,
});
}
if handle.is_null() {
return Err(io::Error::other("Windows relative open returned an invalid handle"));
return Err(io::Error::other("Windows relative open returned an invalid handle").into());
}
Ok(unsafe { winapi_util::Handle::from_raw_handle(handle) })
@@ -6650,7 +6750,7 @@ pub(crate) fn compare_and_update_control_file(
let parent = file_path
.parent()
.ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "conditional file has no parent"))?;
let parent_guard = lock_windows_directory_tree(parent, Some(parent), publication_root)?;
let parent_guard = lock_windows_directory_tree(parent, publication_root)?;
let lock = open_windows_relative(
parent_guard.last_handle()?,
std::ffi::OsStr::new(".rustfs-cas.lock"),
@@ -6714,41 +6814,78 @@ pub(crate) fn compare_and_update_control_file(
Ok(ConditionalFileUpdate::Updated)
}
/// How often a directory pin retries a transient delete-access conflict.
/// Six doubling delays from 1 ms wait about 63 ms in total, which outlasts a
/// prune whose thread is preempted while it holds delete access.
#[cfg(windows)]
const WINDOWS_DIRECTORY_PIN_RETRIES: u32 = 6;
#[cfg(windows)]
const WINDOWS_DIRECTORY_PIN_FIRST_RETRY_DELAY: std::time::Duration = std::time::Duration::from_millis(1);
/// Pin one directory component of a guarded path.
///
/// The handle shares reads and writes but never delete: renaming or removing
/// a directory needs delete access to it, so omitting delete sharing pins its
/// identity. Write sharing must stay granted on every component, ancestors
/// included, because the kernel opens the target parent for write during a
/// rename. A directory pinned here as one object's ancestor is also the
/// publication parent of a nested object (`a/b` holds `a/b/c`), and a
/// read-only share would make that unrelated rename fail with
/// ERROR_SHARING_VIOLATION. Write sharing does not unpin an ancestor: a
/// pinned ancestor always contains its pinned child, and NTFS sets a
/// name-surrogate reparse point only on an empty directory.
///
/// Excluding delete also means the open fails while another handle holds
/// delete access. Pruning an empty prefix directory takes that access only
/// briefly: `RemoveDirectoryW` opens the directory for delete and closes it
/// again, whether or not the directory turns out to be empty. A sharing
/// violation or a pending delete is therefore retried a bounded number of
/// times. After a removal finishes, the retry sees the directory as missing
/// and follows `create_disposition`, so `FILE_OPEN` still reports it missing
/// and `FILE_OPEN_IF` creates it again.
#[cfg(windows)]
fn open_windows_directory_component(
parent: &WindowsDirectoryHandle,
component: &std::ffi::OsStr,
create_disposition: u32,
) -> io::Result<WindowsDirectoryHandle> {
use windows_sys::Win32::Storage::FileSystem::FILE_SHARE_READ;
open_windows_relative_directory_component(parent, component, create_disposition, FILE_SHARE_READ)
}
#[cfg(windows)]
fn open_windows_relative_directory_component(
parent: &WindowsDirectoryHandle,
component: &std::ffi::OsStr,
create_disposition: u32,
share_access: u32,
) -> io::Result<WindowsDirectoryHandle> {
use windows_sys::{
Wdk::Storage::FileSystem::{FILE_DIRECTORY_FILE, FILE_OPEN_REPARSE_POINT},
Win32::Storage::FileSystem::{
FILE_ATTRIBUTE_DIRECTORY, FILE_ATTRIBUTE_REPARSE_POINT, FILE_READ_ATTRIBUTES, FILE_TRAVERSE,
Win32::{
Foundation::{STATUS_DELETE_PENDING, STATUS_SHARING_VIOLATION},
Storage::FileSystem::{
FILE_ATTRIBUTE_DIRECTORY, FILE_ATTRIBUTE_REPARSE_POINT, FILE_READ_ATTRIBUTES, FILE_SHARE_READ, FILE_SHARE_WRITE,
FILE_TRAVERSE,
},
},
};
let anchor = open_windows_relative(
&parent.handle,
component,
FILE_TRAVERSE | FILE_READ_ATTRIBUTES,
share_access,
let request = WindowsRelativeOpen {
desired_access: FILE_TRAVERSE | FILE_READ_ATTRIBUTES,
share_access: FILE_SHARE_READ | FILE_SHARE_WRITE,
create_disposition,
FILE_DIRECTORY_FILE | FILE_OPEN_REPARSE_POINT,
FILE_ATTRIBUTE_DIRECTORY,
true,
)?;
create_options: FILE_DIRECTORY_FILE | FILE_OPEN_REPARSE_POINT,
file_attributes: FILE_ATTRIBUTE_DIRECTORY,
dont_reparse: true,
};
let mut retry_delay = WINDOWS_DIRECTORY_PIN_FIRST_RETRY_DELAY;
let mut retries = 0;
let anchor = loop {
match open_windows_relative_with_status(&parent.handle, component, &request) {
Ok(anchor) => break anchor,
Err(WindowsRelativeOpenError {
status: Some(STATUS_SHARING_VIOLATION | STATUS_DELETE_PENDING),
..
}) if retries < WINDOWS_DIRECTORY_PIN_RETRIES => {
#[cfg(test)]
prepare_rename_test_hooks::run_before_component_retry(component);
std::thread::sleep(retry_delay);
retry_delay = retry_delay.saturating_mul(2);
retries += 1;
}
Err(failure) => return Err(failure.error),
}
};
let info = windows_file_attribute_tag(&anchor)?;
if info.FileAttributes & FILE_ATTRIBUTE_DIRECTORY == 0 {
return Err(io::Error::from(io::ErrorKind::NotADirectory));
@@ -7052,7 +7189,7 @@ pub(crate) fn pin_multipart_source_directory(
#[cfg(windows)]
{
let _ = base;
lock_windows_directory_tree(directory, Some(directory), publication_root)
lock_windows_directory_tree(directory, publication_root)
}
#[cfg(not(any(unix, windows)))]
{
@@ -7142,12 +7279,31 @@ pub(crate) fn rename_multipart_source_owned_std(
{
let _ = &guard;
let source_guard = pin_multipart_source_directory(source_parent, base, root)?;
// The prepared Windows source holds DELETE access and shares only
// read, so the precondition's own opens of that entry (or, for a
// directory, of its guarded tree) would fail with a sharing violation.
// Check it first under a non-exclusive identity anchor, then require
// the exclusive handle to name the same file.
#[cfg(windows)]
let (_source_identity_anchor, expected_source_identity) = {
let anchor = open_windows_rename_source_identity(source, &source_guard)?;
precondition()?;
anchor
};
let preparation = prepare_rename_with_retry(source, destination, base, root)?;
#[cfg(windows)]
if windows_file_identity(&preparation.source)? != expected_source_identity {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
"rename source identity changed after its precondition",
));
}
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
{
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, source);
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, destination);
}
#[cfg(not(windows))]
precondition()?;
#[cfg(unix)]
validate_prepared_rename_source(&pinned_source, source)?;
@@ -7293,6 +7449,7 @@ pub(crate) fn install_multipart_prepared_head(
canonical_head: &Path,
base: &Path,
publication_root: &PublicationRoot,
directory_guard: Option<&RenameCommitGuard>,
) -> io::Result<PreparedRenameSource> {
let original_path = prepared.path.clone();
let parent = canonical_head
@@ -7301,9 +7458,37 @@ pub(crate) fn install_multipart_prepared_head(
if original_path.parent() != Some(parent) || original_path == canonical_head {
return Err(io::Error::new(io::ErrorKind::InvalidInput, "MPU proxy must be a private sibling"));
}
#[cfg(target_os = "linux")]
let guard = if let Some(directory_guard) = directory_guard.filter(|g| g.directories.source_path == parent) {
let dir = &directory_guard.directories.source;
RenameCommitGuard {
directories: Arc::new(LinuxCommitDirectories {
root: directory_guard.directories.root.clone(),
source_path: parent.to_path_buf(),
destination_path: parent.to_path_buf(),
source: LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
},
destination: std::sync::OnceLock::from(LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
}),
destination_base: base.to_path_buf(),
}),
}
} else {
prepare_rename_commit_guard(parent, parent, base, publication_root)?
};
#[cfg(not(target_os = "linux"))]
let _pinned = pin_multipart_source_directory(parent, base, publication_root)?;
let guard = prepare_rename_commit_guard(parent, parent, base, publication_root)?;
#[cfg(not(target_os = "linux"))]
let guard = {
let _ = directory_guard;
prepare_rename_commit_guard(parent, parent, base, publication_root)?
};
rename_prepared_source_std(&prepared, &original_path, canonical_head, base, publication_root, &guard)?;
prepared.path = canonical_head.to_path_buf();
fsync_commit_directory_limited_std(parent, &guard)?;
@@ -7553,12 +7738,15 @@ fn read_multipart_metadata(
let parent = path
.parent()
.ok_or_else(|| io::Error::other("multipart metadata has no parent"))?;
let guard = prepare_rename_commit_guard(parent, parent, base, publication_root)?;
// A read must not recreate a retired upload: pin the existing parent
// instead of taking the write-side guard, which creates missing parents.
existing_base_directory_suffix(parent, base)?;
let guard = pin_multipart_source_directory(parent, base, publication_root)?;
let name = path
.file_name()
.ok_or_else(|| io::Error::other("multipart metadata has no name"))?;
let mut file = open_windows_relative(
guard.source_parent_guard.last_handle()?,
guard.last_handle()?,
name,
SYNCHRONIZE | FILE_READ_ATTRIBUTES | FILE_READ_DATA,
FILE_SHARE_READ,
@@ -7734,7 +7922,7 @@ mod multipart_source_preservation_tests {
let metadata_guard =
prepare_rename_commit_guard(&canonical, &target, &bucket, &publication).expect("canonical metadata guard");
let lease = acquire_namespace_mutation_lease(&target).await;
let payload_guard = multipart_payload_commit_guard(&metadata_guard, &temporary, lease.clone())
let payload_guard = multipart_payload_commit_guard(&metadata_guard, &temporary, &publication, lease.clone())
.await
.expect("distinct pinned payload source");
assert!(
@@ -7769,6 +7957,61 @@ mod multipart_source_preservation_tests {
);
}
#[cfg(windows)]
#[tokio::test]
async fn windows_multipart_payload_guard_publishes_distinct_source_into_metadata_target() {
// Build paths from the configured root: the resolved handle path may
// be verbatim, where `/` in a joined key is not a separator.
let (root, publication, _source, _destination) = fixture();
let root = root.path();
let canonical = root.join("multipart/upload");
let temporary = root.join("tmp/attempt");
let bucket = root.join("bucket");
let target = bucket.join("object");
for directory in [&canonical, &temporary, &target] {
std::fs::create_dir_all(directory).expect("two-source guard directories");
}
let payload = temporary.join("payload");
std::fs::create_dir(&payload).expect("private payload DD");
std::fs::write(payload.join("part.1"), b"native selected shard").expect("private payload shard");
let head = canonical.join("xl.meta");
let destination = target.join("xl.meta");
let metadata_guard =
prepare_rename_commit_guard(&canonical, &target, &bucket, &publication).expect("canonical metadata guard");
let lease = acquire_namespace_mutation_lease(&target).await;
assert!(
rename_all_with_commit_guard(&payload, target.join("payload"), &bucket, &publication, &metadata_guard, lease.clone())
.await
.is_err(),
"the metadata guard does not pin the private payload parent"
);
assert!(
payload.join("part.1").exists(),
"a rejected payload rename must leave its source in place"
);
let payload_guard = multipart_payload_commit_guard(&metadata_guard, &temporary, &publication, lease.clone())
.await
.expect("distinct pinned payload source");
let mut prepared = create_prepared_rename_source_with_commit_guard(&head, &destination, &metadata_guard)
.expect("canonical prepared metadata");
prepared
.write_all(b"prepared native metadata", true)
.expect("durable canonical metadata");
rename_all_with_commit_guard(&payload, target.join("payload"), &bucket, &publication, &payload_guard, lease.clone())
.await
.expect("payload rename uses its own pinned TMP parent");
assert!(head.exists(), "payload publication must not consume the canonical head");
rename_all_with_prepared_source(prepared, &head, &destination, &bucket, &publication, &metadata_guard, lease)
.await
.expect("metadata rename keeps its canonical parent");
assert!(!head.exists());
assert_eq!(std::fs::read(&destination).expect("published metadata"), b"prepared native metadata");
assert_eq!(
std::fs::read(target.join("payload/part.1")).expect("published selected shard"),
b"native selected shard"
);
}
#[cfg(target_os = "linux")]
#[tokio::test]
async fn multipart_payload_guard_rejects_replaced_private_source_or_shared_destination() {
@@ -7790,7 +8033,7 @@ mod multipart_source_preservation_tests {
let metadata_guard =
prepare_rename_commit_guard(&canonical, &target, &bucket, &publication).expect("canonical metadata guard");
let lease = acquire_namespace_mutation_lease(&target).await;
let payload_guard = multipart_payload_commit_guard(&metadata_guard, &temporary, lease.clone())
let payload_guard = multipart_payload_commit_guard(&metadata_guard, &temporary, &publication, lease.clone())
.await
.expect("pin payload and shared target");
let replaced = if replace_source { &temporary } else { &target };
@@ -8047,7 +8290,6 @@ pub(crate) fn mkdir_all_below_existing_base_std(
#[cfg(windows)]
{
use windows_sys::Wdk::Storage::FileSystem::{FILE_OPEN, FILE_OPEN_IF};
use windows_sys::Win32::Storage::FileSystem::{FILE_SHARE_READ, FILE_SHARE_WRITE};
let base_relative = publication_root.relative_path(base_dir)?;
let capacity = base_relative
@@ -8058,13 +8300,12 @@ pub(crate) fn mkdir_all_below_existing_base_std(
let mut handles = Vec::with_capacity(capacity);
handles.push(publication_root.directory.clone());
let mut guard = ExistingBaseDirectoryGuard::new(handles);
let mut components = base_relative
let components = base_relative
.components()
.map(|component| (component, FILE_OPEN))
.chain(relative.components().map(|component| (component, FILE_OPEN_IF)))
.filter(|(component, _)| !matches!(component, Component::CurDir))
.peekable();
while let Some((component, disposition)) = components.next() {
.filter(|(component, _)| !matches!(component, Component::CurDir));
for (component, disposition) in components {
let Component::Normal(component) = component else {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
@@ -8075,22 +8316,16 @@ pub(crate) fn mkdir_all_below_existing_base_std(
.handles
.last()
.ok_or_else(|| io::Error::other("Windows publication root guard is empty"))?;
// The kernel opens the final parent for write during a relative
// rename. Share writes from its first open: a temporary read-only
// share would block another rename into the same trash directory.
// Ancestors stay strict and no handle shares delete access, keeping
// every retained directory identity pinned.
let share_access = if components.peek().is_none() {
FILE_SHARE_READ | FILE_SHARE_WRITE
} else {
FILE_SHARE_READ
};
let child = open_windows_relative_directory_component(parent, component, disposition, share_access)?;
// Every component, including the final parent the kernel opens
// for write during a relative rename, shares writes from its first
// open; see `open_windows_directory_component`.
let child = open_windows_directory_component(parent, component, disposition)?;
guard.handles.push(child);
#[cfg(test)]
if components.peek().is_none() {
prepare_rename_test_hooks::run_after_component_opened(dir_path);
}
}
// The hook observes the window right after the final component opened.
#[cfg(test)]
if guard.handles.len() > 1 {
prepare_rename_test_hooks::run_after_component_opened(dir_path);
}
Ok(guard)
@@ -9496,7 +9731,7 @@ mod tests {
use std::os::windows::fs::OpenOptionsExt;
use std::sync::atomic::{AtomicBool, Ordering};
use windows_sys::Win32::{
Foundation::{ERROR_SHARING_VIOLATION, GENERIC_WRITE},
Foundation::ERROR_SHARING_VIOLATION,
Storage::FileSystem::{
DELETE, FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_OPEN_REPARSE_POINT, FILE_SHARE_DELETE, FILE_SHARE_READ,
FILE_SHARE_WRITE,
@@ -9538,13 +9773,13 @@ mod tests {
let second_preparation = prepare_rename_with_retry(&sources[1], &destinations[1], base, &publication_root)
.expect("prepare the second trash rename");
assert!(interleaved.load(Ordering::Acquire), "the competing parent-open window must be exercised");
for (path, access) in [(&tmp, GENERIC_WRITE), (&trash, DELETE)] {
for path in [&tmp, &trash] {
let err = std::fs::OpenOptions::new()
.access_mode(access)
.access_mode(DELETE)
.share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(path)
.expect_err("ancestor writes and final-parent deletion must remain excluded");
.expect_err("pinned ancestors and final parents must exclude deletion");
assert_eq!(
err.raw_os_error(),
Some(i32::try_from(ERROR_SHARING_VIOLATION).expect("Windows error code must fit i32"))
@@ -9561,6 +9796,115 @@ mod tests {
}
}
#[cfg(windows)]
#[test]
fn windows_rename_into_a_pinned_ancestor_succeeds_while_deletion_stays_excluded() {
use std::os::windows::fs::OpenOptionsExt;
use windows_sys::Win32::{
Foundation::ERROR_SHARING_VIOLATION,
Storage::FileSystem::{
DELETE, FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_OPEN_REPARSE_POINT, FILE_SHARE_DELETE, FILE_SHARE_READ,
FILE_SHARE_WRITE,
},
};
// `bucket/parent` is an object directory and also the prefix of the
// nested object `bucket/parent/child`. A guard on the nested object
// pins `parent` as an ancestor while the parent object publishes a new
// data directory into it by pathname.
let temp_dir = tempdir().expect("create temp dir");
let base = temp_dir.path().join("bucket");
let parent = base.join("parent");
let child = parent.join("child");
std::fs::create_dir_all(&child).expect("create nested object directory");
let staged = temp_dir.path().join("staged-data-dir");
std::fs::create_dir(&staged).expect("create staged data directory");
std::fs::write(staged.join("part.1"), b"payload").expect("write staged part");
let publication_root = PublicationRoot::new(temp_dir.path()).expect("open publication root");
let child_guard =
super::mkdir_all_below_existing_base_std(&child, &base, &publication_root).expect("pin the nested object tree");
let destination = parent.join(uuid::Uuid::new_v4().to_string());
std::fs::rename(&staged, &destination).expect("a pinned ancestor must accept a rename into it");
assert_eq!(std::fs::read(destination.join("part.1")).expect("read published part"), b"payload");
let err = std::fs::OpenOptions::new()
.access_mode(DELETE)
.share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(&parent)
.expect_err("the pinned ancestor must still refuse deletion");
assert_eq!(
err.raw_os_error(),
Some(i32::try_from(ERROR_SHARING_VIOLATION).expect("Windows error code must fit i32"))
);
drop(child_guard);
}
#[cfg(windows)]
#[test]
fn windows_directory_pin_retries_a_transient_delete_holder_and_fails_closed_on_a_lasting_one() {
use std::os::windows::fs::OpenOptionsExt;
use windows_sys::Win32::{
Foundation::ERROR_SHARING_VIOLATION,
Storage::FileSystem::{
DELETE, FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_OPEN_REPARSE_POINT, FILE_READ_ATTRIBUTES, FILE_SHARE_DELETE,
FILE_SHARE_READ, FILE_SHARE_WRITE, SYNCHRONIZE,
},
};
// Open the prefix the way `RemoveDirectoryW` does while it prunes an
// object prefix: delete access, every share mode granted.
let open_like_a_pruner = |path: &Path| {
std::fs::OpenOptions::new()
.access_mode(DELETE | SYNCHRONIZE | FILE_READ_ATTRIBUTES)
.share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(path)
.expect("open the prefix for delete")
};
let temp_dir = tempdir().expect("create temp dir");
let base = temp_dir.path().join("bucket");
let publication_root = PublicationRoot::new(temp_dir.path()).expect("open publication root");
let transient_name = uuid::Uuid::new_v4().to_string();
let transient_prefix = base.join(&transient_name);
std::fs::create_dir_all(&transient_prefix).expect("create transient prefix");
let holder = Arc::new(std::sync::Mutex::new(Some(open_like_a_pruner(&transient_prefix))));
let holder_for_hook = Arc::clone(&holder);
let released = Arc::new(std::sync::atomic::AtomicBool::new(false));
let released_for_hook = Arc::clone(&released);
prepare_rename_test_hooks::install_before_component_retry(std::ffi::OsStr::new(&transient_name), move || {
holder_for_hook.lock().expect("holder mutex poisoned").take();
released_for_hook.store(true, std::sync::atomic::Ordering::Release);
});
let child = transient_prefix.join("child");
let guard = super::mkdir_all_below_existing_base_std(&child, &base, &publication_root)
.expect("a pin must outlast a transient delete holder");
assert!(
released.load(std::sync::atomic::Ordering::Acquire),
"the pin must have met the delete holder"
);
assert!(child.is_dir(), "the retried walk must still create the child");
let err = std::fs::OpenOptions::new()
.access_mode(DELETE)
.share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(&transient_prefix)
.expect_err("the retried pin must still exclude deletion");
assert_eq!(err.raw_os_error(), i32::try_from(ERROR_SHARING_VIOLATION).ok());
drop(guard);
let lasting_prefix = base.join(uuid::Uuid::new_v4().to_string());
std::fs::create_dir_all(&lasting_prefix).expect("create lasting prefix");
let lasting_holder = open_like_a_pruner(&lasting_prefix);
let Err(err) = super::mkdir_all_below_existing_base_std(&lasting_prefix.join("child"), &base, &publication_root) else {
panic!("a pin must not wait forever on a lasting delete holder");
};
assert_eq!(err.raw_os_error(), i32::try_from(ERROR_SHARING_VIOLATION).ok());
drop(lasting_holder);
}
#[cfg(windows)]
#[tokio::test]
async fn windows_rename_all_supports_same_parent_publication() {
@@ -10258,7 +10602,7 @@ mod tests {
let publication_root = test_publication_root(&[&src, &dst, &base]);
let source_parent_guard =
lock_windows_directory_tree(src.parent().expect("source reparse entry must have a parent"), None, &publication_root)
lock_windows_directory_tree(src.parent().expect("source reparse entry must have a parent"), &publication_root)
.expect("pin the source parent tree");
let identity_error = match open_windows_rename_source_identity(&src, &source_parent_guard) {
Ok(_) => panic!("a non-dedup source reparse entry must be rejected by its tag"),
@@ -10292,30 +10636,51 @@ mod tests {
#[cfg(windows)]
#[tokio::test]
async fn windows_rename_all_excludes_source_parent_reparse_writers() {
async fn windows_rename_all_keeps_the_source_parent_from_becoming_a_reparse_point() {
use std::os::windows::fs::OpenOptionsExt;
use windows_sys::Win32::{
Foundation::GENERIC_WRITE,
Storage::FileSystem::{FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_OPEN_REPARSE_POINT, FILE_SHARE_READ, FILE_SHARE_WRITE},
Foundation::{ERROR_DIR_NOT_EMPTY, ERROR_SHARING_VIOLATION, GENERIC_WRITE},
Storage::FileSystem::{
DELETE, FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_OPEN_REPARSE_POINT, FILE_SHARE_DELETE, FILE_SHARE_READ,
FILE_SHARE_WRITE,
},
};
let temp_dir = tempdir().expect("create temp dir");
let base = temp_dir.path().join("bucket");
let source_parent = temp_dir.path().join("staging");
let outside = temp_dir.path().join("outside");
let src = source_parent.join("staged-object");
let dst = base.join("published-object");
std::fs::create_dir(&base).expect("create destination base");
std::fs::create_dir(&source_parent).expect("create source parent");
std::fs::create_dir(&outside).expect("create outside target");
std::fs::write(&src, b"original").expect("write original staged object");
let source_parent_for_hook = source_parent.clone();
let outside_for_hook = outside.clone();
windows_rename_test_hooks::install_before_publication(&dst, move || {
std::fs::OpenOptions::new()
// The pin shares writes, so a writer can open the source parent,
// but the parent still holds the staged entry and NTFS refuses to
// turn a non-empty directory into a mount point.
let writable_parent = std::fs::OpenOptions::new()
.access_mode(GENERIC_WRITE)
.share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(&source_parent_for_hook)
.expect_err("the source-parent guard must exclude a reparse writer");
.map(winapi_util::Handle::from_file)
.expect("a write-sharing pin admits a concurrent writer");
let err = try_set_windows_mount_point(&writable_parent, &outside_for_hook)
.expect_err("the pinned source parent must not become a mount point");
assert_eq!(err.raw_os_error(), i32::try_from(ERROR_DIR_NOT_EMPTY).ok());
let err = std::fs::OpenOptions::new()
.access_mode(DELETE)
.share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(&source_parent_for_hook)
.expect_err("the source-parent guard must exclude deletion and rename");
assert_eq!(err.raw_os_error(), i32::try_from(ERROR_SHARING_VIOLATION).ok());
});
rename_all(&src, &dst, &base)
@@ -10324,6 +10689,7 @@ mod tests {
assert!(!src.exists(), "the original staged entry must be moved");
assert_eq!(std::fs::read(&dst).expect("read published original object"), b"original");
assert!(!outside.join("published-object").exists(), "publication must not escape the base");
}
#[cfg(windows)]
@@ -10403,7 +10769,7 @@ mod tests {
let guard = mkdir_all_below_existing_base_std(&base, &base).expect("guard destination base");
let publication_root = test_publication_root(&[&src, &dst, &base]);
let source_parent_guard =
lock_windows_directory_tree(src.parent().expect("source path must have a parent"), None, &publication_root)
lock_windows_directory_tree(src.parent().expect("source path must have a parent"), &publication_root)
.expect("anchor source parent");
let source = open_windows_rename_source(&src, &source_parent_guard).expect("anchor source entry");
@@ -3636,4 +3636,73 @@ mod tests {
assert_eq!(encode_channel_capacity(16 * 1024 * 1024, 32 * 1024 * 1024), 2);
assert_eq!(encode_channel_capacity(1, usize::MAX), DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BLOCKS);
}
#[tokio::test]
async fn read_full_buf_or_eof_preserves_invalid_data_and_wraps_truncated_io_errors() {
use std::pin::Pin;
use std::task::{Context, Poll};
use tokio::io::{AsyncRead, ReadBuf};
struct StepReader {
prefix: Vec<u8>,
offset: usize,
err_kind: Option<std::io::ErrorKind>,
err_msg: &'static str,
}
impl AsyncRead for StepReader {
fn poll_read(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
if self.offset < self.prefix.len() {
let take = (self.prefix.len() - self.offset).min(buf.remaining());
buf.put_slice(&self.prefix[self.offset..self.offset + take]);
self.offset += take;
return Poll::Ready(Ok(()));
}
if let Some(kind) = self.err_kind.take() {
return Poll::Ready(Err(std::io::Error::new(kind, self.err_msg)));
}
Poll::Ready(Ok(()))
}
}
// 1. Error on the very first read (total == 0) preserves the original ErrorKind.
let mut r1 = StepReader {
prefix: Vec::new(),
offset: 0,
err_kind: Some(std::io::ErrorKind::BrokenPipe),
err_msg: "immediate broken pipe",
};
let mut buf = BytesMut::with_capacity(16);
let err = read_full_buf_or_eof(&mut r1, &mut buf, 16)
.await
.expect_err("immediate read error must fail");
assert_eq!(err.kind(), std::io::ErrorKind::BrokenPipe);
// 2. InvalidData (e.g. HashReader checksum/trailer mismatch) after partial fill preserves InvalidData.
let mut r2 = StepReader {
prefix: b"partial".to_vec(),
offset: 0,
err_kind: Some(std::io::ErrorKind::InvalidData),
err_msg: "Checksum mismatch",
};
let mut buf2 = BytesMut::with_capacity(16);
let err = read_full_buf_or_eof(&mut r2, &mut buf2, 16)
.await
.expect_err("InvalidData after partial fill must fail");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("Checksum mismatch"));
// 3. Non-InvalidData error after partial fill is normalized to UnexpectedEof.
let mut r3 = StepReader {
prefix: b"partial".to_vec(),
offset: 0,
err_kind: Some(std::io::ErrorKind::BrokenPipe),
err_msg: "mid-stream reset",
};
let mut buf3 = BytesMut::with_capacity(16);
let err = read_full_buf_or_eof(&mut r3, &mut buf3, 16)
.await
.expect_err("truncated stream error must fail");
assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof);
}
}
@@ -934,4 +934,124 @@ mod tests {
"cached record must still reject corrupt or mismatched shard payloads"
);
}
#[tokio::test]
async fn shard_verifier_rejects_out_of_bounds_index_and_empty_proof_sources_fail_closed() {
let mut builder = IntegrityBuilder::new(IntegrityLayout::new(2, 2, 8, false).expect("layout"), 1).expect("builder");
builder
.push([b"abcd".as_slice(), b"efgh", b"ijkl", b"mnop"].into_iter())
.await
.expect("stripe");
let prepared = builder.finish(8).await.expect("index");
let proof = Arc::new(PartProofReader {
part: prepared.part,
sources: Vec::new(),
volume: "bucket".to_owned(),
inline: None,
cached: Mutex::new(ProofCache::default()),
});
// coding_index >= layout.shards() is rejected at construction time.
let err = ShardVerifier::new(proof.clone(), 4, 0, None)
.err()
.expect("coding_index out of bounds must fail");
assert_eq!(err.kind(), io::ErrorKind::InvalidData);
assert!(err.to_string().contains("shard integrity verification failed"));
// Valid coding_index with neither inline bytes nor disk sources fails closed when reading stripe record.
let mut verifier = ShardVerifier::new(proof, 0, 0, None).expect("verifier");
let err = verifier
.verify(b"abcd")
.await
.expect_err("empty proof sources with no inline index must fail closed");
assert_eq!(err.kind(), io::ErrorKind::InvalidData);
assert!(err.to_string().contains("shard integrity verification failed"));
}
#[tokio::test]
async fn spilled_index_write_verify_and_restore_proof_replicas_round_trip() {
let (_dirs, store) = crate::services::rebalance::test_store_with_persisted_rebalance_meta(Default::default()).await;
let mut disk_slots = store.pools[0].disk_set[0].get_local_disks().await;
let disks: Vec<_> = disk_slots.iter().flatten().cloned().collect();
assert_eq!(disks.len(), 2);
let volume = "spilled-integrity-bucket";
for disk in &disks {
disk.make_volume(volume).await.expect("make volume");
}
// 2 shards (1 data + 1 parity) matches the 2-disk fixture; 2350 stripes * 448B > 1 MiB spills to Spool::File(_).
let layout = IntegrityLayout::new(1, 1, 8, false).expect("layout");
let mut builder = IntegrityBuilder::new(layout, 1).expect("builder");
let shards = [b"abcdefgh".as_slice(); 2];
for _ in 0..2350 {
builder.push(shards.into_iter()).await.expect("stripe");
}
let part_size = 2350 * 8;
let prepared = builder.finish(part_size).await.expect("spilled index");
assert!(matches!(prepared.index, Spool::File(_)));
let data_dir = uuid::Uuid::new_v4();
let object = "object";
let directory = format!("{object}/{data_dir}");
let part = prepared
.write(&mut disk_slots, volume, volume, &directory)
.await
.expect("write spilled index");
let rel_path = format!("{directory}/{}", part.file_name());
for disk in &disks {
verify_index_file(disk, volume, &rel_path, &part)
.await
.expect("verify spilled index on disk");
}
// Corrupt replica 0; restore_proof_replicas should repair it from healthy replica 1.
disks[0]
.write_all(volume, &rel_path, bytes::Bytes::from_static(b"corrupted-index"))
.await
.expect("corrupt disk 0");
assert!(verify_index_file(&disks[0], volume, &rel_path, &part).await.is_err());
let mut fi = rustfs_filemeta::FileInfo {
volume: volume.to_owned(),
name: object.to_owned(),
data_dir: Some(data_dir),
erasure: rustfs_filemeta::ErasureInfo {
data_blocks: 1,
parity_blocks: 1,
block_size: 8,
..Default::default()
},
parts: vec![rustfs_filemeta::ObjectPartInfo {
number: 1,
size: part_size,
actual_size: part_size as i64,
integrity: Some(part.clone()),
..Default::default()
}],
..Default::default()
};
fi.persist_shard_integrity().expect("persist shard integrity");
let repaired = restore_proof_replicas(&fi, &disk_slots, volume, object)
.await
.expect("restore proof replica 0");
assert_eq!(repaired, 1);
verify_index_file(&disks[0], volume, &rel_path, &part)
.await
.expect("repaired replica 0 must verify");
// Negative durability boundary: when all replicas are corrupted, restore_proof_replicas must fail closed.
for disk in &disks {
disk.write_all(volume, &rel_path, bytes::Bytes::from_static(b"all-corrupted"))
.await
.expect("corrupt all replicas");
}
let err = restore_proof_replicas(&fi, &disk_slots, volume, object)
.await
.expect_err("all corrupted proof replicas must fail closed");
assert!(
err.to_string().contains("shard integrity verification failed"),
"unexpected error when all proof replicas are corrupt: {err}"
);
}
}
+29 -12
View File
@@ -478,6 +478,7 @@ struct ScannerPublicationCommitScopeInner {
#[derive(Clone)]
pub struct ScannerPublicationCommitScope {
inner: Arc<ScannerPublicationCommitScopeInner>,
owns_completion: bool,
}
/// RAII fallback for storage paths that return before their commit closure
@@ -569,6 +570,16 @@ impl ScannerPublicationCommitScope {
movement_permit: Mutex::new(Some(movement_permit)),
lease_release_safe,
}),
owns_completion: true,
}
}
/// A pool may use the shared fence, but only the enclosing operation may
/// report completion and authorize release of its leases.
pub(crate) fn pool_delete_participant(&self) -> Self {
Self {
inner: Arc::clone(&self.inner),
owns_completion: false,
}
}
@@ -607,7 +618,9 @@ impl ScannerPublicationCommitScope {
/// storage mutation owner. Once set, dropping a scanner waiter leaves the
/// scope in-flight until that owner reports committed or indeterminate.
pub fn attach_mutation_owner(&self) {
self.inner.owner_attached.store(true, Ordering::Release);
if self.owns_completion {
self.inner.owner_attached.store(true, Ordering::Release);
}
}
fn owner_attached(&self) -> bool {
@@ -654,16 +667,17 @@ impl ScannerPublicationCommitScope {
}
pub fn mark_aborted_before_commit(&self) -> bool {
if self
.inner
.state
.compare_exchange(
SCANNER_PUBLICATION_SCOPE_ADMITTED,
SCANNER_PUBLICATION_SCOPE_ABORTED_BEFORE_COMMIT,
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
if self.owns_completion
&& self
.inner
.state
.compare_exchange(
SCANNER_PUBLICATION_SCOPE_ADMITTED,
SCANNER_PUBLICATION_SCOPE_ABORTED_BEFORE_COMMIT,
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
self.inner.lease_release_safe.store(true, Ordering::Release);
self.inner.completed.notify_waiters();
@@ -677,6 +691,9 @@ impl ScannerPublicationCommitScope {
}
fn mark_terminal(&self, terminal: ScannerPublicationCommitState) -> bool {
if !self.owns_completion {
return false;
}
self.inner
.state
.compare_exchange(SCANNER_PUBLICATION_SCOPE_IN_FLIGHT, terminal.as_u8(), Ordering::AcqRel, Ordering::Acquire)
@@ -709,7 +726,7 @@ impl ScannerPublicationCommitScope {
/// Release the storage-owned movement permit only after a known-safe
/// terminal result. Returns `false` for in-flight or indeterminate work.
pub async fn release_movement_permit(&self) -> bool {
if !self.state().permits_lease_release() {
if !self.owns_completion || !self.state().permits_lease_release() {
return false;
}
self.inner.movement_permit.lock().await.take().is_some()
+132 -1
View File
@@ -58,7 +58,7 @@ use std::sync::{
Arc, OnceLock,
atomic::{AtomicBool, AtomicU8, AtomicU64, Ordering},
};
use tokio::sync::{Mutex, Notify, OnceCell, OwnedRwLockReadGuard, RwLock};
use tokio::sync::{Mutex, Notify, OnceCell, OwnedRwLockReadGuard, OwnedSemaphorePermit, RwLock, Semaphore};
use tokio::time::Instant;
use tokio_util::sync::CancellationToken;
use tokio_util::task::TaskTracker;
@@ -187,6 +187,38 @@ impl DecommissionCountLedger {
}
}
#[derive(Debug)]
struct ListReadBudget {
permits: Arc<Semaphore>,
capacity: usize,
}
impl ListReadBudget {
fn new(capacity: usize) -> Self {
let capacity = capacity
.min(Semaphore::MAX_PERMITS)
.min(usize::try_from(u32::MAX).unwrap_or(usize::MAX));
Self {
permits: Arc::new(Semaphore::new(capacity)),
capacity,
}
}
async fn acquire(&self, fanout: usize) -> Result<Option<Arc<OwnedSemaphorePermit>>, tokio::sync::AcquireError> {
if self.capacity == 0 {
return Ok(None);
}
// A single walk must remain admissible even when the topology exceeds
// the configured budget. Tokio accepts at most u32::MAX in one grant.
let weight = u32::try_from(fanout.max(1).min(self.capacity)).unwrap_or(u32::MAX);
self.permits
.clone()
.acquire_many_owned(weight)
.await
.map(|permit| Some(Arc::new(permit)))
}
}
/// Runtime state owned by a single `ECStore` instance.
///
/// This is intentionally minimal in the first migration slice; subsequent
@@ -201,6 +233,7 @@ pub struct InstanceContext {
/// drift out of sync). Stored as one value so the three predicates can
/// never observe a torn intermediate state.
erasure_kind: RwLock<SetupType>,
list_read_budget: OnceLock<ListReadBudget>,
/// This instance's namespace lock manager (Phase 5 Slice 3, backlog#939).
///
/// Owned per-instance so two instances no longer share a lock namespace
@@ -354,6 +387,24 @@ impl InstanceContext {
Self::with_lock_manager(Arc::new(GlobalLockManager::new()))
}
/// Bound native listing fanout independently of GET body permits. A walk
/// keeps its shared grant until all of its disk producers have terminated.
/// Acquire before creating the merge channels or taking object locks.
pub(crate) async fn acquire_list_read_budget(
&self,
fanout: usize,
) -> Result<Option<Arc<OwnedSemaphorePermit>>, tokio::sync::AcquireError> {
self.list_read_budget
.get_or_init(|| {
ListReadBudget::new(rustfs_utils::get_env_usize(
rustfs_config::ENV_OBJECT_MAX_CONCURRENT_DISK_READS,
rustfs_config::DEFAULT_OBJECT_MAX_CONCURRENT_DISK_READS,
))
})
.acquire(fanout)
.await
}
/// Build a context bound to a specific lock manager.
///
/// [`bootstrap_ctx`] uses this to alias the process-global lock manager so
@@ -362,6 +413,7 @@ impl InstanceContext {
fn with_lock_manager(lock_manager: Arc<GlobalLockManager>) -> Self {
Self {
erasure_kind: RwLock::new(SetupType::Unknown),
list_read_budget: OnceLock::new(),
lock_manager,
region: OnceLock::new(),
deployment_id: OnceLock::new(),
@@ -1111,6 +1163,85 @@ pub fn bootstrap_ctx() -> Arc<InstanceContext> {
mod tests {
use super::*;
#[tokio::test]
async fn list_read_budget_weights_and_clamps_fanout() {
let budget = ListReadBudget::new(8);
let first = budget.acquire(3).await.expect("first walk").expect("bounded grant");
assert_eq!(budget.permits.available_permits(), 5);
let mut blocked = std::pin::pin!(budget.acquire(6));
assert!(futures::poll!(&mut blocked).is_pending());
drop(first);
let second = blocked.await.expect("queued walk").expect("bounded grant");
assert_eq!(budget.permits.available_permits(), 2);
drop(second);
let oversized = budget
.acquire(usize::MAX)
.await
.expect("oversized walk")
.expect("bounded grant");
assert_eq!(budget.permits.available_permits(), 0);
drop(oversized);
assert_eq!(budget.permits.available_permits(), 8);
}
#[tokio::test]
async fn list_read_budget_zero_preserves_unbounded_configuration() {
let budget = ListReadBudget::new(0);
assert!(budget.acquire(usize::MAX).await.expect("disabled budget").is_none());
}
#[tokio::test]
async fn list_read_budget_cancelled_waiter_returns_partial_grant() {
let budget = ListReadBudget::new(4);
let held = budget.acquire(3).await.expect("first walk");
{
let mut waiter = std::pin::pin!(budget.acquire(4));
assert!(futures::poll!(&mut waiter).is_pending());
}
assert_eq!(budget.permits.available_permits(), 1);
drop(held);
assert!(budget.acquire(4).await.expect("next walk").is_some());
}
#[tokio::test]
async fn list_read_budget_shared_grant_survives_parent_drop() {
let budget = ListReadBudget::new(4);
let parent = budget.acquire(4).await.expect("first walk").expect("bounded grant");
let producer = parent.clone();
drop(parent);
assert_eq!(budget.permits.available_permits(), 0);
drop(producer);
assert_eq!(budget.permits.available_permits(), 4);
}
#[tokio::test]
async fn list_read_budget_is_shared_by_context_and_isolated_between_instances() {
let first = Arc::new(InstanceContext::new());
first.list_read_budget.set(ListReadBudget::new(4)).expect("fresh context");
let same = first.clone();
let other = InstanceContext::new();
other.list_read_budget.set(ListReadBudget::new(4)).expect("fresh context");
let held = first.acquire_list_read_budget(4).await.expect("first context");
let mut waiter = std::pin::pin!(same.acquire_list_read_budget(1));
assert!(futures::poll!(&mut waiter).is_pending());
assert!(
other
.acquire_list_read_budget(4)
.await
.expect("independent context")
.is_some()
);
drop(held);
assert!(waiter.await.expect("shared context resumed").is_some());
}
#[test]
fn list_read_budget_extreme_configuration_is_bounded_without_panicking() {
let budget = ListReadBudget::new(usize::MAX);
assert!(budget.capacity <= Semaphore::MAX_PERMITS);
assert!(u32::try_from(budget.capacity).is_ok());
}
#[test]
fn decommission_count_ledger_bounds_retained_entries() {
let mut ledger = super::DecommissionCountLedger::default();
@@ -385,3 +385,165 @@ async fn pause_and_cancel_are_durable_during_content_read() {
JobState::Cancelled
);
}
#[test]
fn job_request_validation_and_protection_classification_cover_all_negative_branches() {
use crate::object_api::ObjectInfo;
let valid_digest = rustfs_rio::Checksum::new_from_data(rustfs_rio::ChecksumType::SHA256, b"payload")
.expect("sha256")
.encoded;
// 1. Empty items and > MAX_ITEMS (64).
let mut empty_req = request(JobMode::Audit, valid_digest.clone());
empty_req.items.clear();
let err = empty_req.validate().expect_err("empty items must fail");
assert!(
err.to_string().contains("a job requires 1..=64 explicit source versions"),
"unexpected error: {err}"
);
let mut too_many = request(JobMode::Audit, valid_digest.clone());
too_many.items = (0..65)
.map(|i| ItemRequest {
key: format!("key-{i}"),
version_id: None,
expected_sha256: None,
target_key: None,
})
.collect();
let err = too_many.validate().expect_err(">64 items must fail");
assert!(
err.to_string().contains("a job requires 1..=64 explicit source versions"),
"unexpected error: {err}"
);
// 2. bytes_per_second and max_object_bytes bounds.
let mut low_bps = request(JobMode::Audit, valid_digest.clone());
low_bps.bytes_per_second = 64 * 1024 - 1;
let err = low_bps.validate().expect_err("low bytes_per_second must fail");
assert!(err.to_string().contains("invalid byte rate or staging limit"), "unexpected error: {err}");
let mut high_bps = request(JobMode::Audit, valid_digest.clone());
high_bps.bytes_per_second = 1024 * 1024 * 1024 + 1;
let err = high_bps.validate().expect_err("high bytes_per_second must fail");
assert!(err.to_string().contains("invalid byte rate or staging limit"), "unexpected error: {err}");
let mut zero_max = request(JobMode::Audit, valid_digest.clone());
zero_max.max_object_bytes = 0;
let err = zero_max.validate().expect_err("zero max_object_bytes must fail");
assert!(err.to_string().contains("invalid byte rate or staging limit"), "unexpected error: {err}");
let mut huge_max = request(JobMode::Audit, valid_digest.clone());
huge_max.max_object_bytes = model::MAX_OBJECT_BYTES + 1;
let err = huge_max.validate().expect_err("huge max_object_bytes must fail");
assert!(err.to_string().contains("invalid byte rate or staging limit"), "unexpected error: {err}");
// 3. Invalid keys (empty, trailing '/', embedded '\0', > 1024 chars).
for bad_key in ["", "trailing/", "null\0byte", &"x".repeat(1025)] {
let mut req = request(JobMode::Audit, valid_digest.clone());
req.items[0].key = bad_key.to_string();
let err = req.validate().expect_err("invalid key must fail");
assert!(
err.to_string().contains("invalid object key"),
"expected 'invalid object key' for {bad_key:?}, got {err}"
);
}
// 4. Invalid version_id (nil UUID, malformed UUID, empty string) and valid UUID.
for bad_ver in ["", "not-a-uuid", &Uuid::nil().to_string()] {
let mut req = request(JobMode::Audit, valid_digest.clone());
req.items[0].version_id = Some(bad_ver.to_string());
let err = req.validate().expect_err("invalid version_id must fail");
assert!(
err.to_string().contains("invalid source version"),
"expected 'invalid source version' for {bad_ver:?}, got {err}"
);
}
let mut valid_uuid_ver = request(JobMode::Audit, valid_digest.clone());
valid_uuid_ver.items[0].version_id = Some(Uuid::new_v4().to_string());
assert!(valid_uuid_ver.validate().is_ok());
// 5. Invalid expected_sha256 (bad base64, wrong byte length).
let err = request(JobMode::Audit, "!!!not_base64!!!".to_string())
.validate()
.expect_err("non-base64 sha256 must fail");
assert!(err.to_string().contains("SHA256 must be canonical Base64"), "unexpected error: {err}");
let err = request(JobMode::Audit, "AAAA".to_string())
.validate()
.expect_err("wrong-length sha256 must fail");
assert!(
err.to_string().contains("SHA256 must be canonical Base64 of 32 bytes"),
"unexpected error: {err}"
);
// 6. Mode-specific target_key rules (Audit forbids target_key; Migrate requires unique target_key != key).
let mut audit_with_target = request(JobMode::Audit, valid_digest.clone());
audit_with_target.items[0].target_key = Some("target".to_string());
let err = audit_with_target.validate().expect_err("audit with target_key must fail");
assert!(
err.to_string().contains("only migrations require an explicit target key"),
"unexpected error: {err}"
);
let mut migrate_no_target = request(JobMode::Migrate, valid_digest.clone());
migrate_no_target.items[0].target_key = None;
let err = migrate_no_target
.validate()
.expect_err("migrate without target_key must fail");
assert!(
err.to_string().contains("only migrations require an explicit target key"),
"unexpected error: {err}"
);
let mut migrate_same_target = request(JobMode::Migrate, valid_digest.clone());
migrate_same_target.items[0].target_key = Some("source".to_string());
let err = migrate_same_target
.validate()
.expect_err("migrate target == source must fail");
assert!(
err.to_string()
.contains("targets must be distinct from every source and target"),
"unexpected error: {err}"
);
let mut migrate_dup_target = request(JobMode::Migrate, valid_digest);
migrate_dup_target.items.push(ItemRequest {
key: "source-2".into(),
version_id: None,
expected_sha256: None,
target_key: Some("target".into()),
});
let err = migrate_dup_target.validate().expect_err("duplicate migrate target must fail");
assert!(
err.to_string()
.contains("targets must be distinct from every source and target"),
"unexpected error: {err}"
);
// 7. model::unsupported and model::protection classifications.
let mut delete_marker = ObjectInfo {
delete_marker: true,
..Default::default()
};
assert_eq!(model::unsupported(&delete_marker), Some("delete_marker"));
assert_eq!(model::protection(&delete_marker), Protection::NoPayload);
delete_marker.delete_marker = false;
let mut transitioned = ObjectInfo::default();
transitioned.transitioned_object.status = "Complete".to_string();
assert_eq!(model::unsupported(&transitioned), Some("transitioned_object"));
let mut encrypted = ObjectInfo::default();
Arc::make_mut(&mut encrypted.user_defined).insert("x-amz-server-side-encryption".to_string(), "AES256".to_string());
assert_eq!(model::unsupported(&encrypted), Some("transformed_object"));
let mut invalid_decl = ObjectInfo::default();
rustfs_utils::http::insert_str(
Arc::make_mut(&mut invalid_decl.user_defined),
rustfs_filemeta::shard_integrity::SUFFIX_SHARD_INTEGRITY,
"not-json".to_string(),
);
assert_eq!(model::unsupported(&invalid_decl), Some("invalid_protection_declaration"));
assert_eq!(model::protection(&invalid_decl), Protection::InvalidDeclaration);
}
@@ -593,7 +593,7 @@ impl ECStore {
#[cfg(test)]
crate::core::pools::observe_pool_activation_start_attempt(crate::core::pools::PoolActivationStartKind::Rebalance);
let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
let mut pool_meta_guard = self.pool_meta_save_gate.write().await;
pool_meta_guard.ensure_write_safe(stage)?;
let activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), fleet_proof).await?;
let pool_meta = self
@@ -625,7 +625,7 @@ impl ECStore {
{
// Lock order: pool_meta_save_gate -> rebalance.bin -> pool.bin. These
// shared fences admit a peer without waiting for the run's own entries.
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
let mut pool_meta_guard = self.pool_meta_save_gate.write().await;
pool_meta_guard.ensure_write_safe("rebalance worker activation")?;
let mut worker_fence = acquire_pool_rebalance_worker_locks(pool.clone()).await?;
let pool_meta = self
@@ -654,7 +654,7 @@ impl ECStore {
{
// The admission read fences must already be dropped. Reacquire in the
// original order and revalidate every durable boundary across that gap.
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
let mut pool_meta_guard = self.pool_meta_save_gate.write().await;
pool_meta_guard.ensure_write_safe("rebalance worker metadata update")?;
let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?;
let pool_meta = self
@@ -1714,7 +1714,7 @@ mod tests {
persist_pool_meta_identity_for_startup(store.pools.clone(), &mut write_state, true)
.await
.expect("initialized pool metadata identity should persist");
*store.pool_meta_save_gate.lock().await = write_state;
*store.pool_meta_save_gate.write().await = write_state;
// Simulate physical loss; raw deletion of authoritative pool.bin is
// deliberately rejected by the native metadata mutation guard.
for pool in &store.pools {
@@ -1910,6 +1910,10 @@ mod tests {
&ObjectOptions {
version_id: Some(version_id.to_string()),
mod_time: earlier(version.mod_time),
// Without the object lock the rename waits for every
// drive, so no early-ACK tail is still writing the copies
// removed below.
no_lock: true,
..versioned
},
))
@@ -2250,6 +2254,195 @@ mod tests {
.expect("large-stack rebalance test thread should complete");
}
#[test]
#[serial_test::serial]
fn real_rebalance_preserves_encrypted_multipart_etag_proof() {
run_large_stack_async_test("rebalance-encrypted-multipart-etag", || async {
use crate::io_support::rio::{HashReader, WriteEncryption, WritePlan};
use rustfs_utils::http::{SSEC_ALGORITHM_HEADER, SUFFIX_OPAQUE_ENCRYPTED_ETAG, etag_can_be_listed};
const REBALANCE_ID: &str = "rebalance-encrypted-multipart-etag";
for (object, versioned, protected) in [
("protected-version", true, true),
("protected-null", false, true),
("legacy-version", true, false),
("legacy-null", false, false),
] {
// Each case needs a fresh progress counter: the fixture's small
// capacity can reach its rebalance goal after one encrypted object.
let (_temp_dirs, store, _peer) =
crate::services::rebalance::test_two_pool_stores_with_isolated_node_contexts(None).await;
let bucket = "rebalance-encrypted-etag";
Box::pin(store.make_bucket(
bucket,
&MakeBucketOptions {
versioning_enabled: versioned,
..Default::default()
},
))
.await
.expect("create the migration bucket");
prepare_rebalance_test_volumes(&store).await;
let incarnation = store.bucket_incarnation_id(bucket).await.expect("bucket incarnation");
let rebalance_meta = active_rebalance_meta(REBALANCE_ID);
rebalance_meta
.save(Arc::clone(&store.pools[0]))
.await
.expect("persist active rebalance");
*store.rebalance_meta.write().await = Some(rebalance_meta);
let source_set = store.pools[0].get_disks_by_key(object);
let target_set = store.pools[1].get_disks_by_key(object);
let write_opts = ObjectOptions {
versioned,
user_defined: std::collections::HashMap::from([(SSEC_ALGORITHM_HEADER.to_owned(), "AES256".to_owned())]),
..Default::default()
};
let upload = Box::pin(source_set.new_multipart_upload(bucket, object, &write_opts))
.await
.expect("create source encrypted multipart upload");
let plaintext = b"encrypted multipart bytes must survive pool migration";
let size = i64::try_from(plaintext.len()).expect("small fixture length");
let source = HashReader::from_stream(std::io::Cursor::new(plaintext.to_vec()), size, size, None, None, false)
.expect("plaintext source reader");
let encrypted = WritePlan::new()
.with_encryption(WriteEncryption::multipart([7; 32], [9; 12], 1))
.apply(source, size)
.expect("encrypt the source part through the production writer");
let mut part_opts = write_opts.clone();
if !protected {
// Model a retained legacy fingerprint without producer proof.
part_opts.preserve_etag = Some("0123456789abcdef0123456789abcdef".to_owned());
}
let part = Box::pin(source_set.put_object_part(
bucket,
object,
&upload.upload_id,
1,
&mut PutObjReader::new(encrypted),
&part_opts,
))
.await
.expect("persist source encrypted part");
let written = Box::pin(Arc::clone(&source_set).complete_multipart_upload(
bucket,
object,
&upload.upload_id,
vec![CompletePart {
part_num: part.part_num,
etag: part.etag,
..Default::default()
}],
&write_opts,
))
.await
.expect("complete source encrypted multipart upload");
assert_eq!(written.version_id.is_some(), versioned);
assert!(written.is_multipart(), "the actual migration must use its multipart path");
assert_eq!(etag_can_be_listed(&written.user_defined), protected);
let read_opts = ObjectOptions {
version_id: Some(data_movement::data_movement_exact_version_id(written.version_id)),
data_movement: true,
raw_data_movement_read: true,
..Default::default()
};
let source_bytes = {
let mut reader = Box::pin(source_set.get_object_reader(bucket, object, None, HeaderMap::new(), &read_opts))
.await
.expect("read source ciphertext");
let mut bytes = Vec::new();
reader
.stream
.read_to_end(&mut bytes)
.await
.expect("collect source ciphertext");
bytes
};
assert_ne!(source_bytes.as_slice(), plaintext, "the fixture must contain encrypted bytes");
let entry = metacache_entry_from_source(&source_set, bucket, object).await;
let outcome = tokio::time::timeout(
StdDuration::from_secs(60),
Box::pin(Arc::clone(&store).rebalance_entry(
RebalanceEntryTarget {
bucket: bucket.to_owned(),
pool_index: 0,
},
entry,
Arc::clone(&source_set),
Arc::new(RebalanceBucketConfigs {
bucket_incarnation_id: Some(incarnation),
..Default::default()
}),
Arc::from(REBALANCE_ID),
CancellationToken::new(),
)),
)
.await
.expect("rebalance must finish within the test deadline")
.expect("rebalance source entry");
let target = match target_set.get_object_info(bucket, object, &read_opts).await {
Ok(target) => target,
Err(err) => {
let target_versions = target_set
.load_file_info_versions_exact(bucket, object)
.await
.expect("inspect target after metadata read failure")
.map(|versions| {
versions
.versions
.into_iter()
.map(|version| version.version_id)
.collect::<Vec<_>>()
});
let source_versions = source_set
.load_file_info_versions_exact(bucket, object)
.await
.expect("inspect source after metadata read failure")
.map(|versions| {
versions
.versions
.into_iter()
.map(|version| version.version_id)
.collect::<Vec<_>>()
});
panic!(
"read migrated {object}: {err}; outcome={outcome:?}, expected_version={:?}, target_versions={target_versions:?}, source_versions={source_versions:?}",
written.version_id
);
}
};
assert_eq!(target.version_id, written.version_id, "{object}");
assert_eq!(target.etag, written.etag, "{object}");
assert_eq!(
rustfs_utils::http::get_consistent_str(&target.user_defined, SUFFIX_OPAQUE_ENCRYPTED_ETAG),
protected.then_some("v1"),
"aggregate ETag proof must follow the exact source version for {object}"
);
assert_eq!(etag_can_be_listed(&target.user_defined), protected, "{object}");
assert_eq!(outcome, RebalanceEntryOutcome::Completed, "{object}");
let mut target_reader =
Box::pin(target_set.get_object_reader(bucket, object, None, HeaderMap::new(), &read_opts))
.await
.expect("read target ciphertext");
let mut target_bytes = Vec::new();
target_reader
.stream
.read_to_end(&mut target_bytes)
.await
.expect("collect target ciphertext");
assert_eq!(target_bytes, source_bytes, "rebalance preserves ciphertext bytes for {object}");
assert!(
source_set
.load_file_info_versions_exact(bucket, object)
.await
.expect("inspect source after migration")
.is_none(),
"source cleanup follows a verified target for {object}"
);
}
});
}
const TRANSITIONED_REBALANCE_ID: &str = "rebalance-transitioned-source-cleanup";
/// One transitioned source version on pool 0 of a two-pool store with an
+2 -2
View File
@@ -100,7 +100,7 @@ pub async fn test_store_with_persisted_rebalance_meta(
rebalance_meta: tokio::sync::RwLock::new(Some(meta)),
decommission_cancelers: tokio::sync::RwLock::new(vec![None]),
start_gate: tokio::sync::Mutex::new(()),
pool_meta_save_gate: tokio::sync::Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx,
bucket_fence_registry: std::sync::Arc::default(),
});
@@ -335,7 +335,7 @@ async fn test_pool_stores_with_contexts(
rebalance_meta: tokio::sync::RwLock::new(rebalance_meta.clone()),
decommission_cancelers: tokio::sync::RwLock::new(vec![None; pool_count]),
start_gate: tokio::sync::Mutex::new(()),
pool_meta_save_gate: tokio::sync::Mutex::new(pool_meta_write_state.independent_clone_for_test()),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::new(pool_meta_write_state.independent_clone_for_test()),
ctx: store_ctx,
bucket_fence_registry: std::sync::Arc::default(),
})
@@ -3412,7 +3412,7 @@ fn test_store_with_rebalance_meta(meta: RebalanceMeta) -> Arc<crate::store::ECSt
rebalance_meta: tokio::sync::RwLock::new(Some(meta)),
decommission_cancelers: tokio::sync::RwLock::new(Vec::new()),
start_gate: tokio::sync::Mutex::new(()),
pool_meta_save_gate: tokio::sync::Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx: crate::runtime::instance::bootstrap_ctx(),
bucket_fence_registry: std::sync::Arc::default(),
})
+1
View File
@@ -26,6 +26,7 @@ pub(crate) mod tier_probe_intent;
pub mod warm_backend;
pub mod warm_backend_aliyun;
pub mod warm_backend_azure;
pub mod warm_backend_azure_blob;
#[cfg(feature = "gcs")]
pub mod warm_backend_gcs;
pub mod warm_backend_huaweicloud;
+410 -7
View File
@@ -93,6 +93,7 @@ use super::{
delete_tier_mutation_intent_record_if_current, list_tier_coordinator_mutation_intent_records,
list_tier_mutation_intent_records, load_tier_coordinator_mutation_intent_record_with_etag,
load_tier_mutation_intent_record_with_etag, save_tier_coordinator_mutation_intent_record_if_absent,
save_tier_mutation_intent_record_if_current,
},
warm_backend::WarmBackendImpl,
};
@@ -327,6 +328,7 @@ pub const TIER_CONFIG_VERSION: u16 = 2;
const EXTERNAL_TIER_TYPE_UNSUPPORTED: i32 = 0;
const EXTERNAL_TIER_TYPE_S3: i32 = 1;
const EXTERNAL_TIER_TYPE_AZURE: i32 = 2;
const EXTERNAL_TIER_AZURE_BLOB_ENDPOINT_SENTINEL: &str = "rustfs-azure-blob-v1:";
const EXTERNAL_TIER_TYPE_GCS: i32 = 3;
const EXTERNAL_TIER_TYPE_MINIO: i32 = 4;
const EXTERNAL_TIER_VERSION_WASABI_V1: &str = "rustfs-wasabi-v1";
@@ -619,7 +621,11 @@ struct CommittedMutationRecoveryPlan {
impl TierMutationRecoverySnapshot {
fn requires_config_lock(&self) -> bool {
self.has_committed_mutation_blocks || self.intents.iter().any(|recovered| recovered.has_coordinator_record)
self.has_committed_mutation_blocks
|| self.intents.iter().any(|recovered| {
recovered.has_coordinator_record
|| (recovered.has_peer_record && recovered.intent.state == TierMutationIntentState::Prepared)
})
}
fn allowance(&self) -> MutationBlockAllowance {
@@ -2147,6 +2153,7 @@ fn build_coordinator_tier_mutation_intent(
return Ok(None);
}
Ok(Some(TierMutationIntent {
requires_native_azure: candidate.tiers.values().any(TierConfig::is_native_azure),
mutation_id: uuid::Uuid::new_v4(),
revision: 1,
kind,
@@ -2515,11 +2522,11 @@ fn tier_backend_identity(config: &TierConfig) -> io::Result<TierDestinationId> {
}),
TierType::Azure => config.azure.as_ref().map(|value| {
(
"azure",
if config.is_native_azure() { "azure-blob-v1" } else { "azure" },
value.endpoint.as_str(),
value.bucket.as_str(),
value.prefix.as_str(),
value.region.as_str(),
if config.is_native_azure() { "" } else { value.region.as_str() },
value.access_key.as_str(),
)
}),
@@ -2592,7 +2599,7 @@ fn normalized_tier_prefix<'a>(tier_type: &TierType, prefix: &'a str) -> &'a str
}
fn tier_exact_get_delete(config: &TierConfig) -> bool {
ProviderVersionCapabilities::for_tier_type(&config.tier_type.as_lowercase()).exact_get_delete
config.is_native_azure() || ProviderVersionCapabilities::for_tier_type(&config.tier_type.as_lowercase()).exact_get_delete
}
struct TierDriverGeneration {
@@ -3130,7 +3137,15 @@ fn to_external_tier_config(name: &str, tier: &TierConfig) -> io::Result<External
.ok_or_else(|| io::Error::other("tier config missing azure backend payload"))?;
out.tier_type = EXTERNAL_TIER_TYPE_AZURE;
out.azure = Some(ExternalTierAzure {
endpoint: az.endpoint.clone(),
// Keep the v2 Azure payload readable and rewriteable by older
// servers, but make their S3 constructor reject its endpoint.
// An unknown numeric type would discard the Azure payload on
// an unrelated configuration rewrite by an older server.
endpoint: if tier.is_native_azure() {
format!("{EXTERNAL_TIER_AZURE_BLOB_ENDPOINT_SENTINEL}{}", az.endpoint)
} else {
az.endpoint.clone()
},
account_name: az.access_key.clone(),
account_key: az.secret_key.clone(),
bucket: az.bucket.clone(),
@@ -3289,6 +3304,15 @@ fn from_external_tier_config(name: String, ext: ExternalTierConfig) -> io::Resul
..Default::default()
};
let azure_blob = ext.version == super::tier_config::AZURE_BLOB_TIER_VERSION;
let native_endpoint = ext
.azure
.as_ref()
.is_some_and(|azure| azure.endpoint.starts_with(EXTERNAL_TIER_AZURE_BLOB_ENDPOINT_SENTINEL));
if azure_blob != native_endpoint || azure_blob && (ext.tier_type != EXTERNAL_TIER_TYPE_AZURE || ext.tier_type_hint.is_some())
{
return Err(io::Error::other("native Azure Blob tier has inconsistent protocol discriminators"));
}
let wasabi_hint = ext.tier_type_hint.as_deref() == Some("wasabi");
let wasabi_version = ext.version == EXTERNAL_TIER_VERSION_WASABI_V1;
if wasabi_hint && !wasabi_version {
@@ -3309,7 +3333,9 @@ fn from_external_tier_config(name: String, ext: ExternalTierConfig) -> io::Resul
cfg.name
)));
}
let tier_type = if wasabi_version {
let tier_type = if azure_blob {
TierType::Azure
} else if wasabi_version {
TierType::Wasabi
} else {
tier_type_from_hint(ext.tier_type_hint.as_deref()).unwrap_or(match ext.tier_type {
@@ -3387,7 +3413,15 @@ fn from_external_tier_config(name: String, ext: ExternalTierConfig) -> io::Resul
.ok_or_else(|| io::Error::other(format!("tier config '{}' missing azure backend payload", cfg.name)))?;
cfg.azure = Some(crate::services::tier::tier_config::TierAzure {
name: cfg.name.clone(),
endpoint: az.endpoint.clone(),
endpoint: if azure_blob {
az.endpoint
.strip_prefix(EXTERNAL_TIER_AZURE_BLOB_ENDPOINT_SENTINEL)
.filter(|endpoint| !endpoint.is_empty())
.ok_or_else(|| io::Error::other("native Azure Blob endpoint marker is missing"))?
.to_string()
} else {
az.endpoint.clone()
},
access_key: az.account_name.clone(),
secret_key: az.account_key.clone(),
bucket: az.bucket.clone(),
@@ -3558,6 +3592,9 @@ impl TierConfigMgr {
}
pub fn unmarshal(data: &[u8]) -> std::result::Result<TierConfigMgr, std::io::Error> {
if data.starts_with(&TIER_CONFIG_FORMAT.to_le_bytes()) {
return decode_external_tiering_config_blob(data);
}
let mut cfg: TierConfigMgr = serde_json::from_slice(data)?;
for (name, tier) in &mut cfg.tiers {
if matches!(&tier.tier_type, TierType::Wasabi) {
@@ -3575,6 +3612,11 @@ impl TierConfigMgr {
}
pub fn marshal(&self) -> std::result::Result<Bytes, std::io::Error> {
// The legacy JSON shape omits the driver protocol version. Native
// Azure must retain the binary discriminator even through this helper.
if self.tiers.values().any(TierConfig::is_native_azure) {
return encode_external_tiering_config_blob(self);
}
let data = serde_json::to_vec(&self)?;
let mut data = Bytes::from(data);
@@ -3608,6 +3650,17 @@ impl TierConfigMgr {
deadline: Option<Instant>,
) -> std::result::Result<(), AdminError> {
normalize_s3_gcs_add_tier_name(&mut tier_config)?;
if tier_config.tier_type == TierType::Azure {
tier_config.version = super::tier_config::AZURE_BLOB_TIER_VERSION.to_string();
if let Some(azure) = tier_config.azure.as_mut() {
if azure.storage_class.eq_ignore_ascii_case("STANDARD") {
azure.storage_class.clear();
}
if azure.endpoint.is_empty() {
azure.endpoint = format!("https://{}.blob.core.windows.net", azure.access_key);
}
}
}
self.ensure_generation_is_idle(&tier_config.name)?;
let tier_name = tier_config.name.clone();
if tier_name != tier_name.to_uppercase() {
@@ -5054,6 +5107,9 @@ impl TierConfigMgr {
Self::reconcile_terminal_dual_prefix_mutation_intents(api.clone(), &mut snapshot).await?;
Self::recover_prepared_coordinator_mutation_intents(api.clone(), &mut snapshot.coordinator_intents).await?;
if config_lock.is_some() {
Self::recover_expired_peer_only_prepares(api.clone(), &snapshot).await?;
}
snapshot = Self::load_and_reconcile_mutation_recovery_snapshot(handle, api.clone(), true).await?;
if snapshot.requires_config_lock() && config_lock.is_none() {
@@ -5454,6 +5510,58 @@ impl TierConfigMgr {
}
}
async fn recover_expired_peer_only_prepares<S>(api: Arc<S>, snapshot: &TierMutationRecoverySnapshot) -> io::Result<()>
where
S: EcstoreObjectIO,
{
let now = i64::try_from(OffsetDateTime::now_utc().unix_timestamp_nanos()).unwrap_or(i64::MAX);
let orphaned = snapshot
.intents
.iter()
.filter(|recovered| {
recovered.has_peer_record
&& !recovered.has_coordinator_record
&& recovered.intent.state == TierMutationIntentState::Prepared
&& Self::tier_mutation_tombstone_retention_elapsed(&recovered.intent, now)
})
.map(|recovered| recovered.intent.clone())
.collect::<Vec<_>>();
if orphaned.is_empty() {
return Ok(());
}
// The caller holds the tier-config namespace lock. Only the exact
// persisted candidate proves a commit; an old or different config
// leaves the peer fence intact.
let loaded = load_tier_config_for_recovery(api.clone()).await?;
let Some(config_etag) = loaded.etag else {
return Ok(());
};
for intent in orphaned {
if loaded.persisted_digest != Some(intent.candidate_digest) {
continue;
}
let _mutation_guard = acquire_tier_mutation_mutex(intent.mutation_id).await;
let (mut current, record_etag) =
match load_tier_mutation_intent_record_with_etag(api.clone(), intent.mutation_id).await {
Ok(record) => record,
Err(Error::ConfigNotFound) => continue,
Err(err) => return Err(tier_mutation_replay_error(err)),
};
if current != intent {
continue;
}
current
.advance(TierMutationIntentState::Committed, Some(config_etag.clone()))
.map_err(io::Error::other)?;
match save_tier_mutation_intent_record_if_current(api.clone(), &current, &record_etag).await {
Ok(()) | Err(Error::PreconditionFailed) => {}
Err(err) => return Err(tier_mutation_replay_error(err)),
}
}
Ok(())
}
async fn recover_prepared_coordinator_mutation_intents<S>(api: Arc<S>, intents: &mut [TierMutationIntent]) -> io::Result<()>
where
S: EcstoreObjectIO,
@@ -9536,6 +9644,98 @@ mod tests {
assert_eq!(az.sp_auth.client_secret, "secret");
}
#[tokio::test]
async fn native_azure_add_preserves_protocol_through_restart_and_requires_peer_support() {
let mut manager = empty_mgr();
let mut tier = build_azure_tier("account");
let name = tier.name.clone();
let azure = tier.azure.as_mut().expect("Azure payload should exist");
azure.endpoint.clear();
azure.storage_class = "STANDARD".to_string();
azure.secret_key = base64_simd::STANDARD.encode_to_string([7; 32]);
TIER_DRIVER_TEST_FACTORY
.scope(healthy_driver_factory(), manager.add(tier, true))
.await
.expect("current Console Azure payload should be admitted");
let created = &manager.tiers[&name];
assert!(created.is_native_azure());
assert!(tier_exact_get_delete(created));
let azure = created.azure.as_ref().expect("Azure payload should survive");
assert_eq!(azure.endpoint, "https://account.blob.core.windows.net");
assert!(azure.storage_class.is_empty());
let bytes = manager
.marshal()
.expect("native configuration should serialize without losing its protocol");
let external: ExternalTierConfigMgr =
rmp_serde::from_slice(&bytes[4..]).expect("native config should use the binary shape");
assert_eq!(external.tiers[&name].tier_type, EXTERNAL_TIER_TYPE_AZURE);
let stored = external.tiers[&name]
.azure
.as_ref()
.expect("native storage keeps the existing Azure payload");
assert_eq!(stored.endpoint, "rustfs-azure-blob-v1:https://account.blob.core.windows.net");
let legacy_backend_config = crate::services::tier::tier_config::TierAzure {
endpoint: stored.endpoint.clone(),
access_key: stored.account_name.clone(),
secret_key: stored.account_key.clone(),
bucket: stored.bucket.clone(),
..Default::default()
};
assert!(
crate::services::tier::warm_backend_azure::WarmBackendAzure::new(&legacy_backend_config, &name)
.await
.is_err(),
"the unchanged legacy S3 constructor must reject the native endpoint before any request"
);
assert!(
external.tiers[&name].tier_type_hint.is_none(),
"the protocol marker must not be overridden by an extended provider hint"
);
let restarted = TierConfigMgr::unmarshal(&bytes).expect("native configuration should reload");
assert!(restarted.tiers[&name].is_native_azure());
assert_eq!(
tier_backend_identity(created).expect("created identity"),
tier_backend_identity(&restarted.tiers[&name]).expect("reloaded identity")
);
let targets = vec![TierMutationIntentTarget {
tier_name: name,
old_backend_identity: None,
new_backend_identity: Some(tier_backend_identity(created).expect("native destination identity")),
}];
let intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &manager, targets)
.expect("native intent should build")
.expect("native Add must use peer prepare");
assert!(intent.requires_native_azure);
}
#[test]
fn native_azure_protocol_is_distinct_from_legacy_and_rejects_corrupt_discriminators() {
let legacy = build_azure_tier("account");
let mut native = legacy.clone_with_credentials();
native.version = super::super::tier_config::AZURE_BLOB_TIER_VERSION.to_string();
assert_ne!(
tier_backend_identity(&legacy).expect("legacy identity"),
tier_backend_identity(&native).expect("native identity"),
"native Blob must never acquire a legacy S3 destination's cleanup authority"
);
let stored = to_external_tier_config(&legacy.name, &legacy).expect("legacy encoding");
assert_eq!(stored.tier_type, EXTERNAL_TIER_TYPE_AZURE);
let decoded = from_external_tier_config(legacy.name, stored).expect("legacy decoding");
assert!(!decoded.is_native_azure());
assert!(!tier_exact_get_delete(&decoded));
let stored = to_external_tier_config(&native.name, &native).expect("native encoding");
for mutate in [
|config: &mut ExternalTierConfig| config.version = "v1".to_string(),
|config: &mut ExternalTierConfig| config.tier_type = EXTERNAL_TIER_TYPE_S3,
|config: &mut ExternalTierConfig| config.tier_type_hint = Some("azure".to_string()),
] {
let mut corrupt = stored.clone();
mutate(&mut corrupt);
assert!(from_external_tier_config(native.name.clone(), corrupt).is_err());
}
}
#[test]
fn test_external_blob_gcs_writer_stays_raw_for_v2_old_readers() {
const RAW_JSON: &str = r#"{"type":"service_account","project_id":"tier-௿"}"#;
@@ -9944,6 +10144,7 @@ mod tests {
fn prepared_remove_intent(tier_name: &str, mutation_id: uuid::Uuid) -> TierMutationIntent {
TierMutationIntent {
requires_native_azure: false,
mutation_id,
revision: 1,
kind: TierMutationIntentKind::Remove,
@@ -13238,6 +13439,208 @@ mod tests {
);
}
#[tokio::test]
async fn reload_recovers_expired_peer_only_prepare_for_persisted_add() {
use crate::services::tier::tier_mutation_intent::save_tier_mutation_intent_record;
let store = Arc::new(CasConfigStore::default());
let previous = empty_mgr();
let mut candidate = empty_mgr();
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
let targets = build_tier_mutation_affected_targets(
TierMutationIntentKind::Add,
HashSet::from(["COLD-A".to_string()]),
&previous,
&candidate,
)
.expect("added tier should have a target");
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
.expect("intent should build")
.expect("add should require an intent");
intent.expires_at_unix_nanos = 1;
candidate
.save_tiering_config_if_current(store.clone(), None)
.await
.expect("candidate should persist");
save_tier_mutation_intent_record(store.clone(), &intent)
.await
.expect("peer prepare should persist without a coordinator record");
let manager = TierConfigMgr::new();
TIER_MUTATION_TEST_PEERS
.scope(Vec::new(), async {
TIER_DRIVER_TEST_FACTORY
.scope(healthy_driver_factory(), async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect("matching saved candidate should recover");
TierConfigMgr::acquire_operation_lease(&manager, "COLD-A")
.await
.expect("recovered tier should be readable");
})
.await;
})
.await;
assert!(
TierConfigMgr::load_tier_mutation_intents(store)
.await
.expect("peer scan should succeed")
.iter()
.all(|record| record.state != TierMutationIntentState::Prepared),
"the orphaned prepare must no longer block later reloads"
);
}
#[tokio::test]
async fn reload_keeps_peer_only_prepare_without_commit_proof() {
use crate::services::tier::tier_mutation_intent::{
save_tier_mutation_intent_record, tier_mutation_intent_record_object_name,
};
for case in [
"unexpired",
"recently_expired",
"different_candidate",
"missing_config",
"concurrent_change",
] {
let store = Arc::new(CasConfigStore::default());
let previous = empty_mgr();
let mut candidate = empty_mgr();
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
let targets = build_tier_mutation_affected_targets(
TierMutationIntentKind::Add,
HashSet::from(["COLD-A".to_string()]),
&previous,
&candidate,
)
.expect("added tier should have a target");
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
.expect("intent should build")
.expect("add should require an intent");
intent.expires_at_unix_nanos = match case {
"unexpired" => tier_mutation_intent_expiry_unix_nanos(),
"recently_expired" => {
i64::try_from(OffsetDateTime::now_utc().unix_timestamp_nanos()).expect("test clock should fit")
- 1_000_000_000
}
_ => 1,
};
if case == "different_candidate" {
intent.candidate_digest = [0x51; 32];
}
if case != "missing_config" {
candidate
.save_tiering_config_if_current(store.clone(), None)
.await
.expect("candidate should persist");
}
save_tier_mutation_intent_record(store.clone(), &intent)
.await
.expect("peer prepare should persist");
let expected = if case == "concurrent_change" {
let mut changed = intent.clone();
changed.candidate_digest = [0x52; 32];
let object = tier_mutation_intent_record_object_name(intent.mutation_id).expect("record path should build");
store
.rewrite_on_next_if_match(object, changed.encode().expect("changed intent should encode"))
.await;
changed
} else {
intent
};
let manager = TierConfigMgr::new();
TIER_MUTATION_TEST_PEERS
.scope(Vec::new(), async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect("unproven prepare should remain fenced");
})
.await;
let error = match TierConfigMgr::acquire_operation_lease(&manager, "COLD-A").await {
Ok(_) => panic!("{case}: unproven prepare must block reads"),
Err(error) => error,
};
assert!(error.message.contains("being replaced"), "{case}: {error}");
let records = TierConfigMgr::load_tier_mutation_intents(store)
.await
.expect("peer scan should succeed");
assert_eq!(records, vec![expected], "{case}: prepared evidence must remain unchanged");
}
}
#[tokio::test]
async fn peer_only_prepare_recovery_retains_fence_until_peer_commit_succeeds() {
use crate::services::tier::tier_mutation_intent::save_tier_mutation_intent_record;
let store = Arc::new(CasConfigStore::default());
let previous = empty_mgr();
let mut candidate = empty_mgr();
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
let targets = build_tier_mutation_affected_targets(
TierMutationIntentKind::Add,
HashSet::from(["COLD-A".to_string()]),
&previous,
&candidate,
)
.expect("added tier should have a target");
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
.expect("intent should build")
.expect("add should require an intent");
intent.expires_at_unix_nanos = 1;
candidate
.save_tiering_config_if_current(store.clone(), None)
.await
.expect("candidate should persist");
save_tier_mutation_intent_record(store.clone(), &intent)
.await
.expect("peer prepare should persist");
let manager = TierConfigMgr::new();
let calls = Arc::new(Mutex::new(Vec::new()));
TIER_MUTATION_TEST_PEERS
.scope(vec![FakeTierMutationPeer::boxed("peer-a", calls.clone(), Err("offline"))], async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect_err("failed peer commit must leave recovery incomplete");
})
.await;
let retained = TierConfigMgr::load_tier_mutation_intents(store.clone())
.await
.expect("peer scan should succeed");
assert_eq!(retained.len(), 1);
assert_eq!(retained[0].state, TierMutationIntentState::Committed);
let error = match TierConfigMgr::acquire_operation_lease(&manager, "COLD-A").await {
Ok(_) => panic!("failed peer commit must retain the fence"),
Err(error) => error,
};
assert!(error.message.contains("being replaced"), "{error}");
TIER_MUTATION_TEST_PEERS
.scope(
vec![FakeTierMutationPeer::boxed(
"peer-a",
calls.clone(),
Ok(PeerTierMutationState::Committed),
)],
async {
TIER_DRIVER_TEST_FACTORY
.scope(healthy_driver_factory(), async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect("peer commit retry should recover");
TierConfigMgr::acquire_operation_lease(&manager, "COLD-A")
.await
.expect("tier should be readable after the retry");
})
.await;
},
)
.await;
assert_eq!(lock_unpoisoned(&calls).len(), 2, "recovery should retry the peer commit");
}
#[tokio::test]
async fn coordinator_mutation_recovery_scan_blocks_new_operation_lease() {
let store = Arc::new(CasConfigStore::default());
@@ -139,6 +139,7 @@ impl TierType {
}
pub(crate) const TIER_CREDENTIAL_REDACTED: &str = "REDACTED";
pub(crate) const AZURE_BLOB_TIER_VERSION: &str = "rustfs-azure-blob-v1";
#[derive(Default, Serialize, Deserialize)]
#[serde(default)]
@@ -178,6 +179,10 @@ impl Clone for TierConfig {
}
impl TierConfig {
pub(crate) fn is_native_azure(&self) -> bool {
self.tier_type == TierType::Azure && self.version == AZURE_BLOB_TIER_VERSION
}
pub(crate) fn redacted(&self) -> Self {
let mut redacted = Self {
version: self.version.clone(),
@@ -127,6 +127,10 @@ impl TierMutationIntentTarget {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub(crate) struct TierMutationIntent {
// Old peers deny unknown intent fields before installing a prepare block.
// Omit this extension for existing drivers to retain their wire checksum.
#[serde(default, skip_serializing_if = "is_false")]
pub requires_native_azure: bool,
pub mutation_id: Uuid,
pub revision: u64,
pub kind: TierMutationIntentKind,
@@ -282,7 +286,8 @@ impl TierMutationIntent {
}
pub(crate) fn same_identity_as(&self, other: &Self) -> bool {
self.mutation_id == other.mutation_id
self.requires_native_azure == other.requires_native_azure
&& self.mutation_id == other.mutation_id
&& self.kind == other.kind
&& self.old_config_etag == other.old_config_etag
&& self.candidate_digest == other.candidate_digest
@@ -825,6 +830,10 @@ fn identity_is_empty(identity: &TierDestinationId) -> bool {
identity.iter().all(|byte| *byte == 0)
}
fn is_false(value: &bool) -> bool {
!value
}
fn digest_is_empty(digest: &TierMutationDigest) -> bool {
digest.iter().all(|byte| *byte == 0)
}
@@ -840,6 +849,7 @@ mod tests {
fn prepared_intent() -> TierMutationIntent {
TierMutationIntent {
requires_native_azure: false,
mutation_id: Uuid::new_v4(),
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -867,6 +877,49 @@ mod tests {
assert_eq!(shard, tier_mutation_mutex_shard_index(mutation_id));
}
#[test]
fn native_azure_requirement_is_identity_bound_and_rejected_by_the_previous_wire_reader() {
// Preserve the previous strict wire shape as an independent reader.
#[derive(Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
struct LegacyIntent {
mutation_id: Uuid,
revision: u64,
kind: TierMutationIntentKind,
state: TierMutationIntentState,
#[serde(default, skip_serializing_if = "Option::is_none")]
old_config_etag: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
committed_config_etag: Option<String>,
candidate_digest: TierMutationDigest,
affected_targets: Vec<TierMutationIntentTarget>,
expires_at_unix_nanos: i64,
}
let legacy = prepared_intent();
let legacy_bytes = serde_json::to_vec(&legacy).expect("legacy intent should encode");
let old_reader: LegacyIntent =
serde_json::from_slice(&legacy_bytes).expect("ordinary mutations remain readable by old peers");
assert_eq!(
serde_json::to_vec(&old_reader).expect("old encoding"),
legacy_bytes,
"ordinary mutation checksums must stay unchanged"
);
let mut native = legacy.clone();
native.requires_native_azure = true;
let native_bytes = serde_json::to_vec(&native).expect("native intent should encode");
assert!(
serde_json::from_slice::<LegacyIntent>(&native_bytes).is_err(),
"an old peer must reject native Azure before prepare"
);
assert!(!legacy.same_identity_as(&native), "an abort cannot drop the native-driver requirement");
let encoded = native.encode().expect("native durable intent should encode");
assert_eq!(
TierMutationIntent::decode(native.mutation_id, &encoded).expect("new peer should decode"),
native
);
}
#[tokio::test]
async fn mutation_mutex_serializes_the_same_id() {
let mutation_id = Uuid::new_v4();
@@ -25,6 +25,7 @@ use crate::services::tier::{
tier_handlers::{ERR_TIER_BUCKET_NOT_FOUND, ERR_TIER_INVALID_CREDENTIALS, ERR_TIER_NOT_FOUND, ERR_TIER_PERM_ERR},
warm_backend_aliyun::WarmBackendAliyun,
warm_backend_azure::WarmBackendAzure,
warm_backend_azure_blob::{AzureBlobError, WarmBackendAzureBlob},
warm_backend_huaweicloud::WarmBackendHuaweicloud,
warm_backend_minio::WarmBackendMinIO,
warm_backend_r2::WarmBackendR2,
@@ -309,10 +310,10 @@ pub fn build_transition_put_options(storage_class: String, mut metadata: HashMap
/// Connection parameters every S3-compatible warm backend provider supplies.
///
/// The Aliyun, Azure, Huaweicloud, Tencent, MinIO, R2, and RustFS backends all
/// The Aliyun, legacy Azure, Huaweicloud, Tencent, MinIO, R2, and RustFS backends all
/// wrap [`WarmBackendS3`] around a statically-credentialed [`TransitionClient`]
/// built from exactly these values. `bucket_lookup` is a parameter rather than a
/// constant because the providers split into two families: Aliyun, Azure,
/// constant because the providers split into two families: Aliyun, legacy Azure,
/// Huaweicloud, and Tencent pin [`BucketLookupType::BucketLookupDNS`], while
/// MinIO, R2, and RustFS leave it at [`BucketLookupType::BucketLookupAuto`].
pub(crate) struct S3CompatibleWarmBackendParams<'a> {
@@ -673,6 +674,9 @@ fn probe_cleanup_incomplete_error() -> AdminError {
}
fn is_remote_tier_auth_rejection(err: &std::io::Error) -> bool {
if AzureBlobError::from_io(err).is_some() && err.kind() == std::io::ErrorKind::PermissionDenied {
return true;
}
let Some(response) = err.get_ref().and_then(|err| err.downcast_ref::<ErrorResponse>()) else {
return false;
};
@@ -706,10 +710,27 @@ async fn check_warm_backend_with_deadlines(
Ok(Err(err)) if is_remote_tier_auth_rejection(&err) => {
return Err(ERR_TIER_INVALID_CREDENTIALS.clone());
}
Ok(Err(_)) => {
Ok(Err(err)) => {
if let Some(rejection) = AzureBlobError::from_io(&err).filter(|error| error.rejects_write()) {
let mut admin_err = ERR_TIER_INVALID_CONFIG.clone();
admin_err.message = rejection.to_string();
return Err(admin_err);
}
return Err(match compensate_uncertain_probe_put(w, &probe_object, cleanup_deadline).await {
Ok(()) => ERR_TIER_PERM_ERR.clone(),
Err(_) => probe_cleanup_incomplete_error(),
Ok(()) => {
let mut admin_err = ERR_TIER_PERM_ERR.clone();
if let Some(original) = AzureBlobError::from_io(&err) {
admin_err.message = original.to_string();
}
admin_err
}
Err(_) => {
let mut admin_err = probe_cleanup_incomplete_error();
if let Some(original) = AzureBlobError::from_io(&err) {
admin_err.message.push_str(&format!(": {original}"));
}
admin_err
}
});
}
Err(_) => {
@@ -951,24 +972,23 @@ pub async fn new_warm_backend(tier: &TierConfig, probe: bool) -> Result<WarmBack
}
}
TierType::Azure => {
if let Some(azure_config) = tier.azure.as_ref() {
let dd = WarmBackendAzure::new(azure_config, &tier.name).await;
if let Err(err) = dd {
warn!("{}", err);
return Err(AdminError {
code: "XRustFSAdminTierInvalidConfig".to_string(),
message: format!("Unable to setup remote tier, check tier configuration: {err}"),
status_code: StatusCode::BAD_REQUEST,
});
}
d = Some(Box::new(dd.expect("Failed to create Azure backend")));
let azure = tier.azure.as_ref().ok_or_else(|| AdminError {
code: "XRustFSAdminTierInvalidConfig".to_string(),
message: "Azure tier configuration not found".to_string(),
status_code: StatusCode::BAD_REQUEST,
})?;
let driver = if tier.is_native_azure() {
WarmBackendAzureBlob::new(azure).map(|driver| -> WarmBackendImpl { Box::new(driver) })
} else {
return Err(AdminError {
code: "XRustFSAdminTierInvalidConfig".to_string(),
message: "Azure tier configuration not found".to_string(),
status_code: StatusCode::BAD_REQUEST,
});
}
WarmBackendAzure::new(azure, &tier.name)
.await
.map(|driver| -> WarmBackendImpl { Box::new(driver) })
};
d = Some(driver.map_err(|err| AdminError {
code: "XRustFSAdminTierInvalidConfig".to_string(),
message: format!("Unable to setup remote tier, check tier configuration: {err}"),
status_code: StatusCode::BAD_REQUEST,
})?);
}
#[cfg(not(feature = "gcs"))]
TierType::GCS => {
@@ -1042,6 +1062,13 @@ pub(crate) async fn new_transition_candidate_reconciler(
tier: &TierConfig,
) -> Result<Option<Box<dyn TransitionCandidateReconciler + Send + Sync + 'static>>, AdminError> {
let reconciler: Box<dyn TransitionCandidateReconciler + Send + Sync + 'static> = match tier.tier_type {
TierType::Azure if tier.is_native_azure() => Box::new(
WarmBackendAzureBlob::new(tier.azure.as_ref().ok_or_else(|| ERR_TIER_INVALID_CONFIG.clone())?).map_err(|err| {
let mut admin_err = ERR_TIER_INVALID_CONFIG.clone();
admin_err.message = err.to_string();
admin_err
})?,
),
TierType::S3 => Box::new(
WarmBackendS3::new(tier.s3.as_ref().ok_or_else(|| ERR_TIER_INVALID_CONFIG.clone())?, &tier.name)
.await
@@ -88,7 +88,7 @@ impl WarmBackend for WarmBackendAzure {
}
async fn probe_transition_candidate(&self, object: &str) -> Result<TransitionCandidateProbe, std::io::Error> {
// Azure currently uses the shared S3/SigV4 transport, but its normal
// Legacy Azure uses the shared S3/SigV4 transport, but its normal
// object path cannot persist exact remote versions across mixed
// RustFS releases. The mutation probe may still detect and precisely
// remove a versioned test object before rejecting that configuration.
@@ -0,0 +1,881 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Native Azure Blob tier transport, including Azure Stack Hub 2301 and later.
//!
//! Writes use one conditional commit and never replay a failed HTTP request.
//! Larger bodies are staged in bounded blocks with a unique upload namespace;
//! an interrupted staging operation leaves only Azure's expiring uncommitted
//! blocks. Lifecycle uploads use the existing transaction journal after a lost
//! commit acknowledgement; validation probes retain in-process compensation.
//! Versioned GET/DELETE address the persisted version. Current-root cleanup
//! may demote that matching version under a strong ETag before exact deletion.
//! Unversioned DELETE also requires a strong ETag to refuse replacement.
use super::tier_config::TierAzure;
use super::warm_backend::{
TransitionCandidateIdentity, TransitionCandidateProbe, TransitionCandidateReconciler, WarmBackend, WarmBackendGetOpts,
build_transition_put_options, transition_client_timeouts_from_env,
};
use super::warm_backend_s3::transition_candidate_metadata_matches;
use crate::bucket::lifecycle::transition_transaction::canonical_transition_remote_object;
use bytes::Bytes;
use futures_util::stream;
use http::{HeaderMap, HeaderName, HeaderValue, Method, StatusCode};
use quick_xml::{Reader, events::Event};
use rustfs_s3_client::transition_api::{ReadCloser, ReaderImpl};
use rustfs_signer::azure_shared_key_signature;
use rustfs_utils::crypto::hex;
use rustfs_utils::egress::validate_outbound_url;
use rustfs_utils::hash::FastMd5;
use rustfs_utils::http::metadata_compat::{
MINIO_INTERNAL_PREFIX, SUFFIX_TRANSITION_TIER_DESTINATION_ID, SUFFIX_TRANSITION_TRANSACTION_ID, internal_key_rustfs,
};
use std::collections::HashMap;
use std::io::{self, Cursor, ErrorKind};
use std::time::Duration;
use time::OffsetDateTime;
use tokio::io::{AsyncRead, AsyncReadExt};
use url::Url;
const API_VERSION: &str = "2020-10-02";
const MIN_BLOCK_BYTES: u64 = 16 * 1024 * 1024;
const MAX_BLOCK_BYTES: u64 = 100 * 1024 * 1024;
const MAX_BLOCKS: u64 = 50_000;
const MAX_CONTROL_BYTES: usize = 64 * 1024;
const MAX_METADATA_BYTES: usize = 8 * 1024;
const METADATA_PREFIX: &str = "x-ms-meta-rustfs_";
const HTTP_DATE: &[time::format_description::BorrowedFormatItem<'static>] =
time::macros::format_description!("[weekday repr:short], [day] [month repr:short] [year] [hour]:[minute]:[second] GMT");
/// A bounded diagnostic that never contains credentials, URLs or response bodies.
#[derive(Debug, thiserror::Error)]
#[error("Azure Blob {operation} failed (HTTP {status}, code {code}, request ID {request_id})")]
pub(crate) struct AzureBlobError {
operation: &'static str,
status: StatusCode,
code: String,
request_id: String,
}
impl AzureBlobError {
pub(super) fn from_io(error: &io::Error) -> Option<&Self> {
error.get_ref()?.downcast_ref()
}
pub(super) fn rejects_write(&self) -> bool {
matches!(
self.status,
StatusCode::BAD_REQUEST | StatusCode::NOT_FOUND | StatusCode::CONFLICT | StatusCode::PRECONDITION_FAILED
)
}
}
pub struct WarmBackendAzureBlob {
client: reqwest::Client,
endpoint: Url,
account: String,
key: Vec<u8>,
container: String,
prefix: String,
request_timeout: Duration,
}
impl WarmBackendAzureBlob {
/// Build a native Blob backend without performing remote I/O.
///
/// # Errors
///
/// Rejects unsupported credentials or storage classes, an invalid account,
/// container, prefix, endpoint or account key, and TLS/client setup failures.
pub fn new(conf: &TierAzure) -> io::Result<Self> {
validate_config(conf)?;
let endpoint = if conf.endpoint.is_empty() {
format!("https://{}.blob.core.windows.net", conf.access_key)
} else {
conf.endpoint.clone()
};
let endpoint = Url::parse(&endpoint).map_err(|_| invalid_input("Azure Blob endpoint is not a valid URL"))?;
if !matches!(endpoint.scheme(), "https" | "http")
|| endpoint.host_str().is_none()
|| !endpoint.username().is_empty()
|| endpoint.password().is_some()
|| !matches!(endpoint.path(), "" | "/")
|| endpoint.query().is_some()
|| endpoint.fragment().is_some()
{
return Err(invalid_input(
"Azure Blob endpoint must be an HTTP(S) origin without credentials, path or query",
));
}
validate_outbound_url(&endpoint).map_err(|_| invalid_input("Azure Blob endpoint is not allowed"))?;
let key = decode_key(&conf.secret_key)?;
rustfs_tls_runtime::ensure_crypto_provider().map_err(io::Error::other)?;
let timeouts = transition_client_timeouts_from_env();
let client = reqwest::Client::builder()
.connect_timeout(timeouts.connect_timeout)
.read_timeout(timeouts.response_body_idle_timeout)
.redirect(reqwest::redirect::Policy::none())
.retry(reqwest::retry::never())
.no_gzip()
.no_brotli()
.no_zstd()
.no_deflate()
.build()
.map_err(transport_error)?;
Ok(Self {
client,
endpoint,
account: conf.access_key.clone(),
key,
container: conf.bucket.clone(),
prefix: conf.prefix.strip_suffix('/').unwrap_or(&conf.prefix).to_string(),
request_timeout: timeouts.request_timeout,
})
}
fn object_url(&self, object: &str, version: &str) -> io::Result<Url> {
validate_version(version)?;
let name = if self.prefix.is_empty() {
object.to_string()
} else {
format!("{}/{object}", self.prefix)
};
validate_object_name(&name)?;
let mut url = self.endpoint.clone();
{
let mut segments = url
.path_segments_mut()
.map_err(|_| invalid_input("Azure Blob endpoint has no path"))?;
segments.clear().push(&self.container);
for segment in name.split('/') {
segments.push(segment);
}
}
if !version.is_empty() {
url.query_pairs_mut().append_pair("versionid", version);
}
Ok(url)
}
fn container_url(&self) -> io::Result<Url> {
let mut url = self.endpoint.clone();
url.path_segments_mut()
.map_err(|_| invalid_input("Azure Blob endpoint has no path"))?
.clear()
.push(&self.container);
Ok(url)
}
async fn send(&self, method: Method, url: Url, mut headers: HeaderMap, body: Option<Bytes>) -> io::Result<reqwest::Response> {
if let Some(body) = &body {
insert_header(&mut headers, "content-length", &body.len().to_string())?;
let bytes = body.clone();
let digest = tokio::task::spawn_blocking(move || FastMd5::digest(&bytes))
.await
.map_err(|_| io::Error::other("Azure Blob content checksum task failed"))?;
insert_header(&mut headers, "content-md5", &base64_simd::STANDARD.encode_to_string(digest))?;
}
headers.insert("x-ms-version", HeaderValue::from_static(API_VERSION));
headers.insert("accept-encoding", HeaderValue::from_static("identity"));
let date = OffsetDateTime::now_utc()
.format(HTTP_DATE)
.map_err(|_| io::Error::other("Azure Blob request date is not representable"))?;
insert_header(&mut headers, "x-ms-date", &date)?;
let signature = azure_shared_key_signature(&self.key, &self.account, method.as_str(), &url, &headers)
.map_err(|err| io::Error::new(ErrorKind::InvalidInput, err))?;
insert_header(&mut headers, "authorization", &format!("SharedKey {}:{signature}", self.account))?;
let mut request = reqwest::Request::new(method, url);
*request.headers_mut() = headers;
*request.body_mut() = body.map(reqwest::Body::from);
tokio::time::timeout(self.request_timeout, self.client.execute(request))
.await
.map_err(|_| io::Error::new(ErrorKind::TimedOut, "Azure Blob request timed out"))?
.map_err(transport_error)
}
async fn expect_status(
&self,
response: reqwest::Response,
expected: StatusCode,
operation: &'static str,
) -> io::Result<reqwest::Response> {
if response.status() == expected {
return Ok(response);
}
Err(service_error(response, operation, self.request_timeout).await)
}
async fn head(&self, object: &str, version: &str) -> io::Result<Option<HeaderMap>> {
let response = self
.send(Method::HEAD, self.object_url(object, version)?, HeaderMap::new(), None)
.await?;
if response.status() == StatusCode::OK {
if !version.is_empty() {
validate_response_version(response.headers(), version)?;
}
return Ok(Some(response.headers().clone()));
}
let err = service_error(response, "HEAD", self.request_timeout).await;
if err.kind() == ErrorKind::NotFound {
return Ok(None);
}
Err(err)
}
async fn delete(&self, object: &str, version: &str, headers: HeaderMap) -> io::Result<()> {
let response = self
.send(Method::DELETE, self.object_url(object, version)?, headers, None)
.await?;
if response.status() == StatusCode::ACCEPTED {
return Ok(());
}
let error = service_error(response, "DELETE", self.request_timeout).await;
if error.kind() == ErrorKind::NotFound {
return Ok(());
}
Err(error)
}
async fn write(
&self,
object: &str,
reader: ReaderImpl,
length: i64,
metadata: HashMap<String, String>,
) -> io::Result<String> {
let size = u64::try_from(length).map_err(|_| invalid_input("Azure Blob upload requires a known, non-negative length"))?;
let block_size = block_size(size)?;
let url = self.object_url(object, "")?;
let mut headers = metadata_headers(metadata)?;
headers.insert("if-none-match", HeaderValue::from_static("*"));
let mut reader: Box<dyn AsyncRead + Send + Sync + Unpin> = match reader {
ReaderImpl::Body(bytes) => Box::new(Cursor::new(bytes)),
ReaderImpl::ObjectBody(reader) => Box::new(reader),
};
if size <= block_size {
let body = read_part(&mut reader, size).await?;
ensure_eof(&mut reader).await?;
headers.insert("x-ms-blob-type", HeaderValue::from_static("BlockBlob"));
let response = self.send(Method::PUT, url, headers, Some(body)).await?;
let response = self.expect_status(response, StatusCode::CREATED, "PUT").await?;
return response_version(response.headers());
}
let upload = uuid::Uuid::new_v4().simple().to_string();
let mut blocks = Vec::new();
let mut remaining = size;
while remaining > 0 {
let count = remaining.min(block_size);
let body = read_part(&mut reader, count).await?;
let block_id = base64_simd::STANDARD.encode_to_string(format!("{upload}-{:05}", blocks.len()));
let mut block_url = url.clone();
block_url
.query_pairs_mut()
.append_pair("comp", "block")
.append_pair("blockid", &block_id);
let response = self.send(Method::PUT, block_url, HeaderMap::new(), Some(body)).await?;
self.expect_status(response, StatusCode::CREATED, "PUT block").await?;
blocks.push(block_id);
remaining -= count;
}
ensure_eof(&mut reader).await?;
let mut xml = String::from("<?xml version=\"1.0\" encoding=\"utf-8\"?><BlockList>");
for block in blocks {
// IDs are generated base64, so they contain no XML delimiters.
xml.push_str("<Uncommitted>");
xml.push_str(&block);
xml.push_str("</Uncommitted>");
}
xml.push_str("</BlockList>");
let mut commit_url = url;
commit_url.query_pairs_mut().append_pair("comp", "blocklist");
headers.insert("content-type", HeaderValue::from_static("application/xml"));
let response = self.send(Method::PUT, commit_url, headers, Some(Bytes::from(xml))).await?;
let response = self.expect_status(response, StatusCode::CREATED, "PUT block list").await?;
response_version(response.headers())
}
}
#[async_trait::async_trait]
impl WarmBackend for WarmBackendAzureBlob {
fn validate_remote_version_id(&self, version: &str) -> io::Result<()> {
validate_version(version)
}
async fn put(&self, object: &str, reader: ReaderImpl, length: i64) -> io::Result<String> {
self.write(object, reader, length, HashMap::new()).await
}
async fn put_with_meta(
&self,
object: &str,
reader: ReaderImpl,
length: i64,
metadata: HashMap<String, String>,
) -> io::Result<String> {
self.write(object, reader, length, metadata).await
}
async fn get(&self, object: &str, version: &str, opts: WarmBackendGetOpts) -> io::Result<ReadCloser> {
let range = requested_range(&opts)?;
let mut headers = HeaderMap::new();
if let Some((start, end)) = range {
let value = end.map_or_else(|| format!("bytes={start}-"), |end| format!("bytes={start}-{end}"));
insert_header(&mut headers, "range", &value)?;
}
let response = self
.send(Method::GET, self.object_url(object, version)?, headers, None)
.await?;
let expected = if range.is_some() {
StatusCode::PARTIAL_CONTENT
} else {
StatusCode::OK
};
let response = self.expect_status(response, expected, "GET").await?;
validate_response_version(response.headers(), version)?;
let length = response
.headers()
.get("content-length")
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse::<u64>().ok())
.ok_or_else(|| invalid_data("Azure Blob GET omitted a valid Content-Length"))?;
if let Some(range) = range {
validate_content_range(response.headers(), range, length)?;
}
let body = stream::try_unfold((response, length), |(mut response, remaining)| async move {
loop {
match response.chunk().await.map_err(transport_error)? {
Some(chunk) if chunk.is_empty() => continue,
Some(chunk) => {
let count =
u64::try_from(chunk.len()).map_err(|_| invalid_data("Azure Blob response length overflow"))?;
let remaining = remaining
.checked_sub(count)
.ok_or_else(|| invalid_data("Azure Blob returned more bytes than declared"))?;
return Ok::<_, io::Error>(Some((chunk, (response, remaining))));
}
None if remaining == 0 => return Ok(None),
None => {
return Err(io::Error::new(
ErrorKind::UnexpectedEof,
"Azure Blob response ended before its declared length",
));
}
}
}
});
Ok(ReadCloser::new(tokio_util::io::StreamReader::new(Box::pin(body))))
}
async fn remove(&self, object: &str, version: &str) -> io::Result<()> {
let mut headers = HeaderMap::new();
if version.is_empty() {
let Some(current) = self.head(object, "").await? else {
return Ok(());
};
if !response_version(&current)?.is_empty() {
return Err(invalid_data("Azure Blob became versioned; refusing an unversioned delete"));
}
headers = delete_preconditions(&current)?;
}
let Err(error) = self.delete(object, version, headers).await else {
return Ok(());
};
if version.is_empty()
|| !AzureBlobError::from_io(&error).is_some_and(|azure| {
matches!(azure.status, StatusCode::FORBIDDEN | StatusCode::CONFLICT)
&& azure.code == "OperationNotAllowedOnRootBlob"
})
{
return Err(error);
}
// Azure may require the current version to become a previous version
// before exact deletion. Only its matching generation can be demoted.
if let Some(current) = self.head(object, "").await? {
validate_response_version(&current, version)?;
self.delete(object, "", delete_preconditions(&current)?).await?;
}
// A successful base delete alone does not complete version cleanup.
// An error or lost acknowledgement remains retryable by the caller.
self.delete(object, version, HeaderMap::new()).await
}
async fn probe_transition_candidate(&self, object: &str) -> io::Result<TransitionCandidateProbe> {
Ok(match self.head(object, "").await? {
None => TransitionCandidateProbe::Missing,
Some(headers) => classify_head(&headers)?,
})
}
async fn probe_transition_version(&self, object: &str, version: &str) -> io::Result<TransitionCandidateProbe> {
if version.is_empty() {
return Err(invalid_input("Azure Blob exact probe requires a version ID"));
}
Ok(match self.head(object, version).await? {
None => TransitionCandidateProbe::Missing,
Some(_) => TransitionCandidateProbe::VersionedPresent(version.to_string()),
})
}
async fn in_use(&self) -> io::Result<bool> {
let mut url = self.container_url()?;
let prefix = if self.prefix.is_empty() {
String::new()
} else {
format!("{}/", self.prefix)
};
url.query_pairs_mut()
.append_pair("restype", "container")
.append_pair("comp", "list")
.append_pair("prefix", &prefix)
.append_pair("maxresults", "1");
let response = self.send(Method::GET, url, HeaderMap::new(), None).await?;
let response = self.expect_status(response, StatusCode::OK, "LIST").await?;
list_has_objects(&read_control_body(response, self.request_timeout).await?)
}
}
#[async_trait::async_trait]
impl TransitionCandidateReconciler for WarmBackendAzureBlob {
async fn probe_transition_candidate_for(
&self,
object: &str,
identity: TransitionCandidateIdentity,
) -> io::Result<TransitionCandidateProbe> {
let Some(headers) = self.head(object, "").await? else {
return Ok(TransitionCandidateProbe::Missing);
};
let mut metadata = HashMap::new();
for suffix in [SUFFIX_TRANSITION_TRANSACTION_ID, SUFFIX_TRANSITION_TIER_DESTINATION_ID] {
for key in [internal_key_rustfs(suffix), format!("{MINIO_INTERNAL_PREFIX}{suffix}")] {
if let Some(value) = headers.get(metadata_header_name(&key)) {
let decoded = base64_simd::STANDARD
.decode_to_vec(value.as_bytes())
.map_err(|_| invalid_data("Azure Blob candidate metadata is not valid base64"))?;
let value =
String::from_utf8(decoded).map_err(|_| invalid_data("Azure Blob candidate metadata is not UTF-8"))?;
metadata.insert(key, value);
}
}
}
if !transition_candidate_metadata_matches(&metadata, identity)? {
return Ok(TransitionCandidateProbe::Ambiguous);
}
classify_head(&headers)
}
}
fn validate_config(conf: &TierAzure) -> io::Result<()> {
if !(3..=24).contains(&conf.access_key.len())
|| !conf
.access_key
.bytes()
.all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit())
{
return Err(invalid_input("Azure Blob account name must contain 3 to 24 lowercase letters or digits"));
}
let container = conf.bucket.as_bytes();
if !(3..=63).contains(&container.len())
|| !container
.iter()
.all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || *byte == b'-')
|| container.first() == Some(&b'-')
|| container.last() == Some(&b'-')
|| conf.bucket.contains("--")
{
return Err(invalid_input("Azure Blob container name is invalid"));
}
if !(conf.storage_class.is_empty() || conf.storage_class == "STANDARD")
|| !conf.sp_auth.tenant_id.is_empty()
|| !conf.sp_auth.client_id.is_empty()
|| !conf.sp_auth.client_secret.is_empty()
{
return Err(invalid_input(
"Azure Blob tiers currently require an account key and the default storage class",
));
}
let prefix = conf.prefix.strip_suffix('/').unwrap_or(&conf.prefix);
if !prefix.is_empty() {
validate_object_name(prefix)?;
// Canonical suffix length is independent of the bucket and UUID values.
let object = canonical_transition_remote_object(
uuid::Uuid::from_u128(1),
&conf.bucket,
uuid::Uuid::from_u128(2),
uuid::Uuid::from_u128(3),
)
.map_err(io::Error::other)?;
validate_object_name(&format!("{prefix}/{object}"))?;
}
Ok(())
}
fn decode_key(key: &str) -> io::Result<Vec<u8>> {
let key = base64_simd::STANDARD
.decode_to_vec(key.trim().as_bytes())
.map_err(|_| invalid_input("Azure Blob account key must be base64"))?;
if key.is_empty() {
return Err(invalid_input("Azure Blob account key must not be empty"));
}
Ok(key)
}
fn validate_object_name(name: &str) -> io::Result<()> {
if name.is_empty()
|| name.len() > 1760
|| name.chars().count() > 880
|| name.split('/').any(|segment| matches!(segment, "." | ".."))
{
return Err(invalid_input(
"Azure Blob tier name exceeds Stack Hub limits or contains a standalone dot segment",
));
}
Ok(())
}
fn validate_version(version: &str) -> io::Result<()> {
if version.len() > 1024 || version.bytes().any(|byte| !byte.is_ascii_graphic()) {
return Err(invalid_input("Azure Blob version ID is invalid"));
}
Ok(())
}
fn response_version(headers: &HeaderMap) -> io::Result<String> {
let Some(version) = headers.get("x-ms-version-id") else {
return Ok(String::new());
};
let version = version
.to_str()
.map_err(|_| invalid_data("Azure Blob version ID is not ASCII"))?;
validate_version(version)?;
Ok(version.to_string())
}
fn validate_response_version(headers: &HeaderMap, requested: &str) -> io::Result<()> {
if response_version(headers)? != requested {
return Err(invalid_data("Azure Blob did not return the requested exact version"));
}
Ok(())
}
fn delete_preconditions(current: &HeaderMap) -> io::Result<HeaderMap> {
let etag = current
.get("etag")
.ok_or_else(|| invalid_data("Azure Blob HEAD omitted the delete precondition"))?;
let bytes = etag.as_bytes();
// A wildcard, weak tag or list cannot identify a single matching generation.
if bytes.len() < 3
|| bytes.first() != Some(&b'"')
|| bytes.last() != Some(&b'"')
|| !bytes[1..bytes.len() - 1]
.iter()
.all(|byte| *byte == b'!' || (b'#'..=b'~').contains(byte))
{
return Err(invalid_data("Azure Blob HEAD omitted a single strong ETag"));
}
let mut headers = HeaderMap::new();
headers.insert("if-match", etag.clone());
Ok(headers)
}
fn classify_head(headers: &HeaderMap) -> io::Result<TransitionCandidateProbe> {
let version = response_version(headers)?;
Ok(if version.is_empty() {
TransitionCandidateProbe::UnversionedPresent
} else {
TransitionCandidateProbe::VersionedPresent(version)
})
}
fn block_size(size: u64) -> io::Result<u64> {
if size > MAX_BLOCK_BYTES * MAX_BLOCKS {
return Err(invalid_input("Azure Blob exceeds the Stack Hub block count or object size limit"));
}
Ok(size.div_ceil(MAX_BLOCKS).max(MIN_BLOCK_BYTES))
}
async fn read_part(reader: &mut (dyn AsyncRead + Send + Sync + Unpin), length: u64) -> io::Result<Bytes> {
let length = usize::try_from(length).map_err(|_| invalid_input("Azure Blob block length is not representable"))?;
let mut bytes = vec![0; length];
reader.read_exact(&mut bytes).await?;
Ok(Bytes::from(bytes))
}
async fn ensure_eof(reader: &mut (dyn AsyncRead + Send + Sync + Unpin)) -> io::Result<()> {
if reader.read(&mut [0_u8; 1]).await? != 0 {
return Err(invalid_input("Azure Blob upload body exceeds its declared length"));
}
Ok(())
}
fn metadata_header_name(key: &str) -> String {
format!("{METADATA_PREFIX}{}", hex(key.as_bytes()))
}
fn metadata_headers(metadata: HashMap<String, String>) -> io::Result<HeaderMap> {
let opts = build_transition_put_options(String::new(), metadata);
let mut headers = HeaderMap::new();
for (name, value) in [
("x-ms-blob-content-type", opts.content_type),
("x-ms-blob-content-encoding", opts.content_encoding),
("x-ms-blob-content-language", opts.content_language),
("x-ms-blob-content-disposition", opts.content_disposition),
("x-ms-blob-cache-control", opts.cache_control),
] {
if !value.is_empty() {
insert_header(&mut headers, name, &value)?;
}
}
let mut size = 0_usize;
for (key, value) in opts.user_metadata {
let key = key.to_ascii_lowercase();
let key = key.strip_prefix("x-amz-meta-").unwrap_or(&key);
let name = metadata_header_name(key);
let value = base64_simd::STANDARD.encode_to_string(value.as_bytes());
size = size.saturating_add(name.len()).saturating_add(value.len());
if size > MAX_METADATA_BYTES {
return Err(invalid_input("Azure Blob encoded metadata exceeds 8 KiB"));
}
if headers.contains_key(&name) {
return Err(invalid_input("Azure Blob metadata contains conflicting names"));
}
insert_header(&mut headers, &name, &value)?;
}
Ok(headers)
}
fn requested_range(opts: &WarmBackendGetOpts) -> io::Result<Option<(u64, Option<u64>)>> {
let start = u64::try_from(opts.start_offset).map_err(|_| invalid_input("Azure Blob range starts before zero"))?;
let length = u64::try_from(opts.length).map_err(|_| invalid_input("Azure Blob range length is negative"))?;
if length > 0 {
let end = start
.checked_add(length - 1)
.ok_or_else(|| invalid_input("Azure Blob range overflows"))?;
return Ok(Some((start, Some(end))));
}
Ok((start > 0).then_some((start, None)))
}
fn validate_content_range(headers: &HeaderMap, requested: (u64, Option<u64>), length: u64) -> io::Result<()> {
let value = headers
.get("content-range")
.and_then(|value| value.to_str().ok())
.unwrap_or_default();
let range = value
.strip_prefix("bytes ")
.and_then(|value| value.split_once('/'))
.and_then(|(range, total)| {
let (start, end) = range.split_once('-')?;
Some((start.parse::<u64>().ok()?, end.parse::<u64>().ok()?, total.parse::<u64>().ok()?))
});
let Some((start, end, total)) = range else {
return Err(invalid_data("Azure Blob returned an invalid Content-Range"));
};
if start != requested.0
|| end < start
|| end >= total
|| end != requested.1.unwrap_or(total - 1).min(total - 1)
|| end.checked_sub(start).and_then(|size| size.checked_add(1)) != Some(length)
{
return Err(invalid_data("Azure Blob returned a different range from the one requested"));
}
Ok(())
}
async fn read_control_body(mut response: reqwest::Response, timeout: Duration) -> io::Result<Vec<u8>> {
tokio::time::timeout(timeout, async move {
let mut body = Vec::new();
while let Some(chunk) = response.chunk().await.map_err(transport_error)? {
if body.len().saturating_add(chunk.len()) > MAX_CONTROL_BYTES {
return Err(invalid_data("Azure Blob control response exceeds 64 KiB"));
}
body.extend_from_slice(&chunk);
}
Ok(body)
})
.await
.map_err(|_| io::Error::new(ErrorKind::TimedOut, "Azure Blob control response timed out"))?
}
async fn service_error(response: reqwest::Response, operation: &'static str, timeout: Duration) -> io::Error {
let status = response.status();
let request_id = diagnostic_token(
response
.headers()
.get("x-ms-request-id")
.and_then(|value| value.to_str().ok()),
);
let mut code = diagnostic_token(
response
.headers()
.get("x-ms-error-code")
.and_then(|value| value.to_str().ok()),
);
if code == "unknown"
&& let Ok(body) = read_control_body(response, timeout).await
&& let Some(xml_code) = xml_error_code(&body)
{
code = diagnostic_token(Some(&xml_code));
}
let kind = if status == StatusCode::NOT_FOUND && code == "BlobNotFound" {
ErrorKind::NotFound
} else if matches!(status, StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN) {
ErrorKind::PermissionDenied
} else if status == StatusCode::PRECONDITION_FAILED {
ErrorKind::AlreadyExists
} else {
ErrorKind::Other
};
io::Error::new(
kind,
AzureBlobError {
operation,
status,
code,
request_id,
},
)
}
fn diagnostic_token(value: Option<&str>) -> String {
value
.filter(|value| {
!value.is_empty() && value.len() <= 128 && value.bytes().all(|byte| byte.is_ascii_alphanumeric() || byte == b'-')
})
.unwrap_or("unknown")
.to_string()
}
fn xml_error_code(body: &[u8]) -> Option<String> {
let mut reader = Reader::from_reader(body);
let mut depth = 0_u32;
let mut root = false;
let mut code = None;
loop {
match reader.read_event().ok()? {
Event::Start(start) => {
if depth == 0 {
if root || start.local_name().as_ref() != "Error" {
return None;
}
root = true;
}
if depth == 1 && start.local_name().as_ref() == "Code" {
if code.is_some() {
return None;
}
let text = reader.read_text(start.name()).ok()?;
code = Some(quick_xml::escape::unescape(text.as_ref()).ok()?.into_owned());
continue;
}
depth += 1;
if depth > 16 {
return None;
}
}
Event::End(_) => depth = depth.checked_sub(1)?,
Event::Text(text) if depth == 0 && !text.as_ref().trim().is_empty() => return None,
Event::CData(_) if depth == 0 => return None,
Event::Eof => return (root && depth == 0).then_some(code).flatten(),
Event::DocType(_) => return None,
_ => {}
}
}
}
fn list_has_objects(body: &[u8]) -> io::Result<bool> {
let mut reader = Reader::from_reader(body);
let mut depth = 0_u32;
let mut root = false;
let mut blobs = false;
let mut found = false;
loop {
let event = reader
.read_event()
.map_err(|_| invalid_data("Azure Blob listing is not valid XML"))?;
let empty = matches!(event, Event::Empty(_));
match event {
Event::Start(start) | Event::Empty(start) => {
if depth == 0 && (root || start.local_name().as_ref() != "EnumerationResults") {
return Err(invalid_data("Azure Blob listing has an unexpected root"));
}
root |= start.local_name().as_ref() == "EnumerationResults";
if start.local_name().as_ref() == "Blobs" {
if blobs || depth != 1 {
return Err(invalid_data("Azure Blob listing has an invalid Blobs element"));
}
blobs = true;
}
found |= matches!(start.local_name().as_ref(), "Blob" | "BlobPrefix");
if !empty && start.local_name().as_ref() == "NextMarker" {
let marker = reader
.read_text(start.name())
.map_err(|_| invalid_data("Azure Blob listing marker is invalid"))?;
found |= !marker.is_empty();
continue;
}
if !empty {
depth += 1;
if depth > 16 {
return Err(invalid_data("Azure Blob listing exceeds the XML depth limit"));
}
}
}
Event::End(_) => {
depth = depth
.checked_sub(1)
.ok_or_else(|| invalid_data("Azure Blob listing has an invalid closing tag"))?;
}
Event::DocType(_) => return Err(invalid_data("Azure Blob listing must not contain a document type")),
Event::Text(text) if depth == 0 && !text.as_ref().trim().is_empty() => {
return Err(invalid_data("Azure Blob listing contains text outside its root"));
}
Event::CData(_) if depth == 0 => return Err(invalid_data("Azure Blob listing contains text outside its root")),
Event::Eof if root && blobs && depth == 0 => return Ok(found),
Event::Eof => return Err(invalid_data("Azure Blob listing is incomplete")),
_ => {}
}
}
}
fn insert_header(headers: &mut HeaderMap, name: &str, value: &str) -> io::Result<()> {
let name =
HeaderName::from_bytes(name.as_bytes()).map_err(|_| invalid_input("Azure Blob metadata header name is invalid"))?;
let value = HeaderValue::from_str(value).map_err(|_| invalid_input("Azure Blob metadata header value is invalid"))?;
headers.insert(name, value);
Ok(())
}
fn invalid_input(message: &'static str) -> io::Error {
io::Error::new(ErrorKind::InvalidInput, message)
}
fn invalid_data(message: &'static str) -> io::Error {
io::Error::new(ErrorKind::InvalidData, message)
}
fn transport_error(err: reqwest::Error) -> io::Error {
let kind = if err.is_timeout() {
ErrorKind::TimedOut
} else {
ErrorKind::Other
};
io::Error::new(kind, err.without_url())
}
#[cfg(test)]
#[path = "warm_backend_azure_blob_tests.rs"]
mod tests;
@@ -0,0 +1,929 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::*;
use crate::services::tier::warm_backend::{WarmBackendImpl, check_warm_backend};
use rustfs_utils::http::metadata_compat::insert_str;
use tokio::io::AsyncWriteExt;
use tokio::net::{TcpListener, TcpStream};
use tokio::task::JoinHandle;
struct Reply {
status: StatusCode,
headers: HeaderMap,
body: Bytes,
disconnect: bool,
}
fn reply(status: StatusCode, headers: &[(&str, &str)], body: &[u8]) -> Reply {
let mut map = HeaderMap::new();
for (name, value) in headers {
insert_header(&mut map, name, value).expect("fixture headers should be valid");
}
Reply {
status,
headers: map,
body: Bytes::copy_from_slice(body),
disconnect: false,
}
}
fn present(version: Option<&str>) -> Reply {
let mut response = reply(StatusCode::OK, &[("content-length", "6"), ("etag", "\"original\"")], b"");
if let Some(version) = version {
insert_header(&mut response.headers, "x-ms-version-id", version).expect("fixture version should be valid");
}
response
}
fn missing() -> Reply {
reply(StatusCode::NOT_FOUND, &[("x-ms-error-code", "BlobNotFound")], b"")
}
#[derive(Debug)]
struct Captured {
method: String,
target: String,
headers: HeaderMap,
body: Vec<u8>,
}
async fn capture(stream: &mut TcpStream) -> Captured {
let mut bytes = Vec::new();
let boundary = loop {
let mut buffer = [0_u8; 4096];
let count = stream.read(&mut buffer).await.expect("fixture request should be readable");
assert_ne!(count, 0, "request must include a complete HTTP head");
bytes.extend_from_slice(&buffer[..count]);
if let Some(index) = bytes.windows(4).position(|window| window == b"\r\n\r\n") {
break index + 4;
}
assert!(bytes.len() <= 32 * 1024, "request headers must remain bounded");
};
let text = std::str::from_utf8(&bytes[..boundary]).expect("request headers should be ASCII");
let mut lines = text.split("\r\n");
let mut request_line = lines.next().expect("request line").split_whitespace();
let method = request_line.next().expect("request method").to_string();
let target = request_line.next().expect("request target").to_string();
let mut headers = HeaderMap::new();
for line in lines.filter(|line| !line.is_empty()) {
let (name, value) = line.split_once(':').expect("request header should include a colon");
insert_header(&mut headers, name, value.trim()).expect("request headers should parse");
}
let length = headers
.get("content-length")
.map(|value| {
value
.to_str()
.expect("length should be ASCII")
.parse::<usize>()
.expect("length should be numeric")
})
.unwrap_or_default();
assert!(length <= 17 * 1024 * 1024, "fixture only accepts its bounded test bodies");
let mut body = bytes[boundary..].to_vec();
let received = body.len();
body.resize(length, 0);
stream
.read_exact(&mut body[received..])
.await
.expect("request body should match its declared length");
Captured {
method,
target,
headers,
body,
}
}
async fn fixture(responses: Vec<Reply>) -> (WarmBackendAzureBlob, JoinHandle<Vec<Captured>>) {
let listener = TcpListener::bind("127.0.0.1:0").await.expect("fixture listener should bind");
let address = listener.local_addr().expect("fixture address");
let task = tokio::spawn(async move {
let mut requests = Vec::new();
for mut response in responses {
let (mut stream, _) = tokio::time::timeout(Duration::from_secs(10), listener.accept())
.await
.expect("expected request must arrive")
.expect("fixture should accept");
requests.push(capture(&mut stream).await);
if response.disconnect {
continue;
}
if !response.headers.contains_key("content-length") {
insert_header(&mut response.headers, "content-length", &response.body.len().to_string())
.expect("fixture length should be valid");
}
let mut wire = format!(
"HTTP/1.1 {} {}\r\nConnection: close\r\n",
response.status.as_u16(),
response.status.canonical_reason().unwrap_or("")
);
for (name, value) in &response.headers {
wire.push_str(&format!("{name}: {}\r\n", value.to_str().expect("fixture header is ASCII")));
}
wire.push_str("\r\n");
stream.write_all(wire.as_bytes()).await.expect("fixture should send headers");
stream.write_all(&response.body).await.expect("fixture should send body");
}
requests
});
rustfs_tls_runtime::ensure_crypto_provider().expect("fixture TLS provider should initialize");
let client = reqwest::Client::builder()
.no_proxy()
.redirect(reqwest::redirect::Policy::none())
.retry(reqwest::retry::never())
.read_timeout(Duration::from_secs(5))
.build()
.expect("fixture client should build");
(
WarmBackendAzureBlob {
client,
endpoint: Url::parse(&format!("http://{address}")).expect("fixture endpoint"),
account: "account".to_string(),
key: vec![7; 32],
container: "container".to_string(),
prefix: "cold".to_string(),
request_timeout: Duration::from_secs(5),
},
task,
)
}
#[tokio::test]
async fn mutation_probe_uses_blob_shared_key_and_cleans_up_with_an_etag() {
let responses = vec![
reply(StatusCode::CREATED, &[], b""),
present(None),
reply(StatusCode::PARTIAL_CONTENT, &[("content-range", "bytes 0-5/6")], b"RustFS"),
present(None),
reply(StatusCode::ACCEPTED, &[], b""),
missing(),
];
let (backend, task) = fixture(responses).await;
let backend: WarmBackendImpl = Box::new(backend);
check_warm_backend(Some(&backend))
.await
.expect("native Blob mutation probe should succeed");
let requests = task.await.expect("fixture should join");
assert_eq!(
requests.iter().map(|request| request.method.as_str()).collect::<Vec<_>>(),
["PUT", "HEAD", "GET", "HEAD", "DELETE", "HEAD"]
);
assert!(requests[0].target.starts_with("/container/cold/rustfs-tier-probe-"));
assert_eq!(requests[0].body, b"RustFS");
assert_eq!(requests[0].headers["x-ms-blob-type"], "BlockBlob");
assert_eq!(requests[0].headers["if-none-match"], "*");
assert_eq!(requests[4].headers["if-match"], "\"original\"");
for request in &requests {
assert_eq!(request.target, requests[0].target, "the whole probe must address the same blob");
assert_eq!(request.headers["x-ms-version"], API_VERSION);
assert!(
request.headers["authorization"]
.to_str()
.expect("authorization is ASCII")
.starts_with("SharedKey account:")
);
assert!(!request.headers.contains_key("x-amz-date"));
assert!(!request.target.contains("location"));
assert!(!request.target.contains("versionid"));
}
}
#[tokio::test]
async fn versioned_mutation_probe_cleans_up_after_forbidden_current_version_delete() {
let version = "2026-11-06T01:02:03.0000000Z";
let (backend, task) = fixture(vec![
reply(StatusCode::CREATED, &[("x-ms-version-id", version)], b""),
present(Some(version)),
reply(
StatusCode::PARTIAL_CONTENT,
&[("content-range", "bytes 0-5/6"), ("x-ms-version-id", version)],
b"RustFS",
),
reply(StatusCode::FORBIDDEN, &[], b"<Error><Code>OperationNotAllowedOnRootBlob</Code></Error>"),
present(Some(version)),
reply(StatusCode::ACCEPTED, &[], b""),
reply(StatusCode::ACCEPTED, &[], b""),
missing(),
])
.await;
let backend: WarmBackendImpl = Box::new(backend);
check_warm_backend(Some(&backend))
.await
.expect("a versioned Add/Verify probe must finish cleanup after the specific 403");
let requests = task.await.expect("fixture should join");
assert_eq!(
requests.iter().map(|request| request.method.as_str()).collect::<Vec<_>>(),
["PUT", "HEAD", "GET", "DELETE", "HEAD", "DELETE", "DELETE", "HEAD"]
);
let base = &requests[0].target;
assert!(base.starts_with("/container/cold/rustfs-tier-probe-"));
for index in [1, 4, 5, 7] {
assert_eq!(&requests[index].target, base, "base operations must retain the probe identity");
}
for index in [2, 3, 6] {
let url = Url::parse(&format!("http://fixture{}", requests[index].target)).expect("captured URL");
assert_eq!(url.path(), base);
assert_eq!(
url.query_pairs().collect::<Vec<_>>(),
[("versionid".into(), version.into())],
"exact reads and deletes must retain the timestamp version"
);
}
assert_eq!(requests[5].headers["if-match"], "\"original\"");
for request in &requests {
assert_eq!(request.headers["x-ms-version"], "2020-10-02");
}
}
#[tokio::test]
async fn rejected_put_preserves_azure_error_and_never_deletes_an_existing_blob() {
for (status, code) in [
(StatusCode::BAD_REQUEST, "InvalidHeaderValue"),
(StatusCode::PRECONDITION_FAILED, "ConditionNotMet"),
] {
let (backend, task) = fixture(vec![reply(
status,
&[("x-ms-error-code", code), ("x-ms-request-id", "abc-123")],
b"<Error><Code>PRIVATE-RESPONSE-BODY</Code></Error>",
)])
.await;
let backend: WarmBackendImpl = Box::new(backend);
let err = check_warm_backend(Some(&backend))
.await
.expect_err("deterministic rejected write must fail");
assert!(err.message.contains(code));
assert!(err.message.contains("abc-123"));
assert!(!err.message.contains("PRIVATE-RESPONSE-BODY"));
assert!(!err.message.contains("cleanup is incomplete"));
let requests = task.await.expect("fixture should join");
assert_eq!(requests.len(), 1, "a rejected conditional PUT must not trigger cleanup");
}
}
#[tokio::test]
async fn empty_version_headers_use_the_unversioned_read_and_delete_paths() {
let mut head = present(Some(""));
head.headers.insert("content-length", HeaderValue::from_static("0"));
let (backend, task) = fixture(vec![
reply(StatusCode::CREATED, &[("x-ms-version-id", "")], b""),
reply(StatusCode::OK, &[("x-ms-version-id", "")], b""),
head,
reply(StatusCode::ACCEPTED, &[], b""),
])
.await;
let version = backend
.put("empty", ReaderImpl::Body(Bytes::new()), 0)
.await
.expect("empty version is unversioned");
assert!(version.is_empty());
let mut reader = backend
.get("empty", &version, WarmBackendGetOpts::default())
.await
.expect("unversioned read");
let mut body = Vec::new();
reader.read_to_end(&mut body).await.expect("empty blob should read");
assert!(body.is_empty());
backend
.remove("empty", &version)
.await
.expect("unversioned conditional delete");
for request in task.await.expect("fixture should join") {
assert_eq!(request.target, "/container/cold/empty");
}
}
#[tokio::test]
async fn rejected_credentials_are_reported_without_an_uncertain_cleanup() {
let (backend, task) = fixture(vec![reply(
StatusCode::FORBIDDEN,
&[("x-ms-error-code", "AuthenticationFailed")],
b"",
)])
.await;
let backend: WarmBackendImpl = Box::new(backend);
let err = check_warm_backend(Some(&backend))
.await
.expect_err("invalid credentials must fail");
assert_eq!(err.code, crate::services::tier::tier_handlers::ERR_TIER_INVALID_CREDENTIALS.code);
assert_eq!(task.await.expect("fixture should join").len(), 1);
}
#[tokio::test]
async fn lost_put_response_is_not_replayed_and_its_candidate_is_cleaned() {
let mut lost = reply(StatusCode::CREATED, &[], b"");
lost.disconnect = true;
let (backend, task) = fixture(vec![
lost,
present(None),
present(None),
reply(StatusCode::ACCEPTED, &[], b""),
missing(),
missing(),
])
.await;
let backend: WarmBackendImpl = Box::new(backend);
let err = check_warm_backend(Some(&backend))
.await
.expect_err("a lost write acknowledgement must remain a failure");
assert!(!err.message.contains("cleanup is incomplete"));
let requests = task.await.expect("fixture should join");
assert_eq!(requests.iter().filter(|request| request.method == "PUT").count(), 1);
assert_eq!(requests.iter().filter(|request| request.method == "DELETE").count(), 1);
}
#[tokio::test]
async fn block_upload_commits_only_the_complete_declared_body() {
let (backend, task) = fixture(vec![
reply(StatusCode::CREATED, &[], b""),
reply(StatusCode::CREATED, &[], b""),
reply(StatusCode::CREATED, &[], b""),
])
.await;
let length = usize::try_from(MIN_BLOCK_BYTES + 5).expect("fixture size");
let bytes = Bytes::from(vec![42; length]);
assert_eq!(
backend
.put(
"large",
ReaderImpl::ObjectBody(ReadCloser::new(Cursor::new(bytes.clone()))),
i64::try_from(length).expect("fixture length")
)
.await
.expect("block upload"),
""
);
let requests = task.await.expect("fixture should join");
assert_eq!(requests.len(), 3);
let mut block_ids = Vec::new();
let mut body = Vec::new();
for request in &requests[..2] {
let url = Url::parse(&format!("http://fixture{}", request.target)).expect("captured URL");
let query: HashMap<_, _> = url.query_pairs().into_owned().collect();
assert_eq!(query["comp"], "block");
block_ids.push(query["blockid"].clone());
assert!(request.headers.contains_key("content-md5"));
body.extend_from_slice(&request.body);
}
assert_eq!(body, bytes);
assert_ne!(block_ids[0], block_ids[1]);
assert!(requests[2].target.ends_with("?comp=blocklist"));
assert_eq!(requests[2].headers["if-none-match"], "*");
let expected = format!(
"<?xml version=\"1.0\" encoding=\"utf-8\"?><BlockList><Uncommitted>{}</Uncommitted><Uncommitted>{}</Uncommitted></BlockList>",
block_ids[0], block_ids[1],
);
assert_eq!(requests[2].body, expected.as_bytes());
}
#[tokio::test]
async fn server_error_remains_visible_after_successful_probe_compensation() {
let (backend, task) = fixture(vec![
reply(
StatusCode::INTERNAL_SERVER_ERROR,
&[("x-ms-error-code", "InternalError"), ("x-ms-request-id", "error-123")],
b"",
),
present(None),
present(None),
reply(StatusCode::ACCEPTED, &[], b""),
missing(),
missing(),
])
.await;
let backend: WarmBackendImpl = Box::new(backend);
let error = check_warm_backend(Some(&backend))
.await
.expect_err("failed upload remains an error after cleanup");
assert!(error.message.contains("500"));
assert!(error.message.contains("InternalError"));
assert!(error.message.contains("error-123"));
assert!(!error.message.contains("cleanup is incomplete"));
let requests = task.await.expect("fixture should join");
assert_eq!(requests.iter().filter(|request| request.method == "PUT").count(), 1);
assert_eq!(requests.iter().filter(|request| request.method == "DELETE").count(), 1);
}
#[tokio::test]
async fn empty_blob_and_declared_length_failures_are_handled_before_commit() {
let (backend, task) = fixture(vec![reply(StatusCode::CREATED, &[], b"")]).await;
backend
.put("empty", ReaderImpl::Body(Bytes::new()), 0)
.await
.expect("empty upload");
let requests = task.await.expect("fixture should join");
assert_eq!(requests[0].headers["content-length"], "0");
assert_eq!(requests[0].headers["content-md5"], "1B2M2Y8AsgTpgAmY7PhCfg==");
for (body, kind) in [
(b"short".as_slice(), ErrorKind::UnexpectedEof),
(b"too-long".as_slice(), ErrorKind::InvalidInput),
] {
let (backend, task) = fixture(vec![]).await;
let err = backend
.put("bad", ReaderImpl::Body(Bytes::copy_from_slice(body)), 6)
.await
.expect_err("length mismatch");
assert_eq!(err.kind(), kind);
assert!(task.await.expect("fixture should join").is_empty());
}
let (backend, task) = fixture(vec![reply(StatusCode::CREATED, &[], b"")]).await;
let body = Bytes::from(vec![0; usize::try_from(MIN_BLOCK_BYTES + 1).expect("fixture size")]);
let err = backend
.put(
"short-block",
ReaderImpl::Body(body),
i64::try_from(MIN_BLOCK_BYTES + 2).expect("fixture length"),
)
.await
.expect_err("short final block");
assert_eq!(err.kind(), ErrorKind::UnexpectedEof);
assert_eq!(
task.await.expect("fixture should join").len(),
1,
"short input must never commit its staged blocks"
);
}
#[tokio::test]
async fn range_get_preserves_key_encoding_and_exact_version_for_delete() {
let version = "2026-11-06T01:02:03.0000000Z";
let (backend, task) = fixture(vec![
reply(
StatusCode::PARTIAL_CONTENT,
&[("content-range", "bytes 1-3/6"), ("x-ms-version-id", version)],
b"ust",
),
reply(StatusCode::ACCEPTED, &[], b""),
])
.await;
let object = "dir/a b+?#%2F.bin";
let mut reader = backend
.get(
object,
version,
WarmBackendGetOpts {
start_offset: 1,
length: 3,
},
)
.await
.expect("versioned range read");
let mut body = Vec::new();
reader.read_to_end(&mut body).await.expect("range body");
assert_eq!(body, b"ust");
backend.remove_exact(object, version).await.expect("exact delete");
let requests = task.await.expect("fixture should join");
assert_eq!(requests.len(), 2, "exact delete needs no current-version HEAD");
for request in requests {
let url = Url::parse(&format!("http://fixture{}", request.target)).expect("captured URL");
assert_eq!(
urlencoding::decode(url.path()).expect("decoded path"),
format!("/container/cold/{object}")
);
let query: Vec<_> = url.query_pairs().into_owned().collect();
assert_eq!(query, [("versionid".to_string(), version.to_string())]);
}
}
#[tokio::test]
async fn current_version_delete_demotes_only_the_matching_root_before_exact_cleanup() {
for status in [StatusCode::FORBIDDEN, StatusCode::CONFLICT] {
let version = "target-version";
let (backend, task) = fixture(vec![
reply(status, &[("x-ms-error-code", "OperationNotAllowedOnRootBlob")], b""),
present(Some(version)),
reply(StatusCode::ACCEPTED, &[], b""),
reply(StatusCode::ACCEPTED, &[], b""),
])
.await;
backend
.remove_exact("current", version)
.await
.expect("current version should be removable");
let requests = task.await.expect("fixture should join");
assert_eq!(requests.len(), 4);
assert_eq!(requests[0].target, "/container/cold/current?versionid=target-version");
assert_eq!(requests[1].method, "HEAD");
assert_eq!(requests[1].target, "/container/cold/current");
assert_eq!(requests[2].method, "DELETE");
assert_eq!(requests[2].target, "/container/cold/current");
assert_eq!(requests[2].headers["if-match"], "\"original\"");
assert_eq!(requests[3].method, "DELETE");
assert_eq!(requests[3].target, requests[0].target);
}
}
#[tokio::test]
async fn current_version_delete_refuses_replacements_before_and_after_head() {
for replaced_after_head in [false, true] {
let mut responses = vec![
reply(StatusCode::FORBIDDEN, &[("x-ms-error-code", "OperationNotAllowedOnRootBlob")], b""),
present(Some(if replaced_after_head {
"target-version"
} else {
"replacement-version"
})),
];
if replaced_after_head {
responses.push(reply(StatusCode::PRECONDITION_FAILED, &[("x-ms-error-code", "ConditionNotMet")], b""));
}
let (backend, task) = fixture(responses).await;
assert!(backend.remove_exact("current", "target-version").await.is_err());
let requests = task.await.expect("fixture should join");
if replaced_after_head {
assert_eq!(requests.len(), 3, "a failed condition must stop exact cleanup");
assert_eq!(requests[2].target, "/container/cold/current");
assert_eq!(requests[2].headers["if-match"], "\"original\"");
} else {
assert_eq!(requests.len(), 2, "a different current version must never authorize a base delete");
}
}
}
#[tokio::test]
async fn current_version_delete_retries_the_exact_version_when_the_root_is_already_absent() {
let (backend, task) = fixture(vec![
reply(StatusCode::FORBIDDEN, &[("x-ms-error-code", "OperationNotAllowedOnRootBlob")], b""),
missing(),
reply(StatusCode::ACCEPTED, &[], b""),
])
.await;
backend
.remove_exact("current", "target-version")
.await
.expect("previous version cleanup");
let requests = task.await.expect("fixture should join");
assert_eq!(requests.len(), 3);
assert_eq!(requests[2].target, "/container/cold/current?versionid=target-version");
assert!(
requests
.iter()
.all(|request| request.method != "DELETE" || request.target.contains("versionid="))
);
}
#[tokio::test]
async fn current_version_delete_keeps_partial_or_uncertain_cleanup_retryable() {
for fails_during_exact_cleanup in [false, true] {
for lost_response in [false, true] {
let mut responses = vec![
reply(StatusCode::FORBIDDEN, &[("x-ms-error-code", "OperationNotAllowedOnRootBlob")], b""),
present(Some("target-version")),
];
if fails_during_exact_cleanup {
responses.push(reply(StatusCode::ACCEPTED, &[], b""));
}
let mut failure = reply(StatusCode::INTERNAL_SERVER_ERROR, &[("x-ms-error-code", "InternalError")], b"");
failure.disconnect = lost_response;
responses.push(failure);
responses.push(reply(StatusCode::ACCEPTED, &[], b""));
let (backend, task) = fixture(responses).await;
backend
.remove_exact("current", "target-version")
.await
.expect_err("base removal alone or an uncertain response must not complete cleanup");
backend
.remove_exact("current", "target-version")
.await
.expect("retry exact cleanup");
let requests = task.await.expect("fixture should join");
let expected = if fails_during_exact_cleanup { 5 } else { 4 };
assert_eq!(requests.len(), expected, "uncertain destructive requests must not be replayed internally");
assert_eq!(requests[2].headers["if-match"], "\"original\"");
assert_eq!(requests[expected - 1].target, "/container/cold/current?versionid=target-version");
assert_eq!(
requests
.iter()
.filter(|request| request.method == "DELETE" && !request.target.contains('?'))
.count(),
1,
"a retry must not delete the root again"
);
}
}
}
#[tokio::test]
async fn delete_requires_a_single_strong_etag_and_an_exact_current_version() {
for versioned in [false, true] {
for etag in [
None,
Some(""),
Some("*"),
Some("W/\"weak\""),
Some("\"first\", \"second\""),
Some("\"\""),
] {
let mut head = present(if versioned { Some("target-version") } else { None });
head.headers.remove("etag");
if let Some(etag) = etag {
insert_header(&mut head.headers, "etag", etag).expect("fixture etag should be representable");
}
let mut responses = Vec::new();
if versioned {
responses.push(reply(StatusCode::FORBIDDEN, &[("x-ms-error-code", "OperationNotAllowedOnRootBlob")], b""));
}
responses.push(head);
let (backend, task) = fixture(responses).await;
assert!(
backend
.remove("current", if versioned { "target-version" } else { "" })
.await
.is_err()
);
let requests = task.await.expect("fixture should join");
assert!(
requests
.iter()
.all(|request| request.method != "DELETE" || request.target.contains("versionid=")),
"a missing, wildcard, weak or multi-value ETag cannot authorize root removal"
);
}
}
for current_version in [None, Some("")] {
let (backend, task) = fixture(vec![
reply(StatusCode::FORBIDDEN, &[("x-ms-error-code", "OperationNotAllowedOnRootBlob")], b""),
present(current_version),
])
.await;
assert!(backend.remove_exact("current", "target-version").await.is_err());
assert_eq!(task.await.expect("fixture should join").len(), 2);
}
}
#[tokio::test]
async fn exact_delete_does_not_demote_roots_for_other_service_errors() {
for (status, code) in [
(StatusCode::CONFLICT, "BlobImmutableDueToPolicy"),
(StatusCode::FORBIDDEN, "BlobImmutableDueToPolicy"),
(StatusCode::FORBIDDEN, "AuthenticationFailed"),
(StatusCode::FORBIDDEN, "AuthorizationFailure"),
(StatusCode::FORBIDDEN, "AuthorizationPermissionMismatch"),
(StatusCode::FORBIDDEN, "OperationNotAllowedOnRootBlobOther"),
(StatusCode::UNAUTHORIZED, "OperationNotAllowedOnRootBlob"),
(StatusCode::INTERNAL_SERVER_ERROR, "OperationNotAllowedOnRootBlob"),
] {
let (backend, task) = fixture(vec![reply(status, &[("x-ms-error-code", code)], b"")]).await;
let error = backend
.remove_exact("current", "target-version")
.await
.expect_err("service rejection");
assert_eq!(AzureBlobError::from_io(&error).expect("typed service error").code, code);
assert_eq!(
task.await.expect("fixture should join").len(),
1,
"unrelated errors cannot trigger root removal"
);
}
}
#[test]
fn prefix_reserves_the_complete_canonical_lifecycle_name_before_backend_creation() {
use crate::bucket::lifecycle::transition_transaction::canonical_transition_remote_object;
let mut config = TierAzure {
access_key: "account".to_string(),
secret_key: base64_simd::STANDARD.encode_to_string([7; 32]),
bucket: "container".to_string(),
prefix: "a".repeat(800),
..Default::default()
};
assert!(
WarmBackendAzureBlob::new(&config).is_err(),
"a prefix that fits only the probe is unusable for lifecycle"
);
let object = canonical_transition_remote_object(
uuid::Uuid::from_u128(1),
"source-bucket",
uuid::Uuid::from_u128(2),
uuid::Uuid::from_u128(3),
)
.expect("canonical lifecycle key");
let max_ascii_prefix = 880 - object.chars().count() - 1;
config.prefix = "a".repeat(max_ascii_prefix);
let backend = WarmBackendAzureBlob::new(&config).expect("complete name at the character limit");
assert!(backend.object_url(&object, "").is_ok());
config.prefix.push('/');
let backend = WarmBackendAzureBlob::new(&config).expect("one trailing slash is normalized");
assert!(backend.object_url(&object, "").is_ok());
config.prefix = "a".repeat(max_ascii_prefix + 1);
assert!(WarmBackendAzureBlob::new(&config).is_err(), "reject one extra character before I/O");
let max_unicode_prefix = (1760 - object.len() - 1) / '界'.len_utf8();
config.prefix = "界".repeat(max_unicode_prefix);
let backend = WarmBackendAzureBlob::new(&config).expect("complete Unicode name within the byte limit");
assert!(backend.object_url(&object, "").is_ok());
config.prefix.push('界');
assert!(
WarmBackendAzureBlob::new(&config).is_err(),
"the byte limit must include the canonical suffix"
);
}
#[tokio::test]
async fn version_drift_and_incorrect_ranges_fail_closed() {
let (backend, task) = fixture(vec![present(Some("new-version"))]).await;
let err = backend
.remove("changed", "")
.await
.expect_err("unversioned delete must reject version drift");
assert!(err.to_string().contains("became versioned"));
assert_eq!(task.await.expect("fixture should join").len(), 1);
for headers in [
vec![("content-range", "bytes 0-2/6")],
vec![("content-range", "bytes 1-3/6"), ("x-ms-version-id", "unexpected-version")],
] {
let (backend, task) = fixture(vec![reply(StatusCode::PARTIAL_CONTENT, &headers, b"ust")]).await;
let result = backend
.get(
"object",
"",
WarmBackendGetOpts {
start_offset: 1,
length: 3,
},
)
.await;
assert!(result.is_err(), "wrong offset or an unexpected version cannot satisfy the read");
task.await.expect("fixture should join");
}
}
#[tokio::test]
async fn unversioned_delete_keeps_the_head_precondition_when_the_blob_changes() {
let (backend, task) = fixture(vec![
present(None),
reply(StatusCode::PRECONDITION_FAILED, &[("x-ms-error-code", "ConditionNotMet")], b""),
])
.await;
assert!(backend.remove("changed", "").await.is_err());
let requests = task.await.expect("fixture should join");
assert_eq!(requests[1].headers["if-match"], "\"original\"");
}
#[tokio::test]
async fn absence_requires_blob_not_found_and_container_listing_is_bounded() {
for (response, missing_expected) in [
(missing(), true),
(reply(StatusCode::NOT_FOUND, &[("x-ms-error-code", "ContainerNotFound")], b""), false),
(reply(StatusCode::NOT_FOUND, &[], b""), false),
] {
let (backend, task) = fixture(vec![response]).await;
let result = backend.probe_transition_candidate("object").await;
if missing_expected {
assert_eq!(result.expect("explicit blob absence"), TransitionCandidateProbe::Missing);
} else {
assert!(result.is_err(), "an absent container or an ambiguous HEAD is not blob absence");
}
task.await.expect("fixture should join");
}
for (body, in_use) in [
(b"<EnumerationResults><Blobs/><NextMarker/></EnumerationResults>".as_slice(), false),
(
b"<EnumerationResults><Blobs><Blob><Name>cold/object</Name></Blob></Blobs><NextMarker/></EnumerationResults>"
.as_slice(),
true,
),
(
b"<EnumerationResults><Blobs/><NextMarker>next</NextMarker></EnumerationResults>".as_slice(),
true,
),
] {
let (backend, task) = fixture(vec![reply(StatusCode::OK, &[], body)]).await;
assert_eq!(backend.in_use().await.expect("native container listing"), in_use);
let requests = task.await.expect("fixture should join");
assert!(
requests[0]
.target
.contains("restype=container&comp=list&prefix=cold%2F&maxresults=1")
);
}
let oversized = vec![b' '; MAX_CONTROL_BYTES + 1];
let (backend, task) = fixture(vec![reply(StatusCode::OK, &[], &oversized)]).await;
assert!(backend.in_use().await.is_err(), "oversized listing cannot establish an empty destination");
task.await.expect("fixture should join");
}
#[tokio::test]
async fn recovery_requires_the_uploaded_transaction_and_destination_identity() {
let identity = TransitionCandidateIdentity {
transaction_id: uuid::Uuid::new_v4(),
destination_id: [3; 32],
};
let mut metadata = HashMap::new();
insert_str(&mut metadata, SUFFIX_TRANSITION_TRANSACTION_ID, identity.transaction_id.to_string());
insert_str(&mut metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, hex(identity.destination_id));
let headers = metadata_headers(metadata).expect("identity metadata should encode for Azure");
assert_eq!(
headers
.keys()
.filter(|name| name.as_str().starts_with(METADATA_PREFIX))
.count(),
4
);
let mut owned = present(Some("owned-version"));
owned.headers.extend(headers);
let (backend, task) = fixture(vec![owned, present(None)]).await;
assert_eq!(
backend
.probe_transition_candidate_for("object", identity)
.await
.expect("owned candidate"),
TransitionCandidateProbe::VersionedPresent("owned-version".to_string())
);
assert_eq!(
backend
.probe_transition_candidate_for("object", identity)
.await
.expect("foreign candidate classification"),
TransitionCandidateProbe::Ambiguous
);
task.await.expect("fixture should join");
}
#[test]
fn configuration_xml_and_size_limits_fail_before_network_io() {
let mut config = TierAzure {
access_key: "account".to_string(),
secret_key: base64_simd::STANDARD.encode_to_string([7; 32]),
bucket: "container".to_string(),
..Default::default()
};
assert!(
WarmBackendAzureBlob::new(&config).is_ok(),
"public endpoint should be derived from the account"
);
config.endpoint = "http://127.0.0.1:10000".to_string();
assert!(
WarmBackendAzureBlob::new(&config).is_err(),
"production constructor must apply the outbound guard"
);
config.endpoint = "https://account.blob.example/path".to_string();
assert!(
WarmBackendAzureBlob::new(&config).is_err(),
"an endpoint path cannot rewrite object addressing"
);
for name in [".", "..", "a/../b", "a/./b"] {
assert!(validate_object_name(name).is_err());
}
assert_eq!(
block_size(MAX_BLOCK_BYTES * MAX_BLOCKS).expect("largest supported object"),
MAX_BLOCK_BYTES
);
assert!(block_size(MAX_BLOCK_BYTES * MAX_BLOCKS + 1).is_err());
for xml in [
b"<html>not a container</html>".as_slice(),
b"<EnumerationResults><Blobs>".as_slice(),
b"<!DOCTYPE x><EnumerationResults><Blobs/></EnumerationResults>".as_slice(),
b"<EnumerationResults><Other><Blobs/></Other></EnumerationResults>".as_slice(),
b"garbage<EnumerationResults><Blobs/></EnumerationResults>".as_slice(),
] {
assert!(list_has_objects(xml).is_err());
}
assert_eq!(
xml_error_code(b"<Error><Code>BlobNotFound</Code></Error>").as_deref(),
Some("BlobNotFound")
);
assert!(xml_error_code(b"<Error><Code>BlobNotFound</Code>").is_none());
assert!(xml_error_code(b"<Error><Code>BlobNotFound</Code></Error>garbage").is_none());
}
#[tokio::test]
async fn control_response_has_a_total_deadline_after_headers_arrive() {
let listener = TcpListener::bind("127.0.0.1:0").await.expect("control fixture should bind");
let address = listener.local_addr().expect("control fixture address");
let (release, wait) = tokio::sync::oneshot::channel();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.expect("control fixture should accept");
capture(&mut stream).await;
stream
.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 16\r\n\r\n")
.await
.expect("control response headers");
let _ = wait.await;
});
let (mut backend, empty) = fixture(Vec::new()).await;
empty.await.expect("unused fixture should close");
backend.endpoint = Url::parse(&format!("http://{address}")).expect("control endpoint should parse");
let response = backend
.send(Method::GET, backend.container_url().expect("container URL"), HeaderMap::new(), None)
.await
.expect("headers should arrive before testing the body deadline");
let err = tokio::time::timeout(Duration::from_secs(1), read_control_body(response, Duration::from_millis(20)))
.await
.expect("body deadline must be shorter than the transport idle timeout")
.expect_err("an unfinished control response must time out");
assert_eq!(err.kind(), ErrorKind::TimedOut);
let _ = release.send(());
server.await.expect("control fixture should join");
}
@@ -351,7 +351,7 @@ fn transition_candidate_metadata(info: ObjectInfo) -> HashMap<String, String> {
metadata
}
fn transition_candidate_metadata_matches(
pub(super) fn transition_candidate_metadata_matches(
metadata: &HashMap<String, String>,
identity: TransitionCandidateIdentity,
) -> Result<bool, std::io::Error> {
@@ -284,8 +284,9 @@ pub(in crate::set_disk) fn resolve_read_part_from_responses(
present_count += 1;
let part = &parts[part_idx];
let key = (part.etag.as_str(), part.number, part.size, part.actual_size, part.integrity.as_ref());
let (count, _) = part_quorum.entry(key).or_insert((0, part));
let (count, protected_count, _) = part_quorum.entry(key).or_insert((0, 0, part));
*count += 1;
*protected_count += usize::from(part.encrypted_etag);
continue;
}
@@ -296,12 +297,16 @@ pub(in crate::set_disk) fn resolve_read_part_from_responses(
}
}
let max_part = part_quorum.values().max_by_key(|(count, _)| count);
let max_quorum = max_part.map_or(0, |(count, _)| *count).max(missing_count);
if let Some((count, part)) = max_part
let max_part = part_quorum.values().max_by_key(|(count, _, _)| count);
let max_quorum = max_part.map_or(0, |(count, _, _)| *count).max(missing_count);
if let Some((count, protected_count, part)) = max_part
&& *count >= read_quorum
{
return Ok((*part).clone());
let mut part = (*part).clone();
// Older peers may omit the extension while agreeing on the part itself.
// Require independent proof quorum without reducing part availability.
part.encrypted_etag = *protected_count >= read_quorum;
return Ok(part);
}
if missing_count >= read_quorum {
@@ -13390,6 +13395,53 @@ mod tests {
}
}
#[test]
fn resolve_read_part_requires_matching_quorum_for_encrypted_etag_proof() {
let protected = ObjectPartInfo {
encrypted_etag: true,
..read_part_test_part(1, "protected-etag")
};
let legacy = ObjectPartInfo {
encrypted_etag: false,
..protected.clone()
};
for (parts, quorum, expected) in [
(vec![protected.clone(), legacy.clone(), legacy.clone()], 2, false),
(vec![legacy.clone(), protected.clone(), protected.clone()], 2, true),
(vec![protected.clone(), protected.clone(), legacy.clone(), legacy.clone()], 3, false),
(
vec![
protected.clone(),
ObjectPartInfo {
etag: "other-generation".to_owned(),
..protected.clone()
},
legacy.clone(),
],
2,
false,
),
(
vec![
protected.clone(),
ObjectPartInfo {
error: Some("read failed".to_owned()),
..protected.clone()
},
legacy,
],
2,
false,
),
] {
let responses = parts.into_iter().map(|part| Some(vec![part])).collect::<Vec<_>>();
let part = resolve_read_part_from_responses("bucket", "upload/part.1.meta", 1, 0, 1, &responses, quorum)
.expect("matching metadata remains readable across producer versions");
assert_eq!(part.etag, protected.etag);
assert_eq!(part.encrypted_etag, expected);
}
}
#[test]
fn resolve_read_part_diagnostic_branch_keeps_read_quorum_error() {
temp_env::with_var("RUSTFS_ISSUE3031_DIAG_ENABLE", Some("true"), || {
+64 -53
View File
@@ -7980,21 +7980,23 @@ fn is_min_allowed_part_size(size: i64) -> bool {
}
fn get_complete_multipart_md5(parts: &[CompletePart]) -> String {
let mut buf = Vec::new();
for part in parts.iter() {
if let Some(etag) = &part.etag {
if let Ok(etag_bytes) = hex_simd::decode_to_vec(etag.as_bytes()) {
buf.extend(etag_bytes);
} else {
buf.extend(etag.bytes());
let mut hasher = Md5::new();
for etag in parts.iter().filter_map(|part| part.etag.as_deref()) {
// Native part ETags decode to one fixed-size digest. Preserve the
// historical hex-or-raw interpretation for imported part ETags.
let mut digest = [0u8; 16];
if etag.len() == digest.len() * 2 {
match hex_simd::decode(etag.as_bytes(), hex_simd::Out::from_slice(&mut digest)) {
Ok(decoded) => hasher.update(decoded),
Err(_) => hasher.update(etag.as_bytes()),
}
} else if let Ok(decoded) = hex_simd::decode_to_vec(etag.as_bytes()) {
hasher.update(decoded);
} else {
hasher.update(etag.as_bytes());
}
}
let mut hasher = Md5::new();
hasher.update(&buf);
let digest = hasher.finalize();
let etag_hex = faster_hex::hex_string(digest.as_slice());
format!("{}-{}", etag_hex, parts.len())
@@ -9682,50 +9684,59 @@ mod tests {
}
#[test]
fn test_get_complete_multipart_md5() {
// Test MD5 calculation for multipart upload
let parts = vec![
fn complete_multipart_md5_preserves_legacy_etag_encodings() {
fn part(etag: Option<&str>, part_num: usize) -> CompletePart {
CompletePart {
part_num: 1,
etag: Some("d41d8cd98f00b204e9800998ecf8427e".to_string()),
checksum_crc32: None,
checksum_crc32c: None,
checksum_sha1: None,
checksum_sha256: None,
checksum_crc64nvme: None,
},
CompletePart {
part_num: 2,
etag: Some("098f6bcd4621d373cade4e832627b4f6".to_string()),
checksum_crc32: None,
checksum_crc32c: None,
checksum_sha1: None,
checksum_sha256: None,
checksum_crc64nvme: None,
},
part_num,
etag: etag.map(str::to_owned),
..Default::default()
}
}
fn buffered_digest(parts: &[CompletePart]) -> String {
let mut bytes = Vec::new();
for etag in parts.iter().filter_map(|part| part.etag.as_deref()) {
match hex_simd::decode_to_vec(etag.as_bytes()) {
Ok(decoded) => bytes.extend(decoded),
Err(_) => bytes.extend_from_slice(etag.as_bytes()),
}
}
format!("{}-{}", faster_hex::hex_string(&Md5::digest(bytes)), parts.len())
}
assert_eq!(get_complete_multipart_md5(&[]), "d41d8cd98f00b204e9800998ecf8427e-0");
assert_eq!(get_complete_multipart_md5(&[part(None, 1)]), "d41d8cd98f00b204e9800998ecf8427e-1");
assert_eq!(get_complete_multipart_md5(&[part(Some("00"), 1)]), "93b885adfe0da089cdf634904fd59f71-1");
assert_eq!(
get_complete_multipart_md5(&[
part(Some("d41d8cd98f00b204e9800998ecf8427e"), 1),
part(Some("098f6bcd4621d373cade4e832627b4f6"), 2),
]),
"f666c301a43456b66b9f3a47f696d391-2"
);
let encodings = [
None,
Some(""),
Some("00"),
Some("0aBcFf"),
Some("0123456789abcdef0123456789abcdef"),
Some("0123456789ABCDEF0123456789ABCDEF"),
Some("0123456789abcdef0123456789abcdef00"),
Some("0123456789abcdef0123456789abcdeg"),
Some("0123456789abcdef0123456789abcdeg00"),
Some("f"),
Some("not-hex"),
Some("\"0123456789abcdef0123456789abcdef\""),
Some("0123456789abcdef0123456789abcdef-2"),
Some("café"),
];
let md5 = get_complete_multipart_md5(&parts);
assert!(md5.ends_with("-2")); // Should end with part count
assert!(md5.len() > 10); // Should have reasonable length
// Test with empty parts
let empty_parts = vec![];
let empty_result = get_complete_multipart_md5(&empty_parts);
assert!(empty_result.ends_with("-0"));
// Test with single part
let single_part = vec![CompletePart {
part_num: 1,
etag: Some("d41d8cd98f00b204e9800998ecf8427e".to_string()),
checksum_crc32: None,
checksum_crc32c: None,
checksum_sha1: None,
checksum_sha256: None,
checksum_crc64nvme: None,
}];
let single_result = get_complete_multipart_md5(&single_part);
assert!(single_result.ends_with("-1"));
for first in encodings {
for second in encodings {
let parts = [part(first, 1), part(second, 2)];
assert_eq!(get_complete_multipart_md5(&parts), buffered_digest(&parts), "{first:?}, {second:?}");
}
}
}
#[test]
@@ -311,4 +311,85 @@ mod tests {
.await
.expect("single-disk no-parity PUT should pass self-verify");
}
#[tokio::test]
async fn no_parity_self_verify_rejects_missing_inline_data_missing_disk_shard_and_under_quorum() {
let (_temp_dirs, disks, _set_disks) = hermetic_set_disks_for_pool_with_default_parity(1, 0, 0).await;
let shard_size = 16usize;
let payload = b"self verify negative path";
let encoded = encode_streaming_shard(payload, shard_size).await;
let disks_opt = disks.iter().cloned().map(Some).collect::<Vec<_>>();
// 1. Inline shard with `data: None` fails closed.
let empty_parts = [FileInfo {
data: None,
..Default::default()
}];
let err = verify_written_bitrot_shards(
&disks_opt,
Some(&empty_parts),
BitrotSelfVerifyTarget {
operation: "put_object",
bucket: "bucket",
object: "missing-inline-data",
part_number: None,
volume: RUSTFS_META_TMP_BUCKET,
path: "unused",
logical_shard_size: payload.len(),
shard_size,
write_quorum: 1,
},
)
.await
.expect_err("inline shard missing data must fail");
assert!(err.to_string().contains("missing bitrot data"));
// 2. Disk shard missing on disk (FileNotFound) fails closed with shard error context.
let err = verify_written_bitrot_shards(
&disks_opt,
None,
BitrotSelfVerifyTarget {
operation: "put_object",
bucket: "bucket",
object: "missing-disk-file",
part_number: Some(3),
volume: RUSTFS_META_TMP_BUCKET,
path: "nonexistent-temp-shard",
logical_shard_size: payload.len(),
shard_size,
write_quorum: 1,
},
)
.await
.expect_err("missing disk shard must fail");
assert!(err.to_string().contains("no-parity bitrot self-verify failed for shard 0"));
// 3. Verified shard count below write_quorum fails closed even when available shard is valid.
let valid_parts = [
FileInfo {
data: Some(encoded),
..Default::default()
},
FileInfo::default(),
];
let two_slots = vec![disks_opt[0].clone(), None];
let err = verify_written_bitrot_shards(
&two_slots,
Some(&valid_parts),
BitrotSelfVerifyTarget {
operation: "put_object",
bucket: "bucket",
object: "under-quorum",
part_number: None,
volume: RUSTFS_META_TMP_BUCKET,
path: "unused",
logical_shard_size: payload.len(),
shard_size,
write_quorum: 2,
},
)
.await
.expect_err("under-quorum verification must fail");
assert!(err.to_string().contains("quorum unavailable: 1 shard(s) verified, need 2"));
}
}
+365 -16
View File
@@ -46,8 +46,8 @@ use super::super::{
log_multipart_write_quorum_failure, multipart_part_logical_size, parts_after_marker, path_join_buf,
prepare_complete_multipart_parts, record_compression_total_memory, record_multipart_write_quorum_failure,
reduce_read_quorum_errs, reduce_write_quorum_errs, remove_header_map, resolve_write_layout,
restore_commit_operation_id_from_metadata, should_persist_encryption_original_size, strip_internal_multipart_metadata,
to_object_err, warn,
restore_commit_operation_id_from_metadata, restore_operation_id_from_metadata, should_persist_encryption_original_size,
strip_internal_multipart_metadata, to_object_err, warn,
};
use super::bitrot_self_verify::{BitrotSelfVerifyTarget, drop_failed_writer_disks, verify_written_bitrot_shards};
#[cfg(test)]
@@ -2728,6 +2728,9 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
.map(crate::io_support::rio::compression_index_storage_bytes);
let mut etag = data.stream.try_resolve_etag().unwrap_or_default();
let encrypted_etag = data.stream.encrypted_etag
&& !etag.is_empty()
&& opts.preserve_etag.as_ref().is_none_or(|preserved| preserved == &etag);
if let Some(ref tag) = opts.preserve_etag {
etag = tag.clone();
@@ -2748,6 +2751,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
let part_info = ObjectPartInfo {
etag: etag.clone(),
encrypted_etag,
number: part_id,
size: w_size,
mod_time: Some(OffsetDateTime::now_utc()),
@@ -3647,6 +3651,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
}
let source_condition = capture_source_condition(bucket, object, &upload_id_path, &fi, Uuid::new_v4())?;
let preserved_encrypted_etag = preserves_staged_encrypted_etag_proof(&fi, opts, expected_restore_operation_id);
let upload_data_dir = source_condition.lineage.original_data_dir.to_string();
// The source generation remains in the captured precondition. Its
// ACTIVE/recovery controls are not final user-object metadata.
@@ -4117,11 +4122,17 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
}
// etag
let etag = resolve_complete_etag(opts, &uploaded_parts);
let (etag, encrypted_etag) = resolve_complete_etag(opts, &uploaded_parts, &fi.parts);
let encrypted_etag = encrypted_etag
|| preserved_encrypted_etag
&& fi.version_id.filter(|version| !version.is_nil()) == source_condition.lineage.original_version_id;
// An upload can contain parts from an older writer or a peer. Until
// every part's format is attested, keep its ETag out of keyless lists.
// New writes attest their selected parts. Same-version internal moves
// and restores can retain the proof from quorum-validated source metadata.
rustfs_utils::http::metadata_compat::remove_str(&mut fi.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG);
if encrypted_etag {
insert_str(&mut fi.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
}
fi.metadata.insert("etag".to_owned(), etag);
@@ -4948,19 +4959,57 @@ fn completed_multipart_upload_matches(
Ok(true)
}
// Capture only the original staged proof, before completion applies metadata
// updates. Publication must also retain the source lineage's version.
fn preserves_staged_encrypted_etag_proof(fi: &FileInfo, opts: &ObjectOptions, restore_operation_id: Option<Uuid>) -> bool {
if opts.replication_request {
return false;
}
let movement = opts.data_movement
&& opts.version_id.is_some()
&& get_consistent_str(&fi.metadata, rustfs_utils::http::SUFFIX_DATA_MOVEMENT_UPLOAD)
.is_some_and(|identity| !identity.is_empty());
// Commit rechecks this operation against the live object under its lock.
let restore = restore_operation_id
.is_some_and(|expected| restore_operation_id_from_metadata(&fi.metadata).is_ok_and(|id| id == Some(expected)));
if !movement && !restore {
return false;
}
let Ok(version) = opts.version_id.as_deref().map(Uuid::parse_str).transpose() else {
return false;
};
let Some(etag) = opts.preserve_etag.as_deref().filter(|etag| !etag.is_empty()) else {
return false;
};
version.filter(|version| !version.is_nil()) == fi.version_id.filter(|version| !version.is_nil())
&& fi.metadata.get("etag").map(String::as_str) == Some(etag)
&& get_consistent_str(&fi.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG) == Some("v1")
}
/// Final ETag for a completed multipart object. An authorized replication
/// request preserves the source ETag so the replication HEAD comparison
/// converges even when the source ETag is not derivable from the uploaded
/// parts (foreign-origin objects, ciphertext-derived ETags); the internal
/// metadata override comes next; otherwise the ETag is computed from parts.
fn resolve_complete_etag(opts: &ObjectOptions, uploaded_parts: &[CompletePart]) -> String {
if let Some(etag) = opts.preserve_etag.as_ref().filter(|etag| !etag.is_empty()) {
return etag.clone();
fn resolve_complete_etag(
opts: &ObjectOptions,
uploaded_parts: &[CompletePart],
stored_parts: &[ObjectPartInfo],
) -> (String, bool) {
let preserved = opts
.preserve_etag
.as_ref()
.filter(|etag| !etag.is_empty())
.or_else(|| opts.user_defined.get("etag"));
let protected_parts = !stored_parts.is_empty() && stored_parts.iter().all(|part| part.encrypted_etag);
if let Some(etag) = preserved
&& !protected_parts
{
return (etag.clone(), false);
}
if let Some(etag) = opts.user_defined.get("etag") {
return etag.clone();
}
get_complete_multipart_md5(uploaded_parts)
let computed = get_complete_multipart_md5(uploaded_parts);
let encrypted_etag = protected_parts && preserved.is_none_or(|etag| etag == &computed);
(preserved.cloned().unwrap_or(computed), encrypted_etag)
}
/// Serialize the object-level checksum record that `complete_multipart_upload`
@@ -9093,6 +9142,26 @@ mod tests {
receipts.push(part.etag);
}
}
let (staged, _) = set
.check_upload_id_exists(bucket, "object", &upload.upload_id, false)
.await
.expect("read upload generation");
let upload_path = SetDisks::get_upload_id_dir(bucket, "object", &upload.upload_id);
let data_dir = staged.data_dir.expect("staging data directory");
for disk in &disks {
let metadata = disk
.read_parts(
RUSTFS_META_MULTIPART_BUCKET,
&[
format!("{upload_path}/{data_dir}/part.1.meta"),
format!("{upload_path}/{data_dir}/part.3.meta"),
],
)
.await
.expect("read persisted part proofs");
assert!(metadata[0].encrypted_etag, "rewritten protected part is attested");
assert!(!metadata[1].encrypted_etag, "an overridden part ETag has no producer proof");
}
assert_ne!(receipts[0], receipts[1], "spare parts must not identify another part's plaintext");
assert_ne!(receipts[0], receipts[2], "same-number retries get a new receipt");
for _ in 0..2 {
@@ -9129,6 +9198,230 @@ mod tests {
.await
.expect("committed completion replay is stable");
assert_eq!(replay.etag, committed.etag);
for completed in [committed, replay] {
assert!(
rustfs_utils::http::etag_can_be_listed(&completed.user_defined),
"the selected protected part retains its ETag after completion and replay"
);
assert_eq!(
rustfs_utils::http::get_consistent_str(&completed.user_defined, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG,),
Some("v1")
);
}
}
#[tokio::test]
async fn complete_encrypted_multipart_clears_unproven_upload_etag_marker() {
let (_dirs, disks, set) = hermetic_set_disks(4).await;
let bucket = "legacy-encrypted-etag";
make_bucket_on_all(&disks, bucket).await;
let mut opts = ObjectOptions {
user_defined: HashMap::from([(SSEC_ALGORITHM_HEADER.to_owned(), "AES256".to_owned())]),
..Default::default()
};
insert_str(&mut opts.user_defined, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
// Model an older writer whose part metadata has no protected-format proof.
let (upload_id, parts) = stage_upload_with_create_opts(&set, bucket, "object", b"legacy part", &opts).await;
let completed = set
.clone()
.complete_multipart_upload(bucket, "object", &upload_id, parts, &ObjectOptions::default())
.await
.expect("legacy part can still complete");
let reread = set
.get_object_info(bucket, "object", &ObjectOptions::default())
.await
.expect("read completed metadata");
assert_eq!(completed.etag, reread.etag);
for info in [completed, reread] {
assert!(!rustfs_utils::http::etag_can_be_listed(&info.user_defined));
assert!(!contains_key_str(&info.user_defined, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG));
}
}
#[test]
fn data_movement_etag_proof_requires_verified_source_identity() {
let version = Uuid::new_v4();
let mut source = FileInfo {
version_id: Some(version),
metadata: HashMap::from([("etag".to_owned(), "protected-source-etag".to_owned())]),
..Default::default()
};
insert_str(
&mut source.metadata,
rustfs_utils::http::SUFFIX_DATA_MOVEMENT_UPLOAD,
"source-generation".to_owned(),
);
insert_str(&mut source.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
let opts = ObjectOptions {
data_movement: true,
version_id: Some(version.to_string()),
preserve_etag: source.metadata.get("etag").cloned(),
..Default::default()
};
assert!(preserves_staged_encrypted_etag_proof(&source, &opts, None));
for version_id in [None, Some(Uuid::nil())] {
let null_source = FileInfo {
version_id,
..source.clone()
};
let null_opts = ObjectOptions {
version_id: Some(Uuid::nil().to_string()),
..opts.clone()
};
assert!(preserves_staged_encrypted_etag_proof(&null_source, &null_opts, None));
}
for rejected in [
ObjectOptions {
data_movement: false,
..opts.clone()
},
ObjectOptions {
replication_request: true,
..opts.clone()
},
ObjectOptions {
version_id: None,
..opts.clone()
},
ObjectOptions {
version_id: Some("invalid".to_owned()),
..opts.clone()
},
ObjectOptions {
version_id: Some(Uuid::new_v4().to_string()),
..opts.clone()
},
ObjectOptions {
preserve_etag: None,
..opts.clone()
},
ObjectOptions {
preserve_etag: Some(String::new()),
..opts.clone()
},
ObjectOptions {
preserve_etag: Some("replacement-etag".to_owned()),
..opts.clone()
},
] {
assert!(!preserves_staged_encrypted_etag_proof(&source, &rejected, None));
}
for suffix in [
rustfs_utils::http::SUFFIX_DATA_MOVEMENT_UPLOAD,
rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG,
] {
let mut missing = source.clone();
rustfs_utils::http::remove_str(&mut missing.metadata, suffix);
assert!(!preserves_staged_encrypted_etag_proof(&missing, &opts, None));
insert_str(&mut missing.metadata, suffix, String::new());
assert!(!preserves_staged_encrypted_etag_proof(&missing, &opts, None));
let mut conflicting = source.clone();
conflicting
.metadata
.insert(format!("x-minio-internal-{suffix}"), "conflict".to_owned());
assert!(!preserves_staged_encrypted_etag_proof(&conflicting, &opts, None));
}
for value in [None, Some("unknown")] {
let mut unproven = source.clone();
rustfs_utils::http::remove_str(&mut unproven.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG);
if let Some(value) = value {
insert_str(&mut unproven.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, value.to_owned());
}
let request_claim = ObjectOptions {
user_defined: source.metadata.clone(),
..opts.clone()
};
assert!(!preserves_staged_encrypted_etag_proof(&unproven, &request_claim, None));
}
}
#[test]
fn restore_etag_proof_requires_matching_staged_operation() {
let operation_id = Uuid::new_v4();
for version_id in [None, Some(Uuid::new_v4())] {
let mut staged = FileInfo {
version_id,
metadata: HashMap::from([("etag".to_owned(), "source-etag".to_owned())]),
..Default::default()
};
let opts = ObjectOptions {
version_id: version_id.map(|id| id.to_string()),
preserve_etag: Some("source-etag".to_owned()),
..Default::default()
};
insert_str(&mut staged.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
assert!(!preserves_staged_encrypted_etag_proof(&staged, &opts, Some(operation_id)));
insert_str(&mut staged.metadata, SUFFIX_RESTORE_OPERATION_ID, operation_id.to_string());
assert!(preserves_staged_encrypted_etag_proof(&staged, &opts, Some(operation_id)));
assert!(!preserves_staged_encrypted_etag_proof(&staged, &opts, Some(Uuid::new_v4())));
rustfs_utils::http::remove_str(&mut staged.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG);
assert!(!preserves_staged_encrypted_etag_proof(&staged, &opts, Some(operation_id)));
}
}
#[tokio::test]
async fn complete_data_movement_etag_proof_rejects_version_rewrites_and_late_claims() {
let (_dirs, disks, set) = hermetic_set_disks(4).await;
let bucket = "movement-etag-proof";
make_bucket_on_all(&disks, bucket).await;
for (object, staged_proof, publish_versioned, late_claim, expected_proof) in [
("same-version", true, true, false, true),
("rewritten-version", true, false, false, false),
("late-claim", false, true, true, false),
] {
let version = Uuid::new_v4();
let mut opts = ObjectOptions {
data_movement: true,
versioned: true,
version_id: Some(version.to_string()),
preserve_etag: Some("protected-source-etag".to_owned()),
user_defined: HashMap::from([(SSEC_ALGORITHM_HEADER.to_owned(), "AES256".to_owned())]),
..Default::default()
};
insert_str(
&mut opts.user_defined,
rustfs_utils::http::SUFFIX_DATA_MOVEMENT_UPLOAD,
"source-generation".to_owned(),
);
if staged_proof {
insert_str(&mut opts.user_defined, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
}
let (upload_id, parts) = stage_upload_with_create_opts(&set, bucket, object, b"raw source bytes", &opts).await;
opts.versioned = publish_versioned;
if late_claim {
let mut metadata = HashMap::new();
insert_str(&mut metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
opts.eval_metadata = Some(metadata);
}
let completed = set
.clone()
.complete_multipart_upload(bucket, object, &upload_id, parts, &opts)
.await
.expect("complete raw internal movement");
assert_eq!(completed.version_id, publish_versioned.then_some(version), "{object}");
assert_eq!(completed.etag, opts.preserve_etag, "{object}");
let persisted = set
.get_object_info(
bucket,
object,
&ObjectOptions {
version_id: Some(crate::data_movement::data_movement_exact_version_id(completed.version_id)),
..Default::default()
},
)
.await
.expect("read completed proof from disk");
assert_eq!(
rustfs_utils::http::etag_can_be_listed(&persisted.user_defined),
expected_proof,
"{object}"
);
assert_eq!(
get_consistent_str(&persisted.user_defined, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG),
expected_proof.then_some("v1"),
"{object}"
);
}
}
#[tokio::test]
@@ -16205,7 +16498,7 @@ mod tests {
preserve_etag: Some(foreign_etag.clone()),
..Default::default()
};
assert_eq!(resolve_complete_etag(&opts, &[]), foreign_etag);
assert_eq!(resolve_complete_etag(&opts, &[], &[]), (foreign_etag, false));
// Empty preserve value degrades to the next source.
let opts_empty = ObjectOptions {
@@ -16213,11 +16506,67 @@ mod tests {
user_defined: std::collections::HashMap::from([("etag".to_string(), "override-etag".to_string())]),
..Default::default()
};
assert_eq!(resolve_complete_etag(&opts_empty, &[]), "override-etag");
assert_eq!(resolve_complete_etag(&opts_empty, &[], &[]), ("override-etag".to_string(), false));
// Without either source the ETag is computed from the parts.
let computed = resolve_complete_etag(&ObjectOptions::default(), &[]);
assert_eq!(computed, get_complete_multipart_md5(&[]));
let computed = resolve_complete_etag(&ObjectOptions::default(), &[], &[]);
assert_eq!(computed, (get_complete_multipart_md5(&[]), false));
}
#[test]
fn complete_etag_proof_requires_all_selected_parts_and_matching_overrides() {
for proofs in [&[][..], &[true], &[true, true], &[true, false], &[false]] {
let stored: Vec<_> = proofs
.iter()
.enumerate()
.map(|(index, &encrypted_etag)| ObjectPartInfo {
number: index + 1,
etag: "0123456789abcdef0123456789abcdef".to_owned(),
encrypted_etag,
..Default::default()
})
.collect();
let uploaded: Vec<_> = stored
.iter()
.map(|part| CompletePart {
part_num: part.number,
etag: Some(part.etag.clone()),
..Default::default()
})
.collect();
let computed = get_complete_multipart_md5(&uploaded);
let protected = !proofs.is_empty() && proofs.iter().all(|&proof| proof);
assert_eq!(
resolve_complete_etag(&ObjectOptions::default(), &uploaded, &stored),
(computed.clone(), protected)
);
for preserved in ["foreign-etag", computed.as_str(), ""] {
for opts in [
ObjectOptions {
preserve_etag: Some(preserved.to_owned()),
..Default::default()
},
ObjectOptions {
user_defined: HashMap::from([("etag".to_owned(), preserved.to_owned())]),
..Default::default()
},
ObjectOptions {
preserve_etag: Some(preserved.to_owned()),
user_defined: HashMap::from([("etag".to_owned(), "metadata-etag".to_owned())]),
..Default::default()
},
] {
let expected_etag = if !preserved.is_empty() || opts.preserve_etag.is_none() {
preserved
} else {
opts.user_defined.get("etag").map_or(computed.as_str(), String::as_str)
};
let result = resolve_complete_etag(&opts, &uploaded, &stored);
assert_eq!(result, (expected_etag.to_owned(), protected && expected_etag == computed));
}
}
}
}
/// Accumulate the per-part digests and the running full-object checksum the
+57 -24
View File
@@ -9103,6 +9103,9 @@ impl SetDisks {
let mut futures = Vec::with_capacity(disks.len());
let mut errs = Vec::with_capacity(disks.len());
let requires_bounded_delete_recovery = commit_context
.options
.is_some_and(|options| options.delete_recovery_work_budget.is_some());
for disk in disks.iter() {
let disk_namespace_owner = namespace_owner.clone().map(|owner| owner as Arc<dyn Send + Sync>);
let transaction_guard = commit_context.transaction_guard.as_ref();
@@ -9114,6 +9117,7 @@ impl SetDisks {
let options = DeleteOptions {
old_data_dir: Some(rollback_dir),
delete_recovery_bucket_incarnation: source_bucket_incarnation_id,
requires_bounded_delete_recovery,
..Default::default()
};
let result = if is_meta_bucketname(bucket) {
@@ -10557,6 +10561,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
.collect();
let mut futures = Vec::with_capacity(disks.len());
let lock_lost_during_commit = Arc::new(std::sync::atomic::AtomicBool::new(false));
let requires_bounded_delete_recovery = opts.delete_recovery_work_budget.is_some();
// let mut errors = Vec::with_capacity(disks.len());
@@ -10574,6 +10579,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
let options = DeleteOptions {
old_data_dir: Some(rollback_dir),
delete_recovery_bucket_incarnation: opts.expected_bucket_incarnation_id,
requires_bounded_delete_recovery,
..Default::default()
};
if current_generations
@@ -17173,6 +17179,9 @@ mod transition_commit_failure_tests {
#[tokio::test]
#[serial_test::serial]
async fn multipart_restore_copies_the_stored_representation_back_verbatim() {
use crate::io_support::rio::{WriteEncryption, WritePlan};
use rustfs_utils::http::{SUFFIX_OPAQUE_ENCRYPTED_ETAG, etag_can_be_listed};
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "restore-multipart-stored-coordinates-bucket";
let object = "object.bin";
@@ -17180,9 +17189,8 @@ mod transition_commit_failure_tests {
disk.make_volume(bucket).await.expect("bucket volume should be created");
}
// The minimum-part-size gate reads the PLAINTEXT length, so part one
// clears 5 MiB there while its stored form carries encoding overhead.
let part_shapes = [(6 * 1024 * 1024_usize, 9_216_usize), (256 * 1024_usize, 512_usize)];
// The first plaintext part clears the non-final 5 MiB threshold.
let part_sizes = [6 * 1024 * 1024_usize, 256 * 1024_usize];
let mut user_defined = HashMap::new();
user_defined.insert(rustfs_utils::http::INTERNAL_ENCRYPTION_ALGORITHM_HEADER.to_string(), "AES256".to_string());
user_defined.insert(
@@ -17203,20 +17211,22 @@ mod transition_commit_failure_tests {
.expect("multipart upload should be created");
let mut uploaded_parts = Vec::new();
let mut expected_stored = Vec::new();
for (index, (plaintext_len, overhead)) in part_shapes.iter().enumerate() {
let stored = stored_representation_bytes(index as u32 * 7 + 1, plaintext_len + overhead);
expected_stored.extend_from_slice(&stored);
let stored_len = stored.len() as i64;
let hash_reader =
HashReader::from_stream(std::io::Cursor::new(stored), stored_len, *plaintext_len as i64, None, None, false)
.expect("hash reader over the stored representation");
let mut reader = PutObjReader::new(hash_reader);
for (index, &part_size) in part_sizes.iter().enumerate() {
let plaintext = stored_representation_bytes(index as u32 * 7 + 1, part_size);
let actual_size = part_size as i64;
let plaintext_reader =
HashReader::from_stream(std::io::Cursor::new(plaintext), actual_size, actual_size, None, None, false)
.expect("plaintext part reader");
let encrypted_reader = WritePlan::new()
.with_encryption(WriteEncryption::multipart([7; 32], [9; 12], index + 1))
.apply(plaintext_reader, actual_size)
.expect("native protected encrypted writer");
let mut reader = PutObjReader::new(encrypted_reader);
let info = set_disks
.put_object_part(bucket, object, &upload.upload_id, index + 1, &mut reader, &ObjectOptions::default())
.await
.expect("stored part should be staged");
assert_eq!(info.size as i64, stored_len, "a part is stored in its encoded length");
assert!(info.size as i64 > actual_size, "part contains actual encryption overhead");
uploaded_parts.push(CompletePart {
part_num: info.part_num,
etag: info.etag,
@@ -17240,8 +17250,12 @@ mod transition_commit_failure_tests {
"the fixture must keep the two coordinate systems apart: stored {size} vs plaintext {actual_size}"
);
}
let stored_before = read_stored_representation(&set_disks, bucket, object).await;
assert_stored_representation_eq(&stored_before, &expected_stored, "the fixture must store its encoded bytes verbatim");
assert_eq!(
rustfs_utils::http::get_consistent_str(&original.user_defined, SUFFIX_OPAQUE_ENCRYPTED_ETAG),
Some("v1")
);
assert!(etag_can_be_listed(&original.user_defined), "the source has a real producer-attested ETag");
let expected_stored = read_stored_representation(&set_disks, bucket, object).await;
let tier_name = format!("COLDTIER{}", &Uuid::new_v4().simple().to_string()[..8]).to_uppercase();
register_mock_tier(&runtime_sources::global_tier_config_mgr(), &tier_name).await;
@@ -17265,6 +17279,16 @@ mod transition_commit_failure_tests {
.await
.expect("multipart source should transition before restore");
let transitioned = set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("read metadata after actual tier transition");
assert_eq!(
rustfs_utils::http::get_consistent_str(&transitioned.user_defined, SUFFIX_OPAQUE_ENCRYPTED_ETAG),
Some("v1")
);
assert!(etag_can_be_listed(&transitioned.user_defined), "transition retains the source proof");
let operation_id = Uuid::new_v4();
set_disks
.put_object_metadata(
@@ -17310,6 +17334,11 @@ mod transition_commit_failure_tests {
// backlog#2369 P7.1: the copy-back digests stored bytes, so the object's
// public ETag has to be carried over rather than recomputed.
assert_eq!(restored.etag, original.etag, "restore must preserve the object ETag");
assert_eq!(restored.version_id, original.version_id, "restore keeps the same version");
assert!(
etag_can_be_listed(&restored.user_defined),
"the restored version retains its protected ETag proof"
);
}
#[tokio::test]
@@ -20432,7 +20461,7 @@ mod transition_upload_integrity_tests {
assert_local_source_intact(&set_disks, bucket, object, &payload).await;
}
#[tokio::test(flavor = "multi_thread")]
#[tokio::test(flavor = "current_thread")]
#[serial_test::serial]
async fn commit_lock_acquire_failure_cleans_remote_candidate_and_preserves_source() {
let manager = Arc::new(rustfs_lock::GlobalLockManager::new());
@@ -20441,7 +20470,16 @@ mod transition_upload_integrity_tests {
let bucket = "transition-lock-acquire-failure-bucket";
let object = "object.bin";
let payload = b"transition commit lock failure must clean the remote candidate".repeat(1024);
let original = write_source(&set_disks, &disk_stores, bucket, object, &payload).await;
let original = temp_env::async_with_vars(
[(
crate::set_disk::core::io_primitives::ENV_RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE,
Some("false"),
)],
async { write_source(&set_disks, &disk_stores, bucket, object, &payload).await },
)
.await;
// Complete source I/O before controlling the quorum retry deadline.
tokio::time::pause();
let tier_name = format!("COLDTIER{}", &Uuid::new_v4().simple().to_string()[..8]).to_uppercase();
let backend = register_mock_tier(&runtime_sources::global_tier_config_mgr(), &tier_name).await;
let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::DistErasure).await;
@@ -20453,13 +20491,8 @@ mod transition_upload_integrity_tests {
.await
.expect_err("transition must fail when the commit namespace write lock cannot reach quorum");
assert!(
matches!(
error,
StorageError::NamespaceLockQuorumUnavailable { .. }
| StorageError::Lock(LockError::QuorumNotReached { .. })
| StorageError::Lock(LockError::Internal { .. })
),
"unexpected transition lock-acquire error: {error:?}"
matches!(error, StorageError::NamespaceLockQuorumUnavailable { required: 2, .. }),
"the injected permanent quorum failure must reach the transition commit boundary: {error:?}"
);
assert_eq!(
backend.put_count().await,
+39 -2
View File
@@ -2979,10 +2979,11 @@ fn build_legacy_per_part_fallback_reader(
part_length: usize,
part_size: usize,
read_repair: Option<(DecodeReadRepairContext, bool)>,
) -> Box<dyn AsyncRead + Unpin + Send + Sync> {
) -> Box<LegacyPerPartDecodeReader> {
let buffer = adaptive_duplex_buffer_size(part_size as i64);
let (read_half, mut write_half) = tokio::io::duplex(buffer);
let (read_half, write_half) = tokio::io::duplex(buffer);
let decode = tokio::spawn(async move {
let mut write_half = GetObjectDownstreamWriter::new(write_half);
let outcome = erasure
.decode_with_stripe_handles_and_reopeners_with_diagnostics(
&mut write_half,
@@ -8908,6 +8909,42 @@ mod tests {
assert!(err.to_string().contains("inconsistent read source shards"));
}
#[tokio::test]
async fn legacy_per_part_output_close_retains_downstream_classification() {
let erasure = coding::Erasure::new(2, 2, 64);
let data = (0..64u8).collect::<Vec<_>>();
let setup =
setup_codec_data_blocks_first_encoded_bitrot_readers(&erasure, &data, &[0], &[], &[], HashAlgorithm::HighwayHash256)
.await;
let mut reader = build_legacy_per_part_fallback_reader(
erasure,
setup.readers,
setup.deferred_stripe_handles,
Vec::new(),
None,
0,
data.len(),
data.len(),
None,
);
// Closing the actual output pipe models the body consumer disappearing
// while retaining the executor long enough to inspect its typed result.
assert!(!reader.finished);
let (replacement, _unused) = tokio::io::duplex(1);
drop(std::mem::replace(&mut reader.inner, replacement));
let handle = reader.decode.take().expect("fallback decode task");
let error = handle
.await
.expect("join the fallback task")
.expect("closed pipe must interrupt output");
assert_eq!(error.kind(), std::io::ErrorKind::BrokenPipe);
assert_eq!(
crate::diagnostics::get::classify_io_error(&error),
GetObjectFailureReason::DownstreamClosed,
"fallback output must carry the same marker as the normal GET path"
);
}
// backlog#879: a later multipart part that loses a shard (read quorum still
// holds) must be reconstructed in place through the legacy per-part decode
// bridge instead of failing the stream. This exercises the exact reader the
+27 -19
View File
@@ -527,7 +527,7 @@ impl ECStore {
async fn acquire_heal_format_fence(
&self,
) -> Result<(
tokio::sync::MutexGuard<'_, PoolMetaWriteState>,
crate::core::pools::PoolMetaSaveGateWriteGuard<'_>,
NamespaceLockGuard,
NamespaceLockGuard,
PoolMeta,
@@ -538,7 +538,7 @@ impl ECStore {
.first()
.cloned()
.ok_or_else(|| Error::other("heal format requires at least one storage pool"))?;
let mut write_state = self.pool_meta_save_gate.lock().await;
let mut write_state = self.pool_meta_save_gate.write().await;
write_state.ensure_write_safe("heal format fence failed")?;
// Metadata fence order is part of the decommission/rebalance protocol:
@@ -1079,7 +1079,7 @@ impl ECStore {
pub(super) async fn handle_heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
let movement_gate = self.ctx.data_movement_operation_gate();
let _movement_guard = movement_gate.read().await;
let save_guard = self.pool_meta_save_gate.lock().await;
let save_guard = self.pool_meta_save_gate.write().await;
save_guard.ensure_write_safe("bucket heal cannot run while pool metadata requires recovery")?;
let mut fenced_pools = BTreeSet::new();
{
@@ -2527,7 +2527,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx: crate::runtime::instance::bootstrap_ctx(),
bucket_fence_registry: std::sync::Arc::default(),
}
@@ -3550,10 +3550,14 @@ mod tests {
async fn full_format_heal_preblocked_pool_metadata_never_writes_format() {
let (temp_dir, store, shutdown) = multi_pool_heal_store().await;
remove_heal_test_format(&temp_dir, &store, 0, 3).await;
store.pool_meta_save_gate.lock().await.observe_replicas(PoolMetaReplicaState {
needs_repair: true,
repair_write_safe: false,
});
store
.pool_meta_save_gate
.write()
.await
.observe_replicas(PoolMetaReplicaState {
needs_repair: true,
repair_write_safe: false,
});
let err = store
.handle_heal_format(false)
@@ -3688,7 +3692,7 @@ mod tests {
let target = remove_heal_test_format(&temp_dir, &store, 0, 3).await;
store.pools[0].disk_set[0].disks.write().await[3] = None;
let barrier = HealFormatAfterSaveBarrier::install(&store.pools[0], 3);
let recovery_latch = store.pool_meta_save_gate.lock().await.aborted_transaction_latch_for_test();
let recovery_latch = store.pool_meta_save_gate.write().await.aborted_transaction_latch_for_test();
let mut heal = tokio::spawn({
let store = Arc::clone(&store);
async move { store.handle_heal_format(false).await }
@@ -3848,7 +3852,7 @@ mod tests {
}
}
let endpoints: EndpointServerPools = pools.iter().map(|pool| pool.endpoints.clone()).collect::<Vec<_>>().into();
let state = source.pool_meta_save_gate.lock().await.independent_clone_for_test();
let state = source.pool_meta_save_gate.write().await.independent_clone_for_test();
Arc::new(ECStore {
id: Uuid::new_v4(),
disk_map: Default::default(),
@@ -3858,7 +3862,7 @@ mod tests {
rebalance_meta: tokio::sync::RwLock::new(None),
decommission_cancelers: tokio::sync::RwLock::new(vec![None; 2]),
start_gate: tokio::sync::Mutex::new(()),
pool_meta_save_gate: tokio::sync::Mutex::new(state),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::new(state),
ctx: Arc::clone(&source.ctx),
bucket_fence_registry: Arc::default(),
})
@@ -4040,7 +4044,7 @@ mod tests {
let missing = remove_pool_meta_shard(&store, 0).await;
let (guard, admissions, mut state) = store.acquire_pool_meta_object_heal_fence(&[0]).await.expect("heal authority");
assert!(admissions.into_iter().all(|admission| admission.is_ok()));
assert!(store.pool_meta_save_gate.try_lock().is_err(), "heal must retain the earlier save gate");
assert!(store.pool_meta_save_gate.try_write().is_err(), "heal must retain the earlier save gate");
let writer_store = Arc::clone(&store);
let writer = tokio::spawn(async move { writer_store.save_current_pool_meta_for_test(&[0]).await });
tokio::task::yield_now().await;
@@ -4687,10 +4691,14 @@ mod tests {
#[tokio::test]
async fn bucket_heal_blocks_before_dispatch_after_unreadable_pool_meta_replica() {
let store = minimal_heal_store().await;
store.pool_meta_save_gate.lock().await.observe_replicas(PoolMetaReplicaState {
needs_repair: true,
repair_write_safe: false,
});
store
.pool_meta_save_gate
.write()
.await
.observe_replicas(PoolMetaReplicaState {
needs_repair: true,
repair_write_safe: false,
});
let err = store
.handle_heal_bucket("bucket", &HealOpts::default())
@@ -4724,7 +4732,7 @@ mod tests {
.await
.expect("peer dispatch should start");
assert!(
store.pool_meta_save_gate.try_lock().is_ok(),
store.pool_meta_save_gate.try_write().is_ok(),
"coordinator must release its local save gate before waiting for peers"
);
assert!(
@@ -5116,7 +5124,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx: crate::runtime::instance::bootstrap_ctx(),
bucket_fence_registry: std::sync::Arc::default(),
};
@@ -5136,7 +5144,7 @@ mod tests {
.save_for_startup(store.pools.clone())
.await
.expect("pool metadata should be persisted before format heal");
*store.pool_meta_save_gate.lock().await = PoolMetaWriteState::default();
*store.pool_meta_save_gate.write().await = crate::core::pools::PoolMetaWriteState::default();
let replacement_endpoint = store.pools[1].endpoints.endpoints.as_ref()[2].clone();
tokio::fs::remove_file(
+8 -2
View File
@@ -676,7 +676,7 @@ impl ECStore {
rebalance_meta: RwLock::new(None),
decommission_cancelers,
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::new(pool_meta_write_state),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::new(pool_meta_write_state),
// Adopt the caller's context (the process bootstrap one on the
// legacy path) so startup writes (erasure type recorded before
// this point) and later reads share one cell.
@@ -725,7 +725,7 @@ impl ECStore {
// Lock order is save gate -> distributed metadata writer, matching
// runtime mutations. RPC locks are available before ECStore publication
// and do not depend on the later fleet proof.
let mut startup_write_state = self.pool_meta_save_gate.lock().await;
let mut startup_write_state = self.pool_meta_save_gate.write().await;
let startup_pool_meta_guard = if should_persist_pool_meta {
let pool = self
.pools
@@ -24035,6 +24035,7 @@ mod tests {
without_storage_class_env(build_isolated_test_store(temp_dir.path(), "tier-mutation-intent-record", &[4])).await;
let mutation_id = uuid::Uuid::new_v4();
let intent = TierMutationIntent {
requires_native_azure: false,
mutation_id,
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -24220,6 +24221,7 @@ mod tests {
let (_ctx, store, _shutdown) =
without_storage_class_env(build_isolated_test_store(temp_dir.path(), "tier-mutation-intent-scan", &[4])).await;
let build_intent = |mutation_id: uuid::Uuid, tier_name: &str| TierMutationIntent {
requires_native_azure: false,
mutation_id,
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -24281,6 +24283,7 @@ mod tests {
without_storage_class_env(build_isolated_test_store(temp_dir.path(), "tier-mutation-intent-advance", &[4])).await;
let mutation_id = uuid::Uuid::new_v4();
let intent = TierMutationIntent {
requires_native_azure: false,
mutation_id,
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -24355,6 +24358,7 @@ mod tests {
let abort_id = uuid::Uuid::new_v4();
let abort_intent = TierMutationIntent {
requires_native_azure: false,
mutation_id: abort_id,
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -24397,6 +24401,7 @@ mod tests {
candidate_digest: [u8; 32],
) -> TierMutationIntent {
TierMutationIntent {
requires_native_azure: false,
mutation_id,
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -25473,6 +25478,7 @@ mod tests {
let (_ctx, store, _shutdown) =
without_storage_class_env(build_isolated_test_store(temp_dir.path(), "tier-mutation-intent-scan-page", &[4])).await;
let build_intent = |mutation_id: uuid::Uuid, tier_name: &str| TierMutationIntent {
requires_native_azure: false,
mutation_id,
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -318,7 +318,10 @@ mod multipart_copy_heal_tests {
let mut corrupt = original.clone();
corrupt[0] ^= 0x80;
std::fs::write(&shard, &corrupt).expect("actual bitrot corruption in shared physical shard");
std::fs::File::open(&shard)
// Windows only flushes a handle opened for writing.
std::fs::OpenOptions::new()
.write(true)
.open(&shard)
.expect("open corrupted actual shard")
.sync_all()
.expect("persist actual fault bytes");
+309 -18
View File
@@ -127,28 +127,30 @@ impl AuthoritativeListOperations for ECStore {
marker: Option<String>,
max_keys: i32,
) -> Result<ListObjectVersionsInfo> {
if max_keys <= 0 || max_keys > MAX_OBJECT_LIST {
self.list_unversioned_authoritative_page(bucket, prefix, marker, max_keys, None)
.await
}
async fn list_identifier_tree_leaves_authoritative(
self: Arc<Self>,
bucket: &str,
prefix: &str,
leaf: &str,
marker: Option<String>,
max_keys: i32,
) -> Result<ListObjectVersionsInfo> {
let valid_prefix = prefix
.strip_suffix('/')
.is_some_and(|root| !root.is_empty() && root.split('/').all(|part| !part.is_empty() && !matches!(part, "." | "..")));
if !valid_prefix || !crate::disk::is_identifier_tree_leaf(leaf) {
return Err(StorageError::InvalidArgument(
bucket.to_string(),
prefix.to_string(),
"authoritative page size is out of range".to_string(),
"identifier-tree listing requires a directory prefix and a dotted leaf name".to_string(),
));
}
let opts = ListPathOptions {
bucket: bucket.to_string(),
prefix: prefix.to_string(),
marker: marker.clone(),
limit: max_keys,
recursive: true,
incl_deleted: true,
versioned: true,
authoritative: true,
caller: ListObjectsGatherCaller::S3ListVersions,
..Default::default()
};
let mut page = self.list_path(&opts).await?;
page.entries.forward_past(marker);
unversioned_authoritative_page(bucket, page)
self.list_unversioned_authoritative_page(bucket, prefix, marker, max_keys, Some(leaf.to_string()))
.await
}
async fn walk_authoritative(
@@ -368,6 +370,9 @@ fn log_list_path_finished(
#[derive(Debug, Default, Clone)]
pub struct ListPathOptions {
pub caller: ListObjectsGatherCaller,
/// Internal native-walk admission, shared with every disk producer so
/// aborting an HTTP request cannot release its budget ahead of cleanup.
pub read_budget: Option<Arc<tokio::sync::OwnedSemaphorePermit>>,
pub id: Option<String>,
// Bucket of the listing.
@@ -433,6 +438,8 @@ pub struct ListPathOptions {
// Internal bounded pages require GET-equivalent per-set resolution.
pub authoritative: bool,
pub identifier_tree_leaf: Option<String>,
pub stop_disk_at_limit: bool,
pub pool_idx: Option<usize>,
@@ -1098,6 +1105,39 @@ impl ListPathOptions {
}
impl ECStore {
async fn list_unversioned_authoritative_page(
self: Arc<Self>,
bucket: &str,
prefix: &str,
marker: Option<String>,
max_keys: i32,
identifier_tree_leaf: Option<String>,
) -> Result<ListObjectVersionsInfo> {
if max_keys <= 0 || max_keys > MAX_OBJECT_LIST {
return Err(StorageError::InvalidArgument(
bucket.to_string(),
prefix.to_string(),
"authoritative page size is out of range".to_string(),
));
}
let opts = ListPathOptions {
bucket: bucket.to_string(),
prefix: prefix.to_string(),
marker: marker.clone(),
limit: max_keys,
recursive: true,
incl_deleted: true,
versioned: true,
authoritative: true,
identifier_tree_leaf,
caller: ListObjectsGatherCaller::S3ListVersions,
..Default::default()
};
let mut page = self.list_path(&opts).await?;
page.entries.forward_past(marker);
unversioned_authoritative_page(bucket, page)
}
#[allow(clippy::too_many_arguments)]
// @continuation_token marker
// @start_after as marker when continuation_token empty
@@ -1478,6 +1518,11 @@ impl ECStore {
if o.transient {
o.create = false;
}
let fanout = self.pools.iter().fold(0usize, |total, pool| {
total.saturating_add(pool.disk_set.len().saturating_mul(pool.set_drive_count))
});
o.read_budget = self.ctx.acquire_list_read_budget(fanout).await.map_err(Error::other)?;
let log_context = ListPathLogContext::from_options(&o);
// cancel channel
@@ -2878,6 +2923,12 @@ impl SetDisks {
if o.transient {
o.create = false;
}
o.read_budget = self
.instance_ctx()
.acquire_list_read_budget(self.set_drive_count)
.await
.map_err(Error::other)?;
let log_context = ListPathLogContext::from_options(&o);
let cancel = CancellationToken::new();
@@ -2922,6 +2973,33 @@ impl SetDisks {
Ok(page)
}
fn resolve_authoritative_listed_entry(
bucket: &str,
selection: Option<(&str, &str)>,
entries: MetaCacheEntries,
errors: &[Option<DiskError>],
disk_count: usize,
parity: usize,
) -> crate::disk::error::Result<Option<MetaCacheEntry>> {
if disk_count == 0 || entries.0.len() > disk_count || entries.0.len() != errors.len() {
return Err(DiskError::ErasureReadQuorum);
}
// An old peer may ignore the optional disk selector. Its noncandidate
// rows must be ignored before metadata parsing. Selected keys retain
// the original resolver and all of its quorum/corruption checks.
if selection.is_some_and(|(prefix, leaf)| {
entries
.0
.iter()
.flatten()
.next()
.is_some_and(|entry| !crate::disk::identifier_tree_leaf_matches(prefix, leaf, &entry.name))
}) {
return Ok(None);
}
Self::resolve_listed_versions(bucket, entries, errors, disk_count, parity)
}
/// Version scan for set-scoped callers that act on absence: set walks
/// (tier reference proofs, tier-delete journals, pool maintenance) and the
/// recursive prefix-delete scan. Every online drive is read and each
@@ -2945,10 +3023,13 @@ impl SetDisks {
let first_error = Arc::new(tokio::sync::Mutex::new(None));
let error_sink = Arc::clone(&first_error);
let bucket = opts.bucket.clone();
let prefix = opts.prefix.clone();
let identifier_tree_leaf = opts.identifier_tree_leaf.clone();
let cancel = rx.clone();
let result = list_path_raw_with_claim_tracker(
rx,
ListPathRawOptions {
read_budget: opts.read_budget.clone(),
disks: disks.into_iter().map(Some).collect(),
bucket: opts.bucket,
path: opts.base_dir,
@@ -2961,6 +3042,7 @@ impl SetDisks {
forward_to: opts.marker,
min_disks: read_quorum,
preserve_replica_metadata: true,
identifier_tree_leaf: opts.identifier_tree_leaf,
// A reader's local page may be consumed by stale entries. Only
// the resolved, merged page may stop the authoritative walk.
per_disk_limit: 0,
@@ -2968,7 +3050,14 @@ impl SetDisks {
walkdir_timeout: opts.walkdir_timeout,
walkdir_stall_timeout: opts.walkdir_stall_timeout,
partial: Some(Box::new(move |entries, errors| {
let resolved = Self::resolve_listed_versions(&bucket, entries, errors, disk_count, parity);
let resolved = Self::resolve_authoritative_listed_entry(
&bucket,
identifier_tree_leaf.as_deref().map(|leaf| (prefix.as_str(), leaf)),
entries,
errors,
disk_count,
parity,
);
let sender = sender.clone();
let cancel = cancel.clone();
let first_error = Arc::clone(&error_sink);
@@ -3107,6 +3196,7 @@ impl SetDisks {
let result = list_path_raw_with_outcome(
rx,
ListPathRawOptions {
read_budget: opts.read_budget.clone(),
disks: disks.into_iter().map(Some).collect(),
fallback_disks: fallback_disks.into_iter().map(Some).collect(),
bucket: opts.bucket,
@@ -6788,6 +6878,58 @@ mod test {
(dirs, store)
}
#[tokio::test]
async fn list_read_budget_bounds_store_and_set_entrypoints_and_recovers_after_errors() {
let (_dirs, store) = single_set_store(4).await;
let held = store
.ctx
.acquire_list_read_budget(usize::MAX)
.await
.expect("occupy native budget");
let semaphore = held.as_ref().expect("bounded instance budget").semaphore();
let owners = Arc::strong_count(semaphore);
let opts = ListPathOptions {
bucket: "list-budget-absent-bucket".into(),
limit: 1,
authoritative: true,
..Default::default()
};
let set = store.pools[0].disk_set[0].clone();
set.disks.write().await.fill(None);
{
let mut store_walk = std::pin::pin!(store.clone().list_path(&opts));
let mut set_walk = std::pin::pin!(set.clone().list_path_result(&opts));
assert!(futures::poll!(&mut store_walk).is_pending(), "store walk must await its budget");
assert_eq!(Arc::strong_count(semaphore), owners + 1, "store must queue on the shared budget");
assert!(futures::poll!(&mut set_walk).is_pending(), "set walk shares the same instance budget");
assert_eq!(Arc::strong_count(semaphore), owners + 2, "set must queue on the same budget");
let empty = store
.clone()
.list_path(&ListPathOptions {
limit: 0,
..opts.clone()
})
.await
.expect("empty page needs no disk admission");
assert!(empty.exhausted);
}
assert_eq!(Arc::strong_count(semaphore), owners, "cancelled entry points release the queue ownership");
drop(held);
let result = timeout(Duration::from_secs(5), store.clone().list_path(&opts))
.await
.expect("failed walk terminates");
assert_eq!(
result.err(),
Some(DiskError::ErasureReadQuorum.into()),
"offline drives retain their quorum error"
);
let next = timeout(Duration::from_secs(5), store.ctx.acquire_list_read_budget(usize::MAX))
.await
.expect("failed and cancelled walks release their grants")
.expect("budget remains open");
assert!(next.is_some());
}
fn versioned_object_metadata(bucket: &str, name: &str, versions: &[(u128, i64)], slot: usize) -> Bytes {
let mut meta = FileMeta::new();
for (id, modified) in versions {
@@ -6929,6 +7071,155 @@ mod test {
);
}
#[test]
fn authoritative_identifier_tree_filters_old_peer_rows_before_metadata_resolution() {
let prefix = "table-catalog/namespaces/";
let leaf = "table-entry.json";
let corrupt = |name: &str| {
MetaCacheEntries(vec![
Some(MetaCacheEntry {
name: format!("{prefix}{name}"),
metadata: b"corrupt metadata".to_vec(),
..Default::default()
});
4
])
};
let report = "sales/tables/orders/maintenance/hash/jobs/job.json";
let errors = vec![None; 4];
assert!(
SetDisks::resolve_authoritative_listed_entry(RUSTFS_META_BUCKET, None, corrupt(report), &errors, 4, 2,).is_err(),
"the control must actually parse and reject the corrupt metadata"
);
assert!(
SetDisks::resolve_authoritative_listed_entry(
RUSTFS_META_BUCKET,
Some((prefix, leaf)),
corrupt(report),
&errors,
4,
2,
)
.expect("an old peer's unrelated report is outside the selected scan")
.is_none()
);
assert!(
SetDisks::resolve_authoritative_listed_entry(
RUSTFS_META_BUCKET,
Some((prefix, leaf)),
corrupt("sales/tables/orders/maintenance/tables/owner/table-entry.json"),
&errors,
4,
2,
)
.is_err(),
"a matching nested owner must still fail closed on corruption"
);
}
#[tokio::test]
async fn authoritative_identifier_tree_pages_preserve_nested_owners_and_read_quorum() {
let (dirs, store) = single_set_store(4).await;
let prefix = "table-catalog/identifier-tree/";
let leaf = "table-entry.json";
for (root, selected) in [
("table-catalog/identifier-tree", leaf),
(prefix, "../entry.json"),
(prefix, "table-entry"),
] {
assert!(matches!(
store
.clone()
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, root, selected, None, 2)
.await,
Err(StorageError::InvalidArgument(..))
));
}
let names = [
format!("{prefix}a/tables/first/{leaf}"),
format!("{prefix}sales/tables/orders/maintenance/tables/owner/{leaf}"),
format!("{prefix}z/tables/last/{leaf}"),
];
for name in &names {
write_version_replicas(&dirs, RUSTFS_META_BUCKET, name, |slot| Some(unversioned_catalog_metadata(name, slot))).await;
}
for name in [
format!("{prefix}sales"),
format!("{prefix}sales/tables/orders/maintenance/hash/jobs/job.json"),
] {
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &name, |_| Some(b"corrupt noncandidate metadata".to_vec())).await;
}
assert!(
store
.clone()
.list_unversioned_objects_authoritative(RUSTFS_META_BUCKET, prefix, None, 100)
.await
.is_err(),
"the ordinary authoritative scan must expose the corrupt control objects"
);
let first = store
.clone()
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, None, 2)
.await
.expect("noncandidate metadata must not be read");
assert!(first.is_truncated);
assert_eq!(
first.objects.iter().map(|object| &object.name).collect::<Vec<_>>(),
names[..2].iter().collect::<Vec<_>>()
);
let second = store
.clone()
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, first.next_marker, 2)
.await
.expect("the next page must retain the final owner");
assert!(!second.is_truncated);
assert_eq!(second.objects.iter().map(|object| &object.name).collect::<Vec<_>>(), vec![&names[2]]);
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &names[0], |_| Some(b"corrupt selected metadata".to_vec())).await;
assert!(
store
.clone()
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, None, 2)
.await
.is_err(),
"corruption at a selected key cannot be skipped to fill the page"
);
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &names[0], |slot| {
Some(unversioned_catalog_metadata(&names[0], slot))
})
.await;
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &names[0], |slot| {
Some(versioned_object_metadata(RUSTFS_META_BUCKET, &names[0], &[(2, 1_700_000_002)], slot).to_vec())
})
.await;
assert!(
store
.clone()
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, None, 2)
.await
.is_err(),
"a versioned ownership entry cannot be accepted as an unversioned proof"
);
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &names[0], |slot| {
Some(unversioned_catalog_metadata(&names[0], slot))
})
.await;
let set = &store.pools[0].disk_set[0];
for disk in set.disks.write().await.iter_mut().skip(1) {
*disk = None;
}
assert!(
matches!(
store
.clone()
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, None, 2)
.await,
Err(StorageError::ErasureReadQuorum)
),
"unavailable replicas cannot prove owner absence"
);
}
#[tokio::test]
async fn authoritative_store_walk_includes_reserved_prefix_versions() {
let bucket = "catalog-cleanup-versions";
+8 -6
View File
@@ -34,7 +34,7 @@ use crate::bucket::utils::check_put_object_part_args;
use crate::bucket::utils::{check_valid_bucket_name, check_valid_bucket_name_strict, is_meta_bucketname};
use crate::cluster::rpc::{RemoteClient, S3PeerSys};
use crate::config::storageclass;
use crate::core::pools::{DecommissionCanceler, PoolMeta, PoolMetaWriteState};
use crate::core::pools::{DecommissionCanceler, PoolMeta, PoolMetaSaveGate};
use crate::disk::endpoint::{Endpoint, EndpointType};
use crate::disk::{DiskAPI, DiskInfo, DiskInfoOptions};
use crate::error::{Error, Result};
@@ -480,9 +480,13 @@ pub struct ECStore {
/// Serializes full-document pool metadata saves and retains a fail-closed
/// write block after startup observes an unreadable replica.
///
/// Immutable admission proofs share authority; publication, recovery and
/// authority changes retain exclusive access through their existing I/O.
/// Each writer publishes its final state on release, and health
/// inspection reads that committed state without acquiring the gate.
/// Lock order: acquire `pool_meta_save_gate`, then the distributed
/// `pool.bin` fence, then clone `pool_meta` under a short read lock.
pub(crate) pool_meta_save_gate: Mutex<PoolMetaWriteState>,
pub(crate) pool_meta_save_gate: PoolMetaSaveGate,
/// Per-instance runtime state (Phase 5, backlog#939).
///
/// Carries this instance's identity/runtime out of the process globals so
@@ -555,7 +559,6 @@ impl Default for ScannerDataMovementPauseStatus {
#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize)]
pub struct PoolMetaWriteGateStatus {
pub writes_ready: bool,
pub check_timed_out: bool,
pub write_blocked: bool,
pub transaction_aborted: bool,
pub pool_meta_absent: bool,
@@ -571,7 +574,6 @@ impl Default for PoolMetaWriteGateStatus {
fn default() -> Self {
Self {
writes_ready: true,
check_timed_out: false,
write_blocked: false,
transaction_aborted: false,
pool_meta_absent: false,
@@ -1936,7 +1938,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx,
bucket_fence_registry: Arc::default(),
};
@@ -2011,7 +2013,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx,
bucket_fence_registry: Arc::default(),
})
+1 -1
View File
@@ -2133,7 +2133,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx: crate::runtime::instance::bootstrap_ctx(),
bucket_fence_registry: std::sync::Arc::default(),
}
+20 -5
View File
@@ -3218,7 +3218,12 @@ impl ECStore {
pub async fn object_lock_config_snapshot(&self, bucket: &str) -> Result<Arc<ObjectLockConfigSnapshot>> {
check_valid_bucket_name(bucket)?;
let lifecycle_guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
let metadata_guard = acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await?;
let metadata_guard = crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_read_lock_in_for_purpose(
&self.ctx,
bucket,
"object_lock_admission_metadata_read",
)
.await?;
let (state, bucket_incarnation_id, config_revision) =
get_object_lock_config_and_incarnation_from_disk_with_guard_in(&self.ctx, bucket, Some(&lifecycle_guard)).await?;
if lifecycle_guard.is_lock_lost() || metadata_guard.is_lock_lost() {
@@ -3238,7 +3243,12 @@ impl ECStore {
pub(crate) async fn object_lock_config_snapshot_for_single_put(&self, bucket: &str) -> Result<Arc<ObjectLockConfigSnapshot>> {
check_valid_bucket_name(bucket)?;
let lifecycle_guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
let metadata_guard = acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await?;
let metadata_guard = crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_read_lock_in_for_purpose(
&self.ctx,
bucket,
"object_lock_snapshot_metadata_read",
)
.await?;
let (state, bucket_incarnation_id, config_revision, quota_res) =
get_object_lock_and_quota_config_from_disk_with_guard_in(&self.ctx, bucket, Some(&lifecycle_guard)).await?;
if lifecycle_guard.is_lock_lost() || metadata_guard.is_lock_lost() {
@@ -3267,7 +3277,12 @@ impl ECStore {
if lifecycle_fence.is_lock_lost() {
return Err(Error::other("bucket lifecycle lock was lost before loading the Object Lock snapshot"));
}
let metadata_guard = acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await?;
let metadata_guard = crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_read_lock_in_for_purpose(
&self.ctx,
bucket,
"object_lock_completion_metadata_read",
)
.await?;
let (state, bucket_incarnation_id, config_revision) =
get_object_lock_config_and_incarnation_from_disk_with_guard_in(&self.ctx, bucket, Some(&metadata_guard)).await?;
if lifecycle_fence.is_lock_lost() || metadata_guard.is_lock_lost() {
@@ -9988,7 +10003,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx: crate::runtime::instance::bootstrap_ctx(),
bucket_fence_registry: std::sync::Arc::default(),
}
@@ -10080,7 +10095,7 @@ mod tests {
rebalance_meta: RwLock::new(None),
decommission_cancelers: RwLock::new(Vec::new()),
start_gate: Mutex::new(()),
pool_meta_save_gate: Mutex::default(),
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
ctx,
bucket_fence_registry: std::sync::Arc::default(),
}
+160 -2
View File
@@ -445,6 +445,13 @@ impl ECStore {
return Ok(());
}
let scoped_cleanup = opts
.scanner_publication_commit_scope
.as_ref()
.zip(opts.http_preconditions.as_ref())
.filter(|_| bucket == RUSTFS_META_BUCKET && opts.delete_prefix_object);
let _cleanup_outcome =
scoped_cleanup.map(|(scope, _)| crate::object_api::ScannerPublicationCommitScopeGuard::new(scope.clone()));
let mut first_error = None;
let mut first_volume_error = None;
let mut has_success = false;
@@ -457,9 +464,30 @@ impl ECStore {
}
let mut opts = opts.clone();
opts.delete_prefix = true;
if let Some((scope, preconditions)) = scoped_cleanup {
// The exact-object delete retains its namespace writer across
// this lookup and dispatch. Never remove an unobserved revision.
let lookup = ObjectOptions {
no_lock: true,
suppress_read_repair: true,
preserve_delete_marker: true,
..Default::default()
};
let disks = pool.get_disks_by_key(object);
let current = disks.get_object_info(bucket, object, &lookup).await;
if let Some(error) =
crate::set_disk::SetDisks::evaluate_write_precondition(current, preconditions, bucket, object, &lookup)
{
if matches!(error, StorageError::PreconditionFailed) || crate::error::is_err_object_not_found(&error) {
continue;
}
return Err(error);
}
opts.scanner_publication_commit_scope = Some(scope.pool_delete_participant());
}
match pool.delete_object(bucket, object, opts).await {
Ok(_) => has_success = true,
Err(err) if is_err_strict_volume_not_found(&err) => {
Err(err) if scoped_cleanup.is_none() && is_err_strict_volume_not_found(&err) => {
if first_volume_error.is_none() {
first_volume_error = Some(err);
}
@@ -472,14 +500,21 @@ impl ECStore {
}
}
match first_error {
let result = match first_error {
Some(err) => Err(err),
None if has_success => Ok(()),
None => match first_volume_error {
Some(err) => Err(err),
None if scoped_cleanup.is_some() => Err(StorageError::PreconditionFailed),
None => Ok(()),
},
};
if let Some((scope, _)) = scoped_cleanup
&& result.is_ok()
{
let _ = scope.mark_committed();
}
result
}
pub(super) async fn get_available_pool_idx(&self, bucket: &str, object: &str, size: i64) -> Option<usize> {
@@ -1379,6 +1414,129 @@ mod tests {
(temp_dir, store, shutdown)
}
async fn run_scanner_observed_cleanup(
if_match_override: Option<&str>,
second_payload: Option<&[u8]>,
lose_second_quorum: bool,
) -> (
Result<crate::object_api::ObjectInfo>,
crate::object_api::ScannerPublicationCommitState,
Vec<bool>,
) {
use crate::config::com::save_config_with_opts_and_metadata;
use crate::storage_api_contracts::object::HTTPPreconditions;
use std::time::Duration;
let (_directories, store, shutdown) = setup_multi_pool_test_store("scanner-observed-cleanup", &[4, 4]).await;
let object = "buckets/.usage.observed.json";
let mut tags = Vec::new();
let old_payload = b"old observed snapshot";
for (pool, payload) in store.pools.iter().zip([Some(old_payload.as_slice()), second_payload]) {
let Some(payload) = payload else { continue };
let info = save_config_with_opts_and_metadata(pool.clone(), object, payload.to_vec(), &ObjectOptions::default())
.await
.expect("seed an observed snapshot on each pool");
tags.push(info.etag.expect("seed has CAS tag"));
}
if let Some(second_payload) = second_payload {
assert_eq!(tags[0] == tags[1], old_payload == second_payload);
}
let epoch = store.scanner_data_usage_publication_epoch().await.expect("publication epoch");
let scope = store
.scanner_data_usage_publication_commit_scope(epoch, tokio::time::Instant::now() + Duration::from_secs(30), Vec::new())
.await
.expect("admit observed cleanup");
let disconnected = if lose_second_quorum {
let pool = &store.pools[1];
tokio::time::timeout(Duration::from_secs(10), pool.stop_endpoint_monitor_for_test())
.await
.expect("fixture must own the disk slots");
let set = pool.get_disks_by_key(object);
let original = {
let mut disks = set.disks.write().await;
let offline = vec![None; disks.len()];
std::mem::replace(&mut *disks, offline)
};
Some((set, original))
} else {
None
};
let result = store
.delete_object(
crate::disk::RUSTFS_META_BUCKET,
object,
ObjectOptions {
delete_prefix: true,
delete_prefix_object: true,
http_preconditions: Some(HTTPPreconditions {
if_match: Some(if_match_override.map_or_else(|| tags.remove(0), str::to_owned)),
..Default::default()
}),
scanner_publication_commit_scope: Some(scope.clone()),
..Default::default()
},
)
.await;
if let Some((set, disks)) = disconnected {
*set.disks.write().await = disks;
assert!(!scope.release_movement_permit().await, "partial cleanup cannot release its permit");
}
let mut still_present = Vec::new();
for pool in &store.pools {
still_present.push(
read_config_no_lock_preserve_empty_with_metadata(pool.clone(), object)
.await
.is_ok(),
);
}
shutdown.cancel();
(result, scope.state(), still_present)
}
#[tokio::test]
async fn scanner_observed_cleanup_must_keep_scope_live_across_both_pools() {
let (result, state, still_present) = run_scanner_observed_cleanup(None, Some(b"old observed snapshot"), false).await;
assert!(result.is_ok(), "result {result:?}, scope {state:?}, pool copies {still_present:?}");
assert_eq!(state, crate::object_api::ScannerPublicationCommitState::Committed);
assert_eq!(still_present, vec![false, false]);
}
#[tokio::test]
async fn scanner_observed_cleanup_must_preserve_both_pools_if_cas_changed() {
let (result, state, still_present) =
run_scanner_observed_cleanup(Some("stale-cas-token"), Some(b"old observed snapshot"), false).await;
assert!(
matches!(result, Err(StorageError::PreconditionFailed))
&& state == crate::object_api::ScannerPublicationCommitState::AbortedBeforeCommit
&& still_present == [true, true],
"result {result:?}, scope {state:?}, pool copies {still_present:?}"
);
}
#[tokio::test]
async fn scanner_observed_cleanup_must_preserve_a_nonmatching_pool_copy() {
let (result, state, still_present) = run_scanner_observed_cleanup(None, Some(b"newer observed snapshot"), false).await;
assert!(result.is_ok(), "result {result:?}, scope {state:?}, pool copies {still_present:?}");
assert_eq!(state, crate::object_api::ScannerPublicationCommitState::Committed);
assert_eq!(still_present, vec![false, true]);
}
#[tokio::test]
async fn scanner_observed_cleanup_must_succeed_when_another_pool_has_no_copy() {
let (result, state, still_present) = run_scanner_observed_cleanup(None, None, false).await;
assert!(result.is_ok(), "result {result:?}, scope {state:?}, pool copies {still_present:?}");
assert_eq!(state, crate::object_api::ScannerPublicationCommitState::Committed);
assert_eq!(still_present, vec![false, false]);
}
#[tokio::test]
async fn scanner_observed_cleanup_must_retain_ownership_after_partial_failure() {
let (result, state, still_present) = run_scanner_observed_cleanup(None, Some(b"old observed snapshot"), true).await;
assert!(result.is_err(), "partial cleanup must fail");
assert_eq!(state, crate::object_api::ScannerPublicationCommitState::Indeterminate);
assert_eq!(still_present, vec![false, true]);
}
struct LifecycleDeleteAllFailureGuard;
impl Drop for LifecycleDeleteAllFailureGuard {
+64 -1
View File
@@ -70,13 +70,17 @@ pub struct ObjectPartInfo {
pub error: Option<String>,
#[serde(default)]
pub integrity: Option<crate::shard_integrity::PartIntegrity>,
/// The stored ETag was generated by the protected encrypted writer.
#[serde(default)]
pub encrypted_etag: bool,
}
impl Serialize for ObjectPartInfo {
fn serialize<S: serde::Serializer>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error> {
// Named fields let release readers ignore optional extensions. Appending
// a ninth element to the old positional array makes those readers fail.
let mut map = serializer.serialize_map(Some(8 + usize::from(self.integrity.is_some())))?;
let mut map =
serializer.serialize_map(Some(8 + usize::from(self.integrity.is_some()) + usize::from(self.encrypted_etag)))?;
map.serialize_entry("etag", &self.etag)?;
map.serialize_entry("number", &self.number)?;
map.serialize_entry("size", &self.size)?;
@@ -88,6 +92,9 @@ impl Serialize for ObjectPartInfo {
if let Some(integrity) = &self.integrity {
map.serialize_entry("integrity", integrity)?;
}
if self.encrypted_etag {
map.serialize_entry("encrypted_etag", &true)?;
}
map.end()
}
}
@@ -1143,6 +1150,7 @@ impl FileInfo {
checksums,
error: None,
integrity: None,
encrypted_etag: false,
};
for p in self.parts.iter_mut() {
@@ -2231,6 +2239,59 @@ mod tests {
)
}
#[test]
fn object_part_encrypted_etag_round_trips_with_legacy_compatibility() {
#[derive(Serialize, Deserialize)]
struct LegacyPart {
etag: String,
number: usize,
size: usize,
actual_size: i64,
mod_time: Option<OffsetDateTime>,
index: Option<Bytes>,
checksums: Option<HashMap<String, String>>,
error: Option<String>,
}
let legacy = LegacyPart {
etag: "0123456789abcdef0123456789abcdef".to_owned(),
number: 1,
size: 42,
actual_size: 42,
mod_time: None,
index: None,
checksums: None,
error: None,
};
for encoded in [
rmp_serde::to_vec(&legacy).expect("legacy positional part encodes"),
rmp_serde::to_vec_named(&legacy).expect("legacy named part encodes"),
] {
let decoded = ObjectPartInfo::unmarshal(&encoded).expect("legacy part remains readable");
assert_eq!(decoded.etag, legacy.etag);
assert!(!decoded.encrypted_etag, "an older writer cannot attest a protected ETag");
}
for encrypted_etag in [false, true] {
let current = ObjectPartInfo {
etag: legacy.etag.clone(),
number: legacy.number,
size: legacy.size,
actual_size: legacy.actual_size,
encrypted_etag,
..Default::default()
};
let encoded = current.marshal_msg().expect("current part encodes as a map");
let field_count = rmp::decode::read_map_len(&mut encoded.as_slice()).expect("part format is a map");
assert_eq!(field_count, if encrypted_etag { 9 } else { 8 });
assert_eq!(ObjectPartInfo::unmarshal(&encoded).expect("proof round trips"), current);
let old_reader: LegacyPart = rmp_serde::from_slice(&encoded).expect("old readers ignore optional proof");
assert_eq!(old_reader.etag, current.etag);
assert_eq!(old_reader.number, current.number);
assert_eq!(old_reader.size, current.size);
}
}
fn object_part_info_strategy() -> impl Strategy<Value = ObjectPartInfo> {
(
small_string_strategy(),
@@ -2252,6 +2313,7 @@ mod tests {
checksums,
error,
integrity: None,
encrypted_etag: false,
})
}
@@ -2461,6 +2523,7 @@ mod tests {
),
error: Some("part-error".to_string()),
integrity: None,
encrypted_etag: false,
}],
erasure: ErasureInfo {
algorithm: "erasure-algorithm".to_string(),
+2
View File
@@ -1346,6 +1346,7 @@ impl<'a> BorrowedMetaObject<'a> {
checksums: None,
error: None,
integrity: None,
encrypted_etag: false,
});
}
}
@@ -2625,6 +2626,7 @@ impl From<MetaObjectV1Part> for ObjectPartInfo {
checksums: value.checksums,
error: value.error,
integrity: None,
encrypted_etag: false,
}
}
}
+1 -1
View File
@@ -30,7 +30,7 @@ workspace = true
[dependencies]
serde = { workspace = true, features = ["derive"] }
tokio = { workspace = true, features = ["sync"] }
uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics", "serde"] }
[dev-dependencies]
serde_json = { workspace = true }
+1 -1
View File
@@ -58,7 +58,7 @@ pub enum Error {
TaskExecutionFailed { message: String },
/// Only the qualified MRF Deep check may produce this terminal result.
#[error("Heal task execution failed: Missing exact storage proof for durable MRF repair {target}")]
#[error("Independent verification unavailable for legacy object {target}; durable MRF responsibility retained")]
LegacyMrfUnverified { target: String },
#[error("Replacement failure could not be persisted: {failure}; persistence error: {persistence}")]
+17
View File
@@ -292,6 +292,20 @@ impl HealManager {
"Heal scheduler task completed"
);
}
Err(e @ Error::LegacyMrfUnverified { .. }) => {
warn!(
target: "rustfs::heal::manager",
event = EVENT_HEAL_SCHEDULER_STATE,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_MANAGER,
state = "verification_unavailable",
task_id,
heal_type = %task_type_label_for_spawn,
set = %task_set_label_for_spawn,
detail = %e,
"Heal attempt completed without independent verification"
);
}
Err(e) => {
let will_retry = retry_request.is_some();
if will_retry {
@@ -483,6 +497,9 @@ impl HealManager {
// update statistics
let mut stats = statistics_clone.write().await;
match completed_status {
HealTaskStatus::Completed if matches!(&result, Err(Error::LegacyMrfUnverified { .. })) => {
stats.update_unverified_task_completion();
}
HealTaskStatus::Completed => {
stats.update_task_completion(true);
stats.add_healed_objects(completed_progress.objects_healed, completed_progress.bytes_processed);
+63
View File
@@ -1106,6 +1106,22 @@ impl HealStorageAPI for MockStorage {
if bucket.starts_with("legacy-ambiguity-scheduler-") {
return Err(Error::Storage(EcstoreError::DeleteRecoveryLegacyAmbiguous));
}
if bucket == "legacy-held-pending-scheduler" {
let drives = vec![rustfs_madmin::heal_commands::HealDriveInfo {
state: "ok".to_owned(),
..Default::default()
}];
return Ok((
HealResultItem {
legacy_identity_unverified: true,
detail: rustfs_heal_contracts::heal_channel::LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL.to_owned(),
before: rustfs_madmin::heal_commands::Infos { drives: drives.clone() },
after: rustfs_madmin::heal_commands::Infos { drives },
..Default::default()
},
None,
));
}
if bucket == "completed-retention-failed" {
return Err(Error::TaskExecutionFailed {
message: "retention fixture failure".to_string(),
@@ -2153,6 +2169,53 @@ fn only_typed_mrf_result_can_pause_unverified_legacy() {
}
}
#[tokio::test(flavor = "current_thread")]
async fn mrf_legacy_completed_attempt_retains_responsibility_without_failed_statistics() {
use crate::heal::failure_logging::test_support::FailureLogCapture;
let capture = FailureLogCapture::default();
let _subscriber = tracing::subscriber::set_default(capture.clone());
let (target, _) = unverified_legacy_fixture("legacy-held-pending-scheduler");
let bucket = target.bucket.clone();
let anchor = target.durable_anchor.clone().expect("durable MRF anchor");
let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
let mut request = HealRequest::object(bucket.to_string(), target.object.to_string(), None);
request.source = HealRequestSource::Mrf;
request.expected_mrf_bucket_incarnation_id = Some(anchor.bucket_incarnation_id);
request.options.pool_index = Some(1);
request.options.set_index = Some(2);
let receipt = manager
.submit_mrf_heal_request_with_receipt_and_identity(request, target)
.await
.expect("admit durable MRF check");
assert_eq!(receipt.result, HealAdmissionResult::Accepted);
process_manager_queue_once(&manager).await;
tokio::time::timeout(Duration::from_secs(2), async {
while manager.get_statistics().await.unverified_tasks == 0 {
tokio::task::yield_now().await;
}
})
.await
.expect("publish completed unverified attempt");
let stats = manager.get_statistics().await;
assert_eq!(stats.unverified_tasks, 1);
assert_eq!(stats.failed_tasks, 0);
assert_eq!(stats.successful_tasks, 0);
assert_eq!(stats.total_objects_healed, 0);
let report = manager.get_task_report(&receipt.task_id).await.expect("retained task report");
assert_eq!(report.status, HealTaskStatus::Completed);
let outcome = report.outcome.expect("retained unknown outcome");
assert_eq!(outcome.counters.unknown, 1);
assert_eq!(outcome.counters.failed, 0);
assert!(rustfs_common::mrf_channel::take_mrf_verified_repair_events_for(&bucket).is_empty());
let held = rustfs_common::mrf_channel::take_mrf_unverified_legacy_events_for(&bucket);
assert_eq!(held.len(), 1);
assert_eq!(held[0].anchor, anchor);
assert!(manager.retrying_heals.lock().await.is_empty());
assert_eq!(capture.count(tracing::Level::ERROR, Some("rustfs::heal::manager")), 0);
assert_eq!(capture.count(tracing::Level::WARN, Some("rustfs::heal::manager")), 1);
}
#[test]
fn completed_mrf_notice_publishes_only_verified_positive_events() {
use crate::heal::outcome::{HealObjectIdentity, HealObjectKind, HealObjectOutcome, HealTaskOutcome};
+24
View File
@@ -506,6 +506,9 @@ pub struct HealStatistics {
pub successful_tasks: u64,
/// Failed tasks
pub failed_tasks: u64,
/// Completed attempts without independent verification; not the current durable MRF backlog.
#[serde(default)]
pub unverified_tasks: u64,
/// Running tasks
pub running_tasks: u64,
/// Total healed objects
@@ -528,6 +531,7 @@ impl HealStatistics {
total_tasks: 0,
successful_tasks: 0,
failed_tasks: 0,
unverified_tasks: 0,
running_tasks: 0,
total_objects_healed: 0,
total_bytes_healed: 0,
@@ -549,6 +553,11 @@ impl HealStatistics {
self.last_update_time = SystemTime::now();
}
pub(crate) fn update_unverified_task_completion(&mut self) {
self.unverified_tasks = self.unverified_tasks.saturating_add(1);
self.last_update_time = SystemTime::now();
}
pub fn add_healed_objects(&mut self, count: u64, bytes: u64) {
self.total_objects_healed += count;
self.total_bytes_healed += bytes;
@@ -915,6 +924,21 @@ mod tests {
assert_eq!(stats.failed_tasks, 0);
}
#[test]
fn unverified_statistics_preserve_old_snapshot_compatibility() {
let mut stats = HealStatistics::new();
stats.update_unverified_task_completion();
assert_eq!(stats.unverified_tasks, 1);
assert_eq!(stats.successful_tasks, 0);
assert_eq!(stats.failed_tasks, 0);
let mut wire = serde_json::to_value(&stats).expect("serialize task statistics");
let current: HealStatistics = serde_json::from_value(wire.clone()).expect("read new statistics");
assert_eq!(current.unverified_tasks, 1);
wire.as_object_mut().expect("statistics object").remove("unverified_tasks");
let old: HealStatistics = serde_json::from_value(wire).expect("read statistics without new field");
assert_eq!(old.unverified_tasks, 0);
}
#[test]
fn test_heal_statistics_update_task_completion_success() {
let mut stats = HealStatistics::new();
+23 -2
View File
@@ -291,7 +291,7 @@ pub enum HealTaskStatus {
Running,
/// Retrying after a recoverable failure
Retrying { error: String, retry_attempt: u32 },
/// Completed
/// Execution completed; the outcome and durable MRF ledger retain any unverified responsibility.
Completed,
/// Failed
Failed { error: String },
@@ -1232,6 +1232,22 @@ impl HealTask {
"Heal task completed"
});
}
Err(Error::LegacyMrfUnverified { .. }) => {
// The qualified Deep attempt finished, but its Unknown outcome
// cannot discharge the separately owned durable responsibility.
self.progress.write().await.mark_completed();
*self.status.write().await = HealTaskStatus::Completed;
debug!(
target: "rustfs::heal::task",
event = EVENT_HEAL_TASK_STATE,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_TASK,
state = "verification_unavailable",
task_id = %self.id,
heal_type = self.heal_type.kind_label(),
"Heal attempt completed without independent verification"
);
}
Err(Error::TaskCancelled) => {
let mut status = self.status.write().await;
*status = HealTaskStatus::Cancelled;
@@ -1281,11 +1297,16 @@ impl HealTask {
let terminal_state = match &result {
Ok(_) => "completed",
Err(Error::LegacyMrfUnverified { .. }) => "verification_unavailable",
Err(Error::TaskCancelled) => "cancelled",
Err(Error::TaskTimeout) => "timed_out",
Err(_) => "failed",
};
self.emit_trace_task_state(terminal_state, start_instant.elapsed(), result.as_ref().err());
let terminal_error = result
.as_ref()
.err()
.filter(|error| !matches!(error, Error::LegacyMrfUnverified { .. }));
self.emit_trace_task_state(terminal_state, start_instant.elapsed(), terminal_error);
result
}
+29 -1
View File
@@ -3285,6 +3285,29 @@ async fn durable_mrf_unverified_object_names_cannot_change_retry_classification(
}
}
#[tokio::test]
async fn mrf_legacy_verification_pending_is_a_completed_unverified_attempt() {
let (task, _) = mrf_task_for_item("legacy-pending", healthy_legacy_heal_item());
let error = task.execute().await.expect_err("unverified result must retain MRF ownership");
assert!(matches!(error, Error::LegacyMrfUnverified { .. }));
assert!(!error.is_recoverable_heal());
assert_eq!(task.get_status().await, HealTaskStatus::Completed);
let outcome = task.get_outcome().await;
assert_eq!(outcome.execution, HealExecutionOutcome::Completed);
assert_eq!(outcome.counters.unknown, 1);
assert_eq!(outcome.counters.failed, 0);
assert_eq!(outcome.counters.healed, 0);
assert_eq!(outcome.counters.unchanged, 0);
assert!(matches!(outcome.objects[0].disposition, HealObjectDisposition::Unknown));
let (status, detail) = outcome.legacy_status("finished", None);
assert_eq!(status, "finished");
assert!(
detail
.expect("visible unverified result")
.contains("authoritative storage proof is unavailable")
);
}
#[tokio::test]
async fn mrf_legacy_object_names_cannot_change_retry_classification() {
for object in [
@@ -3304,7 +3327,9 @@ async fn mrf_legacy_object_names_cannot_change_retry_classification() {
assert!(matches!(error, Error::LegacyMrfUnverified { .. }));
assert_eq!(
error.to_string(),
format!("Heal task execution failed: Missing exact storage proof for durable MRF repair bucket-a/{object}")
format!(
"Independent verification unavailable for legacy object bucket-a/{object}; durable MRF responsibility retained"
)
);
}
}
@@ -3362,11 +3387,14 @@ async fn mrf_legacy_pause_requires_storage_owned_healthy_deep_result() {
let outcome = task.get_outcome().await;
assert_eq!(outcome.counters.healed, 0);
if case == "qualified" {
assert_eq!(task.get_status().await, HealTaskStatus::Completed);
assert_eq!(outcome.counters.unknown, 1);
assert_eq!(outcome.objects[0].detail.as_deref(), Some("updated diagnostic wording"));
let opts = storage.object_heal_opts.lock().expect("storage options");
assert_eq!(opts.len(), 1);
assert_eq!(opts[0].scan_mode, HealScanMode::Deep);
} else {
assert!(matches!(task.get_status().await, HealTaskStatus::Failed { .. }), "case={case}");
}
}
}
@@ -30,6 +30,78 @@ use storage_api::integration::{
WriteCommitGuard, WriteCompletion,
};
async fn complete_fixture_replicas(env: &TestECStoreEnv, bucket: &str, object: &str) {
// TailDrained completes fan-out, but a successful quorum write may still
// have minority failures. Fault injection needs a fully repaired baseline.
let result = ECStoreHealStorage::new(env.ecstore.clone())
.heal_object_with_receipt(
bucket,
object,
None,
&HealOpts {
scan_mode: HealScanMode::Deep,
..Default::default()
},
)
.await
.expect("complete protected fixture replicas before fault injection");
assert!(result.error.is_none(), "fixture repair failed: {:?}", result.error);
assert_eq!(result.item.after.drives.len(), env.disk_paths.len());
assert!(
result.item.after.drives.iter().all(|drive| drive.state == "ok"),
"fixture requires every replica, not only read/write quorum: {:?}",
result.item.after.drives
);
assert!(result.item.integrity_verified, "fixture must retain protected payload verification");
}
#[tokio::test]
#[serial]
async fn mixed_batch_fixture_repairs_a_missing_minority_replica() {
let root = tempfile::tempdir().expect("minority replica fixture");
let env = TestECStoreEnv::builder().base_dir(root.path()).build().await;
let bucket = "mixed-fixture-baseline";
let object = "object";
env.make_bucket(bucket, false).await;
let expected = vec![0x5a; 1024 * 1024 + 123];
env.ecstore
.put_object(
bucket,
object,
&mut PutObjReader::from_vec(expected.clone()),
&ObjectOptions {
shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Protected),
write_completion: WriteCompletion::TailDrained,
..Default::default()
},
)
.await
.expect("quorum commit fixture");
// Guarantee a minority failure regardless of whether the original write
// succeeded on every disk, then repair it before any test fault is applied.
let replica = env.disk_paths[0].join(bucket).join(object);
if replica.exists() {
tokio::fs::remove_dir_all(&replica)
.await
.expect("remove one complete replica");
}
complete_fixture_replicas(&env, bucket, object).await;
let disks = env.ecstore.pools[0].get_disks(0);
for disk in disks.disks.read().await.iter().flatten() {
disk.read_version("", bucket, object, "", &ReadOptions::default())
.await
.expect("every fixture replica has metadata after completion");
}
let mut reader = env
.ecstore
.get_object_reader(bucket, object, None, Default::default(), &ObjectOptions::default())
.await
.expect("read the completed fixture");
let mut actual = Vec::new();
reader.stream.read_to_end(&mut actual).await.expect("completed fixture body");
assert_eq!(actual, expected);
}
#[test]
fn disabled_lock_reports_only_quorum_selection_and_deep_check_observations() {
const CHILD_ENV: &str = "RUSTFS_TEST_DISABLED_LOCK_HEAL_CHILD";
@@ -704,6 +776,7 @@ fn admin_mixed_batch_retains_repairs_after_object_retry_exhaustion() {
)
.await
.expect("commit complete fixture");
complete_fixture_replicas(&env, bucket, object).await;
for (slot, disk) in disks.iter().enumerate() {
let meta = disk
.read_version("", bucket, object, "", &ReadOptions::default())
+100 -10
View File
@@ -429,6 +429,7 @@ const INTERNODE_OPERATION_REQUESTS_OUTGOING_TOTAL: &str = "rustfs_system_network
const INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL: &str = "rustfs_system_network_internode_operation_requests_incoming_total";
const INTERNODE_OPERATION_ERRORS_TOTAL: &str = "rustfs_system_network_internode_operation_errors_total";
const INTERNODE_OPERATION_DURATION_MS: &str = "rustfs_system_network_internode_operation_duration_ms";
const INTERNODE_OPERATION_SERVER_DURATION_MS: &str = "rustfs_system_network_internode_operation_server_duration_ms";
const INTERNODE_OPERATION_STAGE_DURATION_MS: &str = "rustfs_system_network_internode_operation_stage_duration_ms";
const INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL: &str = "rustfs_system_network_internode_operation_classified_errors_total";
const INTERNODE_OPERATION_RETRIES_TOTAL: &str = "rustfs_system_network_internode_operation_retries_total";
@@ -562,6 +563,10 @@ pub const INTERNODE_OPERATION_METRICS: &[InternodeOperationMetricDescriptor] = &
name: INTERNODE_OPERATION_LARGE_PAYLOADS_TOTAL,
labels: SERVER_OPERATION_BACKEND_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_SERVER_DURATION_MS,
labels: SERVER_OPERATION_BACKEND_LABELS,
},
];
static STABLE_SERVER_LABEL: OnceLock<String> = OnceLock::new();
@@ -687,6 +692,78 @@ impl GrpcReadVersionMetricHandles {
static GRPC_READ_VERSION_METRIC_HANDLES: LazyLock<GrpcReadVersionMetricHandles> =
LazyLock::new(|| GrpcReadVersionMetricHandles::new(current_server_label()));
#[cfg(not(test))]
static GRPC_READ_MULTIPLE_DURATION: LazyLock<metrics::Histogram> = LazyLock::new(|| {
operation_duration_histogram(
INTERNODE_OPERATION_DURATION_MS,
INTERNODE_OPERATION_GRPC_READ_MULTIPLE,
INTERNODE_TRANSPORT_BACKEND_GRPC,
)
});
#[cfg(not(test))]
static GRPC_BATCH_READ_VERSION_DURATION: LazyLock<metrics::Histogram> = LazyLock::new(|| {
operation_duration_histogram(
INTERNODE_OPERATION_DURATION_MS,
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
)
});
#[cfg(not(test))]
static GRPC_READ_MULTIPLE_SERVER_DURATION: LazyLock<metrics::Histogram> = LazyLock::new(|| {
operation_duration_histogram(
INTERNODE_OPERATION_SERVER_DURATION_MS,
INTERNODE_OPERATION_GRPC_READ_MULTIPLE,
INTERNODE_TRANSPORT_BACKEND_GRPC,
)
});
#[cfg(not(test))]
static GRPC_BATCH_READ_VERSION_SERVER_DURATION: LazyLock<metrics::Histogram> = LazyLock::new(|| {
operation_duration_histogram(
INTERNODE_OPERATION_SERVER_DURATION_MS,
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
)
});
#[cfg(not(test))]
fn grpc_operation_duration_if_ready(operation: &'static str, backend: &'static str) -> Option<&'static metrics::Histogram> {
STABLE_SERVER_LABEL.get()?;
if backend != INTERNODE_TRANSPORT_BACKEND_GRPC {
return None;
}
match operation {
INTERNODE_OPERATION_GRPC_READ_VERSION => Some(&GRPC_READ_VERSION_METRIC_HANDLES.duration),
INTERNODE_OPERATION_GRPC_READ_MULTIPLE => Some(&GRPC_READ_MULTIPLE_DURATION),
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION => Some(&GRPC_BATCH_READ_VERSION_DURATION),
_ => None,
}
}
#[cfg(not(test))]
fn grpc_server_duration_if_ready(operation: &'static str, backend: &'static str) -> Option<&'static metrics::Histogram> {
STABLE_SERVER_LABEL.get()?;
if backend != INTERNODE_TRANSPORT_BACKEND_GRPC {
return None;
}
match operation {
INTERNODE_OPERATION_GRPC_READ_MULTIPLE => Some(&GRPC_READ_MULTIPLE_SERVER_DURATION),
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION => Some(&GRPC_BATCH_READ_VERSION_SERVER_DURATION),
_ => None,
}
}
fn operation_duration_histogram(name: &'static str, operation: &'static str, backend: &'static str) -> metrics::Histogram {
metrics::histogram!(
name,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
}
#[cfg(not(test))]
fn server_metric_handles_if_ready() -> Option<&'static InternodeServerMetricHandles> {
STABLE_SERVER_LABEL.get()?;
@@ -944,17 +1021,28 @@ impl InternodeMetrics {
pub fn record_duration_for_operation_and_backend(&self, operation: &'static str, backend: &'static str, duration: Duration) {
let duration_ms = duration.as_secs_f64() * 1000.0;
#[cfg(not(test))]
if let Some(handles) = grpc_read_version_metric_handles_if_ready(operation, backend) {
handles.duration.record(duration_ms);
if let Some(histogram) = grpc_operation_duration_if_ready(operation, backend) {
histogram.record(duration_ms);
return;
}
metrics::histogram!(
INTERNODE_OPERATION_DURATION_MS,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
.record(duration_ms);
operation_duration_histogram(INTERNODE_OPERATION_DURATION_MS, operation, backend).record(duration_ms);
}
/// Records server-side handler latency in a separate histogram from client RPC duration.
/// The caller supplies the handler lifetime, including any error or cancellation path.
pub fn record_server_duration_for_operation_and_backend(
&self,
operation: &'static str,
backend: &'static str,
duration: Duration,
) {
let duration_ms = duration.as_secs_f64() * 1000.0;
#[cfg(not(test))]
if let Some(histogram) = grpc_server_duration_if_ready(operation, backend) {
histogram.record(duration_ms);
return;
}
operation_duration_histogram(INTERNODE_OPERATION_SERVER_DURATION_MS, operation, backend).record(duration_ms);
}
pub fn record_stage_duration_for_operation_and_backend(
@@ -1699,7 +1787,7 @@ mod tests {
#[test]
fn operation_metric_descriptors_include_backend_and_operation_labels() {
assert_eq!(INTERNODE_OPERATION_METRICS.len(), 22);
assert_eq!(INTERNODE_OPERATION_METRICS.len(), 23);
for metric in &INTERNODE_OPERATION_METRICS[..6] {
assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
}
@@ -1737,6 +1825,8 @@ mod tests {
// Payload histogram + large-payload counter carry operation+backend labels.
assert_eq!(INTERNODE_OPERATION_METRICS[20].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[21].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[22].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[22].name, INTERNODE_OPERATION_SERVER_DURATION_MS);
}
#[test]
@@ -0,0 +1,206 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use metrics::{Counter, Gauge, Histogram, Key, KeyName, Metadata, Recorder, SharedString, Unit, with_local_recorder};
use metrics_util::debugging::{DebugValue, DebuggingRecorder};
use rustfs_io_metrics::internode_metrics::{
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_MULTIPLE,
INTERNODE_OPERATION_GRPC_READ_VERSION, INTERNODE_TRANSPORT_BACKEND_GRPC, INTERNODE_TRANSPORT_BACKEND_UNKNOWN,
InternodeMetrics, set_internode_server_label,
};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
const DURATION_METRIC: &str = "rustfs_system_network_internode_operation_duration_ms";
const SERVER_DURATION_METRIC: &str = "rustfs_system_network_internode_operation_server_duration_ms";
const SERVER_READ_OPERATIONS: [&str; 2] = [
INTERNODE_OPERATION_GRPC_READ_MULTIPLE,
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION,
];
const READ_OPERATIONS: [&str; 3] = [
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_OPERATION_GRPC_READ_MULTIPLE,
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION,
];
#[derive(Default)]
struct RegistrationCountingRecorder {
inner: DebuggingRecorder,
duration_registrations: AtomicUsize,
}
impl Recorder for RegistrationCountingRecorder {
fn describe_counter(&self, key: KeyName, unit: Option<Unit>, description: SharedString) {
self.inner.describe_counter(key, unit, description);
}
fn describe_gauge(&self, key: KeyName, unit: Option<Unit>, description: SharedString) {
self.inner.describe_gauge(key, unit, description);
}
fn describe_histogram(&self, key: KeyName, unit: Option<Unit>, description: SharedString) {
self.inner.describe_histogram(key, unit, description);
}
fn register_counter(&self, key: &Key, metadata: &Metadata<'_>) -> Counter {
self.inner.register_counter(key, metadata)
}
fn register_gauge(&self, key: &Key, metadata: &Metadata<'_>) -> Gauge {
self.inner.register_gauge(key, metadata)
}
fn register_histogram(&self, key: &Key, metadata: &Metadata<'_>) -> Histogram {
if key.name() == DURATION_METRIC || key.name() == SERVER_DURATION_METRIC {
self.duration_registrations.fetch_add(1, Ordering::Relaxed);
}
self.inner.register_histogram(key, metadata)
}
}
// Integration tests exercise the production cache, which is disabled in unit tests
// so individual unit tests can install independent local recorders.
#[test]
fn unary_read_duration_handles_are_cached_only_after_server_label_is_ready() {
let recorder = RegistrationCountingRecorder::default();
let snapshotter = recorder.inner.snapshotter();
let metrics = InternodeMetrics::default();
with_local_recorder(&recorder, || {
for operation in READ_OPERATIONS {
metrics.record_duration_for_operation_and_backend(operation, INTERNODE_TRANSPORT_BACKEND_GRPC, Duration::ZERO);
}
for operation in SERVER_READ_OPERATIONS {
metrics.record_server_duration_for_operation_and_backend(operation, INTERNODE_TRANSPORT_BACKEND_GRPC, Duration::ZERO);
}
set_internode_server_label("cached-duration-test");
for operation in READ_OPERATIONS {
metrics.record_duration_for_operation_and_backend(
operation,
INTERNODE_TRANSPORT_BACKEND_GRPC,
Duration::from_micros(125),
);
let registrations = recorder.duration_registrations.load(Ordering::Relaxed);
metrics.record_duration_for_operation_and_backend(
operation,
INTERNODE_TRANSPORT_BACKEND_GRPC,
Duration::from_micros(250),
);
assert_eq!(
recorder.duration_registrations.load(Ordering::Relaxed),
registrations,
"{operation} should reuse its duration handle after warm-up"
);
}
for operation in SERVER_READ_OPERATIONS {
metrics.record_server_duration_for_operation_and_backend(
operation,
INTERNODE_TRANSPORT_BACKEND_GRPC,
Duration::from_micros(1500),
);
let registrations = recorder.duration_registrations.load(Ordering::Relaxed);
metrics.record_server_duration_for_operation_and_backend(
operation,
INTERNODE_TRANSPORT_BACKEND_GRPC,
Duration::from_micros(2500),
);
assert_eq!(
recorder.duration_registrations.load(Ordering::Relaxed),
registrations,
"{operation} should reuse its separate server duration handle after warm-up"
);
}
// Other backends and operations must retain their own labels and fallback.
for (operation, backend) in READ_OPERATIONS
.map(|operation| (operation, INTERNODE_TRANSPORT_BACKEND_UNKNOWN))
.into_iter()
.chain([(INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC)])
{
let registrations = recorder.duration_registrations.load(Ordering::Relaxed);
metrics.record_duration_for_operation_and_backend(operation, backend, Duration::from_millis(1));
assert_eq!(recorder.duration_registrations.load(Ordering::Relaxed), registrations + 1);
}
for (operation, backend) in SERVER_READ_OPERATIONS
.map(|operation| (operation, INTERNODE_TRANSPORT_BACKEND_UNKNOWN))
.into_iter()
.chain([(INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC)])
{
let registrations = recorder.duration_registrations.load(Ordering::Relaxed);
metrics.record_server_duration_for_operation_and_backend(operation, backend, Duration::from_millis(3));
assert_eq!(recorder.duration_registrations.load(Ordering::Relaxed), registrations + 1);
}
});
let rows = snapshotter.snapshot().into_vec();
let durations: Vec<_> = rows
.iter()
.filter(|(key, _, _, _)| key.key().name() == DURATION_METRIC || key.key().name() == SERVER_DURATION_METRIC)
.collect();
assert_eq!(
durations
.iter()
.filter(|(key, _, _, _)| key.key().name() == DURATION_METRIC)
.count(),
10
);
assert_eq!(
durations
.iter()
.filter(|(key, _, _, _)| key.key().name() == SERVER_DURATION_METRIC)
.count(),
7
);
for (key, _, _, value) in durations {
let server_duration = key.key().name() == SERVER_DURATION_METRIC;
let operations = if server_duration {
SERVER_READ_OPERATIONS.as_slice()
} else {
READ_OPERATIONS.as_slice()
};
let labels: Vec<_> = key.key().labels().map(|label| (label.key(), label.value())).collect();
assert_eq!(labels.len(), 3);
let server = labels.iter().find(|(name, _)| *name == "server").expect("server label").1;
let operation = labels
.iter()
.find(|(name, _)| *name == "operation")
.expect("operation label")
.1;
let backend = labels.iter().find(|(name, _)| *name == "backend").expect("backend label").1;
let expected: &[f64] = if server == "unset" {
assert!(operations.contains(&operation));
assert_eq!(backend, INTERNODE_TRANSPORT_BACKEND_GRPC);
&[0.0]
} else {
assert_eq!(server, "cached-duration-test");
if backend == INTERNODE_TRANSPORT_BACKEND_GRPC && operations.contains(&operation) {
if server_duration { &[1.5, 2.5] } else { &[0.125, 0.25] }
} else {
assert!(
(backend == INTERNODE_TRANSPORT_BACKEND_UNKNOWN && operations.contains(&operation))
|| (backend == INTERNODE_TRANSPORT_BACKEND_GRPC && operation == INTERNODE_OPERATION_GRPC_READ_ALL)
);
if server_duration { &[3.0] } else { &[1.0] }
}
};
match value {
DebugValue::Histogram(samples) => {
assert_eq!(samples.iter().map(|sample| sample.0).collect::<Vec<_>>(), expected);
}
other => panic!("duration metric must be a histogram, got {other:?}"),
}
}
}
+447 -15
View File
@@ -12,12 +12,14 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use parking_lot::{RwLock, RwLockReadGuard, RwLockWriteGuard};
use std::collections::HashMap;
use std::ops::{Deref, DerefMut};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, SystemTime};
use tokio::sync::RwLock;
use tokio::time::{Instant, timeout};
use tokio::sync::Notify;
use tokio::time::{Instant, timeout, timeout_at};
use crate::{
FastLockGuard, GlobalLockManager, LockClient, LockId, LockInfo, LockManager, LockMetadata, LockPriority, LockRequest,
@@ -26,13 +28,98 @@ use crate::{
/// Default shard count for guard storage (must be power of 2)
const DEFAULT_GUARD_SHARD_COUNT: usize = 64;
const EVENT_NAMESPACE_LOCK_ACQUIRE: &str = "namespace_lock_acquire";
const EVENT_NAMESPACE_LOCK_RELEASE: &str = "namespace_lock_release";
const LOG_COMPONENT_LOCK: &str = "lock";
const LOG_SUBSYSTEM_NAMESPACE: &str = "namespace";
type GuardShard = Arc<RwLock<HashMap<LockId, LocalGuardEntry>>>;
type GuardMap = HashMap<LockId, LocalGuardEntry>;
type GuardShard = Arc<GuardRegistry>;
type GuardStorage = Arc<Vec<GuardShard>>;
/// Registry access never reserves the table for a task that has not resumed.
/// Only short synchronous map operations hold the native lock; contention
/// waits on a notification without blocking an executor thread.
#[derive(Debug, Default)]
struct GuardRegistry {
entries: RwLock<GuardMap>,
changed: Notify,
}
struct GuardRegistryWake<'a>(&'a Notify);
impl Drop for GuardRegistryWake<'_> {
fn drop(&mut self) {
self.0.notify_waiters();
}
}
struct GuardRegistryAccess<'a, G> {
// Field drop order unlocks the table before waking its waiters, including
// when a map operation unwinds. Physical guards are removed before Drop.
guard: G,
_wake: GuardRegistryWake<'a>,
}
impl<G: Deref> Deref for GuardRegistryAccess<'_, G> {
type Target = G::Target;
fn deref(&self) -> &Self::Target {
&self.guard
}
}
impl<G: DerefMut> DerefMut for GuardRegistryAccess<'_, G> {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.guard
}
}
impl GuardRegistry {
fn access<G>(&self, guard: G) -> GuardRegistryAccess<'_, G> {
GuardRegistryAccess {
guard,
_wake: GuardRegistryWake(&self.changed),
}
}
async fn read(&self) -> GuardRegistryAccess<'_, RwLockReadGuard<'_, GuardMap>> {
tokio::task::consume_budget().await;
if let Some(guard) = self.entries.try_read() {
return self.access(guard);
}
loop {
tokio::task::consume_budget().await;
let notified = self.changed.notified();
tokio::pin!(notified);
notified.as_mut().enable();
if let Some(guard) = self.entries.try_read() {
return self.access(guard);
}
notified.await;
}
}
async fn write(&self) -> GuardRegistryAccess<'_, RwLockWriteGuard<'_, GuardMap>> {
tokio::task::consume_budget().await;
if let Some(guard) = self.entries.try_write() {
return self.access(guard);
}
loop {
tokio::task::consume_budget().await;
// Register before rechecking so an unlock between the first
// failed try and this wait cannot leave the task asleep.
let notified = self.changed.notified();
tokio::pin!(notified);
notified.as_mut().enable();
if let Some(guard) = self.entries.try_write() {
return self.access(guard);
}
notified.await;
}
}
}
/// Local lock client using FastLock with sharded guard storage for better concurrency
#[derive(Debug)]
pub struct LocalClient {
@@ -94,7 +181,7 @@ impl LocalClient {
pub fn with_shard_count(shard_count: usize) -> Self {
assert!(shard_count.is_power_of_two(), "Shard count must be power of 2");
let guard_storage: Vec<GuardShard> = (0..shard_count).map(|_| Arc::new(RwLock::new(HashMap::new()))).collect();
let guard_storage: Vec<GuardShard> = (0..shard_count).map(|_| Arc::new(GuardRegistry::default())).collect();
Self::with_storage(Arc::new(guard_storage), None, crate::fast_lock::CLEANUP_INTERVAL)
}
@@ -115,7 +202,7 @@ impl LocalClient {
/// This allows simulating multi-node environments where each node has its own lock backend
pub fn with_manager(manager: Arc<GlobalLockManager>) -> Self {
let guard_storage = (0..DEFAULT_GUARD_SHARD_COUNT)
.map(|_| Arc::new(RwLock::new(HashMap::new())))
.map(|_| Arc::new(GuardRegistry::default()))
.collect();
Self::with_storage(Arc::new(guard_storage), Some(manager), crate::fast_lock::CLEANUP_INTERVAL)
}
@@ -123,7 +210,7 @@ impl LocalClient {
#[cfg(test)]
pub(crate) fn with_manager_and_reaper_interval(manager: Arc<GlobalLockManager>, reaper_interval: Duration) -> Self {
let guard_storage = (0..DEFAULT_GUARD_SHARD_COUNT)
.map(|_| Arc::new(RwLock::new(HashMap::new())))
.map(|_| Arc::new(GuardRegistry::default()))
.collect();
Self::with_storage(Arc::new(guard_storage), Some(manager), reaper_interval)
}
@@ -140,7 +227,7 @@ impl LocalClient {
}
/// Get the shard for a given lock ID
fn get_shard(&self, lock_id: &LockId) -> &Arc<RwLock<HashMap<LockId, LocalGuardEntry>>> {
fn get_shard(&self, lock_id: &LockId) -> &GuardShard {
let index = self.get_resource_shard_index(&lock_id.resource);
&self.guard_storage[index]
}
@@ -232,6 +319,7 @@ impl Default for LocalClient {
#[async_trait::async_trait]
impl LockClient for LocalClient {
async fn acquire_lock(&self, request: &LockRequest) -> Result<LockResponse> {
let started = Instant::now();
self.ensure_reaper();
if request.lock_id.resource != request.resource {
return Ok(LockResponse::failure(
@@ -251,6 +339,7 @@ impl LockClient for LocalClient {
.await
.unwrap_or(0)
};
let reclaim_elapsed = started.elapsed();
let build_lock_request = |acquire_timeout| match request.lock_type {
LockType::Exclusive => crate::ObjectLockRequest::new_write(request.resource.clone(), request.owner.clone())
@@ -265,17 +354,65 @@ impl LockClient for LocalClient {
if remaining.is_zero() {
return Ok(LockResponse::failure("Lock acquisition timeout", request.acquire_timeout));
}
let manager_started = Instant::now();
match lock_manager.acquire_lock(build_lock_request(remaining)).await {
Ok(guard) => {
let manager_elapsed = manager_started.elapsed();
let registration_started = Instant::now();
let physical_guard_id = guard.guard_id();
let lock_id = request.lock_id.clone();
let entry = LocalGuardEntry::new(guard, request.ttl);
let acquired_at = entry.acquired_at;
let expires_at = entry.expires_at;
{
let (replaced_entry, registration_wait, registration_mutation, registry_entries, registry_capacity) = {
let shard = self.get_shard(&lock_id);
let mut guards = shard.write().await;
guards.insert(lock_id.clone(), entry);
let Ok(mut guards) = timeout_at(acquire_deadline, shard.write()).await else {
return Ok(LockResponse::failure("Lock acquisition timeout", request.acquire_timeout));
};
// A ready table cannot publish a grant after the peer
// budget expired while its acquisition task was idle.
if Instant::now() >= acquire_deadline {
drop(guards);
return Ok(LockResponse::failure("Lock acquisition timeout", request.acquire_timeout));
}
let registration_wait = registration_started.elapsed();
let mutation_started = Instant::now();
let replaced_entry = guards.insert(lock_id.clone(), entry);
(
replaced_entry,
registration_wait,
mutation_started.elapsed(),
guards.len(),
guards.capacity(),
)
};
// Repeated shared requests can replace an entry. Releasing
// its physical guard must not reenter locks under the table.
drop(replaced_entry);
let registration_elapsed = registration_started.elapsed();
if started.elapsed() >= Duration::from_millis(100) {
tracing::trace!(
target: "rustfs_lock::namespace_owner",
event = EVENT_NAMESPACE_LOCK_ACQUIRE,
component = LOG_COMPONENT_LOCK,
subsystem = LOG_SUBSYSTEM_NAMESPACE,
state = "local_completed",
resource = %request.resource,
lock_id = %lock_id,
mode = ?request.lock_type,
physical_guard_id,
reclaim_ms = reclaim_elapsed.as_millis(),
manager_ms = manager_elapsed.as_millis(),
registration_ms = registration_elapsed.as_millis(),
registration_wait_ms = registration_wait.as_millis(),
registration_mutation_ms = registration_mutation.as_millis(),
registry_entries,
registry_capacity,
acquire_ms = started.elapsed().as_millis(),
"Namespace local lock acquisition completed slowly"
);
}
let lock_info = LockInfo {
@@ -469,6 +606,8 @@ impl LockClient for LocalClient {
mod tests {
use super::*;
use crate::{GlobalLockManager, LockClient, LockRequest, LockType};
use futures::FutureExt;
use std::task::Poll;
fn request(resource: crate::ObjectKey, owner: &str, ttl: Duration) -> LockRequest {
LockRequest::new(resource, LockType::Exclusive, owner)
@@ -476,6 +615,295 @@ mod tests {
.with_acquire_timeout(Duration::from_millis(80))
}
fn hold_guard_shard(shard: GuardShard) -> (std::sync::mpsc::Sender<()>, std::thread::JoinHandle<()>) {
let (entered_tx, entered_rx) = std::sync::mpsc::sync_channel(0);
let (release_tx, release_rx) = std::sync::mpsc::channel();
let thread = std::thread::spawn(move || {
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("build the independent guard-table owner runtime");
let guard = runtime.block_on(shard.write());
entered_tx.send(()).expect("signal that the guard table is occupied");
// A dropped controller also releases the table on test failure.
let _ = release_rx.recv_timeout(Duration::from_secs(3));
drop(guard);
});
entered_rx
.recv_timeout(Duration::from_secs(3))
.expect("guard-table owner must enter before the tested operation");
(release_tx, thread)
}
#[tokio::test(start_paused = true)]
async fn unpolled_registry_writer_does_not_block_compatible_acquisition() {
let manager = Arc::new(GlobalLockManager::new());
let client = LocalClient::with_manager(manager.clone());
client.reaper_started.store(true, Ordering::Release);
let first = LockRequest::new(crate::ObjectKey::new("bucket", ""), LockType::Shared, "first");
assert!(client.acquire_lock(&first).await.expect("first shared lock").success);
let (release_table, table_owner) = hold_guard_shard(client.get_shard(&first.lock_id).clone());
let mut pending_release = Box::pin(client.release(&first.lock_id));
assert!(matches!(futures::poll!(&mut pending_release), Poll::Pending));
release_table.send(()).expect("release the occupied table");
table_owner.join().expect("guard-table owner must finish");
// The earlier writer is deliberately not polled: an unoccupied
// bookkeeping table must remain usable by a currently running peer.
let second = LockRequest::new(first.resource.clone(), LockType::Shared, "second");
let response = client
.acquire_lock(&second)
.now_or_never()
.expect("a sleeping registry writer must not reserve the available table")
.expect("compatible shared acquisition should return a response");
assert!(response.success);
assert_eq!(
manager
.as_fast_lock_manager()
.expect("enabled physical lock backend")
.get_shard(&first.resource)
.active_guard_count(),
2
);
assert!(
pending_release
.await
.expect("the earlier writer still releases its exact grant")
);
assert!(client.release(&second.lock_id).await.expect("release the second grant"));
assert_eq!(
manager
.as_fast_lock_manager()
.expect("enabled physical lock backend")
.get_shard(&first.resource)
.active_guard_count(),
0
);
}
#[tokio::test(start_paused = true)]
async fn acquisition_deadline_rejects_registration_after_physical_grant() {
for unlock_before_poll in [false, true] {
let manager = Arc::new(GlobalLockManager::new());
let client = LocalClient::with_manager(manager.clone());
client.reaper_started.store(true, Ordering::Release);
let resource = crate::ObjectKey::new("bucket", "registration-deadline");
let blocker = manager
.acquire_lock(crate::ObjectLockRequest::new_write(resource.clone(), "blocker"))
.await
.expect("hold the physical lock until guard-table preflight finishes");
let request =
request(resource.clone(), "owner", Duration::from_secs(30)).with_acquire_timeout(Duration::from_millis(20));
let mut acquire = Box::pin(tokio::task::unconstrained(client.acquire_lock(&request)));
assert!(matches!(futures::poll!(&mut acquire), Poll::Pending));
let (release_table, table_owner) = hold_guard_shard(client.get_shard(&request.lock_id).clone());
drop(blocker);
assert!(matches!(futures::poll!(&mut acquire), Poll::Pending));
assert_eq!(
manager
.as_fast_lock_manager()
.expect("enabled physical lock backend")
.get_shard(&resource)
.active_guard_count(),
1,
"the tested future owns a physical grant"
);
tokio::time::advance(Duration::from_millis(21)).await;
if unlock_before_poll {
release_table.send(()).expect("make registration ready after its deadline");
table_owner.join().expect("guard-table owner must finish");
assert!(!acquire.await.expect("expired registration response").success);
} else {
let response = timeout(Duration::from_millis(100), &mut acquire)
.await
.expect("registration must honor its deadline while the table remains occupied")
.expect("expired registration response");
assert!(!response.success);
release_table.send(()).expect("release the occupied table");
table_owner.join().expect("guard-table owner must finish");
}
assert!(
client
.check_status(&request.lock_id)
.await
.expect("check the expired identity")
.is_none()
);
assert_eq!(
manager
.as_fast_lock_manager()
.expect("enabled physical lock backend")
.get_shard(&resource)
.active_guard_count(),
0,
"expired registration must release its grant"
);
assert!(
manager
.acquire_lock(crate::ObjectLockRequest::new_write(resource, "next-owner"))
.await
.is_ok(),
"the next owner must not inherit an abandoned physical grant"
);
}
}
#[tokio::test(start_paused = true)]
async fn cancelling_registration_releases_the_unpublished_physical_grant() {
let manager = Arc::new(GlobalLockManager::new());
let client = LocalClient::with_manager(manager.clone());
client.reaper_started.store(true, Ordering::Release);
let resource = crate::ObjectKey::new("bucket", "cancel-registration");
let blocker = manager
.acquire_lock(crate::ObjectLockRequest::new_write(resource.clone(), "blocker"))
.await
.expect("hold the physical lock through registry preflight");
let request = request(resource.clone(), "cancelled", Duration::from_secs(30));
let mut acquire = Box::pin(tokio::task::unconstrained(client.acquire_lock(&request)));
assert!(matches!(futures::poll!(&mut acquire), Poll::Pending));
let (release_table, table_owner) = hold_guard_shard(client.get_shard(&request.lock_id).clone());
drop(blocker);
assert!(matches!(futures::poll!(&mut acquire), Poll::Pending));
let backend = manager.as_fast_lock_manager().expect("enabled physical lock backend");
assert_eq!(backend.get_shard(&resource).active_guard_count(), 1);
drop(acquire);
assert_eq!(backend.get_shard(&resource).active_guard_count(), 0);
assert!(
manager
.acquire_lock(crate::ObjectLockRequest::new_write(resource, "next-owner"))
.await
.is_ok(),
"cancellation must release the physical lock without waiting for the table"
);
release_table.send(()).expect("release the occupied table");
table_owner.join().expect("guard-table owner must finish");
assert!(
client
.check_status(&request.lock_id)
.await
.expect("check cancelled identity")
.is_none()
);
assert!(client.list_lock_leases().await.is_empty());
}
#[tokio::test]
async fn cancelling_a_registry_waiter_keeps_other_waiters_live() {
let shard = Arc::new(GuardRegistry::default());
let (release_table, table_owner) = hold_guard_shard(shard.clone());
let mut cancelled = Box::pin(shard.write());
let mut survivor = Box::pin(shard.read());
assert!(matches!(futures::poll!(&mut cancelled), Poll::Pending));
assert!(matches!(futures::poll!(&mut survivor), Poll::Pending));
drop(cancelled);
release_table.send(()).expect("wake registry waiters");
table_owner.join().expect("guard-table owner must finish");
let access = timeout(Duration::from_millis(100), survivor)
.await
.expect("a cancelled waiter must not consume another waiter's unlock notification");
assert!(access.is_empty());
}
#[tokio::test]
async fn uncontended_registry_access_retains_cooperative_scheduling() {
let shard = GuardRegistry::default();
let mut yielded = false;
for _ in 0..1024 {
match shard.read().now_or_never() {
Some(access) => drop(access),
None => {
yielded = true;
break;
}
}
}
assert!(yielded, "a loop of ready registry operations must still yield to other executor tasks");
}
#[tokio::test]
async fn repeated_shared_identity_replaces_only_its_previous_physical_guard() {
let manager = Arc::new(GlobalLockManager::new());
let client = LocalClient::with_manager(manager.clone());
let request = LockRequest::new(crate::ObjectKey::new("bucket", ""), LockType::Shared, "owner");
assert!(client.acquire_lock(&request).await.expect("first shared grant").success);
assert!(client.acquire_lock(&request).await.expect("repeated shared identity").success);
let backend = manager.as_fast_lock_manager().expect("enabled physical lock backend");
assert_eq!(backend.get_shard(&request.resource).active_guard_count(), 1);
assert_eq!(client.list_lock_leases().await.len(), 1);
assert!(client.release(&request.lock_id).await.expect("release the replacement grant"));
assert_eq!(backend.get_shard(&request.resource).active_guard_count(), 0);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn hot_shared_registry_keeps_refresh_reaper_and_snapshots_live() {
for width in [64, 128] {
let manager = Arc::new(GlobalLockManager::new());
let client = Arc::new(LocalClient::with_manager_and_reaper_interval(manager.clone(), Duration::from_millis(1)));
let resource = crate::ObjectKey::new("bucket", "");
let expired = LockRequest::new(resource.clone(), LockType::Shared, "expired").with_ttl(Duration::ZERO);
assert!(client.acquire_lock(&expired).await.expect("expired seed grant").success);
let acquired = Arc::new(tokio::sync::Barrier::new(width + 1));
let observed = Arc::new(Notify::new());
let mut tasks = tokio::task::JoinSet::new();
for index in 0..width {
let client = client.clone();
let resource = resource.clone();
let acquired = acquired.clone();
let observed = observed.clone();
tasks.spawn(async move {
for iteration in 0..8 {
let request = LockRequest::new(resource.clone(), LockType::Shared, format!("load-{index}"));
assert!(client.acquire_lock(&request).await.expect("concurrent shared grant").success);
if iteration == 0 {
let notification = observed.notified();
tokio::pin!(notification);
notification.as_mut().enable();
acquired.wait().await;
notification.await;
} else {
tokio::task::yield_now().await;
}
assert!(client.refresh(&request.lock_id).await.expect("refresh concurrent grant"));
assert!(
client
.check_status(&request.lock_id)
.await
.expect("read concurrent grant")
.is_some()
);
assert!(client.release(&request.lock_id).await.expect("release concurrent grant"));
}
});
}
timeout(Duration::from_secs(5), acquired.wait())
.await
.expect("all shared grants must reach the snapshot gate");
let snapshot = client.list_lock_leases().await;
assert_eq!(snapshot.iter().filter(|lease| lease.owner.starts_with("load-")).count(), width);
observed.notify_waiters();
let mut completed = 0;
timeout(Duration::from_secs(5), async {
while let Some(result) = tasks.join_next().await {
result.expect("all shared grant workers must finish");
completed += 1;
let _ = client.list_lock_leases().await;
}
})
.await
.expect("refresh, cleanup, and concurrent snapshots must make progress");
assert_eq!(completed, width);
wait_until_reaped(&client, &expired.lock_id).await;
assert!(client.list_lock_leases().await.is_empty());
assert!(
manager
.as_fast_lock_manager()
.expect("enabled physical lock backend")
.list_locks()
.is_empty()
);
}
}
async fn wait_until_reaped(client: &LocalClient, lock_id: &LockId) {
for _ in 0..80 {
if client.check_status(lock_id).await.unwrap().is_none() {
@@ -502,8 +930,9 @@ mod tests {
}
let lock_request = request(resource, "owner", Duration::from_secs(30));
let response = tokio::time::timeout(Duration::from_millis(100), client.acquire_lock(&lock_request))
.await
let response = client
.acquire_lock(&lock_request)
.now_or_never()
.expect("an acquire must not sweep unrelated guard shards")
.expect("local acquire should return a response");
assert!(response.success);
@@ -530,8 +959,9 @@ mod tests {
}
let replacement = request(resource, "replacement-owner", Duration::from_secs(30));
let response = tokio::time::timeout(Duration::from_millis(100), client.acquire_lock(&replacement))
.await
let response = client
.acquire_lock(&replacement)
.now_or_never()
.expect("expired guard reclaim must only write the resource shard")
.expect("local acquire should return a response");
assert!(response.success);
@@ -547,7 +977,7 @@ mod tests {
client.reaper_started.store(true, Ordering::Release);
let resource = crate::ObjectKey::new("bucket", "resource-shard-deadline");
let target_shard = client.get_resource_shard_index(&resource);
let _target_write = client.guard_storage[target_shard].write().await;
let (release_table, table_owner) = hold_guard_shard(client.guard_storage[target_shard].clone());
let lock_request = request(resource, "owner", Duration::from_secs(30)).with_acquire_timeout(Duration::from_millis(10));
let response = tokio::time::timeout(Duration::from_millis(100), client.acquire_lock(&lock_request))
@@ -555,6 +985,8 @@ mod tests {
.expect("acquire should return after its reclaim budget expires")
.expect("local acquire should return a response");
assert!(!response.success);
release_table.send(()).expect("release the occupied resource table");
table_owner.join().expect("guard-table owner must finish");
}
#[tokio::test]
File diff suppressed because it is too large Load Diff
+45 -4
View File
@@ -66,11 +66,26 @@ pub fn select_effective_memory(host_total: u64, host_available: u64, cgroup: Opt
/// Resolves the effective memory from an already-refreshed system handle.
///
/// `cgroup_limits()` is computed fresh on each call and is only implemented on
/// Linux (it returns `None` elsewhere), so non-Linux hosts always use host
/// values.
/// On Linux, the current process cgroup and its ancestors are read on each call,
/// including limits imposed by systemd services. System-wide cgroup limits are
/// used if process information is unavailable. Other platforms use host values.
pub fn effective_memory_from_system(system: &System) -> EffectiveMemory {
let cgroup = system.cgroup_limits().map(|limits| (limits.total_memory, limits.free_memory));
#[cfg(target_os = "linux")]
let limits = sysinfo::get_current_pid()
.ok()
.and_then(|pid| {
let mut processes = System::new().ok()?;
processes.refresh_processes_specifics(
sysinfo::ProcessesToUpdate::Some(&[pid]),
false,
sysinfo::ProcessRefreshKind::nothing().with_memory(),
);
processes.process(pid)?.cgroup_limits()
})
.or_else(|| system.cgroup_limits());
#[cfg(not(target_os = "linux"))]
let limits = system.cgroup_limits();
let cgroup = limits.map(|limits| (limits.total_memory, limits.free_memory));
select_effective_memory(system.total_memory(), system.available_memory(), cgroup)
}
@@ -89,6 +104,32 @@ mod tests {
const GIB: u64 = 1024 * 1024 * 1024;
#[cfg(target_os = "linux")]
#[test]
fn effective_memory_uses_current_process_cgroup() {
use sysinfo::{ProcessRefreshKind, ProcessesToUpdate, System, get_current_pid};
let pid = get_current_pid().expect("current process ID");
let mut system = System::new().expect("system memory handle");
system.refresh_memory();
let mut process_system = System::new().expect("process memory handle");
process_system.refresh_processes_specifics(
ProcessesToUpdate::Some(&[pid]),
false,
ProcessRefreshKind::nothing().with_memory(),
);
let limits = process_system.process(pid).expect("current process entry").cgroup_limits();
let expected = select_effective_memory(
system.total_memory(),
system.available_memory(),
limits.map(|limits| (limits.total_memory, limits.free_memory)),
);
let actual = super::effective_memory_from_system(&system);
assert_eq!(actual.basis, expected.basis);
assert_eq!(actual.total_bytes, expected.total_bytes);
assert!(actual.available_bytes <= actual.total_bytes);
}
#[test]
fn select_effective_memory_prefers_constraining_cgroup() {
let effective = select_effective_memory(64 * GIB, 40 * GIB, Some((2 * GIB, GIB)));
+1
View File
@@ -642,6 +642,7 @@ impl AdminAction {
self,
AdminAction::GetTableBucketAction
| AdminAction::SetTableBucketAction
| AdminAction::MigrateTableCatalogAction
| AdminAction::GetTableNamespaceAction
| AdminAction::SetTableNamespaceAction
| AdminAction::UpdateTableNamespacePropertiesAction
+140
View File
@@ -1795,6 +1795,146 @@ mod test {
Ok(())
}
async fn assert_catalog_migration_policy_matrix(data: &[u8], cases: &[(&str, AdminAction, bool)]) -> Result<()> {
let policy = Policy::parse_config(data)?;
let conditions = HashMap::new();
let claims = HashMap::new();
let groups = None;
for (bucket, action, expected) in cases {
let args = Args {
account: "catalog-migration-operator",
groups: &groups,
action: Action::AdminAction(*action),
bucket,
conditions: &conditions,
is_owner: false,
object: "",
claims: &claims,
deny_only: false,
};
assert_eq!(policy.is_allowed(&args).await, *expected, "{action:?} on warehouse {bucket:?}");
}
Ok(())
}
#[tokio::test]
async fn catalog_migration_allow_resource_is_limited_to_the_requested_warehouse() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "admin:MigrateTableCatalog",
"Resource": "arn:aws:s3:::warehouse-a"
}]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("", AdminAction::MigrateTableCatalogAction, false),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_allow_not_resource_excludes_only_the_named_warehouse() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "admin:MigrateTableCatalog",
"NotResource": "arn:aws:s3:::warehouse-b"
}]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("warehouse-c", AdminAction::MigrateTableCatalogAction, true),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_explicit_deny_resource_does_not_spread_to_other_warehouses() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Action": "admin:*", "Resource": "*"},
{"Effect": "Deny", "Action": "admin:MigrateTableCatalog", "Resource": "arn:aws:s3:::warehouse-b"}
]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("warehouse-c", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::ServerInfoAdminAction, true),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_explicit_deny_not_resource_preserves_its_exception() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Action": "admin:*", "Resource": "*"},
{"Effect": "Deny", "Action": "admin:MigrateTableCatalog", "NotResource": "arn:aws:s3:::warehouse-a"}
]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("warehouse-c", AdminAction::MigrateTableCatalogAction, false),
("warehouse-b", AdminAction::ServerInfoAdminAction, true),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_admin_wildcard_still_honors_the_requested_action_scope() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "admin:*",
"Resource": "arn:aws:s3:::warehouse-a"
}]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("", AdminAction::MigrateTableCatalogAction, false),
("warehouse-b", AdminAction::ServerInfoAdminAction, true),
("", AdminAction::GetTableCatalogAction, true),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_resource_free_admin_grants_keep_their_global_behavior() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [{"Effect": "Allow", "Action": "admin:MigrateTableCatalog"}]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, true),
("", AdminAction::MigrateTableCatalogAction, true),
],
)
.await
}
#[tokio::test]
async fn test_table_admin_action_with_resource_is_limited_to_bucket() -> Result<()> {
use crate::policy::action::{Action, AdminAction};
@@ -43,14 +43,9 @@ const CONTAINER_META_HEADER_PREFIX: &str = "x-container-meta-";
/// Request header prefix removing a container metadata item.
const CONTAINER_META_REMOVE_HEADER_PREFIX: &str = "x-remove-container-meta-";
/// Bucket-tag namespace holding account metadata items.
pub(crate) const ACCOUNT_META_TAG_PREFIX: &str = "swift-account-meta-";
/// Bucket-tag namespace holding container metadata items.
///
/// Deliberately narrower than the `swift-` tags around it: the container ACL
/// (`swift-acl-*`) and versioning (`swift-versions-location`) tags share this
/// tag set and must survive a metadata POST.
pub(crate) const CONTAINER_META_TAG_PREFIX: &str = "swift-meta-";
pub(crate) use super::storage_api::tagging::{
SWIFT_ACCOUNT_META_TAG_PREFIX as ACCOUNT_META_TAG_PREFIX, SWIFT_CONTAINER_META_TAG_PREFIX as CONTAINER_META_TAG_PREFIX,
};
/// The metadata changes carried by one account or container POST.
///
+3 -5
View File
@@ -61,11 +61,9 @@ pub use errors::{SwiftError, SwiftResult};
pub use metadata_update::MetadataUpdate;
pub use router::{SwiftRoute, SwiftRouter};
/// Maximum number of metadata headers allowed per resource (Swift standard)
pub(crate) const MAX_METADATA_COUNT: usize = 90;
/// Maximum size in bytes for a single metadata value (Swift standard)
pub(crate) const MAX_METADATA_VALUE_SIZE: usize = 256;
pub(crate) use storage_api::tagging::{
MAX_SWIFT_METADATA_COUNT as MAX_METADATA_COUNT, MAX_SWIFT_METADATA_VALUE_SIZE as MAX_METADATA_VALUE_SIZE,
};
/// Validate metadata against Swift limits
///
+19 -2
View File
@@ -16,6 +16,7 @@ use std::collections::HashMap;
use std::sync::Arc;
pub(crate) use rustfs_ecstore::api::bucket::metadata::BucketMetadata as SwiftBucketMetadata;
use rustfs_ecstore::api::bucket::metadata::SWIFT_TAGGING_MARKER;
use rustfs_ecstore::api::bucket::metadata::{BUCKET_TAGGING_CONFIG, is_unreadable_config_error};
use rustfs_ecstore::api::bucket::metadata_sys::{get as get_swift_bucket_metadata_from_backend, update_config_with};
use rustfs_ecstore::api::bucket::utils::serialize as serialize_bucket_config;
@@ -25,7 +26,7 @@ use rustfs_ecstore::api::notification::get_global_notification_sys;
pub(crate) use rustfs_ecstore::api::runtime::object_store_handle as resolve_swift_object_store_handle;
use rustfs_ecstore::api::storage::ECStore as SwiftStore;
use rustfs_storage_api as storage_contracts;
use s3s::dto::Tagging;
use s3s::dto::{Tag, Tagging};
use super::{SwiftError, SwiftResult};
@@ -54,6 +55,12 @@ pub(crate) mod public_api {
};
}
pub(crate) mod tagging {
pub(crate) use rustfs_ecstore::api::bucket::metadata::{
MAX_SWIFT_METADATA_COUNT, MAX_SWIFT_METADATA_VALUE_SIZE, SWIFT_ACCOUNT_META_TAG_PREFIX, SWIFT_CONTAINER_META_TAG_PREFIX,
};
}
pub(crate) mod versioning {
pub(crate) use super::storage_contracts::{ListOperations, ObjectOperations};
}
@@ -105,7 +112,7 @@ where
// including the container ACL and versioning tags — because the rewrite
// closures treat "no parsed tags" as "no tags".
let result = update_config_with(&bucket, BUCKET_TAGGING_CONFIG, |bm| {
let tagging = match rewrite(bm.tagging_config.as_ref()) {
let mut tagging = match rewrite(bm.tagging_config.as_ref()) {
Ok(tagging) => tagging,
Err(err) => {
rejected = Some(err);
@@ -116,6 +123,16 @@ where
if tagging.tag_set.is_empty() {
Ok(Vec::new())
} else {
if !tagging
.tag_set
.iter()
.any(|tag| tag.key.as_deref() == Some(SWIFT_TAGGING_MARKER))
{
tagging.tag_set.push(Tag {
key: Some(SWIFT_TAGGING_MARKER.to_owned()),
value: Some("1".to_owned()),
});
}
// The S3 XML serializer, not quick_xml: the metadata loader's
// parse step must be able to round-trip what we persist.
serialize_bucket_config(&tagging)
+16
View File
@@ -82,6 +82,22 @@ Configure WebDAV via environment variables:
| `RUSTFS_WEBDAV_MAX_BODY_SIZE` | Max upload size (bytes) | 5GB |
| `RUSTFS_WEBDAV_REQUEST_TIMEOUT` | Request timeout (seconds) | 300 |
| `RUSTFS_WEBDAV_MAX_CONNECTIONS` | Max concurrently served connections | 1024 |
| `RUSTFS_CORS_ALLOWED_ORIGINS` | Browser origins allowed by the S3 and WebDAV listeners | empty |
## Browser Access and CORS
WebDAV reads `RUSTFS_CORS_ALLOWED_ORIGINS` at server startup. Set a comma-separated list of exact frontend origins to allow browser clients, for example `https://dav-console.example,https://files.example`. Explicit origins allow credentialed requests. The default empty value disables cross-origin access. A standalone `*` permits any origin with credentials mode `omit` and emits no `Access-Control-Allow-Credentials` header.
Valid `OPTIONS` preflight requests from an allowed origin receive `204 No Content` without Basic authentication. Actual WebDAV requests and ordinary `OPTIONS` requests still use Basic authentication and the existing IAM checks. Allowed origins receive CORS headers on success and error responses. Preflights allow WebDAV methods and explicitly echo valid requested header names, including `Authorization`, `Depth`, `Destination`, and `If-Match`. Responses expose `ETag`, `DAV`, `Allow`, `Lock-Token`, and range headers to browser code.
```bash
curl -i -X OPTIONS http://localhost:8080/ \
-H 'Origin: https://dav-console.example' \
-H 'Access-Control-Request-Method: PROPFIND' \
-H 'Access-Control-Request-Headers: authorization,depth,content-type'
```
Configure an explicit HTTPS frontend origin for credentialed browser access and use HTTPS on the WebDAV listener when sending access keys and secret keys. This listener-wide policy applies across WebDAV paths. S3 bucket CORS rules remain scoped to the S3 API.
## Quick Start
+4
View File
@@ -23,6 +23,10 @@ pub enum WebDavInitError {
Bind(#[from] std::io::Error),
#[error("server error: {0}")]
Server(String),
#[error("WebDAV background task failed: {0}")]
TaskJoin(#[from] tokio::task::JoinError),
#[error("WebDAV connections did not drain before the shutdown deadline")]
ShutdownDeadline,
#[error("invalid WebDAV configuration: {0}")]
InvalidConfig(String),
#[error("TLS error: {0}")]
+697 -38
View File
@@ -19,8 +19,12 @@ use crate::common::session::{Protocol, ProtocolPrincipal, SessionContext, is_tem
use bytes::Bytes;
use dav_server::DavHandler;
use dav_server::fakels::FakeLs;
use http::header::{AUTHORIZATION, REFERER, USER_AGENT};
use http::{HeaderMap, HeaderValue};
use http::header::{
ACCESS_CONTROL_ALLOW_CREDENTIALS, ACCESS_CONTROL_ALLOW_HEADERS, ACCESS_CONTROL_ALLOW_METHODS, ACCESS_CONTROL_ALLOW_ORIGIN,
ACCESS_CONTROL_EXPOSE_HEADERS, ACCESS_CONTROL_REQUEST_HEADERS, ACCESS_CONTROL_REQUEST_METHOD, AUTHORIZATION, ORIGIN, REFERER,
USER_AGENT, VARY,
};
use http::{HeaderMap, HeaderValue, Method};
use http_body_util::{BodyExt, Full, LengthLimitError, Limited};
use hyper::body::Body as HttpBody;
use hyper::server::conn::http1;
@@ -31,18 +35,23 @@ use rustfs_config::{DEFAULT_TLS_RELOAD_ENABLE, DEFAULT_TLS_RELOAD_INTERVAL, ENV_
use rustfs_tls_runtime::{ReloadableServerCertResolver, TlsReloadOptions, spawn_server_cert_reload_loop};
use rustfs_utils::MaskedAccessKey;
use std::convert::Infallible;
use std::future::Future;
use std::io;
use std::net::IpAddr;
use std::pin::Pin;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Duration;
use subtle::ConstantTimeEq;
use tokio::net::TcpListener;
use tokio::sync::{Semaphore, broadcast, watch};
use tokio::task::JoinSet;
use tokio::time::timeout;
use tokio_rustls::TlsAcceptor;
use tracing::{Instrument, debug, error, info, info_span, warn};
const CONNECTION_DRAIN_TIMEOUT: Duration = Duration::from_secs(10);
const LOG_COMPONENT_PROTOCOLS: &str = "protocols";
const LOG_SUBSYSTEM_WEBDAV_SERVER: &str = "webdav_server";
const LOG_SUBSYSTEM_WEBDAV_AUTH: &str = "webdav_auth";
@@ -60,6 +69,177 @@ const EVENT_WEBDAV_CONNECTION_CAP_STATE: &str = "webdav_connection_cap_state";
/// materialise a whole object in memory for every GET.
type WebDavBody = Pin<Box<dyn HttpBody<Data = Bytes, Error = io::Error> + Send>>;
/// Listener-wide browser policy, parsed once and shared across connections.
enum WebDavCors {
Disabled,
Wildcard,
Origins(Vec<HeaderValue>),
}
impl WebDavCors {
const METHODS: &str = "OPTIONS, GET, HEAD, PUT, DELETE, PATCH, PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, UNLOCK";
fn new(origins: &str) -> Self {
match origins.trim() {
"" => Self::Disabled,
"*" => Self::Wildcard,
origins => Self::Origins(
origins
.split(',')
.map(str::trim)
.filter(|origin| !origin.is_empty() && *origin != "*")
.filter_map(|origin| HeaderValue::from_str(origin).ok())
.collect(),
),
}
}
fn allowed_origin(&self, headers: &HeaderMap) -> Option<HeaderValue> {
if matches!(self, Self::Disabled) {
return None;
}
let mut origins = headers.get_all(ORIGIN).iter();
let origin = origins.next()?;
let value = origin.to_str().ok()?;
if origins.next().is_some() || value.is_empty() || value.bytes().any(|byte| byte.is_ascii_whitespace() || byte == b',') {
return None;
}
match self {
Self::Origins(allowed) => allowed.iter().find(|allowed| *allowed == origin).cloned(),
Self::Wildcard => Some(HeaderValue::from_static("*")),
Self::Disabled => None,
}
}
/// Keep the CORS boundary outside authentication and DAV dispatch so every
/// application response, including early errors, follows the same policy.
async fn handle<B, F, Fut>(&self, req: Request<B>, next: F) -> Result<Response<WebDavBody>, Infallible>
where
F: FnOnce(Request<B>) -> Fut,
Fut: Future<Output = Result<Response<WebDavBody>, Infallible>>,
{
let allow_origin = self.allowed_origin(req.headers());
let is_options = req.method() == Method::OPTIONS;
let is_preflight =
is_options && req.headers().contains_key(ORIGIN) && req.headers().contains_key(ACCESS_CONTROL_REQUEST_METHOD);
let mut response = if is_preflight {
self.preflight(req.headers(), allow_origin.is_some())
} else {
next(req).await?
};
// Absent and rejected Origin values also select cache variants. Append
// rather than overwrite the DAV handler's existing Vary fields.
response.headers_mut().append(VARY, HeaderValue::from_static("Origin"));
if is_options {
response.headers_mut().append(
VARY,
HeaderValue::from_static("Access-Control-Request-Method, Access-Control-Request-Headers"),
);
}
if let Some(origin) = allow_origin {
response.headers_mut().insert(ACCESS_CONTROL_ALLOW_ORIGIN, origin);
if matches!(self, Self::Origins(_)) {
response
.headers_mut()
.insert(ACCESS_CONTROL_ALLOW_CREDENTIALS, HeaderValue::from_static("true"));
}
if !is_preflight {
response.headers_mut().insert(
ACCESS_CONTROL_EXPOSE_HEADERS,
HeaderValue::from_static("ETag, DAV, Allow, Lock-Token, Content-Range, Accept-Ranges, WWW-Authenticate"),
);
}
}
Ok(response)
}
fn preflight(&self, headers: &HeaderMap, origin_allowed: bool) -> Response<WebDavBody> {
let mut methods = headers.get_all(ACCESS_CONTROL_REQUEST_METHOD).iter();
let method = methods.next().and_then(|value| value.to_str().ok());
let method_allowed = method.is_some_and(|method| Self::METHODS.split(", ").any(|allowed| allowed == method));
if !origin_allowed || !method_allowed || methods.next().is_some() {
return error_response(StatusCode::FORBIDDEN, "CORS preflight denied");
}
let Ok(allowed_headers) = cors_requested_headers(headers) else {
return error_response(StatusCode::FORBIDDEN, "CORS preflight denied");
};
let mut response = Response::new(fixed_body(Bytes::new()));
*response.status_mut() = StatusCode::NO_CONTENT;
response
.headers_mut()
.insert(ACCESS_CONTROL_ALLOW_METHODS, HeaderValue::from_static(Self::METHODS));
// A wildcard does not cover Authorization, and is not a wildcard for
// credentialed requests. Echo the validated names explicitly instead.
if let Some(value) = allowed_headers {
response.headers_mut().insert(ACCESS_CONTROL_ALLOW_HEADERS, value);
}
response
}
}
/// Validate borrowed field names, reusing an ordinary single header value.
/// RFC 9110 section 5.6.1.2 requires recipients to ignore empty list elements;
/// normalize those and repeated fields into one nonempty response list.
fn cors_requested_headers(headers: &HeaderMap) -> Result<Option<HeaderValue>, ()> {
let values = headers.get_all(ACCESS_CONTROL_REQUEST_HEADERS);
let mut fields = values.iter();
let Some(first) = fields.next() else {
return Ok(None);
};
let mut normalize = fields.next().is_some();
let mut has_name = false;
for value in values.iter() {
for name in value.to_str().map_err(|_| ())?.split(',') {
let name = name.trim_matches([' ', '\t']);
if name.is_empty() {
normalize = true;
continue;
}
if !is_cors_header_name(name) {
return Err(());
}
has_name = true;
}
}
if !has_name {
return Err(());
}
if !normalize {
return Ok(Some(first.clone()));
}
let capacity = values.iter().map(|value| value.as_bytes().len() + 1).sum();
let mut normalized = String::with_capacity(capacity);
for value in values {
for name in value.to_str().map_err(|_| ())?.split(',') {
let name = name.trim_matches([' ', '\t']);
if name.is_empty() {
continue;
}
if !normalized.is_empty() {
normalized.push(',');
}
normalized.push_str(name);
}
}
HeaderValue::try_from(normalized).map(Some).map_err(|_| ())
}
/// RFC 9110 field-name is a nonempty sequence of ASCII tchar bytes.
fn is_cors_header_name(name: &str) -> bool {
!name.is_empty()
&& name.bytes().all(|byte| {
byte.is_ascii_alphanumeric()
|| matches!(
byte,
b'!' | b'#' | b'$' | b'%' | b'&' | b'\'' | b'*' | b'+' | b'-' | b'.' | b'^' | b'_' | b'`' | b'|' | b'~'
)
})
}
fn policy_request_headers(headers: &HeaderMap) -> HeaderMap {
let mut policy_headers = HeaderMap::new();
for name in [USER_AGENT, REFERER] {
@@ -74,6 +254,11 @@ fn policy_request_headers(headers: &HeaderMap) -> HeaderMap {
policy_headers
}
struct ConnectionLimits {
max_body_size: u64,
request_timeout: Duration,
}
/// WebDAV server implementation
pub struct WebDavServer<S>
where
@@ -83,6 +268,7 @@ where
config: WebDavConfig,
/// S3 storage backend
storage: S,
cors: Arc<WebDavCors>,
}
impl<S> WebDavServer<S>
@@ -100,7 +286,13 @@ where
/// Create a new WebDAV server
pub async fn new(config: WebDavConfig, storage: S) -> Result<Self, WebDavInitError> {
config.validate().await?;
Ok(Self { config, storage })
let origins =
rustfs_utils::get_env_str(rustfs_config::ENV_CORS_ALLOWED_ORIGINS, rustfs_config::DEFAULT_CORS_ALLOWED_ORIGINS);
Ok(Self {
config,
storage,
cors: Arc::new(WebDavCors::new(&origins)),
})
}
/// Start the WebDAV server
@@ -135,7 +327,8 @@ where
bind_addr = %self.config.bind_addr,
"WebDAV server listening"
);
let (reload_shutdown_tx, reload_shutdown_rx) = watch::channel(false);
let (shutdown_tx, shutdown) = watch::channel(false);
let mut reload_task = None;
// Setup TLS if enabled
let tls_acceptor = if self.config.tls_enabled {
@@ -151,15 +344,9 @@ where
let resolver = ReloadableServerCertResolver::load_from_directory(cert_dir)
.map_err(|e| WebDavInitError::Tls(format!("Failed to create certificate resolver: {}", e)))?;
let _reload_task = spawn_server_cert_reload_loop(
"webdav",
resolver.clone(),
Self::tls_reload_options(),
reload_shutdown_rx.clone(),
);
let server_config = crate::common::tls::server_config(resolver, self.config.ca_file.as_deref())
let server_config = crate::common::tls::server_config(resolver.clone(), self.config.ca_file.as_deref())
.map_err(|error| WebDavInitError::Tls(format!("Invalid client CA configuration: {error}")))?;
reload_task = spawn_server_cert_reload_loop("webdav", resolver, Self::tls_reload_options(), shutdown.clone());
Some(TlsAcceptor::from(Arc::new(server_config)))
} else {
@@ -173,7 +360,15 @@ where
let request_timeout = Duration::from_secs(self.config.request_timeout_secs);
let connection_limiter = Arc::new(Semaphore::new(self.config.max_connections.min(Semaphore::MAX_PERMITS)));
loop {
let mut connections = JoinSet::new();
let mut failure = None;
'accept: loop {
while let Some(result) = connections.try_join_next() {
if let Err(error) = result {
failure = Some(WebDavInitError::TaskJoin(error));
break 'accept;
}
}
// Admission control: hold a permit before accepting, so at
// saturation the kernel backlog absorbs the burst instead of the
// process spawning an unbounded number of connection tasks. The
@@ -204,7 +399,6 @@ where
state = "shutdown_requested",
"WebDAV shutdown requested"
);
let _ = reload_shutdown_tx.send(true);
break;
}
}
@@ -216,25 +410,35 @@ where
match accept_result {
Ok((stream, addr)) => {
let storage = storage.clone();
let cors = self.cors.clone();
let tls_acceptor = tls_acceptor.clone();
let max_body_size = self.config.max_body_size;
let limits = ConnectionLimits {
max_body_size: self.config.max_body_size,
request_timeout,
};
let source_ip: IpAddr = addr.ip();
let span = info_span!(
"webdav-connection",
peer = %source_ip,
transport = if tls_acceptor.is_some() { "tls" } else { "tcp" },
);
tokio::spawn(
let mut shutdown = shutdown.clone();
connections.spawn(
async move {
let _permit = permit;
if let Some(acceptor) = tls_acceptor {
// A handshake that never completes would otherwise
// hold its connection permit forever.
match timeout(request_timeout, acceptor.accept(stream)).await {
let handshake = tokio::select! {
biased;
_ = shutdown.changed() => return,
result = timeout(request_timeout, acceptor.accept(stream)) => result,
};
match handshake {
Ok(Ok(tls_stream)) => {
let io = TokioIo::new(tls_stream);
if let Err(e) = Self::handle_connection_impl(io, storage, source_ip, true, max_body_size, request_timeout).await {
if let Err(e) = Self::handle_connection_impl(io, storage, source_ip, true, limits, cors, shutdown).await {
debug!(
event = EVENT_WEBDAV_CONNECTION_STATE,
component = LOG_COMPONENT_PROTOCOLS,
@@ -272,7 +476,7 @@ where
}
} else {
let io = TokioIo::new(stream);
if let Err(e) = Self::handle_connection_impl(io, storage, source_ip, false, max_body_size, request_timeout).await {
if let Err(e) = Self::handle_connection_impl(io, storage, source_ip, false, limits, cors, shutdown).await {
debug!(
event = EVENT_WEBDAV_CONNECTION_STATE,
component = LOG_COMPONENT_PROTOCOLS,
@@ -309,13 +513,29 @@ where
state = "shutdown_requested",
"WebDAV shutdown requested"
);
let _ = reload_shutdown_tx.send(true);
break;
}
}
}
let _ = reload_shutdown_tx.send(true);
drop(listener);
shutdown_tx.send_replace(true);
let drain = async {
while let Some(result) = connections.join_next().await {
if let Err(error) = result {
failure.get_or_insert(WebDavInitError::TaskJoin(error));
}
}
};
if timeout(CONNECTION_DRAIN_TIMEOUT, drain).await.is_err() {
connections.shutdown().await;
failure.get_or_insert(WebDavInitError::ShutdownDeadline);
}
if let Some(task) = reload_task
&& let Err(error) = task.await
{
failure.get_or_insert(WebDavInitError::TaskJoin(error));
}
info!(
event = EVENT_WEBDAV_SERVER_STATE,
component = LOG_COMPONENT_PROTOCOLS,
@@ -323,7 +543,7 @@ where
state = "stopped",
"WebDAV server stopped"
);
Ok(())
failure.map_or(Ok(()), Err)
}
/// Handle a single connection with hyper-util TokioIo wrapper
@@ -332,25 +552,49 @@ where
storage: S,
source_ip: IpAddr,
secure_transport: bool,
max_body_size: u64,
request_timeout: Duration,
limits: ConnectionLimits,
cors: Arc<WebDavCors>,
mut shutdown: watch::Receiver<bool>,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>>
where
I: tokio::io::AsyncRead + tokio::io::AsyncWrite + Unpin + Send + 'static,
{
let service = service_fn(move |req: Request<hyper::body::Incoming>| {
let storage = storage.clone();
async move { Self::handle_request(req, storage, source_ip, secure_transport, max_body_size, request_timeout).await }
let ConnectionLimits {
max_body_size,
request_timeout,
} = limits;
// These borrows live for the connection. Preflights complete before
// invoking the closure that clones the backend for an actual request.
let cors = cors.as_ref();
let storage = &storage;
let received_request = AtomicBool::new(false);
let service = service_fn(|req: Request<hyper::body::Incoming>| {
received_request.store(true, Ordering::Relaxed);
cors.handle(req, move |req| {
Self::handle_request(req, storage.clone(), source_ip, secure_transport, max_body_size, request_timeout)
})
});
// A peer that opens a connection and dribbles (or never finishes)
// request headers is disconnected once the configured request
// timeout elapses. The timer is required for the deadline to apply.
http1::Builder::new()
let connection = http1::Builder::new()
.timer(TokioTimer::new())
.header_read_timeout(request_timeout)
.serve_connection(io, service)
.await?;
.serve_connection(io, service);
tokio::pin!(connection);
tokio::select! {
biased;
_ = shutdown.changed() => {
// Hyper starts in a busy state even before parsing the first
// request. An incomplete header has no application work to drain.
if received_request.load(Ordering::Relaxed) {
connection.as_mut().graceful_shutdown();
connection.await?;
}
}
result = &mut connection => result?,
}
Ok(())
}
@@ -693,6 +937,7 @@ mod tests {
use s3s::dto::*;
use std::fmt::{Debug, Formatter};
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::task::{Context, Poll};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
@@ -701,8 +946,19 @@ mod tests {
/// Storage double for the connection-level tests. Those requests are
/// rejected before authentication succeeds, so storage is never reached.
#[derive(Clone)]
struct StubStorage;
#[derive(Default)]
struct StubStorage {
clone_count: Arc<AtomicUsize>,
}
impl Clone for StubStorage {
fn clone(&self) -> Self {
self.clone_count.fetch_add(1, Ordering::Relaxed);
Self {
clone_count: self.clone_count.clone(),
}
}
}
impl Debug for StubStorage {
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
@@ -873,6 +1129,362 @@ mod tests {
.expect("build get request")
}
const BROWSER_ORIGIN: &str = "https://dav-console.example";
async fn cors_http_request(origins: &str, method: &str, headers: &str) -> String {
cors_http_exchange(origins, method, headers).await.0
}
async fn cors_http_exchange(origins: &str, method: &str, headers: &str) -> (String, usize) {
let storage = StubStorage::default();
let clone_count = storage.clone_count.clone();
let (mut client, server) = tokio::io::duplex(8192);
let (_shutdown_tx, shutdown) = tokio::sync::watch::channel(false);
let connection = tokio::spawn(WebDavServer::<StubStorage>::handle_connection_impl(
TokioIo::new(server),
storage,
TEST_IP,
false,
ConnectionLimits {
max_body_size: 16,
request_timeout: Duration::from_secs(30),
},
Arc::new(WebDavCors::new(origins)),
shutdown,
));
let request = format!("{method} / HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n{headers}\r\n");
client.write_all(request.as_bytes()).await.expect("write browser request");
let mut response = Vec::new();
timeout(Duration::from_secs(5), client.read_to_end(&mut response))
.await
.expect("browser response timeout")
.expect("read browser response");
timeout(Duration::from_secs(5), connection)
.await
.expect("connection shutdown timeout")
.expect("connection task")
.expect("serve browser request");
(
String::from_utf8(response).expect("ASCII HTTP response"),
clone_count.load(Ordering::Relaxed),
)
}
fn browser_request(method: &str, path: &str) -> Request<Full<Bytes>> {
Request::builder()
.method(method)
.uri(path)
.header(ORIGIN, BROWSER_ORIGIN)
.body(Full::new(Bytes::new()))
.expect("build browser request")
}
fn assert_browser_headers(response: &Response<WebDavBody>) {
assert_eq!(response.headers()[ACCESS_CONTROL_ALLOW_ORIGIN], BROWSER_ORIGIN);
assert_eq!(response.headers()[ACCESS_CONTROL_ALLOW_CREDENTIALS], "true");
assert!(response.headers().get_all(VARY).iter().any(|value| value == "Origin"));
}
#[tokio::test]
async fn webdav_cors_preflight_bypasses_authentication() {
for method in [
"OPTIONS",
"GET",
"HEAD",
"PUT",
"DELETE",
"PATCH",
"PROPFIND",
"PROPPATCH",
"MKCOL",
"COPY",
"MOVE",
"LOCK",
"UNLOCK",
] {
let headers = format!(
"Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: {method}\r\nAccess-Control-Request-Headers: authorization,depth,content-type,destination,if-match\r\n"
);
let response = cors_http_request(BROWSER_ORIGIN, "OPTIONS", &headers).await;
assert!(response.starts_with("HTTP/1.1 204"), "{method}: {response}");
assert!(response.contains(&format!("access-control-allow-origin: {BROWSER_ORIGIN}\r\n")));
assert!(response.contains("access-control-allow-credentials: true\r\n"));
let allowed_methods = response
.lines()
.find_map(|line| line.strip_prefix("access-control-allow-methods: "))
.expect("preflight must advertise allowed methods");
assert!(allowed_methods.split(',').map(str::trim).any(|allowed| allowed == method));
assert!(response.contains("access-control-allow-headers: authorization,depth,content-type,destination,if-match\r\n"));
assert!(response.contains("vary: Access-Control-Request-Method, Access-Control-Request-Headers\r\n"));
assert!(!response.contains("www-authenticate:"));
}
}
#[tokio::test]
async fn webdav_cors_actual_requests_and_plain_options_require_authentication() {
for method in ["GET", "PUT", "DELETE", "PROPFIND", "MOVE", "OPTIONS"] {
let response = cors_http_request(BROWSER_ORIGIN, method, &format!("Origin: {BROWSER_ORIGIN}\r\n")).await;
assert!(response.starts_with("HTTP/1.1 401"), "{method}: {response}");
assert!(response.contains(&format!("access-control-allow-origin: {BROWSER_ORIGIN}\r\n")));
assert!(response.contains("www-authenticate: Basic"));
}
for method in ["GET", "PROPFIND"] {
for headers in ["", "Origin: https://other.example\r\n"] {
let response = cors_http_request(BROWSER_ORIGIN, method, headers).await;
assert!(response.starts_with("HTTP/1.1 401"), "{method}: {response}");
assert!(!response.contains("access-control-allow-origin:"));
assert!(response.contains("vary: Origin\r\n"));
}
}
for headers in [
format!("Origin: {BROWSER_ORIGIN}\r\nAuthorization: Basic !!!!\r\n"),
format!("Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PUT\r\n"),
] {
let response = cors_http_request(BROWSER_ORIGIN, "PUT", &headers).await;
assert!(response.starts_with("HTTP/1.1 401"));
assert!(response.contains(&format!("access-control-allow-origin: {BROWSER_ORIGIN}\r\n")));
}
let response = cors_http_request(BROWSER_ORIGIN, "OPTIONS", "Access-Control-Request-Method: PROPFIND\r\n").await;
assert!(response.starts_with("HTTP/1.1 401"));
assert!(!response.contains("access-control-allow-origin:"));
assert!(response.contains("vary: Origin\r\n"));
assert!(response.contains("vary: Access-Control-Request-Method, Access-Control-Request-Headers\r\n"));
}
#[tokio::test]
async fn webdav_cors_keeps_early_body_limit_errors_visible() {
let response =
cors_http_request(BROWSER_ORIGIN, "PUT", &format!("Origin: {BROWSER_ORIGIN}\r\nContent-Length: 17\r\n")).await;
assert!(response.starts_with("HTTP/1.1 413"));
assert!(response.contains(&format!("access-control-allow-origin: {BROWSER_ORIGIN}\r\n")));
}
#[tokio::test]
async fn webdav_cors_rejects_unconfigured_and_unmatched_origins() {
for (configuration, origin) in [
("", BROWSER_ORIGIN),
(" ", BROWSER_ORIGIN),
(BROWSER_ORIGIN, "https://dav-console.example.attacker.test"),
(BROWSER_ORIGIN, "null"),
(BROWSER_ORIGIN, "https://other.example"),
("*, https://other.example", BROWSER_ORIGIN),
] {
let headers = format!("Origin: {origin}\r\nAccess-Control-Request-Method: PROPFIND\r\n");
let response = cors_http_request(configuration, "OPTIONS", &headers).await;
assert!(response.starts_with("HTTP/1.1 403"), "{configuration:?}: {response}");
assert!(!response.contains("access-control-allow-origin:"));
assert!(!response.contains("access-control-allow-credentials:"));
assert!(response.contains("vary: Origin\r\n"));
}
}
#[tokio::test]
async fn webdav_cors_rejects_malformed_preflights() {
for headers in [
format!("Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: TRACE\r\n"),
format!("Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: propfind\r\n"),
format!(
"Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PROPFIND\r\nAccess-Control-Request-Method: GET\r\n"
),
format!("Origin: {BROWSER_ORIGIN}\r\nOrigin: https://other.example\r\nAccess-Control-Request-Method: PROPFIND\r\n"),
format!("Origin: {BROWSER_ORIGIN},https://other.example\r\nAccess-Control-Request-Method: PROPFIND\r\n"),
format!(
"Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PROPFIND\r\nAccess-Control-Request-Headers: invalid header\r\n"
),
format!(
"Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PROPFIND\r\nAccess-Control-Request-Headers: , \t,\r\n"
),
] {
let response = cors_http_request(BROWSER_ORIGIN, "OPTIONS", &headers).await;
assert!(response.starts_with("HTTP/1.1 403"), "{headers}: {response}");
assert!(!response.contains("access-control-allow-methods:"));
}
}
#[tokio::test]
async fn webdav_cors_normalizes_header_lists() {
for requested in [
"authorization, ,depth,",
" , authorization,\tdepth , ",
"authorization,\r\nAccess-Control-Request-Headers: , depth,",
] {
let headers = format!(
"Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PROPFIND\r\nAccess-Control-Request-Headers: {requested}\r\n"
);
let response = cors_http_request(BROWSER_ORIGIN, "OPTIONS", &headers).await;
assert!(response.starts_with("HTTP/1.1 204"), "{response}");
assert!(response.contains("access-control-allow-headers: authorization,depth\r\n"));
}
}
#[test]
fn webdav_cors_header_name_validation_matches_http_tokens() {
for byte in u8::MIN..=u8::MAX {
let bytes = [b'x', byte, b'y'];
let actual = std::str::from_utf8(&bytes).is_ok_and(is_cors_header_name);
assert_eq!(actual, http::HeaderName::from_bytes(&bytes).is_ok(), "byte {byte}");
}
assert!(!is_cors_header_name(""));
assert!(is_cors_header_name("Authorization"));
assert!(is_cors_header_name("!#$%&'*+-.^_`|~"));
}
#[tokio::test]
async fn webdav_cors_only_clones_backend_for_actual_requests() {
for (method, requested_method, expected_clones, expected_status) in [
("OPTIONS", Some("PROPFIND"), 0, "204"),
("OPTIONS", Some("TRACE"), 0, "403"),
("OPTIONS", None, 1, "401"),
("PUT", Some("PUT"), 1, "401"),
] {
let mut headers = format!("Origin: {BROWSER_ORIGIN}\r\n");
if let Some(requested_method) = requested_method {
headers.push_str(&format!("Access-Control-Request-Method: {requested_method}\r\n"));
}
let (response, clones) = cors_http_exchange(BROWSER_ORIGIN, method, &headers).await;
assert!(response.starts_with(&format!("HTTP/1.1 {expected_status}")), "{response}");
assert_eq!(clones, expected_clones, "{method} with {requested_method:?}");
}
}
#[tokio::test]
async fn webdav_cors_wildcard_never_allows_credentials() {
let headers = format!(
"Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PROPFIND\r\nAccess-Control-Request-Headers: Authorization\r\n"
);
let preflight = cors_http_request(" * ", "OPTIONS", &headers).await;
assert!(preflight.starts_with("HTTP/1.1 204"));
assert!(preflight.contains("access-control-allow-origin: *\r\n"));
assert!(preflight.contains("access-control-allow-headers: Authorization\r\n"));
assert!(!preflight.contains("access-control-allow-credentials:"));
let actual = cors_http_request("*", "PROPFIND", &format!("Origin: {BROWSER_ORIGIN}\r\n")).await;
assert!(actual.starts_with("HTTP/1.1 401"));
assert!(actual.contains("access-control-allow-origin: *\r\n"));
assert!(!actual.contains("access-control-allow-credentials:"));
}
#[tokio::test]
async fn webdav_cors_allowlist_trims_entries_and_matches_exactly() {
let configuration = format!(" https://other.example , {BROWSER_ORIGIN} ");
let headers = format!("Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PROPFIND\r\n");
let response = cors_http_request(&configuration, "OPTIONS", &headers).await;
assert!(response.starts_with("HTTP/1.1 204"));
assert!(response.contains(&format!("access-control-allow-origin: {BROWSER_ORIGIN}\r\n")));
}
#[tokio::test]
async fn webdav_cors_wraps_dav_multistatus_etag_and_streamed_body() {
let cors = WebDavCors::new(BROWSER_ORIGIN);
let handler = memfs_handler();
let created = dispatch_dav(chunked_request(2, 4), handler.clone(), TEST_IP, 16, Duration::from_secs(30)).await;
assert_eq!(created.status(), StatusCode::CREATED);
let propfind = cors
.handle(browser_request("PROPFIND", "/"), |req| async {
Ok(dispatch_dav(req, handler.clone(), TEST_IP, 16, Duration::from_secs(30)).await)
})
.await
.expect("PROPFIND response");
assert_eq!(propfind.status(), StatusCode::MULTI_STATUS);
assert_browser_headers(&propfind);
let get = cors
.handle(browser_request("GET", "/upload.bin"), |req| async {
Ok(dispatch_dav(req, handler.clone(), TEST_IP, 16, Duration::from_secs(30)).await)
})
.await
.expect("GET response");
assert_eq!(get.status(), StatusCode::OK);
assert_browser_headers(&get);
assert!(get.headers().contains_key("etag"));
assert!(
get.headers()[ACCESS_CONTROL_EXPOSE_HEADERS]
.to_str()
.expect("exposed headers")
.contains("ETag")
);
assert!(get.body().size_hint().upper().is_none(), "CORS must preserve streaming");
let data = get.into_body().collect().await.expect("stream GET body").to_bytes();
assert_eq!(data.as_ref(), b"aaaaaaaa");
let mut put = browser_request("PUT", "/upload.bin");
put.headers_mut()
.insert("if-match", HeaderValue::from_static("\"stale-etag\""));
let conditional = cors
.handle(put, |req| async {
Ok(dispatch_dav(req, handler, TEST_IP, 16, Duration::from_secs(30)).await)
})
.await
.expect("conditional PUT response");
assert_eq!(conditional.status(), StatusCode::PRECONDITION_FAILED);
assert_browser_headers(&conditional);
}
#[tokio::test]
async fn webdav_cors_wraps_dispatch_body_limit_errors() {
let cors = WebDavCors::new(BROWSER_ORIGIN);
let mut request = chunked_request(8, 8);
request.headers_mut().insert(ORIGIN, HeaderValue::from_static(BROWSER_ORIGIN));
let oversized = cors
.handle(request, |req| async {
Ok(dispatch_dav(req, memfs_handler(), TEST_IP, 16, Duration::from_secs(30)).await)
})
.await
.expect("oversized response");
assert_eq!(oversized.status(), StatusCode::PAYLOAD_TOO_LARGE);
assert_browser_headers(&oversized);
}
#[tokio::test(start_paused = true)]
async fn webdav_cors_wraps_dispatch_timeout_errors() {
let cors = WebDavCors::new(BROWSER_ORIGIN);
let request = Request::builder()
.method("PUT")
.uri("/stalled.bin")
.header(ORIGIN, BROWSER_ORIGIN)
.body(StalledBody)
.expect("stalled browser request");
let response = cors
.handle(request, |req| async {
Ok(dispatch_dav(req, memfs_handler(), TEST_IP, 16, Duration::from_secs(30)).await)
})
.await
.expect("timeout response");
assert_eq!(response.status(), StatusCode::REQUEST_TIMEOUT);
assert_browser_headers(&response);
}
#[tokio::test]
async fn webdav_cors_preserves_existing_vary_on_success_and_errors() {
let cors = WebDavCors::new(BROWSER_ORIGIN);
for status in [
StatusCode::OK,
StatusCode::FORBIDDEN,
StatusCode::NOT_FOUND,
StatusCode::INTERNAL_SERVER_ERROR,
] {
let response = cors
.handle(browser_request("GET", "/"), |_| async {
let mut response = error_response(status, "test response");
response
.headers_mut()
.append(VARY, HeaderValue::from_static("Accept-Encoding"));
Ok(response)
})
.await
.expect("wrapped response");
assert_eq!(response.status(), status);
assert_browser_headers(&response);
assert!(
response
.headers()
.get_all(VARY)
.iter()
.any(|value| value == "Accept-Encoding")
);
}
}
#[test]
fn policy_headers_drop_credentials_and_s3_auth_spoofing() {
let mut headers = HeaderMap::new();
@@ -969,13 +1581,18 @@ mod tests {
async fn slow_request_headers_hit_request_timeout() {
let (mut client, server) = tokio::io::duplex(1024);
let (_shutdown_tx, shutdown) = tokio::sync::watch::channel(false);
let conn = tokio::spawn(WebDavServer::<StubStorage>::handle_connection_impl(
TokioIo::new(server),
StubStorage,
StubStorage::default(),
TEST_IP,
false,
1024,
Duration::from_secs(30),
ConnectionLimits {
max_body_size: 1024,
request_timeout: Duration::from_secs(30),
},
Arc::new(WebDavCors::new("")),
shutdown,
));
client
@@ -987,6 +1604,40 @@ mod tests {
assert!(outcome.is_ok(), "connection outlived the configured request timeout");
}
#[tokio::test]
async fn shutdown_closes_connection_with_incomplete_first_headers() {
use std::future::{Future, poll_fn};
use std::task::Poll;
let (mut client, server) = tokio::io::duplex(1024);
let (shutdown_tx, shutdown) = watch::channel(false);
let connection = WebDavServer::<StubStorage>::handle_connection_impl(
TokioIo::new(server),
StubStorage::default(),
TEST_IP,
false,
ConnectionLimits {
max_body_size: 1024,
request_timeout: Duration::from_secs(30),
},
Arc::new(WebDavCors::new("")),
shutdown,
);
tokio::pin!(connection);
client
.write_all(b"GET / HTTP/1.1\r\nHost: localhost\r\n")
.await
.expect("write partial headers");
let pending = poll_fn(|cx| Poll::Ready(connection.as_mut().poll(cx).is_pending())).await;
assert!(pending, "connection should be waiting for the rest of the first header");
shutdown_tx.send_replace(true);
timeout(Duration::from_secs(1), connection.as_mut())
.await
.expect("shutdown should not wait for an incomplete header")
.expect("no request reached the service");
}
/// R05-CAN-097: the accept loop must not serve more connections at once
/// than the configured cap.
#[tokio::test]
@@ -1000,7 +1651,7 @@ mod tests {
request_timeout_secs: 30,
max_connections: 1,
};
let server = WebDavServer::new(config, StubStorage).await.expect("build server");
let server = WebDavServer::new(config, StubStorage::default()).await.expect("build server");
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await.expect("bind listener");
let addr = listener.local_addr().expect("listener addr");
@@ -1035,8 +1686,16 @@ mod tests {
.expect("read second");
assert!(served > 0, "second connection returned no data");
let _ = shutdown_tx.send(());
let _ = timeout(Duration::from_secs(10), serving).await;
shutdown_tx.send(()).expect("signal server");
timeout(Duration::from_secs(5), serving)
.await
.expect("server did not retire the keep-alive connection")
.expect("server task panicked")
.expect("server did not shut down cleanly");
let remaining = timeout(Duration::from_secs(5), second.read_to_end(&mut Vec::new()))
.await
.expect("connection remained open after server exit");
remaining.expect("read remaining response and EOF");
}
#[tokio::test]
@@ -21,5 +21,7 @@
//! Swift implementation.
pub use rustfs_ecstore::api::bucket::metadata::load_bucket_metadata;
pub use rustfs_ecstore::api::bucket::metadata_sys::{get as get_bucket_metadata, set_bucket_metadata};
pub use rustfs_ecstore::api::bucket::metadata_sys::{
get as get_bucket_metadata, get_s3_tagging_config, set_bucket_metadata, update_s3_tagging_config,
};
pub use rustfs_ecstore::api::runtime::object_store_handle;

Some files were not shown because too many files have changed in this diff Show More