Compare commits

...

31 Commits

Author SHA1 Message Date
唐小鸭 b1cdaed49e fix(targets): resolve default queue dirs per node
The built-in notify and audit queue directories /opt/rustfs/events and
/opt/rustfs/audit are persisted cluster-wide as literal defaults. On
native Windows they have no drive prefix, so every target that kept the
default failed validation as a non-absolute path.

Treat those two literals as logical defaults and resolve them on each
node when a queue directory is validated or opened. A new
RUSTFS_TARGET_STORE_DIR relocates both default queues on any platform;
without it, Windows uses %ProgramData%\RustFS. Explicit operator paths,
the persisted configuration, and Linux/macOS behavior are unchanged, and
pending entries are not migrated when the root changes.

Expose the variable through the Helm chart as
config.rustfs.target_store_dir and document the resolution rules.
2026-11-08 23:08:35 +08:00
cxymds addf7a6219 fix: reduce scanner backoff and classify downstream closures (#9235) 2026-11-08 02:18:17 -05:00
Ramakrishna Chilaka 420cf32b2a perf(ecstore): optimize PUT, DELETE, and multipart commit hotpaths (#9227) 2026-11-08 02:12:40 -05:00
Chris An 907410b2f5 fix(select): bound Parquet allocations before decoding (#9231) 2026-11-08 01:36:12 -05:00
Chris An f10079f1a3 fix(rpc): size and reserve replay cache for compact entries (#9233) 2026-11-08 01:35:58 -05:00
Chris An a6c3016493 fix(cache): respect process cgroup memory limits (#9232) 2026-11-08 01:35:49 -05:00
Hauser Zhi 68fb52b752 chore(deps): update flake.lock (#9236) 2026-11-08 01:34:01 -05:00
cxymds 16f618d41b fix: correct scanner pacing and unverified MRF outcomes (#9225)
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-08 06:48:25 +08:00
Henry Guo 565d262ede refactor(table-catalog): retire durable-strong backing (#9130)
* refactor(table-catalog): retire durable-strong backing

* fix(table-catalog): avoid backup locks during S3 authorization

* fix(table-catalog): preserve directory and maintenance behavior

* fix(table-catalog): back up retained v1 snapshot history

* test(table-catalog): cover retained v1 backup recovery

Exercise backup inventory sealing and restore after a v1-to-v2 upgrade, including missing or corrupt references, nonzero inline snapshot sequences, v2 manifests in retained v1 history, and unchanged commit admission rules.

Clarify when direct maintenance can replace reports and discard retry budgets, and require quiescing scheduler admission and workers before invocation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris An <anzhengchao@gmail.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-08 05:03:55 +08:00
Chris An 7b010ddd70 fix(ci,ecstore): restore validation and defer capacity exhaustion (#9223) 2026-11-07 15:17:25 -05:00
Chris An 272b6587e0 fix(ci): bound workspace archive builds to two workers (#9222) 2026-11-07 06:32:41 -05:00
Chris An e3ed2cf2f4 fix(decommission): reserve owned source usage during activation (#9220) 2026-11-07 01:49:42 -05:00
Chris An bf03b1117a fix(swift): preserve bucket metadata with Swift tag values (#9218) 2026-11-07 01:49:12 -05:00
Boyi (Elliot) Li c3fe0f9649 test(rio,ecstore): add negative-branch durability unit tests (#9216)
Expand positive and negative branch unit test coverage across durability-critical modules:
- crates/rio/src/checksum.rs: Checksum::matches (unset hasher, multipart want_parts mismatch, digest mismatch), new_with_type, as_map, obj_type, from_header, add_part (CRC32/CRC32C/CRC64_NVME combine + error branches), and get_content_checksum header validation.
- crates/rio/src/hash_reader.rs: HashReader::new wrapping invariants (already-read inner reader, conflicting MD5 checksum, conflicting positive size, unbounded size propagation) and checksum helper error paths.
- crates/ecstore/src/set_disk/ops/bitrot_self_verify.rs: verify_written_bitrot_shards missing inline data, missing disk shard, and under-quorum rejection.
- crates/ecstore/src/erasure/coding/encode.rs: read_full_buf_or_eof immediate read error, InvalidData preservation after partial fill, and UnexpectedEof normalization.
- crates/ecstore/src/io_support/shard_integrity.rs: ShardVerifier out-of-bounds coding_index, empty proof sources fail-closed, and spilled (>1 MiB) index disk write + verify_index_file + restore_proof_replicas round-trip and all-corrupt fail-closed boundary.
- crates/ecstore/src/services/integrity/tests.rs: JobRequest::validate boundary/format checks with exact error messages and model::protection / model::unsupported classifications.
2026-11-07 02:38:43 +00:00
唐小鸭 6e31f6a183 fix(ecstore): complete and abort native multipart uploads on Windows (#9191) 2026-11-06 22:36:16 -05:00
Hauser Zhi d66d144309 feat(observability): record unary read RPC durations (#9202)
* feat(observability): record unary read RPC durations

Record complete server-side durations for ReadMultiple and BatchReadVersion, including error and cancellation paths.\n\nCo-Authored-By: heihutu <heihutu@gmail.com>\nCo-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(observability): cover unary duration handlers

Exercise ReadMultiple and BatchReadVersion missing-disk paths and assert operation duration histograms.\n\nCo-Authored-By: heihutu <heihutu@gmail.com>\nCo-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(observability): cache unary read RPC duration handles

Reuse cached duration histograms for ReadMultiple and BatchReadVersion once
the stable server label is available. Preserve the existing ReadVersion
handle and the fallback for other operations and backends.

Add an integration regression for handle reuse, startup fallback, labels,
and recorded values using the production cache path.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(observability): separate server duration from client RTT

Record unary ReadMultiple and BatchReadVersion handler lifetimes in a
separate server duration histogram while preserving existing client RTT
series and allocation-free cached handles for both measurements.

Cover client/server sample separation, cache reuse, startup and backend
fallbacks, and missing-disk handlers. Document the new server metric.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-07 09:16:18 +08:00
cxymds 150836d707 feat(tier): support native Azure Blob endpoints (#9200)
* feat(tier): support native Azure Blob endpoints

* fix(tier): guard Azure version cleanup and prefix limits

* fix(tier): preserve Azure quoted headers and document version gate

* fix(tier): handle Azure forbidden current-version deletion

* test(tier): control transition commit lock failure timing
2026-11-07 09:16:06 +08:00
Henry Guo e4057b6746 fix(table-catalog): reject storage credentials for catalog access (#9185)
* fix(table-catalog): reject storage credentials for catalog access

* fix(table-catalog): avoid growing the s3s error macro footprint

Use the existing S3Error constructor for the storage-credential rejection
while preserving its AccessDenied code and message. Keep the merged PR
within the current s3s footprint baseline without relaxing the guard.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-11-07 08:07:58 +08:00
Hauser Zhi cfdd705b62 fix(cors): preserve readiness and complete cache selectors (#9207)
Use a direct response future for originless requests and forward through
the polled-ready service. Nominate every retained CORS header input and
transfer evaluated headers without rebuilding the map.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-07 08:07:04 +08:00
Hauser Zhi a1d0fa41d0 fix: preserve usage generations and unify request tracing (#9204)
* fix(ecstore): prevent cached usage snapshot rollback

Order refreshed snapshots by their authoritative lineage before replacing cached usage. Share the rule with early bucket refreshes while retaining quota baselines, invalidation fencing, legitimate deletions, and clock recovery.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(server): skip trace propagation for disabled RPC spans

Avoid extracting and assigning a remote parent when RPC tracing is disabled after authentication and admission. Cover disabled logging and the production HTTP span propagation path.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve usage ordering across refresh recovery

Retain the last good snapshot independently of negative-cache status. Order observations within the same authoritative baseline by their own coordinated epoch and cycle before wall time, including clock rollback and recovery. Capture persisted baseline identity without cloning its bucket maps.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(server): centralize request trace propagation

Let the HTTP span factory own parent extraction and assignment. Use actual span filtering, preserve span-only subscribers, and remove repeated tracing work from RPC authentication.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-07 08:06:47 +08:00
Chris An 7d4982f206 fix(storage): fence bounded deletes during rolling upgrades (#9214) 2026-11-07 08:06:28 +08:00
Chris An 1a7fb2461d fix(webdav): join connection and reload tasks on shutdown (#9213) 2026-11-07 08:06:14 +08:00
Chris An e1b9d1cb94 fix(ecstore): preserve ownership across multi-pool recovery (#9212) 2026-11-07 08:06:04 +08:00
Chris An 85313fc06f fix(runtime): release retained replay and allocator memory (#9211) 2026-11-07 08:05:46 +08:00
cxymds b5f3207378 fix(lock): preserve acquisition progress during multipart uploads (#9209)
* fix(lock): retain slow responses within the quorum deadline

* fix(lock): retain partial quorum timeout diagnostics

* fix(lock): avoid guard registry scheduling convoys

* chore(ecstore): trace metadata transaction lock purposes

* fix(scanner): keep capability probes on read fences

* fix(lock): preserve renewal and contention progress
2026-11-07 08:05:26 +08:00
Hauser Zhi 5490e493ab fix(s3): retain protected multipart ETags in object listings (#9208)
* fix(s3): retain protected multipart ETags in object listings

Record encrypted part ETag provenance, aggregate it at read quorum, and attest completed multipart ETags for keyless listings. Preserve legacy fingerprint protection and test versioned SSE-KMS and SSE-C upload, listing, and replay paths.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(s3): stream multipart ETag aggregation

Restore the preserved ETag fast path when selected parts lack protected provenance. Stream native part digests through a fixed stack buffer, preserve legacy hex-or-raw semantics, and strengthen exact-output and override regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): preserve encrypted ETag proof during data movement

Capture quorum-backed source proof before multipart completion metadata updates, require exact preserved ETag and version identity, and retain proof only when publication keeps the source version. Cover real two-pool rebalance, null versions, legacy sources, and injected or rewritten identities.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): retain encrypted multipart ETag proof on restore

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-11-07 08:04:54 +08:00
Hauser Zhi 134820c1e0 test(storage): reproduce HTTPS rejoin readiness in Docker (#9206)
* test(storage): reproduce HTTPS rejoin readiness in Docker

Compare cluster-only and node-ready/write gates on immutable native binaries. Preserve first-failure and latest-acknowledged-content evidence across recovery, capture lock diagnostics before restart, and limit cleanup to verified owned resources.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): unify Docker repro ownership and controls

Use typed resource absence and one verified container inventory for staging, partial setup and retained nodes. Reject inherited experiment overrides before creating resources and remove duplicated state.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): guard staging volumes and interrupted capture

Verify volume ownership before privileged staging and disable implicit image-content copying. Preserve failed evidence and continue owned finalization when capture or cleanup is interrupted.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): capture live rejoin logs and verify trace coverage

Read live tmpfs logs inside the verified container and use one explicit stdout sink for diagnostics. Record bounded capture provenance and fail diagnostic coverage when acquisition, guard-drop or physical-release records are missing, without masking HTTP failures.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): reject inherited aliases and finalize on SIGTERM

Reject inherited MINIO compatibility settings before Docker resource creation. Route SIGTERM through interrupt-aware finalization and restore the previous handler after each run. Cover real process signals and failure preservation.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): normalize legacy controls and termination

Reject inherited MINIO compatibility aliases in neutral runtime images. Bridge run-mode SIGTERM into the existing interruption finalization and restore the caller signal handler on return or exception.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-07 08:04:28 +08:00
Chris An 2b9217b6f9 test(e2e): fix quota and conditional write fixtures (#9210) 2026-11-07 08:03:00 +08:00
Hauser Zhi 70a46d0e8c fix(webdav): support configured browser CORS access (#9205)
* fix(webdav): support configured browser CORS access

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser Zhi <housemecn@gmail.com>

* perf(webdav): streamline CORS policy and preflight parsing

Borrow connection-owned policy state and clone the backend only for actual requests. Validate borrowed header names and normalize HTTP lists when required.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser Zhi <housemecn@gmail.com>

---------

Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-06 14:01:06 -05:00
Hauser Zhi a910ef0dd0 fix(cors): isolate cache variants and streamline policy evaluation (#9203)
Apply cache variation at the shared response boundary, preserve existing
Vary selectors, and reuse parsed global policy and compact CORS inputs.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-06 12:40:11 -05:00
Chris An 430ade191f fix(scanner): bound wide bucket-root fanout memory (#9193)
* fix(scanner): bound wide bucket-root fanout memory (#8961)

Root prefixes were excluded from ordinary compaction and force_compact only
flattened the root above a hardcoded 250k child ceiling, so buckets with
tens of thousands of top-level prefixes kept one cache entry each after
#8971. Compact wide roots outside checkpointed forward sweeps, and honor
force_compact's limit for root fanout.

* fix(scanner): keep root fanout on interrupted force_compact

Interrupted sweeps seal a resume frontier after force_compact. Applying
the ordinary 10k root limit there flattened frontier-named children and
made the next cycle replay completed prefixes. Use the emergency FORCE
ceiling on the interrupted path; successful completions still honor the
ordinary limit. Drop clippy assertions_on_constants in the new tests.

Signed-off-by: Chris An <anzhengchao@gmail.com>

* test(scanner): seal reloaded frontier after interrupted compaction

Const assertions keep the wide-root preconditions without tripping
clippy::assertions_on_constants. The interrupted and finished sweep
paths share one limit helper, and a cache wider than the ordinary
10k limit still seals its frontier child after reload.

Co-authored-by: Chris An <overtrue@users.noreply.github.com>

* fix(scanner): isolate unreadable children during root compaction

* fix(scanner): isolate compacted namespace child errors

* fix(scanner): preserve compacted scan accounting and coverage

Commit isolated namespace child usage only after successful traversal, while retaining completed accounting on cancellation and resource deferral. Synchronize reduced root totals before the cancellation-prone flush boundary.

Treat skipped namespaces as incomplete coverage and invalidate legacy sampling baselines until a fresh verification pass succeeds. Keep provisional usage out of progress updates and retain the upstream symlink regression.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): preserve legacy retry coverage

* test(scanner): regress cancel overwrite after wide-root reduce

houseme's 714f590 already refreshes the root accumulator from the
compacted cache entry before flush_pacing. Add a unit regression that
simulates the interrupt replace_hashed path so a future regression
cannot reintroduce dangling child links over the aggregate.

Signed-off-by: overtrue <anzhengchao@gmail.com>

---------

Signed-off-by: Chris An <anzhengchao@gmail.com>
Signed-off-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Chris An <overtrue@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-06 12:36:25 -05:00
235 changed files with 35099 additions and 25756 deletions
+1
View File
@@ -71,6 +71,7 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_nightly_candidate.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_repro_rejoin_readiness_docker.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_validation_registry.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_validation_runner.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_retain_validation_evidence.py
+2 -2
View File
@@ -110,7 +110,7 @@ threads-required = "num-test-threads"
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_object_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.default.overrides]]
@@ -410,7 +410,7 @@ threads-required = "num-test-threads"
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_object_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.ci.overrides]]
+40
View File
@@ -0,0 +1,40 @@
# syntax=docker/dockerfile:1.6
# Copyright 2026 RustFS Team
# Licensed under the Apache License, Version 2.0.
# A compatible native runtime with Python 3.11+, getent and stat is required by
# the isolated fixture's staging/provenance checks. This is a test image.
ARG RUNTIME_BASE=python:3.13-slim-trixie
FROM ${RUNTIME_BASE}
ARG SOURCE_REVISION
ARG BINARY_SHA256
ARG BUILD_PROFILE=dev
COPY rustfs /usr/bin/rustfs
RUN python3 - "$SOURCE_REVISION" "$BINARY_SHA256" "$BUILD_PROFILE" <<'PY'
import hashlib
import os
import re
import shutil
import sys
source, digest, profile = sys.argv[1:]
if not re.fullmatch(r"[0-9a-f]{40}", source):
raise SystemExit("invalid source revision")
if not re.fullmatch(r"[0-9a-f]{64}", digest):
raise SystemExit("invalid binary digest")
if profile not in ("dev", "release"):
raise SystemExit("invalid build profile")
with open("/usr/bin/rustfs", "rb") as binary:
if hashlib.file_digest(binary, "sha256").hexdigest() != digest:
raise SystemExit("binary digest mismatch")
for tool in ("getent", "stat"):
if shutil.which(tool) is None:
raise SystemExit(f"missing runtime tool: {tool}")
os.chmod("/usr/bin/rustfs", 0o755)
PY
LABEL io.rustfs.repro.source_revision="${SOURCE_REVISION}" \
io.rustfs.repro.binary_sha256="${BINARY_SHA256}" \
io.rustfs.repro.build_profile="${BUILD_PROFILE}"
USER 10001:10001
ENTRYPOINT ["/usr/bin/rustfs"]
+2 -1
View File
@@ -215,7 +215,8 @@ jobs:
- name: Archive workspace test binaries
env:
CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
# Workspace test compilation has high per-process memory use.
CARGO_BUILD_JOBS: "2"
run: |
./scripts/ci/resource_sampler.sh start workspace-archive
trap './scripts/ci/resource_sampler.sh stop' EXIT
+26
View File
@@ -107,6 +107,32 @@ jobs:
shell: pwsh
run: cargo test -p rustfs-ecstore --lib multipart_owned_record_parent_barriers_ -- --nocapture
# Native multipart completion publishes its payload from a different
# parent than its metadata, and abort renames under a precondition; both
# need their own Windows handle discipline.
- name: Test native multipart complete and abort publication
shell: pwsh
run: >-
cargo test -p rustfs-ecstore --lib --
native_abort_missing_state_with_actual_owned_records_never_becomes_legacy_or_missing
native_abort_pending_intent_is_not_a_delete_certificate_and_raw_cleanup_preserves_exact_evidence
--nocapture
# Owned-undo reclaim removes a directory it pinned, copy heal persists
# fault bytes through a handle, and metadata reads meet a directory.
- name: Test native multipart reclaim, copy heal, and directory reads
shell: pwsh
run: >-
cargo test -p rustfs-ecstore --lib --
native_multipart_owned_undo
native_multipart_copy_heal
orphan_multipart_cleanup_rejects_metadata_directory_on_one_disk
--nocapture
- name: Test directory removal fallback
shell: pwsh
run: "cargo test -p rustfs-ecstore --lib disk::fs::tests::test_remove -- --nocapture"
- name: Test Windows path conversion
shell: pwsh
run: "cargo test -p rustfs-utils --no-default-features --features path --lib dunce:: -- --nocapture"
+4
View File
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Notifications and audit
- Notify and audit targets that keep the default `queue_dir` now work on native Windows: the defaults `/opt/rustfs/events` and `/opt/rustfs/audit` resolve per node, to `%ProgramData%\RustFS\events` and `%ProgramData%\RustFS\audit` on Windows, instead of failing validation as non-absolute paths. The new `RUSTFS_TARGET_STORE_DIR` (Helm `config.rustfs.target_store_dir`) relocates both default queues on any platform. Linux and macOS behavior is unchanged when it is unset, persisted configuration is unchanged, and pending entries are not migrated when the root changes. See `docs/operations/target-queue-storage.md`.
### Deployment
- Existing volumes created by older root-running container images must be writable by the non-root runtime identity before upgrading. The default container UID/GID is `10001:10001`; stop the affected instances, verify host mount paths, and for ordinary bind mounts run `sudo chown -R 10001:10001 /actual/data /actual/logs` on each RustFS data/log mount. Account for ACLs, NFS, Kubernetes, and rootless UID mappings. Image-build ownership changes do not migrate mounted data. See the Docker Quick Start in [README.md](README.md).
Generated
+7
View File
@@ -9612,6 +9612,7 @@ dependencies = [
"rustfs-log-analyzer",
"rustfs-madmin",
"rustfs-mimalloc",
"rustfs-mimalloc-sys",
"rustfs-notify",
"rustfs-object-capacity",
"rustfs-object-data-cache",
@@ -9862,6 +9863,7 @@ dependencies = [
"path-absolutize",
"pin-project-lite",
"proptest",
"quick-xml",
"rand 0.10.3",
"ratelimit",
"rcgen",
@@ -9894,7 +9896,9 @@ dependencies = [
"rustfs-s3-ops",
"rustfs-s3-types",
"rustfs-scanner-metrics",
"rustfs-signer",
"rustfs-storage-api",
"rustfs-tls-runtime",
"rustfs-uring",
"rustfs-utils",
"rustix",
@@ -10755,6 +10759,7 @@ dependencies = [
"arc-swap",
"async-compression",
"async-trait",
"brotli 9.1.1",
"bytes",
"chrono",
"crc-fast",
@@ -10764,6 +10769,7 @@ dependencies = [
"futures-core",
"hotpath",
"http 1.5.0",
"lz4",
"memchr",
"metrics",
"parking_lot 0.13.0",
@@ -10896,6 +10902,7 @@ dependencies = [
"time",
"tracing",
"tracing-subscriber",
"url",
]
[[package]]
+1 -1
View File
@@ -199,7 +199,7 @@ docker run -d --name rustfs -p 9000:9000 \
Notes:
- `RUSTFS_NOTIFY_ENABLE=true` enables the global notify module switch.
- For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
- If queue dir is omitted, the official image maps `/opt/rustfs/events` into the persistent `/data` volume so it is writable by the runtime user and pending events survive container recreation. Override `queue_dir` when using another deployment layout.
- If queue dir is omitted, the official image maps `/opt/rustfs/events` into the persistent `/data` volume so it is writable by the runtime user and pending events survive container recreation. Override `queue_dir` when using another deployment layout, or set `RUSTFS_TARGET_STORE_DIR` to move every default queue; native Windows defaults to `%ProgramData%\RustFS`. See [Target queue storage](docs/operations/target-queue-storage.md).
- `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
- Since `1.0.0-beta.11`, webhook endpoints on private or container networks
(`Docker Compose service names`, `host.docker.internal`, RFC 1918 addresses) are
+7
View File
@@ -276,6 +276,13 @@ Drive timeout profile preset:
- values must be positive; `0` or an invalid value falls back to the default, and values above `60` are clamped to `60`.
- the setting is read by the aggregating node only; it does not change the internode RPC wire contract. Any retry shares one round deadline rather than receiving a fresh timeout.
## Target queue directory
- `RUSTFS_TARGET_STORE_DIR`
- absolute root for notify and audit target queues whose `queue_dir` is the default `/opt/rustfs/events` or `/opt/rustfs/audit`; they open `<root>/events` and `<root>/audit` on this node.
- unset keeps those paths on Linux and macOS; native Windows uses `%ProgramData%\RustFS\events` and `%ProgramData%\RustFS\audit`.
- explicit `queue_dir` values are never relocated, and the persisted target configuration is unchanged. Changing the root does not migrate pending entries. See `docs/operations/target-queue-storage.md`.
## Startup filesystem boundary policy
- `RUSTFS_UNSUPPORTED_FS_POLICY` controls startup behavior when RustFS detects local endpoint filesystems that are outside the supported production boundary.
+3 -2
View File
@@ -32,8 +32,9 @@ pub const ENV_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD: &str = "RUSTFS_OBJECT_MEDIUM_
/// Environment variable name for maximum concurrent disk reads for object operations.
/// - Purpose: Limit the number of concurrent disk read operations for object reads to prevent I/O saturation.
/// - Unit: request count (usize).
/// - Semantics: Throttling disk reads helps maintain overall system responsiveness under load.
/// - Unit: request count (usize) for GET; estimated drive fanout for native LIST.
/// - Semantics: GET bodies and native LIST walks use separate per-instance budgets of this size.
/// A LIST consumes up to one budget, including its disk-producer cleanup. `0` disables both caps.
/// - Example: `export RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=16`
/// - Note: This setting may interact with OS-level I/O scheduling and should be tuned based on hardware capabilities.
pub const ENV_OBJECT_MAX_CONCURRENT_DISK_READS: &str = "RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS";
+6
View File
@@ -144,6 +144,12 @@ pub const POSTGRES_QUEUE_LIMIT: &str = "queue_limit";
/// Applies to both notify and audit target queue stores.
pub const ENV_TARGET_STORE_COMPRESS: &str = "RUSTFS_TARGET_STORE_COMPRESS";
/// Environment variable that relocates the root of the default notify and audit queue directories.
///
/// It applies only to targets whose `queue_dir` is the logical default (`EVENT_DEFAULT_DIR` or
/// `AUDIT_DEFAULT_DIR`); their queues move to `<root>/events` and `<root>/audit` on this node.
pub const ENV_TARGET_STORE_DIR: &str = "RUSTFS_TARGET_STORE_DIR";
/// Queue-store compression is enabled by default to reduce disk footprint.
pub const DEFAULT_TARGET_STORE_COMPRESS: bool = true;
+55 -2
View File
@@ -675,6 +675,10 @@ impl DataUsageInfo {
/// cycle. The timestamp fallback preserves ordering for legacy snapshots that
/// predate those fields.
pub fn data_usage_snapshot_is_newer(candidate: &DataUsageInfo, baseline: &DataUsageInfo) -> bool {
data_usage_identity_is_newer(candidate.snapshot_identity(), baseline.snapshot_identity())
}
fn data_usage_identity_is_newer(candidate: DataUsageSnapshotIdentity, baseline: DataUsageSnapshotIdentity) -> bool {
match (
candidate.scanner_epoch.zip(candidate.scanner_cycle),
baseline.scanner_epoch.zip(baseline.scanner_cycle),
@@ -697,10 +701,18 @@ pub fn data_usage_snapshot_is_newer(candidate: &DataUsageInfo, baseline: &DataUs
/// already advance the authoritative timestamp when deleting a bucket, so a
/// rollback delete/recreate fences the previous bucket incarnation too.
pub fn observed_data_usage_is_newer(observed: &DataUsageInfo, authoritative: &DataUsageInfo) -> bool {
observed_data_usage_is_newer_than_identity(observed, authoritative.snapshot_identity())
}
/// Return whether a nonconverged observation may supersede `authoritative`.
///
/// This applies the structural, exact-baseline, and generation checks of
/// [`observed_data_usage_is_newer`] without retaining the baseline's bucket maps.
pub fn observed_data_usage_is_newer_than_identity(observed: &DataUsageInfo, authoritative: DataUsageSnapshotIdentity) -> bool {
observed.usage_snapshot_converged == Some(false)
&& (observed.is_complete_bucket_usage_snapshot() || observed.is_valid_partial_snapshot())
&& observed.usage_snapshot_authoritative_baseline.as_ref() == Some(&authoritative.snapshot_identity())
&& data_usage_snapshot_is_newer(observed, authoritative)
&& observed.usage_snapshot_authoritative_baseline == Some(authoritative)
&& data_usage_identity_is_newer(observed.snapshot_identity(), authoritative)
}
/// Metadata describing the status of a disk-level data usage snapshot.
@@ -3140,12 +3152,17 @@ mod tests {
};
assert!(observed_data_usage_is_newer(&observed, &authoritative));
assert!(observed_data_usage_is_newer_than_identity(&observed, authoritative.snapshot_identity()));
authoritative.last_update = Some(SystemTime::UNIX_EPOCH + Duration::from_secs(601));
assert!(
!observed_data_usage_is_newer(&observed, &authoritative),
"an old-binary namespace mutation must fence the prior bucket incarnation regardless of clock skew"
);
assert!(
!observed_data_usage_is_newer_than_identity(&observed, authoritative.snapshot_identity()),
"a captured identity must preserve the exact namespace fence"
);
}
#[test]
@@ -3172,6 +3189,18 @@ mod tests {
assert!(!observed_data_usage_is_newer(&candidate(2, 9, Some(false), true), &authoritative));
assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(true), true), &authoritative));
assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(false), false), &authoritative));
for (observed, expected) in [
(candidate(2, 11, Some(false), true), true),
(candidate(2, 9, Some(false), true), false),
(candidate(2, 11, Some(true), true), false),
(candidate(2, 11, Some(false), false), false),
] {
assert_eq!(
observed_data_usage_is_newer_than_identity(&observed, authoritative.snapshot_identity()),
expected,
"observation identity selection must preserve convergence, completeness, and generation gates"
);
}
let mut partial = candidate(2, 11, Some(false), false);
partial.usage_snapshot_partial = true;
@@ -3186,6 +3215,30 @@ mod tests {
segment_invalidation_proof: None,
}];
assert!(observed_data_usage_is_newer(&partial, &authoritative));
assert!(observed_data_usage_is_newer_than_identity(&partial, authoritative.snapshot_identity()));
}
#[test]
fn observation_identity_selection_keeps_legacy_timestamp_ordering() {
let baseline = DataUsageSnapshotIdentity {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(10)),
// An epoch without a cycle is not a coordinated position.
scanner_epoch: Some(2),
scanner_cycle: None,
};
let mut observed = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(11)),
scanner_epoch: Some(1),
usage_snapshot_complete: true,
usage_snapshot_converged: Some(false),
usage_snapshot_authoritative_baseline: Some(baseline),
..Default::default()
};
assert!(observed_data_usage_is_newer_than_identity(&observed, baseline));
observed.last_update = baseline.last_update;
assert!(!observed_data_usage_is_newer_than_identity(&observed, baseline));
observed.last_update = None;
assert!(!observed_data_usage_is_newer_than_identity(&observed, baseline));
}
#[test]
@@ -260,7 +260,7 @@ async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Clien
let key = "guarded.txt";
admin_create_user(env, &user, user_secret).await?;
let policy = serde_json::json!({
let mut policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
@@ -268,9 +268,8 @@ async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Clien
"Resource": [format!("arn:aws:s3:::{bucket}/*")],
"Condition": { "Null": { "s3:if-match": "false" } }
}]
})
.to_string();
admin_add_and_attach_policy(env, &policy_name, &policy, &user).await?;
});
admin_add_and_attach_policy(env, &policy_name, &policy.to_string(), &user).await?;
admin.create_bucket().bucket(&bucket).send().await?;
let original = admin
@@ -293,6 +292,46 @@ async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Clien
.expect_err("PutObject without If-Match must be denied");
assert_error(&denied, "AccessDenied", 403, "unconditional PutObject");
let denied = client
.put_object()
.bucket(&bucket)
.key(key)
.if_match(&etag)
.body(ByteStream::from_static(b"missing-read-permission"))
.send()
.await
.expect_err("If-Match without GetObject must be denied");
assert_error(&denied, "AccessDenied", 403, "conditional PutObject without GetObject");
// S3 requires GetObject as well as PutObject for an ETag-based write.
policy["Statement"]
.as_array_mut()
.expect("policy statements")
.push(serde_json::json!({
"Effect": "Allow",
"Action": ["s3:GetObject"],
"Resource": [format!("arn:aws:s3:::{bucket}/{key}")]
}));
admin_add_canned_policy_via(
AdminTransport::Signed,
&env.url,
&env.access_key,
&env.secret_key,
&policy_name,
&policy.to_string(),
)
.await?;
let denied = client
.put_object()
.bucket(&bucket)
.key(key)
.body(ByteStream::from_static(b"blind-overwrite"))
.send()
.await
.expect_err("GetObject permission must not allow an unconditional PutObject");
assert_error(&denied, "AccessDenied", 403, "unconditional PutObject with GetObject");
client
.put_object()
.bucket(&bucket)
@@ -301,6 +340,8 @@ async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Clien
.body(ByteStream::from_static(b"v2"))
.send()
.await?;
let written = client.get_object().bucket(&bucket).key(key).send().await?;
assert_eq!(written.body.collect().await?.into_bytes().as_ref(), b"v2");
let _ = admin.delete_object().bucket(&bucket).key(key).send().await;
let _ = admin.delete_bucket().bucket(&bucket).send().await;
+34 -4
View File
@@ -412,6 +412,38 @@ mod integration_tests {
);
}
async fn upload_after_quota_reconciliation(
env: &QuotaTestEnv,
key: &str,
size_bytes: usize,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let upload = async {
loop {
let result = env
.client
.put_object()
.bucket(&env.bucket_name)
.key(key)
.body(aws_sdk_s3::primitives::ByteStream::from(vec![0u8; size_bytes]))
.send()
.await;
match result {
Ok(_) => break,
Err(error) => assert_quota_rejection(
error.raw_response().map(|response| response.status().as_u16()),
error.as_service_error(),
&error,
),
}
sleep(Duration::from_secs(1)).await;
}
};
// The durable V1 ledger can retain the old charge for its 30-second reconciliation cooldown.
timeout(Duration::from_secs(45), upload)
.await
.map_err(|_| format!("PUT {key} did not succeed after quota reconciliation within 45 seconds").into())
}
async fn assert_put_rejected_by_quota(env: &QuotaTestEnv, key: &str, size_bytes: usize) {
let error = env
.client
@@ -573,8 +605,7 @@ mod integration_tests {
assert!(!env.object_exists("file1.txt").await?);
// Now we should be able to upload again (quota freed up)
env.upload_object("file3.txt", 256 * 1024).await?;
upload_after_quota_reconciliation(&env, "file3.txt", 256 * 1024).await?;
assert!(env.object_exists("file3.txt").await?);
env.cleanup_bucket().await?;
@@ -981,8 +1012,7 @@ mod integration_tests {
);
assert_eq!(delete_result.deleted().len(), 2);
// Now should be able to upload again (quota freed up)
env.upload_object("file3.txt", 256 * 1024).await?;
upload_after_quota_reconciliation(&env, "file3.txt", 256 * 1024).await?;
assert!(env.object_exists("file3.txt").await?);
env.cleanup_bucket().await?;
@@ -114,10 +114,10 @@ fn staged_commit(metadata: &Value, location: &str) -> Value {
})
}
async fn location_claim_preserves_other_users_objects_after_drop_and_restart(backing: &str) -> TestResult {
async fn location_claim_preserves_other_users_objects_after_drop_and_restart() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
let server_env = [("RUSTFS_TABLE_CATALOG_BACKING", backing)];
let server_env = [("RUSTFS_TABLE_CATALOG_BACKING", "object")];
env.start_rustfs_server_without_cleanup_with_env(&[("RUSTFS_TABLE_CATALOG_BACKING", "object")])
.await?;
env.wait_for_server_ready().await?;
@@ -147,39 +147,6 @@ async fn location_claim_preserves_other_users_objects_after_drop_and_restart(bac
StatusCode::OK,
)
.await?;
if backing == "durable-strong" {
catalog_request(
&env,
&root,
Method::POST,
&format!("/iceberg/v1/{bucket}/catalog/warehouse-index/backfill"),
None,
StatusCode::NO_CONTENT,
)
.await?;
let preflight = catalog_request(
&env,
&root,
Method::GET,
&format!("/iceberg/v1/{bucket}/catalog/migration"),
None,
StatusCode::OK,
)
.await?;
assert_eq!(preflight["blockers"], json!([]), "{preflight}");
let migration = catalog_request(
&env,
&root,
Method::POST,
&format!("/iceberg/v1/{bucket}/catalog/migration"),
None,
StatusCode::OK,
)
.await?;
assert_eq!(migration["ready_to_enable_durable_strong"], json!(true), "{migration}");
env.restart_server_preserving_data(vec![], &server_env).await?;
env.wait_for_server_ready().await?;
}
admin_create_user(&env, &creator.0, &creator.1).await?;
admin_create_user(&env, &reader.0, &reader.1).await?;
let create_statement = json!({
@@ -341,7 +308,7 @@ async fn location_claim_preserves_other_users_objects_after_drop_and_restart(bac
StatusCode::FORBIDDEN,
)
.await?;
if backing == "object" {
{
catalog_request(
&env,
&creator,
@@ -408,10 +375,5 @@ async fn location_claim_preserves_other_users_objects_after_drop_and_restart(bac
#[tokio::test]
async fn object_catalog_location_claim_preserves_objects_after_drop_and_restart() -> TestResult {
location_claim_preserves_other_users_objects_after_drop_and_restart("object").await
}
#[tokio::test]
async fn durable_strong_catalog_location_claim_preserves_objects_after_drop_and_restart() -> TestResult {
location_claim_preserves_other_users_objects_after_drop_and_restart("durable-strong").await
location_claim_preserves_other_users_objects_after_drop_and_restart().await
}
+4 -1
View File
@@ -31,7 +31,7 @@ workspace = true
[features]
default = []
fips = ["rustfs-s3-client/fips", "rustfs-utils/fips"]
fips = ["rustfs-s3-client/fips", "rustfs-utils/fips", "rustfs-tls-runtime/fips"]
diagnostic-read-version-caller = ["rustfs-io-metrics/diagnostic-read-version-caller"]
# Sampled wall/CPU diagnostics only; does not change the read backend.
diagnostic-local-batch = ["rustix/time"]
@@ -203,6 +203,9 @@ libc.workspace = true
# (backlog#1178); "thread" provides cleanup affinity; "fs" is a workspace default.
rustix = { workspace = true, features = ["process", "fs", "thread"] }
rustfs-madmin.workspace = true
rustfs-signer.workspace = true
rustfs-tls-runtime.workspace = true
quick-xml.workspace = true
reqwest = { workspace = true }
aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
urlencoding = { workspace = true }
+14 -9
View File
@@ -191,8 +191,11 @@ pub mod bucket {
OBJECT_LOCK_CONFIG, UnreadableBucketConfig, is_unreadable_config_error, load_bucket_metadata,
table_catalog_path_hash, unreadable_config_refusal,
};
pub use crate::bucket::metadata::{BUCKET_DURABILITY_CONFIG, BUCKET_ON_DEMAND_MIGRATION_CONFIG};
pub use crate::bucket::metadata::{MAX_BUCKET_TAGS, validate_tag_set_entries};
pub use crate::bucket::metadata::{
BUCKET_DURABILITY_CONFIG, BUCKET_ON_DEMAND_MIGRATION_CONFIG, MAX_SWIFT_METADATA_COUNT, MAX_SWIFT_METADATA_VALUE_SIZE,
SWIFT_ACCOUNT_META_TAG_PREFIX, SWIFT_CONTAINER_META_TAG_PREFIX, SWIFT_TAGGING_MARKER,
};
pub use crate::bucket::metadata::{MAX_BUCKET_TAGS, is_swift_bucket_metadata_tag_key, validate_tag_set_entries};
}
pub mod config_parse_mode {
@@ -210,19 +213,20 @@ pub mod bucket {
pub mod metadata_sys {
pub use crate::bucket::metadata_sys::{
BUCKET_CONFIG_PUBLISH_HOOK, BucketConfigPublishHook, BucketMetadataMutationGuard, BucketMetadataSys,
ObjectLockConfigState, acquire_bucket_metadata_transaction_lock,
ObjectLockConfigState, ScannerBucketIncarnationReadGuard, acquire_bucket_metadata_transaction_lock,
acquire_bucket_metadata_transaction_lock_for_incarnation, acquire_scanner_bucket_incarnation_fence,
capture_bucket_metadata_incarnation, delete, delete_if_incarnation, delete_if_incarnation_at,
delete_under_transaction_lock, get, get_accelerate_config, get_bucket_policy, get_bucket_policy_raw,
get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
acquire_scanner_bucket_incarnation_read_fence, capture_bucket_metadata_incarnation, delete, delete_if_incarnation,
delete_if_incarnation_at, delete_under_transaction_lock, get, get_accelerate_config, get_bucket_policy,
get_bucket_policy_raw, get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
get_global_bucket_metadata_sys, get_lifecycle_config, get_logging_config, get_notification_config,
get_object_lock_config, get_object_lock_config_state, get_on_demand_migration_config,
get_on_demand_migration_config_in, get_public_access_block_config, get_quota_config, get_replication_config,
get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
get_request_payment_config, get_s3_tagging_config, get_sse_config, get_tagging_config, get_versioning_config,
get_website_config, init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
remove_bucket_metadata_if_absent, set_bucket_metadata, update, update_bucket_targets_under_transaction_lock,
update_config_with, update_if_incarnation, update_if_incarnation_at, update_quota_if_incarnation,
update_quota_if_incarnation_at, update_under_transaction_lock, update_under_transaction_lock_at,
update_quota_if_incarnation_at, update_s3_tagging_config, update_under_transaction_lock,
update_under_transaction_lock_at,
};
#[cfg(feature = "test-util")]
pub use crate::bucket::metadata_sys::{ConfigWriteLockProbe, test_support};
@@ -450,6 +454,7 @@ pub mod data_usage {
pub mod disk {
pub use crate::cluster::rpc::background_pipe::{BackgroundPipeReader, BackgroundPipeWriter, background_pipe};
pub use crate::cluster::rpc::background_reader_stream::BackgroundReaderStream;
pub use crate::disk::BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION;
pub use crate::disk::background_memory::BackgroundEncodeBuffer;
pub use crate::disk::disk_store::get_object_disk_read_timeout;
#[cfg(feature = "test-util")]
+139 -22
View File
@@ -256,15 +256,15 @@ pub const BUCKET_POLICY_CONFIG: &str = "policy.json";
pub const BUCKET_NOTIFICATION_CONFIG: &str = "notification.xml";
pub const BUCKET_LIFECYCLE_CONFIG: &str = "lifecycle.xml";
pub const BUCKET_SSECONFIG: &str = "bucket-encryption.xml";
/// Shared tag-entry validation for object requests and persisted bucket tags.
/// Validate tag entries supplied to the S3 API.
pub fn validate_tag_set_entries(tag_set: &[s3s::dto::Tag]) -> std::result::Result<(), &'static str> {
fn is_valid_tag_text(value: &str) -> bool {
static VALID_TAG_TEXT: std::sync::OnceLock<std::result::Result<regex::Regex, regex::Error>> = std::sync::OnceLock::new();
VALID_TAG_TEXT
.get_or_init(|| regex::Regex::new(r"^[\p{L}\p{N}\p{Z}+\-=._:/@]*$"))
.as_ref()
.is_ok_and(|pattern| pattern.is_match(value))
}
validate_tag_entries(tag_set, validate_s3_tag_entry)
}
fn validate_tag_entries(
tag_set: &[s3s::dto::Tag],
mut validate_entry: impl FnMut(&str, &str) -> std::result::Result<(), &'static str>,
) -> std::result::Result<(), &'static str> {
let mut keys = std::collections::HashSet::with_capacity(tag_set.len());
for tag in tag_set {
let key = tag
@@ -272,13 +272,8 @@ pub fn validate_tag_set_entries(tag_set: &[s3s::dto::Tag]) -> std::result::Resul
.as_deref()
.filter(|key| !key.is_empty())
.ok_or("Tag key cannot be empty")?;
if key.encode_utf16().count() > 128 || !is_valid_tag_text(key) {
return Err("The TagKey you have provided is invalid");
}
let value = tag.value.as_deref().ok_or("Tag value cannot be null")?;
if value.encode_utf16().count() > 256 || !is_valid_tag_text(value) {
return Err("The TagValue you have provided is invalid");
}
validate_entry(key, value)?;
if !keys.insert(key) {
return Err("Cannot provide multiple Tags with the same key");
}
@@ -286,8 +281,103 @@ pub fn validate_tag_set_entries(tag_set: &[s3s::dto::Tag]) -> std::result::Resul
Ok(())
}
fn validate_s3_tag_entry(key: &str, value: &str) -> std::result::Result<(), &'static str> {
fn is_valid_tag_text(value: &str) -> bool {
static VALID_TAG_TEXT: std::sync::OnceLock<std::result::Result<regex::Regex, regex::Error>> = std::sync::OnceLock::new();
VALID_TAG_TEXT
.get_or_init(|| regex::Regex::new(r"^[\p{L}\p{N}\p{Z}+\-=._:/@]*$"))
.as_ref()
.is_ok_and(|pattern| pattern.is_match(value))
}
if key.encode_utf16().count() > 128 || !is_valid_tag_text(key) {
return Err("The TagKey you have provided is invalid");
}
if value.encode_utf16().count() > 256 || !is_valid_tag_text(value) {
return Err("The TagValue you have provided is invalid");
}
Ok(())
}
pub const MAX_BUCKET_TAGS: usize = 50;
/// Maximum number of metadata headers on a Swift resource.
pub const MAX_SWIFT_METADATA_COUNT: usize = 90;
/// Maximum byte length of a Swift metadata value.
pub const MAX_SWIFT_METADATA_VALUE_SIZE: usize = 256;
/// Prefix for Swift account metadata persisted in tagging.xml.
pub const SWIFT_ACCOUNT_META_TAG_PREFIX: &str = "swift-account-meta-";
/// Prefix for Swift container metadata persisted in tagging.xml.
pub const SWIFT_CONTAINER_META_TAG_PREFIX: &str = "swift-meta-";
/// Marks a bucket whose shared tag config contains Swift-owned metadata.
/// The key cannot be supplied through S3's tag alphabet.
pub const SWIFT_TAGGING_MARKER: &str = "!rustfs-swift-metadata";
/// Whether a persisted tag belongs to Swift's internal metadata namespace.
pub fn is_swift_bucket_metadata_tag_key(key: &str) -> bool {
matches!(
key,
"swift-acl-read" | "swift-acl-write" | "swift-versions-location" | SWIFT_TAGGING_MARKER
) || key.starts_with(SWIFT_ACCOUNT_META_TAG_PREFIX)
|| key.starts_with(SWIFT_CONTAINER_META_TAG_PREFIX)
}
fn validate_persisted_bucket_tags(tags: &[s3s::dto::Tag]) -> std::result::Result<(), &'static str> {
const SWIFT_ACL_MAX_BYTES: usize = 8192;
const SWIFT_META_NAME_MAX_BYTES: usize = 128;
const MAX_PERSISTED_TAGS: usize = MAX_BUCKET_TAGS + 2 * MAX_SWIFT_METADATA_COUNT + 4;
if tags.len() > MAX_PERSISTED_TAGS {
return Err("Too many persisted bucket tags");
}
// Swift stores ACLs, versioning and metadata in the same XML as S3 tags.
// Only those known keys may carry values outside the S3 tag alphabet.
let mut account_metadata = 0;
let mut container_metadata = 0;
let mut s3_tags = 0;
validate_tag_entries(tags, |key, value| {
let swift_metadata = key
.strip_prefix(SWIFT_ACCOUNT_META_TAG_PREFIX)
.map(|name| (name, &mut account_metadata))
.or_else(|| {
key.strip_prefix(SWIFT_CONTAINER_META_TAG_PREFIX)
.map(|name| (name, &mut container_metadata))
});
if let Some((name, count)) = swift_metadata
&& name.len() <= SWIFT_META_NAME_MAX_BYTES
&& http::HeaderName::from_bytes(name.as_bytes()).is_ok()
&& value.len() <= MAX_SWIFT_METADATA_VALUE_SIZE
&& http::HeaderValue::from_str(value).is_ok()
{
*count += 1;
if *count > MAX_SWIFT_METADATA_COUNT {
return Err("Too many Swift metadata tags");
}
return Ok(());
}
if key == SWIFT_TAGGING_MARKER && value == "1" {
return Ok(());
}
let valid_swift_header = match key {
"swift-acl-read" | "swift-acl-write" => value.len() <= SWIFT_ACL_MAX_BYTES,
"swift-versions-location" => value.len() <= 256 && !value.is_empty() && !value.contains('/'),
_ => false,
};
if valid_swift_header && http::HeaderValue::from_str(value).is_ok() {
return Ok(());
}
s3_tags += 1;
if s3_tags > MAX_BUCKET_TAGS {
return Err("Cannot have more than 50 tags per bucket");
}
validate_s3_tag_entry(key, value)
})
}
pub const BUCKET_TAGGING_CONFIG: &str = "tagging.xml";
pub const BUCKET_QUOTA_CONFIG_FILE: &str = "quota.json";
pub const OBJECT_LOCK_CONFIG: &str = "object-lock.xml";
@@ -1026,14 +1116,7 @@ impl BucketMetadata {
let tagging = deserialize::<Tagging>(&data).map_err(|_| {
Error::InvalidArgument(self.name.clone(), String::new(), "Invalid bucket tagging XML".to_owned())
})?;
if tagging.tag_set.len() > MAX_BUCKET_TAGS {
return Err(Error::InvalidArgument(
self.name.clone(),
String::new(),
"Cannot have more than 50 tags per bucket".to_owned(),
));
}
validate_tag_set_entries(&tagging.tag_set)
validate_persisted_bucket_tags(&tagging.tag_set)
.map_err(|message| Error::InvalidArgument(self.name.clone(), String::new(), message.to_owned()))?;
}
self.tagging_config_xml = data;
@@ -1827,6 +1910,40 @@ mod test {
assert_eq!(metadata.bucket_incarnation_id, incarnation);
}
#[test]
fn persisted_bucket_tags_keep_separate_s3_and_swift_limits() {
use s3s::dto::Tag;
let tag = |key: String, value: &str| Tag {
key: Some(key),
value: Some(value.to_owned()),
};
let acl = tag("swift-acl-read".to_owned(), ".r:*,AUTH_other");
assert_eq!(
validate_tag_set_entries(std::slice::from_ref(&acl)),
Err("The TagValue you have provided is invalid")
);
let mut tags = vec![acl.clone()];
tags.extend((0..MAX_BUCKET_TAGS).map(|i| tag(format!("team-{i}"), "blue")));
tags.extend((0..MAX_SWIFT_METADATA_COUNT).map(|i| tag(format!("{SWIFT_CONTAINER_META_TAG_PREFIX}item!{i}"), "a,b")));
assert_eq!(validate_persisted_bucket_tags(&tags), Ok(()));
tags.push(tag(format!("{SWIFT_CONTAINER_META_TAG_PREFIX}overflow"), "a,b"));
assert_eq!(validate_persisted_bucket_tags(&tags), Err("Too many Swift metadata tags"));
tags.pop();
tags.push(acl);
assert_eq!(
validate_persisted_bucket_tags(&tags),
Err("Cannot provide multiple Tags with the same key")
);
tags.pop();
tags[0].value = Some(".r:*,".repeat(1700));
assert_eq!(validate_persisted_bucket_tags(&tags), Err("The TagValue you have provided is invalid"));
}
/// backlog#2292: a replicated config is stamped with the source
/// `updated_at` it was given, not the local clock, while the plain
/// `update_config` entry keeps stamping the local clock.
+417 -44
View File
@@ -22,6 +22,7 @@ use super::target::BucketTargets;
use crate::bucket::bucket_target_sys::BucketTargetSys;
use crate::bucket::metadata::{load_bucket_metadata_parse, load_bucket_metadata_parse_with_presence};
use crate::bucket::utils::is_meta_bucketname;
use crate::diagnostics::object_lock::ObjectLockAttempt;
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::{Error, Result, is_err_bucket_not_found, is_err_strict_volume_not_found};
use crate::object_api::ObjectOptions;
@@ -599,7 +600,9 @@ pub(crate) async fn get_bucket_incarnation_id_for_options_in(
}
let sys = bucket_metadata_sys_of(ctx)?;
let sys = sys.read().await.clone();
let guard = acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts).await?;
let guard =
acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts, "bucket_incarnation_options_read")
.await?;
sys.get_bucket_incarnation_id_under_transaction_lock(bucket, &guard).await
}
@@ -866,6 +869,30 @@ impl BucketMetadataMutationGuard {
}
}
/// Read-only scanner authority for one bucket incarnation.
///
/// Lock order is bucket lifecycle (read), then metadata transaction (read).
/// This guard cannot authorize a configuration mutation or scanner ACK write.
pub struct ScannerBucketIncarnationReadGuard {
bucket: String,
incarnation_id: Uuid,
lifecycle_guard: rustfs_lock::NamespaceLockGuard,
transaction_guard: rustfs_lock::NamespaceLockGuard,
}
impl ScannerBucketIncarnationReadGuard {
/// Returns the verified identity while both read fences remain valid.
///
/// # Errors
/// Returns an error if either lock has lost its lease.
pub fn checked_bucket_incarnation(&self) -> Result<(&str, Uuid)> {
if self.lifecycle_guard.is_lock_lost() || self.transaction_guard.is_lock_lost() {
return Err(Error::other("scanner bucket incarnation read lock was lost"));
}
Ok((&self.bucket, self.incarnation_id))
}
}
async fn acquire_config_write_guard(sys: Arc<RwLock<BucketMetadataSys>>, bucket: &str) -> Result<BucketMetadataMutationGuard> {
acquire_config_write_guard_for_incarnation(sys, bucket, None).await
}
@@ -892,6 +919,68 @@ pub async fn acquire_scanner_bucket_incarnation_fence(
acquire_config_write_guard_with_migration(sys, bucket, Some(expected_incarnation_id), false).await
}
/// Validate scanner read authority without serializing compatible metadata readers.
///
/// The returned guard retains both read locks. It never migrates metadata or
/// creates an incarnation; mutation callers must use the exclusive scanner fence.
///
/// # Errors
/// Returns an error for invalid bucket/owner identities, unavailable locks or
/// storage, absent incarnation metadata, or a deleted/recreated bucket.
pub async fn acquire_scanner_bucket_incarnation_read_fence(
bucket: &str,
expected_incarnation_id: Uuid,
expected_owner_id: Uuid,
) -> Result<ScannerBucketIncarnationReadGuard> {
acquire_scanner_bucket_incarnation_read_fence_with_sys(
get_bucket_metadata_sys()?,
bucket,
expected_incarnation_id,
expected_owner_id,
)
.await
}
async fn acquire_scanner_bucket_incarnation_read_fence_with_sys(
sys: Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
expected_incarnation_id: Uuid,
expected_owner_id: Uuid,
) -> Result<ScannerBucketIncarnationReadGuard> {
super::utils::check_valid_bucket_name(bucket)?;
let metadata_sys = sys.read().await.clone();
if expected_owner_id.is_nil() || metadata_sys.object_store().id != expected_owner_id || expected_incarnation_id.is_nil() {
return Err(Error::other("scanner bucket incarnation owner does not match"));
}
let lifecycle_guard = metadata_sys.object_store().acquire_bucket_lifecycle_read_lock(bucket).await?;
let transaction_guard = await_bucket_namespace_operation(
Some(&lifecycle_guard),
bucket,
"scanner incarnation read transaction lock acquisition",
acquire_metadata_transaction_guard(
metadata_sys.object_store(),
bucket,
"scanner_incarnation_probe",
rustfs_lock::LockType::Shared,
),
)
.await?;
let incarnation_id = validate_bucket_metadata_guard_incarnation(
&metadata_sys,
bucket,
Some(expected_incarnation_id),
&lifecycle_guard,
&transaction_guard,
)
.await?;
Ok(ScannerBucketIncarnationReadGuard {
bucket: bucket.to_owned(),
incarnation_id,
lifecycle_guard,
transaction_guard,
})
}
async fn acquire_config_write_guard_with_migration(
sys: Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
@@ -917,15 +1006,47 @@ async fn acquire_config_write_guard_with_migration(
Some(&lifecycle_guard),
bucket,
"bucket config transaction lock acquisition",
acquire_transaction_lock_with_sys(&sys, bucket),
acquire_transaction_lock_with_sys_for_purpose(
&sys,
bucket,
if migrate {
"bucket_config_update"
} else {
"scanner_incarnation_validation"
},
),
)
.await?;
let current_incarnation_id = validate_bucket_metadata_guard_incarnation(
&metadata_sys,
bucket,
expected_incarnation_id,
&lifecycle_guard,
&transaction_guard,
)
.await?;
Ok(BucketMetadataMutationGuard {
bucket: bucket.to_string(),
incarnation_id: current_incarnation_id,
lifecycle_guard,
transaction_guard,
})
}
async fn validate_bucket_metadata_guard_incarnation(
metadata_sys: &BucketMetadataSys,
bucket: &str,
expected_incarnation_id: Option<Uuid>,
lifecycle_guard: &rustfs_lock::NamespaceLockGuard,
transaction_guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<Uuid> {
await_bucket_namespace_operation(
Some(&lifecycle_guard),
Some(lifecycle_guard),
bucket,
"bucket config existence validation",
await_bucket_namespace_operation(
Some(&transaction_guard),
Some(transaction_guard),
bucket,
"bucket config existence transaction validation",
async {
@@ -943,11 +1064,11 @@ async fn acquire_config_write_guard_with_migration(
)
.await?;
let current_incarnation_id = await_bucket_namespace_operation(
Some(&lifecycle_guard),
Some(lifecycle_guard),
bucket,
"bucket config incarnation validation",
await_bucket_namespace_operation(
Some(&transaction_guard),
Some(transaction_guard),
bucket,
"bucket config incarnation transaction validation",
load_bucket_incarnation(metadata_sys.object_store(), bucket),
@@ -960,12 +1081,7 @@ async fn acquire_config_write_guard_with_migration(
return Err(Error::BucketNotFound(bucket.to_string()));
}
Ok(BucketMetadataMutationGuard {
bucket: bucket.to_string(),
incarnation_id: current_incarnation_id,
lifecycle_guard,
transaction_guard,
})
Ok(current_incarnation_id)
}
/// Rewrite one config file while the caller already holds this bucket's
@@ -1174,11 +1290,23 @@ async fn delete_under_config_write_guard(
/// targets one, so a writer on another node cannot land a whole-file save in
/// the middle of this read-modify-write.
pub async fn update_config_with<F>(bucket: &str, config_file: &str, mutate: F) -> Result<OffsetDateTime>
where
F: FnOnce(&BucketMetadata) -> Result<Vec<u8>> + Send,
{
update_config_with_expected(bucket, config_file, None, mutate).await
}
async fn update_config_with_expected<F>(
bucket: &str,
config_file: &str,
expected_incarnation_id: Option<Uuid>,
mutate: F,
) -> Result<OffsetDateTime>
where
F: FnOnce(&BucketMetadata) -> Result<Vec<u8>> + Send,
{
let sys = get_bucket_metadata_sys()?;
let guard = Box::pin(acquire_config_write_guard(sys.clone(), bucket)).await?;
let guard = Box::pin(acquire_config_write_guard_for_incarnation(sys.clone(), bucket, expected_incarnation_id)).await?;
guard.ensure_valid(bucket)?;
let metadata_sys = sys.read().await.clone();
let updated = await_bucket_namespace_operation(
@@ -1197,6 +1325,70 @@ where
Ok(updated)
}
fn swift_manages_bucket_tags(tagging: &Tagging) -> bool {
tagging
.tag_set
.iter()
.any(|tag| tag.key.as_deref() == Some(super::metadata::SWIFT_TAGGING_MARKER) && tag.value.as_deref() == Some("1"))
}
/// Read the S3 tag set while keeping Swift's internal tags private.
pub async fn get_s3_tagging_config(bucket: &str) -> Result<(Tagging, OffsetDateTime)> {
let (mut tagging, updated) = get_tagging_config(bucket).await?;
if swift_manages_bucket_tags(&tagging) {
tagging.tag_set.retain(|tag| {
!tag.key
.as_deref()
.is_some_and(super::metadata::is_swift_bucket_metadata_tag_key)
});
}
Ok((tagging, updated))
}
/// Replace or clear S3 tags atomically while retaining Swift metadata.
///
/// The expected incarnation, when present, is checked under the bucket
/// lifecycle and metadata guards. A failed write leaves the previous config
/// intact. Repeated writes replace the same S3 set.
pub async fn update_s3_tagging_config(
bucket: &str,
tagging: Option<Tagging>,
expected_incarnation_id: Option<Uuid>,
) -> Result<OffsetDateTime> {
let tags = tagging.map(|tagging| tagging.tag_set).unwrap_or_default();
let invalid = |message: &str| Error::InvalidArgument(bucket.to_owned(), String::new(), message.to_owned());
if tags.len() > super::metadata::MAX_BUCKET_TAGS {
return Err(invalid("Cannot have more than 50 tags per bucket"));
}
super::metadata::validate_tag_set_entries(&tags).map_err(invalid)?;
update_config_with_expected(bucket, BUCKET_TAGGING_CONFIG, expected_incarnation_id, |metadata| {
let mut merged = metadata.tagging_config.clone().unwrap_or_default();
if swift_manages_bucket_tags(&merged) {
if tags.iter().any(|tag| {
tag.key
.as_deref()
.is_some_and(super::metadata::is_swift_bucket_metadata_tag_key)
}) {
return Err(invalid("Tag key is reserved for Swift metadata on this bucket"));
}
merged.tag_set.retain(|tag| {
tag.key
.as_deref()
.is_some_and(super::metadata::is_swift_bucket_metadata_tag_key)
});
} else {
merged.tag_set.clear();
}
merged.tag_set.extend(tags);
if merged.tag_set.is_empty() {
Ok(Vec::new())
} else {
super::utils::serialize(&merged).map_err(Error::other)
}
})
.await
}
/// Acquire a bucket's metadata transaction lock, held across a whole
/// read-modify-write of its metadata file.
///
@@ -1235,13 +1427,18 @@ pub(crate) async fn acquire_bucket_metadata_transaction_lock_in(
pub(crate) async fn acquire_bucket_metadata_transaction_read_lock_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
) -> Result<rustfs_lock::NamespaceLockGuard> {
acquire_bucket_metadata_transaction_read_lock_in_for_purpose(ctx, bucket, "bucket_metadata_read").await
}
pub(crate) async fn acquire_bucket_metadata_transaction_read_lock_in_for_purpose(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
op: &'static str,
) -> Result<rustfs_lock::NamespaceLockGuard> {
let sys = bucket_metadata_sys_of(ctx)?;
let api = sys.read().await.object_store();
let lock = api
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
Ok(lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?)
acquire_metadata_transaction_guard(api, bucket, op, rustfs_lock::LockType::Shared).await
}
/// Readers already holding an Object Lock snapshot must share its guard:
@@ -1250,6 +1447,7 @@ pub(crate) async fn acquire_bucket_metadata_transaction_read_lock_for_options_in
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
opts: &ObjectOptions,
op: &'static str,
) -> Result<Arc<rustfs_lock::NamespaceLockGuard>> {
if let Some(snapshot) = opts.object_lock_config_snapshot.as_ref() {
let store = object_store_in(ctx).await?;
@@ -1259,12 +1457,22 @@ pub(crate) async fn acquire_bucket_metadata_transaction_read_lock_for_options_in
Error::other("Object Lock snapshot does not hold a valid metadata transaction fence for this bucket")
});
}
Ok(Arc::new(acquire_bucket_metadata_transaction_read_lock_in(ctx, bucket).await?))
Ok(Arc::new(
acquire_bucket_metadata_transaction_read_lock_in_for_purpose(ctx, bucket, op).await?,
))
}
async fn acquire_transaction_lock_with_sys(
sys: &Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
) -> Result<rustfs_lock::NamespaceLockGuard> {
acquire_transaction_lock_with_sys_for_purpose(sys, bucket, "bucket_metadata_transaction_write").await
}
async fn acquire_transaction_lock_with_sys_for_purpose(
sys: &Arc<RwLock<BucketMetadataSys>>,
bucket: &str,
op: &'static str,
) -> Result<rustfs_lock::NamespaceLockGuard> {
// Resolve the store under a short-lived read guard: this runs before the
// write guard in `acquire_config_write_guards`, and must not still hold a
@@ -1273,12 +1481,22 @@ async fn acquire_transaction_lock_with_sys(
let lock = api
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let acquire = lock.get_write_lock(crate::set_disk::get_lock_acquire_timeout());
let timeout = crate::set_disk::get_lock_acquire_timeout();
let mut attempt = ObjectLockAttempt::start(
op,
RUSTFS_META_BUCKET,
&bucket_metadata_transaction_lock_key(bucket),
None,
&lock,
"write",
timeout,
);
let acquire = lock.get_write_lock(timeout);
#[cfg(any(test, feature = "test-util"))]
{
tokio::pin!(acquire);
let mut notified = false;
let guard = futures::future::poll_fn(|cx| match std::future::Future::poll(acquire.as_mut(), cx) {
let result = futures::future::poll_fn(|cx| match std::future::Future::poll(acquire.as_mut(), cx) {
std::task::Poll::Pending => {
if !notified {
notify_config_write_lock_attempt(bucket);
@@ -1288,11 +1506,38 @@ async fn acquire_transaction_lock_with_sys(
}
std::task::Poll::Ready(result) => std::task::Poll::Ready(result),
})
.await?;
Ok(guard)
.await;
attempt.observe_acquisition(&result);
Ok(result?)
}
#[cfg(not(any(test, feature = "test-util")))]
Ok(acquire.await?)
{
let result = acquire.await;
attempt.observe_acquisition(&result);
Ok(result?)
}
}
pub(crate) async fn acquire_metadata_transaction_guard(
api: Arc<ECStore>,
bucket: &str,
op: &'static str,
lock_type: rustfs_lock::LockType,
) -> Result<rustfs_lock::NamespaceLockGuard> {
let object = bucket_metadata_transaction_lock_key(bucket);
let lock = api.new_ns_lock(RUSTFS_META_BUCKET, &object).await?;
let timeout = crate::set_disk::get_lock_acquire_timeout();
let mode = match lock_type {
rustfs_lock::LockType::Shared => "read",
rustfs_lock::LockType::Exclusive => "write",
};
let mut attempt = ObjectLockAttempt::start(op, RUSTFS_META_BUCKET, &object, None, &lock, mode, timeout);
let result = match lock_type {
rustfs_lock::LockType::Shared => lock.get_read_lock(timeout).await,
rustfs_lock::LockType::Exclusive => lock.get_write_lock(timeout).await,
};
attempt.observe_acquisition(&result);
Ok(result?)
}
/// The lock resource name is deliberately still the `bucket-targets` one it
@@ -2794,13 +3039,13 @@ impl BucketMetadataSys {
}
async fn find_bucket_incarnation_id_from_disk(&self, bucket: &str) -> Result<Option<Uuid>> {
let transaction_lock = self
.object_store()
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let _transaction_guard = transaction_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
let _transaction_guard = acquire_metadata_transaction_guard(
self.object_store(),
bucket,
"bucket_incarnation_disk_read",
rustfs_lock::LockType::Shared,
)
.await?;
self.find_bucket_incarnation_id_under_transaction_lock(bucket, &_transaction_guard)
.await
}
@@ -2876,13 +3121,13 @@ impl BucketMetadataSys {
/// revalidated at write quorum, so a stale snapshot can neither revert an
/// acknowledged configuration update nor outlive a delete/recreate.
async fn migrate_bucket_creation_commit(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
let transaction_lock = self
.object_store()
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let transaction_guard = transaction_lock
.get_write_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
let transaction_guard = acquire_metadata_transaction_guard(
self.object_store(),
bucket,
"bucket_creation_commit_migration",
rustfs_lock::LockType::Exclusive,
)
.await?;
let namespace_lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let namespace_guard = namespace_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
@@ -2932,13 +3177,13 @@ impl BucketMetadataSys {
}
async fn migrate_legacy_metadata(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
let transaction_lock = self
.object_store()
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let _transaction_guard = transaction_lock
.get_write_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
let _transaction_guard = acquire_metadata_transaction_guard(
self.object_store(),
bucket,
"bucket_metadata_legacy_migration",
rustfs_lock::LockType::Exclusive,
)
.await?;
let authority = self
.load_authoritative_metadata_from_disk_under_transaction_lock(bucket)
@@ -3977,6 +4222,12 @@ mod tests {
.await
.is_err()
);
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, Uuid::new_v4(), store.id)
.await
.is_err(),
"read-only scanner validation must not migrate an absent incarnation"
);
assert!(load_bucket_incarnation(store, bucket).await.expect("read sidecar").is_none());
assert!(
sys.read()
@@ -4033,6 +4284,128 @@ mod tests {
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn scanner_incarnation_read_fence_shares_reads_and_blocks_mutations() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
let bucket = "scanner-read-fence";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let incarnation = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("bucket incarnation");
let guard = acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, incarnation, store.id)
.await
.expect("read-only scanner fence");
assert_eq!(guard.checked_bucket_incarnation().expect("valid read fences"), (bucket, incarnation));
let transaction_lock = store
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await
.expect("metadata transaction namespace");
assert!(
transaction_lock.get_write_lock(Duration::from_millis(20)).await.is_err(),
"scanner read authority must exclude metadata writers"
);
let lifecycle_lock = store
.new_ns_lock(bucket, crate::storage_api_contracts::bucket::BUCKET_LIFECYCLE_LOCK_OBJECT)
.await
.expect("bucket lifecycle namespace");
assert!(
lifecycle_lock.get_write_lock(Duration::from_millis(20)).await.is_err(),
"scanner read authority must exclude delete and recreation"
);
let another = timeout(
Duration::from_secs(1),
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, incarnation, store.id),
)
.await
.expect("scanner readers must not serialize")
.expect("second read-only scanner fence");
let compatible = timeout(
Duration::from_secs(1),
acquire_bucket_metadata_transaction_read_lock_in(&store.ctx, bucket),
)
.await
.expect("metadata readers must remain compatible")
.expect("metadata transaction reader");
drop(compatible);
drop(another);
let mut mutation = Box::pin(acquire_config_write_guard_with_migration(sys.clone(), bucket, Some(incarnation), false));
assert!(matches!(futures::poll!(&mut mutation), std::task::Poll::Pending));
let mut deletion = Box::pin(store.acquire_bucket_lifecycle_write_lock(bucket));
assert!(matches!(futures::poll!(&mut deletion), std::task::Poll::Pending));
drop(deletion);
drop(guard);
let mutation = timeout(Duration::from_secs(1), mutation)
.await
.expect("metadata mutation must progress after scanner read release")
.expect("exclusive mutation fence");
drop(mutation);
let lifecycle = timeout(Duration::from_secs(1), store.acquire_bucket_lifecycle_write_lock(bucket))
.await
.expect("bucket delete fence must progress after readers release")
.expect("exclusive lifecycle fence");
drop(lifecycle);
for (owner, expected) in [
(Uuid::new_v4(), incarnation),
(Uuid::nil(), incarnation),
(store.id, Uuid::nil()),
] {
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, expected, owner)
.await
.is_err(),
"invalid owner or nil incarnation must not receive authority"
);
}
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, Uuid::new_v4(), store.id)
.await
.is_err(),
"a stale incarnation must not receive authority"
);
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("delete bucket");
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, incarnation, store.id)
.await
.is_err(),
"deleted bucket must not receive authority"
);
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("recreate bucket");
let replacement = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("replacement incarnation");
assert_ne!(replacement, incarnation);
assert!(
acquire_scanner_bucket_incarnation_read_fence_with_sys(sys.clone(), bucket, incarnation, store.id)
.await
.is_err(),
"recreation must reject the previous scanner identity"
);
let guard = acquire_scanner_bucket_incarnation_read_fence_with_sys(sys, bucket, replacement, store.id)
.await
.expect("replacement read-only scanner fence");
assert_eq!(
guard.checked_bucket_incarnation().expect("replacement remains fenced"),
(bucket, replacement)
);
}
#[tokio::test]
async fn old_node_metadata_rewrite_cannot_replace_bucket_incarnation_sidecar() {
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
@@ -886,7 +886,13 @@ pub(crate) async fn begin(
}
let metadata_lock_started = rustfs_io_metrics::put_stage_timer();
let metadata_guard = metadata_sys::acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts).await?;
let metadata_guard = metadata_sys::acquire_bucket_metadata_transaction_read_lock_for_options_in(
ctx,
bucket,
opts,
"quota_context_metadata_read",
)
.await?;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_METADATA_LOCK,
metadata_lock_started,
@@ -217,6 +217,8 @@ impl Drop for NotifyOnDrop {
#[derive(Default)]
pub struct ListPathRawOptions {
/// Retain native listing admission through producer cancellation/draining.
pub read_budget: Option<Arc<tokio::sync::OwnedSemaphorePermit>>,
pub disks: Vec<Option<DiskStore>>,
pub fallback_disks: Vec<Option<DiskStore>>,
pub bucket: String,
@@ -255,6 +257,7 @@ pub struct ListPathRawOptions {
impl Clone for ListPathRawOptions {
fn clone(&self) -> Self {
Self {
read_budget: self.read_budget.clone(),
disks: self.disks.clone(),
fallback_disks: self.fallback_disks.clone(),
bucket: self.bucket.clone(),
@@ -389,7 +392,9 @@ async fn list_path_raw_inner(
let disk_io_context_clone = disk_io_context.clone();
let (rd, wr) = tokio::io::duplex(64 * 1024);
readers.push(MetacacheReader::new(rd));
let read_budget = opts.read_budget.clone();
let producer = with_optional_disk_io_context(disk_io_context_clone, async move {
let _read_budget = read_budget;
let mut wr = PublishedBytesWriter::new(wr);
#[cfg(test)]
if let Some(observer) = &test_disk_io_context_observed {
@@ -1897,6 +1902,42 @@ mod tests {
assert_eq!(seen.lock().expect("seen mutex poisoned").as_slice(), &["bucket/object".to_string()]);
}
#[tokio::test]
async fn list_read_budget_is_retained_by_raw_producer_until_cancellation() {
let semaphore = Arc::new(tokio::sync::Semaphore::new(4));
let grant = Arc::new(semaphore.clone().acquire_many_owned(4).await.expect("listing grant"));
let weak_grant = Arc::downgrade(&grant);
let entered = Arc::new(tokio::sync::Notify::new());
let stopped = Arc::new(tokio::sync::Notify::new());
let worker = tokio::spawn(list_path_raw(
CancellationToken::new(),
ListPathRawOptions {
read_budget: Some(grant),
disks: vec![None],
min_disks: 1,
test_reader_behaviors: vec![TestReaderBehavior::StallWithSignals {
entered: entered.clone(),
stopped: stopped.clone(),
}],
..Default::default()
},
));
timeout(Duration::from_secs(5), entered.notified())
.await
.expect("producer entered");
assert!(weak_grant.strong_count() >= 2, "producer must retain a grant independently of the parent");
assert_eq!(semaphore.available_permits(), 0);
worker.abort();
assert!(worker.await.expect_err("parent cancelled").is_cancelled());
timeout(Duration::from_secs(5), stopped.notified())
.await
.expect("producer stopped");
let _next = timeout(Duration::from_secs(5), semaphore.acquire_many(4))
.await
.expect("cancelled listing releases all permits")
.expect("budget stays open");
}
#[tokio::test]
async fn list_path_raw_aborts_unresponsive_producer_after_quorum_eof() {
let result = timeout(
+285 -18
View File
@@ -127,9 +127,14 @@ const HTTP_PUT_FILE_CAPABILITY_AUTH_DOMAIN: &[u8] = b"rustfs-http-put-file-capab
const UNSIGNED_PAYLOAD: &str = "UNSIGNED-PAYLOAD";
const UNSIGNED_PAYLOAD_NONCE: &str = "unsigned";
const SIGNATURE_VALID_DURATION: i64 = 300; // 5 minutes
const REPLAY_CACHE_RETENTION: Duration = Duration::from_secs(601);
const REPLAY_CACHE_RETENTION_SECS: usize = 601;
const REPLAY_CACHE_ENTRY_BYTES_ESTIMATE: u64 = 128;
// Timestamp validity and the wall-clock high-water mark still gate eviction,
// including future-dated signatures and replay attempts after clock rollback.
const REPLAY_CACHE_RETENTION: Duration = Duration::from_secs(301);
// Preserve capacity for the full validity window at maximum accepted clock skew.
const REPLAY_CACHE_CAPACITY_WINDOW_SECS: usize = 601;
// A reserved index avoids overlapping old/new tables during a rehash. Include
// index slack, compact expiry storage, and page/allocator bookkeeping.
const REPLAY_CACHE_ENTRY_BYTES_ESTIMATE: usize = 2 * std::mem::size_of::<Uuid>() + std::mem::size_of::<RpcNonceExpiration>() + 16;
// Keep 16 CPU / 32 GiB field nodes at the 32M cap without requiring an env override.
const REPLAY_CACHE_AUTO_MEMORY_PERCENT: u64 = 13;
const REPLAY_CACHE_AUTO_RPC_RPS_PER_CPU: usize = 4096;
@@ -222,11 +227,11 @@ fn replay_cache_capacity_from_resources(cpu_count: usize, memory_limit_bytes: Op
let cpu_count = cpu_count.max(1);
let cpu_based_capacity = cpu_count
.saturating_mul(REPLAY_CACHE_AUTO_RPC_RPS_PER_CPU)
.saturating_mul(REPLAY_CACHE_RETENTION_SECS);
.saturating_mul(REPLAY_CACHE_CAPACITY_WINDOW_SECS);
let memory_based_capacity = memory_limit_bytes
.map(|bytes| {
let budget = bytes.saturating_mul(REPLAY_CACHE_AUTO_MEMORY_PERCENT) / 100;
saturating_usize_from_u64(budget / REPLAY_CACHE_ENTRY_BYTES_ESTIMATE)
saturating_usize_from_u64(budget / u64::try_from(REPLAY_CACHE_ENTRY_BYTES_ESTIMATE).unwrap_or(u64::MAX))
})
.unwrap_or(REPLAY_CACHE_AUTO_MAX_CAPACITY);
let capacity = memory_based_capacity
@@ -366,10 +371,84 @@ fn resolve_replay_cache_capacity() -> usize {
decision.capacity
}
const RPC_NONCE_EXPIRATIONS_PER_PAGE: usize = 1_024;
struct RpcNonceExpiration {
nonce: Uuid,
expiry_offset_ns: u32,
valid_until_offset_secs: i32,
}
struct RpcNonceExpirationPage {
expires_at: Instant,
valid_until: i64,
entries: VecDeque<RpcNonceExpiration>,
}
#[derive(Default)]
struct RpcNonceExpirations {
pages: VecDeque<RpcNonceExpirationPage>,
}
impl RpcNonceExpirations {
fn push_back(&mut self, expires_at: Instant, valid_until: i64, nonce: Uuid) {
// Use exact offsets and start another page if either offset is too large.
// Fully drained pages can be freed while newer nonces are still valid.
if let Some(page) = self.pages.back_mut()
&& page.entries.len() < RPC_NONCE_EXPIRATIONS_PER_PAGE
&& let Some(offset) = expires_at.checked_duration_since(page.expires_at)
&& let Ok(expiry_offset_ns) = u32::try_from(offset.as_nanos())
&& let Some(offset) = valid_until.checked_sub(page.valid_until)
&& let Ok(valid_until_offset_secs) = i32::try_from(offset)
{
page.entries.push_back(RpcNonceExpiration {
nonce,
expiry_offset_ns,
valid_until_offset_secs,
});
} else {
self.pages.push_back(RpcNonceExpirationPage {
expires_at,
valid_until,
entries: VecDeque::from([RpcNonceExpiration {
nonce,
expiry_offset_ns: 0,
valid_until_offset_secs: 0,
}]),
});
}
}
fn pop_expired(&mut self, now: Instant, wall_time: i64) -> Option<Uuid> {
let page = self.pages.front_mut()?;
let entry = page.entries.front()?;
let elapsed = now.checked_duration_since(page.expires_at)?;
let valid_until = page.valid_until + i64::from(entry.valid_until_offset_secs);
if elapsed <= Duration::from_nanos(u64::from(entry.expiry_offset_ns)) || wall_time <= valid_until {
return None;
}
let nonce = entry.nonce;
page.entries.pop_front();
if page.entries.is_empty() {
self.pages.pop_front();
if self.pages.is_empty() {
self.pages.shrink_to_fit();
}
}
Some(nonce)
}
#[cfg(test)]
fn capacity(&self) -> usize {
self.pages.iter().map(|page| page.entries.capacity()).sum()
}
}
#[derive(Default)]
struct RpcNonceCache {
nonces: HashSet<Uuid>,
expirations: VecDeque<(Instant, i64, Uuid)>,
expirations: RpcNonceExpirations,
max_wall_time: i64,
}
@@ -423,18 +502,21 @@ fn publish_nonce_cache_metrics(metrics: Option<RpcNonceCacheMetrics<'_>>) {
}
impl RpcNonceCache {
fn reserve_index(&mut self, capacity: usize) -> std::io::Result<()> {
self.nonces
.try_reserve(capacity)
.map_err(|error| std::io::Error::new(std::io::ErrorKind::OutOfMemory, error))
}
fn remove_expired(&mut self, now: Instant, wall_time: i64) -> usize {
let mut removed = 0;
while matches!(
self.expirations.front(),
Some((expires_at, valid_until, _)) if *expires_at < now && *valid_until < wall_time
) {
let Some((_, _, nonce)) = self.expirations.pop_front() else {
break;
};
while let Some(nonce) = self.expirations.pop_expired(now, wall_time) {
self.nonces.remove(&nonce);
removed += 1;
}
if removed > 0 && self.nonces.is_empty() {
self.nonces.shrink_to_fit();
}
removed
}
@@ -466,9 +548,14 @@ impl RpcNonceCache {
}),
);
}
if self.nonces.is_empty()
&& let Err(error) = self.reserve_index(record.capacity.min(REPLAY_CACHE_AUTO_MAX_CAPACITY))
{
return (Err(error), Some(metrics));
}
self.nonces.insert(record.nonce);
self.expirations
.push_back((record.expires_at, record.signed_at.saturating_add(SIGNATURE_VALID_DURATION), record.nonce));
.push_back(record.expires_at, record.signed_at.saturating_add(SIGNATURE_VALID_DURATION), record.nonce);
(
Ok(()),
Some(RpcNonceCacheMetrics {
@@ -1628,6 +1715,9 @@ pub fn verify_tonic_rpc_signature_with_bootstrap(
}
pub fn tonic_rpc_auth_failure_reason(error: &std::io::Error) -> &'static str {
if error.kind() == std::io::ErrorKind::OutOfMemory {
return "replay_cache_allocation";
}
match error.to_string().as_str() {
"Missing RPC audience" => "missing_audience",
"Invalid RPC request path" => "invalid_request_path",
@@ -3067,9 +3157,9 @@ mod tests {
assert_eq!(decision.source, ReplayCacheCapacitySource::Auto);
assert_eq!(decision.memory_basis, Some(MemoryBasis::Host));
assert_eq!(decision.memory_based_capacity, 17_448_304);
assert_eq!(decision.memory_based_capacity, 31_019_208);
assert_eq!(decision.cpu_based_capacity, 19_693_568);
assert_eq!(decision.capacity, 17_448_304);
assert_eq!(decision.capacity, 19_693_568);
}
#[test]
@@ -3079,13 +3169,13 @@ mod tests {
replay_cache_capacity_decision(rustfs_utils::EnvParseOutcome::Absent, 16, Some(32 * gib), Some(MemoryBasis::Host));
assert_eq!(decision.source, ReplayCacheCapacitySource::Auto);
assert_eq!(decision.memory_based_capacity, 34_896_609);
assert_eq!(decision.memory_based_capacity, 62_038_416);
assert_eq!(decision.cpu_based_capacity, 39_387_136);
assert_eq!(decision.capacity, REPLAY_CACHE_AUTO_MAX_CAPACITY);
let observed_field_node =
replay_cache_capacity_decision(rustfs_utils::EnvParseOutcome::Absent, 16, Some(31 * gib), Some(MemoryBasis::Host));
assert_eq!(observed_field_node.memory_based_capacity, 33_806_090);
assert_eq!(observed_field_node.memory_based_capacity, 60_099_715);
assert_eq!(observed_field_node.cpu_based_capacity, 39_387_136);
assert_eq!(observed_field_node.capacity, REPLAY_CACHE_AUTO_MAX_CAPACITY);
}
@@ -3158,6 +3248,138 @@ mod tests {
}
}
#[test]
fn nonce_cache_releases_expired_timestamp_capacity_without_allowing_clock_rollback_replay() {
let now = Instant::now();
let expiry = now + REPLAY_CACHE_RETENTION;
let later = now + Duration::from_secs(302);
let nonce = Uuid::new_v4();
let replacement = Uuid::new_v4();
let mut cache = RpcNonceCache::default();
cache
.check_and_record(test_nonce_record(nonce, 1_000, now, 1_000, expiry, 1))
.0
.expect("record initial nonce");
let boundary = cache
.check_and_record(test_nonce_record(nonce, 1_000, later, 1_300, later + REPLAY_CACHE_RETENTION, 1))
.0
.expect_err("a signature at the inclusive freshness boundary must remain protected");
assert_eq!(boundary.to_string(), "RPC request replay detected");
cache
.check_and_record(test_nonce_record(replacement, 1_302, later, 1_302, later + REPLAY_CACHE_RETENTION, 1))
.0
.expect("an expired signature must release bounded cache capacity");
assert!(!cache.nonces.contains(&nonce));
assert!(cache.nonces.contains(&replacement));
let rollback = cache
.check_and_record(test_nonce_record(nonce, 1_000, later, 1_000, later + REPLAY_CACHE_RETENTION, 1))
.0
.expect_err("clock rollback must not resurrect an evicted signature");
assert_eq!(rollback.to_string(), "RPC request timestamp expired after clock regression");
}
#[test]
fn nonce_cache_keeps_future_timestamp_reserved_until_its_full_validity_window_closes() {
let now = Instant::now();
let expiry = now + REPLAY_CACHE_RETENTION;
let later = now + Duration::from_secs(302);
let nonce = Uuid::new_v4();
let mut cache = RpcNonceCache::default();
cache
.check_and_record(test_nonce_record(nonce, 1_300, now, 1_000, expiry, 1))
.0
.expect("record nonce signed at maximum allowed forward skew");
let replay = cache
.check_and_record(test_nonce_record(nonce, 1_300, later, 1_302, later + REPLAY_CACHE_RETENTION, 1))
.0
.expect_err("monotonic expiry must not remove a still-valid future signature");
assert_eq!(replay.to_string(), "RPC request replay detected");
let full = cache
.check_and_record(test_nonce_record(Uuid::new_v4(), 1_302, later, 1_302, later + REPLAY_CACHE_RETENTION, 1))
.0
.expect_err("a still-valid nonce must continue consuming capacity");
assert_eq!(full.to_string(), "RPC replay cache capacity exceeded");
let expired = now + Duration::from_secs(602);
cache
.check_and_record(test_nonce_record(
Uuid::new_v4(),
1_602,
expired,
1_602,
expired + REPLAY_CACHE_RETENTION,
1,
))
.0
.expect("release capacity only once future timestamp validity has closed");
}
#[test]
fn nonce_index_rejects_unallocatable_capacity_with_typed_error() {
let mut cache = RpcNonceCache::default();
let error = cache
.reserve_index(usize::MAX)
.expect_err("unallocatable capacity must fail without a panic");
assert_eq!(error.kind(), std::io::ErrorKind::OutOfMemory);
assert!(
error
.get_ref()
.is_some_and(|source| source.is::<std::collections::TryReserveError>())
);
assert_eq!(tonic_rpc_auth_failure_reason(&error), "replay_cache_allocation");
assert!(cache.nonces.is_empty());
assert!(cache.expirations.pages.is_empty());
}
#[test]
fn nonce_cache_reserves_the_bounded_index_before_sustained_traffic() {
let now = Instant::now();
let expiry = now + REPLAY_CACHE_RETENTION;
let mut cache = RpcNonceCache::default();
let capacity = 4096;
cache
.check_and_record(test_nonce_record(Uuid::from_u128(1), 1000, now, 1000, expiry, capacity))
.0
.expect("record first nonce");
let allocated = cache.nonces.capacity();
assert!(allocated >= capacity, "the hash index must not rehash while serving the bounded window");
for n in 2..=capacity {
cache
.check_and_record(test_nonce_record(
Uuid::from_u128(u128::try_from(n).expect("nonce index")),
1000,
now,
1000,
expiry,
capacity,
))
.0
.expect("record within bounded capacity");
}
assert_eq!(cache.nonces.capacity(), allocated);
let full = cache
.check_and_record(test_nonce_record(Uuid::from_u128(9999), 1000, now, 1000, expiry, capacity))
.0
.expect_err("preallocation must not relax the logical capacity fence");
assert_eq!(full.to_string(), "RPC replay cache capacity exceeded");
assert_eq!(cache.nonces.len(), capacity);
}
#[test]
fn replay_cache_compact_storage_fits_two_cpu_window_in_2800_mib() {
let decision = replay_cache_capacity_decision(
rustfs_utils::EnvParseOutcome::Absent,
2,
Some(2800 * 1024 * 1024),
Some(MemoryBasis::Cgroup),
);
assert_eq!(decision.cpu_based_capacity, 4_923_392);
assert_eq!(decision.capacity, decision.cpu_based_capacity);
assert!(decision.memory_based_capacity >= decision.capacity);
assert_eq!(decision.source, ReplayCacheCapacitySource::Auto);
}
#[test]
fn nonce_cache_expires_by_monotonic_deadline_and_fails_closed_at_capacity() {
let now = Instant::now();
@@ -3178,6 +3400,51 @@ mod tests {
assert!(cache.nonces.contains(&nonce_b));
}
#[test]
fn nonce_cache_releases_expiration_storage_after_a_burst() {
let start = Instant::now();
let expiry = start + REPLAY_CACHE_RETENTION;
let mut cache = RpcNonceCache::default();
for nonce in 1..=4_096 {
let record = test_nonce_record(Uuid::from_u128(nonce), 100, start, 100, expiry, 4_096);
cache.check_and_record(record).0.expect("first use of nonce");
}
assert!(cache.expirations.capacity() >= 4_096);
assert_eq!(cache.remove_expired(expiry, 701), 0);
assert_eq!(cache.remove_expired(expiry + Duration::from_nanos(1), 400), 0);
assert_eq!(cache.remove_expired(expiry + Duration::from_nanos(1), 701), 4_096);
assert!(cache.nonces.is_empty());
assert_eq!(cache.nonces.capacity(), 0);
assert_eq!(cache.expirations.capacity(), 0, "expired entries must release their expiration storage");
}
#[test]
fn nonce_expirations_preserve_exact_deadlines_across_pages() {
let expiry = Instant::now();
let first = Uuid::from_u128(1);
let second = Uuid::from_u128(2);
let third = Uuid::from_u128(3);
let fourth = Uuid::from_u128(4);
let mut queue = super::RpcNonceExpirations::default();
let next = expiry + Duration::from_nanos(1);
let later = expiry + Duration::from_secs(5);
queue.push_back(expiry, 501, first);
queue.push_back(next, 500, second);
queue.push_back(later, 505, third);
queue.push_back(later, i64::MAX, fourth);
assert_eq!(queue.pages.len(), 3);
assert_eq!(queue.pop_expired(next, 501), None);
assert_eq!(queue.pop_expired(next, 502), Some(first));
assert_eq!(queue.pop_expired(next, 502), None);
assert_eq!(queue.pop_expired(later, 502), Some(second));
assert_eq!(queue.pages.len(), 2);
assert_eq!(queue.pop_expired(later, 506), None);
assert_eq!(queue.pop_expired(later + Duration::from_nanos(1), 506), Some(third));
assert_eq!(queue.pop_expired(later + Duration::from_nanos(1), i64::MAX), None);
}
#[test]
fn nonce_cache_metrics_mark_successful_records_only() {
let now = Instant::now();
+85 -13
View File
@@ -27,13 +27,13 @@ use crate::disk::multipart_source::{
MULTIPART_SOURCE_PROTOCOL_VERSION, MultipartSourceOperation, MultipartSourceRequest, MultipartSourceResponse,
};
use crate::disk::{
BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION,
DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection,
DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation,
DiskOption, FileInfoVersions, FileReader, FileWriter, GuardedDeleteVersionsPayload, GuardedFileInfoVersions,
PartTransactionAction, PoolMetaNativeRecoveryRequest, PoolMetaNativeRecoveryResponse, ReadMultipleReq, ReadMultipleResp,
ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
batch_read_version_one_by_one,
BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION, BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate,
DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest,
DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo,
DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions, FileReader, FileWriter, GuardedDeleteVersionsPayload,
GuardedFileInfoVersions, PartTransactionAction, PoolMetaNativeRecoveryRequest, PoolMetaNativeRecoveryResponse,
ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo,
WalkDirOptions, batch_read_version_one_by_one,
disk_store::{
DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE,
get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout,
@@ -376,6 +376,21 @@ struct WalkCopyError {
source: io::Error,
}
impl WalkCopyError {
fn into_disk_error(self) -> DiskError {
// Only a local output write/flush proves the metacache consumer closed.
// An identical I/O kind from the remote input remains a transport failure.
if matches!(self.reason, RemoteWalkFailureReason::Write | RemoteWalkFailureReason::Flush)
&& self.source.kind() == io::ErrorKind::BrokenPipe
&& self.source.get_ref().is_none()
{
DiskError::metacache_output_stream_closed()
} else {
DiskError::Io(self.source)
}
}
}
async fn copy_stream_with_buffer<R, W>(
reader: &mut R,
writer: &mut W,
@@ -1032,6 +1047,7 @@ impl RemoteDisk {
volume: &str,
payload: Vec<u8>,
probe_only: bool,
bounded_prepare: bool,
) -> Result<DeleteVersionsResponse> {
if payload.len() > rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES {
return Err(DiskError::FileCorrupt);
@@ -1056,6 +1072,8 @@ impl RemoteDisk {
.map_err(|status| {
if status.code() == Code::Unimplemented {
DiskError::GuardedDeleteUnsupported
} else if bounded_prepare && status.code() == Code::InvalidArgument {
DiskError::UnsupportedDisk
} else {
status.into()
}
@@ -1089,6 +1107,14 @@ impl RemoteDisk {
volume: &str,
operation: DeleteRecoveryRpcOperation,
) -> Result<DeleteRecoveryRpcReply> {
let protocol_version = match &operation {
DeleteRecoveryRpcOperation::ApplyVersion { opts, .. } | DeleteRecoveryRpcOperation::ApplyVersions { opts, .. }
if opts.requires_bounded_delete_recovery =>
{
BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION
}
_ => DELETE_RECOVERY_PROTOCOL_VERSION,
};
let payload = rmp_serde::to_vec_named(&operation)?;
if payload.len() > operation.max_wire_payload_bytes() {
return Err(Error::other("delete recovery operation exceeds payload limit"));
@@ -1119,7 +1145,7 @@ impl RemoteDisk {
let mut request = Request::new(RecoverDeleteTransactionRequest {
disk: self.endpoint.to_string(),
volume: volume.to_owned(),
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
protocol_version,
operation: payload.into(),
});
let canonical_body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref())
@@ -1137,7 +1163,7 @@ impl RemoteDisk {
}
})?
.into_inner();
if response.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION {
if response.protocol_version != protocol_version {
return Err(DiskError::UnsupportedDisk);
}
if !response.success {
@@ -2085,6 +2111,7 @@ impl RemoteDisk {
failure_health_action: FailureHealthAction,
) {
if let Err(err) = operation_result
&& !err.is_metacache_output_stream_closed()
&& is_network_like_disk_error(err)
{
counter!(
@@ -2779,7 +2806,7 @@ impl DiskAPI for RemoteDisk {
}
async fn supports_guarded_delete_versions(&self) -> Result<()> {
let response = self.guarded_delete_versions_rpc("", Vec::new(), true).await?;
let response = self.guarded_delete_versions_rpc("", Vec::new(), true, false).await?;
if !response.errors.is_empty() || !response.item_errors.is_empty() {
return Err(DiskError::FileCorrupt);
}
@@ -2792,7 +2819,12 @@ impl DiskAPI for RemoteDisk {
opts: DeleteOptions,
) -> Vec<Option<DiskError>> {
let count = versions.len();
let payload = GuardedDeleteVersionsPayload { versions, opts };
let bounded_prepare = opts.requires_bounded_delete_recovery;
let payload = GuardedDeleteVersionsPayload {
versions,
opts,
bounded_recovery_protocol: bounded_prepare.then_some(BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION),
};
if let Err(error) = payload.validate() {
return vec![Some(error); count];
}
@@ -2800,7 +2832,10 @@ impl DiskAPI for RemoteDisk {
Ok(encoded) => encoded,
Err(error) => return vec![Some(error.into()); count],
};
match self.guarded_delete_versions_rpc(volume, encoded, false).await {
match self
.guarded_delete_versions_rpc(volume, encoded, false, bounded_prepare)
.await
{
Ok(response) => {
// New guarded peers must return typed errors, never lossy strings.
if response.item_errors.len() != count {
@@ -4037,7 +4072,7 @@ impl DiskAPI for RemoteDisk {
Ok(_) => return Ok(()),
Err(error) => {
failure_reason = Some(error.reason);
return Err(DiskError::Io(error.source));
return Err(error.into_disk_error());
}
}
}
@@ -9668,6 +9703,43 @@ mod tests {
}
}
#[tokio::test]
async fn remote_walk_preserves_local_metacache_consumer_close() {
let transport =
RetryingWalkDirInternodeDataTransport::with_steps(vec![WalkDirTestStep::Data(b"metacache payload".to_vec())]);
let disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
let (mut writer, reader) = tokio::io::duplex(1);
drop(reader);
let error = disk
.walk_dir(WalkDirOptions::default(), &mut writer)
.await
.expect_err("a dropped metacache consumer must stop the producer");
assert!(
error.is_metacache_output_stream_closed(),
"preserve the benign completion marker: {error:?}"
);
assert_eq!(transport.calls().len(), 1, "local consumer closure is not a remote open retry");
assert!(!disk.health.is_faulty());
}
#[tokio::test]
async fn remote_walk_keeps_remote_read_broken_pipe_as_transport_failure() {
let transport = RetryingWalkDirInternodeDataTransport::with_steps(vec![WalkDirTestStep::PartialDataThenError {
data: b"partial metacache".to_vec(),
error: io::ErrorKind::BrokenPipe.into(),
}]);
let disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
let mut writer = Vec::new();
let error = disk
.walk_dir(WalkDirOptions::default(), &mut writer)
.await
.expect_err("an upstream close after partial output is a failed walk");
assert!(!error.is_metacache_output_stream_closed());
assert!(error.contains_io_error_kind(io::ErrorKind::BrokenPipe));
assert_eq!(writer, b"partial metacache");
assert_eq!(transport.calls().len(), 1, "partial output must not be replayed");
}
#[tokio::test]
async fn walk_stream_backpressure_limits_read_ahead_and_is_cancellable() {
let mut reader = Cursor::new(vec![5; 4 * DEFAULT_READ_BUFFER_SIZE]);
@@ -423,11 +423,15 @@ impl RemoteClient {
/// peer toward the offline threshold. See issue #4567.
fn is_transport_failure(status: &tonic::Status) -> bool {
use tonic::Code;
// A valid stale-epoch request is rejected before execution. Its signed
// response refreshes the peer epoch on the existing healthy channel.
let epoch_rejection = status.code() == Code::Unavailable && status.message() == "RPC boot epoch changed";
std::error::Error::source(status).is_some()
|| matches!(
status.code(),
Code::Unavailable | Code::DeadlineExceeded | Code::Unknown | Code::Cancelled
)
|| (!epoch_rejection
&& matches!(
status.code(),
Code::Unavailable | Code::DeadlineExceeded | Code::Unknown | Code::Cancelled
))
}
async fn evict_connection(&self, op: &'static str, reason: &str, resource_summary: &str) {
@@ -2827,6 +2831,37 @@ mod tests {
}
}
#[tokio::test]
async fn stale_boot_epoch_rejections_do_not_open_lock_request_breaker() {
let addr = "http://stale-boot-epoch-lock-breaker:9000";
let client = RemoteClient::new(addr.to_string());
reset_lock_peer_health_for_test(addr);
for _ in 0..LOCK_RPC_REQUEST_BACKOFF_THRESHOLD {
let error = client
.execute_rpc::<(), _>(
"lock",
"bucket/new-object",
Duration::from_secs(1),
async { Err(tonic::Status::unavailable("RPC boot epoch changed")) },
RpcCleanup::new(None, None, None),
)
.await
.expect_err("a stale epoch still rejects the operation before execution");
assert!(error.to_string().contains("RPC boot epoch changed"));
}
let health = lock_peer_health_for_test(addr);
assert_eq!(health.request_consecutive_timeouts, 0);
assert!(health.request_backoff_until.is_none());
assert!(health.last_eviction.is_none());
let admission = client
.admit_lock_request("lock")
.expect("authenticated epoch refresh may proceed immediately");
client.release_lock_request("lock", Some(admission));
reset_lock_peer_health_for_test(addr);
}
#[test]
#[serial_test::serial]
fn test_remote_client_rpc_timeout_honors_configured_deadline() {
+455 -59
View File
@@ -2208,9 +2208,14 @@ fn ensure_decommission_capacity_reservations_available(
continue;
}
metrics::counter!(METRIC_DECOMMISSION_CAPACITY_CONFLICTS_TOTAL, "phase" => phase).increment(1);
return Err(decommission_capacity_blocked_error(format!(
let detail = format!(
"target pool {target_pool_index} reservation requires {required} physical bytes, but only {available} bytes remain during {phase}"
)));
);
return Err(if growth_pool.is_some() {
decommission_target_no_longer_fits_error(target_pool_index, detail)
} else {
decommission_capacity_blocked_error(detail)
});
}
Ok(())
}
@@ -2341,36 +2346,41 @@ fn grow_decommission_target_reservation(
let existing_remaining = existing_target
.map(|target| target.remaining_reserved_physical_bytes(reservation.temporary_copies))
.unwrap_or_default();
let existing_reserved = existing_target
.map(|target| target.reserved_physical_bytes)
.unwrap_or_default();
let allocation_limit = existing_target
.map(|target| target.physical_free_at_reservation)
.unwrap_or_default()
.max(target_capacity.physical_free);
if required_peak <= existing_remaining {
return Ok(false);
}
if reservation.source_data_equivalent_bytes > 0 {
return Err(decommission_target_no_longer_fits_error(
target_pool_index,
format!("source pool {source_pool_index} reservation estimate is exhausted"),
));
}
if let Some(target) = existing_target
&& target_capacity.physical_free <= target.physical_free_at_reservation
{
return Err(decommission_capacity_blocked_error(format!(
"source pool {source_pool_index} target pool {target_pool_index} reservation is exhausted"
)));
}
// Scanner estimates can predate source writes. Grow against current
// target capacity, then validate the aggregate reservation below.
let current_peak = reservation.remaining_peak_physical_bytes();
let additional_peak = required_peak.saturating_sub(existing_remaining);
let additional_predicted = additional_peak.div_ceil(1usize.saturating_add(reservation.temporary_copies));
let additional_source_data = capacity_source_data_equivalent(additional_predicted, target_layout)?;
reservation.source_data_equivalent_bytes = reservation
.source_data_equivalent_bytes
.saturating_add(additional_source_data);
reservation.source_physical_bytes = reservation
.source_physical_bytes
.saturating_add(capacity_target_physical_bytes(additional_source_data, reservation.source_layout)?);
// Recompute from the physical total so EC rounding matches the
// persisted reader after every growth, including repeated growth.
reservation.source_data_equivalent_bytes =
capacity_source_data_equivalent(reservation.source_physical_bytes, reservation.source_layout)?;
reservation.predicted_physical_bytes = reservation
.predicted_physical_bytes
.saturating_add(additional_predicted)
.max(capacity_target_physical_bytes(reservation.source_data_equivalent_bytes, target_layout)?);
.targets
.iter()
.map(|target| target.layout)
.chain(std::iter::once(target_layout))
.try_fold(
reservation.predicted_physical_bytes.saturating_add(additional_predicted),
|predicted, layout| {
capacity_target_physical_bytes(reservation.source_data_equivalent_bytes, layout)
.map(|required| predicted.max(required))
},
)?;
reservation.temporary_physical_bytes = reservation
.predicted_physical_bytes
.saturating_mul(reservation.temporary_copies);
@@ -2378,6 +2388,15 @@ fn grow_decommission_target_reservation(
.predicted_physical_bytes
.saturating_add(reservation.temporary_physical_bytes);
let allocation_growth = reservation.remaining_peak_physical_bytes().saturating_sub(current_peak);
let required_allocation = existing_reserved.saturating_add(allocation_growth);
if required_allocation > allocation_limit {
return Err(decommission_target_no_longer_fits_error(
target_pool_index,
format!(
"target pool {target_pool_index} reservation requires {required_allocation} physical bytes, but its observed allocation limit is {allocation_limit} bytes"
),
));
}
if let Some(target) = reservation
.targets
.iter_mut()
@@ -2402,6 +2421,7 @@ fn grow_decommission_target_reservation(
});
}
renew_decommission_capacity_reservation(reservation, now, true);
validate_decommission_capacity_reservation(Some(reservation))?;
pool.last_update = pool.last_update.max(now);
Ok(true)
})();
@@ -8292,16 +8312,18 @@ where
S: EcstoreObjectIO,
{
// Bound the CAS state retained by callers, including ordinary metadata
// transactions that never enter identity repair. Large inline retry states
// transactions that never enter owner repair. Large inline retry states
// otherwise propagate through startup and cross-crate storage futures.
Box::pin(async move {
if object != POOL_META_IDENTITY_NAME {
let native_owner_repair = object == POOL_META_NAME && matches!(phase, "native_heal" | "prepare_cas");
if object != POOL_META_IDENTITY_NAME && !native_owner_repair {
return execute_pool_meta_object_cas_once(pool, object, data, opts, fence, phase).await;
}
// Keep the exact identity owned by this transaction. A quorum read alone
// cannot discharge repair responsibility for a missing disk replica.
let identity = data;
let mut outcome = execute_pool_meta_object_cas_once(pool.clone(), object, identity.clone(), opts, fence, phase).await;
// Repair only the exact payload already published by this owner. A
// quorum read cannot discharge responsibility for a missing replica.
let committed_payload = data;
let mut outcome =
execute_pool_meta_object_cas_once(pool.clone(), object, committed_payload.clone(), opts, fence, phase).await;
for delay_secs in [1, 2, 4] {
if !outcome.source.as_deref().is_some_and(|err| {
matches!(err, Error::Io(inner) if inner.get_ref().is_some_and(|source| source.is::<crate::set_disk::SystemMetadataRepairRequired>()))
@@ -8314,19 +8336,28 @@ where
let repair_options = async {
fence.ensure_held()?;
let (current, info) = read_config_no_lock_preserve_empty_with_metadata(pool.clone(), object).await?;
if current != identity {
if current != committed_payload {
return Err(Error::PreconditionFailed);
}
let token = info
.etag
.filter(|etag| !etag.trim().is_empty())
.map(PoolMetaCasToken::Existing)
.unwrap_or(PoolMetaCasToken::Unsafe);
let token = if native_owner_repair {
match pool_meta_cas_token_from_info(info) {
token @ PoolMetaCasToken::ExistingNative { .. } => token,
_ => return Err(Error::PreconditionFailed),
}
} else {
info.etag
.filter(|etag| !etag.trim().is_empty())
.map(PoolMetaCasToken::Existing)
.unwrap_or(PoolMetaCasToken::Unsafe)
};
pool_meta_cas_options(object, &token, fence)
}
.await;
outcome = match repair_options {
Ok(opts) => execute_pool_meta_object_cas_once(pool.clone(), object, identity.clone(), opts, fence, phase).await,
Ok(opts) => {
execute_pool_meta_object_cas_once(pool.clone(), object, committed_payload.clone(), opts, fence, phase)
.await
}
Err(err) => PoolMetaCasWriteOutcome::not_dispatched(err),
};
// A rejected repair cannot turn the preceding committed write into a
@@ -12134,8 +12165,13 @@ async fn pause_decommission_start_for_test(store_id: uuid::Uuid, point: Decommis
}
#[cfg(any(test, feature = "test-util"))]
type DecommissionCapacityInfoOverrides =
std::sync::Mutex<HashMap<uuid::Uuid, std::collections::VecDeque<Vec<DecommissionPoolCapacityInfo>>>>;
struct DecommissionCapacityInfoOverride {
snapshots: std::collections::VecDeque<Vec<DecommissionPoolCapacityInfo>>,
owned_snapshots: Option<std::collections::VecDeque<Vec<DecommissionPoolCapacityInfo>>>,
}
#[cfg(any(test, feature = "test-util"))]
type DecommissionCapacityInfoOverrides = std::sync::Mutex<HashMap<uuid::Uuid, DecommissionCapacityInfoOverride>>;
#[cfg(any(test, feature = "test-util"))]
static DECOMMISSION_CAPACITY_INFO_OVERRIDES: std::sync::OnceLock<DecommissionCapacityInfoOverrides> = std::sync::OnceLock::new();
@@ -12152,16 +12188,30 @@ pub(crate) fn set_decommission_capacity_info_overrides_for_test(
.get_or_init(|| std::sync::Mutex::new(HashMap::new()))
.lock()
.expect("decommission capacity info override should not be poisoned")
.insert(store_id, snapshots.into());
.insert(
store_id,
DecommissionCapacityInfoOverride {
snapshots: snapshots.into(),
owned_snapshots: None,
},
);
}
#[cfg(any(test, feature = "test-util"))]
fn take_decommission_capacity_info_override_for_test(store_id: uuid::Uuid) -> Option<Vec<DecommissionPoolCapacityInfo>> {
fn take_decommission_capacity_info_override_for_test(
store_id: uuid::Uuid,
use_owned_usage: bool,
) -> Option<Vec<DecommissionPoolCapacityInfo>> {
let mut overrides = DECOMMISSION_CAPACITY_INFO_OVERRIDES
.get_or_init(|| std::sync::Mutex::new(HashMap::new()))
.lock()
.expect("decommission capacity info override should not be poisoned");
let queue = overrides.get_mut(&store_id)?;
let entry = overrides.get_mut(&store_id)?;
let queue = if use_owned_usage {
entry.owned_snapshots.as_mut().unwrap_or(&mut entry.snapshots)
} else {
&mut entry.snapshots
};
if queue.len() > 1 {
queue.pop_front()
} else {
@@ -15231,7 +15281,7 @@ impl ECStore {
ensure_decommission_start_pool_states(&latest_pool_meta, indices)?;
ensure_decommission_ledger_persistence_supported(&latest_pool_meta)?;
ensure_decommission_capacity_writer_supported(&latest_pool_meta)?;
let capacity_infos = self.get_decommission_all_pool_capacity_infos().await?;
let capacity_infos = self.get_decommission_all_pool_capacity_infos_with_owned_usage().await?;
activation_fence.ensure_held()?;
let target_fence_proof = crate::services::notification_sys::acquire_decommission_target_fence_fleet_proof();
@@ -15593,7 +15643,7 @@ impl ECStore {
async fn get_decommission_prewrite_free_capacity(&self, target_pool_index: usize) -> Result<usize> {
#[cfg(any(test, feature = "test-util"))]
if let Some(capacity_infos) = take_decommission_capacity_info_override_for_test(self.id) {
if let Some(capacity_infos) = take_decommission_capacity_info_override_for_test(self.id, false) {
return capacity_infos
.iter()
.find(|capacity| capacity.pool_index == target_pool_index)
@@ -15616,7 +15666,7 @@ impl ECStore {
async fn get_decommission_pool_capacity_infos(&self, use_owned_usage: bool) -> Result<Vec<DecommissionPoolCapacityInfo>> {
#[cfg(any(test, feature = "test-util"))]
if let Some(capacity_infos) = take_decommission_capacity_info_override_for_test(self.id) {
if let Some(capacity_infos) = take_decommission_capacity_info_override_for_test(self.id, use_owned_usage) {
return Ok(capacity_infos);
}
@@ -15834,14 +15884,8 @@ impl ECStore {
log_decommission_adopted_target_choice(reservation, owner, pool_index, &over_budget);
return Ok(pool_index);
}
if reservation.source_data_equivalent_bytes > 0 {
let mut skipped = over_budget.iter().copied().collect::<Vec<_>>();
skipped.sort_unstable();
return Err(decommission_capacity_blocked_error(format!(
"source pool {} has no target allocation for {expected_data_bytes} data bytes; target pools over their temporary-copy budget: {skipped:?}",
owner.source_pool_index
)));
}
// A stale nonzero estimate can exhaust its allocation too. Selection
// is provisional until fenced growth persists the new reservation.
let capacities = self.get_decommission_all_pool_capacity_infos().await?;
let active_sources = active_decommission_source_indices(&pool_meta);
capacities
@@ -25773,6 +25817,55 @@ mod tests {
);
}
#[tokio::test]
#[serial_test::serial]
async fn decommission_activation_reserves_owned_usage_on_a_shared_filesystem() {
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(None).await;
let layout = DecommissionErasureLayout { data: 1, parity: 0 };
let target = DecommissionPoolCapacityInfo::for_test(1, layout, 60, 60, 0);
let filesystem_source = DecommissionPoolCapacityInfo::for_test(0, layout, 0, 400, 400);
let owned_source = DecommissionPoolCapacityInfo::for_test(0, layout, 0, 400, 30);
set_decommission_capacity_info_overrides_for_test(store.id, vec![vec![filesystem_source, target]]);
{
let mut overrides = super::DECOMMISSION_CAPACITY_INFO_OVERRIDES
.get()
.expect("capacity overrides initialized")
.lock()
.expect("capacity overrides lock");
overrides
.get_mut(&store.id)
.expect("this store has an override")
.owned_snapshots = Some(vec![vec![owned_source, target]].into());
}
store
.save_current_pool_meta_for_decommission_start(&[0], Vec::new())
.await
.expect("activation should reserve the owned bytes rather than unrelated mount usage");
let local = store.pool_meta.read().await;
let reservation = local.pools[0]
.decommission
.as_ref()
.and_then(|state| state.capacity_reservation.as_ref())
.expect("activation must retain its capacity reservation");
assert_eq!(reservation.source_physical_bytes, 30);
assert_eq!(reservation.peak_physical_bytes, 60);
drop(local);
let mut persisted = PoolMeta::default();
persisted
.load_no_lock_from_replicas(store.pools.clone())
.await
.expect("activation should be durable");
let reservation = persisted.pools[0]
.decommission
.as_ref()
.and_then(|state| state.capacity_reservation.as_ref())
.expect("persisted activation must retain its capacity reservation");
assert_eq!(reservation.source_physical_bytes, 30);
assert_eq!(reservation.peak_physical_bytes, 60);
}
#[tokio::test]
#[serial_test::serial]
async fn decommission_activation_lock_recheck_rejects_sudden_space_drop_without_persisting() {
@@ -25807,6 +25900,173 @@ mod tests {
assert!(persisted.pools[0].decommission.is_none());
}
#[tokio::test]
#[serial_test::serial]
async fn decommission_copies_native_object_after_nonzero_usage_estimate_is_exhausted() {
use tokio::io::AsyncReadExt as _;
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(None).await;
let bucket = format!("stale-owned-{}", uuid::Uuid::new_v4());
let object = "new-after-scan.bin";
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create source bucket");
let layout = DecommissionErasureLayout { data: 1, parity: 0 };
let capacities = vec![
DecommissionPoolCapacityInfo::for_test(0, layout, 0, 1 << 30, 1),
DecommissionPoolCapacityInfo::for_test(1, layout, 1 << 30, 1 << 30, 0),
];
set_decommission_capacity_info_overrides_for_test(store.id, vec![capacities]);
let expected = vec![0x57; 16 * 1024];
let mut body = crate::object_api::PutObjReader::from_vec(expected.clone());
store.pools[0]
.put_object(&bucket, object, &mut body, &ObjectOptions::default())
.await
.expect("write an object larger than the earlier scanner estimate");
store
.save_current_pool_meta_for_decommission_start(
&[0],
vec![DecomBucketInfo {
name: bucket.clone(),
prefix: String::new(),
}],
)
.await
.expect("activate the stale nonzero reservation");
let canceler = DecommissionCanceler::new(CancellationToken::new());
store.decommission_cancelers.write().await[0] = Some(canceler.clone());
store
.do_decommission_in_routine(canceler, 0, Arc::new(Semaphore::new(1)))
.await
.expect("run the real native-disk mover");
let mut reader = store.pools[1]
.get_object_reader(&bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
.await
.expect("the newer object must reach the target despite the stale estimate");
let mut actual = Vec::new();
reader.stream.read_to_end(&mut actual).await.expect("read migrated payload");
assert_eq!(actual, expected);
assert!(
store.pools[0]
.get_object_info(&bucket, object, &ObjectOptions::default())
.await
.is_err(),
"source cleanup follows the verified target copy"
);
let meta = store.pool_meta.read().await;
let state = meta.pools[0].decommission.as_ref().expect("retirement state is retained");
assert!(state.complete && !state.failed);
assert_eq!(state.items_decommission_failed, 0);
}
#[tokio::test]
#[cfg(feature = "test-util")]
#[serial_test::serial]
async fn decommission_native_ec_growth_keeps_all_metadata_replicas_readable() {
use rustfs_data_usage::{DataUsageCache, DataUsageEntry};
use tokio::io::AsyncReadExt as _;
let (_temp_dirs, store, _other_store) =
crate::services::rebalance::test_three_pool_stores_with_four_disk_sets_with_isolated_node_contexts(None).await;
let bucket = format!("stale-ec-{}", uuid::Uuid::new_v4());
let object = "new-after-scan.bin";
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
for set in &store.pools[0].disk_set {
let encoded = rmp_serde::to_vec_named(&serde_json::json!({
"info": {
"name": DATA_USAGE_ROOT,
"next_cycle": 1,
"last_update": std::time::SystemTime::now(),
"skip_healing": false,
"snapshot_complete": true
},
"cache": {
DATA_USAGE_ROOT: DataUsageEntry { size: 3, objects: 1, ..Default::default() }
}
}))
.expect("encode completed scanner fixture");
let cache = DataUsageCache::unmarshal(&encoded).expect("read scanner fixture");
assert!(cache.dui(DATA_USAGE_ROOT, &[]).is_complete_bucket_usage_snapshot());
set.put_object(
RUSTFS_META_BUCKET,
&format!("{}/{}", crate::disk::BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME),
&mut crate::object_api::PutObjReader::from_vec(encoded),
&ObjectOptions::default(),
)
.await
.expect("persist the earlier complete cache");
}
let expected = vec![0x37; 16 * 1024];
store.pools[0]
.put_object(
&bucket,
object,
&mut crate::object_api::PutObjReader::from_vec(expected.clone()),
&ObjectOptions::default(),
)
.await
.expect("write after the scan");
store
.save_current_pool_meta_for_decommission_start(
&[0],
vec![DecomBucketInfo {
name: bucket.clone(),
prefix: String::new(),
}],
)
.await
.expect("activate from the earlier owned estimate");
{
let meta = store.pool_meta.read().await;
let reservation = meta.pools[0]
.decommission
.as_ref()
.unwrap()
.capacity_reservation
.as_ref()
.unwrap();
assert_eq!(reservation.source_layout, DecommissionErasureLayout { data: 3, parity: 1 });
assert_eq!(reservation.source_physical_bytes, 8);
}
let canceler = DecommissionCanceler::new(CancellationToken::new());
store.decommission_cancelers.write().await[0] = Some(canceler.clone());
store
.do_decommission_in_routine(canceler, 0, Arc::new(Semaphore::new(1)))
.await
.expect("run native EC migration");
for pool in &store.pools {
let mut reloaded = PoolMeta::default();
reloaded
.load_no_lock_from_replicas(vec![pool.clone()])
.await
.expect("each saved metadata replica must reload");
let state = reloaded.pools[0].decommission.as_ref().expect("saved retirement state");
assert!(state.complete && !state.failed);
}
let mut reader = store
.get_object_reader(&bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
.await
.expect("target object must remain readable");
let mut actual = Vec::new();
reader
.stream
.read_to_end(&mut actual)
.await
.expect("read migrated EC payload");
assert_eq!(actual, expected);
assert!(
store.pools[0]
.get_object_info(&bucket, object, &ObjectOptions::default())
.await
.is_err()
);
}
#[tokio::test]
#[serial_test::serial]
async fn decommission_worker_pauses_on_runtime_capacity_shortage_without_source_side_effect() {
@@ -31049,6 +31309,10 @@ mod pools_tests {
std::future::pending::<()>().await;
}
let mut reader = self.inner.get_object_reader(bucket, object, range, headers, opts).await?;
if object == POOL_META_NAME {
reader.object_info.mod_time = Some(time::OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1));
reader.object_info.data_dir = Some(uuid::Uuid::from_u128(7));
}
match self.fault {
IdentityRepairFault::Changed => {
reader.stream = Box::new(Cursor::new(b"other-identity".to_vec()));
@@ -31078,16 +31342,30 @@ mod pools_tests {
.and_then(HTTPPreconditions::if_none_match_value),
None
);
let written = if object == POOL_META_NAME {
&self.inner.stored
} else {
&self.inner.identity
};
let current_etag = written.lock().expect("metadata lock").as_ref().map(|(_, etag)| etag.clone());
assert_eq!(
opts.http_preconditions.as_ref().and_then(HTTPPreconditions::if_match_value),
self.inner
.identity
.lock()
.expect("identity lock")
.as_ref()
.map(|(_, etag)| etag.as_str()),
current_etag.as_deref(),
"repair must use the observed post-commit CAS token",
);
if object == POOL_META_NAME {
let condition = rustfs_utils::http::metadata_compat::get_consistent_str(
&opts.user_defined,
crate::disk::POOL_META_NATIVE_CAS_METADATA_SUFFIX,
)
.expect("native repair requires a physical predecessor condition");
let condition: crate::disk::PoolMetaNativeCasCondition =
serde_json::from_str(condition).expect("parse native CAS condition");
let parent = condition.parent.expect("native CAS uses the actual committed head");
assert_eq!(Some(parent.etag), current_etag);
assert_eq!(parent.mod_time_unix_nanos, 1_000_000_000);
assert_eq!(parent.data_dir, Some(uuid::Uuid::from_u128(7)));
}
if self.fault == IdentityRepairFault::Reject {
return Err(Error::PreconditionFailed);
}
@@ -31192,6 +31470,46 @@ mod pools_tests {
}
}
#[tokio::test(start_paused = true)]
async fn pool_meta_prepare_owner_repair_uses_exact_native_head() {
let payload = b"owned-pending-pool-meta";
for (fault, writes, reads, succeeds) in [
(IdentityRepairFault::Once, 2, 1, true),
(IdentityRepairFault::Changed, 1, 1, false),
(IdentityRepairFault::MissingEtag, 1, 1, false),
(IdentityRepairFault::Unknown, 1, 0, false),
] {
let pool = IdentityRepairStorage::new(fault);
let state = super::PoolMetaWriteState::default();
let mut arm = state.arm_transaction();
let result = save_pool_meta_object_cas(
pool.clone(),
POOL_META_NAME,
payload.to_vec(),
&PoolMetaCasToken::Missing,
&PoolMetaPersistenceFence::OfflineRecovery,
"prepare_cas",
&mut arm,
)
.await;
if succeeds {
result.expect("prepare must repair its own partially replicated payload");
arm.disarm();
assert_eq!(pool.inner.stored.lock().expect("metadata lock").as_ref().expect("written").0, payload);
} else {
let error = result.expect_err("uncertain or changed authority must remain blocked");
let failure = error.pool_metadata_failure().expect("typed transaction uncertainty");
assert_eq!(failure.kind, crate::error::PoolMetadataFailure::TransactionUnknown);
assert_eq!(failure.phase, "prepare_cas");
}
drop(arm);
assert_eq!(state.ensure_write_safe("after prepare owner repair").is_ok(), succeeds);
assert_eq!(pool.writes.load(Ordering::SeqCst), writes, "{fault:?}");
assert_eq!(pool.reads.load(Ordering::SeqCst), reads, "{fault:?}");
}
}
#[tokio::test(start_paused = true)]
async fn pool_identity_owner_repair_cancellation_keeps_the_transaction_armed() {
for fault in [
@@ -37556,10 +37874,88 @@ mod pools_tests {
.as_ref()
.and_then(|info| info.capacity_reservation.as_ref())
.unwrap();
assert_eq!(reservation.source_data_equivalent_bytes, 2);
assert_eq!(reservation.predicted_physical_bytes, 3);
assert_eq!(reservation.source_data_equivalent_bytes, 3);
assert_eq!(reservation.predicted_physical_bytes, 4);
assert_eq!(reservation.targets.len(), 1);
assert_eq!(reservation.targets[0].reserved_physical_bytes, 6);
assert_eq!(reservation.targets[0].reserved_physical_bytes, 8);
validate_decommission_capacity_reservation(Some(reservation)).expect("empty-snapshot growth must remain readable");
}
#[test]
fn decommission_capacity_grows_a_stale_nonzero_estimate_and_rolls_back_if_space_is_short() {
let now = OffsetDateTime::UNIX_EPOCH + Duration::minutes(1);
let layout = DecommissionErasureLayout { data: 3, parity: 1 };
let initial = vec![
DecommissionPoolCapacityInfo::for_test(0, layout, 0, 64, 2),
DecommissionPoolCapacityInfo::for_test(1, layout, 64, 64, 0),
];
let mut meta = PoolMeta {
version: POOL_META_VERSION,
pools: vec![decommission_test_pool_status(0, None), decommission_test_pool_status(1, None)],
..Default::default()
};
meta.decommission(0, initial[0].space).expect("activate source");
reserve_decommission_start_target_capacity(
&mut meta,
&[0],
&initial,
uuid::Uuid::new_v4(),
1,
now,
DECOMMISSION_CAPACITY_MODEL_VERSION,
)
.expect("reserve the stale estimate");
assert!(
grow_decommission_target_reservation(&mut meta, &initial, 0, 1, 16, uuid::Uuid::new_v4(), now)
.expect("a nonzero stale estimate may grow against real free space")
);
let before = serde_json::to_value(&meta).expect("snapshot reservation before rejected growth");
let shortage = vec![initial[0], DecommissionPoolCapacityInfo::for_test(1, layout, 43, 64, 21)];
let error = grow_decommission_target_reservation(&mut meta, &shortage, 0, 1, 32, uuid::Uuid::new_v4(), now)
.expect_err("real free-space shortage still rejects growth");
assert_eq!(super::decommission_capacity_stale_target_index(&error), Some(1));
assert_eq!(serde_json::to_value(&meta).expect("snapshot after rejected growth"), before);
}
#[test]
fn decommission_capacity_growth_keeps_rounded_and_mixed_layout_estimates_loadable() {
let now = OffsetDateTime::UNIX_EPOCH + Duration::minutes(1);
for source_layout in [
DecommissionErasureLayout { data: 3, parity: 1 },
DecommissionErasureLayout { data: 2, parity: 2 },
] {
let capacities = vec![
DecommissionPoolCapacityInfo::for_test(0, source_layout, 0, 8, 8),
DecommissionPoolCapacityInfo::for_test(1, DecommissionErasureLayout { data: 2, parity: 2 }, 16, 16, 0),
DecommissionPoolCapacityInfo::for_test(2, DecommissionErasureLayout { data: 3, parity: 1 }, 1024, 1024, 0),
];
let mut meta = PoolMeta {
version: POOL_META_VERSION,
pools: (0..3).map(|index| decommission_test_pool_status(index, None)).collect(),
..Default::default()
};
meta.decommission(0, capacities[0].space).unwrap();
reserve_decommission_start_target_capacity(
&mut meta,
&[0],
&capacities,
uuid::Uuid::new_v4(),
1,
now,
DECOMMISSION_CAPACITY_MODEL_VERSION,
)
.expect("initial reservation");
grow_decommission_target_reservation(&mut meta, &capacities, 0, 2, 16, uuid::Uuid::new_v4(), now)
.expect("grow against the target's free space");
let reservation = meta.pools[0]
.decommission
.as_ref()
.unwrap()
.capacity_reservation
.as_ref()
.unwrap();
validate_decommission_capacity_reservation(Some(reservation)).expect("growth must satisfy the persisted reader");
}
}
#[test]
+35 -4
View File
@@ -1616,7 +1616,7 @@ mod decommission_lock_order_tests {
run_routine_to_a_deferral_pause(
&store,
1,
"target pool 1 no longer fits the selected mutation: source pool 0 reservation estimate is exhausted",
"target pool 1 no longer fits the selected mutation: target pool 1 reservation requires",
)
.await;
});
@@ -1647,7 +1647,7 @@ mod decommission_lock_order_tests {
run_routine_to_a_deferral_pause(
&store,
1,
"target pool 1 no longer fits the selected mutation: source pool 0 reservation estimate is exhausted",
"target pool 1 no longer fits the selected mutation: target pool 1 reservation requires",
)
.await;
});
@@ -6062,6 +6062,12 @@ mod decommission_lock_order_tests {
)
.await
.expect("read the next source object for the multipart capacity oracle");
// Reconciled bytes remain charged. With no physical room for a new
// reservation, a distinct mutation must fail instead of reusing them.
let mut exhausted_capacity = capacity_snapshot();
exhausted_capacity[2] = DecommissionPoolCapacityInfo::for_test(2, layout, 0, target_total, target_total);
set_decommission_capacity_info_overrides_for_test(other_store.id, vec![exhausted_capacity]);
let next_result = tokio::time::timeout(
Duration::from_secs(30),
data_movement::migrate_decommission_object(
@@ -7827,6 +7833,12 @@ mod decommission_lock_order_tests {
.expect("same-mutation retry should publish a readable target");
assert_eq!(retry_info.size, i64::try_from(body.len()).expect("test body size fits i64"));
// Reconciled bytes remain charged. With no physical room for a new
// reservation, a distinct mutation must fail instead of reusing them.
let mut exhausted_capacity = capacity_snapshot();
exhausted_capacity[2] = DecommissionPoolCapacityInfo::for_test(2, layout, 0, target_total, target_total);
set_decommission_capacity_info_overrides_for_test(other_store.id, vec![exhausted_capacity]);
let next_object = "data-movement-put-after-retry.bin";
let next_version_id = uuid::Uuid::new_v4().to_string();
let mut next_opts = ObjectOptions {
@@ -7841,7 +7853,11 @@ mod decommission_lock_order_tests {
let next_result = other_store
.put_object_for_data_movement(RUSTFS_META_BUCKET, next_object, &mut next_data, &next_opts, None)
.await;
assert!(next_result.is_err(), "a different mutation must not reuse A's recovered capacity");
let next_error = match next_result {
Err(error) | Ok((_, Err(error))) => error,
Ok((_, Ok(_))) => panic!("a different mutation must not reuse A's recovered capacity"),
};
assert!(next_error.to_string().contains("decommission_capacity_blocked"), "{next_error}");
let next_target_err = other_store.pools[2]
.get_object_info(
RUSTFS_META_BUCKET,
@@ -9929,6 +9945,12 @@ mod decommission_lock_order_tests {
)
.await
.expect("read the next source object for the capacity oracle");
// Reconciled bytes remain charged. With no physical room for a new
// reservation, a distinct mutation must fail instead of reusing them.
let mut exhausted_capacity = capacity_snapshot();
exhausted_capacity[2] = DecommissionPoolCapacityInfo::for_test(2, layout, 0, target_total, target_total);
set_decommission_capacity_info_overrides_for_test(other_store.id, vec![exhausted_capacity]);
let next_result = tokio::time::timeout(
Duration::from_secs(30),
data_movement::migrate_decommission_object(
@@ -10190,11 +10212,20 @@ mod decommission_lock_order_tests {
assert_eq!(reconciled_reservation.pending_target_physical_bytes, 0);
assert_eq!(reconciled_reservation.consumed_target_physical_bytes, 1);
// Reconciled bytes remain charged. With no physical room for a new
// reservation, a distinct mutation must fail instead of reusing them.
let mut exhausted_capacity = capacity_snapshot();
exhausted_capacity[2] = DecommissionPoolCapacityInfo::for_test(2, layout, 1, 2, 1);
set_decommission_capacity_info_overrides_for_test(other_store.id, vec![exhausted_capacity]);
let next_after_reconcile = other_store
.delete_object(&bucket, next_object, next_opts)
.await
.expect_err("the consumed exact-fit capacity must reject the next marker");
assert!(next_after_reconcile.to_string().contains("decommission_capacity_blocked"));
assert!(
next_after_reconcile.to_string().contains("decommission_capacity_blocked"),
"{next_after_reconcile}"
);
let next_target_err = other_store.pools[2]
.get_object_info(
&bucket,
+414 -22
View File
@@ -34,7 +34,7 @@ pub use local_snapshot::{LocalUsageSnapshot, read_snapshot as read_local_snapsho
use rustfs_data_usage::{
BucketTargetUsageInfo, BucketUsageInfo, CompressionTotalInfo, DATA_USAGE_OBJECT_NAME, DATA_USAGE_OBSERVED_OBJECT_NAME,
DataUsageCache, DataUsageInfo, DiskUsageStatus, LEGACY_DATA_USAGE_OBJECT_NAME, SizeHistogram, VersionsHistogram,
observed_data_usage_is_newer,
observed_data_usage_is_newer, observed_data_usage_is_newer_than_identity,
};
use rustfs_io_metrics::record_system_path_failure;
use rustfs_utils::path::SLASH_SEPARATOR;
@@ -227,7 +227,10 @@ pub async fn lookup_degraded_bucket_usage_baseline(store: Arc<ECStore>, bucket:
/// endpoints for up to `DATA_USAGE_CACHE_TTL_SECS` between backend reads.
#[derive(Debug, Clone)]
struct CachedDataUsageSnapshot {
/// Last successful load, retained through failures for lineage comparison.
info: Option<DataUsageInfo>,
/// Controls serving independently from the retained ordering evidence.
last_load_failed: bool,
loaded_at: tokio::time::Instant,
/// Pre-discard per-bucket sizes from the same load, retained even when the
/// snapshot is incomplete and its bucket data is discarded. Consumed only
@@ -240,6 +243,7 @@ impl CachedDataUsageSnapshot {
fn result<T>(&self, project: impl FnOnce(&DataUsageInfo) -> T) -> Result<T, Error> {
self.info
.as_ref()
.filter(|_| !self.last_load_failed)
.map(project)
.ok_or_else(|| Error::other("data usage snapshot load recently failed"))
}
@@ -275,6 +279,17 @@ fn cache_data_usage_snapshot_result<T>(
mut degraded_baseline,
observed_unavailable,
}) => {
// A primary read can fall back to an older backup. Refresh the
// retry age without replacing a newer snapshot this process has
// already read; snapshot size itself is not an ordering signal.
if let Some(previous) = cache.as_mut()
&& let Some(cached_info) = previous.info.as_ref()
&& snapshot_predates(&info, cached_info)
{
previous.loaded_at = loaded_at;
previous.last_load_failed = false;
return Some(Ok(project(cached_info)));
}
// A flaky observed read must not shrink quota coverage for a TTL
// window: carry the previous refresh's baseline entries forward,
// letting the fresh (authoritative) values win where they exist.
@@ -286,21 +301,23 @@ fn cache_data_usage_snapshot_result<T>(
let projected = project(&info);
*cache = Some(CachedDataUsageSnapshot {
info: Some(info),
last_load_failed: false,
loaded_at,
degraded_baseline,
});
Ok(projected)
}
Err(e) => {
// Keep the previous baseline through a failed refresh: quota
// admission must not lose its last grounded values because one
// backend read errored.
let degraded_baseline = cache.take().map(|cached| cached.degraded_baseline).unwrap_or_default();
*cache = Some(CachedDataUsageSnapshot {
// A failed load is negatively cached, but cannot erase the last
// good snapshot's ordering proof or grounded quota baseline.
let cached = cache.get_or_insert_with(|| CachedDataUsageSnapshot {
info: None,
last_load_failed: true,
loaded_at,
degraded_baseline,
degraded_baseline: HashMap::new(),
});
cached.loaded_at = loaded_at;
cached.last_load_failed = true;
Err(e)
}
})
@@ -1245,7 +1262,7 @@ async fn load_data_usage_from_backend_with_baseline(store: Arc<ECStore>) -> Resu
let (loaded_snapshot, source) = load_data_usage_snapshot(store.clone()).await?;
// The observed-newness gate below compares against the snapshot as
// persisted, before normalization demotes or discards anything.
let authoritative_as_persisted = loaded_snapshot.clone();
let authoritative_as_persisted = loaded_snapshot.snapshot_identity();
let (info, mut degraded_baseline) = normalize_loaded_data_usage(loaded_snapshot, source.is_authoritative()).await;
// A bucket without a converged scanner cycle behind it — a freshly joined
@@ -1263,7 +1280,7 @@ async fn load_data_usage_from_backend_with_baseline(store: Arc<ECStore>) -> Resu
// read per snapshot refresh.
let mut observed_unavailable = false;
match load_observed_data_usage_snapshot(store).await {
Ok(Some(observed)) if observed_data_usage_is_newer(&observed, &authoritative_as_persisted) => {
Ok(Some(observed)) if observed_data_usage_is_newer_than_identity(&observed, authoritative_as_persisted) => {
backfill_degraded_baseline_from_observed(&mut degraded_baseline, &observed);
}
Ok(_) => {}
@@ -2367,6 +2384,9 @@ fn unknown_bucket_reload_due(cache: &Option<CachedDataUsageSnapshot>, bucket: &s
let Some(cached) = cache.as_ref() else {
return false;
};
if cached.last_load_failed {
return false;
}
let Some(info) = cached.info.as_ref() else {
return false;
};
@@ -2382,11 +2402,35 @@ fn unknown_bucket_reload_due(cache: &Option<CachedDataUsageSnapshot>, bucket: &s
.is_none_or(|attempted_at| now.duration_since(attempted_at) >= refresh_age)
}
/// Whether a freshly loaded snapshot is older than the cached one, as when a
/// primary read fails and the loader falls back to the previous `.bkp` copy.
fn snapshot_predates(loaded: Option<SystemTime>, cached: Option<SystemTime>) -> bool {
/// Compare authoritative lineage by the writer's epoch, cycle, then timestamp
/// order. Within one lineage, compare the observation's coordinated position
/// before its timestamp; a later observation cannot revive an older lineage.
fn snapshot_predates(loaded: &DataUsageInfo, cached: &DataUsageInfo) -> bool {
let loaded_baseline = loaded
.usage_snapshot_authoritative_baseline
.unwrap_or_else(|| loaded.snapshot_identity());
let cached_baseline = cached
.usage_snapshot_authoritative_baseline
.unwrap_or_else(|| cached.snapshot_identity());
let loaded_position = (loaded_baseline.scanner_epoch, loaded_baseline.scanner_cycle);
let cached_position = (cached_baseline.scanner_epoch, cached_baseline.scanner_cycle);
if loaded_position != cached_position {
return loaded_position < cached_position;
}
let (loaded, cached) = if loaded_baseline == cached_baseline {
let loaded_observation_position = loaded.scanner_epoch.zip(loaded.scanner_cycle);
let cached_observation_position = cached.scanner_epoch.zip(cached.scanner_cycle);
if loaded_observation_position != cached_observation_position {
return loaded_observation_position < cached_observation_position;
}
(loaded.last_update, cached.last_update)
} else {
(loaded_baseline.last_update, cached_baseline.last_update)
};
match (loaded, cached) {
(Some(loaded), Some(cached)) => loaded < cached,
(Some(loaded), Some(cached)) => {
loaded < cached && !rustfs_data_usage::usage_last_update_is_untrusted_future(cached, SystemTime::now())
}
(None, Some(_)) => true,
_ => false,
}
@@ -2427,22 +2471,15 @@ where
return false;
};
let mut cache = data_usage_snapshot_cache().write().await;
let cached_last_update = cache
.as_ref()
.and_then(|cached| cached.info.as_ref())
.and_then(|info| info.last_update);
if snapshot_predates(loaded.info.last_update, cached_last_update) {
return false;
}
cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded),
tokio::time::Instant::now(),
refresh_generation,
data_usage_snapshot_generation(),
|_| (),
|info| info.buckets_usage.contains_key(bucket),
)
.is_some()
.is_some_and(|result| matches!(result, Ok(true)))
}
// Calls of `reload_unknown_bucket_usage` per bucket. Unit tests do not run in
@@ -4224,6 +4261,7 @@ mod tests {
let degraded_baseline = info.bucket_sizes.clone();
*data_usage_snapshot_cache().write().await = Some(CachedDataUsageSnapshot {
info: Some(info),
last_load_failed: false,
loaded_at: tokio::time::Instant::now()
.checked_sub(age)
.expect("the monotonic clock must reach back by the requested age"),
@@ -4700,6 +4738,355 @@ mod tests {
assert_snapshot_bucket(&cached, "bucket");
}
#[tokio::test]
#[serial]
async fn expired_usage_snapshot_refresh_keeps_newer_counts_over_an_older_backup() {
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
use crate::config::com::save_config;
let (_dirs, store) = isolated_store_over_temp_disks().await;
let mut current = usage_snapshot_with_new_bucket(SystemTime::UNIX_EPOCH + Duration::from_secs(2), 100);
current.scanner_epoch = Some(4);
current.scanner_cycle = Some(11);
current.usage_snapshot_converged = Some(true);
let mut backup = data_usage_info_for_test("existing", 1, 5, SystemTime::UNIX_EPOCH + Duration::from_secs(1));
backup.scanner_epoch = Some(4);
backup.scanner_cycle = Some(10);
backup.usage_snapshot_converged = Some(true);
for (path, info) in [
(DATA_USAGE_OBJ_NAME_PATH.as_str(), &current),
(DATA_USAGE_OBJ_BACKUP_PATH.as_str(), &backup),
] {
save_config(store.clone(), path, serde_json::to_vec(info).expect("encode usage snapshot"))
.await
.expect("persist initial usage snapshot");
}
invalidate_data_usage_snapshot_cache().await;
for loaded in [
load_data_usage_from_backend_cached(store.clone()).await,
load_admin_data_usage_from_backend_cached(store.clone()).await,
] {
assert_eq!(loaded.expect("prime usage cache"), current);
}
// Bucket cleanup can reduce usage without advancing the scanner
// position. Only its publication timestamp advances.
let mut after_delete = data_usage_info_for_test("existing", 1, 3, SystemTime::UNIX_EPOCH + Duration::from_secs(3));
after_delete.scanner_epoch = current.scanner_epoch;
after_delete.scanner_cycle = current.scanner_cycle;
after_delete.usage_snapshot_converged = Some(true);
let mut stale_observation =
data_usage_info_for_test("retired-incarnation", 99, 999, SystemTime::UNIX_EPOCH + Duration::from_secs(4));
stale_observation.scanner_epoch = Some(4);
stale_observation.scanner_cycle = Some(12);
stale_observation.usage_snapshot_converged = Some(false);
stale_observation.usage_snapshot_authoritative_baseline = Some(backup.snapshot_identity());
assert!(observed_data_usage_is_newer(&stale_observation, &backup));
assert!(!observed_data_usage_is_newer(&stale_observation, &current));
save_config(
store.clone(),
DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&stale_observation).expect("encode observation of an older namespace"),
)
.await
.expect("persist an observation anchored to the old backup");
for (payload, expected) in [
(b"not a usage snapshot".to_vec(), &current),
(serde_json::to_vec(&after_delete).expect("encode post-delete usage"), &after_delete),
] {
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), payload)
.await
.expect("replace primary usage snapshot");
for cache in [data_usage_snapshot_cache(), admin_data_usage_snapshot_cache()] {
cache
.write()
.await
.as_mut()
.expect("usage cache should be populated")
.loaded_at = tokio::time::Instant::now() - Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
}
for loaded in [
load_data_usage_from_backend_cached(store.clone()).await,
load_admin_data_usage_from_backend_cached(store.clone()).await,
] {
assert_eq!(loaded.expect("refresh an expired usage cache"), *expected);
}
assert!(
!data_usage_snapshot_cache()
.read()
.await
.as_ref()
.expect("quota cache should remain populated")
.degraded_baseline
.contains_key("retired-incarnation"),
"a newer observation anchored to an older namespace must not backfill ghost usage"
);
}
invalidate_data_usage_snapshot_cache().await;
}
#[tokio::test]
#[serial]
async fn expired_admin_usage_refresh_orders_observations_by_generation_before_time() {
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
use crate::config::com::save_config;
let (_dirs, store) = isolated_store_over_temp_disks().await;
let mut authoritative = data_usage_info_for_test("bucket", 1, 10, SystemTime::UNIX_EPOCH + Duration::from_secs(100));
authoritative.scanner_epoch = Some(7);
authoritative.scanner_cycle = Some(10);
authoritative.usage_snapshot_converged = Some(true);
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&authoritative).expect("encode authoritative usage"),
)
.await
.expect("persist the common authoritative baseline");
let observation = |epoch, cycle, seconds, objects, size| {
let mut info =
data_usage_info_for_test("bucket", objects, size, SystemTime::UNIX_EPOCH + Duration::from_secs(seconds));
info.scanner_epoch = Some(epoch);
info.scanner_cycle = Some(cycle);
info.usage_snapshot_converged = Some(false);
info.usage_snapshot_authoritative_baseline = Some(authoritative.snapshot_identity());
info
};
let first = observation(7, 11, 200, 11, 1_100);
let newer_cycle = observation(7, 12, 150, 12, 1_200);
let older_cycle = observation(7, 11, 300, 99, 9_900);
let newer_epoch = observation(8, 1, 50, 1, 100);
invalidate_data_usage_snapshot_cache().await;
for (observed, expected) in [
(&first, &first),
(&newer_cycle, &newer_cycle),
(&older_cycle, &newer_cycle),
(&newer_epoch, &newer_epoch),
] {
assert!(
observed_data_usage_is_newer(observed, &authoritative),
"the persisted observation must pass the real loader's selection gate"
);
save_config(
store.clone(),
DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(observed).expect("encode scanner observation"),
)
.await
.expect("persist scanner observation");
if let Some(cached) = admin_data_usage_snapshot_cache().write().await.as_mut() {
cached.loaded_at = tokio::time::Instant::now() - Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
}
let loaded = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("refresh admin usage through the real backend loader");
assert_eq!(
&loaded,
expected,
"same-baseline observation ordering must follow epoch/cycle before wall clock: incoming={:?}",
observed.snapshot_identity()
);
}
invalidate_data_usage_snapshot_cache().await;
}
#[test]
fn expired_usage_snapshot_refresh_accepts_new_authority_over_a_later_old_observation() {
let loaded_at = tokio::time::Instant::now();
let baseline = data_usage_info_for_test("bucket", 3, 42, SystemTime::UNIX_EPOCH + Duration::from_secs(1));
let mut observed = data_usage_info_for_test("bucket", 5, 60, SystemTime::UNIX_EPOCH + Duration::from_secs(4));
observed.usage_snapshot_converged = Some(false);
observed.usage_snapshot_authoritative_baseline = Some(baseline.snapshot_identity());
let mut cache = Some(CachedDataUsageSnapshot {
info: Some(observed),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::new(),
});
let after_delete = data_usage_info_for_test("bucket", 1, 7, SystemTime::UNIX_EPOCH + Duration::from_secs(3));
let refreshed = cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded_usage_for_test(after_delete.clone())),
loaded_at,
1,
1,
DataUsageInfo::clone,
)
.expect("matching refresh generation")
.expect("a newer authority must retire observations of its predecessor");
assert_eq!(refreshed, after_delete);
}
#[test]
fn expired_usage_snapshot_refresh_orders_lineage_by_epoch_cycle_then_time() {
let loaded_at = tokio::time::Instant::now();
for (cached_position, loaded_position, loaded_time, accept) in [
((Some(2), Some(10)), (Some(1), Some(99)), 110, false),
((Some(2), Some(10)), (Some(2), Some(9)), 110, false),
((Some(2), Some(10)), (Some(3), Some(1)), 90, true),
((Some(2), Some(10)), (Some(2), Some(11)), 90, true),
((Some(2), Some(10)), (Some(2), Some(10)), 110, true),
((Some(3), None), (Some(2), Some(10)), 110, false),
((Some(2), Some(10)), (Some(3), None), 90, true),
((None, None), (None, None), 110, true),
((None, None), (None, None), 90, false),
] {
let mut previous = data_usage_info_for_test("bucket", 3, 42, SystemTime::UNIX_EPOCH + Duration::from_secs(100));
(previous.scanner_epoch, previous.scanner_cycle) = cached_position;
let mut loaded = data_usage_info_for_test("bucket", 1, 7, SystemTime::UNIX_EPOCH + Duration::from_secs(loaded_time));
(loaded.scanner_epoch, loaded.scanner_cycle) = loaded_position;
let expected = if accept { &loaded } else { &previous };
let mut cache = Some(CachedDataUsageSnapshot {
info: Some(previous.clone()),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::new(),
});
let refreshed = cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded_usage_for_test(loaded.clone())),
loaded_at,
1,
1,
DataUsageInfo::clone,
)
.expect("matching refresh generation")
.expect("a valid usage snapshot remains available");
assert_eq!(
&refreshed, expected,
"lineage ordering must match scanner publication: cached={cached_position:?}, loaded={loaded_position:?}, loaded_time={loaded_time}"
);
}
}
#[test]
fn expired_usage_snapshot_refresh_preserves_rejection_backoff_and_baseline() {
let loaded_at = tokio::time::Instant::now();
let expected = data_usage_info_for_test("bucket", 3, 42, SystemTime::UNIX_EPOCH + Duration::from_secs(2));
let mut cache = Some(CachedDataUsageSnapshot {
info: Some(expected.clone()),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::from([("bucket".to_string(), 42)]),
});
let refreshed_at = loaded_at + Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
let refreshed = cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded_usage_for_test(data_usage_info_for_test("bucket", 1, 7, SystemTime::UNIX_EPOCH))),
refreshed_at,
1,
1,
DataUsageInfo::clone,
)
.expect("matching refresh generation")
.expect("the newer cached snapshot remains available");
assert_eq!(refreshed, expected);
let retained = cache.as_ref().expect("cache should remain populated");
assert_eq!(retained.degraded_baseline.get("bucket"), Some(&42));
assert_eq!(retained.loaded_at, refreshed_at, "an older fallback must retain the TTL backoff");
let result =
cache_data_usage_snapshot_result(&mut cache, Err(Error::ErasureReadQuorum), refreshed_at, 1, 1, DataUsageInfo::clone)
.expect("matching refresh generation");
assert!(matches!(result, Err(Error::ErasureReadQuorum)));
let failed = cache.as_ref().expect("negative cache should be populated");
assert!(failed.last_load_failed);
assert_eq!(failed.info.as_ref(), Some(&expected), "failure must retain the last good ordering proof");
}
#[test]
fn expired_usage_snapshot_refresh_accepts_clock_recovery() {
let loaded_at = tokio::time::Instant::now();
let now = SystemTime::now();
let future = now + rustfs_data_usage::USAGE_LAST_UPDATE_FUTURE_TOLERANCE + Duration::from_secs(60);
let mut cache = Some(CachedDataUsageSnapshot {
info: Some(data_usage_info_for_test("bucket", 3, 42, future)),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::new(),
});
let recovered = data_usage_info_for_test("bucket", 1, 7, now);
let refreshed = cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded_usage_for_test(recovered.clone())),
loaded_at,
1,
1,
DataUsageInfo::clone,
)
.expect("matching refresh generation")
.expect("a clock recovery snapshot must not be frozen behind an untrusted timestamp");
assert_eq!(refreshed, recovered);
}
#[test]
fn expired_usage_snapshot_refresh_keeps_last_good_lineage_across_a_failed_load() {
let loaded_at = tokio::time::Instant::now();
let current = data_usage_info_for_test("bucket", 3, 42, SystemTime::UNIX_EPOCH + Duration::from_secs(2));
let mut cache = None;
let first = cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded_usage_for_test(current.clone())),
loaded_at,
1,
1,
DataUsageInfo::clone,
)
.expect("matching refresh generation")
.expect("prime the last successful snapshot");
assert_eq!(first, current);
let failed_at = loaded_at + Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
let failure =
cache_data_usage_snapshot_result(&mut cache, Err(Error::ErasureReadQuorum), failed_at, 1, 1, DataUsageInfo::clone)
.expect("matching refresh generation");
assert!(matches!(failure, Err(Error::ErasureReadQuorum)));
assert!(
fresh_cached_data_usage_snapshot(
&cache,
failed_at + Duration::from_secs(DATA_USAGE_UNKNOWN_BUCKET_REFRESH_SECS),
Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS),
|_| panic!("a failed load must not serve its retained snapshot as a success"),
)
.expect("the failed load remains inside its negative TTL")
.is_err()
);
assert!(
!unknown_bucket_reload_due(
&cache,
"unknown",
failed_at + Duration::from_secs(DATA_USAGE_UNKNOWN_BUCKET_REFRESH_SECS),
),
"an unknown bucket must not bypass the failed load's TTL"
);
let recovered_at = failed_at + Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
let recovered = cache_data_usage_snapshot_result(
&mut cache,
Ok(loaded_usage_for_test(data_usage_info_for_test("bucket", 1, 7, SystemTime::UNIX_EPOCH))),
recovered_at,
1,
1,
DataUsageInfo::clone,
)
.expect("matching refresh generation")
.expect("a readable older backup must preserve the last good lineage");
assert_eq!(recovered, current);
assert_eq!(
cache.as_ref().expect("recovered cache").degraded_baseline.get("bucket"),
Some(&42),
"a failed load must not erase the baseline's ordering proof"
);
assert_eq!(
fresh_cached_data_usage_snapshot(&cache, recovered_at, Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS), |info| info
.objects_total_size,)
.expect("the recovered load starts a fresh TTL")
.expect("the last successful snapshot is available again"),
42
);
}
#[test]
#[serial]
fn unavailable_observed_read_keeps_previous_baseline_coverage() {
@@ -4707,6 +5094,7 @@ mod tests {
let refresh_generation = data_usage_snapshot_generation();
let mut cache = Some(CachedDataUsageSnapshot {
info: Some(data_usage_info_for_test("bucket", 1, 42, SystemTime::UNIX_EPOCH)),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::from([("observed-only".to_string(), 7_u64), ("covered".to_string(), 1)]),
});
@@ -4740,6 +5128,7 @@ mod tests {
let refresh_generation = data_usage_snapshot_generation();
let mut cache = Some(CachedDataUsageSnapshot {
info: Some(data_usage_info_for_test("stale", 1, 42, SystemTime::UNIX_EPOCH)),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::new(),
});
@@ -5928,6 +6317,7 @@ mod tests {
let mut snapshot_cache = data_usage_snapshot_cache().write().await;
*snapshot_cache = Some(CachedDataUsageSnapshot {
info: Some(successor),
last_load_failed: false,
loaded_at: tokio::time::Instant::now(),
degraded_baseline: HashMap::new(),
});
@@ -5999,6 +6389,7 @@ mod tests {
let mut snapshot_cache = data_usage_snapshot_cache().write().await;
*snapshot_cache = Some(CachedDataUsageSnapshot {
info: Some(successor),
last_load_failed: false,
loaded_at: tokio::time::Instant::now(),
degraded_baseline: HashMap::new(),
});
@@ -6051,6 +6442,7 @@ mod tests {
let stale = data_usage_info_for_test(BUCKET, 7, 294, SystemTime::now());
*data_usage_snapshot_cache().write().await = Some(CachedDataUsageSnapshot {
info: Some(stale),
last_load_failed: false,
loaded_at: tokio::time::Instant::now(),
degraded_baseline: HashMap::new(),
});
@@ -21,6 +21,7 @@ fn scalar_projection_borrows_snapshot_and_preserves_unknown_and_zero() {
info.buckets_usage.insert("empty".to_string(), BucketUsageInfo::default());
let cache = Some(CachedDataUsageSnapshot {
info: Some(info),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::new(),
});
@@ -66,6 +67,7 @@ fn projection_is_skipped_for_missing_failed_expired_or_invalidated_snapshots() {
cache = Some(CachedDataUsageSnapshot {
info: Some(data_usage_info_for_test("known", 1, 42, SystemTime::UNIX_EPOCH)),
last_load_failed: false,
loaded_at,
degraded_baseline: HashMap::new(),
});
@@ -127,6 +129,33 @@ fn projected_publication_moves_bucket_maps_into_cache() {
cached.buckets_usage.get_key_value("known").expect("cached bucket").0.as_ptr(),
original_key
);
let failure = cache_data_usage_snapshot_result(
&mut cache,
Err(Error::ErasureReadQuorum),
tokio::time::Instant::now(),
1,
1,
|_: &DataUsageInfo| -> u64 { panic!("a failed load must not project its retained snapshot") },
)
.expect("matching refresh generation");
assert!(matches!(failure, Err(Error::ErasureReadQuorum)));
let retained = cache
.as_ref()
.expect("negative cache")
.info
.as_ref()
.expect("last good ordering proof");
assert_eq!(
retained
.buckets_usage
.get_key_value("known")
.expect("retained bucket")
.0
.as_ptr(),
original_key,
"a failed load must retain the snapshot without copying its bucket maps"
);
}
#[test]
@@ -682,15 +682,11 @@ async fn get_pools_info(all_disks: &[Disk]) -> Result<HashMap<i32, HashMap<i32,
return Err(Error::other("ServerNotInitialized"));
};
let mut pools_info: HashMap<i32, HashMap<i32, ErasureSetInfo>> = HashMap::new();
for d in all_disks {
let pool_info = pools_info.entry(d.pool_index).or_default();
let erasure_set = pool_info.entry(d.set_index).or_default();
if erasure_set.id == 0 {
erasure_set.id = d.set_index;
collect_pools_info(all_disks, |pool_index, set_index| {
let store = store.clone();
async move {
let load = load_data_usage_cache(
store.pools[d.pool_index as usize].disk_set[d.set_index as usize].as_ref(),
store.pools[pool_index as usize].disk_set[set_index as usize].as_ref(),
DATA_USAGE_CACHE_NAME,
);
#[cfg(feature = "diagnostic-read-version-caller")]
@@ -698,11 +694,34 @@ async fn get_pools_info(all_disks: &[Disk]) -> Result<HashMap<i32, HashMap<i32,
rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ServerInfoPools,
load,
);
match load.await {
Ok(cache) => apply_erasure_set_usage(&cache, erasure_set),
Err(_) => erasure_set.usage_error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string()),
}
load.await
}
})
.await
}
async fn collect_pools_info<F, Fut>(all_disks: &[Disk], mut load_usage: F) -> Result<HashMap<i32, HashMap<i32, ErasureSetInfo>>>
where
F: FnMut(i32, i32) -> Fut,
Fut: std::future::Future<Output = Result<rustfs_data_usage::DataUsageCache>>,
{
let mut pools_info: HashMap<i32, HashMap<i32, ErasureSetInfo>> = HashMap::new();
for d in all_disks {
let pool_info = pools_info.entry(d.pool_index).or_default();
let erasure_set = match pool_info.entry(d.set_index) {
std::collections::hash_map::Entry::Occupied(entry) => entry.into_mut(),
std::collections::hash_map::Entry::Vacant(entry) => {
let erasure_set = entry.insert(ErasureSetInfo {
id: d.set_index,
..Default::default()
});
match load_usage(d.pool_index, d.set_index).await {
Ok(cache) => apply_erasure_set_usage(&cache, erasure_set),
Err(_) => erasure_set.usage_error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string()),
}
erasure_set
}
};
erasure_set.raw_capacity += d.total_space;
erasure_set.raw_usage += d.used_space;
@@ -1241,6 +1260,59 @@ mod tests {
assert_eq!(info.deployment_id, expected_deployment_id);
}
#[tokio::test]
async fn pool_usage_is_loaded_once_per_set_including_set_zero() {
let mut disks = Vec::new();
for (pool, set, count) in [(0, 0, 16), (1, 0, 16), (1, 1, 4)] {
disks.extend((0..count).map(|_| Disk {
pool_index: pool,
set_index: set,
used_space: 10,
total_space: 100,
..Default::default()
}));
}
let mut loads = Vec::new();
let info = super::collect_pools_info(&disks, |pool, set| {
loads.push((pool, set));
async move {
if pool == 1 && set == 0 {
return Err(crate::error::Error::other("usage unavailable"));
}
let mut cache = rustfs_data_usage::DataUsageCache::default();
cache.info.name = DATA_USAGE_ROOT.to_owned();
cache.info.last_update = Some(std::time::SystemTime::UNIX_EPOCH);
cache.info.snapshot_complete = true;
cache.replace(
DATA_USAGE_ROOT,
"",
rustfs_data_usage::DataUsageEntry {
size: 25,
objects: 2,
..Default::default()
},
);
Ok(cache)
}
})
.await
.expect("collect mixed successful and unavailable usage");
assert_eq!(loads, [(0, 0), (1, 0), (1, 1)]);
for (pool, set, count) in [(0, 0, 16), (1, 0, 16), (1, 1, 4)] {
let set_info = &info[&pool][&set];
assert_eq!(set_info.id, set);
assert_eq!(set_info.raw_usage, count * 10);
assert_eq!(set_info.raw_capacity, count * 100);
if pool == 1 && set == 0 {
assert_eq!(set_info.usage_error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
} else {
assert_eq!(set_info.usage, 25, "logical usage must not multiply by drive count");
assert_eq!(set_info.objects_count, 2);
assert!(set_info.usage_error.is_none());
}
}
}
#[serial]
#[tokio::test]
async fn local_server_property_includes_runtime_stats_without_endpoint_pools() {
@@ -167,6 +167,16 @@ impl ObjectLockAttempt {
}
}
}
/// Correlate a raw guard's acquisition with its peer identities without
/// claiming ownership of the guard's lifetime. Peer release events retain
/// the identities needed to reconstruct when the actual owner releases it.
pub(crate) fn observe_acquisition(&mut self, result: &Result<NamespaceLockGuard, LockError>) {
self.observe(result);
if let Some(context) = self.0.as_mut() {
context.finished = true;
}
}
}
fn duration_ms(duration: Duration) -> u64 {
@@ -500,6 +510,45 @@ pub(crate) mod tests {
)
}
#[test]
#[serial_test::serial]
fn raw_guard_acquisition_diagnostic_does_not_claim_guard_lifetime() {
temp_env::with_vars([(rustfs_config::ENV_OBJECT_LOCK_DIAG_ENABLE, Some("true"))], || {
let capture = CapturedLockEvents::default();
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("raw guard diagnostic runtime");
tracing::subscriber::with_default(capture.subscriber(), || {
runtime.block_on(async {
let lock = namespace();
let mut attempt = ObjectLockAttempt::start(
"bucket_incarnation_disk_read",
"bucket",
"transaction.lock",
None,
&lock,
"read",
Duration::from_secs(1),
);
let result = lock.get_read_lock(Duration::from_secs(1)).await;
attempt.observe_acquisition(&result);
drop(attempt);
let guard = result.expect("retain the raw guard after diagnostic completion");
assert!(!guard.is_lock_lost());
let states = capture
.rows()
.iter()
.map(|row| row["fields"]["state"].as_str().expect("diagnostic state string").to_owned())
.collect::<Vec<_>>();
assert_eq!(states, ["acquiring", "acquired"]);
drop(guard);
assert_eq!(capture.rows().len(), 2, "the acquisition observer must not invent a guard release");
});
});
});
}
#[test]
#[serial_test::serial]
fn object_lock_diagnostics_disabled_does_not_allocate_context_or_emit() {
+31 -20
View File
@@ -148,47 +148,58 @@ pub async fn make_dir_all(path: impl AsRef<Path>) -> io::Result<()> {
native_io(move || create_dir_all_std(&path)).await
}
fn is_dir_error(e: &io::Error) -> bool {
fn is_dir_error(path: &Path, e: &io::Error) -> bool {
e.raw_os_error() == Some(libc::EISDIR)
|| e.kind() == io::ErrorKind::IsADirectory
// macOS: remove_file on a directory returns EPERM
|| (cfg!(target_os = "macos") && e.raw_os_error() == Some(libc::EPERM))
|| is_windows_dir_access_denied(path, e)
}
/// Windows reports `remove_file` on a directory (or directory symlink) as
/// ERROR_ACCESS_DENIED, which is indistinguishable from a real permission
/// failure without inspecting the entry itself.
#[cfg(windows)]
fn is_windows_dir_access_denied(path: &Path, e: &io::Error) -> bool {
use std::os::windows::fs::FileTypeExt;
e.kind() == io::ErrorKind::PermissionDenied
&& std::fs::symlink_metadata(path).is_ok_and(|metadata| {
let file_type = metadata.file_type();
file_type.is_dir() || file_type.is_symlink_dir()
})
}
#[cfg(not(windows))]
fn is_windows_dir_access_denied(_path: &Path, _e: &io::Error) -> bool {
false
}
#[tracing::instrument(level = "debug", skip_all)]
pub async fn remove(path: impl AsRef<Path>) -> io::Result<()> {
// Try remove_file first; fall back to remove_dir if it's a directory
let file_path = path.as_ref().to_path_buf();
match native_io(move || std::fs::remove_file(file_path)).await {
Ok(()) => Ok(()),
Err(e) if is_dir_error(&e) => {
let path = path.as_ref().to_path_buf();
native_io(move || std::fs::remove_dir(path)).await
}
Err(e) => Err(e),
}
let path = path.as_ref().to_path_buf();
native_io(move || remove_std(path)).await
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub async fn remove_all(path: impl AsRef<Path>) -> io::Result<()> {
// Try remove_file first; fall back to remove_dir_all if it's a directory
let file_path = path.as_ref().to_path_buf();
match native_io(move || std::fs::remove_file(file_path)).await {
let path = path.as_ref().to_path_buf();
native_io(move || match std::fs::remove_file(&path) {
Ok(()) => Ok(()),
Err(e) if is_dir_error(&e) => {
let path = path.as_ref().to_path_buf();
native_io(move || std::fs::remove_dir_all(path)).await
}
Err(e) if is_dir_error(&path, &e) => std::fs::remove_dir_all(&path),
Err(e) => Err(e),
}
})
.await
}
#[tracing::instrument(level = "debug", skip_all)]
pub fn remove_std(path: impl AsRef<Path>) -> io::Result<()> {
// Try remove_file first; fall back to remove_dir if it's a directory
match std::fs::remove_file(path.as_ref()) {
let path = path.as_ref();
match std::fs::remove_file(path) {
Ok(()) => Ok(()),
Err(e) if is_dir_error(&e) => std::fs::remove_dir(path.as_ref()),
Err(e) if is_dir_error(path, &e) => std::fs::remove_dir(path),
Err(e) => Err(e),
}
}
+250 -86
View File
@@ -562,12 +562,57 @@ fn remove_dir_all_if_exists(path: &Path) -> std::io::Result<()> {
fn snapshot_part_transaction_file(src: &Path, backup: &Path, absent: &Path) -> std::io::Result<bool> {
match std::fs::symlink_metadata(src) {
Ok(metadata) if metadata.is_file() => std::fs::hard_link(src, backup).map(|()| true),
Ok(metadata) if metadata.file_type().is_symlink() => {
Err(std::io::Error::new(ErrorKind::InvalidInput, "multipart transaction source is a symlink"))
}
Ok(_) => Err(std::io::Error::new(ErrorKind::InvalidData, "multipart transaction source is not a file")),
Err(err) if err.kind() == ErrorKind::NotFound => std::fs::write(absent, []).map(|()| false),
Err(err) => Err(err),
}
}
struct PreparedPartTxnReceipt {
src_volume_dir: PathBuf,
dst_volume_dir: PathBuf,
src_file_path: PathBuf,
dst_file_path: PathBuf,
dst_meta_path: PathBuf,
transaction_path: PathBuf,
meta: Bytes,
had_old_meta: bool,
}
struct PublishedPartTxnReceipt {
current_data_path: PathBuf,
current_meta_path: PathBuf,
transaction_path: PathBuf,
had_old_meta: bool,
}
const PART_TXN_RECEIPTS_PER_SHARD: usize = 64;
static PREPARED_PART_TXN_RECEIPTS: std::sync::LazyLock<os::ShardedPathCache<PreparedPartTxnReceipt>> =
std::sync::LazyLock::new(|| os::ShardedPathCache::new(PART_TXN_RECEIPTS_PER_SHARD));
static PUBLISHED_PART_TXN_RECEIPTS: std::sync::LazyLock<os::ShardedPathCache<PublishedPartTxnReceipt>> =
std::sync::LazyLock::new(|| os::ShardedPathCache::new(PART_TXN_RECEIPTS_PER_SHARD));
fn put_prepared_part_txn_receipt(transaction_path: &Path, entry: PreparedPartTxnReceipt) {
PREPARED_PART_TXN_RECEIPTS.insert(transaction_path, entry);
}
fn take_prepared_part_txn_receipt(transaction_path: &Path) -> Option<PreparedPartTxnReceipt> {
PREPARED_PART_TXN_RECEIPTS.remove(transaction_path)
}
fn put_published_part_txn_receipt(transaction_path: &Path, entry: PublishedPartTxnReceipt) {
PUBLISHED_PART_TXN_RECEIPTS.insert(transaction_path, entry);
}
fn take_published_part_txn_receipt(transaction_path: &Path) -> Option<PublishedPartTxnReceipt> {
PUBLISHED_PART_TXN_RECEIPTS.remove(transaction_path)
}
fn restore_part_transaction_file(current: &Path, backup: &Path, absent: &Path, restore: &Path) -> std::io::Result<()> {
match std::fs::symlink_metadata(backup) {
Ok(metadata) if metadata.is_file() => {
@@ -3160,6 +3205,10 @@ pub(super) fn has_rename_data_test_failpoints(dst_path: &str) -> bool {
|| os::has_dir_fsync_test_failures()
|| os::file_sync_probe::is_active()
|| os::prepared_publication_test_hooks::has_any_hooks()
|| !OWNED_FILE_BEFORE_OPEN
.lock()
.expect("test file open hook lock should not be poisoned")
.is_empty()
}
#[cfg(all(not(test), feature = "test-util", target_os = "linux"))]
@@ -9142,11 +9191,15 @@ impl LocalDisk {
opts: DeleteOptions,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Vec<Option<DiskError>> {
let payload = super::GuardedDeleteVersionsPayload { versions, opts };
let payload = super::GuardedDeleteVersionsPayload {
versions,
opts,
bounded_recovery_protocol: None,
};
if let Err(error) = payload.validate() {
return vec![Some(error); payload.versions.len()];
}
let super::GuardedDeleteVersionsPayload { versions, mut opts } = payload;
let super::GuardedDeleteVersionsPayload { versions, mut opts, .. } = payload;
opts.recoverable_delete = true;
let mut errors = Vec::with_capacity(versions.len());
for version in &versions {
@@ -9216,15 +9269,14 @@ impl LocalDisk {
.await;
}
let volume_dir = self.io_get_bucket_path(volume)?;
let file_path = self.io_get_object_path(volume, path)?;
let (volume_dir, file_path) = self.io_get_bucket_and_object_path(volume, path)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
let xl_path = path_join(&[file_path.as_path(), Path::new(STORAGE_FORMAT_FILE)]);
let object_path = self.get_object_path(volume, path)?;
let mut namespace_owner: Option<Arc<dyn Send + Sync>> =
Some(os::acquire_metadata_mutation_lease(&self.get_object_path(volume, path)?, namespace_owner).await);
Some(os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await);
if opts.recoverable_delete || (opts.undo_write && opts.undo_delete) {
let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await;
namespace_owner = Some(Arc::new((namespace_owner, volume_guard)));
@@ -9249,7 +9301,15 @@ impl LocalDisk {
let rollback_dir = opts.old_data_dir;
if let Some(rollback_dir) = rollback_dir
&& let Some(()) = self
.try_fused_recoverable_delete_version(volume, path, &fi, &opts, rollback_dir, namespace_owner.clone())
.try_fused_recoverable_delete_version_with_paths(
volume,
path,
&fi,
&opts,
rollback_dir,
namespace_owner.clone(),
Some((object_path, volume_dir.clone(), file_path.clone())),
)
.await?
{
return Ok(());
@@ -9989,7 +10049,7 @@ impl LocalDisk {
if !std::fs::symlink_metadata(&receipt_path)?.is_file() {
return Err(std::io::Error::new(ErrorKind::InvalidData, "delete receipt is not a regular file"));
}
std::fs::File::open(receipt_path)?
os::open_file_for_sync_std(&receipt_path)?
}
Err(err) => return Err(err),
};
@@ -14314,7 +14374,8 @@ impl DiskAPI for LocalDisk {
all_cached = false;
break;
};
let Ok(source) = self.io_get_object_path(bucket, multipart_staging::upload_scope_root(path_str, true)) else {
let Ok(source) = self.io_get_object_open_path(bucket, multipart_staging::upload_scope_root(path_str, true))
else {
all_cached = false;
break;
};
@@ -14441,8 +14502,8 @@ impl DiskAPI for LocalDisk {
multipart_staging::reject_noncanonical_scope(&self.root, self.io_root(), src_volume, src_path)?;
let source_owner = self.multipart_source_owner(dst_volume, dst_path, true, false, None).await?;
let admission = self.multipart_source_admission(dst_volume, dst_path, true, false)?;
let src_volume_dir = self.io_get_bucket_path(src_volume)?;
let dst_volume_dir = self.io_get_bucket_path(dst_volume)?;
let (src_volume_dir, src_file_path) = self.io_get_bucket_and_object_path(src_volume, src_path)?;
let (dst_volume_dir, dst_file_path) = self.io_get_bucket_and_object_path(dst_volume, dst_path)?;
if !skip_access_checks(src_volume) {
super::fs::access_std(&src_volume_dir).map_err(|err| to_access_error(err, DiskError::VolumeAccessDenied))?;
}
@@ -14450,10 +14511,8 @@ impl DiskAPI for LocalDisk {
super::fs::access_std(&dst_volume_dir).map_err(|err| to_access_error(err, DiskError::VolumeAccessDenied))?;
}
let src_file_path = self.io_get_object_path(src_volume, src_path)?;
let dst_file_path = self.io_get_object_path(dst_volume, dst_path)?;
let dst_meta_path = self.io_get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
let transaction_path = self.io_get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?;
let dst_meta_path = self.io_get_object_open_path(dst_volume, &format!("{dst_path}.meta"))?;
let transaction_path = self.io_get_object_open_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?;
for path in [&src_file_path, &dst_file_path, &dst_meta_path, &transaction_path] {
check_path_length(path.to_string_lossy().as_ref())?;
}
@@ -14498,8 +14557,11 @@ impl DiskAPI for LocalDisk {
if !source.is_file() {
return Ok(Err(DiskError::FileAccessDenied));
}
if transaction_path.exists() {
return Ok(Err(DiskError::FileAccessDenied));
match std::fs::symlink_metadata(&transaction_path) {
Err(err) if err.kind() == ErrorKind::NotFound => {}
Ok(m) if m.file_type().is_symlink() => return Ok(Err(DiskError::InvalidPath)),
Ok(_) => return Ok(Err(DiskError::FileAccessDenied)),
Err(err) => return Ok(Err(to_file_error(err).into())),
}
if let Some((integrity, source, destination, expected_len)) = integrity_prep {
let stat = match std::fs::symlink_metadata(&source) {
@@ -14510,7 +14572,7 @@ impl DiskAPI for LocalDisk {
return Ok(Err(DiskError::FileCorrupt));
}
let verify_and_sync = (|| -> std::io::Result<bool> {
let mut file = std::fs::File::open(&source)?;
let mut file = os::open_file_for_sync_std(&source)?;
let mut header = [0u8; rustfs_filemeta::shard_integrity::INDEX_HEADER_SIZE];
std::io::Read::read_exact(&mut file, &mut header)?;
if header != integrity.index_header() {
@@ -14556,7 +14618,7 @@ impl DiskAPI for LocalDisk {
return Ok(Err(to_file_error(err).into()));
}
let prepare_result = (|| -> std::io::Result<()> {
let prepare_result = (|| -> std::io::Result<bool> {
let had_old_data = snapshot_part_transaction_file(
&dst_file_path,
&staging_path.join(PART_TRANSACTION_OLD_DATA),
@@ -14568,10 +14630,11 @@ impl DiskAPI for LocalDisk {
&staging_path.join(PART_TRANSACTION_OLD_META_ABSENT),
)?;
let new_meta_path = staging_path.join(PART_TRANSACTION_NEW_META);
let mut new_meta = std::fs::OpenOptions::new()
.create_new(true)
.write(true)
.open(staging_path.join(PART_TRANSACTION_NEW_META))?;
.open(&new_meta_path)?;
std::io::Write::write_all(&mut new_meta, &meta)?;
if durability.syncs_commit_metadata() {
new_meta.sync_data()?;
@@ -14579,6 +14642,7 @@ impl DiskAPI for LocalDisk {
// Windows rejects renaming a directory while one of its children is
// still open, even when the child handle shares delete access.
drop(new_meta);
let _ = std::fs::hard_link(&new_meta_path, staging_path.join(PART_TRANSACTION_PUBLISH_META));
let sync_overwrite_backup = durability.syncs_commit_metadata() && (had_old_data || had_old_meta);
if sync_overwrite_backup {
os::fsync_dir_std(&staging_path)?;
@@ -14587,13 +14651,29 @@ impl DiskAPI for LocalDisk {
if sync_overwrite_backup {
os::fsync_dir_std(transaction_parent)?;
}
Ok(())
Ok(had_old_meta)
})();
if let Err(err) = prepare_result {
let _ = remove_dir_all_if_exists(&staging_path);
return Ok(Err(to_file_error(err).into()));
}
let had_old_meta = match prepare_result {
Ok(had_old_meta) => had_old_meta,
Err(err) => {
let _ = remove_dir_all_if_exists(&staging_path);
return Ok(Err(to_file_error(err).into()));
}
};
put_prepared_part_txn_receipt(
&transaction_path,
PreparedPartTxnReceipt {
src_volume_dir,
dst_volume_dir,
src_file_path,
dst_file_path,
dst_meta_path,
transaction_path: transaction_path.clone(),
meta,
had_old_meta,
},
);
Ok(Ok(()))
})
@@ -14612,13 +14692,28 @@ impl DiskAPI for LocalDisk {
let source_owner = self.multipart_source_owner(volume, path, true, false, None).await?;
let admission = self.multipart_source_admission(volume, path, true, false)?;
let admission_source = admission.as_ref().map(|a| a.source().to_path_buf());
self.io_get_bucket_path(volume)?;
let current_data_path = self.io_get_object_path(volume, path)?;
let current_meta_path = self.io_get_object_path(volume, &format!("{path}.meta"))?;
let transaction_path = self.io_get_object_path(volume, &crate::disk::part_transaction_path(path))?;
for candidate in [&current_data_path, &current_meta_path, &transaction_path] {
check_path_length(candidate.to_string_lossy().as_ref())?;
}
let txn_rel_path = crate::disk::part_transaction_path(path);
let cached_published = self
.io_get_object_open_path(volume, &txn_rel_path)
.ok()
.and_then(|txn_open| take_published_part_txn_receipt(&txn_open));
let (current_data_path, current_meta_path, transaction_path, skip_obsolete_read_on_commit) = match cached_published {
Some(entry) => (
entry.current_data_path,
entry.current_meta_path,
entry.transaction_path,
!entry.had_old_meta,
),
None => {
let (_, current_data_path) = self.io_get_bucket_and_object_path(volume, path)?;
let current_meta_path = self.io_get_object_path(volume, &format!("{path}.meta"))?;
let transaction_path = self.io_get_object_path(volume, &txn_rel_path)?;
for candidate in [&current_data_path, &current_meta_path, &transaction_path] {
check_path_length(candidate.to_string_lossy().as_ref())?;
}
(current_data_path, current_meta_path, transaction_path, false)
}
};
let durability = effective_durability(volume);
let sync = durability.syncs_commit_metadata();
let part_number = current_data_path
@@ -14684,7 +14779,8 @@ impl DiskAPI for LocalDisk {
PartTransactionAction::Commit => (PART_TRANSACTION_NEW_META, PART_TRANSACTION_OLD_META),
PartTransactionAction::Rollback => (PART_TRANSACTION_OLD_META, PART_TRANSACTION_NEW_META),
};
if let Ok(obsolete) = std::fs::read(transaction_path.join(obsolete_name))
if !(action == PartTransactionAction::Commit && skip_obsolete_read_on_commit && !cfg!(test))
&& let Ok(obsolete) = std::fs::read(transaction_path.join(obsolete_name))
&& let Ok(obsolete) = ObjectPartInfo::unmarshal(&obsolete)
&& let Some(integrity) = obsolete.integrity
&& integrity.validate().is_ok()
@@ -14750,15 +14846,6 @@ impl DiskAPI for LocalDisk {
let admission_source = admission.as_ref().map(|a| a.source().to_path_buf());
pool_meta_native::reject_raw_path(&self.root, src_volume, src_path)?;
pool_meta_native::reject_raw_path(&self.root, dst_volume, dst_path)?;
let src_volume_dir = self.io_get_bucket_path(src_volume)?;
let dst_volume_dir = self.io_get_bucket_path(dst_volume)?;
if !skip_access_checks(src_volume) {
super::fs::access_std(&src_volume_dir).map_err(|e| to_access_error(e, DiskError::VolumeAccessDenied))?
}
if !skip_access_checks(dst_volume) {
super::fs::access_std(&dst_volume_dir).map_err(|e| to_access_error(e, DiskError::VolumeAccessDenied))?
}
let src_is_dir = has_suffix(src_path, SLASH_SEPARATOR);
let dst_is_dir = has_suffix(dst_path, SLASH_SEPARATOR);
@@ -14775,25 +14862,74 @@ impl DiskAPI for LocalDisk {
return Err(DiskError::FileAccessDenied);
}
let src_file_path = self.io_get_object_path(src_volume, src_path)?;
let dst_file_path = self.io_get_object_path(dst_volume, dst_path)?;
// warn!("rename_part src_file_path:{:?}, dst_file_path:{:?}", &src_file_path, &dst_file_path);
check_path_length(src_file_path.to_string_lossy().as_ref())?;
check_path_length(dst_file_path.to_string_lossy().as_ref())?;
let dst_meta_path = if src_is_dir {
None
let cached_prepared = if !src_is_dir {
let txn_rel = crate::disk::part_transaction_path(dst_path);
self.io_get_object_open_path(dst_volume, &txn_rel)
.ok()
.and_then(|txn_open| take_prepared_part_txn_receipt(&txn_open))
.filter(|entry| {
entry.meta == meta
&& self
.io_get_object_open_path(src_volume, src_path)
.is_ok_and(|src_open| src_open == entry.src_file_path)
})
} else {
let path = self.io_get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
check_path_length(path.to_string_lossy().as_ref())?;
Some(path)
None
};
let transaction_path = if src_is_dir {
None
} else {
Some(self.io_get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?)
let (
src_volume_dir,
dst_volume_dir,
src_file_path,
dst_file_path,
dst_meta_path,
transaction_path,
preverified_meta_and_old_flag,
) = match cached_prepared {
Some(entry) => (
entry.src_volume_dir,
entry.dst_volume_dir,
entry.src_file_path,
entry.dst_file_path,
Some(entry.dst_meta_path),
Some(entry.transaction_path),
Some(entry.had_old_meta),
),
None => {
let (src_volume_dir, src_file_path) = self.io_get_bucket_and_object_path(src_volume, src_path)?;
let (dst_volume_dir, dst_file_path) = self.io_get_bucket_and_object_path(dst_volume, dst_path)?;
if !skip_access_checks(src_volume) {
super::fs::access_std(&src_volume_dir).map_err(|e| to_access_error(e, DiskError::VolumeAccessDenied))?
}
if !skip_access_checks(dst_volume) {
super::fs::access_std(&dst_volume_dir).map_err(|e| to_access_error(e, DiskError::VolumeAccessDenied))?
}
check_path_length(src_file_path.to_string_lossy().as_ref())?;
check_path_length(dst_file_path.to_string_lossy().as_ref())?;
let dst_meta_path = if src_is_dir {
None
} else {
let path = self.io_get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
check_path_length(path.to_string_lossy().as_ref())?;
Some(path)
};
let transaction_path = if src_is_dir {
None
} else {
Some(self.io_get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?)
};
(
src_volume_dir,
dst_volume_dir,
src_file_path,
dst_file_path,
dst_meta_path,
transaction_path,
None,
)
}
};
// UploadPart is acknowledged once this rename lands, so the part data and
@@ -14852,35 +14988,43 @@ impl DiskAPI for LocalDisk {
let transaction_publish_meta = if let Some(transaction_path) = transaction_path.as_ref() {
let transaction_meta_path = transaction_path.join(PART_TRANSACTION_NEW_META);
match std::fs::File::open(&transaction_meta_path) {
Ok(mut file) => {
let file_len = file.metadata().map_err(to_file_error)?.len();
if file_len != meta.len() as u64 {
return Err(DiskError::FileCorrupt.into());
}
let mut expected_meta = Vec::with_capacity(meta.len());
std::io::Read::read_to_end(&mut std::io::Read::take(&mut file, file_len), &mut expected_meta)
.map_err(to_file_error)?;
drop(file);
if expected_meta.as_slice() != meta.as_ref() {
return Err(DiskError::FileCorrupt.into());
}
let publish_meta_path = transaction_path.join(PART_TRANSACTION_PUBLISH_META);
if !cfg!(test)
&& preverified_meta_and_old_flag.is_some()
&& publish_meta_path.is_file()
&& std::fs::metadata(&transaction_meta_path).is_ok_and(|m| m.is_file() && m.len() == meta.len() as u64)
{
Some(publish_meta_path)
} else {
match std::fs::File::open(&transaction_meta_path) {
Ok(mut file) => {
let file_len = file.metadata().map_err(to_file_error)?.len();
if file_len != meta.len() as u64 {
return Err(DiskError::FileCorrupt.into());
}
let mut expected_meta = Vec::with_capacity(meta.len());
std::io::Read::read_to_end(&mut std::io::Read::take(&mut file, file_len), &mut expected_meta)
.map_err(to_file_error)?;
drop(file);
if expected_meta.as_slice() != meta.as_ref() {
return Err(DiskError::FileCorrupt.into());
}
let publish_meta_path = transaction_path.join(PART_TRANSACTION_PUBLISH_META);
remove_file_if_exists(&publish_meta_path).map_err(to_file_error)?;
std::fs::hard_link(&transaction_meta_path, &publish_meta_path).map_err(to_file_error)?;
Some(publish_meta_path)
remove_file_if_exists(&publish_meta_path).map_err(to_file_error)?;
std::fs::hard_link(&transaction_meta_path, &publish_meta_path).map_err(to_file_error)?;
Some(publish_meta_path)
}
// Old peers know only RenamePart. The new coordinator never
// reaches rename unless prepare succeeded, so an absent
// transaction directory identifies the rolling-upgrade legacy
// path. A present directory without new.meta is corruption.
Err(err) if err.kind() == ErrorKind::NotFound => match std::fs::metadata(transaction_path) {
Ok(_) => return Err(DiskError::FileCorrupt.into()),
Err(meta_err) if meta_err.kind() == ErrorKind::NotFound => None,
Err(meta_err) => return Err(to_file_error(meta_err)),
},
Err(err) => return Err(to_file_error(err)),
}
// Old peers know only RenamePart. The new coordinator never
// reaches rename unless prepare succeeded, so an absent
// transaction directory identifies the rolling-upgrade legacy
// path. A present directory without new.meta is corruption.
Err(err) if err.kind() == ErrorKind::NotFound => match std::fs::metadata(transaction_path) {
Ok(_) => return Err(DiskError::FileCorrupt.into()),
Err(meta_err) if meta_err.kind() == ErrorKind::NotFound => None,
Err(meta_err) => return Err(to_file_error(meta_err)),
},
Err(err) => return Err(to_file_error(err)),
}
} else {
None
@@ -14943,6 +15087,19 @@ impl DiskAPI for LocalDisk {
}
}
}
if let (Some(had_old_meta), Some(txn_path), Some(meta_path)) =
(preverified_meta_and_old_flag, transaction_path.as_ref(), dst_meta_path.as_path().into())
{
put_published_part_txn_receipt(
txn_path,
PublishedPartTxnReceipt {
current_data_path: dst_file_path,
current_meta_path: meta_path.to_path_buf(),
transaction_path: txn_path.clone(),
had_old_meta,
},
);
}
Ok(true)
} else {
Ok(false)
@@ -19751,6 +19908,9 @@ mod test {
);
}
// Only the Unix grouped dst dir fsync worker consults the injected
// failure; the non-Unix worker syncs by path without the hook.
#[cfg(unix)]
#[tokio::test]
#[serial_test::serial(dst_dir_fsync_group_commit)]
async fn rename_data_non_inline_dst_dir_fsync_group_commit_failure_rolls_back_fresh_put() {
@@ -19839,6 +19999,9 @@ mod test {
);
}
// Only the Unix grouped dst dir fsync worker consults the injected
// failure; the non-Unix worker syncs by path without the hook.
#[cfg(unix)]
#[tokio::test]
#[serial_test::serial(dst_dir_fsync_group_commit)]
async fn rename_data_inline_dst_dir_fsync_group_commit_failure_rolls_back_fresh_put() {
@@ -30444,6 +30607,7 @@ mod test {
expected_delete_marker: None,
delete_recovery_bucket_incarnation: None,
recoverable_delete: false,
requires_bounded_delete_recovery: false,
orphan_data_dir: false,
};
disk.delete("test-volume", "test-file.txt", delete_opts)
+149 -51
View File
@@ -15,8 +15,6 @@
//! Single-disk object rename publication and rollback. The shared execution core
//! retains instrumentation, mutation leases, and commit guards through the syscall.
#[cfg(target_os = "linux")]
use super::has_rename_data_test_failpoints;
#[cfg(all(test, windows))]
use super::run_destination_commit_directory_preparation;
#[cfg(any(not(windows), test))]
@@ -24,8 +22,8 @@ use super::should_fail_local_inline_rollback_hardlink;
use super::{
EVENT_DISK_LOCAL_ACCESS_FAILED, EVENT_DISK_LOCAL_HEAL_PURGE_FAILED, EVENT_DISK_LOCAL_RENAME_REJECTED, LOG_COMPONENT_ECSTORE,
LOG_SUBSYSTEM_DISK_LOCAL, LocalDisk, SyncMode, effective_durability, has_inline_commit_test_failpoints,
inline_metadata_rollback_dir, observe_old_current, remove_dir_all_if_exists, remove_dst_base_before_commit,
remove_file_if_exists, rename_data_versions_signature, run_inline_preparation_before_backup,
has_rename_data_test_failpoints, inline_metadata_rollback_dir, observe_old_current, remove_dir_all_if_exists,
remove_dst_base_before_commit, remove_file_if_exists, rename_data_versions_signature, run_inline_preparation_before_backup,
should_fail_after_metadata_commit, should_fail_before_old_metadata_backup, should_fail_commit_rename,
should_remove_staged_meta_before_commit, skip_access_checks, tmp_staging_mutation_path,
};
@@ -172,6 +170,44 @@ impl RenameCommitContext<'_> {
}
}
#[cfg(any(not(windows), test))]
fn inline_rollback_backup_path(staging_file_path: &Path) -> std::io::Result<PathBuf> {
let Some(staging_parent) = staging_file_path.parent() else {
return Err(std::io::Error::new(ErrorKind::InvalidInput, "missing staging metadata parent"));
};
let backup_path = if staging_file_path.file_name().and_then(|n| n.to_str()) == Some(STORAGE_FORMAT_FILE) {
staging_parent.join(STORAGE_FORMAT_FILE_BACKUP)
} else {
staging_file_path.with_extension("meta.bkp")
};
remove_file_if_exists(&backup_path)?;
Ok(backup_path)
}
#[cfg(any(not(windows), test))]
fn write_local_inline_rollback_backup_from_bytes(
staging_file_path: &Path,
old_metadata: &[u8],
sync: bool,
) -> std::io::Result<PathBuf> {
let backup_path = inline_rollback_backup_path(staging_file_path)?;
let write_res = (|| -> std::io::Result<()> {
#[cfg(test)]
run_owned_file_before_open(&backup_path);
let mut file = std::fs::File::create(&backup_path)?;
std::io::Write::write_all(&mut file, old_metadata)?;
if sync && !old_metadata.is_empty() {
file.sync_data()?;
}
Ok(())
})();
if let Err(err) = write_res {
let _ = remove_file_if_exists(&backup_path);
return Err(err);
}
Ok(backup_path)
}
#[cfg(any(not(windows), test))]
pub(super) fn create_local_inline_rollback_backup(
dst_file_path: &Path,
@@ -179,28 +215,13 @@ pub(super) fn create_local_inline_rollback_backup(
old_metadata: &[u8],
sync: bool,
) -> std::io::Result<PathBuf> {
let Some(staging_parent) = staging_file_path.parent() else {
return Err(std::io::Error::new(ErrorKind::InvalidInput, "missing staging metadata parent"));
};
let backup_path = staging_parent.join(STORAGE_FORMAT_FILE_BACKUP);
remove_file_if_exists(&backup_path)?;
let backup_path = inline_rollback_backup_path(staging_file_path)?;
if should_fail_local_inline_rollback_hardlink(dst_file_path) || std::fs::hard_link(dst_file_path, &backup_path).is_err() {
let write_res = (|| -> std::io::Result<()> {
#[cfg(test)]
run_owned_file_before_open(&backup_path);
let mut file = std::fs::File::create(&backup_path)?;
std::io::Write::write_all(&mut file, old_metadata)?;
if sync && !old_metadata.is_empty() {
file.sync_data()?;
}
Ok(())
})();
if let Err(err) = write_res {
let _ = remove_file_if_exists(&backup_path);
return Err(err);
}
} else if sync && !old_metadata.is_empty() {
std::fs::File::open(&backup_path)?.sync_data()?;
return write_local_inline_rollback_backup_from_bytes(staging_file_path, old_metadata, sync);
} else if cfg!(test) && sync && !old_metadata.is_empty() {
// Skip redundant fdatasync on hardlinked rollback backups in production;
// keep a Windows-safe sync open in tests.
os::open_file_for_sync_std(&backup_path)?.sync_data()?;
}
Ok(backup_path)
}
@@ -658,7 +679,14 @@ impl LocalDisk {
})
.collect()
};
let src_volume_dir = self.io_get_bucket_path(src_volume)?;
let src_volume_dir = if cfg!(target_os = "linux")
&& src_volume == super::super::RUSTFS_META_TMP_BUCKET
&& skip_access_checks(src_volume)
{
self.io_root().join(src_volume)
} else {
self.io_get_bucket_path(src_volume)?
};
if !skip_access_checks(src_volume)
&& let Err(e) = super::super::fs::access_std(&src_volume_dir)
{
@@ -702,8 +730,24 @@ impl LocalDisk {
} else {
STORAGE_FORMAT_FILE
};
let flat_inline_tmp = cfg!(target_os = "linux")
&& !no_inline
&& src_volume == super::super::RUSTFS_META_TMP_BUCKET
&& !preserve_upload_metadata
&& multipart_publication.is_none()
&& pool_meta_condition.is_none()
&& !fi_healing
&& !os::dst_dir_fsync_group_commit_enabled()
&& !os::file_fdatasync_group_commit_enabled()
&& !has_rename_data_test_failpoints(dst_path);
let src_file_path = match multipart_publication.as_ref() {
Some(publication) => publication.canonical_head(),
None if flat_inline_tmp => {
self.io_get_object_open_path(src_volume, format!("{src_path}.{source_metadata_name}").as_str())?
}
None if cfg!(target_os = "linux") => {
self.io_get_object_open_path(src_volume, format!("{src_path}/{source_metadata_name}").as_str())?
}
None => self.io_get_object_path(src_volume, format!("{src_path}/{source_metadata_name}").as_str())?,
};
let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
@@ -714,14 +758,13 @@ impl LocalDisk {
.data_dir
.map(|dir| rustfs_utils::path::retain_slash(dir.to_string().as_str()))
{
let src_data_path = self.io_get_object_path(
src_volume,
rustfs_utils::path::retain_slash(format!("{}/{}", src_path, data_dir).as_str()).as_str(),
)?;
let dst_data_path = self.io_get_object_path(
dst_volume,
rustfs_utils::path::retain_slash(format!("{}/{}", dst_path, data_dir).as_str()).as_str(),
)?;
let src_rel = rustfs_utils::path::retain_slash(format!("{}/{}", src_path, data_dir).as_str());
let src_data_path = if cfg!(target_os = "linux") && multipart_publication.is_none() && !fi_healing {
self.io_get_object_open_path(src_volume, src_rel.as_str())?
} else {
self.io_get_object_path(src_volume, src_rel.as_str())?
};
let dst_data_path = destination_object_path.join(data_dir.trim_matches('/'));
Some((src_data_path, dst_data_path))
} else {
@@ -804,7 +847,7 @@ impl LocalDisk {
let fused_attempt = move |is_cancelled: &dyn Fn() -> bool| -> std::io::Result<FusedCommitAttempt> {
let (guard, has_dst_buf) = prepare_rename_commit_and_destination_metadata_std(
!no_inline,
!no_inline && !flat_inline_tmp,
&src_parent_owned,
&dst_parent_owned,
&dst_file_path_owned,
@@ -876,7 +919,16 @@ impl LocalDisk {
run_owned_file_before_open(&src_file_path_owned);
let mut prepared_metadata_source =
os::create_prepared_rename_source_with_commit_guard(&src_file_path_owned, &dst_file_path_owned, &guard)?;
prepared_metadata_source.write_all(&new_dst_buf, false)?;
let cleanup_flat_tmp = || {
if flat_inline_tmp {
let _ = remove_file_if_exists(&src_file_path_owned);
}
};
if let Err(err) = prepared_metadata_source.write_all(&new_dst_buf, false) {
drop(prepared_metadata_source);
cleanup_flat_tmp();
return Err(err);
}
if !no_inline {
run_inline_preparation_before_backup(&dst_path_str);
}
@@ -884,14 +936,21 @@ impl LocalDisk {
let mut staged_rollback_path = None;
if (!no_inline || rollback_data_dir.is_none())
&& let Some(ref old_metadata) = has_dst_buf
&& (rollback_data_dir.is_some() || sync_commit_metadata || cfg!(test))
&& (rollback_data_dir.is_some() || cfg!(test))
{
let backup_path = create_local_inline_rollback_backup(
let backup_path = match create_local_inline_rollback_backup(
&dst_file_path_owned,
&src_file_path_owned,
old_metadata,
sync_commit_metadata,
)?;
) {
Ok(path) => path,
Err(err) => {
drop(prepared_metadata_source);
cleanup_flat_tmp();
return Err(err);
}
};
staged_rollback_path = Some(StagedRollbackBackup(backup_path));
}
@@ -920,6 +979,8 @@ impl LocalDisk {
return Err(err);
}
} else if sync_commit_metadata && let Err(err) = prepared_metadata_source.sync_data() {
drop(prepared_metadata_source);
cleanup_flat_tmp();
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
@@ -928,6 +989,7 @@ impl LocalDisk {
if is_cancelled() {
drop(prepared_metadata_source);
cleanup_flat_tmp();
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
@@ -944,6 +1006,8 @@ impl LocalDisk {
) {
Ok(guard) => guard,
Err(err) => {
drop(prepared_metadata_source);
cleanup_flat_tmp();
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
@@ -1027,6 +1091,8 @@ impl LocalDisk {
None,
sync_commit_metadata,
) {
drop(prepared_metadata_source);
cleanup_flat_tmp();
let _ = remove_file_if_exists(staged_backup);
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
}
@@ -1047,6 +1113,7 @@ impl LocalDisk {
)
};
if let Err(err) = rename_res {
cleanup_flat_tmp();
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
@@ -1075,8 +1142,38 @@ impl LocalDisk {
return Ok(FusedCommitAttempt::CommitErr(DiskError::Unexpected));
}
if sync_commit_metadata
&& let Err(err) = fsync_inline_commit_std(
if sync_commit_metadata {
if staged_rollback_path.is_none()
&& rollback_data_dir.is_none()
&& let Some(ref old_metadata) = has_dst_buf
&& (!no_inline || rollback_data_dir.is_none())
{
let fsync_res = (|| -> std::io::Result<()> {
if let Some(dst_parent) = dst_file_path_owned.parent() {
os::fsync_commit_directory_limited_std(dst_parent, &guard)?;
}
Ok(())
})();
if let Err(err) = fsync_res {
let rollback_ok =
write_local_inline_rollback_backup_from_bytes(&src_file_path_owned, old_metadata, true)
.and_then(|backup_path| {
let backup = StagedRollbackBackup(backup_path);
let res =
rollback_inline_metadata_std(&dst_file_path_owned, rollback_data_dir, Some(&backup));
let _ = remove_file_if_exists(backup.path());
res
})
.is_ok();
if no_inline {
if rollback_ok && let Some((_, new_data_path)) = has_data_dir_path_owned.as_ref() {
let _ = remove_dir_all_if_exists(new_data_path);
}
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
}
return Ok(FusedCommitAttempt::CommitErr(DiskError::from(err)));
}
} else if let Err(err) = fsync_inline_commit_std(
&dst_file_path_owned,
&dst_volume_dir_owned,
destination_was_absent,
@@ -1087,19 +1184,19 @@ impl LocalDisk {
.filter(|_| no_inline)
.map(|(_, new_data_path)| new_data_path.as_path()),
&guard,
)
{
if no_inline {
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
) {
if no_inline {
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
}
return Ok(FusedCommitAttempt::CommitErr(DiskError::from(err)));
}
return Ok(FusedCommitAttempt::CommitErr(DiskError::from(err)));
}
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
if let Some(parent) = src_file_path_owned.parent() {
if !flat_inline_tmp && let Some(parent) = src_file_path_owned.parent() {
let _ = std::fs::remove_dir(parent);
}
@@ -1364,7 +1461,7 @@ impl LocalDisk {
}
}
}
match publication_owned.finish_on_thread(sync_commit_metadata, staging_cleanup) {
match publication_owned.finish_on_thread(sync_commit_metadata, !cfg!(test), staging_cleanup) {
Ok(result) => Ok(MultipartFusedCommitOutcome::Committed(Box::new(RenameDataResp {
multipart_source: Some(Box::new(result)),
old_data_dir: has_old_data_dir,
@@ -1658,12 +1755,13 @@ impl LocalDisk {
);
}
if let Some((src_data_path, dst_data_path)) = has_data_dir_path.as_ref() {
#[cfg(target_os = "linux")]
#[cfg(any(target_os = "linux", windows))]
let payload_guard = if multipart_publication.is_some() {
Some(
os::multipart_payload_commit_guard(
&commit.directory_guard,
src_data_path.parent().ok_or(DiskError::InvalidPath)?,
commit.publication_root,
commit.mutation_lease.clone(),
)
.await
@@ -1672,9 +1770,9 @@ impl LocalDisk {
} else {
None
};
#[cfg(target_os = "linux")]
#[cfg(any(target_os = "linux", windows))]
let directory_guard = payload_guard.as_ref().unwrap_or(&commit.directory_guard);
#[cfg(not(target_os = "linux"))]
#[cfg(not(any(target_os = "linux", windows)))]
let directory_guard = &commit.directory_guard;
if let Err(err) = os::rename_all_with_commit_guard(
src_data_path,
+342 -58
View File
@@ -856,6 +856,12 @@ pub(super) fn inspection_with_markers_bounded(
let prepared_digest = prepared.as_deref().map(digest);
let target_digest = target.as_deref().map(digest);
let receipt_digest = receipt_bytes.as_deref().map(digest);
let finished = finished
|| (committed
&& prepared.is_none()
&& before.is_none()
&& target.is_none()
&& current_digest == journal.as_ref().and_then(|j| j.target_digest));
if let Some(journal) = &journal {
if (!finished
&& journal.target_digest != target_digest
@@ -1113,13 +1119,34 @@ pub(super) fn inspection_with_markers_bounded(
Ok((inspection, has_physical_data_dirs_or_markers))
}
type PreparedDeleteReceiptEntry = (DeleteRecoveryJournal, bool);
#[derive(Clone)]
struct PreparedDeleteReceiptEntry {
object_path: PathBuf,
bucket_path: PathBuf,
file_path: PathBuf,
staging_root: PathBuf,
journal_raw: Vec<u8>,
journal: DeleteRecoveryJournal,
sole_transaction: bool,
}
#[derive(Clone)]
struct PublishedDeleteReceiptEntry {
object_path: PathBuf,
bucket_path: PathBuf,
file_path: PathBuf,
staging_root: PathBuf,
journal_raw: Vec<u8>,
}
const PREPARED_DELETE_RECEIPTS_PER_SHARD: usize = 64;
static PREPARED_DELETE_RECEIPTS: std::sync::LazyLock<os::ShardedPathCache<PreparedDeleteReceiptEntry>> =
std::sync::LazyLock::new(|| os::ShardedPathCache::new(PREPARED_DELETE_RECEIPTS_PER_SHARD));
static PUBLISHED_DELETE_RECEIPTS: std::sync::LazyLock<os::ShardedPathCache<PublishedDeleteReceiptEntry>> =
std::sync::LazyLock::new(|| os::ShardedPathCache::new(PREPARED_DELETE_RECEIPTS_PER_SHARD));
fn put_prepared_delete_receipt(rollback: impl AsRef<Path>, entry: PreparedDeleteReceiptEntry) {
PREPARED_DELETE_RECEIPTS.insert(rollback.as_ref(), entry);
}
@@ -1128,6 +1155,27 @@ fn take_prepared_delete_receipt(rollback: &Path) -> Option<PreparedDeleteReceipt
PREPARED_DELETE_RECEIPTS.remove(rollback)
}
fn put_published_delete_receipt(rollback: impl AsRef<Path>, entry: PublishedDeleteReceiptEntry) {
PUBLISHED_DELETE_RECEIPTS.insert(rollback.as_ref(), entry);
}
fn take_published_delete_receipt(rollback: &Path) -> Option<PublishedDeleteReceiptEntry> {
PUBLISHED_DELETE_RECEIPTS.remove(rollback)
}
fn format_commit_receipt_bytes(journal_raw: &[u8], finished: bool) -> Vec<u8> {
let suffix: &[u8] = if finished {
b",\"finished\":true}"
} else {
b",\"finished\":false}"
};
let mut out = Vec::with_capacity(11 + journal_raw.len() + suffix.len());
out.extend_from_slice(b"{\"journal\":");
out.extend_from_slice(journal_raw);
out.extend_from_slice(suffix);
out
}
impl LocalDisk {
// Native MPU source closure has its own conditional protocol. Generic
// object recovery must never restore its head or acquire SourceRW under
@@ -2376,6 +2424,7 @@ impl LocalDisk {
barrier(volume, object, transaction, stage, &self.root).await
}
#[cfg(test)]
pub(super) async fn try_fused_recoverable_delete_version(
&self,
volume: &str,
@@ -2384,6 +2433,21 @@ impl LocalDisk {
opts: &DeleteOptions,
transaction: Uuid,
owner: Option<Arc<dyn Send + Sync>>,
) -> Result<Option<()>> {
self.try_fused_recoverable_delete_version_with_paths(volume, object, fi, opts, transaction, owner, None)
.await
}
#[expect(clippy::too_many_arguments)]
pub(super) async fn try_fused_recoverable_delete_version_with_paths(
&self,
volume: &str,
object: &str,
fi: &FileInfo,
opts: &DeleteOptions,
transaction: Uuid,
owner: Option<Arc<dyn Send + Sync>>,
prevalidated_paths: Option<(PathBuf, PathBuf, PathBuf)>,
) -> Result<Option<()>> {
if !opts.recoverable_delete
|| opts.undo_write
@@ -2396,11 +2460,22 @@ impl LocalDisk {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
self.wait_for_startup_cleanup().await;
let (_, file_path) = self.io_get_bucket_and_object_path(volume, object)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
let (object_path, bucket_path, file_path) = match prevalidated_paths {
Some(paths) => paths,
None => {
let obj = self.get_object_path(volume, object)?;
let (b, f) = self.io_get_bucket_and_object_path(volume, object)?;
check_path_length(f.to_string_lossy().as_ref())?;
(obj, b, f)
}
};
let xl_path = path_join(&[file_path.as_path(), Path::new(STORAGE_FORMAT_FILE)]);
let destination = file_path.join(transaction.to_string());
let staging_root = self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?;
let staging_root = if cfg!(target_os = "linux") {
self.io_root().join(RUSTFS_META_TMP_BUCKET)
} else {
self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?
};
let sync = effective_durability(volume).syncs_commit_metadata();
let incarnation = opts.delete_recovery_bucket_incarnation;
let lease = os::acquire_namespace_mutation_lease_with_owner(&destination, owner).await;
@@ -2587,6 +2662,8 @@ impl LocalDisk {
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
#[cfg(all(test, unix))]
fused_prepare_checkpoint::run(&file_path);
let mut reserved_data_path = None;
if let Some(old_path) = old_data_path {
let reserved_marker = old_path.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"));
@@ -2606,6 +2683,7 @@ impl LocalDisk {
reserved_data_path = Some((old_path, reserved_marker));
}
if let Some(ref target_bytes) = target {
// A provisional head must stay outside the published recovery transaction.
let target_in_rollback = destination.join(TARGET);
let linked_xl = staging_root.join(format!("delete-xl-{transaction}"));
let _ = std::fs::remove_file(&linked_xl);
@@ -2653,7 +2731,18 @@ impl LocalDisk {
Err(err) => return Err(err),
}
}
put_prepared_delete_receipt(destination, (journal, sole_transaction));
put_prepared_delete_receipt(
destination,
PreparedDeleteReceiptEntry {
object_path,
bucket_path,
file_path,
staging_root,
journal_raw,
journal,
sole_transaction,
},
);
Ok(Ok(Some(old_dir)))
},
)
@@ -2690,19 +2779,45 @@ impl LocalDisk {
namespace_owner: Option<Arc<dyn Send + Sync>>,
work_budget: Option<Arc<crate::object_api::DeleteRecoveryWorkBudget>>,
) -> Result<Option<(DeleteRecoveryJournal, bool, bool)>> {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) || transaction.is_nil() {
if transaction.is_nil() {
return Err(DiskError::InvalidPath);
}
let object_path = self.get_object_path(volume, object)?;
let rollback_open_path = if volume != RUSTFS_META_MULTIPART_BUCKET {
self.io_get_object_open_path(volume, object)
.ok()
.map(|p| p.join(transaction.to_string()))
} else {
None
};
let cached_prepared = rollback_open_path.as_deref().and_then(take_prepared_delete_receipt);
let (object_path, bucket_path, root, staging_root, cached_receipt) = match cached_prepared {
Some(entry) => (
entry.object_path,
Some(entry.bucket_path),
entry.file_path,
entry.staging_root,
Some((entry.journal_raw, entry.journal, entry.sole_transaction)),
),
None => {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) {
return Err(DiskError::InvalidPath);
}
let obj_path = self.get_object_path(volume, object)?;
(
obj_path,
None,
self.io_get_object_path(volume, object)?,
self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?,
None,
)
}
};
let metadata_owner = os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await;
let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await;
let owner: Option<Arc<dyn Send + Sync>> = Some(Arc::new((metadata_owner, volume_guard)));
let root = self.io_get_object_path(volume, object)?;
let rollback = root.join(transaction.to_string());
let receipt_path = rollback.join(DELETE_QUORUM_COMMITTED_FILE);
let staging_root = self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?;
let sync = effective_durability(volume).syncs_commit_metadata();
let lease = os::acquire_namespace_mutation_lease_with_owner(&receipt_path, owner).await;
let volume_str = volume.to_owned();
@@ -2712,23 +2827,38 @@ impl LocalDisk {
lease,
sync,
move || -> std::io::Result<Result<Option<(DeleteRecoveryJournal, bool, bool)>>> {
if let Some((journal, sole_transaction)) = take_prepared_delete_receipt(&rollback)
if let Some((journal_raw, journal, sole_transaction)) = cached_receipt
&& rollback.join(PREPARED).is_file()
{
let receipt = CommitReceipt {
journal: journal.clone(),
finished: false,
};
let receipt_bytes = serde_json::to_vec(&receipt).map_err(std::io::Error::other)?;
std::fs::create_dir_all(&staging_root)?;
let mut staged = tempfile::NamedTempFile::new_in(&staging_root)?;
std::io::Write::write_all(staged.as_file_mut(), &receipt_bytes)?;
if sync && !receipt_bytes.is_empty() {
staged.as_file().sync_data()?;
let tmp_receipt = staging_root.join(format!("receipt-{transaction}.tmp"));
let receipt_bytes = format_commit_receipt_bytes(&journal_raw, false);
let write_res = (|| -> std::io::Result<()> {
let mut file = std::fs::File::create(&tmp_receipt)?;
std::io::Write::write_all(&mut file, &receipt_bytes)?;
if sync && !receipt_bytes.is_empty() {
file.sync_data()?;
}
std::fs::rename(&tmp_receipt, &receipt_path)?;
if sync {
os::fsync_dir_std(&rollback)?;
}
Ok(())
})();
if let Err(err) = write_res {
let _ = std::fs::remove_file(&tmp_receipt);
return Err(err);
}
std::fs::rename(staged.path(), &receipt_path)?;
if sync {
os::fsync_dir_std(&rollback)?;
if let Some(bucket_path) = bucket_path {
put_published_delete_receipt(
rollback,
PublishedDeleteReceiptEntry {
object_path,
bucket_path,
file_path: root,
staging_root,
journal_raw,
},
);
}
return Ok(Ok(Some((journal, sole_transaction, true))));
}
@@ -2851,14 +2981,46 @@ impl LocalDisk {
Err(err) => Err(err),
}
}
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) || transaction.is_nil() {
if transaction.is_nil() {
return Err(DiskError::InvalidPath);
}
let object_path = self.get_object_path(volume, object)?;
let rollback_open_path = if volume != RUSTFS_META_MULTIPART_BUCKET {
self.io_get_object_open_path(volume, object)
.ok()
.map(|p| p.join(transaction.to_string()))
} else {
None
};
let cached_published = rollback_open_path.as_deref().and_then(take_published_delete_receipt);
let (object_path, bucket_path, staging_root, root, cached_journal_raw) = match cached_published {
Some(entry) => (
entry.object_path,
entry.bucket_path,
entry.staging_root,
entry.file_path,
Some(entry.journal_raw),
),
None => {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) {
return Err(DiskError::InvalidPath);
}
let obj_path = self.get_object_path(volume, object)?;
(
obj_path,
self.io_get_bucket_path(volume)?,
self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?,
self.io_get_object_path(volume, object)?,
None,
)
}
};
let metadata_owner = os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await;
let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await;
let owner: Option<Arc<dyn Send + Sync>> = Some(Arc::new((metadata_owner, volume_guard)));
let rollback = root.join(transaction.to_string());
let receipt_path = rollback.join(DELETE_QUORUM_COMMITTED_FILE);
let sync = effective_durability(volume).syncs_commit_metadata();
let mut claims: Vec<SnapshotLeaseDeleteClaim> = Vec::with_capacity(journal.data_dirs.len());
for data_dir in &journal.data_dirs {
@@ -2880,15 +3042,9 @@ impl LocalDisk {
claims.push(claim);
}
let bucket_path = self.io_get_bucket_path(volume)?;
let staging_root = self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?;
let root = self.io_get_object_path(volume, object)?;
let rollback = root.join(transaction.to_string());
let receipt_path = rollback.join(DELETE_QUORUM_COMMITTED_FILE);
let sync = effective_durability(volume).syncs_commit_metadata();
let data_dirs = journal.data_dirs.clone();
let has_target = journal.target_digest.is_some();
let journal_for_receipt = journal.clone();
let journal_for_receipt = cached_journal_raw.is_none().then(|| journal.clone());
let lease = os::acquire_namespace_mutation_lease_with_owner(&receipt_path, owner).await;
let prune_paths: Vec<_> = rollback
.ancestors()
@@ -2897,6 +3053,31 @@ impl LocalDisk {
.collect();
let res = os::run_blocking_namespace_operation_with_sync(lease, sync, move || -> std::io::Result<()> {
let persist_receipt = |finished: bool| -> std::io::Result<()> {
let receipt_bytes = match &cached_journal_raw {
Some(raw) => format_commit_receipt_bytes(raw, finished),
None => {
let receipt = CommitReceipt {
journal: journal_for_receipt
.clone()
.ok_or_else(|| std::io::Error::other("missing journal for receipt"))?,
finished,
};
serde_json::to_vec(&receipt).map_err(std::io::Error::other)?
}
};
std::fs::create_dir_all(&staging_root)?;
let mut staged_receipt = tempfile::NamedTempFile::new_in(&staging_root)?;
std::io::Write::write_all(staged_receipt.as_file_mut(), &receipt_bytes)?;
if sync && !receipt_bytes.is_empty() {
staged_receipt.as_file().sync_data()?;
}
std::fs::rename(staged_receipt.path(), &receipt_path)?;
if sync {
let _ = os::fsync_dir_std(&rollback);
}
Ok(())
};
let committed = format!("{DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
let recovery_committed = format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
let reserved = format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
@@ -2951,38 +3132,38 @@ impl LocalDisk {
os::fsync_dir_std(&root)?;
}
for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] {
ignore_not_found(std::fs::remove_file(rollback.join(name)))?;
}
if has_target || had_physical_data_dirs {
let finished_receipt = CommitReceipt {
journal: journal_for_receipt,
finished: true,
};
let receipt_bytes = serde_json::to_vec(&finished_receipt).map_err(std::io::Error::other)?;
std::fs::create_dir_all(&staging_root)?;
let mut staged_receipt = tempfile::NamedTempFile::new_in(&staging_root)?;
std::io::Write::write_all(staged_receipt.as_file_mut(), &receipt_bytes)?;
if sync && !receipt_bytes.is_empty() {
staged_receipt.as_file().sync_data()?;
}
std::fs::rename(staged_receipt.path(), &receipt_path)?;
let _ = std::fs::File::create(rollback.join(FINISHED))?;
if sync {
os::fsync_dir_std(&rollback)?;
if let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name))) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
}
for name in [TARGET, PREPARED, FINISHED] {
for name in [PREPARED, TARGET, FINISHED] {
let path = rollback.join(name);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
ignore_not_found(std::fs::remove_file(path))?;
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
}
if sync {
os::fsync_dir_std(&rollback)?;
if sync && let Err(err) = os::fsync_dir_std(&rollback) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &receipt_path);
ignore_not_found(std::fs::remove_file(&receipt_path))?;
if let Err(err) = ignore_not_found(std::fs::remove_file(&receipt_path)) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
for dir_path in &prune_paths {
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, dir_path);
@@ -3105,6 +3286,42 @@ pub(super) async fn restore_barrier(volume: &str, object_dir: &Path, transaction
Ok(())
}
#[cfg(all(test, unix))]
mod fused_prepare_checkpoint {
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::{LazyLock, Mutex};
type Hook = Box<dyn FnOnce() + Send>;
static HOOKS: LazyLock<Mutex<HashMap<PathBuf, Hook>>> = LazyLock::new(|| Mutex::new(HashMap::new()));
pub(super) struct Guard(PathBuf);
impl Drop for Guard {
fn drop(&mut self) {
HOOKS.lock().expect("checkpoint hooks").remove(&self.0);
}
}
pub(super) fn install(root: &Path, hook: impl FnOnce() + Send + 'static) -> Guard {
assert!(
HOOKS
.lock()
.expect("checkpoint hooks")
.insert(root.to_path_buf(), Box::new(hook))
.is_none()
);
Guard(root.to_path_buf())
}
pub(super) fn run(root: &Path) {
let hook = HOOKS.lock().expect("checkpoint hooks").remove(root);
if let Some(hook) = hook {
hook();
}
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -7020,6 +7237,73 @@ mod tests {
marker.deleted = false;
}
#[cfg(unix)]
#[tokio::test]
async fn delete_recovery_fused_prepare_checkpoint_reopens_before_head_publication() {
let (_temp, disk, mut first, _) = fixture().await;
first.version_id = Some(Uuid::new_v4());
let mut second = first.clone();
second.version_id = Some(Uuid::new_v4());
let mut meta = FileMeta::new();
meta.add_version(first).expect("add surviving version");
meta.add_version(second.clone()).expect("add deleted version");
let original = meta.marshal_msg().expect("encode original versions");
let (_, root) = disk.io_get_bucket_and_object_path("recovery", "object").expect("object path");
fs::write(root.join(STORAGE_FORMAT_FILE), &original)
.await
.expect("write versions");
let (checkpoint_temp, checkpoint_disk, _, _) = fixture().await;
let checkpoint_root = checkpoint_temp.path().join("recovery/object");
drop(checkpoint_disk);
let transaction = Uuid::new_v4();
let source_root = root.clone();
let copy_root = checkpoint_root.clone();
let _capture = fused_prepare_checkpoint::install(&root, move || {
let source = source_root.join(transaction.to_string());
let destination = copy_root.join(transaction.to_string());
std::fs::create_dir(&destination).expect("create checkpoint transaction");
std::fs::copy(source_root.join(STORAGE_FORMAT_FILE), copy_root.join(STORAGE_FORMAT_FILE))
.expect("capture head before publication");
for entry in std::fs::read_dir(source).expect("read published preparation") {
let entry = entry.expect("prepared file");
assert!(entry.file_type().expect("prepared file type").is_file());
std::fs::copy(entry.path(), destination.join(entry.file_name())).expect("capture prepared file");
}
});
let opts = DeleteOptions {
recoverable_delete: true,
..Default::default()
};
assert_eq!(
disk.try_fused_recoverable_delete_version("recovery", "object", &second, &opts, transaction, None)
.await
.expect("finish original preparation"),
Some(())
);
assert!(checkpoint_root.join(transaction.to_string()).join(PREPARED).is_file());
let endpoint =
Endpoint::try_from(checkpoint_temp.path().to_str().expect("checkpoint path")).expect("checkpoint endpoint");
let reopened = LocalDisk::new(&endpoint, false)
.await
.expect("reopen checkpoint as a fresh disk");
let inspection = reopened
.delete_recovery_inspect("recovery", "object", transaction)
.await
.expect("actual preparation checkpoint must remain recoverable");
assert_eq!(inspection.state, DeleteRecoveryState::NewPrepared);
reopened
.settle_delete_transaction("recovery", "object", transaction, inspection.fingerprint, DeleteRecoveryAction::Rollback)
.await
.expect("roll back interrupted preparation");
assert_eq!(
fs::read(checkpoint_root.join(STORAGE_FORMAT_FILE))
.await
.expect("read rolled-back versions"),
original
);
}
#[tokio::test]
async fn delete_recovery_fused_versioned_delete_with_surviving_version_and_interrupted_target_cleanup() {
let (_temp, disk, mut v1, _) = fixture().await;
@@ -372,29 +372,32 @@ impl IndexOwner {
#[cfg(unix)]
{
use rustix::fs::{Mode, OFlags, mkdirat, openat};
let mut directories = vec![self._identity_pin.try_clone()?];
for component in [".rustfs.sys", "multipart-index-v1", &sha(bucket)] {
let parent = directories.last().ok_or_else(|| io::Error::other("index parent missing"))?;
match mkdirat(parent, component, Mode::RWXU | Mode::RWXG | Mode::RWXO) {
Ok(()) | Err(rustix::io::Errno::EXIST) => {}
Err(error) => return Err(error.into()),
let open_dir = |parent: rustix::fd::BorrowedFd<'_>, component: &str| -> io::Result<rustix::fd::OwnedFd> {
let flags = OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC;
match openat(parent, component, flags, Mode::empty()) {
Ok(dir) => Ok(dir),
Err(rustix::io::Errno::NOENT) => {
match mkdirat(parent, component, Mode::RWXU | Mode::RWXG | Mode::RWXO) {
Ok(()) | Err(rustix::io::Errno::EXIST) => {}
Err(error) => return Err(error.into()),
}
openat(parent, component, flags, Mode::empty()).map_err(Into::into)
}
Err(error) => Err(error.into()),
}
let directory = openat(
parent,
component,
OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC,
Mode::empty(),
)?;
directories.push(std::fs::File::from(directory));
}
let parent = directories.last().ok_or_else(|| io::Error::other("index parent missing"))?;
};
use std::os::fd::AsFd;
let bucket_sha = sha(bucket);
let sys_dir = open_dir(self._identity_pin.as_fd(), ".rustfs.sys")?;
let index_dir = open_dir(sys_dir.as_fd(), "multipart-index-v1")?;
let parent = open_dir(index_dir.as_fd(), &bucket_sha)?;
// The index is boot-local: every start rescans the staging area
// before serving it, so a marker lost to a crash is recreated before
// it can be consulted. Creation is enough; no fsync of the marker or
// its ancestors. An existing marker is touched so pruning sees the
// new upload's age, not the previous one's.
match openat(
parent,
&parent,
source_root,
OFlags::CREATE | OFlags::EXCL | OFlags::WRONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC,
Mode::RUSR | Mode::WUSR,
@@ -409,7 +412,7 @@ impl IndexOwner {
// the previous upload's age, which only shortens the prune
// grace for this key; index completeness does not depend on it.
if let Ok(marker) =
openat(parent, source_root, OFlags::WRONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC, Mode::empty())
openat(&parent, source_root, OFlags::WRONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC, Mode::empty())
{
let _ = std::fs::File::from(marker).set_modified(std::time::SystemTime::now());
}
@@ -430,6 +433,18 @@ impl IndexOwner {
}
fn ensure_current_root(&self) -> io::Result<()> {
#[cfg(unix)]
let current = {
use std::os::unix::fs::MetadataExt;
let metadata = std::fs::symlink_metadata(&*self.root_path)?;
if !metadata.is_dir() {
return Err(io::Error::other("multipart index physical root is not a directory"));
}
let mut inode = [0; 16];
inode[..8].copy_from_slice(&metadata.ino().to_le_bytes());
(metadata.dev(), inode)
};
#[cfg(not(unix))]
let current = os::PublicationRoot::new(&self.root_path)?.physical_identity()?;
if current != self.physical {
self.disabled.store(true, Ordering::Release);
@@ -1517,7 +1517,7 @@ impl LocalDisk {
}
Err(error) => return Err(error),
};
let _directory = os::pin_multipart_source_directory(&directory, base, root)?;
let directory_pin = os::pin_multipart_source_directory(&directory, base, root)?;
if !parts.is_empty() {
let anchor_path = directory.join(RETIREMENT_ANCHOR);
let bytes = os::read_multipart_source_metadata(&anchor_path, base, root)?;
@@ -1550,7 +1550,13 @@ impl LocalDisk {
}
// An anchor-less empty DD is only rmdir'ed. A new child makes
// this fail closed; no recursive removal can consume its bytes.
// A Windows pin refuses DELETE sharing, so the leaf pin is
// released first; the object pin still holds the parent.
#[cfg(windows)]
drop(directory_pin);
std::fs::remove_dir(&directory)?;
#[cfg(not(windows))]
drop(directory_pin);
if sync {
os::fsync_dir_std(&object_path)?;
}
@@ -1973,25 +1979,28 @@ impl NativeMultipartPublication {
let canonical = self.canonical_head();
let prepared = (|| -> std::io::Result<os::PreparedRenameSource> {
if let Some((ref original_bytes, _, _)) = deferred_initial {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let snapshot = current
.snapshot
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_head = current
.head
.as_ref()
.and_then(SourceHead::raw)
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
if !current.head_matches_state
|| snapshot.phase != MultipartSourcePhase::Active
|| snapshot.capability_sha256.is_some()
|| snapshot.lineage != expected_lineage
|| snapshot.generation != expected_generation
|| snapshot.revision.checked_add(1) != Some(expected_revision)
|| current_head != original_bytes.as_slice()
{
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
if cfg!(test) || !can_mark_presynced {
let current =
load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let snapshot = current
.snapshot
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_head = current
.head
.as_ref()
.and_then(SourceHead::raw)
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
if !current.head_matches_state
|| snapshot.phase != MultipartSourcePhase::Active
|| snapshot.capability_sha256.is_some()
|| snapshot.lineage != expected_lineage
|| snapshot.generation != expected_generation
|| snapshot.revision.checked_add(1) != Some(expected_revision)
|| current_head != original_bytes.as_slice()
{
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
}
} else {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
@@ -2044,7 +2053,7 @@ impl NativeMultipartPublication {
let _ = std::fs::remove_file(prepared_path);
return Err(err);
}
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, base, root)
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, base, root, Some(directory_guard))
})()
.map_err(DiskError::from)?;
*self.state.lock() = state;
@@ -2154,7 +2163,7 @@ impl NativeMultipartPublication {
let mut prepared_metadata_source =
os::create_prepared_rename_source_with_commit_guard(&prepared_path, &dst_file_path, &owned_directory_guard)?;
prepared_metadata_source.write_all(&new_bytes, true)?;
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, &base, &root)
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, &base, &root, Some(&owned_directory_guard))
})
.await
.map_err(DiskError::from)?;
@@ -2201,7 +2210,7 @@ impl NativeMultipartPublication {
let base = self.store.base.clone();
let root = self.store.publication_root.clone();
os::run_blocking_namespace_operation(self.store.lease.clone(), move || {
os::install_multipart_prepared_head(prepared, &canonical, &base, &root)
os::install_multipart_prepared_head(prepared, &canonical, &base, &root, None)
})
.await
.map_err(DiskError::from)
@@ -2258,6 +2267,7 @@ impl NativeMultipartPublication {
pub(super) fn finish_on_thread(
&self,
target_already_synced: bool,
fused_in_place: bool,
staging_cleanup: Option<(PathBuf, Arc<std::sync::atomic::AtomicBool>)>,
) -> Result<MultipartPublicationResult> {
let mut state = self.state.lock().clone();
@@ -2287,62 +2297,64 @@ impl NativeMultipartPublication {
let disk_ids = &self.store.disk_ids;
let terminal = terminal_record_path(source, &state);
(|| -> std::io::Result<()> {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let current_state = current
.state
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_raw = encode(current_state).map_err(IoError::other)?;
if current.head.is_some() || current_raw != expected_state_raw {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
if target_already_synced {
let pinned = os::pin_multipart_source_directory(&target_dir, base, root).map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
let target = os::read_multipart_source_metadata_in_pinned(
&pinned,
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
if target.len() as u64 != expected_length || raw_sha(&target) != expected_sha256 {
if !fused_in_place {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let current_state = current
.state
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_raw = encode(current_state).map_err(IoError::other)?;
if current.head.is_some() || current_raw != expected_state_raw {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
} else {
os::read_and_sync_multipart_target_metadata(
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
|target| {
if target.len() as u64 != expected_length || raw_sha(target) != expected_sha256 {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
if target_already_synced {
let pinned = os::pin_multipart_source_directory(&target_dir, base, root).map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
Ok(())
},
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
})?;
let target = os::read_multipart_source_metadata_in_pinned(
&pinned,
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
if target.len() as u64 != expected_length || raw_sha(&target) != expected_sha256 {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
})?;
} else {
os::read_and_sync_multipart_target_metadata(
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
|target| {
if target.len() as u64 != expected_length || raw_sha(target) != expected_sha256 {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
Ok(())
},
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
}
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), true)?;
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), !fused_in_place)?;
if let Some((io_path, cleaned)) = staging_cleanup {
match std::fs::remove_dir(&io_path) {
Ok(()) => cleaned.store(true, std::sync::atomic::Ordering::Release),
@@ -2375,7 +2387,7 @@ impl NativeMultipartPublication {
) -> Result<MultipartPublicationResult> {
let this = Arc::clone(self);
os::run_blocking_namespace_operation_with_sync(target_lease, true, move || {
this.finish_on_thread(target_already_synced, staging_cleanup)
this.finish_on_thread(target_already_synced, false, staging_cleanup)
.map_err(IoError::other)
})
.await
+41
View File
@@ -2125,10 +2125,19 @@ pub struct GuardedFileInfoVersions {
pub struct GuardedDeleteVersionsPayload {
pub versions: Vec<GuardedFileInfoVersions>,
pub opts: DeleteOptions,
// The strict legacy decoder rejects this field before applying any change.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub bounded_recovery_protocol: Option<u32>,
}
impl GuardedDeleteVersionsPayload {
pub fn validate(&self) -> Result<()> {
if self
.bounded_recovery_protocol
.is_some_and(|version| version != BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION)
{
return Err(DiskError::UnsupportedDisk);
}
validate_delete_recovery_options(&self.opts)?;
if self.versions.is_empty()
|| self.versions.len() > 1000
@@ -2306,6 +2315,9 @@ pub struct DeleteOptions {
pub delete_recovery_bucket_incarnation: Option<Uuid>,
#[serde(default)]
pub recoverable_delete: bool,
/// Transport requirement for preparing a delete that will use bounded settlement.
#[serde(skip)]
pub requires_bounded_delete_recovery: bool,
/// Reclaim only plain erasure-part leaves, preserving every other entry.
/// Keep `recursive` false so an older peer cannot widen this operation.
#[serde(default)]
@@ -2313,6 +2325,8 @@ pub struct DeleteOptions {
}
pub const DELETE_RECOVERY_PROTOCOL_VERSION: u32 = 1;
/// Accepted only for prepare RPCs. Legacy peers must refuse before staging a delete.
pub const BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION: u32 = 2;
pub(crate) fn get_delete_recovery_timeout() -> Result<std::time::Duration> {
let value = std::env::var("RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS");
match value {
@@ -2909,6 +2923,7 @@ mod tests {
insert_str(&mut fi.metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, "ab".repeat(32));
let generation = CurrentDeleteGeneration::from_file_info(&fi).expect("capture complete remote identity");
let payload = GuardedDeleteVersionsPayload {
bounded_recovery_protocol: None,
versions: vec![GuardedFileInfoVersions {
group: FileInfoVersions {
name: "object".into(),
@@ -2963,6 +2978,7 @@ mod tests {
expected_current: CurrentDeleteGeneration::Absent,
};
let mut payload = GuardedDeleteVersionsPayload {
bounded_recovery_protocol: None,
versions: vec![group],
opts: DeleteOptions {
old_data_dir: Some(Uuid::new_v4()),
@@ -2974,6 +2990,25 @@ mod tests {
let decoded: GuardedDeleteVersionsPayload =
rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode guarded batch");
decoded.validate().expect("round-trip guarded batch remains valid");
#[derive(Debug, Deserialize)]
#[serde(deny_unknown_fields)]
struct LegacyGuardedDeletePayload {
#[serde(rename = "versions")]
_versions: Vec<GuardedFileInfoVersions>,
#[serde(rename = "opts")]
_opts: DeleteOptions,
}
rustfs_protos::decode_delete_recovery_payload::<LegacyGuardedDeletePayload>(&encoded)
.expect("legacy reader accepts legacy prepare");
payload.bounded_recovery_protocol = Some(BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION);
let bounded = rmp_serde::to_vec_named(&payload).expect("serialize bounded guarded prepare");
assert!(rustfs_protos::decode_delete_recovery_payload::<LegacyGuardedDeletePayload>(&bounded).is_err());
let bounded: GuardedDeleteVersionsPayload =
rustfs_protos::decode_delete_recovery_payload(&bounded).expect("current reader accepts bounded prepare");
bounded.validate().expect("valid bounded guarded prepare");
payload.bounded_recovery_protocol = Some(BOUNDED_DELETE_PREPARE_PROTOCOL_VERSION + 1);
assert_eq!(payload.validate(), Err(DiskError::UnsupportedDisk));
payload.bounded_recovery_protocol = None;
let mut trailing = encoded;
trailing.push(0xc0);
assert!(rustfs_protos::decode_delete_recovery_payload::<GuardedDeleteVersionsPayload>(&trailing).is_err());
@@ -3168,9 +3203,15 @@ mod tests {
expected_delete_marker: None,
delete_recovery_bucket_incarnation: None,
recoverable_delete: false,
requires_bounded_delete_recovery: true,
orphan_data_dir: false,
};
let wire = serde_json::to_value(&opts).expect("encode legacy wire options");
assert!(wire.get("requires_bounded_delete_recovery").is_none());
let decoded: DeleteOptions = serde_json::from_value(wire).expect("decode legacy wire options");
assert!(!decoded.requires_bounded_delete_recovery);
assert!(opts.recursive);
assert!(!opts.immediate);
assert!(opts.undo_write);
@@ -1149,6 +1149,8 @@ fn normalized_info(fi: &FileInfo, completion: bool) -> FileInfo {
part.checksums = None;
part.index = None;
part.integrity = None;
// The completed object carries the aggregate proof in metadata.
part.encrypted_etag = false;
}
}
fi
@@ -1739,6 +1741,12 @@ mod multipart_proof_authority_tests {
completed.mod_time = Some(original.mod_time.unwrap() + time::Duration::seconds(1));
completed.size = 15;
completed.add_object_part(1, "etag".into(), 15, completed.mod_time, 15, None, None);
completed.parts[0].encrypted_etag = true;
rustfs_utils::http::metadata_compat::insert_str(
&mut completed.metadata,
rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG,
"v1".to_owned(),
);
let publication = condition.attach(&mut completed).unwrap();
assert_eq!(publication.version_id, None);
assert_eq!(
@@ -1748,6 +1756,11 @@ mod multipart_proof_authority_tests {
);
let mut read_completed = persist_null_fixture(&completed);
assert_eq!(read_completed.version_id, None);
assert!(!read_completed.parts[0].encrypted_etag, "xl.meta stores the aggregate proof");
assert_eq!(
rustfs_utils::http::get_consistent_str(&read_completed.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG),
Some("v1")
);
let reread_publication = reread_condition.attach(&mut read_completed).unwrap();
assert_eq!(
publication, reread_publication,
+504 -138
View File
@@ -610,6 +610,26 @@ pub(crate) mod prepare_rename_test_hooks {
hook();
}
}
/// Hooks keyed by a directory component name, run before a Windows
/// directory pin retries a transient sharing conflict on it.
#[cfg(windows)]
static BEFORE_COMPONENT_RETRY: LazyLock<Mutex<HashMap<std::ffi::OsString, Hook>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
#[cfg(windows)]
pub(crate) fn install_before_component_retry(component: &std::ffi::OsStr, hook: impl FnOnce() + Send + 'static) {
BEFORE_COMPONENT_RETRY.lock().insert(component.to_os_string(), Box::new(hook));
}
#[cfg(windows)]
pub(crate) fn run_before_component_retry(component: &std::ffi::OsStr) {
let hook = BEFORE_COMPONENT_RETRY.lock().remove(component);
if let Some(hook) = hook {
let _pause = super::enter_blocking_test_hook();
hook();
}
}
}
/// Fsync a directory so recently created or renamed entries survive power loss.
@@ -644,6 +664,16 @@ pub(crate) fn fsync_directory_handle_std(dir: &Path, opened: Option<&std::fs::Fi
Ok(())
}
/// Open an existing file to read it and flush it. Windows flushes only a
/// handle with write access, so the file is opened read-write there; other
/// platforms keep the read-only open that `sync_all` already accepts.
pub(crate) fn open_file_for_sync_std(path: &Path) -> io::Result<std::fs::File> {
#[cfg(windows)]
return std::fs::OpenOptions::new().read(true).write(true).open(path);
#[cfg(not(windows))]
std::fs::File::open(path)
}
/// Async wrapper around [`fsync_dir_std`]; runs the blocking Unix fsync off the runtime.
pub async fn fsync_dir(dir: impl AsRef<Path>) -> io::Result<()> {
fsync_dir_with_owner(dir.as_ref(), None).await
@@ -2170,6 +2200,7 @@ pub(crate) const MAX_FSYNC_WORKERS_PER_DISK: usize = 8;
// Return busy workers to the shared pool so other disks can acquire a slot.
const MAX_BLOCKING_JOBS_PER_WORKER: usize = 64;
const WORKER_IDLE_SPIN_ITERS: usize = 256;
const FSYNC_WORKER_IDLE_SPIN_ITERS: usize = 1024;
const FALLBACK_NAMESPACE_QUEUE_SHARDS: usize = 8;
#[derive(Clone, Copy, PartialEq, Eq)]
@@ -2378,7 +2409,7 @@ impl DiskBlockingQueue {
.ready
.store(!state.jobs.is_empty(), std::sync::atomic::Ordering::Relaxed);
if job.is_none() {
if state.idle_workers > 0 {
if lane != DiskBlockingLane::Fsync && state.idle_workers > 0 {
state.active_workers = state.active_workers.saturating_sub(1);
guard.state = DiskBlockingWorkerState::Retired;
return;
@@ -2396,7 +2427,12 @@ impl DiskBlockingQueue {
}
idle_polled = false;
} else {
for _ in 0..WORKER_IDLE_SPIN_ITERS {
let spin_iters = if lane == DiskBlockingLane::Fsync {
FSYNC_WORKER_IDLE_SPIN_ITERS
} else {
WORKER_IDLE_SPIN_ITERS
};
for _ in 0..spin_iters {
if queue.lane(lane).ready.load(std::sync::atomic::Ordering::Relaxed) {
break;
}
@@ -4584,7 +4620,7 @@ pub(crate) fn rename_source_is_missing(src_file_path: &Path, publication_root: &
let Some(source_parent) = src_file_path.parent() else {
return false;
};
let source_parent_guard = match lock_windows_directory_tree(source_parent, None, publication_root) {
let source_parent_guard = match lock_windows_directory_tree(source_parent, publication_root) {
Ok(guard) => guard,
Err(err) => return err.kind() == io::ErrorKind::NotFound,
};
@@ -5023,7 +5059,7 @@ fn prepare_rename_with_retry(
open_windows_rename_source_identity(src_file_path, &source_parent_guard)?;
(source_parent_guard, parent_guard, source_identity_anchor, expected_source_identity)
} else {
let source_parent_guard = lock_windows_directory_tree(source_parent, destination_parent, publication_root)?;
let source_parent_guard = lock_windows_directory_tree(source_parent, publication_root)?;
let (source_identity_anchor, expected_source_identity) =
open_windows_rename_source_identity(src_file_path, &source_parent_guard)?;
let parent_guard = prepare_destination_parent()?;
@@ -5444,7 +5480,9 @@ impl PublicationRoot {
pub(crate) fn open_readonly(&self, path: &Path) -> io::Result<std::fs::File> {
#[cfg(target_os = "linux")]
return self.open_relative(path, rustix::fs::OFlags::RDONLY, rustix::fs::Mode::empty());
#[cfg(not(target_os = "linux"))]
#[cfg(windows)]
return open_windows_readonly(path);
#[cfg(not(any(target_os = "linux", windows)))]
std::fs::File::open(path)
}
@@ -5908,14 +5946,11 @@ impl RenameCommitGuard {
}
/// Reopen a destination tree for handle-relative child publication.
/// Ancestors remain write-exclusive while the final parent shares writes
/// required by the kernel's relative rename. Delete sharing stays omitted
/// throughout, so every retained directory identity remains pinned.
/// Every component is pinned by `open_windows_directory_component`.
fn destination_directory_guard(&self, directory: &Path, create_missing: bool) -> io::Result<RenameDestinationPathGuard> {
#[cfg(windows)]
{
use windows_sys::Wdk::Storage::FileSystem::{FILE_OPEN, FILE_OPEN_IF};
use windows_sys::Win32::Storage::FileSystem::{FILE_SHARE_READ, FILE_SHARE_WRITE};
let relative = directory.strip_prefix(&self.destination_parent).map_err(|_| {
io::Error::new(
@@ -5948,35 +5983,17 @@ impl RenameCommitGuard {
let parent = handles
.last()
.ok_or_else(|| io::Error::other("Windows destination guard lost its parent handle"))?;
let mut relative_components = relative
.components()
.filter_map(|component| match component {
Component::Normal(component) => Some(component),
_ => None,
})
.peekable();
let destination_parent_share = if relative_components.peek().is_none() {
FILE_SHARE_READ | FILE_SHARE_WRITE
} else {
FILE_SHARE_READ
};
handles.push(open_windows_relative_directory_component(
parent,
component,
FILE_OPEN,
destination_parent_share,
)?);
while let Some(component) = relative_components.next() {
handles.push(open_windows_directory_component(parent, component, FILE_OPEN)?);
let relative_components = relative.components().filter_map(|component| match component {
Component::Normal(component) => Some(component),
_ => None,
});
for component in relative_components {
let parent = handles
.last()
.ok_or_else(|| io::Error::other("Windows destination path guard lost its parent handle"))?;
let disposition = if create_missing { FILE_OPEN_IF } else { FILE_OPEN };
let share_access = if relative_components.peek().is_none() {
FILE_SHARE_READ | FILE_SHARE_WRITE
} else {
FILE_SHARE_READ
};
handles.push(open_windows_relative_directory_component(parent, component, disposition, share_access)?);
handles.push(open_windows_directory_component(parent, component, disposition)?);
}
Ok(RenameDestinationPathGuard {
directory: directory.to_path_buf(),
@@ -6020,18 +6037,47 @@ pub(crate) fn multipart_payload_commit_guard_std(
})
}
#[cfg(windows)]
fn multipart_payload_commit_guard_std(
metadata_guard: &RenameCommitGuard,
source_parent: &Path,
publication_root: &PublicationRoot,
) -> io::Result<RenameCommitGuard> {
// Pin the payload staging tree like a distinct-tree metadata source; the
// target stays the exact parent handle the metadata transaction holds.
let source_parent_guard = lock_windows_directory_tree(source_parent, publication_root)?;
Ok(RenameCommitGuard {
source_parent: source_parent.to_path_buf(),
destination_parent: metadata_guard.destination_parent.clone(),
source_parent_guard,
destination_parent_guard: metadata_guard.destination_parent_guard.clone(),
})
}
/// Native MPU metadata and payload have distinct source parents. Reuse the
/// exact target identity already held by the metadata transaction, while
/// independently pinning the private payload source under the same root.
#[cfg(target_os = "linux")]
#[cfg(any(target_os = "linux", windows))]
pub(crate) async fn multipart_payload_commit_guard(
metadata_guard: &RenameCommitGuard,
source_parent: &Path,
publication_root: &PublicationRoot,
lease: Arc<NamespaceMutationLease>,
) -> io::Result<RenameCommitGuard> {
let metadata_guard = metadata_guard.clone();
let source_parent = source_parent.to_path_buf();
run_blocking_namespace_operation(lease, move || multipart_payload_commit_guard_std(&metadata_guard, &source_parent)).await
#[cfg(target_os = "linux")]
let operation = {
// The Linux guard already carries its publication root.
let _ = publication_root;
move || multipart_payload_commit_guard_std(&metadata_guard, &source_parent)
};
#[cfg(windows)]
let operation = {
let publication_root = publication_root.clone();
move || multipart_payload_commit_guard_std(&metadata_guard, &source_parent, &publication_root)
};
run_blocking_namespace_operation(lease, operation).await
}
pub(crate) fn prepare_rename_commit_guard(
@@ -6073,20 +6119,16 @@ pub(crate) fn prepare_rename_commit_guard(
}
#[cfg(windows)]
{
// A same-directory rename must use one shared-write parent handle:
// retaining a second read-only-share handle would block the kernel's
// relative target open. Delete sharing stays excluded, so identity is
// still pinned. Distinct source trees remain strict.
// A same-directory rename reuses one guard for both sides. Every pin
// shares writes and excludes delete; see
// `open_windows_directory_component`.
let same_parent = publication_root.relative_path(source_parent)? == publication_root.relative_path(destination_parent)?;
let (source_parent_guard, destination_parent_guard) = if same_parent {
let destination_parent_guard =
mkdir_all_below_existing_base_std(destination_parent, destination_base, publication_root)?;
(destination_parent_guard.clone(), destination_parent_guard)
} else {
// The source parent also hosts private rollback staging files.
// Their handle-relative publication needs write sharing on this
// final directory while delete sharing remains excluded.
let source_parent_guard = lock_windows_directory_tree(source_parent, Some(source_parent), publication_root)?;
let source_parent_guard = lock_windows_directory_tree(source_parent, publication_root)?;
let destination_parent_guard =
mkdir_all_below_existing_base_std(destination_parent, destination_base, publication_root)?;
(source_parent_guard, destination_parent_guard)
@@ -6106,6 +6148,28 @@ pub(crate) fn prepare_rename_commit_guard(
}
}
/// Unix opens a directory read-only and lets the reader classify it, while
/// Windows refuses the plain open with ACCESS_DENIED. Reopen only an actual
/// directory with backup semantics so the same `is_dir` checks apply.
#[cfg(windows)]
fn open_windows_readonly(path: &Path) -> io::Result<std::fs::File> {
use std::os::windows::fs::OpenOptionsExt;
use windows_sys::Win32::Storage::FileSystem::FILE_FLAG_BACKUP_SEMANTICS;
match std::fs::File::open(path) {
Err(error)
if error.kind() == io::ErrorKind::PermissionDenied
&& std::fs::symlink_metadata(path).is_ok_and(|metadata| metadata.is_dir()) =>
{
std::fs::OpenOptions::new()
.read(true)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS)
.open(path)
}
result => result,
}
}
#[cfg(windows)]
fn open_windows_publication_root(path: &Path) -> io::Result<(PathBuf, WindowsDirectoryHandle)> {
use std::os::windows::fs::OpenOptionsExt;
@@ -6292,22 +6356,12 @@ fn windows_file_attribute_tag(
}
#[cfg(windows)]
fn lock_windows_directory_tree(
path: &Path,
shared_write_ancestor: Option<&Path>,
publication_root: &PublicationRoot,
) -> io::Result<ExistingBaseDirectoryGuard> {
fn lock_windows_directory_tree(path: &Path, publication_root: &PublicationRoot) -> io::Result<ExistingBaseDirectoryGuard> {
use windows_sys::Wdk::Storage::FileSystem::FILE_OPEN;
use windows_sys::Win32::Storage::FileSystem::{FILE_SHARE_READ, FILE_SHARE_WRITE};
let relative = publication_root.relative_path(path)?;
let shared_write_relative = shared_write_ancestor
.map(|ancestor| publication_root.relative_path(ancestor))
.transpose()?
.filter(|ancestor| relative.starts_with(*ancestor));
let mut handles = Vec::with_capacity(relative.components().count().saturating_add(1));
handles.push(publication_root.directory.clone());
let mut opened_relative = PathBuf::new();
for component in relative.components() {
let Component::Normal(component) = component else {
@@ -6322,26 +6376,13 @@ fn lock_windows_directory_tree(
let parent = handles
.last()
.ok_or_else(|| io::Error::other("Windows directory guard lost its root handle"))?;
opened_relative.push(component);
let child = if shared_write_relative.is_some_and(|ancestor| opened_relative.as_path() == ancestor) {
// A handle-relative rename opens the target parent for write. When
// that parent is also a source ancestor, this source-side handle
// must share write access or the transaction blocks itself. Delete
// sharing remains omitted, so the directory identity stays pinned.
open_windows_relative_directory_component(parent, component, FILE_OPEN, FILE_SHARE_READ | FILE_SHARE_WRITE)?
} else {
open_windows_directory_component(parent, component, FILE_OPEN)?
};
handles.push(child);
handles.push(open_windows_directory_component(parent, component, FILE_OPEN)?);
}
Ok(ExistingBaseDirectoryGuard::new(handles))
}
#[cfg(windows)]
// SAFETY: the object attributes borrow a checked UTF-16 component and live
// parent handle for the duration of the synchronous NtCreateFile call.
#[allow(unsafe_code)]
fn open_windows_relative(
parent: &winapi_util::Handle,
component: &std::ffi::OsStr,
@@ -6352,6 +6393,64 @@ fn open_windows_relative(
file_attributes: u32,
dont_reparse: bool,
) -> io::Result<winapi_util::Handle> {
open_windows_relative_with_status(
parent,
component,
&WindowsRelativeOpen {
desired_access,
share_access,
create_disposition,
create_options,
file_attributes,
dont_reparse,
},
)
.map_err(|failure| failure.error)
}
/// `NtCreateFile` arguments for one handle-relative open.
#[cfg(windows)]
struct WindowsRelativeOpen {
desired_access: u32,
share_access: u32,
create_disposition: u32,
create_options: u32,
file_attributes: u32,
dont_reparse: bool,
}
/// A failed relative open that keeps the NTSTATUS, which tells transient
/// states such as a pending delete apart from errors that share its Win32 code.
#[cfg(windows)]
struct WindowsRelativeOpenError {
status: Option<windows_sys::Win32::Foundation::NTSTATUS>,
error: io::Error,
}
#[cfg(windows)]
impl From<io::Error> for WindowsRelativeOpenError {
fn from(error: io::Error) -> Self {
Self { status: None, error }
}
}
#[cfg(windows)]
// SAFETY: the object attributes borrow a checked UTF-16 component and live
// parent handle for the duration of the synchronous NtCreateFile call.
#[allow(unsafe_code)]
fn open_windows_relative_with_status(
parent: &winapi_util::Handle,
component: &std::ffi::OsStr,
request: &WindowsRelativeOpen,
) -> std::result::Result<winapi_util::Handle, WindowsRelativeOpenError> {
let &WindowsRelativeOpen {
desired_access,
share_access,
create_disposition,
create_options,
file_attributes,
dont_reparse,
} = request;
use std::{
mem::size_of,
os::windows::{ffi::OsStrExt, io::AsRawHandle, io::FromRawHandle},
@@ -6366,10 +6465,7 @@ fn open_windows_relative(
let mut name = component.encode_wide().collect::<Vec<_>>();
if name.is_empty() || name.contains(&0) {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
"relative Windows file name is empty or contains a NUL",
));
return Err(io::Error::new(io::ErrorKind::InvalidInput, "relative Windows file name is empty or contains a NUL").into());
}
let name_bytes = name
.len()
@@ -6409,13 +6505,17 @@ fn open_windows_relative(
)
};
if status < 0 {
return match i32::try_from(unsafe { RtlNtStatusToDosError(status) }) {
Ok(code) => Err(io::Error::from_raw_os_error(code)),
Err(_) => Err(io::Error::other(format!("Windows relative open failed with NTSTATUS {status:#x}"))),
let error = match i32::try_from(unsafe { RtlNtStatusToDosError(status) }) {
Ok(code) => io::Error::from_raw_os_error(code),
Err(_) => io::Error::other(format!("Windows relative open failed with NTSTATUS {status:#x}")),
};
return Err(WindowsRelativeOpenError {
status: Some(status),
error,
});
}
if handle.is_null() {
return Err(io::Error::other("Windows relative open returned an invalid handle"));
return Err(io::Error::other("Windows relative open returned an invalid handle").into());
}
Ok(unsafe { winapi_util::Handle::from_raw_handle(handle) })
@@ -6650,7 +6750,7 @@ pub(crate) fn compare_and_update_control_file(
let parent = file_path
.parent()
.ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "conditional file has no parent"))?;
let parent_guard = lock_windows_directory_tree(parent, Some(parent), publication_root)?;
let parent_guard = lock_windows_directory_tree(parent, publication_root)?;
let lock = open_windows_relative(
parent_guard.last_handle()?,
std::ffi::OsStr::new(".rustfs-cas.lock"),
@@ -6714,41 +6814,78 @@ pub(crate) fn compare_and_update_control_file(
Ok(ConditionalFileUpdate::Updated)
}
/// How often a directory pin retries a transient delete-access conflict.
/// Six doubling delays from 1 ms wait about 63 ms in total, which outlasts a
/// prune whose thread is preempted while it holds delete access.
#[cfg(windows)]
const WINDOWS_DIRECTORY_PIN_RETRIES: u32 = 6;
#[cfg(windows)]
const WINDOWS_DIRECTORY_PIN_FIRST_RETRY_DELAY: std::time::Duration = std::time::Duration::from_millis(1);
/// Pin one directory component of a guarded path.
///
/// The handle shares reads and writes but never delete: renaming or removing
/// a directory needs delete access to it, so omitting delete sharing pins its
/// identity. Write sharing must stay granted on every component, ancestors
/// included, because the kernel opens the target parent for write during a
/// rename. A directory pinned here as one object's ancestor is also the
/// publication parent of a nested object (`a/b` holds `a/b/c`), and a
/// read-only share would make that unrelated rename fail with
/// ERROR_SHARING_VIOLATION. Write sharing does not unpin an ancestor: a
/// pinned ancestor always contains its pinned child, and NTFS sets a
/// name-surrogate reparse point only on an empty directory.
///
/// Excluding delete also means the open fails while another handle holds
/// delete access. Pruning an empty prefix directory takes that access only
/// briefly: `RemoveDirectoryW` opens the directory for delete and closes it
/// again, whether or not the directory turns out to be empty. A sharing
/// violation or a pending delete is therefore retried a bounded number of
/// times. After a removal finishes, the retry sees the directory as missing
/// and follows `create_disposition`, so `FILE_OPEN` still reports it missing
/// and `FILE_OPEN_IF` creates it again.
#[cfg(windows)]
fn open_windows_directory_component(
parent: &WindowsDirectoryHandle,
component: &std::ffi::OsStr,
create_disposition: u32,
) -> io::Result<WindowsDirectoryHandle> {
use windows_sys::Win32::Storage::FileSystem::FILE_SHARE_READ;
open_windows_relative_directory_component(parent, component, create_disposition, FILE_SHARE_READ)
}
#[cfg(windows)]
fn open_windows_relative_directory_component(
parent: &WindowsDirectoryHandle,
component: &std::ffi::OsStr,
create_disposition: u32,
share_access: u32,
) -> io::Result<WindowsDirectoryHandle> {
use windows_sys::{
Wdk::Storage::FileSystem::{FILE_DIRECTORY_FILE, FILE_OPEN_REPARSE_POINT},
Win32::Storage::FileSystem::{
FILE_ATTRIBUTE_DIRECTORY, FILE_ATTRIBUTE_REPARSE_POINT, FILE_READ_ATTRIBUTES, FILE_TRAVERSE,
Win32::{
Foundation::{STATUS_DELETE_PENDING, STATUS_SHARING_VIOLATION},
Storage::FileSystem::{
FILE_ATTRIBUTE_DIRECTORY, FILE_ATTRIBUTE_REPARSE_POINT, FILE_READ_ATTRIBUTES, FILE_SHARE_READ, FILE_SHARE_WRITE,
FILE_TRAVERSE,
},
},
};
let anchor = open_windows_relative(
&parent.handle,
component,
FILE_TRAVERSE | FILE_READ_ATTRIBUTES,
share_access,
let request = WindowsRelativeOpen {
desired_access: FILE_TRAVERSE | FILE_READ_ATTRIBUTES,
share_access: FILE_SHARE_READ | FILE_SHARE_WRITE,
create_disposition,
FILE_DIRECTORY_FILE | FILE_OPEN_REPARSE_POINT,
FILE_ATTRIBUTE_DIRECTORY,
true,
)?;
create_options: FILE_DIRECTORY_FILE | FILE_OPEN_REPARSE_POINT,
file_attributes: FILE_ATTRIBUTE_DIRECTORY,
dont_reparse: true,
};
let mut retry_delay = WINDOWS_DIRECTORY_PIN_FIRST_RETRY_DELAY;
let mut retries = 0;
let anchor = loop {
match open_windows_relative_with_status(&parent.handle, component, &request) {
Ok(anchor) => break anchor,
Err(WindowsRelativeOpenError {
status: Some(STATUS_SHARING_VIOLATION | STATUS_DELETE_PENDING),
..
}) if retries < WINDOWS_DIRECTORY_PIN_RETRIES => {
#[cfg(test)]
prepare_rename_test_hooks::run_before_component_retry(component);
std::thread::sleep(retry_delay);
retry_delay = retry_delay.saturating_mul(2);
retries += 1;
}
Err(failure) => return Err(failure.error),
}
};
let info = windows_file_attribute_tag(&anchor)?;
if info.FileAttributes & FILE_ATTRIBUTE_DIRECTORY == 0 {
return Err(io::Error::from(io::ErrorKind::NotADirectory));
@@ -7052,7 +7189,7 @@ pub(crate) fn pin_multipart_source_directory(
#[cfg(windows)]
{
let _ = base;
lock_windows_directory_tree(directory, Some(directory), publication_root)
lock_windows_directory_tree(directory, publication_root)
}
#[cfg(not(any(unix, windows)))]
{
@@ -7142,12 +7279,31 @@ pub(crate) fn rename_multipart_source_owned_std(
{
let _ = &guard;
let source_guard = pin_multipart_source_directory(source_parent, base, root)?;
// The prepared Windows source holds DELETE access and shares only
// read, so the precondition's own opens of that entry (or, for a
// directory, of its guarded tree) would fail with a sharing violation.
// Check it first under a non-exclusive identity anchor, then require
// the exclusive handle to name the same file.
#[cfg(windows)]
let (_source_identity_anchor, expected_source_identity) = {
let anchor = open_windows_rename_source_identity(source, &source_guard)?;
precondition()?;
anchor
};
let preparation = prepare_rename_with_retry(source, destination, base, root)?;
#[cfg(windows)]
if windows_file_identity(&preparation.source)? != expected_source_identity {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
"rename source identity changed after its precondition",
));
}
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
{
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, source);
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, destination);
}
#[cfg(not(windows))]
precondition()?;
#[cfg(unix)]
validate_prepared_rename_source(&pinned_source, source)?;
@@ -7293,6 +7449,7 @@ pub(crate) fn install_multipart_prepared_head(
canonical_head: &Path,
base: &Path,
publication_root: &PublicationRoot,
directory_guard: Option<&RenameCommitGuard>,
) -> io::Result<PreparedRenameSource> {
let original_path = prepared.path.clone();
let parent = canonical_head
@@ -7301,9 +7458,37 @@ pub(crate) fn install_multipart_prepared_head(
if original_path.parent() != Some(parent) || original_path == canonical_head {
return Err(io::Error::new(io::ErrorKind::InvalidInput, "MPU proxy must be a private sibling"));
}
#[cfg(target_os = "linux")]
let guard = if let Some(directory_guard) = directory_guard.filter(|g| g.directories.source_path == parent) {
let dir = &directory_guard.directories.source;
RenameCommitGuard {
directories: Arc::new(LinuxCommitDirectories {
root: directory_guard.directories.root.clone(),
source_path: parent.to_path_buf(),
destination_path: parent.to_path_buf(),
source: LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
},
destination: std::sync::OnceLock::from(LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
}),
destination_base: base.to_path_buf(),
}),
}
} else {
prepare_rename_commit_guard(parent, parent, base, publication_root)?
};
#[cfg(not(target_os = "linux"))]
let _pinned = pin_multipart_source_directory(parent, base, publication_root)?;
let guard = prepare_rename_commit_guard(parent, parent, base, publication_root)?;
#[cfg(not(target_os = "linux"))]
let guard = {
let _ = directory_guard;
prepare_rename_commit_guard(parent, parent, base, publication_root)?
};
rename_prepared_source_std(&prepared, &original_path, canonical_head, base, publication_root, &guard)?;
prepared.path = canonical_head.to_path_buf();
fsync_commit_directory_limited_std(parent, &guard)?;
@@ -7553,12 +7738,15 @@ fn read_multipart_metadata(
let parent = path
.parent()
.ok_or_else(|| io::Error::other("multipart metadata has no parent"))?;
let guard = prepare_rename_commit_guard(parent, parent, base, publication_root)?;
// A read must not recreate a retired upload: pin the existing parent
// instead of taking the write-side guard, which creates missing parents.
existing_base_directory_suffix(parent, base)?;
let guard = pin_multipart_source_directory(parent, base, publication_root)?;
let name = path
.file_name()
.ok_or_else(|| io::Error::other("multipart metadata has no name"))?;
let mut file = open_windows_relative(
guard.source_parent_guard.last_handle()?,
guard.last_handle()?,
name,
SYNCHRONIZE | FILE_READ_ATTRIBUTES | FILE_READ_DATA,
FILE_SHARE_READ,
@@ -7734,7 +7922,7 @@ mod multipart_source_preservation_tests {
let metadata_guard =
prepare_rename_commit_guard(&canonical, &target, &bucket, &publication).expect("canonical metadata guard");
let lease = acquire_namespace_mutation_lease(&target).await;
let payload_guard = multipart_payload_commit_guard(&metadata_guard, &temporary, lease.clone())
let payload_guard = multipart_payload_commit_guard(&metadata_guard, &temporary, &publication, lease.clone())
.await
.expect("distinct pinned payload source");
assert!(
@@ -7769,6 +7957,61 @@ mod multipart_source_preservation_tests {
);
}
#[cfg(windows)]
#[tokio::test]
async fn windows_multipart_payload_guard_publishes_distinct_source_into_metadata_target() {
// Build paths from the configured root: the resolved handle path may
// be verbatim, where `/` in a joined key is not a separator.
let (root, publication, _source, _destination) = fixture();
let root = root.path();
let canonical = root.join("multipart/upload");
let temporary = root.join("tmp/attempt");
let bucket = root.join("bucket");
let target = bucket.join("object");
for directory in [&canonical, &temporary, &target] {
std::fs::create_dir_all(directory).expect("two-source guard directories");
}
let payload = temporary.join("payload");
std::fs::create_dir(&payload).expect("private payload DD");
std::fs::write(payload.join("part.1"), b"native selected shard").expect("private payload shard");
let head = canonical.join("xl.meta");
let destination = target.join("xl.meta");
let metadata_guard =
prepare_rename_commit_guard(&canonical, &target, &bucket, &publication).expect("canonical metadata guard");
let lease = acquire_namespace_mutation_lease(&target).await;
assert!(
rename_all_with_commit_guard(&payload, target.join("payload"), &bucket, &publication, &metadata_guard, lease.clone())
.await
.is_err(),
"the metadata guard does not pin the private payload parent"
);
assert!(
payload.join("part.1").exists(),
"a rejected payload rename must leave its source in place"
);
let payload_guard = multipart_payload_commit_guard(&metadata_guard, &temporary, &publication, lease.clone())
.await
.expect("distinct pinned payload source");
let mut prepared = create_prepared_rename_source_with_commit_guard(&head, &destination, &metadata_guard)
.expect("canonical prepared metadata");
prepared
.write_all(b"prepared native metadata", true)
.expect("durable canonical metadata");
rename_all_with_commit_guard(&payload, target.join("payload"), &bucket, &publication, &payload_guard, lease.clone())
.await
.expect("payload rename uses its own pinned TMP parent");
assert!(head.exists(), "payload publication must not consume the canonical head");
rename_all_with_prepared_source(prepared, &head, &destination, &bucket, &publication, &metadata_guard, lease)
.await
.expect("metadata rename keeps its canonical parent");
assert!(!head.exists());
assert_eq!(std::fs::read(&destination).expect("published metadata"), b"prepared native metadata");
assert_eq!(
std::fs::read(target.join("payload/part.1")).expect("published selected shard"),
b"native selected shard"
);
}
#[cfg(target_os = "linux")]
#[tokio::test]
async fn multipart_payload_guard_rejects_replaced_private_source_or_shared_destination() {
@@ -7790,7 +8033,7 @@ mod multipart_source_preservation_tests {
let metadata_guard =
prepare_rename_commit_guard(&canonical, &target, &bucket, &publication).expect("canonical metadata guard");
let lease = acquire_namespace_mutation_lease(&target).await;
let payload_guard = multipart_payload_commit_guard(&metadata_guard, &temporary, lease.clone())
let payload_guard = multipart_payload_commit_guard(&metadata_guard, &temporary, &publication, lease.clone())
.await
.expect("pin payload and shared target");
let replaced = if replace_source { &temporary } else { &target };
@@ -8047,7 +8290,6 @@ pub(crate) fn mkdir_all_below_existing_base_std(
#[cfg(windows)]
{
use windows_sys::Wdk::Storage::FileSystem::{FILE_OPEN, FILE_OPEN_IF};
use windows_sys::Win32::Storage::FileSystem::{FILE_SHARE_READ, FILE_SHARE_WRITE};
let base_relative = publication_root.relative_path(base_dir)?;
let capacity = base_relative
@@ -8058,13 +8300,12 @@ pub(crate) fn mkdir_all_below_existing_base_std(
let mut handles = Vec::with_capacity(capacity);
handles.push(publication_root.directory.clone());
let mut guard = ExistingBaseDirectoryGuard::new(handles);
let mut components = base_relative
let components = base_relative
.components()
.map(|component| (component, FILE_OPEN))
.chain(relative.components().map(|component| (component, FILE_OPEN_IF)))
.filter(|(component, _)| !matches!(component, Component::CurDir))
.peekable();
while let Some((component, disposition)) = components.next() {
.filter(|(component, _)| !matches!(component, Component::CurDir));
for (component, disposition) in components {
let Component::Normal(component) = component else {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
@@ -8075,22 +8316,16 @@ pub(crate) fn mkdir_all_below_existing_base_std(
.handles
.last()
.ok_or_else(|| io::Error::other("Windows publication root guard is empty"))?;
// The kernel opens the final parent for write during a relative
// rename. Share writes from its first open: a temporary read-only
// share would block another rename into the same trash directory.
// Ancestors stay strict and no handle shares delete access, keeping
// every retained directory identity pinned.
let share_access = if components.peek().is_none() {
FILE_SHARE_READ | FILE_SHARE_WRITE
} else {
FILE_SHARE_READ
};
let child = open_windows_relative_directory_component(parent, component, disposition, share_access)?;
// Every component, including the final parent the kernel opens
// for write during a relative rename, shares writes from its first
// open; see `open_windows_directory_component`.
let child = open_windows_directory_component(parent, component, disposition)?;
guard.handles.push(child);
#[cfg(test)]
if components.peek().is_none() {
prepare_rename_test_hooks::run_after_component_opened(dir_path);
}
}
// The hook observes the window right after the final component opened.
#[cfg(test)]
if guard.handles.len() > 1 {
prepare_rename_test_hooks::run_after_component_opened(dir_path);
}
Ok(guard)
@@ -9496,7 +9731,7 @@ mod tests {
use std::os::windows::fs::OpenOptionsExt;
use std::sync::atomic::{AtomicBool, Ordering};
use windows_sys::Win32::{
Foundation::{ERROR_SHARING_VIOLATION, GENERIC_WRITE},
Foundation::ERROR_SHARING_VIOLATION,
Storage::FileSystem::{
DELETE, FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_OPEN_REPARSE_POINT, FILE_SHARE_DELETE, FILE_SHARE_READ,
FILE_SHARE_WRITE,
@@ -9538,13 +9773,13 @@ mod tests {
let second_preparation = prepare_rename_with_retry(&sources[1], &destinations[1], base, &publication_root)
.expect("prepare the second trash rename");
assert!(interleaved.load(Ordering::Acquire), "the competing parent-open window must be exercised");
for (path, access) in [(&tmp, GENERIC_WRITE), (&trash, DELETE)] {
for path in [&tmp, &trash] {
let err = std::fs::OpenOptions::new()
.access_mode(access)
.access_mode(DELETE)
.share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(path)
.expect_err("ancestor writes and final-parent deletion must remain excluded");
.expect_err("pinned ancestors and final parents must exclude deletion");
assert_eq!(
err.raw_os_error(),
Some(i32::try_from(ERROR_SHARING_VIOLATION).expect("Windows error code must fit i32"))
@@ -9561,6 +9796,115 @@ mod tests {
}
}
#[cfg(windows)]
#[test]
fn windows_rename_into_a_pinned_ancestor_succeeds_while_deletion_stays_excluded() {
use std::os::windows::fs::OpenOptionsExt;
use windows_sys::Win32::{
Foundation::ERROR_SHARING_VIOLATION,
Storage::FileSystem::{
DELETE, FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_OPEN_REPARSE_POINT, FILE_SHARE_DELETE, FILE_SHARE_READ,
FILE_SHARE_WRITE,
},
};
// `bucket/parent` is an object directory and also the prefix of the
// nested object `bucket/parent/child`. A guard on the nested object
// pins `parent` as an ancestor while the parent object publishes a new
// data directory into it by pathname.
let temp_dir = tempdir().expect("create temp dir");
let base = temp_dir.path().join("bucket");
let parent = base.join("parent");
let child = parent.join("child");
std::fs::create_dir_all(&child).expect("create nested object directory");
let staged = temp_dir.path().join("staged-data-dir");
std::fs::create_dir(&staged).expect("create staged data directory");
std::fs::write(staged.join("part.1"), b"payload").expect("write staged part");
let publication_root = PublicationRoot::new(temp_dir.path()).expect("open publication root");
let child_guard =
super::mkdir_all_below_existing_base_std(&child, &base, &publication_root).expect("pin the nested object tree");
let destination = parent.join(uuid::Uuid::new_v4().to_string());
std::fs::rename(&staged, &destination).expect("a pinned ancestor must accept a rename into it");
assert_eq!(std::fs::read(destination.join("part.1")).expect("read published part"), b"payload");
let err = std::fs::OpenOptions::new()
.access_mode(DELETE)
.share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(&parent)
.expect_err("the pinned ancestor must still refuse deletion");
assert_eq!(
err.raw_os_error(),
Some(i32::try_from(ERROR_SHARING_VIOLATION).expect("Windows error code must fit i32"))
);
drop(child_guard);
}
#[cfg(windows)]
#[test]
fn windows_directory_pin_retries_a_transient_delete_holder_and_fails_closed_on_a_lasting_one() {
use std::os::windows::fs::OpenOptionsExt;
use windows_sys::Win32::{
Foundation::ERROR_SHARING_VIOLATION,
Storage::FileSystem::{
DELETE, FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_OPEN_REPARSE_POINT, FILE_READ_ATTRIBUTES, FILE_SHARE_DELETE,
FILE_SHARE_READ, FILE_SHARE_WRITE, SYNCHRONIZE,
},
};
// Open the prefix the way `RemoveDirectoryW` does while it prunes an
// object prefix: delete access, every share mode granted.
let open_like_a_pruner = |path: &Path| {
std::fs::OpenOptions::new()
.access_mode(DELETE | SYNCHRONIZE | FILE_READ_ATTRIBUTES)
.share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(path)
.expect("open the prefix for delete")
};
let temp_dir = tempdir().expect("create temp dir");
let base = temp_dir.path().join("bucket");
let publication_root = PublicationRoot::new(temp_dir.path()).expect("open publication root");
let transient_name = uuid::Uuid::new_v4().to_string();
let transient_prefix = base.join(&transient_name);
std::fs::create_dir_all(&transient_prefix).expect("create transient prefix");
let holder = Arc::new(std::sync::Mutex::new(Some(open_like_a_pruner(&transient_prefix))));
let holder_for_hook = Arc::clone(&holder);
let released = Arc::new(std::sync::atomic::AtomicBool::new(false));
let released_for_hook = Arc::clone(&released);
prepare_rename_test_hooks::install_before_component_retry(std::ffi::OsStr::new(&transient_name), move || {
holder_for_hook.lock().expect("holder mutex poisoned").take();
released_for_hook.store(true, std::sync::atomic::Ordering::Release);
});
let child = transient_prefix.join("child");
let guard = super::mkdir_all_below_existing_base_std(&child, &base, &publication_root)
.expect("a pin must outlast a transient delete holder");
assert!(
released.load(std::sync::atomic::Ordering::Acquire),
"the pin must have met the delete holder"
);
assert!(child.is_dir(), "the retried walk must still create the child");
let err = std::fs::OpenOptions::new()
.access_mode(DELETE)
.share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(&transient_prefix)
.expect_err("the retried pin must still exclude deletion");
assert_eq!(err.raw_os_error(), i32::try_from(ERROR_SHARING_VIOLATION).ok());
drop(guard);
let lasting_prefix = base.join(uuid::Uuid::new_v4().to_string());
std::fs::create_dir_all(&lasting_prefix).expect("create lasting prefix");
let lasting_holder = open_like_a_pruner(&lasting_prefix);
let Err(err) = super::mkdir_all_below_existing_base_std(&lasting_prefix.join("child"), &base, &publication_root) else {
panic!("a pin must not wait forever on a lasting delete holder");
};
assert_eq!(err.raw_os_error(), i32::try_from(ERROR_SHARING_VIOLATION).ok());
drop(lasting_holder);
}
#[cfg(windows)]
#[tokio::test]
async fn windows_rename_all_supports_same_parent_publication() {
@@ -10258,7 +10602,7 @@ mod tests {
let publication_root = test_publication_root(&[&src, &dst, &base]);
let source_parent_guard =
lock_windows_directory_tree(src.parent().expect("source reparse entry must have a parent"), None, &publication_root)
lock_windows_directory_tree(src.parent().expect("source reparse entry must have a parent"), &publication_root)
.expect("pin the source parent tree");
let identity_error = match open_windows_rename_source_identity(&src, &source_parent_guard) {
Ok(_) => panic!("a non-dedup source reparse entry must be rejected by its tag"),
@@ -10292,30 +10636,51 @@ mod tests {
#[cfg(windows)]
#[tokio::test]
async fn windows_rename_all_excludes_source_parent_reparse_writers() {
async fn windows_rename_all_keeps_the_source_parent_from_becoming_a_reparse_point() {
use std::os::windows::fs::OpenOptionsExt;
use windows_sys::Win32::{
Foundation::GENERIC_WRITE,
Storage::FileSystem::{FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_OPEN_REPARSE_POINT, FILE_SHARE_READ, FILE_SHARE_WRITE},
Foundation::{ERROR_DIR_NOT_EMPTY, ERROR_SHARING_VIOLATION, GENERIC_WRITE},
Storage::FileSystem::{
DELETE, FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_OPEN_REPARSE_POINT, FILE_SHARE_DELETE, FILE_SHARE_READ,
FILE_SHARE_WRITE,
},
};
let temp_dir = tempdir().expect("create temp dir");
let base = temp_dir.path().join("bucket");
let source_parent = temp_dir.path().join("staging");
let outside = temp_dir.path().join("outside");
let src = source_parent.join("staged-object");
let dst = base.join("published-object");
std::fs::create_dir(&base).expect("create destination base");
std::fs::create_dir(&source_parent).expect("create source parent");
std::fs::create_dir(&outside).expect("create outside target");
std::fs::write(&src, b"original").expect("write original staged object");
let source_parent_for_hook = source_parent.clone();
let outside_for_hook = outside.clone();
windows_rename_test_hooks::install_before_publication(&dst, move || {
std::fs::OpenOptions::new()
// The pin shares writes, so a writer can open the source parent,
// but the parent still holds the staged entry and NTFS refuses to
// turn a non-empty directory into a mount point.
let writable_parent = std::fs::OpenOptions::new()
.access_mode(GENERIC_WRITE)
.share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(&source_parent_for_hook)
.expect_err("the source-parent guard must exclude a reparse writer");
.map(winapi_util::Handle::from_file)
.expect("a write-sharing pin admits a concurrent writer");
let err = try_set_windows_mount_point(&writable_parent, &outside_for_hook)
.expect_err("the pinned source parent must not become a mount point");
assert_eq!(err.raw_os_error(), i32::try_from(ERROR_DIR_NOT_EMPTY).ok());
let err = std::fs::OpenOptions::new()
.access_mode(DELETE)
.share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)
.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT)
.open(&source_parent_for_hook)
.expect_err("the source-parent guard must exclude deletion and rename");
assert_eq!(err.raw_os_error(), i32::try_from(ERROR_SHARING_VIOLATION).ok());
});
rename_all(&src, &dst, &base)
@@ -10324,6 +10689,7 @@ mod tests {
assert!(!src.exists(), "the original staged entry must be moved");
assert_eq!(std::fs::read(&dst).expect("read published original object"), b"original");
assert!(!outside.join("published-object").exists(), "publication must not escape the base");
}
#[cfg(windows)]
@@ -10403,7 +10769,7 @@ mod tests {
let guard = mkdir_all_below_existing_base_std(&base, &base).expect("guard destination base");
let publication_root = test_publication_root(&[&src, &dst, &base]);
let source_parent_guard =
lock_windows_directory_tree(src.parent().expect("source path must have a parent"), None, &publication_root)
lock_windows_directory_tree(src.parent().expect("source path must have a parent"), &publication_root)
.expect("anchor source parent");
let source = open_windows_rename_source(&src, &source_parent_guard).expect("anchor source entry");
@@ -3636,4 +3636,73 @@ mod tests {
assert_eq!(encode_channel_capacity(16 * 1024 * 1024, 32 * 1024 * 1024), 2);
assert_eq!(encode_channel_capacity(1, usize::MAX), DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BLOCKS);
}
#[tokio::test]
async fn read_full_buf_or_eof_preserves_invalid_data_and_wraps_truncated_io_errors() {
use std::pin::Pin;
use std::task::{Context, Poll};
use tokio::io::{AsyncRead, ReadBuf};
struct StepReader {
prefix: Vec<u8>,
offset: usize,
err_kind: Option<std::io::ErrorKind>,
err_msg: &'static str,
}
impl AsyncRead for StepReader {
fn poll_read(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
if self.offset < self.prefix.len() {
let take = (self.prefix.len() - self.offset).min(buf.remaining());
buf.put_slice(&self.prefix[self.offset..self.offset + take]);
self.offset += take;
return Poll::Ready(Ok(()));
}
if let Some(kind) = self.err_kind.take() {
return Poll::Ready(Err(std::io::Error::new(kind, self.err_msg)));
}
Poll::Ready(Ok(()))
}
}
// 1. Error on the very first read (total == 0) preserves the original ErrorKind.
let mut r1 = StepReader {
prefix: Vec::new(),
offset: 0,
err_kind: Some(std::io::ErrorKind::BrokenPipe),
err_msg: "immediate broken pipe",
};
let mut buf = BytesMut::with_capacity(16);
let err = read_full_buf_or_eof(&mut r1, &mut buf, 16)
.await
.expect_err("immediate read error must fail");
assert_eq!(err.kind(), std::io::ErrorKind::BrokenPipe);
// 2. InvalidData (e.g. HashReader checksum/trailer mismatch) after partial fill preserves InvalidData.
let mut r2 = StepReader {
prefix: b"partial".to_vec(),
offset: 0,
err_kind: Some(std::io::ErrorKind::InvalidData),
err_msg: "Checksum mismatch",
};
let mut buf2 = BytesMut::with_capacity(16);
let err = read_full_buf_or_eof(&mut r2, &mut buf2, 16)
.await
.expect_err("InvalidData after partial fill must fail");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("Checksum mismatch"));
// 3. Non-InvalidData error after partial fill is normalized to UnexpectedEof.
let mut r3 = StepReader {
prefix: b"partial".to_vec(),
offset: 0,
err_kind: Some(std::io::ErrorKind::BrokenPipe),
err_msg: "mid-stream reset",
};
let mut buf3 = BytesMut::with_capacity(16);
let err = read_full_buf_or_eof(&mut r3, &mut buf3, 16)
.await
.expect_err("truncated stream error must fail");
assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof);
}
}
@@ -934,4 +934,124 @@ mod tests {
"cached record must still reject corrupt or mismatched shard payloads"
);
}
#[tokio::test]
async fn shard_verifier_rejects_out_of_bounds_index_and_empty_proof_sources_fail_closed() {
let mut builder = IntegrityBuilder::new(IntegrityLayout::new(2, 2, 8, false).expect("layout"), 1).expect("builder");
builder
.push([b"abcd".as_slice(), b"efgh", b"ijkl", b"mnop"].into_iter())
.await
.expect("stripe");
let prepared = builder.finish(8).await.expect("index");
let proof = Arc::new(PartProofReader {
part: prepared.part,
sources: Vec::new(),
volume: "bucket".to_owned(),
inline: None,
cached: Mutex::new(ProofCache::default()),
});
// coding_index >= layout.shards() is rejected at construction time.
let err = ShardVerifier::new(proof.clone(), 4, 0, None)
.err()
.expect("coding_index out of bounds must fail");
assert_eq!(err.kind(), io::ErrorKind::InvalidData);
assert!(err.to_string().contains("shard integrity verification failed"));
// Valid coding_index with neither inline bytes nor disk sources fails closed when reading stripe record.
let mut verifier = ShardVerifier::new(proof, 0, 0, None).expect("verifier");
let err = verifier
.verify(b"abcd")
.await
.expect_err("empty proof sources with no inline index must fail closed");
assert_eq!(err.kind(), io::ErrorKind::InvalidData);
assert!(err.to_string().contains("shard integrity verification failed"));
}
#[tokio::test]
async fn spilled_index_write_verify_and_restore_proof_replicas_round_trip() {
let (_dirs, store) = crate::services::rebalance::test_store_with_persisted_rebalance_meta(Default::default()).await;
let mut disk_slots = store.pools[0].disk_set[0].get_local_disks().await;
let disks: Vec<_> = disk_slots.iter().flatten().cloned().collect();
assert_eq!(disks.len(), 2);
let volume = "spilled-integrity-bucket";
for disk in &disks {
disk.make_volume(volume).await.expect("make volume");
}
// 2 shards (1 data + 1 parity) matches the 2-disk fixture; 2350 stripes * 448B > 1 MiB spills to Spool::File(_).
let layout = IntegrityLayout::new(1, 1, 8, false).expect("layout");
let mut builder = IntegrityBuilder::new(layout, 1).expect("builder");
let shards = [b"abcdefgh".as_slice(); 2];
for _ in 0..2350 {
builder.push(shards.into_iter()).await.expect("stripe");
}
let part_size = 2350 * 8;
let prepared = builder.finish(part_size).await.expect("spilled index");
assert!(matches!(prepared.index, Spool::File(_)));
let data_dir = uuid::Uuid::new_v4();
let object = "object";
let directory = format!("{object}/{data_dir}");
let part = prepared
.write(&mut disk_slots, volume, volume, &directory)
.await
.expect("write spilled index");
let rel_path = format!("{directory}/{}", part.file_name());
for disk in &disks {
verify_index_file(disk, volume, &rel_path, &part)
.await
.expect("verify spilled index on disk");
}
// Corrupt replica 0; restore_proof_replicas should repair it from healthy replica 1.
disks[0]
.write_all(volume, &rel_path, bytes::Bytes::from_static(b"corrupted-index"))
.await
.expect("corrupt disk 0");
assert!(verify_index_file(&disks[0], volume, &rel_path, &part).await.is_err());
let mut fi = rustfs_filemeta::FileInfo {
volume: volume.to_owned(),
name: object.to_owned(),
data_dir: Some(data_dir),
erasure: rustfs_filemeta::ErasureInfo {
data_blocks: 1,
parity_blocks: 1,
block_size: 8,
..Default::default()
},
parts: vec![rustfs_filemeta::ObjectPartInfo {
number: 1,
size: part_size,
actual_size: part_size as i64,
integrity: Some(part.clone()),
..Default::default()
}],
..Default::default()
};
fi.persist_shard_integrity().expect("persist shard integrity");
let repaired = restore_proof_replicas(&fi, &disk_slots, volume, object)
.await
.expect("restore proof replica 0");
assert_eq!(repaired, 1);
verify_index_file(&disks[0], volume, &rel_path, &part)
.await
.expect("repaired replica 0 must verify");
// Negative durability boundary: when all replicas are corrupted, restore_proof_replicas must fail closed.
for disk in &disks {
disk.write_all(volume, &rel_path, bytes::Bytes::from_static(b"all-corrupted"))
.await
.expect("corrupt all replicas");
}
let err = restore_proof_replicas(&fi, &disk_slots, volume, object)
.await
.expect_err("all corrupted proof replicas must fail closed");
assert!(
err.to_string().contains("shard integrity verification failed"),
"unexpected error when all proof replicas are corrupt: {err}"
);
}
}
+29 -12
View File
@@ -478,6 +478,7 @@ struct ScannerPublicationCommitScopeInner {
#[derive(Clone)]
pub struct ScannerPublicationCommitScope {
inner: Arc<ScannerPublicationCommitScopeInner>,
owns_completion: bool,
}
/// RAII fallback for storage paths that return before their commit closure
@@ -569,6 +570,16 @@ impl ScannerPublicationCommitScope {
movement_permit: Mutex::new(Some(movement_permit)),
lease_release_safe,
}),
owns_completion: true,
}
}
/// A pool may use the shared fence, but only the enclosing operation may
/// report completion and authorize release of its leases.
pub(crate) fn pool_delete_participant(&self) -> Self {
Self {
inner: Arc::clone(&self.inner),
owns_completion: false,
}
}
@@ -607,7 +618,9 @@ impl ScannerPublicationCommitScope {
/// storage mutation owner. Once set, dropping a scanner waiter leaves the
/// scope in-flight until that owner reports committed or indeterminate.
pub fn attach_mutation_owner(&self) {
self.inner.owner_attached.store(true, Ordering::Release);
if self.owns_completion {
self.inner.owner_attached.store(true, Ordering::Release);
}
}
fn owner_attached(&self) -> bool {
@@ -654,16 +667,17 @@ impl ScannerPublicationCommitScope {
}
pub fn mark_aborted_before_commit(&self) -> bool {
if self
.inner
.state
.compare_exchange(
SCANNER_PUBLICATION_SCOPE_ADMITTED,
SCANNER_PUBLICATION_SCOPE_ABORTED_BEFORE_COMMIT,
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
if self.owns_completion
&& self
.inner
.state
.compare_exchange(
SCANNER_PUBLICATION_SCOPE_ADMITTED,
SCANNER_PUBLICATION_SCOPE_ABORTED_BEFORE_COMMIT,
Ordering::AcqRel,
Ordering::Acquire,
)
.is_ok()
{
self.inner.lease_release_safe.store(true, Ordering::Release);
self.inner.completed.notify_waiters();
@@ -677,6 +691,9 @@ impl ScannerPublicationCommitScope {
}
fn mark_terminal(&self, terminal: ScannerPublicationCommitState) -> bool {
if !self.owns_completion {
return false;
}
self.inner
.state
.compare_exchange(SCANNER_PUBLICATION_SCOPE_IN_FLIGHT, terminal.as_u8(), Ordering::AcqRel, Ordering::Acquire)
@@ -709,7 +726,7 @@ impl ScannerPublicationCommitScope {
/// Release the storage-owned movement permit only after a known-safe
/// terminal result. Returns `false` for in-flight or indeterminate work.
pub async fn release_movement_permit(&self) -> bool {
if !self.state().permits_lease_release() {
if !self.owns_completion || !self.state().permits_lease_release() {
return false;
}
self.inner.movement_permit.lock().await.take().is_some()
+132 -1
View File
@@ -58,7 +58,7 @@ use std::sync::{
Arc, OnceLock,
atomic::{AtomicBool, AtomicU8, AtomicU64, Ordering},
};
use tokio::sync::{Mutex, Notify, OnceCell, OwnedRwLockReadGuard, RwLock};
use tokio::sync::{Mutex, Notify, OnceCell, OwnedRwLockReadGuard, OwnedSemaphorePermit, RwLock, Semaphore};
use tokio::time::Instant;
use tokio_util::sync::CancellationToken;
use tokio_util::task::TaskTracker;
@@ -187,6 +187,38 @@ impl DecommissionCountLedger {
}
}
#[derive(Debug)]
struct ListReadBudget {
permits: Arc<Semaphore>,
capacity: usize,
}
impl ListReadBudget {
fn new(capacity: usize) -> Self {
let capacity = capacity
.min(Semaphore::MAX_PERMITS)
.min(usize::try_from(u32::MAX).unwrap_or(usize::MAX));
Self {
permits: Arc::new(Semaphore::new(capacity)),
capacity,
}
}
async fn acquire(&self, fanout: usize) -> Result<Option<Arc<OwnedSemaphorePermit>>, tokio::sync::AcquireError> {
if self.capacity == 0 {
return Ok(None);
}
// A single walk must remain admissible even when the topology exceeds
// the configured budget. Tokio accepts at most u32::MAX in one grant.
let weight = u32::try_from(fanout.max(1).min(self.capacity)).unwrap_or(u32::MAX);
self.permits
.clone()
.acquire_many_owned(weight)
.await
.map(|permit| Some(Arc::new(permit)))
}
}
/// Runtime state owned by a single `ECStore` instance.
///
/// This is intentionally minimal in the first migration slice; subsequent
@@ -201,6 +233,7 @@ pub struct InstanceContext {
/// drift out of sync). Stored as one value so the three predicates can
/// never observe a torn intermediate state.
erasure_kind: RwLock<SetupType>,
list_read_budget: OnceLock<ListReadBudget>,
/// This instance's namespace lock manager (Phase 5 Slice 3, backlog#939).
///
/// Owned per-instance so two instances no longer share a lock namespace
@@ -354,6 +387,24 @@ impl InstanceContext {
Self::with_lock_manager(Arc::new(GlobalLockManager::new()))
}
/// Bound native listing fanout independently of GET body permits. A walk
/// keeps its shared grant until all of its disk producers have terminated.
/// Acquire before creating the merge channels or taking object locks.
pub(crate) async fn acquire_list_read_budget(
&self,
fanout: usize,
) -> Result<Option<Arc<OwnedSemaphorePermit>>, tokio::sync::AcquireError> {
self.list_read_budget
.get_or_init(|| {
ListReadBudget::new(rustfs_utils::get_env_usize(
rustfs_config::ENV_OBJECT_MAX_CONCURRENT_DISK_READS,
rustfs_config::DEFAULT_OBJECT_MAX_CONCURRENT_DISK_READS,
))
})
.acquire(fanout)
.await
}
/// Build a context bound to a specific lock manager.
///
/// [`bootstrap_ctx`] uses this to alias the process-global lock manager so
@@ -362,6 +413,7 @@ impl InstanceContext {
fn with_lock_manager(lock_manager: Arc<GlobalLockManager>) -> Self {
Self {
erasure_kind: RwLock::new(SetupType::Unknown),
list_read_budget: OnceLock::new(),
lock_manager,
region: OnceLock::new(),
deployment_id: OnceLock::new(),
@@ -1111,6 +1163,85 @@ pub fn bootstrap_ctx() -> Arc<InstanceContext> {
mod tests {
use super::*;
#[tokio::test]
async fn list_read_budget_weights_and_clamps_fanout() {
let budget = ListReadBudget::new(8);
let first = budget.acquire(3).await.expect("first walk").expect("bounded grant");
assert_eq!(budget.permits.available_permits(), 5);
let mut blocked = std::pin::pin!(budget.acquire(6));
assert!(futures::poll!(&mut blocked).is_pending());
drop(first);
let second = blocked.await.expect("queued walk").expect("bounded grant");
assert_eq!(budget.permits.available_permits(), 2);
drop(second);
let oversized = budget
.acquire(usize::MAX)
.await
.expect("oversized walk")
.expect("bounded grant");
assert_eq!(budget.permits.available_permits(), 0);
drop(oversized);
assert_eq!(budget.permits.available_permits(), 8);
}
#[tokio::test]
async fn list_read_budget_zero_preserves_unbounded_configuration() {
let budget = ListReadBudget::new(0);
assert!(budget.acquire(usize::MAX).await.expect("disabled budget").is_none());
}
#[tokio::test]
async fn list_read_budget_cancelled_waiter_returns_partial_grant() {
let budget = ListReadBudget::new(4);
let held = budget.acquire(3).await.expect("first walk");
{
let mut waiter = std::pin::pin!(budget.acquire(4));
assert!(futures::poll!(&mut waiter).is_pending());
}
assert_eq!(budget.permits.available_permits(), 1);
drop(held);
assert!(budget.acquire(4).await.expect("next walk").is_some());
}
#[tokio::test]
async fn list_read_budget_shared_grant_survives_parent_drop() {
let budget = ListReadBudget::new(4);
let parent = budget.acquire(4).await.expect("first walk").expect("bounded grant");
let producer = parent.clone();
drop(parent);
assert_eq!(budget.permits.available_permits(), 0);
drop(producer);
assert_eq!(budget.permits.available_permits(), 4);
}
#[tokio::test]
async fn list_read_budget_is_shared_by_context_and_isolated_between_instances() {
let first = Arc::new(InstanceContext::new());
first.list_read_budget.set(ListReadBudget::new(4)).expect("fresh context");
let same = first.clone();
let other = InstanceContext::new();
other.list_read_budget.set(ListReadBudget::new(4)).expect("fresh context");
let held = first.acquire_list_read_budget(4).await.expect("first context");
let mut waiter = std::pin::pin!(same.acquire_list_read_budget(1));
assert!(futures::poll!(&mut waiter).is_pending());
assert!(
other
.acquire_list_read_budget(4)
.await
.expect("independent context")
.is_some()
);
drop(held);
assert!(waiter.await.expect("shared context resumed").is_some());
}
#[test]
fn list_read_budget_extreme_configuration_is_bounded_without_panicking() {
let budget = ListReadBudget::new(usize::MAX);
assert!(budget.capacity <= Semaphore::MAX_PERMITS);
assert!(u32::try_from(budget.capacity).is_ok());
}
#[test]
fn decommission_count_ledger_bounds_retained_entries() {
let mut ledger = super::DecommissionCountLedger::default();
@@ -385,3 +385,165 @@ async fn pause_and_cancel_are_durable_during_content_read() {
JobState::Cancelled
);
}
#[test]
fn job_request_validation_and_protection_classification_cover_all_negative_branches() {
use crate::object_api::ObjectInfo;
let valid_digest = rustfs_rio::Checksum::new_from_data(rustfs_rio::ChecksumType::SHA256, b"payload")
.expect("sha256")
.encoded;
// 1. Empty items and > MAX_ITEMS (64).
let mut empty_req = request(JobMode::Audit, valid_digest.clone());
empty_req.items.clear();
let err = empty_req.validate().expect_err("empty items must fail");
assert!(
err.to_string().contains("a job requires 1..=64 explicit source versions"),
"unexpected error: {err}"
);
let mut too_many = request(JobMode::Audit, valid_digest.clone());
too_many.items = (0..65)
.map(|i| ItemRequest {
key: format!("key-{i}"),
version_id: None,
expected_sha256: None,
target_key: None,
})
.collect();
let err = too_many.validate().expect_err(">64 items must fail");
assert!(
err.to_string().contains("a job requires 1..=64 explicit source versions"),
"unexpected error: {err}"
);
// 2. bytes_per_second and max_object_bytes bounds.
let mut low_bps = request(JobMode::Audit, valid_digest.clone());
low_bps.bytes_per_second = 64 * 1024 - 1;
let err = low_bps.validate().expect_err("low bytes_per_second must fail");
assert!(err.to_string().contains("invalid byte rate or staging limit"), "unexpected error: {err}");
let mut high_bps = request(JobMode::Audit, valid_digest.clone());
high_bps.bytes_per_second = 1024 * 1024 * 1024 + 1;
let err = high_bps.validate().expect_err("high bytes_per_second must fail");
assert!(err.to_string().contains("invalid byte rate or staging limit"), "unexpected error: {err}");
let mut zero_max = request(JobMode::Audit, valid_digest.clone());
zero_max.max_object_bytes = 0;
let err = zero_max.validate().expect_err("zero max_object_bytes must fail");
assert!(err.to_string().contains("invalid byte rate or staging limit"), "unexpected error: {err}");
let mut huge_max = request(JobMode::Audit, valid_digest.clone());
huge_max.max_object_bytes = model::MAX_OBJECT_BYTES + 1;
let err = huge_max.validate().expect_err("huge max_object_bytes must fail");
assert!(err.to_string().contains("invalid byte rate or staging limit"), "unexpected error: {err}");
// 3. Invalid keys (empty, trailing '/', embedded '\0', > 1024 chars).
for bad_key in ["", "trailing/", "null\0byte", &"x".repeat(1025)] {
let mut req = request(JobMode::Audit, valid_digest.clone());
req.items[0].key = bad_key.to_string();
let err = req.validate().expect_err("invalid key must fail");
assert!(
err.to_string().contains("invalid object key"),
"expected 'invalid object key' for {bad_key:?}, got {err}"
);
}
// 4. Invalid version_id (nil UUID, malformed UUID, empty string) and valid UUID.
for bad_ver in ["", "not-a-uuid", &Uuid::nil().to_string()] {
let mut req = request(JobMode::Audit, valid_digest.clone());
req.items[0].version_id = Some(bad_ver.to_string());
let err = req.validate().expect_err("invalid version_id must fail");
assert!(
err.to_string().contains("invalid source version"),
"expected 'invalid source version' for {bad_ver:?}, got {err}"
);
}
let mut valid_uuid_ver = request(JobMode::Audit, valid_digest.clone());
valid_uuid_ver.items[0].version_id = Some(Uuid::new_v4().to_string());
assert!(valid_uuid_ver.validate().is_ok());
// 5. Invalid expected_sha256 (bad base64, wrong byte length).
let err = request(JobMode::Audit, "!!!not_base64!!!".to_string())
.validate()
.expect_err("non-base64 sha256 must fail");
assert!(err.to_string().contains("SHA256 must be canonical Base64"), "unexpected error: {err}");
let err = request(JobMode::Audit, "AAAA".to_string())
.validate()
.expect_err("wrong-length sha256 must fail");
assert!(
err.to_string().contains("SHA256 must be canonical Base64 of 32 bytes"),
"unexpected error: {err}"
);
// 6. Mode-specific target_key rules (Audit forbids target_key; Migrate requires unique target_key != key).
let mut audit_with_target = request(JobMode::Audit, valid_digest.clone());
audit_with_target.items[0].target_key = Some("target".to_string());
let err = audit_with_target.validate().expect_err("audit with target_key must fail");
assert!(
err.to_string().contains("only migrations require an explicit target key"),
"unexpected error: {err}"
);
let mut migrate_no_target = request(JobMode::Migrate, valid_digest.clone());
migrate_no_target.items[0].target_key = None;
let err = migrate_no_target
.validate()
.expect_err("migrate without target_key must fail");
assert!(
err.to_string().contains("only migrations require an explicit target key"),
"unexpected error: {err}"
);
let mut migrate_same_target = request(JobMode::Migrate, valid_digest.clone());
migrate_same_target.items[0].target_key = Some("source".to_string());
let err = migrate_same_target
.validate()
.expect_err("migrate target == source must fail");
assert!(
err.to_string()
.contains("targets must be distinct from every source and target"),
"unexpected error: {err}"
);
let mut migrate_dup_target = request(JobMode::Migrate, valid_digest);
migrate_dup_target.items.push(ItemRequest {
key: "source-2".into(),
version_id: None,
expected_sha256: None,
target_key: Some("target".into()),
});
let err = migrate_dup_target.validate().expect_err("duplicate migrate target must fail");
assert!(
err.to_string()
.contains("targets must be distinct from every source and target"),
"unexpected error: {err}"
);
// 7. model::unsupported and model::protection classifications.
let mut delete_marker = ObjectInfo {
delete_marker: true,
..Default::default()
};
assert_eq!(model::unsupported(&delete_marker), Some("delete_marker"));
assert_eq!(model::protection(&delete_marker), Protection::NoPayload);
delete_marker.delete_marker = false;
let mut transitioned = ObjectInfo::default();
transitioned.transitioned_object.status = "Complete".to_string();
assert_eq!(model::unsupported(&transitioned), Some("transitioned_object"));
let mut encrypted = ObjectInfo::default();
Arc::make_mut(&mut encrypted.user_defined).insert("x-amz-server-side-encryption".to_string(), "AES256".to_string());
assert_eq!(model::unsupported(&encrypted), Some("transformed_object"));
let mut invalid_decl = ObjectInfo::default();
rustfs_utils::http::insert_str(
Arc::make_mut(&mut invalid_decl.user_defined),
rustfs_filemeta::shard_integrity::SUFFIX_SHARD_INTEGRITY,
"not-json".to_string(),
);
assert_eq!(model::unsupported(&invalid_decl), Some("invalid_protection_declaration"));
assert_eq!(model::protection(&invalid_decl), Protection::InvalidDeclaration);
}
@@ -1910,6 +1910,10 @@ mod tests {
&ObjectOptions {
version_id: Some(version_id.to_string()),
mod_time: earlier(version.mod_time),
// Without the object lock the rename waits for every
// drive, so no early-ACK tail is still writing the copies
// removed below.
no_lock: true,
..versioned
},
))
@@ -2250,6 +2254,195 @@ mod tests {
.expect("large-stack rebalance test thread should complete");
}
#[test]
#[serial_test::serial]
fn real_rebalance_preserves_encrypted_multipart_etag_proof() {
run_large_stack_async_test("rebalance-encrypted-multipart-etag", || async {
use crate::io_support::rio::{HashReader, WriteEncryption, WritePlan};
use rustfs_utils::http::{SSEC_ALGORITHM_HEADER, SUFFIX_OPAQUE_ENCRYPTED_ETAG, etag_can_be_listed};
const REBALANCE_ID: &str = "rebalance-encrypted-multipart-etag";
for (object, versioned, protected) in [
("protected-version", true, true),
("protected-null", false, true),
("legacy-version", true, false),
("legacy-null", false, false),
] {
// Each case needs a fresh progress counter: the fixture's small
// capacity can reach its rebalance goal after one encrypted object.
let (_temp_dirs, store, _peer) =
crate::services::rebalance::test_two_pool_stores_with_isolated_node_contexts(None).await;
let bucket = "rebalance-encrypted-etag";
Box::pin(store.make_bucket(
bucket,
&MakeBucketOptions {
versioning_enabled: versioned,
..Default::default()
},
))
.await
.expect("create the migration bucket");
prepare_rebalance_test_volumes(&store).await;
let incarnation = store.bucket_incarnation_id(bucket).await.expect("bucket incarnation");
let rebalance_meta = active_rebalance_meta(REBALANCE_ID);
rebalance_meta
.save(Arc::clone(&store.pools[0]))
.await
.expect("persist active rebalance");
*store.rebalance_meta.write().await = Some(rebalance_meta);
let source_set = store.pools[0].get_disks_by_key(object);
let target_set = store.pools[1].get_disks_by_key(object);
let write_opts = ObjectOptions {
versioned,
user_defined: std::collections::HashMap::from([(SSEC_ALGORITHM_HEADER.to_owned(), "AES256".to_owned())]),
..Default::default()
};
let upload = Box::pin(source_set.new_multipart_upload(bucket, object, &write_opts))
.await
.expect("create source encrypted multipart upload");
let plaintext = b"encrypted multipart bytes must survive pool migration";
let size = i64::try_from(plaintext.len()).expect("small fixture length");
let source = HashReader::from_stream(std::io::Cursor::new(plaintext.to_vec()), size, size, None, None, false)
.expect("plaintext source reader");
let encrypted = WritePlan::new()
.with_encryption(WriteEncryption::multipart([7; 32], [9; 12], 1))
.apply(source, size)
.expect("encrypt the source part through the production writer");
let mut part_opts = write_opts.clone();
if !protected {
// Model a retained legacy fingerprint without producer proof.
part_opts.preserve_etag = Some("0123456789abcdef0123456789abcdef".to_owned());
}
let part = Box::pin(source_set.put_object_part(
bucket,
object,
&upload.upload_id,
1,
&mut PutObjReader::new(encrypted),
&part_opts,
))
.await
.expect("persist source encrypted part");
let written = Box::pin(Arc::clone(&source_set).complete_multipart_upload(
bucket,
object,
&upload.upload_id,
vec![CompletePart {
part_num: part.part_num,
etag: part.etag,
..Default::default()
}],
&write_opts,
))
.await
.expect("complete source encrypted multipart upload");
assert_eq!(written.version_id.is_some(), versioned);
assert!(written.is_multipart(), "the actual migration must use its multipart path");
assert_eq!(etag_can_be_listed(&written.user_defined), protected);
let read_opts = ObjectOptions {
version_id: Some(data_movement::data_movement_exact_version_id(written.version_id)),
data_movement: true,
raw_data_movement_read: true,
..Default::default()
};
let source_bytes = {
let mut reader = Box::pin(source_set.get_object_reader(bucket, object, None, HeaderMap::new(), &read_opts))
.await
.expect("read source ciphertext");
let mut bytes = Vec::new();
reader
.stream
.read_to_end(&mut bytes)
.await
.expect("collect source ciphertext");
bytes
};
assert_ne!(source_bytes.as_slice(), plaintext, "the fixture must contain encrypted bytes");
let entry = metacache_entry_from_source(&source_set, bucket, object).await;
let outcome = tokio::time::timeout(
StdDuration::from_secs(60),
Box::pin(Arc::clone(&store).rebalance_entry(
RebalanceEntryTarget {
bucket: bucket.to_owned(),
pool_index: 0,
},
entry,
Arc::clone(&source_set),
Arc::new(RebalanceBucketConfigs {
bucket_incarnation_id: Some(incarnation),
..Default::default()
}),
Arc::from(REBALANCE_ID),
CancellationToken::new(),
)),
)
.await
.expect("rebalance must finish within the test deadline")
.expect("rebalance source entry");
let target = match target_set.get_object_info(bucket, object, &read_opts).await {
Ok(target) => target,
Err(err) => {
let target_versions = target_set
.load_file_info_versions_exact(bucket, object)
.await
.expect("inspect target after metadata read failure")
.map(|versions| {
versions
.versions
.into_iter()
.map(|version| version.version_id)
.collect::<Vec<_>>()
});
let source_versions = source_set
.load_file_info_versions_exact(bucket, object)
.await
.expect("inspect source after metadata read failure")
.map(|versions| {
versions
.versions
.into_iter()
.map(|version| version.version_id)
.collect::<Vec<_>>()
});
panic!(
"read migrated {object}: {err}; outcome={outcome:?}, expected_version={:?}, target_versions={target_versions:?}, source_versions={source_versions:?}",
written.version_id
);
}
};
assert_eq!(target.version_id, written.version_id, "{object}");
assert_eq!(target.etag, written.etag, "{object}");
assert_eq!(
rustfs_utils::http::get_consistent_str(&target.user_defined, SUFFIX_OPAQUE_ENCRYPTED_ETAG),
protected.then_some("v1"),
"aggregate ETag proof must follow the exact source version for {object}"
);
assert_eq!(etag_can_be_listed(&target.user_defined), protected, "{object}");
assert_eq!(outcome, RebalanceEntryOutcome::Completed, "{object}");
let mut target_reader =
Box::pin(target_set.get_object_reader(bucket, object, None, HeaderMap::new(), &read_opts))
.await
.expect("read target ciphertext");
let mut target_bytes = Vec::new();
target_reader
.stream
.read_to_end(&mut target_bytes)
.await
.expect("collect target ciphertext");
assert_eq!(target_bytes, source_bytes, "rebalance preserves ciphertext bytes for {object}");
assert!(
source_set
.load_file_info_versions_exact(bucket, object)
.await
.expect("inspect source after migration")
.is_none(),
"source cleanup follows a verified target for {object}"
);
}
});
}
const TRANSITIONED_REBALANCE_ID: &str = "rebalance-transitioned-source-cleanup";
/// One transitioned source version on pool 0 of a two-pool store with an
+1
View File
@@ -26,6 +26,7 @@ pub(crate) mod tier_probe_intent;
pub mod warm_backend;
pub mod warm_backend_aliyun;
pub mod warm_backend_azure;
pub mod warm_backend_azure_blob;
#[cfg(feature = "gcs")]
pub mod warm_backend_gcs;
pub mod warm_backend_huaweicloud;
+147 -6
View File
@@ -327,6 +327,7 @@ pub const TIER_CONFIG_VERSION: u16 = 2;
const EXTERNAL_TIER_TYPE_UNSUPPORTED: i32 = 0;
const EXTERNAL_TIER_TYPE_S3: i32 = 1;
const EXTERNAL_TIER_TYPE_AZURE: i32 = 2;
const EXTERNAL_TIER_AZURE_BLOB_ENDPOINT_SENTINEL: &str = "rustfs-azure-blob-v1:";
const EXTERNAL_TIER_TYPE_GCS: i32 = 3;
const EXTERNAL_TIER_TYPE_MINIO: i32 = 4;
const EXTERNAL_TIER_VERSION_WASABI_V1: &str = "rustfs-wasabi-v1";
@@ -2147,6 +2148,7 @@ fn build_coordinator_tier_mutation_intent(
return Ok(None);
}
Ok(Some(TierMutationIntent {
requires_native_azure: candidate.tiers.values().any(TierConfig::is_native_azure),
mutation_id: uuid::Uuid::new_v4(),
revision: 1,
kind,
@@ -2515,11 +2517,11 @@ fn tier_backend_identity(config: &TierConfig) -> io::Result<TierDestinationId> {
}),
TierType::Azure => config.azure.as_ref().map(|value| {
(
"azure",
if config.is_native_azure() { "azure-blob-v1" } else { "azure" },
value.endpoint.as_str(),
value.bucket.as_str(),
value.prefix.as_str(),
value.region.as_str(),
if config.is_native_azure() { "" } else { value.region.as_str() },
value.access_key.as_str(),
)
}),
@@ -2592,7 +2594,7 @@ fn normalized_tier_prefix<'a>(tier_type: &TierType, prefix: &'a str) -> &'a str
}
fn tier_exact_get_delete(config: &TierConfig) -> bool {
ProviderVersionCapabilities::for_tier_type(&config.tier_type.as_lowercase()).exact_get_delete
config.is_native_azure() || ProviderVersionCapabilities::for_tier_type(&config.tier_type.as_lowercase()).exact_get_delete
}
struct TierDriverGeneration {
@@ -3130,7 +3132,15 @@ fn to_external_tier_config(name: &str, tier: &TierConfig) -> io::Result<External
.ok_or_else(|| io::Error::other("tier config missing azure backend payload"))?;
out.tier_type = EXTERNAL_TIER_TYPE_AZURE;
out.azure = Some(ExternalTierAzure {
endpoint: az.endpoint.clone(),
// Keep the v2 Azure payload readable and rewriteable by older
// servers, but make their S3 constructor reject its endpoint.
// An unknown numeric type would discard the Azure payload on
// an unrelated configuration rewrite by an older server.
endpoint: if tier.is_native_azure() {
format!("{EXTERNAL_TIER_AZURE_BLOB_ENDPOINT_SENTINEL}{}", az.endpoint)
} else {
az.endpoint.clone()
},
account_name: az.access_key.clone(),
account_key: az.secret_key.clone(),
bucket: az.bucket.clone(),
@@ -3289,6 +3299,15 @@ fn from_external_tier_config(name: String, ext: ExternalTierConfig) -> io::Resul
..Default::default()
};
let azure_blob = ext.version == super::tier_config::AZURE_BLOB_TIER_VERSION;
let native_endpoint = ext
.azure
.as_ref()
.is_some_and(|azure| azure.endpoint.starts_with(EXTERNAL_TIER_AZURE_BLOB_ENDPOINT_SENTINEL));
if azure_blob != native_endpoint || azure_blob && (ext.tier_type != EXTERNAL_TIER_TYPE_AZURE || ext.tier_type_hint.is_some())
{
return Err(io::Error::other("native Azure Blob tier has inconsistent protocol discriminators"));
}
let wasabi_hint = ext.tier_type_hint.as_deref() == Some("wasabi");
let wasabi_version = ext.version == EXTERNAL_TIER_VERSION_WASABI_V1;
if wasabi_hint && !wasabi_version {
@@ -3309,7 +3328,9 @@ fn from_external_tier_config(name: String, ext: ExternalTierConfig) -> io::Resul
cfg.name
)));
}
let tier_type = if wasabi_version {
let tier_type = if azure_blob {
TierType::Azure
} else if wasabi_version {
TierType::Wasabi
} else {
tier_type_from_hint(ext.tier_type_hint.as_deref()).unwrap_or(match ext.tier_type {
@@ -3387,7 +3408,15 @@ fn from_external_tier_config(name: String, ext: ExternalTierConfig) -> io::Resul
.ok_or_else(|| io::Error::other(format!("tier config '{}' missing azure backend payload", cfg.name)))?;
cfg.azure = Some(crate::services::tier::tier_config::TierAzure {
name: cfg.name.clone(),
endpoint: az.endpoint.clone(),
endpoint: if azure_blob {
az.endpoint
.strip_prefix(EXTERNAL_TIER_AZURE_BLOB_ENDPOINT_SENTINEL)
.filter(|endpoint| !endpoint.is_empty())
.ok_or_else(|| io::Error::other("native Azure Blob endpoint marker is missing"))?
.to_string()
} else {
az.endpoint.clone()
},
access_key: az.account_name.clone(),
secret_key: az.account_key.clone(),
bucket: az.bucket.clone(),
@@ -3558,6 +3587,9 @@ impl TierConfigMgr {
}
pub fn unmarshal(data: &[u8]) -> std::result::Result<TierConfigMgr, std::io::Error> {
if data.starts_with(&TIER_CONFIG_FORMAT.to_le_bytes()) {
return decode_external_tiering_config_blob(data);
}
let mut cfg: TierConfigMgr = serde_json::from_slice(data)?;
for (name, tier) in &mut cfg.tiers {
if matches!(&tier.tier_type, TierType::Wasabi) {
@@ -3575,6 +3607,11 @@ impl TierConfigMgr {
}
pub fn marshal(&self) -> std::result::Result<Bytes, std::io::Error> {
// The legacy JSON shape omits the driver protocol version. Native
// Azure must retain the binary discriminator even through this helper.
if self.tiers.values().any(TierConfig::is_native_azure) {
return encode_external_tiering_config_blob(self);
}
let data = serde_json::to_vec(&self)?;
let mut data = Bytes::from(data);
@@ -3608,6 +3645,17 @@ impl TierConfigMgr {
deadline: Option<Instant>,
) -> std::result::Result<(), AdminError> {
normalize_s3_gcs_add_tier_name(&mut tier_config)?;
if tier_config.tier_type == TierType::Azure {
tier_config.version = super::tier_config::AZURE_BLOB_TIER_VERSION.to_string();
if let Some(azure) = tier_config.azure.as_mut() {
if azure.storage_class.eq_ignore_ascii_case("STANDARD") {
azure.storage_class.clear();
}
if azure.endpoint.is_empty() {
azure.endpoint = format!("https://{}.blob.core.windows.net", azure.access_key);
}
}
}
self.ensure_generation_is_idle(&tier_config.name)?;
let tier_name = tier_config.name.clone();
if tier_name != tier_name.to_uppercase() {
@@ -9536,6 +9584,98 @@ mod tests {
assert_eq!(az.sp_auth.client_secret, "secret");
}
#[tokio::test]
async fn native_azure_add_preserves_protocol_through_restart_and_requires_peer_support() {
let mut manager = empty_mgr();
let mut tier = build_azure_tier("account");
let name = tier.name.clone();
let azure = tier.azure.as_mut().expect("Azure payload should exist");
azure.endpoint.clear();
azure.storage_class = "STANDARD".to_string();
azure.secret_key = base64_simd::STANDARD.encode_to_string([7; 32]);
TIER_DRIVER_TEST_FACTORY
.scope(healthy_driver_factory(), manager.add(tier, true))
.await
.expect("current Console Azure payload should be admitted");
let created = &manager.tiers[&name];
assert!(created.is_native_azure());
assert!(tier_exact_get_delete(created));
let azure = created.azure.as_ref().expect("Azure payload should survive");
assert_eq!(azure.endpoint, "https://account.blob.core.windows.net");
assert!(azure.storage_class.is_empty());
let bytes = manager
.marshal()
.expect("native configuration should serialize without losing its protocol");
let external: ExternalTierConfigMgr =
rmp_serde::from_slice(&bytes[4..]).expect("native config should use the binary shape");
assert_eq!(external.tiers[&name].tier_type, EXTERNAL_TIER_TYPE_AZURE);
let stored = external.tiers[&name]
.azure
.as_ref()
.expect("native storage keeps the existing Azure payload");
assert_eq!(stored.endpoint, "rustfs-azure-blob-v1:https://account.blob.core.windows.net");
let legacy_backend_config = crate::services::tier::tier_config::TierAzure {
endpoint: stored.endpoint.clone(),
access_key: stored.account_name.clone(),
secret_key: stored.account_key.clone(),
bucket: stored.bucket.clone(),
..Default::default()
};
assert!(
crate::services::tier::warm_backend_azure::WarmBackendAzure::new(&legacy_backend_config, &name)
.await
.is_err(),
"the unchanged legacy S3 constructor must reject the native endpoint before any request"
);
assert!(
external.tiers[&name].tier_type_hint.is_none(),
"the protocol marker must not be overridden by an extended provider hint"
);
let restarted = TierConfigMgr::unmarshal(&bytes).expect("native configuration should reload");
assert!(restarted.tiers[&name].is_native_azure());
assert_eq!(
tier_backend_identity(created).expect("created identity"),
tier_backend_identity(&restarted.tiers[&name]).expect("reloaded identity")
);
let targets = vec![TierMutationIntentTarget {
tier_name: name,
old_backend_identity: None,
new_backend_identity: Some(tier_backend_identity(created).expect("native destination identity")),
}];
let intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &manager, targets)
.expect("native intent should build")
.expect("native Add must use peer prepare");
assert!(intent.requires_native_azure);
}
#[test]
fn native_azure_protocol_is_distinct_from_legacy_and_rejects_corrupt_discriminators() {
let legacy = build_azure_tier("account");
let mut native = legacy.clone_with_credentials();
native.version = super::super::tier_config::AZURE_BLOB_TIER_VERSION.to_string();
assert_ne!(
tier_backend_identity(&legacy).expect("legacy identity"),
tier_backend_identity(&native).expect("native identity"),
"native Blob must never acquire a legacy S3 destination's cleanup authority"
);
let stored = to_external_tier_config(&legacy.name, &legacy).expect("legacy encoding");
assert_eq!(stored.tier_type, EXTERNAL_TIER_TYPE_AZURE);
let decoded = from_external_tier_config(legacy.name, stored).expect("legacy decoding");
assert!(!decoded.is_native_azure());
assert!(!tier_exact_get_delete(&decoded));
let stored = to_external_tier_config(&native.name, &native).expect("native encoding");
for mutate in [
|config: &mut ExternalTierConfig| config.version = "v1".to_string(),
|config: &mut ExternalTierConfig| config.tier_type = EXTERNAL_TIER_TYPE_S3,
|config: &mut ExternalTierConfig| config.tier_type_hint = Some("azure".to_string()),
] {
let mut corrupt = stored.clone();
mutate(&mut corrupt);
assert!(from_external_tier_config(native.name.clone(), corrupt).is_err());
}
}
#[test]
fn test_external_blob_gcs_writer_stays_raw_for_v2_old_readers() {
const RAW_JSON: &str = r#"{"type":"service_account","project_id":"tier-௿"}"#;
@@ -9944,6 +10084,7 @@ mod tests {
fn prepared_remove_intent(tier_name: &str, mutation_id: uuid::Uuid) -> TierMutationIntent {
TierMutationIntent {
requires_native_azure: false,
mutation_id,
revision: 1,
kind: TierMutationIntentKind::Remove,
@@ -139,6 +139,7 @@ impl TierType {
}
pub(crate) const TIER_CREDENTIAL_REDACTED: &str = "REDACTED";
pub(crate) const AZURE_BLOB_TIER_VERSION: &str = "rustfs-azure-blob-v1";
#[derive(Default, Serialize, Deserialize)]
#[serde(default)]
@@ -178,6 +179,10 @@ impl Clone for TierConfig {
}
impl TierConfig {
pub(crate) fn is_native_azure(&self) -> bool {
self.tier_type == TierType::Azure && self.version == AZURE_BLOB_TIER_VERSION
}
pub(crate) fn redacted(&self) -> Self {
let mut redacted = Self {
version: self.version.clone(),
@@ -127,6 +127,10 @@ impl TierMutationIntentTarget {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub(crate) struct TierMutationIntent {
// Old peers deny unknown intent fields before installing a prepare block.
// Omit this extension for existing drivers to retain their wire checksum.
#[serde(default, skip_serializing_if = "is_false")]
pub requires_native_azure: bool,
pub mutation_id: Uuid,
pub revision: u64,
pub kind: TierMutationIntentKind,
@@ -282,7 +286,8 @@ impl TierMutationIntent {
}
pub(crate) fn same_identity_as(&self, other: &Self) -> bool {
self.mutation_id == other.mutation_id
self.requires_native_azure == other.requires_native_azure
&& self.mutation_id == other.mutation_id
&& self.kind == other.kind
&& self.old_config_etag == other.old_config_etag
&& self.candidate_digest == other.candidate_digest
@@ -825,6 +830,10 @@ fn identity_is_empty(identity: &TierDestinationId) -> bool {
identity.iter().all(|byte| *byte == 0)
}
fn is_false(value: &bool) -> bool {
!value
}
fn digest_is_empty(digest: &TierMutationDigest) -> bool {
digest.iter().all(|byte| *byte == 0)
}
@@ -840,6 +849,7 @@ mod tests {
fn prepared_intent() -> TierMutationIntent {
TierMutationIntent {
requires_native_azure: false,
mutation_id: Uuid::new_v4(),
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -867,6 +877,49 @@ mod tests {
assert_eq!(shard, tier_mutation_mutex_shard_index(mutation_id));
}
#[test]
fn native_azure_requirement_is_identity_bound_and_rejected_by_the_previous_wire_reader() {
// Preserve the previous strict wire shape as an independent reader.
#[derive(Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
struct LegacyIntent {
mutation_id: Uuid,
revision: u64,
kind: TierMutationIntentKind,
state: TierMutationIntentState,
#[serde(default, skip_serializing_if = "Option::is_none")]
old_config_etag: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
committed_config_etag: Option<String>,
candidate_digest: TierMutationDigest,
affected_targets: Vec<TierMutationIntentTarget>,
expires_at_unix_nanos: i64,
}
let legacy = prepared_intent();
let legacy_bytes = serde_json::to_vec(&legacy).expect("legacy intent should encode");
let old_reader: LegacyIntent =
serde_json::from_slice(&legacy_bytes).expect("ordinary mutations remain readable by old peers");
assert_eq!(
serde_json::to_vec(&old_reader).expect("old encoding"),
legacy_bytes,
"ordinary mutation checksums must stay unchanged"
);
let mut native = legacy.clone();
native.requires_native_azure = true;
let native_bytes = serde_json::to_vec(&native).expect("native intent should encode");
assert!(
serde_json::from_slice::<LegacyIntent>(&native_bytes).is_err(),
"an old peer must reject native Azure before prepare"
);
assert!(!legacy.same_identity_as(&native), "an abort cannot drop the native-driver requirement");
let encoded = native.encode().expect("native durable intent should encode");
assert_eq!(
TierMutationIntent::decode(native.mutation_id, &encoded).expect("new peer should decode"),
native
);
}
#[tokio::test]
async fn mutation_mutex_serializes_the_same_id() {
let mutation_id = Uuid::new_v4();
@@ -25,6 +25,7 @@ use crate::services::tier::{
tier_handlers::{ERR_TIER_BUCKET_NOT_FOUND, ERR_TIER_INVALID_CREDENTIALS, ERR_TIER_NOT_FOUND, ERR_TIER_PERM_ERR},
warm_backend_aliyun::WarmBackendAliyun,
warm_backend_azure::WarmBackendAzure,
warm_backend_azure_blob::{AzureBlobError, WarmBackendAzureBlob},
warm_backend_huaweicloud::WarmBackendHuaweicloud,
warm_backend_minio::WarmBackendMinIO,
warm_backend_r2::WarmBackendR2,
@@ -309,10 +310,10 @@ pub fn build_transition_put_options(storage_class: String, mut metadata: HashMap
/// Connection parameters every S3-compatible warm backend provider supplies.
///
/// The Aliyun, Azure, Huaweicloud, Tencent, MinIO, R2, and RustFS backends all
/// The Aliyun, legacy Azure, Huaweicloud, Tencent, MinIO, R2, and RustFS backends all
/// wrap [`WarmBackendS3`] around a statically-credentialed [`TransitionClient`]
/// built from exactly these values. `bucket_lookup` is a parameter rather than a
/// constant because the providers split into two families: Aliyun, Azure,
/// constant because the providers split into two families: Aliyun, legacy Azure,
/// Huaweicloud, and Tencent pin [`BucketLookupType::BucketLookupDNS`], while
/// MinIO, R2, and RustFS leave it at [`BucketLookupType::BucketLookupAuto`].
pub(crate) struct S3CompatibleWarmBackendParams<'a> {
@@ -673,6 +674,9 @@ fn probe_cleanup_incomplete_error() -> AdminError {
}
fn is_remote_tier_auth_rejection(err: &std::io::Error) -> bool {
if AzureBlobError::from_io(err).is_some() && err.kind() == std::io::ErrorKind::PermissionDenied {
return true;
}
let Some(response) = err.get_ref().and_then(|err| err.downcast_ref::<ErrorResponse>()) else {
return false;
};
@@ -706,10 +710,27 @@ async fn check_warm_backend_with_deadlines(
Ok(Err(err)) if is_remote_tier_auth_rejection(&err) => {
return Err(ERR_TIER_INVALID_CREDENTIALS.clone());
}
Ok(Err(_)) => {
Ok(Err(err)) => {
if let Some(rejection) = AzureBlobError::from_io(&err).filter(|error| error.rejects_write()) {
let mut admin_err = ERR_TIER_INVALID_CONFIG.clone();
admin_err.message = rejection.to_string();
return Err(admin_err);
}
return Err(match compensate_uncertain_probe_put(w, &probe_object, cleanup_deadline).await {
Ok(()) => ERR_TIER_PERM_ERR.clone(),
Err(_) => probe_cleanup_incomplete_error(),
Ok(()) => {
let mut admin_err = ERR_TIER_PERM_ERR.clone();
if let Some(original) = AzureBlobError::from_io(&err) {
admin_err.message = original.to_string();
}
admin_err
}
Err(_) => {
let mut admin_err = probe_cleanup_incomplete_error();
if let Some(original) = AzureBlobError::from_io(&err) {
admin_err.message.push_str(&format!(": {original}"));
}
admin_err
}
});
}
Err(_) => {
@@ -951,24 +972,23 @@ pub async fn new_warm_backend(tier: &TierConfig, probe: bool) -> Result<WarmBack
}
}
TierType::Azure => {
if let Some(azure_config) = tier.azure.as_ref() {
let dd = WarmBackendAzure::new(azure_config, &tier.name).await;
if let Err(err) = dd {
warn!("{}", err);
return Err(AdminError {
code: "XRustFSAdminTierInvalidConfig".to_string(),
message: format!("Unable to setup remote tier, check tier configuration: {err}"),
status_code: StatusCode::BAD_REQUEST,
});
}
d = Some(Box::new(dd.expect("Failed to create Azure backend")));
let azure = tier.azure.as_ref().ok_or_else(|| AdminError {
code: "XRustFSAdminTierInvalidConfig".to_string(),
message: "Azure tier configuration not found".to_string(),
status_code: StatusCode::BAD_REQUEST,
})?;
let driver = if tier.is_native_azure() {
WarmBackendAzureBlob::new(azure).map(|driver| -> WarmBackendImpl { Box::new(driver) })
} else {
return Err(AdminError {
code: "XRustFSAdminTierInvalidConfig".to_string(),
message: "Azure tier configuration not found".to_string(),
status_code: StatusCode::BAD_REQUEST,
});
}
WarmBackendAzure::new(azure, &tier.name)
.await
.map(|driver| -> WarmBackendImpl { Box::new(driver) })
};
d = Some(driver.map_err(|err| AdminError {
code: "XRustFSAdminTierInvalidConfig".to_string(),
message: format!("Unable to setup remote tier, check tier configuration: {err}"),
status_code: StatusCode::BAD_REQUEST,
})?);
}
#[cfg(not(feature = "gcs"))]
TierType::GCS => {
@@ -1042,6 +1062,13 @@ pub(crate) async fn new_transition_candidate_reconciler(
tier: &TierConfig,
) -> Result<Option<Box<dyn TransitionCandidateReconciler + Send + Sync + 'static>>, AdminError> {
let reconciler: Box<dyn TransitionCandidateReconciler + Send + Sync + 'static> = match tier.tier_type {
TierType::Azure if tier.is_native_azure() => Box::new(
WarmBackendAzureBlob::new(tier.azure.as_ref().ok_or_else(|| ERR_TIER_INVALID_CONFIG.clone())?).map_err(|err| {
let mut admin_err = ERR_TIER_INVALID_CONFIG.clone();
admin_err.message = err.to_string();
admin_err
})?,
),
TierType::S3 => Box::new(
WarmBackendS3::new(tier.s3.as_ref().ok_or_else(|| ERR_TIER_INVALID_CONFIG.clone())?, &tier.name)
.await
@@ -88,7 +88,7 @@ impl WarmBackend for WarmBackendAzure {
}
async fn probe_transition_candidate(&self, object: &str) -> Result<TransitionCandidateProbe, std::io::Error> {
// Azure currently uses the shared S3/SigV4 transport, but its normal
// Legacy Azure uses the shared S3/SigV4 transport, but its normal
// object path cannot persist exact remote versions across mixed
// RustFS releases. The mutation probe may still detect and precisely
// remove a versioned test object before rejecting that configuration.
@@ -0,0 +1,881 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Native Azure Blob tier transport, including Azure Stack Hub 2301 and later.
//!
//! Writes use one conditional commit and never replay a failed HTTP request.
//! Larger bodies are staged in bounded blocks with a unique upload namespace;
//! an interrupted staging operation leaves only Azure's expiring uncommitted
//! blocks. Lifecycle uploads use the existing transaction journal after a lost
//! commit acknowledgement; validation probes retain in-process compensation.
//! Versioned GET/DELETE address the persisted version. Current-root cleanup
//! may demote that matching version under a strong ETag before exact deletion.
//! Unversioned DELETE also requires a strong ETag to refuse replacement.
use super::tier_config::TierAzure;
use super::warm_backend::{
TransitionCandidateIdentity, TransitionCandidateProbe, TransitionCandidateReconciler, WarmBackend, WarmBackendGetOpts,
build_transition_put_options, transition_client_timeouts_from_env,
};
use super::warm_backend_s3::transition_candidate_metadata_matches;
use crate::bucket::lifecycle::transition_transaction::canonical_transition_remote_object;
use bytes::Bytes;
use futures_util::stream;
use http::{HeaderMap, HeaderName, HeaderValue, Method, StatusCode};
use quick_xml::{Reader, events::Event};
use rustfs_s3_client::transition_api::{ReadCloser, ReaderImpl};
use rustfs_signer::azure_shared_key_signature;
use rustfs_utils::crypto::hex;
use rustfs_utils::egress::validate_outbound_url;
use rustfs_utils::hash::FastMd5;
use rustfs_utils::http::metadata_compat::{
MINIO_INTERNAL_PREFIX, SUFFIX_TRANSITION_TIER_DESTINATION_ID, SUFFIX_TRANSITION_TRANSACTION_ID, internal_key_rustfs,
};
use std::collections::HashMap;
use std::io::{self, Cursor, ErrorKind};
use std::time::Duration;
use time::OffsetDateTime;
use tokio::io::{AsyncRead, AsyncReadExt};
use url::Url;
const API_VERSION: &str = "2020-10-02";
const MIN_BLOCK_BYTES: u64 = 16 * 1024 * 1024;
const MAX_BLOCK_BYTES: u64 = 100 * 1024 * 1024;
const MAX_BLOCKS: u64 = 50_000;
const MAX_CONTROL_BYTES: usize = 64 * 1024;
const MAX_METADATA_BYTES: usize = 8 * 1024;
const METADATA_PREFIX: &str = "x-ms-meta-rustfs_";
const HTTP_DATE: &[time::format_description::BorrowedFormatItem<'static>] =
time::macros::format_description!("[weekday repr:short], [day] [month repr:short] [year] [hour]:[minute]:[second] GMT");
/// A bounded diagnostic that never contains credentials, URLs or response bodies.
#[derive(Debug, thiserror::Error)]
#[error("Azure Blob {operation} failed (HTTP {status}, code {code}, request ID {request_id})")]
pub(crate) struct AzureBlobError {
operation: &'static str,
status: StatusCode,
code: String,
request_id: String,
}
impl AzureBlobError {
pub(super) fn from_io(error: &io::Error) -> Option<&Self> {
error.get_ref()?.downcast_ref()
}
pub(super) fn rejects_write(&self) -> bool {
matches!(
self.status,
StatusCode::BAD_REQUEST | StatusCode::NOT_FOUND | StatusCode::CONFLICT | StatusCode::PRECONDITION_FAILED
)
}
}
pub struct WarmBackendAzureBlob {
client: reqwest::Client,
endpoint: Url,
account: String,
key: Vec<u8>,
container: String,
prefix: String,
request_timeout: Duration,
}
impl WarmBackendAzureBlob {
/// Build a native Blob backend without performing remote I/O.
///
/// # Errors
///
/// Rejects unsupported credentials or storage classes, an invalid account,
/// container, prefix, endpoint or account key, and TLS/client setup failures.
pub fn new(conf: &TierAzure) -> io::Result<Self> {
validate_config(conf)?;
let endpoint = if conf.endpoint.is_empty() {
format!("https://{}.blob.core.windows.net", conf.access_key)
} else {
conf.endpoint.clone()
};
let endpoint = Url::parse(&endpoint).map_err(|_| invalid_input("Azure Blob endpoint is not a valid URL"))?;
if !matches!(endpoint.scheme(), "https" | "http")
|| endpoint.host_str().is_none()
|| !endpoint.username().is_empty()
|| endpoint.password().is_some()
|| !matches!(endpoint.path(), "" | "/")
|| endpoint.query().is_some()
|| endpoint.fragment().is_some()
{
return Err(invalid_input(
"Azure Blob endpoint must be an HTTP(S) origin without credentials, path or query",
));
}
validate_outbound_url(&endpoint).map_err(|_| invalid_input("Azure Blob endpoint is not allowed"))?;
let key = decode_key(&conf.secret_key)?;
rustfs_tls_runtime::ensure_crypto_provider().map_err(io::Error::other)?;
let timeouts = transition_client_timeouts_from_env();
let client = reqwest::Client::builder()
.connect_timeout(timeouts.connect_timeout)
.read_timeout(timeouts.response_body_idle_timeout)
.redirect(reqwest::redirect::Policy::none())
.retry(reqwest::retry::never())
.no_gzip()
.no_brotli()
.no_zstd()
.no_deflate()
.build()
.map_err(transport_error)?;
Ok(Self {
client,
endpoint,
account: conf.access_key.clone(),
key,
container: conf.bucket.clone(),
prefix: conf.prefix.strip_suffix('/').unwrap_or(&conf.prefix).to_string(),
request_timeout: timeouts.request_timeout,
})
}
fn object_url(&self, object: &str, version: &str) -> io::Result<Url> {
validate_version(version)?;
let name = if self.prefix.is_empty() {
object.to_string()
} else {
format!("{}/{object}", self.prefix)
};
validate_object_name(&name)?;
let mut url = self.endpoint.clone();
{
let mut segments = url
.path_segments_mut()
.map_err(|_| invalid_input("Azure Blob endpoint has no path"))?;
segments.clear().push(&self.container);
for segment in name.split('/') {
segments.push(segment);
}
}
if !version.is_empty() {
url.query_pairs_mut().append_pair("versionid", version);
}
Ok(url)
}
fn container_url(&self) -> io::Result<Url> {
let mut url = self.endpoint.clone();
url.path_segments_mut()
.map_err(|_| invalid_input("Azure Blob endpoint has no path"))?
.clear()
.push(&self.container);
Ok(url)
}
async fn send(&self, method: Method, url: Url, mut headers: HeaderMap, body: Option<Bytes>) -> io::Result<reqwest::Response> {
if let Some(body) = &body {
insert_header(&mut headers, "content-length", &body.len().to_string())?;
let bytes = body.clone();
let digest = tokio::task::spawn_blocking(move || FastMd5::digest(&bytes))
.await
.map_err(|_| io::Error::other("Azure Blob content checksum task failed"))?;
insert_header(&mut headers, "content-md5", &base64_simd::STANDARD.encode_to_string(digest))?;
}
headers.insert("x-ms-version", HeaderValue::from_static(API_VERSION));
headers.insert("accept-encoding", HeaderValue::from_static("identity"));
let date = OffsetDateTime::now_utc()
.format(HTTP_DATE)
.map_err(|_| io::Error::other("Azure Blob request date is not representable"))?;
insert_header(&mut headers, "x-ms-date", &date)?;
let signature = azure_shared_key_signature(&self.key, &self.account, method.as_str(), &url, &headers)
.map_err(|err| io::Error::new(ErrorKind::InvalidInput, err))?;
insert_header(&mut headers, "authorization", &format!("SharedKey {}:{signature}", self.account))?;
let mut request = reqwest::Request::new(method, url);
*request.headers_mut() = headers;
*request.body_mut() = body.map(reqwest::Body::from);
tokio::time::timeout(self.request_timeout, self.client.execute(request))
.await
.map_err(|_| io::Error::new(ErrorKind::TimedOut, "Azure Blob request timed out"))?
.map_err(transport_error)
}
async fn expect_status(
&self,
response: reqwest::Response,
expected: StatusCode,
operation: &'static str,
) -> io::Result<reqwest::Response> {
if response.status() == expected {
return Ok(response);
}
Err(service_error(response, operation, self.request_timeout).await)
}
async fn head(&self, object: &str, version: &str) -> io::Result<Option<HeaderMap>> {
let response = self
.send(Method::HEAD, self.object_url(object, version)?, HeaderMap::new(), None)
.await?;
if response.status() == StatusCode::OK {
if !version.is_empty() {
validate_response_version(response.headers(), version)?;
}
return Ok(Some(response.headers().clone()));
}
let err = service_error(response, "HEAD", self.request_timeout).await;
if err.kind() == ErrorKind::NotFound {
return Ok(None);
}
Err(err)
}
async fn delete(&self, object: &str, version: &str, headers: HeaderMap) -> io::Result<()> {
let response = self
.send(Method::DELETE, self.object_url(object, version)?, headers, None)
.await?;
if response.status() == StatusCode::ACCEPTED {
return Ok(());
}
let error = service_error(response, "DELETE", self.request_timeout).await;
if error.kind() == ErrorKind::NotFound {
return Ok(());
}
Err(error)
}
async fn write(
&self,
object: &str,
reader: ReaderImpl,
length: i64,
metadata: HashMap<String, String>,
) -> io::Result<String> {
let size = u64::try_from(length).map_err(|_| invalid_input("Azure Blob upload requires a known, non-negative length"))?;
let block_size = block_size(size)?;
let url = self.object_url(object, "")?;
let mut headers = metadata_headers(metadata)?;
headers.insert("if-none-match", HeaderValue::from_static("*"));
let mut reader: Box<dyn AsyncRead + Send + Sync + Unpin> = match reader {
ReaderImpl::Body(bytes) => Box::new(Cursor::new(bytes)),
ReaderImpl::ObjectBody(reader) => Box::new(reader),
};
if size <= block_size {
let body = read_part(&mut reader, size).await?;
ensure_eof(&mut reader).await?;
headers.insert("x-ms-blob-type", HeaderValue::from_static("BlockBlob"));
let response = self.send(Method::PUT, url, headers, Some(body)).await?;
let response = self.expect_status(response, StatusCode::CREATED, "PUT").await?;
return response_version(response.headers());
}
let upload = uuid::Uuid::new_v4().simple().to_string();
let mut blocks = Vec::new();
let mut remaining = size;
while remaining > 0 {
let count = remaining.min(block_size);
let body = read_part(&mut reader, count).await?;
let block_id = base64_simd::STANDARD.encode_to_string(format!("{upload}-{:05}", blocks.len()));
let mut block_url = url.clone();
block_url
.query_pairs_mut()
.append_pair("comp", "block")
.append_pair("blockid", &block_id);
let response = self.send(Method::PUT, block_url, HeaderMap::new(), Some(body)).await?;
self.expect_status(response, StatusCode::CREATED, "PUT block").await?;
blocks.push(block_id);
remaining -= count;
}
ensure_eof(&mut reader).await?;
let mut xml = String::from("<?xml version=\"1.0\" encoding=\"utf-8\"?><BlockList>");
for block in blocks {
// IDs are generated base64, so they contain no XML delimiters.
xml.push_str("<Uncommitted>");
xml.push_str(&block);
xml.push_str("</Uncommitted>");
}
xml.push_str("</BlockList>");
let mut commit_url = url;
commit_url.query_pairs_mut().append_pair("comp", "blocklist");
headers.insert("content-type", HeaderValue::from_static("application/xml"));
let response = self.send(Method::PUT, commit_url, headers, Some(Bytes::from(xml))).await?;
let response = self.expect_status(response, StatusCode::CREATED, "PUT block list").await?;
response_version(response.headers())
}
}
#[async_trait::async_trait]
impl WarmBackend for WarmBackendAzureBlob {
fn validate_remote_version_id(&self, version: &str) -> io::Result<()> {
validate_version(version)
}
async fn put(&self, object: &str, reader: ReaderImpl, length: i64) -> io::Result<String> {
self.write(object, reader, length, HashMap::new()).await
}
async fn put_with_meta(
&self,
object: &str,
reader: ReaderImpl,
length: i64,
metadata: HashMap<String, String>,
) -> io::Result<String> {
self.write(object, reader, length, metadata).await
}
async fn get(&self, object: &str, version: &str, opts: WarmBackendGetOpts) -> io::Result<ReadCloser> {
let range = requested_range(&opts)?;
let mut headers = HeaderMap::new();
if let Some((start, end)) = range {
let value = end.map_or_else(|| format!("bytes={start}-"), |end| format!("bytes={start}-{end}"));
insert_header(&mut headers, "range", &value)?;
}
let response = self
.send(Method::GET, self.object_url(object, version)?, headers, None)
.await?;
let expected = if range.is_some() {
StatusCode::PARTIAL_CONTENT
} else {
StatusCode::OK
};
let response = self.expect_status(response, expected, "GET").await?;
validate_response_version(response.headers(), version)?;
let length = response
.headers()
.get("content-length")
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse::<u64>().ok())
.ok_or_else(|| invalid_data("Azure Blob GET omitted a valid Content-Length"))?;
if let Some(range) = range {
validate_content_range(response.headers(), range, length)?;
}
let body = stream::try_unfold((response, length), |(mut response, remaining)| async move {
loop {
match response.chunk().await.map_err(transport_error)? {
Some(chunk) if chunk.is_empty() => continue,
Some(chunk) => {
let count =
u64::try_from(chunk.len()).map_err(|_| invalid_data("Azure Blob response length overflow"))?;
let remaining = remaining
.checked_sub(count)
.ok_or_else(|| invalid_data("Azure Blob returned more bytes than declared"))?;
return Ok::<_, io::Error>(Some((chunk, (response, remaining))));
}
None if remaining == 0 => return Ok(None),
None => {
return Err(io::Error::new(
ErrorKind::UnexpectedEof,
"Azure Blob response ended before its declared length",
));
}
}
}
});
Ok(ReadCloser::new(tokio_util::io::StreamReader::new(Box::pin(body))))
}
async fn remove(&self, object: &str, version: &str) -> io::Result<()> {
let mut headers = HeaderMap::new();
if version.is_empty() {
let Some(current) = self.head(object, "").await? else {
return Ok(());
};
if !response_version(&current)?.is_empty() {
return Err(invalid_data("Azure Blob became versioned; refusing an unversioned delete"));
}
headers = delete_preconditions(&current)?;
}
let Err(error) = self.delete(object, version, headers).await else {
return Ok(());
};
if version.is_empty()
|| !AzureBlobError::from_io(&error).is_some_and(|azure| {
matches!(azure.status, StatusCode::FORBIDDEN | StatusCode::CONFLICT)
&& azure.code == "OperationNotAllowedOnRootBlob"
})
{
return Err(error);
}
// Azure may require the current version to become a previous version
// before exact deletion. Only its matching generation can be demoted.
if let Some(current) = self.head(object, "").await? {
validate_response_version(&current, version)?;
self.delete(object, "", delete_preconditions(&current)?).await?;
}
// A successful base delete alone does not complete version cleanup.
// An error or lost acknowledgement remains retryable by the caller.
self.delete(object, version, HeaderMap::new()).await
}
async fn probe_transition_candidate(&self, object: &str) -> io::Result<TransitionCandidateProbe> {
Ok(match self.head(object, "").await? {
None => TransitionCandidateProbe::Missing,
Some(headers) => classify_head(&headers)?,
})
}
async fn probe_transition_version(&self, object: &str, version: &str) -> io::Result<TransitionCandidateProbe> {
if version.is_empty() {
return Err(invalid_input("Azure Blob exact probe requires a version ID"));
}
Ok(match self.head(object, version).await? {
None => TransitionCandidateProbe::Missing,
Some(_) => TransitionCandidateProbe::VersionedPresent(version.to_string()),
})
}
async fn in_use(&self) -> io::Result<bool> {
let mut url = self.container_url()?;
let prefix = if self.prefix.is_empty() {
String::new()
} else {
format!("{}/", self.prefix)
};
url.query_pairs_mut()
.append_pair("restype", "container")
.append_pair("comp", "list")
.append_pair("prefix", &prefix)
.append_pair("maxresults", "1");
let response = self.send(Method::GET, url, HeaderMap::new(), None).await?;
let response = self.expect_status(response, StatusCode::OK, "LIST").await?;
list_has_objects(&read_control_body(response, self.request_timeout).await?)
}
}
#[async_trait::async_trait]
impl TransitionCandidateReconciler for WarmBackendAzureBlob {
async fn probe_transition_candidate_for(
&self,
object: &str,
identity: TransitionCandidateIdentity,
) -> io::Result<TransitionCandidateProbe> {
let Some(headers) = self.head(object, "").await? else {
return Ok(TransitionCandidateProbe::Missing);
};
let mut metadata = HashMap::new();
for suffix in [SUFFIX_TRANSITION_TRANSACTION_ID, SUFFIX_TRANSITION_TIER_DESTINATION_ID] {
for key in [internal_key_rustfs(suffix), format!("{MINIO_INTERNAL_PREFIX}{suffix}")] {
if let Some(value) = headers.get(metadata_header_name(&key)) {
let decoded = base64_simd::STANDARD
.decode_to_vec(value.as_bytes())
.map_err(|_| invalid_data("Azure Blob candidate metadata is not valid base64"))?;
let value =
String::from_utf8(decoded).map_err(|_| invalid_data("Azure Blob candidate metadata is not UTF-8"))?;
metadata.insert(key, value);
}
}
}
if !transition_candidate_metadata_matches(&metadata, identity)? {
return Ok(TransitionCandidateProbe::Ambiguous);
}
classify_head(&headers)
}
}
fn validate_config(conf: &TierAzure) -> io::Result<()> {
if !(3..=24).contains(&conf.access_key.len())
|| !conf
.access_key
.bytes()
.all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit())
{
return Err(invalid_input("Azure Blob account name must contain 3 to 24 lowercase letters or digits"));
}
let container = conf.bucket.as_bytes();
if !(3..=63).contains(&container.len())
|| !container
.iter()
.all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || *byte == b'-')
|| container.first() == Some(&b'-')
|| container.last() == Some(&b'-')
|| conf.bucket.contains("--")
{
return Err(invalid_input("Azure Blob container name is invalid"));
}
if !(conf.storage_class.is_empty() || conf.storage_class == "STANDARD")
|| !conf.sp_auth.tenant_id.is_empty()
|| !conf.sp_auth.client_id.is_empty()
|| !conf.sp_auth.client_secret.is_empty()
{
return Err(invalid_input(
"Azure Blob tiers currently require an account key and the default storage class",
));
}
let prefix = conf.prefix.strip_suffix('/').unwrap_or(&conf.prefix);
if !prefix.is_empty() {
validate_object_name(prefix)?;
// Canonical suffix length is independent of the bucket and UUID values.
let object = canonical_transition_remote_object(
uuid::Uuid::from_u128(1),
&conf.bucket,
uuid::Uuid::from_u128(2),
uuid::Uuid::from_u128(3),
)
.map_err(io::Error::other)?;
validate_object_name(&format!("{prefix}/{object}"))?;
}
Ok(())
}
fn decode_key(key: &str) -> io::Result<Vec<u8>> {
let key = base64_simd::STANDARD
.decode_to_vec(key.trim().as_bytes())
.map_err(|_| invalid_input("Azure Blob account key must be base64"))?;
if key.is_empty() {
return Err(invalid_input("Azure Blob account key must not be empty"));
}
Ok(key)
}
fn validate_object_name(name: &str) -> io::Result<()> {
if name.is_empty()
|| name.len() > 1760
|| name.chars().count() > 880
|| name.split('/').any(|segment| matches!(segment, "." | ".."))
{
return Err(invalid_input(
"Azure Blob tier name exceeds Stack Hub limits or contains a standalone dot segment",
));
}
Ok(())
}
fn validate_version(version: &str) -> io::Result<()> {
if version.len() > 1024 || version.bytes().any(|byte| !byte.is_ascii_graphic()) {
return Err(invalid_input("Azure Blob version ID is invalid"));
}
Ok(())
}
fn response_version(headers: &HeaderMap) -> io::Result<String> {
let Some(version) = headers.get("x-ms-version-id") else {
return Ok(String::new());
};
let version = version
.to_str()
.map_err(|_| invalid_data("Azure Blob version ID is not ASCII"))?;
validate_version(version)?;
Ok(version.to_string())
}
fn validate_response_version(headers: &HeaderMap, requested: &str) -> io::Result<()> {
if response_version(headers)? != requested {
return Err(invalid_data("Azure Blob did not return the requested exact version"));
}
Ok(())
}
fn delete_preconditions(current: &HeaderMap) -> io::Result<HeaderMap> {
let etag = current
.get("etag")
.ok_or_else(|| invalid_data("Azure Blob HEAD omitted the delete precondition"))?;
let bytes = etag.as_bytes();
// A wildcard, weak tag or list cannot identify a single matching generation.
if bytes.len() < 3
|| bytes.first() != Some(&b'"')
|| bytes.last() != Some(&b'"')
|| !bytes[1..bytes.len() - 1]
.iter()
.all(|byte| *byte == b'!' || (b'#'..=b'~').contains(byte))
{
return Err(invalid_data("Azure Blob HEAD omitted a single strong ETag"));
}
let mut headers = HeaderMap::new();
headers.insert("if-match", etag.clone());
Ok(headers)
}
fn classify_head(headers: &HeaderMap) -> io::Result<TransitionCandidateProbe> {
let version = response_version(headers)?;
Ok(if version.is_empty() {
TransitionCandidateProbe::UnversionedPresent
} else {
TransitionCandidateProbe::VersionedPresent(version)
})
}
fn block_size(size: u64) -> io::Result<u64> {
if size > MAX_BLOCK_BYTES * MAX_BLOCKS {
return Err(invalid_input("Azure Blob exceeds the Stack Hub block count or object size limit"));
}
Ok(size.div_ceil(MAX_BLOCKS).max(MIN_BLOCK_BYTES))
}
async fn read_part(reader: &mut (dyn AsyncRead + Send + Sync + Unpin), length: u64) -> io::Result<Bytes> {
let length = usize::try_from(length).map_err(|_| invalid_input("Azure Blob block length is not representable"))?;
let mut bytes = vec![0; length];
reader.read_exact(&mut bytes).await?;
Ok(Bytes::from(bytes))
}
async fn ensure_eof(reader: &mut (dyn AsyncRead + Send + Sync + Unpin)) -> io::Result<()> {
if reader.read(&mut [0_u8; 1]).await? != 0 {
return Err(invalid_input("Azure Blob upload body exceeds its declared length"));
}
Ok(())
}
fn metadata_header_name(key: &str) -> String {
format!("{METADATA_PREFIX}{}", hex(key.as_bytes()))
}
fn metadata_headers(metadata: HashMap<String, String>) -> io::Result<HeaderMap> {
let opts = build_transition_put_options(String::new(), metadata);
let mut headers = HeaderMap::new();
for (name, value) in [
("x-ms-blob-content-type", opts.content_type),
("x-ms-blob-content-encoding", opts.content_encoding),
("x-ms-blob-content-language", opts.content_language),
("x-ms-blob-content-disposition", opts.content_disposition),
("x-ms-blob-cache-control", opts.cache_control),
] {
if !value.is_empty() {
insert_header(&mut headers, name, &value)?;
}
}
let mut size = 0_usize;
for (key, value) in opts.user_metadata {
let key = key.to_ascii_lowercase();
let key = key.strip_prefix("x-amz-meta-").unwrap_or(&key);
let name = metadata_header_name(key);
let value = base64_simd::STANDARD.encode_to_string(value.as_bytes());
size = size.saturating_add(name.len()).saturating_add(value.len());
if size > MAX_METADATA_BYTES {
return Err(invalid_input("Azure Blob encoded metadata exceeds 8 KiB"));
}
if headers.contains_key(&name) {
return Err(invalid_input("Azure Blob metadata contains conflicting names"));
}
insert_header(&mut headers, &name, &value)?;
}
Ok(headers)
}
fn requested_range(opts: &WarmBackendGetOpts) -> io::Result<Option<(u64, Option<u64>)>> {
let start = u64::try_from(opts.start_offset).map_err(|_| invalid_input("Azure Blob range starts before zero"))?;
let length = u64::try_from(opts.length).map_err(|_| invalid_input("Azure Blob range length is negative"))?;
if length > 0 {
let end = start
.checked_add(length - 1)
.ok_or_else(|| invalid_input("Azure Blob range overflows"))?;
return Ok(Some((start, Some(end))));
}
Ok((start > 0).then_some((start, None)))
}
fn validate_content_range(headers: &HeaderMap, requested: (u64, Option<u64>), length: u64) -> io::Result<()> {
let value = headers
.get("content-range")
.and_then(|value| value.to_str().ok())
.unwrap_or_default();
let range = value
.strip_prefix("bytes ")
.and_then(|value| value.split_once('/'))
.and_then(|(range, total)| {
let (start, end) = range.split_once('-')?;
Some((start.parse::<u64>().ok()?, end.parse::<u64>().ok()?, total.parse::<u64>().ok()?))
});
let Some((start, end, total)) = range else {
return Err(invalid_data("Azure Blob returned an invalid Content-Range"));
};
if start != requested.0
|| end < start
|| end >= total
|| end != requested.1.unwrap_or(total - 1).min(total - 1)
|| end.checked_sub(start).and_then(|size| size.checked_add(1)) != Some(length)
{
return Err(invalid_data("Azure Blob returned a different range from the one requested"));
}
Ok(())
}
async fn read_control_body(mut response: reqwest::Response, timeout: Duration) -> io::Result<Vec<u8>> {
tokio::time::timeout(timeout, async move {
let mut body = Vec::new();
while let Some(chunk) = response.chunk().await.map_err(transport_error)? {
if body.len().saturating_add(chunk.len()) > MAX_CONTROL_BYTES {
return Err(invalid_data("Azure Blob control response exceeds 64 KiB"));
}
body.extend_from_slice(&chunk);
}
Ok(body)
})
.await
.map_err(|_| io::Error::new(ErrorKind::TimedOut, "Azure Blob control response timed out"))?
}
async fn service_error(response: reqwest::Response, operation: &'static str, timeout: Duration) -> io::Error {
let status = response.status();
let request_id = diagnostic_token(
response
.headers()
.get("x-ms-request-id")
.and_then(|value| value.to_str().ok()),
);
let mut code = diagnostic_token(
response
.headers()
.get("x-ms-error-code")
.and_then(|value| value.to_str().ok()),
);
if code == "unknown"
&& let Ok(body) = read_control_body(response, timeout).await
&& let Some(xml_code) = xml_error_code(&body)
{
code = diagnostic_token(Some(&xml_code));
}
let kind = if status == StatusCode::NOT_FOUND && code == "BlobNotFound" {
ErrorKind::NotFound
} else if matches!(status, StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN) {
ErrorKind::PermissionDenied
} else if status == StatusCode::PRECONDITION_FAILED {
ErrorKind::AlreadyExists
} else {
ErrorKind::Other
};
io::Error::new(
kind,
AzureBlobError {
operation,
status,
code,
request_id,
},
)
}
fn diagnostic_token(value: Option<&str>) -> String {
value
.filter(|value| {
!value.is_empty() && value.len() <= 128 && value.bytes().all(|byte| byte.is_ascii_alphanumeric() || byte == b'-')
})
.unwrap_or("unknown")
.to_string()
}
fn xml_error_code(body: &[u8]) -> Option<String> {
let mut reader = Reader::from_reader(body);
let mut depth = 0_u32;
let mut root = false;
let mut code = None;
loop {
match reader.read_event().ok()? {
Event::Start(start) => {
if depth == 0 {
if root || start.local_name().as_ref() != "Error" {
return None;
}
root = true;
}
if depth == 1 && start.local_name().as_ref() == "Code" {
if code.is_some() {
return None;
}
let text = reader.read_text(start.name()).ok()?;
code = Some(quick_xml::escape::unescape(text.as_ref()).ok()?.into_owned());
continue;
}
depth += 1;
if depth > 16 {
return None;
}
}
Event::End(_) => depth = depth.checked_sub(1)?,
Event::Text(text) if depth == 0 && !text.as_ref().trim().is_empty() => return None,
Event::CData(_) if depth == 0 => return None,
Event::Eof => return (root && depth == 0).then_some(code).flatten(),
Event::DocType(_) => return None,
_ => {}
}
}
}
fn list_has_objects(body: &[u8]) -> io::Result<bool> {
let mut reader = Reader::from_reader(body);
let mut depth = 0_u32;
let mut root = false;
let mut blobs = false;
let mut found = false;
loop {
let event = reader
.read_event()
.map_err(|_| invalid_data("Azure Blob listing is not valid XML"))?;
let empty = matches!(event, Event::Empty(_));
match event {
Event::Start(start) | Event::Empty(start) => {
if depth == 0 && (root || start.local_name().as_ref() != "EnumerationResults") {
return Err(invalid_data("Azure Blob listing has an unexpected root"));
}
root |= start.local_name().as_ref() == "EnumerationResults";
if start.local_name().as_ref() == "Blobs" {
if blobs || depth != 1 {
return Err(invalid_data("Azure Blob listing has an invalid Blobs element"));
}
blobs = true;
}
found |= matches!(start.local_name().as_ref(), "Blob" | "BlobPrefix");
if !empty && start.local_name().as_ref() == "NextMarker" {
let marker = reader
.read_text(start.name())
.map_err(|_| invalid_data("Azure Blob listing marker is invalid"))?;
found |= !marker.is_empty();
continue;
}
if !empty {
depth += 1;
if depth > 16 {
return Err(invalid_data("Azure Blob listing exceeds the XML depth limit"));
}
}
}
Event::End(_) => {
depth = depth
.checked_sub(1)
.ok_or_else(|| invalid_data("Azure Blob listing has an invalid closing tag"))?;
}
Event::DocType(_) => return Err(invalid_data("Azure Blob listing must not contain a document type")),
Event::Text(text) if depth == 0 && !text.as_ref().trim().is_empty() => {
return Err(invalid_data("Azure Blob listing contains text outside its root"));
}
Event::CData(_) if depth == 0 => return Err(invalid_data("Azure Blob listing contains text outside its root")),
Event::Eof if root && blobs && depth == 0 => return Ok(found),
Event::Eof => return Err(invalid_data("Azure Blob listing is incomplete")),
_ => {}
}
}
}
fn insert_header(headers: &mut HeaderMap, name: &str, value: &str) -> io::Result<()> {
let name =
HeaderName::from_bytes(name.as_bytes()).map_err(|_| invalid_input("Azure Blob metadata header name is invalid"))?;
let value = HeaderValue::from_str(value).map_err(|_| invalid_input("Azure Blob metadata header value is invalid"))?;
headers.insert(name, value);
Ok(())
}
fn invalid_input(message: &'static str) -> io::Error {
io::Error::new(ErrorKind::InvalidInput, message)
}
fn invalid_data(message: &'static str) -> io::Error {
io::Error::new(ErrorKind::InvalidData, message)
}
fn transport_error(err: reqwest::Error) -> io::Error {
let kind = if err.is_timeout() {
ErrorKind::TimedOut
} else {
ErrorKind::Other
};
io::Error::new(kind, err.without_url())
}
#[cfg(test)]
#[path = "warm_backend_azure_blob_tests.rs"]
mod tests;
@@ -0,0 +1,929 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::*;
use crate::services::tier::warm_backend::{WarmBackendImpl, check_warm_backend};
use rustfs_utils::http::metadata_compat::insert_str;
use tokio::io::AsyncWriteExt;
use tokio::net::{TcpListener, TcpStream};
use tokio::task::JoinHandle;
struct Reply {
status: StatusCode,
headers: HeaderMap,
body: Bytes,
disconnect: bool,
}
fn reply(status: StatusCode, headers: &[(&str, &str)], body: &[u8]) -> Reply {
let mut map = HeaderMap::new();
for (name, value) in headers {
insert_header(&mut map, name, value).expect("fixture headers should be valid");
}
Reply {
status,
headers: map,
body: Bytes::copy_from_slice(body),
disconnect: false,
}
}
fn present(version: Option<&str>) -> Reply {
let mut response = reply(StatusCode::OK, &[("content-length", "6"), ("etag", "\"original\"")], b"");
if let Some(version) = version {
insert_header(&mut response.headers, "x-ms-version-id", version).expect("fixture version should be valid");
}
response
}
fn missing() -> Reply {
reply(StatusCode::NOT_FOUND, &[("x-ms-error-code", "BlobNotFound")], b"")
}
#[derive(Debug)]
struct Captured {
method: String,
target: String,
headers: HeaderMap,
body: Vec<u8>,
}
async fn capture(stream: &mut TcpStream) -> Captured {
let mut bytes = Vec::new();
let boundary = loop {
let mut buffer = [0_u8; 4096];
let count = stream.read(&mut buffer).await.expect("fixture request should be readable");
assert_ne!(count, 0, "request must include a complete HTTP head");
bytes.extend_from_slice(&buffer[..count]);
if let Some(index) = bytes.windows(4).position(|window| window == b"\r\n\r\n") {
break index + 4;
}
assert!(bytes.len() <= 32 * 1024, "request headers must remain bounded");
};
let text = std::str::from_utf8(&bytes[..boundary]).expect("request headers should be ASCII");
let mut lines = text.split("\r\n");
let mut request_line = lines.next().expect("request line").split_whitespace();
let method = request_line.next().expect("request method").to_string();
let target = request_line.next().expect("request target").to_string();
let mut headers = HeaderMap::new();
for line in lines.filter(|line| !line.is_empty()) {
let (name, value) = line.split_once(':').expect("request header should include a colon");
insert_header(&mut headers, name, value.trim()).expect("request headers should parse");
}
let length = headers
.get("content-length")
.map(|value| {
value
.to_str()
.expect("length should be ASCII")
.parse::<usize>()
.expect("length should be numeric")
})
.unwrap_or_default();
assert!(length <= 17 * 1024 * 1024, "fixture only accepts its bounded test bodies");
let mut body = bytes[boundary..].to_vec();
let received = body.len();
body.resize(length, 0);
stream
.read_exact(&mut body[received..])
.await
.expect("request body should match its declared length");
Captured {
method,
target,
headers,
body,
}
}
async fn fixture(responses: Vec<Reply>) -> (WarmBackendAzureBlob, JoinHandle<Vec<Captured>>) {
let listener = TcpListener::bind("127.0.0.1:0").await.expect("fixture listener should bind");
let address = listener.local_addr().expect("fixture address");
let task = tokio::spawn(async move {
let mut requests = Vec::new();
for mut response in responses {
let (mut stream, _) = tokio::time::timeout(Duration::from_secs(10), listener.accept())
.await
.expect("expected request must arrive")
.expect("fixture should accept");
requests.push(capture(&mut stream).await);
if response.disconnect {
continue;
}
if !response.headers.contains_key("content-length") {
insert_header(&mut response.headers, "content-length", &response.body.len().to_string())
.expect("fixture length should be valid");
}
let mut wire = format!(
"HTTP/1.1 {} {}\r\nConnection: close\r\n",
response.status.as_u16(),
response.status.canonical_reason().unwrap_or("")
);
for (name, value) in &response.headers {
wire.push_str(&format!("{name}: {}\r\n", value.to_str().expect("fixture header is ASCII")));
}
wire.push_str("\r\n");
stream.write_all(wire.as_bytes()).await.expect("fixture should send headers");
stream.write_all(&response.body).await.expect("fixture should send body");
}
requests
});
rustfs_tls_runtime::ensure_crypto_provider().expect("fixture TLS provider should initialize");
let client = reqwest::Client::builder()
.no_proxy()
.redirect(reqwest::redirect::Policy::none())
.retry(reqwest::retry::never())
.read_timeout(Duration::from_secs(5))
.build()
.expect("fixture client should build");
(
WarmBackendAzureBlob {
client,
endpoint: Url::parse(&format!("http://{address}")).expect("fixture endpoint"),
account: "account".to_string(),
key: vec![7; 32],
container: "container".to_string(),
prefix: "cold".to_string(),
request_timeout: Duration::from_secs(5),
},
task,
)
}
#[tokio::test]
async fn mutation_probe_uses_blob_shared_key_and_cleans_up_with_an_etag() {
let responses = vec![
reply(StatusCode::CREATED, &[], b""),
present(None),
reply(StatusCode::PARTIAL_CONTENT, &[("content-range", "bytes 0-5/6")], b"RustFS"),
present(None),
reply(StatusCode::ACCEPTED, &[], b""),
missing(),
];
let (backend, task) = fixture(responses).await;
let backend: WarmBackendImpl = Box::new(backend);
check_warm_backend(Some(&backend))
.await
.expect("native Blob mutation probe should succeed");
let requests = task.await.expect("fixture should join");
assert_eq!(
requests.iter().map(|request| request.method.as_str()).collect::<Vec<_>>(),
["PUT", "HEAD", "GET", "HEAD", "DELETE", "HEAD"]
);
assert!(requests[0].target.starts_with("/container/cold/rustfs-tier-probe-"));
assert_eq!(requests[0].body, b"RustFS");
assert_eq!(requests[0].headers["x-ms-blob-type"], "BlockBlob");
assert_eq!(requests[0].headers["if-none-match"], "*");
assert_eq!(requests[4].headers["if-match"], "\"original\"");
for request in &requests {
assert_eq!(request.target, requests[0].target, "the whole probe must address the same blob");
assert_eq!(request.headers["x-ms-version"], API_VERSION);
assert!(
request.headers["authorization"]
.to_str()
.expect("authorization is ASCII")
.starts_with("SharedKey account:")
);
assert!(!request.headers.contains_key("x-amz-date"));
assert!(!request.target.contains("location"));
assert!(!request.target.contains("versionid"));
}
}
#[tokio::test]
async fn versioned_mutation_probe_cleans_up_after_forbidden_current_version_delete() {
let version = "2026-11-06T01:02:03.0000000Z";
let (backend, task) = fixture(vec![
reply(StatusCode::CREATED, &[("x-ms-version-id", version)], b""),
present(Some(version)),
reply(
StatusCode::PARTIAL_CONTENT,
&[("content-range", "bytes 0-5/6"), ("x-ms-version-id", version)],
b"RustFS",
),
reply(StatusCode::FORBIDDEN, &[], b"<Error><Code>OperationNotAllowedOnRootBlob</Code></Error>"),
present(Some(version)),
reply(StatusCode::ACCEPTED, &[], b""),
reply(StatusCode::ACCEPTED, &[], b""),
missing(),
])
.await;
let backend: WarmBackendImpl = Box::new(backend);
check_warm_backend(Some(&backend))
.await
.expect("a versioned Add/Verify probe must finish cleanup after the specific 403");
let requests = task.await.expect("fixture should join");
assert_eq!(
requests.iter().map(|request| request.method.as_str()).collect::<Vec<_>>(),
["PUT", "HEAD", "GET", "DELETE", "HEAD", "DELETE", "DELETE", "HEAD"]
);
let base = &requests[0].target;
assert!(base.starts_with("/container/cold/rustfs-tier-probe-"));
for index in [1, 4, 5, 7] {
assert_eq!(&requests[index].target, base, "base operations must retain the probe identity");
}
for index in [2, 3, 6] {
let url = Url::parse(&format!("http://fixture{}", requests[index].target)).expect("captured URL");
assert_eq!(url.path(), base);
assert_eq!(
url.query_pairs().collect::<Vec<_>>(),
[("versionid".into(), version.into())],
"exact reads and deletes must retain the timestamp version"
);
}
assert_eq!(requests[5].headers["if-match"], "\"original\"");
for request in &requests {
assert_eq!(request.headers["x-ms-version"], "2020-10-02");
}
}
#[tokio::test]
async fn rejected_put_preserves_azure_error_and_never_deletes_an_existing_blob() {
for (status, code) in [
(StatusCode::BAD_REQUEST, "InvalidHeaderValue"),
(StatusCode::PRECONDITION_FAILED, "ConditionNotMet"),
] {
let (backend, task) = fixture(vec![reply(
status,
&[("x-ms-error-code", code), ("x-ms-request-id", "abc-123")],
b"<Error><Code>PRIVATE-RESPONSE-BODY</Code></Error>",
)])
.await;
let backend: WarmBackendImpl = Box::new(backend);
let err = check_warm_backend(Some(&backend))
.await
.expect_err("deterministic rejected write must fail");
assert!(err.message.contains(code));
assert!(err.message.contains("abc-123"));
assert!(!err.message.contains("PRIVATE-RESPONSE-BODY"));
assert!(!err.message.contains("cleanup is incomplete"));
let requests = task.await.expect("fixture should join");
assert_eq!(requests.len(), 1, "a rejected conditional PUT must not trigger cleanup");
}
}
#[tokio::test]
async fn empty_version_headers_use_the_unversioned_read_and_delete_paths() {
let mut head = present(Some(""));
head.headers.insert("content-length", HeaderValue::from_static("0"));
let (backend, task) = fixture(vec![
reply(StatusCode::CREATED, &[("x-ms-version-id", "")], b""),
reply(StatusCode::OK, &[("x-ms-version-id", "")], b""),
head,
reply(StatusCode::ACCEPTED, &[], b""),
])
.await;
let version = backend
.put("empty", ReaderImpl::Body(Bytes::new()), 0)
.await
.expect("empty version is unversioned");
assert!(version.is_empty());
let mut reader = backend
.get("empty", &version, WarmBackendGetOpts::default())
.await
.expect("unversioned read");
let mut body = Vec::new();
reader.read_to_end(&mut body).await.expect("empty blob should read");
assert!(body.is_empty());
backend
.remove("empty", &version)
.await
.expect("unversioned conditional delete");
for request in task.await.expect("fixture should join") {
assert_eq!(request.target, "/container/cold/empty");
}
}
#[tokio::test]
async fn rejected_credentials_are_reported_without_an_uncertain_cleanup() {
let (backend, task) = fixture(vec![reply(
StatusCode::FORBIDDEN,
&[("x-ms-error-code", "AuthenticationFailed")],
b"",
)])
.await;
let backend: WarmBackendImpl = Box::new(backend);
let err = check_warm_backend(Some(&backend))
.await
.expect_err("invalid credentials must fail");
assert_eq!(err.code, crate::services::tier::tier_handlers::ERR_TIER_INVALID_CREDENTIALS.code);
assert_eq!(task.await.expect("fixture should join").len(), 1);
}
#[tokio::test]
async fn lost_put_response_is_not_replayed_and_its_candidate_is_cleaned() {
let mut lost = reply(StatusCode::CREATED, &[], b"");
lost.disconnect = true;
let (backend, task) = fixture(vec![
lost,
present(None),
present(None),
reply(StatusCode::ACCEPTED, &[], b""),
missing(),
missing(),
])
.await;
let backend: WarmBackendImpl = Box::new(backend);
let err = check_warm_backend(Some(&backend))
.await
.expect_err("a lost write acknowledgement must remain a failure");
assert!(!err.message.contains("cleanup is incomplete"));
let requests = task.await.expect("fixture should join");
assert_eq!(requests.iter().filter(|request| request.method == "PUT").count(), 1);
assert_eq!(requests.iter().filter(|request| request.method == "DELETE").count(), 1);
}
#[tokio::test]
async fn block_upload_commits_only_the_complete_declared_body() {
let (backend, task) = fixture(vec![
reply(StatusCode::CREATED, &[], b""),
reply(StatusCode::CREATED, &[], b""),
reply(StatusCode::CREATED, &[], b""),
])
.await;
let length = usize::try_from(MIN_BLOCK_BYTES + 5).expect("fixture size");
let bytes = Bytes::from(vec![42; length]);
assert_eq!(
backend
.put(
"large",
ReaderImpl::ObjectBody(ReadCloser::new(Cursor::new(bytes.clone()))),
i64::try_from(length).expect("fixture length")
)
.await
.expect("block upload"),
""
);
let requests = task.await.expect("fixture should join");
assert_eq!(requests.len(), 3);
let mut block_ids = Vec::new();
let mut body = Vec::new();
for request in &requests[..2] {
let url = Url::parse(&format!("http://fixture{}", request.target)).expect("captured URL");
let query: HashMap<_, _> = url.query_pairs().into_owned().collect();
assert_eq!(query["comp"], "block");
block_ids.push(query["blockid"].clone());
assert!(request.headers.contains_key("content-md5"));
body.extend_from_slice(&request.body);
}
assert_eq!(body, bytes);
assert_ne!(block_ids[0], block_ids[1]);
assert!(requests[2].target.ends_with("?comp=blocklist"));
assert_eq!(requests[2].headers["if-none-match"], "*");
let expected = format!(
"<?xml version=\"1.0\" encoding=\"utf-8\"?><BlockList><Uncommitted>{}</Uncommitted><Uncommitted>{}</Uncommitted></BlockList>",
block_ids[0], block_ids[1],
);
assert_eq!(requests[2].body, expected.as_bytes());
}
#[tokio::test]
async fn server_error_remains_visible_after_successful_probe_compensation() {
let (backend, task) = fixture(vec![
reply(
StatusCode::INTERNAL_SERVER_ERROR,
&[("x-ms-error-code", "InternalError"), ("x-ms-request-id", "error-123")],
b"",
),
present(None),
present(None),
reply(StatusCode::ACCEPTED, &[], b""),
missing(),
missing(),
])
.await;
let backend: WarmBackendImpl = Box::new(backend);
let error = check_warm_backend(Some(&backend))
.await
.expect_err("failed upload remains an error after cleanup");
assert!(error.message.contains("500"));
assert!(error.message.contains("InternalError"));
assert!(error.message.contains("error-123"));
assert!(!error.message.contains("cleanup is incomplete"));
let requests = task.await.expect("fixture should join");
assert_eq!(requests.iter().filter(|request| request.method == "PUT").count(), 1);
assert_eq!(requests.iter().filter(|request| request.method == "DELETE").count(), 1);
}
#[tokio::test]
async fn empty_blob_and_declared_length_failures_are_handled_before_commit() {
let (backend, task) = fixture(vec![reply(StatusCode::CREATED, &[], b"")]).await;
backend
.put("empty", ReaderImpl::Body(Bytes::new()), 0)
.await
.expect("empty upload");
let requests = task.await.expect("fixture should join");
assert_eq!(requests[0].headers["content-length"], "0");
assert_eq!(requests[0].headers["content-md5"], "1B2M2Y8AsgTpgAmY7PhCfg==");
for (body, kind) in [
(b"short".as_slice(), ErrorKind::UnexpectedEof),
(b"too-long".as_slice(), ErrorKind::InvalidInput),
] {
let (backend, task) = fixture(vec![]).await;
let err = backend
.put("bad", ReaderImpl::Body(Bytes::copy_from_slice(body)), 6)
.await
.expect_err("length mismatch");
assert_eq!(err.kind(), kind);
assert!(task.await.expect("fixture should join").is_empty());
}
let (backend, task) = fixture(vec![reply(StatusCode::CREATED, &[], b"")]).await;
let body = Bytes::from(vec![0; usize::try_from(MIN_BLOCK_BYTES + 1).expect("fixture size")]);
let err = backend
.put(
"short-block",
ReaderImpl::Body(body),
i64::try_from(MIN_BLOCK_BYTES + 2).expect("fixture length"),
)
.await
.expect_err("short final block");
assert_eq!(err.kind(), ErrorKind::UnexpectedEof);
assert_eq!(
task.await.expect("fixture should join").len(),
1,
"short input must never commit its staged blocks"
);
}
#[tokio::test]
async fn range_get_preserves_key_encoding_and_exact_version_for_delete() {
let version = "2026-11-06T01:02:03.0000000Z";
let (backend, task) = fixture(vec![
reply(
StatusCode::PARTIAL_CONTENT,
&[("content-range", "bytes 1-3/6"), ("x-ms-version-id", version)],
b"ust",
),
reply(StatusCode::ACCEPTED, &[], b""),
])
.await;
let object = "dir/a b+?#%2F.bin";
let mut reader = backend
.get(
object,
version,
WarmBackendGetOpts {
start_offset: 1,
length: 3,
},
)
.await
.expect("versioned range read");
let mut body = Vec::new();
reader.read_to_end(&mut body).await.expect("range body");
assert_eq!(body, b"ust");
backend.remove_exact(object, version).await.expect("exact delete");
let requests = task.await.expect("fixture should join");
assert_eq!(requests.len(), 2, "exact delete needs no current-version HEAD");
for request in requests {
let url = Url::parse(&format!("http://fixture{}", request.target)).expect("captured URL");
assert_eq!(
urlencoding::decode(url.path()).expect("decoded path"),
format!("/container/cold/{object}")
);
let query: Vec<_> = url.query_pairs().into_owned().collect();
assert_eq!(query, [("versionid".to_string(), version.to_string())]);
}
}
#[tokio::test]
async fn current_version_delete_demotes_only_the_matching_root_before_exact_cleanup() {
for status in [StatusCode::FORBIDDEN, StatusCode::CONFLICT] {
let version = "target-version";
let (backend, task) = fixture(vec![
reply(status, &[("x-ms-error-code", "OperationNotAllowedOnRootBlob")], b""),
present(Some(version)),
reply(StatusCode::ACCEPTED, &[], b""),
reply(StatusCode::ACCEPTED, &[], b""),
])
.await;
backend
.remove_exact("current", version)
.await
.expect("current version should be removable");
let requests = task.await.expect("fixture should join");
assert_eq!(requests.len(), 4);
assert_eq!(requests[0].target, "/container/cold/current?versionid=target-version");
assert_eq!(requests[1].method, "HEAD");
assert_eq!(requests[1].target, "/container/cold/current");
assert_eq!(requests[2].method, "DELETE");
assert_eq!(requests[2].target, "/container/cold/current");
assert_eq!(requests[2].headers["if-match"], "\"original\"");
assert_eq!(requests[3].method, "DELETE");
assert_eq!(requests[3].target, requests[0].target);
}
}
#[tokio::test]
async fn current_version_delete_refuses_replacements_before_and_after_head() {
for replaced_after_head in [false, true] {
let mut responses = vec![
reply(StatusCode::FORBIDDEN, &[("x-ms-error-code", "OperationNotAllowedOnRootBlob")], b""),
present(Some(if replaced_after_head {
"target-version"
} else {
"replacement-version"
})),
];
if replaced_after_head {
responses.push(reply(StatusCode::PRECONDITION_FAILED, &[("x-ms-error-code", "ConditionNotMet")], b""));
}
let (backend, task) = fixture(responses).await;
assert!(backend.remove_exact("current", "target-version").await.is_err());
let requests = task.await.expect("fixture should join");
if replaced_after_head {
assert_eq!(requests.len(), 3, "a failed condition must stop exact cleanup");
assert_eq!(requests[2].target, "/container/cold/current");
assert_eq!(requests[2].headers["if-match"], "\"original\"");
} else {
assert_eq!(requests.len(), 2, "a different current version must never authorize a base delete");
}
}
}
#[tokio::test]
async fn current_version_delete_retries_the_exact_version_when_the_root_is_already_absent() {
let (backend, task) = fixture(vec![
reply(StatusCode::FORBIDDEN, &[("x-ms-error-code", "OperationNotAllowedOnRootBlob")], b""),
missing(),
reply(StatusCode::ACCEPTED, &[], b""),
])
.await;
backend
.remove_exact("current", "target-version")
.await
.expect("previous version cleanup");
let requests = task.await.expect("fixture should join");
assert_eq!(requests.len(), 3);
assert_eq!(requests[2].target, "/container/cold/current?versionid=target-version");
assert!(
requests
.iter()
.all(|request| request.method != "DELETE" || request.target.contains("versionid="))
);
}
#[tokio::test]
async fn current_version_delete_keeps_partial_or_uncertain_cleanup_retryable() {
for fails_during_exact_cleanup in [false, true] {
for lost_response in [false, true] {
let mut responses = vec![
reply(StatusCode::FORBIDDEN, &[("x-ms-error-code", "OperationNotAllowedOnRootBlob")], b""),
present(Some("target-version")),
];
if fails_during_exact_cleanup {
responses.push(reply(StatusCode::ACCEPTED, &[], b""));
}
let mut failure = reply(StatusCode::INTERNAL_SERVER_ERROR, &[("x-ms-error-code", "InternalError")], b"");
failure.disconnect = lost_response;
responses.push(failure);
responses.push(reply(StatusCode::ACCEPTED, &[], b""));
let (backend, task) = fixture(responses).await;
backend
.remove_exact("current", "target-version")
.await
.expect_err("base removal alone or an uncertain response must not complete cleanup");
backend
.remove_exact("current", "target-version")
.await
.expect("retry exact cleanup");
let requests = task.await.expect("fixture should join");
let expected = if fails_during_exact_cleanup { 5 } else { 4 };
assert_eq!(requests.len(), expected, "uncertain destructive requests must not be replayed internally");
assert_eq!(requests[2].headers["if-match"], "\"original\"");
assert_eq!(requests[expected - 1].target, "/container/cold/current?versionid=target-version");
assert_eq!(
requests
.iter()
.filter(|request| request.method == "DELETE" && !request.target.contains('?'))
.count(),
1,
"a retry must not delete the root again"
);
}
}
}
#[tokio::test]
async fn delete_requires_a_single_strong_etag_and_an_exact_current_version() {
for versioned in [false, true] {
for etag in [
None,
Some(""),
Some("*"),
Some("W/\"weak\""),
Some("\"first\", \"second\""),
Some("\"\""),
] {
let mut head = present(if versioned { Some("target-version") } else { None });
head.headers.remove("etag");
if let Some(etag) = etag {
insert_header(&mut head.headers, "etag", etag).expect("fixture etag should be representable");
}
let mut responses = Vec::new();
if versioned {
responses.push(reply(StatusCode::FORBIDDEN, &[("x-ms-error-code", "OperationNotAllowedOnRootBlob")], b""));
}
responses.push(head);
let (backend, task) = fixture(responses).await;
assert!(
backend
.remove("current", if versioned { "target-version" } else { "" })
.await
.is_err()
);
let requests = task.await.expect("fixture should join");
assert!(
requests
.iter()
.all(|request| request.method != "DELETE" || request.target.contains("versionid=")),
"a missing, wildcard, weak or multi-value ETag cannot authorize root removal"
);
}
}
for current_version in [None, Some("")] {
let (backend, task) = fixture(vec![
reply(StatusCode::FORBIDDEN, &[("x-ms-error-code", "OperationNotAllowedOnRootBlob")], b""),
present(current_version),
])
.await;
assert!(backend.remove_exact("current", "target-version").await.is_err());
assert_eq!(task.await.expect("fixture should join").len(), 2);
}
}
#[tokio::test]
async fn exact_delete_does_not_demote_roots_for_other_service_errors() {
for (status, code) in [
(StatusCode::CONFLICT, "BlobImmutableDueToPolicy"),
(StatusCode::FORBIDDEN, "BlobImmutableDueToPolicy"),
(StatusCode::FORBIDDEN, "AuthenticationFailed"),
(StatusCode::FORBIDDEN, "AuthorizationFailure"),
(StatusCode::FORBIDDEN, "AuthorizationPermissionMismatch"),
(StatusCode::FORBIDDEN, "OperationNotAllowedOnRootBlobOther"),
(StatusCode::UNAUTHORIZED, "OperationNotAllowedOnRootBlob"),
(StatusCode::INTERNAL_SERVER_ERROR, "OperationNotAllowedOnRootBlob"),
] {
let (backend, task) = fixture(vec![reply(status, &[("x-ms-error-code", code)], b"")]).await;
let error = backend
.remove_exact("current", "target-version")
.await
.expect_err("service rejection");
assert_eq!(AzureBlobError::from_io(&error).expect("typed service error").code, code);
assert_eq!(
task.await.expect("fixture should join").len(),
1,
"unrelated errors cannot trigger root removal"
);
}
}
#[test]
fn prefix_reserves_the_complete_canonical_lifecycle_name_before_backend_creation() {
use crate::bucket::lifecycle::transition_transaction::canonical_transition_remote_object;
let mut config = TierAzure {
access_key: "account".to_string(),
secret_key: base64_simd::STANDARD.encode_to_string([7; 32]),
bucket: "container".to_string(),
prefix: "a".repeat(800),
..Default::default()
};
assert!(
WarmBackendAzureBlob::new(&config).is_err(),
"a prefix that fits only the probe is unusable for lifecycle"
);
let object = canonical_transition_remote_object(
uuid::Uuid::from_u128(1),
"source-bucket",
uuid::Uuid::from_u128(2),
uuid::Uuid::from_u128(3),
)
.expect("canonical lifecycle key");
let max_ascii_prefix = 880 - object.chars().count() - 1;
config.prefix = "a".repeat(max_ascii_prefix);
let backend = WarmBackendAzureBlob::new(&config).expect("complete name at the character limit");
assert!(backend.object_url(&object, "").is_ok());
config.prefix.push('/');
let backend = WarmBackendAzureBlob::new(&config).expect("one trailing slash is normalized");
assert!(backend.object_url(&object, "").is_ok());
config.prefix = "a".repeat(max_ascii_prefix + 1);
assert!(WarmBackendAzureBlob::new(&config).is_err(), "reject one extra character before I/O");
let max_unicode_prefix = (1760 - object.len() - 1) / '界'.len_utf8();
config.prefix = "界".repeat(max_unicode_prefix);
let backend = WarmBackendAzureBlob::new(&config).expect("complete Unicode name within the byte limit");
assert!(backend.object_url(&object, "").is_ok());
config.prefix.push('界');
assert!(
WarmBackendAzureBlob::new(&config).is_err(),
"the byte limit must include the canonical suffix"
);
}
#[tokio::test]
async fn version_drift_and_incorrect_ranges_fail_closed() {
let (backend, task) = fixture(vec![present(Some("new-version"))]).await;
let err = backend
.remove("changed", "")
.await
.expect_err("unversioned delete must reject version drift");
assert!(err.to_string().contains("became versioned"));
assert_eq!(task.await.expect("fixture should join").len(), 1);
for headers in [
vec![("content-range", "bytes 0-2/6")],
vec![("content-range", "bytes 1-3/6"), ("x-ms-version-id", "unexpected-version")],
] {
let (backend, task) = fixture(vec![reply(StatusCode::PARTIAL_CONTENT, &headers, b"ust")]).await;
let result = backend
.get(
"object",
"",
WarmBackendGetOpts {
start_offset: 1,
length: 3,
},
)
.await;
assert!(result.is_err(), "wrong offset or an unexpected version cannot satisfy the read");
task.await.expect("fixture should join");
}
}
#[tokio::test]
async fn unversioned_delete_keeps_the_head_precondition_when_the_blob_changes() {
let (backend, task) = fixture(vec![
present(None),
reply(StatusCode::PRECONDITION_FAILED, &[("x-ms-error-code", "ConditionNotMet")], b""),
])
.await;
assert!(backend.remove("changed", "").await.is_err());
let requests = task.await.expect("fixture should join");
assert_eq!(requests[1].headers["if-match"], "\"original\"");
}
#[tokio::test]
async fn absence_requires_blob_not_found_and_container_listing_is_bounded() {
for (response, missing_expected) in [
(missing(), true),
(reply(StatusCode::NOT_FOUND, &[("x-ms-error-code", "ContainerNotFound")], b""), false),
(reply(StatusCode::NOT_FOUND, &[], b""), false),
] {
let (backend, task) = fixture(vec![response]).await;
let result = backend.probe_transition_candidate("object").await;
if missing_expected {
assert_eq!(result.expect("explicit blob absence"), TransitionCandidateProbe::Missing);
} else {
assert!(result.is_err(), "an absent container or an ambiguous HEAD is not blob absence");
}
task.await.expect("fixture should join");
}
for (body, in_use) in [
(b"<EnumerationResults><Blobs/><NextMarker/></EnumerationResults>".as_slice(), false),
(
b"<EnumerationResults><Blobs><Blob><Name>cold/object</Name></Blob></Blobs><NextMarker/></EnumerationResults>"
.as_slice(),
true,
),
(
b"<EnumerationResults><Blobs/><NextMarker>next</NextMarker></EnumerationResults>".as_slice(),
true,
),
] {
let (backend, task) = fixture(vec![reply(StatusCode::OK, &[], body)]).await;
assert_eq!(backend.in_use().await.expect("native container listing"), in_use);
let requests = task.await.expect("fixture should join");
assert!(
requests[0]
.target
.contains("restype=container&comp=list&prefix=cold%2F&maxresults=1")
);
}
let oversized = vec![b' '; MAX_CONTROL_BYTES + 1];
let (backend, task) = fixture(vec![reply(StatusCode::OK, &[], &oversized)]).await;
assert!(backend.in_use().await.is_err(), "oversized listing cannot establish an empty destination");
task.await.expect("fixture should join");
}
#[tokio::test]
async fn recovery_requires_the_uploaded_transaction_and_destination_identity() {
let identity = TransitionCandidateIdentity {
transaction_id: uuid::Uuid::new_v4(),
destination_id: [3; 32],
};
let mut metadata = HashMap::new();
insert_str(&mut metadata, SUFFIX_TRANSITION_TRANSACTION_ID, identity.transaction_id.to_string());
insert_str(&mut metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, hex(identity.destination_id));
let headers = metadata_headers(metadata).expect("identity metadata should encode for Azure");
assert_eq!(
headers
.keys()
.filter(|name| name.as_str().starts_with(METADATA_PREFIX))
.count(),
4
);
let mut owned = present(Some("owned-version"));
owned.headers.extend(headers);
let (backend, task) = fixture(vec![owned, present(None)]).await;
assert_eq!(
backend
.probe_transition_candidate_for("object", identity)
.await
.expect("owned candidate"),
TransitionCandidateProbe::VersionedPresent("owned-version".to_string())
);
assert_eq!(
backend
.probe_transition_candidate_for("object", identity)
.await
.expect("foreign candidate classification"),
TransitionCandidateProbe::Ambiguous
);
task.await.expect("fixture should join");
}
#[test]
fn configuration_xml_and_size_limits_fail_before_network_io() {
let mut config = TierAzure {
access_key: "account".to_string(),
secret_key: base64_simd::STANDARD.encode_to_string([7; 32]),
bucket: "container".to_string(),
..Default::default()
};
assert!(
WarmBackendAzureBlob::new(&config).is_ok(),
"public endpoint should be derived from the account"
);
config.endpoint = "http://127.0.0.1:10000".to_string();
assert!(
WarmBackendAzureBlob::new(&config).is_err(),
"production constructor must apply the outbound guard"
);
config.endpoint = "https://account.blob.example/path".to_string();
assert!(
WarmBackendAzureBlob::new(&config).is_err(),
"an endpoint path cannot rewrite object addressing"
);
for name in [".", "..", "a/../b", "a/./b"] {
assert!(validate_object_name(name).is_err());
}
assert_eq!(
block_size(MAX_BLOCK_BYTES * MAX_BLOCKS).expect("largest supported object"),
MAX_BLOCK_BYTES
);
assert!(block_size(MAX_BLOCK_BYTES * MAX_BLOCKS + 1).is_err());
for xml in [
b"<html>not a container</html>".as_slice(),
b"<EnumerationResults><Blobs>".as_slice(),
b"<!DOCTYPE x><EnumerationResults><Blobs/></EnumerationResults>".as_slice(),
b"<EnumerationResults><Other><Blobs/></Other></EnumerationResults>".as_slice(),
b"garbage<EnumerationResults><Blobs/></EnumerationResults>".as_slice(),
] {
assert!(list_has_objects(xml).is_err());
}
assert_eq!(
xml_error_code(b"<Error><Code>BlobNotFound</Code></Error>").as_deref(),
Some("BlobNotFound")
);
assert!(xml_error_code(b"<Error><Code>BlobNotFound</Code>").is_none());
assert!(xml_error_code(b"<Error><Code>BlobNotFound</Code></Error>garbage").is_none());
}
#[tokio::test]
async fn control_response_has_a_total_deadline_after_headers_arrive() {
let listener = TcpListener::bind("127.0.0.1:0").await.expect("control fixture should bind");
let address = listener.local_addr().expect("control fixture address");
let (release, wait) = tokio::sync::oneshot::channel();
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.expect("control fixture should accept");
capture(&mut stream).await;
stream
.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 16\r\n\r\n")
.await
.expect("control response headers");
let _ = wait.await;
});
let (mut backend, empty) = fixture(Vec::new()).await;
empty.await.expect("unused fixture should close");
backend.endpoint = Url::parse(&format!("http://{address}")).expect("control endpoint should parse");
let response = backend
.send(Method::GET, backend.container_url().expect("container URL"), HeaderMap::new(), None)
.await
.expect("headers should arrive before testing the body deadline");
let err = tokio::time::timeout(Duration::from_secs(1), read_control_body(response, Duration::from_millis(20)))
.await
.expect("body deadline must be shorter than the transport idle timeout")
.expect_err("an unfinished control response must time out");
assert_eq!(err.kind(), ErrorKind::TimedOut);
let _ = release.send(());
server.await.expect("control fixture should join");
}
@@ -351,7 +351,7 @@ fn transition_candidate_metadata(info: ObjectInfo) -> HashMap<String, String> {
metadata
}
fn transition_candidate_metadata_matches(
pub(super) fn transition_candidate_metadata_matches(
metadata: &HashMap<String, String>,
identity: TransitionCandidateIdentity,
) -> Result<bool, std::io::Error> {
@@ -284,8 +284,9 @@ pub(in crate::set_disk) fn resolve_read_part_from_responses(
present_count += 1;
let part = &parts[part_idx];
let key = (part.etag.as_str(), part.number, part.size, part.actual_size, part.integrity.as_ref());
let (count, _) = part_quorum.entry(key).or_insert((0, part));
let (count, protected_count, _) = part_quorum.entry(key).or_insert((0, 0, part));
*count += 1;
*protected_count += usize::from(part.encrypted_etag);
continue;
}
@@ -296,12 +297,16 @@ pub(in crate::set_disk) fn resolve_read_part_from_responses(
}
}
let max_part = part_quorum.values().max_by_key(|(count, _)| count);
let max_quorum = max_part.map_or(0, |(count, _)| *count).max(missing_count);
if let Some((count, part)) = max_part
let max_part = part_quorum.values().max_by_key(|(count, _, _)| count);
let max_quorum = max_part.map_or(0, |(count, _, _)| *count).max(missing_count);
if let Some((count, protected_count, part)) = max_part
&& *count >= read_quorum
{
return Ok((*part).clone());
let mut part = (*part).clone();
// Older peers may omit the extension while agreeing on the part itself.
// Require independent proof quorum without reducing part availability.
part.encrypted_etag = *protected_count >= read_quorum;
return Ok(part);
}
if missing_count >= read_quorum {
@@ -13390,6 +13395,53 @@ mod tests {
}
}
#[test]
fn resolve_read_part_requires_matching_quorum_for_encrypted_etag_proof() {
let protected = ObjectPartInfo {
encrypted_etag: true,
..read_part_test_part(1, "protected-etag")
};
let legacy = ObjectPartInfo {
encrypted_etag: false,
..protected.clone()
};
for (parts, quorum, expected) in [
(vec![protected.clone(), legacy.clone(), legacy.clone()], 2, false),
(vec![legacy.clone(), protected.clone(), protected.clone()], 2, true),
(vec![protected.clone(), protected.clone(), legacy.clone(), legacy.clone()], 3, false),
(
vec![
protected.clone(),
ObjectPartInfo {
etag: "other-generation".to_owned(),
..protected.clone()
},
legacy.clone(),
],
2,
false,
),
(
vec![
protected.clone(),
ObjectPartInfo {
error: Some("read failed".to_owned()),
..protected.clone()
},
legacy,
],
2,
false,
),
] {
let responses = parts.into_iter().map(|part| Some(vec![part])).collect::<Vec<_>>();
let part = resolve_read_part_from_responses("bucket", "upload/part.1.meta", 1, 0, 1, &responses, quorum)
.expect("matching metadata remains readable across producer versions");
assert_eq!(part.etag, protected.etag);
assert_eq!(part.encrypted_etag, expected);
}
}
#[test]
fn resolve_read_part_diagnostic_branch_keeps_read_quorum_error() {
temp_env::with_var("RUSTFS_ISSUE3031_DIAG_ENABLE", Some("true"), || {
+64 -53
View File
@@ -7980,21 +7980,23 @@ fn is_min_allowed_part_size(size: i64) -> bool {
}
fn get_complete_multipart_md5(parts: &[CompletePart]) -> String {
let mut buf = Vec::new();
for part in parts.iter() {
if let Some(etag) = &part.etag {
if let Ok(etag_bytes) = hex_simd::decode_to_vec(etag.as_bytes()) {
buf.extend(etag_bytes);
} else {
buf.extend(etag.bytes());
let mut hasher = Md5::new();
for etag in parts.iter().filter_map(|part| part.etag.as_deref()) {
// Native part ETags decode to one fixed-size digest. Preserve the
// historical hex-or-raw interpretation for imported part ETags.
let mut digest = [0u8; 16];
if etag.len() == digest.len() * 2 {
match hex_simd::decode(etag.as_bytes(), hex_simd::Out::from_slice(&mut digest)) {
Ok(decoded) => hasher.update(decoded),
Err(_) => hasher.update(etag.as_bytes()),
}
} else if let Ok(decoded) = hex_simd::decode_to_vec(etag.as_bytes()) {
hasher.update(decoded);
} else {
hasher.update(etag.as_bytes());
}
}
let mut hasher = Md5::new();
hasher.update(&buf);
let digest = hasher.finalize();
let etag_hex = faster_hex::hex_string(digest.as_slice());
format!("{}-{}", etag_hex, parts.len())
@@ -9682,50 +9684,59 @@ mod tests {
}
#[test]
fn test_get_complete_multipart_md5() {
// Test MD5 calculation for multipart upload
let parts = vec![
fn complete_multipart_md5_preserves_legacy_etag_encodings() {
fn part(etag: Option<&str>, part_num: usize) -> CompletePart {
CompletePart {
part_num: 1,
etag: Some("d41d8cd98f00b204e9800998ecf8427e".to_string()),
checksum_crc32: None,
checksum_crc32c: None,
checksum_sha1: None,
checksum_sha256: None,
checksum_crc64nvme: None,
},
CompletePart {
part_num: 2,
etag: Some("098f6bcd4621d373cade4e832627b4f6".to_string()),
checksum_crc32: None,
checksum_crc32c: None,
checksum_sha1: None,
checksum_sha256: None,
checksum_crc64nvme: None,
},
part_num,
etag: etag.map(str::to_owned),
..Default::default()
}
}
fn buffered_digest(parts: &[CompletePart]) -> String {
let mut bytes = Vec::new();
for etag in parts.iter().filter_map(|part| part.etag.as_deref()) {
match hex_simd::decode_to_vec(etag.as_bytes()) {
Ok(decoded) => bytes.extend(decoded),
Err(_) => bytes.extend_from_slice(etag.as_bytes()),
}
}
format!("{}-{}", faster_hex::hex_string(&Md5::digest(bytes)), parts.len())
}
assert_eq!(get_complete_multipart_md5(&[]), "d41d8cd98f00b204e9800998ecf8427e-0");
assert_eq!(get_complete_multipart_md5(&[part(None, 1)]), "d41d8cd98f00b204e9800998ecf8427e-1");
assert_eq!(get_complete_multipart_md5(&[part(Some("00"), 1)]), "93b885adfe0da089cdf634904fd59f71-1");
assert_eq!(
get_complete_multipart_md5(&[
part(Some("d41d8cd98f00b204e9800998ecf8427e"), 1),
part(Some("098f6bcd4621d373cade4e832627b4f6"), 2),
]),
"f666c301a43456b66b9f3a47f696d391-2"
);
let encodings = [
None,
Some(""),
Some("00"),
Some("0aBcFf"),
Some("0123456789abcdef0123456789abcdef"),
Some("0123456789ABCDEF0123456789ABCDEF"),
Some("0123456789abcdef0123456789abcdef00"),
Some("0123456789abcdef0123456789abcdeg"),
Some("0123456789abcdef0123456789abcdeg00"),
Some("f"),
Some("not-hex"),
Some("\"0123456789abcdef0123456789abcdef\""),
Some("0123456789abcdef0123456789abcdef-2"),
Some("café"),
];
let md5 = get_complete_multipart_md5(&parts);
assert!(md5.ends_with("-2")); // Should end with part count
assert!(md5.len() > 10); // Should have reasonable length
// Test with empty parts
let empty_parts = vec![];
let empty_result = get_complete_multipart_md5(&empty_parts);
assert!(empty_result.ends_with("-0"));
// Test with single part
let single_part = vec![CompletePart {
part_num: 1,
etag: Some("d41d8cd98f00b204e9800998ecf8427e".to_string()),
checksum_crc32: None,
checksum_crc32c: None,
checksum_sha1: None,
checksum_sha256: None,
checksum_crc64nvme: None,
}];
let single_result = get_complete_multipart_md5(&single_part);
assert!(single_result.ends_with("-1"));
for first in encodings {
for second in encodings {
let parts = [part(first, 1), part(second, 2)];
assert_eq!(get_complete_multipart_md5(&parts), buffered_digest(&parts), "{first:?}, {second:?}");
}
}
}
#[test]
@@ -311,4 +311,85 @@ mod tests {
.await
.expect("single-disk no-parity PUT should pass self-verify");
}
#[tokio::test]
async fn no_parity_self_verify_rejects_missing_inline_data_missing_disk_shard_and_under_quorum() {
let (_temp_dirs, disks, _set_disks) = hermetic_set_disks_for_pool_with_default_parity(1, 0, 0).await;
let shard_size = 16usize;
let payload = b"self verify negative path";
let encoded = encode_streaming_shard(payload, shard_size).await;
let disks_opt = disks.iter().cloned().map(Some).collect::<Vec<_>>();
// 1. Inline shard with `data: None` fails closed.
let empty_parts = [FileInfo {
data: None,
..Default::default()
}];
let err = verify_written_bitrot_shards(
&disks_opt,
Some(&empty_parts),
BitrotSelfVerifyTarget {
operation: "put_object",
bucket: "bucket",
object: "missing-inline-data",
part_number: None,
volume: RUSTFS_META_TMP_BUCKET,
path: "unused",
logical_shard_size: payload.len(),
shard_size,
write_quorum: 1,
},
)
.await
.expect_err("inline shard missing data must fail");
assert!(err.to_string().contains("missing bitrot data"));
// 2. Disk shard missing on disk (FileNotFound) fails closed with shard error context.
let err = verify_written_bitrot_shards(
&disks_opt,
None,
BitrotSelfVerifyTarget {
operation: "put_object",
bucket: "bucket",
object: "missing-disk-file",
part_number: Some(3),
volume: RUSTFS_META_TMP_BUCKET,
path: "nonexistent-temp-shard",
logical_shard_size: payload.len(),
shard_size,
write_quorum: 1,
},
)
.await
.expect_err("missing disk shard must fail");
assert!(err.to_string().contains("no-parity bitrot self-verify failed for shard 0"));
// 3. Verified shard count below write_quorum fails closed even when available shard is valid.
let valid_parts = [
FileInfo {
data: Some(encoded),
..Default::default()
},
FileInfo::default(),
];
let two_slots = vec![disks_opt[0].clone(), None];
let err = verify_written_bitrot_shards(
&two_slots,
Some(&valid_parts),
BitrotSelfVerifyTarget {
operation: "put_object",
bucket: "bucket",
object: "under-quorum",
part_number: None,
volume: RUSTFS_META_TMP_BUCKET,
path: "unused",
logical_shard_size: payload.len(),
shard_size,
write_quorum: 2,
},
)
.await
.expect_err("under-quorum verification must fail");
assert!(err.to_string().contains("quorum unavailable: 1 shard(s) verified, need 2"));
}
}
+365 -16
View File
@@ -46,8 +46,8 @@ use super::super::{
log_multipart_write_quorum_failure, multipart_part_logical_size, parts_after_marker, path_join_buf,
prepare_complete_multipart_parts, record_compression_total_memory, record_multipart_write_quorum_failure,
reduce_read_quorum_errs, reduce_write_quorum_errs, remove_header_map, resolve_write_layout,
restore_commit_operation_id_from_metadata, should_persist_encryption_original_size, strip_internal_multipart_metadata,
to_object_err, warn,
restore_commit_operation_id_from_metadata, restore_operation_id_from_metadata, should_persist_encryption_original_size,
strip_internal_multipart_metadata, to_object_err, warn,
};
use super::bitrot_self_verify::{BitrotSelfVerifyTarget, drop_failed_writer_disks, verify_written_bitrot_shards};
#[cfg(test)]
@@ -2728,6 +2728,9 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
.map(crate::io_support::rio::compression_index_storage_bytes);
let mut etag = data.stream.try_resolve_etag().unwrap_or_default();
let encrypted_etag = data.stream.encrypted_etag
&& !etag.is_empty()
&& opts.preserve_etag.as_ref().is_none_or(|preserved| preserved == &etag);
if let Some(ref tag) = opts.preserve_etag {
etag = tag.clone();
@@ -2748,6 +2751,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
let part_info = ObjectPartInfo {
etag: etag.clone(),
encrypted_etag,
number: part_id,
size: w_size,
mod_time: Some(OffsetDateTime::now_utc()),
@@ -3647,6 +3651,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
}
let source_condition = capture_source_condition(bucket, object, &upload_id_path, &fi, Uuid::new_v4())?;
let preserved_encrypted_etag = preserves_staged_encrypted_etag_proof(&fi, opts, expected_restore_operation_id);
let upload_data_dir = source_condition.lineage.original_data_dir.to_string();
// The source generation remains in the captured precondition. Its
// ACTIVE/recovery controls are not final user-object metadata.
@@ -4117,11 +4122,17 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
}
// etag
let etag = resolve_complete_etag(opts, &uploaded_parts);
let (etag, encrypted_etag) = resolve_complete_etag(opts, &uploaded_parts, &fi.parts);
let encrypted_etag = encrypted_etag
|| preserved_encrypted_etag
&& fi.version_id.filter(|version| !version.is_nil()) == source_condition.lineage.original_version_id;
// An upload can contain parts from an older writer or a peer. Until
// every part's format is attested, keep its ETag out of keyless lists.
// New writes attest their selected parts. Same-version internal moves
// and restores can retain the proof from quorum-validated source metadata.
rustfs_utils::http::metadata_compat::remove_str(&mut fi.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG);
if encrypted_etag {
insert_str(&mut fi.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
}
fi.metadata.insert("etag".to_owned(), etag);
@@ -4948,19 +4959,57 @@ fn completed_multipart_upload_matches(
Ok(true)
}
// Capture only the original staged proof, before completion applies metadata
// updates. Publication must also retain the source lineage's version.
fn preserves_staged_encrypted_etag_proof(fi: &FileInfo, opts: &ObjectOptions, restore_operation_id: Option<Uuid>) -> bool {
if opts.replication_request {
return false;
}
let movement = opts.data_movement
&& opts.version_id.is_some()
&& get_consistent_str(&fi.metadata, rustfs_utils::http::SUFFIX_DATA_MOVEMENT_UPLOAD)
.is_some_and(|identity| !identity.is_empty());
// Commit rechecks this operation against the live object under its lock.
let restore = restore_operation_id
.is_some_and(|expected| restore_operation_id_from_metadata(&fi.metadata).is_ok_and(|id| id == Some(expected)));
if !movement && !restore {
return false;
}
let Ok(version) = opts.version_id.as_deref().map(Uuid::parse_str).transpose() else {
return false;
};
let Some(etag) = opts.preserve_etag.as_deref().filter(|etag| !etag.is_empty()) else {
return false;
};
version.filter(|version| !version.is_nil()) == fi.version_id.filter(|version| !version.is_nil())
&& fi.metadata.get("etag").map(String::as_str) == Some(etag)
&& get_consistent_str(&fi.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG) == Some("v1")
}
/// Final ETag for a completed multipart object. An authorized replication
/// request preserves the source ETag so the replication HEAD comparison
/// converges even when the source ETag is not derivable from the uploaded
/// parts (foreign-origin objects, ciphertext-derived ETags); the internal
/// metadata override comes next; otherwise the ETag is computed from parts.
fn resolve_complete_etag(opts: &ObjectOptions, uploaded_parts: &[CompletePart]) -> String {
if let Some(etag) = opts.preserve_etag.as_ref().filter(|etag| !etag.is_empty()) {
return etag.clone();
fn resolve_complete_etag(
opts: &ObjectOptions,
uploaded_parts: &[CompletePart],
stored_parts: &[ObjectPartInfo],
) -> (String, bool) {
let preserved = opts
.preserve_etag
.as_ref()
.filter(|etag| !etag.is_empty())
.or_else(|| opts.user_defined.get("etag"));
let protected_parts = !stored_parts.is_empty() && stored_parts.iter().all(|part| part.encrypted_etag);
if let Some(etag) = preserved
&& !protected_parts
{
return (etag.clone(), false);
}
if let Some(etag) = opts.user_defined.get("etag") {
return etag.clone();
}
get_complete_multipart_md5(uploaded_parts)
let computed = get_complete_multipart_md5(uploaded_parts);
let encrypted_etag = protected_parts && preserved.is_none_or(|etag| etag == &computed);
(preserved.cloned().unwrap_or(computed), encrypted_etag)
}
/// Serialize the object-level checksum record that `complete_multipart_upload`
@@ -9093,6 +9142,26 @@ mod tests {
receipts.push(part.etag);
}
}
let (staged, _) = set
.check_upload_id_exists(bucket, "object", &upload.upload_id, false)
.await
.expect("read upload generation");
let upload_path = SetDisks::get_upload_id_dir(bucket, "object", &upload.upload_id);
let data_dir = staged.data_dir.expect("staging data directory");
for disk in &disks {
let metadata = disk
.read_parts(
RUSTFS_META_MULTIPART_BUCKET,
&[
format!("{upload_path}/{data_dir}/part.1.meta"),
format!("{upload_path}/{data_dir}/part.3.meta"),
],
)
.await
.expect("read persisted part proofs");
assert!(metadata[0].encrypted_etag, "rewritten protected part is attested");
assert!(!metadata[1].encrypted_etag, "an overridden part ETag has no producer proof");
}
assert_ne!(receipts[0], receipts[1], "spare parts must not identify another part's plaintext");
assert_ne!(receipts[0], receipts[2], "same-number retries get a new receipt");
for _ in 0..2 {
@@ -9129,6 +9198,230 @@ mod tests {
.await
.expect("committed completion replay is stable");
assert_eq!(replay.etag, committed.etag);
for completed in [committed, replay] {
assert!(
rustfs_utils::http::etag_can_be_listed(&completed.user_defined),
"the selected protected part retains its ETag after completion and replay"
);
assert_eq!(
rustfs_utils::http::get_consistent_str(&completed.user_defined, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG,),
Some("v1")
);
}
}
#[tokio::test]
async fn complete_encrypted_multipart_clears_unproven_upload_etag_marker() {
let (_dirs, disks, set) = hermetic_set_disks(4).await;
let bucket = "legacy-encrypted-etag";
make_bucket_on_all(&disks, bucket).await;
let mut opts = ObjectOptions {
user_defined: HashMap::from([(SSEC_ALGORITHM_HEADER.to_owned(), "AES256".to_owned())]),
..Default::default()
};
insert_str(&mut opts.user_defined, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
// Model an older writer whose part metadata has no protected-format proof.
let (upload_id, parts) = stage_upload_with_create_opts(&set, bucket, "object", b"legacy part", &opts).await;
let completed = set
.clone()
.complete_multipart_upload(bucket, "object", &upload_id, parts, &ObjectOptions::default())
.await
.expect("legacy part can still complete");
let reread = set
.get_object_info(bucket, "object", &ObjectOptions::default())
.await
.expect("read completed metadata");
assert_eq!(completed.etag, reread.etag);
for info in [completed, reread] {
assert!(!rustfs_utils::http::etag_can_be_listed(&info.user_defined));
assert!(!contains_key_str(&info.user_defined, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG));
}
}
#[test]
fn data_movement_etag_proof_requires_verified_source_identity() {
let version = Uuid::new_v4();
let mut source = FileInfo {
version_id: Some(version),
metadata: HashMap::from([("etag".to_owned(), "protected-source-etag".to_owned())]),
..Default::default()
};
insert_str(
&mut source.metadata,
rustfs_utils::http::SUFFIX_DATA_MOVEMENT_UPLOAD,
"source-generation".to_owned(),
);
insert_str(&mut source.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
let opts = ObjectOptions {
data_movement: true,
version_id: Some(version.to_string()),
preserve_etag: source.metadata.get("etag").cloned(),
..Default::default()
};
assert!(preserves_staged_encrypted_etag_proof(&source, &opts, None));
for version_id in [None, Some(Uuid::nil())] {
let null_source = FileInfo {
version_id,
..source.clone()
};
let null_opts = ObjectOptions {
version_id: Some(Uuid::nil().to_string()),
..opts.clone()
};
assert!(preserves_staged_encrypted_etag_proof(&null_source, &null_opts, None));
}
for rejected in [
ObjectOptions {
data_movement: false,
..opts.clone()
},
ObjectOptions {
replication_request: true,
..opts.clone()
},
ObjectOptions {
version_id: None,
..opts.clone()
},
ObjectOptions {
version_id: Some("invalid".to_owned()),
..opts.clone()
},
ObjectOptions {
version_id: Some(Uuid::new_v4().to_string()),
..opts.clone()
},
ObjectOptions {
preserve_etag: None,
..opts.clone()
},
ObjectOptions {
preserve_etag: Some(String::new()),
..opts.clone()
},
ObjectOptions {
preserve_etag: Some("replacement-etag".to_owned()),
..opts.clone()
},
] {
assert!(!preserves_staged_encrypted_etag_proof(&source, &rejected, None));
}
for suffix in [
rustfs_utils::http::SUFFIX_DATA_MOVEMENT_UPLOAD,
rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG,
] {
let mut missing = source.clone();
rustfs_utils::http::remove_str(&mut missing.metadata, suffix);
assert!(!preserves_staged_encrypted_etag_proof(&missing, &opts, None));
insert_str(&mut missing.metadata, suffix, String::new());
assert!(!preserves_staged_encrypted_etag_proof(&missing, &opts, None));
let mut conflicting = source.clone();
conflicting
.metadata
.insert(format!("x-minio-internal-{suffix}"), "conflict".to_owned());
assert!(!preserves_staged_encrypted_etag_proof(&conflicting, &opts, None));
}
for value in [None, Some("unknown")] {
let mut unproven = source.clone();
rustfs_utils::http::remove_str(&mut unproven.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG);
if let Some(value) = value {
insert_str(&mut unproven.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, value.to_owned());
}
let request_claim = ObjectOptions {
user_defined: source.metadata.clone(),
..opts.clone()
};
assert!(!preserves_staged_encrypted_etag_proof(&unproven, &request_claim, None));
}
}
#[test]
fn restore_etag_proof_requires_matching_staged_operation() {
let operation_id = Uuid::new_v4();
for version_id in [None, Some(Uuid::new_v4())] {
let mut staged = FileInfo {
version_id,
metadata: HashMap::from([("etag".to_owned(), "source-etag".to_owned())]),
..Default::default()
};
let opts = ObjectOptions {
version_id: version_id.map(|id| id.to_string()),
preserve_etag: Some("source-etag".to_owned()),
..Default::default()
};
insert_str(&mut staged.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
assert!(!preserves_staged_encrypted_etag_proof(&staged, &opts, Some(operation_id)));
insert_str(&mut staged.metadata, SUFFIX_RESTORE_OPERATION_ID, operation_id.to_string());
assert!(preserves_staged_encrypted_etag_proof(&staged, &opts, Some(operation_id)));
assert!(!preserves_staged_encrypted_etag_proof(&staged, &opts, Some(Uuid::new_v4())));
rustfs_utils::http::remove_str(&mut staged.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG);
assert!(!preserves_staged_encrypted_etag_proof(&staged, &opts, Some(operation_id)));
}
}
#[tokio::test]
async fn complete_data_movement_etag_proof_rejects_version_rewrites_and_late_claims() {
let (_dirs, disks, set) = hermetic_set_disks(4).await;
let bucket = "movement-etag-proof";
make_bucket_on_all(&disks, bucket).await;
for (object, staged_proof, publish_versioned, late_claim, expected_proof) in [
("same-version", true, true, false, true),
("rewritten-version", true, false, false, false),
("late-claim", false, true, true, false),
] {
let version = Uuid::new_v4();
let mut opts = ObjectOptions {
data_movement: true,
versioned: true,
version_id: Some(version.to_string()),
preserve_etag: Some("protected-source-etag".to_owned()),
user_defined: HashMap::from([(SSEC_ALGORITHM_HEADER.to_owned(), "AES256".to_owned())]),
..Default::default()
};
insert_str(
&mut opts.user_defined,
rustfs_utils::http::SUFFIX_DATA_MOVEMENT_UPLOAD,
"source-generation".to_owned(),
);
if staged_proof {
insert_str(&mut opts.user_defined, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
}
let (upload_id, parts) = stage_upload_with_create_opts(&set, bucket, object, b"raw source bytes", &opts).await;
opts.versioned = publish_versioned;
if late_claim {
let mut metadata = HashMap::new();
insert_str(&mut metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
opts.eval_metadata = Some(metadata);
}
let completed = set
.clone()
.complete_multipart_upload(bucket, object, &upload_id, parts, &opts)
.await
.expect("complete raw internal movement");
assert_eq!(completed.version_id, publish_versioned.then_some(version), "{object}");
assert_eq!(completed.etag, opts.preserve_etag, "{object}");
let persisted = set
.get_object_info(
bucket,
object,
&ObjectOptions {
version_id: Some(crate::data_movement::data_movement_exact_version_id(completed.version_id)),
..Default::default()
},
)
.await
.expect("read completed proof from disk");
assert_eq!(
rustfs_utils::http::etag_can_be_listed(&persisted.user_defined),
expected_proof,
"{object}"
);
assert_eq!(
get_consistent_str(&persisted.user_defined, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG),
expected_proof.then_some("v1"),
"{object}"
);
}
}
#[tokio::test]
@@ -16205,7 +16498,7 @@ mod tests {
preserve_etag: Some(foreign_etag.clone()),
..Default::default()
};
assert_eq!(resolve_complete_etag(&opts, &[]), foreign_etag);
assert_eq!(resolve_complete_etag(&opts, &[], &[]), (foreign_etag, false));
// Empty preserve value degrades to the next source.
let opts_empty = ObjectOptions {
@@ -16213,11 +16506,67 @@ mod tests {
user_defined: std::collections::HashMap::from([("etag".to_string(), "override-etag".to_string())]),
..Default::default()
};
assert_eq!(resolve_complete_etag(&opts_empty, &[]), "override-etag");
assert_eq!(resolve_complete_etag(&opts_empty, &[], &[]), ("override-etag".to_string(), false));
// Without either source the ETag is computed from the parts.
let computed = resolve_complete_etag(&ObjectOptions::default(), &[]);
assert_eq!(computed, get_complete_multipart_md5(&[]));
let computed = resolve_complete_etag(&ObjectOptions::default(), &[], &[]);
assert_eq!(computed, (get_complete_multipart_md5(&[]), false));
}
#[test]
fn complete_etag_proof_requires_all_selected_parts_and_matching_overrides() {
for proofs in [&[][..], &[true], &[true, true], &[true, false], &[false]] {
let stored: Vec<_> = proofs
.iter()
.enumerate()
.map(|(index, &encrypted_etag)| ObjectPartInfo {
number: index + 1,
etag: "0123456789abcdef0123456789abcdef".to_owned(),
encrypted_etag,
..Default::default()
})
.collect();
let uploaded: Vec<_> = stored
.iter()
.map(|part| CompletePart {
part_num: part.number,
etag: Some(part.etag.clone()),
..Default::default()
})
.collect();
let computed = get_complete_multipart_md5(&uploaded);
let protected = !proofs.is_empty() && proofs.iter().all(|&proof| proof);
assert_eq!(
resolve_complete_etag(&ObjectOptions::default(), &uploaded, &stored),
(computed.clone(), protected)
);
for preserved in ["foreign-etag", computed.as_str(), ""] {
for opts in [
ObjectOptions {
preserve_etag: Some(preserved.to_owned()),
..Default::default()
},
ObjectOptions {
user_defined: HashMap::from([("etag".to_owned(), preserved.to_owned())]),
..Default::default()
},
ObjectOptions {
preserve_etag: Some(preserved.to_owned()),
user_defined: HashMap::from([("etag".to_owned(), "metadata-etag".to_owned())]),
..Default::default()
},
] {
let expected_etag = if !preserved.is_empty() || opts.preserve_etag.is_none() {
preserved
} else {
opts.user_defined.get("etag").map_or(computed.as_str(), String::as_str)
};
let result = resolve_complete_etag(&opts, &uploaded, &stored);
assert_eq!(result, (expected_etag.to_owned(), protected && expected_etag == computed));
}
}
}
}
/// Accumulate the per-part digests and the running full-object checksum the
+57 -24
View File
@@ -9103,6 +9103,9 @@ impl SetDisks {
let mut futures = Vec::with_capacity(disks.len());
let mut errs = Vec::with_capacity(disks.len());
let requires_bounded_delete_recovery = commit_context
.options
.is_some_and(|options| options.delete_recovery_work_budget.is_some());
for disk in disks.iter() {
let disk_namespace_owner = namespace_owner.clone().map(|owner| owner as Arc<dyn Send + Sync>);
let transaction_guard = commit_context.transaction_guard.as_ref();
@@ -9114,6 +9117,7 @@ impl SetDisks {
let options = DeleteOptions {
old_data_dir: Some(rollback_dir),
delete_recovery_bucket_incarnation: source_bucket_incarnation_id,
requires_bounded_delete_recovery,
..Default::default()
};
let result = if is_meta_bucketname(bucket) {
@@ -10557,6 +10561,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
.collect();
let mut futures = Vec::with_capacity(disks.len());
let lock_lost_during_commit = Arc::new(std::sync::atomic::AtomicBool::new(false));
let requires_bounded_delete_recovery = opts.delete_recovery_work_budget.is_some();
// let mut errors = Vec::with_capacity(disks.len());
@@ -10574,6 +10579,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
let options = DeleteOptions {
old_data_dir: Some(rollback_dir),
delete_recovery_bucket_incarnation: opts.expected_bucket_incarnation_id,
requires_bounded_delete_recovery,
..Default::default()
};
if current_generations
@@ -17173,6 +17179,9 @@ mod transition_commit_failure_tests {
#[tokio::test]
#[serial_test::serial]
async fn multipart_restore_copies_the_stored_representation_back_verbatim() {
use crate::io_support::rio::{WriteEncryption, WritePlan};
use rustfs_utils::http::{SUFFIX_OPAQUE_ENCRYPTED_ETAG, etag_can_be_listed};
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "restore-multipart-stored-coordinates-bucket";
let object = "object.bin";
@@ -17180,9 +17189,8 @@ mod transition_commit_failure_tests {
disk.make_volume(bucket).await.expect("bucket volume should be created");
}
// The minimum-part-size gate reads the PLAINTEXT length, so part one
// clears 5 MiB there while its stored form carries encoding overhead.
let part_shapes = [(6 * 1024 * 1024_usize, 9_216_usize), (256 * 1024_usize, 512_usize)];
// The first plaintext part clears the non-final 5 MiB threshold.
let part_sizes = [6 * 1024 * 1024_usize, 256 * 1024_usize];
let mut user_defined = HashMap::new();
user_defined.insert(rustfs_utils::http::INTERNAL_ENCRYPTION_ALGORITHM_HEADER.to_string(), "AES256".to_string());
user_defined.insert(
@@ -17203,20 +17211,22 @@ mod transition_commit_failure_tests {
.expect("multipart upload should be created");
let mut uploaded_parts = Vec::new();
let mut expected_stored = Vec::new();
for (index, (plaintext_len, overhead)) in part_shapes.iter().enumerate() {
let stored = stored_representation_bytes(index as u32 * 7 + 1, plaintext_len + overhead);
expected_stored.extend_from_slice(&stored);
let stored_len = stored.len() as i64;
let hash_reader =
HashReader::from_stream(std::io::Cursor::new(stored), stored_len, *plaintext_len as i64, None, None, false)
.expect("hash reader over the stored representation");
let mut reader = PutObjReader::new(hash_reader);
for (index, &part_size) in part_sizes.iter().enumerate() {
let plaintext = stored_representation_bytes(index as u32 * 7 + 1, part_size);
let actual_size = part_size as i64;
let plaintext_reader =
HashReader::from_stream(std::io::Cursor::new(plaintext), actual_size, actual_size, None, None, false)
.expect("plaintext part reader");
let encrypted_reader = WritePlan::new()
.with_encryption(WriteEncryption::multipart([7; 32], [9; 12], index + 1))
.apply(plaintext_reader, actual_size)
.expect("native protected encrypted writer");
let mut reader = PutObjReader::new(encrypted_reader);
let info = set_disks
.put_object_part(bucket, object, &upload.upload_id, index + 1, &mut reader, &ObjectOptions::default())
.await
.expect("stored part should be staged");
assert_eq!(info.size as i64, stored_len, "a part is stored in its encoded length");
assert!(info.size as i64 > actual_size, "part contains actual encryption overhead");
uploaded_parts.push(CompletePart {
part_num: info.part_num,
etag: info.etag,
@@ -17240,8 +17250,12 @@ mod transition_commit_failure_tests {
"the fixture must keep the two coordinate systems apart: stored {size} vs plaintext {actual_size}"
);
}
let stored_before = read_stored_representation(&set_disks, bucket, object).await;
assert_stored_representation_eq(&stored_before, &expected_stored, "the fixture must store its encoded bytes verbatim");
assert_eq!(
rustfs_utils::http::get_consistent_str(&original.user_defined, SUFFIX_OPAQUE_ENCRYPTED_ETAG),
Some("v1")
);
assert!(etag_can_be_listed(&original.user_defined), "the source has a real producer-attested ETag");
let expected_stored = read_stored_representation(&set_disks, bucket, object).await;
let tier_name = format!("COLDTIER{}", &Uuid::new_v4().simple().to_string()[..8]).to_uppercase();
register_mock_tier(&runtime_sources::global_tier_config_mgr(), &tier_name).await;
@@ -17265,6 +17279,16 @@ mod transition_commit_failure_tests {
.await
.expect("multipart source should transition before restore");
let transitioned = set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("read metadata after actual tier transition");
assert_eq!(
rustfs_utils::http::get_consistent_str(&transitioned.user_defined, SUFFIX_OPAQUE_ENCRYPTED_ETAG),
Some("v1")
);
assert!(etag_can_be_listed(&transitioned.user_defined), "transition retains the source proof");
let operation_id = Uuid::new_v4();
set_disks
.put_object_metadata(
@@ -17310,6 +17334,11 @@ mod transition_commit_failure_tests {
// backlog#2369 P7.1: the copy-back digests stored bytes, so the object's
// public ETag has to be carried over rather than recomputed.
assert_eq!(restored.etag, original.etag, "restore must preserve the object ETag");
assert_eq!(restored.version_id, original.version_id, "restore keeps the same version");
assert!(
etag_can_be_listed(&restored.user_defined),
"the restored version retains its protected ETag proof"
);
}
#[tokio::test]
@@ -20432,7 +20461,7 @@ mod transition_upload_integrity_tests {
assert_local_source_intact(&set_disks, bucket, object, &payload).await;
}
#[tokio::test(flavor = "multi_thread")]
#[tokio::test(flavor = "current_thread")]
#[serial_test::serial]
async fn commit_lock_acquire_failure_cleans_remote_candidate_and_preserves_source() {
let manager = Arc::new(rustfs_lock::GlobalLockManager::new());
@@ -20441,7 +20470,16 @@ mod transition_upload_integrity_tests {
let bucket = "transition-lock-acquire-failure-bucket";
let object = "object.bin";
let payload = b"transition commit lock failure must clean the remote candidate".repeat(1024);
let original = write_source(&set_disks, &disk_stores, bucket, object, &payload).await;
let original = temp_env::async_with_vars(
[(
crate::set_disk::core::io_primitives::ENV_RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE,
Some("false"),
)],
async { write_source(&set_disks, &disk_stores, bucket, object, &payload).await },
)
.await;
// Complete source I/O before controlling the quorum retry deadline.
tokio::time::pause();
let tier_name = format!("COLDTIER{}", &Uuid::new_v4().simple().to_string()[..8]).to_uppercase();
let backend = register_mock_tier(&runtime_sources::global_tier_config_mgr(), &tier_name).await;
let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::DistErasure).await;
@@ -20453,13 +20491,8 @@ mod transition_upload_integrity_tests {
.await
.expect_err("transition must fail when the commit namespace write lock cannot reach quorum");
assert!(
matches!(
error,
StorageError::NamespaceLockQuorumUnavailable { .. }
| StorageError::Lock(LockError::QuorumNotReached { .. })
| StorageError::Lock(LockError::Internal { .. })
),
"unexpected transition lock-acquire error: {error:?}"
matches!(error, StorageError::NamespaceLockQuorumUnavailable { required: 2, .. }),
"the injected permanent quorum failure must reach the transition commit boundary: {error:?}"
);
assert_eq!(
backend.put_count().await,
+39 -2
View File
@@ -2979,10 +2979,11 @@ fn build_legacy_per_part_fallback_reader(
part_length: usize,
part_size: usize,
read_repair: Option<(DecodeReadRepairContext, bool)>,
) -> Box<dyn AsyncRead + Unpin + Send + Sync> {
) -> Box<LegacyPerPartDecodeReader> {
let buffer = adaptive_duplex_buffer_size(part_size as i64);
let (read_half, mut write_half) = tokio::io::duplex(buffer);
let (read_half, write_half) = tokio::io::duplex(buffer);
let decode = tokio::spawn(async move {
let mut write_half = GetObjectDownstreamWriter::new(write_half);
let outcome = erasure
.decode_with_stripe_handles_and_reopeners_with_diagnostics(
&mut write_half,
@@ -8908,6 +8909,42 @@ mod tests {
assert!(err.to_string().contains("inconsistent read source shards"));
}
#[tokio::test]
async fn legacy_per_part_output_close_retains_downstream_classification() {
let erasure = coding::Erasure::new(2, 2, 64);
let data = (0..64u8).collect::<Vec<_>>();
let setup =
setup_codec_data_blocks_first_encoded_bitrot_readers(&erasure, &data, &[0], &[], &[], HashAlgorithm::HighwayHash256)
.await;
let mut reader = build_legacy_per_part_fallback_reader(
erasure,
setup.readers,
setup.deferred_stripe_handles,
Vec::new(),
None,
0,
data.len(),
data.len(),
None,
);
// Closing the actual output pipe models the body consumer disappearing
// while retaining the executor long enough to inspect its typed result.
assert!(!reader.finished);
let (replacement, _unused) = tokio::io::duplex(1);
drop(std::mem::replace(&mut reader.inner, replacement));
let handle = reader.decode.take().expect("fallback decode task");
let error = handle
.await
.expect("join the fallback task")
.expect("closed pipe must interrupt output");
assert_eq!(error.kind(), std::io::ErrorKind::BrokenPipe);
assert_eq!(
crate::diagnostics::get::classify_io_error(&error),
GetObjectFailureReason::DownstreamClosed,
"fallback output must carry the same marker as the normal GET path"
);
}
// backlog#879: a later multipart part that loses a shard (read quorum still
// holds) must be reconstructed in place through the legacy per-part decode
// bridge instead of failing the stream. This exercises the exact reader the
+6
View File
@@ -24035,6 +24035,7 @@ mod tests {
without_storage_class_env(build_isolated_test_store(temp_dir.path(), "tier-mutation-intent-record", &[4])).await;
let mutation_id = uuid::Uuid::new_v4();
let intent = TierMutationIntent {
requires_native_azure: false,
mutation_id,
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -24220,6 +24221,7 @@ mod tests {
let (_ctx, store, _shutdown) =
without_storage_class_env(build_isolated_test_store(temp_dir.path(), "tier-mutation-intent-scan", &[4])).await;
let build_intent = |mutation_id: uuid::Uuid, tier_name: &str| TierMutationIntent {
requires_native_azure: false,
mutation_id,
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -24281,6 +24283,7 @@ mod tests {
without_storage_class_env(build_isolated_test_store(temp_dir.path(), "tier-mutation-intent-advance", &[4])).await;
let mutation_id = uuid::Uuid::new_v4();
let intent = TierMutationIntent {
requires_native_azure: false,
mutation_id,
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -24355,6 +24358,7 @@ mod tests {
let abort_id = uuid::Uuid::new_v4();
let abort_intent = TierMutationIntent {
requires_native_azure: false,
mutation_id: abort_id,
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -24397,6 +24401,7 @@ mod tests {
candidate_digest: [u8; 32],
) -> TierMutationIntent {
TierMutationIntent {
requires_native_azure: false,
mutation_id,
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -25473,6 +25478,7 @@ mod tests {
let (_ctx, store, _shutdown) =
without_storage_class_env(build_isolated_test_store(temp_dir.path(), "tier-mutation-intent-scan-page", &[4])).await;
let build_intent = |mutation_id: uuid::Uuid, tier_name: &str| TierMutationIntent {
requires_native_azure: false,
mutation_id,
revision: 1,
kind: TierMutationIntentKind::Edit,
@@ -318,7 +318,10 @@ mod multipart_copy_heal_tests {
let mut corrupt = original.clone();
corrupt[0] ^= 0x80;
std::fs::write(&shard, &corrupt).expect("actual bitrot corruption in shared physical shard");
std::fs::File::open(&shard)
// Windows only flushes a handle opened for writing.
std::fs::OpenOptions::new()
.write(true)
.open(&shard)
.expect("open corrupted actual shard")
.sync_all()
.expect("persist actual fault bytes");
+68
View File
@@ -368,6 +368,9 @@ fn log_list_path_finished(
#[derive(Debug, Default, Clone)]
pub struct ListPathOptions {
pub caller: ListObjectsGatherCaller,
/// Internal native-walk admission, shared with every disk producer so
/// aborting an HTTP request cannot release its budget ahead of cleanup.
pub read_budget: Option<Arc<tokio::sync::OwnedSemaphorePermit>>,
pub id: Option<String>,
// Bucket of the listing.
@@ -1478,6 +1481,11 @@ impl ECStore {
if o.transient {
o.create = false;
}
let fanout = self.pools.iter().fold(0usize, |total, pool| {
total.saturating_add(pool.disk_set.len().saturating_mul(pool.set_drive_count))
});
o.read_budget = self.ctx.acquire_list_read_budget(fanout).await.map_err(Error::other)?;
let log_context = ListPathLogContext::from_options(&o);
// cancel channel
@@ -2878,6 +2886,12 @@ impl SetDisks {
if o.transient {
o.create = false;
}
o.read_budget = self
.instance_ctx()
.acquire_list_read_budget(self.set_drive_count)
.await
.map_err(Error::other)?;
let log_context = ListPathLogContext::from_options(&o);
let cancel = CancellationToken::new();
@@ -2949,6 +2963,7 @@ impl SetDisks {
let result = list_path_raw_with_claim_tracker(
rx,
ListPathRawOptions {
read_budget: opts.read_budget.clone(),
disks: disks.into_iter().map(Some).collect(),
bucket: opts.bucket,
path: opts.base_dir,
@@ -3107,6 +3122,7 @@ impl SetDisks {
let result = list_path_raw_with_outcome(
rx,
ListPathRawOptions {
read_budget: opts.read_budget.clone(),
disks: disks.into_iter().map(Some).collect(),
fallback_disks: fallback_disks.into_iter().map(Some).collect(),
bucket: opts.bucket,
@@ -6788,6 +6804,58 @@ mod test {
(dirs, store)
}
#[tokio::test]
async fn list_read_budget_bounds_store_and_set_entrypoints_and_recovers_after_errors() {
let (_dirs, store) = single_set_store(4).await;
let held = store
.ctx
.acquire_list_read_budget(usize::MAX)
.await
.expect("occupy native budget");
let semaphore = held.as_ref().expect("bounded instance budget").semaphore();
let owners = Arc::strong_count(semaphore);
let opts = ListPathOptions {
bucket: "list-budget-absent-bucket".into(),
limit: 1,
authoritative: true,
..Default::default()
};
let set = store.pools[0].disk_set[0].clone();
set.disks.write().await.fill(None);
{
let mut store_walk = std::pin::pin!(store.clone().list_path(&opts));
let mut set_walk = std::pin::pin!(set.clone().list_path_result(&opts));
assert!(futures::poll!(&mut store_walk).is_pending(), "store walk must await its budget");
assert_eq!(Arc::strong_count(semaphore), owners + 1, "store must queue on the shared budget");
assert!(futures::poll!(&mut set_walk).is_pending(), "set walk shares the same instance budget");
assert_eq!(Arc::strong_count(semaphore), owners + 2, "set must queue on the same budget");
let empty = store
.clone()
.list_path(&ListPathOptions {
limit: 0,
..opts.clone()
})
.await
.expect("empty page needs no disk admission");
assert!(empty.exhausted);
}
assert_eq!(Arc::strong_count(semaphore), owners, "cancelled entry points release the queue ownership");
drop(held);
let result = timeout(Duration::from_secs(5), store.clone().list_path(&opts))
.await
.expect("failed walk terminates");
assert_eq!(
result.err(),
Some(DiskError::ErasureReadQuorum.into()),
"offline drives retain their quorum error"
);
let next = timeout(Duration::from_secs(5), store.ctx.acquire_list_read_budget(usize::MAX))
.await
.expect("failed and cancelled walks release their grants")
.expect("budget remains open");
assert!(next.is_some());
}
fn versioned_object_metadata(bucket: &str, name: &str, versions: &[(u128, i64)], slot: usize) -> Bytes {
let mut meta = FileMeta::new();
for (id, modified) in versions {
+18 -3
View File
@@ -3218,7 +3218,12 @@ impl ECStore {
pub async fn object_lock_config_snapshot(&self, bucket: &str) -> Result<Arc<ObjectLockConfigSnapshot>> {
check_valid_bucket_name(bucket)?;
let lifecycle_guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
let metadata_guard = acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await?;
let metadata_guard = crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_read_lock_in_for_purpose(
&self.ctx,
bucket,
"object_lock_admission_metadata_read",
)
.await?;
let (state, bucket_incarnation_id, config_revision) =
get_object_lock_config_and_incarnation_from_disk_with_guard_in(&self.ctx, bucket, Some(&lifecycle_guard)).await?;
if lifecycle_guard.is_lock_lost() || metadata_guard.is_lock_lost() {
@@ -3238,7 +3243,12 @@ impl ECStore {
pub(crate) async fn object_lock_config_snapshot_for_single_put(&self, bucket: &str) -> Result<Arc<ObjectLockConfigSnapshot>> {
check_valid_bucket_name(bucket)?;
let lifecycle_guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
let metadata_guard = acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await?;
let metadata_guard = crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_read_lock_in_for_purpose(
&self.ctx,
bucket,
"object_lock_snapshot_metadata_read",
)
.await?;
let (state, bucket_incarnation_id, config_revision, quota_res) =
get_object_lock_and_quota_config_from_disk_with_guard_in(&self.ctx, bucket, Some(&lifecycle_guard)).await?;
if lifecycle_guard.is_lock_lost() || metadata_guard.is_lock_lost() {
@@ -3267,7 +3277,12 @@ impl ECStore {
if lifecycle_fence.is_lock_lost() {
return Err(Error::other("bucket lifecycle lock was lost before loading the Object Lock snapshot"));
}
let metadata_guard = acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await?;
let metadata_guard = crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_read_lock_in_for_purpose(
&self.ctx,
bucket,
"object_lock_completion_metadata_read",
)
.await?;
let (state, bucket_incarnation_id, config_revision) =
get_object_lock_config_and_incarnation_from_disk_with_guard_in(&self.ctx, bucket, Some(&metadata_guard)).await?;
if lifecycle_fence.is_lock_lost() || metadata_guard.is_lock_lost() {
+160 -2
View File
@@ -445,6 +445,13 @@ impl ECStore {
return Ok(());
}
let scoped_cleanup = opts
.scanner_publication_commit_scope
.as_ref()
.zip(opts.http_preconditions.as_ref())
.filter(|_| bucket == RUSTFS_META_BUCKET && opts.delete_prefix_object);
let _cleanup_outcome =
scoped_cleanup.map(|(scope, _)| crate::object_api::ScannerPublicationCommitScopeGuard::new(scope.clone()));
let mut first_error = None;
let mut first_volume_error = None;
let mut has_success = false;
@@ -457,9 +464,30 @@ impl ECStore {
}
let mut opts = opts.clone();
opts.delete_prefix = true;
if let Some((scope, preconditions)) = scoped_cleanup {
// The exact-object delete retains its namespace writer across
// this lookup and dispatch. Never remove an unobserved revision.
let lookup = ObjectOptions {
no_lock: true,
suppress_read_repair: true,
preserve_delete_marker: true,
..Default::default()
};
let disks = pool.get_disks_by_key(object);
let current = disks.get_object_info(bucket, object, &lookup).await;
if let Some(error) =
crate::set_disk::SetDisks::evaluate_write_precondition(current, preconditions, bucket, object, &lookup)
{
if matches!(error, StorageError::PreconditionFailed) || crate::error::is_err_object_not_found(&error) {
continue;
}
return Err(error);
}
opts.scanner_publication_commit_scope = Some(scope.pool_delete_participant());
}
match pool.delete_object(bucket, object, opts).await {
Ok(_) => has_success = true,
Err(err) if is_err_strict_volume_not_found(&err) => {
Err(err) if scoped_cleanup.is_none() && is_err_strict_volume_not_found(&err) => {
if first_volume_error.is_none() {
first_volume_error = Some(err);
}
@@ -472,14 +500,21 @@ impl ECStore {
}
}
match first_error {
let result = match first_error {
Some(err) => Err(err),
None if has_success => Ok(()),
None => match first_volume_error {
Some(err) => Err(err),
None if scoped_cleanup.is_some() => Err(StorageError::PreconditionFailed),
None => Ok(()),
},
};
if let Some((scope, _)) = scoped_cleanup
&& result.is_ok()
{
let _ = scope.mark_committed();
}
result
}
pub(super) async fn get_available_pool_idx(&self, bucket: &str, object: &str, size: i64) -> Option<usize> {
@@ -1379,6 +1414,129 @@ mod tests {
(temp_dir, store, shutdown)
}
async fn run_scanner_observed_cleanup(
if_match_override: Option<&str>,
second_payload: Option<&[u8]>,
lose_second_quorum: bool,
) -> (
Result<crate::object_api::ObjectInfo>,
crate::object_api::ScannerPublicationCommitState,
Vec<bool>,
) {
use crate::config::com::save_config_with_opts_and_metadata;
use crate::storage_api_contracts::object::HTTPPreconditions;
use std::time::Duration;
let (_directories, store, shutdown) = setup_multi_pool_test_store("scanner-observed-cleanup", &[4, 4]).await;
let object = "buckets/.usage.observed.json";
let mut tags = Vec::new();
let old_payload = b"old observed snapshot";
for (pool, payload) in store.pools.iter().zip([Some(old_payload.as_slice()), second_payload]) {
let Some(payload) = payload else { continue };
let info = save_config_with_opts_and_metadata(pool.clone(), object, payload.to_vec(), &ObjectOptions::default())
.await
.expect("seed an observed snapshot on each pool");
tags.push(info.etag.expect("seed has CAS tag"));
}
if let Some(second_payload) = second_payload {
assert_eq!(tags[0] == tags[1], old_payload == second_payload);
}
let epoch = store.scanner_data_usage_publication_epoch().await.expect("publication epoch");
let scope = store
.scanner_data_usage_publication_commit_scope(epoch, tokio::time::Instant::now() + Duration::from_secs(30), Vec::new())
.await
.expect("admit observed cleanup");
let disconnected = if lose_second_quorum {
let pool = &store.pools[1];
tokio::time::timeout(Duration::from_secs(10), pool.stop_endpoint_monitor_for_test())
.await
.expect("fixture must own the disk slots");
let set = pool.get_disks_by_key(object);
let original = {
let mut disks = set.disks.write().await;
let offline = vec![None; disks.len()];
std::mem::replace(&mut *disks, offline)
};
Some((set, original))
} else {
None
};
let result = store
.delete_object(
crate::disk::RUSTFS_META_BUCKET,
object,
ObjectOptions {
delete_prefix: true,
delete_prefix_object: true,
http_preconditions: Some(HTTPPreconditions {
if_match: Some(if_match_override.map_or_else(|| tags.remove(0), str::to_owned)),
..Default::default()
}),
scanner_publication_commit_scope: Some(scope.clone()),
..Default::default()
},
)
.await;
if let Some((set, disks)) = disconnected {
*set.disks.write().await = disks;
assert!(!scope.release_movement_permit().await, "partial cleanup cannot release its permit");
}
let mut still_present = Vec::new();
for pool in &store.pools {
still_present.push(
read_config_no_lock_preserve_empty_with_metadata(pool.clone(), object)
.await
.is_ok(),
);
}
shutdown.cancel();
(result, scope.state(), still_present)
}
#[tokio::test]
async fn scanner_observed_cleanup_must_keep_scope_live_across_both_pools() {
let (result, state, still_present) = run_scanner_observed_cleanup(None, Some(b"old observed snapshot"), false).await;
assert!(result.is_ok(), "result {result:?}, scope {state:?}, pool copies {still_present:?}");
assert_eq!(state, crate::object_api::ScannerPublicationCommitState::Committed);
assert_eq!(still_present, vec![false, false]);
}
#[tokio::test]
async fn scanner_observed_cleanup_must_preserve_both_pools_if_cas_changed() {
let (result, state, still_present) =
run_scanner_observed_cleanup(Some("stale-cas-token"), Some(b"old observed snapshot"), false).await;
assert!(
matches!(result, Err(StorageError::PreconditionFailed))
&& state == crate::object_api::ScannerPublicationCommitState::AbortedBeforeCommit
&& still_present == [true, true],
"result {result:?}, scope {state:?}, pool copies {still_present:?}"
);
}
#[tokio::test]
async fn scanner_observed_cleanup_must_preserve_a_nonmatching_pool_copy() {
let (result, state, still_present) = run_scanner_observed_cleanup(None, Some(b"newer observed snapshot"), false).await;
assert!(result.is_ok(), "result {result:?}, scope {state:?}, pool copies {still_present:?}");
assert_eq!(state, crate::object_api::ScannerPublicationCommitState::Committed);
assert_eq!(still_present, vec![false, true]);
}
#[tokio::test]
async fn scanner_observed_cleanup_must_succeed_when_another_pool_has_no_copy() {
let (result, state, still_present) = run_scanner_observed_cleanup(None, None, false).await;
assert!(result.is_ok(), "result {result:?}, scope {state:?}, pool copies {still_present:?}");
assert_eq!(state, crate::object_api::ScannerPublicationCommitState::Committed);
assert_eq!(still_present, vec![false, false]);
}
#[tokio::test]
async fn scanner_observed_cleanup_must_retain_ownership_after_partial_failure() {
let (result, state, still_present) = run_scanner_observed_cleanup(None, Some(b"old observed snapshot"), true).await;
assert!(result.is_err(), "partial cleanup must fail");
assert_eq!(state, crate::object_api::ScannerPublicationCommitState::Indeterminate);
assert_eq!(still_present, vec![false, true]);
}
struct LifecycleDeleteAllFailureGuard;
impl Drop for LifecycleDeleteAllFailureGuard {
+64 -1
View File
@@ -70,13 +70,17 @@ pub struct ObjectPartInfo {
pub error: Option<String>,
#[serde(default)]
pub integrity: Option<crate::shard_integrity::PartIntegrity>,
/// The stored ETag was generated by the protected encrypted writer.
#[serde(default)]
pub encrypted_etag: bool,
}
impl Serialize for ObjectPartInfo {
fn serialize<S: serde::Serializer>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error> {
// Named fields let release readers ignore optional extensions. Appending
// a ninth element to the old positional array makes those readers fail.
let mut map = serializer.serialize_map(Some(8 + usize::from(self.integrity.is_some())))?;
let mut map =
serializer.serialize_map(Some(8 + usize::from(self.integrity.is_some()) + usize::from(self.encrypted_etag)))?;
map.serialize_entry("etag", &self.etag)?;
map.serialize_entry("number", &self.number)?;
map.serialize_entry("size", &self.size)?;
@@ -88,6 +92,9 @@ impl Serialize for ObjectPartInfo {
if let Some(integrity) = &self.integrity {
map.serialize_entry("integrity", integrity)?;
}
if self.encrypted_etag {
map.serialize_entry("encrypted_etag", &true)?;
}
map.end()
}
}
@@ -1143,6 +1150,7 @@ impl FileInfo {
checksums,
error: None,
integrity: None,
encrypted_etag: false,
};
for p in self.parts.iter_mut() {
@@ -2231,6 +2239,59 @@ mod tests {
)
}
#[test]
fn object_part_encrypted_etag_round_trips_with_legacy_compatibility() {
#[derive(Serialize, Deserialize)]
struct LegacyPart {
etag: String,
number: usize,
size: usize,
actual_size: i64,
mod_time: Option<OffsetDateTime>,
index: Option<Bytes>,
checksums: Option<HashMap<String, String>>,
error: Option<String>,
}
let legacy = LegacyPart {
etag: "0123456789abcdef0123456789abcdef".to_owned(),
number: 1,
size: 42,
actual_size: 42,
mod_time: None,
index: None,
checksums: None,
error: None,
};
for encoded in [
rmp_serde::to_vec(&legacy).expect("legacy positional part encodes"),
rmp_serde::to_vec_named(&legacy).expect("legacy named part encodes"),
] {
let decoded = ObjectPartInfo::unmarshal(&encoded).expect("legacy part remains readable");
assert_eq!(decoded.etag, legacy.etag);
assert!(!decoded.encrypted_etag, "an older writer cannot attest a protected ETag");
}
for encrypted_etag in [false, true] {
let current = ObjectPartInfo {
etag: legacy.etag.clone(),
number: legacy.number,
size: legacy.size,
actual_size: legacy.actual_size,
encrypted_etag,
..Default::default()
};
let encoded = current.marshal_msg().expect("current part encodes as a map");
let field_count = rmp::decode::read_map_len(&mut encoded.as_slice()).expect("part format is a map");
assert_eq!(field_count, if encrypted_etag { 9 } else { 8 });
assert_eq!(ObjectPartInfo::unmarshal(&encoded).expect("proof round trips"), current);
let old_reader: LegacyPart = rmp_serde::from_slice(&encoded).expect("old readers ignore optional proof");
assert_eq!(old_reader.etag, current.etag);
assert_eq!(old_reader.number, current.number);
assert_eq!(old_reader.size, current.size);
}
}
fn object_part_info_strategy() -> impl Strategy<Value = ObjectPartInfo> {
(
small_string_strategy(),
@@ -2252,6 +2313,7 @@ mod tests {
checksums,
error,
integrity: None,
encrypted_etag: false,
})
}
@@ -2461,6 +2523,7 @@ mod tests {
),
error: Some("part-error".to_string()),
integrity: None,
encrypted_etag: false,
}],
erasure: ErasureInfo {
algorithm: "erasure-algorithm".to_string(),
+2
View File
@@ -1346,6 +1346,7 @@ impl<'a> BorrowedMetaObject<'a> {
checksums: None,
error: None,
integrity: None,
encrypted_etag: false,
});
}
}
@@ -2625,6 +2626,7 @@ impl From<MetaObjectV1Part> for ObjectPartInfo {
checksums: value.checksums,
error: value.error,
integrity: None,
encrypted_etag: false,
}
}
}
+1 -1
View File
@@ -30,7 +30,7 @@ workspace = true
[dependencies]
serde = { workspace = true, features = ["derive"] }
tokio = { workspace = true, features = ["sync"] }
uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics", "serde"] }
[dev-dependencies]
serde_json = { workspace = true }
+1 -1
View File
@@ -58,7 +58,7 @@ pub enum Error {
TaskExecutionFailed { message: String },
/// Only the qualified MRF Deep check may produce this terminal result.
#[error("Heal task execution failed: Missing exact storage proof for durable MRF repair {target}")]
#[error("Independent verification unavailable for legacy object {target}; durable MRF responsibility retained")]
LegacyMrfUnverified { target: String },
#[error("Replacement failure could not be persisted: {failure}; persistence error: {persistence}")]
+17
View File
@@ -292,6 +292,20 @@ impl HealManager {
"Heal scheduler task completed"
);
}
Err(e @ Error::LegacyMrfUnverified { .. }) => {
warn!(
target: "rustfs::heal::manager",
event = EVENT_HEAL_SCHEDULER_STATE,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_MANAGER,
state = "verification_unavailable",
task_id,
heal_type = %task_type_label_for_spawn,
set = %task_set_label_for_spawn,
detail = %e,
"Heal attempt completed without independent verification"
);
}
Err(e) => {
let will_retry = retry_request.is_some();
if will_retry {
@@ -483,6 +497,9 @@ impl HealManager {
// update statistics
let mut stats = statistics_clone.write().await;
match completed_status {
HealTaskStatus::Completed if matches!(&result, Err(Error::LegacyMrfUnverified { .. })) => {
stats.update_unverified_task_completion();
}
HealTaskStatus::Completed => {
stats.update_task_completion(true);
stats.add_healed_objects(completed_progress.objects_healed, completed_progress.bytes_processed);
+63
View File
@@ -1106,6 +1106,22 @@ impl HealStorageAPI for MockStorage {
if bucket.starts_with("legacy-ambiguity-scheduler-") {
return Err(Error::Storage(EcstoreError::DeleteRecoveryLegacyAmbiguous));
}
if bucket == "legacy-held-pending-scheduler" {
let drives = vec![rustfs_madmin::heal_commands::HealDriveInfo {
state: "ok".to_owned(),
..Default::default()
}];
return Ok((
HealResultItem {
legacy_identity_unverified: true,
detail: rustfs_heal_contracts::heal_channel::LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL.to_owned(),
before: rustfs_madmin::heal_commands::Infos { drives: drives.clone() },
after: rustfs_madmin::heal_commands::Infos { drives },
..Default::default()
},
None,
));
}
if bucket == "completed-retention-failed" {
return Err(Error::TaskExecutionFailed {
message: "retention fixture failure".to_string(),
@@ -2153,6 +2169,53 @@ fn only_typed_mrf_result_can_pause_unverified_legacy() {
}
}
#[tokio::test(flavor = "current_thread")]
async fn mrf_legacy_completed_attempt_retains_responsibility_without_failed_statistics() {
use crate::heal::failure_logging::test_support::FailureLogCapture;
let capture = FailureLogCapture::default();
let _subscriber = tracing::subscriber::set_default(capture.clone());
let (target, _) = unverified_legacy_fixture("legacy-held-pending-scheduler");
let bucket = target.bucket.clone();
let anchor = target.durable_anchor.clone().expect("durable MRF anchor");
let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
let mut request = HealRequest::object(bucket.to_string(), target.object.to_string(), None);
request.source = HealRequestSource::Mrf;
request.expected_mrf_bucket_incarnation_id = Some(anchor.bucket_incarnation_id);
request.options.pool_index = Some(1);
request.options.set_index = Some(2);
let receipt = manager
.submit_mrf_heal_request_with_receipt_and_identity(request, target)
.await
.expect("admit durable MRF check");
assert_eq!(receipt.result, HealAdmissionResult::Accepted);
process_manager_queue_once(&manager).await;
tokio::time::timeout(Duration::from_secs(2), async {
while manager.get_statistics().await.unverified_tasks == 0 {
tokio::task::yield_now().await;
}
})
.await
.expect("publish completed unverified attempt");
let stats = manager.get_statistics().await;
assert_eq!(stats.unverified_tasks, 1);
assert_eq!(stats.failed_tasks, 0);
assert_eq!(stats.successful_tasks, 0);
assert_eq!(stats.total_objects_healed, 0);
let report = manager.get_task_report(&receipt.task_id).await.expect("retained task report");
assert_eq!(report.status, HealTaskStatus::Completed);
let outcome = report.outcome.expect("retained unknown outcome");
assert_eq!(outcome.counters.unknown, 1);
assert_eq!(outcome.counters.failed, 0);
assert!(rustfs_common::mrf_channel::take_mrf_verified_repair_events_for(&bucket).is_empty());
let held = rustfs_common::mrf_channel::take_mrf_unverified_legacy_events_for(&bucket);
assert_eq!(held.len(), 1);
assert_eq!(held[0].anchor, anchor);
assert!(manager.retrying_heals.lock().await.is_empty());
assert_eq!(capture.count(tracing::Level::ERROR, Some("rustfs::heal::manager")), 0);
assert_eq!(capture.count(tracing::Level::WARN, Some("rustfs::heal::manager")), 1);
}
#[test]
fn completed_mrf_notice_publishes_only_verified_positive_events() {
use crate::heal::outcome::{HealObjectIdentity, HealObjectKind, HealObjectOutcome, HealTaskOutcome};
+24
View File
@@ -506,6 +506,9 @@ pub struct HealStatistics {
pub successful_tasks: u64,
/// Failed tasks
pub failed_tasks: u64,
/// Completed attempts without independent verification; not the current durable MRF backlog.
#[serde(default)]
pub unverified_tasks: u64,
/// Running tasks
pub running_tasks: u64,
/// Total healed objects
@@ -528,6 +531,7 @@ impl HealStatistics {
total_tasks: 0,
successful_tasks: 0,
failed_tasks: 0,
unverified_tasks: 0,
running_tasks: 0,
total_objects_healed: 0,
total_bytes_healed: 0,
@@ -549,6 +553,11 @@ impl HealStatistics {
self.last_update_time = SystemTime::now();
}
pub(crate) fn update_unverified_task_completion(&mut self) {
self.unverified_tasks = self.unverified_tasks.saturating_add(1);
self.last_update_time = SystemTime::now();
}
pub fn add_healed_objects(&mut self, count: u64, bytes: u64) {
self.total_objects_healed += count;
self.total_bytes_healed += bytes;
@@ -915,6 +924,21 @@ mod tests {
assert_eq!(stats.failed_tasks, 0);
}
#[test]
fn unverified_statistics_preserve_old_snapshot_compatibility() {
let mut stats = HealStatistics::new();
stats.update_unverified_task_completion();
assert_eq!(stats.unverified_tasks, 1);
assert_eq!(stats.successful_tasks, 0);
assert_eq!(stats.failed_tasks, 0);
let mut wire = serde_json::to_value(&stats).expect("serialize task statistics");
let current: HealStatistics = serde_json::from_value(wire.clone()).expect("read new statistics");
assert_eq!(current.unverified_tasks, 1);
wire.as_object_mut().expect("statistics object").remove("unverified_tasks");
let old: HealStatistics = serde_json::from_value(wire).expect("read statistics without new field");
assert_eq!(old.unverified_tasks, 0);
}
#[test]
fn test_heal_statistics_update_task_completion_success() {
let mut stats = HealStatistics::new();
+23 -2
View File
@@ -291,7 +291,7 @@ pub enum HealTaskStatus {
Running,
/// Retrying after a recoverable failure
Retrying { error: String, retry_attempt: u32 },
/// Completed
/// Execution completed; the outcome and durable MRF ledger retain any unverified responsibility.
Completed,
/// Failed
Failed { error: String },
@@ -1232,6 +1232,22 @@ impl HealTask {
"Heal task completed"
});
}
Err(Error::LegacyMrfUnverified { .. }) => {
// The qualified Deep attempt finished, but its Unknown outcome
// cannot discharge the separately owned durable responsibility.
self.progress.write().await.mark_completed();
*self.status.write().await = HealTaskStatus::Completed;
debug!(
target: "rustfs::heal::task",
event = EVENT_HEAL_TASK_STATE,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_TASK,
state = "verification_unavailable",
task_id = %self.id,
heal_type = self.heal_type.kind_label(),
"Heal attempt completed without independent verification"
);
}
Err(Error::TaskCancelled) => {
let mut status = self.status.write().await;
*status = HealTaskStatus::Cancelled;
@@ -1281,11 +1297,16 @@ impl HealTask {
let terminal_state = match &result {
Ok(_) => "completed",
Err(Error::LegacyMrfUnverified { .. }) => "verification_unavailable",
Err(Error::TaskCancelled) => "cancelled",
Err(Error::TaskTimeout) => "timed_out",
Err(_) => "failed",
};
self.emit_trace_task_state(terminal_state, start_instant.elapsed(), result.as_ref().err());
let terminal_error = result
.as_ref()
.err()
.filter(|error| !matches!(error, Error::LegacyMrfUnverified { .. }));
self.emit_trace_task_state(terminal_state, start_instant.elapsed(), terminal_error);
result
}
+29 -1
View File
@@ -3285,6 +3285,29 @@ async fn durable_mrf_unverified_object_names_cannot_change_retry_classification(
}
}
#[tokio::test]
async fn mrf_legacy_verification_pending_is_a_completed_unverified_attempt() {
let (task, _) = mrf_task_for_item("legacy-pending", healthy_legacy_heal_item());
let error = task.execute().await.expect_err("unverified result must retain MRF ownership");
assert!(matches!(error, Error::LegacyMrfUnverified { .. }));
assert!(!error.is_recoverable_heal());
assert_eq!(task.get_status().await, HealTaskStatus::Completed);
let outcome = task.get_outcome().await;
assert_eq!(outcome.execution, HealExecutionOutcome::Completed);
assert_eq!(outcome.counters.unknown, 1);
assert_eq!(outcome.counters.failed, 0);
assert_eq!(outcome.counters.healed, 0);
assert_eq!(outcome.counters.unchanged, 0);
assert!(matches!(outcome.objects[0].disposition, HealObjectDisposition::Unknown));
let (status, detail) = outcome.legacy_status("finished", None);
assert_eq!(status, "finished");
assert!(
detail
.expect("visible unverified result")
.contains("authoritative storage proof is unavailable")
);
}
#[tokio::test]
async fn mrf_legacy_object_names_cannot_change_retry_classification() {
for object in [
@@ -3304,7 +3327,9 @@ async fn mrf_legacy_object_names_cannot_change_retry_classification() {
assert!(matches!(error, Error::LegacyMrfUnverified { .. }));
assert_eq!(
error.to_string(),
format!("Heal task execution failed: Missing exact storage proof for durable MRF repair bucket-a/{object}")
format!(
"Independent verification unavailable for legacy object bucket-a/{object}; durable MRF responsibility retained"
)
);
}
}
@@ -3362,11 +3387,14 @@ async fn mrf_legacy_pause_requires_storage_owned_healthy_deep_result() {
let outcome = task.get_outcome().await;
assert_eq!(outcome.counters.healed, 0);
if case == "qualified" {
assert_eq!(task.get_status().await, HealTaskStatus::Completed);
assert_eq!(outcome.counters.unknown, 1);
assert_eq!(outcome.objects[0].detail.as_deref(), Some("updated diagnostic wording"));
let opts = storage.object_heal_opts.lock().expect("storage options");
assert_eq!(opts.len(), 1);
assert_eq!(opts[0].scan_mode, HealScanMode::Deep);
} else {
assert!(matches!(task.get_status().await, HealTaskStatus::Failed { .. }), "case={case}");
}
}
}
@@ -30,6 +30,78 @@ use storage_api::integration::{
WriteCommitGuard, WriteCompletion,
};
async fn complete_fixture_replicas(env: &TestECStoreEnv, bucket: &str, object: &str) {
// TailDrained completes fan-out, but a successful quorum write may still
// have minority failures. Fault injection needs a fully repaired baseline.
let result = ECStoreHealStorage::new(env.ecstore.clone())
.heal_object_with_receipt(
bucket,
object,
None,
&HealOpts {
scan_mode: HealScanMode::Deep,
..Default::default()
},
)
.await
.expect("complete protected fixture replicas before fault injection");
assert!(result.error.is_none(), "fixture repair failed: {:?}", result.error);
assert_eq!(result.item.after.drives.len(), env.disk_paths.len());
assert!(
result.item.after.drives.iter().all(|drive| drive.state == "ok"),
"fixture requires every replica, not only read/write quorum: {:?}",
result.item.after.drives
);
assert!(result.item.integrity_verified, "fixture must retain protected payload verification");
}
#[tokio::test]
#[serial]
async fn mixed_batch_fixture_repairs_a_missing_minority_replica() {
let root = tempfile::tempdir().expect("minority replica fixture");
let env = TestECStoreEnv::builder().base_dir(root.path()).build().await;
let bucket = "mixed-fixture-baseline";
let object = "object";
env.make_bucket(bucket, false).await;
let expected = vec![0x5a; 1024 * 1024 + 123];
env.ecstore
.put_object(
bucket,
object,
&mut PutObjReader::from_vec(expected.clone()),
&ObjectOptions {
shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Protected),
write_completion: WriteCompletion::TailDrained,
..Default::default()
},
)
.await
.expect("quorum commit fixture");
// Guarantee a minority failure regardless of whether the original write
// succeeded on every disk, then repair it before any test fault is applied.
let replica = env.disk_paths[0].join(bucket).join(object);
if replica.exists() {
tokio::fs::remove_dir_all(&replica)
.await
.expect("remove one complete replica");
}
complete_fixture_replicas(&env, bucket, object).await;
let disks = env.ecstore.pools[0].get_disks(0);
for disk in disks.disks.read().await.iter().flatten() {
disk.read_version("", bucket, object, "", &ReadOptions::default())
.await
.expect("every fixture replica has metadata after completion");
}
let mut reader = env
.ecstore
.get_object_reader(bucket, object, None, Default::default(), &ObjectOptions::default())
.await
.expect("read the completed fixture");
let mut actual = Vec::new();
reader.stream.read_to_end(&mut actual).await.expect("completed fixture body");
assert_eq!(actual, expected);
}
#[test]
fn disabled_lock_reports_only_quorum_selection_and_deep_check_observations() {
const CHILD_ENV: &str = "RUSTFS_TEST_DISABLED_LOCK_HEAL_CHILD";
@@ -704,6 +776,7 @@ fn admin_mixed_batch_retains_repairs_after_object_retry_exhaustion() {
)
.await
.expect("commit complete fixture");
complete_fixture_replicas(&env, bucket, object).await;
for (slot, disk) in disks.iter().enumerate() {
let meta = disk
.read_version("", bucket, object, "", &ReadOptions::default())
+100 -10
View File
@@ -429,6 +429,7 @@ const INTERNODE_OPERATION_REQUESTS_OUTGOING_TOTAL: &str = "rustfs_system_network
const INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL: &str = "rustfs_system_network_internode_operation_requests_incoming_total";
const INTERNODE_OPERATION_ERRORS_TOTAL: &str = "rustfs_system_network_internode_operation_errors_total";
const INTERNODE_OPERATION_DURATION_MS: &str = "rustfs_system_network_internode_operation_duration_ms";
const INTERNODE_OPERATION_SERVER_DURATION_MS: &str = "rustfs_system_network_internode_operation_server_duration_ms";
const INTERNODE_OPERATION_STAGE_DURATION_MS: &str = "rustfs_system_network_internode_operation_stage_duration_ms";
const INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL: &str = "rustfs_system_network_internode_operation_classified_errors_total";
const INTERNODE_OPERATION_RETRIES_TOTAL: &str = "rustfs_system_network_internode_operation_retries_total";
@@ -562,6 +563,10 @@ pub const INTERNODE_OPERATION_METRICS: &[InternodeOperationMetricDescriptor] = &
name: INTERNODE_OPERATION_LARGE_PAYLOADS_TOTAL,
labels: SERVER_OPERATION_BACKEND_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_SERVER_DURATION_MS,
labels: SERVER_OPERATION_BACKEND_LABELS,
},
];
static STABLE_SERVER_LABEL: OnceLock<String> = OnceLock::new();
@@ -687,6 +692,78 @@ impl GrpcReadVersionMetricHandles {
static GRPC_READ_VERSION_METRIC_HANDLES: LazyLock<GrpcReadVersionMetricHandles> =
LazyLock::new(|| GrpcReadVersionMetricHandles::new(current_server_label()));
#[cfg(not(test))]
static GRPC_READ_MULTIPLE_DURATION: LazyLock<metrics::Histogram> = LazyLock::new(|| {
operation_duration_histogram(
INTERNODE_OPERATION_DURATION_MS,
INTERNODE_OPERATION_GRPC_READ_MULTIPLE,
INTERNODE_TRANSPORT_BACKEND_GRPC,
)
});
#[cfg(not(test))]
static GRPC_BATCH_READ_VERSION_DURATION: LazyLock<metrics::Histogram> = LazyLock::new(|| {
operation_duration_histogram(
INTERNODE_OPERATION_DURATION_MS,
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
)
});
#[cfg(not(test))]
static GRPC_READ_MULTIPLE_SERVER_DURATION: LazyLock<metrics::Histogram> = LazyLock::new(|| {
operation_duration_histogram(
INTERNODE_OPERATION_SERVER_DURATION_MS,
INTERNODE_OPERATION_GRPC_READ_MULTIPLE,
INTERNODE_TRANSPORT_BACKEND_GRPC,
)
});
#[cfg(not(test))]
static GRPC_BATCH_READ_VERSION_SERVER_DURATION: LazyLock<metrics::Histogram> = LazyLock::new(|| {
operation_duration_histogram(
INTERNODE_OPERATION_SERVER_DURATION_MS,
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
)
});
#[cfg(not(test))]
fn grpc_operation_duration_if_ready(operation: &'static str, backend: &'static str) -> Option<&'static metrics::Histogram> {
STABLE_SERVER_LABEL.get()?;
if backend != INTERNODE_TRANSPORT_BACKEND_GRPC {
return None;
}
match operation {
INTERNODE_OPERATION_GRPC_READ_VERSION => Some(&GRPC_READ_VERSION_METRIC_HANDLES.duration),
INTERNODE_OPERATION_GRPC_READ_MULTIPLE => Some(&GRPC_READ_MULTIPLE_DURATION),
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION => Some(&GRPC_BATCH_READ_VERSION_DURATION),
_ => None,
}
}
#[cfg(not(test))]
fn grpc_server_duration_if_ready(operation: &'static str, backend: &'static str) -> Option<&'static metrics::Histogram> {
STABLE_SERVER_LABEL.get()?;
if backend != INTERNODE_TRANSPORT_BACKEND_GRPC {
return None;
}
match operation {
INTERNODE_OPERATION_GRPC_READ_MULTIPLE => Some(&GRPC_READ_MULTIPLE_SERVER_DURATION),
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION => Some(&GRPC_BATCH_READ_VERSION_SERVER_DURATION),
_ => None,
}
}
fn operation_duration_histogram(name: &'static str, operation: &'static str, backend: &'static str) -> metrics::Histogram {
metrics::histogram!(
name,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
}
#[cfg(not(test))]
fn server_metric_handles_if_ready() -> Option<&'static InternodeServerMetricHandles> {
STABLE_SERVER_LABEL.get()?;
@@ -944,17 +1021,28 @@ impl InternodeMetrics {
pub fn record_duration_for_operation_and_backend(&self, operation: &'static str, backend: &'static str, duration: Duration) {
let duration_ms = duration.as_secs_f64() * 1000.0;
#[cfg(not(test))]
if let Some(handles) = grpc_read_version_metric_handles_if_ready(operation, backend) {
handles.duration.record(duration_ms);
if let Some(histogram) = grpc_operation_duration_if_ready(operation, backend) {
histogram.record(duration_ms);
return;
}
metrics::histogram!(
INTERNODE_OPERATION_DURATION_MS,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
.record(duration_ms);
operation_duration_histogram(INTERNODE_OPERATION_DURATION_MS, operation, backend).record(duration_ms);
}
/// Records server-side handler latency in a separate histogram from client RPC duration.
/// The caller supplies the handler lifetime, including any error or cancellation path.
pub fn record_server_duration_for_operation_and_backend(
&self,
operation: &'static str,
backend: &'static str,
duration: Duration,
) {
let duration_ms = duration.as_secs_f64() * 1000.0;
#[cfg(not(test))]
if let Some(histogram) = grpc_server_duration_if_ready(operation, backend) {
histogram.record(duration_ms);
return;
}
operation_duration_histogram(INTERNODE_OPERATION_SERVER_DURATION_MS, operation, backend).record(duration_ms);
}
pub fn record_stage_duration_for_operation_and_backend(
@@ -1699,7 +1787,7 @@ mod tests {
#[test]
fn operation_metric_descriptors_include_backend_and_operation_labels() {
assert_eq!(INTERNODE_OPERATION_METRICS.len(), 22);
assert_eq!(INTERNODE_OPERATION_METRICS.len(), 23);
for metric in &INTERNODE_OPERATION_METRICS[..6] {
assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
}
@@ -1737,6 +1825,8 @@ mod tests {
// Payload histogram + large-payload counter carry operation+backend labels.
assert_eq!(INTERNODE_OPERATION_METRICS[20].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[21].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[22].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[22].name, INTERNODE_OPERATION_SERVER_DURATION_MS);
}
#[test]
@@ -0,0 +1,206 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use metrics::{Counter, Gauge, Histogram, Key, KeyName, Metadata, Recorder, SharedString, Unit, with_local_recorder};
use metrics_util::debugging::{DebugValue, DebuggingRecorder};
use rustfs_io_metrics::internode_metrics::{
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_MULTIPLE,
INTERNODE_OPERATION_GRPC_READ_VERSION, INTERNODE_TRANSPORT_BACKEND_GRPC, INTERNODE_TRANSPORT_BACKEND_UNKNOWN,
InternodeMetrics, set_internode_server_label,
};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
const DURATION_METRIC: &str = "rustfs_system_network_internode_operation_duration_ms";
const SERVER_DURATION_METRIC: &str = "rustfs_system_network_internode_operation_server_duration_ms";
const SERVER_READ_OPERATIONS: [&str; 2] = [
INTERNODE_OPERATION_GRPC_READ_MULTIPLE,
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION,
];
const READ_OPERATIONS: [&str; 3] = [
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_OPERATION_GRPC_READ_MULTIPLE,
INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION,
];
#[derive(Default)]
struct RegistrationCountingRecorder {
inner: DebuggingRecorder,
duration_registrations: AtomicUsize,
}
impl Recorder for RegistrationCountingRecorder {
fn describe_counter(&self, key: KeyName, unit: Option<Unit>, description: SharedString) {
self.inner.describe_counter(key, unit, description);
}
fn describe_gauge(&self, key: KeyName, unit: Option<Unit>, description: SharedString) {
self.inner.describe_gauge(key, unit, description);
}
fn describe_histogram(&self, key: KeyName, unit: Option<Unit>, description: SharedString) {
self.inner.describe_histogram(key, unit, description);
}
fn register_counter(&self, key: &Key, metadata: &Metadata<'_>) -> Counter {
self.inner.register_counter(key, metadata)
}
fn register_gauge(&self, key: &Key, metadata: &Metadata<'_>) -> Gauge {
self.inner.register_gauge(key, metadata)
}
fn register_histogram(&self, key: &Key, metadata: &Metadata<'_>) -> Histogram {
if key.name() == DURATION_METRIC || key.name() == SERVER_DURATION_METRIC {
self.duration_registrations.fetch_add(1, Ordering::Relaxed);
}
self.inner.register_histogram(key, metadata)
}
}
// Integration tests exercise the production cache, which is disabled in unit tests
// so individual unit tests can install independent local recorders.
#[test]
fn unary_read_duration_handles_are_cached_only_after_server_label_is_ready() {
let recorder = RegistrationCountingRecorder::default();
let snapshotter = recorder.inner.snapshotter();
let metrics = InternodeMetrics::default();
with_local_recorder(&recorder, || {
for operation in READ_OPERATIONS {
metrics.record_duration_for_operation_and_backend(operation, INTERNODE_TRANSPORT_BACKEND_GRPC, Duration::ZERO);
}
for operation in SERVER_READ_OPERATIONS {
metrics.record_server_duration_for_operation_and_backend(operation, INTERNODE_TRANSPORT_BACKEND_GRPC, Duration::ZERO);
}
set_internode_server_label("cached-duration-test");
for operation in READ_OPERATIONS {
metrics.record_duration_for_operation_and_backend(
operation,
INTERNODE_TRANSPORT_BACKEND_GRPC,
Duration::from_micros(125),
);
let registrations = recorder.duration_registrations.load(Ordering::Relaxed);
metrics.record_duration_for_operation_and_backend(
operation,
INTERNODE_TRANSPORT_BACKEND_GRPC,
Duration::from_micros(250),
);
assert_eq!(
recorder.duration_registrations.load(Ordering::Relaxed),
registrations,
"{operation} should reuse its duration handle after warm-up"
);
}
for operation in SERVER_READ_OPERATIONS {
metrics.record_server_duration_for_operation_and_backend(
operation,
INTERNODE_TRANSPORT_BACKEND_GRPC,
Duration::from_micros(1500),
);
let registrations = recorder.duration_registrations.load(Ordering::Relaxed);
metrics.record_server_duration_for_operation_and_backend(
operation,
INTERNODE_TRANSPORT_BACKEND_GRPC,
Duration::from_micros(2500),
);
assert_eq!(
recorder.duration_registrations.load(Ordering::Relaxed),
registrations,
"{operation} should reuse its separate server duration handle after warm-up"
);
}
// Other backends and operations must retain their own labels and fallback.
for (operation, backend) in READ_OPERATIONS
.map(|operation| (operation, INTERNODE_TRANSPORT_BACKEND_UNKNOWN))
.into_iter()
.chain([(INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC)])
{
let registrations = recorder.duration_registrations.load(Ordering::Relaxed);
metrics.record_duration_for_operation_and_backend(operation, backend, Duration::from_millis(1));
assert_eq!(recorder.duration_registrations.load(Ordering::Relaxed), registrations + 1);
}
for (operation, backend) in SERVER_READ_OPERATIONS
.map(|operation| (operation, INTERNODE_TRANSPORT_BACKEND_UNKNOWN))
.into_iter()
.chain([(INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC)])
{
let registrations = recorder.duration_registrations.load(Ordering::Relaxed);
metrics.record_server_duration_for_operation_and_backend(operation, backend, Duration::from_millis(3));
assert_eq!(recorder.duration_registrations.load(Ordering::Relaxed), registrations + 1);
}
});
let rows = snapshotter.snapshot().into_vec();
let durations: Vec<_> = rows
.iter()
.filter(|(key, _, _, _)| key.key().name() == DURATION_METRIC || key.key().name() == SERVER_DURATION_METRIC)
.collect();
assert_eq!(
durations
.iter()
.filter(|(key, _, _, _)| key.key().name() == DURATION_METRIC)
.count(),
10
);
assert_eq!(
durations
.iter()
.filter(|(key, _, _, _)| key.key().name() == SERVER_DURATION_METRIC)
.count(),
7
);
for (key, _, _, value) in durations {
let server_duration = key.key().name() == SERVER_DURATION_METRIC;
let operations = if server_duration {
SERVER_READ_OPERATIONS.as_slice()
} else {
READ_OPERATIONS.as_slice()
};
let labels: Vec<_> = key.key().labels().map(|label| (label.key(), label.value())).collect();
assert_eq!(labels.len(), 3);
let server = labels.iter().find(|(name, _)| *name == "server").expect("server label").1;
let operation = labels
.iter()
.find(|(name, _)| *name == "operation")
.expect("operation label")
.1;
let backend = labels.iter().find(|(name, _)| *name == "backend").expect("backend label").1;
let expected: &[f64] = if server == "unset" {
assert!(operations.contains(&operation));
assert_eq!(backend, INTERNODE_TRANSPORT_BACKEND_GRPC);
&[0.0]
} else {
assert_eq!(server, "cached-duration-test");
if backend == INTERNODE_TRANSPORT_BACKEND_GRPC && operations.contains(&operation) {
if server_duration { &[1.5, 2.5] } else { &[0.125, 0.25] }
} else {
assert!(
(backend == INTERNODE_TRANSPORT_BACKEND_UNKNOWN && operations.contains(&operation))
|| (backend == INTERNODE_TRANSPORT_BACKEND_GRPC && operation == INTERNODE_OPERATION_GRPC_READ_ALL)
);
if server_duration { &[3.0] } else { &[1.0] }
}
};
match value {
DebugValue::Histogram(samples) => {
assert_eq!(samples.iter().map(|sample| sample.0).collect::<Vec<_>>(), expected);
}
other => panic!("duration metric must be a histogram, got {other:?}"),
}
}
}
+447 -15
View File
@@ -12,12 +12,14 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use parking_lot::{RwLock, RwLockReadGuard, RwLockWriteGuard};
use std::collections::HashMap;
use std::ops::{Deref, DerefMut};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, SystemTime};
use tokio::sync::RwLock;
use tokio::time::{Instant, timeout};
use tokio::sync::Notify;
use tokio::time::{Instant, timeout, timeout_at};
use crate::{
FastLockGuard, GlobalLockManager, LockClient, LockId, LockInfo, LockManager, LockMetadata, LockPriority, LockRequest,
@@ -26,13 +28,98 @@ use crate::{
/// Default shard count for guard storage (must be power of 2)
const DEFAULT_GUARD_SHARD_COUNT: usize = 64;
const EVENT_NAMESPACE_LOCK_ACQUIRE: &str = "namespace_lock_acquire";
const EVENT_NAMESPACE_LOCK_RELEASE: &str = "namespace_lock_release";
const LOG_COMPONENT_LOCK: &str = "lock";
const LOG_SUBSYSTEM_NAMESPACE: &str = "namespace";
type GuardShard = Arc<RwLock<HashMap<LockId, LocalGuardEntry>>>;
type GuardMap = HashMap<LockId, LocalGuardEntry>;
type GuardShard = Arc<GuardRegistry>;
type GuardStorage = Arc<Vec<GuardShard>>;
/// Registry access never reserves the table for a task that has not resumed.
/// Only short synchronous map operations hold the native lock; contention
/// waits on a notification without blocking an executor thread.
#[derive(Debug, Default)]
struct GuardRegistry {
entries: RwLock<GuardMap>,
changed: Notify,
}
struct GuardRegistryWake<'a>(&'a Notify);
impl Drop for GuardRegistryWake<'_> {
fn drop(&mut self) {
self.0.notify_waiters();
}
}
struct GuardRegistryAccess<'a, G> {
// Field drop order unlocks the table before waking its waiters, including
// when a map operation unwinds. Physical guards are removed before Drop.
guard: G,
_wake: GuardRegistryWake<'a>,
}
impl<G: Deref> Deref for GuardRegistryAccess<'_, G> {
type Target = G::Target;
fn deref(&self) -> &Self::Target {
&self.guard
}
}
impl<G: DerefMut> DerefMut for GuardRegistryAccess<'_, G> {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.guard
}
}
impl GuardRegistry {
fn access<G>(&self, guard: G) -> GuardRegistryAccess<'_, G> {
GuardRegistryAccess {
guard,
_wake: GuardRegistryWake(&self.changed),
}
}
async fn read(&self) -> GuardRegistryAccess<'_, RwLockReadGuard<'_, GuardMap>> {
tokio::task::consume_budget().await;
if let Some(guard) = self.entries.try_read() {
return self.access(guard);
}
loop {
tokio::task::consume_budget().await;
let notified = self.changed.notified();
tokio::pin!(notified);
notified.as_mut().enable();
if let Some(guard) = self.entries.try_read() {
return self.access(guard);
}
notified.await;
}
}
async fn write(&self) -> GuardRegistryAccess<'_, RwLockWriteGuard<'_, GuardMap>> {
tokio::task::consume_budget().await;
if let Some(guard) = self.entries.try_write() {
return self.access(guard);
}
loop {
tokio::task::consume_budget().await;
// Register before rechecking so an unlock between the first
// failed try and this wait cannot leave the task asleep.
let notified = self.changed.notified();
tokio::pin!(notified);
notified.as_mut().enable();
if let Some(guard) = self.entries.try_write() {
return self.access(guard);
}
notified.await;
}
}
}
/// Local lock client using FastLock with sharded guard storage for better concurrency
#[derive(Debug)]
pub struct LocalClient {
@@ -94,7 +181,7 @@ impl LocalClient {
pub fn with_shard_count(shard_count: usize) -> Self {
assert!(shard_count.is_power_of_two(), "Shard count must be power of 2");
let guard_storage: Vec<GuardShard> = (0..shard_count).map(|_| Arc::new(RwLock::new(HashMap::new()))).collect();
let guard_storage: Vec<GuardShard> = (0..shard_count).map(|_| Arc::new(GuardRegistry::default())).collect();
Self::with_storage(Arc::new(guard_storage), None, crate::fast_lock::CLEANUP_INTERVAL)
}
@@ -115,7 +202,7 @@ impl LocalClient {
/// This allows simulating multi-node environments where each node has its own lock backend
pub fn with_manager(manager: Arc<GlobalLockManager>) -> Self {
let guard_storage = (0..DEFAULT_GUARD_SHARD_COUNT)
.map(|_| Arc::new(RwLock::new(HashMap::new())))
.map(|_| Arc::new(GuardRegistry::default()))
.collect();
Self::with_storage(Arc::new(guard_storage), Some(manager), crate::fast_lock::CLEANUP_INTERVAL)
}
@@ -123,7 +210,7 @@ impl LocalClient {
#[cfg(test)]
pub(crate) fn with_manager_and_reaper_interval(manager: Arc<GlobalLockManager>, reaper_interval: Duration) -> Self {
let guard_storage = (0..DEFAULT_GUARD_SHARD_COUNT)
.map(|_| Arc::new(RwLock::new(HashMap::new())))
.map(|_| Arc::new(GuardRegistry::default()))
.collect();
Self::with_storage(Arc::new(guard_storage), Some(manager), reaper_interval)
}
@@ -140,7 +227,7 @@ impl LocalClient {
}
/// Get the shard for a given lock ID
fn get_shard(&self, lock_id: &LockId) -> &Arc<RwLock<HashMap<LockId, LocalGuardEntry>>> {
fn get_shard(&self, lock_id: &LockId) -> &GuardShard {
let index = self.get_resource_shard_index(&lock_id.resource);
&self.guard_storage[index]
}
@@ -232,6 +319,7 @@ impl Default for LocalClient {
#[async_trait::async_trait]
impl LockClient for LocalClient {
async fn acquire_lock(&self, request: &LockRequest) -> Result<LockResponse> {
let started = Instant::now();
self.ensure_reaper();
if request.lock_id.resource != request.resource {
return Ok(LockResponse::failure(
@@ -251,6 +339,7 @@ impl LockClient for LocalClient {
.await
.unwrap_or(0)
};
let reclaim_elapsed = started.elapsed();
let build_lock_request = |acquire_timeout| match request.lock_type {
LockType::Exclusive => crate::ObjectLockRequest::new_write(request.resource.clone(), request.owner.clone())
@@ -265,17 +354,65 @@ impl LockClient for LocalClient {
if remaining.is_zero() {
return Ok(LockResponse::failure("Lock acquisition timeout", request.acquire_timeout));
}
let manager_started = Instant::now();
match lock_manager.acquire_lock(build_lock_request(remaining)).await {
Ok(guard) => {
let manager_elapsed = manager_started.elapsed();
let registration_started = Instant::now();
let physical_guard_id = guard.guard_id();
let lock_id = request.lock_id.clone();
let entry = LocalGuardEntry::new(guard, request.ttl);
let acquired_at = entry.acquired_at;
let expires_at = entry.expires_at;
{
let (replaced_entry, registration_wait, registration_mutation, registry_entries, registry_capacity) = {
let shard = self.get_shard(&lock_id);
let mut guards = shard.write().await;
guards.insert(lock_id.clone(), entry);
let Ok(mut guards) = timeout_at(acquire_deadline, shard.write()).await else {
return Ok(LockResponse::failure("Lock acquisition timeout", request.acquire_timeout));
};
// A ready table cannot publish a grant after the peer
// budget expired while its acquisition task was idle.
if Instant::now() >= acquire_deadline {
drop(guards);
return Ok(LockResponse::failure("Lock acquisition timeout", request.acquire_timeout));
}
let registration_wait = registration_started.elapsed();
let mutation_started = Instant::now();
let replaced_entry = guards.insert(lock_id.clone(), entry);
(
replaced_entry,
registration_wait,
mutation_started.elapsed(),
guards.len(),
guards.capacity(),
)
};
// Repeated shared requests can replace an entry. Releasing
// its physical guard must not reenter locks under the table.
drop(replaced_entry);
let registration_elapsed = registration_started.elapsed();
if started.elapsed() >= Duration::from_millis(100) {
tracing::trace!(
target: "rustfs_lock::namespace_owner",
event = EVENT_NAMESPACE_LOCK_ACQUIRE,
component = LOG_COMPONENT_LOCK,
subsystem = LOG_SUBSYSTEM_NAMESPACE,
state = "local_completed",
resource = %request.resource,
lock_id = %lock_id,
mode = ?request.lock_type,
physical_guard_id,
reclaim_ms = reclaim_elapsed.as_millis(),
manager_ms = manager_elapsed.as_millis(),
registration_ms = registration_elapsed.as_millis(),
registration_wait_ms = registration_wait.as_millis(),
registration_mutation_ms = registration_mutation.as_millis(),
registry_entries,
registry_capacity,
acquire_ms = started.elapsed().as_millis(),
"Namespace local lock acquisition completed slowly"
);
}
let lock_info = LockInfo {
@@ -469,6 +606,8 @@ impl LockClient for LocalClient {
mod tests {
use super::*;
use crate::{GlobalLockManager, LockClient, LockRequest, LockType};
use futures::FutureExt;
use std::task::Poll;
fn request(resource: crate::ObjectKey, owner: &str, ttl: Duration) -> LockRequest {
LockRequest::new(resource, LockType::Exclusive, owner)
@@ -476,6 +615,295 @@ mod tests {
.with_acquire_timeout(Duration::from_millis(80))
}
fn hold_guard_shard(shard: GuardShard) -> (std::sync::mpsc::Sender<()>, std::thread::JoinHandle<()>) {
let (entered_tx, entered_rx) = std::sync::mpsc::sync_channel(0);
let (release_tx, release_rx) = std::sync::mpsc::channel();
let thread = std::thread::spawn(move || {
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("build the independent guard-table owner runtime");
let guard = runtime.block_on(shard.write());
entered_tx.send(()).expect("signal that the guard table is occupied");
// A dropped controller also releases the table on test failure.
let _ = release_rx.recv_timeout(Duration::from_secs(3));
drop(guard);
});
entered_rx
.recv_timeout(Duration::from_secs(3))
.expect("guard-table owner must enter before the tested operation");
(release_tx, thread)
}
#[tokio::test(start_paused = true)]
async fn unpolled_registry_writer_does_not_block_compatible_acquisition() {
let manager = Arc::new(GlobalLockManager::new());
let client = LocalClient::with_manager(manager.clone());
client.reaper_started.store(true, Ordering::Release);
let first = LockRequest::new(crate::ObjectKey::new("bucket", ""), LockType::Shared, "first");
assert!(client.acquire_lock(&first).await.expect("first shared lock").success);
let (release_table, table_owner) = hold_guard_shard(client.get_shard(&first.lock_id).clone());
let mut pending_release = Box::pin(client.release(&first.lock_id));
assert!(matches!(futures::poll!(&mut pending_release), Poll::Pending));
release_table.send(()).expect("release the occupied table");
table_owner.join().expect("guard-table owner must finish");
// The earlier writer is deliberately not polled: an unoccupied
// bookkeeping table must remain usable by a currently running peer.
let second = LockRequest::new(first.resource.clone(), LockType::Shared, "second");
let response = client
.acquire_lock(&second)
.now_or_never()
.expect("a sleeping registry writer must not reserve the available table")
.expect("compatible shared acquisition should return a response");
assert!(response.success);
assert_eq!(
manager
.as_fast_lock_manager()
.expect("enabled physical lock backend")
.get_shard(&first.resource)
.active_guard_count(),
2
);
assert!(
pending_release
.await
.expect("the earlier writer still releases its exact grant")
);
assert!(client.release(&second.lock_id).await.expect("release the second grant"));
assert_eq!(
manager
.as_fast_lock_manager()
.expect("enabled physical lock backend")
.get_shard(&first.resource)
.active_guard_count(),
0
);
}
#[tokio::test(start_paused = true)]
async fn acquisition_deadline_rejects_registration_after_physical_grant() {
for unlock_before_poll in [false, true] {
let manager = Arc::new(GlobalLockManager::new());
let client = LocalClient::with_manager(manager.clone());
client.reaper_started.store(true, Ordering::Release);
let resource = crate::ObjectKey::new("bucket", "registration-deadline");
let blocker = manager
.acquire_lock(crate::ObjectLockRequest::new_write(resource.clone(), "blocker"))
.await
.expect("hold the physical lock until guard-table preflight finishes");
let request =
request(resource.clone(), "owner", Duration::from_secs(30)).with_acquire_timeout(Duration::from_millis(20));
let mut acquire = Box::pin(tokio::task::unconstrained(client.acquire_lock(&request)));
assert!(matches!(futures::poll!(&mut acquire), Poll::Pending));
let (release_table, table_owner) = hold_guard_shard(client.get_shard(&request.lock_id).clone());
drop(blocker);
assert!(matches!(futures::poll!(&mut acquire), Poll::Pending));
assert_eq!(
manager
.as_fast_lock_manager()
.expect("enabled physical lock backend")
.get_shard(&resource)
.active_guard_count(),
1,
"the tested future owns a physical grant"
);
tokio::time::advance(Duration::from_millis(21)).await;
if unlock_before_poll {
release_table.send(()).expect("make registration ready after its deadline");
table_owner.join().expect("guard-table owner must finish");
assert!(!acquire.await.expect("expired registration response").success);
} else {
let response = timeout(Duration::from_millis(100), &mut acquire)
.await
.expect("registration must honor its deadline while the table remains occupied")
.expect("expired registration response");
assert!(!response.success);
release_table.send(()).expect("release the occupied table");
table_owner.join().expect("guard-table owner must finish");
}
assert!(
client
.check_status(&request.lock_id)
.await
.expect("check the expired identity")
.is_none()
);
assert_eq!(
manager
.as_fast_lock_manager()
.expect("enabled physical lock backend")
.get_shard(&resource)
.active_guard_count(),
0,
"expired registration must release its grant"
);
assert!(
manager
.acquire_lock(crate::ObjectLockRequest::new_write(resource, "next-owner"))
.await
.is_ok(),
"the next owner must not inherit an abandoned physical grant"
);
}
}
#[tokio::test(start_paused = true)]
async fn cancelling_registration_releases_the_unpublished_physical_grant() {
let manager = Arc::new(GlobalLockManager::new());
let client = LocalClient::with_manager(manager.clone());
client.reaper_started.store(true, Ordering::Release);
let resource = crate::ObjectKey::new("bucket", "cancel-registration");
let blocker = manager
.acquire_lock(crate::ObjectLockRequest::new_write(resource.clone(), "blocker"))
.await
.expect("hold the physical lock through registry preflight");
let request = request(resource.clone(), "cancelled", Duration::from_secs(30));
let mut acquire = Box::pin(tokio::task::unconstrained(client.acquire_lock(&request)));
assert!(matches!(futures::poll!(&mut acquire), Poll::Pending));
let (release_table, table_owner) = hold_guard_shard(client.get_shard(&request.lock_id).clone());
drop(blocker);
assert!(matches!(futures::poll!(&mut acquire), Poll::Pending));
let backend = manager.as_fast_lock_manager().expect("enabled physical lock backend");
assert_eq!(backend.get_shard(&resource).active_guard_count(), 1);
drop(acquire);
assert_eq!(backend.get_shard(&resource).active_guard_count(), 0);
assert!(
manager
.acquire_lock(crate::ObjectLockRequest::new_write(resource, "next-owner"))
.await
.is_ok(),
"cancellation must release the physical lock without waiting for the table"
);
release_table.send(()).expect("release the occupied table");
table_owner.join().expect("guard-table owner must finish");
assert!(
client
.check_status(&request.lock_id)
.await
.expect("check cancelled identity")
.is_none()
);
assert!(client.list_lock_leases().await.is_empty());
}
#[tokio::test]
async fn cancelling_a_registry_waiter_keeps_other_waiters_live() {
let shard = Arc::new(GuardRegistry::default());
let (release_table, table_owner) = hold_guard_shard(shard.clone());
let mut cancelled = Box::pin(shard.write());
let mut survivor = Box::pin(shard.read());
assert!(matches!(futures::poll!(&mut cancelled), Poll::Pending));
assert!(matches!(futures::poll!(&mut survivor), Poll::Pending));
drop(cancelled);
release_table.send(()).expect("wake registry waiters");
table_owner.join().expect("guard-table owner must finish");
let access = timeout(Duration::from_millis(100), survivor)
.await
.expect("a cancelled waiter must not consume another waiter's unlock notification");
assert!(access.is_empty());
}
#[tokio::test]
async fn uncontended_registry_access_retains_cooperative_scheduling() {
let shard = GuardRegistry::default();
let mut yielded = false;
for _ in 0..1024 {
match shard.read().now_or_never() {
Some(access) => drop(access),
None => {
yielded = true;
break;
}
}
}
assert!(yielded, "a loop of ready registry operations must still yield to other executor tasks");
}
#[tokio::test]
async fn repeated_shared_identity_replaces_only_its_previous_physical_guard() {
let manager = Arc::new(GlobalLockManager::new());
let client = LocalClient::with_manager(manager.clone());
let request = LockRequest::new(crate::ObjectKey::new("bucket", ""), LockType::Shared, "owner");
assert!(client.acquire_lock(&request).await.expect("first shared grant").success);
assert!(client.acquire_lock(&request).await.expect("repeated shared identity").success);
let backend = manager.as_fast_lock_manager().expect("enabled physical lock backend");
assert_eq!(backend.get_shard(&request.resource).active_guard_count(), 1);
assert_eq!(client.list_lock_leases().await.len(), 1);
assert!(client.release(&request.lock_id).await.expect("release the replacement grant"));
assert_eq!(backend.get_shard(&request.resource).active_guard_count(), 0);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn hot_shared_registry_keeps_refresh_reaper_and_snapshots_live() {
for width in [64, 128] {
let manager = Arc::new(GlobalLockManager::new());
let client = Arc::new(LocalClient::with_manager_and_reaper_interval(manager.clone(), Duration::from_millis(1)));
let resource = crate::ObjectKey::new("bucket", "");
let expired = LockRequest::new(resource.clone(), LockType::Shared, "expired").with_ttl(Duration::ZERO);
assert!(client.acquire_lock(&expired).await.expect("expired seed grant").success);
let acquired = Arc::new(tokio::sync::Barrier::new(width + 1));
let observed = Arc::new(Notify::new());
let mut tasks = tokio::task::JoinSet::new();
for index in 0..width {
let client = client.clone();
let resource = resource.clone();
let acquired = acquired.clone();
let observed = observed.clone();
tasks.spawn(async move {
for iteration in 0..8 {
let request = LockRequest::new(resource.clone(), LockType::Shared, format!("load-{index}"));
assert!(client.acquire_lock(&request).await.expect("concurrent shared grant").success);
if iteration == 0 {
let notification = observed.notified();
tokio::pin!(notification);
notification.as_mut().enable();
acquired.wait().await;
notification.await;
} else {
tokio::task::yield_now().await;
}
assert!(client.refresh(&request.lock_id).await.expect("refresh concurrent grant"));
assert!(
client
.check_status(&request.lock_id)
.await
.expect("read concurrent grant")
.is_some()
);
assert!(client.release(&request.lock_id).await.expect("release concurrent grant"));
}
});
}
timeout(Duration::from_secs(5), acquired.wait())
.await
.expect("all shared grants must reach the snapshot gate");
let snapshot = client.list_lock_leases().await;
assert_eq!(snapshot.iter().filter(|lease| lease.owner.starts_with("load-")).count(), width);
observed.notify_waiters();
let mut completed = 0;
timeout(Duration::from_secs(5), async {
while let Some(result) = tasks.join_next().await {
result.expect("all shared grant workers must finish");
completed += 1;
let _ = client.list_lock_leases().await;
}
})
.await
.expect("refresh, cleanup, and concurrent snapshots must make progress");
assert_eq!(completed, width);
wait_until_reaped(&client, &expired.lock_id).await;
assert!(client.list_lock_leases().await.is_empty());
assert!(
manager
.as_fast_lock_manager()
.expect("enabled physical lock backend")
.list_locks()
.is_empty()
);
}
}
async fn wait_until_reaped(client: &LocalClient, lock_id: &LockId) {
for _ in 0..80 {
if client.check_status(lock_id).await.unwrap().is_none() {
@@ -502,8 +930,9 @@ mod tests {
}
let lock_request = request(resource, "owner", Duration::from_secs(30));
let response = tokio::time::timeout(Duration::from_millis(100), client.acquire_lock(&lock_request))
.await
let response = client
.acquire_lock(&lock_request)
.now_or_never()
.expect("an acquire must not sweep unrelated guard shards")
.expect("local acquire should return a response");
assert!(response.success);
@@ -530,8 +959,9 @@ mod tests {
}
let replacement = request(resource, "replacement-owner", Duration::from_secs(30));
let response = tokio::time::timeout(Duration::from_millis(100), client.acquire_lock(&replacement))
.await
let response = client
.acquire_lock(&replacement)
.now_or_never()
.expect("expired guard reclaim must only write the resource shard")
.expect("local acquire should return a response");
assert!(response.success);
@@ -547,7 +977,7 @@ mod tests {
client.reaper_started.store(true, Ordering::Release);
let resource = crate::ObjectKey::new("bucket", "resource-shard-deadline");
let target_shard = client.get_resource_shard_index(&resource);
let _target_write = client.guard_storage[target_shard].write().await;
let (release_table, table_owner) = hold_guard_shard(client.guard_storage[target_shard].clone());
let lock_request = request(resource, "owner", Duration::from_secs(30)).with_acquire_timeout(Duration::from_millis(10));
let response = tokio::time::timeout(Duration::from_millis(100), client.acquire_lock(&lock_request))
@@ -555,6 +985,8 @@ mod tests {
.expect("acquire should return after its reclaim budget expires")
.expect("local acquire should return a response");
assert!(!response.success);
release_table.send(()).expect("release the occupied resource table");
table_owner.join().expect("guard-table owner must finish");
}
#[tokio::test]
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -159,7 +159,7 @@ pub(crate) fn notify_configuration_hint() -> String {
format!(
"No notify targets configured. Check {}=true and instance-scoped target env vars (for example {webhook_enable_primary} + {webhook_endpoint_primary} for arn:rustfs:sqs::primary:webhook). If using default queue_dir, ensure {} is writable.",
rustfs_config::ENV_NOTIFY_ENABLE,
rustfs_config::EVENT_DEFAULT_DIR,
rustfs_targets::store::resolve_queue_dir(rustfs_config::EVENT_DEFAULT_DIR),
)
}
+45 -4
View File
@@ -66,11 +66,26 @@ pub fn select_effective_memory(host_total: u64, host_available: u64, cgroup: Opt
/// Resolves the effective memory from an already-refreshed system handle.
///
/// `cgroup_limits()` is computed fresh on each call and is only implemented on
/// Linux (it returns `None` elsewhere), so non-Linux hosts always use host
/// values.
/// On Linux, the current process cgroup and its ancestors are read on each call,
/// including limits imposed by systemd services. System-wide cgroup limits are
/// used if process information is unavailable. Other platforms use host values.
pub fn effective_memory_from_system(system: &System) -> EffectiveMemory {
let cgroup = system.cgroup_limits().map(|limits| (limits.total_memory, limits.free_memory));
#[cfg(target_os = "linux")]
let limits = sysinfo::get_current_pid()
.ok()
.and_then(|pid| {
let mut processes = System::new().ok()?;
processes.refresh_processes_specifics(
sysinfo::ProcessesToUpdate::Some(&[pid]),
false,
sysinfo::ProcessRefreshKind::nothing().with_memory(),
);
processes.process(pid)?.cgroup_limits()
})
.or_else(|| system.cgroup_limits());
#[cfg(not(target_os = "linux"))]
let limits = system.cgroup_limits();
let cgroup = limits.map(|limits| (limits.total_memory, limits.free_memory));
select_effective_memory(system.total_memory(), system.available_memory(), cgroup)
}
@@ -89,6 +104,32 @@ mod tests {
const GIB: u64 = 1024 * 1024 * 1024;
#[cfg(target_os = "linux")]
#[test]
fn effective_memory_uses_current_process_cgroup() {
use sysinfo::{ProcessRefreshKind, ProcessesToUpdate, System, get_current_pid};
let pid = get_current_pid().expect("current process ID");
let mut system = System::new().expect("system memory handle");
system.refresh_memory();
let mut process_system = System::new().expect("process memory handle");
process_system.refresh_processes_specifics(
ProcessesToUpdate::Some(&[pid]),
false,
ProcessRefreshKind::nothing().with_memory(),
);
let limits = process_system.process(pid).expect("current process entry").cgroup_limits();
let expected = select_effective_memory(
system.total_memory(),
system.available_memory(),
limits.map(|limits| (limits.total_memory, limits.free_memory)),
);
let actual = super::effective_memory_from_system(&system);
assert_eq!(actual.basis, expected.basis);
assert_eq!(actual.total_bytes, expected.total_bytes);
assert!(actual.available_bytes <= actual.total_bytes);
}
#[test]
fn select_effective_memory_prefers_constraining_cgroup() {
let effective = select_effective_memory(64 * GIB, 40 * GIB, Some((2 * GIB, GIB)));
+1
View File
@@ -642,6 +642,7 @@ impl AdminAction {
self,
AdminAction::GetTableBucketAction
| AdminAction::SetTableBucketAction
| AdminAction::MigrateTableCatalogAction
| AdminAction::GetTableNamespaceAction
| AdminAction::SetTableNamespaceAction
| AdminAction::UpdateTableNamespacePropertiesAction
+140
View File
@@ -1795,6 +1795,146 @@ mod test {
Ok(())
}
async fn assert_catalog_migration_policy_matrix(data: &[u8], cases: &[(&str, AdminAction, bool)]) -> Result<()> {
let policy = Policy::parse_config(data)?;
let conditions = HashMap::new();
let claims = HashMap::new();
let groups = None;
for (bucket, action, expected) in cases {
let args = Args {
account: "catalog-migration-operator",
groups: &groups,
action: Action::AdminAction(*action),
bucket,
conditions: &conditions,
is_owner: false,
object: "",
claims: &claims,
deny_only: false,
};
assert_eq!(policy.is_allowed(&args).await, *expected, "{action:?} on warehouse {bucket:?}");
}
Ok(())
}
#[tokio::test]
async fn catalog_migration_allow_resource_is_limited_to_the_requested_warehouse() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "admin:MigrateTableCatalog",
"Resource": "arn:aws:s3:::warehouse-a"
}]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("", AdminAction::MigrateTableCatalogAction, false),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_allow_not_resource_excludes_only_the_named_warehouse() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "admin:MigrateTableCatalog",
"NotResource": "arn:aws:s3:::warehouse-b"
}]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("warehouse-c", AdminAction::MigrateTableCatalogAction, true),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_explicit_deny_resource_does_not_spread_to_other_warehouses() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Action": "admin:*", "Resource": "*"},
{"Effect": "Deny", "Action": "admin:MigrateTableCatalog", "Resource": "arn:aws:s3:::warehouse-b"}
]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("warehouse-c", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::ServerInfoAdminAction, true),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_explicit_deny_not_resource_preserves_its_exception() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Action": "admin:*", "Resource": "*"},
{"Effect": "Deny", "Action": "admin:MigrateTableCatalog", "NotResource": "arn:aws:s3:::warehouse-a"}
]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("warehouse-c", AdminAction::MigrateTableCatalogAction, false),
("warehouse-b", AdminAction::ServerInfoAdminAction, true),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_admin_wildcard_still_honors_the_requested_action_scope() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "admin:*",
"Resource": "arn:aws:s3:::warehouse-a"
}]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, false),
("", AdminAction::MigrateTableCatalogAction, false),
("warehouse-b", AdminAction::ServerInfoAdminAction, true),
("", AdminAction::GetTableCatalogAction, true),
],
)
.await
}
#[tokio::test]
async fn catalog_migration_resource_free_admin_grants_keep_their_global_behavior() -> Result<()> {
assert_catalog_migration_policy_matrix(
br#"{
"Version": "2012-10-17",
"Statement": [{"Effect": "Allow", "Action": "admin:MigrateTableCatalog"}]
}"#,
&[
("warehouse-a", AdminAction::MigrateTableCatalogAction, true),
("warehouse-b", AdminAction::MigrateTableCatalogAction, true),
("", AdminAction::MigrateTableCatalogAction, true),
],
)
.await
}
#[tokio::test]
async fn test_table_admin_action_with_resource_is_limited_to_bucket() -> Result<()> {
use crate::policy::action::{Action, AdminAction};
@@ -43,14 +43,9 @@ const CONTAINER_META_HEADER_PREFIX: &str = "x-container-meta-";
/// Request header prefix removing a container metadata item.
const CONTAINER_META_REMOVE_HEADER_PREFIX: &str = "x-remove-container-meta-";
/// Bucket-tag namespace holding account metadata items.
pub(crate) const ACCOUNT_META_TAG_PREFIX: &str = "swift-account-meta-";
/// Bucket-tag namespace holding container metadata items.
///
/// Deliberately narrower than the `swift-` tags around it: the container ACL
/// (`swift-acl-*`) and versioning (`swift-versions-location`) tags share this
/// tag set and must survive a metadata POST.
pub(crate) const CONTAINER_META_TAG_PREFIX: &str = "swift-meta-";
pub(crate) use super::storage_api::tagging::{
SWIFT_ACCOUNT_META_TAG_PREFIX as ACCOUNT_META_TAG_PREFIX, SWIFT_CONTAINER_META_TAG_PREFIX as CONTAINER_META_TAG_PREFIX,
};
/// The metadata changes carried by one account or container POST.
///
+3 -5
View File
@@ -61,11 +61,9 @@ pub use errors::{SwiftError, SwiftResult};
pub use metadata_update::MetadataUpdate;
pub use router::{SwiftRoute, SwiftRouter};
/// Maximum number of metadata headers allowed per resource (Swift standard)
pub(crate) const MAX_METADATA_COUNT: usize = 90;
/// Maximum size in bytes for a single metadata value (Swift standard)
pub(crate) const MAX_METADATA_VALUE_SIZE: usize = 256;
pub(crate) use storage_api::tagging::{
MAX_SWIFT_METADATA_COUNT as MAX_METADATA_COUNT, MAX_SWIFT_METADATA_VALUE_SIZE as MAX_METADATA_VALUE_SIZE,
};
/// Validate metadata against Swift limits
///
+19 -2
View File
@@ -16,6 +16,7 @@ use std::collections::HashMap;
use std::sync::Arc;
pub(crate) use rustfs_ecstore::api::bucket::metadata::BucketMetadata as SwiftBucketMetadata;
use rustfs_ecstore::api::bucket::metadata::SWIFT_TAGGING_MARKER;
use rustfs_ecstore::api::bucket::metadata::{BUCKET_TAGGING_CONFIG, is_unreadable_config_error};
use rustfs_ecstore::api::bucket::metadata_sys::{get as get_swift_bucket_metadata_from_backend, update_config_with};
use rustfs_ecstore::api::bucket::utils::serialize as serialize_bucket_config;
@@ -25,7 +26,7 @@ use rustfs_ecstore::api::notification::get_global_notification_sys;
pub(crate) use rustfs_ecstore::api::runtime::object_store_handle as resolve_swift_object_store_handle;
use rustfs_ecstore::api::storage::ECStore as SwiftStore;
use rustfs_storage_api as storage_contracts;
use s3s::dto::Tagging;
use s3s::dto::{Tag, Tagging};
use super::{SwiftError, SwiftResult};
@@ -54,6 +55,12 @@ pub(crate) mod public_api {
};
}
pub(crate) mod tagging {
pub(crate) use rustfs_ecstore::api::bucket::metadata::{
MAX_SWIFT_METADATA_COUNT, MAX_SWIFT_METADATA_VALUE_SIZE, SWIFT_ACCOUNT_META_TAG_PREFIX, SWIFT_CONTAINER_META_TAG_PREFIX,
};
}
pub(crate) mod versioning {
pub(crate) use super::storage_contracts::{ListOperations, ObjectOperations};
}
@@ -105,7 +112,7 @@ where
// including the container ACL and versioning tags — because the rewrite
// closures treat "no parsed tags" as "no tags".
let result = update_config_with(&bucket, BUCKET_TAGGING_CONFIG, |bm| {
let tagging = match rewrite(bm.tagging_config.as_ref()) {
let mut tagging = match rewrite(bm.tagging_config.as_ref()) {
Ok(tagging) => tagging,
Err(err) => {
rejected = Some(err);
@@ -116,6 +123,16 @@ where
if tagging.tag_set.is_empty() {
Ok(Vec::new())
} else {
if !tagging
.tag_set
.iter()
.any(|tag| tag.key.as_deref() == Some(SWIFT_TAGGING_MARKER))
{
tagging.tag_set.push(Tag {
key: Some(SWIFT_TAGGING_MARKER.to_owned()),
value: Some("1".to_owned()),
});
}
// The S3 XML serializer, not quick_xml: the metadata loader's
// parse step must be able to round-trip what we persist.
serialize_bucket_config(&tagging)
+16
View File
@@ -82,6 +82,22 @@ Configure WebDAV via environment variables:
| `RUSTFS_WEBDAV_MAX_BODY_SIZE` | Max upload size (bytes) | 5GB |
| `RUSTFS_WEBDAV_REQUEST_TIMEOUT` | Request timeout (seconds) | 300 |
| `RUSTFS_WEBDAV_MAX_CONNECTIONS` | Max concurrently served connections | 1024 |
| `RUSTFS_CORS_ALLOWED_ORIGINS` | Browser origins allowed by the S3 and WebDAV listeners | empty |
## Browser Access and CORS
WebDAV reads `RUSTFS_CORS_ALLOWED_ORIGINS` at server startup. Set a comma-separated list of exact frontend origins to allow browser clients, for example `https://dav-console.example,https://files.example`. Explicit origins allow credentialed requests. The default empty value disables cross-origin access. A standalone `*` permits any origin with credentials mode `omit` and emits no `Access-Control-Allow-Credentials` header.
Valid `OPTIONS` preflight requests from an allowed origin receive `204 No Content` without Basic authentication. Actual WebDAV requests and ordinary `OPTIONS` requests still use Basic authentication and the existing IAM checks. Allowed origins receive CORS headers on success and error responses. Preflights allow WebDAV methods and explicitly echo valid requested header names, including `Authorization`, `Depth`, `Destination`, and `If-Match`. Responses expose `ETag`, `DAV`, `Allow`, `Lock-Token`, and range headers to browser code.
```bash
curl -i -X OPTIONS http://localhost:8080/ \
-H 'Origin: https://dav-console.example' \
-H 'Access-Control-Request-Method: PROPFIND' \
-H 'Access-Control-Request-Headers: authorization,depth,content-type'
```
Configure an explicit HTTPS frontend origin for credentialed browser access and use HTTPS on the WebDAV listener when sending access keys and secret keys. This listener-wide policy applies across WebDAV paths. S3 bucket CORS rules remain scoped to the S3 API.
## Quick Start
+4
View File
@@ -23,6 +23,10 @@ pub enum WebDavInitError {
Bind(#[from] std::io::Error),
#[error("server error: {0}")]
Server(String),
#[error("WebDAV background task failed: {0}")]
TaskJoin(#[from] tokio::task::JoinError),
#[error("WebDAV connections did not drain before the shutdown deadline")]
ShutdownDeadline,
#[error("invalid WebDAV configuration: {0}")]
InvalidConfig(String),
#[error("TLS error: {0}")]
+697 -38
View File
@@ -19,8 +19,12 @@ use crate::common::session::{Protocol, ProtocolPrincipal, SessionContext, is_tem
use bytes::Bytes;
use dav_server::DavHandler;
use dav_server::fakels::FakeLs;
use http::header::{AUTHORIZATION, REFERER, USER_AGENT};
use http::{HeaderMap, HeaderValue};
use http::header::{
ACCESS_CONTROL_ALLOW_CREDENTIALS, ACCESS_CONTROL_ALLOW_HEADERS, ACCESS_CONTROL_ALLOW_METHODS, ACCESS_CONTROL_ALLOW_ORIGIN,
ACCESS_CONTROL_EXPOSE_HEADERS, ACCESS_CONTROL_REQUEST_HEADERS, ACCESS_CONTROL_REQUEST_METHOD, AUTHORIZATION, ORIGIN, REFERER,
USER_AGENT, VARY,
};
use http::{HeaderMap, HeaderValue, Method};
use http_body_util::{BodyExt, Full, LengthLimitError, Limited};
use hyper::body::Body as HttpBody;
use hyper::server::conn::http1;
@@ -31,18 +35,23 @@ use rustfs_config::{DEFAULT_TLS_RELOAD_ENABLE, DEFAULT_TLS_RELOAD_INTERVAL, ENV_
use rustfs_tls_runtime::{ReloadableServerCertResolver, TlsReloadOptions, spawn_server_cert_reload_loop};
use rustfs_utils::MaskedAccessKey;
use std::convert::Infallible;
use std::future::Future;
use std::io;
use std::net::IpAddr;
use std::pin::Pin;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Duration;
use subtle::ConstantTimeEq;
use tokio::net::TcpListener;
use tokio::sync::{Semaphore, broadcast, watch};
use tokio::task::JoinSet;
use tokio::time::timeout;
use tokio_rustls::TlsAcceptor;
use tracing::{Instrument, debug, error, info, info_span, warn};
const CONNECTION_DRAIN_TIMEOUT: Duration = Duration::from_secs(10);
const LOG_COMPONENT_PROTOCOLS: &str = "protocols";
const LOG_SUBSYSTEM_WEBDAV_SERVER: &str = "webdav_server";
const LOG_SUBSYSTEM_WEBDAV_AUTH: &str = "webdav_auth";
@@ -60,6 +69,177 @@ const EVENT_WEBDAV_CONNECTION_CAP_STATE: &str = "webdav_connection_cap_state";
/// materialise a whole object in memory for every GET.
type WebDavBody = Pin<Box<dyn HttpBody<Data = Bytes, Error = io::Error> + Send>>;
/// Listener-wide browser policy, parsed once and shared across connections.
enum WebDavCors {
Disabled,
Wildcard,
Origins(Vec<HeaderValue>),
}
impl WebDavCors {
const METHODS: &str = "OPTIONS, GET, HEAD, PUT, DELETE, PATCH, PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, UNLOCK";
fn new(origins: &str) -> Self {
match origins.trim() {
"" => Self::Disabled,
"*" => Self::Wildcard,
origins => Self::Origins(
origins
.split(',')
.map(str::trim)
.filter(|origin| !origin.is_empty() && *origin != "*")
.filter_map(|origin| HeaderValue::from_str(origin).ok())
.collect(),
),
}
}
fn allowed_origin(&self, headers: &HeaderMap) -> Option<HeaderValue> {
if matches!(self, Self::Disabled) {
return None;
}
let mut origins = headers.get_all(ORIGIN).iter();
let origin = origins.next()?;
let value = origin.to_str().ok()?;
if origins.next().is_some() || value.is_empty() || value.bytes().any(|byte| byte.is_ascii_whitespace() || byte == b',') {
return None;
}
match self {
Self::Origins(allowed) => allowed.iter().find(|allowed| *allowed == origin).cloned(),
Self::Wildcard => Some(HeaderValue::from_static("*")),
Self::Disabled => None,
}
}
/// Keep the CORS boundary outside authentication and DAV dispatch so every
/// application response, including early errors, follows the same policy.
async fn handle<B, F, Fut>(&self, req: Request<B>, next: F) -> Result<Response<WebDavBody>, Infallible>
where
F: FnOnce(Request<B>) -> Fut,
Fut: Future<Output = Result<Response<WebDavBody>, Infallible>>,
{
let allow_origin = self.allowed_origin(req.headers());
let is_options = req.method() == Method::OPTIONS;
let is_preflight =
is_options && req.headers().contains_key(ORIGIN) && req.headers().contains_key(ACCESS_CONTROL_REQUEST_METHOD);
let mut response = if is_preflight {
self.preflight(req.headers(), allow_origin.is_some())
} else {
next(req).await?
};
// Absent and rejected Origin values also select cache variants. Append
// rather than overwrite the DAV handler's existing Vary fields.
response.headers_mut().append(VARY, HeaderValue::from_static("Origin"));
if is_options {
response.headers_mut().append(
VARY,
HeaderValue::from_static("Access-Control-Request-Method, Access-Control-Request-Headers"),
);
}
if let Some(origin) = allow_origin {
response.headers_mut().insert(ACCESS_CONTROL_ALLOW_ORIGIN, origin);
if matches!(self, Self::Origins(_)) {
response
.headers_mut()
.insert(ACCESS_CONTROL_ALLOW_CREDENTIALS, HeaderValue::from_static("true"));
}
if !is_preflight {
response.headers_mut().insert(
ACCESS_CONTROL_EXPOSE_HEADERS,
HeaderValue::from_static("ETag, DAV, Allow, Lock-Token, Content-Range, Accept-Ranges, WWW-Authenticate"),
);
}
}
Ok(response)
}
fn preflight(&self, headers: &HeaderMap, origin_allowed: bool) -> Response<WebDavBody> {
let mut methods = headers.get_all(ACCESS_CONTROL_REQUEST_METHOD).iter();
let method = methods.next().and_then(|value| value.to_str().ok());
let method_allowed = method.is_some_and(|method| Self::METHODS.split(", ").any(|allowed| allowed == method));
if !origin_allowed || !method_allowed || methods.next().is_some() {
return error_response(StatusCode::FORBIDDEN, "CORS preflight denied");
}
let Ok(allowed_headers) = cors_requested_headers(headers) else {
return error_response(StatusCode::FORBIDDEN, "CORS preflight denied");
};
let mut response = Response::new(fixed_body(Bytes::new()));
*response.status_mut() = StatusCode::NO_CONTENT;
response
.headers_mut()
.insert(ACCESS_CONTROL_ALLOW_METHODS, HeaderValue::from_static(Self::METHODS));
// A wildcard does not cover Authorization, and is not a wildcard for
// credentialed requests. Echo the validated names explicitly instead.
if let Some(value) = allowed_headers {
response.headers_mut().insert(ACCESS_CONTROL_ALLOW_HEADERS, value);
}
response
}
}
/// Validate borrowed field names, reusing an ordinary single header value.
/// RFC 9110 section 5.6.1.2 requires recipients to ignore empty list elements;
/// normalize those and repeated fields into one nonempty response list.
fn cors_requested_headers(headers: &HeaderMap) -> Result<Option<HeaderValue>, ()> {
let values = headers.get_all(ACCESS_CONTROL_REQUEST_HEADERS);
let mut fields = values.iter();
let Some(first) = fields.next() else {
return Ok(None);
};
let mut normalize = fields.next().is_some();
let mut has_name = false;
for value in values.iter() {
for name in value.to_str().map_err(|_| ())?.split(',') {
let name = name.trim_matches([' ', '\t']);
if name.is_empty() {
normalize = true;
continue;
}
if !is_cors_header_name(name) {
return Err(());
}
has_name = true;
}
}
if !has_name {
return Err(());
}
if !normalize {
return Ok(Some(first.clone()));
}
let capacity = values.iter().map(|value| value.as_bytes().len() + 1).sum();
let mut normalized = String::with_capacity(capacity);
for value in values {
for name in value.to_str().map_err(|_| ())?.split(',') {
let name = name.trim_matches([' ', '\t']);
if name.is_empty() {
continue;
}
if !normalized.is_empty() {
normalized.push(',');
}
normalized.push_str(name);
}
}
HeaderValue::try_from(normalized).map(Some).map_err(|_| ())
}
/// RFC 9110 field-name is a nonempty sequence of ASCII tchar bytes.
fn is_cors_header_name(name: &str) -> bool {
!name.is_empty()
&& name.bytes().all(|byte| {
byte.is_ascii_alphanumeric()
|| matches!(
byte,
b'!' | b'#' | b'$' | b'%' | b'&' | b'\'' | b'*' | b'+' | b'-' | b'.' | b'^' | b'_' | b'`' | b'|' | b'~'
)
})
}
fn policy_request_headers(headers: &HeaderMap) -> HeaderMap {
let mut policy_headers = HeaderMap::new();
for name in [USER_AGENT, REFERER] {
@@ -74,6 +254,11 @@ fn policy_request_headers(headers: &HeaderMap) -> HeaderMap {
policy_headers
}
struct ConnectionLimits {
max_body_size: u64,
request_timeout: Duration,
}
/// WebDAV server implementation
pub struct WebDavServer<S>
where
@@ -83,6 +268,7 @@ where
config: WebDavConfig,
/// S3 storage backend
storage: S,
cors: Arc<WebDavCors>,
}
impl<S> WebDavServer<S>
@@ -100,7 +286,13 @@ where
/// Create a new WebDAV server
pub async fn new(config: WebDavConfig, storage: S) -> Result<Self, WebDavInitError> {
config.validate().await?;
Ok(Self { config, storage })
let origins =
rustfs_utils::get_env_str(rustfs_config::ENV_CORS_ALLOWED_ORIGINS, rustfs_config::DEFAULT_CORS_ALLOWED_ORIGINS);
Ok(Self {
config,
storage,
cors: Arc::new(WebDavCors::new(&origins)),
})
}
/// Start the WebDAV server
@@ -135,7 +327,8 @@ where
bind_addr = %self.config.bind_addr,
"WebDAV server listening"
);
let (reload_shutdown_tx, reload_shutdown_rx) = watch::channel(false);
let (shutdown_tx, shutdown) = watch::channel(false);
let mut reload_task = None;
// Setup TLS if enabled
let tls_acceptor = if self.config.tls_enabled {
@@ -151,15 +344,9 @@ where
let resolver = ReloadableServerCertResolver::load_from_directory(cert_dir)
.map_err(|e| WebDavInitError::Tls(format!("Failed to create certificate resolver: {}", e)))?;
let _reload_task = spawn_server_cert_reload_loop(
"webdav",
resolver.clone(),
Self::tls_reload_options(),
reload_shutdown_rx.clone(),
);
let server_config = crate::common::tls::server_config(resolver, self.config.ca_file.as_deref())
let server_config = crate::common::tls::server_config(resolver.clone(), self.config.ca_file.as_deref())
.map_err(|error| WebDavInitError::Tls(format!("Invalid client CA configuration: {error}")))?;
reload_task = spawn_server_cert_reload_loop("webdav", resolver, Self::tls_reload_options(), shutdown.clone());
Some(TlsAcceptor::from(Arc::new(server_config)))
} else {
@@ -173,7 +360,15 @@ where
let request_timeout = Duration::from_secs(self.config.request_timeout_secs);
let connection_limiter = Arc::new(Semaphore::new(self.config.max_connections.min(Semaphore::MAX_PERMITS)));
loop {
let mut connections = JoinSet::new();
let mut failure = None;
'accept: loop {
while let Some(result) = connections.try_join_next() {
if let Err(error) = result {
failure = Some(WebDavInitError::TaskJoin(error));
break 'accept;
}
}
// Admission control: hold a permit before accepting, so at
// saturation the kernel backlog absorbs the burst instead of the
// process spawning an unbounded number of connection tasks. The
@@ -204,7 +399,6 @@ where
state = "shutdown_requested",
"WebDAV shutdown requested"
);
let _ = reload_shutdown_tx.send(true);
break;
}
}
@@ -216,25 +410,35 @@ where
match accept_result {
Ok((stream, addr)) => {
let storage = storage.clone();
let cors = self.cors.clone();
let tls_acceptor = tls_acceptor.clone();
let max_body_size = self.config.max_body_size;
let limits = ConnectionLimits {
max_body_size: self.config.max_body_size,
request_timeout,
};
let source_ip: IpAddr = addr.ip();
let span = info_span!(
"webdav-connection",
peer = %source_ip,
transport = if tls_acceptor.is_some() { "tls" } else { "tcp" },
);
tokio::spawn(
let mut shutdown = shutdown.clone();
connections.spawn(
async move {
let _permit = permit;
if let Some(acceptor) = tls_acceptor {
// A handshake that never completes would otherwise
// hold its connection permit forever.
match timeout(request_timeout, acceptor.accept(stream)).await {
let handshake = tokio::select! {
biased;
_ = shutdown.changed() => return,
result = timeout(request_timeout, acceptor.accept(stream)) => result,
};
match handshake {
Ok(Ok(tls_stream)) => {
let io = TokioIo::new(tls_stream);
if let Err(e) = Self::handle_connection_impl(io, storage, source_ip, true, max_body_size, request_timeout).await {
if let Err(e) = Self::handle_connection_impl(io, storage, source_ip, true, limits, cors, shutdown).await {
debug!(
event = EVENT_WEBDAV_CONNECTION_STATE,
component = LOG_COMPONENT_PROTOCOLS,
@@ -272,7 +476,7 @@ where
}
} else {
let io = TokioIo::new(stream);
if let Err(e) = Self::handle_connection_impl(io, storage, source_ip, false, max_body_size, request_timeout).await {
if let Err(e) = Self::handle_connection_impl(io, storage, source_ip, false, limits, cors, shutdown).await {
debug!(
event = EVENT_WEBDAV_CONNECTION_STATE,
component = LOG_COMPONENT_PROTOCOLS,
@@ -309,13 +513,29 @@ where
state = "shutdown_requested",
"WebDAV shutdown requested"
);
let _ = reload_shutdown_tx.send(true);
break;
}
}
}
let _ = reload_shutdown_tx.send(true);
drop(listener);
shutdown_tx.send_replace(true);
let drain = async {
while let Some(result) = connections.join_next().await {
if let Err(error) = result {
failure.get_or_insert(WebDavInitError::TaskJoin(error));
}
}
};
if timeout(CONNECTION_DRAIN_TIMEOUT, drain).await.is_err() {
connections.shutdown().await;
failure.get_or_insert(WebDavInitError::ShutdownDeadline);
}
if let Some(task) = reload_task
&& let Err(error) = task.await
{
failure.get_or_insert(WebDavInitError::TaskJoin(error));
}
info!(
event = EVENT_WEBDAV_SERVER_STATE,
component = LOG_COMPONENT_PROTOCOLS,
@@ -323,7 +543,7 @@ where
state = "stopped",
"WebDAV server stopped"
);
Ok(())
failure.map_or(Ok(()), Err)
}
/// Handle a single connection with hyper-util TokioIo wrapper
@@ -332,25 +552,49 @@ where
storage: S,
source_ip: IpAddr,
secure_transport: bool,
max_body_size: u64,
request_timeout: Duration,
limits: ConnectionLimits,
cors: Arc<WebDavCors>,
mut shutdown: watch::Receiver<bool>,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>>
where
I: tokio::io::AsyncRead + tokio::io::AsyncWrite + Unpin + Send + 'static,
{
let service = service_fn(move |req: Request<hyper::body::Incoming>| {
let storage = storage.clone();
async move { Self::handle_request(req, storage, source_ip, secure_transport, max_body_size, request_timeout).await }
let ConnectionLimits {
max_body_size,
request_timeout,
} = limits;
// These borrows live for the connection. Preflights complete before
// invoking the closure that clones the backend for an actual request.
let cors = cors.as_ref();
let storage = &storage;
let received_request = AtomicBool::new(false);
let service = service_fn(|req: Request<hyper::body::Incoming>| {
received_request.store(true, Ordering::Relaxed);
cors.handle(req, move |req| {
Self::handle_request(req, storage.clone(), source_ip, secure_transport, max_body_size, request_timeout)
})
});
// A peer that opens a connection and dribbles (or never finishes)
// request headers is disconnected once the configured request
// timeout elapses. The timer is required for the deadline to apply.
http1::Builder::new()
let connection = http1::Builder::new()
.timer(TokioTimer::new())
.header_read_timeout(request_timeout)
.serve_connection(io, service)
.await?;
.serve_connection(io, service);
tokio::pin!(connection);
tokio::select! {
biased;
_ = shutdown.changed() => {
// Hyper starts in a busy state even before parsing the first
// request. An incomplete header has no application work to drain.
if received_request.load(Ordering::Relaxed) {
connection.as_mut().graceful_shutdown();
connection.await?;
}
}
result = &mut connection => result?,
}
Ok(())
}
@@ -693,6 +937,7 @@ mod tests {
use s3s::dto::*;
use std::fmt::{Debug, Formatter};
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::task::{Context, Poll};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
@@ -701,8 +946,19 @@ mod tests {
/// Storage double for the connection-level tests. Those requests are
/// rejected before authentication succeeds, so storage is never reached.
#[derive(Clone)]
struct StubStorage;
#[derive(Default)]
struct StubStorage {
clone_count: Arc<AtomicUsize>,
}
impl Clone for StubStorage {
fn clone(&self) -> Self {
self.clone_count.fetch_add(1, Ordering::Relaxed);
Self {
clone_count: self.clone_count.clone(),
}
}
}
impl Debug for StubStorage {
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
@@ -873,6 +1129,362 @@ mod tests {
.expect("build get request")
}
const BROWSER_ORIGIN: &str = "https://dav-console.example";
async fn cors_http_request(origins: &str, method: &str, headers: &str) -> String {
cors_http_exchange(origins, method, headers).await.0
}
async fn cors_http_exchange(origins: &str, method: &str, headers: &str) -> (String, usize) {
let storage = StubStorage::default();
let clone_count = storage.clone_count.clone();
let (mut client, server) = tokio::io::duplex(8192);
let (_shutdown_tx, shutdown) = tokio::sync::watch::channel(false);
let connection = tokio::spawn(WebDavServer::<StubStorage>::handle_connection_impl(
TokioIo::new(server),
storage,
TEST_IP,
false,
ConnectionLimits {
max_body_size: 16,
request_timeout: Duration::from_secs(30),
},
Arc::new(WebDavCors::new(origins)),
shutdown,
));
let request = format!("{method} / HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n{headers}\r\n");
client.write_all(request.as_bytes()).await.expect("write browser request");
let mut response = Vec::new();
timeout(Duration::from_secs(5), client.read_to_end(&mut response))
.await
.expect("browser response timeout")
.expect("read browser response");
timeout(Duration::from_secs(5), connection)
.await
.expect("connection shutdown timeout")
.expect("connection task")
.expect("serve browser request");
(
String::from_utf8(response).expect("ASCII HTTP response"),
clone_count.load(Ordering::Relaxed),
)
}
fn browser_request(method: &str, path: &str) -> Request<Full<Bytes>> {
Request::builder()
.method(method)
.uri(path)
.header(ORIGIN, BROWSER_ORIGIN)
.body(Full::new(Bytes::new()))
.expect("build browser request")
}
fn assert_browser_headers(response: &Response<WebDavBody>) {
assert_eq!(response.headers()[ACCESS_CONTROL_ALLOW_ORIGIN], BROWSER_ORIGIN);
assert_eq!(response.headers()[ACCESS_CONTROL_ALLOW_CREDENTIALS], "true");
assert!(response.headers().get_all(VARY).iter().any(|value| value == "Origin"));
}
#[tokio::test]
async fn webdav_cors_preflight_bypasses_authentication() {
for method in [
"OPTIONS",
"GET",
"HEAD",
"PUT",
"DELETE",
"PATCH",
"PROPFIND",
"PROPPATCH",
"MKCOL",
"COPY",
"MOVE",
"LOCK",
"UNLOCK",
] {
let headers = format!(
"Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: {method}\r\nAccess-Control-Request-Headers: authorization,depth,content-type,destination,if-match\r\n"
);
let response = cors_http_request(BROWSER_ORIGIN, "OPTIONS", &headers).await;
assert!(response.starts_with("HTTP/1.1 204"), "{method}: {response}");
assert!(response.contains(&format!("access-control-allow-origin: {BROWSER_ORIGIN}\r\n")));
assert!(response.contains("access-control-allow-credentials: true\r\n"));
let allowed_methods = response
.lines()
.find_map(|line| line.strip_prefix("access-control-allow-methods: "))
.expect("preflight must advertise allowed methods");
assert!(allowed_methods.split(',').map(str::trim).any(|allowed| allowed == method));
assert!(response.contains("access-control-allow-headers: authorization,depth,content-type,destination,if-match\r\n"));
assert!(response.contains("vary: Access-Control-Request-Method, Access-Control-Request-Headers\r\n"));
assert!(!response.contains("www-authenticate:"));
}
}
#[tokio::test]
async fn webdav_cors_actual_requests_and_plain_options_require_authentication() {
for method in ["GET", "PUT", "DELETE", "PROPFIND", "MOVE", "OPTIONS"] {
let response = cors_http_request(BROWSER_ORIGIN, method, &format!("Origin: {BROWSER_ORIGIN}\r\n")).await;
assert!(response.starts_with("HTTP/1.1 401"), "{method}: {response}");
assert!(response.contains(&format!("access-control-allow-origin: {BROWSER_ORIGIN}\r\n")));
assert!(response.contains("www-authenticate: Basic"));
}
for method in ["GET", "PROPFIND"] {
for headers in ["", "Origin: https://other.example\r\n"] {
let response = cors_http_request(BROWSER_ORIGIN, method, headers).await;
assert!(response.starts_with("HTTP/1.1 401"), "{method}: {response}");
assert!(!response.contains("access-control-allow-origin:"));
assert!(response.contains("vary: Origin\r\n"));
}
}
for headers in [
format!("Origin: {BROWSER_ORIGIN}\r\nAuthorization: Basic !!!!\r\n"),
format!("Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PUT\r\n"),
] {
let response = cors_http_request(BROWSER_ORIGIN, "PUT", &headers).await;
assert!(response.starts_with("HTTP/1.1 401"));
assert!(response.contains(&format!("access-control-allow-origin: {BROWSER_ORIGIN}\r\n")));
}
let response = cors_http_request(BROWSER_ORIGIN, "OPTIONS", "Access-Control-Request-Method: PROPFIND\r\n").await;
assert!(response.starts_with("HTTP/1.1 401"));
assert!(!response.contains("access-control-allow-origin:"));
assert!(response.contains("vary: Origin\r\n"));
assert!(response.contains("vary: Access-Control-Request-Method, Access-Control-Request-Headers\r\n"));
}
#[tokio::test]
async fn webdav_cors_keeps_early_body_limit_errors_visible() {
let response =
cors_http_request(BROWSER_ORIGIN, "PUT", &format!("Origin: {BROWSER_ORIGIN}\r\nContent-Length: 17\r\n")).await;
assert!(response.starts_with("HTTP/1.1 413"));
assert!(response.contains(&format!("access-control-allow-origin: {BROWSER_ORIGIN}\r\n")));
}
#[tokio::test]
async fn webdav_cors_rejects_unconfigured_and_unmatched_origins() {
for (configuration, origin) in [
("", BROWSER_ORIGIN),
(" ", BROWSER_ORIGIN),
(BROWSER_ORIGIN, "https://dav-console.example.attacker.test"),
(BROWSER_ORIGIN, "null"),
(BROWSER_ORIGIN, "https://other.example"),
("*, https://other.example", BROWSER_ORIGIN),
] {
let headers = format!("Origin: {origin}\r\nAccess-Control-Request-Method: PROPFIND\r\n");
let response = cors_http_request(configuration, "OPTIONS", &headers).await;
assert!(response.starts_with("HTTP/1.1 403"), "{configuration:?}: {response}");
assert!(!response.contains("access-control-allow-origin:"));
assert!(!response.contains("access-control-allow-credentials:"));
assert!(response.contains("vary: Origin\r\n"));
}
}
#[tokio::test]
async fn webdav_cors_rejects_malformed_preflights() {
for headers in [
format!("Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: TRACE\r\n"),
format!("Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: propfind\r\n"),
format!(
"Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PROPFIND\r\nAccess-Control-Request-Method: GET\r\n"
),
format!("Origin: {BROWSER_ORIGIN}\r\nOrigin: https://other.example\r\nAccess-Control-Request-Method: PROPFIND\r\n"),
format!("Origin: {BROWSER_ORIGIN},https://other.example\r\nAccess-Control-Request-Method: PROPFIND\r\n"),
format!(
"Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PROPFIND\r\nAccess-Control-Request-Headers: invalid header\r\n"
),
format!(
"Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PROPFIND\r\nAccess-Control-Request-Headers: , \t,\r\n"
),
] {
let response = cors_http_request(BROWSER_ORIGIN, "OPTIONS", &headers).await;
assert!(response.starts_with("HTTP/1.1 403"), "{headers}: {response}");
assert!(!response.contains("access-control-allow-methods:"));
}
}
#[tokio::test]
async fn webdav_cors_normalizes_header_lists() {
for requested in [
"authorization, ,depth,",
" , authorization,\tdepth , ",
"authorization,\r\nAccess-Control-Request-Headers: , depth,",
] {
let headers = format!(
"Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PROPFIND\r\nAccess-Control-Request-Headers: {requested}\r\n"
);
let response = cors_http_request(BROWSER_ORIGIN, "OPTIONS", &headers).await;
assert!(response.starts_with("HTTP/1.1 204"), "{response}");
assert!(response.contains("access-control-allow-headers: authorization,depth\r\n"));
}
}
#[test]
fn webdav_cors_header_name_validation_matches_http_tokens() {
for byte in u8::MIN..=u8::MAX {
let bytes = [b'x', byte, b'y'];
let actual = std::str::from_utf8(&bytes).is_ok_and(is_cors_header_name);
assert_eq!(actual, http::HeaderName::from_bytes(&bytes).is_ok(), "byte {byte}");
}
assert!(!is_cors_header_name(""));
assert!(is_cors_header_name("Authorization"));
assert!(is_cors_header_name("!#$%&'*+-.^_`|~"));
}
#[tokio::test]
async fn webdav_cors_only_clones_backend_for_actual_requests() {
for (method, requested_method, expected_clones, expected_status) in [
("OPTIONS", Some("PROPFIND"), 0, "204"),
("OPTIONS", Some("TRACE"), 0, "403"),
("OPTIONS", None, 1, "401"),
("PUT", Some("PUT"), 1, "401"),
] {
let mut headers = format!("Origin: {BROWSER_ORIGIN}\r\n");
if let Some(requested_method) = requested_method {
headers.push_str(&format!("Access-Control-Request-Method: {requested_method}\r\n"));
}
let (response, clones) = cors_http_exchange(BROWSER_ORIGIN, method, &headers).await;
assert!(response.starts_with(&format!("HTTP/1.1 {expected_status}")), "{response}");
assert_eq!(clones, expected_clones, "{method} with {requested_method:?}");
}
}
#[tokio::test]
async fn webdav_cors_wildcard_never_allows_credentials() {
let headers = format!(
"Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PROPFIND\r\nAccess-Control-Request-Headers: Authorization\r\n"
);
let preflight = cors_http_request(" * ", "OPTIONS", &headers).await;
assert!(preflight.starts_with("HTTP/1.1 204"));
assert!(preflight.contains("access-control-allow-origin: *\r\n"));
assert!(preflight.contains("access-control-allow-headers: Authorization\r\n"));
assert!(!preflight.contains("access-control-allow-credentials:"));
let actual = cors_http_request("*", "PROPFIND", &format!("Origin: {BROWSER_ORIGIN}\r\n")).await;
assert!(actual.starts_with("HTTP/1.1 401"));
assert!(actual.contains("access-control-allow-origin: *\r\n"));
assert!(!actual.contains("access-control-allow-credentials:"));
}
#[tokio::test]
async fn webdav_cors_allowlist_trims_entries_and_matches_exactly() {
let configuration = format!(" https://other.example , {BROWSER_ORIGIN} ");
let headers = format!("Origin: {BROWSER_ORIGIN}\r\nAccess-Control-Request-Method: PROPFIND\r\n");
let response = cors_http_request(&configuration, "OPTIONS", &headers).await;
assert!(response.starts_with("HTTP/1.1 204"));
assert!(response.contains(&format!("access-control-allow-origin: {BROWSER_ORIGIN}\r\n")));
}
#[tokio::test]
async fn webdav_cors_wraps_dav_multistatus_etag_and_streamed_body() {
let cors = WebDavCors::new(BROWSER_ORIGIN);
let handler = memfs_handler();
let created = dispatch_dav(chunked_request(2, 4), handler.clone(), TEST_IP, 16, Duration::from_secs(30)).await;
assert_eq!(created.status(), StatusCode::CREATED);
let propfind = cors
.handle(browser_request("PROPFIND", "/"), |req| async {
Ok(dispatch_dav(req, handler.clone(), TEST_IP, 16, Duration::from_secs(30)).await)
})
.await
.expect("PROPFIND response");
assert_eq!(propfind.status(), StatusCode::MULTI_STATUS);
assert_browser_headers(&propfind);
let get = cors
.handle(browser_request("GET", "/upload.bin"), |req| async {
Ok(dispatch_dav(req, handler.clone(), TEST_IP, 16, Duration::from_secs(30)).await)
})
.await
.expect("GET response");
assert_eq!(get.status(), StatusCode::OK);
assert_browser_headers(&get);
assert!(get.headers().contains_key("etag"));
assert!(
get.headers()[ACCESS_CONTROL_EXPOSE_HEADERS]
.to_str()
.expect("exposed headers")
.contains("ETag")
);
assert!(get.body().size_hint().upper().is_none(), "CORS must preserve streaming");
let data = get.into_body().collect().await.expect("stream GET body").to_bytes();
assert_eq!(data.as_ref(), b"aaaaaaaa");
let mut put = browser_request("PUT", "/upload.bin");
put.headers_mut()
.insert("if-match", HeaderValue::from_static("\"stale-etag\""));
let conditional = cors
.handle(put, |req| async {
Ok(dispatch_dav(req, handler, TEST_IP, 16, Duration::from_secs(30)).await)
})
.await
.expect("conditional PUT response");
assert_eq!(conditional.status(), StatusCode::PRECONDITION_FAILED);
assert_browser_headers(&conditional);
}
#[tokio::test]
async fn webdav_cors_wraps_dispatch_body_limit_errors() {
let cors = WebDavCors::new(BROWSER_ORIGIN);
let mut request = chunked_request(8, 8);
request.headers_mut().insert(ORIGIN, HeaderValue::from_static(BROWSER_ORIGIN));
let oversized = cors
.handle(request, |req| async {
Ok(dispatch_dav(req, memfs_handler(), TEST_IP, 16, Duration::from_secs(30)).await)
})
.await
.expect("oversized response");
assert_eq!(oversized.status(), StatusCode::PAYLOAD_TOO_LARGE);
assert_browser_headers(&oversized);
}
#[tokio::test(start_paused = true)]
async fn webdav_cors_wraps_dispatch_timeout_errors() {
let cors = WebDavCors::new(BROWSER_ORIGIN);
let request = Request::builder()
.method("PUT")
.uri("/stalled.bin")
.header(ORIGIN, BROWSER_ORIGIN)
.body(StalledBody)
.expect("stalled browser request");
let response = cors
.handle(request, |req| async {
Ok(dispatch_dav(req, memfs_handler(), TEST_IP, 16, Duration::from_secs(30)).await)
})
.await
.expect("timeout response");
assert_eq!(response.status(), StatusCode::REQUEST_TIMEOUT);
assert_browser_headers(&response);
}
#[tokio::test]
async fn webdav_cors_preserves_existing_vary_on_success_and_errors() {
let cors = WebDavCors::new(BROWSER_ORIGIN);
for status in [
StatusCode::OK,
StatusCode::FORBIDDEN,
StatusCode::NOT_FOUND,
StatusCode::INTERNAL_SERVER_ERROR,
] {
let response = cors
.handle(browser_request("GET", "/"), |_| async {
let mut response = error_response(status, "test response");
response
.headers_mut()
.append(VARY, HeaderValue::from_static("Accept-Encoding"));
Ok(response)
})
.await
.expect("wrapped response");
assert_eq!(response.status(), status);
assert_browser_headers(&response);
assert!(
response
.headers()
.get_all(VARY)
.iter()
.any(|value| value == "Accept-Encoding")
);
}
}
#[test]
fn policy_headers_drop_credentials_and_s3_auth_spoofing() {
let mut headers = HeaderMap::new();
@@ -969,13 +1581,18 @@ mod tests {
async fn slow_request_headers_hit_request_timeout() {
let (mut client, server) = tokio::io::duplex(1024);
let (_shutdown_tx, shutdown) = tokio::sync::watch::channel(false);
let conn = tokio::spawn(WebDavServer::<StubStorage>::handle_connection_impl(
TokioIo::new(server),
StubStorage,
StubStorage::default(),
TEST_IP,
false,
1024,
Duration::from_secs(30),
ConnectionLimits {
max_body_size: 1024,
request_timeout: Duration::from_secs(30),
},
Arc::new(WebDavCors::new("")),
shutdown,
));
client
@@ -987,6 +1604,40 @@ mod tests {
assert!(outcome.is_ok(), "connection outlived the configured request timeout");
}
#[tokio::test]
async fn shutdown_closes_connection_with_incomplete_first_headers() {
use std::future::{Future, poll_fn};
use std::task::Poll;
let (mut client, server) = tokio::io::duplex(1024);
let (shutdown_tx, shutdown) = watch::channel(false);
let connection = WebDavServer::<StubStorage>::handle_connection_impl(
TokioIo::new(server),
StubStorage::default(),
TEST_IP,
false,
ConnectionLimits {
max_body_size: 1024,
request_timeout: Duration::from_secs(30),
},
Arc::new(WebDavCors::new("")),
shutdown,
);
tokio::pin!(connection);
client
.write_all(b"GET / HTTP/1.1\r\nHost: localhost\r\n")
.await
.expect("write partial headers");
let pending = poll_fn(|cx| Poll::Ready(connection.as_mut().poll(cx).is_pending())).await;
assert!(pending, "connection should be waiting for the rest of the first header");
shutdown_tx.send_replace(true);
timeout(Duration::from_secs(1), connection.as_mut())
.await
.expect("shutdown should not wait for an incomplete header")
.expect("no request reached the service");
}
/// R05-CAN-097: the accept loop must not serve more connections at once
/// than the configured cap.
#[tokio::test]
@@ -1000,7 +1651,7 @@ mod tests {
request_timeout_secs: 30,
max_connections: 1,
};
let server = WebDavServer::new(config, StubStorage).await.expect("build server");
let server = WebDavServer::new(config, StubStorage::default()).await.expect("build server");
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await.expect("bind listener");
let addr = listener.local_addr().expect("listener addr");
@@ -1035,8 +1686,16 @@ mod tests {
.expect("read second");
assert!(served > 0, "second connection returned no data");
let _ = shutdown_tx.send(());
let _ = timeout(Duration::from_secs(10), serving).await;
shutdown_tx.send(()).expect("signal server");
timeout(Duration::from_secs(5), serving)
.await
.expect("server did not retire the keep-alive connection")
.expect("server task panicked")
.expect("server did not shut down cleanly");
let remaining = timeout(Duration::from_secs(5), second.read_to_end(&mut Vec::new()))
.await
.expect("connection remained open after server exit");
remaining.expect("read remaining response and EOF");
}
#[tokio::test]
@@ -21,5 +21,7 @@
//! Swift implementation.
pub use rustfs_ecstore::api::bucket::metadata::load_bucket_metadata;
pub use rustfs_ecstore::api::bucket::metadata_sys::{get as get_bucket_metadata, set_bucket_metadata};
pub use rustfs_ecstore::api::bucket::metadata_sys::{
get as get_bucket_metadata, get_s3_tagging_config, set_bucket_metadata, update_s3_tagging_config,
};
pub use rustfs_ecstore::api::runtime::object_store_handle;
@@ -103,11 +103,110 @@ async fn swift_metadata_writes_are_durable() {
posts_survive_disk_truth_reload(&env).await;
tag_writers_preserve_each_others_state(&env).await;
s3_tag_writes_preserve_swift_metadata(&env).await;
unrelated_account_posts_preserve_the_tempurl_key().await;
acl_only_posts_leave_container_metadata_alone(&env).await;
versioning_writes_reject_missing_containers().await;
}
async fn s3_tag_writes_preserve_swift_metadata(env: &TestECStoreEnv) {
use ecstore_test_compat::{get_s3_tagging_config, update_s3_tagging_config};
let s3_only = "s3-only-swift-like-keys";
env.make_bucket(s3_only, false).await;
let mut ordinary_tags = get_bucket_metadata(s3_only)
.await
.expect("S3-only bucket metadata")
.tagging_config
.clone()
.unwrap_or_default();
ordinary_tags.tag_set =
serde_json::from_value(json!([{"key": "swift-meta-owner", "value": "blue"}])).expect("ordinary tag fixture");
update_s3_tagging_config(s3_only, Some(ordinary_tags.clone()), None)
.await
.expect("S3-only buckets accept Swift-like names");
reload_bucket_metadata_from_disk(s3_only).await;
assert_eq!(
get_s3_tagging_config(s3_only).await.expect("S3-only tag read").0.tag_set,
ordinary_tags.tag_set
);
let project_id = "swifts3tagproj";
let account = format!("AUTH_{project_id}");
let container = "shared-s3-view";
let bucket = ContainerMapper::default().swift_to_s3_bucket(container, project_id);
let credentials = keystone_credentials(project_id);
env.make_bucket(&bucket, false).await;
let mut metadata = MetadataUpdate::default();
for index in 0..51 {
metadata = metadata.set(&format!("item-{index}"), "a,b");
}
update_container_metadata(&account, container, &credentials, metadata)
.await
.expect("51 Swift metadata values should persist");
container::set_container_acl(&account, container, Some(".r:*"), None, &credentials)
.await
.expect("public ACL should persist");
let mut s3_tags = get_s3_tagging_config(&bucket).await.expect("initial S3 view").0;
assert!(s3_tags.tag_set.is_empty(), "Swift fields must be absent from the initial S3 view");
s3_tags.tag_set = (0..50)
.map(|index| serde_json::from_value(json!({"key": format!("team-{index}"), "value": "blue"})).expect("S3 tag fixture"))
.collect();
update_s3_tagging_config(&bucket, Some(s3_tags.clone()), None)
.await
.expect("S3 tags must coexist with Swift metadata");
reload_bucket_metadata_from_disk(&bucket).await;
let (read_back, _) = get_s3_tagging_config(&bucket).await.expect("S3 view should read");
assert_eq!(read_back.tag_set, s3_tags.tag_set, "the S3 view must contain only the 50 S3 tags");
for forbidden_key in ["swift-meta-item-0", "!rustfs-swift-metadata"] {
let mut forbidden = s3_tags.clone();
forbidden.tag_set.truncate(1);
forbidden.tag_set[0].key = Some(forbidden_key.to_owned());
forbidden.tag_set[0].value = Some("1".to_owned());
assert!(
update_s3_tagging_config(&bucket, Some(forbidden), None).await.is_err(),
"S3 must reject overwriting Swift metadata or forging its marker"
);
assert_eq!(
get_s3_tagging_config(&bucket)
.await
.expect("rejected writes preserve S3 tags")
.0
.tag_set,
read_back.tag_set
);
}
update_s3_tagging_config(&bucket, Some(read_back), None)
.await
.expect("S3 read-back must be writable");
assert!(
update_s3_tagging_config(&bucket, Some(s3_tags), Some(uuid::Uuid::new_v4()))
.await
.is_err(),
"a stale bucket incarnation must reject the mutation"
);
update_s3_tagging_config(&bucket, None, None)
.await
.expect("S3 delete should preserve Swift metadata");
reload_bucket_metadata_from_disk(&bucket).await;
assert!(
get_s3_tagging_config(&bucket)
.await
.expect("Swift-only config has an empty S3 view")
.0
.tag_set
.is_empty()
);
assert_eq!(persisted_container_metadata(&bucket).await.len(), 51);
assert!(
container::get_container_acl(&account, container, &credentials)
.await
.expect("Swift ACL should load")
.check_read_access(None, None, None),
"S3 tag mutations must preserve the public Swift ACL"
);
}
async fn posts_survive_disk_truth_reload(env: &TestECStoreEnv) {
// --- Container metadata POST (X-Container-Meta-*) ---
let project_id = "swiftpersistproj";
@@ -234,13 +333,18 @@ async fn tag_writers_preserve_each_others_state(env: &TestECStoreEnv) {
env.make_bucket(&ContainerMapper::default().swift_to_s3_bucket(archive, project_id), false)
.await;
update_container_metadata(&swift_account, container, &credentials, MetadataUpdate::default().set("color", "blue"))
.await
.expect("container metadata POST should succeed");
update_container_metadata(
&swift_account,
container,
&credentials,
MetadataUpdate::default().set("color", "blue,green"),
)
.await
.expect("container metadata POST should succeed");
container::enable_versioning(&swift_account, container, archive, &credentials)
.await
.expect("enable versioning should succeed");
container::set_container_acl(&swift_account, container, Some(".r:*"), Some("AUTH_other"), &credentials)
container::set_container_acl(&swift_account, container, Some(".r:*,AUTH_other"), Some("AUTH_other"), &credentials)
.await
.expect("set container ACL should succeed");
@@ -248,7 +352,7 @@ async fn tag_writers_preserve_each_others_state(env: &TestECStoreEnv) {
// All three writers' state coexists after a disk-truth reload.
let meta = persisted_container_metadata(&bucket).await;
assert_eq!(meta.get("color").map(String::as_str), Some("blue"));
assert_eq!(meta.get("color").map(String::as_str), Some("blue,green"));
assert_eq!(
container::get_versions_location(&swift_account, container, &credentials)
.await
@@ -259,7 +363,7 @@ async fn tag_writers_preserve_each_others_state(env: &TestECStoreEnv) {
let acl = container::get_container_acl(&swift_account, container, &credentials)
.await
.expect("container ACL should load");
assert!(!acl.read.is_empty(), "read ACL must survive the reload");
assert!(acl.check_read_access(None, None, None), "public read ACL must survive the reload");
assert!(!acl.write.is_empty(), "write ACL must survive the reload");
// Disabling versioning drops only the versioning tag.
@@ -279,7 +383,7 @@ async fn tag_writers_preserve_each_others_state(env: &TestECStoreEnv) {
let meta = persisted_container_metadata(&bucket).await;
assert_eq!(
meta.get("color").map(String::as_str),
Some("blue"),
Some("blue,green"),
"disable_versioning must not disturb container metadata"
);
let acl = container::get_container_acl(&swift_account, container, &credentials)
+145
View File
@@ -1728,4 +1728,149 @@ mod tests {
let xxh3_multipart = ChecksumType(ChecksumType::XXHASH3.0 | ChecksumType::MULTIPART.0);
assert!(!xxh3_multipart.is_s3s_typed());
}
#[test]
fn checksum_matches_covers_positive_and_negative_branches() {
let payload = b"durability check payload";
let valid = Checksum::new_from_data(ChecksumType::CRC32C, payload).expect("valid crc32c");
// Positive match with no multipart count.
assert!(valid.matches(payload, 0).is_ok());
// Empty encoded checksum is a no-op success.
let mut empty_encoded = valid.clone();
empty_encoded.encoded.clear();
assert!(empty_encoded.matches(b"anything", 0).is_ok());
// Non-empty encoded checksum with unset/no-hasher type fails closed.
let no_hasher = Checksum {
checksum_type: ChecksumType::NONE,
encoded: valid.encoded.clone(),
raw: valid.raw.clone(),
want_parts: 0,
};
let err = no_hasher.matches(payload, 0).expect_err("NONE hasher must fail");
let err_msg = err.to_string();
assert!(err_msg.contains("no hasher available"), "unexpected error: {err_msg}");
// Multipart want_parts mismatch fails with formatted "<encoded>-<parts>" want/got.
let mp = Checksum::new_with_type(ChecksumType::CRC32C, &format!("{}-3", valid.encoded)).expect("multipart checksum");
assert_eq!(mp.want_parts, 3);
let err = mp.matches(payload, 2).expect_err("parts count mismatch must fail");
let err_msg = err.to_string();
assert!(err_msg.contains(&format!("{}-3", valid.encoded)), "missing want parts: {err_msg}");
assert!(err_msg.contains(&format!("{}-2", valid.encoded)), "missing got parts: {err_msg}");
assert!(mp.matches(payload, 3).is_ok(), "matching parts count should succeed");
// Payload digest mismatch fails closed.
let err = valid
.matches(b"corrupted payload", 0)
.expect_err("corrupted payload must fail");
assert_eq!(err.want, valid.encoded);
assert_ne!(err.got, valid.encoded);
}
#[test]
fn checksum_constructors_add_part_and_header_negative_branches() {
use super::get_content_checksum;
use http::{HeaderMap, HeaderValue};
// new_with_type negative branches
assert!(Checksum::new_with_type(ChecksumType::NONE, "AAAA").is_none());
assert!(Checksum::new_with_type(ChecksumType::CRC32, "AAAA-1-2").is_none());
assert!(Checksum::new_with_type(ChecksumType::CRC32, "AAAA-notanint").is_none());
assert!(Checksum::new_with_type(ChecksumType::CRC32, "!!!not_base64!!!").is_none());
let short_b64 = base64_simd::STANDARD.encode_to_string([1u8, 2u8]);
assert!(Checksum::new_with_type(ChecksumType::CRC32, &short_b64).is_none());
// as_map returns None when raw length does not match raw_byte_len()
let mut invalid_raw = Checksum::new_from_data(ChecksumType::CRC32, b"hello").expect("crc32");
assert!(invalid_raw.as_map().is_some());
invalid_raw.raw.pop();
assert!(invalid_raw.as_map().is_none());
// obj_type and from_header coverage
assert_eq!(ChecksumType::CRC64_NVME.obj_type(), "FULL_OBJECT");
assert_eq!(ChecksumType(ChecksumType::SHA256.0 | ChecksumType::MULTIPART.0).obj_type(), "COMPOSITE");
assert_eq!(ChecksumType::NONE.obj_type(), "");
let mut headers = HeaderMap::new();
assert_eq!(ChecksumType::from_header(&headers), ChecksumType::NONE);
headers.insert("x-amz-checksum-algorithm", HeaderValue::from_static("CRC32C"));
headers.insert("x-amz-checksum-type", HeaderValue::from_static("FULL_OBJECT"));
let parsed = ChecksumType::from_header(&headers);
assert!(parsed.is(ChecksumType::CRC32C) && parsed.full_object_requested());
// add_part positive and negative branches
let part_a = b"hello ";
let part_b = b"world!";
let part_b_len = part_b.len() as i64;
for t in [ChecksumType::CRC32, ChecksumType::CRC32C, ChecksumType::CRC64_NVME] {
let mut acc = Checksum::new_from_data(t, part_a).expect("part a");
let b_sum = Checksum::new_from_data(t, part_b).expect("part b");
let full = Checksum::new_from_data(t, b"hello world!").expect("full");
// size == 0 is a no-op
acc.add_part(&b_sum, 0).expect("zero size no-op");
assert_eq!(acc.encoded, Checksum::new_from_data(t, part_a).unwrap().encoded);
// size > 0 combines CRCs into full-object digest
acc.add_part(&b_sum, part_b_len).expect("combine");
assert_eq!(acc.encoded, full.encoded, "combined CRC mismatch for {t:?}");
}
// add_part rejects unmergeable algorithm, mismatched types, and invalid raw lengths
let mut sha_a = Checksum::new_from_data(ChecksumType::SHA256, part_a).expect("sha256");
let sha_b = Checksum::new_from_data(ChecksumType::SHA256, part_b).expect("sha256");
let err = sha_a
.add_part(&sha_b, part_b_len)
.expect_err("unmergeable algorithm must fail");
assert!(err.contains("checksum type cannot be merged"), "unexpected error: {err}");
let mut crc32_a = Checksum::new_from_data(ChecksumType::CRC32, part_a).expect("crc32");
let crc32c_b = Checksum::new_from_data(ChecksumType::CRC32C, part_b).expect("crc32c");
let err = crc32_a
.add_part(&crc32c_b, part_b_len)
.expect_err("mismatched types must fail");
assert!(err.contains("checksum type does not match"), "unexpected error: {err}");
let mut bad_base = crc32_a.clone();
bad_base.raw.truncate(1);
let err = bad_base
.add_part(&crc32_a, part_b_len)
.expect_err("invalid base raw len must fail");
assert!(err.contains("invalid base checksum"), "unexpected error: {err}");
let mut bad_part = crc32_a.clone();
bad_part.raw.truncate(1);
let err = crc32_a
.add_part(&bad_part, part_b_len)
.expect_err("invalid part raw len must fail");
assert!(err.contains("invalid part checksum"), "unexpected error: {err}");
// get_content_checksum negative branches
let mut dup_trailer = HeaderMap::new();
dup_trailer.insert("x-amz-trailer", HeaderValue::from_static("x-amz-checksum-crc32, x-amz-checksum-crc32c"));
let err = get_content_checksum(&dup_trailer).expect_err("duplicate trailer checksums must fail");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("Invalid checksum"), "unexpected error: {err}");
let mut unmergeable_full_trailer = HeaderMap::new();
unmergeable_full_trailer.insert("x-amz-trailer", HeaderValue::from_static("x-amz-checksum-sha256"));
unmergeable_full_trailer.insert("x-amz-checksum-type", HeaderValue::from_static("FULL_OBJECT"));
let err = get_content_checksum(&unmergeable_full_trailer).expect_err("FULL_OBJECT SHA256 trailer must fail");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("Invalid checksum"), "unexpected error: {err}");
let mut invalid_type_header = HeaderMap::new();
invalid_type_header.insert("x-amz-trailer", HeaderValue::from_static("x-amz-checksum-crc32"));
invalid_type_header.insert("x-amz-checksum-type", HeaderValue::from_static("BOGUS_TYPE"));
let err = get_content_checksum(&invalid_type_header).expect_err("BOGUS_TYPE checksum-type must fail");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("Invalid checksum"), "unexpected error: {err}");
let mut malformed_direct = HeaderMap::new();
malformed_direct.insert("x-amz-checksum-crc32", HeaderValue::from_static("not_valid_base64!"));
let err = get_content_checksum(&malformed_direct).expect_err("malformed base64 header must fail");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("invalid checksum value"), "unexpected error: {err}");
}
}
+100
View File
@@ -1269,4 +1269,104 @@ mod trailer_tests {
let err = read_all(&mut outer).await.expect_err("mismatch through a wrapper must fail");
assert_checksum_mismatch(&err);
}
#[tokio::test]
async fn wrapping_hash_reader_rejects_already_read_conflicting_checksum_and_size() {
let data = b"wrapping validation payload";
let size = data.len() as i64;
// 1. Wrapping a HashReader that has already been partially read must fail.
let mut inner = HashReader::from_stream(Cursor::new(data.to_vec()), size, size, None, None, false).expect("inner");
let mut one = [0u8; 1];
inner.read_exact(&mut one).await.expect("read one byte");
assert_eq!(inner.bytes_read(), 1);
let err = HashReader::new(Box::new(inner), size, size, None, None, false)
.err()
.expect("wrapping after read must fail");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("already read"));
// 2. Wrapping a HashReader with a conflicting MD5 checksum must fail.
let inner = HashReader::from_stream(
Cursor::new(data.to_vec()),
size,
size,
Some("11111111111111111111111111111111".to_string()),
None,
false,
)
.expect("inner with md5");
let err = HashReader::new(
Box::new(inner),
size,
size,
Some("22222222222222222222222222222222".to_string()),
None,
false,
)
.err()
.expect("conflicting md5 checksum must fail");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("checksum mismatch"));
// 3. Wrapping a HashReader with a conflicting positive size must fail.
let inner = HashReader::from_stream(Cursor::new(data.to_vec()), size, size, None, None, false).expect("inner");
let err = HashReader::new(Box::new(inner), size + 10, size, None, None, false)
.err()
.expect("conflicting positive size must fail");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("size mismatch"));
// 4. Wrapping an unbounded (-1) HashReader propagates newly discovered positive size and actual_size.
let inner = HashReader::from_stream(Cursor::new(data.to_vec()), -1, -1, None, None, false).expect("unbounded inner");
let outer = HashReader::new(Box::new(inner), size, size, Some("abcd".to_string()), None, false).expect("outer");
assert_eq!(outer.size(), size);
assert_eq!(outer.actual_size(), size);
}
#[tokio::test]
async fn hash_reader_checksum_helpers_reject_invalid_inputs() {
let data = b"checksum helper validation";
let size = data.len() as i64;
let mut reader = HashReader::from_stream(Cursor::new(data.to_vec()), size, size, None, None, false).expect("reader");
// add_calculated_checksum rejects both NONE and INVALID checksum types.
let err = reader
.add_calculated_checksum(ChecksumType::NONE)
.expect_err("NONE checksum type must fail");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
let err = reader
.add_calculated_checksum(ChecksumType::INVALID)
.expect_err("INVALID checksum type must fail");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
// add_non_trailing_checksum handles None as a no-op and rejects a Checksum with no hasher.
reader.add_non_trailing_checksum(None, false).expect("None is no-op");
let invalid_checksum = Checksum {
checksum_type: ChecksumType::INVALID,
encoded: "AAAA".to_string(),
raw: vec![1, 2, 3, 4],
want_parts: 0,
};
let err = reader
.add_non_trailing_checksum(Some(invalid_checksum), false)
.expect_err("invalid checksum_type must fail");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
// add_checksum_no_trailer attaches a direct header checksum without trailer reader.
let mut reader =
HashReader::from_stream(Cursor::new(data.to_vec()), size, size, None, None, false).expect("fresh reader");
assert!(reader.checksum().is_none(), "HashReader::checksum() is None when no content_hash is set");
let crc32_val = encoded(ChecksumType::CRC32C, data);
let mut headers = HeaderMap::new();
headers.insert("x-amz-checksum-crc32c", HeaderValue::from_str(&crc32_val).expect("header"));
reader
.add_checksum_no_trailer(&headers, false)
.expect("direct header checksum");
assert!(
reader.checksum().is_some(),
"HashReader::checksum() returns Some after add_checksum_no_trailer"
);
assert_eq!(read_all(&mut reader).await.expect("verified"), data);
}
}
+2
View File
@@ -73,6 +73,8 @@ rustfs-storage-api.workspace = true
futures = { workspace = true }
futures-core = { workspace = true }
flate2.workspace = true
brotli.workspace = true
lz4.workspace = true
http.workspace = true
memchr.workspace = true
s3s = { workspace = true, features = ["minio"] }
+32
View File
@@ -27,6 +27,7 @@ mod csv_input;
mod input_stream;
mod metrics;
pub mod object_store;
pub mod parquet_budget;
pub mod query;
pub mod server;
mod storage_api;
@@ -218,6 +219,14 @@ fn classify_select_error_source(err: &(dyn StdError + 'static)) -> Option<Select
if let Some(error) = err.downcast_ref::<SelectError>() {
return Some(error.clone());
}
if let Some(error) = err.downcast_ref::<parquet_budget::ParquetBudgetError>() {
return Some(match error {
parquet_budget::ParquetBudgetError::Exceeded => SelectError::ResourceExhausted,
parquet_budget::ParquetBudgetError::Invalid(_) | parquet_budget::ParquetBudgetError::Codec(_) => {
SelectError::ParquetParsingError
}
});
}
if let Some(error) = err.downcast_ref::<object_store::SelectObjectStoreError>() {
return Some(error.select_error());
}
@@ -332,9 +341,32 @@ impl Display for ResolvedTable {
#[cfg(test)]
mod tests {
use super::*;
use datafusion::common::DataFusionError;
use datafusion::sql::sqlparser::parser::ParserError;
#[test]
fn parquet_admission_errors_preserve_select_error_codes() {
for (source, expected) in [
(parquet_budget::ParquetBudgetError::Exceeded, SelectError::ResourceExhausted),
(
parquet_budget::ParquetBudgetError::Invalid("malformed footer".into()),
SelectError::ParquetParsingError,
),
(
parquet_budget::ParquetBudgetError::Codec(std::io::Error::from(std::io::ErrorKind::InvalidData)),
SelectError::ParquetParsingError,
),
] {
let error = QueryError::from(DataFusionError::External(Box::new(source)));
assert_eq!(error.select_error(), expected);
}
let nested = QueryError::from(DataFusionError::ParquetError(Box::new(ParquetError::External(Box::new(
parquet_budget::ParquetBudgetError::Exceeded,
)))));
assert_eq!(nested.select_error(), SelectError::ResourceExhausted);
}
#[test]
fn test_query_error_display() {
let err = QueryError::NotImplemented {
File diff suppressed because it is too large Load Diff
@@ -424,7 +424,8 @@ impl SimpleQueryDispatcher {
tracker: &QueryExecutionTracker,
) -> QueryResult<MetadataProvider> {
if self.input.request.input_serialization.parquet.is_some() {
let provider = ParquetSelectTable::try_new(session.inner(), self.input.as_ref()).await?;
let query_guard = tracker.query_guard().ok_or(QueryError::Cancel)?;
let provider = ParquetSelectTable::try_new(session.inner(), self.input.as_ref(), Some(query_guard)).await?;
let current_session_table_provider = self.build_table_handle_provider()?;
let metadata_provider =
MetadataProvider::new(provider, current_session_table_provider, self.func_manager.clone(), session.clone());
@@ -27,7 +27,10 @@ use datafusion::{
datasource::{
TableProvider,
listing::{ListingTableUrl, PartitionedFile},
physical_plan::{FileScanConfigBuilder, ParquetSource, parquet::ParquetAccessPlan},
physical_plan::{
FileScanConfigBuilder, ParquetSource,
parquet::{ParquetAccessPlan, ParquetFileReaderFactory},
},
source::DataSourceExec,
},
execution::{memory_pool::MemoryLimit, object_store::ObjectStoreUrl},
@@ -40,6 +43,7 @@ use datafusion::{
},
physical_plan::{
ExecutionPlan, SendableRecordBatchStream,
metrics::ExecutionPlanMetricsSet,
stream::RecordBatchStreamAdapter,
streaming::{PartitionStream, StreamingTableExec},
},
@@ -48,6 +52,8 @@ use futures::{FutureExt, TryFutureExt, TryStreamExt, future::BoxFuture, stream};
use rustfs_s3select_api::{
QueryResult,
object_store::{SelectScanRange, scan_range_from_bounds},
parquet_budget::{ParquetBudgetError, page_decode_budget, validate_footer},
query::session::QueryExecutionGuard,
};
use s3s::dto::SelectObjectContentInput;
@@ -59,6 +65,7 @@ pub struct ParquetSelectTable {
object_size: u64,
access_plan: Option<Arc<ParquetAccessPlan>>,
expanded_row_budget: usize,
reader_factory: Arc<BudgetParquetReaderFactory>,
}
impl fmt::Debug for ParquetSelectTable {
@@ -72,11 +79,30 @@ impl fmt::Debug for ParquetSelectTable {
}
}
#[derive(Debug)]
struct ObjectStoreParquetReader {
store: Arc<dyn ObjectStore>,
path: Path,
file_size: u64,
metadata_read_limit: usize,
metadata_decode_limit: usize,
metadata: Option<Arc<ParquetMetaData>>,
query_guard: Option<QueryExecutionGuard>,
}
async fn validate_in_worker<T: Send + 'static>(
query_guard: Option<QueryExecutionGuard>,
operation: impl FnOnce() -> Result<T, ParquetBudgetError> + Send + 'static,
) -> ParquetResult<T> {
tokio::task::spawn_blocking(move || {
// Cancellation cannot release admission while the decoder preflight
// still owns input buffers or codec allocations.
let _query_guard = query_guard;
operation()
})
.await
.map_err(|error| ParquetError::External(Box::new(error)))?
.map_err(|error| ParquetError::External(Box::new(error)))
}
impl AsyncFileReader for ObjectStoreParquetReader {
@@ -94,7 +120,16 @@ impl AsyncFileReader for ObjectStoreParquetReader {
if total.is_none_or(|total| total > self.metadata_read_limit as u64) {
return async { Err(ParquetError::External(Box::new(parquet_budget_error()))) }.boxed();
}
async move { self.store.get_ranges(&self.path, &ranges).await.map_err(parquet_store_error) }.boxed()
async move {
// Read requested ranges directly so sparse offsets cannot amplify
// a small request into a large coalesced allocation.
let mut bytes = Vec::with_capacity(ranges.len());
for range in ranges {
bytes.push(self.store.get_range(&self.path, range).await.map_err(parquet_store_error)?);
}
Ok(bytes)
}
.boxed()
}
fn get_metadata<'a>(
@@ -102,6 +137,27 @@ impl AsyncFileReader for ObjectStoreParquetReader {
options: Option<&'a ArrowReaderOptions>,
) -> BoxFuture<'a, ParquetResult<Arc<ParquetMetaData>>> {
async move {
if let Some(metadata) = self.metadata.as_ref() {
return Ok(Arc::clone(metadata));
}
let tail_start = self
.file_size
.checked_sub(8)
.ok_or_else(|| ParquetError::EOF("Parquet footer is truncated".into()))?;
let tail = self.get_bytes(tail_start..self.file_size).await?;
let tail: &[u8; 8] = tail
.as_ref()
.try_into()
.map_err(|_| ParquetError::EOF("Parquet footer is truncated".into()))?;
let footer = datafusion::parquet::file::metadata::FooterTail::try_new(tail)?;
let length =
u64::try_from(footer.metadata_length()).map_err(|_| ParquetError::External(Box::new(parquet_budget_error())))?;
let start = tail_start
.checked_sub(length)
.ok_or_else(|| ParquetError::EOF("Parquet metadata is truncated".into()))?;
let bytes = self.get_bytes(start..tail_start).await?;
let limit = self.metadata_decode_limit;
validate_in_worker(self.query_guard.clone(), move || validate_footer(&bytes, limit)).await?;
let metadata_options = options.map(|options| options.metadata_options().clone());
let mut metadata_reader = ParquetMetaDataReader::new().with_metadata_options(metadata_options);
@@ -119,8 +175,84 @@ impl AsyncFileReader for ObjectStoreParquetReader {
}
}
#[derive(Debug)]
struct BudgetParquetReaderFactory {
store: Arc<dyn ObjectStore>,
path: Path,
file_size: u64,
metadata: Arc<ParquetMetaData>,
read_limit: usize,
}
impl ParquetFileReaderFactory for BudgetParquetReaderFactory {
fn create_reader(
&self,
_partition_index: usize,
file: PartitionedFile,
_metadata_size_hint: Option<usize>,
_metrics: &ExecutionPlanMetricsSet,
) -> DFResult<Box<dyn AsyncFileReader + Send>> {
if file.object_meta.location != self.path || file.object_meta.size != self.file_size {
return Err(DataFusionError::Execution("Parquet reader input differs from its admitted object".into()));
}
Ok(Box::new(ObjectStoreParquetReader {
store: Arc::clone(&self.store),
path: self.path.clone(),
file_size: self.file_size,
metadata_read_limit: self.read_limit,
metadata_decode_limit: self.read_limit,
metadata: Some(Arc::clone(&self.metadata)),
query_guard: None,
}))
}
}
async fn validate_parquet_pages(
store: &Arc<dyn ObjectStore>,
path: &Path,
file_size: u64,
metadata: &ParquetMetaData,
plan: Option<&ParquetAccessPlan>,
limit: usize,
query_guard: &Option<QueryExecutionGuard>,
) -> DFResult<()> {
for (index, group) in metadata.row_groups().iter().enumerate() {
if plan.is_some_and(|plan| !plan.should_scan(index)) {
continue;
}
let mut resident = 0usize;
for column in group.columns() {
let start = u64::try_from(column.dictionary_page_offset().unwrap_or(column.data_page_offset()))
.map_err(|_| parquet_budget_error())?;
let length = u64::try_from(column.compressed_size()).map_err(|_| parquet_budget_error())?;
let end = start
.checked_add(length)
.filter(|end| *end <= file_size)
.ok_or_else(parquet_budget_error)?;
if length > u64::try_from(limit).map_err(|_| parquet_budget_error())? {
return Err(parquet_budget_error());
}
let bytes = store.get_range(path, start..end).await?;
let declared = usize::try_from(column.uncompressed_size()).map_err(|_| parquet_budget_error())?;
let codec = column.compression();
let required = validate_in_worker(query_guard.clone(), move || page_decode_budget(&bytes, declared, limit, codec))
.await
.map_err(DataFusionError::from)?;
resident = resident
.checked_add(required)
.filter(|total| *total <= limit)
.ok_or_else(parquet_budget_error)?;
}
}
Ok(())
}
impl ParquetSelectTable {
pub async fn try_new(state: &dyn Session, input: &SelectObjectContentInput) -> QueryResult<Arc<dyn TableProvider>> {
pub async fn try_new(
state: &dyn Session,
input: &SelectObjectContentInput,
query_guard: Option<QueryExecutionGuard>,
) -> QueryResult<Arc<dyn TableProvider>> {
let table_path = ListingTableUrl::parse(format!("s3://{}/{}", input.bucket, input.key))?;
let object_store_url = table_path.object_store();
let object_location = Path::from(input.key.clone());
@@ -136,9 +268,12 @@ impl ParquetSelectTable {
};
let reader = ObjectStoreParquetReader {
store: Arc::clone(&store),
path: object_location,
path: object_location.clone(),
file_size: object_meta.size,
metadata_read_limit: (memory_budget / 16).max(1),
metadata_decode_limit: (memory_budget / 4).max(1),
metadata: None,
query_guard: query_guard.clone(),
};
let builder = ParquetRecordBatchStreamBuilder::new(reader)
.await
@@ -153,11 +288,26 @@ impl ParquetSelectTable {
builder.schema().metadata().clone(),
));
let metadata = Arc::clone(builder.metadata());
// This is admission for valid Parquet declarations, not a replacement
// for limits inside the footer parser or native compression codecs.
let expanded_row_budget = (memory_budget / 16).max(1);
let access_plan = parquet_access_plan(input, object_meta.size, metadata.as_ref())?;
admit_parquet_decode(metadata.as_ref(), access_plan.as_deref(), memory_budget / 4)?;
validate_parquet_pages(
&store,
&object_location,
object_meta.size,
&metadata,
access_plan.as_deref(),
memory_budget / 4,
&query_guard,
)
.await?;
let reader_factory = Arc::new(BudgetParquetReaderFactory {
store,
path: object_location,
file_size: object_meta.size,
metadata,
read_limit: (memory_budget / 4).max(1),
});
Ok(Arc::new(Self {
schema,
@@ -166,6 +316,7 @@ impl ParquetSelectTable {
object_size: object_meta.size,
access_plan,
expanded_row_budget,
reader_factory,
}))
}
@@ -202,7 +353,12 @@ impl TableProvider for ParquetSelectTable {
));
}
let scan_limit = if filters.is_empty() { limit } else { None };
let file_source = Arc::new(ParquetSource::new(Arc::clone(&self.schema)));
let file_source = Arc::new(
ParquetSource::new(Arc::clone(&self.schema))
.with_parquet_file_reader_factory(self.reader_factory.clone())
.with_enable_page_index(false)
.with_bloom_filter_on_read(false),
);
let config = FileScanConfigBuilder::new(self.object_store_url.clone(), file_source)
.with_file(self.partitioned_file())
.with_projection_indices(projection.cloned())?
@@ -439,6 +595,39 @@ pub(super) fn dictionary_parquet_for_test(value_bytes: usize, rows: usize) -> Ve
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn parquet_cancelled_preflight_retains_admission_until_worker_finishes() {
let admission = Arc::new(tokio::sync::Semaphore::new(1));
let permit = Arc::clone(&admission).acquire_owned().await.expect("initial admission");
let (started_tx, started_rx) = tokio::sync::oneshot::channel();
let (release_tx, release_rx) = std::sync::mpsc::channel();
let task = tokio::spawn(validate_in_worker(Some(Arc::new(permit)), move || {
started_tx.send(()).expect("report running worker");
release_rx
.recv_timeout(std::time::Duration::from_secs(5))
.expect("worker release");
Ok(())
}));
tokio::time::timeout(std::time::Duration::from_secs(5), started_rx)
.await
.expect("worker starts before deadline")
.expect("worker started");
task.abort();
assert!(task.await.expect_err("cancelled request task").is_cancelled());
assert!(Arc::clone(&admission).try_acquire_owned().is_err(), "running worker still owns admission");
release_tx.send(()).expect("release worker");
tokio::time::timeout(std::time::Duration::from_secs(5), async {
loop {
if Arc::clone(&admission).try_acquire_owned().is_ok() {
break;
}
tokio::task::yield_now().await;
}
})
.await
.expect("worker releases admission after completion");
}
use datafusion::{
arrow::{
array::Int32Array,
@@ -519,7 +708,7 @@ mod tests {
let context = parquet_session(store);
let state = context.state();
let error = match ParquetSelectTable::try_new(&state, &parquet_input("missing.parquet")).await {
let error = match ParquetSelectTable::try_new(&state, &parquet_input("missing.parquet"), None).await {
Ok(_) => panic!("missing parquet object must fail"),
Err(error) => error,
};
@@ -538,7 +727,7 @@ mod tests {
let context = parquet_session(store);
let state = context.state();
let error = match ParquetSelectTable::try_new(&state, &parquet_input(object.as_ref())).await {
let error = match ParquetSelectTable::try_new(&state, &parquet_input(object.as_ref()), None).await {
Ok(_) => panic!("corrupt parquet metadata must fail"),
Err(error) => error,
};
@@ -581,7 +770,7 @@ mod tests {
.put(&Path::from("dictionary.parquet"), Bytes::from(bytes).into())
.await
.expect("fixture");
let table = ParquetSelectTable::try_new(&context.state(), &parquet_input("dictionary.parquet"))
let table = ParquetSelectTable::try_new(&context.state(), &parquet_input("dictionary.parquet"), None)
.await
.expect("valid dictionary metadata");
context.register_table("source", table).expect("table");
@@ -752,7 +941,7 @@ mod tests {
start: Some(starts[0]),
end: Some(starts[0]),
});
let table = ParquetSelectTable::try_new(&context.state(), &input)
let table = ParquetSelectTable::try_new(&context.state(), &input, None)
.await
.expect("an unselected large row group must not reject the small range");
context.register_table("source", table).expect("table");
@@ -770,7 +959,7 @@ mod tests {
start: Some(starts[1]),
end: Some(starts[1]),
});
let error = match ParquetSelectTable::try_new(&context.state(), &input).await {
let error = match ParquetSelectTable::try_new(&context.state(), &input, None).await {
Ok(_) => panic!("the selected oversized row group must still be rejected before decoding"),
Err(error) => error,
};
@@ -806,13 +995,117 @@ mod tests {
.put(&Path::from("large.parquet"), Bytes::from(bytes).into())
.await
.expect("fixture");
let error = match ParquetSelectTable::try_new(&context.state(), &parquet_input("large.parquet")).await {
let error = match ParquetSelectTable::try_new(&context.state(), &parquet_input("large.parquet"), None).await {
Ok(_) => panic!("valid declared decoded size must be admitted before page decode"),
Err(error) => error,
};
assert!(error.to_string().contains("query memory budget"));
}
#[tokio::test]
async fn parquet_footer_collection_budget_is_checked_before_metadata_decode() {
// A root schema with no leaves and 50 empty, valid row groups. The
// encoded footer fits the read budget; its reservations do not.
let mut footer = vec![0x15, 2, 0x19, 0x1c, 0x48, 4];
footer.extend(b"root");
footer.extend([0x15, 0, 0, 0x16, 0, 0x19, 0xfc, 50]);
for _ in 0..50 {
footer.extend([0x19, 0x0c, 0x16, 0, 0x16, 0, 0]);
}
footer.push(0);
let decoded = ParquetMetaDataReader::decode_metadata(&footer).expect("valid compact footer fixture");
assert_eq!(decoded.num_row_groups(), 50);
let mut bytes = b"PAR1".to_vec();
bytes.extend(&footer);
bytes.extend(u32::try_from(footer.len()).expect("small footer").to_le_bytes());
bytes.extend(b"PAR1");
let store = Arc::new(InMemory::new());
let context = parquet_session_with_memory(store.clone(), 8192);
store
.put(&Path::from("footer.parquet"), Bytes::from(bytes).into())
.await
.expect("fixture");
let error = match ParquetSelectTable::try_new(&context.state(), &parquet_input("footer.parquet"), None).await {
Ok(_) => panic!("footer collection reservations exceed the query budget"),
Err(error) => error,
};
assert_eq!(error.select_error(), SelectError::ResourceExhausted);
}
#[tokio::test]
async fn parquet_forged_footer_cannot_hide_oversized_compressed_pages() {
use datafusion::parquet::file::metadata::ParquetMetaDataWriter;
let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Utf8, false)]));
let batch = RecordBatch::try_new(
Arc::clone(&schema),
vec![Arc::new(datafusion::arrow::array::StringArray::from(vec![
"x".repeat(32 * 1024 * 1024),
]))],
)
.expect("large compressible value");
let properties = datafusion::parquet::file::properties::WriterProperties::builder()
.set_dictionary_page_size_limit(64 * 1024 * 1024)
.set_compression(datafusion::parquet::basic::Compression::ZSTD(Default::default()))
.build();
let mut encoded = Vec::new();
let mut writer = ArrowWriter::try_new(&mut encoded, schema, Some(properties)).expect("writer");
writer.write(&batch).expect("compressed value");
writer.close().expect("footer");
let original = Bytes::from(encoded);
let builder = ParquetRecordBatchReaderBuilder::try_new(original.clone()).expect("valid compressed fixture");
let metadata = builder.metadata();
let groups = metadata
.row_groups()
.iter()
.map(|group| {
let columns = group
.columns()
.iter()
.cloned()
.map(|column| {
column
.into_builder()
.set_total_uncompressed_size(64)
.build()
.expect("forged column")
})
.collect();
group
.clone()
.into_builder()
.set_column_metadata(columns)
.build()
.expect("forged group")
})
.collect();
let forged = metadata.as_ref().clone().into_builder().set_row_groups(groups).build();
let footer_size = u32::from_le_bytes(
original[original.len() - 8..original.len() - 4]
.try_into()
.expect("footer size"),
);
let mut bytes = original[..original.len() - 8 - usize::try_from(footer_size).expect("footer length")].to_vec();
ParquetMetaDataWriter::new(&mut bytes, &forged)
.finish()
.expect("forged footer");
assert!(bytes.len() < 256 * 1024, "compressed input remains small");
admit_parquet_decode(&forged, None, 4 * 1024 * 1024).expect("footer-only admission misses the page size");
let store = Arc::new(InMemory::new());
let context = parquet_session_with_memory(store.clone(), 16 * 1024 * 1024);
store
.put(&Path::from("forged.parquet"), Bytes::from(bytes).into())
.await
.expect("fixture");
let result = ParquetSelectTable::try_new(&context.state(), &parquet_input("forged.parquet"), None).await;
let error = match result {
Ok(_) => panic!("actual query admission must reject oversized page buffers before decoding"),
Err(error) => error,
};
assert_eq!(error.select_error(), SelectError::ResourceExhausted);
}
#[tokio::test]
async fn parquet_nested_views_decode_through_the_actual_file_schema() {
use datafusion::arrow::array::{ListArray, StringArray};
+6 -1
View File
@@ -1823,7 +1823,12 @@ impl DataUsageCache {
return;
};
if top_e.children.len() > 250_000 {
// Wide bucket roots previously only flattened above a hardcoded 250k
// fanout, so caches with 10k–250k direct children (common with many
// top-level prefixes) never shrank and retained one entry per prefix
// after #8971's non-root compaction. Honor `limit` here so completed
// sweeps and emergency compaction actually bound root fanout (#8961).
if top_e.children.len() > limit {
self.reduce_children_of(&hash_path(&self.info.name), limit, true);
}
if self.cache.len() <= limit {
@@ -3223,6 +3223,200 @@ fn test_reduce_children_of_usize_underflow_saturates() {
assert!(cache.find("bucket/large/b").is_some());
}
#[test]
fn test_force_compact_flattens_wide_root_above_limit() {
// Regression for #8961: a bucket root with fanout between the ordinary
// compact limit and the old hardcoded 250_000 ceiling must shrink.
// Before the fix, force_compact kept one entry per prefix forever.
const PREFIXES: usize = 64;
const LIMIT: usize = 16;
const {
assert!(PREFIXES > LIMIT);
}
let root = hash_path("bucket");
let mut cache = DataUsageCache {
info: DataUsageCacheInfo {
name: "bucket".to_string(),
..Default::default()
},
..Default::default()
};
cache.replace_hashed(&root, &None, &DataUsageEntry::default());
for index in 0..PREFIXES {
let child = hash_path(&format!("bucket/prefix-{index:04}"));
cache.replace_hashed(
&child,
&Some(root.clone()),
&DataUsageEntry {
objects: 1,
size: 1,
compacted: true,
..Default::default()
},
);
}
assert_eq!(cache.find("bucket").map(|entry| entry.children.len()), Some(PREFIXES));
assert_eq!(cache.cache.len(), PREFIXES + 1);
cache.force_compact(LIMIT);
let root_entry = cache.find("bucket").expect("root retained");
assert!(root_entry.compacted, "wide root must flatten under force_compact");
assert!(root_entry.children.is_empty(), "flattened root must drop per-prefix children");
assert_eq!((root_entry.objects, root_entry.size), (PREFIXES, PREFIXES));
assert_eq!(cache.cache.len(), 1, "prefix leaves must be released after root flatten");
}
#[test]
fn test_force_compact_keeps_narrow_root_children() {
const PREFIXES: usize = 4;
const LIMIT: usize = 16;
const {
assert!(PREFIXES < LIMIT);
}
let root = hash_path("bucket");
let mut cache = DataUsageCache {
info: DataUsageCacheInfo {
name: "bucket".to_string(),
..Default::default()
},
..Default::default()
};
cache.replace_hashed(
&root,
&None,
&DataUsageEntry {
objects: PREFIXES,
..Default::default()
},
);
for index in 0..PREFIXES {
let child = hash_path(&format!("bucket/prefix-{index:04}"));
cache.replace_hashed(
&child,
&Some(root.clone()),
&DataUsageEntry {
objects: 1,
compacted: true,
..Default::default()
},
);
}
cache.force_compact(LIMIT);
let root_entry = cache.find("bucket").expect("root retained");
assert!(!root_entry.compacted);
assert_eq!(root_entry.children.len(), PREFIXES);
assert_eq!(cache.cache.len(), PREFIXES + 1);
}
#[test]
fn test_force_compact_interrupted_ceiling_retains_wide_root_children() {
// Interrupted sweeps pass the emergency FORCE ceiling into force_compact so
// a resume frontier under a 10k–250k-wide root stays addressable.
const PREFIXES: usize = 64;
const ORDINARY_LIMIT: usize = 16;
const INTERRUPTED_CEILING: usize = 250_000;
const {
assert!(PREFIXES > ORDINARY_LIMIT);
assert!(PREFIXES < INTERRUPTED_CEILING);
}
let root = hash_path("bucket");
let mut cache = DataUsageCache {
info: DataUsageCacheInfo {
name: "bucket".to_string(),
..Default::default()
},
..Default::default()
};
cache.replace_hashed(&root, &None, &DataUsageEntry::default());
for index in 0..PREFIXES {
let child = hash_path(&format!("bucket/prefix-{index:04}"));
cache.replace_hashed(
&child,
&Some(root.clone()),
&DataUsageEntry {
objects: 1,
size: 1,
compacted: true,
..Default::default()
},
);
}
cache.force_compact(INTERRUPTED_CEILING);
let root_entry = cache.find("bucket").expect("root retained");
assert!(!root_entry.compacted, "interrupted ceiling must not flatten a 64-wide root");
assert_eq!(root_entry.children.len(), PREFIXES);
assert_eq!(cache.cache.len(), PREFIXES + 1);
// Sanity: the ordinary completion limit would have flattened this layout.
let mut completed = cache.clone();
completed.force_compact(ORDINARY_LIMIT);
assert!(
completed.find("bucket").is_some_and(|entry| entry.compacted),
"ordinary limit still flattens the same wide root after completion"
);
}
#[test]
fn test_force_compact_interrupted_ceiling_seals_reloaded_frontier() {
// The interrupted call runs before seal_scan_frontier. Flattening at the
// ordinary limit deletes the child that frontier names, so a saved cache
// reloads with no resume position and the next cycle replays finished prefixes.
const PREFIXES: usize = 64;
const ORDINARY_LIMIT: usize = 16;
const INTERRUPTED_CEILING: usize = 250_000;
const {
assert!(PREFIXES > ORDINARY_LIMIT);
assert!(PREFIXES < INTERRUPTED_CEILING);
}
let frontier = format!("bucket/prefix-{:04}", PREFIXES / 2);
let mut cache = DataUsageCache::default();
cache.prepare_bucket_checkpoint("bucket", 8, 1, DataUsageCacheSource::new(0, 0), TEST_PLAN_DIGEST, valid_scan_identity());
let root = hash_path("bucket");
cache.replace_hashed(&root, &None, &DataUsageEntry::default());
for index in 0..PREFIXES {
let child = hash_path(&format!("bucket/prefix-{index:04}"));
cache.replace_hashed(
&child,
&Some(root.clone()),
&DataUsageEntry {
objects: 1,
size: 1,
compacted: true,
..Default::default()
},
);
}
let mut interrupted = cache.clone();
interrupted.force_compact(INTERRUPTED_CEILING);
assert!(interrupted.find(&frontier).is_some(), "interrupted ceiling must keep the frontier child");
interrupted
.seal_scan_frontier(Some(&frontier))
.expect("seal frontier after interrupted compaction");
let encoded = rmp_serde::to_vec(&interrupted).expect("partial cache should encode");
let reloaded: DataUsageCache = rmp_serde::from_slice(&encoded).expect("partial cache should reload");
assert_eq!(reloaded.validated_scan_frontier(), Some(frontier.as_str()));
assert!(reloaded.find(&frontier).is_some(), "reloaded frontier child must stay addressable");
cache.force_compact(ORDINARY_LIMIT);
assert!(cache.find(&frontier).is_none(), "ordinary limit removes the frontier child");
cache
.seal_scan_frontier(Some(&frontier))
.expect("seal after a completed flatten");
assert!(
cache.validated_scan_frontier().is_none(),
"a flattened root cannot seal the removed frontier child"
);
}
#[tokio::test]
async fn test_retry_save_op_times_out_and_returns_error_after_retries() {
let attempts = Arc::new(AtomicUsize::new(0));

Some files were not shown because too many files have changed in this diff Show More